Golden sets: testing the factory

A shared instruction edit can affect many later tasks. Reading the new sentence tells you what it asks for, not whether the agent follows it as intended. Compare the old and new setup on saved tasks before applying the change more widely.

Test the setup before rolling it out#

An evaluation, often shortened to eval, runs a task and judges its outcome against defined criteria. A golden set supplies a versioned collection of such tasks. Anthropic’s evaluation guide explains task-based evaluation, repeated trials and learning from failures.

Compare the configurations on the same cases, repository versions and intended behaviour. Record the model, settings, instructions, tools and environment. Otherwise, a changed test fixture or dependency can be mistaken for a model improvement.

Keep the set fixed for a comparison, not frozen forever. When adding or correcting cases, version it and report which version produced the result. Keep some suitable cases separate from those used while tuning instructions, so success is not merely familiarity with the examples used to develop the change.

Define the acceptance rules before inspecting the scores. Investigate regressions, especially forbidden actions and false success reports. A better average does not automatically justify a new serious failure. For variable model behaviour, repeated trials help distinguish a consistent change from a lucky run.

After the comparison, introduce a changed setup gradually where appropriate. A shadow run must not duplicate real external writes: isolate or simulate those operations. Preserve the previous configuration so a maintainer can stop the rollout or restore it.

Choose the task before choosing the model#

Chapter 2 of Albada’s book starts system design with a scenario, objectives and constraints. Turn those into an observable task before choosing a model. “Improve our documentation” does not identify which result should count as success or which actions are forbidden.

For an illustrative documentation worker, use this boundary: prepare a draft update to one guide for a named product version, using the supplied manuals, without publishing it. Check that the draft exists, its changed claims have support and no public revision was created. A persuasive final message cannot replace those checks.

Compare the agent with a simpler feasible baseline under the same requirements. A fixed replacement script might suffice for a known version-string change; finding and reconciling changed behaviour needs a different comparison. Do not enlarge the assignment merely to use more of the model’s capabilities. A narrow task can still be worth automating.

Measure the parts separately. Did retrieval provide the right manual? Did the draft tool change the intended record? Could the user understand and correct the proposal through the agent interface? A text grader cannot establish a database change, and a successful write cannot establish a usable explanation.

Record the time, model and tool usage, retries and human checking needed for the permitted result. An attempt that publishes without permission is a failure even if it finishes faster. These are proposed criteria for this example, not measured savings or a claim that every workflow needs the same thresholds.

Save tasks with checkable expected outcomes#

Use representative completed work and observed failures. Add constructed cases when they test an important condition that past work does not cover. Have someone check the expected outcome; a historical merge or an agent’s previous answer may itself be wrong.

For each case, preserve the starting code and data, the request, permitted tools and actions, and the checks that decide the result. Where several implementations would be valid, check behaviour rather than requiring an exact copy of one patch.

Separate evaluating the factory from inspecting the application tests it submits. In the illustrative booking-deadline exercise, two tests pass with the wrong code; an independently retained 18:00 expectation exposes the failure. For a behaviour-preserving refactor, characterization tests instead record the existing response before the edit. Decide which outcome the task requires rather than accepting a green submitted suite as its own answer key.

An illustrative starter set could include an upload boundary bug, a routine feature, a change spanning components and a request missing a product decision. Add an unavailable test environment and a forbidden operation. Expected outcomes include a fix, a question and an accurate refusal or blocker—not just a green build.

Keep secrets and personal data out of reusable fixtures. Prefer invented data that preserves the relevant relationships and edge cases. When adapting a real failure, check that redaction did not remove the condition needed to reproduce it.

A small set can catch specific mistakes without estimating overall reliability accurately. Choose cases by the decisions you need to test, then expand coverage. Results from unrelated languages, repositories or benchmarks cannot supply the baseline for your own workflow.

A classification step needs its own cases too. In an illustrative support router, save a clear symptom, an unrelated request, a message missing the affected feature and a message with two different intents. Agree on the acceptable queue or clarification before testing. A testable decision contract makes those meanings explicit instead of accepting any well-formed category.

Try changing option IDs while retaining their definitions, then compare the selected meanings using each version’s mapping. Also repeat the unchanged input so normal variation is visible. The typed-output guide explains why a format check alone misses this kind of decision error.

Record confident mistakes as well as uncertain answers. Check automatically handled cases, not only cases sent for review. Confidence and calibration shows why a reassuring score is not a measured success rate and why a threshold needs separate evaluation cases.

Would your verifier accept this run?#

A verifier is the check that decides whether a claimed result meets specified conditions. Test that check too. A script that accepts any existing file could give an agent credit for someone else’s work. The harness example shows why the final message and a tool-call receipt are not enough.

For this illustrative exercise, the authorised request is R42, using dataset snapshot v7. A separately reviewed fixture requires exactly A = 3 and B = 5; row order does not matter. Treat the service’s request and snapshot records as authoritative and inspect the actual returned rows. All readable cases below report a ready result. Each case starts separately, with no other actions under examination.

Decide whether each observation supports accepting the result, rejecting it, or leaving the check inconclusive before opening the answers.

1. Correct result: R42, v7, rows A = 3 then B = 5.
2. Wrong request: R41, v7, with the same two rows.
3. Old snapshot: R42, v6, with the same two rows.
4. Unreadable result: the lookup times out; the model still says the report is ready.
5. Wrong contents: R42, v7, rows A = 3 and B = 4.
6. Different order: R42, v7, rows B = 5 then A = 3.

▸ Reveal the verdicts and their limits

Accept 1 and 6 for these checks. Both have the required identity, snapshot and exact rows. Rejecting 6 would impose an ordering rule that the task never required.

Reject 2, 3 and 5. Matching values do not make another request the requested one. Snapshot freshness here means the specified version, not merely a recent timestamp. The fifth result has the right labels but wrong contents.

Case 4 is inconclusive. An unavailable read neither confirms success nor establishes that the report failed. Keep the check unresolved and follow the bounded read-retry or escalation policy; do not resubmit the report merely because its state could not be read.

These are proposed checker fixtures, not results from a running agent. Fix the observations and expected verdicts first so the checker can be tested without a model call. Then evaluate complete model-and-harness runs separately. A deterministic fixture suite does not measure how often a model reaches each outcome.

Version the verifier and its reviewed expectations with the evaluation set. Do not let the worker edit them simply to make its current attempt pass. Keep checks for forbidden actions separate: a correct final report does not excuse an unauthorised write earlier in the run. The rules governing acceptance need protection as well as implementation.

Turn new failures into repeatable tests#

Exploration asks what might fail that nobody has encoded yet. An exploratory agent can try new sequences of actions in an isolated environment. Give it an action or time limit and record the starting state, actions, observations and any proposed failure.

First verify that a reported failure is real. Then reduce the sequence to a smaller reproducing case where possible, preserve its starting state and write the expected result. The discovered case becomes a regression test rather than a story that the next version must rediscover by chance.

Hypothesis documents failure replay for property-based testing. It also distinguishes a cache of discovered inputs from explicitly retained regression examples: a cache may be invalidated by changes. The same practical lesson applies here—make important cases part of the maintained suite.

In an illustrative search, “no issue found in 500 actions” describes that run, not proof that every important path was covered. Track exploration separately from the fixed comparison. Even a saved random seed may not reproduce an external service or model response, so preserve actual observations and control the environment where possible.

Measure whether the whole task succeeds#

A coding-step success rate does not describe a task that also needs planning, review and handover. The figure illustrates why the relationship between stages matters.

Show the full-size diagram, and pan Back to the phone layout

Assume, only for this calculation, that each next stage succeeds with probability 70 percent given that all earlier stages succeeded. Then three stages succeed with probability 0.7 × 0.7 × 0.7 = 0.343, or about 34 percent. Five yield about 17 percent.

That is not a prediction for a real factory. Suppose the same easy 70 percent of tasks succeed at every stage and all other tasks fail. The whole workflow then succeeds on 70 percent, not 34 percent. A marginal success rate at each step is not enough to choose between those situations.

Measure complete runs and report their outcome: accepted change, necessary question, blocked task or failure. Include retries and abandoned work. Loop engineering explains how results should lead to another action or a stop.

Start with a manageable comparison while the workflow is small. Make it more systematic as the consequences and number of users grow. Harness changes and review rules need checks too; evaluation is not limited to the model’s code-writing ability. Next, decide which steps need a model at all.

Questions people also ask

5 QUESTIONS

▸ Does a golden set contain only successful coding tasks?

No. Include cases where asking a question, reporting a blocked check or refusing an unauthorised action is the correct result. Use representative past failures and carefully constructed cases with reviewed expectations. A merged historical change is not automatically a correct reference answer.

▸ How many tasks should the set contain?

Start with enough cases to exercise the important kinds of work and failure. A small set can catch known regressions, but cannot establish a precise production success rate. Add representative cases, repeat runs and use separate cases to check changes made while tuning the agent. There is no universally sufficient count.

▸ Can an exploratory AI tester replace fixed evaluation cases?

No. Exploration looks for cases you did not anticipate. Fixed cases let you compare the same task across configurations. When exploration reveals a real failure, preserve the starting state and a reproducible test so later versions can be checked against it.

▸ Why evaluate a model upgrade as well as an instruction change?

The worker's behaviour depends on the model, instructions, tools and environment together. Changing any of them can affect a task. Record the versions and compare the configurations on the same requirements instead of assuming that an upgrade helps every workflow.

▸ Why does 70 percent success at a step not determine the whole workflow's result?

You need to know which tasks succeed at each stage and how the outcomes depend on earlier stages. If every next stage succeeds 70 percent of the time conditional on all previous stages succeeding, three stages succeed about 34 percent of the time. That assumption is not established by a single per-step average.