Golden sets: testing the factory

There is a moment in every factory where somebody edits the shared instruction file, reads the edit, agrees with it, and ships it to the whole fleet. It is the widest-reaching untested change in the system, it happens weekly, and it is the most likely explanation for a week in which the agents are mysteriously worse.

A harness change is a code change#

The published practice is a golden set: a fixed collection of real work items that have already been completed, with outcomes known to be good, replayed against every change to the instructions, the reusable skills or the model. If the replay comes out worse, the change does not go live. That is the whole mechanism, and its value is entirely in the word fixed — two runs are comparable only when the input did not move between them.

Rollout is staged for the same reason a code change is. A changed instruction set runs in shadow on a handful of items before it applies to everything, so a regression the replay did not cover shows up on five items rather than five hundred. And the version of the harness in force gets recorded against every run, which is what makes a bad week attributable to something rather than to a general feeling that the agents have got worse.

A model upgrade goes through the identical gate, and this is the part teams skip. It feels like a dependency bump, and dependency bumps are routine. But everything you have ever measured is the model and the instructions together — swapping either changes the system, and the instructions were written against the behaviour of the model that was in place when somebody wrote them. Gating instruction edits while waving model versions through has the risk backwards.

What goes in the set#

Real completed work, not written scenarios. The commonly cited range is twenty to fifty items, but the count matters less than the spread — what the set has to cover is the ways your work actually differs from itself. A bug with a stack trace and a reproduction. A feature that follows an existing pattern. Something that touches several services at once. Something with an ambiguity in it that should come back as a question rather than a change. That last one is easy to leave out and it is the one that catches an instruction edit which made the agents more confident rather than more correct.

Two properties are worth insisting on early, because retrofitting either is painful. The set is screened by a person, so nobody is measuring against an outcome that was itself wrong. And it does not contain personal data — real work items and real logs frequently do, and a fixture that gets replayed on every change is exactly the wrong place for it. Synthetic where synthetic will do, redacted where it will not.

One caution about importing published numbers as a starting point. The widely quoted agent benchmarks lean heavily on Python, and the multilingual work that followed reports materially lower resolve rates on other languages — Java second to Python and with a visible gap. Whatever the headline figure is, your baseline is the number your own set produces on your own code, and there is no way to obtain it except by running it.

One attempt is not the number you need#

Here is the arithmetic that reframes every success rate you will be quoted. Reliability is measured per attempt, and a factory is a chain of attempts.

Nothing in that figure is a worse agent than the one in the first bar. It is the same agent, asked to be right three times running, then five. Plan, build, review, verify is four steps before anything reaches a person, and at seven-in-ten per step the chain delivers about a quarter of the time. That is not a reason to abandon the design — it is the reason the repair loops exist, and the reason the number to improve is the per-step rate rather than the model.

It is also why reliability has to be measured across the whole set and across repeated runs rather than demonstrated once. A single successful pass tells you the ceiling. What governs how much work arrives usable is the rate, and a rate needs repetition to observe.

Which is the argument for building the gate before you need it, though not necessarily on day one. Early on, a handful of instruction changes can be reviewed by a person the way any small change is, and building a formal replay harness before a single workflow has proved itself is effort spent on the wrong thing. The moment it stops being optional is the moment more than one person is editing the harness, because that is when nobody can hold the whole of it in their head and the file starts contradicting itself in ways only a replay will find. The same discipline is what keeps the person at the merge gate reading a stream of proposals worth their attention, rather than a stream that quietly got worse in March.

A golden set only tells you about the steps that can vary. Which steps those are — and which have no business being a model call at all — is the line between the deterministic and the AI steps.