Measuring a factory honestly

Six months in, somebody will ask whether the factory worked. If every number you hold starts on the day you switched it on, there is no honest answer available — each figure is a level with nothing behind it to compare against. The work that makes the question answerable happens before anything visible exists, is unrewarding while you do it, and cannot be done afterwards, which is why almost nobody does it.

The baseline nobody collects#

Two kinds of goal get written down at the start of a programme like this, and neither survives a review. Semi-automatic by spring names a feeling and a date. Faster names a direction with no origin. Both become testable the moment there are numbers from before, and stay untestable forever if there are not, because last quarter’s cycle times cannot be reconstructed in April.

What to collect is dull and short. Volume today: work items and merged changes per month, split by kind — feature, defect, maintenance — and by team, since an average across teams hides the only signal you will care about later. Cycle times today, cut at the three places work actually waits: request to ready, ready to merged, merged to in production. Effort today, in hours per month, spent on review, on manual testing, on deployment and on triage — the number an agent is supposed to reduce, and the one nobody has. Quality today: production incidents per month, time to resolution, and the share of defects that get reopened. And current spend on AI tooling, because that is the reference the true cost of a merged change eventually has to be argued against.

The item everybody omits is a way of counting delivered capability that does not run through item counts or change counts. Without one, the headline number is throughput under a different name, and every argument about whether the factory produced value collapses into an argument about whether it produced more pull requests. It does not have to be sophisticated — capabilities released per quarter, counted the same way each quarter, by somebody who is not being measured on the count. It has to be independent.

Asking the team is no substitute for any of this, and there is a clean result showing why. METR ran a randomised controlled trial: sixteen experienced open-source developers, 246 tasks, on repositories they knew well. With AI tools allowed they were 19% slower. They had expected to be 24% faster, and afterwards they still believed they had been 20% faster. The people inside the loop could not feel the direction of the effect, never mind its size. A survey measures sentiment, which is worth knowing and is a different quantity.

Target it, track it, or guard it#

Not every measure is the same kind of object, and sorting them is the most consequential decision in the design. A target is something you are trying to move. A tracked measure is something you watch and deliberately never aim at, because aiming at it would produce the wrong behaviour. A guardrail is not supposed to improve at all — it stays in bounds, and it exists so the targets cannot be met by quietly breaking something else.

The middle band is where a programme goes wrong on paper while going wrong in reality too. Percentage of changes written by agents is a genuinely interesting number and exactly the one a steering group wants on the front page. The moment it becomes a goal, the cheapest way to move it is to generate more low-value changes — a refactor split across six pull requests, one dependency bump per package, a comment tidy — and it will move. The metric improves. The review queue lengthens, the codebase grows, and everything that mattered gets worse while the front-page line goes up and to the right.

That is the general shape of the risk. The specific defence is a rule about reading rather than about collection: the targets are read as a set, and not one of them means anything alone.

You can always ship faster by shipping worse, and a programme measured on speed alone will find that out without anybody deciding to. DORA’s 2025 report puts a name on the mechanism: AI acts as an amplifier, and adoption raises throughput and instability together. A factory that gets quicker and less stable at once is behaving as the evidence predicts rather than failing unusually — but only a paired reading tells you which of the two you are looking at.

Guardrails are the smaller half of the design and the easier half to defend, since nobody has to argue about what a good value would be. Unreviewed merges stay at zero, a policy rather than an aspiration, and one that belongs to the gate a person actually stands at. Duplicated code stays flat, because more changes, faster is precisely the pressure that produces four near-identical helpers in four services, and no target in the set would notice it happening.

Where measurement goes wrong#

The most common failure is the dashboard that shows everything and answers nothing. Forty tiles, all live, all correct, and no stated question that any of them was built to answer — so nobody opens it after the second week and the programme goes back to being assessed on anecdote. The repair is unglamorous: write the question first, build the smallest thing that answers it, and delete a tile when its question stops being asked.

Three further failures look like diligence from a distance. A target lifted from a vendor’s published figure instead of from your own baseline, so a result produced by a ten-day greenfield build becomes a commitment for a brownfield monorepo with twenty-odd services. A measure whose definition changes halfway through the year — lead time sliding from ready-to-merged to request-to-production — leaving the series discontinuous at precisely the point where the improvement appears. And the pilot declared a success on a metric chosen after the results came in, which is storytelling with a chart attached. The defence against the last two is the one you already apply to prompts: a definition is written down, versioned, dated and replayed, the same discipline that a fixed set of already-solved work imposes on the agents themselves.

Then there is the boundary the rest of the design has to be built around, and it deserves more than a footnote. Factory metrics measure agents and flow. Several of them do not stay that way on contact with an organisation: review load, escalation response time, time to answer a clarification. Each is a legitimate flow measure, and each can be read as a statement about a person whether or not that was anyone’s intent — and once a measure can be read that way it will be, usually by somebody two levels from the work who never saw the caveat.

Reading a measure as a statement about a person is a different conversation with different obligations, and in a number of jurisdictions legal ones: employee performance monitoring is regulated, and a metric introduced as an engineering diagnostic does not stop being monitoring because of how it was introduced. Decide in advance rather than in the moment. Aggregate deliberately, at the level of a team or a queue rather than a name. Write down which measures are never reported per person, and keep that list wherever the dashboard is built rather than in a memo nobody can find. Treat a request to break one of them out by individual as a decision needing its own sign-off, not as a filter change. That costs an afternoon before anyone asks, and a negotiation afterwards.

None of this makes a programme succeed. It makes the question answerable, which is a smaller claim and a more useful one, because a factory is expensive enough that we think it is going well should not be the best answer available at the end of the year. What it does not settle is who is accountable for acting on any of it — which is the last thing left to name.