Measuring a factory honestly

“The agent writes code quickly” and “the team delivers useful changes faster” are different claims. A pilot should establish which parts improved, what extra work appeared and whether the result still meets the same requirements.

Choose the comparison before declaring success#

A baseline describes the process you are comparing with. Record it before the pilot where practical: comparable task types, time spent at each stage, human effort, costs and quality outcomes. Define what would count as a useful improvement before choosing a flattering result afterwards.

Separate elapsed time from active effort. A change may spend ten minutes in a worker and two days waiting for review. Record request-to-ready, ready-to-accepted and accepted-to-deployed time when those stages matter. Include the time people spend preparing tasks, answering questions, reviewing and correcting work.

Use established metric names carefully. DORA’s delivery metrics include change lead time from a code commit to production. Request-to-production time is useful too, but it starts earlier. Do not silently switch between the two within one chart.

Missing pre-pilot measurements do not make evaluation impossible. Reliable tracker and deployment history can reconstruct some timings. A suitable concurrent comparison can help, as can alternating methods on comparable tasks. Record what is missing instead of presenting a guessed history as measured data.

Choose a result beyond counting changes. In an illustrative export pilot, it could be whether users can finish a report that previously failed, and how much manual work that removes. A raw count of released features can be manipulated through changing how features are split, just like a pull-request count.

Ask people about the experience, but keep perception separate from timing. METR’s early-2025 study assigned 246 tasks across 16 experienced open-source developers. In that setting, AI access increased completion time by 19 percent, even though participants afterwards estimated a speedup. This is a dated result about those tools, developers and tasks, not a current universal effect.

METR’s February 2026 follow-up reported indications of faster work with later tools but substantial selection and measurement problems. Its uncertainty is another reason to test your own process rather than adopt either study’s point estimate as a promised saving.

Decide what to improve, observe and protect#

An improvement target states a change you want. An observation helps explain what is happening without necessarily being something to maximise. A policy or quality limit identifies conditions that speed must not override. The table is an example selection, not a required scorecard.

For example, fewer clarification questions could mean better-prepared tickets. It could also mean that agents stopped asking about missing decisions. Read the count alongside missed requirements and reviewer corrections.

Likewise, ten additional pull requests may be ten useful changes or ten proposals nobody needed. A higher merge rate can reflect better output, easier selected tasks or weaker review. Keep the acceptance standard visible while interpreting the number.

Show the full-size diagram, and pan Back to the phone layout

The figure uses illustrative values to show why speed and quality need to be read together. DORA also evaluates delivery throughput alongside instability. This does not mean that faster delivery must cause worse reliability; the purpose is to observe both rather than assume a tradeoff or an improvement.

Set policy checks according to the workflow actually approved. In a supervised workflow, a merge without the required person is a violation. An explicitly authorised automatic workflow has different approval rules. Neither is permission to let a favourable dashboard redefine the policy.

Maintenance needs more than one duplicated-code percentage. A shared rule copied into several places can make a follow-up change hard; superficially similar code may also serve intentionally different rules. Inspect related later tasks and the design problems passing tests can miss.

Check what else could explain the result#

A before-and-after difference does not isolate the effect of the factory. The task mix, team, backlog or release process may have changed too. Compare similar work, use suitable random or matched assignment when feasible and record factors that could affect the result.

Include failed and abandoned tasks. A report containing only the changes that merged quickly selects the successes. Keep unfinished work visible and define a follow-up period for defects that appear after release.

Report sample sizes and the spread of results, not only an average. A few easy fixes cannot establish the outcome for large migrations. A median can conceal a long tail of expensive failures, so inspect those cases too.

Version the measurement definitions. If the start point changes from “ready for coding” to “request submitted”, annotate the break or recalculate comparable history where possible. New questions discovered during analysis can be useful, but label them exploratory and test them in later work instead of pretending they were the original success criteria.

Keep the dashboard focused on decisions people make. A reviewer needs to know which work is waiting and why. A maintainer deciding whether to expand the pilot needs quality and complete task costs. Neither needs every available counter on one screen.

Agree on access, aggregation and retention before collecting identifiable workflow data. A long wait may reflect a missing requirement or an overloaded queue, not poor individual performance. Do not quietly turn a team-level diagnostic into a ranking of named employees.

Use repeatable evaluation tasks to compare agent configurations, and operational measurements to evaluate how the whole workflow performs with people. They answer related but different questions. Finally, name who will act on the findings; collecting them does not improve the process by itself.

Questions people also ask

5 QUESTIONS

▸ What should we measure before a pilot?

Record the work types, elapsed time, human preparation and review effort, costs, defects and rework. Define the start and end of each measure. Choose at least one outcome tied to the user's problem, not only the number of code changes. A small pilot needs a manageable set of questions, not every available metric.

▸ Can a pilot still be evaluated without a pre-pilot baseline?

Yes, but do not invent missing history. Reliable issue and deployment records may reconstruct part of it. You can also compare suitable concurrent groups or alternate the workflow on comparable tasks. State the limits of the comparison; recalled review hours are not the same as recorded hours.

▸ Why is AI-written code share a poor productivity target?

It measures where code came from, not whether the result helped. The share can rise through unnecessary edits or a changed counting rule. Track adoption when useful, but assess productivity with useful outcomes, total effort and quality under consistent definitions.

▸ Can a team's impressions show whether the pilot worked?

They can reveal usability, confidence and frustration, which matter. They do not replace measuring task time or defects. Keep these observations alongside measured outcomes rather than assuming that feeling faster establishes how much time was saved.

▸ Should these metrics rank individual developers?

That is not the purpose of this workflow evaluation. Review time and waiting often reflect dependencies and task difficulty, not one person's performance. Prefer team or process-level reporting, limit access and retention, and separately assess any proposed use of identifiable employee data with the responsible privacy and employment specialists.