Deterministic steps and AI steps

The bill for an automated run is not spread evenly across it. Nearly all the elapsed time goes on work a shell script has been able to do for twenty years, and nearly all the money goes on a few minutes in which a model reads something and forms an opinion. So the first design question in a factory is not which model to use. It is how little of the run you can give it.

What a script should own#

The list is longer than most teams assume. Running the build. Running the test suite. Applying formatting and checking that it held. Running static analysis. Collecting build and test logs as artifacts. Driving a browser through a flow and capturing screenshots of it. Uploading those artifacts and attaching them to the change. Every one of them produces exactly one answer for a given input, and none of them improves if something reasons about it first.

Decisions at the edges belong on the same side, and they are the ones most often handed away by accident. Whether a run starts at all is a condition on facts already in hand — a change that touches only documentation does not need a build, and deciding that is a path match, not an assessment. What happens after a merge is the same shape: poll the deployment until the running revision matches the commit that was merged, then run the checks. Neither of those questions has a second opinion worth paying for.

Three things are genuinely left. Deciding how to implement a change. Diagnosing why a test failed. Deciding whether a change actually meets the acceptance criteria it was given. Those are open-ended, they depend on context no rule enumerates, and there is no closed form that produces them. That is the ambiguous part in the middle, and it is the only part worth a model.

None of this is about capability. A model can run a build perfectly well; it can read a diff and tell you the indentation is wrong. The split exists because of cost, repeatability and blast radius. A script returns the same verdict every time, costs nothing to run again, and cannot be talked out of it by anything it happens to read. A model returns a very good verdict most of the time, costs money on each call, and is influenced by its input — which is least acceptable in exactly the places you would most like certainty.

The ordering follows from that, and it shows up on the invoice. The deterministic slice runs first, before any model reads anything, because what the model is handed is what you pay for. A script that reduces a forty-thousand-line build log to the two hundred lines around the first failure is worth more than a larger model reading all of it, and the same argument applies to the map of the system that the shared instruction set carries: a lookup that names the three services a change touches removes the exploration that would otherwise have been billed by the token.

A hook stops what a file only asks#

The strongest form of the rule is not about which steps you script. It is about where the rules live. Hard rules belong in enforcement the run cannot route around — a tool allowlist naming what may be executed at all, and pre-execution hooks that inspect a proposed action and refuse it. Conventions belong in the instruction files: naming, structure, which reference implementation to copy, how a commit message reads.

It is not a difference of tone. An instruction file guides the model, competing for attention with everything else in the context window. A hook physically stops it, outside the model, on a rule the model has no vote on. Anything that must never happen should therefore be something the run cannot do, rather than something it has been asked not to do — a distinction that stops being academic the moment you accept that some of what a run reads was written by somebody who is not on your team.

That same rule settles a much more ordinary argument: which tests a repair loop should run. “Run the full suite every time” sounds like the cautious answer, and on a large repository it is unaffordable. A loop that spends forty minutes of build capacity before the model sees a result is a loop nobody iterates on, so it gets weakened, and the weakening is done under time pressure by whoever is waiting.

Affected-scope verification is the structure that survives contact with a real repository. Compile and unit-test the module that changed. Test the dependent modules the dependency graph actually names. Run the integration and contract tests around the boundary that moved. Run the broader suite once before merge, or per risk tier. Keep the full regression run scheduled separately, where its cost is predictable and nobody is waiting on it. The trade is explicit: you accept a narrower net inside the loop in exchange for a loop that can run ten times an hour.

And the part that is quietly load-bearing: the builder must never choose its own test profile. Which tests are required is derived — from the changed paths, the service metadata, the risk tier and the acceptance criteria — and handed to the run as a fact it did not get a say in. An agent under pressure to turn a check green is the worst available judge of whether it has tested enough, for the same reason a reviewer should not push its own fixes.

Where the line slips#

Nobody decides to hand the model the whole run. It arrives one reasonable-looking delegation at a time, because each individual one saves an afternoon of writing a script and each is defensible on its own terms. The pull is always in the same direction, and there is never a moment where refusing feels proportionate.

Security, routing and “good enough” are the three worth naming, because each arrives with a plausible argument. Security: the model is already reading the change, so let it judge whether this command is safe to run. Routing: the labels on incoming work are inconsistent, so let it pick which queue an item belongs in. And the quietest one, whether something is good enough: a check came back with a warning rather than a failure, so let the model decide whether the warning matters. In all three, an allowlist, a lookup table and a threshold already had the answer — and in all three the model’s answer is one a sufficiently persuasive input can change.

What makes this hard to argue against in the moment is that the failure is gradual. The deterministic core does not collapse; it hollows out. The bill rises without any single decision that caused it. Two runs on identical input start returning different verdicts, which is first noticed as flakiness and blamed on the tests. Nobody can point at the week it happened, because it never happened in a week — only across a long series of days on which the system got slightly more expensive and slightly less repeatable than the day before.

Holding the line takes something deliberate, and it is cheap to state. Before giving the model a decision, write down the check that would answer it instead. If the check is writable, write the check; if it genuinely is not, you have found one of the few decisions that belongs to the model and you should say so in the design. The decisions you keep on the deterministic side are also the ones worth backing with credentials narrow enough to make the wrong action impossible, because a rule the run cannot break is worth more than a rule it has been told about.

All of this assumes the step has somewhere to run. That turns out to be the part teams underestimate: not the checkout, but a running slice of the system and data that was never a customer’s.