Verifying after the deploy

The build went green and the change merged, and nothing in that sequence has touched the environment your users are in. Continuous integration proved that the code compiles and the tests pass on a machine that was configured to make them pass. Whether the change works is a different question, asked of a different computer, and there is no honest way to ask it until the revision serving traffic is the commit you merged.

What a green build cannot see#

Start with the failures a test suite is structurally incapable of producing, because it does not run where they live. An environment variable that was never added to the deployment. A database the code reaches happily from a developer’s laptop and cannot reach at all from where it now runs. A routing rule that was written, reviewed, merged and then never applied. A runtime whose library versions differ from the ones the image was built against. And integration between two services, which does not exist as a testable thing until both of them are deployed somewhere together.

Every one of those produces a flawless pipeline and a broken system. That is not a gap in the tests to be closed by writing more of them; it is a category difference. A suite answers questions about code. These are questions about a deployment, and the only place to ask them is the deployment.

Underneath all of it sits a prerequisite that teams usually discover late enough for it to hurt. “Is the thing I merged the thing that is running” only has an answer if deployments are uniquely versioned and traceable back to a commit. Where a release is a snapshot that cannot be identified or redeployed on its own, the question is not difficult, it is meaningless — and the verifier cannot be built at all until the delivery pipeline changes. That is a weeks-long piece of platform work discovered in the middle of what looked like an afternoon of agent design, so it is worth checking first rather than second.

Wait for the revision, not the clock#

The verifier’s first action is a comparison rather than a wait. It asks the deployment target what revision is serving, and it keeps asking until that revision equals the merged commit or a timeout expires. Nothing else runs before that answer comes back.

A timeout is not a failed check and must never be filed as one. If the deployed revision never becomes the merged commit, nothing whatsoever has been learnt about the change; what has been learnt is that delivery did not finish. That goes to a person as a deployment problem, carrying the revision it found and the revision it wanted, and the agent stops there. Diagnosing it would mean reading infrastructure the agent has deliberately not been given, and the credentials it holds should make that refusal literal rather than leaving it to good behaviour.

Once the two match, the checks run cheapest-first. The service reports healthy. An authentication smoke test proves a caller can actually get in, which catches the whole class of failures where a secret rotated and nothing downstream noticed. Then the specific path the change affected is exercised, and its result is asserted against what the work item said should happen. That last step carries all the value. A verification suite that stops at “the service returned 200” is telling you the process is alive, which the health check already said, and nothing at all about whether the change did what it was merged to do.

Both outcomes get written down. A pass attaches its evidence to the work item — the revision, the timestamps, the requests, the responses it asserted on — so that “done” has something behind it beyond a status field. A failure opens a defect with that same evidence and hands it on. The agent does not attempt a repair, for the same reason a review agent does not push its own fixes: the moment a step can write to the thing it checks, its verdict is a verdict on its own work.

Read-only is not side-effect-free#

Now the part that gets waved through. “Exercise the path the change affected” sounds like reading, and it almost never is. A flow that everybody in the room describes as read-only reaches a service that writes an audit row, increments a usage counter, fires a webhook, takes a lock, or wakes a downstream job. Against production, every one of those lands in real business data belonging to real customers.

Take a dispatch endpoint. Verifying a change to it means booking a real delivery — not a pretend one, but a booking with a number, in the sequence, visible to whoever loads the van in the morning, quite possibly emailed to the customer. Or take a notification flow, where asserting that the message was sent is asserting that somebody received it. Neither of those is a test. Both are transactions, executed by an agent, on a schedule, several times a day, and the first person to notice will be in finance rather than in engineering.

So the phasing that actually holds is unglamorous. Verification that exercises real flows runs against an integration or QA environment with synthetic data, where a written row is a written row in something built to be thrown away. In production the checks stay passive: correct revision deployed, service healthy, error rate unmoved, the expected log lines and metrics present, and read-only synthetic endpoints called where somebody has built them. Production validation in an early factory means deployment and health verification. It does not mean replaying a business transaction and hoping.

Going further has to be earned, and the bar is higher than it looks from the outside. Dedicated synthetic users and tenants that are identifiable as synthetic in every downstream system — not only the one under test, but analytics, the warehouse, the customer mailer, and whatever sends the email. Verification endpoints that are safe to call twice. A rate limit on the verification calls themselves. Automatic cleanup of whatever the check created. An explicit written list of workflows the verifier may never exercise. And a named human approving anything that touches a workflow with real-world consequences. That list is a safety case for one workflow; a second exception needs its own, because there is no blanket version of this argument.

The failures cluster into four, and three of them are quiet. A fixed sleep that has been green for eight months because it has been reading the previous build for eight months, so the one time it should have caught something it did not. A smoke test that only ever proved the process was running. A verification flow somebody assured you was harmless, which turns out to write to an audit log, found when a recurring entry gets queried. And the loud one, which costs the most goodwill: a verifier that files the same defect every ten minutes, because somebody wired it to report and nobody wired it to the de-duplication the incident path already does. Two hundred identical tickets are not two hundred problems. They are one problem, buried.