Verifying after the deploy
Check that the intended version is serving before judging a release. Distinguish missing version information, rollout delays and failed behaviour checks.
Post-deploy verification checks the software in the environment where it was released. First identify the version serving the request, then run the permitted checks. A passing test against the previous version does not show that the new change works.
A fix has merged, but the server is still running yesterday's build. A checker that calls it immediately may report a pass for the wrong version. Read the deployed build's identifier first, then check the new version when it is available.
The build passed and the change merged. Those are useful results, but they do not show which version users are receiving. Check the deployed version before calling the release verified.
Check the actual deployment, not just the build#
A test establishes what happened with its code, configuration and environment. Production may have a missing environment variable, a different network rule or an unavailable dependency. A test environment can catch many such problems, but passing there does not prove that the actual production configuration is correct.
The SRE Workbook’s canarying chapter explains why teams also evaluate a release under real traffic. A canary releases a change to a limited part of the service and compares its behaviour before expanding the rollout. It complements testing rather than replacing it.
For a simpler first workflow, identify the running version and check its permitted health signals. This does not require a model to guess whether two revision identifiers match. The release system should record which build corresponds to the accepted source change.
Keep the version identifier with the released files#
A static site can emit a small version.json. A service can expose build metadata through an endpoint or its monitoring data. OpenTelemetry’s service conventions, for example, include a service-version attribute. The value still has to come from the correct build.
Write that identifier when creating the released output, and keep it with the files or image it describes. A separate script that announces the latest repository commit can get ahead of the deployment. Two build outputs need identifiers that describe their own contents, not a shared marker updated independently.
The release may be built from a merge commit, a release branch or a bundle of accepted changes. Record that mapping rather than assuming every pull request’s last commit must be the exact deployed identifier. An immutable build identifier or image digest can distinguish outputs made from the same source with different inputs.
Missing information needs its own result. Do not stamp HEAD, unknown or undefined and treat it as a usable version. A checker can report the missing marker and continue authorised health monitoring without claiming to have tested the intended release.
Wait for the intended version, with a deadline#
Use a deployment-complete event or poll the target with a sensible interval and deadline. Confirm the serving version before attributing a behaviour check to the release. Sleeping for ten minutes and checking without identifying the version only assumes that deployment finished in time.
The figure illustrates a release that takes time to appear. If the deadline expires while a valid older version is still observed, report the rollout delay and both identifiers. If the marker is absent, report missing identification. Neither result says that a functional test of the new version failed.
A newer release may supersede the expected one before the verifier sees it. Record that situation and follow the release policy; do not keep waiting forever for a version that is no longer meant to serve. Avoid marking the newer version as checked using results from its predecessor.
Account for multiple instances and caches. One response identifies the instance or cached output that answered, not necessarily the whole fleet. Check rollout status and route verification to the relevant version where the platform supports it.
Once the target is identified, run the approved checks. Service health answers a different question from a test of the changed feature. Record the target, version, time, actual result and any checks that could not run. Keep secrets and customer data out of routine reports.
In this supervised example, the verifier reports problems rather than repairing them. An urgent production failure goes through the team’s incident route; a non-urgent defect can become a backlog item. Repeated checks should update the existing problem rather than open another identical ticket.
Decide which checks are safe to run in production#
A request described as a check may still write an audit record, send a message or trigger work elsewhere. An order-placement check can create an actual order. A notification check can contact a real recipient. Find those effects before scheduling the request.
A limited starting policy can allow version, health and operational-metric checks while keeping business-flow tests in an isolated environment with invented data. Even health requests need sensible rate limits and access rules.
Production synthetic tests are possible, but require deliberate design. Use recognisable test accounts and ensure that downstream services also treat them as tests. Control delivery, payments, notifications and analytics where applicable. Specify cleanup, repeat-request behaviour and workflows that must never be exercised automatically.
Approve those operations for the particular workflow and enforce them through restricted tools and credentials. A permission to read service health should not also permit changing configuration or sending customer notifications.
When verification finds a problem, save enough information to investigate without repeating a harmful action. The incident-to-backlog guide explains how to group related failures, preserve useful evidence and send the work to the right people.
IF YOU REMEMBER ONE THING
Identify what is serving, check it using permitted operations and report the result precisely. A rollout delay, an unidentified version and a failed feature test need different responses.
Questions people also ask
6 QUESTIONSHow long should a verifier wait after deployment?
Use a deployment event or bounded polling to find the intended version instead of assuming a fixed delay is enough. Choose a polling interval and deadline appropriate to the deployment. Report an old version, missing version information and a failed behaviour check as different results.
How should a service report its version?
Include a build identifier in the released files or service metadata and link that identifier to the source revision. Read it from the deployed output. Do not update a separate latest-version marker that can advance while older application files are still serving.
What happens when version information is missing?
Report that the checked deployment cannot be identified. General health checks can still be useful, but do not attribute their result to the intended change. An invalid placeholder such as unknown is not a real version to wait for indefinitely.
Can verification exercise a real business flow in production?
Only under an explicitly designed and authorised test policy. A request may send a notification, create an order or trigger downstream work. Start with limited health and version checks, and use an isolated test environment for business flows. Production synthetic tests need suitable accounts, downstream controls and cleanup.
Should the verifier repair a failure itself?
In the supervised design here, it records the failure and hands it to the responsible team rather than editing code or infrastructure. A separate recovery workflow may have explicit permission to act. Do not give repair permission merely because a checker reported a problem.
Does one correct version response prove that rollout is complete?
No. A load balancer may send the request to just one instance, and a cache may serve an older marker. Check the rollout scope and serving instances through the appropriate deployment information. Record which target the behaviour check actually reached.