Deterministic steps and AI steps
Choose scripts for explicit rules and AI for investigation. See when one model call is enough, when a workflow fits, and which checks code must enforce.
Use ordinary code when the rule is explicit, such as running a required test or checking a repository permission. Use AI to investigate questions that need interpretation. The model can propose an action, but it should not decide whether to ignore the rules governing that action.
A program can compare the deployed revision with the revision expected for a release. No model needs to guess whether the two identifiers match. Explaining why a test failed may require reading code and trying an experiment—that is a different kind of work.
Running a test and explaining its failure are different jobs. A test runner follows a known command and reports a result. An agent may need to inspect the code to understand that result. Connect the two rather than asking the model to invent both the rules and the verdict.
Put explicit rules in ordinary code#
Build commands, formatters, static checks and result collection can run without a model deciding how to perform them each time. A browser test can follow a defined sequence and capture a screenshot. Checking what that screenshot reveals about an unfamiliar design is a separate task.
The same distinction applies at the start and end of a run. Code can apply an approved rule about which changed paths require a build. Code can compare the deployed revision with the intended release revision. A model can investigate an exception, but it should not silently replace the recorded condition.
The figure shows an illustrative division of work, not measured time or cost. It does not say that all scripts run first. A test for a generated change necessarily runs after that change exists.
Here, deterministic describes an explicitly implemented decision rather than a model judgement. The complete environment may still vary: a network service can time out and a test can be flaky. Record the inputs and results instead of assuming that using a script guarantees identical execution.
Use the model where interpretation helps: investigate an unfamiliar failure, propose an implementation or assess whether a requirement is missing a case. It may also draft rules or tests. Review those proposals before making them the rules that decide acceptance.
Prepare useful input at each step. A tool can return the failing assertion and a reference to the full log. HumanLayer’s compact-feedback example demonstrates that approach. Do not promise that the first 200 lines always contain the cause, or hide skipped tests and warnings just to save tokens. The harness connects those tools to the agent.
Choose the smallest useful amount of AI#
Needing language interpretation does not automatically mean needing an agent. Anthropic distinguishes model-directed agents from predefined workflows and recommends testing simpler solutions first. Compare the actual job, not the appeal of its label.
In an illustrative support system, summarising a supplied customer message may need one model call. The application already has the text and asks for a bounded output. There is no need to invent further investigation steps.
Classifying the message, looking up an order and drafting a reply can be a fixed workflow when the sequence and branches are already known. A classifier or language model can interpret the message while ordinary code controls the following steps. Traditional machine learning and agent systems are not alternatives for every part of an application.
Investigating conflicting delivery records can justify an agent when the next useful lookup depends on what the previous result revealed. It might need another source or a question to the customer. The worked agent example follows that route without granting permission to refund or modify the order.
Test whether the extra investigation improves the supported answer enough to justify its time, cost and review burden. Include missing records and contradictory evidence. Repetition, unstructured text and a large document are properties of the task; none alone proves that a continuing agent loop is the right solution.
Handle a known prerequisite outside the model#
Kumar’s harness example puts login handling in application code instead of giving the model a password. The useful boundary is between choosing the task and carrying out a known prerequisite. It is not permission to log in anywhere the browser happens to land.
In an illustrative report tool, a read request receives a documented session-expired response. The runtime may renew the already authorised session once for the configured service and account, then repeat that read. One renewal is this example’s limit, not a general standard. A permission denial, an unexpected destination or a different account needs a separate decision, not a broader credential.
Keep the credential in a component the model cannot inspect through its tools. Omitting it from the prompt is insufficient if a shell can print the same environment variable or a log contains the token. OWASP’s secrets guidance supports restricting access and preventing disclosure in logs. These are additional requirements for this proposed design, not a security assessment of the demonstration.
Return the renewed read’s actual result or a specific failure, without secret values. Authentication restores access; it does not establish that the report exists or is correct. Run the result check afterwards. Repeating a read and repeating a possibly completed write need different rules; the lost-response example shows why.
Generate exact settings from approved inputs#
Consider a static site whose build emits approved inline JavaScript. Its Content Security Policy can list hashes of those exact scripts. MDN explains CSP hash sources: the browser can allow an inline script when its contents match a permitted hash. Rebuilding the list avoids maintaining hashes by hand after each legitimate script change.
This generates values for a policy; it does not decide which code deserves approval. Hashing every script in compromised output could authorise the unwanted script too. Review the trusted inputs and build process before relying on the generated list.
Similarly, generate allowed service origins from explicitly approved configuration, not every URL found in the output. An injected address must not grant itself network permission by appearing in a page.
Check the generated policy separately. Verify that expected values are present, forbidden broad exceptions are absent and unresolved template markers did not reach the final result. Exercise the served page and headers in a browser as well.
A generator and checker that share the same faulty extraction code can agree on the wrong result. Recomputing values catches stale or missing policy but not every shared bug. Use known fixtures and behaviour checks for that limitation. The example is a way to generate settings reliably, not proof that the whole site is secure.
Check permission before running the action#
Keep conventions in project instructions: naming, useful reference code and supported commands. Enforce consequential restrictions in the tools and environment. A proposed production command should be rejected when the run is allowed only a test environment.
A blocking pre-execution hook is one possible enforcement point. Claude Code’s documentation describes the events and responses for that tool. Other clients differ. Confirm that the relevant action triggers the check, that rejection blocks it and that another shell or connector cannot bypass it.
A hook that only logs an event or warns after execution does not prevent the action. Nor does an instruction saying “ask first”. Untrusted text may try to persuade the model to cross that boundary, which is why the permission check must not rely on the same persuasion.
Apply this reasoning to required tests too. The builder can run additional checks and propose coverage, but should not unilaterally waive required acceptance tests. Derive the minimum checks from maintained project rules, actual dependencies and the change’s consequences.
Focused tests can give quick feedback inside a repair loop. Broader integration and regression checks may still be required before merge. When the dependency map is incomplete, do not pretend that a path filter proves the rest of the system is unaffected. A full suite may be appropriate when it is inexpensive or the impact is uncertain.
Do not let a model waive a failed check#
Suppose a required scan reports a problem. The agent can investigate whether the finding is valid and propose a correction. It must not declare the finding irrelevant merely to reach the next step. Use the existing exception process when a person with the right authority decides an exception is justified.
Routing has the same distinction. A lookup table can route work when ownership is explicit. An ambiguous request may need classification or a question. A model’s suggested queue is not authority to change another team’s code.
For an illustrative support request, a known component ID can map directly to its owning team. A free-text symptom may instead need classification. Finding the underlying defect is a further investigation, not something the queue choice has already accomplished. Decision models show how to separate that bounded choice from writing an explanation and executing an action.
Returning an allowed queue name only establishes the shape of the result. The queue can still be wrong, and moving the request can still be forbidden. Type-safe output versus correctness follows those checks separately; testable decision questions make the available evidence and missing-context outcome explicit.
Before assigning a decision to AI, ask which rule and inputs would settle it. Implement the rule when it is known. When interpretation remains, record the uncertainty and decide who accepts the result. Do not invent a threshold merely to make a judgement look deterministic.
Use restricted credentials and stop controls to enforce the resulting policy. Then provide an isolated environment with suitable test data so the permitted checks can actually run.
IF YOU REMEMBER ONE THING
Use code to enforce a rule that is already decided. Let an agent investigate and propose, but do not let it turn a required failure into a pass by changing the acceptance rule.
Questions people also ask
5 QUESTIONSWhich pipeline steps usually do not need a model?
Running known build and test commands, formatting, collecting results and checking explicit identifiers or permissions can be ordinary code. Choosing a new implementation or investigating an unexplained failure may need interpretation. Some tasks combine both: a model proposes a test and a test runner executes it.
Can code generate a security policy?
It can generate exact values from approved rules and inputs. For example, a static-site build can hash its approved inline scripts for a Content Security Policy. Hashing arbitrary output does not decide whether the script should be trusted in the first place.
Should every repair attempt run the full test suite?
It depends on the cost and required coverage. Focused checks can give faster feedback while broader checks run before acceptance. Derive required coverage from maintained rules and actual dependencies, and treat uncertain impact conservatively. A scheduled test after merge cannot retroactively satisfy a required pre-merge check.
How does a hook differ from an instruction file?
An instruction file tells the model what it should do. A hook is program code run at a defined event. A blocking pre-execution hook can reject an action, but only when the runtime enforces it and the agent cannot bypass it. Not every hook blocks execution.
Must every scripted step happen before the first model call?
No. Some checks prepare the input; others test code the model has not written yet. Run each check when its inputs are ready. Return enough detail to support the next decision without assuming that an arbitrary log excerpt always preserves the cause.