Harness engineering: the tools and checks around an agent

Consider two agents given the same request: “Fix the upload-size bug.” One has the current code and working tests. The other has an outdated note, a broken build command and access to production. The request is identical, but the work they can do is very different. Improving that setup is harness engineering.

The same request can have a very different setup#

In the agent decision loop, the model proposes actions. The surrounding software supplies the information and tools, decides which requests may run and returns their results. It also handles limits and stopping. That software and its configuration form the harness.

Some of it belongs to the coding-tool provider: the program that calls the model and runs tools. Some belongs to your team: repository instructions, test commands, temporary environments and access rules. Birgitta Böckeler describes these as the inner and outer harness. Improving your team’s setup does not require building a new coding tool.

Context engineering focuses on what the model receives. The harness also determines what it can do. Loop engineering deals with what happens after the result: another attempt, a question or a stop.

Separate the jobs before choosing the tools

In Software Factory Design Patterns, Dex Horthy and Vaibhav Gupta separate the execution setup into compute, development environment, harness and orchestration. In this narrower use, harness means the coding runtime; the broader outer-harness discussion can also include environment and workflow configuration. The useful distinction is who supplies each capability, not which label wins.

Try it: find the responsibility behind a stopped upload task

The same upload fix stops at a different point. Which observation changes the next investigation? Worker process never startsWorker starts; database blockedRuntime cannot block a toolChecks finish; review missing

Start with compute availability.

1. The task has no running worker.
2. Inspect allocation and process-start records.
3. Restore a worker before claiming that its tests or model failed.

No application check ran. Changing the upload code cannot repair an unavailable worker.

Illustrative responsibility map, not an automatic diagnosis or a tested runtime comparison. Real failures can involve several layers; inspect their records. No worker or tool runs here.

These are responsibilities, not four mandatory deployments. One existing product may cover several; an existing CI pipeline may handle the orchestration you need. Buying a coding runtime does not require buying its entire surrounding stack, and owning a workflow does not require rebuilding the runtime. Keep a component replaceable where a real requirement justifies it, rather than wrapping every SDK in another interface.

For example, an unreachable test database is an environment-readiness problem, not evidence that the model needs a better coding prompt. A finished worker whose result never reaches review points to a different handoff. The orchestration guide asks which existing system should own that state.

What each part gives the agent#

Instructions explain the project. An AGENTS.md file, or the equivalent for your tool, can name the supported test command and explain which generated files should not be edited. Check how your tool finds these files and handles conflicting instructions. The filename alone does not guarantee that every agent reads it.

Skills provide task-specific procedures. A migration skill can contain instructions, example migrations and helper scripts. The Agent Skills format packages this material for compatible tools. Loading the full procedure only when it is needed is called progressive disclosure. A spelling correction should not need every migration instruction.

Tools perform actions. Reading a file, running a test and publishing a pull request need different operations and permissions. MCP, the Model Context Protocol, is one way to connect an application to tools and information. Connecting a tool does not decide whether the current task is allowed to use it.

Hooks run work at particular events. A hook might run a check after an edit or examine a request before execution. The available events and blocking behaviour depend on the coding tool. Claude Code’s hook reference documents one implementation. A hook that prints a warning is not necessarily a hook that prevents an action.

The environment makes the task runnable. Provide a known code version, dependencies, invented test data and working startup commands. Keep production credentials out of that workspace. Otherwise the agent may spend the task repairing its environment or act on real data when it should be testing.

Prevent the action, not just warn against it#

“Do not run production migrations” explains the rule. It does not stop a tool from running one. The difference matters when the model requests the wrong operation despite the warning.

A

Instructions only

Production access is still available.

1. Instruction “Do not change production.”
2. Mistaken request Migrate the production database.
3. Permissive tool Production writes are allowed.

The request can passThe tool can write to production.

B

An enforced boundary

Only the test database is permitted.

1. Same instruction “Do not change production.”
2. Same mistaken request Migrate the production database.
3. Scoped tool Production target rejected. No production credential.

Stopped before executionThis tool does not attempt the write.

Same request→

What changedThe model made the same mistake. Target validation and scoped access changed what the tool could do.

Hypothetical execution paths. Instructions guide the model; enforced scope limits this tool’s effects.

In this illustrative setup, the tool checks its target and rejects production before running the command. The workspace also lacks the production credential. The instruction explains what should happen; the access restrictions limit what can happen when the model misunderstands.

Conflicting information needs a decision too. An old example may disagree with current project rules, or a ticket may request something a security policy prohibits. Make the governing source clear and ask when the conflict cannot be resolved. Retrieved text must not become permission to override these rules. The prompt-injection guide covers deliberately hostile instructions in such material.

Follow a bug from reproduction to test results#

In the upload example, the agreed rule allows a file exactly at the size limit and rejects a larger file. The harness supplies the requirement, the relevant code and a test environment where the failure can be reproduced.

The agent changes the comparison on its branch. The runtime runs checks for a file below the limit, exactly at it and one byte above it. Testing the actual upload path can reveal another validation layer that a local unit test missed.

When a check fails, return the actual assertion and enough detail to investigate it. Keep the full log available by reference. HumanLayer’s compact-feedback example describes making failures easier to find without sending every line of routine output to the model.

The next step depends on the failure. A wrong comparison can justify another coding attempt. An unavailable test database means the check is blocked. A missing decision about whether the limit includes request metadata needs a question. Automatically making another edit in all three cases would be a mistake.

After the required checks pass, save the changed code, its revision and the commands and results for review. The statement “fixed” is the agent’s conclusion, not a substitute for those records. It also does not mean that anyone accepted or deployed the change.

Check the result before reporting success#

In Tejas Kumar’s harness demonstration, a browser agent reaches a login page but reports a successful upvote. Adding a verifier makes the failure visible; a later authentication handler addresses the missing prerequisite. Detecting the failure and repairing it are separate improvements.

Consider a separate illustrative example: an agent requests report R42 for dataset snapshot v7. The approved test fixture expects exactly two rows, A = 3 and B = 5, in either order. This deliberately small task isolates the completion check; it does not justify using an agent to generate a simple report.

Without a result check: the tool returns a report identifier and the model says the file is ready. A receipt establishes that the service accepted a request under this example’s contract. It does not establish that the requested file can be read.

With a result check: application code reads the service’s record and finds R42 still pending. The harness reports that no completed report has been verified. It has not repaired the task, but it no longer turns the model’s completion message into success.

After an appropriate next step: the runtime waits within its configured time limit and reads again. If the service now reports the requested result as ready, the verifier checks its request identity, source snapshot and actual rows. Those checks can pass without asking the model to judge its own work. If the read fails, the outcome remains unknown; another report submission is not a safe default. The retry example explains that distinction.

The request and snapshot binding must come from the trusted service record, not labels invented in generated text. Compare the returned rows with the separately reviewed fixture. A matching result still does not establish that every action during the run was permitted; access controls and checks for forbidden effects have their own jobs.

A deterministic check can consistently check the wrong thing. Kumar’s shortened verifier checks a click and the resulting URL, rather than directly reading a stored vote. That is a limit of the displayed snippet, not a reproduced defect in his complete application. For our report, merely finding a file would likewise accept the wrong request or an old snapshot. Test the verifier with those counterexamples before relying on its verdict.

Replace a part without replacing the rules#

Albada’s introductory chapter recommends modular components with clear interfaces so an agent system can change. The useful question is which change should stay local. A separate module need not be a separate service, and no interface makes every provider interchangeable.

Consider an illustrative document-search tool whose storage provider is being replaced. Keep the agent-facing meaning stable: a successful search returns matching documents and their source versions; an unavailable provider returns an explicit failure. Do not translate a timeout into an empty result list. “Nothing matched” and “the search did not finish” justify different next actions.

Put the provider-specific request and response conversion behind the tool. Keep access checks in the enforced execution path, so changing the search provider cannot accidentally expose another customer’s documents. OpenAI’s custom-function documentation makes the relevant boundary visible: the model supplies a request, while application code performs the operation and returns its result.

Before switching, test a known match, no match, a denied request and an unavailable service. Then rerun representative agent tasks. Matching the old response shape does not establish matching retrieval quality, error behaviour or task outcomes.

Version the tool, its instructions and the configuration used for the comparison. Preserve a way to restore the previous setup. Replacing a tool or revising its instructions is application maintenance; it is not automatically the agent learning or the model being trained.

Replacing the coding runtime needs its own checks

Two runtimes that both emit JSON are not necessarily interchangeable. The episode’s discussion of different hook systems suggests a more useful acceptance test: can your workflow observe and control the states it actually depends on? Test the installed versions rather than treating a protocol name or an old product comparison as a capability guarantee.

For the upload task, try starting on a pinned workspace, identifying the run, surfacing a permission wait, reporting a tool failure separately from a failed assertion, stopping further actions, and exporting the resulting diff and check records. Test interrupted sessions too. Declare whether resuming them is supported; a restarted conversation must not silently repeat a write whose outcome is unknown.

A progress event sent to a user interface is not the same as a blocking hook before execution. When an adapter cannot preserve a required restriction or state, reject that workflow or send it to a person rather than quietly dropping the requirement. Keep plans, session references and check results linked to the run and code revision outside disposable workers. The artifact-handoff guide explains how the next worker can use those records without relying on one provider’s conversation history.

Test how the agent handles different tasks#

Application tests check a particular software change. An agent evaluation tests how the configured model and harness handle a collection of tasks. One successful fix does not show how the same setup handles missing requirements or a broken environment.

Use the upload boundary as one case. Add a task that needs clarification and one whose required test environment cannot start. Their expected results differ: a tested fix, a useful question and an accurate blocker report. Treating every opened pull request as success would hide those differences.

Record the model settings, instruction version, tool configuration, repository revision and environment. Compare changes against the same task requirements. Without this record, a changed dependency might be mistaken for a prompt regression.

Fix the part that caused the failure#

Repeatedly running the wrong test command may call for clearer instructions. Editing the wrong module may call for a repository map or a structural check. An unanswered product question needs the person who can decide it. Adding more text to the prompt is not the answer to every failure.

OpenAI’s harness-engineering report describes repository guidance and automated architecture checks in its own project. Treat the mechanisms as ideas to test, not a promise that copying the setup will reproduce its results.

Try the smallest relevant correction and compare it on the affected task and other representative work. A reusable evaluation set helps reveal whether the fix causes another problem. For the next stage in the learning path, preparing a ready ticket explains what the agent needs to know before coding begins.

Questions people also ask

3 QUESTIONS

▸ Is an agent harness just AGENTS.md or CLAUDE.md?

No. Those files can explain project rules and commands. The harness also runs tools, loads information, manages the workspace and controls permissions. Writing a test command in a file does not prove that the agent ran it. Writing a restriction does not enforce it.

▸ How does harness engineering differ from context engineering?

Context engineering chooses what information the model receives. Harness engineering includes the software and configuration it uses to act. They overlap: returning the relevant part of a test failure is both a tool feature and an improvement to the model's next input.

▸ Do I need to build my own harness?

Usually you can start by configuring the coding tool and development tools you already have. Supply working setup commands, project instructions and suitable permissions. Build something new when those tools cannot meet a specific requirement.