Loop engineering: when an AI agent should retry or stop

A second attempt is useful when it addresses something the first attempt revealed. A failing test identifies a boundary case. A reviewer points out a missing permission check. Without a useful new action, “try again” may just spend more time on the same mistake.

Give the next attempt a reason#

A coding tool repeatedly calls the model, runs permitted actions and returns the results. The wider workflow decides which task to start and when to try again or stop. Here, loop engineering means designing those choices, not simply running the same prompt on a schedule.

For an illustrative maintenance task, replace a deprecated upload validator without changing the files accepted by the application. A failed boundary test suggests a code investigation. An unavailable test database calls for a blocker report. The same retry action does not fit both results.

Fix this task or choose the next improvement#

A fast repair loop works on one task: edit, check, inspect the failure and correct it. A slow improvement loop returns to a continuing concern: find useful work, propose a checked change, receive the review result and inspect the current system again later.

Recurring goal: replace a deprecated validator without changing accepted uploads.

Revisit the current repository

Unresolved proposal for this work?

YES · wait for review; create no duplicate

NO · choose one eligible change

Fast loop · one task

1. Edit the selected call site
2. Run protected behaviour checks

↶ FAIL + useful new evidence · repair within budget

Missing decision, no progress or limit reached · stop or ask

PASS · submit the checked proposal

Accept, reject or request a specific change

↶ Slow loop · reconsider later using the review outcome and current code

Two different reasons to hold workA failing check prevents a bad change from advancing. A full review queue prevents more work from arriving than people can assess.

A proposed maintenance workflow. Repetition is useful only when results change the next action; waiting and no-op outcomes are part of the design.

“Slow” describes the wider cycle, not the model’s speed. A daily job that opens another proposal without noticing that yesterday’s was rejected is missing important feedback. The review should affect what the next run chooses.

Let an experiment result guide the next attempt#

The AutoResearch project discussed in Andrej Karpathy’s No Priors interview gives this pattern a concrete setting. Its experiment instructions ask an agent to change training code, run it under a fixed training-time budget and record a validation score where lower is better. The agent keeps or discards candidates using the results; the instructions also ask it to consider simplicity and memory use.

The same instructions forbid changing the evaluation code. That is a written rule, not evidence of a tool-enforced restriction. In a workflow that relies on a protected evaluator, enforce that boundary outside the agent’s editable files and run the checks against the actual candidate. Otherwise, an improved reported score might come from changing the measurement rather than improving the work.

A per-experiment time limit is not a limit on the whole sequence. For an adapted workflow, set an overall spending or time budget, preserve the accepted version and record unsuccessful attempts. Permission to continue experimenting should not also grant permission to release a result.

This setup gives the next attempt a defined comparison. It does not establish that every product decision has an adequate numerical test. The automation-limits guide explains why choosing the upload policy still needs a different kind of decision.

What a Ralph loop needs between runs#

Geoffrey Huntley’s Ralph pattern repeats agent work towards a goal and uses observed failures to improve the process. Implementations differ. Repetition alone does not guarantee completion, and it does not require a group of agents talking to one another.

For the validator task, a later run needs the current requirement, accepted changes, remaining callers and actual check results. A new conversation can leave old search steps behind. It must not forget an unfinished diff or a pull request already waiting for review.

Permission to retry a local test is not permission to release. Decide separately which actions can proceed automatically and which still need a person.

Write down the goal and stopping rules#

“Improve code quality” gives the worker no clear stopping point. “Replace one remaining use of this validator while preserving accepted uploads and documented errors” supplies a task and a condition the change must meet.

Also decide what starts the work, what the agent may change, which results count as progress and how much time or money the task may use. Name the person who handles missing decisions and unfinished work.

The loop-contract example collects these choices. It is a worksheet, not executable configuration. Its example limits must be adapted and enforced in the actual runtime.

Follow one maintenance task#

Start from the current code and look for an existing task or proposal addressing the same caller. If it is still waiting for review, wait or work on the requested correction. If no eligible caller remains, record that there is nothing to change.

Inspect one candidate and choose the smallest coherent change. An ordinary code transformation may suffice. Use a model when the local differences need investigation. Check the existing behaviour before the edit so an earlier failure is not mistaken for a new defect.

Run the relevant checks after the change. If the old and new validators disagree, explain the difference; do not quietly weaken the requirement. The proposal should include the changed code, test results and any decision still needed.

When creating the pull request, account for a response that may be lost after the server accepts the request. AWS’s safe-retry guidance explains why a timeout is not proof that nothing happened. Check for the existing result before repeating a possibly completed write.

Follow this illustrative trace: the create request times out; the runtime records an uncertain write; a read-only lookup finds a proposal for the intended task, repository and branches. The runtime checks the actual head revision and proposed changes, then saves the proposal identifier and stops trying to create it. A matching title alone would not establish that it found the intended work.

If the lookup is unavailable or its result is ambiguous, keep the write unresolved and use a bounded read-retry or escalation policy. Even “not found” may be insufficient when the service’s reads can lag behind writes. Repeating creation requires a supported safe-retry contract, not the model’s guess. Finding the proposal also does not mean its code was accepted, merged or deployed. The harness result check should report only what the observation establishes.

Add another task type or change more at once#

Adding accessibility checks creates another kind of work, with its own eligibility and review rules. Updating several related callers in one change increases the amount of work per run. These are different decisions and need different tests.

Keep related edits together when that makes them easier to understand. Do not insist on one file per task if the behaviour spans two files. But do not expand task types or patch size simply to produce more work while review is already backed up.

Use failed checks and full queues to control the work#

Agent discussions sometimes use backpressure for checks that prevent an incorrect change from advancing. HumanLayer’s example returns the useful failure without filling the next model call with routine test output. Keep the full results available and preserve the command’s actual success or failure.

A full review queue provides another kind of feedback. A failing test says this change is not ready. A full queue says the system should stop producing more proposals for now.

Protect important acceptance checks from casual edits. Removing a failing assertion may turn the build green without fixing the requested behaviour. Another model agreeing with the builder does not supply the missing test.

Make CI faster without skipping tests#

Consider a second proposed loop: reduce unnecessary test-environment setup. Measure the current job with its runner type, dependency versions, cache state and test workload. Separate waiting for a runner from executing the job.

Try one explanation for the delay, such as repeated dependency setup. Compare multiple equivalent runs. A warm cache on the candidate and a cold cache on the baseline would make the difference hard to attribute to the edit.

A helper agent can summarise logs, but it must distinguish successful, failed, cancelled and missing runs. A job that finishes sooner because it skipped half the tests did less work. The goal is the same required testing with less avoidable delay, not simply a lower duration. This is an evaluation example, not a reported saving.

Report why the loop stopped#

A ready proposal, nothing eligible, pending review, missing decision and exhausted budget are different outcomes. Record which one occurred. Otherwise, an appropriate wait can look like a broken scheduler and an abandoned task can look complete.

Apply limits across child workers and replacement sessions, not separately each time an agent starts. Prevent two workers from owning the same task. Preserve edits and attempted external actions after an interruption. The stop controls must restrict the underlying tools, not merely send a request to stop.

Decide whether more model work will help#

You can give a task more attempts or workers. You can also change the input, narrow the task or run a useful check earlier. Informal phrases such as “token harder” and “token smarter” describe those choices; neither is a measurement of success.

For example, generating ten proposals when people can review two does not complete ten changes. More generation would increase waiting. On another task, spending more effort on a missing investigation may prevent hours of rework. Measure the actual constraint before optimising token use.

Count useful results and the work left for people#

Count accepted changes alongside review effort, duplicates, reversions, missed defects and total cost. Include unsuccessful runs and valid no-change results. Try the next related requirement too: today’s passing patch may make tomorrow’s maintenance harder.

Repeated failures can justify a changed instruction or an added check. Test that change before applying it widely. Logs do not improve future runs by themselves. Deterministic steps and AI steps helps choose which checks belong outside the model; the measurement guide covers comparison with the existing process.

Questions people also ask

3 QUESTIONS

▸ How often should a maintenance loop run?

Match it to how often eligible work appears and how quickly proposals can be reviewed. A daily run need not produce another proposal while yesterday's is unresolved. Start from an event when that fits the work better than a fixed schedule.

▸ Does a slow loop use a slower model?

No. Here, slow refers to the wider cycle of finding work, proposing a change, receiving review and checking the system again later. Choose a model based on the task and measured results, not the name of that cycle.

▸ Is a retry count enough to control an unattended agent?

No. Also limit elapsed time, spending, tool calls and simultaneous work where relevant. Stop earlier for a missing decision, forbidden action or repeated failure without a useful next step. Enforce the limits outside the instructions the agent may edit.