Ephemeral environments and seed data

Ask what an agent needs to do a piece of work and the answer sounds cheap: a copy of the repository and a branch to work on. Then the first integration test runs, finds no database, and the plan stops. What a run needs is not a checkout. It is a running slice of the system, and standing that slice up is engineering with its own schedule rather than a line in somebody’s setup script.

The environment is the expensive part#

The shape that works is unglamorous. A container is created for one run, from an image that already carries the toolchain — compiler, package manager, test runner, database client — so nothing is being installed while the clock runs. It clones the repository and checks out the branch. It receives its credentials at run time rather than carrying them in the image, which is the only arrangement under which a credential you can narrow and revoke means anything. When the run ends the container is destroyed, and nothing survives it.

Every property you want from a fleet falls out of that last clause. Runs are independent, because no run can leave state behind for the next one to find. They are parallel, because there is nothing to queue behind. A run that goes badly wrong — a dependency half-installed, a migration half-applied — cannot contaminate its successor, because it has no successor in that container. The alternative is a long-lived worker reset between jobs, which works until a reset misses something, and then produces a class of failure nobody can reproduce on purpose.

What gets underestimated is the word toolchain. Unit tests want a compiler. Integration tests want a database, and usually a message broker, an object store, and a stand-in for whichever third party the code talks to. Reproducing a reported defect wants all of that plus data shaped like the data the defect happened in. So the environment is not the container; the container is the wrapper around it. The environment is a runnable slice of the system, and on a brownfield estate of twenty-odd services, deciding how thin that slice can be while still failing honestly is design work with a schedule attached.

It is also where published success rates stop transferring. The agent benchmarks people quote are Python-heavy, and Multi-SWE-bench — which puts the same task shape across several languages — finds resolve rates on Java and others falling materially below Python. Some of that is the model. Some of it is that the stacks whose environments are slowest and most awkward to stand up are the same stacks where a run has more ways to fail before the agent has read a line. A number measured where the environment boots in seconds is not a forecast for one where it does not.

A database per run, not a database for everyone#

Give every run its own database, started from a container image and thrown away with the container. Not a schema inside a shared instance, not a tenant, not a set of tables with the run identifier in their names. A separate database process, alive for one run and nobody else’s.

Two things rest on that, and both are load-bearing. The first is the step that makes a bug-fixing agent worth trusting at all: reproduce the failure before changing anything, then show the same reproduction passing afterwards. Against a database somebody else is writing to, that reproduction is evidence of nothing — the failure may have appeared and vanished for reasons unrelated to the code, and the agent will happily attribute the change to its own fix. The second is parallelism. A shared test database serialises the whole fleet, and it does so invisibly: nobody announces the constraint, runs simply start interfering, the interference is filed as flakiness, and somebody adds a queue to make the flakiness stop. The factory is now single-file and nobody decided that.

The bill for this is wall-clock time on every run, and the mistake is to file it under overhead. It is not overhead around the unit of work, it is part of the unit cost, and it is charged whether the run produces a merged change or an apology. Teams meter tokens because tokens arrive as an invoice with the word on it, and quietly do not meter the compute that ran for six minutes before the agent said anything. Then someone asks what a merged change actually costs and the answer gets assembled from one measured number and one guess.

There are levers, and every one of them is a trade. A slimmer image starts faster and carries fewer tools. A shallow clone is quicker and blinds anything that needs history. A database restored from a snapshot skips the migrations and stops proving that the migrations work. A warm pool of pre-built containers removes most of the wait in exchange for some of the isolation the pool was built to protect. Choose between them on measurements rather than taste, and meter them beside the tokens — because on a stack where startup dominates, a cheaper model saves you almost nothing.

Seed data is where this goes wrong#

The genuinely hard constraint is not the container. It is what goes in the database, and the tempting answer — a copy of production, scrubbed a bit — is the one to refuse.

Pseudonymised production data is still personal data under the GDPR. Swapping names for tokens lowers the risk; it does not change what the data is, because the substitution is reversible by whoever holds the mapping. A production copy therefore carries every obligation the production database carries, into an environment built to be disposable and treated accordingly. That is the legal half. The engineering half is worse: real customer records now sit inside a process that reads untrusted text — issue descriptions, review comments, dependency documentation, stack traces from a third party — and can reach the network. That is exactly the arrangement that turns an injected instruction into an exfiltration path instead of a rude comment.

Data generated to match the real schemas avoids both problems at once. It has no data subject, so it can go anywhere a container can go. It lives in the repository, versioned alongside the code that reads it, so a schema change and its seed change arrive in the same review. And it has a property nobody values until they need it: a person can read it. Nobody reviews a production dump. Somebody can read a seed file that says this customer has no delivery address and this order has forty lines, disagree with it, and add the case that got through last month.

Four ways this decays, none of which announce themselves. The first is the shared integration database everybody agrees is temporary. It exists because a run needed somewhere to point on a Thursday, and eighteen months later it is the reason nothing can run twice at once, and it holds hand-written rows that a dozen tests silently depend on. The second is synthetic data that drifts: the schema moves, the generator does not, the seed still loads because the migration filled the new column with a default, and the suite goes green against a shape of data the system stopped seeing in spring.

The third is a seed set too thin to fail. Three neat rows per table, every foreign key satisfied, no nulls, no long strings, no unusual characters, no record with a history behind it. Every agent run passes and every real request finds the case nobody seeded — and because the runs are green, the factory keeps producing confident changes at volume, which is a worse outcome than producing none. The fourth is the restore. Somebody needs to debug one thing, production has the data, the dump takes twenty minutes, and the exception becomes the practice. It never gets undone, because nothing breaks when it stays.

All of it is infrastructure that has to be scheduled rather than assumed, and it is the reason a factory takes months and not a fortnight. Once each run can be created and destroyed on its own, though, the question changes shape: something still has to decide which run happens when, and whether that needs an orchestrator or the pipeline you already have is the next thing worth arguing about.