Incidents back into the backlog

A production error that fires ten thousand times overnight is not ten thousand problems. It is one problem with a very loud voice, and until something in the pipeline says so, the response to it is a person reading a dashboard at seven in the morning and deciding by eye what is new. That decision is mechanical and almost entirely pattern-matching — and it is also the decision that determines whether an automated feedback path helps at all, because an agent that files what it reads will file ten thousand items.

One fault is one piece of work#

This is the return arrow made concrete. A factory has an inner loop that turns a ready work item into a merged change, and an outer loop that decides what the inner loop should be building next — and the outer loop only closes if what production learns comes back as planned work rather than as a message in a channel. That is the arrow the two-loop picture draws and then does not explain. What follows is the mechanism behind it.

Two small details come first, and they are the difference between this working and it either spamming or missing things. The agent polls the error tracker from a stored cursor — a durable marker for how far it has already read — so a run does not re-process what the last run handled. And it reads back from a little before that marker, an overlap window of a few minutes, because events arrive out of order and a cursor advanced to the exact boundary quietly drops whatever lands late. Duplicates from the overlap are harmless; the next step removes them anyway. A gap at the boundary is not, because nothing ever reports it.

Then the ordering: fingerprint each event and collapse the duplicates, assess severity against frequency, map the top stack frame to a service and a file, search the backlog for an item already carrying that fingerprint, and create or update exactly one prioritised defect.

The word carrying the weight in that sequence is update. Most implementations are written as a create, and a create-only path has no way to say we already know — so the second occurrence of a known fault produces a second item, and by Friday the backlog holds more copies of one defect than it holds defects. The item that gets written, new or updated, carries the stack trace, the route, the frequency, the release and whatever evidence exists for reproducing it. The rest is decoration.

What the fingerprint is made of#

A fingerprint is a short string derived from an event such that two events describing the same fault produce the same string. Every other design question in this agent is downstream of getting it right, so it deserves stating in full: the error type, a normalised stack location, the route or operation it occurred on, the release it occurred in, and the status of any work item already holding that fingerprint.

Normalised is doing real work in the second of those. A raw stack frame carries a line number that moves every time somebody adds an import above it, and an absolute path that differs between a container and a developer’s machine. Fingerprint on the raw frame and one unrelated edit turns a known fault into a new one; normalise to the module and the function and it survives ordinary churn. Overdo the normalisation — strip down to the file alone — and two genuinely different faults in the same file become indistinguishable.

The last two ingredients are the ones people leave out, and they matter more than they look. Including the release makes the same error before and after a fix two different things. Without it, a fault that returns three releases later lands silently on the old item and reads as leftover noise; with it, the return files its own item and names the release that reintroduced it — a large part of the investigation, done before anyone opens an editor. Including the status of any existing item stops two symmetrical mistakes: a closed item reopened by a straggler that was in flight when the fix shipped, and a fresh occurrence appended to something already marked done. A closed item plus a new occurrence in a later release is a regression, and should say so in an item of its own.

What it costs, and where it goes wrong#

Production logs are enormous, and the naive shape of this agent — hand the model the log and ask what happened — is the most expensive thing you can build. The fix is unglamorous. A plain deterministic script slices the logs before any model reads anything: by time window, by request identifier, by service, by level. The model then reads a fragment measured in kilobytes instead of the volume it was cut from. This is the clearest case in a whole factory of the split between what a script should decide and what a model should, and it is where the bill is genuinely won rather than trimmed.

The security shape follows from what the agent handles. It reads text an attacker can influence — a stack trace can contain a request parameter, and a request parameter can contain instructions — and it holds real access to system internals. Two of the three legs of an exfiltration path are present by design and cannot be removed, so the leg you give up is the third: read-only against production, and no outbound network route except to the tracker it writes to. That is the same reasoning applied elsewhere to content an attacker can reach into the pipeline, and it bites harder here than anywhere, because the untrusted content is the input.

A separate obligation is easy to skip. Production logs routinely contain personal data — an email address in a validation error, a customer identifier in a parameter, an address in a serialised payload — and it has to be redacted before anything is written into a work item. A defect ticket is long-lived and widely readable, so a customer’s details in one are a disclosure that outlives the incident by years. Redaction belongs in the deterministic slicing step, before the model sees the fragment, not in a rule asking the model to be careful.

Four ways this goes wrong, in rough order of how often. A fingerprint too narrow: one fault opens forty items because the line number moved, and the backlog becomes unusable in a fortnight. A fingerprint too broad: three unrelated faults merge into one item with three stack traces and no coherent reproduction, which nobody can pick up. A cursor that resets on redeploy, because it lived in memory or in a container’s local disk, and re-files six months of history in one run — the failure that gets the feature switched off the same afternoon. And the item that is created but never given enough evidence to reproduce: no route, no release, no payload shape, just a type and a count. That is a notification wearing a work item’s clothes, and it costs a person the triage they thought they had automated.

Three measures tell you which of those you have. The de-duplication ratio — events in, items out — catches both fingerprint failures, in opposite directions. Mean time from first occurrence to work item says whether the loop closes fast enough to matter. And the false-item rate, the proportion of filed items a person closes without action, is the one that decides whether the team keeps the feature switched on; track it from the first week rather than from the first complaint. All three read better against what a single merged change costs to produce, because an item nobody acts on has still consumed a slice of that.

One last thing, worth being blunt about: this agent is dependency-heavy and belongs late. It needs error tracking, aggregated logs and deployments correlated to releases, and it needs all three to be trustworthy rather than merely chosen — a fingerprint built on a release field nobody maintains produces confident nonsense. It looks like an early win because the value is so legible. It is not one. Build the deterministic slicing script first, point it at a person, and let it earn its keep before anything writes to the backlog on its own.