Skip to the content
Software Made Clear Diagrams that show the mechanism About

Distributed tracing

ANSWER

The whole mechanism is one identifier that survives every hop, plus a second saying who called you. Standardising those two is what lets tools from different vendors reassemble a request none of them saw the whole of.

IN PLAIN TERMS

A parcel keeps its original tracking number through every depot it passes, while each depot also stamps on which van it just came off. The first number says which parcel this is. The second is what lets someone rebuild the route afterwards. A request is followed the same way.

A request crossing a fleet of independently deployed services produces that many sets of logs, and no service can answer the question anyone actually has, which is what happened to this request. Distributed tracing solves that with a mechanism far smaller than the tooling around it suggests — one identifier that has to survive every hop, and a second that records who made the call.

What the header has to carry#

The W3C Trace Context specification defines a traceparent header with four dash-separated fields: a version, a 16-byte trace-id, an 8-byte parent-id, and a byte of trace-flags. Its own example is 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01, and everything a tracing interface later shows you is rebuilt from those four values as they arrive at each service.

The two identifiers do different jobs, and the difference is the design. The trace-id is fixed for the whole request and copied unchanged into every outbound call, which is what makes all the work one story. The parent-id changes at every hop — each service replaces it with an identifier for its own unit of work before calling the next one — which is what makes the story a tree rather than a list. Without the first there is nothing to group by. Without the second you know nine things happened and not which of them caused the others.

A second header, tracestate, carries vendor-specific key-value pairs alongside the standard fields. It exists so that a vendor can propagate its own data without changing the part everyone has agreed on — the boundary between what is shared and what is proprietary, drawn in the protocol rather than left to convention.

Why it had to be standardised#

The specification is unusually direct about the problem it exists to solve: traces collected by different tracing vendors cannot be correlated, because there is no shared unique identifier. Before agreement on a format, propagation worked perfectly within one vendor’s instrumentation and stopped at the first service somebody else’s tooling had touched. A trace ended not where the request ended but where the tooling changed.

That is a strange kind of problem for a technical standard, because there is nothing difficult about the format — it is two identifiers in a header. The hard part was agreeing which header and which layout, and the value delivered is interoperability rather than capability. Standardising it means a request can cross a boundary between two organisations’ instrumentation and stay one trace.

One hop drops the header#

The failure that produces useless data rather than no data is a service that receives the header and does not pass it on. Everything downstream of that point starts a fresh trace, so the picture shows a request that ends where it did not end and several unrelated ones beginning at the same instant. Nothing errors, the traces look plausible individually, and the break is only visible if you already know the call happened — which is the same shape as any other gap that is invisible from inside the system that has it.

The specification anticipates a subtler version and rules on it. All bytes zero is an invalid trace-id, and vendors must ignore a traceparent carrying one — the same for the parent-id. Without that rule, a field left at its default would be indistinguishable from a genuine identifier, and every service that failed to populate it would file its work under one enormous shared trace. Making the zero value invalid converts a silent data-corruption bug into a header that is simply discarded.

The last one is sampling, which is a single bit in the flags and a large operational decision. Tracing every request at full volume is usually impractical, so a fraction is kept — and the fraction has to be chosen consistently along a request, much as a decision taken once has to be honoured by everything downstream, or a trace ends up with holes where one service decided to record and another did not. That is why the sampled bit travels in the header with the identifiers rather than being decided independently at each hop.

IF YOU REMEMBER ONE THING

One identifier that never changes, one that changes at every hop. The first makes the work one trace; the second makes it a shape — and a service that drops the header quietly deletes everything after itself.

Questions people also ask

3 QUESTIONS
What is actually in a traceparent header?

Four dash-separated fields: a version, a 16-byte trace-id, an 8-byte parent-id identifying the calling span, and a byte of flags whose only defined bit today records whether the trace was sampled. The example in the W3C specification is 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01. Everything else a tracing system shows you is reconstructed from those pieces arriving at each service.

Why does the specification call an all-zero trace-id invalid?

Because a default-initialised field would otherwise look like a real identifier. The spec states that all bytes as zero is an invalid value and that vendors must ignore the traceparent when it appears, which turns a silent bug — thousands of unrelated requests collapsing into one trace — into a header that is simply discarded. The same rule applies to the parent-id.

Does distributed tracing need every service to use the same vendor?

That was the problem the standard was written for. The specification names it directly: traces collected by different tracing vendors cannot be correlated because there is no shared unique identifier. Agreeing on the header format is what allows one hop instrumented by one tool and the next by another to end up in the same trace.