Model cascades: when a cheap first decision saves money
Calculate a two-stage AI workflow's complete cost, then count the relevant material its first filter misses. Compare savings without quietly lowering quality.
A model cascade sends selected work through additional models. A cheaper first stage saves money only when the work it avoids costs more than screening, extra checks and rework. Measure missed cases too: a later model cannot recover information that the workflow never gives it.
In an illustrative document review, a cheap filter sends 3,000 of 10,000 passages to a more expensive reader. The bill can fall. But when the filter wrongly excludes an important passage, the reader never gets the chance to recognise it.
“Use the expensive model only when needed” leaves the hardest question unanswered: how will the application recognise those cases? A filter that saves money by missing useful material has changed the result, not just its price. Evaluate the choice to stop as carefully as the choice to continue.
Define what the first stage may decide#
A model cascade is a sequence in which an earlier result helps determine whether another model is used. The 2023 FrugalGPT paper explores LLM cascades that trade off model cost and task performance. It establishes a research precedent, not a benchmark for the workflow below or evidence that every cheap-first design improves quality.
Our original illustrative workflow reviews 10,000 passages for possible additions to a documentation library. A first model marks each passage for deeper review or no further processing. The next model examines selected passages in more detail. Original source text is retained; “excluded” means excluded from this review pass, not deleted.
Define uncertain and failed cases separately. In this proposed design, missing comparison context or an unavailable screening service sends the passage to a waiting or review path. It does not silently become “nothing useful here.” That policy costs work, which belongs in the comparison.
A testable screening question states what information is supplied and what each outcome means. A decision model is one possible implementation of the first stage, not a requirement.
Calculate the complete bill#
Use deliberately invented prices to make the arithmetic inspectable. Assume a deep assessment costs €0.02 per passage and screening costs €0.001 per passage. These are flat costs chosen for the example, not provider quotes. Actual inputs, output lengths and billing arrangements can differ.
Sending all 10,000 passages directly for deep assessment costs 10,000 × €0.02 = €200. For this comparison, assume the baseline needs no additional charged calls. A real baseline’s retries and checks would need to be counted too.
With the cascade, screening every passage costs 10,000 × €0.001 = €10. Sending 30% onward adds 3,000 × €0.02 = €60. Assume another €15 of direct charges for retry and checking work across this batch. The total is €85, not €60.
The direct-charge difference is €200 − €85 = €115, or 57.5% of this example’s baseline. That is arithmetic under assumptions, not an expected saving. We have not included human time, shared infrastructure or maintenance in either figure.
Record those costs separately instead of treating them as free. Complete-task cost accounting explains how human effort and unsuccessful work change a comparison.
Change the forwarding rate#
Let N be the number of passages and r the fraction sent for deep assessment. Under the flat-cost assumptions, the cascade’s cost is N × screening cost + N × r × deep cost + extra charges. Count the screening fee for every passage, including ones sent onward.
For the same invented costs, the expression becomes €10 + €200 × r + €15. It equals the €200 baseline at r = 0.875: 87.5% forwarded. Below that fraction it is cheaper on direct charges; above it, it is more expensive.
For example, forwarding 95% costs €10 + €190 + €15 = €215. The extra stage has made the bill larger. The break-even point changes when passage lengths, model prices, retries or review work change; 87.5% is not a reusable routing threshold.
Try it: change how much work passes the screen
For 10,000 constructed passages, screening costs €10, deeper review costs €200 if every passage is sent, and extra retry/check charges are €15.
Invented flat charges, not provider prices or a savings forecast. Cost alone cannot show whether the screen hid important passages.
Count relevant passages that disappear#
Now give the same constructed batch a reviewed answer key. Suppose 2,000 passages genuinely deserve consideration under an agreed relevance rule. Of the 3,000 forwarded passages, 1,800 are relevant and 1,200 are not. The 7,000 excluded passages therefore contain 200 relevant passages and 6,800 irrelevant ones.
The first stage retains 1,800 of the 2,000 relevant passages: 90% recall. Recall means the fraction of relevant cases recovered. The remaining 200 are false negatives: relevant material incorrectly excluded. A low forwarding rate alone says nothing about that loss.
Suppose the deeper review retains 95% of the relevant passages it actually receives. It keeps 1,710 of those 1,800. End-to-end recall is then 1,710 divided by the original 2,000, or 85.5%. Multiplying 90% by 95% is valid here because the second percentage is explicitly conditional on the forwarded relevant passages, not an unrelated model benchmark.
Even a perfect downstream decision on every forwarded passage would leave the first 200 misses undiscovered in this pass. To find them, the workflow needs another route to the original material, such as a separate review sample. A stronger model does not repair an omission simply by being stronger.
These counts do not establish that direct deep assessment is perfect either. Measure its errors on the same answer key. Compare both routes under the same acceptable loss, rather than using the cascade’s lower bill as the verdict.
Compare quality, time and cost together#
Create repeatable cases with checked expectations, including difficult positives and plausible but irrelevant passages. Keep tuning cases separate from the final comparison. Record first-stage decisions, downstream outcomes and the passages that never reached the second stage.
Audit some confidently excluded material as well as uncertain material. Otherwise, the cases most likely to escape review are absent from the evidence about the filter. Keep the sampling method visible; reviewing only an unusual subgroup does not estimate the error rate of the entire excluded batch.
Measure elapsed time separately from cost. A forwarded passage incurs the first call before the next one can start. Parallel capacity, batching, retries and human waiting affect when the whole job finishes. A small model’s individual latency cannot stand in for that end-to-end measurement.
Keep the cascade when the complete comparison meets the task’s quality requirements at an acceptable cost and delay. Change or remove it when the first decision adds more missed material or review work than the saved processing justifies. Confidence evaluation helps test a routing signal; it does not remove the need to inspect excluded cases.
Questions people also ask
2 QUESTIONSIs a model cascade the same as a router?
Not always. A router may select one model before the task is attempted. A cascade can use an earlier model's result to decide whether more processing is needed. Designs can combine both. State which calls each input actually receives before estimating cost.
Does a faster screening model make the whole task faster?
Not necessarily. A forwarded request waits for screening and then for the next stage. A batch may finish sooner when enough expensive work is avoided, but serial latency, concurrency, review and retries must be measured for the actual workflow.