Turn vague AI requests into decisions you can test

An overall usefulness score does not tell an editor whether a paragraph adds an example, corrects a mistake or repeats an existing explanation. It also hides whether the model had the existing explanation in the first place. Start with the decision the next person or program needs to make.

Replace useful with a checkable question#

Consider an original illustrative documentation task. The existing note says, “The demonstration cache holds at most 100 entries.” A candidate paragraph says, “The demonstration cache holds at most 100 entries. It does not limit the total bytes stored.” These sentences are invented here to demonstrate comparison, not quotations from a product manual.

Instead of “Is this valuable?”, ask: Does the candidate state a limitation that the supplied note does not state? Define a positive result as a new limitation in the candidate, not merely a different wording of the same fact. For these two texts, the byte-limit statement is a candidate addition.

That answer does not establish that the byte-limit statement is true of an actual implementation. Novelty relative to a note and factual correctness are different questions. The editor would need implementation evidence or an appropriate public source before publishing the claim.

TypeSafe recommends narrowly scoped questions and composing their results in code. Our documentation example applies that idea; it is not a claim that the proposed questions are validated for Jev or that decomposition guarantees a better answer.

Supply the evidence needed for that question#

Give the comparison step the candidate text, relevant existing text, their identifiers and versions, and the precise question. In this proposed contract, those are named input fields. The model should not have to infer which paragraph is current guidance and which is a proposed addition.

TypeSafe’s state documentation describes supplying related material together and separating that material from the questions. It also says that questions in one request see the same state. That interface does not imply access to documents the application did not supply.

An ordinary input check can establish that comparison text is present. It cannot establish that the retrieved text covers every relevant page. Record retrieval separately: a successful search that found no matching text differs from a search failure or a passage lost through truncation.

Use context preparation to select and preserve the relevant evidence. Keep the output claim narrow: “not stated in these supplied excerpts” is supportable by that comparison; “never explained anywhere in the library” requires broader coverage.

Try it: change the comparison evidence

Does a candidate paragraph add a missing CSV-export condition? Current guide supplied; condition is absentCurrent guide supplied; condition already presentNo current guide supplied

The candidate adds a condition absent from the supplied guide.

That is a checkable comparison against this version, not proof of novelty across every guide.

Constructed example; no model or library search runs here.

Separate facts that can be true together#

A paragraph can repeat one fact and add another. It can also add an example while contradicting an existing rule. A single choice between “duplicate”, “example” and “contradiction” loses information when several meanings apply.

For this task, ask separately whether the candidate adds a limitation, adds an example, or conflicts with a statement in the comparison text. Each question can have present, absent and needs_context outcomes with explicit definitions. These are our proposed application outcomes, not built-in Jev categories.

Keep a missing prerequisite separate from a negative answer. With no existing text available, the application cannot evaluate whether a condition is absent from that text. Return a blocked comparison or fetch the missing material rather than guessing that the candidate is new.

Questions evaluated separately against the same input do not automatically receive one another’s answers. If a later comparison needs the result of an earlier extraction, arrange that dependency in the workflow or supply the extracted material first. Also, separate questions do not imply statistically independent errors: both can misread the same passage.

Write examples before tuning the model#

For the limitation question, agree on these illustrative cases before choosing a model or changing instructions. The comparison target is the supplied note, not undocumented knowledge of a real cache.

The last case changes only the comparison text. It checks whether the model actually compares the two inputs rather than classifying the candidate in isolation. A useful answer must be allowed to change when the evidence changes.

If reviewers disagree about the expected answer, inspect the definition before calling the model wrong. Perhaps “limitation” includes resource consequences for one reviewer but not another. Clarify the criterion, record the change and recheck affected cases instead of silently changing the answer key to match the model.

Maintain a versioned evaluation set with the complete inputs and acceptable outcomes. Keep separate cases for final evaluation after tuning. Check output meaning, not just the presence of one of the allowed strings.

Keep the next action in the application#

In our proposed workflow, a new limitation creates a review candidate with references to both texts. A possible contradiction sends both claims for factual checking. Missing context triggers retrieval or a saved blocker. None of those outcomes publishes an article automatically.

A decision model that returns only choices does not thereby write the candidate explanation or quote the supporting sentence. Preserve the original passages for the next stage. A separate extraction or writing step can be added when needed, with its own checks.

Compare this design with a simpler baseline on the same cases. Extra questions are worthwhile when they make the next action clearer or improve measured results, not because more model calls look thorough. The cascade walkthrough shows how to count both their cost and the useful material a first-stage decision might exclude.

Questions people also ask

2 QUESTIONS

▸ Must every task be split into many tiny questions?

No. Split questions when the answers mean different things or lead to different actions. Extra questions add work and can share the same mistaken assumptions. Keep a single question when it already has a clear, testable meaning.

▸ Can a model decide whether a passage is new to an entire website?

Only within the evidence actually available to it. A comparison with selected excerpts supports a narrower conclusion than a verified comparison with every relevant page. Keep retrieval coverage and the model's comparison result separate.