Evals for agent output

Before you start

• An agent with written instructions whose answers affect something: routing, decisions or code.

• A list of the failures already seen or feared.

• Two ways to run the agent: the real setting and a fast preflight through the API.

Steps

1. Pick the failures that have consequences

   The suite tests the smallest set of distinct failures with a material consequence. Each base case tests one decision boundary: for example whether the agent asks the owner or decides itself.

   Check: Every case names the wrong answer it catches.

2. Write self-contained prompts

   A prompt carries everything the right answer needs. The answer must not depend on hidden state or an assumed authorisation; an ambiguous prompt is rewritten rather than explained to the grader.

   Check: An outsider could grade the answer from the prompt and the expected answer alone.

3. Consolidate near-duplicate cases

   Two cases that fail for the same reason are one case. Rare variants move out of the base suite as optional ones.

   Check: Removing a case would lose one failure.

4. Repeat adaptively

   The smoke set runs twice. Only a case whose two runs disagree gets a third, deciding run. The full suite runs once when the instructions change significantly. A passing case is not rerun to make more evidence.

   Check: Every run is justified by a change or a disagreement.

5. Test in the real setting, preflight fast

   The real setting is the integration test; the suite run through the API is a fast preflight. A green preflight does not replace a run in the real setting.

   Check: Published instructions have been run in the real setting.

6. Change the instructions only on repeated evidence

   The instructions change when a repeated run shows a defect in the instructions rather than in the test case. The change records the runs it rests on, and the smoke set runs again after it.

   Check: Every instruction change points at runs.

When not to use this

• Output a program can check directly, such as code or data: write a test, not an eval.

• A one-off prompt that will not be maintained.

• The grader is the same model with no written expected answer: the suite then measures the model's opinion of itself.

Where it comes from

The suite began as a regression test for the instructions of an AI team's control room: prompts, expected answers, smoke runs and deciding runs. The same idea recurs as blind-reviewed benchmarks in the Searchint work.