Blog

How to build agent evals from production traces

To build agent evals from production traces, keep the full trace, pick cases on purpose, and write down what the agent should have done instead of copying what it did. Check the end state and the tool calls in code first, use a judge only for what code cannot check, and repeat each case several times before you compare versions. Hold back part of the set so a fix is tested on cases it was not tuned on. The figures on this page come from public papers, not Slancha results.

Your traces already hold the cases an agent gets wrong. The work is turning them into tests whose expected answer comes from a person or a rule, not from the run you are trying to improve, and doing it without paying for a judge call on every step of every trace.

By Slancha. Published .

Why traces make good test cases

A trace records one real run: the request, each model call, each tool call with its arguments and result, and the final reply. Real traffic shows what users actually ask, which a developer's list of imagined cases rarely does. It also shows where the agent went wrong.

A trace is not an answer key. It says what the agent did, not what it should have done. If you paste the old reply in as the expected answer, you freeze today's behavior, bugs included, and every later improvement reads as a regression.

1. Keep the whole trace, and scrub it

Store the input, every model and tool call with its arguments and result, and the versions of the prompt, model and tools that produced the run. Without versions you cannot tell whether a change in results came from your edit or from something else. Give each run an ID that follows it through every step, so a failing case leads back to the exact run.

Remove personal data and secrets before a trace becomes a test case, and keep access to the raw traces narrower than access to the test set. Scrubbing is never perfect, so have a person look at every case you keep.

2. Pick traces on purpose

A random slice of traffic is mostly easy successes. Build the set from several groups:

  • Failures and complaints: runs a user reported, retried or handed to a person.
  • Rare tool paths: tools or argument shapes that appear in few runs but matter when they do.
  • High-risk actions: anything that changes data, spends money or sends a message.
  • A spread of ordinary runs, so you can see that a fix does not break the common case.

Remove near-duplicates so one popular request does not fill the set. Then split it: use one part while you work on the agent, and keep the other part untouched until you want to know whether a fix holds. If you tune against every case, the set stops telling you anything.

3. Write the expected behavior, not the old output

For each case, decide what a good run must do. A person who knows the task should write it, because only they know what the agent should have done. The paper Who Validates the Validators? (Shankar and co-authors, UIST 2024) describes criteria drift: people need criteria to grade outputs, but grading outputs is how they work out the criteria. Expect to revise your expectations as you read more traces, and keep them in a versioned file a reviewer can read.

Write expectations as checks on the outcome and the steps, not as exact wording:

  • The end state: the record that should have changed, and nothing else.
  • Tool rules: tools that must be called, tools that must not be, and a cap on how many calls the task needs.
  • Required facts in the reply, such as an amount the user asked for.
  • A short rubric line only for what is fuzzy, such as tone.

τ-bench (Yao and co-authors, 2024) uses this shape for agents that talk to users and call tools. It compares the database at the end of a conversation with the goal state an annotator specified, so many different conversations can reach the same end state, and a wrong write fails.

4. Check in code first, and judge what is left

Most agent failures do not need a model to spot. Validate structured output against a schema, compare the end state to the expected one, check tool names and arguments, and enforce limits on calls, steps and time. These run locally and return at once. Keep the LLM judge for the parts code cannot check, such as whether an explanation was clear or a refusal was polite.

That order is also where the cost goes. A judge call that has to read a long trace is the slow, expensive step, so every case code can settle is a judge call you skip. The speed guide and the cost guide list more tactics, such as caching the stable rubric and sending runs that can wait through a batch tier.

5. Look at steps, not only the final answer

A plausible final reply can hide a wrong tool call earlier in the run. The Agent-as-a-Judge paper (Zhuge and co-authors, 2024) argues that checking only final outcomes ignores the step-by-step nature of agents, and proposes judging intermediate steps too. Rules on tool calls give you much of this in code. Where a step needs judgment, give the judge that step and its context, not the whole trace, so each call stays short.

6. Run each case more than once before you compare versions

An agent can pass a case once and fail it the next time. τ-bench reports pass^k, the chance that an agent solves a task on all k repeated tries, averaged over tasks. The authors report that even the state-of-the-art function-calling agents they tested succeed on fewer than 50 percent of tasks, and that in the retail domain pass^8 falls below 25 percent. An agent that often succeeds once can still fail often when the same task repeats.

So run each case a few times and record how often it passes. Compare two versions on the same cases, and look at which cases changed, not only at the average. The sample-size guide shows how many independent cases a given margin needs, and why a paired comparison on the same cases beats two separate error bars.

7. Keep feeding the set

Each new failure you fix becomes a new case, so the same bug cannot return unnoticed. Review the set now and then, drop cases that no longer match how the product is used, and re-read the expectations when the task changes. Rerun the whole set before a model or prompt change reaches users, using a batch tier if nobody is waiting.

Where Slancha fits

Slancha aims to make the expensive part of evals faster and cheaper. For agents, that part is the judge reading long traces. You keep your current eval or observability platform for storing traces and reviewing runs. Pilots by arrangement.

Request a pilot

Related: how to cut what each run costs and the eval cost calculator.

Questions people ask

How many production traces do I need to start?

There is no fixed number. Begin with the cases you already know are wrong, add one for each new failure, and use the sample-size guide on this site to see how wide the margin is for the size you have. A small set can catch a repeat of a known failure. It cannot show a small difference between two versions.

Can I use the old agent output as the expected answer?

Not as the answer. The trace shows what the agent did, so using it as the expected output freezes its mistakes. Have a person write what a good run must do, as checks on the end state, the tool calls and the required facts.

Should an LLM judge read the whole trace?

Only when it has to. Settle what you can in code first. Where a step needs judgment, give the judge that step and the context it needs rather than the full trace, because a judge call that reads a long trace is the slow, costly part of a run. Check the judge against your own labeled examples before you trust it; the cheaper-judge guide on this site shows how.

How do I handle personal data in traces?

Remove personal data and secrets before a trace becomes a test case, keep raw traces behind narrower access than the test set, and have a person review each case you keep. A scrub is never perfect, so do not treat a scrubbed trace as risk free.

Do I need to replace my eval or observability platform?

No. Keep it for storing traces, datasets and review. The steps here are about which cases you keep and how you check them.

Sources