How to speed up LLM evaluation

To run LLM evals faster, make fewer slow calls and send the ones you keep concurrently. Run a curated subset day to day, put the stable rubric first so it can be cached, catch format failures in code before a judge sees them, try a smaller judge on the easy cases, and send calls in parallel or through a batch tier. Keep the full run for release gates. You don't need to change eval platforms to do any of it.

Make fewer slow calls, and stop waiting on them one at a time. Run a curated subset day to day, keep the stable rubric at the front of the prompt so it can be cached, catch format failures in code before a judge sees them, try a smaller judge on the easy cases, and send the calls concurrently or through a batch tier. You don’t need to change platforms to do any of it.

Request a pilot

Figures below come from the cited papers. They are not Slancha results.

Why an eval run is slow

A unit test runs local code. An LLM judge call is a remote request, and the judge writes its answer one token at a time. A long explanation before the verdict means a long wait for every example.

Three habits make it worse. A loop sends one call, waits, then sends the next. The same rubric goes out in full with every example. And every small prompt change triggers the whole suite, so the run that should take minutes sits in a queue.

Your existing eval or observability platform stores, compares and reviews runs. It doesn’t change how long each judge call takes. The tactics below are about the calls. Eval platforms and eval optimization do different jobs, and you can keep both.

1. Run a curated subset day to day

Maia Polo and co-authors, in tinyBenchmarks (ICML 2024), showed that 100 curated examples were enough to estimate a model’s result on MMLU, a benchmark of about 14,000 questions. Across the benchmarks they studied, about 100 curated examples per scenario landed within about 2 percent error on average.

That result is for curated subsets of public benchmarks, not a random 100 of your own tickets. Pick examples that cover your main user intents, known failure modes and edge cases. Before a subset gates a merge, check that it ranks your recent versions in the same order as the full set. Keep the full run for release gates and scheduled sweeps.

2. Put the stable rubric first so it can be cached

In a judge prompt, the instructions, rubric and worked examples are the same on every call. Only the input and the answer under review change. Major model providers offer prompt caching, which reuses the processed prefix of a prompt when the next request starts with the same text.

Put everything static at the start of the prompt and everything that varies at the end. Don’t put a timestamp, run ID or other changing value ahead of the rubric, because any change in the prefix misses the cache. Minimum cacheable lengths and prices differ by provider. The eval cost guide cites the published rates.

3. Fail obvious problems in code first

The fastest judge call is the one you skip. Many failures don’t need a model to spot. Run these checks before a judge sees the answer:

  • Validate structured output against a schema. Malformed JSON or a missing required field is a failure on its own.
  • Enforce length limits and required formats with plain string checks.
  • Match banned terms and leftover placeholders with a pattern list.
  • Compare against a known answer, exactly or after normalizing, when the task has one.

Each check runs locally and returns at once. The judge is left with the cases that need judgment.

4. Send easy cases to a smaller judge

A strong judge can stand in for human preference labels. Zheng and co-authors, in Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (NeurIPS 2023), found that their strongest judge agreed with human experts on 85 percent of non-tied votes, above the 81 percent agreement among the experts themselves.

Smaller judges are faster. Zhu, Wang, and Wang report in a 2023 paper on fine-tuned LLM judges (ICLR 2025) that their 7B judge graded 5,000 samples in 3 minutes on 8 GPUs, and agreed with the stronger judge it was trained on 81.11 percent of the time on their benchmark without reference answers. Their higher agreement figures are for the larger 13B and 33B judges.

Speed doesn’t make a small judge general. Huang and co-authors, in An Empirical Study of LLM-as-a-Judge (Findings of ACL 2025), found that fine-tuned judges did well on in-domain tests but fell behind a strong proprietary judge on generalizability, fairness and adaptability. An earlier version of the same paper (arXiv v3) described a cascade that sent low-confidence cases on to the strong judge and reported accuracy on par with it at 50 percent of the API expense. The published version doesn’t include that result.

So cascade, and calibrate. Have the small judge return a verdict and a confidence. Keep the confident verdicts and send the uncertain ones to the strong judge. Before you route a whole run this way, check agreement between the two judges on a sample of your own tasks.

5. Run calls concurrently, or batch the ones that can wait

A loop that awaits each call before sending the next spends most of its time idle on the network. Send calls concurrently through a worker pool with a cap on calls in flight. Set the cap from your request and token limits, and back off on rate-limit errors.

For runs nobody is waiting on, such as nightly regressions or a comparison before a model upgrade, use a provider batch tier. Batch jobs trade a slower turnaround for a lower price and their own capacity, so they don’t compete with production traffic. The eval cost guide cites one provider’s published batch terms.

Where Slancha fits

Slancha pilots aim to make the expensive part of evals faster and cheaper, so you can spend less per eval and run one whenever a change calls for it. Keep your existing eval or observability platform.

Pilots by arrangement.

Request a pilot

Estimate a run from your own inputs, or read how to cut what each run costs.

Questions

How do I speed up LLM evaluation?

Make fewer slow calls, and stop waiting on them one at a time. Run a curated subset day to day, keep the stable rubric at the front of the prompt so the provider can cache it, catch format failures in code before a judge sees them, try a smaller judge on the easy cases, and run the calls concurrently or through a batch tier.

Why are evals so much slower than unit tests?

A unit test runs local code. An LLM judge call is a remote request that generates its answer one token at a time, and the rubric is sent again with every example. A loop that waits for each call before it sends the next adds those delays up.

Will a smaller set of examples give the wrong answer?

It can if the set is random or too small for the change you are testing. Maia Polo and co-authors (tinyBenchmarks, ICML 2024) found that about 100 curated examples per scenario estimated results within about 2 percent error on average on the benchmarks they studied. Check that your subset ranks your own versions in the same order as the full set before you rely on it.

Can a smaller judge replace the one I trust?

On the tasks it was tuned for, it can do well. As a general substitute, no. Huang and co-authors (Findings of ACL 2025) found that fine-tuned judges did well on in-domain tests but fell behind a strong proprietary judge on generalization, fairness and adaptability. Send the confident cases to the small judge, send the uncertain ones to the strong judge, and check agreement on a sample of your own tasks.

Does running calls in parallel hit rate limits?

Unbounded parallelism will. Cap the number of calls in flight so you stay under your request and token limits, and back off when the provider says to. For runs that can wait, use a batch tier instead. The cost guide cites one provider’s published batch terms.

Do I need to replace my eval platform to make evals faster?

No. Keep your existing eval or observability platform for datasets, traces and review. The tactics here change how many judge calls you make and how you send them.

Sources