How to reduce LLM evaluation cost

The bill is why the eval does not get run

A full judge pass repeats the same rubric on every example, and often more than once. Teams then skip the run. The expensive moment is a model upgrade: something changed, and nobody can say whether the product did. Sampling makes that worse. One answer from one prompt is a weak measurement of a system that does not return the same text twice.

An eval platform stores and reviews that run. It does not, by itself, make the judge call cheap. The tactics below are about the call. Eval platforms and eval optimization do different jobs, and you can keep both.

1. Judge a curated subset, not the whole set

To accurately estimate the performance of an LLM on MMLU, a popular multiple-choice QA benchmark consisting of 14K examples, it is sufficient to evaluate this LLM on 100 curated examples.

That is Maia Polo and co-authors, tinyBenchmarks (ICML 2024). Across the benchmarks they studied, about 100 curated examples per scenario estimated performance within about 2 percent error on average. The paper is about those benchmarks and about curated examples, not a random 100 of your production tickets.

Use it as a reason to measure how small a set can be and still rank the change you care about. A pre-upgrade check on a curated slice is a run you can afford to repeat. Price that slice in the eval cost calculator.

2. Cache the rubric, not the example

The instructions and the rubric are the same on every judge call. The example is not. On the OpenAI pricing page (16 October 2026), short-context standard rates for gpt-5.4 are $2.50 per million input tokens and $15 per million output tokens. Cached input for that model is listed at $0.25 per million, one tenth of the input rate. gpt-5.4-mini is listed at $0.75 input, $0.075 cached input, and $4.50 output. Those cached rates apply to tokens the API marks as cached. Output tokens are unchanged.

Anthropic’s prompt-caching docs price a cache read at 0.1 times the base input price on the standard multiplier, with lower multipliers on some models. A 5-minute cache write is 1.25 times the base input price, and a 1-hour write is 2 times. A prefix used once does not pay for itself. Minimum cacheable lengths apply and differ by model; the docs are the source.

Put the stable rubric first. Leave the example outside the cached prefix. Do not treat a hoped-for hit rate as a saving until you measure it.

3. Use a batch tier when the run can wait

A regression check before a model upgrade does not need a synchronous reply. The OpenAI Batch API lists a 50 percent cost discount compared with the synchronous APIs, a separate pool of higher rate limits, and completion within 24 hours. That fits a nightly or pre-merge eval. It does not fit a live user request.

Halving the token price in the calculator is the right shape for this discount. Confirm the rate on your own account before you budget it. Other providers publish their own batch terms.

4. Try a smaller judge, then check it on your tasks

gpt-5.4-mini’s short-context standard rates above are 30 percent of the gpt-5.4 input and output rates on the same pricing page. That ratio is arithmetic on the listed prices. It is not a quality result.

Strong LLM judges like GPT-4 can match both controlled and crowdsourced human preferences well, achieving over 80% agreement, the same level of agreement between humans.

Zheng and co-authors wrote that in Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (NeurIPS 2023). On non-tied MT-Bench votes, their GPT-4 judge agreed with human experts 85 percent of the time, above the 81 percent agreement among those experts. A strong judge can stand in for a preference label. A smaller model does not inherit that result.

Zhu, Wang, and Wang, JudgeLM (ICLR 2025), report 81.11 percent agreement with GPT-4 for JudgeLM-7B on their no-reference benchmark. The same paper’s higher agreement figures are for larger judges and more training data, not for the 7B model.

Huang and co-authors, An Empirical Study of LLM-as-a-Judge (Findings of ACL 2025), found that fine-tuned judges did well on in-domain tests but fell behind a strong proprietary judge on generalizability, fairness and adaptability. An earlier version of the same paper (arXiv v3) described a cascade that sent low-confidence cases on to the strong judge and reported accuracy on par with it at 50 percent of the API expense. The published version doesn’t include that result. The practical tactic is the same as the paper’s caution. Keep a sample of your own tasks, compare the smaller judge with the one you trust, and route the uncertain cases back until that sample agrees.

5. Repeat a small run when the answer moves

Two checks are easy to skip when each judge call is expensive. The first is the model upgrade: run the same set before and after the change. The second is consistency: run the same item more than once, because a single sample cannot show whether the output is stable.

Zheng and co-authors also document position bias in pairwise judging. A swap of the two answers is a second judge call. Fairer comparisons cost more calls, which is why a smaller set and a cheaper call belong together. The calculator’s runs field is where you price that repetition. It does not tell you how many runs your task needs.

Where Slancha fits

Slancha pilots aim to work alongside the eval stack you already run. Keep the eval platform where the run lives. The work here is to make the expensive part of that run cheaper and faster, so the eval happens.

Pilots by arrangement. Bring a scrubbed trace, the relevant source, representative tasks and quality criteria to a conversation. The scrub is not perfect. A person inspects the evidence before anything is adopted.

Estimate a run from your own inputs, or read how the platforms differ. If the run is too slow rather than too expensive, read the eval speed guide.

Sources

• tinyBenchmarks, Maia Polo and co-authors, ICML 2024. 100 curated examples, and about 2 percent average error on the benchmarks they studied.
• OpenAI API pricing, retrieved 16 October 2026. gpt-5.4 and gpt-5.4-mini short-context standard rates, including cached input.
• OpenAI Batch API. 50 percent discount versus synchronous APIs, completion within 24 hours.
• Anthropic prompt caching. Cache reads at 0.1 times base input on the standard multiplier; writes at 1.25 times (5 minutes) or 2 times (1 hour).
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Zheng and co-authors, NeurIPS 2023.
• JudgeLM, Zhu, Wang, and Wang, ICLR 2025.
• An Empirical Study of LLM-as-a-Judge, Huang and co-authors, Findings of ACL 2025. The cascade result is from version 3 of the preprint.

Related guides

• How to build agent evals from production traces
• How much do LLM evals cost?
• How many examples does an LLM eval need?
• Can a cheaper LLM judge replace an expensive one?
• Eval speed guide
• Eval cost calculator