How much do LLM evals cost?
LLM evals are expensive because every run repeats a judge call for every example and every criterion: cost per run is calls × tokens per call × price per token, multiplied by the number of runs. At the illustrative prices on this page, one 6,000-call run costs $2.07 to $138, and public benchmarks cost far more (the HELM paper reports $85 to $10,926 per model through paid APIs). To make them cheaper, judge a smaller, well-chosen set, check a cheaper judge on your own labels, cache the stable rubric and batch the runs that can wait.
An eval costs judge calls × tokens per call × price per token, repeated for every run. At the illustrative prices below, one 6,000-call run costs $2.07 to $138. Public benchmarks cost far more: the HELM paper reports $85 to $10,926 per model through paid APIs.
Updated
The formula
Cost per run = calls × (input tokens × input price + output tokens × output price).
Calls = examples × judge calls per example × runs. Judge each criterion separately and every criterion is one more call. The model under test adds its own calls on top.
A worked example
Assumptions: 2,000 examples, 3 criteria judged separately (6,000 calls per run), 1,500 input and 200 output tokens per call, 50 runs a month.
| Price tier (illustrative) | Per million tokens, input / output | Per 1,000 calls | One run | 50 runs |
|---|---|---|---|---|
| Small | $0.15 / $0.60 | $0.345 | $2.07 | $103.50 |
| Mid | $1 / $4 | $2.30 | $13.80 | $690.00 |
| Large | $10 / $40 | $23.00 | $138.00 | $6,900.00 |
Tiers are illustrative, not any provider's price list. For published rates see the cost guide, or change the inputs in the calculator.
What public benchmarks cost
- HELM (Liang and co-authors, TMLR, August 2023): running its 42 scenarios cost $85 to $10,926 per model for the paid-API models it tested, and 540 to 4,200 GPU hours for open models.
- Efficient Benchmarking (Perlitz and co-authors, NAACL 2024): on HELM, a procedure cut computation by 100× or more with minimal loss of reliability in the model ranking.
- Holistic Agent Leaderboard (Kapoor and co-authors, October 2025): about $40,000 for 21,730 agent runs across 9 models and 9 benchmarks. Average cost per benchmark run ranged from $13 to over $450. On one benchmark, two agent designs cost $171 and $1,577 for a two-point accuracy difference, and cost forced single runs without error bars on most benchmarks.
Where the bill hides
- Criteria: each separately judged criterion multiplies calls.
- Runs: every prompt, model or rubric change is another full run.
- Prompt length: a long rubric is paid for on every call.
- Judge size: a stronger judge than the task needs multiplies the price per call.
How to bring it down
Judge a smaller, well-chosen set (how many examples you need); check a cheaper judge on your own labels (cheaper LLM judge); cache the stable rubric and batch non-urgent runs (reduce eval cost); run calls in parallel (speed up evals).
Questions
How much does an LLM eval cost?
Judge calls times tokens per call times price, repeated for every run. At the illustrative prices on this page, one run of 2,000 examples judged on 3 criteria (6,000 calls of 1,500 input and 200 output tokens) costs between $2.07 and $138. Fifty runs a month cost between $103.50 and $6,900. Real prices differ by model and change often.
How do I estimate the cost of an LLM-as-judge run?
Count the judge calls: examples times criteria judged separately, times runs. Multiply by the input and output tokens in one call, priced at your judge's published rates. Add the calls the model under test makes. The calculator on this site does the arithmetic from inputs you edit.
Why do agent evals cost more than text evals?
Each agent run makes many model calls and tool calls. The Holistic Agent Leaderboard authors (October 2025) report average costs from $13 to over $450 per benchmark run, and two agent designs on one benchmark that cost $171 and $1,577 for a two-point accuracy difference.
What does a full public benchmark cost?
The HELM paper (TMLR, August 2023) reports $85 to $10,926 per model for the paid-API models it ran, and 540 to 4,200 GPU hours for open models.
Does a cheaper eval mean a worse one?
Not if you check it. A cheaper judge, a smaller example set or a cascade each changes the measurement, so compare it with a sample you labeled yourself before you rely on it. See /cheaper-llm-judge and /how-many-examples-llm-eval.
Sources
Checked
- HELM, Percy Liang and co-authors. Transactions on Machine Learning Research, August 2023. arXiv:2211.09110 (posted November 2022).
- Efficient Benchmarking (of Language Models), Yotam Perlitz and co-authors. NAACL 2024. arXiv:2308.11696 (August 2023).
- Holistic Agent Leaderboard, Sayash Kapoor and co-authors. arXiv:2510.11977 (October 2025).
Where Slancha fits
Slancha pilots aim to make the expensive part of evals faster and cheaper, so you can spend less per eval and run one whenever a change calls for it. Keep your existing eval or observability platform.
Pilots by arrangement.
Estimate a run from your own inputs, or read how to cut what each run costs.