LLM eval cost calculator
A judge run costs examples × runs × the token bill of one call. The figure below uses the inputs in the form. It's an estimate from your inputs, not a measured result.
Assumptions
- Default baseline prices are the short-context standard rates for gpt-5.4 on the OpenAI pricing page, retrieved 16 October 2026: $2.50 input and $15 output per million tokens. Cached input on that row is $0.25.
- The cheaper column starts at gpt-5.4-mini on the same page: $0.75 input, $4.50 output, and $0.075 cached input. That is 30 percent of the gpt-5.4 input and output rates. The ratio is arithmetic on the listed prices, not a quality claim.
- The set size starts at 500 examples and 3 runs. Change both. A published research scale for a curated benchmark slice is 100 examples, which is not a recommendation for your task. See the guide.
- Cache-read share starts at 0. If you set it, that share of input tokens is billed at the cache-read price and the rest at the input price. The cache-half button sets that share to 50 on the cheaper column and leaves prices unchanged. The share is your assumption, not a measured hit rate. Output tokens are never cached in this formula.
- OpenAI’s Batch API lists a 50 percent discount versus synchronous prices. The half-price button applies that shape to whatever is in the baseline column. It does not call an API, and other providers differ.
- Omitted on purpose: the model under test, retries, failed calls, platform fees, and minimum cache lengths.
Cost
Baseline $12.38
Cheaper approach $3.71
Difference $8.67
1,500 baseline judge calls, 1,500 in the cheaper approach. The cheaper approach is 70 percent less than the baseline.
Cost = examples × runs × (uncached input tokens × input price + cached input tokens × cache-read price + output tokens × output price) ÷ 1,000,000. Prices are dollars per million tokens. Totals of at least one cent are shown to the cent; the difference keeps sub-cent precision when either total needs it.
Pilots by arrangement.