Can a cheaper LLM judge replace an expensive one?
Sometimes, and you check it first. To cut the cost of LLM-as-a-judge evals, label 100 to 200 of your own examples, run the cheaper judge and the one you trust on them, and switch only where the cheaper judge's agreement with your labels, shown with an interval, and the failures it catches clear your bar. To choose a smaller judge without losing accuracy, keep the strong judge or a person for the cases where the small one falls short; a cascade lets a cheap judge rule first and escalate when it is unsure. A fine-tuned 7B judge agreed with its teacher 81.11% of the time on a narrow task (Zhu and co-authors, ICLR 2025) but fine-tuned judges can fall behind a strong judge on generalization (Huang and co-authors, Findings of ACL 2025), so test on your own tasks.
Sometimes, and you check it first. Label a sample of your own, measure how often the cheaper judge agrees with your labels, and switch only where the margin clears your bar. Papers show a cheap judge can hold up on narrow tasks, and a cascade can keep a cheap judge on easy cases and escalate hard ones.
Updated
What the research shows
- A strong judge can match people. Zheng and co-authors (NeurIPS 2023 Datasets and Benchmarks, June 2023): the strongest judge agreed with human experts on 85% of non-tied votes, above the 81% agreement among the experts.
- Judges have biases. In the same paper, with the answer order swapped, the strongest judge gave consistent verdicts on 65.0% of pairs. Few-shot examples raised consistency to 77.5% but made API calls 4× more expensive.
- Small judges can work on a narrow task. A 7B judge fine-tuned from a strong judge's labels (Zhu and co-authors, ICLR 2025) judged 5,000 samples in 3 minutes on 8 GPUs and agreed with its teacher 81.11% of the time without reference answers and 84.08% with them. Agreement above 90% is the paper's larger judge.
- They do not always transfer. An empirical study (Huang and co-authors, Findings of ACL 2025) found fine-tuned judges did well in-domain but fell behind a strong judge on generalization, fairness and adaptability.
- Cascades save the strong judge for hard cases. Jung and co-authors (July 2024): a cheap judge rules first and escalates when unsure. On a Chatbot Arena subset where the strongest judge almost never reached 80% human agreement, their method guaranteed over 80% while covering 79.1% of cases, and left the rest for people.
How to check a cheaper judge
- Label 100 to 200 of your own examples. Include known failures. Label more if failures are rare.
- Run the cheap judge and the one you trust now on the same examples.
- Report each judge's agreement with your labels, with an interval. At 90% agreement, 100 labels give about ±5.9 points, 200 give ±4.2 and 500 give ±2.6.
- Report separately how many true failures each judge catches. A judge that passes everything agrees with you 90% of the time when 90% of answers are good.
- Swap the answer order, rerun, and count the flips.
- Check again when either judge model changes.
The switch rule: move where the cheap judge's interval stays above your bar on both agreement and failures caught; keep the strong judge or a person where it does not.
Related: what an eval costs, how many examples, speed up evals, calculator.
Questions
Can a smaller model be an LLM judge?
Sometimes, on the narrow task it was tuned for. A 7B judge fine-tuned from a strong judge's labels agreed with its teacher 81.11% of the time without reference answers and 84.08% with them (ICLR 2025); agreement above 90% is the paper's larger judge. Across other tasks a fine-tuned judge can fall behind a strong judge on generalization, fairness and adaptability (Findings of ACL 2025). Check it on your own labeled sample.
How much human labeling do I need to check a judge?
100 to 200 of your own examples is a workable start. At 90% agreement, 100 labels give a 95% interval of about plus or minus 5.9 points, 200 give 4.2 and 500 give 2.6. Include known failures, and label more if failures are rare.
What is a judge cascade?
A cheap judge rules first, and a stronger judge or a person handles the cases where it is unsure. Jung and co-authors (July 2024) guaranteed over 80% human agreement on a Chatbot Arena subset where the strongest judge almost never reached it, covering 79.1% of cases.
Do LLM judges have biases?
Yes. Zheng and co-authors (NeurIPS 2023) found position bias: with the answer order swapped, the strongest judge gave consistent verdicts on 65.0% of pairs. Swap the order, rerun and count flips before you trust a verdict.
How do I know a cheaper judge is still accurate?
Run it and the judge you trust now on the same labeled sample, and compare each with your labels. Report agreement with an interval, and report how many true failures each catches. Check again when either judge model changes.
Sources
Checked
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Lianmin Zheng and co-authors. NeurIPS 2023 Datasets and Benchmarks. arXiv:2306.05685 (June 2023).
- A 2023 paper on fine-tuned LLM judges, Lianghui Zhu and co-authors. ICLR 2025. arXiv:2310.17631 (October 2023).
- An Empirical Study of LLM-as-a-Judge, Hui Huang and co-authors. Findings of ACL 2025. arXiv:2403.02839 (March 2024).
- Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement, Jaehun Jung and co-authors. arXiv:2407.18370 (July 2024).
- Adding Error Bars to Evals, Evan Miller. arXiv:2411.00640 (November 2024).
Where Slancha fits
Slancha pilots aim to make the expensive part of evals faster and cheaper, so you can spend less per eval and run one whenever a change calls for it. Keep your existing eval or observability platform.
Pilots by arrangement.
Estimate a run from your own inputs, or read how to cut what each run costs.