How many examples does an LLM eval need?
For a pass rate near 80%, about 246 independent examples give a 95% interval of ±5 points, about 683 give ±3 and about 1,537 give ±2; with no idea of the pass rate, use about 385, about 1,068 and about 2,401. To compare two versions, run both on the same examples and test the per-example differences (a paired comparison) instead of comparing two separate error bars, because paired differences remove the variance both versions share (Miller, November 2024). The sizes follow n = 1.96² × p × (1 − p) / E² and are approximate.
Enough for the margin you can live with. For a pass rate near 80%, about 246 independent examples give a 95% interval of ±5 points, about 683 give ±3 and about 1,537 give ±2. With no idea of the pass rate, use about 385, about 1,068 and about 2,401. A curated subset can need far fewer, but only for the comparison it was built for.
Updated
The formula
Standard error = √(p × (1 − p) / n). The 95% interval is p ± 1.96 × standard error (Miller, November 2024). Solve for n: n = 1.96² × p × (1 − p) / E², where E is the margin you want.
These sizes use the normal approximation, so treat them as approximate; they are less accurate for pass rates near 0% or 100% and for small sets.
Approximate examples needed for a margin (95% confidence)
| Margin | Pass rate near 80% | Pass rate unknown (worst case, 50%) |
|---|---|---|
| ±5 points | 246 | 385 |
| ±3 points | 683 | 1,068 |
| ±2 points | 1,537 | 2,401 |
What a set you already have gives you (approximate 95% margin, ± points)
| Examples | Pass rate 50% | 80% | 90% |
|---|---|---|---|
| 50 | 13.9 | 11.1 | 8.3 |
| 100 | 9.8 | 7.8 | 5.9 |
| 200 | 6.9 | 5.5 | 4.2 |
| 500 | 4.4 | 3.5 | 2.6 |
| 1,000 | 3.1 | 2.5 | 1.9 |
| 2,000 | 2.2 | 1.8 | 1.3 |
Note: These assume independent examples. Related examples, such as several questions on one document, behave like fewer. Miller reports clustered standard errors over 3 times larger than naive ones on two popular evals.
Comparing two versions
To tell whether version B beats version A, run both on the same examples and test the per-example differences (a paired comparison) rather than comparing two separate error bars. Miller shows paired differences remove the variance both versions share, a free gain when the two are correlated. Power analysis then gives the number of examples for the gap you care about.
When a smaller set works
- tinyBenchmarks (Polo and co-authors, ICML 2024): 100 curated examples per scenario estimated a model's accuracy on the full MMLU benchmark with average error under 2%, from under 1% of the examples.
- Efficient Benchmarking (Perlitz and co-authors, NAACL 2024): on HELM, computation fell 100× or more with minimal loss of reliability in the model ranking.
- The caveat: these are curated subsets of public benchmarks that estimate results across many models. Stratified random sampling, the simplest strategy, can give a large estimation error (tinyBenchmarks). Before you trust a subset of your own, check that it ranks your recent versions the way the full set does.
A checklist
- Pick the margin that would change your decision.
- Read the example count from the table.
- Remove duplicates and group related examples.
- Compare versions on the same examples, paired.
- Report the interval, not only the pass rate.
Related: what an eval costs, speed up evals, calculator.
Questions
How many examples do I need for an LLM eval?
For a pass rate near 80% and a 95% interval of plus or minus 5 points, about 246 independent examples. Plus or minus 3 points needs about 683 and plus or minus 2 needs about 1,537. If you do not know the pass rate, use about 385, about 1,068 and about 2,401. The formula is n = 1.96² × p × (1 − p) / E².
Are 100 examples enough?
For a curated subset of a public benchmark, the tinyBenchmarks authors (ICML 2024) report average error under 2% with 100 examples per scenario. For a random set of 100 of your own examples at an 80% pass rate, the 95% interval is about plus or minus 7.8 points.
How do I know whether two prompts or models really differ?
Compare them on the same examples and test the per-example differences, not two separate error bars (Miller, November 2024). Paired differences remove the variance both versions share. A power analysis tells you how many examples the gap you care about needs.
Why might my interval be too narrow?
Examples that come in groups are not independent. Miller reports clustered standard errors over 3 times larger than naive ones on two popular evals. Cluster by document or user before you trust the interval.
Does repeating a run help?
Partly. Resampling the model's answer on each example lowers the noise from the model, not the noise from which examples you picked (Miller). More distinct examples lower both.
Sources
Checked
Where Slancha fits
Slancha pilots aim to make the expensive part of evals faster and cheaper, so you can spend less per eval and run one whenever a change calls for it. Keep your existing eval or observability platform.
Pilots by arrangement.
Estimate a run from your own inputs, or read how to cut what each run costs.