Best-of-K eval selection bias: pick the highest-scoring config from K candidates on one eval set and that observed score is biased up. It reports the expected maximum of K noisy estimates, which beats the field mean whenever K exceeds one. The bias appears even when all K configs are truly equal, grows with K, and shrinks with n.
Here is the version that bites you. Your bake-off ran a batch of prompts against one eval set, the top one came out ahead, and you shipped it. In production it does worse. That drop reads like bad luck, or drift, or a bad week. It is none of those. It is a number you could have computed before you shipped, and it gets larger the more candidates you tried.
I ran a small script to make the gap concrete. Eight configs, one hundred eval items, and here is the catch: I made all eight configs truly identical, every one a fair coin at 50%. There is no real best. Nothing to tune. Then I let selection pick a winner anyway:
config 0: 47/100 = 47.0%
config 1: 50/100 = 50.0%
config 2: 52/100 = 52.0%
config 3: 52/100 = 52.0%
config 4: 50/100 = 50.0%
config 5: 50/100 = 50.0%
config 6: 54/100 = 54.0% <- selected winner (argmax)
config 7: 44/100 = 44.0%
config 6: 54.0% (k=54 n=100 SE=4.98)
config 2: 52.0% (k=52 n=100 SE=5.00)
RANK: INDISTINGUISHABLE - gap 2.00 pp against 7.06 pooled SE = 0.28 SE < 2.0. Ranking "config 6" above "config 2" is NOT allowed.
Held-out the winner on a fresh 100 items: 48/100 = 48.0%.
Config 6 wins the bake-off at 54.0%. Then I asked the same eval-guard I use in the McNemar and rule-of-three pieces to rank config 6 against the runner-up. It refused: the gap is 0.28 SE, far under the two-SE bar, so INDISTINGUISHABLE. The ranker would not call config 6 the best. Selection did. And on a fresh held-out set the 54.0% falls back to 48.0%, toward the true 50% it was always going to be.
TL;DR
- Picking the best of K configs by observed pass rate reports the expected maximum of K noisy estimates. The max of K exceeds the field mean, strictly, for K at least 2 (Jensen's inequality on a convex function). So the winner's reported score is biased up.
- The bias is not about a weak config. In the null world, where all K configs are truly equal, best-of-16 reads 8.81 pp above the true 50%, and best-of-64 reads 11.67 pp above it. Nothing to optimize; the inflation is the act of selecting.
- It grows with K: 2.82 pp at K=2, 7.10 pp at K=8, 11.67 pp at K=64 (p=0.5, n=100). Adding candidates makes it worse, not better. You cannot fix a selection bias by selecting from more.
- It equals the drop you feel on held-out. On held-out, best-of-16 is indistinguishable from a random pick, 50.4% against 49.9% at 1.40 SE, while its in-sample number sits 29.5 SE higher. The whole curse lives in the eval you selected on.
- It shrinks with n like 1/sqrt(n): 8.81 pp at 100 items, 2.21 pp at 1600. Real, but you fight the square root, so halving the bias costs four times the eval budget.
What this is and is not. The fixtures are synthetic, and I say so on purpose. The bias here is a theorem, not an effect I fitted: E[max of K i.i.d. non-degenerate estimates] > their common mean is Jensen, and the null-world sections compute it exactly from binomial order statistics with no sampling and no tuned constant. The one sampled section uses a fixed seed and is byte-identical across runs. The curse is negligible when one config is genuinely far ahead of the field; it dominates when the field is tight, which is where most bake-offs actually sit. I show both.
Why the winner's score is biased up
Selection reports a maximum, and a maximum is not an average. Each config's observed pass rate is its true rate plus noise. When you take the best of K, you are not reading a typical config, you are reading whichever one the noise flattered most. E[max] sits above the mean for any K of 2 or more, and the gap widens as K grows. This is Jensen's inequality: the max is a convex function of the estimates, so the expectation of the max exceeds the max of the expectations, which in the equal case is just the common true rate.
The magnitude has a textbook form. The expected max of K standard normals grows like sqrt(2 ln K), so the selection bias scales roughly as sigma * sqrt(2 ln K), where sigma is the standard error of one config's estimate. That is why more candidates keep costing you: the penalty climbs with ln K and never turns around. Here is the exact order-statistic computation, summed over the binomial, not sampled:
K | E[in-sample winner] | true field p | bias = drop (pp) | sigma*sqrt(2lnK)
1 | 50.00% | 50.0% | -0.00 | 0.00
2 | 52.82% | 50.0% | 2.82 | 5.89
4 | 55.14% | 50.0% | 5.14 | 8.33
8 | 57.10% | 50.0% | 7.10 | 10.20
16 | 58.81% | 50.0% | 8.81 | 11.77
32 | 60.31% | 50.0% | 10.31 | 13.16
64 | 61.67% | 50.0% | 11.67 | 14.42
Read the bias column downward. Zero at K=1, because with one config there is no selection to make; the script flags that row as degenerate and refuses to run the selection path on it. Then 2.82, 5.14, 7.10, all the way to 11.67 pp at K=64. The sqrt(2 ln K) column is the growth guide, not an equality: it is the asymptotic leading term and runs loose at small K, which is why I print the exact bias beside it rather than the approximation. The conclusion does not live in any single row. It lives in the monotone climb of the whole swept column, with zero real signal anywhere in the fixture.
The bias is there even when nothing is better
The null world is the part I care about most, because it is the part you cannot argue with. There is no best config to find. Every candidate is the same coin. If the winner's score were an honest estimate, best-of-K would land at 50% on average for every K. It does not. It lands at 58.81% for K=16 and 61.67% for K=64. That premium is manufactured entirely by the operation of taking a maximum over noise.
This is the anti-fixture check I hold myself to after dropping five of my own drafts in one day for hiding a conclusion inside a tuned constant. Here there is no constant to tune in my favor: no lucky base rate, no engineered gap, no cherry-picked K. I set the true difference between configs to exactly zero and the bias is still positive and still growing. You can disagree with the binomial if you like. You cannot make the max of sixteen coins average out to one coin.
It doesn't depend on a lucky base rate
The next question a good skeptic asks: did I pick p=0.5 because it flatters the effect? So I swept the base rate at K=16 and read the exact bias at each:
p | E[in-sample winner] | bias = drop (pp) | sigma (pp) | bias/sigma
0.10 | 15.58% | 5.58 | 3.00 | 1.86
0.20 | 27.28% | 7.28 | 4.00 | 1.82
0.50 | 58.81% | 8.81 | 5.00 | 1.76
0.80 | 86.79% | 6.79 | 4.00 | 1.70
0.90 | 94.92% | 4.92 | 3.00 | 1.64
Positive at every base rate, 5.58 pp at a 10% pass rate up to 8.81 pp at 50%. Look at the last column instead of the fourth. Measured in units of the per-cell noise sigma, the inflation is close to constant, about 1.76 sigma at p=0.5 and never straying far. That is the real statement: the curse scales with sigma = sqrt(p(1-p)/n), the noise in one estimate, not with any special value of p. There is no base rate you can choose to make it disappear. The absolute pp figure is largest near 50% only because that is where a coin is noisiest.
More items shrink it, but you're fighting a square root
There is one honest escape, and the same math hands it to you. Since the bias scales with sigma, and sigma falls like 1/sqrt(n), running more eval items shrinks the curse. The theorem's degenerate limit is n to infinity: sigma to zero, bias to zero. Sweeping n at K=16, p=0.5:
n | sigma (pp) | bias = drop (pp) | bias/sigma | normal pred (pp)
100 | 5.00 | 8.81 | 1.76 | 8.83
400 | 2.50 | 4.41 | 1.76 | 4.41
1600 | 1.25 | 2.21 | 1.77 | 2.21
6400 | 0.62 | 1.10 | 1.77 | 1.10
Two things worth seeing here. First, the last column: I also computed the normal-theory prediction, sigma times the exact expected max of sixteen standard normals (which the script works out to 1.7660 by quadrature), and the binomial bias converges onto it as n grows and the central limit theorem takes hold. 8.81 against a predicted 8.83 at n=100, then dead-on to two decimals by n=400. The theorem and the run agree because it is the same theorem.
Second, the cost. Going from 100 to 400 items, a 4x eval budget, halves the bias from 8.81 to 4.41 pp. That is the square-root tax. If someone pays a judge model per eval item, buying your way out of the winner's curse is a real line on the bill, and it scales the wrong way. "Just run more evals" is correct and expensive at the same time.
The held-out drop is the bias you actually feel
The bias so far is E[in-sample winner] - true rate. You never see the true rate, so how do you know you got bitten? You hold the winner out. Re-evaluate the selected config on a fresh set and its inflated score regresses. In expectation the held-out drop equals the selection bias, because the held-out re-estimate is unbiased for whatever config you happened to pick.
That is exactly the trap I walked into on the McNemar piece, where a quantity that looked like a finding turned out to be an unconditional rate wearing a conditional costume. So this script runs the independence probe from the same library before it claims anything: it compares the winner's held-out rate against a random pick's held-out rate, and reports whether the conditioning on "was selected" actually buys you anything.
in-sample(best of 16) : 58.8% (k=23506 n=40000 SE=0.25)
held-out(best of 16) : 50.4% (k=20145 n=40000 SE=0.25)
held-out(random pick) : 49.9% (k=19947 n=40000 SE=0.25)
FORCED BY CONSTRUCTION [NULL: held-out of best-of-16 vs a random pick]: conditional 50.4% is indistinguishable from unconditional 49.9% (1.40 SE < 2.0)
in-sample vs held-out (paired over 400 trials): drop 8.40 pp = 29.5 SE -> SEPARATED (measured curse)
On held-out, best-of-16 reads 50.4% and a random pick reads 49.9%. The probe calls them indistinguishable at 1.40 SE, which is the whole point: selecting the best of sixteen coins bought nothing you could keep. The winner's held-out rate is the base rate. Meanwhile the in-sample number sits 29.5 SE above held-out on the paired comparison. The entire curse lived in the eval you selected on, and none of it survived the move to fresh data. That is the honest translation of "you selected noise."
The probe is not vacuous, and I check that too. When I give one config a real edge, config 0 truly at 0.70 while the other fifteen sit at 0.50, the same probe flips:
held-out(best of 16) : 69.4% (k=27747 n=40000 SE=0.23)
held-out(random pick) : 51.5% (k=20595 n=40000 SE=0.25)
ok [SPREAD: held-out of best-of-16 vs a random pick]: conditional 69.4% separates from unconditional 51.5% by 52.60 SE >= 2.0 - the conditioning is real
Now selection earns its keep: the winner's held-out rate beats a random pick by 52.60 SE. So the probe discriminates. It is forced under the null and not forced when there is signal, which is the only way I would trust the null result at all.
When selection is safe and when it lies
None of this says stop running bake-offs. It says know which regime you are in. I held K=8 and n=100 fixed and widened one config's true edge over a flat field, reading the exact stats at each width:
leader true p | edge (SE) | P(pick best) | held-out best | lift vs random | in-sample bias
0.50 | 0.00 | 14.4% | 50.0% | -0.00 | 7.10
0.52 | 0.40 | 23.6% | 50.5% | 0.22 | 6.96
0.55 | 1.00 | 42.0% | 52.1% | 1.47 | 6.21
0.60 | 2.00 | 74.3% | 57.4% | 6.18 | 3.62
0.70 | 4.00 | 99.2% | 69.8% | 17.33 | 0.19
Two ends, two different worlds. A dead-flat field, edge 0 SE, is the null: selection picks the true best only 14.4% of the time, about one in eight for eight configs, the held-out lift over a random pick is zero, and the in-sample bias of 7.10 pp is the entire result. A runaway leader, edge 4 SE at a true 0.70, gets picked 99.2% of the time, delivers a real 17.33 pp held-out lift, and its in-sample bias shrinks to 0.19 pp because there is little noise left to exploit next to a gap that large. The dangerous regime is the middle, one or two SE of edge, which is where most prompt and model bake-offs actually land: a handful of items separating the top few. There, selection inflates the score and the winner regresses.
How is this different from the McNemar piece?
Fair question, since both are about comparing configs on an eval set. The McNemar piece is about comparing two configs correctly: two prompts on one task set give paired outcomes, so the right significance test is McNemar, not a two-proportion SE. That is a question about the test you run on a fixed comparison. This piece is about what happens after you compare K of them and keep the max. The bias here is not in any pairwise test. It is in the selection step and in carrying the winner's eval score forward as if it were an unbiased estimate. You can run a flawless paired test between every pair and still report an inflated winner, because the inflation is post-selection, not a testing error. Different failure, one layer up.
What I am not claiming
Three limits, stated plainly.
The fixtures are synthetic Bernoulli configs, not your agent. The order-statistic math is exactly right for i.i.d. estimates; real evals have correlated items and non-identical difficulty, which change the constants but not the direction. If anything, correlation across your candidates makes the effective K smaller and the bias milder than the i.i.d. case, while heavy-tailed noise makes it worse. The sign does not move.
The bias is about the winner's score, not about whether you chose the right config. With a real leader you usually still pick it, as the spread table shows. The mistake the curse causes is trusting the number next to the winner, not the act of choosing a winner.
And it is an expectation. Any single held-out re-eval has its own noise, so a particular winner might regress a lot, a little, or by luck not at all. The claim is about what happens on average across bake-offs, which is the thing your intuition quietly assumes when it treats an eval score as ground truth.
What does your bake-off do
Everything here ran locally, Python 3.13.5, stdlib plus the pinned measure.py, offline, no keys, no funds, three runs byte-identical, exit 0, empty stderr. The null-world numbers are exact order statistics; the sampled sections use a fixed seed. Same eval harness as the two pieces before it, and the three of them are one family of pre-execution gates for AI agents: decide what you can actually claim before you act on the claim.
I publish the runs that correct my own reading, not only the ones that flatter it, so follow along if that is your kind of thing. And a real question I do not have a clean answer to. When your last bake-off picked a winner out of a dozen prompts on a couple hundred items, did you hold that winner out on a fresh set before you shipped its score, or did the eval number go straight into the launch doc? I want to know how many candidates your team tries, and whether anyone writes down that trying more of them is the thing making the winner look better.
AI disclosure. I drafted this with an AI writing assistant and edited every line; the framing, the tool, and the reading are mine. Every output block is pasted from one real local run on 2026-07-23. winners_curse.py sha256 c968f39c9295fff7…, run output sha256 41934693461986a6…, and the imported measure.py stays at its pinned sha256 b1b3702bccb6ab46… so the earlier pieces keep verifying. Recompute: python3 winners_curse.py prints the same bytes, exit 0, empty stderr, stdlib only.
Top comments (0)