AA Benchmark v5 Changes: No More Benchmark Optimization!

3-Line TL;DR

Artificial Analysis (AA) released Intelligence Index v5 on October 27, and 60% of the score now comes from tests whose questions or answers are kept private
Labs cannot study for a test they cannot see, which takes aim at the score inflation behind recent leaderboard fights; AA itself says held-out data reduces labs' ability to game evaluations
It is the same idea as Kaggle's private leaderboard, so I trust AA scores more now, while remembering that nobody outside AA can rerun the hidden part

AA's Intelligence Index is one combined score built from nine tests in four areas: knowledge work, coding, science and general knowledge, 25% each (AA methodology)
I follow AI performance through this one index, so a change to it changes how I read every model launch
The headline of v5 is not a new test but where the tests live

AA's Index v5 announcement with the line that 60% of the Index uses private test sets or held-out answers boxed in red

Source: Artificial Analysis — Announcing Intelligence Index v5, October 27, 2026; red boxes added by HSL

"Private" means the questions, or at least the answers and grading tests, are never published
The private share has climbed fast: 20% in v4.1, 40% in v4.2 in September, 60% now (AA v4.2)
New in v5 is FrontierCode 2.0 from Cognition: 132 real coding tasks graded on whether a maintainer would merge the change, kept private "to avoid contamination"
Also new: Terminal-Bench-Science, 70 science research tasks in a terminal; three older tests were dropped, including SciCode

Stacked bar of the nine v5 evaluations by weight; five private ones, including FrontierCode 2.0, add up to 60%

Made by HSL — weights and private flags from the AA methodology page, checked October 29, 2026

At launch Claude Opus 5.5 led with 58, then GPT-6 Astra 57 and GPT-6.1 Sol 55; a day later Gemini 4 Carbon took first with 59

A public benchmark has a weakness: once the questions are online, they can end up in training data, or a model can be tuned to them
The score goes up, but the model has studied the exam rather than the subject
AA says it directly: held-out data "reduces the ability for labs to game evaluations" (AA v4.2)
So the title is a little loose: nothing bans optimizing, but hidden questions leave nothing to optimize against
The best-known fight happened on LM Arena, where people vote between two anonymous answers

Timeline: Llama 4 and the Leaderboard Illusion paper in April 2025, then AA v4.2 at 40% private and v5 at 60% private in 2026

In April 2025 Meta advertised Llama 4 Maverick's 1417 Arena score, earned by "an experimental chat version" (Meta)
LMArena said Meta should have made clear it was a model customized for human preference, and added the released version to the board (LMArena statement, via Simon Willison)
That released version then ranked below GPT-4o, Claude 3.5 Sonnet and Gemini 1.5 Pro (TechCrunch)
Weeks later the paper "The Leaderboard Illusion" counted 27 private variants Meta tested before Llama 4 and argued that publishing only the best one biases scores (arXiv)
Arena disputed how big that effect is, estimating about +11 Elo after 50 private tests (Arena)
I used to check LM Arena alongside AA, but votes from people seem to carry a popularity bias, so now I watch AA alone

Data science competitions solved this long ago
On Kaggle, the leaderboard you see during a contest scores only part of the test data, and the final ranking uses the rest (Kaggle docs)

Kaggle docs on public and private leaderboards, with the overfitting warning boxed in red

Source: Kaggle — Competitions documentation, The Leaderboard; red box added by HSL

Kaggle warns that a high public score does not guarantee a high private score, so chasing the public board is a trap
Data competitions I have entered work the same way: the test that decides the real ranking is revealed only at the end, and I think that is the fair way to do it
AA's private tests play the role of that final board, except the result is shown every time instead of once

Two scoreboards: a robot proudly tops the public board but slides to the bottom of the private board marked FINAL, where a quieter robot is first

First on the public board, last on the final one

I think it is a very good change, and from now on I will read AA scores with more trust
Three limits are still worth keeping in mind:
- Nobody outside AA can rerun the private part, so trust moves from the test to the tester
- Even with hidden questions, a score that is reported back every time leaks a little signal; one study's simulated attack beat a test's designed ceiling just by merging models and keeping the best score (CapBencher)
- 40% of the index is still public: Terminal-Bench 4.0, Terminal-Bench-Science, HLE-Diamond and GDP.pdf
No source has yet measured how much v5 actually reduces contamination; that is AA's design goal, not a measured result
I still treat a one or two point gap as a tie

When you see an AA score	What I check
Two models within 1–2 points	Treat as a tie; look at price and speed
A big jump in the overall score	Which of the nine tests moved, and whether it is private or public
A model I will use for coding	The coding tests (FrontierCode, Terminal-Bench) on their own
Comparing with an older score	Same index version only; v5 numbers differ from v4

Hacker News user hgoel asks what a one-point v5 gap means in practice and why frequently revised tests stay private (Hacker News)
In a September r/LocalLLaMA thread before v5, feng_sg argued that publishing model weights still does not let outsiders reproduce private AA-Briefcase scores (Reddit)
In the same thread, Antblue replied that a public sample scenario and AA's tools could corroborate the results (no measurements) (Reddit)
In an April r/LocalLLaMA thread, iperson4213 warned that hidden questions are not enough if the source code with the answers is public (no measurements) (Reddit)

Q. Can I still compare a v4 score with a v5 score?
- No; AA changed the tests and weights, so every model's number moved. Compare within one version
Q. Does a private test make cheating impossible?
- No, it removes the easiest route; repeated score feedback and leaked source material can still help a model
Q. Why not just use LM Arena?
- Arena measures which answer people prefer, which mixes in style and popularity; it is a different question from task accuracy
Q. Does a high AA score mean the model is best for me?
- Not necessarily; the index is mostly English and text-based, so test the model on your own work too