AIdataNewsGlobal General: Artificial Analysis Intelligence Index v5 Expands 2 Benchmarks, Revises 3 Sets, and Reaches 60% Private Test Coverage

By David Nakamura, AIdataNewsGlobal

Artificial Analysis introduced Artificial Analysis Intelligence Index v5 today, a composite measure of model capabilities on tasks that reflect real-world usage. The release calls it the largest revision of the Artificial Analysis Intelligence Index since v4 launched in January. The updated composite uses 2 added benchmark suites, 3 upgraded suites, and, for the first time, draws 60% of the Intelligence Index from private datasets, private test sets, or held-out answers. The Artificial Analysis Progress Index v1 arrives at the same time, putting model capability over time on one continuous scale.

Artificial Analysis separates the present frontier from historical progress: the Intelligence Index is positioned for current capability, while the Progress Index is positioned for change across earlier releases.

Snapshot: Equal Category Weights

Capability area	Index weight	How it is composed
Knowledge Work	25%	AA-Briefcase 15%, AutomationBench-AA 5%, GDP.pdf 5%
Coding	25%	FrontierCode 2.0 15%, Terminal-Bench 4.0 10%
Science	25%	Terminal-Bench-Science 15%, CritPt v2 hard subset 10%
General	25%	AA-Omniscience v1.1 15%, split 10% accuracy and 5% non-hallucination; HLE-Diamond 10%

Benchmark Inventory

New: FrontierCode 2.0 and Terminal-Bench-Science. FrontierCode 2.0, from Cognition, tests real-world coding in 132 open-source repository tasks and grades whether a maintainer would merge the agent's change. Terminal-Bench-Science contains 70 agentic scientific research tasks run in a terminal environment.

Upgraded: HLE-Diamond, AA-Omniscience v1.1, CritPt v2. HLE-Diamond takes the place of Humanity's Last Exam with a refined 1,000-question set from the Center for AI Safety (CAIS) and Scale AI, evenly split between reasoning and knowledge. AA-Omniscience v1.1 removes saturated and public set questions, adds new questions, and revises ambiguous or erroneous questions. CritPt v2 uses a private 40-question hard subset, revised by CritPt authors and run with a code-execution tool.

Retained: AA-Briefcase, AutomationBench-AA, GDP.pdf, Terminal-Bench 4.0.

Removed: GDPval-AA v2.1, SciCode, AA-LCR v1.1.

Added Benchmark Detail

FrontierCode 2.0

Cognition released FrontierCode 2.0 earlier today as a private coding benchmark. Tasks remain private to avoid contamination. A failed blocking check scores 0, and each task is run three times in Stirrup, the Artificial Analysis agentic harness.

Results: Claude Opus 5.5 (max) leads at 46%. GPT-6 Astra (max), GPT-6.1 Terra (max), and Claude Sonnet 5.5 (max) follow at 42%. DeepSeek V4.1 Flash 1017 (max) is the best open weights model at 28%, 18 points behind the leader.

Terminal-Bench-Science

Terminal-Bench-Science tests scientific research workflows in terminal environments. Experts curate and review each task, and all tests in a task must pass.

Results: GPT-6 Astra (max) leads at 63%; Claude Opus 5.5 (max) reaches 59%; GPT-6.1 Sol (max) reaches 58%. Kimi K3.1 Pro (weights not yet released) reaches 34%, Claude Haiku 5.5 (max) reaches 20%, and DeepSeek V4.1 Flash 1017 (max), the best model with released weights, reaches 13%. Most benchmarked open weights models score under 10%.

Upgraded Benchmark Detail

HLE-Diamond

HLE-Diamond replaces Humanity's Last Exam. The refined 1,000-question version is provided by the Center for AI Safety (CAIS) and Scale AI, with reasoning and knowledge questions split evenly. Claude Opus 5.5 (max) scores 68%, GPT-6 Astra (max) 64%, and Claude Fable 5.1 (max) 60%.

For the leaders, knowledge is harder than reasoning: Claude Opus 5.5 (max) and GPT-6 Astra (max) both reach 78% on reasoning, versus 58% and 50% on knowledge.

AA-Omniscience v1.1

AA-Omniscience v1.1 is the refreshed version of the Artificial Analysis knowledge and hallucination benchmark. The revision removes saturated and public set questions, adds new questions, and revises ambiguous or erroneous questions. Average scores are close to v1, but 2 points lower on the AA-Omniscience v1.1 Index, median 0.6. The public subset, AA-Omniscience-Public-v1.1, was released on Hugging Face and recalibrated so accuracy, hallucination rate, and overall performance align across public and private sets.

CritPt v2

CritPt v2 in the Index uses a private 40-question hard subset. It subtracts the 30 questions that seven frontier models solved in all five attempts at launch from the original 70 research-level physics questions. Models run in an offline Python sandbox in Stirrup, the Artificial Analysis agentic harness.

Results: GPT-6 Astra leads with 72% on high effort; GPT-6.1 Sol follows with 69% on max effort. Claude Opus 5.5 (max), the overall Index leader, scores 61%. Kimi K3.1 Pro (weights not yet released) is the highest open weights model at 45%.

Overall Leaderboard

Claude Opus 5.5 (max) is first at 58, GPT-6 Astra (max) second at 57, and GPT-6.1 Sol (max) third at 55. Anthropic and OpenAI own the top five places, and Gemini 4 Argon (high) comes next at 50.

No Universal Winner

Across all nine evaluations, v5 still has room for progress because no single model wins every test. Claude Opus 5.5 wins three; five other models win the remaining six:

• Claude Opus 5.5: FrontierCode 2.0 46%, HLE-Diamond 68%, AA-Omniscience v1.1 48
• GPT-6 Astra: Terminal-Bench-Science 63%, CritPt v2 72%
• Claude Sonnet 5.5: AA-Briefcase 1812 Elo
• Kimi K3.1 Pro: Terminal-Bench 4.0 67%
• Gemini 4 Argon: AutomationBench-AA 78%
• Muse Wonder Full: GDP.pdf 33% all-pass

The open-weights gap widened to 12 points from 3 points in v4.3. Kimi K3.1 Pro (weights not yet released) tops open weights at 46. Among released open weights models, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) reach 32, 26 points behind the leader; Claude Haiku 5.5 (max) is ahead at 36. The gap is widest on HLE-Diamond and Terminal-Bench-Science: Kimi K3.1 Pro scores 37% and 34%, compared with 68% for Claude Opus 5.5 (max) and 63% for GPT-6 Astra (max).

Cost and Effort Profile

Cost varies widely at the top of the table. Across the 10 highest-scoring models shown, cost per Intelligence Index task differs by more than 13x: GPT-6.1 Terra (max) is cheapest at $0.96, while Claude Fable 5.1 (max) is most expensive at $13.04.

Claude Opus 5.5 (max) costs $10.04 per task, GPT-6 Astra (max) $6.24, and GPT-6.1 Sol (max) $1.27. GPT-6.1 Sol (max) is within 4 points of Claude Opus 5.5 (max) at about an eighth of the cost per task.

Effort settings matter more than in v4.3. Claude Opus 5.5 moves from 35 at low effort to 58 at max, with 10x cost per task ($0.98 vs $10.04). Claude Sonnet 5.5 spans 29 points from low to max, up from 20 in v4.3. The spread for GPT-6.1 Sol doubles from 10 to 20 points.

v5 also demands more generation. Every model measured produces more output tokens per Intelligence Index task than in v4.3, with the median rising by two thirds, from 35k to 59k. Agentic evaluations account for most token use: Terminal-Bench-Science, Terminal-Bench 4.0, AA-Briefcase and FrontierCode 2.0 make up 55% of the Index weight but a median ~90% of output tokens. Terminal-Bench-Science alone accounts for about a third of output Tokens per Task.

Progress Index

The Artificial Analysis Progress Index v1 tracks model capability since November 2022 on one capability scale across many benchmark families. Use it to compare historical capability movement, including between releases years apart. Use the Intelligence Index for current use cases and the present frontier; it remains the primary Artificial Analysis measure of current model intelligence. The release credits Epoch AI and Anthropic and notes that the Progress Index was built on learnings from the Epoch Capabilities Index.

Credits

The update credits Cognition, Steven Dillmann and the Terminal-Bench team, the Center for AI Safety (CAIS), Scale AI, and Minhui Zhu, Minyang Tian, Hao Peng and the CritPt team.

FAQ

• What changed in Artificial Analysis Intelligence Index v5? It adds 2 benchmarks, upgrades 3 suites, and makes private evaluation data the majority share at 60%.
• Which model leads the overall Intelligence Index? Claude Opus 5.5 (max) at 58, ahead of GPT-6 Astra (max) at 57 and GPT-6.1 Sol (max) at 55.
• How large is the open-weights gap? Kimi K3.1 Pro (weights not yet released) reaches 46, 12 points behind the leader; DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) reach 32.
• What should readers use the Progress Index for? Comparing capability movement from November 2022 onward, not only current-frontier ranking.

Closing Read

For AIdataNewsGlobal readers, the update shows a benchmark shift toward private, agentic, and tool-use evaluation. The headline scores remain informative, but the design change matters just as much: v5 reduces public contamination risk, widens the open-weights distance, and makes effort settings a central part of both score and cost.