Topic

AI Benchmarks

13 stories

OpinionThe Human Factor
Meta says Muse Wonder has caught up to the AI frontier, but user results remain unproven
Artificial Analysis scored the models on its Intelligence Index, but neither those results nor Meta’s announcement establishes how reliably they handle leases, insurance claims or work.
Yesterday
Research
AI responses to terrorist prompts: what to know about a new safety test
CBS reported that responses varied with a prompt’s stated purpose. The full prompt set and response-level records were not available for independent review.
Oct 9
Models
Mistral Large 4 trails four Chinese open-weight models on a composite index
The index favors four Chinese models, while separate legal and cybersecurity evaluations give Mistral’s preview stronger results on particular tasks.
Oct 9

More on AI Benchmarks

OpinionThe Editorial Board
Haiku 5.5’s lower listed rates need independent task tests
Haiku 5.5 starts at $0.10 per million input tokens for prompts up to 100,000 tokens, but published rates and company estimates do not show cost or quality on a buyer’s workload.
Oct 7
ModelsExplainer
Claude Haiku 5.5: what to know about its features, prices and performance claims
Haiku’s token rates drop 90% for prompts up to 100,000 tokens, but Anthropic estimates average running costs fall about 75% after accounting for a tokenizer that uses more tokens.
Oct 7
Models
Mistral opens ML4 API preview, with weights planned for Oct. 27
Artificial Analysis lists ML4 at 38, one point below DeepSeek V4.1 Flash; Simon Willison reports two reasoning levels in Mistral’s API.
Oct 6
OpinionThe Argument
Beam should be judged by evidence, not the U.S.-China AI race
Reflection’s table puts Beam at 80.1 on Terminal Bench 2.1, below Kimi K3 and DeepSeek V4.1 Flash; public weights and the promised technical report are still pending.
Oct 6
Models
How Reflection AI is making its case for Beam against Chinese models
Reflection’s table puts Beam near GLM-5.2 but behind Qwen, Kimi and DeepSeek on two benchmarks. Public weights and developer materials are still pending.
Oct 6
OpinionThe Human Factor
Reflection AI’s Beam is not yet a public alternative to Chinese models
Beam’s early version is limited to selected users; Reflection plans to publish its weights and evaluation materials later in October, without a specific release date.
Oct 5

ModelsAnalysis
How Reflection AI is positioning Beam against Chinese models
Reflection offered limited early access and says public weights, a technical report and developer materials will follow later in October, without a specific release date.
Oct 5
Models
Cyber defenders come first in Google’s Gemini 4 Argon rollout as safeguards develop
Google says Argon can find and patch vulnerabilities. An independent benchmark ranks it second of 44 models, but its performance in live defense operations is unproven.
Oct 2
OpinionReality Check
GPT-6.1 Sol's price claim holds up; its near-Astra intelligence claim still needs outside proof
Sol's list prices are one-fifth of Astra's. The near-Astra evidence is mostly OpenAI's own, and a secondhand report of the one outside index has Astra winning most of its tests.
Oct 1
ModelsExplainer
How close does OpenAI's GPT-6.1 Sol get to Astra for a fifth of the price?
OpenAI says Sol nearly matches Astra at $2/$10 per million tokens against $10/$50. An independent test puts it 1 point behind, and Sol's Ultrafast tier was only promised at launch.
Oct 1