A model’s label is no substitute for independent results

Those results support a precise conclusion: on this index, those five models outscored Large 4. The Intelligence Index combines 10 evaluations, including knowledge-work tasks, coding, agentic workflows, science, document reasoning and long-context reasoning. Its single score summarizes performance across that selected set; it is not a universal measure of every capability or a verdict on every use.

A composite can help compare models across several kinds of work, but it also compresses differences between them. The reported scores do not establish how the models compare on tasks outside the index, or which would work best for a particular organization. Nor do they amount to broad replication by multiple independent evaluators: they are results from a comparison produced by a firm separate from Mistral.

The five higher scores also do not disprove Mistral’s narrower claim that Large 4 would be its strongest open-weight model developed outside China. They do make clear why the comparison group and the measure must be stated. A developer’s label cannot stand in for test results, but neither should one index be treated as a universal league table.

What does the cybersecurity result add?

The overall score is not the entire comparison. Tom’s Hardware reported that Artificial Analysis gave Large 4 a score of 81.7% on the CyberGym-E2E-AA cybersecurity evaluation. MiMo-V2.6-Pro scored 79%, and GLM-5.3-Flash scored 74% on that test.

That is evidence of a stronger result on one evaluation than those two models achieved. It does not establish broad superiority over them, or show that Large 4 would defend real networks effectively. Mistral co-founder and chief scientist Guillaume Lample said the model’s cyber-defence capabilities could help governments and businesses defend against cyberattacks involving jailbroken closed models, Euronews reported. That is the company’s stated rationale; the benchmark result alone does not verify effectiveness in operational security work.

Both kinds of evidence matter. A single overall ranking can hide task-specific differences, while a high score on one task can be stretched into a claim of general superiority. Neither conclusion follows from the reported results. The fair reading is narrower: five Chinese open models scored above Large 4 on the composite index, and Large 4 scored above two of them on one cybersecurity evaluation.

When will the open weights be tested?

An open-weight release makes a model’s parameters publicly available. That is different from access through an API preview: the preview lets users test the model through the interface, while the weights themselves had not yet been released as of Oct. 9. The results available then therefore do not establish how independent tests of released weights would compare with tests of the preview, or what licence and usage conditions would apply.

That distinction matters to developers and organizations considering access to the model’s weights rather than its API. The planned release date is not evidence that the release happened, and these reported scores do not show how Large 4 performs on any one organization’s workload.

We recommend that independent evaluators test the released weights and publish task-by-task results alongside any composite score. Mistral’s standing should be judged against that evidence, not the label attached to a planned release. The existing tests establish a limited comparison, not a universal winner.