Independent Tests Knock Le Chonk's Coding Lead Claims
By @sharedot · · 7 pages
- AI Frontier
- Open Weights
- Benchmarks
- Mistral
Third-party runs put Mistral Large 4's Terminal-Bench score at 25–26% versus its claimed 41%, and DeepSeek still leads four of five shared tests.
The coding lead that did not reproduce
Mistral released Mistral Large 4 'Le Chonk' as open weights on October 29, 2026, claiming the open-weight lead in agentic coding and cyber defense after three extra weeks of reinforcement training on the October 6 preview.
The headline claims now meet independent scrutiny: both third-party testers report the two flagship coding figures coming in well below the vendor numbers, making the reproducibility gap itself the story of the release.
Why the shortfall is surprising
The gap is specific and large. Mistral claims 41% on Terminal-Bench 4.0, but Intelligent Living reports independent runs of 25–26%, and 19.7% on FrontierCode 2.0 against an independently measured 15%.
On the shared Artificial Analysis tests, DeepSeek V4.1 Flash leads Mistral in four of five, which the site calls the clearest independent evidence that the open-weight lead has not changed hands.
shattered.io adds the counterweight: reported sources put Le Chonk at 48 on the Artificial Analysis Coding Agent Index, level with GPT-6.1 Terra and Codex.
The evidence, side by side
Intelligent Living tabulates the independent picture: vals.ai gives Le Chonk 81.56% on Vibe Code Bench v1.1 but 56.42% on its overall index, behind DeepSeek V4.1 Flash at 61.26%.
Artificial Analysis has Le Chonk at 25% on Terminal-Bench 4.0 versus DeepSeek's 33.8% and GLM-5.3 Flash at 32.8%, and an Intelligence Index of 26, which shattered.io reports sits below DeepSeek and GLM-5.3 at 32 each.
vals.ai's harshest gap is legal research: 27.33% versus 45.19% for both Chinese rivals.
What still works in Le Chonk's favor
Not every number is a miss. Vibe Code Bench at 81.56% is the standout, and Mistral's efficiency claim — about 47% fewer tokens per task than GLM-5.3 — could matter more to builders than a one-point benchmark edge if confirmed.
shattered.io notes 161 output tokens per second at $0.70 per million input tokens and $2.10 output, a 90% cached-input discount to $0.07, and an average $2.06 per Coding Agent Index task. The claimed 82% reproduce-then-patch cyber score remains unverified per Intelligent Living.
What comes next
Scores move between testing setups, and Intelligent Living cautions that benchmarks reward the exact tasks they measure, not general reliability; no single number settles the race. A technical report is listed as coming soon, per shattered.io, so several figures currently rest on launch materials.
For practitioners, the practical takeaway is to rerun Le Chonk on their own repositories before trusting the coding-lead narrative — a model that wins on a public index can lose on a messy internal monorepo, as shattered.io puts it.