LuminaBench
Mathematics benchmark / MathArena / ETH SRI

BrokenArXiv — September 2026

Lumina VerifiedLumina checked the published sources, versions, model labels, score extraction and conversion. External evaluations were run by their publishers; Lumina has not independently rerun or reproduced them.

Accuracy

Version MathArena-arxiv_false--september-20261106Verified 6 November 202612 published resultsOfficial source

BrokenArXiv — September 2026

LuminaBench
Show
12 of 12 models · best variant per model
03006009001,2001,139GPT-6.1 SolMax1,115Claude Opus 5.5High1,060Kimi K3.1 Pro1,028DeepSeek V4.1 Flash 1017Max1,023DeepSeek V4.1 FlashMax1,002DeepSeek-V4.1-Flash-1017 (Max, DSH)1,002Gemini 4 Argon968Muse Wonder933Qwen3.8 2.4T A95B911Muse Spark Open911Grok 4.7xHigh907Gemini 3.8 Flash

Lumina Score vs Cost per Task

Original mean per-problem costs, token counts and time with monthly cohort; no total/budget values substituted.

LuminaBench
12 of 12 families
Lower resource use and higher score (green); higher resource use and lower score (red). Guides split at the visible selection’s medians.$0.3$1$3$109001,0001,1001,200Cost per task · USD (log scale)Lumina ScoreDeepSeek-V4.1-Flash-1017 (Max, DSH)GPT-6.1 SolClaude Opus 5.5Kimi K3.1 ProDeepSeek V4.1 Flash 1017DeepSeek V4.1 Flash

Lumina Score vs Tokens

Original mean per-problem costs, token counts and time with monthly cohort; no total/budget values substituted.

LuminaBench
12 of 12 families
200K400K600K9001,0001,1001,200Output tokens per taskLumina ScoreGPT-6.1 SolClaude Opus 5.5DeepSeek V4.1 Flash 1017Kimi K3.1 ProDeepSeek-V4.1-Flash-1017 (Max, DSH)DeepSeek V4.1 Flash

Lumina Score vs Time per Task

Original mean per-problem costs, token counts and time with monthly cohort; no total/budget values substituted.

LuminaBench
12 of 12 families
Lower resource use and higher score (green); higher resource use and lower score (red). Guides split at the visible selection’s medians.501009001,0001,1001,200Time per task · minLumina ScoreGemini 3.8 FlashGPT-6.1 SolMuse Spark OpenClaude Opus 5.5Kimi K3.1 ProDeepSeek V4.1 Flash 1017

Original mean per-problem costs, token counts and time with monthly cohort; no total/budget values substituted. Missing metrics are omitted from their graph; quality scores remain available.

Results table

Method & coverage

September 2026 detecting/correcting deliberately broken mathematical arguments; preserve model configuration, native effort, warning flags and monthly task cohort. No visual-math competition imported.

Original mean per-problem costs, token counts and time with monthly cohort; no total/budget values substituted.

How the Lumina conversion works

1,000 + 75 × ((original score − 49.13499999999999) / 18.062062682133877). The result is rounded once. Scores may tie.

Frozen initial observed text-only cohort; one mean per exact publisher release/API, balancing repeated settings/routes within that release. Original score and protocol remain separate. No inferred variants or future refitting.

Calibration
matharena-september-broken-z75-v1 · experimental
Reference mean
49.13499999999999
Population SD
18.062062682133877
Coverage
Standalone benchmark.
Identity, exclusions & attribution

Original measurements and exact source settings retained; unknown API, effort and harness versions remain unknown.

September 2026 detecting/correcting deliberately broken mathematical arguments; preserve model configuration, native effort, warning flags and monthly task cohort. No visual-math competition imported.

Standalone text benchmark; no new combined index. Source identity/effort/protocol qualifications remain in Data and CSV.

All published reasoning settings, including Claude Max, are included by default. Display labels use Low and xHigh; source labels remain in the retained records.

Official source

Publisher terms. Scores and resource figures are attributed to the original publisher.