LuminaBench
Coding benchmark / Sakana AI

ALE-Bench

Lumina VerifiedLumina checked the published sources, versions, model labels, score extraction and conversion. External evaluations were run by their publishers; Lumina has not independently rerun or reproduced them.

Algorithm engineering on 40 AtCoder heuristic contest problems.

Version 40 problemsVerified 2 November 2026750 published resultsOfficial source

ALE-Bench · Lumina Score

Self-refine ×1

LuminaBench
Show
20 of 136 models · best variant per model
03507001,0501,4001,257GPT-6 AstraMax1,230GPT-6.1 SolMax1,213GPT 6.1 Terra Max1,193GPT-6 SolMax1,156GPT-5.6 SolMax1,155Claude Opus 5High1,152Claude Opus 5.5High1,152Claude Fable 5.1High1,146Gemini 4 ArgonHigh1,138Claude Fable 5High1,127GPT-5.6 TerraMax1,126GPT 5.5 2026 04 23xHigh1,124Kimi k3.1 Pro Max1,109Claude Sonnet 5.5High1,092Muse Spark 1.3xHigh1,089GPT-5.6 LunaMax1,088GPT 5.3 codexxHigh1,084DeepSeek V4.1 Flash 1017Max1,082GPT-5.4High1,079Muse Spark OpenxHigh

Lumina Score vs Cost per Problem

Publisher-reported means at the selected protocol · Self-refine ×1

LuminaBench
15 of 136 families
$0.1$0.3$1$39001,0001,1001,2001,300Mean cost/task · USD (log scale)Lumina ScoreGPT-6 AstraGPT-6.1 SolGPT 6.1 Terra MaxGPT-6 SolGPT-5.6 SolClaude Opus 5Claude Fable 5.1Claude Opus 5.5Gemini 4 ArgonClaude Sonnet 5.5Claude Fable 5GPT-5.6 TerraGPT 5.5 2026 04 23Kimi k3.1 Pro MaxMuse Spark 1.3

Lumina Score vs Output Tokens per Problem

Measured output tokens; execution budgets are separate · Self-refine ×1

LuminaBench
15 of 136 families
3K10K30K100K9001,0001,1001,2001,300Mean output tokens/problem (log scale)Lumina ScoreGPT-6 AstraGPT-6.1 SolGPT 6.1 Terra MaxGPT-6 SolGPT-5.6 TerraGemini 4 ArgonClaude Opus 5GPT-5.6 SolClaude Fable 5.1Claude Opus 5.5Claude Fable 5Claude Sonnet 5.5Muse Spark 1.3

Mean AtCoder performance across all 40 problems. Refinement budgets stay separate; rank and raw optimization scores are not the primary score. Self-refine ×1 selects the median-performing response from 15 samples using 50 local cases. Published problem variation is not a confidence interval. Missing metrics are omitted from their graph; quality scores remain available.

Results table

Method & coverage

Mean AtCoder performance across all 40 problems. Refinement budgets stay separate; rank and raw optimization scores are not the primary score. Self-refine ×1 selects the median-performing response from 15 samples using 50 local cases. Published problem variation is not a confidence interval.

Publisher mean cost and output tokens per problem at the selected refinement budget; program execution time is not generation time.

How the Lumina conversion works

1,000 + 75 × ((original score − 983.9363357843135) / 573.0262354499242). The result is rounded once. Scores may tie.

Frozen initial primary-protocol cohort; one mean per exact release/system, equally weighted. ALE reference uses self-refine ×1 only. Subsequent refreshes never refit. Strictly increasing conversion preserves source ordering at full precision; rounded display scores may tie.

Calibration
ale-bench-z75-v1 · experimental
Reference mean
983.9363357843135
Population SD
573.0262354499242
Coverage
Standalone benchmark.
Identity, exclusions & attribution

Exact source release, effort, route and harness retained. Unknown evaluation dates stay unknown; no cross-source score estimates or combined indexes.

Code licensing is not assumed to clear separately published result redistribution.

All published reasoning settings, including Claude Max, are included by default. Display labels use Low and xHigh; source labels remain in the retained records.

Official results

Publisher terms. Scores and resource figures are attributed to the original publisher.