LuminaBench
Long context benchmark / NVIDIA Research

RULER

Lumina VerifiedLumina checked the published sources, versions, model labels, score extraction and conversion. External evaluations were run by their publishers; Lumina has not independently rerun or reproduced them.

Mean task score

Version RULER-13tasks-4K128K-20261106Verified 6 November 202644 published resultsOfficial source

RULER

LuminaBench
Show
20 of 44 models · best variant per model
03006009001,2001,076Jamba-1.5-large (94B/398B)1,075Gemini-1.5-pro,-Text%20and%20images)1,074Qwen2.5-14B-Instruct-1M (14B)1,071Qwen3-235B-A22B1,068Qwen3-14B (14B)1,065Jamba-1.5-mini (12B/52B)1,064Qwen3-32B (32B)1,060EXAONE-4.0-32B (32B)1,053Qwen2.5-7B-Instruct-1M (7B)1,052Qwen3-30B-A3B (30B)1,052GPT-4 1106 Preview1,043GLM4 (9B)1,041Llama3.1 (70B)1,039Qwen3-8B (8B)1,039MegaBeam-Mistral (7B)1,037Command-R-0824 (32B)1,037ProLong (8B)1,034Llama3.1 (8B)1,034Command-R (35B)1,032Command-R-plus-0824 (104B)

No comparable per-task resource measurements were published. Missing metrics are omitted from their graph; quality scores remain available.

Results table

Method & coverage

13 synthetic text tasks across retrieval, multi-hop tracing, aggregation and question answering; publisher Avg. across 4K to 128K context lengths. Partial context rows without Avg. excluded.

No comparable per-task resource measurements were published.

How the Lumina conversion works

1,000 + 75 × ((original score − 82.00227272727271) / 13.815184147036513). The result is rounded once. Scores may tie.

Frozen initial observed text-only cohort; one mean per exact publisher release/API, balancing repeated settings/routes within that release. Original score and protocol remain separate. No inferred variants or future refitting.

Calibration
ruler-z75-v1 · experimental
Reference mean
82.00227272727271
Population SD
13.815184147036513
Coverage
Standalone benchmark.
Identity, exclusions & attribution

13 synthetic text tasks across retrieval, multi-hop tracing, aggregation and question answering; publisher Avg. across 4K to 128K context lengths. Partial context rows without Avg. excluded.

Historical published model cohort. Context-length settings are not native reasoning-effort variants.

Text synthetic retrieval, long-context documents and reasoning; no visual inputs.

Standalone text benchmark; no new combined index. Source identity/effort/protocol qualifications remain in Data and CSV.

All published reasoning settings, including Claude Max, are included by default. Display labels use Low and xHigh; source labels remain in the retained records.

Official source · Publication terms / evidence

Publisher terms. Scores and resource figures are attributed to the original publisher.