Bench to the Future 3
Lumina VerifiedLumina checked the published sources, versions, model labels, score extraction and conversion. External evaluations were run by their publishers; Lumina has not independently rerun or reproduced them.Pooled forecast loss
Bench to the Future 3
No comparable per-task resource measurements were published. Missing metrics are omitted from their graph; quality scores remain available.
Results table
Method & coverage
2,386 resolved pastcasting questions: 2,015 binary Brier and 371 numeric normalized RPS, frozen web corpus. Original pooled loss weights numeric questions three times; lower is better. Exact original pooled value, not recomputed by Lumina.
No comparable per-task resource measurements were published.
How the Lumina conversion works
1,000 − 75 × ((original score − 0.13566666666666669) / 0.010913803695829928). The result is rounded once. Scores may tie.
Frozen initial observed text-only cohort; one mean per exact publisher release/API, balancing repeated settings/routes within that release. Original score and protocol remain separate. No inferred variants or future refitting.
The source metric is lower-is-better; its standardized difference is negated before adding the baseline.
- Calibration
- btf3-z75-v1 · experimental
- Reference mean
- 0.13566666666666669
- Population SD
- 0.010913803695829928
- Coverage
- Standalone benchmark.
Identity, exclusions & attribution
2,386 resolved pastcasting questions: 2,015 binary Brier and 371 numeric normalized RPS, frozen web corpus. Original pooled loss weights numeric questions three times; lower is better. Exact original pooled value, not recomputed by Lumina.
Source costs on BTF-3 use an August subset while accuracy spans June–August; not substituted as full-cohort resource measurements. DRB runtime is step-estimated, not a measured wall-clock comparison.
Text forecasting/research questions and archived text web-search tools.
Standalone text benchmark; no new combined index. Source identity/effort/protocol qualifications remain in Data and CSV.
All published reasoning settings, including Claude Max, are included by default. Display labels use Low and xHigh; source labels remain in the retained records.
Official source · Publication terms / evidence
Publisher terms. Scores and resource figures are attributed to the original publisher.