LuminaBench
Tool use benchmark / OpenRouter

τ²-Bench Airline — OpenRouter

Lumina VerifiedLumina checked the published sources, versions, model labels, score extraction and conversion. External evaluations were run by their publishers; Lumina has not independently rerun or reproduced them.

Accuracy

Version OpenRouter-tau_bench_verified_airline-v1-2026-11-06Verified 6 November 20261093 published resultsOfficial source

τ²-Bench Airline — OpenRouter

LuminaBench
Show
20 of 142 models · best variant per model
03006009001,2001,094Claude Fable 51,088DeepSeek V4 Flash 07311,085Gemini 3.7 Flash1,081GPT-5.11,080DeepSeek V4 Flash 04231,078Qwen3.5 397B A17B1,078Ling 3.1 Flash1,077Claude Fable 5.11,075MiMo-V2.6-Pro1,075Claude Opus 51,075GLM 5.31,075GLM 5.3 Flash1,075Claude Sonnet 5.51,075Qwen3.6 27B1,074DeepSeek V4 Pro 04231,074Kimi K3.1 Pro1,074Gemini 4 Argon1,074DeepSeek V4 Pro 08131,072Gemini 3.5 Flash Lite1,072Kimi K2.7 Code

Lumina Score vs Cost per Task

USD per task; time is mean generation time per task for GPQA/airline, mean search-task latency for search lanes. These definitions are not interchangeable. Positive published telemetry only; missing/zero telemetry is not represented as free or instantaneous.

LuminaBench
15 of 142 families
Lower resource use and higher score (green); higher resource use and lower score (red). Guides split at the visible selection’s medians.$0.003$0.01$0.03$0.1$0.3$17008009001,0001,100Cost per task · USD (log scale)Lumina ScoreClaude Fable 5DeepSeek V4 Flash 0731Gemini 3.7 Flash

Lumina Score vs Tokens

USD per task; time is mean generation time per task for GPQA/airline, mean search-task latency for search lanes. These definitions are not interchangeable. Positive published telemetry only; missing/zero telemetry is not represented as free or instantaneous.

LuminaBench
15 of 142 families
3K10K30K7008009001,0001,100Output tokens per task (log scale)Lumina ScoreClaude Fable 5DeepSeek V4 Flash 0731GPT-5.1

Lumina Score vs Time per Task

USD per task; time is mean generation time per task for GPQA/airline, mean search-task latency for search lanes. These definitions are not interchangeable. Positive published telemetry only; missing/zero telemetry is not represented as free or instantaneous.

LuminaBench
15 of 142 families
Lower resource use and higher score (green); higher resource use and lower score (red). Guides split at the visible selection’s medians.1310307008009001,0001,100Time per task · min (log scale)Lumina ScoreClaude Fable 5Gemini 3.7 FlashGPT-5.1Ling 3.1 Flash

USD per task; time is mean generation time per task for GPQA/airline, mean search-task latency for search lanes. These definitions are not interchangeable. Positive published telemetry only; missing/zero telemetry is not represented as free or instantaneous. Missing metrics are omitted from their graph; quality scores remain available.

Results table

Method & coverage

Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.

USD per task; time is mean generation time per task for GPQA/airline, mean search-task latency for search lanes. These definitions are not interchangeable. Positive published telemetry only; missing/zero telemetry is not represented as free or instantaneous.

How the Lumina conversion works

1,000 + 75 × ((original score − 64.7055832999172) / 14.106894767506427). The result is rounded once. Scores may tie.

Frozen initial observed text-only cohort; one mean per exact publisher release/API, balancing repeated settings/routes within that release. Original score and protocol remain separate. No inferred variants or future refitting.

Calibration
openrouter-tau-bench-verified-airline-z75-v1 · experimental
Reference mean
64.7055832999172
Population SD
14.106894767506427
Coverage
Standalone benchmark.
Identity, exclusions & attribution

Rows aggregate the benchmark-runs dataset exactly as the leaderboard pages do, over each benchmark's non-zero-score runs from the newest 90 days: accuracy is the task-weighted mean of per-run accuracy, accuracy_std_dev is the population standard deviation of per-run accuracy (null with a single run), and the per-task averages divide summed cost, generation time, and output tokens by summed tasks. Rows are ordered by accuracy descending, then model name.

Routing and search-lane observations are distinct configurations, not independent model releases. No effort is inferred from an unlabelled default or a turn budget.

Source standard deviation is retained in original evidence only, not presented as an inferred confidence interval.

Text questions and text function/search tools. HLE uses the explicitly documented text-only subset; multimodal questions excluded.

Standalone text benchmark; no new combined index. Source identity/effort/protocol qualifications remain in Data and CSV.

All published reasoning settings, including Claude Max, are included by default. Display labels use Low and xHigh; source labels remain in the retained records.

Official source · Publication terms / evidence

Publisher terms. Scores and resource figures are attributed to the original publisher.