The short version
Score, Confidence and Eligibility answer separate questions. Keeping them separate prevents evidence breadth or uncertainty from quietly changing the order.
Central capability estimate
A measured estimate built from compatible benchmark evidence. The highest eligible central Score ranks first.
Strength of the evidence
How complete, connected and independent the supporting evidence is. Confidence never changes the order.
Enough evidence to rank
Whether compatible evidence is sufficient for an ordinal position. A diagnostic estimate alone is not a rank.
Methodology 2.3 foundation
Standard-eligible models retain their exact Methodology 2.2 Score, category scores, evidence selection, Confidence and uncertainty. Methodology 2.3 independently checks additional canonical, current and non-deprecated models without using a stored public ranking status as an input, then merges every eligible central Score into the same ordered Top 100.
Calibration boundary
Unsafe launch-only scoring remains shadow-locked
Route 3 launch scoring is not active. Its current decision is shadow-locked with no frozen production parameters. No named model can tune or waive that boundary.
Standard direct rows meet the exact 2.2 protocol. Relative anchored rows support compatible within-table comparisons only. Safe non-scoring values remain visible as public reference. Mirrors and copied tables are provenance carriers; quarantine is reserved for unresolved or unsafe evidence.
A source carrier is not a new measurement. Exact source-native observations are deduplicated by underlying event, model, configuration, benchmark and evaluation system. A provider's own subject row can be direct; competitor cells from that provider cannot become independent direct evidence.
A finite 2.2-compatible diagnostic Score may become ordinal-eligible only for a current canonical model with at least 8 safe deduplicated families, 3 categories and 2 row-valid anchors when comparison evidence exists, with no identity, configuration or safety conflict.
Multi-benchmark provider matrices remain one parent publication event with benchmark-specific comparison groups. A one-pillar category-local successor signal receives half weight and a four-point movement cap; two or more local pillars use the normal rule, and evidence never crosses category boundaries.
Ranked names use the ordinary public rank and Score only. There is no early, launch or provisional badge, no alternative leaderboard and no public changelog entry for an internal eligibility-basis transition alone.
Models receive a public Score and rank only through the unchanged Methodology 2.2 standard route or the generic evidence-breadth route. Unranked models may still show source-supported benchmark results, provenance, specifications, pricing and availability, while internal diagnostic Overall and category estimates remain hidden.
A future Route 3 review requires at least 30 genuine launch cases across 10 providers and 20 model families, with independent mature targets and an untouched grouped holdout. Meeting that evidence-history trigger permits review only; it never activates Route 3 automatically or on a fixed date.
Evidence to leaderboard
Eight connected stages keep collection, compatible evidence, scoring, uncertainty and publication separate and reproducible.
Reviewed aliases first join legitimate mirrors; each product then contributes at most one result to each genuinely compatible protocol.
Direction-aware scores put exact protocols on 0–100; frozen reviewed family offsets then make different benchmark families more comparable.
Full-calibration families are corrected by the hash-pinned 2.1 artifact, source-capped inside pillars, then averaged equally across available pillars.
Shared exact protocols create margin-sensitive comparisons with strength-of-schedule adjustment.
Compatible shared-protocol gains or regressions form a separate one-hop successor component; the generic fallback prior remains distinct.
Complete official tables contribute bounded margin-aware relative evidence only after already-consumed semantic comparison edges are removed.
Observed category weight is capped at 1.67× nominal; bounded missing-category estimates retain unallocated weight, while Confidence remains separate.
The highest 100 eligible central scores receive ranks. Confidence and intervals never reorder them.
Published before scoring
The category weights total 100%. Inside each category, the model's available capability pillars contribute equally.
Price, speed and popularity stay separate. Overall adds no external composite, hierarchy score or editorial correction.
Read the result
The six category scores combine using the published weights above. Missing categories never become zero, and price or speed never enter the calculation.
A model receives an ordinal rank only when its evidence is compatible and broad enough. Strong but sparse diagnostic estimates can remain unranked.
Absent benchmarks never become zero. Available pillars re-normalise; anchored, successor and controlled launch estimates are labeled and capped, while a decaying generic prior covers only what is still unobserved.
A provider launch table may contribute absolute rows and bounded relative edges, but the release remains one source event and adds no Confidence or eligibility.
Price, output speed, advertising, sponsorship, popularity, News coverage and Arena votes never affect Score or rank.
Generic launch path
Price, output speed, advertising, sponsorship, popularity, News coverage, Arena votes, the Artificial Analysis Intelligence Index, other aggregate Overall leaderboards and manually assigned frontier classes.
Frozen structural calibration
Methodology 2.2 reuses the frozen Methodology 2.1 family offsets learned offline from broad models that genuinely overlap multiple benchmark families. Normal ranking and refresh commands only load that artifact. The historical v0.3 tables remain diagnostics outside production scoring; no rank fixture, model name, provider identity or desired relationship enters the learner or live Score.
Methodology 2.2 validation: accepted · Simulation parity, sparse regression, category invariance and protected-data checks pass.
model/category overlaps
frozen calibrations
normalised points
SHA-256 pinned
Source URLs, provider launch pages, aggregators, captures and publication hashes are provenance—not benchmark identity. The reviewed map keeps incompatible versions, tracks and evaluation protocols separate, including MRCR context regimes and distinct agent harnesses.
identity = owner + benchmark + exact version + metric/unit
+ genuine track/tool mode + evaluation protocol
≥5 models: full calibration · 2–4: pairwise only · 1: display onlyMethodology 2.1 preserves the reviewed 2.0 canonical protocols and their direction-aware normalization as immutable history. Ordinary ranking and data refreshes cannot relearn either protocol calibration or family offsets.
p = (count below + ½ count tied) ÷ n
P = 15 + 70p
z = direction(x − median) ÷ max(1.4826·MAD, minimum spread)
M = 50 + 35·tanh(z ÷ 2)
BenchmarkScore = clamp(0.55P + 0.45M, 0, 100)Discrimination uses normalized p90–p10 spread; saturation measures metric-endpoint clustering; support uses log-scaled canonical-model count. Version dates—not check dates—drive freshness, while aliases and mirrors reduce independence. Quality factors are normalized within a pillar and also weight pairwise edges.
Q = discrimination^.30 · saturation^.20 · support^.15
· freshness^.10 · source quality^.20 · independence^.05
freshness half-life: 4 years · missing bounds/dates: neutral 0.5Only broad direct-evidence cohort rows with four pillars, two source clusters and owner or independent support enter the offline learner. Fewer than three overlaps shrink to zero; three, four and five-plus overlaps use 25%, 50% and 75%. The reviewed artifact is versioned and hash-pinned, so adding one model cannot recalibrate the leaderboard.
residual = family score − weighted median(≥2 other families)
offset = clamp(median residual × overlap shrinkage, −15, +15)
calibrated family = family score − frozen offsetRelated versions remain separate but share family, duplicate, mirror, configuration and source-cluster caps. Existing pairwise, small-table anchored and one-hop successor routes run first and reserve the exact semantic comparison edges they consume. Missing pillars remain missing rather than becoming zero.
PillarScore = source-capped quality-weighted mean(calibrated families)
Category PillarScore = mean(available pillar scores)
normal relative routes = peer + anchored + successor/sharedFull and pairwise-only canonical protocols create quality-weighted edges. Larger margins matter more, correlated families are capped, and a win against stronger opposition carries greater fitted value.
outcome = 0.5 + 0.5·tanh(score margin ÷ 8)
all canonical-protocol pairs · regularized Bradley–Terry fit
PeerScore = 0.55 percentile score + 0.45 rating-margin scoreExact compatible protocols are combined inside pillars, then across at least two pillars. A two-pillar category-local delta is preferred; otherwise a global lineage delta may inform Score but never category eligibility. Positive and negative deltas are symmetric, one predecessor hop is allowed, and the frozen fallback prior stays a separate component.
SuccessorDelta = equal-pillar reliability-weighted median(shared deltas)
SuccessorScore = predecessor category + clamp(delta, −8, +8)
max successor weight: 25% × [1, .8, .5, .2, 0] for 0,1,2,3,4+ direct pillarsIf PeerScore is unavailable, its weight moves to PillarScore—never to the prior. Successor evidence is reported separately, its weight decays with direct breadth, and neither successor nor anchored evidence is counted as a full-direct pillar.
4+ direct: .65 Pillar + .35 Peer + 0 Prior
2–3 direct: .65 Pillar + .30 Peer + .05 Prior
1 direct: .70 Pillar + .20 Peer + .10 Prior
0 direct: generic Prior only · Confidence D · no ordinal rank
final = (1 − successor weight)·base + successor weight·SuccessorScoreA raw result can still contribute normally to its absolute family and also supply new relative information. Deduplication occurs at exact benchmark/protocol/configuration/evaluation-system comparison-edge level. A launch edge already used by peer, anchored, successor/shared or another relative route is rejected. Mirrors collapse inside one release event, and launch evidence creates no pillars, eligibility, connectivity or Confidence.
margin = clamp(direction·(candidate − anchor) ÷ max(p90−p10, floor), −1, 1)
estimate = clamp(anchor pre-launch category + 8·margin, 0, 100)
launch weight: 20% at 1–2 direct pillars · 10% at 3 · 0% at 0 or 4+The safeguard is symmetric and does nothing merely because a category is sparse. It switches off at three direct pillars, requires an actual comparison replacement signal and never reuses a launch margin numerically.
trigger: ≤2 direct pillars + ≥2 families + leave-one-family range > 8
replacement: real cross-family peer signal backed by ≥2 non-zero-overlap families
movement = clamp(.15 × clamp(replacement − score, −8, +8), −1.2, +1.2)Unallocated weight uses nominally combined global/peer category estimates bounded within ±6 of the model prior and no higher than one point below the weakest observed category. Zero-direct-category models retain a diagnostic prior but receive no competitive Overall Score or rank. Eligibility, Confidence and nominal weights remain unchanged.
C = observed nominal category coverage
effective observed weight = nominal × min(1/C, 1.67)
p = .15 × (1 − C)
Overall = (1 − p) × (.85 fixed composite + .15 observed median) + p × bounded model priorSee Methodology 2.3 in practice
One public Top 100 combines preserved standard Scores with any model that passes the generic evidence gate; reference evidence stays visible and separately labeled.