Reasoning benchmark / Epoch AI

GPQA Diamond

Lumina Verified

Expert-level science reasoning on GPQA Diamond. Published evaluations by Epoch AI.

Version Diamond-198Verified 31 October 2026326 published resultsOfficial source

GPQA Diamond · Lumina Score

Epoch AI · 198 questions

LuminaBench

20 of 326 configurations

Epoch AI mean accuracy and published standard error; unspecified effort remains unknown. Missing metrics are omitted from their graph; quality scores remain available.

Results table

GPQA Diamond Diamond-198. Original publisher values and provisional Lumina conversion.
Rank
1	Gemini 4 ArgonHigh	1,096	96.4%	±1.19 pp SE
2	GPT-6 AstraMax	1,094	95.77%	±1.37 pp SE
2	Claude Sonnet 5.5Max	1,094	95.58%	±1.37 pp SE
4	Gemini 3.8 FlashHigh	1,093	95.39%	±1.4 pp SE
4	GPT-6.1 SolMax	1,093	95.39%	±1.38 pp SE
6	Gemini 3.7 FlashHigh	1,091	94.82%	±1.35 pp SE
7	GPT-5.4 ProxHigh	1,090	94.6%	±1.6 pp SE
7	Gemini 3.1 Pro (Preview)High	1,090	94.44%	±1.63 pp SE
9	GPT-6.1 TerraMax	1,089	94.32%	±1.49 pp SE
9	GPT-6 SolMax	1,089	94.32%	±1.49 pp SE
9	Gemini 3.6 FlashHigh	1,089	94.13%	±1.4 pp SE
9	Gemini 3.1 Pro (Preview)Not specified	1,089	94.1%	±1.7 pp SE
13	GPT-5.5 Pre ReleasexHigh	1,088	94%	±1.55 pp SE
13	Grok 4.6High	1,088	94%	±1.45 pp SE
13	GPT-5.5 Pro Pre ReleasexHigh	1,088	93.92%	±1.58 pp SE
13	Claude Opus 5Max	1,088	93.88%	±1.48 pp SE
17	GPT-5.6 SolMax	1,087	93.5%	±1.57 pp SE
18	Grok 4.5High	1,086	93.43%	±1.43 pp SE
18	GPT-5.6 TerraMax	1,086	93.31%	±1.54 pp SE
18	GPT-5.4xHigh	1,086	93.3%	±1.8 pp SE
21	Grok 4.6xHigh	1,085	93.18%	±1.52 pp SE
21	DeepSeek V4.1 Flash 1017Max	1,085	93.18%	±1.57 pp SE
21	Kimi K3Max	1,085	93.12%	±1.49 pp SE
21	Claude Opus 5Not specified	1,085	92.93%	±1.83 pp SE
25	Gemini 3.5 FlashHigh	1,084	92.8%	±1.64 pp SE
25	Grok 4.7xHigh	1,084	92.74%	±1.55 pp SE
25	qwen3.8 MaxxHigh	1,084	92.68%	±1.69 pp SE
25	Gemini 3 Pro PreviewNot specified	1,084	92.61%	±1.65 pp SE
29	qwen3.8 Max 0902xHigh	1,082	92.3%	±1.72 pp SE
30	Kimi K3High	1,081	91.92%	±1.94 pp SE
30	GLM-5.2Max	1,081	91.86%	±1.61 pp SE
32	DeepSeek V4 Pro 0813Max	1,080	91.67%	±1.52 pp SE
32	GPT-5.6 LunaMax	1,080	91.6%	±1.73 pp SE
34	GPT-5.2xHigh	1,079	91.4%	±1.8 pp SE
35	Claude Opus 4.8Max	1,078	91.04%	±1.92 pp SE
35	DeepSeek V4 Flash 0731Max	1,078	91.04%	±1.77 pp SE
35	GLM 5.3Max	1,078	90.91%	±1.59 pp SE
35	MiniMax M3Not specified	1,078	90.91%	±2.05 pp SE
35	DeepSeek V4 ProHigh	1,078	90.91%	±2.05 pp SE
35	qwen3.7 MaxNot specified	1,078	90.91%	±2.05 pp SE
41	Kimi k2.6Not specified	1,077	90.78%	±1.72 pp SE
41	GPT-5.5Low	1,077	90.66%	±1.75 pp SE
41	Claude Opus 5.5Max	1,077	90.59%	±1.94 pp SE
44	Claude Opus 4.632K	1,076	90.53%	±1.69 pp SE
44	Claude Sonnet 5xHigh	1,076	90.53%	±1.78 pp SE
44	GPT-6 LunaMax	1,076	90.47%	±1.85 pp SE
47	GLM 5.3 FlashMax	1,075	90.15%	±1.72 pp SE
47	Claude 4.7xHigh	1,075	90.15%	±1.85 pp SE
49	GPT-5.6 SolLow	1,074	89.9%	±2.15 pp SE
49	GLM 5.1Not specified	1,074	89.9%	±2.15 pp SE
49	GPT-5.4High	1,074	89.9%	±2.15 pp SE
49	DeepSeek V4.1 FlashMax	1,074	89.84%	±1.73 pp SE
49	Muse SparkNot specified	1,074	89.8%	±2.2 pp SE
54	DeepSeek V4 ProMax	1,073	89.65%	±1.75 pp SE
54	Claude Haiku 5.5Max	1,073	89.65%	±1.98 pp SE
54	Gemini 3 Flash PreviewHigh	1,073	89.39%	±2.19 pp SE
57	Grok 4.20 (Reasoning)Not specified	1,072	89.33%	±1.86 pp SE
58	Gemini 3.5 FlashLow	1,071	88.89%	±2.24 pp SE
58	GPT-5.4Medium	1,071	88.89%	±2.24 pp SE
58	Grok 4.3High	1,071	88.83%	±1.96 pp SE
61	Claude Opus 4.664K	1,070	88.76%	±1.94 pp SE
61	Muse Spark OpenxHigh	1,070	88.57%	±1.87 pp SE
63	Inkling SmallxHigh	1,069	88.51%	±1.91 pp SE
63	qwen3.6 PlusNot specified	1,069	88.38%	±2.28 pp SE
63	Claude Opus 4.8Low	1,069	88.38%	±2.28 pp SE
63	Claude Opus 4.6Max	1,069	88.38%	±2.28 pp SE
63	InklingxHigh	1,069	88.26%	±1.94 pp SE
68	GPT-5.2High	1,068	88.19%	±1.91 pp SE
68	Mistral Large 4 0High	1,068	88.07%	±1.93 pp SE
70	GPT-5.2Medium	1,067	87.88%	±1.91 pp SE
70	GLM-5.2Low	1,067	87.88%	±2.33 pp SE
70	qwen3.7 PlusNot specified	1,067	87.88%	±2.33 pp SE
70	Claude Opus 5Low	1,067	87.88%	±2.33 pp SE
70	Kimi k2.7 CodeNot specified	1,067	87.88%	±2.33 pp SE
70	GLM-5Not specified	1,067	87.82%	±2.34 pp SE
76	GPT-5.1High	1,066	87.63%	±1.88 pp SE
76	Kimi k2p5Not specified	1,066	87.6%	±1.9 pp SE
76	Claude Sonnet 4.632K	1,066	87.37%	±2 pp SE
76	GPT-5.6 TerraLow	1,066	87.37%	±2.37 pp SE
76	qwen3.6 Max PreviewNot specified	1,066	87.37%	±2.37 pp SE
81	Grok 4Not specified	1,064	87%	±2 pp SE
81	GPT-5.4 MinixHigh	1,064	86.87%	±2.41 pp SE
83	qwen3.5 397b a17bNone	1,062	86.36%	±2.45 pp SE
83	Gemini 3.6 FlashLow	1,062	86.36%	±2.45 pp SE
83	Claude 4.7Max	1,062	86.36%	±2.45 pp SE
83	Gemini 3.5 FlashMinimal	1,062	86.36%	±2.45 pp SE
83	GPT-5High	1,062	86.17%	±2.08 pp SE
88	Claude Opus 4.532K	1,061	86.05%	±2.07 pp SE
89	Gemini 3.6 FlashMinimal	1,060	85.86%	±2.48 pp SE
89	qwen3.6 27bNot specified	1,060	85.86%	±2.48 pp SE
89	qwen3.5 397b a17bNot specified	1,060	85.86%	±2.48 pp SE
89	Claude Fable 5Max	1,060	85.86%	±2.48 pp SE
93	Claude Opus 4.516K	1,059	85.48%	±2.15 pp SE
93	Nemotron 3 UltraNot specified	1,059	85.35%	±2.52 pp SE
93	Claude Opus 4.8None	1,059	85.35%	±2.52 pp SE
93	GPT-5Medium	1,059	85.35%	±2.13 pp SE
97	Gemini 2.5 ProNot specified	1,058	85.29%	±2.12 pp SE
97	GPT-5.1Medium	1,058	85.04%	±2.13 pp SE
99	qwen3.6 35b a3bNone	1,057	84.85%	±2.55 pp SE
99	GPT-5.4Low	1,057	84.85%	±2.55 pp SE
99	qwen3.5 PlusNot specified	1,057	84.85%	±2.55 pp SE
99	Kimi K3Low	1,057	84.85%	±2.55 pp SE
99	Gemini 2.5 Pro Preview 06 05Not specified	1,057	84.85%	±2.55 pp SE
99	qwen3.6 27bNone	1,057	84.85%	±2.55 pp SE
105	Kimi k2 Thinking TurboNot specified	1,055	84.22%	±2.09 pp SE
106	qwen3.6 35b a3bNot specified	1,054	83.84%	±2.62 pp SE
106	Gemini 2.5 Pro Exp 03 25Not specified	1,054	83.84%	±2.62 pp SE
108	GPT-5.4 MiniHigh	1,053	83.59%	±2.23 pp SE
109	qwen3.5 35b a3bNot specified	1,052	83.46%	±2.18 pp SE
109	DeepSeek ReasonerNot specified	1,052	83.42%	±2.02 pp SE
109	GLM 4.7Not specified	1,052	83.33%	±2.37 pp SE
109	Gemini 3.5 Flash-LiteHigh	1,052	83.33%	±2.66 pp SE
109	Claude Fable 5High	1,052	83.33%	±2.66 pp SE
109	Claude Sonnet 4.6High	1,052	83.33%	±2.66 pp SE
109	Claude Sonnet 4.6Medium	1,052	83.33%	±2.66 pp SE
109	qwen3.6 FlashNot specified	1,052	83.33%	±2.66 pp SE
117	Gemini 3 Flash PreviewNot specified	1,051	83.21%	±1.9 pp SE
118	GPT-5.6 SolNone	1,050	82.83%	±2.69 pp SE
118	GPT-5.2Low	1,050	82.7%	±2.27 pp SE
120	GPT-5.5 InstantNot specified	1,049	82.51%	±2.25 pp SE
121	qwen3.7 FlashNot specified	1,048	82.32%	±2.72 pp SE
121	GPT-5.6 LunaLow	1,048	82.32%	±2.72 pp SE
121	qwen3.5 FlashNot specified	1,048	82.32%	±2.72 pp SE
121	Claude Sonnet 4.559K	1,048	82.32%	±2.72 pp SE
125	qwen3.7 PlusNone	1,047	81.82%	±2.75 pp SE
125	o3High	1,047	81.82%	±2.13 pp SE
125	Gemini 3.1 Flash LiteHigh	1,047	81.82%	±2.75 pp SE
128	Claude Sonnet 4.532K	1,046	81.73%	±2.76 pp SE
129	MiniMax M3None	1,045	81.31%	±2.78 pp SE
130	qwen3.5 35b a3bNone	1,044	81.19%	±2.32 pp SE
131	o3Medium	1,043	80.81%	±2.81 pp SE
131	qwen3.7 FlashNone	1,043	80.81%	±2.81 pp SE
131	Claude Opus 4.5Not specified	1,043	80.68%	±2.39 pp SE
134	Claude Sonnet 5Max	1,041	80.3%	±2.83 pp SE
134	Qwen3 235B A22B Thinking 2507Not specified	1,041	80.05%	±2.6 pp SE
136	o3Low	1,040	79.8%	±2.86 pp SE
137	o4-miniHigh	1,039	79.61%	±2.4 pp SE
138	qwen3.5 9BNot specified	1,037	78.98%	±2.37 pp SE
138	qwen3.5 9BNone	1,037	78.91%	±2.39 pp SE
140	Claude Sonnet 4.516K	1,036	78.79%	±2.91 pp SE
140	Claude Sonnet 4.6Max	1,036	78.79%	±2.91 pp SE
140	Claude Fable 5Low	1,036	78.79%	±2.91 pp SE
143	Claude 3.764K	1,035	78.5%	±2.65 pp SE
143	GPT-5.4 NanoHigh	1,035	78.47%	±2.44 pp SE
143	Claude Sonnet 432K	1,035	78.28%	±2.94 pp SE
146	Claude Sonnet 459K	1,033	77.82%	±2.67 pp SE
146	o4-miniMedium	1,033	77.78%	±2.96 pp SE
148	Claude Opus 4 1 2025080516K	1,031	77.27%	±2.99 pp SE
148	GPT-5.6 TerraNone	1,031	77.27%	±2.99 pp SE
148	GPT-5.5None	1,031	77.27%	±2.99 pp SE
151	o3-miniHigh	1,030	77.02%	±2.59 pp SE
152	Claude Opus 4 1 2025080527K	1,029	76.77%	±3.01 pp SE
152	Claude 3.732K	1,029	76.77%	±3.01 pp SE
152	o1 2024 12 17High	1,029	76.77%	±3.01 pp SE
152	Claude 3.716K	1,029	76.77%	±3.01 pp SE
156	Deepseek R1 (05/28)Not specified	1,028	76.33%	±2.4 pp SE
156	Claude Opus 416K	1,028	76.26%	±3.03 pp SE
156	Grok 3 MiniLow	1,028	76.26%	±3.03 pp SE
159	Gemini 3.5 Flash-LiteLow	1,026	75.76%	±3.05 pp SE
159	GPT-Oss 120bHigh	1,026	75.76%	±2.74 pp SE
159	Gemma 4 31b ItMinimal	1,026	75.76%	±3.05 pp SE
159	Claude Sonnet 416K	1,026	75.76%	±3.05 pp SE
159	o1 2024 12 17Medium	1,026	75.76%	±3.05 pp SE
164	o4-miniLow	1,024	75.25%	±3.07 pp SE
165	GPT-5 MiniHigh	1,023	75%	±2.28 pp SE
166	GPT-5.4None	1,022	74.75%	±3.1 pp SE
166	Grok 3 MiniHigh	1,022	74.62%	±2.77 pp SE
168	o3-miniMedium	1,021	74.27%	±2.54 pp SE
168	Gemini 3.5 Flash-LiteMinimal	1,021	74.24%	±3.12 pp SE
168	Gemini 3.1 Flash LiteLow	1,021	74.24%	±3.12 pp SE
168	o1 2024 12 17Low	1,021	74.24%	±3.12 pp SE
172	Gemini 3.1 Flash LiteMinimal	1,019	73.74%	±3.14 pp SE
172	Claude Sonnet 4.5Not specified	1,019	73.74%	±3.14 pp SE
174	GPT-5.2None	1,017	73.23%	±3.15 pp SE
174	DeepSeek V4 ProNone	1,017	73.23%	±3.15 pp SE
174	Claude Opus 4 1 20250805Not specified	1,017	73.23%	±3.15 pp SE
174	Gemma 4 26b a4b ItMinimal	1,017	73.23%	±3.15 pp SE
178	qwen3 Max 2025 09 23Not specified	1,015	72.6%	±2.73 pp SE
179	GPT-5.4 NanoLow	1,014	72.22%	±3.19 pp SE
180	GPT-5 MiniMinimal	1,012	71.72%	±3.21 pp SE
180	GPT-5Minimal	1,012	71.72%	±3.21 pp SE
180	GPT-5 MiniMedium	1,012	71.65%	±2.32 pp SE
183	Seed Oss 36b InstructNot specified	1,011	71.53%	±2.74 pp SE
184	DeepSeek ChatNot specified	1,010	71.21%	±3.23 pp SE
184	Claude Haiku 4.532K	1,010	71.21%	±3.23 pp SE
184	GLM-5.2None	1,010	71.21%	±3.23 pp SE
187	qwen3 235b a22bNot specified	1,009	70.71%	±2.71 pp SE
188	qwen3 30b a3b Thinking 2507Not specified	1,006	70.08%	±2.83 pp SE
189	GPT-5 NanoHigh	1,004	69.44%	±2.76 pp SE
189	Deepseek R1Not specified	1,004	69.22%	±3.07 pp SE
191	Claude Opus 4Not specified	1,003	69.19%	±3.29 pp SE
192	GPT-4.5 Preview 2025 02 27Not specified	1,002	68.69%	±3.3 pp SE
193	o3-miniLow	1,000	68.18%	±3.32 pp SE
194	DeepSeek V3 0324Not specified	998	67.61%	±2.68 pp SE
194	Grok 3Not specified	998	67.58%	±2.69 pp SE
196	GPT-5 NanoMedium	997	67.42%	±2.66 pp SE
197	Llama 4 MaverickNot specified	996	66.98%	±2.82 pp SE
197	GPT-4.1 2025 04 14Not specified	996	66.92%	±2.82 pp SE
199	Gemini 2.5 Pro Preview 05 06Not specified	995	66.67%	±3.36 pp SE
199	GPT-5.1None	995	66.67%	±3.36 pp SE
199	Claude Sonnet 4Not specified	995	66.67%	±3.36 pp SE
202	Claude 3.7Not specified	993	66.04%	±2.67 pp SE
203	GPT-4.1 Mini 2025 04 14Not specified	992	65.85%	±2.71 pp SE
203	Qwen3 32BNot specified	992	65.72%	±2.79 pp SE
205	Gemini 2.0 Pro Exp 02 05Not specified	991	65.66%	±3.38 pp SE
206	QwQ PlusNot specified	990	65.4%	±2.85 pp SE
206	QwQ 32BNot specified	990	65.34%	±2.81 pp SE
208	DeepSeek R1 Distill Qwen 32BNot specified	986	64.14%	±2.75 pp SE
208	Gemini 2.0 Flash 001Not specified	986	64.14%	±2.74 pp SE
208	GPT-5.4 MiniNone	986	64.14%	±3.42 pp SE
211	qwen3 14bNot specified	985	63.76%	±2.93 pp SE
212	GPT-5.6 LunaNone	984	63.64%	±3.43 pp SE
213	o1-miniHigh	980	62.37%	±2.74 pp SE
214	Qwen3 30B A3BNot specified	978	61.68%	±2.97 pp SE
215	GPT-Oss 20bMedium	975	60.8%	±2.49 pp SE
216	GLM 4.7 FlashNot specified	974	60.54%	±2.93 pp SE
216	Claude Haiku 4.5Not specified	974	60.48%	±2.82 pp SE
218	Mistral Medium 2505Not specified	970	59.53%	±2.82 pp SE
218	o1-miniMedium	970	59.5%	±2.8 pp SE
220	GPT-5 NanoLow	964	57.58%	±3.52 pp SE
221	Gemini 1.5 Pro 002Not specified	963	57.23%	±2.8 pp SE
222	Gemini 2.0 Flash Thinking Exp 01 21Not specified	962	57.07%	±3.53 pp SE
223	qwen3 8bNot specified	961	56.76%	±2.93 pp SE
224	DeepSeek V3Not specified	960	56.53%	±2.76 pp SE
225	Qwen Max 2025 01 25Not specified	959	56.12%	±2.79 pp SE
225	Magistral Small 2506Not specified	959	56.06%	±3.54 pp SE
225	Phi 4Not specified	959	56.06%	±2.59 pp SE
228	DeepSeek R1 Distill Llama 70BNot specified	957	55.74%	±2.97 pp SE
228	qwen3 30b a3b Instruct 2507Not specified	957	55.62%	±2.88 pp SE
228	GPT-5.4 NanoNone	957	55.56%	±3.54 pp SE
231	Claude 3 5 Sonnet 20241022Not specified	956	55.3%	±2.82 pp SE
232	Qwen3 32BNone	952	54.1%	±2.81 pp SE
232	Claude 3 5 Sonnet 20240620Not specified	952	54.04%	±2.75 pp SE
234	Grok 2 1212Not specified	951	53.79%	±2.71 pp SE
235	qwen3 14bNone	949	53.41%	±2.75 pp SE
235	GPT-Oss 20bLow	949	53.16%	±2.5 pp SE
237	qwen3 4bNot specified	946	52.34%	±3.01 pp SE
238	Llama 4 ScoutNot specified	944	51.83%	±3.15 pp SE
239	Mistral Large 2411Not specified	942	51.33%	±2.69 pp SE
240	Llama 3.1 405B InstructNot specified	941	50.92%	±2.59 pp SE
241	Qwen3 30B A3BNone	939	50.38%	±2.83 pp SE
241	o1 Preview 2024 09 12Not specified	939	50.32%	±2.83 pp SE
243	GPT-4o 2024 08 06Not specified	935	49.21%	±2.6 pp SE
243	qwen2.5 72b InstructNot specified	935	49.15%	±2.69 pp SE
243	Mistral Small 3.2 2506Not specified	935	49.05%	±2.77 pp SE
246	Mistral Large 2407Not specified	934	49.02%	±2.59 pp SE
246	GPT-4.1 Nano 2025 04 14Not specified	934	48.93%	±2.48 pp SE
246	GPT-4o 2024 05 13Not specified	934	48.9%	±2.63 pp SE
249	GPT-5 NanoMinimal	933	48.48%	±3.56 pp SE
250	Qwen Plus 2025 01 25Not specified	931	48.11%	±2.71 pp SE
250	GPT-4o 2024 11 20Not specified	931	47.89%	±2.63 pp SE
252	Gemma 3 27b ItNot specified	930	47.73%	±2.7 pp SE
252	Magistral Small 2509Not specified	930	47.6%	±2.6 pp SE
254	qwen3 8bNone	929	47.47%	±2.62 pp SE
254	Mistral Small 2503Not specified	929	47.47%	±2.76 pp SE
254	Llama 3.3 70B InstructNot specified	929	47.44%	±2.85 pp SE
254	Gemini 1.5 Flash 002Not specified	929	47.32%	±2.77 pp SE
254	Mistral Small 3 2501Not specified	929	47.29%	±2.79 pp SE
259	Claude 3 Opus 20240229Not specified	928	47.16%	±2.64 pp SE
260	GPT-4 Turbo 2024 04 09Not specified	926	46.59%	±2.66 pp SE
261	Llama 3.1 Tulu 3 70B DpoNot specified	925	46.28%	±2.41 pp SE
262	qwen2.5 32b InstructNot specified	924	46.09%	±2.6 pp SE
262	GPT-Oss 20bHigh	924	45.96%	±3.55 pp SE
262	Gemini 1.5 Pro 001Not specified	924	45.86%	±2.62 pp SE
262	qwen3 4b Instruct 2507Not specified	924	45.83%	±2.85 pp SE
266	Mistral Small 2501Not specified	922	45.3%	±2.48 pp SE
267	GLM 4.7 FlashNone	921	45.14%	±2.18 pp SE
268	DeepSeek R1 Distill Qwen 14BNot specified	920	44.7%	±2.86 pp SE
269	Llama 3.1 70B InstructNot specified	918	44.19%	±2.45 pp SE
270	qwen3 4bNone	916	43.56%	±2.69 pp SE
271	Wizardlm 2 8x22BNot specified	915	43.43%	±2.32 pp SE
272	GPT-4 1106 PreviewNot specified	912	42.36%	±2.41 pp SE
273	GPT-4 0125 PreviewNot specified	911	42.27%	±2.68 pp SE
274	Mistral Small 3.1 2503Not specified	910	41.92%	±2.35 pp SE
274	Qwen Turbo 2024 11 01Not specified	910	41.79%	±2.62 pp SE
276	Llama 3.2 90B Vision InstructNot specified	907	41.04%	±2.29 pp SE
277	qwen2 72b InstructNot specified	906	40.78%	±2.57 pp SE
277	Claude 3 Sonnet 20240229Not specified	906	40.59%	±2.39 pp SE
277	Meta Llama 3 70B InstructNot specified	906	40.56%	±2.62 pp SE
280	Gemini 1.5 Flash 001Not specified	905	40.37%	±2.39 pp SE
281	Gemma 3 12b ItNot specified	902	39.46%	±2.53 pp SE
282	Mistral Large 2402Not specified	899	38.76%	±2.48 pp SE
283	Claude 3 5 Haiku 20241022Not specified	897	38.13%	±2.64 pp SE
283	qwen3 1.7bNot specified	897	38.01%	±2.75 pp SE
285	GPT-4o Mini 2024 07 18Not specified	896	37.72%	±2.41 pp SE
286	Hermes 2 Theta Llama 3 70BNot specified	895	37.47%	±2.48 pp SE
287	Gemma 2 27b ItNot specified	892	36.49%	±2.39 pp SE
288	Claude 3 Haiku 20240307Not specified	891	36.3%	±2.42 pp SE
289	GPT-4 0314Not specified	889	35.73%	±2.36 pp SE
290	qwen2.5 7b InstructNot specified	888	35.48%	±2.29 pp SE
291	Claude 2.0Not specified	885	34.66%	±2.51 pp SE
292	Open Mixtral 8x22bNot specified	883	34.06%	±2.07 pp SE
292	Gemini 1.0 Pro 001Not specified	883	33.96%	±2.01 pp SE
292	Eurus 2 7B PrimeNot specified	883	33.9%	±2.11 pp SE
295	DeepSeek R1 Distill Qwen 1.5BNot specified	882	33.59%	±2.06 pp SE
296	Claude 2.1Not specified	880	32.95%	±2.32 pp SE
296	Gemini 1.5 Flash 8b 001Not specified	880	32.95%	±2.42 pp SE
298	DBRX InstructNot specified	879	32.89%	±3.26 pp SE
299	Yi 1.5 34B ChatNot specified	876	31.98%	±2 pp SE
300	qwen1.5 32b ChatNot specified	872	30.74%	±2.02 pp SE
300	GPT-4 0613Not specified	872	30.65%	±2.34 pp SE
300	qwen3 1.7bNone	872	30.62%	±2.26 pp SE
303	Mixtral 8x7B Instruct v0.1Not specified	871	30.59%	±1.94 pp SE
304	Open Mistral Nemo 2407Not specified	869	29.89%	±2.1 pp SE
304	Open Mixtral 8x7bNot specified	869	29.83%	±1.91 pp SE
306	qwen1.5 72b ChatNot specified	865	28.82%	±1.99 pp SE
307	Granite 4.0 MicroNot specified	864	28.28%	±1.82 pp SE
308	GPT-3.5 Turbo 1106Not specified	863	28.03%	±1.87 pp SE
309	Phi 3 Medium 128k InstructNot specified	861	27.59%	±1.61 pp SE
309	Gemma 2 9b ItNot specified	861	27.46%	±2.3 pp SE
311	GPT-3.5 Turbo 0125Not specified	860	27.18%	±1.95 pp SE
311	Ministral 8b 2410Not specified	860	27.15%	±2.09 pp SE
313	Llama 3.1 8B InstructNot specified	859	26.96%	±1.86 pp SE
314	Llama 2 70b Chat HfNot specified	857	26.33%	±1.99 pp SE
315	Meta Llama 3 8B InstructNot specified	856	26.07%	±1.7 pp SE
316	Ministral 3b 2410Not specified	853	25.25%	±1.93 pp SE
317	DeepSeek Llm 67b ChatNot specified	851	24.62%	±1.36 pp SE
318	Granite 4.0 1bNot specified	849	23.99%	±1.62 pp SE
318	Llama 3.2 1B InstructNot specified	849	23.93%	±1.63 pp SE
320	Gemma 3 4b ItNot specified	846	23.23%	±1.87 pp SE
321	Gemma 3 1b ItNot specified	835	19.95%	±1.52 pp SE
322	Mistral 7B Instruct v0.3Not specified	819	15.18%	±1.01 pp SE
323	Yi 34B ChatNot specified	817	14.74%	±1.14 pp SE
324	Open Mistral 7bNot specified	812	13.23%	±0.99 pp SE
325	Granite 4.0 350mNot specified	805	11.24%	±0.91 pp SE
326	DeepSeek r1 0528 qwen3 8bNot specified	799	9.28%	±1.07 pp SE

1 · Gemini 4 ArgonHigh1096

Accuracy: 96.4%
Standard error: ±1.19 pp SE
Evidence: Model profile ↗

2 · GPT-6 AstraMax1094

Accuracy: 95.77%
Standard error: ±1.37 pp SE
Evidence: Model profile ↗

2 · Claude Sonnet 5.5Max1094

Accuracy: 95.58%
Standard error: ±1.37 pp SE
Evidence: Model profile ↗

4 · Gemini 3.8 FlashHigh1093

Accuracy: 95.39%
Standard error: ±1.4 pp SE
Evidence: Model profile ↗

4 · GPT-6.1 SolMax1093

Accuracy: 95.39%
Standard error: ±1.38 pp SE
Evidence: Model profile ↗

6 · Gemini 3.7 FlashHigh1091

Accuracy: 94.82%
Standard error: ±1.35 pp SE
Evidence: Model profile ↗

7 · GPT-5.4 ProxHigh1090

Accuracy: 94.6%
Standard error: ±1.6 pp SE
Evidence: Model profile ↗

7 · Gemini 3.1 Pro (Preview)High1090

Accuracy: 94.44%
Standard error: ±1.63 pp SE
Evidence: Model profile ↗

9 · GPT-6.1 TerraMax1089

Accuracy: 94.32%
Standard error: ±1.49 pp SE
Evidence: Model profile ↗

9 · GPT-6 SolMax1089

Accuracy: 94.32%
Standard error: ±1.49 pp SE
Evidence: Model profile ↗

9 · Gemini 3.6 FlashHigh1089

Accuracy: 94.13%
Standard error: ±1.4 pp SE
Evidence: Model profile ↗

9 · Gemini 3.1 Pro (Preview)Not specified1089

Accuracy: 94.1%
Standard error: ±1.7 pp SE
Evidence: Model profile ↗

13 · GPT-5.5 Pre ReleasexHigh1088

Accuracy: 94%
Standard error: ±1.55 pp SE
Evidence: Model profile ↗

13 · Grok 4.6High1088

Accuracy: 94%
Standard error: ±1.45 pp SE
Evidence: Model profile ↗

13 · GPT-5.5 Pro Pre ReleasexHigh1088

Accuracy: 93.92%
Standard error: ±1.58 pp SE
Evidence: Model profile ↗

13 · Claude Opus 5Max1088

Accuracy: 93.88%
Standard error: ±1.48 pp SE
Evidence: Model profile ↗

17 · GPT-5.6 SolMax1087

Accuracy: 93.5%
Standard error: ±1.57 pp SE
Evidence: Model profile ↗

18 · Grok 4.5High1086

Accuracy: 93.43%
Standard error: ±1.43 pp SE
Evidence: Model profile ↗

18 · GPT-5.6 TerraMax1086

Accuracy: 93.31%
Standard error: ±1.54 pp SE
Evidence: Model profile ↗

18 · GPT-5.4xHigh1086

Accuracy: 93.3%
Standard error: ±1.8 pp SE
Evidence: Model profile ↗

21 · Grok 4.6xHigh1085

Accuracy: 93.18%
Standard error: ±1.52 pp SE
Evidence: Model profile ↗

21 · DeepSeek V4.1 Flash 1017Max1085

Accuracy: 93.18%
Standard error: ±1.57 pp SE
Evidence: Model profile ↗

21 · Kimi K3Max1085

Accuracy: 93.12%
Standard error: ±1.49 pp SE
Evidence: Model profile ↗

21 · Claude Opus 5Not specified1085

Accuracy: 92.93%
Standard error: ±1.83 pp SE
Evidence: Model profile ↗

25 · Gemini 3.5 FlashHigh1084

Accuracy: 92.8%
Standard error: ±1.64 pp SE
Evidence: Model profile ↗

25 · Grok 4.7xHigh1084

Accuracy: 92.74%
Standard error: ±1.55 pp SE
Evidence: Model profile ↗

25 · qwen3.8 MaxxHigh1084

Accuracy: 92.68%
Standard error: ±1.69 pp SE
Evidence: Model profile ↗

25 · Gemini 3 Pro PreviewNot specified1084

Accuracy: 92.61%
Standard error: ±1.65 pp SE
Evidence: Model profile ↗

29 · qwen3.8 Max 0902xHigh1082

Accuracy: 92.3%
Standard error: ±1.72 pp SE
Evidence: Model profile ↗

30 · Kimi K3High1081

Accuracy: 91.92%
Standard error: ±1.94 pp SE
Evidence: Model profile ↗

30 · GLM-5.2Max1081

Accuracy: 91.86%
Standard error: ±1.61 pp SE
Evidence: Model profile ↗

32 · DeepSeek V4 Pro 0813Max1080

Accuracy: 91.67%
Standard error: ±1.52 pp SE
Evidence: Model profile ↗

32 · GPT-5.6 LunaMax1080

Accuracy: 91.6%
Standard error: ±1.73 pp SE
Evidence: Model profile ↗

34 · GPT-5.2xHigh1079

Accuracy: 91.4%
Standard error: ±1.8 pp SE
Evidence: Model profile ↗

35 · Claude Opus 4.8Max1078

Accuracy: 91.04%
Standard error: ±1.92 pp SE
Evidence: Model profile ↗

35 · DeepSeek V4 Flash 0731Max1078

Accuracy: 91.04%
Standard error: ±1.77 pp SE
Evidence: Model profile ↗

35 · GLM 5.3Max1078

Accuracy: 90.91%
Standard error: ±1.59 pp SE
Evidence: Model profile ↗

35 · MiniMax M3Not specified1078

Accuracy: 90.91%
Standard error: ±2.05 pp SE
Evidence: Model profile ↗

35 · DeepSeek V4 ProHigh1078

Accuracy: 90.91%
Standard error: ±2.05 pp SE
Evidence: Model profile ↗

35 · qwen3.7 MaxNot specified1078

Accuracy: 90.91%
Standard error: ±2.05 pp SE
Evidence: Model profile ↗

41 · Kimi k2.6Not specified1077

Accuracy: 90.78%
Standard error: ±1.72 pp SE
Evidence: Model profile ↗

41 · GPT-5.5Low1077

Accuracy: 90.66%
Standard error: ±1.75 pp SE
Evidence: Model profile ↗

41 · Claude Opus 5.5Max1077

Accuracy: 90.59%
Standard error: ±1.94 pp SE
Evidence: Model profile ↗

44 · Claude Opus 4.632K1076

Accuracy: 90.53%
Standard error: ±1.69 pp SE
Evidence: Model profile ↗

44 · Claude Sonnet 5xHigh1076

Accuracy: 90.53%
Standard error: ±1.78 pp SE
Evidence: Model profile ↗

44 · GPT-6 LunaMax1076

Accuracy: 90.47%
Standard error: ±1.85 pp SE
Evidence: Model profile ↗

47 · GLM 5.3 FlashMax1075

Accuracy: 90.15%
Standard error: ±1.72 pp SE
Evidence: Model profile ↗

47 · Claude 4.7xHigh1075

Accuracy: 90.15%
Standard error: ±1.85 pp SE
Evidence: Model profile ↗

49 · GPT-5.6 SolLow1074

Accuracy: 89.9%
Standard error: ±2.15 pp SE
Evidence: Model profile ↗

49 · GLM 5.1Not specified1074

Accuracy: 89.9%
Standard error: ±2.15 pp SE
Evidence: Model profile ↗

49 · GPT-5.4High1074

Accuracy: 89.9%
Standard error: ±2.15 pp SE
Evidence: Model profile ↗

49 · DeepSeek V4.1 FlashMax1074

Accuracy: 89.84%
Standard error: ±1.73 pp SE
Evidence: Model profile ↗

49 · Muse SparkNot specified1074

Accuracy: 89.8%
Standard error: ±2.2 pp SE
Evidence: Model profile ↗

54 · DeepSeek V4 ProMax1073

Accuracy: 89.65%
Standard error: ±1.75 pp SE
Evidence: Model profile ↗

54 · Claude Haiku 5.5Max1073

Accuracy: 89.65%
Standard error: ±1.98 pp SE
Evidence: Model profile ↗

54 · Gemini 3 Flash PreviewHigh1073

Accuracy: 89.39%
Standard error: ±2.19 pp SE
Evidence: Model profile ↗

57 · Grok 4.20 (Reasoning)Not specified1072

Accuracy: 89.33%
Standard error: ±1.86 pp SE
Evidence: Model profile ↗

58 · Gemini 3.5 FlashLow1071

Accuracy: 88.89%
Standard error: ±2.24 pp SE
Evidence: Model profile ↗

58 · GPT-5.4Medium1071

Accuracy: 88.89%
Standard error: ±2.24 pp SE
Evidence: Model profile ↗

58 · Grok 4.3High1071

Accuracy: 88.83%
Standard error: ±1.96 pp SE
Evidence: Model profile ↗

61 · Claude Opus 4.664K1070

Accuracy: 88.76%
Standard error: ±1.94 pp SE
Evidence: Model profile ↗

61 · Muse Spark OpenxHigh1070

Accuracy: 88.57%
Standard error: ±1.87 pp SE
Evidence: Model profile ↗

63 · Inkling SmallxHigh1069

Accuracy: 88.51%
Standard error: ±1.91 pp SE
Evidence: Model profile ↗

63 · qwen3.6 PlusNot specified1069

Accuracy: 88.38%
Standard error: ±2.28 pp SE
Evidence: Model profile ↗

63 · Claude Opus 4.8Low1069

Accuracy: 88.38%
Standard error: ±2.28 pp SE
Evidence: Model profile ↗

63 · Claude Opus 4.6Max1069

Accuracy: 88.38%
Standard error: ±2.28 pp SE
Evidence: Model profile ↗

63 · InklingxHigh1069

Accuracy: 88.26%
Standard error: ±1.94 pp SE
Evidence: Model profile ↗

68 · GPT-5.2High1068

Accuracy: 88.19%
Standard error: ±1.91 pp SE
Evidence: Model profile ↗

68 · Mistral Large 4 0High1068

Accuracy: 88.07%
Standard error: ±1.93 pp SE
Evidence: Model profile ↗

70 · GPT-5.2Medium1067

Accuracy: 87.88%
Standard error: ±1.91 pp SE
Evidence: Model profile ↗

70 · GLM-5.2Low1067

Accuracy: 87.88%
Standard error: ±2.33 pp SE
Evidence: Model profile ↗

70 · qwen3.7 PlusNot specified1067

Accuracy: 87.88%
Standard error: ±2.33 pp SE
Evidence: Model profile ↗

70 · Claude Opus 5Low1067

Accuracy: 87.88%
Standard error: ±2.33 pp SE
Evidence: Model profile ↗

70 · Kimi k2.7 CodeNot specified1067

Accuracy: 87.88%
Standard error: ±2.33 pp SE
Evidence: Model profile ↗

70 · GLM-5Not specified1067

Accuracy: 87.82%
Standard error: ±2.34 pp SE
Evidence: Model profile ↗

76 · GPT-5.1High1066

Accuracy: 87.63%
Standard error: ±1.88 pp SE
Evidence: Model profile ↗

76 · Kimi k2p5Not specified1066

Accuracy: 87.6%
Standard error: ±1.9 pp SE
Evidence: Model profile ↗

76 · Claude Sonnet 4.632K1066

Accuracy: 87.37%
Standard error: ±2 pp SE
Evidence: Model profile ↗

76 · GPT-5.6 TerraLow1066

Accuracy: 87.37%
Standard error: ±2.37 pp SE
Evidence: Model profile ↗

76 · qwen3.6 Max PreviewNot specified1066

Accuracy: 87.37%
Standard error: ±2.37 pp SE
Evidence: Model profile ↗

81 · Grok 4Not specified1064

Accuracy: 87%
Standard error: ±2 pp SE
Evidence: Model profile ↗

81 · GPT-5.4 MinixHigh1064

Accuracy: 86.87%
Standard error: ±2.41 pp SE
Evidence: Model profile ↗

83 · qwen3.5 397b a17bNone1062

Accuracy: 86.36%
Standard error: ±2.45 pp SE
Evidence: Model profile ↗

83 · Gemini 3.6 FlashLow1062

Accuracy: 86.36%
Standard error: ±2.45 pp SE
Evidence: Model profile ↗

83 · Claude 4.7Max1062

Accuracy: 86.36%
Standard error: ±2.45 pp SE
Evidence: Model profile ↗

83 · Gemini 3.5 FlashMinimal1062

Accuracy: 86.36%
Standard error: ±2.45 pp SE
Evidence: Model profile ↗

83 · GPT-5High1062

Accuracy: 86.17%
Standard error: ±2.08 pp SE
Evidence: Model profile ↗

88 · Claude Opus 4.532K1061

Accuracy: 86.05%
Standard error: ±2.07 pp SE
Evidence: Model profile ↗

89 · Gemini 3.6 FlashMinimal1060

Accuracy: 85.86%
Standard error: ±2.48 pp SE
Evidence: Model profile ↗

89 · qwen3.6 27bNot specified1060

Accuracy: 85.86%
Standard error: ±2.48 pp SE
Evidence: Model profile ↗

89 · qwen3.5 397b a17bNot specified1060

Accuracy: 85.86%
Standard error: ±2.48 pp SE
Evidence: Model profile ↗

89 · Claude Fable 5Max1060

Accuracy: 85.86%
Standard error: ±2.48 pp SE
Evidence: Model profile ↗

93 · Claude Opus 4.516K1059

Accuracy: 85.48%
Standard error: ±2.15 pp SE
Evidence: Model profile ↗

93 · Nemotron 3 UltraNot specified1059

Accuracy: 85.35%
Standard error: ±2.52 pp SE
Evidence: Model profile ↗

93 · Claude Opus 4.8None1059

Accuracy: 85.35%
Standard error: ±2.52 pp SE
Evidence: Model profile ↗

93 · GPT-5Medium1059

Accuracy: 85.35%
Standard error: ±2.13 pp SE
Evidence: Model profile ↗

97 · Gemini 2.5 ProNot specified1058

Accuracy: 85.29%
Standard error: ±2.12 pp SE
Evidence: Model profile ↗

97 · GPT-5.1Medium1058

Accuracy: 85.04%
Standard error: ±2.13 pp SE
Evidence: Model profile ↗

99 · qwen3.6 35b a3bNone1057

Accuracy: 84.85%
Standard error: ±2.55 pp SE
Evidence: Model profile ↗

99 · GPT-5.4Low1057

Accuracy: 84.85%
Standard error: ±2.55 pp SE
Evidence: Model profile ↗

99 · qwen3.5 PlusNot specified1057

Accuracy: 84.85%
Standard error: ±2.55 pp SE
Evidence: Model profile ↗

99 · Kimi K3Low1057

Accuracy: 84.85%
Standard error: ±2.55 pp SE
Evidence: Model profile ↗

99 · Gemini 2.5 Pro Preview 06 05Not specified1057

Accuracy: 84.85%
Standard error: ±2.55 pp SE
Evidence: Model profile ↗

99 · qwen3.6 27bNone1057

Accuracy: 84.85%
Standard error: ±2.55 pp SE
Evidence: Model profile ↗

105 · Kimi k2 Thinking TurboNot specified1055

Accuracy: 84.22%
Standard error: ±2.09 pp SE
Evidence: Model profile ↗

106 · qwen3.6 35b a3bNot specified1054

Accuracy: 83.84%
Standard error: ±2.62 pp SE
Evidence: Model profile ↗

106 · Gemini 2.5 Pro Exp 03 25Not specified1054

Accuracy: 83.84%
Standard error: ±2.62 pp SE
Evidence: Model profile ↗

108 · GPT-5.4 MiniHigh1053

Accuracy: 83.59%
Standard error: ±2.23 pp SE
Evidence: Model profile ↗

109 · qwen3.5 35b a3bNot specified1052

Accuracy: 83.46%
Standard error: ±2.18 pp SE
Evidence: Model profile ↗

109 · DeepSeek ReasonerNot specified1052

Accuracy: 83.42%
Standard error: ±2.02 pp SE
Evidence: Model profile ↗

109 · GLM 4.7Not specified1052

Accuracy: 83.33%
Standard error: ±2.37 pp SE
Evidence: Model profile ↗

109 · Gemini 3.5 Flash-LiteHigh1052

Accuracy: 83.33%
Standard error: ±2.66 pp SE
Evidence: Model profile ↗

109 · Claude Fable 5High1052

Accuracy: 83.33%
Standard error: ±2.66 pp SE
Evidence: Model profile ↗

109 · Claude Sonnet 4.6High1052

Accuracy: 83.33%
Standard error: ±2.66 pp SE
Evidence: Model profile ↗

109 · Claude Sonnet 4.6Medium1052

Accuracy: 83.33%
Standard error: ±2.66 pp SE
Evidence: Model profile ↗

109 · qwen3.6 FlashNot specified1052

Accuracy: 83.33%
Standard error: ±2.66 pp SE
Evidence: Model profile ↗

117 · Gemini 3 Flash PreviewNot specified1051

Accuracy: 83.21%
Standard error: ±1.9 pp SE
Evidence: Model profile ↗

118 · GPT-5.6 SolNone1050

Accuracy: 82.83%
Standard error: ±2.69 pp SE
Evidence: Model profile ↗

118 · GPT-5.2Low1050

Accuracy: 82.7%
Standard error: ±2.27 pp SE
Evidence: Model profile ↗

120 · GPT-5.5 InstantNot specified1049

Accuracy: 82.51%
Standard error: ±2.25 pp SE
Evidence: Model profile ↗

121 · qwen3.7 FlashNot specified1048

Accuracy: 82.32%
Standard error: ±2.72 pp SE
Evidence: Model profile ↗

121 · GPT-5.6 LunaLow1048

Accuracy: 82.32%
Standard error: ±2.72 pp SE
Evidence: Model profile ↗

121 · qwen3.5 FlashNot specified1048

Accuracy: 82.32%
Standard error: ±2.72 pp SE
Evidence: Model profile ↗

121 · Claude Sonnet 4.559K1048

Accuracy: 82.32%
Standard error: ±2.72 pp SE
Evidence: Model profile ↗

125 · qwen3.7 PlusNone1047

Accuracy: 81.82%
Standard error: ±2.75 pp SE
Evidence: Model profile ↗

125 · o3High1047

Accuracy: 81.82%
Standard error: ±2.13 pp SE
Evidence: Model profile ↗

125 · Gemini 3.1 Flash LiteHigh1047

Accuracy: 81.82%
Standard error: ±2.75 pp SE
Evidence: Model profile ↗

128 · Claude Sonnet 4.532K1046

Accuracy: 81.73%
Standard error: ±2.76 pp SE
Evidence: Model profile ↗

129 · MiniMax M3None1045

Accuracy: 81.31%
Standard error: ±2.78 pp SE
Evidence: Model profile ↗

130 · qwen3.5 35b a3bNone1044

Accuracy: 81.19%
Standard error: ±2.32 pp SE
Evidence: Model profile ↗

131 · o3Medium1043

Accuracy: 80.81%
Standard error: ±2.81 pp SE
Evidence: Model profile ↗

131 · qwen3.7 FlashNone1043

Accuracy: 80.81%
Standard error: ±2.81 pp SE
Evidence: Model profile ↗

131 · Claude Opus 4.5Not specified1043

Accuracy: 80.68%
Standard error: ±2.39 pp SE
Evidence: Model profile ↗

134 · Claude Sonnet 5Max1041

Accuracy: 80.3%
Standard error: ±2.83 pp SE
Evidence: Model profile ↗

134 · Qwen3 235B A22B Thinking 2507Not specified1041

Accuracy: 80.05%
Standard error: ±2.6 pp SE
Evidence: Model profile ↗

136 · o3Low1040

Accuracy: 79.8%
Standard error: ±2.86 pp SE
Evidence: Model profile ↗

137 · o4-miniHigh1039

Accuracy: 79.61%
Standard error: ±2.4 pp SE
Evidence: Model profile ↗

138 · qwen3.5 9BNot specified1037

Accuracy: 78.98%
Standard error: ±2.37 pp SE
Evidence: Model profile ↗

138 · qwen3.5 9BNone1037

Accuracy: 78.91%
Standard error: ±2.39 pp SE
Evidence: Model profile ↗

140 · Claude Sonnet 4.516K1036

Accuracy: 78.79%
Standard error: ±2.91 pp SE
Evidence: Model profile ↗

140 · Claude Sonnet 4.6Max1036

Accuracy: 78.79%
Standard error: ±2.91 pp SE
Evidence: Model profile ↗

140 · Claude Fable 5Low1036

Accuracy: 78.79%
Standard error: ±2.91 pp SE
Evidence: Model profile ↗

143 · Claude 3.764K1035

Accuracy: 78.5%
Standard error: ±2.65 pp SE
Evidence: Model profile ↗

143 · GPT-5.4 NanoHigh1035

Accuracy: 78.47%
Standard error: ±2.44 pp SE
Evidence: Model profile ↗

143 · Claude Sonnet 432K1035

Accuracy: 78.28%
Standard error: ±2.94 pp SE
Evidence: Model profile ↗

146 · Claude Sonnet 459K1033

Accuracy: 77.82%
Standard error: ±2.67 pp SE
Evidence: Model profile ↗

146 · o4-miniMedium1033

Accuracy: 77.78%
Standard error: ±2.96 pp SE
Evidence: Model profile ↗

148 · Claude Opus 4 1 2025080516K1031

Accuracy: 77.27%
Standard error: ±2.99 pp SE
Evidence: Model profile ↗

148 · GPT-5.6 TerraNone1031

Accuracy: 77.27%
Standard error: ±2.99 pp SE
Evidence: Model profile ↗

148 · GPT-5.5None1031

Accuracy: 77.27%
Standard error: ±2.99 pp SE
Evidence: Model profile ↗

151 · o3-miniHigh1030

Accuracy: 77.02%
Standard error: ±2.59 pp SE
Evidence: Model profile ↗

152 · Claude Opus 4 1 2025080527K1029

Accuracy: 76.77%
Standard error: ±3.01 pp SE
Evidence: Model profile ↗

152 · Claude 3.732K1029

Accuracy: 76.77%
Standard error: ±3.01 pp SE
Evidence: Model profile ↗

152 · o1 2024 12 17High1029

Accuracy: 76.77%
Standard error: ±3.01 pp SE
Evidence: Model profile ↗

152 · Claude 3.716K1029

Accuracy: 76.77%
Standard error: ±3.01 pp SE
Evidence: Model profile ↗

156 · Deepseek R1 (05/28)Not specified1028

Accuracy: 76.33%
Standard error: ±2.4 pp SE
Evidence: Model profile ↗

156 · Claude Opus 416K1028

Accuracy: 76.26%
Standard error: ±3.03 pp SE
Evidence: Model profile ↗

156 · Grok 3 MiniLow1028

Accuracy: 76.26%
Standard error: ±3.03 pp SE
Evidence: Model profile ↗

159 · Gemini 3.5 Flash-LiteLow1026

Accuracy: 75.76%
Standard error: ±3.05 pp SE
Evidence: Model profile ↗

159 · GPT-Oss 120bHigh1026

Accuracy: 75.76%
Standard error: ±2.74 pp SE
Evidence: Model profile ↗

159 · Gemma 4 31b ItMinimal1026

Accuracy: 75.76%
Standard error: ±3.05 pp SE
Evidence: Model profile ↗

159 · Claude Sonnet 416K1026

Accuracy: 75.76%
Standard error: ±3.05 pp SE
Evidence: Model profile ↗

159 · o1 2024 12 17Medium1026

Accuracy: 75.76%
Standard error: ±3.05 pp SE
Evidence: Model profile ↗

164 · o4-miniLow1024

Accuracy: 75.25%
Standard error: ±3.07 pp SE
Evidence: Model profile ↗

165 · GPT-5 MiniHigh1023

Accuracy: 75%
Standard error: ±2.28 pp SE
Evidence: Model profile ↗

166 · GPT-5.4None1022

Accuracy: 74.75%
Standard error: ±3.1 pp SE
Evidence: Model profile ↗

166 · Grok 3 MiniHigh1022

Accuracy: 74.62%
Standard error: ±2.77 pp SE
Evidence: Model profile ↗

168 · o3-miniMedium1021

Accuracy: 74.27%
Standard error: ±2.54 pp SE
Evidence: Model profile ↗

168 · Gemini 3.5 Flash-LiteMinimal1021

Accuracy: 74.24%
Standard error: ±3.12 pp SE
Evidence: Model profile ↗

168 · Gemini 3.1 Flash LiteLow1021

Accuracy: 74.24%
Standard error: ±3.12 pp SE
Evidence: Model profile ↗

168 · o1 2024 12 17Low1021

Accuracy: 74.24%
Standard error: ±3.12 pp SE
Evidence: Model profile ↗

172 · Gemini 3.1 Flash LiteMinimal1019

Accuracy: 73.74%
Standard error: ±3.14 pp SE
Evidence: Model profile ↗

172 · Claude Sonnet 4.5Not specified1019

Accuracy: 73.74%
Standard error: ±3.14 pp SE
Evidence: Model profile ↗

174 · GPT-5.2None1017

Accuracy: 73.23%
Standard error: ±3.15 pp SE
Evidence: Model profile ↗

174 · DeepSeek V4 ProNone1017

Accuracy: 73.23%
Standard error: ±3.15 pp SE
Evidence: Model profile ↗

174 · Claude Opus 4 1 20250805Not specified1017

Accuracy: 73.23%
Standard error: ±3.15 pp SE
Evidence: Model profile ↗

174 · Gemma 4 26b a4b ItMinimal1017

Accuracy: 73.23%
Standard error: ±3.15 pp SE
Evidence: Model profile ↗

178 · qwen3 Max 2025 09 23Not specified1015

Accuracy: 72.6%
Standard error: ±2.73 pp SE
Evidence: Model profile ↗

179 · GPT-5.4 NanoLow1014

Accuracy: 72.22%
Standard error: ±3.19 pp SE
Evidence: Model profile ↗

180 · GPT-5 MiniMinimal1012

Accuracy: 71.72%
Standard error: ±3.21 pp SE
Evidence: Model profile ↗

180 · GPT-5Minimal1012

Accuracy: 71.72%
Standard error: ±3.21 pp SE
Evidence: Model profile ↗

180 · GPT-5 MiniMedium1012

Accuracy: 71.65%
Standard error: ±2.32 pp SE
Evidence: Model profile ↗

183 · Seed Oss 36b InstructNot specified1011

Accuracy: 71.53%
Standard error: ±2.74 pp SE
Evidence: Model profile ↗

184 · DeepSeek ChatNot specified1010

Accuracy: 71.21%
Standard error: ±3.23 pp SE
Evidence: Model profile ↗

184 · Claude Haiku 4.532K1010

Accuracy: 71.21%
Standard error: ±3.23 pp SE
Evidence: Model profile ↗

184 · GLM-5.2None1010

Accuracy: 71.21%
Standard error: ±3.23 pp SE
Evidence: Model profile ↗

187 · qwen3 235b a22bNot specified1009

Accuracy: 70.71%
Standard error: ±2.71 pp SE
Evidence: Model profile ↗

188 · qwen3 30b a3b Thinking 2507Not specified1006

Accuracy: 70.08%
Standard error: ±2.83 pp SE
Evidence: Model profile ↗

189 · GPT-5 NanoHigh1004

Accuracy: 69.44%
Standard error: ±2.76 pp SE
Evidence: Model profile ↗

189 · Deepseek R1Not specified1004

Accuracy: 69.22%
Standard error: ±3.07 pp SE
Evidence: Model profile ↗

191 · Claude Opus 4Not specified1003

Accuracy: 69.19%
Standard error: ±3.29 pp SE
Evidence: Model profile ↗

192 · GPT-4.5 Preview 2025 02 27Not specified1002

Accuracy: 68.69%
Standard error: ±3.3 pp SE
Evidence: Model profile ↗

193 · o3-miniLow1000

Accuracy: 68.18%
Standard error: ±3.32 pp SE
Evidence: Model profile ↗

194 · DeepSeek V3 0324Not specified998

Accuracy: 67.61%
Standard error: ±2.68 pp SE
Evidence: Model profile ↗

194 · Grok 3Not specified998

Accuracy: 67.58%
Standard error: ±2.69 pp SE
Evidence: Model profile ↗

196 · GPT-5 NanoMedium997

Accuracy: 67.42%
Standard error: ±2.66 pp SE
Evidence: Model profile ↗

197 · Llama 4 MaverickNot specified996

Accuracy: 66.98%
Standard error: ±2.82 pp SE
Evidence: Model profile ↗

197 · GPT-4.1 2025 04 14Not specified996

Accuracy: 66.92%
Standard error: ±2.82 pp SE
Evidence: Model profile ↗

199 · Gemini 2.5 Pro Preview 05 06Not specified995

Accuracy: 66.67%
Standard error: ±3.36 pp SE
Evidence: Model profile ↗

199 · GPT-5.1None995

Accuracy: 66.67%
Standard error: ±3.36 pp SE
Evidence: Model profile ↗

199 · Claude Sonnet 4Not specified995

Accuracy: 66.67%
Standard error: ±3.36 pp SE
Evidence: Model profile ↗

202 · Claude 3.7Not specified993

Accuracy: 66.04%
Standard error: ±2.67 pp SE
Evidence: Model profile ↗

203 · GPT-4.1 Mini 2025 04 14Not specified992

Accuracy: 65.85%
Standard error: ±2.71 pp SE
Evidence: Model profile ↗

203 · Qwen3 32BNot specified992

Accuracy: 65.72%
Standard error: ±2.79 pp SE
Evidence: Model profile ↗

205 · Gemini 2.0 Pro Exp 02 05Not specified991

Accuracy: 65.66%
Standard error: ±3.38 pp SE
Evidence: Model profile ↗

206 · QwQ PlusNot specified990

Accuracy: 65.4%
Standard error: ±2.85 pp SE
Evidence: Model profile ↗

206 · QwQ 32BNot specified990

Accuracy: 65.34%
Standard error: ±2.81 pp SE
Evidence: Model profile ↗

208 · DeepSeek R1 Distill Qwen 32BNot specified986

Accuracy: 64.14%
Standard error: ±2.75 pp SE
Evidence: Model profile ↗

208 · Gemini 2.0 Flash 001Not specified986

Accuracy: 64.14%
Standard error: ±2.74 pp SE
Evidence: Model profile ↗

208 · GPT-5.4 MiniNone986

Accuracy: 64.14%
Standard error: ±3.42 pp SE
Evidence: Model profile ↗

211 · qwen3 14bNot specified985

Accuracy: 63.76%
Standard error: ±2.93 pp SE
Evidence: Model profile ↗

212 · GPT-5.6 LunaNone984

Accuracy: 63.64%
Standard error: ±3.43 pp SE
Evidence: Model profile ↗

213 · o1-miniHigh980

Accuracy: 62.37%
Standard error: ±2.74 pp SE
Evidence: Model profile ↗

214 · Qwen3 30B A3BNot specified978

Accuracy: 61.68%
Standard error: ±2.97 pp SE
Evidence: Model profile ↗

215 · GPT-Oss 20bMedium975

Accuracy: 60.8%
Standard error: ±2.49 pp SE
Evidence: Model profile ↗

216 · GLM 4.7 FlashNot specified974

Accuracy: 60.54%
Standard error: ±2.93 pp SE
Evidence: Model profile ↗

216 · Claude Haiku 4.5Not specified974

Accuracy: 60.48%
Standard error: ±2.82 pp SE
Evidence: Model profile ↗

218 · Mistral Medium 2505Not specified970

Accuracy: 59.53%
Standard error: ±2.82 pp SE
Evidence: Model profile ↗

218 · o1-miniMedium970

Accuracy: 59.5%
Standard error: ±2.8 pp SE
Evidence: Model profile ↗

220 · GPT-5 NanoLow964

Accuracy: 57.58%
Standard error: ±3.52 pp SE
Evidence: Model profile ↗

221 · Gemini 1.5 Pro 002Not specified963

Accuracy: 57.23%
Standard error: ±2.8 pp SE
Evidence: Model profile ↗

222 · Gemini 2.0 Flash Thinking Exp 01 21Not specified962

Accuracy: 57.07%
Standard error: ±3.53 pp SE
Evidence: Model profile ↗

223 · qwen3 8bNot specified961

Accuracy: 56.76%
Standard error: ±2.93 pp SE
Evidence: Model profile ↗

224 · DeepSeek V3Not specified960

Accuracy: 56.53%
Standard error: ±2.76 pp SE
Evidence: Model profile ↗

225 · Qwen Max 2025 01 25Not specified959

Accuracy: 56.12%
Standard error: ±2.79 pp SE
Evidence: Model profile ↗

225 · Magistral Small 2506Not specified959

Accuracy: 56.06%
Standard error: ±3.54 pp SE
Evidence: Model profile ↗

225 · Phi 4Not specified959

Accuracy: 56.06%
Standard error: ±2.59 pp SE
Evidence: Model profile ↗

228 · DeepSeek R1 Distill Llama 70BNot specified957

Accuracy: 55.74%
Standard error: ±2.97 pp SE
Evidence: Model profile ↗

228 · qwen3 30b a3b Instruct 2507Not specified957

Accuracy: 55.62%
Standard error: ±2.88 pp SE
Evidence: Model profile ↗

228 · GPT-5.4 NanoNone957

Accuracy: 55.56%
Standard error: ±3.54 pp SE
Evidence: Model profile ↗

231 · Claude 3 5 Sonnet 20241022Not specified956

Accuracy: 55.3%
Standard error: ±2.82 pp SE
Evidence: Model profile ↗

232 · Qwen3 32BNone952

Accuracy: 54.1%
Standard error: ±2.81 pp SE
Evidence: Model profile ↗

232 · Claude 3 5 Sonnet 20240620Not specified952

Accuracy: 54.04%
Standard error: ±2.75 pp SE
Evidence: Model profile ↗

234 · Grok 2 1212Not specified951

Accuracy: 53.79%
Standard error: ±2.71 pp SE
Evidence: Model profile ↗

235 · qwen3 14bNone949

Accuracy: 53.41%
Standard error: ±2.75 pp SE
Evidence: Model profile ↗

235 · GPT-Oss 20bLow949

Accuracy: 53.16%
Standard error: ±2.5 pp SE
Evidence: Model profile ↗

237 · qwen3 4bNot specified946

Accuracy: 52.34%
Standard error: ±3.01 pp SE
Evidence: Model profile ↗

238 · Llama 4 ScoutNot specified944

Accuracy: 51.83%
Standard error: ±3.15 pp SE
Evidence: Model profile ↗

239 · Mistral Large 2411Not specified942

Accuracy: 51.33%
Standard error: ±2.69 pp SE
Evidence: Model profile ↗

240 · Llama 3.1 405B InstructNot specified941

Accuracy: 50.92%
Standard error: ±2.59 pp SE
Evidence: Model profile ↗

241 · Qwen3 30B A3BNone939

Accuracy: 50.38%
Standard error: ±2.83 pp SE
Evidence: Model profile ↗

241 · o1 Preview 2024 09 12Not specified939

Accuracy: 50.32%
Standard error: ±2.83 pp SE
Evidence: Model profile ↗

243 · GPT-4o 2024 08 06Not specified935

Accuracy: 49.21%
Standard error: ±2.6 pp SE
Evidence: Model profile ↗

243 · qwen2.5 72b InstructNot specified935

Accuracy: 49.15%
Standard error: ±2.69 pp SE
Evidence: Model profile ↗

243 · Mistral Small 3.2 2506Not specified935

Accuracy: 49.05%
Standard error: ±2.77 pp SE
Evidence: Model profile ↗

246 · Mistral Large 2407Not specified934

Accuracy: 49.02%
Standard error: ±2.59 pp SE
Evidence: Model profile ↗

246 · GPT-4.1 Nano 2025 04 14Not specified934

Accuracy: 48.93%
Standard error: ±2.48 pp SE
Evidence: Model profile ↗

246 · GPT-4o 2024 05 13Not specified934

Accuracy: 48.9%
Standard error: ±2.63 pp SE
Evidence: Model profile ↗

249 · GPT-5 NanoMinimal933

Accuracy: 48.48%
Standard error: ±3.56 pp SE
Evidence: Model profile ↗

250 · Qwen Plus 2025 01 25Not specified931

Accuracy: 48.11%
Standard error: ±2.71 pp SE
Evidence: Model profile ↗

250 · GPT-4o 2024 11 20Not specified931

Accuracy: 47.89%
Standard error: ±2.63 pp SE
Evidence: Model profile ↗

252 · Gemma 3 27b ItNot specified930

Accuracy: 47.73%
Standard error: ±2.7 pp SE
Evidence: Model profile ↗

252 · Magistral Small 2509Not specified930

Accuracy: 47.6%
Standard error: ±2.6 pp SE
Evidence: Model profile ↗

254 · qwen3 8bNone929

Accuracy: 47.47%
Standard error: ±2.62 pp SE
Evidence: Model profile ↗

254 · Mistral Small 2503Not specified929

Accuracy: 47.47%
Standard error: ±2.76 pp SE
Evidence: Model profile ↗

254 · Llama 3.3 70B InstructNot specified929

Accuracy: 47.44%
Standard error: ±2.85 pp SE
Evidence: Model profile ↗

254 · Gemini 1.5 Flash 002Not specified929

Accuracy: 47.32%
Standard error: ±2.77 pp SE
Evidence: Model profile ↗

254 · Mistral Small 3 2501Not specified929

Accuracy: 47.29%
Standard error: ±2.79 pp SE
Evidence: Model profile ↗

259 · Claude 3 Opus 20240229Not specified928

Accuracy: 47.16%
Standard error: ±2.64 pp SE
Evidence: Model profile ↗

260 · GPT-4 Turbo 2024 04 09Not specified926

Accuracy: 46.59%
Standard error: ±2.66 pp SE
Evidence: Model profile ↗

261 · Llama 3.1 Tulu 3 70B DpoNot specified925

Accuracy: 46.28%
Standard error: ±2.41 pp SE
Evidence: Model profile ↗

262 · qwen2.5 32b InstructNot specified924

Accuracy: 46.09%
Standard error: ±2.6 pp SE
Evidence: Model profile ↗

262 · GPT-Oss 20bHigh924

Accuracy: 45.96%
Standard error: ±3.55 pp SE
Evidence: Model profile ↗

262 · Gemini 1.5 Pro 001Not specified924

Accuracy: 45.86%
Standard error: ±2.62 pp SE
Evidence: Model profile ↗

262 · qwen3 4b Instruct 2507Not specified924

Accuracy: 45.83%
Standard error: ±2.85 pp SE
Evidence: Model profile ↗

266 · Mistral Small 2501Not specified922

Accuracy: 45.3%
Standard error: ±2.48 pp SE
Evidence: Model profile ↗

267 · GLM 4.7 FlashNone921

Accuracy: 45.14%
Standard error: ±2.18 pp SE
Evidence: Model profile ↗

268 · DeepSeek R1 Distill Qwen 14BNot specified920

Accuracy: 44.7%
Standard error: ±2.86 pp SE
Evidence: Model profile ↗

269 · Llama 3.1 70B InstructNot specified918

Accuracy: 44.19%
Standard error: ±2.45 pp SE
Evidence: Model profile ↗

270 · qwen3 4bNone916

Accuracy: 43.56%
Standard error: ±2.69 pp SE
Evidence: Model profile ↗

271 · Wizardlm 2 8x22BNot specified915

Accuracy: 43.43%
Standard error: ±2.32 pp SE
Evidence: Model profile ↗

272 · GPT-4 1106 PreviewNot specified912

Accuracy: 42.36%
Standard error: ±2.41 pp SE
Evidence: Model profile ↗

273 · GPT-4 0125 PreviewNot specified911

Accuracy: 42.27%
Standard error: ±2.68 pp SE
Evidence: Model profile ↗

274 · Mistral Small 3.1 2503Not specified910

Accuracy: 41.92%
Standard error: ±2.35 pp SE
Evidence: Model profile ↗

274 · Qwen Turbo 2024 11 01Not specified910

Accuracy: 41.79%
Standard error: ±2.62 pp SE
Evidence: Model profile ↗

276 · Llama 3.2 90B Vision InstructNot specified907

Accuracy: 41.04%
Standard error: ±2.29 pp SE
Evidence: Model profile ↗

277 · qwen2 72b InstructNot specified906

Accuracy: 40.78%
Standard error: ±2.57 pp SE
Evidence: Model profile ↗

277 · Claude 3 Sonnet 20240229Not specified906

Accuracy: 40.59%
Standard error: ±2.39 pp SE
Evidence: Model profile ↗

277 · Meta Llama 3 70B InstructNot specified906

Accuracy: 40.56%
Standard error: ±2.62 pp SE
Evidence: Model profile ↗

280 · Gemini 1.5 Flash 001Not specified905

Accuracy: 40.37%
Standard error: ±2.39 pp SE
Evidence: Model profile ↗

281 · Gemma 3 12b ItNot specified902

Accuracy: 39.46%
Standard error: ±2.53 pp SE
Evidence: Model profile ↗

282 · Mistral Large 2402Not specified899

Accuracy: 38.76%
Standard error: ±2.48 pp SE
Evidence: Model profile ↗

283 · Claude 3 5 Haiku 20241022Not specified897

Accuracy: 38.13%
Standard error: ±2.64 pp SE
Evidence: Model profile ↗

283 · qwen3 1.7bNot specified897

Accuracy: 38.01%
Standard error: ±2.75 pp SE
Evidence: Model profile ↗

285 · GPT-4o Mini 2024 07 18Not specified896

Accuracy: 37.72%
Standard error: ±2.41 pp SE
Evidence: Model profile ↗

286 · Hermes 2 Theta Llama 3 70BNot specified895

Accuracy: 37.47%
Standard error: ±2.48 pp SE
Evidence: Model profile ↗

287 · Gemma 2 27b ItNot specified892

Accuracy: 36.49%
Standard error: ±2.39 pp SE
Evidence: Model profile ↗

288 · Claude 3 Haiku 20240307Not specified891

Accuracy: 36.3%
Standard error: ±2.42 pp SE
Evidence: Model profile ↗

289 · GPT-4 0314Not specified889

Accuracy: 35.73%
Standard error: ±2.36 pp SE
Evidence: Model profile ↗

290 · qwen2.5 7b InstructNot specified888

Accuracy: 35.48%
Standard error: ±2.29 pp SE
Evidence: Model profile ↗

291 · Claude 2.0Not specified885

Accuracy: 34.66%
Standard error: ±2.51 pp SE
Evidence: Model profile ↗

292 · Open Mixtral 8x22bNot specified883

Accuracy: 34.06%
Standard error: ±2.07 pp SE
Evidence: Model profile ↗

292 · Gemini 1.0 Pro 001Not specified883

Accuracy: 33.96%
Standard error: ±2.01 pp SE
Evidence: Model profile ↗

292 · Eurus 2 7B PrimeNot specified883

Accuracy: 33.9%
Standard error: ±2.11 pp SE
Evidence: Model profile ↗

295 · DeepSeek R1 Distill Qwen 1.5BNot specified882

Accuracy: 33.59%
Standard error: ±2.06 pp SE
Evidence: Model profile ↗

296 · Claude 2.1Not specified880

Accuracy: 32.95%
Standard error: ±2.32 pp SE
Evidence: Model profile ↗

296 · Gemini 1.5 Flash 8b 001Not specified880

Accuracy: 32.95%
Standard error: ±2.42 pp SE
Evidence: Model profile ↗

298 · DBRX InstructNot specified879

Accuracy: 32.89%
Standard error: ±3.26 pp SE
Evidence: Model profile ↗

299 · Yi 1.5 34B ChatNot specified876

Accuracy: 31.98%
Standard error: ±2 pp SE
Evidence: Model profile ↗

300 · qwen1.5 32b ChatNot specified872

Accuracy: 30.74%
Standard error: ±2.02 pp SE
Evidence: Model profile ↗

300 · GPT-4 0613Not specified872

Accuracy: 30.65%
Standard error: ±2.34 pp SE
Evidence: Model profile ↗

300 · qwen3 1.7bNone872

Accuracy: 30.62%
Standard error: ±2.26 pp SE
Evidence: Model profile ↗

303 · Mixtral 8x7B Instruct v0.1Not specified871

Accuracy: 30.59%
Standard error: ±1.94 pp SE
Evidence: Model profile ↗

304 · Open Mistral Nemo 2407Not specified869

Accuracy: 29.89%
Standard error: ±2.1 pp SE
Evidence: Model profile ↗

304 · Open Mixtral 8x7bNot specified869

Accuracy: 29.83%
Standard error: ±1.91 pp SE
Evidence: Model profile ↗

306 · qwen1.5 72b ChatNot specified865

Accuracy: 28.82%
Standard error: ±1.99 pp SE
Evidence: Model profile ↗

307 · Granite 4.0 MicroNot specified864

Accuracy: 28.28%
Standard error: ±1.82 pp SE
Evidence: Model profile ↗

308 · GPT-3.5 Turbo 1106Not specified863

Accuracy: 28.03%
Standard error: ±1.87 pp SE
Evidence: Model profile ↗

309 · Phi 3 Medium 128k InstructNot specified861

Accuracy: 27.59%
Standard error: ±1.61 pp SE
Evidence: Model profile ↗

309 · Gemma 2 9b ItNot specified861

Accuracy: 27.46%
Standard error: ±2.3 pp SE
Evidence: Model profile ↗

311 · GPT-3.5 Turbo 0125Not specified860

Accuracy: 27.18%
Standard error: ±1.95 pp SE
Evidence: Model profile ↗

311 · Ministral 8b 2410Not specified860

Accuracy: 27.15%
Standard error: ±2.09 pp SE
Evidence: Model profile ↗

313 · Llama 3.1 8B InstructNot specified859

Accuracy: 26.96%
Standard error: ±1.86 pp SE
Evidence: Model profile ↗

314 · Llama 2 70b Chat HfNot specified857

Accuracy: 26.33%
Standard error: ±1.99 pp SE
Evidence: Model profile ↗

315 · Meta Llama 3 8B InstructNot specified856

Accuracy: 26.07%
Standard error: ±1.7 pp SE
Evidence: Model profile ↗

316 · Ministral 3b 2410Not specified853

Accuracy: 25.25%
Standard error: ±1.93 pp SE
Evidence: Model profile ↗

317 · DeepSeek Llm 67b ChatNot specified851

Accuracy: 24.62%
Standard error: ±1.36 pp SE
Evidence: Model profile ↗

318 · Granite 4.0 1bNot specified849

Accuracy: 23.99%
Standard error: ±1.62 pp SE
Evidence: Model profile ↗

318 · Llama 3.2 1B InstructNot specified849

Accuracy: 23.93%
Standard error: ±1.63 pp SE
Evidence: Model profile ↗

320 · Gemma 3 4b ItNot specified846

Accuracy: 23.23%
Standard error: ±1.87 pp SE
Evidence: Model profile ↗

321 · Gemma 3 1b ItNot specified835

Accuracy: 19.95%
Standard error: ±1.52 pp SE
Evidence: Model profile ↗

322 · Mistral 7B Instruct v0.3Not specified819

Accuracy: 15.18%
Standard error: ±1.01 pp SE
Evidence: Model profile ↗

323 · Yi 34B ChatNot specified817

Accuracy: 14.74%
Standard error: ±1.14 pp SE
Evidence: Model profile ↗

324 · Open Mistral 7bNot specified812

Accuracy: 13.23%
Standard error: ±0.99 pp SE
Evidence: Model profile ↗

325 · Granite 4.0 350mNot specified805

Accuracy: 11.24%
Standard error: ±0.91 pp SE
Evidence: Model profile ↗

326 · DeepSeek r1 0528 qwen3 8bNot specified799

Accuracy: 9.28%
Standard error: ±1.07 pp SE
Evidence: Model profile ↗

Source & conversion

Epoch AI internal evaluations on the 198-question GPQA Diamond subset.

Publisher-reported mean accuracy and standard error are preserved. Resource measurements were not included in the source export.

How the Lumina conversion works

1,000 + 75 × ((original score − 68.1830944842617) / 21.93641724240376). The result is rounded once. Scores may tie.

Frozen initial official Epoch internal-run CSV cohort of 326 model/configuration results. Every published row is retained; original run dates, exact model version, unspecified settings and explicit token budgets remain distinct. Scores use mean_score, never a selected best scorer. Historical runs have no per-row harness version in the export.

Calibration: gpqa-diamond-z75-v1 · experimental
Reference mean: 68.1830944842617
Population SD: 21.93641724240376
Coverage: Contributes through the Lumina Reasoning Index.

Identity, exclusions & attribution

Exact published model versions and explicit effort suffixes are retained. Unknown effort is not assigned to a known variant.

Dated previews, pre-release models and explicit token budgets remain separate.

Original standard error is not relabelled as a confidence interval.

Official benchmark · Epoch AI data and attribution · CC BY 4.0

Publisher terms. Scores and resource figures are attributed to the original publisher.