About Testing the Limits of Chain-of-thought with Multistep Soft Reasoning
Definition and scoring
- Organisation
- Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, Greg Durrett
- Category
- Reasoning
- Version
- 2023
- Direction
- higher is better
- Ranking use
- Reference
- Contamination risk
- Unknown
A dataset for evaluating language models on multistep soft reasoning tasks specified in natural language narratives. Tests the ability to perform complex, structured reasoning. Every genuine source row stays tied to its exact model label, configuration, benchmark version and evaluation system. The summary chart shows one best compatible score per canonical product; Score 2.0 admits only explicitly mapped, frozen protocols and keeps incompatible configurations separate.
Open benchmark source ↗