What Changed
A recent evaluation revealed that enabling retained reasoning and compaction settings in the Responses API tripled scores for GPT‑5.6 Sol on the public task set of a specific benchmark while cutting output tokens by six times.
Why It Matters for Practical AI Work
The official harness discarded private reasoning after each action and used rolling truncation that made older actions invisible as history grew, causing models to struggle with learning over time. Retaining reasoning allowed the model to remember past thoughts rather than reinterpreting the game from scratch every turn.
What A Reader Can Verify
Readers can verify that GPT‑5.6 Sol scored 7.8% with standard settings versus 38.3% when both retained reasoning and compaction were enabled on the public task set, a metric comparing performance to a human baseline of approximately 48%. The source material confirms these specific percentages for the six-level game series.
What Remains Unknown
It remains unknown whether other commercial developers optimize their harnesses differently or if similar improvements apply to models beyond GPT‑5.6 Sol without further testing on this specific benchmark configuration and task set.
Source-grounded facts
- GPT‑5.6 Sol scored just 7.8% with standard settings.
- Enabling retained reasoning and compaction tripled scores.
What remains unknown
- The exact impact of these harness changes on models other than GPT-5.6 Sol is not specified in the source text.
Frequently asked questions
What is the purpose of the ARC-AGI-3 benchmark?
The benchmark measures how well agents learn and reason in unfamiliar 2D games without explicit instructions or special tools.
How are scores calculated for this evaluation task set?
Scores measure Relative Human Action Efficiency, comparing model performance to an estimated average human tester score of 48% based on official logs.