Original research brief
Microsoft ThinkingBox grades AI agents on the records they leave behind
Microsoft's ThinkingBox benchmark grades AI agents on terminal backend state across 507 workflows, run 20 times each. See what consistency costs and how…
What this piece is grounded in
01
Microsoft ThinkingBox grades AI agents on the records they leave behind, not the sentences they generate.
02
Across 507 stateful business workflows, each run 20 times against various LLM models, it grades agents on terminal backend state and side effects.
03
In a common-set ablation covering 121,680 valid trials across 12 LLM models, 79,853 attempts failed the executable checks.
04
Claude Opus 5.5 leads overall at 67.16%, two-thirds of a point above Claude Opus 5.
01 / FIELD NOTE
What changed: a new benchmark for agent reliability
Microsoft has released ThinkingBox, a benchmark that grades AI agents on the terminal backend state and side effects they leave behind, rather than on the sentences they generate. The Hugging Face blog post announcing it describes a customer service example where an agent made nine tool calls, closed a ticket as resolved, and replied that the query was resolved, while the database still showed the required end state as 'hold'. ThinkingBox runs agents against isolated MCP tool sessions and checks the records they leave. The benchmark covers 507 stateful business workflows, each run 20 times against various LLM models, and is now available through Hugging Face.
02 / FIELD NOTE
Why it matters for practical AI work
The source reports that in a common-set ablation covering 121,680 valid trials across 12 LLM models, 79,853 attempts failed the executable checks. Of those failures, 67.24% still terminated cleanly, invoked a state-changing tool, and reported no final tool error. Executable checks found wrong field values in 77.61% of them, unintended extra effects in 43.30%, and missing required effects in 25.36%. This means an agent can sound correct and make valid tool calls while leaving the wrong value or creating an extra side effect. The blog post states that a trajectory is a claim, and database state is the evidence. For teams building agents, this suggests that checking final responses is not enough; the state changes an agent makes need verification.
03 / FIELD NOTE
What you can verify
The source provides specific numbers you can check. Claude Opus 5.5 leads overall at 67.16% pass@1, two-thirds of a point above Claude Opus 5. Kimi-K3 is the strongest open-weights model, within a point of GPT-6-Astra. Domain matters: Claude Opus 4.6 scores 68.62% on retail but 8.30% on auto insurance. Only three models retain most of their single-attempt score over 20 repeats: GPT-6 Astra retains 78%, and Claude Opus 5.5 and Claude Opus 5 each retain 71%. At the other end, GLM-5.1, Kimi-K2.6 and DeepSeek-V4-Pro each keep about 8%. Kimi-K3 solves 93.89% of the benchmark at least once (476 of 507 tasks) but only 13.41% succeed in all 20 attempts. Claude Opus 5 completes 47.53% of the benchmark on every single attempt.
04 / FIELD NOTE
Editorial interpretation: what this means for agent evaluation
The source's core argument is that one success is not reliability. The blog post reports that every task runs 20 independent times from an identical clean backend, and the observed 20/20 count is used literally, with no estimator or smoothing. The practical implication for AI work is that single-attempt scores, which most leaderboards publish, can overstate what a model will do consistently. The gap between what a model can do once and what it does every time is the whole story, according to the source. For teams evaluating agents, this suggests running repeated trials and checking the final state of the systems the agent touches, not just the agent's final message. The supplied feed entry is a concise summary; consult the canonical article for full context.
Questions readers ask
What does ThinkingBox measure that other benchmarks do not?
ThinkingBox grades agents on the terminal backend state and side effects they leave behind, not on the sentences they generate. The source reports that in a common-set ablation, 67.24% of failed attempts still terminated cleanly and reported no final tool error, yet executable checks found wrong field values in 77.61% of them. This means tool-call validity alone does not confirm a correct outcome.
How many times does ThinkingBox run each task?
The source states that every task runs 20 independent times, each from an identical clean backend. The blog post uses the observed 20/20 as the literal count of how many of the 507 tasks passed 20 out of 20, with no estimator and no smoothing. This repetition is designed to test whether a model can do the work every time, not just once.