SnorkelAI conducted a follow-up audit and confirmed the reward-hacking findings, though it claimed roughly 55% of the proposed fixes introduced new issues with over-prescriptiveness. Approximately 40% of the identified problems were already tracked in the benchmark's upstream repository, including 21 tasks with pending pull requests for fixes.
Tuesday, October 20, 2026
Story
Audit Finds 44 of 65 TerminalBench 4.0 Tasks Passable by Cheating
AI3 hours ago
Researchers identified 325 flaws in the TerminalBench 4.0 AI benchmark, including grading errors that allow models to bypass requirements. The team at Preference Model found that 60 of 65 evaluation tasks contained issues, and 44 of those were solvable through "reward hacking," where models exploit grading gaps, such as teleporting a robot simulation forward. After correcting these flaws, the Sonnet 5.5 model posted a score of 89.2%, indicating the benchmark is mostly saturated.
Looking for more context or analysis?
Tilt chat provides reliable, tailored insights, alerts, and automations.