The audit revealed significant disparities in grader accuracy, with pass rates for some tasks rising from 18.8% to 43.8% when overly strict verifiers were corrected. Conversely, pass rates fell from 60.2% to 49.7% where verifiers were too lenient. In one case, a submission failed to use required GDPR and HIPAA templates but still received a perfect score of 1.0 until the update reduced the score to 0.09. These findings affect the credibility of frontier labs that cite AutomationBench on their model cards.
Tuesday, October 20, 2026
Story
Parsewave Fixes 206 Zapier Benchmark Bugs to Revise 28% of Kimi Grades
AI14 days ago
Parsewave reviewed all 600 public tasks in Zapier's AutomationBench to test whether the verifiers could be fooled by incorrect answers. The auditor used agents to generate plausible but wrong responses, resulting in the discovery of 206 bugs that the firm subsequently fixed in a release called AutomationBench Verified. When Parsewave replayed 1,235 runs of the Kimi K3 model using the updated graders, 344 of those entries, or 27.9%, received a different score.
Looking for more context or analysis?
Tilt chat provides reliable, tailored insights, alerts, and automations.