The dataset was developed with open source maintainers spending over 40 hours per task to establish qualitative and unit tests for mergeability. OpenAI's GPT-6.1 Sol (max) showed the strongest performance for its cost, scoring 49% at $0.81 per task, compared to $10.15 for the top Claude Opus 5.5. GPT-6 models also exhibited higher token efficiency, with the Astra and Sol versions averaging 27k and 33k output tokens per task, respectively, while Claude Sonnet 5.5 averaged 236k tokens.
Tuesday, October 20, 2026
Story
Claude Opus 5.5 Leads First AI Code Merge Test with 52% Score
AI8 days ago
Artificial Analysis has published its results for FrontierCode 1.1, a private evaluation created by Cognition to see if AI-generated code would actually be accepted into real-world software repositories. Nine models from six labs scored between 43% and 52%, with Anthropic's Claude Opus 5.5 (max with fallback) achieving the highest mark. Despite this lead, models across the board failed more than 40% of the tasks, and nine of the 100 items in the primary set were not passed by any model. MiMo-V2.6-Pro was the top open weights model with a 45% success rate.
Looking for more context or analysis?
Tilt chat provides reliable, tailored insights, alerts, and automations.