🔍 Read the full analysis: FrontierCode: What Maintainers Think When AI Contributes Code on ThorstenMeyerAI.com
Get hardware and tech essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026, testing 23 models on repository tasks judged against maintainer-style criteria. Claude Opus 5.5 led with a 52.2% score and cleared every blocking check in 57.7% of attempts; results show that AI-generated code still requires human review.
Artificial Analysis added FrontierCode to its model evaluations on October 12, testing 23 AI models on code changes assessed against maintainers’ criteria rather than test results alone. Claude Opus 5.5 ranked first with a 52.2% score, but it passed every blocking check in only 57.7% of attempts, underscoring that the benchmark does not support treating AI-generated pull requests as ready to merge without review.
FrontierCode evaluates whether a proposed code change would meet a repository maintainer’s acceptance bar. The test set used in the Artificial Analysis run contains 100 tasks drawn from a wider 150-task benchmark. The source report says tasks were written by more than 20 maintainers for projects including Celery, Budibase, uppy and Mattermost; task prompts are private to reduce the chance models have encountered the solutions during training.
The leading models were closely grouped: GPT-6.1 Sol scored 49.1%, while Claude Fable 5.1 scored 49.0%. Artificial Analysis reported that nine models from six labs scored between 43% and 52%. Scores fell sharply farther down the listed results: Grok 4.7 scored 33.8%, MiniMax-M3 23.0% and Nemotron 3 Ultra 10.5%.
Every model was run through the same Stirrup agent harness, with up to 250 turns per attempt and three attempts per task, according to the report. A separate GPT-6 Luna model handled the code-review step. The reported price per task ranged from $0.09 for GPT-6 Luna to $12 for Claude Fable 5.1. Claude Opus 5.5 cost $10.15 per task in this evaluation, while GPT-6.1 Sol cost $0.81.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Tests Is Not Enough
Many coding benchmarks focus on whether code passes tests. FrontierCode also checks issues that affect whether a patch is practical to review and maintain. Its reported criteria include hidden tests, command checks, tests designed to catch changes that would not fail on the original code, constraints on the files and size of a patch, and an LLM review against maintainers’ stated requirements.
Failing any blocking check gives an attempt a score of zero, even if other checks pass. That design makes a high score a measure of more than basic functional correctness, but the results still show substantial failure rates: the top model did not clear all blocking checks in 42.3% of attempts. For engineering teams, the findings are a reminder that benchmark ranking alone cannot establish whether a system is safe to merge into a live project.
The cost figures also put a price on incremental performance in this particular setup. The source report calculates that moving from GPT-6.1 Sol to Claude Opus 5.5 buys 3.1 score points at about 12.5 times the per-task cost. Those figures describe this benchmark run, not a universal cost or productivity advantage; results can vary with tasks, configuration and review needs.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Scores Patches
FrontierCode was developed by Cognition and open-source maintainers, announced in June 2026, and updated to version 1.1 in July, according to the source report. The update added rules against looking up task solutions online. Maintainers reportedly spent more than 40 hours on each task, which is intended to make the benchmark reflect work in established repositories rather than short, isolated programming exercises.
Scores are not simply the share of tasks solved. The source says attempts that fail a blocking check score zero; those that pass all blocking checks receive scores from 70% to 100%, with quality checks affecting the final result. Attempts flagged by Cognition’s trajectory scanner for looking up a solution also receive zero. The benchmark additionally penalizes oversized diffs, edits beyond relevant files and unhelpful tests.
The source report identifies Xiaomi’s MiMo-V2.6-Pro as the leading open-weights model, scoring 45.4%, with DeepSeek V4.1 Flash at 44.0% and Kimi K3 at 43.6%. It also compares FrontierCode standings with a general intelligence index, where some models rank differently. That comparison suggests general-purpose scores do not necessarily predict performance on this repository-focused evaluation; it does not establish how the models will perform on every coding task.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, in Cognition’s announcement, as quoted by ThorstenMeyerAI.com
As an affiliate, we earn on qualifying purchases.
What the Results Cannot Show
The published results are limited to the models and configuration tested. The source says Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview did not have FrontierCode results as of October 13, 2026. Their absence means this evaluation cannot establish how they would rank against the listed models.
The reported scores also do not show how often developers would accept patches in their own projects, or how performance changes under different prompts, tools and review processes. The source gives aggregate results but does not provide enough detail here to independently assess variation across individual tasks or attempts. Nine of the benchmark’s 100 tasks were reportedly unsolved by any model on any attempt; the report does not identify those tasks in the supplied material.
software development AI assistants
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
More Model Results to Watch
The immediate next development is whether additional labs and models are added to the FrontierCode results. The source report specifically points to pending evaluations for Gemini 4 Argon and several Muse models, though it gives no release dates. Their eventual results could change the leaderboard, but no outcome can be inferred from performance on other coding benchmarks.
For now, the evaluation offers a comparison under one shared harness and set of rules, not a guarantee about production performance. Readers and engineering teams can watch for new benchmark entries and fuller task-level reporting. The present results support a narrower conclusion: even the leading tested model failed at least one blocking check in a substantial share of attempts, so human review remains part of the reported picture.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is FrontierCode?
FrontierCode is a coding benchmark developed by Cognition with open-source maintainers. It tests code changes against repository-specific checks and review criteria, rather than relying only on whether tests pass.
Which model ranked first in the reported evaluation?
Claude Opus 5.5, in the max-with-fallback configuration, ranked first with a 52.2% FrontierCode score. It cleared all blocking checks in 57.7% of attempts, according to Artificial Analysis’s results as described in the source report.
Does the score mean an AI patch is safe to merge?
No. A benchmark score summarizes performance on a defined set of tasks and checks. In this evaluation, even the top model failed to clear every blocking check in 42.3% of attempts, and the results do not replace project-specific review.
How much did model performance cost per task?
The source report lists costs from $0.09 per task for GPT-6 Luna to $12 for Claude Fable 5.1. Claude Opus 5.5 was listed at $10.15 and GPT-6.1 Sol at $0.81. These are reported benchmark costs, not guaranteed prices for other uses.
Which models were missing from the results?
As of October 13, the source report said there were no FrontierCode results for Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. It did not give dates for their possible inclusion.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
