🔍 Read the full analysis: FrontierCode: What Goes Into Reviewing A Pull Request From AI? on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026, and tested 23 AI models on repository changes judged for more than test success. Claude Opus 5.5 led with a 52.2% benchmark score and cleared all blocking checks in 57.7% of attempts. The results suggest AI-generated pull requests still need human review; costs and some model results vary widely or remain unavailable.
Artificial Analysis tested 23 AI models on FrontierCode, a coding benchmark that evaluates whether a repository change would pass a maintainer-style review rather than merely pass tests. Claude Opus 5.5 scored highest at 52.2%, but cleared every blocking check in only 57.7% of attempts, according to the evaluation published October 12. The results offer a practical measure of AI coding reliability—and show why human review remains relevant.
FrontierCode scores changes across checks for correctness, regressions, test quality, scope and review quality. A failure on any blocking check gives an attempt a score of zero, even if ordinary tests pass. Attempts that pass all blocking checks receive scores from 70% to 100%, depending on additional quality checks. The benchmark also penalizes oversized diffs, edits outside relevant files and tests that do not distinguish the changed code from the original.
In Artificial Analysis’s results, GPT-6.1 Sol ranked second at 49.1%, followed closely by Claude Fable 5.1 at 49.0%. Claude Sonnet 5.5 scored 45.8%, while Xiaomi’s MiMo-V2.6-Pro reached 45.4%. The published table puts nine models from six labs between 43% and 52%; below them, scores drop, including 33.8% for Grok 4.7 and 10.5% for Nemotron 3 Ultra.
The evaluation used the same open-source agent harness, Stirrup, for each model, with up to 250 turns per attempt and three attempts per task. Artificial Analysis assessed the benchmark’s 100-task Main set. Nine tasks were not solved by any tested model on any attempt, the source reports. A GPT-6 Luna model served as the automated code-review judge; the results do not establish how the models would perform in every team or repository.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Tests Is Not Enough
Many coding evaluations focus on whether a program passes tests. FrontierCode tests a broader question: whether a proposed change is correct, appropriately scoped and suitable for review. That distinction matters to engineering teams because a patch can pass visible tests yet introduce regressions, include unnecessary edits or provide weak tests of its own.
The leading model’s 57.7% blocking-check pass rate means it failed at least one required check in the remaining attempts, under this benchmark’s scoring rules. That is not a direct estimate of failure rates in production, but it does caution against treating benchmark performance as evidence that generated pull requests can be merged without review. Results also point to a cost trade-off: Artificial Analysis lists Opus at $10.15 per task and GPT-6.1 Sol at $0.81, with a 3.1-point score gap between them. These are reported benchmark costs, not guarantees of real-world operating costs or outcomes.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Judges Changes
FrontierCode was built by Cognition with open-source maintainers and announced in June 2026; version 1.1 followed in July with rules against searching online for task solutions. More than 20 maintainers contributed tasks from 36 well-known repositories, including Celery, Budibase, uppy and Mattermost. The source says maintainers spent more than 40 hours on each task. The full benchmark contains 150 tasks, while Artificial Analysis evaluated the 100-task Main set.
Tasks are private, according to the source, to reduce the chance that models encountered their solutions during training. Checks include hidden tests, command-based regression and cleanliness checks, reverse tests to assess whether an agent’s tests fail on the original code, adaptive tests for complex behavior, limits on files and diff size, and an LLM review against maintainers’ criteria. Cognition also applies a trajectory scanner; attempts flagged for looking up solutions score zero. These rules are designed to measure more than whether a final patch can satisfy a test suite.
The rankings also differ from a general intelligence leaderboard. In the comparison provided by the source, Claude Sonnet 5.5 drops from second on the Intelligence Index to fifth on FrontierCode, while DeepSeek V4.1 Flash ranks higher on FrontierCode than its intelligence score alone might suggest. That comparison is specific to the models and rankings reported; it does not establish a universal relationship between general capability and coding performance.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, in Cognition’s announcement
As an affiliate, we earn on qualifying purchases.
Gaps in the Current Results
The published scores cover a defined benchmark setup, not every software team’s workflow. It remains unclear how closely the results predict performance on private production code, with different review standards, tools or human oversight. The automated review step also uses GPT-6 Luna (medium) as the judge, so the reported quality scores depend partly on that evaluator and the benchmark’s criteria.
Several models had no FrontierCode result as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview, according to the source. Their absence means the current rankings are incomplete across the wider model field. The source also provides benchmark cost figures, but those figures alone do not show what deployment would cost for a particular organization or workload.
As an affiliate, we earn on qualifying purchases.
Models and Results to Watch
The next useful comparison will come as missing models are evaluated and the benchmark’s results are updated. In particular, the source identifies Muse Wonder Full as a model whose FrontierCode result will be worth watching, given its performance on other coding evaluations. Any new score should be read alongside its pass rate, cost and evaluation setup rather than as a ranking alone.
For teams considering AI coding agents, the immediate takeaway is to test proposed tools against their own repositories and review standards. FrontierCode supplies evidence about performance on a fixed set of maintainer-authored tasks; it does not settle whether a model can safely handle a team’s pull requests. Further results may clarify how models compare, but the current figures support keeping human review in the process.
software development code analysis
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What does FrontierCode measure?
It evaluates whether AI-generated repository changes satisfy checks for correctness, regressions, test quality, scope and maintainers’ review criteria—not just whether tests pass.
Which model ranked highest?
Claude Opus 5.5 (max with fallback) led Artificial Analysis’s results with a 52.2% FrontierCode score. It cleared every blocking check in 57.7% of attempts.
Does the result mean AI pull requests can be merged automatically?
No. The benchmark does not establish that generated changes are safe to merge without review. The leading model failed at least one blocking check in a substantial share of attempts under this evaluation.
How much did the tested models cost per task?
Artificial Analysis reported costs ranging from $0.09 per task for GPT-6 Luna (max) to $10.15 for Claude Opus 5.5 (max with fallback), among the figures provided. These are benchmark figures and may not match costs in other workloads.
Were all major models included?
No. As of October 13, 2026, the source said results were unavailable for Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
