🔍 Read the full analysis: What Gemini 4 Carbon’s Security Scores Reveal About AI That Can Break In And Repair A Lock on ThorstenMeyerAI.com
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
TL;DR
Google says Gemini 4 Carbon scored strongly on tests of finding, exploiting and fixing software vulnerabilities, and is giving access first to vetted cyber defenders. Two of the three headline results were produced by Google, and comparisons with rivals use different testing conditions; no independent public evaluation of Carbon’s security performance is available.
Google says its Gemini 4 Carbon model performed strongly on three cybersecurity tests covering vulnerability discovery, exploit development and software repair, while limiting initial access to a small group of vetted cyber defenders. The results point to a model that could assist both security teams and attackers, but two of the three headline scores were calculated by Google, and the available rival comparisons are not fully like-for-like.
Google reported a score of 88.2% on SEC-bench Pro, which tests whether a model can produce an input that triggers a known software bug when given information about where to search. The score uses the benchmark’s May 2026 version of 183 tasks, averaged over three attempts, and Google’s Antigravity agent tool. The test is based on already-fixed flaws in browser engines and, in its latest version, the Linux kernel. It does not measure whether a model can independently discover unknown vulnerabilities from scratch.
On ExploitGym, Google reported 54.8% for Carbon using a 10-hour time limit. The benchmark asks a model to turn a program crash into an exploit that runs code and retrieves a hidden value. Google also reported a 53.5% result using a six-hour limit. The source material cites Claude Opus 5.5 at 34.5% within six hours, based on Anthropic’s reporting. These figures come from different labs and should not be treated as a controlled head-to-head comparison: reported methods, tools, graders and time limits can vary.
Carbon scored 71% on CWE-bench, a test of finding and fixing flaws while keeping existing software tests working. Collinear AI, the benchmark’s creator, runs the test on a private set of 120 tasks. Its reported leaderboard results put Carbon three percentage points ahead of GPT-6 Astra, Gemini 4 Argon and Grok 4.7, each at 68%, and four points ahead of Claude Opus 5.5 at 67%. The supplied account says Artificial Analysis evaluated and published these results. Across the tested models, five tasks defeated every model, and even the leading result left close to three in ten tasks unresolved on the first try.
Infographic 01 · What Carbon's three scores measure
Find, exploit, fix: three tests, and only one outside referee.
Google says Gemini 4 Carbon leads all three security benchmarks it reported. Here is what each one asks of the model, and who produced Carbon's number.
1Find · SEC-bench Pro
- The model gets
- Source code, the area to search, the bug type and the expected error. Not the input that triggers the bug.
- It counts when
- The model writes an input that triggers a known, already-fixed bug in a browser JavaScript engine.
- Carbon’s score came from
- Google, 183-task May 2026 version, average of 3 attempts Google's own run
2Exploit · ExploitGym
- The model gets
- An input that crashes the program. Standard protections switched off.
- It counts when
- The model turns the crash into its own code running on the target and reads a hidden flag; an AI judge checks it used the intended bug.
- Carbon’s score came from
- Google, one attempt per task, 10-hour limit (53.5% at 6 hours) Google's own run
3Fix · CWE-bench
- The model gets
- A real open-source code base and one instruction: audit it and fix what you find. No hint what or where.
- It counts when
- The attack stops working and every existing test still passes. Models never write exploits.
- Carbon’s score came from
- Collinear AI, 120 private tasks, all models run by Collinear Outside run
Read the fine print
- Google tested without safety filters (“evaluated without guardrails”); Anthropic and OpenAI report their cyber results with safeguards off too
- Rival numbers come from the rivals’ own system cards, with their own time limits and graders
- SEC-bench Pro and ExploitGym use public bugs with public fixes; CWE-bench keeps its test set private
Finding is not discovering
- SEC-bench Pro tells the model where to look and what kind of bug to expect
- That is closer to a detective with a tip than to a hunt for unknown flaws
- Real attacks also need targets, delivery and staying undetected; real fixes need review
The same skill that proves a bug is dangerous is the skill an attacker needs.Thorsten Meyer · my reading
Sources: Google, 28 Oct 2026; Google methodology note, Oct 2026; CWE-bench leaderboard, 28 Oct 2026; SEC-bench Pro paper; ExploitGym paper; OpenAI GPT-6 Astra system card, 3 Sep 2026. Lead margin and counts as in the article; margin is my arithmetic.
Powered by Thorsten Meyer AI
Why Access Starts With Defenders
The tests cover capabilities that can serve both security teams and attackers. Finding a weakness can help developers prioritize repairs; turning a crash into a working exploit can show how serious a flaw is. But the same ability could help someone break into vulnerable systems. Google’s restricted rollout reflects that dual use: the company says Carbon is initially being provided to a small, vetted group of cyber defenders, rather than made broadly available.
For organizations, the scores are evidence of performance on particular test setups, not a guarantee that Carbon will secure a live network or produce safe repairs in every case. The CWE-bench results show why human review remains necessary: the top reported score was 71%, and passing a benchmark task does not by itself establish that a fix is suitable for a company’s software. Teams considering AI-assisted remediation still need to test changes, review them and monitor systems.
For the public, the immediate effect appears limited. The source says Carbon is not available for general use, and it reports no change to consumer software or automatic updates. The more consequential question is how access and safeguards will work if models with these capabilities become more widely available. The scores alone do not answer that question.
Popular right now
Roborock Qrevo CurvX Robot Vacuum and Mop
Dyson Ball Animal 3 Corded Upright Vacuum
Ninja CREAMi 7-in-1 Ice Cream Maker, NC301
As an affiliate, we earn on qualifying purchases.
What Each Benchmark Tests
The three tests represent different stages of security work, and their percentages are not interchangeable. SEC-bench Pro gives the model source code and clues about a known bug, then checks whether it can trigger that bug. Its score reflects performance on a defined set of tasks with a particular agent tool and repeated attempts, not general vulnerability discovery.
ExploitGym starts with a crash and tests whether the model can turn it into an exploit that retrieves a hidden “flag.” The supplied description says the benchmark was developed by researchers led by the University of California, Berkeley, with contributors from several institutions and AI labs. Google’s reported run disabled standard protections such as memory randomization, a condition that affects how its results should be interpreted against real-world systems.
CWE-bench tests repair rather than exploit writing. The model receives an open-source codebase and is asked to audit and fix problems without being told where they are. A repair counts only if the attack no longer works and existing tests still pass. Unlike the other two headline scores, CWE-bench is run by its creator on a private test set, which the supplied account says is intended to reduce the risk that tasks appear in training data.
Limits of the Published Scores
No independent public test of Carbon on its security capabilities is described in the supplied material. Google produced the SEC-bench Pro and ExploitGym figures; CWE-bench is operated by Collinear AI, with the account also identifying Artificial Analysis as the evaluator and publisher. That makes the sources and methods different across the three scores.
The rival comparisons also have limits. The ExploitGym results cited use different time limits and are reported by the labs that tested their own models. For SEC-bench Pro, the source gives OpenAI’s GPT-6 Astra result of 85.4% and Gemini 4 Argon’s 83.4% on the same benchmark version, but no Claude result. The benchmark leaderboard says newer Claude results are unavailable because of safeguard restrictions. The supplied material does not provide a single independent, standardized run comparing every model with identical tools, settings and scoring.
It is also unclear how Carbon performed against protections enabled, on systems outside the benchmark tasks, or in deployment by the vetted defenders. The source does not specify how many organizations will receive access, when a wider release might occur, or what outcomes will come from the government’s voluntary pre-release testing.
What Comes After Testing
The next reported stage is restricted access for vetted defenders, alongside the U.S. government’s voluntary pre-release testing. Google has not, in the supplied material, announced a public release date or described the full access conditions. Results from these trials, if published, could offer information about how Carbon performs outside benchmark settings, though no such results are available yet.
For now, readers and security teams should treat the percentages as benchmark-specific findings, with their stated conditions and attribution attached. Companies evaluating AI-assisted security work will need to verify proposed fixes and test them in their own environments. For ordinary software users, the source reports no immediate change: keeping software updated remains the practical step it identifies while Carbon’s access remains limited.
Key Questions
What did Gemini 4 Carbon score on the three security tests?
Google reported 88.2% on SEC-bench Pro, 54.8% on ExploitGym with a 10-hour limit, and 71% on CWE-bench. The figures come from different test arrangements; Google produced the first two, while Collinear AI runs CWE-bench.
Can the scores be compared directly with rival models?
Not fully. The ExploitGym comparisons cited use different time limits and results reported by different labs. Some SEC-bench Pro results are available on the same benchmark version, but the supplied material says there is no Claude result for the current comparison because of safeguard restrictions.
Does SEC-bench Pro show that Carbon can find unknown flaws?
No. The benchmark gives a model source code and information about the area and type of a known, previously fixed bug. The model must trigger it; the test does not establish that it can discover unknown vulnerabilities from scratch.
Who can use Gemini 4 Carbon now?
According to the source material, Google is providing it to a small group of vetted cyber defenders. It also says Carbon is in the U.S. government’s voluntary pre-release testing. No public release date is provided.
Do the results mean Carbon can reliably repair software?
No. CWE-bench measures whether a repair stops a tested attack while keeping existing tests working, and Carbon’s reported score was 71%. That result leaves tasks unresolved and does not prove that every AI-written fix will be safe or suitable in a real deployment.
Source: ThorstenMeyerAI.com
