Google says Gemini 4 Carbon found software flaws at 88.2% on one security benchmark

Google reported on October 28 that its new Gemini 4 Carbon model scored 88.2% on SEC-bench Pro, a benchmark in which AI agents hunt for security bugs in major open-source software. Koray Kavukcuoglu, Google DeepMind’s senior vice president and Google’s chief AI architect, signed the announcement, which calls the result state-of-the-art. The number is Google’s own and was not accompanied by a published rival score on the same test.

AI, explained every day: The Deep View is the go-to daily newsletter for 600,000+ founders, knowledge workers and professionals who want to stay up to date with artificial intelligence.

Get The Deep View →

We may earn a commission if you sign up.

Two other results sit beside it, one for exploiting flaws and one for patching them, and each comes from a different benchmark.

What SEC-bench Pro measures

The benchmark’s own project site describes SEC-bench Pro as a self-evolving test for agents that look for security bugs in critical software, covering the V8 JavaScript engine, Firefox and the Linux kernel. It evaluates vulnerability discovery and the generation of proof-of-concept inputs. Each target comes packaged with a container image, a rendered prompt and a harness that runs the agent under a timeout.

The leaderboard defines its headline metric as success, the share of benchmark instances an agent solved, and gives an example of 201 solved out of 344, or 58.4%. The page does not spell out the exact success criterion for the Pro version. In the original SEC-bench, a proof-of-concept counted only if it triggered a valid sanitizer error, which gives a reasonable idea of the standard but should not be assumed to carry over unchanged.

What Google claims, and how it frames the comparison

In the Gemini 4 Carbon announcement, Google says the model “achieves a state-of-the-art 88.2% on SEC-bench Pro” for finding real vulnerabilities, and lists two companion results: 54.8% on ExploitGym, which Google describes as an exploitation benchmark, and 71.0% on Collinear’s CWE-bench, a patching benchmark. The post says the external benchmarks were evaluated without guardrails, meaning the safety restrictions that would apply in a public product were not in place for these runs.

The post names no competitor for any of the three external scores. Its direct comparisons are internal, with Carbon posting more than twice the success rate of Gemini 4 Argon at generating secure code, and a Chrome-team result in which Carbon produced high-quality patches more than ten times better than another lab’s model, left unnamed. The Hacker News, in a report by Ravie Lakshmanan, summarized the three scores as leads for finding, exploiting and patching.

The three scores also describe three different jobs. Finding a flaw is what a defender or an attacker does first, exploiting it is the step that turns a bug into a working attack, and patching is the repair. Google puts Carbon’s exploitation score at 54.8%, well below its finding score, and reports a separate patching score of 71.0%. Because the benchmarks differ, the three percentages cannot be compared with one another as if they measured the same thing.

Why 88.2% is a vendor figure, not a verdict

The project site’s public leaderboard did not list Gemini 4 Carbon when it was checked, and its top entry was a GPT-5.5 Codex configuration at 58.4% overall. That gap should not be read as Carbon beating a rival by thirty points. The leaderboard and Google’s figure may use different task subsets, timeouts or agent harnesses, and Google does not say which it used. Until a third party reruns the model, 88.2% is a number Google reports about its own model under conditions Google chose.

One measure of Carbon does come from outside the company. Artificial Analysis, an independent evaluator, lists Carbon at 59 on its Intelligence Index, dated October 28, against a median of 40 for comparable models. That index is a general-capability score and says nothing about SEC-bench Pro, so it cannot confirm the security result.

Google’s post also lists a price, which it says is introductory: $2 per million input tokens and $10 per million output tokens, rising to $4 and $20 after the introductory period. It reports a 0.0% attack success rate for Carbon on the Gray Swan indirect prompt injection benchmark, another result that rests on Google’s own testing. Neither figure bears on the 88.2%, but both come from the same self-reported evaluation package.

Who can actually use it

Access is narrow. Google says Carbon is initially going to a subset of cyber defenders and will widen to other trusted partners through the Fairwind Program, and its post says the model has not yet been released to developers, businesses or consumers. Fairwind, described in a September post by Four Flynn, Google’s vice president for security and privacy, is a limited-access program for governments and trusted partners that Google said had more than 650 participants.

The unresolved question is what happens to the numbers when guardrails return. Google reported its benchmark results without them, and defenders who get early access will work with a model that was measured in a configuration the public may never see. The Hacker News also reported that Google is taking part in the U.S. government’s voluntary pre-release testing process, which could produce an outside read on the model before any broad release.

This article was produced with the assistance of AI and reviewed by Morning Overview editors prior to publication.

AI, explained every day: Get The Deep View →

We may earn a commission if you sign up.