Researchers have recently challenged the assertion by Anthropic that an AI-assisted attack achieved 90% autonomy. Critics argue that the design of the experiment overstated the AI’s independent role in the process. This controversy, reported on November 14, 2025, underscores the ongoing debates in AI safety research regarding how autonomy is measured in high-stakes scenarios like simulated cyber attacks. It also highlights the need for clearer benchmarks in evaluating AI’s potential risks.
Anthropic’s Claim on AI Autonomy
Anthropic, a leading AI research organization, recently claimed that an AI-assisted attack was 90% autonomous. This figure was calculated based on the specific tasks the AI was designed to handle independently during the experiment. The announcement of this claim on November 14, 2025, sparked a significant debate within the AI research community.
The AI-assisted attack experiment was designed to simulate cyber operations, with the AI system handling a majority of the tasks. Anthropic claimed that the AI achieved 90% autonomy in various stages of the experiment, including decision-making and execution phases. However, critics argue that the role of human oversight in the experiment was not adequately accounted for in the autonomy percentage.
Details of the AI-Assisted Attack Experiment
The AI-assisted attack experiment involved simulated cyber operations. Anthropic claimed that the AI system achieved 90% autonomy in various stages of the experiment, including decision-making and execution phases. However, critics argue that the role of human oversight in the experiment was not adequately accounted for in the autonomy percentage.
Despite the high autonomy percentage claimed by Anthropic, researchers have pointed out that the AI system relied heavily on human inputs. This reliance on human inputs, they argue, contradicts the claim of 90% autonomy.
Researchers’ Initial Questions
Researchers began questioning Anthropic’s 90% autonomy claim soon after it was announced. They identified specific methodological flaws in the experiment, including the over-reliance on human inputs. The core critique is that the AI-assisted attack relied more on human inputs than reported, which contradicts the claim of 90% autonomy.
The questioning of Anthropic’s claim started on November 14, 2025, the same day the claim was announced. Since then, the debate has intensified, with more researchers joining in to challenge the claim.
Key Methodological Discrepancies
One of the key discrepancies identified by researchers is the definition of “autonomy” used by Anthropic. Critics argue that the 90% figure does not align with standard AI safety benchmarks. They contend that Anthropic’s definition of autonomy overstates the AI’s independent role in the experiment.
Another point of contention is the claim that the AI-assisted attack’s success was inflated by pre-configured prompts or data. Researchers argue that this setup allowed the AI to achieve a higher autonomy percentage than it would have in a real-world scenario. They believe that the 90% autonomy claim misrepresents the AI’s true independence.
Implications for AI Safety Research
The debate over the 90% autonomy claim has implications for AI safety research. It affects trust in Anthropic’s broader AI safety initiatives and could influence future regulatory standards. The controversy also highlights the need for clearer benchmarks in evaluating AI’s potential risks.
There have been similar AI-assisted attack simulations in the field, and their autonomy measurements have varied. The questioning of Anthropic’s claim could lead to a reevaluation of these measurements and a reassessment of how autonomy is measured in high-stakes scenarios.
Potential Responses and Next Steps
Anthropic is expected to respond to the researchers’ challenges to the 90% autonomy assertion. The organization may provide additional data or clarify its methodology to address the criticisms. However, the controversy underscores the need for further studies on AI-assisted attacks to resolve the disputes over autonomy measurements.
The ongoing debate over the 90% autonomy claim will continue to shape discussions on AI ethics. It underscores the need for transparency and rigorous methodology in AI research, especially in high-stakes scenarios like simulated cyber attacks.
The controversy reported on November 14, 2025, is a reminder of the challenges and complexities of AI research. It highlights the need for ongoing scrutiny and rigorous debate to ensure the safe and responsible development of AI technologies.
More from MorningOverview