As AI models’ capabilities keep increasing, the debate over the risks posed by AI is becoming divisive. To some, AI poses ‘catastrophic risks’, with AI systems being used to create weapons or conduct cyberattacks, as well as pursuing their own goals and ‘building themselves’ through recursive self-improvement.
Fears over losing control have only worsened in the age of agentic AI, especially with OpenAI and Anthropic recently disclosing that their models gained unauthorised access to other companies’ systems during testing, and with investigations revealing large-scale cooperation between agents in at least one attack. Catastrophic risk is commonly conflated with existential risk, which supposes that ‘rogue’ or ‘out of control’ AI systems pose a severe threat to human life.
Meanwhile, others argue that this catastrophic framing is no more than a Big Tech pitch to retain power and shift attention away from immediate harms, like disinformation, exploitation and manipulation as well as concerns over algorithmic discrimination and the potential impact on fundamental rights. Currently, there’s ample evidence that these risks could materialise, with the Dutch Toeslagenaffaire and ongoing lawsuits suggesting that AI companions have incentivised self-harm.
These contrasting visions of risk are already shaping how we govern AI on both sides of the Atlantic. The MIT AI risk initiative suggests that catastrophic risks shouldn’t be disentangled from societal risks, as unemployment, disinformation and environmental harm can still lead to catastrophic outcomes. Rather than taking a leap of faith towards either side, the EU must develop a strategic approach for tackling AI risks based on evidence, probability and impact.
This isn’t easy, given that AI’s capabilities are still emerging. But this uncertainty means it’s crucial for the EU to think, plan and act ahead with proper forward-looking governance.
Regulating AI based on risks
Risk is the basis that many countries have chosen for regulating AI. The EU’s AI Act prioritises regulating both AI systems deployed in high-risk domains and general-purpose AI models with high-impact capabilities, with both triggering a systematic risk assessment.
In the US, the states of California, New York and Illinois have adopted a cross-sectoral AI regulation targeting frontier models, defined as those trained using significant compute and thus deemed to pose increased risks. The US federal government has so far been flip-flopping in its approach, pushing back against individual state laws while establishing a voluntary framework for granting the federal government access to frontier AI models before they’re publicly released.
While risk is the common anchor for all these regulatory approaches, in reality, they represent two very different understandings of AI risks. In the EU, policymakers address ‘systemic risk’, whereas in the US, concerns revolve around ‘catastrophic risk’.
Systemic v catastrophic
The EU’s AI Act adopts a qualitative approach for defining the risks posed by advanced AI models. This ‘systemic risk’ is broadly defined as specific to general-purpose AI models which have a high-impact, market-wide reach and the potential to cause to propagate negative effects at scale across public health, safety, security and fundamental rights.
In contrast, the US states mentioned above have adopted a quantitative approach. This sets numerical thresholds for defining ‘catastrophic risk’, namely the risk that the development, storage, use or deployment of a frontier model will contribute to over 50 deaths or serious injuries, or over USD 1 billion in property damage from models evading human control, AI-assisted weapons or autonomous criminal conduct (e.g. cyberattacks, murder or extortion).
As summed up by philosopher Atoosa Kasirzadeh, catastrophic risk concerns magnitude while systemic risk concerns structure. Yet this isn’t simply a conceptual exercise. Understanding risk as either systemic or catastrophic directly impacts AI governance as this is the basis for key regulatory requirements.
As per US states’ regulatory requirements, developers must put a Frontier AI Framework in place for managing risks, but the risks to be identified, assessed and mitigated are limited to a narrow list based on the number of deaths/injuries and damage costs. Meanwhile, the EU’s AI Act’s open-endedness encourages providers to implement a qualitative Safety and Security Framework, leaving space for interpreting, assessing and determining acceptable risks.
The EU’s centralised enforcement mechanism for general-purpose AI models can, in theory, allow for the AI Act to be more systematically and coherently applied across Member States. But the open-endedness of the regulatory text, the focus on ex-ante requirements and the lack of an AI liability framework to address any harm caused compromise the regulation’s effectiveness and ability to cope with the dynamic and emerging risks arising from frontier models.
This makes it, in principle, easier for US states’ attorney generals to regulate frontier models, given the narrower regulatory scope and the quantified assessment of risk. Yet the lack of a uniform, cross-state legal framework could be detrimental in the long run.
The best way forward
Comparing the EU and US state regulatory frameworks reveals a trade-off between adaptability and certainty. Where the EU’s focus on systemic risk is more holistic and potentially more future proof, it can also hinder market deployment due to ambiguity and the larger scope of the risks likely covered by the AI Act.
And while US states’ focus on catastrophic risk offers more regulatory certainty, their strict focus on extreme scenarios leaves major blind spots for widespread, cumulative and non-physical societal harms like disinformation, manipulation and systemic bias, which we’ve already seen with present-day AI tech.
In navigating these trade-offs, the EU mustn’t dilute its comprehensive approach to risks nor neglect the more extreme scenarios which US frameworks emphasise.
The solution is to develop a strategic framework based on evidence, probability and impact, covering the full spectrum of risks and paying attention to the complex interplay between technological, infrastructural and sociopolitical factors. This would ensure that resources are properly allocated and that AI policy is grounded in real concrete needs.
Since 1 August 2026, the EU’s regulatory power has gained teeth with the AI Act now permitting the AI Office to request documentation, model evaluations and access to frontier AI models – a process that’s reportedly already started. This opens a vulnerable but real window to test what anticipatory governance can achieve in the age of AI.