文章探讨了当前关于禁止通用人工智能（AGI）的提议所面临的法律和定义上的困境。作者指出，许多提议将AGI定义为可能导致人类灭绝的AI，这种基于结果的定义在法律上是无效的，因为法律需要在损害发生前介入。文章强调了严格责任和清晰阈值的重要性，类比了核武器条约的成功经验，即通过禁止具体的“前体”能力而非模糊的“灭绝风险武器”来达到监管目的。文章呼吁将法律专家纳入AGI监管的讨论中，以制定出切实可行、可执行的法律框架，而非依赖于易被规避的计算能力上限等代理指标。最终目标是制定出与核条约在核材料管控上同等精确和可执行的AGI前体能力禁令。

⚖️ 基于结果的AGI禁令定义存在法律缺陷：文章指出，将AGI定义为“可能导致人类灭绝”的AI，其法律效力有限，因为监管需要在潜在危害发生前触发，而一旦发生灭绝，将无人能够执行法律。因此，法律必须关注导致危害发生的具体前体能力，而非最终结果。

💡 严格责任与清晰阈值是关键：借鉴核武器条约的成功经验，文章强调AGI禁令应设定具体、可衡量的能力阈值，如自主复制、规模化资源获取和系统性欺骗等，而不是依赖模糊的定义。这类似于核条约禁止零当量的核试验、特定数量的核材料或交付系统，从而实现可执行的监管。

🏛️ 法律专家的参与至关重要：文章认为，AGI监管的定义工作不应仅限于政策制定者，必须引入法律专家。否则，政策制定者可能倾向于使用易于测量的代理指标（如计算能力上限），而企业内部的法律顾问则会找到规避禁令的方法。法律专家的参与能确保禁令的精确性和可执行性，避免其沦为形式化的规定。

Published on September 20, 2025 11:01 AM GMT

TL;DR

Most “AGI ban” proposals define AGI by outcome: whatever potentially leads to human extinction. That’s legally insufficient: regulation has to act before harm occurs, not after.

Strict liability is essential. High-stakes domains (health & safety, product liability, export controls) already impose liability for risky precursor states, not outcomes or intent. AGI regulation must do the same.Fuzzy definitions won’t work here. Courts can tolerate ambiguity in ordinary crimes because errors aren’t civilisation-ending and penalties bite. An AGI ban will likely follow the EU AI Act model (civil fines, ex post enforcement), which companies can Goodhart around. We cannot afford an “80% avoided” ban.Define crisp thresholds. Nuclear treaties succeeded by banning concrete precursors (zero-yield tests, 8kg plutonium, 25kg HEU, 500kg/300km delivery systems), not by banning “extinction-risk weapons.” AGI bans need analogous thresholds: capabilities like autonomous replication, scalable resource acquisition, and systematic deception.Bring lawyers in. If this definitional work is left to policymakers, they’ll default to blunt proxies (compute caps). In-house counsels will then map the gray zones and advise on how to bypass the ban.

Bottom line: If we want a credible AGI ban, we must define and prohibit precursor capabilities with the same precision and enforceability that nuclear treaties applied to fissile material. Anything less collapses into Goodharting and fines-as-business-cost.

Why outcome-based AGI bans' proposals don’t work

I keep seeing proposals framed as “international bans on AGI,” where AGI is defined extremely broadly, often something like “whatever AI companies develop that could lead to human extinction.” As a lawyer, I can’t overstate how badly this type of definition fails to accomplish its purpose. To enable a successful ban on AGI, regulation has to operate ex ante: before the harm materialises. 

If the prohibited category is defined only by reference to the ultimate outcome (in this case, human extinction), then by construction the rule cannot trigger until after the damage is done. At that point the law is meaningless; extinction leaves no survivors to enforce it.

That’s why the definitional work must focus not on the outcome itself, but on the specific features that make the outcome possible: the capabilities, thresholds, or risk factors that causally lead to the catastrophic result. Other high-stakes tech domains already use this model. Under the European General Data Protection Regulation, companies can be fined simply for failing to implement adequate security measures, regardless of whether an intentional breach has occurred. Under European product liability law, a manufacturer is liable for a defective product even if they exercised all possible care to prevent such defect. And even under U.S. export-control law, supplying restricted software without a licence is an offence regardless of intent.

The same logic applies here: to ban AGI responsibly, we need to ban the precursors that make extinction-plausible systems possible, not "the possibility of extinction" itself.

The luxury of "defining the thing" ex post

TsviTB, in What could a policy banning AGI look like?, poses a fair question: “Is it actually a problem to have fuzzy definitions for AGI, when the legal system uses fuzzy definitions all the time?”

In ordinary law, the fuzziness is tolerable because society can absorb the error. If a court gets it wrong on whether a death was murder or manslaughter, the consequences are tragic but not civilisation-ending. And crucially, many of these offences carry criminal penalties (actual prison time) which creates a strong incentive not to dance around the line.

But an AGI ban is likely to sit under product safety law, at least initially (more on this next), where penalties are usually monetary fines. That leaves the door wide open for companies to “Goodhart” their way into development: ticking compliance boxes while still building systems that edge toward the prohibited zone.

This is a painfully common dynamic in corporate law. For example, multinationals routinely practice tax avoidance right up to the legal line of tax evasion. They can prove with audited books that what they’re doing is legal, even if it clearly undermines the spirit of the law. They still avoid paying far more tax than they would under a normal corporate structure, one SMEs can’t afford to set up. 

In practice, they achieve the very outcome the law was designed to prevent, but they do it legally. They don’t get away with all of it, but they get away with maybe 80%.

We don't have this luxury: we cannot afford an AGI ban that is “80% avoided.” Whether the framework sits under civil or criminal law, it will only work if it sets a robust, precise threshold and attaches penalties strong enough to change incentives, not just fines companies can write off as a cost of doing business.

Actually defining the thing we want to ban

If an agreement like this is to work, the first item on the agenda must be to define what counts as the thing you want to ban. 

Why do I think an AGI ban will default to a product safety framework rather than a criminal law framework? Because that’s the path the EU AI Act has already taken. It sets the first precedent for “banning” AI: certain systems are prohibited from being put on the market or deployed when they pose irreversible societal harms (e.g. manipulative systems, exploitative targeting, biometric categorisation, social scoring).

For readers less familiar with the EU AI Act, here’s the exact list of practices it bans under Article 5:

Summary of Prohibited AI Practices under Art.5 EU AI Act

The EU AI Act prohibits placing on the market, putting into service, or using AI systems that:

Manipulate behaviour: use subliminal, manipulative, or deceptive techniques that significantly impair decision-making and cause harm.Exploit vulnerabilities: target people due to age, disability, or socio-economic status, distorting behaviour and causing harm.Social scoring: evaluate/classify people based on behaviour or traits, leading to unjustified or disproportionate detrimental treatment.Predict criminality: assess risk of someone committing crimes based solely on profiling or personality traits.Facial scraping: create or expand biometric databases by untargeted scraping from the internet or CCTV.Emotion recognition: infer emotions in workplaces or schools (except for medical/safety use).Biometric categorisation: classify individuals by sensitive attributes (race, religion, sexual orientation, etc.), with limited law enforcement exceptions.Real-time biometric identification (RBI): use RBI in public spaces for law enforcement (with exceptions).

 

But notice the structure:

It doesn’t ban development of these systems. Technically, a company can still build them internally.It creates civil liability, mainly monetary fines. That’s deterrence, not prevention.Enforcement is ex post: the ban only bites once the system has been deployed and the harm has become measurable.

This is exactly the failure mode I worry about. If an “AGI ban” is drafted the same way, it will look tough on paper but in practice it will be little more than a product-safety regulation: companies treating fines as a cost of doing business, and governments tolerate it for the sake of “innovation". 

That’s why the definitional work matters so much. If the ban is going to be enforceable, we can’t define AGI in terms of its final outcome (extinction) or leave it to vague product-safety language. 

If the AI Safety field fails to define the thing we want to ban, the task will be left up to policymakers, who will reach for the "most measurable" available proxy ( likely compute thresholds) and the entire ecosystem will Goodhart against that.

We need a crisp proxy for what makes a system cross the danger line: capability evaluations, autonomy thresholds, demonstrable ability to replicate, deceive, or seize resources. Something specific enough that strict liability can attach before catastrophe, not after. 

Credible bans depend on bright lines

The political reality is that states are unlikely to agree to halt all frontier development. This is clear for anyone who has read the U.S.' AI Action plan. Even Europe, often seen as the strictest regulator, is taking an ambitious "pro-innovation" stance. 

If a proposal for an AGI ban is to succeed, it has to be precise enough to block “AGI coup”-class systems while still permitting beneficial progress. Otherwise, it will be dismissed as too restrictive or difficult to actually enforce without expensive innovation caps.

Learning from nuclear treaties

Nuclear treaties don’t ban “whatever weapons might end humanity.” They ban specific precursor states and activities with crisp, enforceable thresholds: zero-yield tests, “significant quantities” of fissile material (8 kg of plutonium, 25 kg of HEU), and delivery systems above 500 kg/300 km. These bright lines allow intrusive verification and enforcement, while still permitting conventional weapons to exist^[1]. The principle is clear: regulate by measurable inputs and capabilities, not by catastrophic outcomes.

Until similar definitional work is done for AGI, talk of an “AGI ban” is rhetoric. Useful rhetoric, yes!: It shifts the Overton window, it mobilises groups like PauseAI and ControlAI, it keeps extinction-level risk on the policy agenda. But as law, it will fail unless we solve the definitional problem and bring legal experts into the room (not just policymakers^[2].). Otherwise, in-house counsels at AI labs will simply map the gray areas and find compliant-looking ways to bypass the ban.

If we want a credible ban, we need to put the same intellectual effort into defining precursor thresholds for AI that we once put into fissile material, missile ranges, and test yields. Anything less will collapse into Goodharting and "product safety rules" with fines as the only compliance incentive. 

 

1. ^{^}

   I do not endorse the manufacturing of any weapons, I am merely using this as an example for illustrative purposes. 

2. ^{^}

   One blind spot I notice is how rarely tech lawyers are brought into AI safety strategy. Lawyers in large firms or in-house roles often sit at the chokepoints of real leverage: they can delay deals, rewrite contractual terms, and demand transparency in ways that policymakers often cannot. In the EU especially, an AGI ban (or any ambitious legislative action) would ultimately be implemented, interpreted, and either undermined or strengthened by these lawyers. If they are left out of the conversation, the path of least resistance will be to map the gray areas and advise clients on how to bypass the spirit of the law. 

Discuss