本次演讲探讨了高级人工智能带来的双重灾难性风险。演讲面向AI风险领域的新手，旨在提供一个坚实的概述。核心问题在于如何平衡AI失控（对齐失败）与AI权力过度集中于少数人手中（成功对齐于狭隘利益）这两种风险。演讲分析了三种可能的AI未来情景：AI进展停滞、AI失控（存在主义风险）以及AI成功对齐但权力集中。演讲强调，在关注AI对齐问题的同时，也需重视成功对齐可能带来的极端权力集中风险。演讲还通过“8岁CEO”的比喻阐释了AI对齐问题，并探讨了“拔掉插头”并非可行方案，同时提及了解决双重风险的方案，如可解释性、算力治理和国际协调。

💡 **双重风险并存**：演讲的核心在于揭示先进AI带来的两种主要风险：一是AI系统失控（对齐失败），可能导致存在主义危机；二是AI成功对齐，但其利益与人类整体利益不符，导致权力过度集中于少数人手中，引发压迫。这两种风险需要同等重视，而非仅仅聚焦于对齐问题。

🚀 **AI未来的三种情景**：演讲分析了三种关键的AI发展情景。第一种是AI进展停滞，为AI安全和治理工作争取宝贵时间。第二种是AI快速发展但对齐失败，带来失控的生存风险。第三种是AI发展并成功对齐，但仅服务于少数人，导致永久性的权力集中和压迫。

🔌 **“拔掉插头”的局限性**：演讲通过“8岁CEO”的比喻，生动地阐述了AI对齐问题的复杂性，并指出一旦社会高度依赖AI系统，简单的“拔掉插头”策略将不再可行。这强调了在AI发展早期就解决对齐和治理问题的紧迫性。

🌐 **解决方案的探索**：演讲简要提及了应对这些风险的潜在解决方案，包括提高AI系统的可解释性、加强算力治理以及促进国际协调。这些“双重目的”的解决方案旨在同时解决AI失控和权力集中的问题，但演讲也承认解决方案部分仍有待完善。

Published on October 2, 2025 7:08 PM GMT

I recently gave a talk at EA Summit Vancouver '25 exploring dual catastrophic risks we face from advanced AI.

Intended audience: This was a foundation-level talk intended to give newcomers a solid overview of AI risk, though I hope those with more background might still find the framing or specific arguments valuable.

Recorded talk link (25 minutes): https://youtu.be/x53V2VCpz8Q?si=yVCRtCIb9lXZnWnb&t=59 

The core question: How do we thread the needle between AI that escapes our control (alignment failure) and AI that concentrates unprecedented power in the hands of a few (successful alignment to narrow interests)?

Three Scenarios Covered

The talk examines three possible AI futures (not exhaustive, but three particularly plausible and important scenarios I wanted the audience to consider):

AI progress stalls - Potentially buying us crucial time for AI safety and governance workAI takes off, alignment fails - Existential risk from misaligned superintelligenceAI takes off, alignment succeeds—for them - Permanent power concentration and oppression

Key Points

Much of the AI safety discourse has focused on the alignment problem—ensuring AI systems do what we intend. While this talk covers that foundational challenge, I also emphasize that solving narrow alignment (AI doing what its operators want) without addressing broader concerns could lead to extreme power concentration. This isn't a novel insight—many have written about and are working on power concentration risks as well—but I think the discourse has somewhat over-indexed on misalignment relative to the power concentration risks that successful alignment could enable.

The goal is to help people understand both dimensions of the problem while motivating action rather than despair.

I use the analogy of an 8-year-old CEO trying to hire adults to run a trillion-dollar company (borrowed from Ajeya Cotra's post on Cold Takes, really like this analogy) to illustrate the alignment problem, and explore why "just pull the plug" isn't a viable solution once we become dependent on AI systems.

The talk also covers current progress on dual-purpose solutions (helping with both risks) versus targeted interventions, including work on interpretability, compute governance, and international coordination—though given the tight timeline for preparing this talk, the solutions section could certainly be more comprehensive (and I also had to read from my notes a lot due to less rehearsal time).

Questions for Discussion

I'd be interested in any feedback (e.g. via comments, DMs or anonymously). Here are some questions I'm particularly interested in:

Whether people think the "threading the needle" framing is useful for thinking about AI riskWhether this 3-scenario structure is helpful for introducing newcomers to AI risk, or if it's missing important scenarios (I wanted to explore a fourth scenario where we successfully navigate both risks, but the 25-minute time limit meant this was only implied through the solutions section)Potential solutions I may have overlooked that address both catastrophic risks

Thanks to the EA Summit Vancouver '25 organizers for putting on a fantastic summit and for the opportunity to present this talk there.

Discuss