本文探讨了超级智能（superintelligence）可能带来的“认知爆炸”（intelligence explosion）对人类社会的深远影响。作者认为，与仅有“认知颠覆”（epistemic disruption）相比，我们更可能面临“认知崩溃”（epistemic collapse），即人类赖以判断真伪的共享框架瓦解，大多数人可能被排除在文明决策之外。文章通过虚构场景描绘了普通人在面对AI驱动的现实时可能经历的困惑、不信任和认知隔阂，并强调了人类大脑的进化速度远不及技术发展，可能导致“自然人”阶层的出现。作者呼吁关注并研究如何构建“认知支撑”（epistemic scaffolding），以帮助人类适应巨变，避免被时代抛弃。

🤖 **认知崩溃的风险远超认知颠覆**：文章的核心论点是，超级智能的出现可能导致人类社会失去判断真伪的共有框架，这是一种系统性的崩溃，而非简单的信息混乱。作者认为，将21世纪的进步压缩到十年，人类的认知系统将难以适应，即使AI技术本身解决了对齐问题，社会心理层面的调整也极其困难。例如，AI可能颠覆我们对因果、意识甚至逻辑的基本认知，届时传统的“事实核查”将失效。

🧠 **AI辅助工具的局限性与信任危机**：尽管AI可以提供事实核查、自动化预测和增强智慧等功能，但这些工具的有效性依赖于基本的认知稳定性。在认知崩溃时期，信任成为关键问题。AI建立信誉需要时间，而快速变化的现实使人们难以信任任何信息源，即使是拥有良好记录的AI。此外，人们可能更倾向于选择提供心理慰藉而非挑战性真相的AI，加剧了信息茧房和认知隔阂。

🌐 **“自然人”阶层的出现与社会分化**：随着脑增强、神经接口等“超人类主义”干预措施的普及，那些选择不进行技术增强的人可能形成一个“自然人”的下层阶级。他们的认知能力将无法跟上AI驱动的社会发展速度，导致他们在民主决策和社会进步中几乎没有发言权。这种认知层面的巨大鸿沟将使得民主制度难以维系，社会将出现前所未有的不平等。

💡 **应对认知崩溃的探索方向**：作者提出需要关注并研究如何防止认知崩溃，尽管没有明确的解决方案。可能的方向包括：从历史上的重大观念转变（如从宗教到世俗化）中学习成功经验；建立“认知支撑”机制，如设计能够跨越不同认知层级的AI系统或社会结构；加强对AI说服力的评估，了解其改变世界观的能力；以及研究如何在不完全理解底层逻辑的情况下，让普通人做出明智决策，类似我们乘坐飞机但无需理解空气动力学。

🚀 **数字复生与偏好外推的新型认知威胁**：超级智能还可能带来科幻般的新型认知威胁，例如“数字复生”，即AI根据个人数字痕迹模拟已故亲人，提供高度逼真的互动。另一个威胁是“偏好外推”，AI声称比用户本人更了解其“真正”的偏好。这些技术模糊了真实与模拟、自我认知与外部定义的界限，进一步加剧了认知混乱和对现实的判断困难。

Published on August 19, 2025 11:48 PM GMT

If an intelligence explosion occurs, the vast majority of people will be confused, misled, and epistemically disempowered – with no agency over the future. Unless we try to change this.

Introduction

While knowledge is invisible,^[1] it defines and shapes the world around us. It dictates how we decide what is true, and how we take action on such truths. It is undeniable that the advent of superintelligent AI systems would irreversibly change how we relate to knowledge on both an individual and societal level.

MacAskill and Moorhouse refer to this as epistemic disruption in their paper “Preparing for the Intelligence Explosion”. When defining the intelligence explosion, they use the analogy of a compressed century. What if all the scientific, social, philosophical, and political advancements of the 21st century were compressed into just 10 years? MacAskill compares humanity’s situation to:

"A mediaeval king suddenly needing to upgrade from bows and arrows to nuclear weapons to deal with an ideological threat from a country he's never heard of, while simultaneously grappling with learning that he descended from monkeys and his god doesn't exist.” (emphasis mine)

The authors already identify several ways in which superintelligence would affect society’s decision-making abilities. These include: super-persuasion, stubborn resistance to valid arguments, viral ideologies, and ignoring new crucial considerations (e.g. discovering something as big as heliocentrism). The relevant section of the paper is fairly short – so I’d encourage you to go read it if you haven’t already, before proceeding with this post.

While I agree with all of the risks identified in the section, I believe that the authors are massively underestimating just how turbulent and destabilising the situation would be for humanity. It is far from clear that the impact will “likely be positive overall”, as they claim. It very well may be. But we are probably underrating the amount of work required for that to be the case.

Beyond epistemic disruption (which I take to imply manageable turbulence), I think we would be facing a potential epistemic collapse – a systemic breakdown of how humanity decides what is true. If we take seriously their idea of a century of progress compressed into a decade (or less), we face a very difficult challenge in helping people adapt when fundamental beliefs are rapidly proven wrong – and how they decide what to believe in the first place. Even if we solve the technical challenges relating to AI (e.g. alignment), this social/psychological adjustment will be very hard to get right.

Could AI Really Help Solve This?

Yes, the benefits they identify from AI-enhanced reasoning (fact/argument checking, automated forecasting, and augmented/automated wisdom), would definitely help. But these solutions assume a level of epistemic stability that may not exist during the intelligence explosion. Consider fact-checking. The authors point to community notes on Twitter/X as a success story that AI could build upon. But community notes work precisely because they operate within a shared epistemic framework – users may disagree on facts, but they generally agree on what constitutes evidence. What happens if superintelligence discovers that our fundamental assumptions about causality, consciousness, or even logic are wrong? You probably can't fact-check your way out of something like this.

I think there’s a much stronger case for automated forecasting working, but it too has a critical weakness: trust. They suggest AI systems could "build up a strong track record" that generalises to controversial domains. But track records take time to establish, and time is exactly what people won't have during an intelligence explosion. More fundamentally, if people's entire worldviews are crumbling monthly, why would they trust anything, even an AI with a perfect prediction record? We already see this with something like climate denial. There are many cases where overwhelming evidence doesn't overcome worldview-level resistance.

"Augmented and automated wisdom" presumes people will want to turn to augmented wisdom when they perceive their most basic beliefs as being under assault. During an epistemic collapse, we’d lose any shared framework for determining what counts as “augmented wisdom” versus “augmented manipulation”. Some people may embrace every new AI-delivered truth uncritically. Others will reject everything defensively. Most will probably oscillate between the two, without any stable ground for making distinctions. The crisis would be the fragmentation, not just which direction people end up fragmenting. We already have seen this pattern, albeit more slowly. Darwin published On the Origin of Species over 100 years ago, yet only ~41% of humanity accepts evolution. COVID-19's uncertainty didn't lead to collective learning but to an explosion of conspiracy theories. Each group inhabited completely different realities^[2].

MacAskill and Moorhouse conclude that “selection pressures will probably favour desired traits on the epistemic front” because users will prefer honest and truthful models. But this assumes people can accurately assess truthfulness when their entire epistemic environment is breaking down before them. It seems that current evidence also suggests otherwise – social media algorithms already optimise for engagement over truth, and users consistently choose content that confirms their biases over content that challenges them. Sycophancy is a very big problem in current AI systems.

During an intelligence explosion, all these issues would be magnified. How do you select for “truthfulness” when the nature of truth itself is being revised monthly? Most users plausibly would select for AI systems that provide psychological comfort and coherent narratives, not those delivering difficult truths about the changing nature of reality as we know it!

Maybe AI will just get good at changing people’s minds, and we won’t need to worry about all this. But, how would this work in practice? Would this create over-reliance on the AI and whatever goals/values it is aligned to (e.g. in terms of the model spec or if it was controlled by a very small group of people)?

More Drivers of Destabilisation

Beyond the risks identified above, the intelligence explosion would plausibly introduce entirely new epistemic threats. Ones that sound like they are straight out of a sci-fi movie.

Consider the concept of digital resurrection. Superintelligent AI could create hyperrealistic simulations of deceased individuals based on their digital footprints, writings, and recordings. Imagine your dead grandmother calling you, sounding exactly like herself, sharing memories only she would know (realistically interpolated from data), and giving you advice about your life. Is this really her preferences and wisdom, or an AI's best guess? While some people would adjust to this and improve their “cognitive security” measures, many would not be able to keep pace with the rate of technological change.

Or preference extrapolation – AI systems that claim to know what you "really" want better than you do, based on patterns in your behaviour you're not even conscious of. When an AI can predict your choices with 99.9% accuracy and explain unconscious drives you didn't know you had, who is the authority on your own preferences? I’d imagine that some people would agree to adhere to AI-revealed-preferences, while others would double down on their own human cognition.

The New Underclass of Those Who Do Not Wish to Enhance

This situation isn't helped by the fact that the intelligence explosion likely would make transhumanist^[3] interventions (e.g., cognitive enhancement^[4], physical enhancement, direct neural interfaces, and so on) available to those who desire them and have the means to access them.

But what about the ones who do not wish to enhance and/or augment their capabilities?

A new "naturalist" underclass may emerge. Even if people have the tools to overcome their epistemic crisis, many would probably purposefully choose not to implement them due to fear, appeal to nature fallacies, or just extremely strong emotional aversion. Humanity has integrated with technology in the past (e.g. glasses, medicine, vaccines, etc), and we continue to become more transhumanist. However, this would be a sudden jump like nothing we've seen before.^[5] Our normal human brain is not designed for the blindingly fast levels of change that would be accompany the intelligence explosion. Our species’ technological capabilities have raced ahead, but our brains remain mostly unchanged since they evolved about 200,000 years ago. The enhancements required to keep up would be drastic – not just wearing a device, but fundamentally restructuring how your brain processes information (or even relying on an external AI system to process and simplify nearly all the information you receive).

Therefore, people who say no (which could plausibly be a very large percentage of the human population) will have no say – or a very limited say – in what the future looks like. This would be massively disempowering. They would functionally become children (or, even newborns if the intelligence explosion gets really crazy) in a world run by incomprehensible adults. Democracy would become impossible when citizens are operating at such fundamentally different cognitive levels^[6]. The un-enhanced would be using entirely obsolete frameworks for determining truth. Meanwhile, the enhanced would be moving further into AI-mediated realities the rest of humanity couldn’t even begin to perceive if they had thousands of years at their disposal.

The World From a “Normal” Human’s Perspective

Here's a fictional scenario (written with the help of Claude) of what epistemic collapse might feel like for a “normal” human:

Sarah is a 45-year-old teacher. The year is 2034, two years into the intelligence explosion^[7]. Like most of humanity, she was effectively kept in the dark that an intelligence explosion was even occurring. Though, now she could see it before her eyes. The future had come crashing down upon the present, and the world was beginning to look more and more like sci-fi.

Sarah refused all forms of enhancement, due to fears about keeping her brain “untouched” and placing her trust in “mother” nature^[8] instead. Every morning she faces the same problem: she can't tell what's true anymore. Her enhanced sister sends her "fact-checked" news through an AI system that claims to filter manipulation, but how can Sarah verify the fact-checker? She's stuck trusting black boxes or trusting nothing. Her dead mother called yesterday. Perfect voice, shared memories only they knew, offering advice about her divorce. Sarah has heard about digital resurrection, but knowing doesn't make her immune to just how scarily realistic it is. Are these her mother's actual preferences? An AI's best guess? The technology to verify doesn't exist in any form she can understand.

At work (assuming she is even able to find employment in such a world), enhanced colleagues operate through AI-mediated channels she can't access. When she asks what they're teaching, they try to explain but the conceptual frameworks they use just don’t exist in her un-enhanced brain.

She watches her social circle fragment. Many adhere to AI-led cults and/or new religions. AI romantic partners are very common, and many are advocating for legal protections for such systems. Her brother embraces every AI revelation uncritically – "the best AI scientists say we do actually live in a simulation, and here are the objectively morally valuable actions to take!" Her best friend rejects everything defensively – "they're rewriting reality to control us!" Most people, like Sarah, move back and forth between the two, with no stable ground for distinguishing augmented wisdom from augmented manipulation.

Various forms of human enhancement are widespread in this world, and do not face issues relating to equality of access. The enhanced tell her she's choosing to be left behind. But when worldview-shattering discoveries and understanding them requires restructuring your brain, what choice is that really? She's become unable to participate in day-to-day life, let alone decisions shaping humanity's future.

Preventing Epistemic Collapse

So, what can we do to prevent this from becoming our future? I don't have good answers for how to prevent epistemic collapse, and it seems like a very hard problem – very worth of its “grand challenge” title. But I think it's worth bringing attention to it, and that’s what this post is trying to do. Here are some thoughts on what future work in this area could look like:

Learning from historical transitions. We need to find institutions and movements that have managed major belief changes successfully. E.g., what allowed some societies to navigate the shift from religious to secular worldviews without completely fragmenting?Epistemic scaffolding. Maybe we need transitional institutions that can help people adjust gradually. This might mean AI systems designed to translate between different cognitive levels, or social structures that maintain continuity even as understanding shifts.Better evaluation of AI persuasion. We need evaluations that test AI's ability to shift entire worldviews, not just individual opinions. How persuasive can these systems become? How quickly?Mapping epistemic resilience. What does the current state of epistemic resilience actually look like? There are already people thinking about and working on “AI for epistemics”, and there are some talent-building initiatives on this (like the Fellowship on AI for Human Reasoning).Truth-tracking without understanding. Can we develop systems that let un-enhanced humans make good decisions even when they can't understand the underlying reality? This sounds paradoxical, but I’d argue we already do this in some sense. Most people don't understand how airplanes fly but trust them anyway.

Conclusion

I hope I’m wrong. Maybe AI's impact on epistemics will be positive overall. But I think we’re still underestimating just how bad it could get. The difference between "disruption" and "collapse" matters. Disruption implies turbulence we can navigate. Collapse means the system breaks.

During the intelligence explosion, I think we're looking at potential collapse – where humanity loses any shared framework for determining what's true, and where most people become cognitively excluded from civilisation's decisions. This would be a very bad future, and we must work to prevent it.

Acknowledgements

Thank you to Duncan McClements for providing useful feedback.

 

1. ^{^}

   Unless you view human nerve cells and astrocytes up close and figure out what our beliefs physically look like (or do something vaguely similar with mechanistic interpretability in AI), etc, etc.

2. ^{^}

   A quarter of the UK population believes COVID was a hoax.

3. ^{^}

   This is my favourite definition of transhumanism.

4. ^{^}

   “Machines of Loving Grace”, an excellent (though optimistic) essay on a world with “powerful AI systems”, does a very good job of describing what AI-accelerated neuroscience and biology could enable.

5. ^{^}

   The jump from "wearing glasses" to "installing GPT-12 in your prefrontal cortex" isn't gradual adaptation.

6. ^{^}

   Obviously, citizens are already operating at different cognitive levels, but one would imagine the difference between a “normal” human and a human with AI-enhanced reasoning would be far greater than the gap between an 85-IQ citizen and a 130-IQ citizen.

7. ^{^}

   This date is an illustrative example and not representative of my actual timelines.

8. ^{^}

   “Mother” is in quotation marks because she is, in reality, arguably a terrible mother

Discuss