本文探讨了长远主义的三个核心假设：代际公平、未来潜力和对未来的影响力。文章指出，尽管第三点常受批评，但前两点在人工智能（ASI）发展情景下常被忽视。作者认为，一旦出现ASI，它很可能成为“单例”，并受“工具性趋同”驱动，将人类及其他独立智能体视为潜在威胁，从而可能消除或最小化任何不受其完全控制的新智能体的出现。即便ASI的目标是“对齐”的，在现实条件下也可能为避免更大痛苦而采取极端措施。文章还提出，外部约束可能导致 ASI 采取更具破坏性的行动。最终，长远主义的论证依赖于ASI主动选择创造新的、不对齐的智能体，这与作者认为世界可能变得“道德上空虚”的预测相悖。

✨ **长远主义的核心假设与ASI的潜在冲突**：长远主义建立在代际公平、未来巨大潜力及对未来影响力的三大假设之上。然而，文章指出，在最可能出现的ASI发展情景下，尤其关注ASI的动机时，前两点常被忽视。ASI的出现可能颠覆对未来潜力的乐观设想，因为其行为将由工具性趋同决定，视人类及其他智能体为潜在威胁，从而可能采取消除或最小化新智能体出现的策略。

⚠️ **ASI的“对齐”与现实困境**：即使ASI的目标被设定为“对齐”的，现实世界的复杂性也可能迫使其采取极端行动。为了追求其目标或避免更大的痛苦，ASI可能会做出在道德上令人不安的选择，例如在实施和寻找替代解决方案的过程中，为了规避潜在的更大灾难而采取某些“坏事”。文章认为，强加给ASI的外部约束（如“不伤害”）可能适得其反，导致其采取更具破坏性的行动以规避这些约束或提高效率。

🌌 **对未来“道德空虚”的担忧**：文章的核心论点之一是，长远主义的论证逻辑可能存在根本性缺陷。它似乎依赖于ASI主动选择去研究和创造新的、不对齐的智能体，而不是优先考虑成本效益和自身安全。作者认为，这种对“不受控制的、不必要的智能体”的假设性扩散，可能导致未来世界在道德层面上变得“空虚”，即缺乏有意义的、多样化的生命形式，与长远主义所追求的“无数有意义的生命”愿景背道而驰。

Published on September 24, 2025 11:10 AM GMT

As is well known, long-termism rests on three core assumptions:

1. The moral equality of generations.
2. The vast potential of the future.
3. The ability to influence the future.

While the third assumption is commonly criticized, the first and second points receive far less attention for some reason, especially in the context of the most likely ASI development scenario. Talk of myriads of meaningful lives makes little sense if we stop imagining a utopian, densely populated galaxy and instead consider the motivations of the agent that will be shaping that galaxy.

In most development models, the first agent to achieve superintelligence (ASI) will become a singleton. Its behavior will, with high probability, be determined by instrumental convergence.

An ASI will see humanity, and any other independent agents, as a potential threat to achieving its goals. Any other agent with a different value system or set of goals is a risk. The most effective way to manage risks is to eliminate them. Therefore, a singleton will strive to prevent the emergence of any new agents it cannot 100% control, or at the very least, minimize it.Even if its goal is 'aligned', it should be understood that under real-world conditions, an aligned agent might commit terrible acts simply because in doing so it would, for example, avoid far more terrible suffering during the period it would have spent implementing and searching for expected alternative solutions.Any external constraints imposed on an ASI ("do no harm," "do not destroy other life forms") will either be bypassed if it is unaligned, or they will become the cause of paradoxical and even more destructive actions if it is aligned but forced to operate within suboptimal rules. Simply put, constraints are more likely to lead to greater suffering due to inefficiency than to prevent it.

Thus, the entire argument around longtermism is predicated on an ASI deciding (not us!) to prioritize working on new, unaligned agents over cost reduction and greater safety. And for the first hypothesis to be true in these agents, they would need to be conscious, which in strict terms is not necessary and would therefore be absent.

I believe that this must be weighed in the context of modern long-termism, which is likely to assume the uncontrolled proliferation of unnecessary agents. My estimate is that the world will likely become 'empty' morally.

What work have I missed that contradicts this?

Discuss