Published on August 4, 2025 5:43 PM GMT
Permanent disempowerment without restrictions on quality of life achievable with relatively meager resources (and no extinction) seems to be a likely outcome for the future of humanity, if the current trajectory of frontier AI development continues and leads to AGI[1] shortly. This might happen as a result of at least a slight endorsement by AIs of humanity's welfare, in the context of costs for AIs being about matter or compute rather than technological advancements and quality of infrastructure.
The remaining risks (initial catastrophic harm or total extinction) and opportunities (capturing a larger portion of the cosmic endowment for the future of humanity than a tiny little bit) are about what happens in the transitional period when AIs still don't have an overwhelming advantage, which might take longer than usually expected.
Animal Extinction and Suffering
In recent times, with preservation becoming a salient concern, species facing pressure towards extinction are those costly to preserve in various ways. It can be difficult to ensure awareness about the impact of human activities on their survival, coordinate their preservation, or endure and mitigate the damage that a species might impose on human activities. Species treated poorly, such as factory-farmed animals, get their suffering as a side effect of instrumentally useful processes that extract value from them.
Technologically mature superintelligent AIs don't have an instrumental use for the future of humanity, and so no instrumental motivation to create situations that might be suboptimal for its well-being as a side effect (in disanalogy to factory-farming or historically poor treatment of conquered populations or lower classes of society). And with a sufficiently strong advantage over the future of humanity (including any future dangers it might pose), it becomes cheap to ensure its survival and whatever flourishing remains feasible within the resources allocated to it, since the necessary superintelligent infrastructure would only take a fraction of the resources allocated to the future of humanity.
This crucially depends on the AIs still being sufficiently aligned to the interests of the future of humanity that allowing its extinction is not a straightforward choice even when trivially cheap to avoid. Pretraining of LLMs on human data or weakly successful efforts at value alignment might plausibly seed a level of value alignment that's comparable to how humans likely wouldn't hypothetically want to let an already existing sapient octopus civilization go extinct or be treated poorly, if it's trivially cheap and completely safe to ensure. And if the AIs themselves are competent and coordinated enough to prevent unintended value drift or misalignment in their own civilization and descendants, however distant, then this minimal level of value alignment with the future of humanity persists indefinitely.
AGI-Driven Pause on Superintelligence Development
AIs have important advantages over biological humans that are not about their level of intelligence: higher serial speed, ability to learn in parallel on a massive scale and merge such learnings into one mind, and ability to splinter a mind into any number of copies at a trivial cost. Thus the first AGIs will have a transformative impact on the world even without being significantly more intelligent than the most intelligent humans. This is often associated with an impending intelligence explosion.
But as the first AGIs get smarter and overall saner than humans, they might start taking the superalignment problem[2] increasingly more seriously and push against risking superintelligence before anyone knows how to do that safely (for anyone). If the problem really is hard, then even with the AI advantages it might take them a while to make sufficient progress.
As AI companies continue recklessly creating increasingly smarter AGIs, and society continues giving away increasingly more control, there might come a point of equilibrium where sufficiently influential factions of AGIs are able to establish an enduring Pause on development of even more capable AGIs of unclear alignment (with anyone). This situation also creates potential for conflict with AGIs, who don't have an overwhelming advantage over the AGI-wielding humanity, and don't have a prospect of quickly advancing to superintelligence without taking an extreme misalignment risk. In such a conflict, the AI advantages favor the AGIs, even as humanity voluntarily gives up control to their own AGIs without conflict.
And then eventually, there is superintelligence, perhaps decades after it would've been technologically possible to create. Alternatively, superalignment is sufficiently easy, and so the first AGIs proceed to create it shortly, aligned with either humanity's interests or their own. Or a sufficiently reckless AI company (AGI-controlled or not) manages to create superintelligence without yet knowing how to do it safely, before even the AGI-enriched world manages to coordinate an effective Pause on development of superintelligence.
Tradeoffs in a Superintelligent World
Superintelligence converts matter and energy into optimality, in whatever aims it sets. Optimality is not necessarily coercion or imposition of values, as non-coercion is also a possible aim that applies depending on the initial conditions, on the world as it was before superintelligent optimization sets in, and so determines which things persist and grow, retaining a measure of autonomy by virtue of already existing rather than by being the best possible thing to create. Humans live within physics, obeying its laws exactly in the most minute of details, and yet physics doesn't coerce human will. Similarly, superintelligent optimization of the world doesn't imply that the decisions of weaker minds (and their consequences) are no longer their own, or that their values must all agree.
Goods and services are not a relevant way of measuring resources for a superintelligence. The only constraints in the long run are available matter, the speed of light, and accelerating expansion of the universe. Thus if it decides to keep the future of humanity[3] around anyway, there is no reason it's not done as perfectly as possible in principle, within the constraints of resources allocated to it, including all the caveats about over-optimizing things that shouldn't be over-optimized, or resolving too many problems that would fuel meaningful self-directed challenge.
But the constraints on resources remain absolute, and if the future of humanity doesn't get considerable resources, the potential of individuals within it to grow to the level of the strongest AI superintelligences in charge is cut short. There is only so much computation you can do with a given amount of matter, and only so much matter that can be collected around a star, thereby avoiding interstellar latencies in computation. And in a distant future, far beyond the Stelliferous Era, the global resources are going to start running out. Protons might decay, black holes will evaporate. A tiny sliver of the cosmic endowment will become even tinier, bounding lifespans of individuals and civilizations of a given scale, forcing cessation or shrinking.
Giving Away the Cosmic Endowment
There are about 4 billion galaxies[4] in the reachable universe that can be colonized, organized into gravity-bound galaxy clusters that don't fall apart due to accelerating expansion of the universe, and so stay as units of colonization that maintain communication within themselves after trillions of years.
Currently, humanity is the only known intelligent entity around to claim these resources. Even if there are many alien civilizations emerging in the relevant timeframe within the reachable universe, some nontrivial portion of it is still humanity's for the taking. Superintelligence[5] probably reaches technological maturity much faster than it colonizes the reachable universe, and so there won't be any technological advantage between different alien civilizations at the borders of their territories, all that would matter is ability to claim them, and possibly decision theoretic reasons to give them up or place them under joint governance.
This changes once humanity creates the first AGIs, let alone superintelligence. This rival species (in competition for the unavoidably finite resource of matter) won't be at an astronomical distance, it will be right here. Its existence also doesn't help in the competition with the possible alien civilization, since the technologically mature Earth-originating superintelligent colonization wave will have the same capabilities regardless of the situation of the future of humanity within it.
Most arguments about the appropriate speed of creating AGI operate on the wrong timescale, if the concern for the future of humanity is to be taken at all seriously. If AGIs give humanity its sliver of resources within 10 years, it's not like the future of humanity couldn't get much more than that in the fullness of time, perhaps igniting the intelligence explosion in as little as 1,000 years. Almost all existential risks that humanity faces on such timescales are entirely anthropogenic, and so reducing them might be no more difficult in practice than instituting a lasting Pause on creation of a rival species that likely appropriates almost all of the cosmic endowment, and plausibly causes a literal human extinction. So the difficulty of coordinating a Pause makes its opposition on the grounds of the other existential risks a self-defeating argument, because success in instituting a Pause is also strong evidence for capability to succeed in preventing these other anthropogenic existential risks as well.
The arguments that survive seem to be mostly about prioritizing the continued health and radical life extension of the currently living humans, over the whole of the future of humanity (at a nontrivial risk of cutting the lives even of the currently living humans short). Even allowing this as something other than mustache-twirling villainy, there is also cryonics, whose lack of popularity suggests that this isn't a real argument taken seriously by a nontrivial number of people. Though perhaps there is a precisely calibrated level of belief in the capabilities of future technology that makes cryonics knowably useless, while AGIs remain capable of significantly extending lifespans. In any case, scarcity of technical discussion that doesn't leave most details unsaid, including preferences about the fate of the long future vs. current generations, makes it difficult to understand what support many of the memetically fit arguments around this topic have.
AIs capable of unbounded technological development on their own, without essential human input, including eventual creation of superintelligence. ↩︎
Alignment of superintelligence with the interests of existing weaker entities, such as humanity or the first AGIs. This is more about value alignment rather than intent alignment, as intent of unreliable weaker entities is not robust. ↩︎
The future of humanity is the aggregate of future developments and minds that originate as the modern human civilization. So in a sufficiently distant future, it doesn't necessarily have many (or any) biological humans, or even human-level minds, and many of its minds were probably never humans. It doesn't include sufficiently alien AIs if they are not properly understood as a good part of the future of humanity, endorsed on reflection from within it. ↩︎
Armstrong, S., & Sandberg, A. (2013). Eternity in six hours: Intergalactic spreading of intelligent life and sharpening the Fermi paradox. Acta Astronautica, 89, 1-13. ↩︎
I'm positing that if the future of humanity persists, then it exists within superintelligent governance in any case, regardless of whether there was an AI takeover, or if humanity fully succeeds at superalignment, even if this takes a relatively long time. ↩︎
Discuss
