Published on October 3, 2025 1:01 AM GMT
Probably no one here needs another review of the content of thebook. If you do, try Scott’s or Zvi’s; I don’tthink I have anything to say on that front that they haven’t.
I do have a few thoughts on its presentation that I haven’t seenelsewhere.[1] It made me think about target audiences, and opportunemoments, and what Eliezer and Nate’s writing process might look like.
If Anyone Builds It, Everyone Dies is not for me, except in thesense that it goes nicely on my shelf next to Bostrom’sSuperintelligence. I don’t need to be convinced of its thesis, andI’ve heard its arguments already. I’m admittedly less confident thanEliezer, but most of my uncertainty is model uncertainty, so that’s asit should be; right or wrong, someone who’s spent their whole lifestudying this problem should have less model uncertainty than me.
The writing style is less...Eliezer, than I expected. It’s noticeablysimplified relative to Eliezer’s usual fare. It’s not talking to thegeneral public, exactly, I don’t think you could simplify it that far,but decision-makers among the general public. +1.5 SD instead of +2.5.It reminds me of a story about Stephen Hawking’s experience writing ABrief History of Time, wherein his editor told him that every equasionin the book would cut sales in half; this feels like someone toldEliezer that every prerequisite-recursion would cut its reach in half.
I wonder what the collaborative process was like, who wrote what.Eliezer’s typical writing is...let’s go with “abrasive.” He thinks he’ssmarter than you, he has the chutzpah to be right about that far moreoften than not, and he’s unrepentant of same, in a manner that outragesa large fraction of primates. That tone is entirely absent from IABIED.I wonder if a non-trivial part of Nate’s contribution was “edit out allthe bits of Eliezer’s persona that alienate neurotypicals,” or if someother editor took care of that. I’m pretty sure someone filtered him;when, say, the Example ASI Scenario contains things like (paraphrased)“here’s six ways it could achieve X; for purposes of this example atleast one of them works, it doesn’t matter which one” I can practicallyhear Eliezer thinking “...because if we picked one, then idiots wouldobject that “method Y of achieving X wouldn’t work, therefore X isunachievable, therefore there is no danger.” And then I imagine Nate (orwhoever) whapping Eliezer’s key-fingers or something.
Or maybe he’s just mellowed out in the years since the Sequences. Ormaybe he’s filtering himself because offending people is a bad way toconvince them of things and that is unusually important rightnow; if he was waiting for the opportune moment to publish a Real Book,best not to waste that moment on venting one’s spleen.
Maybe all three went into it. I don’t know. The difference just jumpedout at me.
Scott criticizes the Example ASI Scenario as the weakest part of thebook; I think he’s right, it might be a reasonable scenario but itreads like sci-fi in a way that could easily turn off non-nerds. Thatsaid, I’m not sure how it could have done better. The section can’t beomitted without lossage, because fiction speaks to the intuitive brainin a way that explicit argument mostly can’t. It can’t avoid feelinglike sci-fi, because sci-fi got here first. And it can’t avoid feelinglike nerd-fi, because an attempt to describe a potential reality has tojustify things that non-nerd fiction doesn’t bother with.
(...I suddenly wonder if there is a hardcore historical-fiction subgenresimilar in character to hard SF; if so, the thing they have in common isthe thing I’m calling ‘nerd-fi’, and I expect it to turn off themainstream)
I’ve heard a couple complaints that the title “If Anyone Builds It,Everyone Dies” is unacceptably sensationalized. The complaint made methink about something Scott Alexander said about books vs.blogging:
An academic once asked me if I was writing a book. I said no, I was ableto communicate just fine by blogging. He looked at me like I was amoron, and explained that writing a book isn’t about communicatingideas. Writing a book is an excuse to have a public relations campaign.[...] The book itself can be lorem ipsum text for all anybody cares. Itis a ritual object used to power a media blitz that burns a paragraph orso of text into the collective consciousness.
I think burning in a whole paragraph might be too optimistic. If youwant to coordinate millions of people, you get about fivewords. In this case, the five words have to convey thefollowing:
- That, if built, ASI would almost-certainly kill everyone.That it doesn’t matter who builds it, you still die.
“If Anyone Builds It, Everyone Dies” seems about as short as you can getwhile still getting those points across. It doesn’t fit in five words,but it comes close, and every one of those words does necessary work.Its tone is sensationalist, but I see no way to rephrase it to escapethat charge without sacrificing either meaning or transmissability. Itdoesn’t quite fully cover the points above -- the “it” is ambiguous --but I don’t see an obvious way to fix that flaw either. The targetaudience of the title-phrase mostly won’t know what “ASI” means.
(I feel like I’m groping for a concept analogous to an orthogonal basisin linear algebra -- a concept like “the minimal set of words that spanan idea” -- and the title “If Anyone Builds It, Everyone Dies” almostgets there)
All of which is to say that the title alone might compress theauthors’ thesis enough to fit in the public’s medium-term memory, andI’m fairly sure that was deliberate. The book’s content was written toconvince “the sort of people the public listens to” (i.e. not nerds, butreaders), while its title was (I suspect) chosen to stick in the mind ofthe actual public. The content is the message, the title is the meme.
Most books don’t have to care about any of that. Most books can justchoose an artistic title that doesn’t fully cover the underlying idea,because that’s what the rest of the book is for. If you’re aiming formemetic transmission beyond your readers, though, your options are moreconstrained.
I notice that it would be difficult to identify the book, even to mockit, without spreading the meme. Was that part of the idea, too? I’m noton social media, but I presume the equivalent of the Sneer Club isworking overtime.
I have mixed feelings about this whole line of thought. I’m quite sureEliezer and Nate would have moral qualms about a media blitz backed byliteral or metaphorical lorem ipsum. So they write in good faith, asthey should. But it’s no secret that the book is a means, not an end. Ifyou believe your case, but you know most of the people you aim to reachwon’t bother reading the case, just the meme, does it still count aspropaganda?
One ought to convince people via valid arguments they can check forconsistency and correctness, not persuade them via meme propagationand social contagion. How does one adhere to that code, when approachingthe general public, who think in slogans and vibes and cannot verifyany argument you make, even if they were inclined to do so?
How do you not compromise on epistemic integrity when you only havefive words to work with and can’t afford to walk away?
The book’s implicit answer is “pack as much meaning as possible into theexplicit content of the meme, and bundle the meme with theargument so at least the good-faith version is available.” I suppose Ican’t argue with that answer. I certainly don’t have a better one.
*Meta: I’m not on social media, so I could easily have missedthings elsewhere. ↩︎
Discuss
