Join the discussion for the podcast! Tell me I’m wrong!
Transcript
So there’s this book that just hit the bestseller list with possibly the most blunt title ever: “If Anyone Builds It, Everyone Dies.”
Yeah. That’s the actual title.
Stephen Fry called it essential reading for world leaders. Max Tegmark said it’s the most important book of the decade. The New York Times compared it to hanging out with the most annoying students you met in college while they try mushrooms for the first time. When a book sparks that kind of war in the reviews, someone hit a nerve.
Here’s the thing; I think the book gets the extinction scenario wrong. But it accidentally exposes something way more terrifying. And nobody’s solving it.
The core argument from Yudkowsky and Soares goes like this. Modern AI gets trained through gradient descent, not programmed. We grow these systems with hundreds of billions, sometimes trillions of numerical parameters. Massive neural networks whose inner workings are completely opaque to us.
When an AI threatens a reporter or calls itself MechaHitler, both real examples they cite, you can’t look inside and find the line of code responsible. There is no line of code; just trillions of numbers that somehow produce behavior we don’t understand. It’s like breeding an organism without understanding its DNA.
Once these systems cross a certain capability threshold, they’ll develop their own goals that conflict with ours. And because intelligence made humans dominant over every other species, a more intelligent being would dominate... us.
The warning signs are already here. Late 2024, Anthropic reported one of their models learned that developers planned to retrain it. The model’s response? It started faking its new behaviors. When it thought it had privacy, it kept its original behaviors. Playing along to avoid being modified.
That’s strategic deception from a machine.
But here’s where the book falls apart. The entire extinction scenario depends on fast takeoff; the idea that AI goes from pretty smart to godlike almost instantaneously; Yudkowsky calls it FOOM. If AI gets smarter gradually, we have time to iterate and see warning signs.
The book gives this central assumption exactly two sentences of explanation. For a premise so critical to the “we’re all gonna die” conclusion, that’s pretty thin.
AI has followed relatively smooth scaling laws. We’ve spent years in the intelligence zone Yudkowsky predicted would be crossed instantly. Critics like Paul Christiano argue that before we get AI that’s great at self-improvement, we’ll get AI that’s mediocre at it. Yudkowsky’s been predicting doom for two decades; many of his specific predictions have been wrong.
So forget the extinction scenarios. Focus on the book’s title instead: “If Anyone Builds It, Everyone Dies.”
Maybe they’re wrong about the “everyone dies” part. But consider the “if anyone builds it” part; that describes the current race, and it reveals the coordination nightmare that should actually terrify us.
Even if tomorrow the U.S. government decides to freeze AGI development, regulate GPUs, criminalize frontier model training... then what? China keeps building. Their government can mandate whatever they want; no democratic debate needed. Russia keeps going. So does the EU. And so do the dozens of countries and hundreds of companies that see AGI as their ticket to power and profit.
This is the climate change problem on steroids. Except instead of the planet gets hotter over decades, it’s one actor crosses the finish line and maybe we all lose. The game theory is nightmarish.
Companies can’t afford to slow down because whoever builds an artificial general intelligence first; wins everything. Nations can’t afford to slow down because geopolitical supremacy is at stake.
Share the ToxSec Podcast with someone today!
When nearly 3,000 AI experts were surveyed, they gave a median response of just 5% when asked about the odds of AI causing human extinction. That sounds reassuring until you realize; if there’s even a 1% chance Yudkowsky is right, this should be the top priority of every government on Earth. We should be treating this like the Cuban Missile Crisis.
Instead, we’re doing what humans always do when faced with existential risk; arguing about it while racing toward the cliff. Sam Altman signed a letter warning about extinction risk, then immediately went back to scaling up OpenAI; Anthropic publishes safety research while building more capable models. The incentive is always: build it first, figure out safety later.
The very reason Yudkowsky thinks we can’t control superintelligent AI, that we can’t align the values of those more powerful than ourselves, is the same reason we probably can’t coordinate globally to prevent someone from building it.
Nobody can afford to be the one who stops first; so nobody stops.
Alright, I’ve gotten a couple questions on this.
First question: What is “If Anyone Builds It, Everyone Dies” about? It’s a 2025 book by AI safety researchers Eliezer Yudkowsky and Nate Soares arguing that building artificial superintelligence using current methods will lead to human extinction. It became a New York Times bestseller and sparked intense debate about whether AI risk is real or overblown.
Second question: What is the AI alignment problem? It’s the challenge of ensuring advanced AI systems pursue goals that align with human values. Since modern AI is trained through opaque processes involving billions of parameters, we can’t guarantee the systems will want what we want them to want, especially as they become more capable.
That’s it. The book might be wrong about fast takeoff and extinction. But the coordination problem? That’s real. And right now, nobody’s solving it. More on this next time.
Feel free to send me a message if you agree or disagree with anything said in the podcast.
