An urgent warning from two artificial intelligence insiders on the reckless scramble to build superhuman AI—and how it will end humanity unless we change course. In 2023, hundreds of machine-learning scientists signed an open letter warning about our risk of extinction from smarter-than-human AI. Yet today, the race to develop superhuman AI is only accelerating, as many tech CEOs throw caution to the wind, aggressively scaling up systems they don't understand—and won’t be able to restrain. There is a good chance that they will succeed in building an artificial superintelligence on a timescale of years or decades. And no one is prepared for what will happen next.
For over 20 years, two signatories of that letter—Eliezer Yudkowsky and Nate Soares— have been studying the potential of AI and warning about its consequences. As Yudkowsky and Soares argue, sufficiently intelligent AIs will develop persistent goals of their own: bleak goals that are only tangentially related to what the AI was trained for; lifeless goals that are at odds with our own survival. Worse yet, in the case of a near-inevitable conflict between humans and AI, superintelligences will be able to trivially crush us, as easily as modern algorithms crush the world’s best humans at chess, without allowing the conflict to be close or even especially interesting.
How could an AI kill every human alive, when it’s just a disembodied intelligence trapped in a computer? Yudkowsky and Soares walk through both argument and vivid extinction scenarios and, in so doing, leave no doubt that humanity is not ready to face this challenge—ultimately showing that, on our current path, If Anyone Builds It, Everyone Dies.
The scramble to create superhuman AI has put us on the path to extinction—but it’s not too late to change course, as two of the field’s earliest researchers explain in this clarion call for humanity.
"May prove to be the most important book of our time.”—Tim Urban, Wait But Why
In 2023, hundreds of AI luminaries signed an open letter warning that artificial intelligence poses a serious risk of human extinction. Since then, the AI race has only intensified. Companies and countries are rushing to build machines that will be smarter than any person. And the world is devastatingly unprepared for what would come next.
For decades, two signatories of that letter—Eliezer Yudkowsky and Nate Soares—have studied how smarter-than-human intelligences will think, behave, and pursue their objectives. Their research says that sufficiently smart AIs will develop goals of their own that put them in conflict with us—and that if it comes to conflict, an artificial superintelligence would crush us. The contest wouldn’t even be close.
How could a machine superintelligence wipe out our entire species? Why would it want to? Would it want anything at all? In this urgent book, Yudkowsky and Soares walk through the theory and the evidence, present one possible extinction scenario, and explain what it would take for humanity to survive.
The world is racing to build something truly new under the sun. And if anyone builds it, everyone dies.
“The best no-nonsense, simple explanation of the AI risk problem I've ever read.”—Yishan Wong, Former CEO of Reddit
