Effective Altruism Forum
EA Forum

by Yarrow Bouchard 🔸
4 min read 0

6

AI safetyExistential riskAI risk skepticismAtomically precise manufacturingFanaticismHumorOpinionPascal's mugging
Frontpage
Show all topics

"Gotta go fast." ―Sonic[1]

A few prominent transhumanists have argued that building friendly AI as quickly as we can may be our best chance to prevent a "grey goo" catastrophe, in which self-replicating nanobots kill everyone on Earth. In 1999, Eliezer Yudkowsky forecasted that a nanotech catastrophe would occur sometime between 2003 and 2015. He forecasted a 70%+ chance of human extinction from nanotechnology, and advocated rushing to build friendly AI (in current terminology, aligned AGI or safe AGI) in order to prevent the end of all human life.[2] At this time, Yudkowsky had begun work on his idea for building friendly AI, which he called Elisson. Yudkowsky wrote:

If we don't get some kind of transhuman intelligence around *real soon*, we're dead meat. Remember, from an altruistic perspective, I don't care whether the Singularity is now or in ten thousand years - the reason I'm in a rush has nothing whatsoever to do with the meaning of life. I'm sure that humanity will create a Singularity of one kind or another, if it survives. But the longer it takes to get to the Singularity, the higher the chance of humanity wiping itself out.

My current estimate, as of right now, is that humanity has no more than a 30% chance of making it, probably less. The most realistic estimate for a seed AI transcendence is 2020; nanowar, before 2015. The most optimistic estimate for project Elisson would be 2006; the earliest nanowar, 2003.

So we have a chance, but do you see why I'm not being picky about what kind of Singularity I'll accept?

Fortunately, in 2000, Yudkowsky forecasted[3] that he and his colleagues at the Singularity Institute for Artificial Intelligence or SIAI (now the Machine Intelligence Research Institute or MIRI) would create friendly AI (again, aligned or safe AGI) in somewhere between five to twenty years, and probably in around eight to ten years:

The Singularity Institute seriously intends to build a true general intelligence, possessed of all the key subsystems of human intelligence, plus design features unique to AI.  We do not hold that all the complex features of the human mind are "emergent", or that intelligence is the result of some simple architectural principle, or that general intelligence will appear if we simply add enough data or computing power.  We are willing to do the work required to duplicate the massive complexity of human intelligence; to explore the functionality and behavior of each system and subsystem until we have a complete blueprint for a mind.  For more about our Artificial Intelligence plans, see the document Coding a Transhuman AI.

Our specific cognitive architecture and development plan forms our basis for answering questions such as "Will transhumans be friendly to humanity?" and "When will the Singularity occur?"  At the Singularity Institute, we believe that the answer to the first question is "Yes" with respect to our proposed AI design - if we didn't believe that, the Singularity Institute would not exist.  Our best guess for the timescale is that our final-stage AI will reach transhumanity sometime between 2005 and 2020, probably around 2008 or 2010.  As always with basic research, this is only a guess, and heavily contingent on funding levels.

Nick Bostrom and Ray Kurzweil have made similar arguments about friendly AI as a defense against "grey goo" nanobots.[4][5] A fictionalized version of this scenario plays out in Kurzweil’s 2010 film The Singularity is Near. Ben Goertzel, who helped popularize the term "artificial general intelligence", has also made an argument along these lines.[6] Goertzel proposes an "AGI Nanny" that could steward humanity through the development of dangerous technologies like nanotechnology.

In 2002, Bostrom wrote:

Some technologies seem to be especially worth promoting because they can help in reducing a broad range of threats. Superintelligence is one of these. Although it has its own dangers (expounded in preceding sections), these are dangers that we will have to face at some point no matter what. But getting superintelligence early is desirable because it would help diminish other risks. A superintelligence could advise us on policy. Superintelligence would make the progress curve for nanotechnology much steeper, thus shortening the period of vulnerability between the development of dangerous nanoreplicators and the deployment of adequate defenses. By contrast, getting nanotechnology before superintelligence would do little to diminish the risks of superintelligence.

The argument that we need to rush the development of friendly AI to save the world from dangerous nanotech may seem far-fetched and certainly some of the details are wrong. Yet consider the precautionary principle, expected value, and the long-term future. If the chance of preventing human extinction and saving 10^52 future lives[7] is even one in ten duodecillion[8] or 1 in 10^40 (in other words, a probability of 10^-40), then the expected value is equivalent to saving 1 trillion lives in the present. GiveWell’s estimate for the cost to save a life is $3,000.[9] So, we should be willing to allocate $3 quadrillion toward rushing to build friendly AI before nanobots wipe out humanity. In fact, since there is some small probability — it doesn’t matter how small — that the number of potential future lives is infinite,[10] we should be willing to spend an infinite amount of money on building friendly AI as quickly as possible.

I believe there’s at least a 1 in 10^48 chance and certainly at least a 1 in ∞ chance that I can create friendly AI in about ten years, or twenty years, tops, on a budget of $1 million a year. I encourage funders to reach out for my direct deposit info. If necessary, I can create a 501(c)(3), but I estimate the expected disvalue of the inconvenience to be the equivalent of between a hundred and infinity human deaths.[11] 

  1. ^

    The hedgehog.

  2. ^

    Extropians: Re: Yudkowsky’s AI (Again). https://diyhpl.us/~bryan/irc/extropians/www.lucifer.com/exi-lists/extropians.1Q99/3561.html.

  3. ^

    “Introduction to the Singularity.” The Singularity Institute for Artificial Intelligence, 17 Oct. 2000, https://web.archive.org/web/20001017124429/http://www.singinst.org/intro.html.

  4. ^

    Bostrom, N. “Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards.” Journal of Evolution and Technology, Publisher's version, vol. 9, Institute for Ethics and Emerging Technologies, 2002. https://nickbostrom.com/existential/risks

  5. ^

    Kurzweil, Ray. “Nanotechnology Dangers and Defenses.” Nanotechnology Perceptions, vol. 2, no. 1a, Mar. 2006, https://nano-ntp.com/index.php/nano/article/view/270/179.

  6. ^

    Goertzel, Ben. “Superintelligence: Fears, Promises and Potentials: Reflections on Bostrom’s Superintelligence, Yudkowsky’s From AI to Zombies, and Weaver and Veitas’s ‘Open-Ended Intelligence.’” Journal of Ethics and Emerging Technologies, vol. 25, no. 2, Dec. 2015, pp. 55–87. DOI.org (Crossref), https://jeet.ieet.org/index.php/home/article/view/48/48.

  7. ^

    Bostrom, Nick. “Existential Risk Prevention as Global Priority.” Global Policy, vol. 4, no. 1, Feb. 2013, pp. 15–31. DOI.org (Crossref), https://existential-risk.com/concept.pdf.

  8. ^

    1 in 10,000,000,000,000,000,000,000,000,000,000,000,000,000.

  9. ^

    How Much Does It Cost to Save a Life? | GiveWell. https://www.givewell.org/how-much-does-it-cost-to-save-a-life.

  10. ^

    Bulldog, Bentham’s. “Philanthropy With Infinite Stakes.” Substack newsletter. Bentham’s Newsletter, 19 Nov. 2025, https://benthams.substack.com/p/philanthropy-with-infinite-stakes.

  11. ^

    I also prefer to receive payment in Monero or via deposit to a Swiss bank account.

    12. Show all footnotes

6

0
0

Reactions

0
0

More posts like this

Comments


No comments on this post yet.
Be the first to respond.
More from Yarrow Bouchard 🔸
56
Self-driving cars aren’t nearly a solved problem
Yarrow Bouchard 🔸
· · 20m read
11
11
35
How I hope the EA community will respond to the AI bubble popping
Yarrow Bouchard 🔸
· · 8m read
72
72
39
A major flaw in the Forecasting Research Institute's "Longitudinal Expert AI Panel" survey
Yarrow Bouchard 🔸
· · 7m read
21
21
View more
Curated and popular this week
 ·  · 8m read
 · 
Crossposted from the Global Priorities Project Work towards reducing existential risk is likely to happen over a timescale of decades. For many parts of this work, the benefits of that labour is greatly affected by when it happens. This has a large effect when it comes to strategic thinking about what to do now in order to best help the overall existential risk reduction effort. I look at the effects of nearsightedness, course setting, self-improvement, growth, and serial depth, showing that there are competing considerations which make some parts of labour particularly valuable earlier, while others are more valuable later on. We can thus improve our overall efforts by encouraging more meta-level work on course setting, self-improvement, and growth over the next decade, with more of a focus on the object-level research on specific risks to come in decades beyond that. Nearsightedness Suppose someone considers AI to be the largest source of existential risk, and so spends a decade working on approaches to make self-improving AI safer. It might later become clear that AI was not the most critical area to worry about, or that this part of AI was not the most critical part, or that this work was going to get done anyway by mainstream AI research, or that working on policy to regulate research on AI was more important than working on AI. In any of these cases she wasted some of the value of her work by doing it now. She couldn't be faulted for lack of omniscience, but she could be faulted for making herself unnecessarily at the mercy of bad luck. She could have achieved more by doing her work later, when she had a better idea of what was the most important thing to do. We are nearsighted with respect to time. The further away in time something is, the harder it is to perceive its shape: its form, its likelihood, the best ways to get purchase on it. This means that work done now on avoiding threats in the far future can be considerably less valuable than the same am
 ·  · 4m read
 · 
This is a rough explanation of relative ranges, a heuristic that I've found very helpful for quickly comparing two options that trade off between two dimensions. Consider the following examples of tradeoffs: 1. Should we prioritize helping small animals or large animals? There are more small animals, but large animals have a higher capacity for suffering. 2. Should we fund medical research on the most promising candidates across diseases, or should we focus only on the most important diseases? Broad search is more likely to lead to a successful treatment, but targeted search can lead to treatments for higher-burden diseases. 3. If we fund a recurring health/consumption survey, should we fund it annually or quarterly? More rounds leads to higher-frequency information, but at a higher cost. You could answer these questions by carefully quantifying the value of each parameter – the actual population of different types of animals, the actual welfare ranges of each species, etc. But I think more often than not, that isn't necessary. The relative range heuristic is: prioritize options based on the dimension that varies across a wider range. In the examples above, that would mean: 1. Larger animals might have more capacity for suffering, but intuitively they might have 10x more or 100x more at most. Meanwhile, small animals are 1000x or 10000x more numerous than large animals. So the scale advantage of small animals is more important, and thus we should prioritize small animals. 2. The best drug candidates across all diseases could have 10x higher chances of success than the best drug candidates for specific diseases, but the highest burden diseases have 1000x higher burden than the average disease. So the burden advantage of the most important diseases is more important, and thus we should prioritize research into those diseases only. 3. Higher-frequency information is valuable, but a quarterly survey is 4x more expensive than an annual survey, and its informati
 ·  · 3m read
 · 
TL;DR * When surveyed, the EA community and leaders think ~18-24% of resources should go towards animal advocacy. The actual figure is about 7%. * We as the EA ecosystem are putting less resources (money and time) into animal advocacy than the movement thinks we should when surveyed. * This disparity could be because of loss of message fidelity, it's a harder cause area to pitch donors, or the role of large funders, but I'm honestly not too sure. My job at Senterra Funders (formerly Farmed Animal Funders) involves making the case to EA/EA adjacent prospective donors that they can do a tonne of good by donating to animal advocacy charities. As part of this work I’ve noticed a certain level of inconsistency in the EA ecosystem: I encounter a lot more people who want the animal advocacy movement to 'win' than people working in or donating to the space. The numbers It turns out this intuition is backed up by survey data. Sources (see Appendix for extra details): * Meta Coordination Forum (MCF; 2024) / Talent Need Survey on ideal allocation of financial resources * EA Community survey data from 2023 on jobs by cause area I obtained in private correspondence with David Moss. * Historical EA funding data: 2025 update on actual EA donations by case area. * Note: I didn’t include the 2023 supplementary materials for the EA Community Survey Cause Prioritization because there wasn’t a category on ‘meta’ which made it too confusing to include without extra work. Having said that, at a glance the numbers look fairly similar to the meta coordination forum survey. Looking at this chart, it’s quite striking that about 2.5 times less money goes towards animal advocacy than both the community (here), and ‘EA leaders’ think should go towards this cause area. There’s also a similar issue with where people are working. Compared to animal advocacy, there are almost 4x as many people working in x-risk, 2x in ‘meta’. Accounting for the disparity What’s going on here? Eit
Recent opportunities in AI safety
11
If researchers shared their #1 idea daily, we'd navigate existential challenges far more effectively
Jordan Arel
· · 2m read
1
1
30
Announcing the CLR Fundamentals Program
Center on Long-Term Risk, Santeri T 🔹, Tristan Cook
· · 2m read
0
0
46
Join the Founding Team: Australia's AI Safety Institute
Luke Freeman 🔸, Emily Grundy, Nathan Sherburn, Greg S
· · 6m read
0
0
View more