Unfortunately, I think the No Derivatives clause would probably disallow recording on audiobook (but I'm not 100% sure). 

This is crazy!! I just read your Nature article last month!! This charity is so exciting to me!!

Let me ask two really simple questions (hopefully not too simple):

1) If you had to choose just one thing, what is the single most important thing that would help volcanologists predict when/if a catastrophic volcanic eruption is going to occur?

2) If volcanologists knew a catastrophic eruption was likely, what is the most promising idea — even if speculative or unlikely to work — for how to prevent or reduce an eruption? (E.g. siphoning off the magma to use for geothermal energy??) 

Mitigating risk from large volcanic eruptions seems important, tractable, and neglected, so I wish you the best of luck!!
 

Do you have any response to the arguments made in the post? I would be curious to hear if you have any interesting counterarguments. 

As for the rest, I think it's been addressed at sufficient length already.

I find this comment fairly confusing, so I'm going to try to hopefully clear up some of the confusion.

Here’s why I don’t think “start time for LeCun’s research program is 2022” is true in any sense relevant to this conversation.

Was the intention of the comment I made about Yann LeCun's and Richard Sutton's research roadmaps unclear? It has nothing to do with the question of how far in advance we should start preparing for AGI. I was just giving a different point of comparison than your example of the progress in LLMs from 2018 to 2025. These were examples of how two successful AI researchers think about the amount of time between formulating the fundamental concepts — or at least the fundamental research directions — necessary to build AGI in a paper and actually building AGI. How much in advance of AGI you'd want to prepare is a separate question.

Similarly, I don't think your example of the amount of progress in LLMs from 2018 to 2025 was intended to make an argument about how long in advance of AGI to start preparing, was it? I thought you were simply trying to argue that the time between a novel AI paradigm being conceptualized and AGI being created could indeed be 7 years, contrary to what I asserted in the conclusion to my post.

Am I misunderstanding something? This response doesn't seem to be a response of what I was trying to say in the comment it's responding to. Am I missing the point?

IIUC, the subtext of your OP and this whole conversation is that you think people shouldn’t be urgently trying to prepare for AGI / ASI right now.

The topic of how much in advance we should be preparing for AGI and what, specifically, we should be doing to prepare is, of course, related to the topic of when we think AGI is likely to happen, but someone could make the argument that it's important to start preparing for AGI now even if it's 50 or 100 years away. The correctness or incorrectness of that argument wouldn't depend on whether AGI by 2032 is extremely unlikely. My post is about whether AGI by 2032 is extremely unlikely and isn't intended to comment on the question of how far in advance of AGI we should prepare, or what we should do to prepare.

If we really should be preparing for AGI 50 or 100 years in advance, then whether I think we should start preparing for AGI now really doesn't depend on whether I think AGI is likely within 7 years.

I think it’s nuts that we weren’t doing more work on AGI x-risk in 2015, and 2005, and 1995 etc.

If you think there is a strong argument for doing work on AGI safety or alignment 35+ years in advance of when AGI is expected to be created, then you can make that argument without arguing that AGI is likely to be created within 7 years, so that argument could be correct even if my thesis is correct that AGI by 2032 is extremely unlikely. Forgive me if I'm repeating myself here.

You seem to be acting as if (A) is less than 7 years, but you haven’t justified that, and I don’t think you can.

I didn't say anything about that in the post. As I said just above, if it's true, as you say, that we should start preparing for AGI long before we think it's likely to arrive, then this wouldn't be a logical inference from what I've argued.

I am concerned that what you’re actually thinking is more like: “AGI doesn’t feel imminent, therefore (B)<(A)”.

Is "feel" supposed to be pejorative here? Is "AGI doesn't feel imminent" supposed to mean something other than "I don't think AGI is imminent"? Are your opinions about AGI timelines also something you "feel"?

Does the clock start in 2022 when LeCun published A Path Towards Autonomous Machine Intelligence (APTAMI)?

Are you asking me whether I think Yann LeCun has published the roadmap that will, in fact, lead to AGI? I brought up LeCun's roadmap as an example. I brought up Richard Sutton's Alberta Plan as another example. As far as I can tell, these are mutually incompatible roadmaps to AGI. They could also both be wrong. But I just brought these up as examples. I wasn't saying one of them will actually lead to the invention of AGI. 

...if your sense of urgency will be triggered by obvious signals of impressiveness like using language and solving problems beyond current LLMs. If you have some other trigger that you’re looking for, what is it?

In the post, I mentioned a few different broad areas where I think current AI systems do poorly and used this as evidence to argue that AGI is unlikely within 7 years. It would stand to reason, therefore, that I think if AI systems started significantly improving in these areas, it would be a reason for me to believe AGI is closer than I currently think it is. 

I would at least be curious to know what you think about the reasons I gave in the post, even if I disagree.

(If you think we should be urgently preparing for AGI / ASI x-risk right now, despite AGI being extremely unlikely by 2032, then great, we would be in much more agreement than I assumed. If that’s the situation, then I think your post does not convey that mood, and I think that almost all readers will interpret it as having that subtext unless you explicitly say otherwise.)

How far in advance of AGI we should start preparing for it is logically independent from the thesis of this post — which is about the likelihood of near-term AGI — and I didn't say anything in this post about whether we should start preparing now or not. I would prefer to discuss that in the context of a post that does make an argument about how far in advance we should start preparing (and, if so, what kind of preparation would be useful or even possible). 

That topic depends on a lot of things other than AGI timelines, e.g., hard takeoff vs. soft takeoff, the "MIRI worldview" on AI alignment vs. other views, and the scientific/technological paradigm used to build AGI. 

I made this post because I had certain ideas I wanted to talk about that I wanted to hear what people thought about. If you have thoughts about what I said in this post, I would be curious to hear them. If I'm wrong about what I said in the post, why am I wrong? Tell me!

I am desperate to hear good counterarguments. 

I don't really have a gripe with people who want to put relatively small probabilities on near-term AGI, like the superforecasters who guessed there's a 1% chance of AGI by 2030. Who knows anything about anything? Maybe Jill Stein has a 1% chance of winning in 2028! But 50% by 2032 is definitely way too high and I actually don't think there's a rational basis for thinking that.

To me, that quote really sounds like it's about code in general, not code at Anthropic. 

Dario's own interpretation of the prediction, even now that it's come false, seems to be about code in general, based on this defense:

I made this prediction that, you know, in six months, 90% of code would be written by AI models. Some people think that prediction is wrong, but within Anthropic and within a number of companies that we work with, that is absolutely true now.

If the prediction was just about Anthropic's code, you'd think he would just say:

I made this prediction that in six months 90% of Anthropic's code would be written by AI and now within Anthropic that is absolutely true now.

What he actually said comes across as a defense of a prediction he knows was at least partially falsified or is at least in doubt. If he just meant 90% of Anthropic's code would be written by AI, he could just say he was unambiguously right and there's no doubt about it.

Edit:

To address the part of your comment that changed after you edited it, in my interpretation, "we are finding" just means "we are learning" or "we are gaining information that" and is general enough that it doesn't by itself support any particular interpretation. For example, he could have said:

...what we are finding is we are not far from the world—I think we'll be there in three to six months—where AI is writing 90 percent of grant applications.

I wouldn't interpret this to mean that Anthropic is writing any grant applications at all. My interpretation wouldn't be different with or without the "what we are finding" part. If he just said, "I think we are not far from the world...", to me, that would mean exactly the same thing.

I fear we have yet to truly refute Robin Hanson’s claim that EA is primarily a youth movement.

Wow. This is my first time reading that Robin Hanson blog post from 2015. 

When I was around 18 to 20 or 21, I was swept up in political radicalism, and then I became a pretty strong skeptic of political radicalism afterward — although it always bears mentioning that such things are too complex to cram into an either/or binary and the only way to do them justice is try to sort the good from the bad.

I think largely because of this experience I was pretty skeptical of radicalism in EA when I got involved with my university EA group from around age 23 to 25 or 26. I don't like it when ideas become hungry and try to take over everything. Going from a view on charity effectiveness and our moral obligation to donate 10% of our income to charity to a worldview that encompassed more and more and more was never a move I supported or felt comfortable with.^[1]

It has always seemed to me that the more EA tried to stretch beyond its original scope of charity effectiveness and an obligation to give, which Peter Singer articulated in The Life You Can Save in 2009,^[2] the more it was either endorsing dubious, poorly-supported conclusions or trying to reinvent the wheel from first principles for no particularly good reason. 

I think this paragraph from Hanson's blog post is devastatingly accurate:

Some observers see effective altruism as being about using formal statistics or applying consensus scientific theories. But in fact effective altruists embrace contrarian concerns about AI “foom” (discussed often on this blog), concerns based neither on formal statistics nor on applying consensus theories. Instead this community just trusts its own judgment on what reasoning is “careful,” without worrying much if outsiders disagree. This community has a strong overlap with a “rationalist” community wherein people take classes on and much discuss how to be “rational”, and then decide that they have achieved enough rationality to justify embracing many quite contrarian conclusions.

If you think that effective altruism has discovered or invented radically novel and radically superior general-purpose principles for how to think, live, be rational, or be moral, I'm sorry, but that's ludicrous. EA is a mish-mash of ideas from analytic moral philosophy, international development, public health, a bit of economics and finance, and a bit of a few other things. That's all. 

I think the trajectory that is healthy is when people who have strong conviction in EA start with a radical critique of the status quo (e.g. a lot of things like cancer research or art or politics or volunteering with lonely seniors seem a lot less effective than GiveWell charities or the like, so we should scorn them), then see the rationales for the status quo (e.g. ultimately, society would start to fall apart if tried to divert too many resources to GiveWell charities and the like by taking them away from everything else), and then come full circle back around to some less radical position (e.g. as many people as possible should donate 10-20% of their income to effective charities, and some people should try to work directly in high-priority cause areas). 

This healthy trajectory is what I thought of when Hanson said that youth movements eventually "moderate their positions" and "become willing to compromise".

I think the trajectory that is unhealthy is when people repudiate the status quo in some overall sense, seemingly often at least partially because it fills certain emotional needs to make the world other than oneself and to condemn its wicked ways. 

Many (though not all) effective altruists seem content to accept the consensus view on most topics, to more or less trust people in general, to trust most mainstream institutions like academia, journalism, and the civil service (of liberal democratic countries), and they don't particularly seek out being contrarian or radical or to reject the world. 

On the other hand, this impulse to reject the world and be other than it is probably the central impulse that characterizes LessWrong and the rationalist community. EA/rationalist blogosphere writer Ozy Brennan wrote an insightful blog post about rationalists and the "cultic milieu", a concept from sociology that refers to new religious movements rather than the high-control groups we typically think of when we think of "cults". (Read the post if you want more context.) They wrote:

People become rationalists because they are attracted to the cultic milieu—that is, people who distrust authority and want to figure things out for themselves and like knowing secrets that no one else knows. People who are attracted to the cultic milieu are attracted to stigmatized knowledge whether or not it is in fact correct.

In a similar vein, the EA Forum member Maniano wrote a post where they conveyed their impression of EAs and rationalists (abbreviating "rationalists" to "rats", as is not uncommon for rationalists to do):

If I’d have to vaguely point to a specific difference in the vibes of an EAs and those of rats, I would say EAs feel more innocent whereas rats might, with possibly a little bit too much generalization, feel like they’d rank higher in some dark triad traits and feature more of chuunibyou tendencies sprinkled with a dash of narrative addiction.

I don't know for sure what "narrative addiction" means, but I suspect what the author meant is something similar to the sort of psychological tendencies Ozy Brennan described in the post about the cultic milieu. Namely, the same sort of tendency often seen among people who buy into conspiracy theories or the paranoid style in politics to think about the world narratively rather than causally, to favour narratively compelling accounts of events (especially those containing intrigue, secrets, betrayal, and danger) rather than awkward, clunky, uncertain, confusing, and narratively unsatisfying accounts of events.

From the linked Wikipedia article:

Chūnibyō (中二病; lit. 'middle-school second-year syndrome') is a Japanese colloquial term typically used to describe adolescents with delusions of grandeur. These teenagers are thought to desperately want to stand out and convince themselves that they have hidden knowledge or secret powers. It is sometimes called "eighth-grader syndrome" in the United States, usually in the context of localizations of anime which feature the concept as a significant plot element.

I think seeing oneself as other than the wicked world is not a tendency that is inherent to effective altruism or a necessary part of the package. But it is a fundamental part of rationalism. Similarly, EA can be kept safely in one corner of your life, even as some people might try to convince you it needs to eat more of your life. But it seems like the whole idea of rationalism is that it takes over. The whole idea is that it's a radical new way to think, live, be rational, and be moral and/or successful.

I wonder if the kind of boredom you described, Michael, that might eventually set in from a simpler The Life You Can Save-style effective altruism is part of what has motivated people to seek a more expansive (and eventually maybe even totalizing) version of effective altruism — because that bigger version is more exciting (even if it's wrong, and even if it's wrong and harmful).

Personally, I would love to be involved in a version of effective altruism that felt more like a wholesome, warm, inclusive liberal church with an emphasis on community, social ties, and participation. (Come to think of it, one of the main people at the university EA group I was involved in said he learned how to be warm and welcoming to people through church. And he was good at it!) I am not really interested in the postmodernist cyberpunk novel version of effective altruism, which is cold, mean, and unhappy.

1. ^{^}

   I think we should be willing to entertain radical ideas but have a very high bar for accepting them, noting that many ideas considered foundational today were once radical, but also noting that most radical ideas are wrong and some can lead to dangerous or harmful consequences. 
   
   Another thing to consider is how hungry these ideas are, as I mentioned. Some radical ideas have a limited scope of application. For example, polyamory is a radical idea for romantic relationships, but it only affects your romantic relationships. Polyamory doesn't tell you to quit your current job and find a new job where you convince monogamous people to become polyamorous. Or provide services to people who are already polyamorous. Polyamory doesn't tell you to have any particular opinions about politics — besides maybe narrow things like rights (e.g. hospital visitation rights) for people in polyamorous relationships — or technology or culture or the fate of the world. 
   
   When radical ideas become totalizing and want to be the axis around which the world turns, that's when I start to feel alarmed.

2. ^{^}

   The Life You Can Save is an example of a radical idea — one I think we should accept — that, similar to polyamory, may affect our lives in a significant way, but is naturally limited in scope. The Life You Can Save is an expression of a simple and straightforward version of effective altruism. As people have wanted the scope of effective altruism to get larger and larger over time, that has led to the accretion of a more complicated and eclectic version of effective altruism that I think is a lot more dubious.

I'll have to look at that safety report later and see what the responses are to it. At a glance, this seems to be a bigger and more rigorous disclosure than what I've seen previously and Waymo has taken the extra step of publishing in a journal. 

[Edit, added on October 20, 2025 at 12:40pm Eastern: There are probably going to be limitations with any safety data and we shouldn't expect perfection, nor should that get in the way of us lauding companies for being more open with their safety data. However, just one thing to think about: if autonomous vehicles are geofenced to safer areas but they're being compared to humans driving in all areas, ranging from the safest to the most dangerous, then this isn't a strict apples-to-apples comparison.]

However, I'm not ready to jump to any conclusions just yet because it was a similar report by Waymo (not published in a journal, however) that I paid someone with a PhD in a relevant field to help me analyze and, despite Waymo's report initially looking promising and interesting to me, that person's conclusion was that there was not enough data to actually make a determination one way or the other whether Waymo's autonomous vehicles were actually safer than the average human driver.

I was coming at that report from the perspective of wanting it to show that Waymo's vehicles were safer than human drivers (although I didn't tell the person with the PhD that because I didn't want to bias them). I was disappointed that the result was inconclusive. 

If it turns out Waymo's autonomous vehicles are indeed safer than the average human driver, I would celebrate that. Sadly, however, it would not really make me feel more than marginally more optimistic about the near-term prospects of autonomous vehicle technology for widespread commercialization.

The bigger problem for this overall argument about autonomous vehicles (that they show data efficiency or the ability to deal with novelty isn't important) is that safety is only one component of competence (as I said, a parked car is 100% safe) and autonomous vehicles are not as competent as human drivers overall. If they were, there would be a huge commercial opportunity in automating human driving in a widespread fashion — by some estimations, possibly the largest commercial opportunity in the history of capitalism. The reason this can't be done is not regulatory or social or anything like that. It's because the technology simply can't do the job. 

The technology as it's deployed today is not only helped along by geofencing, it's also supported by a high ratio of human labour to the amount of autonomous driving. That's not only safety drivers in the car or remote monitors and operators, but also engineers doing a lot of special casing for specific driving environments. 

If you want to use autonomous vehicles as an example of AI automating significant human labour, first they would have to automate significant human labour — practically, not just in theory — but that hasn't happened yet.

Moreover, driving should, at least in theory, be a low bar. Driving is considered to be routine, boring, repetitive, not particularly complex — exactly the sort of thing we would think should be easier to automate. So, if approaches to AI that have low data efficiency and don't deal well with novelty can't even handle driving, then it stands to reason that more complex forms of human labour such as science, philosophy, journalism, politics, economics, management, social work, and so on would be even less susceptible to automation by these approaches.

Just to be clear on this point: if we had a form of AI that could drive cars, load dishwashers, and work an assembly line but not do those other things (like science, etc.), I think that would be wonderful and it would certainly be economically transformative, but it wouldn't be AGI.

Edited to add on October 20, 2025 at 12:30pm Eastern:

Don’t take my word for it. Andrej Karpathy, an AI researcher formerly at OpenAI who led Tesla’s autonomous driving AI from 2017 to 2022, recently said on a podcast that he doesn’t think fully autonomous driving is nearly solved yet:

…self-driving cars are nowhere near done still. The deployments are pretty minimal. Even Waymo and so on has very few cars. … Also, when you look at these cars and there’s no one driving, I actually think it’s a little bit deceiving because there are very elaborate teleoperation centers of people kind of in a loop with these cars. I don’t have the full extent of it, but there’s more human-in-the-loop than you might expect. There are people somewhere out there beaming in from the sky. I don’t know if they’re fully in the loop with the driving. Some of the time they are, but they’re certainly involved and there are people. In some sense, we haven’t actually removed the person, we’ve moved them to somewhere where you can’t see them.

Things like Docker containers or cloud VMs that can be, in principle, applied to any sort of software or computation could be helpful for all sorts of applications we can't anticipate. They are very general-purpose. That makes sense to me.

The extent to which things designed for deep learning, such as PyTorch, could be applied to ideas outside deep learning seems much more dubious. 

And if we're thinking about ideas that fall within deep learning, but outside what is currently mainstream and popular, then I simply don't know.