This comment deserves some kind of EA Forum award. My goodness, I envy and admire the breezy style with which you wrote this. I wish we had some analogue of Reddit gold, an award in limited supply, we could use to recognize exceptional contributions like this.

I agree with David Mathers that it's simply psychologically implausible that David Thorstad, a sharp professional philosopher and an expert on bounded rationality, existential risk, longtermism, and effective altruism, doesn't understand the concept of expected value. I think we need to jettison such accusations, which have more of personal insult about them than substantive argument. Such accusations, besides just being ridiculous on their face, are corrosive to productive, charitable discussion about substantive disagreements on important topics.

I typically don’t agree with much that Dwarkesh Patel, a popular podcaster, says about AI,^[1] but his recent Substack post makes several incisive points, such as:

Somehow this automated researcher is going to figure out the algorithm for AGI - a problem humans have been banging their head against for the better part of a century - while not having the basic learning capabilities that children have? I find this super implausible.

Yes, exactly. The idea of a non-AGI AI researcher inventing AGI is a skyhook. It’s pulling yourself up by your bootstraps, a borderline supernatural idea. It’s retrocausal. It just doesn’t make sense.

There are more great points in the post besides that, such as:

Currently the labs are trying to bake in a bunch of skills into these models through “mid-training” - there’s an entire supply chain of companies building RL environments which teach the model how to navigate a web browser or use Excel to write financial models.

Either these models will soon learn on the job in a self directed way - making all this pre-baking pointless - or they won’t - which means AGI is not imminent. Humans don’t have to go through a special training phase where they need to rehearse every single piece of software they might ever need to use.

… You don’t need to pre-bake the consultant’s skills at crafting Powerpoint slides in order to automate Ilya [Sutskever, an AI researcher]. So clearly the labs’ actions hint at a world view where these models will continue to fare poorly at generalizing and on-the-job learning, thus making it necessary to build in the skills that they hope will be economically valuable.

And:

It is not possible to automate even a single job by just baking in some predefined set of skills, let alone all the jobs.

We are in an AI bubble, and AGI hype is totally misguided.

1. ^{^}

   There are some important things I disagree with in Dwarkesh's post, too. For example, he says that AI has solved "general understanding, few shot learning, reasoning", but AI has absolutely not solved any of those things. 

   Models lack general understanding, and the best way to see that is they can't do much useful in complex, real world contexts — which is one of the points Dwarkesh is making in the post. Few-shot learning only works well in situations where a model has already been trained on a giant amount of similar training examples. The "reasoning" in "reasoning models" is, in Melanie Mitchell's terminology, a wishful mnemonic. In other words, just naming an AI system something doesn't mean it can actually do the thing it's named after. If Meta renamed Llama 5 to Superintelligence 1, that wouldn't make Llama 5 a superintelligence.  
   
   I also think Dwarkesh is astronomically too optimistic about how economically impactful AI will by 2030. And he's overfocusing on continual learning as the only research problem that needs to be solved, to the neglect of others.

Good question. I’m less familiar with the self-driving car industry in China, but my understanding is that the story there has been the same as in the United States. Lots of hype, lots of demos, lots of big promises and goals, very little success. I don’t think plans count for anything at this point, since there’s been around 6-10 years of companies making ambitious plans that never materialized.

Regulation is not the barrier. The reason why self-driving cars aren’t a solved problem and aren’t close to being a solved problem is that current AI techniques aren’t up to the task; there are open problems in fundamental AI research that would need to be solved for self-driving to be solved. If governments can accelerate progress, it’s in funding fundamental AI research, not in making the rules on the road more lenient.

Seeing the amount of private capital wasted on generative AI has been painful. (OpenAI alone has raised about $80 billion and the total, global, cumulative investment in generative AI seems like it’s into the hundreds of billions.) It’s made me wonder what could have been accomplished if that money had been spent on fundamental AI research instead. Maybe instead of being wasted and possibly even nudging the U.S. slightly toward a recession (along with tariffs and all the rest), we would have gotten the kind of fundamental research progress needed for useful AI robots like self-driving cars.

To quote you from another thread:

Going back to the OP's claims about what is or isn't "a good way to argue," I think it's important to pay attention to the actual text of what someone wrote. That's what my blog post did, and it's annoying to be subject to criticism (and now downvoting) from people who aren't willing to extend the same basic courtesy to me.

You misunderstood my argument based on a misreading of the text. Simple as that.

Please extend the same courtesy to others that you request for yourself. Otherwise, it's just 'rules for thee but for not me'.

@Richard Y Chappell🔸, would you please do me the courtesy of acknowledging that you misunderstood my argument? I think this was a rather uncharitable reading on your part and would have been fairly easy to avoid. Your misreading was not explicitly forestalled by the text but not supported by the text, either, and there was much in the text to suggest I did not hold the view that you took to be the thesis or argument. I found your misreading discourteous for that reason. 

Much of the post is focused on bad intellectual practices, such as:

• Not admitting you got a prediction wrong after you got it wrong
• Repeating the same prediction multiple times in a row and repeatedly getting it wrong, and seemingly not learning anything
• Making fake graphs with false data, no data, dubious units of measurement, no units of measurements, and other problems or inaccuracies
• Psychological or social psychological biases like millennialist cognitive bias, bias resulting from the intellectual and social insularity of the EA community, and possible confirmation bias (e.g. why hasn't Toby Ord's RL scaling post gotten much more attention?)
• Acceptance or tolerance of arguments and assertions that are really weak, unsupported, or sometimes just bad

I don't interpret your comment as a defense or endorsement of any of these practices (although I could if I wanted to be combative and discourteous). I'm assuming you don't endorse these practices and your comment was not intended as a defense of them.

So, why reply to a post that is largely focused on those things as if the thesis or argument or thrust of the post is something other than that, and which was not said in the text? 

On the somewhat more narrow point of AI capabilities optimism, I think the AI bubble popping within the next 5 years or so would be strong evidence that the EA community's AI capabilities optimism has been misplaced. If the large majority of people in the EA community only thought there's a 0.1% chance or a 1% chance of AGI within a decade, then the AI bubble popping might not be that surprising from their point of view. But the actual majority view seems to be more like a 50%+ chance of AGI within a decade. My impression from discussions with various people in the EA community is that many of them would find it surprising if the AI bubble popped.

The difference between a 50%+ chance of AGI within a decade and a 0.1% chance is a lot from an epistemic perspective, even if, just for the sake of argument, it makes absolutely no difference for precautionary arguments about AI safety. So, I think misestimating the probability by that much would be worthy of discussion, even if — for the sake of argument — it doesn't change the underlying case for AI safety.

It is especially worthy of discussion if the misestimation is influenced by bad intellectual practices, such as those listed above. All the information needed to diagnose those intellectual practices as bad is available today, so the AI bubble popping isn't necessary. However, people in the EA community may be reluctant to give a hard look at them without some big external event like an AI bubble popping shaking them up. As I said in the post, I'm pessimistic that even after the AI bubble pops, people in the EA community will, even then, be willing to examine these intellectual practices and acknowledge that they're bad. But it's worth a shot for me to say something about it anyway. 

There are many practical reasons to worry about bad intellectual practices. For example, people in AI safety should worry about whether they're making existential risk from AGI better or worse, and having bad intellectual practices on a systemic or widespread level will make it more likely they'll screw this up. Or, given that, according to Denkenberger in another comment on this post, funding around existential risk from AGI has significantly taken away funding around other existential risks, overestimating existential risk from AGI based on bad intellectual practices might (counterfactually) increase total existential risk just by causing funding to be less wisely allocated. And, of course, there are many other reasons to worry about bad intellectual practices, especially if they are prevalent in a community and culturally supported by that community. 

We both could list reasons on and on why thinking badly might lead to doing badly. Just one more example I'll bring up is that, in practice, most AI safety work seems to make rather definite, specific assumptions about the underlying technical nature of AGI. If AI safety has (by and large) identified an implausible AI paradigm to underlie AGI out of at least several far more plausible and widely-known candidates (largely as a result of the bad intellectual practices listed above), then AI safety will be far less effective at achieving its goals. There might still be a strong precautionary argument for doing AI safety work on even that implausible AI paradigm, but given that AI safety, has, in practice, for the most part, bet on one specific horse and not the others, it is a problem to pick the wrong paradigm. You could maybe argue for an allocation of resources weighted to different AI paradigms based on their perceived plausibility, but that would still result in a large reallocation of resources if the paradigm AI safety is betting on is highly implausible and there are several other candidates that are much more plausible. So, I think this is a fair line of argument.

What matters is not just some unidimensional measure of the EA community’s beliefs like the median year of AGI or the probability of AGI within a certain timeframe or the probability of global catastrophe from AGI (conditional on its creation, or within a certain timeframe). If bad intellectual practices make that number go up too high, it's not necessarily just fine on precautionary grounds, it can mean existential risk is increased. 

That’s a good and interesting point about environmentalism. I took an environmental philosophy class sometime in the early-to-mid-2010s and very long-term thinking was definitely part of the conversation. As in, thinking many centuries, millennia, or even millions of years in the future. One paper (published in 2010) we read imagined humans in the fourth millennium (i.e. from the year 3000 to 4000) living in "civilization reserves", the inverse of wilderness reverses.

My problem with interventions like improving institutional decision-making is that we are already maximally motivated to do this based on neartermist concerns. Everyone wants governments and other powerful institutions to do a better a job making decisions, to do as good a job as possible.

Let’s say you are alarmed about the Trump administration’s illiberalism or creeping authoritarianism in the United States. Does thinking about the future in 1,000 or 10,000 years actually motivate you to care about this more, to do more about it, to try harder? I don’t see how it would. Even if it did make you care a little bit more about it inside yourself, I don’t see how it would make a practical difference to what you do about it.

And taking such a long-term perspective might bring to mind all the nations and empires that have risen and fallen over the ages, and wonder if what happens this decade or the next might fade away just as easily. So, the effect on how much you care might be neutral, or it might make you care a little less. I don’t know — it depends on subjective gut intuition and each individual’s personal perspective.

Also, something like improving governments or institutions is a relay race where the baton is passed between generations, each of which makes its own contribution and has its own impact. Deflecting a big asteroid heading toward Earth is a way for a single organization like NASA to have a direct impact on the far future. But there are very few interventions of that kind. The clearest cases are existential risks or global catastrophic risks originating from natural sources, such as asteroids and pandemics. Every step you take to widen the circle of interventions you consider introduces more irreducible uncertainty and fundamental unpredictability.

I think asteroids and anti-asteroid interventions like NASA’s NEO Surveyor should be a global priority for governments and space agencies (and anyone else who can help). The total cost of solving like 95% of the problem (or whatever it is) is in the ballpark of the cost of building a bridge. I think people look at the asteroid example and think 'ah, there must be a hundred more examples of things just like that'. But in reality it’s a very short list, something like: asteroids, pandemics, nuclear weapons, bioterror, climate change, and large volcanoes. And each of these varies a lot in terms of how neglected they are.

So, I think longtermism is an instance of taking a good idea — protect the world from asteroids for the price of building a bridge, and maybe a half a dozen other things like that such as launch a satellite to observe volcanoes — and running with it way too far. I don’t think there is enough meat on this bone to constitute a worldview or a life philosophy that can be generally embraced (although hat’s off to the few who make keeping the world safe from asteroids or big volcanoes). Which overall is the mistake of effective altruism over the last decade: take one good idea or a few — like donating a lot of money to cost-effective global health charities — and try to turn it into an all-encompassing worldview or life philosophy. People are hungry for meaning in their lives, I get it, I am too, but there are healthier and unhealthier ways to pursue that, ways that are more constructive and more destructive.