The context for what I'm discussing is explained in two Reflective Altruism posts: part 1 here and part 2 here.

Warning: This is a polemic that uses harsh language. I still completely, sincerely mean everything I say here and I consciously endorse it.

It has never stopped shocking and disgusting me that the EA Forum is a place where someone can write a post arguing that Black Africans need Western-funded programs to edit their genomes to increase their intelligence in order to overcome global poverty and can cite overtly racist and white supremacist sources to support this argument (even a source with significant connections to the 1920s, 1930s, and 1940s German Nazi Party and the American Nazi Party, a neo-Nazi party) and that post can receive a significant amount of approval and defense from people in EA, even after the thin disguise over top of the racism is removed by perceptive readers. That is such a bonkers thing and such a morally repugnant thing, I keep struggling to find words to express my exasperation and disbelief. Effective altruism as a movement probably deserves to fail for that, if it can't correct it. 

My loose, general impression is that people who got involved in EA because of global poverty and animal welfare tend to be broadly liberal or centre-left and tend to be at least sympathetic toward arguments about social justice and anti-racism. Conversely, my impression of LessWrong and the online/Bay Area rationalist community is that they don't like social justice, anti-racism, or socially/culturally progressive views. One of the most bewildering things I ever read on LessWrong was one of the site admins (an employee of Lightcone Infrastructure) arguing that closeted gay people probably tend to have low moral integrity because being closeted is a form of deception. I mean, what?! This is the "rationalist" community?? What are you talking about?! As I recall based on votes, a majority of forum users who voted on the comment agreed. 

Overall, LessWrong users seem broadly sympathetic to racist arguments and views. Same for sexist or anti-feminist views, and extremely so for anti-LGBT (especially anti-trans) views. Personally, I find it to be the most unpleasant website I've spent more than ten hours reading. When I think of LessWrong, I picture a dark, dingy corner of a house. I truly find it to be awful. 

The more I've thought about it, the more truth I find in the blogger Ozy Brennan's interpretation the LessWrong and the rationalist community through the concept of the "cultic milieu" and a comparison to new religious movements (not cults in the more usual sense connoting high-control groups). Ozy Brennan self-identifies as a rationalist and is active in the community, which makes this analysis far more believable than if it came from an outsider. The way I'd interpret Ozy's blog post, which Ozy may or may not agree with, is that rationalists are in some sense fundamentally devoted to being incorrect, since they're fundamentally devoted to being against consensus or majority views on many major topics — regardless of whether those views are correct or incorrect — and inevitably that will lead to having a lot of incorrect views. 

I see very loose, very limited analogies between LessWrong and online communities devoted discussing conspiracy theories like QAnon or to online incel communities. Conspiracy theories because LessWrong has a suspicious, distrustful, at least somewhat paranoid or hypervigilant view on people and the world, this impulse to turn over rocks to find where the bad stuff is. Also, there's the impulse to connect too much and subsume too much under one theory or worldview, and too much reliance on one's own fringe community to explain the world and interpret everything. Both, in a sense, are communities built around esoteric knowledge. And, indeed, I've seen some typical sort of conspiracy theory-seeming stuff on LessWrong related to American intelligence agencies and so on. 

Incel communities because the atmosphere of LessWrong feels rather bitter, resentful, angry, unhappy, isolated, desperate, arrogant, and hateful, and in its own way is also a sort of self-help or commiseration community for young men who feel left out of the normal social world. But rather than encouraging healthy, adaptive responses to that experience, instead both communities encourage anti-social behaviour, leaning into distorted thinking, resentment, and disdainful views of other people.

I just noticed that Ozy recently published a much longer article in Asterisk Magazine on the topic of actual high-control groups or high-demand groups with some connection to the rationalist community. It will take me a while to properly read the whole thing and to think about it. But at a glance, there are some aspects of the article that are relevant to what I'm discussing here, such as this quote:

Nevertheless, some groups within the community have wound up wildly dysfunctional–a term I’m using to sidestep definitional arguments about what is and isn’t a cult. And some of the blame can be put on the rationalist community’s marketing.

The Sequences make certain implicit promises. There is an art of thinking better, and we’ve figured it out. If you learn it, you can solve all your problems, become brilliant and hardworking and successful and happy, and be one of the small elite shaping not only society but the entire future of humanity. 

This is, not to put too fine a point on it, not true. 

Multiple interviewees remarked that the Sequences create the raw material for a cult. To his credit, their author, Eliezer Yudkowsky, shows little interest in running one.

In principle, you could have the view that the typical or median person is benefitted by the Sequences or by LessWrong or the rationalist community, and it's just an unfortunate but uncommon side-effect for people to slip into cults or high-control groups. It sounds like that's what Ozy believes. My view is much harsher: by and large, the influence that LessWrong/the rationalist community has on people is bad, and people who take these ideas and this subculture to an extreme are just experiencing a more extreme version of the bad that happens to pretty much everyone who is influenced by these ideas and this subculture. (There might be truly minor exceptions to this, but I still see this as the overall trend.) 

Obviously, there is now a lot of overlap between the EA Forum and LessWrong and between EA and the rationalist community. I think to the extent that LessWrong and the rationalist community have influenced EA, EA has become something much worse. It's become something repelling to me. I don't want any of this cult stuff. I don't want any of this racist stuff. Or conspiracy theory stuff. Or harmful self-help stuff for isolated young men. I'm happy to agree with the consensus view most of the time because I care about being correct much more than I care about being counter-consensus. I am extremely skeptical toward esoteric knowledge and I think it's virtually always either nonsense or prosaic stuff repackaged to look esoteric. I don't buy these promises of unlocking powerful secrets through obscure websites.

There was always a little bit of overlap between EA and the rationalist community, starting very early on, but it wasn't a ruinous amount. And it's not like EA didn't independently have its own problems before the rationalist community's influence increased a lot, but those problems seemed more manageable. The situation now feels like the rationalist community is unloading more and more of its cargo onto the boat that is EA, and EA is just sinking further and further into the water over time. I feel sour and queasy about this because EA was once something I loved and it's becoming increasingly laden with things I oppose in the strongest possible terms, like racism, interpersonal cruelty, and extremely irrational thinking patterns. How can people who were in EA because of global poverty and animal welfare, who had no previous affiliation with the rationalist community, stand this? Are they all gone already? Have they opted to recede from public arguments and just focus on their own particular niches? What gives?

And to the extent that the racism in EA is independently EA's problem and has nothing to do with the influence of the rationalist community (which obviously has to be more than nothing), then that is 100% EA's problem, but I can't imagine racism in EA could be satisfactorily solved without significant conflict with and alienation of many people who overlap between the rationalist community and EA and who either endorse or strongly sympathize with racist views. That's why the diversion from talking about racism on the EA Forum into discussing the rationalist community's influence on EA.

Racism is paradigmatically evil and there is no moral or rational justification for it. Don't lose sight of something so fundamental and clear. Don't let EA drown under the racism and all the other bad stuff people want to bring to it. (Hey, now EA is the drowning child! Talk about irony!) 

I'm not sure that Toby was wrong to work on this, but if he was, it's because if he hadn't, then someone else with more comparative advantage for working on this problem (due to lacking training or talent for philosophy) would have done so shortly afterwards.

How shortly? We're discussing this in October 2025. What's the newest piece of data that Toby's analysis is dependent on? Maybe the Grok 4 chart from July 2025? Or possibly qualitative impressions from the GPT-5 launch in August 2025? Who else is doing high-quality analysis of this kind and publishing it, even using older data? 

I guess I don't automatically buy the idea that even in a few months we'll see someone else independently go through the same reasoning steps as this post and independently come to the same conclusion. But there are plenty of people who could, in theory, do it and who are, in theory, motivated to do this kind of analysis and who also will probably not see this post (e.g. equity research analysts, journalists covering AI, AI researchers and engineers independent of LLM companies). 

I certainly don't buy the idea that if Toby hadn't done this analysis, then someone else in effective altruism would have done it. I don't see anybody else in effective altruism doing similar analysis. (I chalk that up largely to confirmation bias.)

Why do you think this work has less value than solving philosophical problems in AI safety? If LLM scaling is sputtering out, isn't that important to know? In fact, isn't it a strong contender for the most important fact about AI that could be known right now? 

I suppose you could ask why this work hasn't been done by somebody else already and that's a really good question. For instance, why didn't anyone doing equity research or AI journalism notice this already? 

Among people who are highly concerned about near-term AGI, I don't really expect such insights to be surfaced. There is strong confirmation bias. People tend to look for confirmation that AGI is coming soon and not for evidence against. So, I'm not surprised that Toby Ord is the first person within effective altruism to notice this. Most people aren't looking. But this doesn't explain why equity research analysts, AI journalists, or others who are interested in LLM scaling (such as AI researchers or engineers not working for one of the major LLM companies and not bound by an NDA) missed this. I am surprised an academic philosopher is the first person to notice this! And kudos to him for that!

Are you referring to the length of tasks that LLMs are able to complete with a 50% success rate? I don't see that as a meaningful indicator of AGI. Indeed, I would say it's practically meaningless. It truly just doesn't make sense an indicator of progress toward AGI. I find it strange that anyone thinks otherwise. Why should we see that metric as indicating AGI progress anymore than, say, the length of LLMs' context windows?

I think a much more meaningful indicator from METR would be the rate at which AI coding assistants speeds up coding tasks for human coders. Currently, METR's finding is that it slows them down by 19%. But this is asymmetric. Failing to clear a low bar like being an unambiguously useful coding assistant in such tests is strong evidence against models nearing human-level capabilities, but clearing a low bar is not strong evidence for models nearing human-level capabilities. By analogy, we might take an AI system being bad at chess as evidence that it has much less than human-level general intelligence. But we shouldn't take an AI system (such as Deep Blue or AlphaZero) being really good at chess as evidence that it has human-level or greater general intelligence.

If I wanted to settle for an indirect proxy for progress toward AGI, I could short companies like Nvidia, Microsoft, Google, or Meta (e.g. see my recent question about this), but, of course, those companies stock prices' don't directly measure AGI progress. Conversely, someone who wanted to take the other side of the bet could take a long position in those stocks. But then this isn't much of an improvement on the above. If LLMs became much more useful coding assistants, then this could help justify these companies' stock prices, but it wouldn't say much about progress toward AGI. Likewise for other repetitive, text-heavy tasks, like customer support via web chat.

It seems like the flip side should be different: if you do think AGI is very likely to be created within 7 years, shouldn't that imply a long position in stocks like Nvidia, Microsoft, Google, or Meta would be lucrative? In principle, you could believe that LLMs are some number of years away from being able to make a lot of money and at most 7 years away from progressing to AGI, and that the market will give up on LLMs making a lot of money just a few years too soon. But I would find this to be a strange and implausible view.

This is, of course, sensitive to your assumptions.

In principle, yes, but in a typical bet structure, there is no upside for the person taking the other side of that bet, so what would be the point of it for them? I would gladly accept a bet where someone has to pay me an amount of money on January 1, 2033 if AGI isn't created by then (and vice versa), but why would they accept that bet? There's only downside for them.

Sometimes these bets are structured as loans. As in, I would loan someone money and they would promise to pay me that money back plus a premium after 7 years. But I don’t want to give a stranger from another country a 7-year loan that I wouldn’t be able to compel them to repay once the time is up. From my point of view, that would just be me giving a cash gift to a stranger for no particularly good reason.

There is Long Bets, which is a nice site, but since everything goes to charity, it’s largely symbolic. (Also, the money is paid up by both sides in advance, and the Long Now Foundation just holds onto it until the bet is resolved. So, it's a little bit wasteful in that respect. The money is tied up for the duration of the bet and there is a time value of money.)

What part do you think is uncertain? Do you think RL training could become orders of magnitude more compute efficient? 

The recent anthology Essays on Longtermism, which is open access and free to read here, has several essays with good criticisms of longtermism. You might find some of those essays interesting. The authors included in that anthology are a mix of proponents of longtermism and critics of longtermism.

This is not necessarily to disagree with any of your specific arguments or your conclusion, but I think for people who have not been extremely immersed in effective altruist discourse for years, what has been happening with effective altruism over the last 5-10 years can easily be mis-diagnosed.

In the last 5-10 years, has EA shifted significantly toward prioritizing very long-term outcomes (i.e. outcomes more than 1,000 years in the future) over relatively near-term outcomes (i.e. outcomes within the next 100 years)? My impression is no, not really.

Instead, what has happened is that a large number of people in EA have come to believe that there’s more than a 50% chance of artificial general intelligence being created within the next 20 years, with many thinking there’s more than a 50% chance of it being created within 10 years. If AGI is created, many people in EA believe there is a significant risk of human extinction (or another really, really bad outcome). "Significant risk" could mean anywhere from 10% to over 50%. People vary on that. 

This is not really about the very long-term future. It’s actually about the near-term future: what happens within the next 10-20 years. It’s not a pivot from the near-term to the very long-term, it’s a pivot from global poverty and factory farming to near-term AGI. So, it’s not really about longtermism at all. 

The people who are concerned about existential risk from near-term AGI don’t think it’s only a justified worry if you account for lives in the distant future. They think it’s a justified worry if you only account for people who already alive right now. The shift in opinion is not anything to do with arguments about longtermism, but about people thinking AGI is much more likely much sooner than they previously did, and also them accepting arguments that AGI would be incredibly dangerous if created. 

The pivot in EA over the last 5-10 years has also not, in my observation, been a pivot from global poverty and factory farming to existential risk in general, but a pivot to only specifically existential risk from near-term AGI. 

To put my cards on the table, my own personal view is:

• Longtermism is, in principle, correct (i.e. all else being equal, future lives matter as much as present lives or past lives and, if we can, we should try to ensure there are a lot of good future lives), but even after years of discussion, it seems really hard for anyone to give a good example of what actions longtermism justifies that we wouldn’t have already been doing anyway — other than actions related to existential risk, the discussion of which predates the discussion of longtermism by many years.
• Whereas many people in EA seem to think the probability of AGI being created within the next 7 years is 50% or more, I think that probability is significantly less than 0.1%.
• The arguments that AGI would be extremely dangerous rely on assumptions about the underlying technologies used to create AGI. I don’t think we know yet what those technologies will be, so I don’t accept these arguments. I’m not necessarily saying I know for sure AGI won’t be dangerous, either, I’m just saying knowing for sure would require information we don’t have.
• I miss when EA was much more focused on global poverty and factory farming. (So, I agree with you there.)
• Some existential risks are probably neglected, but not existential risk from near-term AGI. I think the world most likely still underinvests in global catastrophic risks of the humans vs. nature variety: asteroids, large volcanos, and natural pandemics. These risks are easier to calculate rigorous probabilities for. 

The x-risks you discussed in your post are humans vs. humans risks: nuclear war, bioweapons, and the humans creating AGI. These are far more complex. Asteroids don’t respond to our space telescopes by attempting to disguise themselves to evade detection. But with anything to do with humans, humans will always respond to what we do, and that response is always at least somewhat unpredictable. 

I still think we should do things to reduce the risk from nuclear war and bioweapons. I’m just saying that these risks are more complex and uncertain than risks from nature. So, it’s more harder to do the cost-effectiveness math that shows spending to reduce these risks is justified. However, so much in the world can’t be rigorously analyzed with that kind of math, so that’s not necessarily an argument against it!

As for climate change, I agree it's important, and maybe some people in EA have done some good work in this area — I don't really know — but it seems like there's already so much focus on it from so many people, many of whom are extremely competent, it's hard to see what EA would contribute by focusing on it. By contrast, global poverty charity effectiveness wasn't a topic many people outside of international development thought about — or at least felt they could do anything about — before GiveWell and effective altruism. Moreover, there wasn't any social movement advocating for people to donate 10% of their income to help the global poor. 

The Grok chart contains no numbers, which is so strange I don't think you can conclude much from it except "we used more RL than last time."

Isn't the point just that the amount of compute used for RL training is now roughly the same as the amount of compute used for self-supervised pre-training? Because if this is true, then obviously scaling up RL training compute another 1,000,000x is obviously not feasible.

My main takeaway from this post is not whether RL training would continue to provide benefits if it were scaled up another 1,000,000x, just that the world doesn't have nearly enough GPUs, electricity, or investment capital for that to be possible.

I disagree with the claim that pre-training returns declined much

Could you elaborate or link to somewhere where someone makes this argument? I'm curious to see if a strong defense can be made of self-supervised pre-training of LLMs continuing to scale and deliver worthwhile, significant benefits.