I can’t thank titotal enough for writing this post and for talking to the Forecasting Research Institute about the error described in this post.

I’m also incredibly thankful to the Forecasting Research Institute for listening to and integrating feedback from me and, in this case, mostly from titotal. It’s not nothing to be responsive to criticism and correction. I can only express appreciation for people who are willing to do this. Nobody loves criticism, but the acceptance of criticism is what it takes to move science, philosophy, and other fields forward. So, hallelujah for that.

I want to be clear that, as titotal noted, we’re just zeroing in here on one specific question discussed in the report, out of 18 total. It is an unfortunate thing that you can work hard on something that is quite large in scope and it can be almost entirely correct (I haven’t reviewed the rest of the report, but I’ll give the benefit of the doubt), but then the discussion focuses around the one mistake you made. I don’t want research or writing to be a thankless task that only elicits criticism, and I want to be thoughtful about how to raise criticism in the future.

For completeness, to make sure readers have a full understanding, I actually made three distinct and independent criticisms of this survey question and how it was reported. First, I noted that the probability of the rapid scenario was reported as an unqualified probability, rather than the probability of the scenario being the best matching of the three — “best matching” is the wording the question used. The Forecasting Research Institute was quick to accept this point and promise to revise the report.

Second, I raised the problem around the intersubjective resolution/metaprediction framing that titotal describes in this post. After a few attempts, I passed the baton to titotal, figuring that titotal’s reputation and math knowledge would make them more convincing. The Forecasting Research Institute has now revised the report in response, as well as their EA Forum post about the report.

Third, the primary issue I raised in my original post on this topic is about a potential anchoring effect or question wording bias with the survey question.^[1] The slow progress scenario is extremely aggressive and optimistic about the amount of progress in AI capabilities between now and the end of 2030. I would personally guess the probability of AI gaining the sort of capabilities described in the slow progress scenario by the end of 2030 is significantly less than 0.1% or 1 in 1,000. I imagine most AI experts would say it’s unlikely, if presented with the scenario in isolation and asked directly about its probability.

For example, here is what is said about household robots in the slow progress scenario:

By the end of 2030 in this slower-progress future, AI is a capable assisting technology for humans; it can … conduct relatively standard tasks that are currently (2025) performed by humans in homes and factories.

Also:

Meanwhile, household robots can make a cup of coffee and unload and load a dishwasher in some modern homes—but they can’t do it as fast as most humans and they require a consistent environment and occasional human guidance.

Even Metaculus, which is known to be aggressive and optimistic about AI capabilities, and which is heavily used by people in the effective altruist community and the LessWrong community, where belief in near-term AGI is strong, puts the median date for the question “When will a reliable and general household robot be developed?” in mid-2032. The resolution criteria for the Metaculus question are compatible with the sentence in the slow progress scenario, although those criteria also stipulate a lot of details that are not stipulated in the slow progress scenario.

An expert panel surveyed in 2020 and 2021 was asked, “[5/10] years from now, what percentage of the time that currently goes into this task can be automated?” and answered 47% for dish washing in 10 years, so in 2030 or 2031. I find this to be a somewhat confusing framing — what does it mean for 47% of the time involved in dish washing to be automated? — but it points to the baseline scenario in the LEAP survey involving contested questions and not just things we can take for granted.

Adam Jonas, a financial analyst at Morgan Stanley who has a track record of being extremely optimistic about AI and robotics (sometimes mistakenly so), and who the financial world interprets as having aggressive, optimistic forecasts, predicts that a “general-purpose humanoid” robot for household chores will require “technological progress in both hardware and AI models, which should take about another decade”, meaning around 2035. So, on Wall Street, even an optimist seems to be less optimistic than the LEAP survey’s slow progress scenario.

If the baseline scenario is more optimistic about AI capabilities progress than Metaculus, the results of a previous expert survey, and a Wall Street analyst on the optimistic end of the spectrum, then it seems plausible that the baseline scenario is already more optimistic than what the LEAP panelists would have reported as their median forecast if they had been asked in a different way. It seems way too aggressive as a baseline scenario. This makes it hard to know how to to interpret the panelists' answers (in addition to the interpretative difficulty raised by the problem described in titotal's post above).

1. ^{^}

   I have also used the term “framing effect” to describe this before — following the Forecasting Research Institute and AI Impacts — but now checking again the definition of that term in psychology, it seems to specifically refer to framing the same information as positive or negative, which doesn’t apply here.

Update #2: titotal has published a full breakdown of the error involving the intersubjective resolution/metaprediction framing of the survey question. It’s a great post that explains the error very well. Many thanks to titotal for taking the time to write the post and for talking to the Forecasting Research Institute about this. Thanks again to the Forecasting Research Institute for revising the report and this post.

If I want to know what “utilitarianism” means, including any disagreements among scholars about the meaning of the term (I have a philosophy degree, I have studied ethics, and I don’t have the impression there are meaningful disagreements among philosophers on the definition of “utilitarianism”), I can find this information in many places, such as:

• The Stanford Encyclopedia of Philosophy
• Encyclopedia Britannica
• Wikipedia
• The book Utilitarianism: A Very Short Introduction co-authored by Peter Singer and published by Oxford University Press
• A textbook like Normative Ethics or an anthology like Ethical Theory
• Philosophy journals
• An academic philosophy podcast like Philosophy Bites
• Academic lectures on YouTube and Crash Course (a high-quality educational resource)

So, it’s easy for me to find out what “utilitarianism” means. There is no shortage of information about that.

Where do I go to find out what “truth-seeking” means? Even if some people disagree on the definition, can I go somewhere and read about, say, the top 3 most popular definitions of the term and why people prefer one definition over the other? 

It seems like an important word. I notice people keep using it. So, what does it mean? Where has it been defined? Is there a source you can cite that attempts to define it?

I have tried to find a definition for “truth-seeking” before, more than once. I’ve asked what the definition is before, more than once. I don’t know if there is a definition. I don’t know if the term means anything definite and specific. I imagine it probably doesn’t have a clear definition or meaning, and that different people who say “truth-seeking” mean different things when they say it — and so people are largely talking past each other when they use this term. 

Incidentally, I think what I just said about “truth-seeking” probably also largely applies to “epistemics”. I suspect “epistemics” probably either means epistemic practices or epistemology, but it’s not clear, and there is evidently some confusion on its intended meaning. Looking at the actual use of “epistemics”, I’m not sure different people mean the same thing by it.

Do you stand by your accusation of bad faith?

Your accusation of bad faith seems to rest on your view that the restraints imposed by the laws of physics on space travel make an alien invasion or attack extremely improbable. Such an event may indeed be extremely improbable, but the laws of physics do not say so.

I have to imagine that you are referring to the speeds of spacecraft and the distances involved. The Milky Way Galaxy is 100,000 light-years in diameter organized along a plane in a disc shape that is 1,000 light-years thick. NASA’s Parker Space Probe has travelled at 0.064% the speed of light. Let’s round it to 0.05% of the speed of light for simplicity. At 0.05% the speed of light, the Parker Space Probe could travel between the two farthest points in the Milky Way Galaxy in 200 million years.

That means that if the maximum speed of spacecraft in the galaxy were limited to only the top speed of NASA’s fastest space probe today, an alien civilization that reached an advanced stage of science and technology — perhaps including things like AGI, advanced nanotechnology/atomically precise manufacturing, cheap nuclear fusion, interstellar spaceships, and so on — more than 200 million years ago would have had plenty of time to establish a presence in every star system of the Milky Way. At 1% the speed of light, the window of time shrinks to 10 million years, and so on.

Designs for spacecraft that credible scientists and engineers thought Earth could actually build in the near future include a light sail-based probe that would supposedly travel at 15-20% the speed of light. Such a probe could traverse the diameter of the Milky Way in under 1 million years at top speed. Acceleration and deceleration complicate the picture somewhat, but the fundamental idea still holds.

If there are alien civilizations in our galaxy, we don’t have any clear, compelling scientific reason to think they wouldn’t be many millions of years older than our civilization. The Earth formed 4.5 billion years ago, so if a habitable planet elsewhere in the galaxy formed just 10% sooner and put life on that planet on the same trajectory as on ours, the aliens would be 450 million years ahead of us. Plenty of time to reach everywhere in the galaxy.

The Fermi paradox has been considered and discussed by people working in physics, astronomy, rocket/spacecraft engineering, SETI, and related fields for decades. There is no consensus on the correct resolution to the paradox. Certainly, there is no consensus that the laws of physics resolve it.

So, if I’m understanding your reasoning correctly — that surely I must be behaving in a dishonest or deceitful way, i.e. engaging in bad faith, because obviously everyone knows the restraints imposed by the laws of physics on space travel make an alien attack on Earth extremely improbable — then your accusation of bad faith seems to rest on a mistake.

Thanks for giving me the opportunity to talk about this because the Fermi paradox is always so much fun to talk about.

My list is very similar to yours. I believe items 1, 2, 3, 4, and 5 have already been achieved to substantial degrees and we continue to see progress in the relevant areas on a quarterly basis. I don't know about the status of 6.

It’s hard to know what "to substantial degrees" means. That sounds very subjective. Without the "to substantial degrees" caveat, it would be easy to prove that 1, 3, 4, and 5 have not been achieved, and fairly straightforward to make a strong case that 2 has not been achieved.

For example, it is simply a fact that Waymo vehicles have a human in the loop — Waymo openly says so — so Waymo has not achieved Level 4/5 autonomy without a human in the loop. Has Waymo achieved Level 4/5 autonomy without humans in the loop "to a substantial degree"? That seems subjective. I don’t know what "to a substantial degree" means to you, and it might mean something different to me, or to other people.

Humanoid robots have not achieved any profitable new applications in recent years, as far as I’m aware. Again, I don’t know what achieving this "to a substantial degree" might mean to you.

I would be curious to know what progress you think has been made recently on the fundamental research problems I mentioned, or what the closest examples are to LLMs engaging in the sort of creative intellectual act I described. I imagine the examples you have in mind are not something the majority of AI experts would agree fit the descriptions I gave.

For clarity on item 1, AI company revenues in 2025 are on track to cover 2024 costs, so on a product basis, AI models are profitable; it's the cost of new models that pull annual figures into the red. I think this will stop being true soon, but that's my speculation, not evidence, so I remain open that scaling will continue to make progress towards AGI, potentially soon.

Distinguish here between gold mining vs. selling picks and shovels. I’m talking about applications of LLMs and AI tools that are profitable for end users. Nvidia is extremely profitable because it sells GPUs to AI companies. In theory, in a hypothetical scenario, AI companies could become profitable by selling AI models as a service (e.g. API tokens, subscriptions) to businesses. But then would those business customers see any profit from the use of LLMs (or other AI tools)? That’s what I’m talking about. Nvidia is selling picks and shovels, and to some extent even the AI companies are selling picks and shovels. Where’s the gold?

The six-item list I gave was a list of some things that — each on their own but especially in combination — would go a long way toward convincing me that I’m wrong and my near-term AGI skepticism is a mistake. When you say your list is similar, I’m not quite sure what you mean. Do you mean that if those things didn’t happen, that would convince you that the probability or level of credence you assign to near-term AGI is way too high? I was trying to ask you what evidence would convince you that you’re wrong.

This is directly answered in the post. Edit: Can you explain why you don’t find what is said in the post about this satisfactory?

Are you presupposing that good practical reasoning involves (i) trying to picture the most-likely future, and then (ii) doing what would be best in that event (while ignoring other credible possibilities, no matter their higher stakes)?

No, of course not.

I have written about this at length before, on multiple occasions (e.g. here and here, to give just two examples). I don’t expect everyone who reads one of my posts for the first time to know all that context and background — why would they? — but, also, the amount of context and background I have to re-explain every time I make a new post is already high because if I don’t, people will just raise the obvious objections I didn’t already anticipate and respond to in the post.

But, in, short: no.

I'm just very dubious of the OP's apparent assumption that losing such a bet ought to trigger deep "soul-searching". It's just not that easy to resolve deep disagreements about what priors / epistemic practices are reasonable.

I agree, but I didn’t say the AI bubble popping should settle the matter, only that I hoped it would motivate people to revisit the topic of near-term AGI with more open-mindedness and curiosity, and much less hostility toward people with dissenting opinions, given that there are already clear, strong objections — and some quite prominently made, as in the case of Toby Ord’s post on RL scaling — to the majority view of the EA community that seem to have mostly escaped serious consideration.

You don’t need an external economic event to see that the made-up graphs in "Situational Awareness" are ridiculous or that AI 2027 could not rationally convince anyone of anything who is not already bought-in to the idea of near-term AGI for other reasons not discussed in AI 2027. And so on. And if the EA community hasn’t noticed these glaring problems, what else hasn’t it noticed?

These are examples that anyone can (hopefully) easily understand with a few minutes of consideration. Anyone can click on one of the "Situational Awareness" graphs and very quickly see that the numbers and lines are just made-up, or that the y-axis has an ill-defined unit of measurement (“effective compute”, which is relative the tasks/problems compute is used for) or no unit of measurement (just “orders of magnitude”, but orders of magnitude of what?) and also no numbers. Plus other ridiculous features, such as claiming that GPT-4 is an AGI.

With AI 2027, it takes more like 10-20 minutes to see that the whole thing is just based on a few guys’ gut intuitions and nothing else. There are other glaring problems in EA discourse around AGI that take more time to explain, such as objections around benchmark construct validity. Even in cases where errors are clear, straightforward, objective, and relatively quick and simple to explain (see below), people often just ignore it when someone points them out. More complex or subtle errors will probably never be considered, even if they are consequential. 

The EA community doesn’t have any analogue of peer review — or it just barely does — where people play the role of rigorously scrutinizing work to catch errors and make sure it meets a certain quality threshold. Some people in the community (probably a minority, but a vocal and aggressive minority) are disdainful of academic science in general and peer review in particular, and don’t think peer review or an analogue of it would actually be helpful. This makes things a little more difficult.

I recently caught two methodological errors in a survey question asked by the Forecasting Research Institute. Pointing them out was an absolutely thankless task and was deeply unpleasant. I got dismissed and downvoted, and if not for titotal’s intervention one of the errors probably never would have gotten fixed. This is very discouraging.

I’m empathetic to the fact that producing research or opinion writing and getting criticized to death also feels deeply unpleasant and thankless, and I’m not entirely sure on the nuances of how to make both sides of the coin feel rewarded rather than punished, but surely there must be a way. I’ve seen it work out well before (and it’s not like this is a new problem no one has dealt with before).

The FRI survey is one example, but one of many. In my observation, people in the EA community are not receptive to the sort of scrutiny that is commonplace in academic contexts. This could be anything from correcting someone on a misunderstanding of the definitions of technical terms used in machine learning or pointing out that Waymo vehicles still have a human in the loop (Waymo calls it "fleet response"). The community pats itself on the back for "loving criticism". I don’t think anybody really loves criticism — only rarely — and maybe the best we can hope for is to begrudgingly accept criticism. But that involves setting up a social and maybe even institutional process of criticism that currently doesn’t exist in the EA community.

When I say "not receptive", I don’t just mean that people hear the scrutiny and just disagree — that’s not inherently problematic, and could be what being receptive to scrutiny looks like — I mean that, for example, they downvote posts/comments and engage in personal insults or accusations (e.g. explicit accusations of "bad faith", of which there is one on this very post), or other hostile behaviour that discourages the scrutiny. Only my masochism allows me to continue posting and commenting on the EA Forum. I honestly don’t know if I have the stomach to do this long-term. It's probably a bad idea to try.

The Unjournal seems like it could be a really promising project in the area of scrutiny and sober second thought. I love the idea of commissioning outside experts to review EA research. I think for organizations with the money to pay for this, this should be the default.

Is there any high-quality evidence or even good anecdotes about how successful creators are at getting people off the platform? I only know anecdotally things like, e.g., Hank Green complaining about the algorithm aggressively downranking his posts about his charity store.

I also feel like I’ve heard comedians say that Twitter is fine with their jokes, but when they want to promote a show — for many of them, the main purpose of being on Twitter — their followers barely see those tweets. Also, when I used TikTok, I noticed a few sketch comedy creators who had large followings on TikTok but had barely any conversions to YouTube.

I think probably the algorithm is behind a lot of this, but also I think probably most users don’t want the friction of clicking through to another platform.

My cynical take on this is that people scroll Twitter and TikTok to numb out and engage their limbic system, not their prefrontal cortex, so it’s a losing game for all involved.

@Bella that’s part of the answer I owe you. I will give the other part soon. 

The market is a curious thing. The market is efficient, yet if no one disagreed with the market, it wouldn’t be. Human life is suffused with paradox and mystery.

Your accusation of bad faith is incorrect. You shouldn’t be so quick to throw the term "bad faith" around (it means something specific and serious, involving deception or dishonesty) just because you disagree with something — that’s a bad habit that closes you off to different perspectives.

I think it’s an entirely apt analogy. We do not have an argument from the laws of physics that shows Avi Loeb is wrong about the possible imminent threat from aliens, or the probability of it. The most convincing argument against Loeb’s conclusions is about the epistemology of science. That same argument applies, mutatis mutandis, to near-term AGI discourse.

With the work you mentioned, there is often an ambiguity involved. To the extent it’s scientifically defensible, it’s mostly not about AGI. To the extent it’s about AGI, it’s mostly not scientifically defensible.

For example, the famous METR graph about the time horizons of tasks AI systems can complete 80% of the time is probably perfectly fine if you only take it for what it is, which is a fairly narrow, heavily caveated series of measurements of current AI systems on artificially simplified benchmark tasks. That’s scientifically defensible, but it’s not about AGI.

When people outside of METR make an inference from this graph to conclusions about imminent AGI, that is not scientifically defensible. This is not a complaint about METR’s research — which is not directly about AGI (at least not in this case) — but about the interpretation of it by people outside of METR to draw conclusions the research does not support. That interpretation is just a hand-wavy philosophical argument, not a scientifically defensible piece of research.

Just to be clear, this is not a criticism of METR, but a criticism of people who misinterpret their work and ignore the caveats that people at METR themselves give.

I suppose it’s worth asking: what evidence, scientific or otherwise, would convince you that this all has been a mistake? That the belief in a significant probability of near-term AGI actually wasn’t well-supported after all?

I can give many possible answers to the opposite question, such as (weighted out of 5 in terms of how important they would be to me deciding that I was wrong):

• Profitable applications of LLMs or other AI tools that justify current investment levels (3/5)
• Evidence of significant progress on fundamental research problems such as generalization, data inefficiency, hierarchical planning, continual learning, reliability, and so on (5/5)
• Any company such as Waymo or Tesla solving Level 4 or 5 autonomy without a human in the loop and without other things that make the problem artificially easy (4/5)
• Profitable and impressive new applications of humanoid robots in real world applications (4/5)
• Any sort of significant credible evidence of a major increase in AI capabilities, such as LLMs being able to autonomously and independently come up with new correct ideas in science, technology, engineering, medicine, philosophy, economics, psychology, etc. (not as a tool for human researchers to more easily search the research literature or anything along those lines, but doing the actual creative intellectual act itself) (5/5)
• A pure reinforcement learning agent learning to play StarCraft II at an above-average level without first bootstrapping via imitation learning, using no more experience to learn this than AlphaStar (3/5)