Metaculus accepts predictions from just anybody, so Metaculus is not an aggregator of expert predictions. It’s not even a prediction market.

I don’t have to tell you that scaling inputs like compute like money, compute, labour, and so on isn’t the same as scaling outputs like capabilities or intelligence. So, evidence that inputs have been increasing a lot is not evidence that outputs have been increasing a lot. We should avoid ambiguating between these two things.

I’m actually not convinced AI can drive a car today in any sense that was not also true 5 years ago or 10 years ago. I have followed the self-driving car industry closely and, internally, companies have a lot of metrics about safety and performance. These are closely held and rarely is anything disclosed to the public.

We also have no idea how much human labour is required in operating autonomous vehicle prototypes, e.g., how often a human has to intervene remotely.

Self-driving car companies are extremely secretive about the information that is the most interesting for judging technological progress. And they simultaneously have strong and aggressive PR and marketing. So, I’m skeptical. Especially since there is a history of companies like Cruise making aggressive, optimistic pronouncements and then abruptly announcing that the company is over.

Elon Musk has said full autonomy is one year away every year since 2015. That’s an extreme case, but others in the self-driving car industry have also set timelines and then blown past them.

There’s a big difference between behaviours, if a human can do them, that indicate a high level of human intelligence versus behaviours that we would need to see from a machine to conclude that it has human-level intelligence or something close to it.

For example, if a human can play grandmaster-level chess, that indicates high intelligence. But computers have played grandmaster-level chess since the 1990s. And yet clearly artificial intelligence (AGI) or human-level artificial intelligence (HLAI) has not existed since the 1990s.

The same idea applies to taking exams. Large language models (LLMs) are good at answering written exam questions, but their success on these questions does not indicate they have an equivalent level of intelligence to humans who score similarly on those exams. This is just a fundamental error, akin to saying IBM’s Deep Blue is AGI.

If you look at a test like ARC-AGI-2, frontier AI systems score well below the human average.

On average, it doesn’t appear like AI experts do in fact agree that AGI is likely to arrive within 5 or 10 years, although of course some AI experts do think that. One survey of AI experts found their median prediction is a 50% chance of AGI by 2047 (23 years from now) — which is actually compatible with the prediction from Geoffrey Hinton you cited, who’s thrown out 5 to 20 years with 50% confidence as his prediction.

Another survey found an aggregated prediction that there’s a 50% chance of AI being capable of automating all human jobs by 2116 (91 years from now). I don’t know why those two predictions are so far apart.

If it seems to you like there’s a consensus around short-term AGI, that probably has more to do with who you’re asking or who you’re listening to than what people, in general, actually believe. I think a lot of AGI discourse is an echo chamber where people continuously hear their existing views affirmed and re-affirmed and reasonable criticism of these views, even criticism from reputable experts, is often not met warmly.

Many people do not share the intuition that frontier AI systems are particularly smart or useful. I wrote a post here that points out, so far, AI does not seem to have had much of an impact on either firm-level productivity or economic growth, and has achieved only the most limited amount of labour automation.

LLM-based systems have multiple embarrassing failure modes that seem to reveal they are much less intelligent than they might otherwise appear. These failures seem like fundamental problems with LLM-based systems and not something that anyone currently knows how to solve.

What does this have to do with effective altruism?

So, you want to try to lock in AI forecasters to onerous and probably illegal contracts that forbid them from founding an AI startup after leaving the forecasting organization? Who would sign such a contract? This is even worse than only hiring people who are intellectually pre-committed to certain AI forecasts. Because it goes beyond a verbal affirmation of their beliefs to actually attempting to legally force them to comply with the (putative) ethical implications of certain AI forecasts.

If the suggestion is simply promoting "social norms" against starting AI startups, well, that social norm already exists to some extent in this community, as evidenced by the response on the EA Forum. But if the norm is too weak, it won’t prevent the undesired outcome (the creation of an AI startup), and if the norm is too strong, I don’t see how it doesn’t end up selecting forecasters for intellectual conformity. Because non-conformists would not want to go along with such a norm (just like they wouldn’t want to sign a contract telling them what they can and can’t do after they leave the forecasting company).

One of the authors responds to the comment you linked to and says he was already aware of the concept of the multiple stages fallacy when writing the paper.

But the point I was making in my comment above is how easy it is for reasonable, informed people to generate different intuitions that form the fundamental inputs of a forecasting model like AI 2027. For example, the authors intuit that something would take years, not decades, to solve. Someone else could easily intuit it will take decades, not years.

The same is true for all the different intuitions the model relies on to get to its thrilling conclusion.

Since the model can only exist by using many such intuitions as inputs, ultimately the model is effectively a re-statement of these intuitions, and putting these intuitions into a model doesn’t make them any more correct.

In 2-3 years, when it turns out the prediction of AGI in 2027 is wrong, it probably won’t be because of a math error in the model but rather because the intuitions the model is based on are wrong.

I don’t know how Epoch AI can both "hire people with a diversity of viewpoints in order to counter bias" and ensure that your former employees won’t try to "cash in on the AI boom in an acceleratory way". These seem like incompatible goals.

I think Epoch has to either:

1. Accept that people have different views and will have different ideas about what actions are ethical, e.g., they may view creating an AI startup focused on automating labour as helpful to the world and benign

or

2. Only hire people who believe in short AGI timelines and high AGI risk and, as a result, bias its forecasts towards those conclusions

Is there a third option?

There’s also a big difference between what’s technically illegal and what a court would realistically punish a person or an organization for doing, since the courts rely on discernment or, more fittingly, judgment. The latter is much more relevant for deciding whether you should use the word "fraud" in the title of a post about a charity.

TFD, I think your analysis is correct and incisive. I’m grateful to you for writing these comments on this post.

It seems clear that if Jaime had different views about the risk-reward of hypothetical 21st century AGI, nobody would be complaining about him loving his family.

Accusing Jaime of "selfishness", even though he used that term himself in (what I interpret to be) a self-deprecating way, seems really unfair and unreasonable, and just excessively mean. As you and Jeff Kaufman pointed out, many people who are accepted into the EA movement have the same or similar views as Jaime on who to prioritize and so on. These criticisms would not be levied against Jaime if he were not an AI risk skeptic.

The social norms of EA or at least the EA Forum are different today than they were ten years ago. Ten years ago, if you said you only care about people who are either alive today or who will be born in the next 100 years, and you don’t think much about AGI because global poverty seems a lot more important, then you would be fully qualified to be the president of a university EA group, get a job at a meta-EA organization, or represent the views of the EA movement to a public audience.

Today, it seems like there are a lot more people who self-identify as EAs who see focusing on global poverty as more or less a waste of time relative to the only thing that matters, which is that the Singularity is coming in about 2-5 years (unless we take drastic action), and all our efforts should be focused on making sure the Singularity goes good and not bad — including trying to delay it if that helps. People who disagree with this view have not yet been fully excluded from EA but it seems like some people are pretty mean to people who disagree. (I am one of the people who disagrees.)

As a side note, it’s also strange to me that people are treating the founding of Mechanize as if it has a realistic chance to accelerate AGI progress more than a negligible amount — enough of a chance of enough of an acceleration to be genuinely concerning. AI startups are created all the time. Some of them state wildly ambitious goals, like Mechanize. They typically fail to achieve these goals. The startup Vicarious comes to mind.

There are many startups trying to automate various kinds of physical and non-physical labour. Some larger companies like Tesla and Alphabet are also working on this. Why would Mechanize be particularly concerning or be particularly likely to succeed?

Did you let Sinergia know their website still shows the old, incorrect estimate of 354 instead of the new, updated estimate of 285? What reason do you have to believe that staff at Sinergia have an intent to deceive? Is it possible they forgot to update their website or haven’t gotten around to it yet?