To me, that quote really sounds like it's about code in general, not code at Anthropic. 

Dario's own interpretation of the prediction, even now that it's come false, seems to be about code in general, based on this defense:

I made this prediction that, you know, in six months, 90% of code would be written by AI models. Some people think that prediction is wrong, but within Anthropic and within a number of companies that we work with, that is absolutely true now.

If the prediction was just about Anthropic's code, you'd think he would just say:

I made this prediction that in six months 90% of Anthropic's code would be written by AI and now within Anthropic that is absolutely true now.

What he actually said comes across as a defense of a prediction he knows was at least partially falsified or is at least in doubt. If he just meant 90% of Anthropic's code would be written by AI, he could just say he was unambiguously right and there's no doubt about it.

Edit:

To address the part of your comment that changed after you edited it, in my interpretation, "we are finding" just means "we are learning" or "we are gaining information that" and is general enough that it doesn't by itself support any particular interpretation. For example, he could have said:

...what we are finding is we are not far from the world—I think we'll be there in three to six months—where AI is writing 90 percent of grant applications.

I wouldn't interpret this to mean that Anthropic is writing any grant applications at all. My interpretation wouldn't be different with or without the "what we are finding" part. If he just said, "I think we are not far from the world...", to me, that would mean exactly the same thing.

I fear we have yet to truly refute Robin Hanson’s claim that EA is primarily a youth movement.

Wow. This is my first time reading that Robin Hanson blog post from 2015. 

When I was around 18 to 20 or 21, I was swept up in political radicalism, and then I became a pretty strong skeptic of political radicalism afterward — although it always bears mentioning that such things are too complex to cram into an either/or binary and the only way to do them justice is try to sort the good from the bad.

I think largely because of this experience I was pretty skeptical of radicalism in EA when I got involved with my university EA group from around age 23 to 25 or 26. I don't like it when ideas become hungry and try to take over everything. Going from a view on charity effectiveness and our moral obligation to donate 10% of our income to charity to a worldview that encompassed more and more and more was never a move I supported or felt comfortable with.^[1]

It has always seemed to me that the more EA tried to stretch beyond its original scope of charity effectiveness and an obligation to give, which Peter Singer articulated in The Life You Can Save in 2009,^[2] the more it was either endorsing dubious, poorly-supported conclusions or trying to reinvent the wheel from first principles for no particularly good reason. 

I think this paragraph from Hanson's blog post is devastatingly accurate:

Some observers see effective altruism as being about using formal statistics or applying consensus scientific theories. But in fact effective altruists embrace contrarian concerns about AI “foom” (discussed often on this blog), concerns based neither on formal statistics nor on applying consensus theories. Instead this community just trusts its own judgment on what reasoning is “careful,” without worrying much if outsiders disagree. This community has a strong overlap with a “rationalist” community wherein people take classes on and much discuss how to be “rational”, and then decide that they have achieved enough rationality to justify embracing many quite contrarian conclusions.

If you think that effective altruism has discovered or invented radically novel and radically superior general-purpose principles for how to think, live, be rational, or be moral, I'm sorry, but that's ludicrous. EA is a mish-mash of ideas from analytic moral philosophy, international development, public health, a bit of economics and finance, and a bit of few other things. That's all. 

I think the trajectory that is healthy is when people who have strong conviction in EA start with a radical critique of the status quo (e.g. a lot of things like cancer research or art or politics or volunteering with lonely seniors seem a lot less effective than GiveWell charities or the like, so we should scorn them), then see the rationales for the status quo (e.g. ultimately, society would start to fall apart if tried to divert too many resources to GiveWell charities and the like by taking them away from everything else), and then come full circle back around to some less radical position (e.g. as many people as possible should donate 10-20% of their income to effective charities, and some people should try to work directly in high-priority cause areas). 

This healthy trajectory is what I thought of when Hanson said that youth movements eventually "moderate their positions" and "become willing to compromise".

I think the trajectory that is unhealthy is when people repudiate the status quo in some overall sense, seemingly often at least partially because it fills certain emotional needs to make the world other than oneself and to condemn its wicked ways. 

Many (though not all) effective altruists seem content to accept the consensus view on most topics, to more or less trust people in general, to trust most mainstream institutions like academia, journalism, and the civil service (of liberal democratic countries), and they don't particularly seek out being contrarian or radical or to reject the world. 

On the other hand, this impulse to reject the world and be other than it is probably the central impulse that characterizes LessWrong and the rationalist community. EA/rationalist blogosphere writer Ozy Brennan wrote an insightful blog post about rationalists and the "cultic milieu", a concept from sociology that refers to new religious movements rather than the high-control groups we typically think of when we think of "cults". (Read the post if you want more context.) They wrote:

People become rationalists because they are attracted to the cultic milieu—that is, people who distrust authority and want to figure things out for themselves and like knowing secrets that no one else knows. People who are attracted to the cultic milieu are attracted to stigmatized knowledge whether or not it is in fact correct.

In a similar vein, the EA Forum member Maniano wrote a post where they conveyed their impression of EAs and rationalists (abbreviating "rationalists" to "rats", as is not uncommon for rationalists to do):

If I’d have to vaguely point to a specific difference in the vibes of an EAs and those of rats, I would say EAs feel more innocent whereas rats might, with possibly a little bit too much generalization, feel like they’d rank higher in some dark triad traits and feature more of chuunibyou tendencies sprinkled with a dash of narrative addiction.

I don't know for sure what "narrative addiction" means, but I suspect what the author meant is something similar to the sort of psychological tendencies Ozy Brennan described in the post about the cultic milieu. Namely, the same sort of tendency often seen among people who buy into conspiracy theories or the paranoid style in politics to think about the world narratively rather than causally, to favour narratively compelling accounts of events (especially those containing intrigue, secrets, betrayal, and danger) rather than awkward, clunky, uncertain, confusing, and narratively unsatisfying accounts of events.

From the linked Wikipedia article:

Chūnibyō (中二病; lit. 'middle-school second-year syndrome') is a Japanese colloquial term typically used to describe adolescents with delusions of grandeur. These teenagers are thought to desperately want to stand out and convince themselves that they have hidden knowledge or secret powers. It is sometimes called "eighth-grader syndrome" in the United States, usually in the context of localizations of anime which feature the concept as a significant plot element.

I think seeing oneself as other than the wicked world is not a tendency that is inherent to effective altruism or a necessary part of the package. But it is a fundamental part of rationalism. Similarly, EA can be kept safely in one corner of your life, even as some people might try to convince you it needs to eat more of your life. But it seems like the whole idea of rationalism is that it takes over. The whole idea is that it's a radical new way to think, live, be rational, and be moral and/or successful.

I wonder if the kind of boredom you described, Michael, that might eventually set in from a simpler The Life You Can Save-style effective altruism is part of what has motivated people to seek a more expansive (and eventually maybe even totalizing) version of effective altruism — because that bigger version is more exciting (even if it's wrong, and even if it's wrong and harmful).

Personally, I would love to be involved in a version of effective altruism that felt more like a wholesome, warm, inclusive liberal church with an emphasis on community, social ties, and participation. (Come to think of it, one of the main people at the university EA group I was involved in said he learned how to be warm and welcoming to people through church. And he was good at it!) I am not really interested in the postmodernist cyberpunk novel version of effective altruism, which is cold, mean, and unhappy.

1. ^{^}

   I think we should be willing to entertain radical ideas but have a very high bar for accepting them, noting that many ideas considered foundational today were once radical, but also noting that most radical ideas are wrong and some can lead to dangerous or harmful consequences. 
   
   Another thing to consider is how hungry these ideas are, as I mentioned. Some radical ideas have a limited scope of application. For example, polyamory is a radical idea for romantic relationships, but it only affects your romantic relationships. Polyamory doesn't tell you to quit your current job and find a new job where you convince monogamous people to become polyamorous. Or provide services to people who are already polyamorous. Polyamory doesn't tell you to have any particular opinions about politics — besides maybe narrow things like rights (e.g. hospital visitation rights) for people in polyamorous relationships — or technology or culture or the fate of the world. 
   
   When radical ideas become totalizing and want to be the axis around which the world turns, that's when I start to feel alarmed.

2. ^{^}

   The Life You Can Save is an example of a radical idea — one I think we should accept — that, similar to polyamory, may affect our lives in a significant way, but is naturally limited in scope. The Life You Can Save is an expression of a simple and straightforward version of effective altruism. As people have wanted the scope of effective altruism to get larger and larger over time, that has led to the accretion of a more complicated and eclectic version of effective altruism that I think is a lot more dubious.

I'll have to look at that safety report later and see what the responses are to it. At a glance, this seems to be a bigger and more rigorous disclosure than what I've seen previously and Waymo has taken the extra step of publishing in a journal. 

However, I'm not ready to jump to any conclusions just yet because it was a similar report by Waymo (not published in a journal, however) that I paid someone with a PhD in a relevant field to help me analyze and, despite Waymo's report initially looking promising and interesting to me, that person's conclusion was that there was not enough data to actually make a determination one way or the other whether Waymo's autonomous vehicles were actually safer than the average human driver.

I was coming at that report from the perspective of wanting it to show that Waymo's vehicles were safer than human drivers (although I didn't tell the person with the PhD that because I didn't want to bias them). I was disappointed that the result was inconclusive. 

If it turns out Waymo's autonomous vehicles are indeed safer than the average human driver, I would celebrate that. Sadly, however, it would not really make me feel more than marginally more optimistic about the near-term prospects of autonomous vehicle technology for widespread commercialization.

The bigger problem for this overall argument about autonomous vehicles (that they show data efficiency or the ability to deal with novelty isn't important) is that safety is only one component of competence (as I said, a parked car is 100% safe) and autonomous vehicles are not as competent as human drivers overall. If they were, there would be a huge commercial opportunity in automating human driving in a widespread fashion — by some estimations, possibly the largest commercial opportunity in the history of capitalism. The reason this can't be done is not regulatory or social or anything like that. It's because the technology simply can't do the job. 

The technology as it's deployed today is not only helped along by geofencing, it's also supported by a high ratio of human labour to the amount of autonomous driving. That's not only safety drivers in the car or remote monitors and operators, but also engineers doing a lot of special casing for specific driving environments. 

If you want to claim that autonomous vehicles are as an example of AI automating significant human labour, first they would have to automate significant human labour — practically, not just in theory — but that hasn't happened yet.

Moreover, driving should, at least in theory, be a low bar. Driving is considered to be routine, boring, repetitive, not particularly complex — exactly the sort of thing we would thing should be easier to automate. So, if approaches to AI that have low data efficiency and don't deal well with novelty can't even handle driving, then it stands to reason that more complex forms of human labour such as science, philosophy, journalism, politics, economics, management, social work, and so on would be even less susceptible to automation by these approaches.

Just to be clear on this point: if we had a form of AI that could drive cars, load dishwashers, and work an assembly line but not do those other things (like science, etc.), I think that would be wonderful and it would certainly be economically transformative, but it wouldn't be AGI.

Things like Docker containers or cloud VMs that can be, in principle, applied to any sort of software or computation could be helpful for all sorts of applications we can't anticipate. They are very general-purpose. That makes sense to me.

The extent to which things designed for deep learning, such as PyTorch, could be applied to ideas outside deep learning seems much more dubious. 

And if we're thinking about ideas that fall within deep learning, but outside what is currently mainstream and popular, then I simply don't know.

I should add, fairly belatedly, another point of comparison. Two Turing Award-winning AI researchers, Yann LeCun and Richard Sutton, each have novel fundamental ideas — not based on scaling LLMs or other comparably mainstream ideas — for how to get to AGI. (A few days ago, I wrote a comment about this here.)

In a 2024 interview, Yann LeCun said he thought it would take "at least a decade and probably much more" to get to AGI or human-level AI by executing his research roadmap. Trying to pinpoint when ideas first started is a fraught exercise. If we say the start time is the 2022 publication of LeCun's position paper "A Path Towards Autonomous Machine Intelligence", then by LeCun's own estimate, the time from publication to human-level AI is at least 12 years and "probably much more". 

In another 2024 interview, Richard Sutton said he thinks there's a 25% chance by 2030 we'll "understand intelligence", although it's unclear to me if he imagines by 2030 there's a 25% chance we'll actually build AGI (or be in a position to do so straightforwardly) or just have the fundamental theoretical knowledge required to do so. The equivalent paper co-authored by Sutton is "The Alberta Plan for AI Research", coincidentally also published in 2022. So, Sutton's own estimate is a 25% chance of success in 8 years, although it's not clear if success here means actually building AGI or a different goal. 

But, crucially, I also definitely don't think we should just automatically accept these numbers. (I also discussed this in my previous comment about this here.) Researchers like Yann LeCun and Richard Sutton have a very high level of self-belief, which I think is psychologically healthy and rational. It is good to be this ambitious. But we shouldn't think of these as predictions or forecasts, but rather as goals. 

LeCun himself has explicitly said you should be skeptical of anyone who says they have found the secret to AGI and will deliver it ten years, including him (as I discussed here). Which of course is very reasonable! 

In the 2024 interview, Sutton said:

I think we should strive for, like, you know, 2030, and knowing that we probably won't succeed, but you have to try.

This was in response to one of the interviewers noting that Sutton had said "decades", plural, when he said "these are the decades when we're going to figure out how the mind works." 

We have good reason to be skeptical if we look at predictions from people in AI that have now come false, such as Dario Amodei's incorrect prediction about AI writing 90% of code by mid-September 2025 or, for that matter, his prediction made 2 years and 2 months ago that we could have something that sounds a lot like AGI in 2 or 3 years, which still has 10 months left to go but looks extremely dubious. As I mentioned in the post, there's also Geoffrey Hinton's prediction about radiology getting automated and various wrong predictions from various people in AI about widespread fully autonomous driving. 

So, to summarize: what Yann LeCun and Richard Sutton are saying is already much more conservative than a trajectory from publishing a paper to building AGI within 7 years. They both tell us to be skeptical of even the timelines they lay out. And, independent of whether they tell us to be skeptical or not, based on the track record of similar predictions, we have good reason to be skeptical.

To me, this seems to be the much more apt point of comparison than the progress of LLMs from 2018 to 2025. 

right now we have lots of resources that did not exist in 2018, like dramatically more compute, better tooling and frameworks like PyTorch and JAX, armies of experts on parallelization, and on and on. These were bottlenecks in 2018, without which we presumably would have gotten the LLMs of today years earlier.

I fear this may be pointless nitpicking, but if I'm getting the timeline right, PyTorch's initial alpha release was in September 2016, its initial proper public release was in January 2017, and PyTorch version 1.0 was released in October 2018. I'm much less familiar with JAX, but apparently it was released in December 2018. Maybe you simply intended to say that PyTorch and JAX are better today than they were in 2018. I don't know. This just stuck out to me as I was re-reading your comment just now. 

For context, OpenAI published a paper about GPT-1 (or just GPT) in 2018, released GPT-2 in 2019, and released GPT-3 in 2020. (I'm going off the dates on the Wikipedia pages for each model.) GPT-1 apparently used TensorFlow, which was initially released in 2015, the same year OpenAI was founded. TensorFlow had a version 1.0 release in 2017, the year before the GPT-1 paper. (In 2020, OpenAI said in a blog post they would be switching to using PyTorch exclusively.)

LLMs may have some niches in which they enhance productivity, such as by serving as an advanced search engine or text search tool for mathematicians. This is quite different than AGI and quite different from either:

a) LLMs having a broad impact on productivity across the economy (which would not necessarily amount to AGI but which would be economically significant) 

or

b) LLMs fully automating jobs by acting autonomously and doing hierarchical planning over very long time horizons (which is the sort of thing AGI would have to be capable of doing to meet the conventional definition of AGI). 

If you want to argue LLMs will get from their current state where they can’t do (a) or (b) to a state where they will be able to do (a) and/or (b), then I think you have to address my arguments in the post about LLMs’ apparent fundamental weaknesses (e.g. the Tower of Hanoi example seems stark to me) and what I said about the obstacles to scaling LLMs further (e.g. Epoch AI estimates that data may run out around 2028).

If the people arguing that there is an AI bubble turn out to be correct and the bubble pops, to what extent would that change people's minds about near-term AGI? 

I strongly suspect there is an AI bubble because the financial expectations around AI seem to be based on AI significantly enhancing productivity and the evidence seems to show it doesn't do that yet. This could change — and I think that's what a lot of people in the business world are thinking and hoping. But my view is a) LLMs have fundamental weaknesses that make this unlikely and b) scaling is running out of steam.

Scaling running out of steam actually means three things:

1) Each new 10x increase in compute is less practically or qualitatively valuable than previous 10x increases in compute.

2) Each new 10x increase in compute is getting harder to pull off because the amount of money involved is getting unwieldy.

3) There is an absolute ceiling to the amount of data LLMs can train on that they are probably approaching.

So, AI investment is dependent on financial expectations that are depending on LLMs enhancing productivity, which isn't happening and probably won't happen due to fundamental problems with LLMs and due to scaling becoming less valuable and less feasible. This implies an AI bubble, which implies the bubble will eventually pop. 

So, if the bubble pops, will that lead people who currently have a much higher estimation than I do of LLMs' current capabilities and near-term prospects to lower that estimation? If AI investment turns out to be a bubble, and it pops, would you change your mind about near-term AGI? Would you think it's much less likely? Would you think AGI is probably much farther away?

Okay, since you're giving me the last word, I'll take it.

There are some ambiguities in terms of how to interpret the concept of the Turing test. People have disagreed about what the rules should be. I will say that in Turing's original paper, he did introduce the concept of testing the computer via sub-games:

Q: Do you play chess?

A: Yes.

Q: I have K at my K1, and no other pieces. You have only K at K6 and R at R1. It is your move. What do you play?

A: (After a pause of 15 seconds) R-R8 mate.

Including other games or puzzles, like the ARC-AGI 2 puzzles, seems in line with this. 

My understanding of the Turing test has always been that there should be basically no restrictions at all — no time limit, no restrictions on what can be asked, no word limit, no question limit. 

In principle, I don't see why you wouldn't allow sending of images, but if you only allowed text-based questions, I suppose even then a judge could tediously write out the ARC-AGI 2 tasks, since they consist of coloured squares in a 30 x 30 grid, and ask the interlocutor to re-create them in Paint. 

To be clear, I don't think ARC-AGI 2 is nearly the only thing you could use to make an LLM fail the Turing test, it's just an easy example.

In Daniel Dennett's 1985 essay "Can Machines Think?" on the Turing test (included in the anthology Brainchildren), Dennett says that "the unrestricted test" is "the only test that is of any theoretical interest at all". He emphasizes that judges should be able to ask anything:

People typically ignore the prospect of having the judge ask off-the-wall questions in the Turing test, and hence they underestimate the competence a computer would have to have to pass the test. But remember, the rules of the imitation game as Turing presented it permit the judge to ask any question that could be asked of a human being—no holds barred.

He also warns:

Cheapened versions of the Turing test are everywhere in the air. Turing's test is not just effective, it is entirely natural—this is, after all, the way we assay the intelligence of each other every day. And since incautious use of such judgments and such tests is the norm, we are in some considerable danger of extrapolating too easily, and judging too generously, about the understanding of the systems we are using.

It's true that before we had LLMs we had lower expectations of what computers can do and asked easier questions. But it doesn't seem right to me to say that as computers get better at natural language, we shouldn't be able to ask harder questions.

I do think the definition and conception of the Turing test is important. If people say that LLMs have passed the Turing test and that's not true, it gives a false impression of LLMs' capabilities, just like when people falsely claim LLMs are AGI.

You could qualify this by saying LLMs can pass a restricted, weak version of the Turing test — but not an unrestricted, adversarial Turing test — which was also true of older computer systems before deep learning. This would sidestep the question of defining the "true" Turing test and still give accurate information.

I don't think the LLMs in this case are clicking them together. Rather, it seems like the LLMs are being used as a search tool for human mathematicians who are clicking them together. 

If you could give the LLM a prompt along the lines of, "Read the mathematics literature and come up with some new proofs based on that," and it could do it, then I would count that as an LLM successfully coming up with a proof, and with a novel idea.

Based on the tweets you linked to, what seems to be happening is that the LLMs are being used as a search tool like Google Scholar, and it's the mathematicians coming up with the proofs, not the search engine.