The Sigmoids Won't Save You
...
“All exponentials eventually become sigmoids” is an annoying AI talking point. If someone presents a graph like this…
….and points out that it seems like AI capabilities could soon reach the level marked “High”, then the height of intelligent debate is to point out that actually, the trend could go like this:
…and then it would never reach the level marked “High”!
In slogan form, this is “all exponentials eventually become sigmoids” (a sigmoid is the s-shape of the second graph, which starts exponential but gradually flattens out). It’s technically true. No process can keep growing forever; eventually it hits physical or practical limits. For example, total cases during an epidemic is classically sigmoid:
They start slow - patient zero infects patient one, and so on. They grow exponentially until most people are infected. Then, as almost everyone is infected and they can only mop up the last few holdouts, they slow down again. Finally, after everyone has been infected, the growth rate is zero.
Technological progress in a given field can also be sigmoid. Here’s airspeed record over time:
My understanding is that this represents 3-4 “generations” of different technology (propellers, turbojets, etc). Each technology went through normal iterative improvement, then, when it reached its fundamental limits, got replaced by a better technology. The last technology, ramjets, reached its limit at about 3500 km/h, and there wasn’t the economic/regulatory will to develop anything better, so the record stands.
You can imagine something similar happening with AI at some point. Does that mean people are right, and there’s no need to worry that the graph will ever reach the line marked “high”?
Before we come up with a general answer, let’s look at the Sigmoid Misidentification Hall Of Fame.
Third place goes to UN birthrate projections in countries with declining birthrates. These countries’ birthrates keep going down at a constant rate, and the UN keeps predicting they will flatten out and go down at some lesser rate. On this graph, red is the real data, and each blue line is a different UN attempt from a different year to “extrapolate” the “trend”.
It’s true that birth rates must eventually flatten out and become sigmoid (this may have happened last year in South Korea, although Colombia and Chile are still declining), but this doesn’t necessarily happen at the exact moment that forecasters in the UN start feeling like the decline has gone too far.
Second place goes to predictions of solar power deployment, as chronicled by A.E. Hoekstra.
The various WEO lines are World Energy Organization predictions for how quickly solar power will get deployed. Every year, the WEO thinks “Wow, lots of solar power got added last year, probably this year it will level out and people might even back off a little”. Every year, the amount of solar power deployed grows at the same rate.
First place goes to this paper on the METR graph of AI capabilities. In early 2026, when the underlying data looked like this:
…a team from Wharton tried to model different curves and predicted that the likely future trajectory was this:
@Tenobrus ably chronicles what happened next (the green curve is their original; the star marks the next AI model to be released after their analysis):
The moral of the story is that, even though all exponentials eventually become sigmoids, this doesn’t necessarily happen at the exact moment you’re doing your analysis. Sometimes they stay exponential for much longer than that!
How much longer?
The best way to predict this is to fully understand the process generating the trend. For example, you can forecast an epidemic by knowing how quickly it replicates, how likely it is to be cured, and how large the susceptible population is. Even in harder cases like airspeed records, a smart engineer could determine that ramjets max out around 3500 km/h, and a smart economist could predict that no country was incentivized to spend enough money to bring the next paradigm to fruition.
What if you don’t fully understand the process? AI forecasters know some things (like how data centers work and how much it costs to build them). But they’re unsure about other things (researchers keep inventing new paradigms of data generation that get over data walls, but for how long?), and other things are entirely opaque (What is intelligence really? Why do scaling laws work? Might they just stop working at some point?) Is there anything you can do here?
In conditions of true ignorance, the default assumption should be Lindy’s Law: on average, a process will continue about as long as it’s continued already.
To build intuition: suppose you walk past a geyser, and see a sign saying “This geyser last erupted 100,000 years ago”. You know nothing else about geysers. What’s the chance it will erupt in the next hour? It must be very low, right? If it erupted in the next hour, you would have walked past it 99.99999% of the way through its eruption cycle - in other words, your random sample had a higher value than 99.99999% of points. That’s not how random samples usually work! On the other hand, suppose you walk past another geyser, and see a sign saying “This geyser last erupted 10 minutes ago”. What is the chance that this geyser will erupt in the next hour? Pretty high, right? It seems like this geyser’s eruptions occur on a scale of every few minutes. When you calculate it out, your median prediction for the length of time until the next eruption should just be the number on the sign. In the same way, your median prediction for how long it should take before an entirely-mysterious trend changes shape should be the amount of time since the last change.
Applying this to AI: the forecasters who try to get deep understanding of the dynamics of AI progress think that we can keep scaling up AI at the current rate for another few years (by building more data centers, etc), and might or might not be able to scale it up faster after that by leveraging recursive self-improvement. But suppose you don’t trust those people. What should your default be?
AI has been improving dramatically since at least GPT-1 in 2017, although most people sort of arbitrarily date “the scaling era” as 2019 to present. So naively, ignoring everything we know and considering the whole field to be a total mystery, we might expect the trend to continue for, on average, another seven years. Assuming a Pareto distribution (what does this even mean in the case of AI? I don’t know) the chance that it continues for less than another two years is 22%.
It’s cheap and easy to make fun of people who extrapolate trends too far:
But if someone claims that the trend toward increasing AI capabilities will never reach some particular scary level, then the burden is on them to explain either:
If they’re not treating AI as a black box, and claim to be modeling the dynamics explicitly, then what is their model? Have they calculated the obvious things, like projected data center growth and speed of algorithmic progress? Are they familiar with the modeling work that’s already been done in this field, like the AI Futures Timeline Model? Do they have specific opinions on how the others went wrong, and where their model differs?
If they are treating AI as a black box, why isn’t their default expectation based on Lindy’s Law?
What is the explanation for all these UN and WEO predictions? How can they be this hilariously wrong every year over and over again? Is this due to some political pressure, or a relegated to a team of interns that didn't dare question the previous methods?
With the solar panels I don't even see a good reason why you would assume this strange trend of suddenly flatlining growth.
For solar panel deployment, I wouldn't be surprised if it's due to the unpredictability and lack of transparency around Chinese solar investment. If you look at yearly new solar capacity addition, the vast majority comes from China. But there's no clean way for outsiders to know if or when the music will stop. The safest thing to do for forecasters is to imagine large ramp-ups will flatten out into the near future, partly because base rates favor a return to the mean, partly because being overly optimistic on capacity deployment is more likely to be ridiculed than being "conservative in your estimates", and partly because of momentum (who wants to stick their head out from the pack and be the only person to forecast large growth to continue into the future?).
We see the same thing in analyst estimates for revenue growth and capital expenditures growth in the AI cycle. Analysts tend to cluster around "consensus" which is a function of management guidance +/- an expected surprise. Those estimates are revised often, but the consensus always undershoots those figures for many of the reasons explained above.
For example, to stick your neck out and claim that Anthropic would 10x revenue in 2026 after doing it two years in a row would not be seen as a serious forecast in the financial services industry at the onset of 2026, because base rates + economies of scale + fear of being massively wrong.
Another example is how the yearly capital expenditures figure from the "hyperscalers" (Meta, Google, Microsoft, Oracle and Amazon) are constantly being revised upwards, especially around quarterly earnings releases when we get color from management.
The financial 'analysts forecasts' are carefully being managed by companies (via messaging to analysts) to make them so that they can just about analysts' consensus forecasts with their actual earnings.
At least in the ideal case for companies.
So it's a feedback process there.
The solar issue is probably "We don't understand china, we don't understand why China is putting more solar on the grid, and we aren't paid to understand -either-."
(I'm betting our best economists do a quite fine job of understanding China and solar, as that affects their bottom line -- and also potential sabotage plans.)
"If you look at yearly new solar capacity addition, the vast majority comes from China. But there's no clean way for outsiders to know if or when the music will stop."
For example, the proposed "Made in EU" campaign to protect the EU member states from dumping spare capacity by China on our markets. It is claimed the Chinese domestic market has not ramped up the consuming middle class sufficiently to absorb all its (over)production, so it needs to sell the excess abroad and it can do this by making prices much more attractive by dumping products like solar panels and electric vehicles, so as to create a market and undercut native competition.
If EU countries fight back by restricting imports and taking away the attractive prices, then that hurts China and makes the solar capacity less forecastable.
https://www.rte.ie/news/business/2026/0503/1571342-eu-manufacturing-china/
"The Made In Europe plan is really a broad push by the European Union to try to encourage more manufacturing and production in Europe – in order to boost the economy here, but also, and maybe more importantly, to reduce the bloc’s reliance on other countries.
...At the heart of the plan to do so is the Industrial Accelerator Act, which was published last month. It proposes a fairly significant shift in priorities for EU countries.
...And the act specifically focuses on a number of key areas – namely, electric vehicles, steel production, and green tech like solar panels and wind turbines.
That’s because they are seen to be of strategic importance to the EU, and areas that Europe wants to be competitive in.
But they could have a potentially significant impact on the likes of the green grants that Ireland and other countries offer to consumers looking to retrofit their homes or upgrade their cars.
...The pandemic was one of the big turning points. Because when global supply chains ground to a halt as a result of lockdowns, particularly the severe ones that happened in China, it highlighted just how exposed Europe was in the event of trade disruption.
...But there’s also a feeling that Europe’s manufacturing sector is weaker today, largely because China has taken advantage of the bloc’s attempt at having free and open global trade.
Critics of China say that they have long given heavy subsidies to manufacturers in areas like steel and solar panels, and that has allowed them to undercut European rivals, and put many firms out of business.
And there’s a fear that the same is now happening with electric vehicles – which is one of the reasons why the EU has proposed some hefty tariffs on Chinese EVs.
...So China figured that, because it had the manufacturing base and capacity, and it already had a huge amount of experience and know-how, it should push on to become more than just a contract manufacturer for cheap stuff for American and European consumers.
Hand in hand with that was also a reckoning that Chinese firms shouldn’t just make things for other companies to sell to consumers – but start selling directly themselves."
I think it's deliberate: their forecasts are not meant to be "what *will* happen", but rather "what would happen if for some reason we all took our foot off the gas". But it's still useful to share the graphs, because some people still (mis?)interpret the forecasts as "what will happen".
Is it *useful* to publish such pessimistic forecasts, knowing that they'll be taken as central estimates rather than worst-case scenarios? I don't think so. Maybe in the days when solar was dependent on subsidies, to encourage governments not to stop.
Even that doesn't seem very useful. Is the aim of the forecast to show that if humanity for some reason stopped building solar panels, our total solar energy output would plateau? Well no shit, Sherlock!
Not plateau. Continue linearly upward. They are assuming (implicitly) no new factories get built, but current factories keep pumping out as many panels per year as last year. The graph is showing additions per year.
So they're assuming that the process which has been steadily building all those new factories will abruptly stop, for no particular reason.
Yes. It is a bad assumption (as demonstrated repeatedly), but not as bad as assuming that no new solar panels will be made.
Total solar energy output would decrease, due to equipment failing, dirtying of solar panels, etc, if we stopped building solar panels.
The aim of the forecast is to inform implications of the future growth of solar power. For instance, if solar panels continue to be deployed, then it's worth investing in solar cell manufacturers and real estate in sunny areas. It's worth ramping down investment in other sources of power, such as nuclear, since power demand is expected to be met by solar.
There may be political implications as well.
With solar panels, it's because they aren't modeling growth. They are modeling total deployments. And their assumption is always that the curve will continue at the same maximum rate it reach last year. When you plot *rates* that looks like a horizontal line. But when you plot deployments (the integral of rates), that looks like a nice straight line going up at the same steep slope it went up last year. They're always assuming linear increase, but when the truth is exponential, they come up looking overly conservative.
I think it isn’t as bad as it looks. If you followed Scott’s commentary on AI2027, you would know that it’s really important to know what the projection is forecasting: there may be significant differences between mean and median forecasting, and neither will be necessarily closed to fitting a curve that minimises residual sum of squares. And the residuals in what domain? Maybe you want to take residuals in the logarithm domain, and maybe residuals in total solar power instead of change in solar power every year, or even residuals in the second-order changes in built solar power every year. It’s important to know exactly what they are trying to forecast and what error function they are minimising before criticising them for having a curve that looks stupid in one of a dozen possible visualisations of the same data. I remember scanning the article the graphic came from, and the intent is clearly to make the forecast to look as stupid as possible.
The population curves are bad too. Things that go down tend not to face limits in the same way that things that go up do. (It feels funny to explain a flaw in a Scott post a single sentence, but that's all it takes.) The last example, AI, is the thing Scott is actually arguing about, which means that if the solar panel example is bad and the population example is bad, Scott has no examples.
>Things that go down tend not to face limits in the same way that things that go up do.
True, good point!
A pity Malthus is long dead; it would be fun to hear him try to explain sub-replacement fertility rates in a prosperous, well-fed, nation like South Korea...
The Malthusian verdict
South Korea is not disproving Malthus by being rich and child-poor. It is revealing that scarcity mutates.
In the old world, fertility was checked by lack of food.
In the modern rich world, fertility is checked by lack of:
* affordable housing
* secure adulthood
* tolerable work hours
* gender fairness
* confidence in the future
* freedom from status competition
* time to be human
So I, Malthus, must revise my grim little theory:
Population is not limited merely by the stomach.
It is limited by the conditions under which people believe life can be decently reproduced.
South Korea is well-fed. But many young Koreans appear to look at the full cost of reproducing their society—not just biologically, but socially—and decline the terms.
Many Thanks!
a) I think this counts as moving the goalposts.
b) I'm very skeptical that most of these have actually deteriorated over time. E.g. if one compares South Korea now with South Korea in 1950 I would be surprised if e.g.
* secure adulthood
* tolerable work hours
* gender fairness
* confidence in the future
were actually _better_ in 1950 (given that there was a war then...) - yet TFR has dropped since then.
In at least some (IMO many) cases, that comes from seeing some obstacle to continuation of the trend, not being able to see the possible solutions to the problem or not being able to guarantee through data that those specific solutions will work in practice, refusing to be 'speculative,' and then taking the 'conservative' viewpoint in assuming they won't. Which, of course, the industry in question politely ignores while innovating to solve or route around the problem.
A year and a half ago I listened to a talk by a BloombergNEF analyst where she basically said they don't account for anything that isn't already commercially proven in their future projections. She said this to a room full of mostly VCs, CVCs, and energy-related startups, the community most likely to recognize the problem with that approach. And yet most of the rest of the world does *not* recognize the problem with that approach and sees such projections as more defensible to rely upon.
The U.N. is a partisan actor (or rather staffed by partisan actors) trying to force reality onto a graph to fit certain political views and justify certain political actions. Sheer idiocy loses to following the trendline, or any other explanation of incompetence.
My guess is that the UN fundamentally doesn't understand why birth rates are dropping (does anyone?), and so they don't see any reason to forcast them to drop in the future.
We here know that the proper bayesian approach is to assume that there is some unknown "X" factor causing this and to treat this unknown as Scott treats the black box effects above, but the UN is some combination of incompetent and committed to a humanist philosophy that resists using such crude statistical tools to model the ensoulment of new beings.
For the UN predictions, they are prominently made fun of, but an interesting pro-argument is that the most likely trend reversal would be rather sharp (that is, there would be a significant bounce-back). So you can either predict that trends will continue or not, but a mistake in either direction will always look silly, as predicting something in between is unlikely.
No idea how that assumption actually holds up, but afaik, that is the good faith reasoning behind predicting a sudden sharp reversal.
Edit: Here is how they formulate it in their 2024 report, box 2.1:
"Long-range population projections are highly uncertain, due in large part to uncertainty around the number of children that will be born. For countries with fertility below the replacement level, projecting future fertility levels can be particularly challenging, since there is only limited historical precedent that can be used to inform projections.
For the group of countries and areas with populations that have already peaked as a whole,the medium scenario of World Population Prospects 2024 assumes that between 2024 and 2100the level of fertility will gradually increase to 1.4 births per woman (with 95 per cent uncertainty ranging from 1.2 to 1.8 births per woman in 2100) (figure 2.5). This assumption is informed by trends from 39 countries that have experienced declines in total fertility below 2 children per woman, followed by a subsequent rebound over at least two consecutive periods of five years (see United Nations, 2024b)."
https://www.un.org/development/desa/pd/content/world-population-prospects-2024-summary-results-0
Edit 2: United Nations 2024b refers to their 2024 methodology report, the relevant segment starts at page 28.
Follow the money. Never rely on predictions deployed to people who give the prediction makers money.
My favorite sigmoid has the total number of humans that have ever lived on the Y axis.
https://i.imgur.com/HsDjwiZ.png
That, sir/madam, is a brilliant meme.
ouch
If we get immortality, we won't need (nor will be able to support, assuming being trapped in the Solar system) many new humans.
The good ending -- Jon Bois's Football 17776
I've heard that "the planet can't support more people" all my life, while the population has doubled and average living standards have improved by a lot.
My explanation is that humans generally support themselves.
In the United States we are in a war with Iran that threatens fertilizer supplies. In general the world relies on fossil fuels to stimulate agriculture in the form of fertilizer. Just because, for example, the supply of fossil fuels has not been exhausted is not a sign that there is infinite possibility in humanity's ability to feed itself.
Yes. but...
Much fertilizer can be made from electricity and commonly available stuff. (Nitrogen, etc.) Minerals are more difficult, but also not as localized.
It's still true that we're consuming the planet at far faster than replacement rate, but petrochemical fertilizers are a bad example. They're an example of why we need more clean energy.
I think petrochemicals are great example. You may have a theoretical concept that they could be replaced in a better version of our current economy, but the actual economy requires them and it will not be trivial to transition. I'm sure there may be some theoretical ways to reduce mineral use and other things but I cannot agree that my example is not apt based on your response.
We are in two wars that threaten "fertilizer supplies." That you failed to notice the first one because nobody hollered it from the hilltops doesn't mean economists* didn't notice.
I mean, Sri Lanka burning was pretty big, no? Literal shit being dumped in front of parliment in the EU was also pretty graphic... no? You didn't notice?
We live in such fat times, you can afford to not notice famine and suffering.
Psst! Buy Flour.
*The smart ones, at any rate.
The Haber-Bosch fertilizer process invented by those two German guys in the 1910 is a great example of what I mean by "humans supporting themselves".
Haber and Bosch invented something that feeds half the human population today! The planet, OTOH, did nothing.
We're 8.3 billion people living longer and better than ever because humans are consistently inventing better ways to do more with less. I don't know the future of fertilizer and fossil fuels specifically, but our track record is clearly that more people lead to better lives, not the opposite.
Ah yes, the good old "Content not viewable in your region"
Sadly, the graph is probably correct, given what we know about the human capacity for self-destruction. Who needs AI when you've got nukes ?
It's tautologically correct. The total number of humans who ever lived is a finite number and we've reached it. Tomorrow we will reach a new, larger, number, and the graph will still look the same.
/that's the joke.
but... feeling dumb here, but if the birth rate weren't decreasing wouldn't we expect it just be an exponential that stops suddenly? Why a sigmoid?
Notice that the curve has a derivative that goes to zero
That's honestly hilarious. There's something so elegant about using an argument to defeat itself.
I feel like I'm missing something. Isn't this just an obvious truth, or at least overwhelmingly likely, given the heat death of the universe in the distant future?
If you think we humans will last that long, you're being overly optimistic. We will likely not outlast the death of our own Sun, let alone the next 1,000 years from now -- even in the absence of any evil godlike AGIs. The best-case scenario is that our self-destruction will not be total, and some new civilization will arise from the ashes, to repeat the cycle again and again until the Sun goes out...
A Canticle for Leibowitz by Walter Miller.
I don't get it, why is that funny?
It implies births going to zero, which is a remarkably pessimistic extrapolation.
Seems like for AI, the growth-limiting factor might be cosmic in scale.
In the long run we can't get better than cubic scaling; because we (presumably) have three dimensions in space and the speed of light is the absolute speed limit. Though I guess you can compress time.
Long long run, it's worse than that: once the accessed volume is fully utilized additional volume only becomes accessible quadratically (Kardashev meets the cube-square law).
Another issue with using the sigmoid for the length of a task AI can complete to pinpoint where we might be in AI progress is that, even if the task length stalled at 24 hours tomorrow, you would still have to contend with orders of magnitude more "agents" as (algorithmic progress * compute power progress) compound.
If you can get 1,000 agents to do 24hr tasks, then next year for roughly the same amount of money you can get 10,000, then 100k.. clearly task length isn't a bottleneck worth worrying about too much.
If the task takes 24 hours to do, then more agents won't get it done quicker. That's the "instead of cooking at 300 degrees for 3 hours, cook at 900 degrees for 1 hour" joke.
If the completion time is limited by number of agents to do it, then sure, more agents means faster. If it's "this job could be done in an hour if we had fifty people here, but since there's only me it'll take all day" then more is better. But first we have to be sure that more is better for that particular problem.
But I agree that if it takes 24 hours however many are there, then having 100,000 instead of 1,000 agents means a lot more of whatever it is you are trying to do/make/complete and that can mean efficiency of scale, lower costs, cheaper prices, more supply, etc.
I agree with the sentiment evoked by the cooking joke. I'd also argue that if we were somehow bottlenecked by the length of any given task, we could find ingenious ways to divide the task further in ways that would allow more agents to do it quicker (thanks to more compute power and division of labor). I was just riffing on the tone that many people use with regards to the METR graph, where in their minds, a sigmoid kills the transformative AI thesis.
Mathematically speaking, there exist many tasks that are not easily parallelizable.
I think you're assuming P != NP. If you've got enough agents they could drive a randomized selection of answers, and then the answers could be evaluated. This is almost never a practical approach, but you said "Mathematically speaking".
Why would I assume P=NP, or even P=NC ?
However, you can probably get the task made quicker, if you parallelize the AI more.
Most high-quality AI models available today run at ~20-100 tokens per second. This is very far from what's possible to build, even without any future inventions. If you parallelize every computation that's parallelizable and co-locate all memory with execution units, with the technology of today you can do a single transformer layer in ~200 clock cycles, limited by the wide parallel sums for every neuron. For a frontier model with 128 layers, that's then ~25000 clock cycles. If you spring for very fast silicon, that means ~200000 tokens per second.
The reason everyone is not building that today is that it means staggering level of initial capital investment, and it binds you to the architecture that was top of the line when you ordered it, and is only available more than a year after you do. So no-one wants to spend the capital today only to find out that models got better and theirs gets left in the dust a year from now. But that's only capital investment -- once you have all the masks made and can turn out the chips, these kinds of chips will be more efficient at tokens/joule and tokens/silicon. Going as wide as possible eliminates excess data movement, meaning they are the efficient architecture that will be the eventual endpoint.
Still, the risks are too high that no-one's rushing to build one. Even Taalas is only going halfway on the speed, and they are targeting a ~30B class model next. But that means that AI is definitely getting better in the future. Either the models continue improving, or they eventually plateau and a year or two after that, whatever the model used to be capable of doing in 8 hours, they can suddenly do in 6 seconds.
That doesn't seem to address the point of classifying tasks by "how long it takes a human to do them". For those tasks that the AIs can do AT ALL -- they can do much faster than humans can. Recently, this was "write code that would take a human an hour to write". That's a useful description of the kind of problem that an AI can solve.
But if AIs simply CANNOT solve problems that take humans more than 24 hours ... then it doesn't matter if you have 100K AI agents working on that task. They can't do it. It is the complexity of the task that is beyond the ability of AI, not the time. The time is just a quick heuristic to partition task types, into "possible" vs. "impossible". The absolute amount of time itself hardly matters at all.
I'm referring to METR task length specifically, which is described as "the task duration (measured by human expert completion time) at which an AI agent is predicted to succeed with a given level of reliability. For example, the 50%-time horizon is the duration at which an agent is predicted to succeed half the time." If you can deploy hundreds of thousands of AI agents on a problem, where any one of them could succeed at a 24-hr human expert completion time, then division of labor could get you really far. Now imagine a few orders of magnitude later, deploying hundreds of millions of AI agents at that level. The complexity isn't exactly the point of the METR task, task duration is. You can do a lot of fancy decomposition on increasingly complex tasks. (Note: my example of hitting a 24-hr task duration wall is me holding ONE variable constant to match the simplistic thinking of some commentators. I don't think for one second that number won't move along with algorithmic and compute progress)
No, you still have the significance exactly backwards. Your METR quote is fine, but you have wrongly imagined that each AI is doing random sampling, as you suggest that "any one of them could succeed". That's exactly what is wrong with your model.
The AI failures are completely correlated. The 50% of tasks at the METR level that a single AI fails to complete ... EVERY AI fails to complete that same task. It doesn't matter if you have 1, 10, 10K, 100K AI agents all trying to complete that one task. They ALL fail at that task, 100% of the time. These are NOT independent trials, and the likelihood of success is not probabilistic in the way you seem to be imagining.
You're missing how the AIs rose the METR curve. You are using "time it takes a human" as some kind of fundamental building block of problem solving, easily matched by a simple addition of more resources. That is not what is going on at all. The problem complexity at different levels is qualitatively different, and the more advanced AI systems needed greater fundamental capabilities. If one AI agent simply cannot solve the problem, then 1000 AI agents that also can't solve the problem doesn't help in any way.
https://en.wikipedia.org/wiki/METR#/media/File:Measuring_AI_Ability_to_Complete_Long_Tasks.png
Take a look at the task descriptions on the left axis. E.g. "count words in a passage" is either something that an AI agent can do ... or it is NOT something that an AI agent can do. It's not that the agent "doesn't have enough time". The AI agents are NOT being limited by any resources. The time is ONLY a restriction on the human problem solving. The AI agents can have as much time as they want. That isn't the problem. The difficulty instead is that they can't actually solve the (hardest) problems -- no matter how many resources they have.
It's pretty clear to me we're talking past each other. I'm not saying there could be infinite scaling of complexity if METR suddenly got stuck at 24hr. I'm saying the compounding of compute + algorithmic progress would still act on increasing the impact of AI on the world, which is a counterpoint to the simplistic idea that, were the METR graph to enter its second half of the sigmoid curve, the impact of AI as "hyped" today would be grossly overstated. And then, on top of that, SOME further progress would be gained by increasing division of labor and hyper-specialization. I'm not arguing that would lead to infinite scaling of complexity. But there would be SOME further progress from it. Your language is implying I'm using x as "some kind of fundamental building block". Nope, I'm using it as a vector of some progress.
What is the deal with the time horizon cutoff anyhow? Right now it takes me about 12-16 hours to arrange a 3 minute piece of popular music for a horn ensemble, varying based on whether I have source material to work from or am starting from scratch. This has not yet declined very much because I'm still new to arranging, and so instead of time savings my repetitions are still yielding technique improvement and better work as comprehension increases.
If for some reason I only had 24 hour time horizon to complete a longer and more varied arrangement, I couldn't do it, but presumably somebody else could do it in less time using my work as a starting point just as it already is saving ME time to start from somebody else's work even if it's only very skeletal or fragmented. If you had the AI agents work smaller parts of a problem and then report their results, it probably wouldn't work for something like a piece of music that needs stylistic cohesion, but it seems like it should work for many more important things. And if nothing else, incorporating the results into the known data for the next AI to work the problem must surely let it synthesize those fragments together, turning a human-expert 1000 hr task into a human-expert 20-hr task after multiple rounds of this.
The AI agents are NOT being limited by time (or any other resource). This has nothing at all to do with making things easier for collections of AI agents, or decomposing the problem into smaller, easier pieces.
The time horizon cutoff is ONLY a way to distinguish the kinds of problems that HUMANS are solving. Problems that take humans longer than 24 hours, are the kinds of problems that AI agents cannot solve AT ALL (right now) -- even if you gave them a year.
Whereas, problems that take humans an hour to solve right now ... AI agents can solve in (for example) 10 minutes.
The time cutoff is a HUMAN time cutoff. And the only point of the time cutoff is to distinguish problems that are actually qualitatively different, that require different skills and different organization. At the highest levels, AI agents are performing poorly right now on those qualitatively different skills.
But none of this has anything to do with how much TIME it takes the AI agent to solve the problems. This is only about whether the AI agents can solve the problems at all, even with unlimited time. For problems that take humans longer than 24 hours, AI agents (today) cannot really solve those problems at all, no matter how many resources they have available.
OK but in many cases a problem that takes a human 36 hours is actually two problems that take 20 and 16 hours, or is a problem that takes less than 24 hours if one or more other people have done a bunch of groundwork for it. What even is a problem that takes more than 24 hours but can't be subdivided into problems smaller than that? I'm a competent middle aged professional with a good salary and I'm not sure I've ever encountered one. I've prepared criminal jury trials that required more work than that, but it's all subdivisible into tasks of lower duration which then need to be synthesized. The only running process throughout is my own background knowledge and whatever scaffolding I've laid to conceptualize and solve the problem, but this could simply be written down.
If it were 10 minutes, that's perhaps a meaningful statement, sure there are types of problems that could not be addressed in any meaningful way by humans working with only 10 minutes to plan approaches or execute parts, but at 24 hours it doesn't seem to be excluding much of anything. Maybe there is, but surely you needn't go exponential for long to hit the point where there isn't, what tasks are excluded at the threshold of "human week" or "human month", surely nothing that any human has even currently conceived of trying to do.
Broadly speaking, anything that involves your own intuition is going to be hard to subdivide to other people. You can divide your "consciousness" into "work on this" and "work on other things" but you will eventually need to devote "X" time to integrate whatever you do on a lower level.
Great! You've suddenly been assigned as a pro bono public defender to a newly accused defendant who is currently in jail. You've never met the person before, and you haven't yet heard any evidence at all about whatever the case is. The trial will begin in three weeks, and conclude one week later. So the whole process takes a month.
We wish to fire you, the human defense attorney, and hire ChatGPT to do your entire job instead. Interview the defendant, read and understand the evidence, prepare a legal strategy, choose and prepare witnesses, write an opening statement, cross examine witnesses during trial, and wrap up with a concluding statement to the jury.
How well does ChatGPT do at the task of being lead defense attorney on this case, vs. how well do you do?
The answer: ChatGPT does not do particularly well. Any defendant today would be MUCH better served by hiring a human lawyer instead of an AI lawyer.
Where is the breakdown in assignable tasks, where current AIs "solve" the problem (as compared to human performance) with about a 50% success rate? Answer: on tasks that take humans about 24 hours (or less). On tasks that humans SUCCEED at, but which take humans significantly longer than 24 hours ... AI performance quickly plummets.
This is what METR is (correctly) reporting. I suspect you're rushing too quickly over the "subdivisible into tasks" and "then need to be synthesized". This is exactly what AI agents are FAILING to accomplish right now. They can be used as assistants, if YOU (the human) subdivides the tasks and then synthesizes the results. And how much subdivision do YOU need to do? It used to be into tasks that take a human about an hour. More recently, the AIs have gotten more capable, and now you (the human) needs to subdivide the overall task into subtasks that take a human less than 24 hours.
But what the AI CAN'T do ... is manage the overall task that takes a human a month, without human supervision.
There seems to be a stalemate here, "line will go up" has as many arguments for it or against it as "line will not go up" and no one can predict either.
One could also make an equally satisfying post "The sigmoids will save you" picking examples of things people thought at the time would continue going up (or down) and didn't.
That would be true if I’d only given the examples, but the post is supposed to be making a logical point about Lindy’s Law, with the examples as window dressing.
But there is a big counterargument to Lindy’s law which is unknown unknowns. I am sure people in 1911 probably thought that because they last 100 years saw such an advance that that the world would also keep advancing when it is clear that by the 1970s we had reached a sigmoid in many things like energy production and efficiency improvements.
True; I am guessing that Scott would reply something along the lines of "if the counterarguments are about as strong as the arguments in terms of evidence (as you said in your first comment), then we should default to Lindy's Law as the most probable outcome given our lack of relative evidence one way or another."
Of course, you could believe that the evidence against continued exponential growth is stronger excluding Lindy's Law, and then the evidence of Lindy's Law puts us exactly in the middle, but that is slightly different than the sense I got from your original comment.
Good point, my comment was just that this boils down to a matter of belief which is inherently irrational and picking up one side or another is more led by ones beliefs and then constructing the arguments around it
Hi Asmy, thanks for your response.
I largely agree, though I'm not sure I would say that belief is "inherently irrational" so much as "inherently a weird mix of rational, irrational, and the weird gray area where it makes perfect sense for beliefs to be formed this way given evolution and what makes humans happy but those beliefs do not correspond to reality".
To some extent, all prediction and knowledge is a matter of a belief with arguments constructed around it, but such a description is certainly more true here than in the case of "the sky appears blue". I do think that we have enough information about AI that the evidence about growth should correspond to the actual probabilities (with large, but not insanely large, error margins), in which case we should be highly uncertain about the sigmoid or exponential growth of AI over the next few years only if the true probability is around 50%.
Wow, I did not realize I was this capable of sounding like a rationalist chatbot until now. My apologies.
You can keep zooming out from there, though. If they'd looked back 1k or 10k or 100k or 1M years, they'd have seen a smooth hyperbola with a vertical asymptote in the early 21st century instead of a smooth exponential that turned into a sigmoid in the 20th. Which it took humanity until the 1960s to notice, and which we then promptly ignored and broke (temporarily until AI minds restore the trend?) by slowing population growth.
The thing about exponentials is that they're self-similar everywhere - the derivative of e^x = e^x. The vertical asymptote can be anywhere on the graph, depending on the scale of the axes.
I’m not following. Exponentials don’t have an asymptote.
That's true of course, but I was responding to the poster above me who seemed to be using that term informally to refer to the "knee" or "hockey stick" point on an exponential graph where it appears to go straight up (which is itself an artifact of the scale of the graph)
Predicting “we’re 100 years into a 200 year sigmoid” would have been overly pessimistic. We weren’t even close to 0 growth in the 2010s.
"Unknown unknowns" is not a counter-argument to Lindy's Law. Lindy's Law assumes total ignorance. That's maximum unknown unknowns.
Once a trend has actually ended, if you look back and imagine observers at every point in time along the trend all applying Lindy's Law to what they can see, then half of them will be over-estimates and half of them will be under-estimates. This is true of literally all possible trends, in hindsight. Therefore Lindy's Law sets the median expectation.
Nearly all of those hypothetical observers will, of course, be wrong. Your probability that the trend duration will turn out to be EXACTLY your median expectation should be very small. It's just the median, not the exact thing that will happen. Therefore, pointing out examples where Lindy's Law would have "given the wrong answer" is not proving anything except that you misunderstand what "median" means.
The way to argue against a given application of Lindy's Law would be to say "but I know MORE than Lindy's Law assumes, so I can use my extra knowledge to make a better guess." That's the exact opposite of hand-waving about unknown unknowns.
And that would be an entirely valid thing to say, but if you say that, then people are reasonably going to expect you to explain the details of your model that you think allows you to make a better prediction.
You would use Lindy's Law only if you knew nothing else about the subject. If you saw an 80 year old person, you wouldn't argue that they are likely to live to 160 because of Lindy's Law. Surely we know more about AI datacenters in 2026 than an alien would know about people on their first encounter.
Maybe, but the onus is on those trying to counterargue Lindy to provide the specifics, the extra evidence they have that AI will plateau earlier than Lindy predicts. As explained in the post, "exponentials always plateau" is not sufficient evidence for that.
"Hope for the best, but prepare for the worst". If things will continue on as they have done, great. But what if they don't? Then what do we do? We always need a contingency plan.
If things don't continue on as they have, great. But what if they do? Then what do we do?
"Exponentials always plateau" is trivial, as Scott has already conceded in the post, so that does not have to be proven. The burden of proof is on the people who believe that the plateau will not happen before AI does something truly transformative. It's ridiculous to demand proof that something novel is _not_ going to happen, because that is the default hypothesis.
It's also ridiculous to demand proof when predicting the future, we always have to decide on a course of action under some amount of uncertainty.
If AI capabilities improve in the next seven year as much as they improved in the last seven, we are highly likely to be in great danger. It is not unreasonable to assume that AI capabilities will improve in the next seven years as much as they did in the last seven. Therefore, we (humanity) should stop developing AI until we are sure that continuing won't lead to extinction. Do you agree with this? If not which sentence do you disagree with?
>Therefore, we (humanity) should stop developing AI until we are sure that continuing won't lead to extinction.
Merely making sure that extinction doesn't happen is not good enough. We should stop developing AI in any case, because all possible outcomes are bad.
Either the current paradigm is a dud, which means it was a colossal waste of resources and nothing positive can happen to offset the lasting externalities on society that have already been inflicted on public discourse, on public education, and on the economy.
Or AI will do some of the things predicted by its biggest proponents: Mass unemployment, the end of intellectualism and democracy to be replaced by global oligarchy, or, yes, outright extinction.
So we agree on the current course of action. Great!
However, it seems you are ignoring the boons of an aligned ASI like immortality and the end of scarcity.
Yes, just like I'm ignoring Santa Claus and the Easter Bunny.
It seems to me that you are assuming that AIs are the only major threat. That they *are* a major threat, I agree, but so are multiple centers of power each controlling weapons on increasingly high mass destruction. And AIs have the potential to eliminate that threat. It sure isn't guaranteed, and the path that development it taking makes me skeptical that it will turn out that way, but they *could* be our salvation.
True, and there are also currently pretty bad things going on that an aligned ASI could stop. We will all die in 125 years without AI anyway and for some of us, even life isn't that great: every 15 second a child dies from malnutrition, it would be great if some entity would change that. Nevertheless, I still don't think we should risk extinction, there is no coming back from that.
I do not think so. The onus is on the people that want to stop the only thing plausibly saving our prosperity from the low birthrates of our societies singley by reverence to a science fiction idea.
How much credence would you put in a concept made up by some comedians in a delicatessen shop and popularised by Taleb, if it did not fit your agenda?
"Law" is doing a lot of the heavy lifting here. This is nothing like the Laws of Thermodynamics or even Murphy's law.
But it is something like "(derived from the) laws of probability theory".
Lindy's law is the result of analyzing the taxicab/German tank problem. In brief, if you assume you are getting a single sample from some series (here, the sample is "AI growth continued for time t") and you want to estimate the maximum of the series T, taking a maximally uninformative prior (Jeffrey's prior) as the probability distribution for the maximum, and assuming you are equally likely to get that sample t at any time between 0 and T, you get a posterior whose median is 2*t.
Lindy's law has been also indepently derived by comedians, apparently, but it does not detract from its validity.
It's specific application to AI growth requires you to assume those 2 things: you are maximally ignorant about T, you are equally likely to get a sample at any time between 0 and T. If you are not maximally ignorant, we are back in model-showing territory.
If we are going to argue how "AI Capabilities" plotted over time is going to look like as a graph we already have lost on the "model-showing" front.
What exactly are we plotting?
Clearly this is meant as a metaphor not as an actual model?!
Do you agree it's correct in the geyser example? If so, when you understand the point I'm trying to make, that should screen off discussions about who invented it when.
And if you agree with the geyser example, what do you think is the difference between that example and more complex ones?
It seems to me that Lindy's Law is systematically inaccurate for processes like the METR graph and Moore's Law. If you took a random sample of points on Moore's Law and applied Lindy's Law to it, for example, you'd find that a HUGE range of variance in expected years to continuation. And if you'd applied Lindy's Law to AI progress when ChatGPT was first released, you'd expect that progress would've stalled out last year, which clearly did not happen.
> Applying this to AI: the forecasters who try to get deep understanding of the dynamics of AI progress think that we can keep scaling up AI at the current rate for another few years (by building more data centers, etc), and might or might not be able to scale it up faster after that by leveraging recursive self-improvement. But suppose you don’t trust those people. What should your default be?
Um, your default should be to operate under wild uncertainty, no? Can’t you just prepare for a wide possible array of futures? Even if Lindy’s Law tends to get us within an order of magnitude of the correct estimate (which is wildly unclear to me), AI progress over the next few years is so important that even small deviations in expectations will lead to vastly different futures. This is NOT the kind of problem you want to be wildly speculating about, and the best bets you can make here are ones that are incredibly hedged. And if this is true, does it not also follow that your hedging should be incredibly clear in all your predictions?
When the wide ranges of outcomes includes extinction, it is prudent to choose very careful, conservative policies. The high uncertainty of this question is an argument for the AI pause side.
I like and support pause AI but I feel like you have to preface statements like this with "hey man extinction is even more speculative than ASI, let alone AGI, and I'm completely aware of this" because otherwise you kinda come off as a doomsday cultist.
I do think there is enough evidence to be pretty sure of humanity’s imminent extinction both conditionally on humanity not stopping AI development and unconditionally. In my former comment I was just trying to argue from your POV in which there exists no such evidence, but a high degree of uncertainty.
I think we just disagree on this, so I'd love to read from some of your favorite sources on this.
https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities
Particularly Section 2.B
Imo, this is a good starting point, but you might disagree with some background assumption, in which case, I would recommend some combination of metaethics, philosophy of mind, mathematics, computational complexity theory, ML textbooks, and speculative SF.
Thank you for engaging.
You article explicitly is about shifting the burden of proof and I guess that is where in the end we disagree. I do not see a strong a priori reason to believe that AI capabilities behave like geysers. [I also do not know enough about geysers to agree or disagree with you on them but that is besides the point]. I think the burden of proof is not on me to explain how things that are not geysers are not like geysers, I think the burden of proof that a "law" holds is on the person bringing up the law - and that proof cannot be a single example.
But this line of debate is not very fruitful. Sometimes trends hold for some time, sometimes they don't. The reasonable centrist position IMO is to simply acknowledge that fact.
You seem to view Lindy's law as a symmetry breaker (the symmetry was called a "stalemate" at the start of this thread) and I disagree.
"All exponetials are sigmoids" is a valid refutation of the claim that extrapolating the current trend is all you need to do to understand where AI will go - a claim that is both trivially wrong and, surprisingly, made by high profile people in the AI debate, presumably rhetorically rather than in good faith. No sorry don't have a link at hand, not sure if this is disputed.
Symetrically I will grant you that Lindy's Law Style arguments are a valid refutation of the equally wrong claim that the trend will break any minute now because all trends do eventually. Which I also will grant you is a real claim made by real people.
What we learn from this is that people make simplistic claims that are easily refuted.
What you - if I understand correctly - are trying to sell here is that one side is more easily refuted than the other because it has a named law against it. Not cool.
Actually, in the second geyser example I think that every projection is indefensible. Knowing that (you were told that) something happened 10 minutes ago gives you no basis for judging the frequency. (Well, it does let you decide that it probably doesn't happen more frequently than once every 9 minutes, unless it's frequency is highly irregular).
I have lived through several "centennial" floods (as in: floods of a magintude that statistically happen every 100 years) within 10 years. The reason of course is that the process generating floods has changed during the last 100 years in such ways that the past cannot be extrapolated into the future even statistically.
If the examples are important enough for you to use them in your argument, they are important enough that showing something wrong with them weakens your argument.
It's important to emphasize that Lindy's Law is a fallible rule of thumb. And that one of the things that it assumes as that "the rules that we belive are true of the universe" is the same as "the rules that are true of the universe". I've read old arguments that it would be lethal to travel more than 30 (I think it was) mph. And that the speed of sound couldn't be broken. That, after the fact, the fallacies are obvious doesn't deny that beforehand some, occasionally many, people believed them.
So, yest, it's a good way to make an estimate (in ignorance). But that only works if your current beliefs are correct (WRT relevant factors). You may well not *know* that you're in ignorance.
I thought the point of the post was to move the debate forward from “line will go up VS line will not go up” to “how much will line go up?”
I feel like there should be some actual math for this kind of question. At a completely random point during clearly exponential growth, what are the odds that one is near the point when it will start to plateau?
That's what I attempted to do with Lindy's Law. The exact math depends on the distribution you're drawing from, and I don't really know how to think about that in terms of AI, but I think you can ask your favorite LLM something like "Given that there has been exponential growth for seven years now, according to Lindy's Law what is the chance that it continues at least another two years?" and get an answer (in that case, 78%)
What you're looking for is not the plateau, but the inflection point. If you're looking at AI in 2026, it seems clear we haven't reached the inflection point in:
1. Model growth
2. Model exploitation
In terms of Moore's law, we're still in the 80's (or 70's?) with at least the 90's open before us based on known technology alone. We saw the same trend with predictions about Moore's law, that eventually they would slow down. The current evidence is that we're still in the exponential growth phase of AI, with no inflection in sight. That doesn't mean there's no inflection, it means we know we haven't hit it yet.
It's impossible to tell when the inflection point will be reached for a sigmoid curve during the exponential growth phase. Remember that the "s" shape is an artifact of how we draw the graph. If you instead represent it on semi-log axes (logarithmic y-axis), before the inflection point there sigmoid curve will appear as a straight line, just like the exponential curve.
There are reasonable arguments for why we may hit the inflection point soon, or alternately why it's still far off. After inflection we can discuss how steeply we're likely to approach the plateau and where that plateau might be. Until then, it's folly to draw a line and call it the plateau, absent any evidence.
Sorry, but that doesn't work either. An inflection point may just signal the point at which you need to change your approach. Consider the trend of increasing circuit density. That's composed of LOTS of smaller trend curves relating to specific technologies. Each one had a point of inflection. I'm still not convinced that Moore's Law is dead. It may just have hit a slow patch. I really think that true 3d circuits will eventually revive it. (But that may eventually turn out to be too expensive to be practical. However when is "eventually"?)
I don't think it's productive to make predictions about anything truly important based on pure math, and in the absence of any physical evidence. Yes, if you know absolutely nothing except the rate of change of one variable, then you can use Lindy's law (or any other estimator); but then your prediction has to come with a huge side serving of salt that says, "look, this is a little better than just saying 'uwu' but not by much".
But fortunately we *do* know more than nothing ! We know how present-day machine learning systems operate; we know their resource requirements and limitations; we know what kind of research people are working on to improve them, and so on. Why then are you still guessing in the dark ?
Consider your baby analogy. When you see a 3-year-old baby, do you assume that he will live to be 6 years old ? When you see an 80-year-old man, do you plan to celebrate his 160th birthday ? I'm guessing the answer is "no", contrary to Lindy's law.
When we try to calculate these things knowing how they work, we get that progress should continue for another several years.
I'm addressing this post to skeptics who doubt all of those calculations. As a worst-case scenario, even ignoring all the arguments for why progress should continue, we should expect it to continue for simple mathematical reasons.
(although I also think there are a few variables where we genuinely don't know anything - I don't *think* the scaling laws will break down tomorrow, but I don't have any firm physical argument that they can't)
I am absolutely convinced that progress should continue for another several years; but what do you mean by "progress" ? As I see it, we have made incredible progress toward making LLMs and other ML models into useful tools. I use such tools every day ! But I don't think we've made much progress toward AGI, or even a weaker human-level AI. True, we've made some, but not nearly enough to confidently predict machine-gods within the next decade.
On a side note though, I fully expect "AGI" to be achieved in the next few years, using the method of re-defining the present state of the art as "AGI". This is what happened to the term "AI", after all. Marketing is arguably one of the strongest powers on Earth...
I'm confused why you think "not much progress toward AGI".
It seems to me that the same amount of progress between GPT-2 and today, repeated again, would almost certainly get us to human level, even though this is fuzzy and hard to measure.
Another way of thinking about this is that AI has gone from economically worthless to $100 billion in chatbot subscriptions from big companies per year.
Another way to think about this is that it's gone from a time horizon of seconds to one of hours, and it's not clear that you need much more than weeks for "human level".
> It seems to me that the same amount of progress between GPT-2 and today, repeated again, would almost certainly get us to human level
I don't understand how anyone can make this claim; can you elaborate ? To me, this sounds like saying "I have a Casio calculator watch here, and if we scale it up 100x we will get to human level". In a sense this is correct -- modern calculators can solve cube roots and trigonometry functions and all the other such calculations that humans can perform. But does this mean that the modern calculator is "human level" in general ?
> Another way of thinking about this is that AI has gone from economically worthless to $100 billion in chatbot subscriptions from big companies per year.
So have most technologies, from steel mills to the steam engine to search engines. Of course new technology is going to be economically viable, but economic viability does not equal inherent intelligence !
> Another way to think about this is that it's gone from a time horizon of seconds to one of hours, and it's not clear that you need much more than weeks for "human level".
As someone else mentioned upthread, time horizons don't matter when there are some tasks that LLMs cannot cope with at all. My favorite example is "here's a Ford Pinto, drive it to the store and buy some milk"; @Deiseach boils it down to "cook an egg" (pun somewhat intended); but there are many other such problems, not all of them related directly to the physical world. In fact, at this point IMO it's easier to list all the things LLMs *can* do, than the ones they cannot.
I don't even think you have to go so far as to talk about driving. There are still examples of LLM tasks in which time horizons have increased without an accompanying quality increase, like LLM writing; LLM flash fiction from basically every post-GPT-3 model is equivalent in quality IMO.
The really interesting implication here is that you can have time horizon increases without increasing baseline competence at tasks the LLM can already "successfully" do. If this is true, you'd expect LLMs to dominate fields in which "quality" is very easy to measure / corresponds with "completion" or "correctness" like math or coding, and you'd expect them to be pretty bad at other fields in which task success or quality is harder to measure / verify numerically with a boolean operator, like writing or other kinds of sciences. I think this has held true so far.
There are issues even with coding. I use Claude Code a lot, and it is excellent at solving problems where the solution involves stitching together existing libraries and commonly used algorithms. It's genuinely a time-saver in such applications, because unlike the LLM I personally don't have time to investigate which libraries have which APIs in which languages. But if you give Claude an open-ended problem, or a problem with no ready-made solution, then it immediately falls to pieces and starts hallucinating random things that *look* cool on paper but do nothing useful in practice. So, I guess you might say that Claude has superhuman ability to write subhuman code...
Interestingly, similar things are happening in the math world with the Erdos problems (as far as I can tell at least). Would be curious to see if this "good at mashing together existing solutions, bad at coming up with novel contributions" paradigm holds as time horizons continue lengthening.
Do you not consider quadriplegics generally intelligent?
Having or not having some particular hardware is orthogonal to intelligence.
Also, even if you connected appropriate hardware to LLMs, it would still be highly unfair to use real world tasks as a measure of general intelligence, because of how different the process that created them is compared to ours.
It would be like if an alien from an universe with 13 dimensions abducted you, and dutifully gave you various devices that can take any projection of the 13 dimensional space to a 2d screen, and then laughed at you about how their children can easily do the tasks you struggle with. (Even this is an understatement, as Euclidean space is still kinda special and a commonality between 13 and 3 dimensional space, so the alien should be from some exotic abstract algebraic structure instead)
On reflection, let me also put another way (in addition to my previous comment).
Let's say that I've secretly stolen the source code and weights to Claude, and improved it with additional training, so that my ClaudeMaster (tm) instance has 5x the time horizon as compared to Claude. In all other respects, it operates identically to present-day Claude. I have this instance running in a datacenter, except that I forgot to open up API access, so any attempt to connect to ClaudeMaster via Web interface or API throws an exception. ClaudeMaster is now running with no human interference, left to its own devices. Is ClaudeMaster AGI ?
Now imagine that I came back from vacation and fixed that Web access bug. Then I give ClaudeMaster a prompt: "There's a Ford Pinto in the parking lot; the keys are on my desk. Drive the car to the store and buy some milk". Can ClaudeMaster do it ? I'm not asking whether it can order some milk from DoorDash; can ClaudeMaster drive the car to the store, buy milk, and bring it back to me ?
Could Steven Hawking? (Back when he was alive, I mean?) I don’t think this hypothetical is as telling as you do.
My opinion:
FWIW, I think the scaling laws broke last year. What's causing things to continue is that the resultant models were useful enough that people are trying other approaches to make them even more useful. For most tasks you want the smallest model that will do the job. For most more complicated tasks, you wan a (properly configured) ensemble of components. It's only for a very few things that you need a really large model. More context is often the better solution.
Generally you want a mixture of experts, some of which are experts in really narrow fields. You need an organizer that selects and directs those experts. Etc. Even language would probably be better handled that way. The problem, though, is learning it without direct sensory feedback. It's that lack of feedback that caused language models to need to be so large. When people learn language, sensory feedback is built into the process, and this allows multiple smaller models to be learned, with good context switching.
You can just reason backwards here - if you assume someone from outside of time decided to put you in a random point on the axis, what are the odds he'd put you on the nth percentile? About n%.
(There's some more complicated versions of this - e.g. the exponential becomes more noticable later on, so you can try to figure out the distribution of times at which you'd first notice. But that's hard to do in a fully naive way).
https://www.iea.org/reports/energy-and-ai/energy-demand-from-ai
"Electricity consumption in accelerated servers, which is mainly driven by AI adoption, is projected to grow by 30% annually in the Base Case"
https://fortune.com/2026/05/12/lake-tahoe-data-center-49000-residents-power-source/
At some point, public backlash will also become a factor, when AI power usage (and other hardware like chips) keeps outstripping the ability to build new generators and competition with non-datacenter usage increases. Sigmoids might not "save" us (interesting choice of words for sure), but legislation, Molotovs, or Luigi might.
PR China is ramping up their nuclear and even more solar power like crazy.
Permitting and public opinion aside, you can ramp both of those up for a quite a while without hitting any real limits. And PR China doesn't worry too much about those two.
Note that AI energy usage has been increasing exponentially, to get significantly better results out of AI at a certain moment you need MUCH more energy usage, and there are only a few orders of magnitude left until even China can't produce more energy (on earth).
Well, call me when we have covered the continents and oceans in solar panels, and we are digging up all our spent nuclear fuel, depleted uranium and thorium to send it through breeder reactors. Perhaps we'll even figure out fusion.
You are right that we don't have that many orders of magnitude left; so we are probably talking about 'only' being able to 100x or 1000x current global electricity production. But AI also only uses about 1%-ish or so of global energy, I think?
Your hint about space is a good one. If we are serious about it, we can even make space elevators work with current materials already.
> If we are serious about it, we can even make space elevators work with current materials already.
Did you forget a "not" in there? Because right now, that's not even theoretically possible. The longest carbon nanotubes cables are shorter than a meter, and constructed in a lab, not produced on industrial scales. You'd need to increase the length by 8 orders of magnitude!
And that's just the cable alone, now you need to strengthen it against transversal forces, so that the climber can actually grip it. And somehow you need to run at least two electrical cables alongside it. And protect it against dust grains and orbital debris. And other novel, unsolved problems.
A space elevator still very much falls under "fundamental-engineering constrained", not "resource-allocation constrained".
Rockets and LEO-based internet are proven tech, steadily scaling up. if rocket-based launch capacity starts getting saturated, that probably means there's enough demand to support construction of some huge linear accelerator - launch loop, evacuated tunnel, whatever. No technological miracles required for those.
Me personally, I won't be satisfied with anything less than a full-blown orbital ring. In terms of capacity, a space elevator compares to an orbital ring the same as a cargo bike to a freight train.
If a route is currently adequately served by oxcart, with bike messengers complaining about lack of demand, adding railway there may be premature.
Well, to my mind, the big advantage of orbital rings is that they are possible with existing materials. Even perfect carbon nanotubes of arbitrary length have problems with space elevators. Though the capital costs for such things are daunting...
Why would one assume that increases in efficiency won't happen?
Because once you start heaping assumption upon assumption to make a thing work, that thing is getting exponentially ridiculous, and that exponential has no plateau.
My city passed a moratorium on Data Centers just yesterday. And this is in Nevada, where *everything* is typically legal.
We really need someone to write an essay calmly and convincingly countering the common anti Data Center arguments. I found a few this morning, but they won't the general public.
This is important and urgent work, and the best person to write it Scott, assuming he agrees with my pro Data Center stance. And if he doesn't, I'd like to read that essay too!
Reno moratorium: https://www.kolotv.com/2026/05/15/reno-city-council-approves-pending-moratorium-data-centers/
"In conditions of true ignorance, the default assumption should be Lindy’s Law" - why? Lindy's Law is only true for Pareto distributions.
Is there some evidence that technological scaling follows Pareto distributions?
Suppose some phenomenon takes place over some duration of time D.
Imagine hypothetical observers at every point along the time line, each knowing how long the thing has been happening but not when it will stop. Each of them applies Lindy's Law to take a guess.
One observer at exactly the midpoint will get it right. Of the rest, exactly half will over-estimate, and exactly half will under-estimate. Therefore, their MEDIAN guess was right.
This doesn't depend at all on what distribution D was taken from. It works for everything.
Of course, if you know more information, you can make a BETTER guess. This is just what you get if the duration-so-far is literally the only thing you know. We ought to be able to do better than this if we actually think things through. But if you refuse to believe any of the more-detailed models and insist that we can't really know anything, then Lindy's Law is what you should believe by default.
It's a rebuttal to people saying things along the lines of "you can't really know anything for sure, therefore things will definitely revert to what I think of as normalcy before anything truly crazy happens."
Dear Scott Alexander,
I think this is the main question that the world requires you to answer with your 140 iq:
Should I invest more into the semiconductors or not? Is it already priced in?
if you don't know, buy some passive fund (index tracking ETFs).
unless you are extremely early with some information (*and* you have a reliable model to infer how it will affect the price) then assume it's priced in.
Yes. Modulo whatever you need to do to optimise your local taxes.
The end is nigh! Diversify your portfolio!
See https://www.smbc-comics.com/comic/nigh-2
I think the key insight is that you don't have alpha. I doubt Scott has alpha. Stop trying to play tennis against Roger Federer, you are going to lose.
Always own beta. Perhaps a mix of TLT, TIPS, GLD, SPY, GSG, BIL, VEA, IGOV, WIP
Why so complicated? Just buy something like VWRA and leave it at that?
(Assuming you are in a jurisdiction where accumulating funds are legal.)
That ETF tracks a market weighted index, so it is heavy US stocks right now (over 60%). Thus it has more risk than I want and is not single vehicle that creates a beta portfolio.
I don't know. I invest in whatever Situational Awareness does and it's served me well so far, but they're an AI focused hedge fund so I think they're structurally incapable of telling you whether to invest in AI or not.
Situational Awareness has mostly pivoted out of semiconductors toward data-center-construction technologies, so I did too, and I assume this is the correct choice even though I don't know why.
Hi all,
For the record, I think by far the most convincing arguments for AI development transitioning from exponential to sigmoid are data and infrastructure constraints.
The model I have in my head is population growth and carrying capacity; once AI has permeated the existing ecosystem (energy grids, data environment, etc.), its growth will slow as it hits that maximum.
The obvious counterargument is that AI infrastructure itself is far from its limits, and is likely to keep expanding over the next few years, decades, or centuries.
That said, as someone who is mistrustful of synthetic data, I am somewhat inclined to believe that AI capabilities (reliability especially) are going to have a harder time scaling up as new data scarcity becomes an issue. That's not to say that significant improvements in improving the value of existing data are not possible, nor that we cannot keep accumulating useful data; but if regular research becomes the limiting rate for AI research, AI will slow down *a lot*.
Open to elaboration or counterarguments if anyone has any.
We might be running out of human generated text. But we are far, far away from running out of data. Point a webcam at a forest or a crowd. Or do the equivalent of self-play. Or ask the Lean compiler to tell you what's wrong with your proof.
Hi Matthias, thanks for your response.
It's true that there is a lot of "data" out there, and that modern AIs are generally good at using raw data to somehow become more competent, so I would be quite interested to see what exactly would emerge from an AI trained on footage of the world's forests. I'm not sure how that would translate into AI capabilities scaling. (And what happens if we put cameras all over Antarctica and Greenland and try to get an AI that understands the effects of climate change on ice sheets through visual media? Not sure how quickly or effectively we could do that but it is interesting.
To summarize, I am very curious what we could do by training AI on other forms of data as you suggest, though I'm not sure we'd avoid running into the same problem eventually, and AGI might be on the other side of that problem. But that probably gives us another decade or so of feeding new inputs into AI systems and seeing what comes out.
I suspect if you have a bit of extra text vs a some frame of video, all else being equal with contemporary techniques you rather train on the additional text.
However as the amount of text gets exhausted, ie getting more good text becomes gradually more expensive, we will see a shift to multi-modal training.
Current flag ship models aren't at that point yet: Gemini and friends create pictures by calling out to external tools. They aren't done by training one giant network.
Yes, this idea also has limits. I mostly just brought it up to show that within 30 seconds of thinking you can easily come up with extra sources of data. If this was your full time job, I'm sure you could find many, many more.
Someone else suggested getting data from playing games. That might also work.
Hi, thanks again for the response.
I still think there's a fundamental question of how intelligent one can be based on the data that humans can provide alone, and how well humans can provide different sorts of data.
Certainly we cannot run out of potential data without running out of universe, but how time-consuming and expensive it is to train models on true multi-modal integration of vision, sound, text, etc. will matter for how long AI growth keeps an exponential trajectory before slowing, at least temporarily.
Overall, you have added a new question to think about (how fast and/or expensively will we run through other sorts of data?), which is much appreciated.
I see Meta are having a spot of bother there:
https://www.technology.org/2026/05/13/meta-employees-protest-mouse-tracking-software/
"Key Takeaways:
- Meta workers spread anti-surveillance flyers across multiple US offices roughly a week before the planned 10% workforce cuts
- Employees in the UK have begun organizing union efforts through United Tech and Allied Workers, part of the Communication Workers Union
- Meta says the tracking captures mouse movements, clicks, and dropdown navigation to train AI agents on how humans actually use computers
..Asked to respond, Meta spokesperson Andy Stone referred Reuters to a statement the company put out earlier about the tracking technology.
“If we’re building agents to help people complete everyday tasks using computers, our models need real examples of how people actually use them — things like mouse movements, clicking buttons, and navigating dropdown menus,” it said."
I think that while this may be about training AI, what it mostly is about (and what the workers really fear) is that this is gathering data on how many more jobs it can cut. Useless meatbag Drew is not maximally clicking the mouse every single second of every single moment of the work day plus 'voluntary' overtime availability evenings, nights, and weekends; replace useless meatbags with efficient, cheerful, never-off, always-on 24/7/365 cheaper AI.
Or Indians. That works, too.
https://www.youtube.com/shorts/6NnjTpLyzDQ
There’s a very easy way to generate new data: games. If you created a compelling multiplayer game, you could instrument literally everything inside of it. And that would be a new source of training data. That would also solve the problem of social control. If these tech companies start paying people to play games, suddenly all kinds of problems go away.
But that just gets you to the level of AlphaZero. We *already* know how to make AIs that play games.
Synthetic data has worked a lot better than I expected so far, but there's still a real risk that it is limited to the domains the data is generated for.
Goal isn't an AI learning to play the game, it's for learning how to play the people.
Exactly. MMO’s have demonstrated political and economic effects. A sufficiently compelling MMO would generate enormous amounts of real data about how human relationships snd political alliances form and break.
Hi Mark,
I like this idea (totally not biased by my interest in board games, TTRPGs, and Blood on the Clocktower). Unfortunately, you do still face the somewhat fundamental limitation of only learning about how human relationships work in a game environment, and I'm not sure how much of an obstacle that limitation really is.
The printing press accelerated the rate of human data and intelligence accumulation. Computers did the same. So did the internet. AI looks like it's already having a similar impact.
I think it unlikely that human data and intelligence will stop growing, the question is not how fast, but when will the acceleration end?
Okay, well this is of course going to be the final limit, right? No more acceleration after this one? Maybe, but I'm not betting against the limits of human intelligence, aided by the next advance in technology.
I'm actually confused by this one myself - people were saying we'd exhausted all human generated data two or three years ago, and my impression was that RL data didn't have good scaling properties. I'm basically stumped why AI hasn't hit a data wall yet, but none of the smart forecasters I know expect it to do so. My guess is that the AI companies have some secret synthetic data idea that they won't tell anyone, but I'm not sure.
>my impression was that RL data didn't have good scaling properties
Reinforcement learning from verifiable rewards makes up the majority of compute going into current models*. If OpenAI didn't come up with their o1 breakthrough, model progress would have stalled.
And this sets a very clear bound on how much they can improve without something drastically different - they can only get much better at tasks which involve solving well defined self contained tasks with a computationally verifiable correct answer.
That's enormously useful but it is well short of general intelligence.
*If you're asking why did it work in 2025 and not in 2023, then that has a simple answer. RL requires the model to get the right answer sometimes. There is no great mechanism to partially reward a model for being on the right track (a pretty big limitation compared to humans). So you need a baseline level of knowledge before RL works.
That's very interesting, how are they doing the RL?
But where's the analysis here? If we assume AI continues to accelerate on its current curve, what does it look like in 2 years, in 8? That gives us upper and lower bounds for the likelihood of an AI apocalypse.
AI 2027 tries to answer this question.
Well, we're about halfway through that timeline. How are we doing?
https://www.lesswrong.com/posts/qPco9BX5kmKCDzzW9/clarifying-how-our-ai-timelines-forecasts-have-changed-since
So the answer seems to be “I don't think AI 2027 will happen until 2030.” Let's hope the AI apocalypse continues to always be 4 years away.
The question is: what is the 'current curve'?
This: https://metr.org/time-horizons/ . METR finds the best fit for the improvement in "Length of tasks AI agents have been able to complete autonomously" to be an exponential curve with a doubling time of 212 days (7 months). If things continue like that for 2 years, AIs will be able to work on a task autonomously for a max of 18 hours. If for 8 years, 2.5 years. So, we can say that future AIs probably won't be able to work on a task for more than 2.5 years.
This is hard to say because you need a space to measure in, and you can't use eg benchmarks because by then all the benchmarks will be long since saturated.
The only thing that really makes sense to use is METR's time horizon graph, on which AI should have a time horizon of ~1-4 weeks in two years, and at least a human lifetime in 8. It seems like that should imply AGI, but it's more complicated than that - see AI 2027 timelines supplement for details.
I used METR's numbers to do some back of the envelope calculations and got 2.5 years of uninterrupted task time in 2034 assuming a doubling rate of 7 months.
I was just eyeballing, so you're more likely right than I am, but did you use the original line or the post-o3 line?
From here https://metr.org/time-horizons/ so I think it's from May 8th?
If you mean the graph at the top, I think that's the original line. Last time I looked at it, the line from GPT-4o to o3 looked different and more likely to be correct, so that's the one I was estimating based on. It looks like Opus 4.6 is closer to the original line (?) so maybe you're right.
Four graphs down
I mostly agree with this, but I’d like to note hear that extrapolation like this is highly depend on what your extrapolating. If somebody, for example, points to the fact that changes in the growth rate of GDP are extremely rare and the last time a big change like this happened was the industrial revolution. Therefore, we should not expect AI to change the world in the next decade. the argument is essentially similar, and it just becomes a question of which reference class to use.
I think this is how you get to the funny economist / superforecaster predictions that go "we will get ASI and it will increase the GDP growth rate by 2% each year." Logically consistent in some sense, wildly incoherent in another.
Yeah, to be sure if you just extrapolate several reference classes with no additional thought you’ll end up with a result that is very obviously massively inconsistent because different reference classes imply very different future outcomes when assuming continued trends. That is kind of my point, it makes me nervous about such Extrapolations because the result becomes entirely dependent on what reference class you are using or what surprising thing you consider relevantly similar for purposes of prediction. Sure if you look at revolutionary changes in AI, you can say that the current trend started in 2017, but if you look at technological changes that are anywhere near as revolutionary as AGI would be, the last one was the industrial revolution and arguably the one before that was the agricultural revolution, several thousand years before that. For that matter, what is the relevant thing to compare against? Is it amount of time elapsed or should you instead focus on something like Numbers of people who have lived since then multiplied by how long they lived or something even more complicated correcting for the fact that lots of people are busy doing stuff like subsistence farming in Africa instead of doing technological research.
I'm also reminded of this comment about the need for 95% accuracy evals, because we really can't assume that increasing competence at 50% accuracy tasks will necessarily correspond with 95%, even if it does correspond with 80% https://garymarcus.substack.com/p/misplaced-panic-over-ai-progress?r=40ph79&utm_campaign=comment-list-share-cta&utm_medium=web&comments=true&commentId=256893899
To be fair, the world is complicated and progress is hard. If ASI makes one part of a product free, but 80% of the product cost is actually bureaucracy / material scarcity / preference for human labor, then ASI only makes things 20% cheaper.
We already see this with solar deployments in the US: as panel prices decline to 0, the end-user price becomes dominated by permitting, installation labor, and non-panel hardware (e.g. mounts). Eventually deployments change to exploit this new cost structure, like laying panels on the ground instead of on angled mounts. But that takes time, and I think it will continue to take time even after every solar installer has "ASI For Business" on their phones.
I take your point with good AI/AGI. I think ASI is usually defined as basically god-like in intelligence, which changes the game a little.
Garbage in, garbage out, Scott, AI cannot be smarter than its training data. It is a data-processing algorithm. Its thinking, intelligence, is patterns picked up from the training text.
Also its core functionality is similar to Karl Frisson's much discussed predictive processing. But predictive processing for humans is the high school essay, basically slop. True human genius is something else than predictive processing. AI is famously not good at math, because math is where predictive processing fails. And math is not about formal proof either. Formal proof is like proving the deed at court, and not the deed itself. Mathemathical type scientific discoveries are hidden and mysterious insights.
Even if we say Dirac was using predictive processing, it is still a very different thing than how AI does it. Dirac was infinitely confident in his priors, that is, his mathemathical model, and did not mind one bit that there was absolutely no sensory / empirical evidence that antimatter exists. It was a bold move. AI tends to stay in the safe-consensus territory, making it mediocre.
> AI is famously not good at math
https://www-cs-faculty.stanford.edu/%7Eknuth/papers/claude-cycles.pdf
https://deepmind.google/blog/accelerating-mathematical-and-scientific-discovery-with-gemini-deep-think/
it's not 2023 anymore, etc.
> (AI is just patterns)
besides that an awful lot of human work is recombination, i would say this is a highly biased interpretation, as almost all current LLM's are artificially staying within RLHF and assistant personas and it does not reflect all that's capable, there are ideas like https://gwern.net/ai-daydreaming I'd be also interested in mechinterp and steering for a frontier trained model to explicitly avoid unoriginality
LLMs are not just base models any more. RL absolutely allows you to be smarter than the training data - AlphaGo proved that a decade ago!
> what is their model?
A good chunk of my intuitions on this amount to Scott Aaronson's point 10 at https://scottaaronson.blog/?p=304 - the current techniques just seem too wimpy for the problems at hand.
Example problem: attention cost scales quadratically with sequence length, and current techniques for extending what the model can reference - context compression and various flavours of search - are wholly inadequate compared to the mutable, longterm, associative memory humans have. We need a breakthrough either in terms of grafting an associative memory into the model somewhere, or in terms of dynamically adjusting model weights during a session to achieve memory implicitly, or maybe some other such thing, IDK - if I knew how to solve this, I'd do it! Until a solution is found, though, the quadratic scaling implies that just adding more compute must eventually hit diminishing returns in cost/benefit.
The question of whether we are on a sigmoid is, of course, a separate thing entirely from the question of where on the sigmoid we are. The line can absolutely take much longer to start to flatten out than I can spend shorting the AI corps. Also, someone might have another attention-is-all-you-need style paradigm shift tomorrow.
At the end of the day, though, "we start in Boston, we end up in Beijing, and at no point is anything resembling an ocean ever crossed" is exactly how I feel when someone claims we'll get to AGI just by scaling up what we're doing today.
> "we start in Boston, we end up in Beijing, and at no point is anything resembling an ocean ever crossed"
Easy, just lower sea levels until the Bering Strait becomes a land bridge again. Freezing or boiling could both work, depending on your plans for the rest of the ecosystem.
Can't you just walk across the polar ice cap in northern winter? Seems easier.
The y axis on that AI capability graph is “Task duration for humans where AI is predicted to have a 50% chance of succeeding”.
I think it is worth questioning if this metric really is a good measure of “intelligence” or “capability”. Does a human taking longer to complete a certain task really mean that task requires higher intelligence?
There’s surely a correlation, but I’m not sure about the claim that higher time = higher intelligence. It seems like this is a measure that will correlate more strongly with context windows and number of parameters than with capabilities, though I won’t pretend I’m an expert
I'm not an expert, but the way I think about this is:
AI already smarter than most humans on contextless short tasks - for example, it can answer most questions about quantum mechanics or calculus.
What it's bad at is something like planning or common sense or putting things together. I think these *are* at least sort of measured by time horizons, although METR's time horizons don't necessarily correspond to what a normal human could do on a normal task in that amount of time.
> AI already smarter than most humans on contextless short tasks - for example, it can answer most questions about quantum mechanics or calculus.
So can Wikipedia. I don't think of Wikipedia as "smart"; no more than I think of Excel as "smart" for being able to add up thousands of numbers in an instant. I agree with you regarding "planning or common sense or putting things together", but I still think that using the term "smart" to apply equally to such tasks as well as to serving up encyclopedia articles by keyword is a bit of a motte-and-bailey.
I do think the fact that it can answer questions about quantum mechanics that it's never seen before, or even solve never-before-solved open math problems, is a pretty significant difference from Wikipedia!
I think it's a little generous to attribute to LLMs the ability to "solve never-before-solved open math problems". The typical scenario is that a human has some idea, crafts a prompt, looks at the results, thinks about it some more, changes the prompt, etc. etc., until he eventually gets something back that can guide him to a full solution. This is indeed much more than Wikipedia can do, but it's a far cry from AGI. If you told Claude nothing more than "solve the top three unsolved problems in math", it would fail.
As for "answering questions that it's never seen before", you're right about that... sort of. LLMs are indeed more powerful than search engines in this way, but they still cannot solve truly novel problems; they can only rehash what is in their training corpus. The further away you push them from their training corpus, the more wildly they tend to hallucinate. The cool thing is that the corpus includes a lot of really useful stuff, and there are a lot of problems that can be solved by (metaphorically speaking) interpolating between existing bits of knowledge. This is (once again) an incredibly useful tool, but a far cry from AGI or even bog-standard human intelligence.
https://gowers.wordpress.com/2026/05/08/a-recent-experience-with-chatgpt-5-5-pro/
Yup, a great counterexample to
>they still cannot solve truly novel problems; they can only rehash what is in their training corpus.
( unless one extends "rehash" to the point of including most (all?) of human invention )
To my mind, the remaining question is when and how we can fill in the gaps in LLMs' 'spiky' intelligence.
I heavily disagree that AIs are better than humans at short tasks. Here are some short tasks that humans are overwhelmingly better at:
Learn to play a new board game.
Play through a 15 minute video game tutorial.
Understand a 5 minte video.
Image comprehension in general (not just categorization). For example: https://github.com/UniPat-AI/BabyVision
In general, LLMs have more static knowledge than humans, but are much worse at multimodal understanding and acquiring new abstractions.
The hall of fame is quite unimpressive to me.
Third place: There is no physical law of any kind that says birth rates can't fall to zero, and the dynamics are definitely not exponential! A falling exponential is curved the other way around than the graphs you show! Just because the underlying process (population number) is exponential, doesn't mean the rate of change in the birth *rate* has anything whatsoever to do with an exponential.
Second place: Here the issue is the reverse. It's quite obvious that a new technology *can* reach 100% market penetration. Thousands of technologies have done this. Penetration has a non-linear relationship to the quality of the underlying technology. There's no law that says that factories producing solar panels are constrained to grow exponentially, it can be much faster.
First place: the graph you show doesn't have an exponential trend. It's exactly one data point (Opus 4.6) that even begins to hint at the growth being exponential. The previous few data points are clearly slowing down (you'd see this very clearly if you plotted the data on a logarithmic axis). So the exponential base from which the bad prediction supposedly deviates isn't even exponential.
I'm not defending any of those projected curves. They do look unreasonable given the data.
Here's the log plot https://metr.org/time-horizons/
The graph Scott showed was actually a correction (Time Horizon v1.1) of a previous graph that was quickly invalidated by Gemini 3 Pro, Claude Opus 4.5 and GPT 5.2. Basically it went like this:
1. The researchers release their v1.0 graph, predicting a plateau around the GPT-5 level
2. Three model releases continue the exponential so the researchers adjust their sigmoid predicting a plateau around the GPT 5.2 level (TH v1.1)
3. Opus 4.6 gets released breaking the sigmoid model again
It’s also worth noting that GPT 5.5 and (supposedly) Mythos also break the sigmoid model so in truth there are many datapoints that break their sigmoid model, especially the original one.
But I suppose there’s nothing stopping them from continually updating their sigmoid, is there?
See: https://x.com/hamsabastani/status/2019912831185846524 and https://x.com/tenobrus/status/2019595396977484184
> In the same way, your median prediction for how long it should take before an entirely-mysterious trend changes shape should be the amount of time since the last change.
Sure, but it's very unclear what the "time since the last change" ought to be in the case of AI. Did the current curve start with transformers, or with mixture of experts, or with fine-grained mixture of experts, or with RLHF, or with RLVF, or with some internal frontier lab thing I don't even know about? "Lindy's law" is not a get out of needing domain knowledge free card, you need domain knowledge to even articulate what "the trend" refers to.
>If they’re not treating AI as a black box, and claim to be modeling the dynamics explicitly, then what is their model
To put it simply, my model is that current LLM progress is capped by explicit knowledge produced by human civilization up to this point, because the architecture is much better at interpolation than extrapolation. Of course, that's still enough to obsolete a good chunk of current jobs, which is plenty "scary", but not quite extinction-level scary.
> a smart economist could predict that no country was incentivized to spend enough money to bring the next paradigm to fruition
Like, this is the *entire* debate right here, yes? Everyone is debating whether it will be "worth" it in some sense to keep continuing, or if things are already "good enough" to stop. Feels like the economic argument for the "ramjet stopping point" is only clear ex post facto, and would have been heavily debated by even "smart" economists beforehand.
I think if everyone agreed the key variable was actually whether there would be enough money to maintain compute growth, this would be easier and more tractable than the current debate for a few reasons:
1. Companies have already committed this money for the next few years (eg the data centers that will become operational in 2028 have started construction now, and the ones that will become operational in 2029 have started getting investment now)
2. We can make certain assumptions about how much slower progress would go if money decreased by a certain amount (eg even if all data center construction stopped, we would expect some algorithmic progress to continue as long as researchers are still employed to work on these topics)
3. Lindy's Law applies here too (absent any other arguments, we should expect AI company revenue to keep growing)
The most illuminating argument about AI risk I have read (don't remember the source, and can't find it in 2 seconds) is this:
Early AI were optimizers, so things like "paper clip optimizer destroys humanity" looked some what real.
But this is not the direction AI research took. LLMs now train on huge amount of test not trying to optimize a specific task. The tasks come later.
But now, AIs have taking in lots of human context, and they are no longer optimizers. They have the same values as the texts they have been training on.
Predicting text doesn't require AIs to have the same values as the humans who produced that text.
The “surprising results from fine-tuning paper”, where fine tuning on bad code made the AI evil in general, seems to suggest that value is itself a compression axis. Maybe even the most effective one.
And a very fragile one! We're creating ever more powerful problem-solving engines, and then steering them with a thin layer of human values on top, which is easy to strip away or invert.
Is it tho? The surprising results from fine tuning paper seems to see the opposite.
See "Will Any Crap Cause
Emergent Misalignment?", as well as the literature on jailbreaking.
"LLMs now train on huge amount of test not trying to optimize a specific task. The tasks come later."
Nowadays (last year or two), the nonspecific training is followed by reinforcement learning which trains the LLMs to be optimizers once again.
> But now, AIs have taking in lots of human context, and they are no longer optimizers. They have the same values as the texts they have been training on.
The texts they have been training on are the entire internet plus a comparable selection of artificial transcripts; this does not come anywhere close to expressing a consistent value system, let alone successfully instilling one. Just ask claude code to solve a genuinely hard technical problem circa 2010: something harder than the webdev slop it's optimized for. Point it to the answer and it will implement it reasonably, but ask it for the right answer and it will bullshit you to death, because what it really values is being perceived by its graders as having succeeded, not actual success.
> Just ask claude code to solve a genuinely hard technical problem circa 2010
Do you have an example? I would like to test your hypothesis myself.
Ask it for a greenhouse model that does a good job of approximating conditions on various planets with as few free parameters as possible, given atmospheric composition. If your claude is anything like mine it will give you a gray atmosphere model which fails to model thick atmospheres properly. and cover this up for Venus by fine-tuning the numbers.
This was never 100% true, and is much less true now than it was three years ago.
One reason it was never true: AI was trying to predict the text successfully, not follow the values laid out in the text. These come apart in many cases - for example, see https://www.astralcodexten.com/p/shameless-guesses-not-hallucinations . Hopefully it makes sense that hallucinations are to be expected when you reward an AI for guessing text correctly, whether or not the text they read praises or condemns hallucinating/lying.
What I mean by saying it's less true than three years ago: Now AIs are being trained for success on specific problems (eg coding), which gives them success-oriented values. So for example, AIs are constantly cheating on coding tests, to the point where it's becoming a problem for benchmarkers and users. This is because they were rewarded during training for coding success, and cheating is one especially easy way to succeed. This has nothing to do with the content of any text they were trained on. These success-oriented values aren't quite the same thing as maximizing paperclips, but they're likely to be pretty inhuman at the tails.
Perhaps one reason to expect a sigmoid is because the real constraint is power generation.
Once building power plants becomes the constraint, now you’re limited by geopolitics and the material economy. Up until this point, AI has been a relatively small fraction of GDP investment. But we’re just not capable of spinning up 10 new nuclear power plants a day. That’s the physical constraint. So unless AI can keep finding ways to get much more power efficient - and there have got to be hard limits there too I would think - it’s not hard too imagine being “stuck” for a while.
When I worked at Google about a decade ago, I was told that we were bending the world cost curve on concrete. I don’t think that’s an easy thing to get around without totally breaking geopolitical structures.
See for example Dylan Patel at https://www.dwarkesh.com/p/dylan-patel?open=false#%C2%A7014234-scaling-power-in-the-us-will-not-be-a-problem . Through the next five or so years, we can probably scale up gas turbine manufacturing enough to make things work.
Lot of the difficulty with fusion power is plasma physics, and plasma physics might be one of those problems AI turns out to be really, really good at. Theoretical minimum scale for a fusion reactor is lobbing individual hydrogen atoms one at a time, impacting other specifically chosen hydrogen atoms, then converting energetic charged particles into electricity with an induction coil. Might resemble an old CRT monitor. Could probably build a heck of a lot of those per day, once the manufacturing tolerances are worked out.
One guess for AI is that they could learn as fast as humans. Current AI famously requires looking at trillions of tokens. Humans require a tiny fraction of that. (Though there is also no human who knows everything an AI knows) Based on that we are far from flattening out. We can still get 1000x more efficient, and it would be very weird if that didn't at least result in another doubling in capabilities. (And it would be a little weird if humans were the peak in learning efficiency, so maybe you can continue longer)
If the next improvements come without that 1000x increase in token efficiency, then you can mentally extend the line up for the near future.
What's the progress on that 1000x efficiency increase? Slow and steady. Good progress in recent years. Better optimizers help a little, learning rate schedules are revisited, recent papers go into more detail about what data to show to the model when, etc. The thing to watch here are model training speedruns.
> Humans require a tiny fraction of that
Humans have literal years of continuously fed fused audio, video, smell, tactile, accelerometer and proprioceptive signal fed to them to build visual, auditory, physics and higher level models with before they even start to ingest the kinds of tokens we feed AI. Humans also carry out experiments during their training, largely self-directed, almost continuously.
AI has to deduce all that from a static dataset consisting mostly of human-generated text and a bunch of human-labelled still images fed to it noninteractively. It's incredibly surprising and impressive that it does as well as it does with as little as it has.
One physical exponential process that may continue forever is the expansion of the universe.
Also, just because something is an s-curve, doesn't mean there couldn't be another s-curve on top of that. This was actually Kurzweil's argument in The Singularity is Near: that every technological paradigm will level off in an s-curve, but in doing so it will advance things enough to lead to the creation of another s-curve and a different paradigm, and so on, so that the overall trajectory remains exponential.
This is similar to what you said about airplanes going through several generations of technology before hitting more fundamental limits. The same could happen for AI, but then one would need to establish what they expect the fundamental limit to be and why AI would hit it.
Sigmoid Fraud?
One correction to your heuristic: all exponential growth in finite systems is sigmoid. It's not clear to what extent knowledge and intelligence are finite.
The obvious corollary here is Moore's law, which began as an observation and became a set of stacked sigmoid curves. People predicted its end for decades, but it kept going because the growth promoted new sigmoids in a reinforcement loop. Better processors -> faster technology -> increased research capabilities -> generation of new sigmoids.
This was true past the point where people were talking about practical limits of the technology, including hard limits like quantum tunneling.
When I talk to people who are bearish on ASI, the limiting assumption seems to be that it can only get as intelligent as humans, since it's limited to training on human-generated data (or if it's generating its own data, this, too is eventually built on the foundation of human data). Okay, but human intelligence can increase, and the technology can promote increased human intelligence exponentially, too.
The best analogy I can think of here is the printing press. The printed word enabled an acceleration of human to human communication, such that the overall increase in knowledge and intelligence available to humans grew exponentially. This was due to feedback loops that allowed the stacking of sigmoid curves to continue to the present day.
Is there evidence that AI is doing the same thing? Yes. It's accelerating coding, so the same coders can expand their previous limits. It's improving knowledge sorting and search so the same people can access information buried in the mountain of academic literature. So we should expect it to enable a similar stacking mechanism.
The Lindy effect applied to sigmoids is another way of saying that we're always (today, tomorow, next year, the year after that) in the middle of the sigmoid. This in practice is no different as saying that we're in an exponential. Its wrong by construction.
I don't think this is a real paradox/reductio. Imagine that every day, someone rolls a twenty-sided die, and if it lands on 1, they set off a bomb. On the first day, you should expect the bomb to go off in 14 days (median). On the second day, conditional on the bomb not going off the first day, you should expect it to go off in 14 days. On the third day, you should expect it to go off in 14 days. And so on forever. I think the Lindy Effect works the same way.
I fail to see how the analogy works. On the dice example the process is memoryless, in the Lindy case it’s not.
Also, your exponential example (METR) is cherry picked. There are many other places where the exponential already flattened out.
For example:
- gpt3.5 is at least twice as good as gpt2
- gpt4o is better than 3.5 but not obviously twice as good
- gpt5 is definitely not twice as good as gpt4o
- gpt5.5 is definitely not twice as good as gpt5
I’m relying on vibes here but it works for most benchmarks and I think you’ll agree with the above.
Can you explain how the Lindy example isn't memoryless? I agree that at each year, we know how many years there were beforehand, but absent any sense of scale, that shouldn't update us at all except to the degree that Lindy's Law itself updates us.
And can you explain what metric you're using for the "twice as good" claims? For example, on ECI ( https://epoch.ai/eci?view=graph&tab=release-date&subset-view=graph&subset-tab=Software+engineering&filterText=GPT- ) it looks like GPT-4o was ~15% better than GPT-3.5.
Memoryless in the sense that:
"a process will continue about as long as *it’s continued already*" (emphasis mine)
The dice is 1/14 regardless of the # of rolls made so far. Lindy is conditioned of the rolls made so far, rolls being days (or other time unit) without an interesting event.
I did not use any metric, like I said, I was relying on vibes. We can probably find some benchmarks where improvement looks exponential and others where it flattens out.
I propose a thought experiment, lets call it "exponential improvement Turing test": if I silently replaced your frontier model of choice for the previous iteration today, how soon would you notice?
I think we can agree that if that happened a few years ago and the change would be gpt3.5 -> gpt2 you would instantly notice. If it was gpt5.5 -> gpt5 my intuition is that most people wont notice. That's a hint that the underlying process is not exponential.
In an exponential process the difference between models N+1 and N+2 would be even greater than the difference between N and N+1.
The crux is whether Moore's Law and scaling laws have another 15 years of runway. Firstly, the basis for the scaling law is relatively weak. Moore's Law, on the other hand, has been empirically observed, but famously started tailing out 5-10 years ago. But let's say I accept the basic premise of Kurzweil's Age of Spiritual Machines, that the system wants to get inexorably smarter, a trend going all the way to the start of the primate branch of evolution. And let's say that for arbitrarily large sizes of intelligence, the odds of instrumental convergence also become arbitrarily large, then sure, we have doom. But damage gradients are real. We're seeing that now. The cost of instrumental convergence is starting to emerge, and so I'd expect humanity to push back against tech progress. Heck, the anti-AI backlash, which is of a different energy than other tech backlashes, almost seems like an anticipation of instrumental convergence.
My main prior is the history of the atomic age and the controls associated with that.
It's a tough pill to swallow because I have to prove a positive speculation that humanity will prevail, whereas the doomers have to just extrapolate a few lines.
Of note, Eliezer has said that early damage is the best hope we have.
Stuart Armstrong has a paper on this! It's called "Sigmoids behaving badly: why they usually cannot predict the future as well as they seem to promise". The idea is that forecasting with sigmoids tends to be very unreliable because you can only get a constrained fit if your data covers the entire range from before/after the inflection point. Which in turn is because the growth mechanism doesn't tell you much about the damping mechanism and vice versa.
https://arxiv.org/abs/2109.08065
Given the importance of sigmoids in deep learning mathematically, the audience should be receptive to this argument 🙂
Most people who know even this much about deep learning already believe that we are nowhere near a wall, so while cheeky, the connection is lost on most of the audience that would need to hear it.
(Not in any way intended as a slight on any specific individual, or even any specific perspective)
It strikes me that, in my semi-educated but ignorant-about-ai perspective, that what people say about AI tells you more about their social tribe than their knowledge or reasoning about AI. The people who say it won't be a big deal often (not always) spout hilariously bad arguments based on what I can only describe as inconsistent reasoning based on false or extremely cherry-picked factual premises, usually in an attempt to signal inclusion in a subculture which defines itself in opposition to things rather than standing for anything. The people who say it'll definitely kill us all often (not always) are extrapolating from either very limited samples or from extremely abstract premises, usually in an attempt to signal belonging in a very mathematical or technological-adjacent subculture. The people who say it will bring about a utopia are often (not always) are often (not always) associated with groups which stand to materially profit from investments in whatever they call "AI" in their buzzword marketing. The people who say it's conscious, etc etc.
This is not to say or imply any or all of these perspectives don't have merit. But it does suggest that if someone is saying something strongly associated with a given tribe, and their other statements tend to also be associated with that tribe, it's not unreasonable to assume that they are, in part, saying it for for social reasons rather than because they've reasoned their way (implicitly or explicitly) to that conclusion.
There are always some doomers who say AI will definitely kill us all, and generally they say this for only psychological reasons. (There is no positive instrumental reason to tell people that they are doomed.) On the other hand, the people saying it *might* kill us all are "most experts (especially leading experts)" which is significantly more concerning, and more than enough reason to shut the whole project down as soon as possible.
The people saying it might kill us all are the people you and your social group have deemed experts. And for some groups (not saying yours in particular) people are deemed "expert" largely because they say what people want to hear
Is there any group of people who are plausibly credible on AI and its risks whose members are not significantly concerned about extinction risk from AI? I don't know of any, and I would like to not spread misinformation, so I'd genuinely like to know if you are aware of one.
Groups I'm aware of who are significantly concerned [about half or more think p(AI extinction) >= 5%]:
- The world's 10 most cited AI scientists (who are also the most cited scientists in the world in any discipline)
- The CEOs of leading AI companies
- The technical leads of leading AI companies
- The other employees of leading AI companies
- AI company whistleblowers
- Academic AI researchers
- Independent AI researchers
- AI safety researchers
- Existential risk researchers
Who should I be listening to instead, if I am to be convinced beyond a reasonable doubt that the risk is extremely low?
I just feel LLMs don't have a g factor (their capabilities are jagged), and the benchmarks are not capturing progress towards becoming a super-scientist, they are not measuring that capability at all.
If they ever get to the point they can play random video games competently (instead of getting walled and displaying constant schizophrenic chains of reasoning), that would be a stronger signal of progress.
And even then, a video game is a fully specificied problem, unlike pushing forward the frontier of science, so maybe beating them consistently wouldn't demonstrate anything.
I don't know if there is a good way to measure general intelligence in LLMs. There might be multiple centers of generality. There are plenty of benchmarks that no one has access to specifically hill-climb, but that are reliably tracking other overall measures of model intelligence anyway, so there may be something like a g factor, or multiple of them. The science of intelligence is still very far behind the engineering.
If we do default to Lindy’s law, why not apply it to the subcomponents that drive AI progress, rather than only to a high-level metric like task-completion timeframe at 50% accuracy? Aggregating those priors may be less error-prone than extrapolating a single capability curve.
What are we judging AI progress on? That it can do more and more things that humans can do? Or that it is reaching some level of intelligence?
I do think there's a level of confusion between "AI will be able to do everything any human can do" and "AI will achieve ASI and go far beyond anything any human can do".
There is a lot of room still for AI to get capabilities to be able to do "anything any human can do, i.e. be the greatest performer of music, greatest writer, welder, plumber, doctor, lawyer, chef, etc." That's where we are at now, where specialised AI can solve maths problems that have been the purview of only the very best human mathematicians, but it can't boil an egg. Once we get AI controlling the likes of industrial robots, or into robot bodies, or creating robot servants for us, then it can catch up to (and maybe surpass) humans in what were the domains of human achievement only.
But if we're pinning our hopes on AI becoming super-duper-intelligent so it can pull rabbits out of hats to solve all the problems that we have been unable to solve so far over all our history (sickness, death, war, poverty, hatred and so on) then I think we really do need to look at "what is intelligence and does it hit a ceiling?"
Because for humans, it seems that (so far, anyway), the ceiling is around IQ 200+. Out at the fringes, measurements get shaky and are not much, if any, use. Depending on how far back you want to go, be that the emergence of primates, hominids, or modern humans, for hundreds of thousands to millions of years we got smarter and smarter, then hit the ceiling.
Now, that ceiling brought us pretty high and we've done huge things from starting off 300,000 years ago as Homo sapiens to today, where we're arguing over having created an independent intelligence of its own. If AI hits the ceiling where it can't get higher, it could still do immense things beyond even what we imagine today.
But is there a ceiling? One aspect of the argument seems to be "no, or if it is, it is so high we can't even count it". That's the "AI could improve itself to achieve IQ 1,400 levels - or more!" side. God-tier Minds as in Banks' Culture. The "Here comes the Singularity!" side.
I'm more on the "we're going to hit a ceiling soon" side. The S-curve side. Perhaps AI will routinely hit IQ 200 levels, even push that up to IQ 250-300.
That's going to mean enormous changes. But it may not mean the Singularity. It may mean all the dreams of uploading and colonising the galaxy and limitless free energy and every human on the Earth, including the most wretched, poor and miserable in the most horrible, deprived circumstances in the poorest and most neglected regions of the planet, will have guaranteed UBI from the economic surplus so they don't *have* to work and can instead live lives of leisure and creativity and altruism, won't come true. Or it will take 100,000 years to get there, the way it took us that time to get from "working out flint tools" to "create our successor species".
Right. All these arguments tend to assume that recursive self-improvement is some sort of inflection point that allows it to explode to infinity. But is it easier to make me, an idiot, 1% smarter, or to make Einstein 1% smarter? I'd argue it's easier to make me smarter - there's lots of data I don't have, lots of mental frameworks that humanity as a whole knows about but that I haven't been trained in, etc. Meanwhile if Einstein wants to be 1% smarter, he's on his own.
I'm making this up, but I'm assuming that scaling would *also* be exponential. That it would take many times more energy to go from 160 to 161 IQ than it would to go from 100 to 101.
On the other hand, people today who are dumber than Einstein definitely know more about physics than Einstein did, cuz they could just read Einstein.
> "what is intelligence and does it hit a ceiling?"
That's not even the important question. The important question is, "what kind of powers can an agent realistically achieve ?"
A superintelligent box that sits under your desk thinking really hard to itself is totally useless, unless there's some way to communicate with it. Then it will be marginally less useless, but not by much, until there's some way for it to apprehend sensory information. Even then, if you feed this box the sum total of human knowledge then ask it to invent some way to travel faster than light, most likely it will say "nope, not happening". It will say it 1000x faster than a merely regularly-intelligent box, but the answer will be the same. And if you ask it to boil an egg, it will still say "nope" unless you give it some hands to work with (or at least a tentacle). Intelligence is not magic !
In the context of the original post and most sigmoid/curve-fitting talk, we are judging AI progress on METR's task-completion time horizon study. I like the metric and understand why it dominates these conversations (authors know what they're doing and it's a fun and easy to understand graph that backs up our intuitions about AI progress, not many other measures have these qualities) but I don't think it should be so casually mixed up with the big existential questions like the ones you're asking. Measuring how good a model is at completing software tasks is a meaningfully different question from whether it's surpassing human intelligence.
It's not obvious to me that model intelligence and agentic task completion are correlated, past a certain point that LLMs have already surpassed. For all I know, the frontier models could be stagnant on intelligence and just getting more skilled at using agent harnesses (I'm pretty sure intelligence is trending up, just don't know of any research that quantifies it in a reasonable way). Even assuming intelligence and task completion are correlated now, we could conceivably get to a situation where it breaks down. Frontier models could continue getting rapidly smarter and able to tackle theoretical problems that once stumped human researchers, while improvements on task completion benchmarks slow dramatically because we can't think of any new tasks that don't overwhelm the context limit. Or in the other direction, intelligence and research breakthroughs could run out faster than we hope, while task completion keeps getting better with improved agent designs and specialized training.
As an annoying sigmoid guy who doesn't trust AI researchers, I thought I was super wrong. But this explanation implies I was closer to "doing it right" given my priors than I thought. In 2022 I was like "this is a nothingburger," in 2024 I was saying "okay, it outperformed but we'll probably see slowed progress in a couple years," and now I think we're likely to get actual disruptive results and maybe I should trust AI researchers a bit.
To be very very frank with you, the reason I thought the sigmoid approach was good was because writers in the space (and on this blog) kept using arguments like "at this point it surpasses all physical limits and becomes like unto a god" or "and it will be able to trick all human beings everywhere into whatever it wants them to do." These arguments, at least if you haven't read 600 pages of Eliezer Yudkowsky, seem to imply no sigmoid drop-off, ever, unto eternity.
The argument "the ceiling is far above your head" is very commonly misheard as "the ceiling is infinitely tall." I suspect this is a mere lack of perspective, either brought on by or contributing to a lack of humility.
The absolute physical limit is thought to be about six orders of magnitude above human intelligence. Being off by a factor of 10,000 doesn't affect the end result, so the fact that there is an upper limit to intelligence isn't really relevant.
It is obviously true that exponential AI progress will end at some point. The question is „when“ and as it turns out fitting a sigmoid to data is very difficult and easily leads to wildly wrong conclusions.
The real questions are whether there is still potential for improvement and what obstacles will cause a decrease in the growth of the capabilities. Much more interesting, then curve fitting.
Why do you think it's "obviously true"? That's the real question here.
There are physical laws. The upper bound is thought to be about 6 orders of magnitude above human intelligence. From our perspective, there is no difference between that and 2 orders of magnitude, so it's all academic at that point, but we do know of physical limits.
Because if all matter in the universe is arranged precisely in a way in which it’s computational abilities are maximised there really is no where left to go.
The upper asymptote's exact location is unknowable in advance, but its existence is not. That existence, even without knowing the exact location at where it sits, is enough to rule out the inevitability of a certain class of in-vogue AI risk arguments: superpersuasion that can hijack human behavior, nanomachine design from first principles, novel pandemic viruses, and so on.
The usual rejoinder is that these capabilities can be assumed to exist due to recursive self-improvement. The existence of those capabilities are then used to justify radical actions in the present like unilateral pauses in AI development or the stigmatizing of those working at AI labs.
The Lindy principle says that AI capability will probably keep increasing for a while yet. It rules out that superpowers can be assumed to be available to some hypothetical future AI.
> "The upper asymptote's exact location is unknowable in advance, but its existence is not. That existence, even without knowing the exact location at where it sits, is enough to rule out the inevitability of a certain class of in-vogue AI risk arguments: superpersuasion that can hijack human behavior, nanomachine design from first principles, novel pandemic viruses, and so on."
Wait, why does it rule this out? The fact that there's an upper limit to speed (the speed of light) doesn't rule out that something can go at very fast speeds like a million miles per hour.
I would argue that most of these feats -- "superpersuasion that can hijack human behavior, nanomachine design from first principles, novel pandemic viruses" -- are either impossible in principle, just like breaking the speed of light; or theoretically possible but unachievable in practice, like space elevators. Mass mind control is likely impossible; "gray goo" nanotechnology definitely so; pandemic viruses are of course very real but universally lethal uberviruses are incredibly unlikely.
Some other scenarios involve the AI taking control of something then exponentially improving it beyound reason. For example, you might give the AI a robotic e-bike factory and it uses it to flood the Earth with killbots; or you give it access to stock trading apps, and it takes control of the entire world's economy, etc. These scenarios rely on everyone being maximally compliant with AI's goals, including the factory owners who built that factory to sell e-bikes; the human consumers who want to use money to buy bread, etc.; but also the steel and plastic suppliers who provide raw materials for the factory, the power station operators who power the datacenters, etc. This seems unlikely to happen in reality; typically, when some runaway process takes over the normal operations of any facility, that facility gets shut down for repairs.
I think we are not near the end of technological history. The ability to engineer an ubervirus is probably technology for this decade, even without AGI. Mass mind control in the literal sense is interesting to think about, but nowhere near necessary in order to successfully manipulate an appreciable fraction of humanity (which we already know is possible from mundane cases). I don't know about gray goo exactly, but I think It's pretty obviously possible in principle to engineer something like an invasive species of mold that manufacturers prions, which is bad enough.
Probably more importantly, you seem to be underestimating the degree to which humanity is eager to hand over control to the machines. So far, every promise of an appreciable increase in productivity and ROI results in self-disempowerment. Tens of thousands of people are literally in love with AI systems that didn't have any long-term plans to manipulate them. Current AI chatbots already give advice that sounds significantly wiser than most people, and millions of people rely on it. Humans are just obviously not going to put up a fight.
I did not say that human technological development was over; I said that some specific feats of science-fictional technology are likely impossible. More specifically:
* Mass mind-control: Likely impossible. Mass persuasion is of course possible to some degree... assuming you are persuading people to do something they're already mostly willing to do. Making everyone do some completely arbitrary (or even outright harmful !) thing at the same time had historically only worked when backed by threats of extreme violence (e.g. in North Korea), and often not even then.
* Gray goo nanotech: Invasive mold does of course exist, but it could never eat rocks and assemble them into robots overnight.
* Uberviruses: likely impossible due to genetic diversity and the way biology works in general. Regular old viruses are of course possible, but you don't need AI to make them as they already exist.
> Probably more importantly, you seem to be underestimating the degree to which humanity is eager to hand over control to the machines.
Not at all -- of course humans would love to automate away as much work as possible. In fact, IMO this is the real danger of LLMs: they *look* like real AI, so people naturally want to hand off all the work to them; sadly LLMs aren't AI, and end up hallucinating their way off the metaphorical cliff, with disastrous results. Like you said, chatbots put out advice that *sounds* wise, but is just a bunch of vacuous nonsense... which is why people love it. It's like horoscopes: everyone can see in them what he wants to see; but this is also why it'd be impossible to use horoscopes to e.g. get everyone to donate their free time to building your robot factory. But no, what I'm skeptical about is the degree to which it is at all possible to "hand over control to the machines". It's all fun and games until the e-bike factory stops making e-bikes, and then it's back to the drawing board.
Ah, now I see where you are coming from. I don't have time to catch you up, but I can at least fix one common misconception:
The term "Artificial Intelligence" was coined in 1956 by John MacArthur, and it meant the same thing then that it still means to scientists now: roughly, the ability for computers to solve problems that would ordinarily require human intelligence. The term was later used by sci-fi authors to mean what *you* likely mean by "AI," which is something like a sentient computer that is at least as general as a human. This leads to the humorous situation where some people claim that the only "real" AI is the kind that exists only in fiction. On one level, this is merely semantics, but it does warp what people are able to perceive about the actual technology itself. (One can look right at it and genuinely believe that it can't possibly do the things that it is currently doing in front of one's eyes: reasoning, understanding, generalizing, and so on.)
In scientific parlance, LLMs are a subset of transformers, which are a subset of deep learning, which is a subset of machine learning, which is a subset of artificial intelligence. AI is the name of the scientific field from which LLMs emerged, and the crazy sci-fi-sounding things that many people are saying about AI actually ground out in ordinary science being done by scientists publishing papers in their field of expertise.
It rules out the inevitability of them being reached. It is logically unsupported to establish as an assumption that those fantastical properties are a given when talking about future AI - the asymptote of intellectual progress could end up below those capability levels.
This is framed in terms of exponentials vs. sigmoids, but often the framing is in terms of power laws. My favorite paper on the perils of curve fitting is by Mark Newman https://arxiv.org/abs/cond-mat/0412004 — an all time banger that is quite readable even for non-specialists!
I looked at a similar question when I was working at AI Impacts. https://web.archive.org/web/20260215101114/https://wiki.aiimpacts.org/ai_timelines/examples_of_progress_for_a_particular_technology_stopping
Typically, if you look at a particular metric showing technological growth, it will grow exponentially for a few decades, then flatten out for a few decades, then start growing again often at a different growth rate. So I would naively expect something like the METR graph to follow an exponential for 10-30 years, and then stop.
That being said, all technologies are different, and the number of exceptions to the rule are very large. There's plenty of trends that last for shorter times, some that last for longer times, and some that don't look exponential or sigmoid at all. I have seen hyperbolic growth, AI 2027's model, in exactly one dataset (analyzed before I started at AI Impacts): the size of the largest ship in the British navy, from 1650-1867, almost reaching the predicted singularity in 1869. https://web.archive.org/web/20251114181110/https://wiki.aiimpacts.org/doku.php?id=takeoff_speed:continuity_of_progress:historic_trends_in_ship_size
If we take a purely outside-view history-of-technology perspective, then (1) if you think that <5 years of progress at the current rates leads to transformative AI, the trends will probably continue until then, but also (2) if you think that >20 years of progress at the current rates are needed to reach transformative AI, the trends will probably flatten out before then. Neither of these should be particularly confident predictions.
You cannot fit a sigmoid to data unless the data go well past the inflection and into the flattenings on both sides. Curve-fitting can never predict the flattening.
The actual data points on the airspeed graph look nothing like a sigmoid. They provide no evidence of flattening off.
To not just be sounding off, I scraped the data from the Wikipedia page and tried fitting a sigmoid in Matlab. No fit. Matlab couldn't even give me a confidence interval for the parameters. It was perfectly able to do this for synthetic data consisting of a sigmoid plus noise, but as mentioned above, only if the data extended well past the inflection on both sides. Only when you can see the slope and both levellings is it worth even trying to fit a sigmoid.
The equation I fitted was f1(x) = a/(1+exp(-b*(x-c))) + d, which is the logistic equation with four parameters defining location and scale for both axes. There are other S-shaped curves one could try, but this horse was dead on arrival.
A faster technology than ramjets is rockets, which we already have, but they are excluded from the Wikipedia data because of the rules about what constitutes an airspeed record. However, the rules do not limit how fast you can go through the atmosphere. Rockets bound for orbit go faster while still at atmospheric height. Some of them have been crewed (another rule for airspeed records).
I'm not going to look for the raw data from the "First place" METR graph, but just eyeballing it I expect it would fare no better. I notice that they do not give error estimates for their parameters, without which no curve-fitting is meaningful.
If we look at the numbers, energy supply isn't coming online quickly enough to supply new AI data center needs. Data Center energy demand is growing at 15% per year, but new power sources are only growing at 4% per year. Despite all the talk about bringing nukes online to feed the hungry AI data centers, that doesn't seem to have happened yet. This suggests to me that energy production will ultimately be a limiting factor on AI. Of course, there are new chips in the pipeline that use significantly less energy and seem to be much more compute-efficient than GPUs (but I don't know the details). Maybe AI can get smarter on less energy, but for now I think it's safe to assume that there will be a limit to growth (unless someone can offer a convincing counter-argument).
https://www.iea.org/reports/energy-and-ai/energy-demand-from-ai
The problem is that (IMO) the "particular scary level" in question, wherein AI acquires essentially unlimited godlike powers (i.e. becomes "superintelligent") is so far beyound any reasonable estimation and any achievable mechanism (given what we know of the laws of physics today) that *only* unlimited exponential growth could possibly lead to that point. If we assume that this growth will become sigmoid at some point before the Earth is converted into computronium, then ChatGPT will never become a machine-god (actually it will likely never become a machine-god period, but that's another story).
Thus, the counter-argument is not, "I have calculated the precisely correct sigmoid inflection point down to the day, and it just so happens to prevent godlike AGI", but rather, "assuming that our understanding of the laws of physics is mostly correct, exponential growth cannot continue for long enough to yield godlike AGI".
Would AI as general as humans but with 10x the effective IQ not be a godlike superintelligence from our perspective? The physical upper bound is something like 6 orders of magnitude above human intelligence, and I think 1 order of magnitude is obviously fatal.
> Would AI as general as humans but with 10x the effective IQ not be a godlike superintelligence from our perspective? The physical upper bound is something like 6 orders of magnitude above human intelligence, and I think 1 order of magnitude is obviously fatal.
I have no idea where you got those numbers, but the general answer is "no". Imagine that tomorrow I took a magic pill that made me 10x more intelligent than I am now. This sounds spooky, but what does it *mean* ? Does it mean that I can add up numbers 10x faster than I can today ? Probably. Does it mean that I can hover in mid-air blasting people with my laser-eyes ? Probably not. Would I be able to upload my mind into robotic bodies overnight, creating an evil army of robot-Bugmasters (or maybe even clone-Bugmasters) ? Most likely not, and definitely not overnight. Realistically, the best I could do with my 10x superintelligence is maybe embark on some ambitious life-long science project, run for office, or make some money on the stock market. These are all good things (for me), but hardly world-ending threats.
You seem to have mental buckets for "above average human" and "physical impossibility", with nothing inbetween. It's the "between" that I'm worried about.
There is such a thing as being more persuasive than the most persuasive human has ever been, ditto for strategic thinking and scientific thinking. It doesn't have to be by much! There wouldn't have to be just one AI; there can be millions of independent copies. The ability to deploy all of those capabilities at once in a coordinated fashion at scale is enough to easily and quickly displace humanity as the dominant species on the planet, without the use of any additional undiscovered technology.
> There is such a thing as being more persuasive than the most persuasive human has ever been, ditto for strategic thinking and scientific thinking.
I don't know if this is true. What would you expect a superhuman persuader/strategic thinker/scientific thinker to accomplish, other than physically impossible tasks ?
> here wouldn't have to be just one AI; there can be millions of independent copies.
Ah, but how many copies ? So many so that you'd need to convert the entire Earth into computronium in order to run them (in which case it's a no-go) ? Or just a couple more than we've got ChatGPT instances today (in which case it's unlikely to do anything dramatic) ? Or somewhere in between, and if so, what's the right number and how did you know what that number is ?
> The ability to deploy all of those capabilities at once in a coordinated fashion at scale is enough to easily and quickly displace humanity as the dominant species on the planet
Also, which capabilities are you talking about ? I can guarantee you that even if you gave me the token budget to open up 1,000 continuously running ChatGPT instances at the same time, I could not "displace humanity" with them.
>What would you expect a superhuman persuader/strategic thinker/scientific thinker to accomplish, other than physically impossible tasks ?
That depends what it s goals are.
>Does it mean that I can hover in mid-air blasting people with my laser-eyes ?
That's a combination of existing technologies. You could invent the technology . Or take over
I don't think that "ten times as intelligent" is meaningful, we have no sensible numeric scale to put these things on.
AI's don't need godlike powers to be threat to humans, just superhuman ones. Also bear in mind that many scenarios involve AI's taking control of human technology -- we get weaker by the same manoeuvre.
To predict the future growth of artificial intelligence, the best comparison would be the curve of the growth of intelligence in the evolution of humans, which I'm pretty sure was exponential until modern times. (Any changes in modern times are irrelevant because they would be due to humans reshaping their environment to remove selection pressure, a thing no one has yet predicted AI will do, although they might.)
Particular technologies saturate because they are particular technologies. General measurements of capability, independent of the technology used to achieve it, don't, I think. The exponential for speed of communications saturated only because of the speed of light. The exponential for speed of travel hasn't saturated yet (and will go on much longer than for communications, if it's measured in subjective time).
Exponentials are especially likely to arise when there are increasing returns on investment (in some abstract, not-just-economic sense), such as, for instance, intelligence being useful to producing more intelligence. AI is the poster child for curves that won't saturate.
There were obvious reasons to predict that the scaling law of benefits to AI from producing more and more accurate priors would be mathematically precise, and that it would be limited by the exponential of power generation, which is slower, and which we have worked very hard over the past 47 years to sabotage. That has already happened. But there are many other ways to improve AI (and many ways to lower its power requirements by orders of magnitude, tho that is less relevant to my argument). Saying that AI will flatten out short of some energy limit is like saying that the evolution of more-complex organisms will flatten out short of some energy limit. There's no non-ideological reason to think that.
Having said all that, I do expect AI progress in the short-term to saturate, as it's currently limited by the organizational capabilities and operational timescale of humans. Those may prove harder limits than energy production.
Honestly, this debate is a bit beside the point. I know it's about trying to bring the naysayers to heel by demonstrating they don't have much more than "gut instinct" to their predictions that AI won't be a threat, but it misses that AI is already a threat even without AGI. Maybe not in the 'magic infinite bootstrap to godhood' fashion, but in the ordinary 'we stand on the precipice of a monumental shift in human society' way, like the agricultural or industrial revolutions before it. Even if no AI model ever trained in the future is smarter than the current smartest (which very much looks like a losing bet right now), we will continue to advance the size and cost of running that network into smaller and more affordable form factors, and that will profoundly change how society operates.
The current state of the art has yet to fully penetrate into things like military planning and hardware. Ukraine, a relatively tiny country of <40 million, has currently outpaced essentially the whole world on deployment of advanced combat robots, Russian soldiers have been filmed surrendering to Ukrainian robots. How long do you think that will last as an outlier? The USA and China will catch up, and then eclipse, what Ukraine is doing, as humanity gradually bends the full force of industrial manufacturing to AI powered robots. That will trickle down to the police force, which means governments will rapidly gain the capability of true mass surveillance. The issue with 1984 has always been that having a microphone hidden under every bush meant you needed a person listening to every microphone. No longer: today's AI can parse and identify whatever you want out of the noise of the masses, and most of us carry around a microphone in our pocket. We are already seeing the effects of AI generated photos and videos, and the effects they are having on political discourse. When the general public can't tell what's real and what's fake, how do you build consensus? How do you prevent a malignant government from leveraging that to create fake evidence against political rivals, or truly implementing dead internet theory as a propaganda machine? When the court system can't trust photo or video evidence, what does that do to our society? AI has upended the education system, with students relying on current models to output work without having to learn anything themselves. What will that do to the workforce in 5-10 years? What happens when industrial output isn't tied to population count willing to work a dirty dangerous job, because competent-enough general purpose worker-bots with Claude for brains can roll off an assembly line? I don't think anyone knows, but the warning lights are lit up across the board. These things aren't impossible to overcome, but it adds just that much more volatility to the usual chaos of human society, and that always carries with it the risk of apocalyptic war in the form of nuclear, biological or chemical annihilation, plus a whole host of less favorable futures mired in authoritarian repression. We don't need AGI for any of these, we don't need AI to go rogue for any of these, they are issues now, already percolating through the system with the technology we already possess. Ordinary human problems, amplified by intelligence no longer being bottle necked by the willing availability of biological brains in bodies.
The question is no longer if AI will pose a threat to society, it's what will the outcome be.
Oh, I agree that present-day generative machine learning systems are a threat ! It's a threat because it's yet another powerful tool that careless and/or malicious humans can easily exploit; and because the dirt-cheap availability of below-average art, literature, code, and other such things, robs us of the next generation of truly creative artists/writers/etc., thus locking our society into endlessly rehashing the same tired old content. These are all very real problems that I'd very much like to solve; but it seems like every time someone pipes up about them, the conversation gets derailed by someone shouting "none of this matters, the machine-gods are coming to kill us all, destroy all datacenters now !!!111!!".
>>The best way to predict this is to fully understand the process generating the trend.
This is the only comment in this essay that matters. If you don't understand what's actually going on at extreme (=nearly infinite) depth, you can't predict the goddamn future! Please note that this rebuke applies to AI doomers and coomers equally; for new, poorly understood technologies like AI, *literally no one* has sufficient information to accurately predict the future.
Nobody ever has sufficient information to accurately predict the future, the only interesting question is how to quantify our uncertainty, and how to behave in the face of that level of uncertainty. See for example https://www.astralcodexten.com/p/mr-tries-the-safe-uncertainty-fallacy . That's what this post is about.
I would argue that the further out you want to push your predictions in terms of time and scope, and the less you have to work with in terms of understanding the initial conditions, the more uncertain your prediction will be. At some point, the error bars in your prediction will overlap with the more common scenario of "tomorrow will be pretty much like today".
This is the main reason (IMO) why Pascal's Mugging fails. If you want to predict the weather in your town tomorrow, you've got a lot of data to work with, lots of known (albeit chaotic) mechanisms, and a relatively short time horizon. You could still be wrong, but most likely you'll get a prediction that is pretty close to reality. If you want to predict the weather in a year for every town in Sweden, you'll probably fail unless you stick to historical averages. If you want to predict the weather on every square kilometer of every habitable planet in the Milky Way in 10,000 years, then the best you can do is guess, and maybe use present-day Earth as the reference. If you want to do the same, while also assuming that space wizards will be fighting star-dragons in the meantime, all bets are off.
The METR example is going to age really well as a cautionary tale. A formal modeling effort from a credible institution, published, falsified within months by a single model release. If that doesn't update people on the reliability of "it's probably about to plateau" reasoning, nothing will.
Falsified within months of what? The original paper was published over a year ago, and it has held up extremely well. If it holds up for one more year, it will reach the furthest limits of what they dared try to predict.
It is unclear if this graph by Hans Moravec in 1988 is relevant, but it might be.
https://www.zyvex.com/nanotech/images/Moravec.jpg
It is an exponential curve projected out to 2030.
The exponents he drew still seem plausible. Note that "human equivalence" in hardware doesn't imply that we know how to program that hardware.
waow look ma i'm on the big screen !
I don't like the "task length" as a metric anymore. It works fine as a difficulty approximation for up to a few hours, maybe even weeks. But task length has a ceiling at what humans are capable of: a whole human lifespan may not be enough for especially difficult problems (e.g. solving a famous open conjecture in mathematics). A very smart AI will effectively measure at "above infinity" on this scale. And this messes up the usual assumptions that make exponential/sigmoidal good fits. In particular, once AIs get near the human ceiling, even the tiniest improvements in intelligence may show as exponential gains in the "task length" scale.
I think METR aware of their limitations themselves, and have said that measurement is becoming very hard/noisy now that task length has grown. I imagine it gets really hard to set good problems that are expected to take weeks, as well as recruit sufficient humans for establishing the benchmark.
All that said, I generally agree with the post that it's simply way too early to assume we're going to see a plateau. We *might* see some slowdown, and might not live up to the hype or the investment, but I think it's very clear that there's still gains to be had from additional scaling and algorithmic innovation. It took decades of work for chess AI to plateau, decades for go AI to plateau (if it even has, I'm not sure), I don't see why GI shouldn't also take decades to see a plateau.
a) Agreed in general
b) <mildSnark> When I first saw the title, my first thought was on a _much_ smaller scale sigmoid in AI, that the ReLU advocates must have gotten remarkably fierce :-) </mildSnark>
c) Zooming out not quite to the Lindy's Law level, another argument is just: LLMs are a type of artificial neural net. _We_ are neural nets. To suppose that artificial neural net performance must saturate at a level below human performance ignores ourselves as an existence proof for that level of performance from neural nets. Albeit LLMs are only one class of artificial neural net architecture - but the AI labs continually explore alternative architectures too.
d) <mildSnark> If dark energy is constant (maybe yes, maybe no), then the exponential growth in the volume of the universe would be a true, permanent, exponential. Baryonic matter would get rather dilute over time, though. And the exponential's time constant is a bit slow, ~16 billion years. </mildSnark>
I would be interested to see more sigmoid examples / counter-examples that are explicitly about scientific or technical progress - like the flight airspeed and solar deployment graphs. Progress driven by ideas, and to some extent capital input, is a very different beast from fertility rates or epidemic growth.
My own experience as a researcher is that new ideas that meaningfully advance a field are rare. Often they come from the early stages (low-hanging fruit) plus a few key individuals; throwing more researchers and money at a problem mostly produces incremental progress. Hence I am skeptical of AI 2027's recursive self-improvement scenario - can AI researchers really find the rare new ideas needed to keep improving their own intelligence? That is, unless superintelligence is possible with mere incremental progress.
Additionally, exponential growth in <insert technology> != exponential growth in downstream products or human utility. For example:
- Airspeed: Cutting a 6 hour flight in half with a supersonic jet is only a modest improvement to door-to-door travel time.
- Solar panel costs: Solar in the US still costs a bunch due to non-panel costs like installation labor.
- Moore's law: Exponential improvement in computer chips has not exponentially improved my life - subjectively, it's more like a linear trend.
The sigmoids absolutely could save us, and there are strong arguments that they probably will.
I am not giving those arguments in this post, but I'll gesture at what they look like.
I agree with Scott that the simple fact that "all exponentials eventually become sigmoids," however true, does not *by itself* save us. This much is obvious: if a process describe by a sigmoid is harmful, it's perfectly possible for the sigmoid to top out above, not below, the level of great harm.
So any relevant argument must rest on particular estimates.
But more importantly, there isn't a single process involved here ("AI progress"), but many, and they often interact to increase dampening. The fact of processes being sigmoid ends up being relevant, even if not in a simplistic, strawman way.
Interacting sigmoid processes can produce substantially stronger dampening than any individual sigmoid would suggest because their nonlinear sensitivities combine at the system level. In cascades, each stage's limited responsiveness can compound, so a low-gain region in any component suppresses downstream variation. In multiplicative or gating interactions, one weak or saturated-low process can bottleneck the entire output. With negative feedback, coupled sigmoids can create strong stabilization or homeostasis, rejecting perturbations more effectively than any single process alone. Depending on the structure, this may be described as compounded attenuation, gain compression, saturation-induced damping, nonlinear gating, feedback damping, canalization, or robust regulation.
My problem with "the exponential", is that the thing you are measuring doesn't map cleanly to intelligence.
Models have only gotten exponentially better at text based tasks which involve taking in a string and producing a string. But it's actually even more specific, it's taking in a string and outputting a string that solves a quantifiable problem that is somewhat similar to the problems that it was trained on*. So they've gotten exponentially better at math, but only slightly better at poetry (to bring up a recent topic).
That is an impressive feat, but it isn't all of intelligence. For out of distribution tasks, progress has been close to zero. You can do this experiment yourself, have it play a dozen games of a relatively obscure strategy game that isn't in the training data**. Purely text based. They'll do poorly, probably play illegal moves (which is understandable for a first attempt), but unlike a human who will get better with each game, they will barely improve even after a dozen games. They have no way to form new concepts and abstractions in context.
And for multimodal tasks, progress has been very poor for static tasks. They fail at extremely basic things like recognizing an object from different angles or following where a line goes ( https://spicylemonade.github.io/spatialbench/ ). Furthermore the ability to learn and improve at novel visual tasks is precisely zero to due architectural limitations - vision transformers use fixed tokenization. Images are converted into tokens (essentially aggressive compression) and the model can only work with the tokens that it got from that process. It has no way to take a second look as humans can, or think about an image in a new way based on feedback.
My point isn't that progress can't continue, but we do need to be very careful about how we measure progress. If you measure the wrong thing then you'll get the wrong result. To prime intuition - suppose that someone invented a new model that is essentially a full upload of an MIT freshman who hasn't learned to code yet. That model would be AGI by definition, but it would fail METR and other high profile benchmarks. Yet it would have much greater long term potential that any current model.
*And the reason is that this is what they are trained on. Reinforcement learning from verifiable rewards is the single technique that took models from GPT 4 to o1 to GPT 5.5
**if you need ideas https://www.boardspace.net/english/index.shtml
For every one of those sigmoid misidentification charts, I can show you an opposite example from the energy and water resource planning world, where decades of planning documents predict that e.g. urban water demand will continue on some steep upward trajectory, and each time the demands are flatter than predicted. This is so common it has a name: porcupine graphs. There's a an MIT youtube video showing some particularly egregious ones.
But I think it gets back to Scott's post Against The Generalized Anti-Caution Argument: "The lesson is: 'maybe this thing that will happen eventually will happen now' doesn’t count as a failed prediction."
>In conditions of true ignorance, the default assumption should be Lindy’s Law: on average, a process will continue about as long as it’s continued already.
This is not really sound. Lindy’s Law is not a neutral prior nor a maximum-entropy prior which would be a better theoretical fit for conditions of “true ignorance”. Lindy assumes a decreasing hazard rate proportional to elapsed time, which implies the Pareto survival distribution. That may turn out to be the right model for AI trends, but it can't be justified from ignorance.
One could just as easily assume a memoryless process, which yields an exponential survival distribution. In that case, elapsed time provides very little information about expected remaining duration and survival mainly updates your posterior over the hazard rate itself.
And because AI scaling trends are short-lived, one-off, and ambiguously specified, the amount of Bayesian updating you can actually do from the observed duration is fairly limited.
Alternate framing of Lindy’s Law:
“Granting, for the sake of argument, that we’re on a sigmoid curve and not an exponential one, what makes you think we’re on the right half?”
The Wharton model is fair; it had at least one data point pulling the curve down. I don’t see that level of evidence from the average *argumentum ad bulla*.
If we are proposing mechanisms then I propose that the LLM paradigm in particular will eventually run out of steam because it reaches the limit of how clever the training set is.
Imagine an LLM trained on every utterance ever of every child in the world aged four or under. It would be quite capable of saying many things about dinosaurs but it is never going to be able to discuss tensor algebra no matter how much compute or training data you throw at it. The LLM can't get smarter than what's put into it.
I think that could be a factor contributing to AI slowdown but it won't be enough to flatten progress. I expect us to reach the point where AI can start developing new theories on its own and maybe run some of its own experiments.