Society has long operated on a simple bargain: we contribute our labor and, in return, gain income, security, and a stake in the economy. Governments tax our wages to fund public services; corporations rely on human workers to create value; and in return, workers expect their efforts to be rewarded with opportunity and security. This centuries-old implicit bargain will soon come under strain.

Maybe this is a small detail to focus on, but I often see a problem when people try to tell a story along the lines of "society was stable and harmonious since time immemorial and then this new, disruptive, dangerous technology came along". I see a problem with this story.

Slavery wasn't abolished in the United States until 1866. Two centuries ago would be 1825. So, is anything resembling a fair bargain exchanging labour for "income, security, and a stake in the economy" really "centuries-old"?

North America and other parts of the world also have a history of indentured servitude. 

Even well into the 1900s, workers were treated in a way that was exploitative and violent. For example, Ford's management and security tried to violently suppress union activities, in at least one case killing some of the workers. 

The brief overview of the history of labour under the heading "Revisiting Our Social Contract" also doesn't mention slavery, indentured servitude, or other ugly, violent parts of this history. 

AGI or transformative AI would, in theory, cause fundamental changes to the way society organizes itself around productive labour, capital investment, government revenue, the welfare state, and so on. Yes. This possibility does raise social, political, and ethical questions. Yes. I get that when you're writing an article like this, you often just need to quickly put together a framing device to get the conversation off the ground. 

But this framing device just seemed a little too whitewashed for my taste.

Thanks.

Sorry for replying to this ancient post now. (I was looking at my old EA Forum posts after not being active on the forum for about a year.)

Here's why this answer feels unsatisfying to me. An incredibly mainstream view is to care about everyone alive today and everyone who will be born in the next 100 years. I have to imagine over 90% of people in the world would agree to that view or a view very close to that if you asked them.

That's already a reason to care about existential risks and a reason people do care about what they perceive as existential risks or global catastrophic risks. It's the reason most people who care about climate change care about climate change.

I don't really know what the best way to express the most mainstream view(s) would be. I don't think most people have tried to form a rigorous view on the ethics of far future people. (I have a hard enough time translating my own intuitions into a rigorous view, even with exposure to academic philosophy and to these sorts of ideas.) But maybe we could conjecture that most people mentally apply a "discount rate" to future lives, so that they care less and less about future lives as the centuries stretch into the future, and at some point it reaches zero.

Future lives in the distant future (i.e. people born significantly later than 100 years from now) only make an actionable different to existential risk when the estimated risk is so low that it changes the expected value math to account for 10^16 or 10^52 or whatever it is hypothetical future lives. That feels like an important insight to me, but its applicability feels limited.

So: people who don't take a longtermist view of existential risk already have a good reason to care about existential risk.

Also: people who take a longtermist view of ethics don't seem to have a good reason to think differently about any other subject than existential risk. At least, that's the impression I get from trying to engage open-mindedly and charitably with this new idea of "longtermism".

Ultimately, I'm still kind of annoyed (or at least perplexed) by "longtermism" being promoted as if it's a new idea with broad applicability, when:

1. A longtermist view of existential risk has been promoted in discussions of existential risk for a very long time. Like, decades.

2. If longtermism is actionable for anything, it's for existential risk and very little (if anything) else.

3. Most people are already bought in to caring about existential risk for relatively "neartermist" reasons.

When I heard the hype about longtermism, I was expecting there to be more meat on the bone.

What AI model did you use to generate these summaries?

Looking at the methodology section you linked to, this really just confirms the accuracy of nostalgebraist's critique, for me. (nostalgebraist is the Tumblr blogger.) There are a lot of guesses and intuitions. Such as:

Overall we’d guess that this is the sort of limitation that would take years to overcome—but not decades; just look at the past decade of progress and consider how many similar barriers have been overcome. E.g. in the history of game-playing RL AIs, we went from AlphaGo to EfficientZero in about a decade.

Remember, we are assuming SC is reached in Mar 2027. We think that most possible barriers that would block SAR from being feasible in 2027 would also block SC from being feasible in 2027.

So in this case we guess that with humans doing the AI R&D, it would take about 2-15 years.

Okay? I'm not necessarily saying this is an unreasonable opinion. I don't really know. But this is fundamentally a process of turning intuitions into numbers and turning numbers into a mathematical model. The mathematical model doesn't make the intuitions any more (or less) correct.

Why not 2-15 months? Why not 20-150 years? Why not 4-30 years? It's ultimately about what the authors intuitively find plausible. Other well-informed people could reasonably find very different numbers plausible.

And if you swap out more of the authors' intuitions for other people's intuitions, the end result might be AGI in 2047 or 2077 or 2177 instead of 2027.

Edit: While looking up something else, I found this paper which attempts a similar sort of exercise as the AI 2027 report and gets a very different result.

A trend showing small, increasing numbers, just above 0, is very different (qualitatively) to a trend that is all flat 0s

Then it's a good thing I didn't claim there was "a trend that is all flat 0s" in the comment you called "disingenuous". I said:

It's only with the o3-low and o1-pro models we see scores above 0% — but still below 5%. Getting above 0% on ARC-AGI-2 is an interesting result and getting much higher scores on the previous version of the benchmark, ARC-AGI, is an interesting result. There's a nuanced discussion to be had about that topic.

This feels like such a small detail to focus on. It feels ridiculous.

I don't think it's a good idea to engage with criticism of an idea in the form of meme videos from Reddit designed to dunk on the critic. Is that intellectually healthy?

I don't think the person who made that video or other people who want to dunk on Yann LeCun for that quote understand what he was trying to say. (Benjamin Todd recently made the same mistake here.) I think people are interpreting this quote hyper-literally and missing the broader point LeCun was trying to make.

Even today, in April 2025, models like GPT-4o and o3-mini don't have a robust understanding of things like time, causality, and the physics of everyday objects. They will routinely tell you absurd things like that an event that happened in 2024 was caused by an event in 2025, while listing the dates of the events. Why don't LLMs, still, in April 2025 consistently understand that causes precede effects and not vice versa?

If anything, this makes it seem like what LeCun said in January 2022 seem prescient. Despite a tremendous amount of scaling of training data and training compute, and, more recently, significant scaling of test-time compute, the same fundamental flaw LeCun called out over 3 years ago remains a flaw in the latest LLMs.

All that being said... I think even if LeCun had made the claim that I think people are mistakenly interpreting him as making and he had turned out to have been wrong about that, discrediting him based on him being wrong about that one thing would be ridiculously uncharitable.

That seems like a non-sequitur and it seems like a calculated insult and not a good faith effort to engage in the substance of my argument.

I disagree with what you said about ARC-AGI and ARC-AGI-2, but it doesn't seem worth getting into.

I think OpenAI revenue is, on average, more aggressive than Moore's law.

I tried to frame the question to avoid counting the revenue or profit of AI companies that sell AI as a product or service. I said:

the ability of AI systems to generate profit for their users by displacing human labour.

Generating profit for users is different from generating profit for vendors. Generating profit for users would mean, for example, that OpenAI's customers are generating more profit for themselves by using OpenAI's models than they were before using LLMs.

I'd guess that LM ability to automate intellectual work is more aggressive than Moore's law, too, but it started from a very low baseline, so it's hard to see.

I realized in some other comments on this post (here and here) that trying to compare these kinds of things to Moore's law is a mistake. As you mentioned, if you start from a low enough baseline, all kinds of things are faster than Moore's law, at least for a while. Also, if you measure all kinds of normal trends within a selective window of time (e.g. number of sandwiches eaten per day from Monday to Tuesday increased from 1 to 2, indicating an upward trajectory many orders of magnitude faster than Moore's law), then you can get a false picture of astronomically fast growth.

Back to the topic of profit... In an interview from sometime in the past few years, Demis Hassabis said that LLMs are mainly being used for "entertainment". I was so surprised by this because you wouldn't expect a statement that sounds so dismissive from someone in his position.

And yet, when I thought about it, that does accurately characterize a lot of what people have used LLMs for, especially initially in 2022 and 2023.

So, to try to measure the usefulness of LLMs, we have to exclude entertainment use cases. To me, one simple, clean way to do that is to measure the profit that people generate by using LLMs. If a corporation, a small business, or a self-employed person pays to use (for example) OpenAI's models, for example, can they increase their profits? And, if so, how much has that increase in profitability changed (if it all) over time, e.g., from 2023 to 2025?

(We would still have to close some loopholes. For example, if a company pays to use OpenAI's API and then just re-packages OpenAI's models for entertainment purposes, then that shouldn't count, since that's the same function I wanted to exclude from the beginning and the only thing that's different is an intermediary has been added.)

I haven't seen much hard data on changes in firm-level profitability or firm-level productivity among companies that adopt LLMs. One of the few sources of data I can find is this study about customer support agents: https://academic.oup.com/qje/article/140/2/889/7990658 The paper is open access.

Here's an interesting quote:

In Figure III, Panels B–E we show that less skilled agents consistently see the largest gains across our other outcomes as well. For the highest-skilled workers, we find mixed results: a zero effect on AHT [Average Handle Time] (Panel B); a small but positive effect for CPH [Chats Per Hour] (Panel C); and, interestingly, small but statistically significant decreases in RRs [Resolution Rates] and customer satisfaction (Panels D and E).

These results are consistent with the idea that generative AI tools may function by exposing lower-skill workers to the best practices of higher-skill workers. Lower-skill workers benefit because AI assistance provides new solutions, whereas the best performers may see little benefit from being exposed to their own best practices. Indeed, the negative effects along measures of chat quality—RR and customer satisfaction—suggest that AI recommendations may distract top performers or lead them to choose the faster or less cognitively taxing option (following suggestions) rather than taking the time to come up with their own responses. Addressing this outcome is potentially important because the conversations of top agents are used for ongoing AI training.

My main takeaway from this study is that this seems really underwhelming. Maybe worse than underwhelming.

Unless I misinterpreted what Steven was trying to say, this supports my observation in the OP about insularity:

There were a number of people, all quite new to the fields of AI and AI safety / alignment, for whom it seems to have never crossed their mind until they talked to me that maybe foundation models won’t scale to AGI, and likewise who didn’t seem to realize that the field of AI is broader than just foundation models.

How could you possibly never encounter the view that "foundation models won't scale to AGI"? How could an intellectually healthy community produce this outcome?