Beware the AI Experimentation Trap

Earlier this month, the MIT Media Lab/Project NANDA released a new report that found that 95% of investments in gen AI have produced zero returns. The headline is just the latest to feed a wave of skepticism that AI will produce results at scale. The underwhelming launch of OpenAI’s GPT-5 has provided fodder to the view that AI’s progress is slowing. Gartner has suggested that generative AI is entering its “trough of disillusionment” era—the third step in the firm’s five-phase hype cycle framework for technological adoption.

To be sure, the MIT report is actually a bit more nuanced than the headline finding suggests: It argues that while individuals are successfully adopting gen AI tools that increase their productivity, such results aren’t measurable at a P&L level, and companies are struggling with enterprise-wide deployments. Moreover, its authors found that most spending on AI experiments goes to sales and marketing initiatives, despite the fact that back-end transformations tend to produce the biggest ROI.

Even so, headlines like these worry leaders. If 95% of the tens of billions invested in experimentation has failed to produce value, is the effort to experiment with AI a complete waste? On the other hand, how will companies learn how to use these tools without running experiments? How are leaders supposed to interpret these results?

As researchers who study AI and teach about AI transformation and technology, we believe that many leaders are making the same mistake they made a decade earlier with digital transformation: encouraging experimentation, which is good, but falling into the trap of letting experimentation run wild, which is counterproductive. For context, in the previous wave of digital transformation, when many leaders felt confused about digital transformation and the path forward, they embraced innovation and experimentation. Leaders embraced a “let 10,000 flowers bloom” approach, hoping that a few experiments produced unicorn-level returns.

The lack of focus proved to be a blunder, however. Without a clear connection to the real business opportunity—the way to create meaningful value for users—the result was a morass of unfocused, under-resourced teams that produced few scalable results. Facing such disappointing returns, many leaders naturally concluded that experimentation with digital was broken and shut down the experiments. In its place they either returned to business as usual or refocused on a few safer bets: perhaps replacing an aging IT system or a near-term payoff like a digital asset management system.

What went wrong? While experimentation is good, without a connection to the true business opportunity—e.g., transforming the core to serve existing and new customers—experiments inevitably fall short of hopes and expectations. It sounds obvious, but by framing AI as radical and disruptive we often lose sight of the connection to the most fundamental objective of business: to solve problems for customers. The way out of this trap is to 1) understand this AI moment in the larger arc of transformation, 2) focus on AI’s potential to help better serve customers, 3) experiment with a focused set of opportunities to prove them out (with an eye toward scaling), and then 4) scale them up. Here’s how.

Understand AI in the Larger Arc of Transformation

Although the world is talking about AI right now, pull back the frame to recall that AI is a recent conversation that is part of a larger shift. The true change we are all wrestling with is a fundamental shift from digital technology operating at the periphery of organizations (e.g., IT was about laptops, wifi, printing, and IT databases for registry of core activities) to digital at the very core of organizations (e.g., an organization built around digital workflows and customer journeys rather than its own production activities). Said differently, in many senses, every company is becoming a technology company. Rather than people performing tasks based on human judgement and intuition, we are moving to a world of data- and AI-driven decisions, overseen by humans but not necessarily with people as the core engine of the activity.

Consider, for example, how Ant Financial makes lending decisions or Amazon makes pricing decisions with humans only overseeing, not doing, the activity. This is a truly profound shift, and we’re only in the middle of it—in the end, it will take many years but it will lead to a fundamentally different kind of organization. Understanding this bigger picture helps remind leaders that the point is to transform the business to use technology to serve customers better, faster, easier, cheaper, etc. All forms of AI (including gen AI) are just a tool—one of many—that can help accomplish that. Just as the internet fundamentally changed how customers are served, but not why they’re served, AI adoption needs to be seen through this laser-like focus to succeed.

Focus on Serving Customers Better

Recognizing that the point of AI is simply to serve customers better can help leaders focus on what they want to achieve through experimentation. The future is fundamentally uncertain. No one knows exactly where we’re headed and no one can give you a playbook (despite what they may claim). As such, experimentation is essential to figuring out how to use AI to transform the company to serve customers better.

In these moments of strategic uncertainty, where many lack focus and direction, it is tempting to adopt a 10,000 flowers approach. But this scatted approach doesn’t usually address the core need. We see so many leaders swamped by headlines about the miracles of gen AI miss the true potential of AI transformation in exactly the same way they missed the point of digital transformation. The headlines about disruption and revolution (which were similar in tone 10 years ago) create panic and distraction.

Your company isn’t going to become Meta, Amazon, or Google, and that shouldn’t be your goal anyway. Quit stressing about some company you read about that supposedly fired 80% of its workforce to become AI-ready. The real opportunity—the one that will actually generate returns—is to look carefully at your internal operations and the external customer journey and start with how you can create real value, in the near term, using AI tools.

Use this lens to focus your experiments. Start with your strategy, and the value-creating opportunities to either transform a core activity or serve new customers better. We often call this activity the Trojan Horse for AI adoption, because it creates real value today and thereby kickstarts the positive cycle of learning by doing. Then explore the experiments that will prove out the best way to do this. As it turns out, this is one of the key, if under-reported findings of the MIT Media Lab findings: that leaders are pursuing more cosmetic applications in marketing while avoiding transforming the core.

Run Experiments to Prove Value with an Eye to Future Scaling

How should experiments proceed? They should check three boxes: 1) connected to true value creation, 2) as low cost as is possible so that you have multiple cycles to learn and improve, and 3) designed with a connection to how they could eventually scale to create value. This sounds simple but is hard to do in practice. At one end of the spectrum, many leaders charge ahead with experiments without thinking about how they could scale. At the other end are leaders getting bogged down, obsessing about being enterprise-ready from day one.

The magic is to strike a balance: Is this a big enough problem to solve? And, if solved, what would it take to scale the solution? To help you assess which problems to solve among the many that might be proposed, we often use our IFD framework: intensity (how intense is the problem), frequency (how frequently does it occur), and density (how many users or instances of the problem occur). We developed this framework by studying established companies as they attempted to choose which ideas to pursue. For example, an innovator comes up with the idea of developing digital tools to help apartment managers order repair services, whereas another comes up with the idea of a baby monitor to alert parents when a baby stops breathing. How frequently do these customers experience the problems? Occasionally for apartment managers, but parents want to ensure their child is safe every night. How intense is the problem? For apartment managers, probably only of modest intensity since they already have go-to service providers, but the intensity for parents could be very high. Finally, how dense is the problem? There are far fewer apartment managers of the right scale to adopt the tool than parents with children. This gives you a good method to weigh the value of ideas that solve different problems.

Scale Up with a Ninja Team

Once you prove out the value of the experiment, then you can scale it up, but doing so requires careful attention to scaling. Moving something from experiment to scale almost always comes with unexpected new challenges. Someone with power to create change needs to own the initiative, leading a team with time and resources to make change happen. These “ninja” scaling teams—we have interviewed teams at Amazon, Qualtrics, 7-Eleven, and elsewhere—have the air cover (i.e., support from the senior leadership), company-wide connections to get resources, and focus to scale.

As you reach the appropriate scale for the project, repeat the process. Of course, there is more to this to transform at enterprise scale, which includes laying the right digital foundations—a transformation the vast majority of companies have not truly completed. But this process outlines the core machine of the transformation that can help companies avoid the waste and tap into the real value of this fundamental transformation.

. . .

As we enter the post-enthusiasm wave of AI, many leaders are in danger of misinterpreting the challenges implementing AI as a signal that AI cannot be used to create value. They run the risk of falling behind on their AI transformation, just as so many leaders fell behind on their digital transformation (and are now playing catch up). The truth is that AI can create value and we are seeing significant progress—how multi-agent systems can improve back-end operations, for instance. But creating value always comes back to the initial moment of experiment design, when a team is able to see how a new tool can create value for customers, because, no matter what new tools come in the future, the purpose of business will always remain the same: to solve important problems for customers.