Sakana claims Fable-level performance from Fugu. But @LLMJunky showed it can barely make a simple game. Why? Here’s what actually made Fable special, and why Fugu falls short for building apps.
Fugu uses multiple models and agents to improve performance. This works because different models are trained with different data and techniques, leading to different token prediction probability distributions for the same input
Thus, if a model might fail at one task, another might not. If they check each other’s work, you improve the odds of avoiding a mistake. By taking the best of each model’s distribution, you create a bigger distribution that covers more tasks and knowledge
This lets you score better on tests, so you really can compete with Fable on benchmarks like HLE
So what’s the problem? Well, taste isn’t something that emerges from ensembles like this. Design by committee is famously bad
When I think of taste or good design, it’s opinionated and consistent. You have a vision of an aesthetic or feel you’re trying to land, and you align everything around it. One model with one distribution has one set of opinions. Maybe they’re bad (GPT overdoes glass cards, for example), but they’re consistent, predictable, controllable
Model ensembles blur this. They overthink things. One model might influence another sometimes but not others. You get more noise. Dissenting opinions.
Fable was the ideal solution: one unified distribution, but an absolutely gigantic one. The knowledge of multiple models but all brought into one big one. As a result, it could act with internal consistency while scaling to a wider array of challenges
I don’t believe these ensemble approaches will be able to ever achieve that “big model smell” because of this. Big models will continue to have their place in the world. These ensembles can and do add value in cases where correctness matters most. But they’re absolutely the wrong choice for matters of taste and aesthetics, like building an app or game. Stick to the basics for that.
00:29