Value Generalisation 2: The Missing Hole in AIs’ abilities

Summary

A human superpower hidden from even ourselves I though GPT 3.5 was on the verge of Artificial General Intelligence (AGI). It certainly seemed that way – it could combine and extend ideas in ways that were far beyond narrow rigid computing. Sure, it had some flaws, but with its general capabilities and humans working tirelessly on improving the model, these flaws would soon be resolved and a full AGI would be born. It didn’t turn out that way. We are still where we were back in 2022 – as in, it feels we are on the verge of AGI, but not there yet. I think that humans have an ability, or a mix of abilities, that I’m calling strong generalisation. And LLMs and their derived models lack strong generalisation. This would explain some of the mysteries about LLMs: The fact that subject matter experts get much more out of an LLM than an amateur. The LLMs’ strange mix of ability and stupidity. Why they need so much data and constant retraining and tweaking. Why they always seem on the verge of being AGIs while never reaching it. There’s a fifth mystery, which at first blush seems like it’s actually an LLM strength: 5. The amazing ability of LLMs to succeed on almost any benchmark, including long-horizon task benchmarks. Why is this a mystery and problem rather than a strength? Well, it’s maybe the strongest counterargument to the bitter lesson and the scaling argument. Because these models saturate long-horizon benchmarks without being AGIs and without being able to transpose this performance to the real world. Thus, whatever it is they’re learning, it isn’t generalisation. So the mystery, rephrased: Why do LLMs not learn long-term planning and generalisation while succeeding on benchmarks that purport to teach them those abilities? Strong Generalisation From one perspective, strong generalisation is a bit of a bucket category – it’s a mix of situational awareness, out-of-distribution generalisation, symbol grounding, adaptive world modelling, long-range planning, hierarchical-planing, anomaly detection, and a few other abilities. From another perspective, it’s precisely one thing: the mix of abilities that allow humans to work and plan in novel situations, pursuing objectives without losing track of what’s going on. It’s essentially an adaptive set of abilities, finding the right mental tool for the job and no more. And it’s mainly a semi-instinctive process, not amenable to direct conscious oversight. Which means that we don’t really understand how we do strong generalisation, we don’t know how to code it[1]up, and we don’t recognise its absence or presence in other entities. So I model LLMs as devices lacking strong generalisation but trained on vast amounts of data and capable of remixing this data in non-trivial ways. Human engineers push them ever further, but never fill the central hole. From this perspective: Subject matter experts provide the strong generalisation, using their understanding of the problem to keep the LLM on track. LLM abilities are spiky; but their main stupidity is that their failures are generally obvious. Why did the LLM break here, or hallucinate there, or go off the rails at this point? If they lack strong generalisation, then their failure does not happen at the moment of failure; it’s because the LLM is attempting a task that is impossible to it, so if it hadn’t failed at this point, it would have failed soon after. Since they lack strong generalisation, they can’t infer very far beyond their training data. So they need vast amounts of this training data to plug the holes they would otherwise have in their abilities. Since humans don’t consciously perceive our strong generalisation, we don’t detect its absence in other entities. So in an LLM we see a vastly knowledgeable and articulate entity that seems able to do, in principle, anything. So when LLMs do fail we pay extra attention to the specific failure mode, thinking “that’s a stupid mistake, it will soon be patched, and once it doesn’t make those mistakes, it will truly be generally intelligent”. Since we can’t define strong generalisation or its various components (like long-term planning) the best we can do is produce examples (benchmarks) that illustrate it. And LLMs will succeed on these benchmarks using the simplest pattern they can find, which is not strong generalisation. Engineers have been pushing LLMs to go further, using tricks like chain-of-thought and reasoning agents (which can be seen as hoping that the secrets of strong generalisation can be found in human textual reasoning), training on yet more data, getting a lot of examples of successes and failures from interactions with clients, and so on. They construct harnesses, which are essentially designed restrictions and guide-rails for the LLM, constructed with care to make the LLM powerful and reliable within the confines of the harness and the task. And, of course, LLMs will always have great power at solving problems that might seem new to some, but are actu