The narrow window to distill the world

There is a tight window of opportunity in which it is possible to distill the world, and that window is closing. It therefore needs to be explored fast. In this essay I want to look at that window, particularly through the lens of Bunshin and memory systems. There are a few threads here: introspection and reflection, which today are overtly output-based; the traceability of looped models; and memory, which will be needed even as we move on to world models and world action models.

Memory cannot be replaced

The fundamental claim is that memory cannot be replaced, and that it is needed even for the most powerful AI. This is the whole argument for distributed, differentiated AI. I don't expect any generic AI to be able to adapt to the world without memory of its local environment, because without it, it simply reverts to the mean.

You need disturbances from the mean. You need outliers, and outliers come from unique thinking styles: personalities, unique reasoning styles. On top of that, a personality puts a person in a unique environment with unique experiences, which gives them unique learning points, which in turn drives the flywheel of a unique personality. A person who is really ambitious might push themselves into situations that are risky and hard, and work on very hard problems. Through that process they develop a lot of personality, and they also develop learnings in a specific area that most people would never even try in the first place. It is that initial personality, the willingness to try and to explore really difficult things, that gives them insights and learnings no one else has.

In the future, big general AIs will try to bake much of this into their weights. I think that is possible because of the simulation layers I have written about previously: multiple layers of roleplay and multi-level routing behavior, where the model is roleplaying someone roleplaying someone roleplaying someone. Such a model can roleplay a person with outlier experiences too, as long as those experiences are within the training data. But before that can happen, models need to explore the world and gather those outlier experiences in the first place.

The limits of context

Exploring and understanding the world requires memory, whether you are a large language model or a world action model. The fundamental property of these models is that they use forward passes, they are autoregressive, and they have context. Because of their autoregressive nature, local memory is stored within the context. There are experiments to get memory into the weights on the fly, which is the broad study of continual learning. But in terms of the base model, what everyone is building right now is a simple autoregressive model that outputs tokens, and those tokens live in the context. It is without doubt that the more context you have, the more context rot there is, the more diffuse attention becomes, and the harder it is to bring up the right attention.

There are many layers going on at once. Attention is required to speak English. Attention is required to think, which is why we have reasoning traces that give the model more space and time to reason. And attention is also required to recall things that are very far away. That is a lot for a single attention mechanism to do.

In humans, there is conscious reasoning, and, interestingly, memory recall sits largely outside of consciousness. Yes, there is conscious recall, where you ask me a question and I think about it. But a lot of the time recall is a background process that happens subconsciously. If I think of a solution to a problem because it reminds me of a similar problem I solved in the past, the surfacing of that past problem was not within my conscious processing.

So there is an opportunity for a harness that helps with memory. That is extremely important, because if you want to distill information about the natural world, about outlier situations, about any specific scenario in a company, in a hospital, anywhere, you need this kind of memory system.

The window is closing

This window of opportunity is closing. It is hard to predict when it will close, though there is also a rescue to that closure, which I will come to.

Increasingly, we see companies moving to looped models. OpenAI is the first to start using them. Looped models don't produce a chain of thought, a reasoning trace, as their output. Instead they loop tensors, more like a recurrent neural network than a fully autoregressive model, which makes it really hard to introspect on what they have been doing. If the looped tensors are not saved in context, how do we recall and introspect on why a certain decision was made? Without reasoning traces, how do we know? Even if you asked the model why it made a decision, it might find it difficult to tell you.

As we move to looped models, it becomes harder and harder for models to introspect, to explain why they did something, and to learn lessons from their experiences: where the reasoning was wrong, where it went off course. All of this is important for a good reflective model, because it needs to generate the lessons that will help it answer the next question. As harness-type behavior moves into the model itself, the model becomes more and more opaque. All the reasoning happens inside the model, there are no clear token-based outputs to analyze, and the window to capture and distill from the world narrows.

Memory moves into the model

The same applies to memory. Today our memory systems are augmentations around the model: retrieval-augmented generation, systems for recall, systems for memory capture and reflective capture. I think it is without doubt that these will eventually be merged into the model itself rather than living in an external harness. Rather than the model outputting tokens and calling tools to save memories, imagine a fast-weight module, akin to the hippocampus, connected to the model through tensors rather than tokens.

The benefit is that these tensors can be far higher dimensional than the language space. And because it is a new substrate, it can mold the memory structure and storage into a form native to the model itself: changing the embedding space, molding or folding it in whatever way makes it easier to access what it wants, or whatever makes sense to it.

This matters because we have seen in neuroscience how the understanding of a memory can change very quickly with a single word. Memory is likely input-conditioned and learning-conditioned. You can imagine the entire embedding space being changed and conditioned on tokens. Currently the embedding space is fixed, and we simply project tokens, English words, into it. But a hippocampus-like module that plugs into the model and communicates with it through tensors would allow much more flexible, higher-level behavior: words shaping the context and the embedding space itself, flexible storage, restructuring when new knowledge arrives, a different worldview based on the information you are given and the things you are learning. Memory is fluid and flexible, and theories such as memory consolidation theory and scene construction theory try to describe how it works. What I am pointing at is a token-conditioned memory embedding space.

The problem with a neural substrate for memory is that it is really hard to distill, to understand what has been learned in it. It is a far more flexible substrate, but how do you distill it for frontier model training runs? Today we have token-based continual learning, because tokens are the common substrate across all models. Every model speaks the same language, which makes tokens a good channel for communicating information across model types. A neural substrate, by contrast, is tuned to its particular model: how it was trained, what its weights are. Once memory lives there, it becomes much harder to find a common structure through which to distill the world.

The rescue

Even so, I am optimistic that the window stays open when memory becomes neural.

The quality of that memory will be far higher than token-based memory, because it is an innate neural substrate. It is extremely rich and comes with full understanding. Think about how a human remembers something: the memory is so general and so flexible. That matters because it controls when you recall things.

Currently you have a single embedding space. Whatever your prompt is, you use it to find close similarity, and that doesn't work very well. Using the weights of the model and doing a full forward pass with all the context inside often works better. But what if the prompt itself modified the embedding space? Then, depending on the prompt, what is close to you could change based on the model's understanding of the prompt. I notice in my daily life that the generalizations available from a neural-substrate memory are far richer than anything you can get from a simple token-based memory or a RAG system with a fixed embedding space. The system I currently use for Bunshin is a token-based memory system that relies on the full forward pass of the model, a middle ground between RAG, forward passes, and a fully flexible embedding space.

Because this kind of memory is richer and input-dependent, distilling a person's memory, their rich outlier information, into a bigger model would look different. You would send the tensors, the weights learned in the memory module, into an RL environment and probe them. You could run a fixed test: here is the same task, how do you think about it? With the memory module attached, the model produces flexible, outlier, out-of-distribution outputs, and those outputs become a unique training set. As the big model optimizes toward these outlier outputs, it continuously brings them in-distribution, and you get distinct routing behaviors through which the model learns what kind of profile this memory collector has.

This aligns closely with Bunshin. Bunshin is your shadow clones: clones roaming the world, collecting information, and developing flexible, rich, multi-dimensional, easily molded memories. Those memory weights can then be sent to an RL environment and probed to train models in a controlled setting. So the window can remain open even when memory is a neural substrate.

Why differentiation matters

The entire thesis of Bunshin is that we want to distill the world, and it rests on the belief that you want agent differentiation. We know, from what I have written in previous essays and from what has been found in the legal AI space (Engram's study with Harvey, Understanding a Law Firm through Study), that context changes behavior, and behavior changes outputs and results. If you use a fixed model to explore the world, it will collect a fixed kind of data. But if you allow for agent differentiation, letting different types of agents with different personalities roam the environment, you discover different things. That gives a much richer understanding of the world and a more in-depth distillation of it.

Lessons from reflective practice

My experience in medicine and surgery is one of the best examples of why we need Bunshin, of why we need flexible, reflective agents in the natural world. The GMC and medical schools place enormous emphasis on reflective practice. It is part of the GMC's core Good Medical Practice guidance, because you can never learn enough from textbooks to do well in the real world.

The transition from textbook to real world is jarring, and most medical students encounter it. Your internal distribution of disease, what you understand about a condition and how common each condition is, is often very different from the distribution you see in real life. The treatment plans stated in textbooks differ from how things actually work. There are practical concerns. As John Salvatier put it, reality has a surprising amount of detail, and much of that detail cannot be captured by textbooks because it is just too rich. There is so much interplay between everything that if you tried to record it all, no one would read the textbook, because it would be 50 million pages long.

Take a patient with peripheral arterial disease. The typical symptom is intermittent claudication, pain in the calves on walking, and this patient doesn't have it. From a naive textbook view, he lacks the classical symptom, so perhaps he doesn't have the condition. But look at his other conditions. He has heart problems and lung problems, and he smokes heavily, so he is short of breath. No textbook tells you this, but you can infer it: because he is short of breath, how far he can walk is limited by the capacity of his heart and lungs. They set a lower threshold on his walking than his leg muscles do, which is why he never experiences claudication. He is limited by his heart and lungs first. This kind of interplay makes perfect sense when you reason about it in real life, but it can never be fully captured in a textbook, because there are too many combinations to consider.

Then there are the minute details. There is a set order to a clinical examination. You might ask the patient to take a deep breath in and hold it, but if you ask them to do it too many times in a row, they become breathless and uncomfortable. So in real life you space out the steps, mix in different steps, and reorder them to optimize for patient comfort. You don't learn this from textbooks. No one is going to write a textbook containing all of this, because it is too much detail and might not apply to many situations, so most of it is never written down or captured. Token-based memory can capture it, because the AI is the one reading and using it, and AI can process far more information than a human can in a short time. For humans, this is why reflective practice is so important: you learn these things from the real world, and you need to think back on them and extract as much as you can.

Two axes of world distillation

There are, then, two bottlenecks to distilling the world.

The first is having outlier agents in outlier situations, outlier locations, and outlier times, so that they gather unique data.

The second is maximizing the learning and insight extracted from whatever data you have. This is the core of reflective practice: how much can you learn from a single experience so that it changes your behavior for the better next time? It depends heavily on the in-context learning capability of the model, because you are taking an existing episode and extracting as much useful, generalizable information from it as possible. There is room for deeper training of models that are better at reflective practice: better at inferring and learning from situations, with stronger learning, reflective and introspective capabilities.

The benchmark here is obvious. What you ultimately want is to minimize the number of experiences needed to maximize future performance. A model with very good reflective practice, combined with a memory harness, will improve far faster than one that is slightly worse. So you can benchmark directly on that: with the least data, how fast can the model improve?

Breaking the decompression limit

Humans operate on an episodic, individual basis, and it is hard to distill information from person to person. That is what teaching is, and its output is limited. There is a decompression limit. Senior consultants in medicine like to tell stories about past cases and what they learned from them, an extremely compact distillation of their knowledge. But to get learning at full granularity, you would have to compress loads and loads of episodic memory into another human. No professor gives a lecture spanning an entire year that walks through their whole career and every case they have seen. Some people write books, but most don't.

AI changes that. For models, this kind of transfer is actually possible. That is how we are going to distill the world, and it is why it is only now becoming possible.