Distilling the world with the invisible hand

Real-World Feedback and the Distillation Bottleneck

This essay follows what was discussed in the previous essays about the physical world bottleneck (On Time Complexity in Technological Revolutions and Mapping Opportunities in the Hyperspeed Timescale). We are seeing this happen right now with AI achieving massive breakthroughs in mathematics that exceed everything humans have accomplished previously. As a next step, people are trying to tackle the things AI is not yet good at and apply AI across more domains in the world. Many areas of life and the economy still cannot be handled by AI. In a software company, for example, you need to understand the customers, interact with customer-facing teams, and gather customer needs and feedback. Customer feedback is real-world data that guides the product, alongside business parameters like revenue, engagement, and churn rate. All of these metrics are real feedback from the world, and they belong to a time-locked domain that cannot simply be sped up. You can speed up how fast you build the company, ship new features, and write code, but knowing what to build is contingent on having the right real-world data.

Because of this, there will be a massive distillation effort to bake these signals and intuitions—knowing what works and what does not—into the models. User interface design is a clear example. Claude and other frontier large language models are already very good at generating UI designs that look nice, but how do we know if those designs actually work? Ultimately, UI design is not about making pretty interfaces; it is about achieving a desired behavioral outcome in the human using the UI. If you want to upsell a product offering, you need to design the interface so the user is most likely to click and pay more. That requires an in-depth understanding of human psychology and behavior that is not yet captured by AI models. While AI can make good-looking interfaces, it does not yet have a strong grasp of human behavior and psychology because there is very little training data capturing that link.

How do real human designers do this? Human designers have the benefit of personal embodied experience—years of using interfaces themselves and building an intuition for how humans behave. Beyond personal experience, schools of thought in UX design have been distilled from decades of human research and psychology into core principles that designers follow. Gathering that knowledge through real-world interaction and experimentation is inherently slow, which is why behavioral science took so long to develop. On top of that existing knowledge, companies also generate on-the-job data as real customers use their apps and provide live feedback on their designs. Current AI models are not trained on this data and remain weak at this layer.

If we want AI to get better at these tasks, we need to bake all of this knowledge and feedback into the model. This is where the idea of distilling the world comes in. We want to distill every part of human society, human behavior, and human experience into these models so that in the future they can one-shot a UI that achieves near-100% conversion or dramatically reduces churn. Frontier AI companies are currently approaching this by creating reinforcement learning (RL) environments with verifiable rewards, collecting human feedback, and harvesting agent traces where humans critique a design. That human feedback draws directly upon decades of UX research and lived experience, embedding that information into the AI through training environments.

As highly theoretical, self-contained fields like mathematics and software engineering are increasingly replaced by AI, there is a push to make AI permeate all parts of society. I call this the slow diffusion process of AI. Hyperscalers often build financial plans assuming AI will immediately excel at everything by extrapolating cleanly from its mathematical abilities. In reality, automating the rest of the economy requires real-world data, real-world experience, and strong data collection systems.

The Invisible Hand and Human Swarms

This slow distillation process plays out across physical domains as well. Working as a barista in a coffee shop involves physical manipulation that embodied robots still cannot reliably perform. Earlier attempts to control robotics with deterministic systems fell short of human fluency, which has sparked the current push into embodied intelligence, humanoid robotics, and world models aimed at distilling the physical world.

As AI replaces math and software engineering, people will migrate into the areas where AI is still weak. This is the invisible hand at work: the economy reallocates human resources wherever automation falls short. If your job is automated, your time is freed up. Software engineers used to spend 90% of their time writing code, reviewing pull requests, and merging changes. With a strong software factory system, time spent writing code drops to zero. People temporarily spend more time reviewing code, but in the future no one will review code either—they will simply verify that the final result works. Engineers then move toward architecture design and product discovery, and whatever AI cannot automate becomes the new bottleneck.

With bottlenecks emerging across the world, a massive shift in resource allocation follows. If all digital work is fully automated while physical work remains the bottleneck, physical labor becomes extremely expensive. Millions of fast AI agents end up waiting on physical tasks that they cannot perform themselves, driving a surge of human workers into physical roles. Moving more people into those bottleneck fields increases data collection in those exact domains, because every worker acts as a proxy collecting real-world data. Over the next few years of slow AI diffusion, workers in bottleneck fields will effectively serve as data-collection proxies, generating the training data needed for models to eventually master those tasks.

The invisible hand drives this cycle because money flows wherever the bottleneck is. Supply and demand raise prices in constrained domains, reorganizing the labor market to pull more people in. If you picture the global labor market as blobs of millions of workers across different industries, what emerges is a human swarm. Workers stream into a bottlenecked field to capture high wages, and once their work completely distills that field into AI, the swarm disperses and moves on to the next un-automated frontier.

This cycle will persist for a long time as we distill every remaining pocket of human ability. As I wrote in The AI Squeeze, this trajectory points toward a world where there is eventually nothing left to distill from the human experience. Today, distilling the world mostly means distilling human experience, because humans are the primary economic actors sustaining society. If distillation eventually displaces enough of the labor market that most people no longer have jobs—living on universal basic income or pushed to the margins while sustained by AI output—AI itself becomes the primary actor in the production of goods and services. At that point, fully embodied AIs navigating the world will shift from distilling human experience to distilling the natural environment directly: learning about materials, resources, and the laws of the universe to embark on projects like sending robotic probes to the stars, continuing the process of learning from the real world that humans have carried out throughout history.

The Timescale Pyramid

This brings us to two connected ideas: the pyramid of faster and slower timescales along with its economic implications, and how Bunshin—a system I have been building—fits into this world.

Even in a fully agentic AI world, some real-world processes take a long time while others run much faster. Consider randomized controlled trials for cancer therapies. Building the physical systems and T-cell manufacturing pipelines can take months, followed by a long patient recruitment process. Even after patients receive treatment, they must be monitored for weeks or months, making feedback inherently slow regardless of AI acceleration. Clinical trials are bottlenecked by the limited number of eligible human patients willing to participate and by physical logistics such as setting up trial centers across different countries.

One step down from human trials sit human organoids, such as melanoma organoids used to test drug responses. Because organoids can be grown in a production pipeline, a lab can maintain a continuous stream of organoids on demand and massively parallelize the process. While growing an organoid is still slower than running a digital world model—where you can speed up time simply by spending more compute—it is a major step faster than a clinical trial. Even if each organoid experiment takes four weeks to mature, starting a new batch every hour four weeks in advance yields a continuous pipeline of fresh results every single hour. With far fewer real-world logistical dependencies, everything happens inside a scalable lab environment, and the same pipeline principle applies to mouse models.

Going another step further down brings us to basic chemistry research: testing how receptors bind to one another and manufacturing proteins or antibodies in vitro to see how they react before deciding which drugs to test on organisms. Because there is no organism growth required, receptors and antibodies can be mass-manufactured and mixed together for near-instant results, operating on an even faster timescale.

These timescale nuances exist within the same domain, and every layer that can be optimized for maximum acceleration will be. Expanding basic science research makes every subsequent organoid experiment much smarter, and massively parallelized organoid and mouse pipelines make every human clinical trial far more precise and accurate. Together, these layers form a pyramid. Because human clinical trials at the top are the slowest, hardest to recruit for, and most expensive, capital flows down the pyramid into the faster steps that can optimize and de-risk the top—flowing all the way down to pure mathematics, which is completely self-contained and the cheapest layer to investigate as compute gets cheaper.

This pyramid aids in distilling the world because experiments are scaled to the difficulty of data collection, extracting maximum information at each timescale. It is another example of the economic engine optimizing the acceleration of science and medicine. Although my background biases me toward software and medicine, the same pyramid structure applies to sociology and politics—such as modeling how populations think, react to policy, and place trust in governments. Pyramids exist everywhere, and the key variable is how steep the pyramid is: how many intermediary steps exist to support the final expensive step.

World models are a powerful new intermediary layer in this structure. Where we previously relied on rigid deterministic models, world models provide high-fidelity simulated environments to experiment on, making the final physical step less expensive. There is a massive opportunity to build new intermediaries that add layers to the pyramid and flatten its slope. A gradual pyramid achieves hyper-efficiency by distilling knowledge across many cheap layers to hyper-optimize the tip, whereas a steep pyramid leaves the expensive final step with little theoretical support, causing it to progress slowly.

We can model the speed of AI's diffusion across society by mapping all the intermediary steps in a field from pure math to the final product and measuring the steepness of its pyramid. The gentler the slope of the pyramid, the faster that field will be distilled and replaced. Connecting this back to the human labor swarms moving from bottleneck to bottleneck, every field has a half-life before it is fully distilled, and that half-life is governed by the steepness of its pyramid.

Weights, Context, and Bunshin

In an economy tuned toward distilling the world, Bunshin is my effort to capture this opportunity within software engineering (building on Continual Learning Through Introspection). Distilling the world requires extracting as much information from real-world work as possible so AI can learn from it. While an individual builder does not have the capital to train frontier models and bake knowledge directly into model weights, we can distill that knowledge into the context window.

There is an important architectural difference between putting knowledge into context versus baking it into weights. Putting knowledge into weights creates a new level of abstraction. In context, you get the benefit of attention and MLP layers combining token representations, while the fixed weights determine how tokens project into Query and Key spaces—acting much like a hypernetwork conditioned on the context across layers. The central question is when in-context learning is sufficient versus when fine-tuning or RL weight training is required, and how deep the simulation overhead of context can go in terms of meta-reasoning.

Models are very good at roleplaying a persona when given a role in context. However, can a model act as someone who acts as another who acts as another, or simulate a large language model running another large language model? Because of how the attention mechanism works, there is a limit to how many nested meta-layers a model can simulate in context before it breaks down and requires weight updates. Tokens and weights are fundamentally different: token-based simulation has a ceiling, even if that ceiling rises as base models grow larger.

In software engineering, the end goal is to give an agent a task and have it solve the entire problem in one shot without hand-holding. Today, agents fall short because there are real-world details about the codebase and environment that were never in their training data. A human provides feedback during a session so the agent adapts, but when a new session starts, the agent forgets everything. Tools like Claude have introduced memory files, indexing, and background "dreaming" passes to address this.

Bunshin builds on this foundation for a single engineer running 30 agents in parallel. Because running interactive play with the environment is the primary way humans generate value in the singularity, a tool should deliver immediate utility to the user while simultaneously distilling their work. Bunshin reflects on past sessions, extracts learnings, and compiles them into structured memory files. It then uses fast, hot-cached models like Haiku to retrieve and serve the exact memories needed for each new task. Rather than relying on brittle deterministic RAG, using the full weights of a small model for retrieval makes it possible to serve the right memory at the right time so the agent can one-shot the task without repeating mistakes.

Every time human intervention is needed, it marks a situation the model has not yet distilled. Linking back to the timescale pyramid and human swarms, any point requiring human steering is a goldmine: it signals exactly what the model still lacks and what needs to be saved into memory, with contradictions resolved through the dreaming consolidation pass. When an engineer runs 30 agents and all 30 finish autonomously, that workflow is already distilled; whenever human intervention is required, the memory system captures that gap.

At the bottom layer, individual humans and their agent swarms distill knowledge locally into memory files so their agents improve immediately. At the same time, those memory files and agent traces can be used to build RL environments that eventually push that knowledge into model weights—which is exactly what frontier labs like Anthropic are doing today, and why static memory files will eventually be absorbed into future base weights.

Latency vs. Capability

Continual learning is now playing out across multiple scales. At the company scale, organizations that want agents specialized for their own codebase distill internal traces and fine-tune open-source weights. At the frontier scale, everyone using Claude feeds traces back to Anthropic to be baked into future general models.

A natural question is how a single generalized frontier model handles contradictory practices across thousands of different companies and users. Over time, frontier models will learn higher-order abstractions for how different archetypes of companies operate, using internal microcircuits to switch modes automatically. Through meta-roleplaying, the model infers what kind of company it is operating in from minimal cues and activates the right internal abstractions to one-shot the task. Whenever a human still has to step in and steer the model, that feedback indicates that the model's initial archetype routing was off, tuning the meta-layers until no human intervention is needed.

This sets up a clear timescale question: from the moment a distillation artifact is created, what is the latency before the model can use that learning again?

1. Memory Systems (Context): Feeding distilled memory directly into the context window has a latency of roughly one minute—the time needed to reflect on a session, consolidate the learning, and serve it into the next prompt.
2. Company-Scale RL and Fine-Tuning: Baking learnings into local weights requires massive trace volume—such as 500 employees each running 30 agents—to build RL environments and run training, resulting in a latency of weeks to months on a weaker open-source base model.
3. Frontier Model Runs: Big AI labs distilling traces from the entire world build the strongest models with deep weight abstractions, but their feedback loop takes months or years and focuses first on generic capabilities—like generic coding, generic medicine, or video generation—leaving company-specific workflows dependent on prompting in the meantime.

This creates a fundamental latency-versus-capability tradeoff. Token-based learning gives you near-zero latency and rides on top of the smartest frontier base model at cheap inference cost (avoiding the heavy VRAM overhead of storing weights and gradients during training), but it is bounded by the model's in-context roleplaying ceiling. Local RL puts knowledge into weights faster than frontier labs, but locks you into a weaker base model. Frontier training runs give you both massive scale and weight-level abstractions, but at the longest latency.

Where each approach wins depends on organization size. A 500-employee company accumulates so much distilled knowledge that stuffing it into context causes context rot and retrieval inefficiency, making weight training worthwhile even if the base model's raw roleplaying ability is strong. For individuals and smaller organizations, however, local knowledge fits comfortably below the context and retrieval ceiling—and as frontier models grow larger, their roleplaying capacity may expand fast enough that smaller teams never hit that cap.

GP Clinics, Small Businesses, and Surface Area

Bunshin operates on the token-based side of this spectrum, which applies far beyond software engineering. Consider a General Practitioner (GP) clinic. Every clinic has its own workflow for emailing coordinators, reconciling medications, requesting consultant reviews, informing hospitals, and scheduling patients. A generic model can draft a polite email, but knowing who to email and how a specific GP clinic runs requires workflow distillation.

Computer-use agents make this distillation possible because they operate existing software directly through the screen without custom integrations. As clinic staff and doctors use computer-use agents to handle emails and medication orders, every human correction distills the clinic's specific processes.

Even more importantly, doctors diagnose patients using real-world priors. When a doctor critiques an agent for weighting differentials incorrectly, they are supplying local epidemiological knowledge. The distribution of diseases in medical textbooks over-represents rare, heavily researched conditions while under-representing mild, everyday illnesses. A local GP knows which viral illness is circulating in their town during that specific week and season—priors that textbook-trained models lack and can only acquire by distilling real clinical practice.

Across the broader economy—whether running a waffle shop, working as a barista, or managing a small or medium-sized business—organizations need a lightweight distillation channel. Big AI labs distill at global scale, and enterprise RL companies distill at the 500-person company scale. Bunshin targets individuals, GP clinics, mom-and-pop shops, and SMBs that simply want a frontier model to adapt to their local workflow within a minute while keeping their data locally owned.

Because healthcare faces heavy regulatory friction, the fastest initial wedge is in less regulated workflows like shops and everyday business operations. Maximizing distillation requires getting as many people as possible to use AI in their daily work, which is why frontier labs like Anthropic deploy forward-deployed engineers and consultancy services into enterprises: the goal is to maximize surface area and penetration across real-world workflows. That is the core metaphor behind Bunshin—from Kage Bunshin no Jutsu in Naruto, where shadow clones spread out across the world to gather experience and distill it back to the main body.

The Great Mining of Human Intelligence

We are entering an era of massive data and compute scaling where AI is eating the world. Where models hit hard stops in the physical world, economic incentives will reallocate human labor and capital to those bottlenecks and build new intermediate layers to flatten every industry's timescale pyramid.

What happens to humans after that remains an open question. It is possible that distilling human intelligence will resemble drilling for oil: just as we repeatedly discovered deeper reserves of fossil fuels whenever we thought we would run out, the depth of human tacit knowledge, judgment, and experience may be vastly underestimated. Distilling the world is the great mining of human intelligence, and we may keep discovering deeper layers of human capability that take decades to extract.

Moving into the singularity will be a long road with real regulatory and societal challenges, and policy choices will shape how far and how fast this distillation goes. At the same time, building continuous learning loops across software engineering, robotics, and everyday workflows has enormous potential for good. Ultimately, the winners in this market will be whoever expands their surface area fastest—integrating AI into real workflows so that every moment of human steering continuously signals what to distill next.