Pretraining Lives Between Evolution and Learning

We keep comparing large models to the wrong things.

Sometimes they are framed as students, absorbing lessons. Sometimes as autocomplete engines, remixing text with impressive timing. In a recent conversation with Dwarkesh Patel, Anthropic CEO Dario Amodei offered a cleaner frame. Pretraining, he argued, is neither human learning nor simple memorization. It occupies a strange middle territory between biological evolution and individual learning.

That sounds abstract until you follow the consequences. If the analogy is right, several arguments about AI limits need to be rewritten. The complaints about data hunger look different. The debate over continual learning loses some of its force. Even reinforcement learning starts to look less like a separate discipline and more like the same machine, pointed at a wider range of environments.

The analogy problem

Human intelligence is built in layers that arrived on very different timescales.

At the deepest level, evolution shaped the brain before any of us were born. It handed us a body plan, sensory systems, instincts, reward circuits, and a huge pile of cognitive priors. Above that sits long-term learning, the months and years during which a child acquires language, social models, motor skills, and durable knowledge. On top of that comes short-term adaptation, the fast updating we do in a conversation, a classroom, or a new job. Then there is pure reaction, the almost reflexive layer that answers before reflection fully arrives.

Models have a stack too, but the pieces do not line up cleanly with ours. Pretraining looks deeper than ordinary learning, because it builds general capabilities from a nearly blank starting point. Reinforcement learning often sits in the same band, refining broad behavior rather than teaching a single explicit fact. In-context learning lands higher up, closer to the rapid adaptation humans do in a situation.

That mismatch matters because we usually argue by forcing an equivalence. We ask whether pretraining is “really learning” in the human sense, and then act surprised when the answer feels unsatisfying. The more useful question is what sort of process it actually resembles. Amodei’s answer is unusual but clarifying: pretraining is a synthetic process that compresses some functions of evolution and some functions of learning into one optimization run.

Humans arrive preloaded

A newborn is not a blank slate in any meaningful engineering sense.

Human infants do not know calculus or tax law, but they inherit an absurd amount of structure. Vision is already specialized. Attention is biased. Social inference comes unusually easily. Language acquisition does not begin from random weights in the poetic sense people like to use; it begins from a brain shaped by millions of years of selection pressure. Nature spent a very long time debugging the substrate.

This is why human sample efficiency is easy to romanticize. A person can learn to use a spreadsheet after a short tutorial, but that moment sits on top of a giant stack of previous competence. The person already understands objects, goals, instructions, symbols, causality, error correction, and usually the social fact that somebody wants a task completed before lunch. Watching a human learn Excel and comparing that directly to a base model is like comparing the speed of a Formula 1 pit stop to the time required to build the car.

Rich Sutton’s The Bitter Lesson made a related point from another angle. General methods that leverage compute tend to beat clever hand-built tricks over time. That argument is often treated as a triumph of brute force. It is better understood as a reminder that much of what looks like elegant intelligence is really expensive structure paid for somewhere else. Biology paid earlier. Machine learning often pays during training.

Models compress evolution into training

This is the part of the frame that changes the feel of the whole debate.

A large language model begins from random initialization. It does not inherit visual cortex, motor routines, or an evolved bias for face detection. It has an architecture, an optimizer, and a learning objective. Then we pour in vast amounts of data and compute. From that starting point, the model has to acquire broad statistical structure about language, facts, concepts, plans, and increasingly the latent shape of the world those words describe.

Seen this way, trillions of training tokens are not simply evidence of waste. They are payment for missing priors. The model is doing, inside one training regime, some work that biology split across two stages: species-level adaptation and lifetime learning. It is less sample efficient than humans on many tasks, certainly. But the comparison is unfair if we ignore the invisible inheritance humans bring to every example.

That does not mean the gap is illusory. Humans still dominate in several forms of efficient learning, especially when embodiment and real-world feedback matter. A child can infer a surprising amount from a handful of interactions. Models often need a mountain of gradient updates. Still, Amodei’s taxonomy gives a better explanation than the usual one, which says machines are merely memorizing while humans genuinely understand. Blank slates have to be trained longer.

Sample inefficiency looks different in this frame

Critics often make a fair point with the wrong underlying story.

If a model needed billions of dollars of compute to become useful on office software or web navigation, that seems like a profound defect. If it possessed the same core learning algorithm humans use, surely it would pick up such skills faster. Patel pushed on that tension in the interview, echoing a concern many researchers share.

The best response is not denial. Models are less sample efficient in important ways. The response is to ask efficient relative to what starting line. A human learning a new app is not learning from zero. The human is transferring an entire civilization’s worth of priors packed into a brain, a body, language, culture, and years of lived experience. The model is transferring from pretraining, which can be formidable, but the substrate below it remains much more generic.

That helps explain why long context windows feel so powerful. Once a model is pretrained, giving it a million tokens of local situation can unlock impressive adaptation. It can read the documentation, inspect the codebase, infer conventions, and often operate competently inside that temporary world. That starts to resemble medium-term human learning far more than the old “stochastic parrot” caricature suggests. The bird has apparently learned to grep.

Reinforcement learning is tracing the same path

One of the more interesting claims in Amodei’s recent discussions is that reinforcement learning is beginning to display the same broad pattern pretraining showed earlier.

The history of language models offers a useful comparison. GPT-1 trained on a relatively narrow corpus and generalized weakly. GPT-2 trained on a much wider scrape of the internet, and suddenly the model felt qualitatively different. It was not just bigger. The distribution widened, and generalization improved.

Reinforcement learning may be entering a similar phase. For a while, much public attention went to narrow domains: board games, simulated control, math competitions. Impressive, yes, but often specialized. The claim now is that RL performance continues to improve in a roughly log-linear way with more training across a broader set of tasks, including coding and other open-ended work. If that holds, the important variable may not be a magical new RL trick. It may be the breadth of the task distribution.

This is where the old fight between “clever algorithms” and “scale” becomes less dramatic than people want. Techniques still matter. Architecture choices matter. Stability engineering matters a lot, especially if you would prefer your training run not to become a bonfire. But once a regime is found that scales, the center of gravity shifts. The marginal breakthrough often comes from more compute, more data, wider environments, better objectives, and the numerical plumbing required to keep the system trainable.

The blob still explains too much

Amodei has described a private 2017 intuition he called the “Big Blob of Compute” hypothesis. The phrase sounds like a joke a tired researcher would write at 2 a.m., which probably helps. The substance is more serious. Strip away the folklore, and progress depends heavily on a short list: raw compute, data volume, data distribution, training time, an objective that keeps rewarding scale, and enough stability tricks to let optimization proceed smoothly.

It is tempting to hear that as contempt for invention. That would be too simple. Transformers were not irrelevant. Better optimizers were not irrelevant. RLHF and related post-training methods were not irrelevant. But in hindsight, many celebrated innovations acted less like new engines and more like better fuel lines, steering systems, and road surfaces for a vehicle whose horsepower kept rising.

There is an uncomfortable implication here for a lot of AI commentary. We love stories where one conceptual key unlocks intelligence. They are neat, cinematic, and easy to tweet. The industry reality often looks messier. Once you have a scalable recipe, progress can come from making the training distribution broader and the optimization budget larger. It feels inelegant because it is.

Continual learning may not be the missing piece

A lot of people assume systems will remain fundamentally limited until they can learn continuously from experience the way humans do.

That may still matter for some domains. Personal assistants should remember you. Robots probably need durable adaptation in the physical world. Organizations will want models that accumulate context without retraining from scratch every time a policy changes. Those are real demands, not philosophical decoration.

Even so, continual learning may not be the gate many people think it is. In software work, the codebase itself already acts as externalized memory. An agent does not need to permanently absorb every repo it touches if it can load the relevant files, issues, tests, logs, and docs into context, then reason over them effectively. A good chunk of “learning on the job” turns out to be retrieval plus adaptation, not deep weight updates.

That is a broader pattern than coding alone. Humans also offload memory into notebooks, search engines, colleagues, calendars, diagrams, and institutions. The mind has always been partly outside the skull. Models with large context windows and strong retrieval may be closer to that arrangement than critics admit. Persistent weight changes remain useful, but they may be more of a performance optimization than a prerequisite for broad competence in many cognitive tasks.

The research agenda shifts

If this taxonomy is even half right, it changes what counts as the bottleneck.

The core challenge stops looking like “discover the one missing ingredient that turns pattern matching into intelligence.” It starts looking more like “assemble training processes broad enough to substitute for more of the priors biology gave us for free.” That means wider data distributions, richer task environments, scalable objectives, longer horizons, and interfaces that let models pull large amounts of situational detail into working memory.

It also sharpens the real uncertainties. We still do not know how far text-heavy pretraining can carry world modeling without stronger embodiment. We do not know whether current scaling trends will bend sharply on tasks requiring persistent agency, long-term planning, or robust causal interaction with the physical world. We do know that several allegedly clean boundaries have already blurred. Syntax was supposed to be easy and semantics hard. Then models got better. Reasoning was supposed to require a different mechanism. Then scaled systems started showing fragments of it. Mathematics was supposed to expose the facade. Then reinforcement learning and tool use changed the picture again.

The practical bet emerging from Amodei’s framing is not that every problem disappears into larger training runs. It is that many supposed category errors were really taxonomy errors. We kept asking whether machines learn like humans, when the more interesting possibility is that they learn through a process humans never had: a synthetic fusion of inherited structure and acquired skill, created inside optimization itself. If that is the right frame, progress will come less from discovering a mystical new faculty and more from widening the training distribution until more of the world can fit inside it.