An intuitive understanding of LLM
A different cognition, a lossy simulator.
9/6/20264 min read
Back in school, we learned most of our knowledge through textbooks. First, we learned the expositions: the background knowledge, concepts, and theories. Then came the worked examples, where we saw problems and their solutions and learned standard ways of approaching them. Finally, at the end of each chapter, we faced practice problems on our own. During this process, we used what we had learned to solve new problems. We tried different approaches, learned from our mistakes, and gradually developed our own strategies for solving problems.
Surprisingly, this learning process is similar to how LLMs are trained. We can roughly think of it as: Expositions → Worked Examples → Practice Problems corresponding to: Pretraining → Supervised Fine-Tuning → Reinforcement Learning.
Learning through expositions corresponds to the pre-training stage of an LLM. The open web data are fed to the LLM, and it learns the statistical patterns found in the training data. Specifically, it goes through a sequence of updating so that its predictions match the statistics of what actually happens in the training set, and its probabilities become consistent with statistical patterns of how these tokens follow each other in the data. Through next-token prediction, the model learns statistical regularities in the training data, building internal representations of language, concepts, relationships, and other structures that help it make better predictions. These representations are encoded in the weights of the neural network. You can think of the LLM as a super-powered machine that has been exposed to an enormous collection of textbooks about almost everything. It learns the statistical patterns of the knowledge and stores its knowledge acquisition in the weights of its neural network.
The learning through worked problems corresponds to the post-training stage of Supervised Fine-Tuning (SFT) of an LLM. The model from the pretrained stage is fed curated instruction-response pairs produced by human labelers. Unlike the open web data, these data are curated by human labelers, who follow a given instruction to provide a well-rounded answer to the question, just like the solutions we have for the worked problems. The model learns the conventions of dialogue, how to follow instructions, and how to structure answers clearly. It effectively learns to simulate the answer that a human labeler would provide. The pretrained model gives it broad knowledge. SFT teaches it how to use that knowledge in a conversational setting.
Once a student gains knowledge and understands standard methods, true competence comes from working through practice problems independently. For an LLM, this corresponds to post-training via Reinforcement Learning (RL). Specifically, for problems with verifiable answers (eg: coding, math), we can run the LLM thousands of times for a given problem and reward it when it provides the correct answer. By repeatedly optimizing for this reward, the LLM can discover the most effective trajectory to generate the right answer. What's particularly interesting is that the model may discover strategies that were not explicitly demonstrated during supervised fine-tuning. The model is still generating tokens one at a time, but RL can encourage sequences of tokens that correspond to useful intermediate steps. To put it in another way, the process of RL promotes the LLM to come up with its own strategy to solve the problem.
The similarity between humans and LLMs doesn't stop at training. There is also an interesting parallel when they are actually solving problems. For humans, what we have learned becomes part of our long-term memory. When we solve a problem, we also rely on working memory. We hold the relevant information in our minds while reasoning about it. For an LLM, what it has learned is stored in its weights. The context window is the LLM's working memory. The model's weights provide its learned prior knowledge, while the context provides the information currently available to the model. Better context often leads to better answers. This is one reason techniques such as retrieval-augmented generation (RAG) can be so useful: instead of trying to put every piece of information into the model's weights, we can provide relevant information through the context at inference time.
Despite the similarity between human and LLM learning processes. The cognition of humans and LLMs is actually different.
First, LLM is a lossy simulator. The pretrained model is a token simulator. The post-trained SFT model is a human labeler simulator. At every stage, information is compressed into the model's parameters. The model doesn't store the original training experience. Instead, it learns a function that tries to reproduce useful patterns from that experience. This is fundamentally different from how humans learn.
Additionally, LLMs learn through tokens. We experience the world through language, vision, sound, touch, physical interaction, social relationships, emotions, and many other forms of experience. The pattern of tokens an LLM learns might be different from what humans learn.
Besides, the LLM needs tokens to think. The LLM imitates at this token level with only these specific computation weights for every single token and sequence. It's much harder for the model to predict the final answer directly than to build up the answer step by step. Each generated token becomes part of the context for the next prediction. In this sense, generating intermediate steps provides additional computational space. While as human, we have many ah-ha moments.
Lastly, RL is particularly powerful when the problem has verifiable answers. In reality, we have lots of problems with unverifiable answers. RLHF (a lossy simulator of actual human preference) is an approximate RL. It is learning to optimize against a statistical and computational approximation of what humans want.
So, do LLM actually think? I don't know. Perhaps the most useful way to think about an LLM is not as a student that learns exactly like us, but as a simulator that has been progressively optimized. First, it learns to simulate the patterns of open web text. Then, it learns to simulate high-quality human responses. Finally, reinforcement learning pushes it toward behaviors that score well according to a reward function. And from this process, something remarkable emerges: A system trained to predict and optimize over tokens can develop behaviors that look surprisingly like knowledge, reasoning, and even thinking.