Reflection on Different Neural Network Architectures
From Bigram to Transformer
7/19/20263 min read
The core goal of LLM (Large Language Model) is simple: auto-completes text. It tries to predict the next word based on the words that came before it. To do this, we train the model on a massive amount of text data so it can discover and learn the patterns of language. While the goal is always the same, the neural network architectures we use to learn these patterns have evolved drastically over time. Let's look at four fundamental structures using the example sentence: "The world is so beautiful"
Bigram Model
The Bigram model is the simplest approach. It looks only at the single previous word to predict the next one.
How it works: To predict "world", it only looks at "the". To predict "is", it only looks at "world".
The verdict: It takes far too little context. Human language is rich and complex. It often depends on relationships spanning multiple words, so relying on just the immediately preceding word makes accurate prediction extremely difficult.
Multi-Layer Perceptron (MLP): The Fixed-Window Glance
MLP uses a fixed pre-defined context window to predict the next word.
How it works: If our context window is 4, the model takes "the", "world", "is", "so" all at once to predict "beautiful".
The verdict: Using more context improves prediction, but it introduces two major problems. First, increasing the context window requires a larger input layer, causing the number of model parameters to grow quickly. Furthermore, because it flattens and crams all words into the network at once, every input position connects to a different set of weights. As a result, the model repeatedly learns the same grammatical patterns at different positions, making learning less efficient and increasing the risk of overfitting.
Wavenet: The Hierarchical Step-by-Step
Instead of swallowing the whole context window at once, Wavenet processes the information gradually and hierarchically.
How it works: it groups consecutive words, summarizes the pattern, and then passes the summary up to the next layer. Layer 1 learns the pattern of ("the" and "world") and ("is" and "so") in parallel. Layer 2 combines those two summaries and learns the patterns. Layer 3 predicts "beautiful".
The verdict: By sharing model weights across different groups, it efficiently captures repetitive language structures (e.g., verb-after-subject) without bloating model size or causing overfitting. It captures much longer contexts without dramatically increasing the number of parameters. However, information still has to travel through multiple layers before distant words can influence one another. As sequences become longer, modeling long-range dependencies remains challenging.
Transformer
Transformer trains the model with a context window ranging from 1 to the predefined window. It allows words within a context window to talk to each other simultaneously, and uses the information the last word gained from all previous words and its own information to predict the next word.
How it works:
Imagine our context window is 4; we will have 4 training data points for this example. The model trains on "the" to predict "world", "the" and "world" to predict "is", "the", "world", "is" to predict "so", and "the", "world", "is", "so" to predict "beautiful".
To achieve this, words are converted into content embeddings and combined with positional embeddings so the model knows the order. Then the model projects the embeddings to three new sets of vectors (Query, Key, and Value). The Query represents the information the word searches for. For example, the query of "the" looks for a noun. The Key represents the information the word contains. For instance, the key of "world" advertises itself as a noun. The Value contains the information that the word wants to communicate to other words if Query and Key match.
The affinity score represents the fraction of information each word takes from previous words. It is calculated by using the dot product between Key and Query with the upper triangle masked, and then scaled and normalized by softmax.
The weighted information is added back to the original embedding (known as the residual connection). It allows the gradient to flow back without vanishing for stable model training.
The output flows through a feedforward network (MLP) to process these rich, contextualized representations and learn the pattern, followed by another residual connection. This whole process forms a single transformer block. Modern LLMs stack dozens of these blocks back-to-back to build ultra-deep language understanding.
The verdict: It is worth noting that the Transformer can use the same training data structure as the Bigram. With X being the context of a pre-defined window ["the", "world", "is", "so"] and the target is one position shifted ["world", "is", "so", "beautiful"]. We can calculate all context lengths simultaneously using triangular matrix multiplication. The weight matrices W_Q, W_K, W_V project each position into Query, Key, and Value spaces using the same projection weights at every token position (same for feedforward weights). But the affinity score is calculated dynamically per input.
Model serving: we will need to store 1) Content and position embeddings; 2) Key, Query, Value weights as well as output projection weights; 3) Feedforward weights; 4) Layer Norm weights; 5) Decoding layer weights.