The architecture behind every modern language model
In 2017, a team at Google published "Attention Is All You Need." The architecture they introduced -- the transformer -- is the foundation of GPT, Claude, Llama, BERT, and essentially every major AI model today.
Before transformers, models processed text one word at a time, in order, left to right. This was slow, and the model would start forgetting earlier words by the time it reached the end of a long sentence.
Read this sentence:
What does "it" refer to? You instantly know it means "the cat" -- not the mat. Your brain figured this out by paying more attention to "cat" than to the other words when reading "it."
Attention is this same idea, turned into a mechanism. When the model is processing one word, it looks at all the other words and decides how much each one matters in the current context. Words that are relevant get high attention; irrelevant words get nearly zero.
Before transformers, models had to pass information word-by-word like a game of telephone. Attention lets every word talk directly to every other word -- no middlemen, no forgetting.
Attention works through three roles that every word plays simultaneously:
When processing the word "it", the query represents what kind of information "it" needs -- it's looking for the noun it refers to.
Every other word advertises what it is via its key. The key for "cat" says "I'm an animal, a noun, the subject of this sentence."
Once a match is found, the value carries the actual content. The value of "cat" provides its meaning to the word that asked for it.
Think of a search engine. You type a query ("what is the subject?"). It matches against keys (every word's description of itself). The results you get back are the values (the actual useful information).
Four steps, no scary formulas:
The output: for each word, a new representation that contains information from the words that mattered most. "It" now carries information from "cat" because "cat" got the highest attention weight.
One set of Q, K, V can only capture one type of relationship at a time. But words relate to each other in multiple ways simultaneously.
Multi-head attention runs several attention operations in parallel, each with its own set of learned Q, K, V weights. Each "head" learns to focus on a different kind of relationship.
Their outputs are combined at the end, giving the model a rich, multi-faceted understanding of how each word relates to every other word.
Modern models use 32 to 128 heads. No one designs what each head learns -- they discover useful patterns on their own during training.
Attention treats all words equally -- it has no idea which word comes first, second, or last. It just sees a bag of words.
But word order changes meaning completely:
Same words, completely different meaning. Without position information, the model can't tell these apart.
Positional encoding adds a unique pattern of numbers to each word based on its position in the sequence. Before any attention happens, each word gets tagged with "I'm word #1", "I'm word #2", etc.
The key idea: the model doesn't "see" word order naturally, so we have to inject it as data.
A transformer isn't one giant structure. It's the same small block stacked many times. GPT-3 stacks 96 blocks. Each block has four components:
Each word gathers relevant information from all the other words. This is where relationships between words are captured.
Rescales the numbers so they don't grow too large or too small as they pass through layers. Without this, deep networks become unstable.
Two linear layers with an activation in between (like the ReLU from Episode 1). Each word is processed individually -- this is where the model "thinks" about what it gathered.
The "+" adds the block's input back to its output. This creates a shortcut so information can skip layers. Without it, very deep networks can't learn -- the signal degrades as it passes through dozens of layers.
After attention gathers context from other words, each word passes through the feed-forward network independently. No word-to-word interaction here -- each word is processed on its own.
It has three steps:
The expansion creates a much larger "workspace" where the model can represent complex patterns. The activation then selectively keeps only the signals that matter. The contraction compresses this back down. It's like spreading out puzzle pieces on a big table to find what you need, then packing the answer back into a small box.
Research suggests the feed-forward layers act as a knowledge store. Individual neurons activate for specific concepts -- one might fire for "capital cities," another for "past tense verbs." Attention finds the relevant context; feed-forward retrieves and processes the knowledge.
Attention already handled the word-to-word relationships. The feed-forward step is each word individually "digesting" what it learned. Think of it as: attention is the group discussion, feed-forward is each person thinking quietly about what was said.
Each word sees all other words -- both left and right. Good for understanding input.
BERT, RoBERTa
Used for: classification, search, entity extraction
Each word can only see words before it -- never future words. Good for generating text.
GPT, Claude, Llama
Used for: text generation, chat, code
Encoder reads the full input, decoder generates the output one token at a time.
T5, original Transformer
Used for: translation, summarization
Encoder = reading comprehension (understand the whole text). Decoder = writing (produce text word by word, never peeking ahead). Encoder-Decoder = translation (read the full source, then write the translation).
In a decoder, when the model is processing word 3, it's not allowed to look at words 4 and 5. This is enforced with a mask -- a grid that blocks certain attention connections.
Why? Because during text generation, future words don't exist yet. If the model could see them during training, it would be cheating -- and would fail at generation time when those words aren't there.
The mask is triangular: word 1 sees only itself, word 2 sees words 1-2, word 3 sees words 1-3, and so on. Each position can only attend to itself and everything before it.
Can word (row) see word (column)?
GPT, Claude, Llama, Mistral -- every major large language model today uses the decoder-only architecture.
They generate text one token at a time. A token is a piece of a word -- roughly 3-4 characters. The model predicts the most likely next token, appends it, then predicts the next, and repeats.
This is called autoregressive generation -- each output becomes part of the input for the next step. It's why you see ChatGPT "typing" word by word.
Generating: "The price of Bitcoin"
At each step, the model sees everything generated so far (but nothing ahead) and predicts the next token. The causal mask enforces this.
Here's how all the pieces connect, from raw text to prediction:
Tokenize -- split text into tokens (sub-word pieces)
Embed -- convert each token into a vector of numbers (its "meaning" in number form)
Add positions -- mix in positional encoding so the model knows word order
Transformer blocks -- pass through N stacked blocks (attention + feed-forward), each one refining the representation
Predict -- project the final representation to a score for every token in the vocabulary. The highest score is the predicted next token.
Attention lets every word look at every other word and decide what's relevant. It uses query, key, value to find and retrieve the right information.
Multi-head attention runs multiple attention operations in parallel, each learning a different type of relationship between words.
Positional encoding injects word order into the model, because attention alone has no concept of sequence.
The transformer block is attention + feed-forward + residual connections + normalization, stacked many times to build depth.
Modern LLMs are decoder-only -- they generate text one token at a time, using a causal mask to prevent looking at future tokens.