How Transformers Actually Work: From-Scratch Walkthrough (2026)

Q: What is self-attention in plain terms?

Self-attention answers, for every token, the question: 'which other tokens matter most to understanding me?' Each token creates a Query, Key, and Value; a token's Query is matched against every other token's Key, and the resulting weights mix the Values. In 'the cat sat on the mat because it was tired', 'it' attends most strongly to 'cat'.

The Big Picture: What a Transformer Does

A transformer takes a sequence of tokens and outputs a probability distribution for the next token. That's literally it — and it's why the "just predict the next word" framing is both accurate and underrated. To predict well, the model must learn grammar, facts, reasoning patterns, and style — all as statistical structure. The architecture has three conceptual stages:

Tokenize — text becomes numbers (BPE explained here).
Embed — each token becomes a vector; position gets added.
Transform — N blocks of attention + feed-forward progressively refine the representations, and the final vector predicts the next token.

Step 1: Tokens → Vectors

# Conceptual: token IDs → embedding vectors
import torch

VOCAB, D_MODEL = 32000, 768
embed = torch.nn.Embedding(VOCAB, D_MODEL)

tokens = torch.tensor([42, 17, 9031, 5])   # "the cat sat on"
x = embed(tokens)      # shape (4, 768): one vector per token

# Position matters — add a positional signal (RoPE in modern LLMs)
# (simplified): x = x + positional_encoding(seq_len)

The embedding table is learned: during training, tokens that behave alike end up with similar vectors. "cat" and "dog" end up closer than "cat" and "tax". This is the same machinery that powers semantic embeddings — just trained for prediction rather than similarity.

Step 2: Self-Attention — the Core Idea

For every token, attention answers: "which other tokens should I pay attention to, and how much?" Each token builds three vectors:

Query (Q): "what am I looking for?"
Key (K): "what do I offer?"
Value (V): "what do I actually contain?"

def attention(Q, K, V):
    # Q, K, V: (seq, d) — scores: how well each query matches each key
    scores = Q @ K.T / (K.shape[-1] ** 0.5)   # scaled dot product
    weights = torch.softmax(scores, dim=-1)   # probabilities
    return weights @ V                        # weighted mix of values

# In "the cat sat on the mat because IT was tired":
#   IT's query strongly matches CAT's key → CAT's value dominates

The scaling (√d) keeps the softmax from saturating. The result: every token's new representation is a weighted blend of all tokens' values — context in the literal sense. This is why long context matters and why the KV cache (storing K and V) exists.

Step 3: Multi-Head Attention

One attention pass learns one kind of relationship. Multi-head runs several in parallel, each with its own projections — one head tracking syntax, another pronouns, another positions:

class MultiHeadAttention(torch.nn.Module):
    def __init__(self, d_model, n_heads):
        super().__init__()
        self.heads = torch.nn.ModuleList([
            AttentionHead(d_model, d_model // n_heads)
            for _ in range(n_heads)])

    def forward(self, x):
        return torch.cat([h(x) for h in self.heads], dim=-1)

The outputs of all heads concatenate back into the full width. It's the same compute, organized — and it's why "multi-head" appears in every architecture discussion, from inference to MoE designs.

Step 4: The Transformer Block

class TransformerBlock(torch.nn.Module):
    def __init__(self, d_model, n_heads, d_ff):
        super().__init__()
        self.attn = MultiHeadAttention(d_model, n_heads)
        self.ff   = torch.nn.Sequential(          # feed-forward
            torch.nn.Linear(d_model, d_ff),
            torch.nn.GELU(),
            torch.nn.Linear(d_ff, d_model))

    def forward(self, x):
        x = x + self.attn(norm(x))   # attention: communicate
        x = x + self.ff(norm(x))     # feed-forward: think
        return x

The two sub-layers have distinct roles: attention is where tokens exchange information (the communication layer), and the feed-forward is where each token processes what it learned independently (the computation layer). The residual connections (x + ...) give gradients a highway — which is what makes hundreds of stacked blocks trainable. A 7B model stacks ~28 such blocks with 28–32 heads.

Why This Architecture Won

Property	What it enables
Parallel training	Whole sequences train in one step — the scale that RNNs couldn't reach
No distance decay	Token 1 can attend to token 100,000 directly — long-range dependencies
Stackable	More blocks = more abstraction layers; scale just works
Predictable scaling	Loss falls smoothly with data + compute — the basis of the scaling laws that drive 2026's model releases

And the honest downside, directly from the architecture: the model is predicting tokens, not verifying facts. Everything this blog does — RAG, evaluation, grounding — exists because a next-token predictor, however powerful, needs external structure to be trustworthy.

From Understanding to Building

Build one: the mini-GPT in raw PyTorch tutorial implements every piece here, tokenizer to training loop.
Understand the vocab: BPE tokenizers are the front door of the whole pipeline.
See it in deployment: the architecture's memory profile is exactly what KV cache, quantization, and speculative decoding optimize.
Skip the transformer entirely: the LFM family exists because some teams bet the attention architecture isn't the only path.

Frequently Asked Questions (FAQ)

What is a transformer?

A transformer is the neural network architecture behind modern LLMs, introduced in the 2017 paper "Attention Is All You Need". Its core innovation is self-attention: every token in a sequence can directly attend to every other token, in parallel, which lets models learn relationships across long texts far better than the recurrent networks they replaced.

What is self-attention in plain terms?

Self-attention answers, for every token, the question: "which other tokens matter most to understanding me?" Each token creates a Query, Key, and Value; a token's Query is matched against every other token's Key, and the resulting weights mix the Values. In "the cat sat on the mat because it was tired", "it" attends most strongly to "cat".

Why is attention better than RNNs?

RNNs process sequences step-by-step, so long-range dependencies decay and training is inherently sequential. Attention lets every token see every other token in one parallel step — no distance decay, and the whole sequence trains in parallel, which is why transformers scale so dramatically with data and compute.

What does a transformer block contain?

Each block contains two sub-layers: multi-head self-attention (the communication layer, where tokens exchange information) followed by a feed-forward network (the computation layer, where each token thinks independently), each wrapped with residual connections and layer normalization. LLMs stack dozens to hundreds of these blocks.

Why is multi-head attention multi-head?

One attention pass learns one kind of relationship (syntax, reference, proximity). Multiple heads run parallel attention with different learned projections, so the model can track several relationship types simultaneously — one head tracking grammar, another tracking entity references, another tracking position.

Do transformers understand language?

No — and that's the honest answer. They learn statistical patterns over tokens: which tokens follow which tokens. What looks like understanding is extremely sophisticated pattern matching at scale. This matters practically: it's why grounding (RAG), verification, and eval are necessary, and why hallucination happens.