close

DEV Community

Taha hussein
Taha hussein

Posted on

I Built a Mini-GPT From Scratch to Actually Understand Transformers (Not Just Copy Code)

#ai #python #datascience #machinelearning

**# Full Post Content (Markdown, ready to paste into DEV.to editor)

A few weeks ago, I asked an AI assistant to build me a poetry-generation model using a Transformer architecture. It worked. The output was decent. But when I read through the code, I could only really explain about 80% of it.

That bothered me more than it should have.

So I went back to the fundamentals: Attention, Query/Key/Value, Multi-Head Attention, positional encoding — and rebuilt everything from scratch, line by line, without copying from the original code. No shortcuts this time.

The result is a small, fully working GPT-style model, and a step-by-step video series (in Arabic, with English code) documenting the entire build.

What the series actually covers

Instead of jumping straight to a wall of code, I broke the build into small, focused pieces:

  • Embeddings: turning word IDs into vectors, and why we need a separate positional embedding on top of the token embedding
  • Self-Attention: building Head from scratch — the Q/K/V projections, the scaled dot-product, causal masking, softmax, and the final weighted sum
  • Multi-Head Attention: running several attention heads in parallel and merging them with a projection layer
  • Feed-Forward + Residual Connections: why attention alone isn't enough, and why skip connections matter for training stability
  • The full MiniGPT class: stacking everything into blocks, adding the final LayerNorm and lm_head
  • Training loop: computing cross-entropy loss, running backprop, and actually watching the loss go down on a tiny toy dataset

Here's a snippet from the core attention head, one of the most important 15 lines in the whole project:

class Head(nn.Module):
    def __init__(self, head_size):
        super().__init__()
        self.key = nn.Linear(n_embd, head_size, bias=False)
        self.query = nn.Linear(n_embd, head_size, bias=False)
        self.value = nn.Linear(n_embd, head_size, bias=False)
        self.register_buffer("tril", torch.tril(
            torch.ones(block_size, block_size)))

    def forward(self, x):
        B, T, C = x.shape
        k = self.key(x)
        q = self.query(x)
        v = self.value(x)
        wei = q @ k.transpose(-2, -1) * C**-0.5
        wei = wei.masked_fill(
            self.tril[:T, :T] == 0, float("-inf"))
        wei = F.softmax(wei, dim=-1)
        return wei @ v
Enter fullscreen mode Exit fullscreen mode

Nothing fancy — no external attention libraries, no shortcuts. Just nn.Linear, matrix multiplication, and a triangular mask to prevent the model from "cheating" by looking at future tokens.

Why rebuild something that already works?

Because there's a real difference between:

  • "I can read this code and follow the logic"
  • "I can write this from an empty file and explain every decision"

The first gets you through a tutorial. The second is what actually sticks when you're debugging a production model six months later, or when someone in an interview asks you to explain why attention scores are scaled by 1/sqrt(head_size).

The video series

The full series (18 short episodes, 3–7 minutes each) walks through this build in Arabic, but all the code, variable names, and comments are in English — so it should be followable even if you don't speak Arabic.

📺 https://www.youtube.com/@Tahahussein-Ai

What's next

Now that the Transformer fundamentals are solid, I'm moving on to tokenization (BPE) and then into fine-tuning real pretrained models (LoRA/QLoRA) — building on this same "understand it deeply, then apply it" approach.

If you've ever felt like you understand a model "well enough" but couldn't quite rebuild it yourself, I'd genuinely recommend trying this exercise. It's humbling, but it's the fastest way I've found to close that gap.

Suggested tags: #machinelearning #python #ai #tutorial #showdev

Top comments (0)