train-llm-from-scratch

![main image](https://cdn-images-1.medium.com/max/5200/1*r99Hq3YBd5FTTWLNYKKvPw.png)

<div align="center">

<!-- omit in toc -->
# Train LLM From Scratch

![Python](https://img.shields.io/badge/Python-3.10%2B-blue) [![CI](https://github.com/FareedKhan-dev/train-llm-from-scratch/actions/workflows/ci.yml/badge.svg)](https://github.com/FareedKhan-dev/train-llm-from-scratch/actions/workflows/ci.yml) ![License](https://img.shields.io/badge/License-MIT-green) ![Contributions](https://img.shields.io/badge/Contributions-Welcome-blue) [![Docs](https://img.shields.io/badge/Docs-Available-success)](https://fareedkhan-dev.github.io/train-llm-from-scratch/)

**I am Looking for a PhD position in AI**. [GitHub](https://github.com/FareedKhan-dev)

</div>

I implemented a transformer model from scratch using PyTorch, based on the paper [Attention is All You Need](https://arxiv.org/abs/1706.03762). You can use my scripts to train your own **billion** or **million** parameter LLM using a single GPU, or a small one on a laptop CPU with no GPU at all.

This started as a pretraining tutorial. It now goes all the way from raw text to an aligned, reasoning style model, with every algorithm hand written in plain PyTorch (no `trl`, no `peft`, no `transformers`). The whole journey is one idea repeated: turn text into numbers, predict the next token, then keep changing the data and the loss until the model does what we want.

Next to the original model there is now a modern one, built the way current open models are (rotary positions, RMSNorm, SwiGLU, grouped-query and latent attention, Mixture of Experts, a KV cache), and every stage can train either. The newer training tricks are here too, each written out by hand: the Muon optimizer, LoRA, IPO and SimPO, Dr. GRPO, DAPO and GSPO, speculative decoding and int8 weights.

![From raw text to an aligned reasoning model](images/00_pipeline.png)

Here is the path we will walk, end to end:

```
raw text  ->  tokens  ->  a Transformer  ->  next-token loss  ->  a base model
base model  ->  SFT  ->  Reward Model  ->  {PPO, DPO}  ->  GRPO  ->  evaluation and chat
```

Below is the output of a trained 13 million parameter LLM, just so you can see where the small end of this starts:

```
In ***1978, The park was returned to the factory-plate that
the public share to the lower of the electronic fence that
follow from the Station's cities. The Canal of ancient Western
nations were confined to the city spot. The villages were directly
linked to cities in China that revolt that the US budget and in
Odambinais is uncertain and fortune established in rural areas.
```

<!-- omit in toc -->
## Table of Contents
- [Who this is for](#who-this-is-for)
- [Prerequisites and Training Time](#prerequisites-and-training-time)
- [Setup](#setup)
- [No GPU? Train on Your Laptop](#no-gpu-train-on-your-laptop)
- [Code Structure](#code-structure)
- [Step 1: Preparing the Data](#step-1-preparing-the-data)
- [Step 2: The Model, Built From Small Pieces](#step-2