1+ <p align =" center " >
2+ <img src =" docs/figures/banner.png " alt =" Text-LLM-Training-from-scratch banner " />
3+ </p >
4+
15# Text-LLM-Training-from-scratch
26
37A from-scratch implementation of the language-model training pipeline in PyTorch, covering
@@ -14,6 +18,30 @@ selection is centralized in a single module, so the same code runs on CPU, Apple
1418
1519Author: [ y0oshi] ( https://github.com/Y0oshi ) . License: MIT.
1620
21+ ## Table of contents
22+
23+ - [ System overview] ( #system-overview )
24+ - [ Requirements] ( #requirements )
25+ - [ Installation] ( #installation )
26+ - [ Quickstart] ( #quickstart )
27+ - [ Two interfaces] ( #two-interfaces )
28+ - [ Example: a 17M-parameter model] ( #example-a-17m-parameter-model )
29+ - [ How each stage works] ( #how-each-stage-works )
30+ - [ Tokenizer: byte-level BPE] ( #tokenizer-byte-level-bpe )
31+ - [ Data pipeline: memory-mapped token shards] ( #data-pipeline-memory-mapped-token-shards )
32+ - [ Model: a decoder-only Transformer] ( #model-a-decoder-only-transformer )
33+ - [ Generation: the key/value cache] ( #generation-the-keyvalue-cache )
34+ - [ Alignment] ( #alignment )
35+ - [ Supervised fine-tuning with prompt masking] ( #supervised-fine-tuning-with-prompt-masking )
36+ - [ Reward model] ( #reward-model )
37+ - [ Direct Preference Optimization] ( #direct-preference-optimization )
38+ - [ GRPO with verifiable rewards] ( #grpo-with-verifiable-rewards )
39+ - [ Serving and evaluation] ( #serving-and-evaluation )
40+ - [ Hardware and scaling] ( #hardware-and-scaling )
41+ - [ Repository layout] ( #repository-layout )
42+ - [ Tests] ( #tests )
43+ - [ License] ( #license )
44+
1745## System overview
1846
1947The pipeline proceeds in four phases: data preparation, pretraining, alignment, and
0 commit comments