Skip to content

Commit 3e38338

Browse files
committed
README EDIT
1 parent 066d811 commit 3e38338

2 files changed

Lines changed: 28 additions & 0 deletions

File tree

README.md

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,7 @@
1+
<p align="center">
2+
<img src="docs/figures/banner.png" alt="Text-LLM-Training-from-scratch banner" />
3+
</p>
4+
15
# Text-LLM-Training-from-scratch
26

37
A from-scratch implementation of the language-model training pipeline in PyTorch, covering
@@ -14,6 +18,30 @@ selection is centralized in a single module, so the same code runs on CPU, Apple
1418

1519
Author: [y0oshi](https://github.com/Y0oshi). License: MIT.
1620

21+
## Table of contents
22+
23+
- [System overview](#system-overview)
24+
- [Requirements](#requirements)
25+
- [Installation](#installation)
26+
- [Quickstart](#quickstart)
27+
- [Two interfaces](#two-interfaces)
28+
- [Example: a 17M-parameter model](#example-a-17m-parameter-model)
29+
- [How each stage works](#how-each-stage-works)
30+
- [Tokenizer: byte-level BPE](#tokenizer-byte-level-bpe)
31+
- [Data pipeline: memory-mapped token shards](#data-pipeline-memory-mapped-token-shards)
32+
- [Model: a decoder-only Transformer](#model-a-decoder-only-transformer)
33+
- [Generation: the key/value cache](#generation-the-keyvalue-cache)
34+
- [Alignment](#alignment)
35+
- [Supervised fine-tuning with prompt masking](#supervised-fine-tuning-with-prompt-masking)
36+
- [Reward model](#reward-model)
37+
- [Direct Preference Optimization](#direct-preference-optimization)
38+
- [GRPO with verifiable rewards](#grpo-with-verifiable-rewards)
39+
- [Serving and evaluation](#serving-and-evaluation)
40+
- [Hardware and scaling](#hardware-and-scaling)
41+
- [Repository layout](#repository-layout)
42+
- [Tests](#tests)
43+
- [License](#license)
44+
1745
## System overview
1846

1947
The pipeline proceeds in four phases: data preparation, pretraining, alignment, and

docs/figures/banner.png

146 KB
Loading

0 commit comments

Comments
 (0)