Split the document into chunks of exactly N tokens with M tokens of overlap between consecutive chunks. No awareness of sentences, paragraphs, or meaning — just raw token counts.
It's dumb, but it's fast and deterministic. Most RAG systems start here.
- Document tokenized using the model's tokenizer
- Tokens sliced into chunks of
chunk_size(500 by default) - Each chunk overlaps with the next by
chunk_overlap(50) tokens to avoid cutting off context - Each chunk embedded and stored independently
- You want a fast, predictable baseline before trying smarter strategies
- Documents have uniform density (no headers, no tables)
- You don't want to pay for embedding calls to a semantic splitter
- Token budget is tight and you need exact control over chunk size
- Documents have clear sections or paragraphs — fixed splitting cuts across them
- Sentences matter — a 500-token cut can slice mid-sentence
- You're working with code, tables, or structured content
| Speed | Very fast — no LLM or embedding needed |
| Accuracy | Medium — no semantic awareness, can cut mid-sentence |
| Cost | Zero — pure text processing |
| Predictability | High — exact chunk sizes every time |
Overlap exists because the cut point might split a thought across two chunks. With 50 token overlap, the end of chunk 1 repeats at the start of chunk 2 — so neither chunk loses critical context.
Too much overlap = redundant context. Too little = broken thoughts at boundaries.
# No API key needed
python fixed_size_chunking.pyAdjust CHUNK_SIZE and CHUNK_OVERLAP at the top of the script to experiment.