|
1 | | -# SemanticChunker.NET |
| 1 | +[](https://github.com/GregorBiswanger/SemanticChunker.NET) |
| 2 | + |
| 3 | +# SemanticChunker.NET |
| 4 | + |
| 5 | +**Automatic Semantic Chunking for RAG in .NET |
| 6 | +Transforms long texts into coherent, retrieval ready chunks with a single call - powered by embeddings and fully compatible with Semantic Kernel and Microsoft.Extensions.AI.** |
| 7 | + |
| 8 | +[](https://www.nuget.org/packages/SemanticChunker.NET/) |
| 9 | + |
| 10 | + |
| 11 | +> Split long documents into semantically coherent chunks that fit your LLM’s context window while maximising retrieval precision. |
| 12 | +
|
| 13 | +## Features ✨ |
| 14 | + |
| 15 | +- **Plug‑and‑play API** – One call to `CreateChunksAsync` returns ready‑to‑use `Chunk` objects with ID, text, and embedding. |
| 16 | +- **Model‑agnostic** – Works with any embedding generator supported by `Microsoft.Extensions.AI`; no framework lock‑in. |
| 17 | +- **Four breakpoint strategies** – `Percentile`, `StandardDeviation`, `InterQuartile`, and `Gradient` cover most corpus profiles. |
| 18 | +- **Context buffer window** – Configurable `bufferSize` preserves cross‑sentence semantics. |
| 19 | +- **Target chunk count** – Unique `targetChunkCount` option produces exactly the number of chunks you need. |
| 20 | +- **Multilingual sentence splitting** – ICU4N ensures accurate sentence boundaries in 70+ languages. |
| 21 | +- **Token‑limit safety** – Automatic 10 % safety margin below your model’s context window. |
| 22 | +- **Parallel embedding generation** – Maximises throughput when your embedding provider supports batching. |
| 23 | +- **Zero external overhead** – Pure .NET plus ICU4N; lightweight for microservices and serverless functions. |
| 24 | + |
| 25 | +## Installation 📦 |
| 26 | + |
| 27 | +```bash |
| 28 | +dotnet add package SemanticChunker.NET |
| 29 | +```` |
| 30 | + |
| 31 | +## Quick Start 🛠️ |
| 32 | + |
| 33 | +```csharp |
| 34 | +using Microsoft.Extensions.AI; |
| 35 | +using Microsoft.SemanticKernel; |
| 36 | +using SemanticChunker.NET; |
| 37 | +
|
| 38 | +// 1. Wire an embedding generator (example uses LM Studio) |
| 39 | +var builder = Kernel.CreateBuilder(); |
| 40 | +builder.Services.AddLmStudioEmbeddingGenerator("text-embedding-multilingual-e5-base"); |
| 41 | +using var kernel = builder.Build(); |
| 42 | +
|
| 43 | +// 2. Create Chunker with your model’s token limit (e.g. 512) |
| 44 | +var embeddingGenerator = |
| 45 | + kernel.Services.GetRequiredService<IEmbeddingGenerator<string, Embedding<float>>>(); |
| 46 | +
|
| 47 | +var chunker = new SemanticChunker(embeddingGenerator, tokenLimit: 512); |
| 48 | +
|
| 49 | +// 3. Chunk text |
| 50 | +string input = File.ReadAllText("whitepaper.md"); |
| 51 | +IList<Chunk> chunks = await chunker.CreateChunksAsync(input); |
| 52 | +
|
| 53 | +// 4. Persist embeddings to your vector store |
| 54 | +await myVectorStore.UpsertAsync(chunks); |
| 55 | +``` |
| 56 | + |
| 57 | +## Step‑by‑Step Calibration Guide |
| 58 | + |
| 59 | +This section walks you through finding the *best* settings for **your** corpus and embedding model. |
| 60 | + |
| 61 | +| Step | Action | Why | |
| 62 | +| --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------- | |
| 63 | +| **1 Choose an embedding model** | Prefer models whose training data match your language/domain. | Embedding quality dominates chunk quality. | |
| 64 | +| **2 Set `tokenLimit`** | Use the embedding model token limit. | Leaves headroom for prompts/RAG metadata. | |
| 65 | +| **3 Pick a buffer size** | Start with **1**; raise to 2–3 if individual sentences lose context. | Neighbor sentences improve semantic continuity. | |
| 66 | +| **4 Choose a breakpoint strategy** | `Percentile` 95 % is the industry default. Switch to `StandardDeviation` when your corpus shows heavy-tail distance distributions. | Percentile is robust; SD handles outliers. | |
| 67 | +| **5 Adjust `thresholdAmount`** | Lower value → more chunks, higher recall; Higher value → fewer, longer chunks, better precision. Tune in 5‑point increments (e.g. 90, 95, 98). | Balances retrieval recall vs. answer accuracy. | |
| 68 | +| **6 Optionally set `targetChunkCount`** | If you know how many chunks you need (e.g. for fixed‑budget eval), supply it and skip manual threshold tuning. | Directly controls output size. | |
| 69 | +| **7 Evaluate** | Measure Answer F1/EM and retrieval hit rate on a validation set. Iterate Steps 4–6 until metrics plateau. | Empirical tuning beats rules of thumb. | |
| 70 | +| **8 Lock parameters in production** | Persist calibrated values in app settings or environment variables. | Guarantees reproducibility across builds. | |
| 71 | + |
| 72 | +## Configuration Reference |
| 73 | + |
| 74 | +| Ctor Parameter | Default | Description | |
| 75 | +| ------------------ | ------------ | --------------------------------------------------------------------- | |
| 76 | +| `tokenLimit` | *‑* | Max LLM tokens per chunk (safety margin = 10 %). | |
| 77 | +| `bufferSize` | `1` | Sentences added before/after current sentence during embedding. | |
| 78 | +| `thresholdType` | `Percentile` | Breakpoint metric (`StandardDeviation`, `InterQuartile`, `Gradient`). | |
| 79 | +| `thresholdAmount` | see table | E.g. 95 % for Percentile, 3 σ for Standard Deviation. | |
| 80 | +| `targetChunkCount` | `null` | Overrides thresholds to hit an exact chunk count. | |
| 81 | +| `minChunkChars` | `0` | Skip chunks shorter than this. | |
| 82 | + |
| 83 | +## 👨💻 Author |
| 84 | + |
| 85 | +**[Gregor Biswanger](https://github.com/GregorBiswanger)** - is a leading expert in generative AI, a Microsoft MVP for Azure AI and Web App Development. As an independent consultant, he works closely with the Microsoft product team for GitHub Copilot and supports companies in implementing modern AI solutions. |
| 86 | + |
| 87 | + As a freelance consultant, trainer, and author, he shares his expertise in software architecture and cloud technologies and is a sought-after speaker at international conferences. For several years, he has been live-streaming every Friday evening on [Twitch](https://twitch.tv/GregorBiswanger) with [My Coding Zone](https://www.my-coding-zone.de) in german and is an active [YouTuber](https://youtube.com/GregorBiswanger). |
| 88 | + |
| 89 | +Reach out to Gregor if you need support in the form of consulting, training, or implementing AI solutions using .NET or Node.js. [LinkedIn](https://www.linkedin.com/in/gregor-biswanger-51342011/) or Twitter [@BFreakout](https://www.twitter.com/BFreakout) |
| 90 | + |
| 91 | +See also the list of [contributors](https://github.com/GregorBiswanger/SemanticChunker.NET/graphs/contributors) who participated in this project. |
| 92 | + |
| 93 | +## 🙋♀️🙋♂ Contributing |
| 94 | + |
| 95 | +Feel free to submit a pull request if you find any bugs (to see a list of active issues, visit the [Issues section](https://github.com/GregorBiswanger/SemanticChunker.NET/issues). |
| 96 | +Please make sure all commits are properly documented. |
| 97 | + |
| 98 | +The best thing would be to write about what you plan to do in the issue beforehand. Then there will be no disappointment if we cannot accept your pull request. |
| 99 | + |
| 100 | +## 🙏 Donate |
| 101 | + |
| 102 | +I work on this open-source project in my free time alongside a full-time job and raising three kids. If you`d like to support my work and help me dedicate more time to this project, consider sponsoring me on GitHub: |
| 103 | +
|
| 104 | +- [Gregor Biswanger](https://github.com/sponsors/GregorBiswanger) |
| 105 | +
|
| 106 | +Your sponsorship allows me to invest more time in improving the project and prioritizing important issues or features. Any support is greatly appreciated - thank you! 🍻 |
| 107 | +
|
| 108 | +## 📜 License |
| 109 | +
|
| 110 | +This project is licensed under the [**Apache License 2.0**](https://raw.githubusercontent.com/GregorBiswanger/SemanticChunker.NET/refs/heads/main/LICENSE.txt) - © Gregor Biswanger 2025 |
| 111 | +
|
| 112 | +*Happy chunking!* 🧩 |
0 commit comments