Commit 135c633
committed
feat: implement Strategy Pattern fragmentation with optional parsers
- Add TreeSitterStrategy for 9 languages (Python, JS, TS, Go, Rust, Java, C, C++, Ruby)
- Add PythonAstStrategy as stdlib fallback
- Add MistuneMarkdownStrategy and RegexMarkdownStrategy for Markdown
- Add HTMLStrategy with lxml for HTML/XML semantic blocks
- Add RuamelYamlStrategy for YAML with position tracking
- Add PySBDTextStrategy for sentence-aware text splitting
- Add ConfigStrategy, ParagraphStrategy, GenericStrategy fallbacks
- Add FragmentationEngine with priority-based chain of responsibility
- All parsers are optional with graceful degradation1 parent 06f9161 commit 135c633
11 files changed
Lines changed: 1631 additions & 79 deletions
File tree
- .github/workflows
- docs
- src/treemapper
- diffctx
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
60 | 60 | | |
61 | 61 | | |
62 | 62 | | |
63 | | - | |
| 63 | + | |
64 | 64 | | |
65 | | - | |
| 65 | + | |
66 | 66 | | |
67 | 67 | | |
68 | 68 | | |
| |||
121 | 121 | | |
122 | 122 | | |
123 | 123 | | |
124 | | - | |
125 | 124 | | |
126 | | - | |
| 125 | + | |
127 | 126 | | |
128 | 127 | | |
129 | 128 | | |
| |||
177 | 176 | | |
178 | 177 | | |
179 | 178 | | |
180 | | - | |
| 179 | + | |
181 | 180 | | |
182 | 181 | | |
183 | | - | |
184 | | - | |
| 182 | + | |
185 | 183 | | |
186 | 184 | | |
187 | 185 | | |
| |||
210 | 208 | | |
211 | 209 | | |
212 | 210 | | |
213 | | - | |
| 211 | + | |
214 | 212 | | |
215 | | - | |
216 | 213 | | |
217 | 214 | | |
218 | 215 | | |
| |||
297 | 294 | | |
298 | 295 | | |
299 | 296 | | |
300 | | - | |
301 | 297 | | |
302 | 298 | | |
303 | 299 | | |
| |||
344 | 340 | | |
345 | 341 | | |
346 | 342 | | |
347 | | - | |
0 commit comments