Commit aa67ed2
authored
fix(memory): prevent infinite loop in chunk_text when overlap rewinds start (#3027)
When rfind("\n\n") found a match near the beginning of the sliding window,
`end` advanced by fewer bytes than CHUNK_OVERLAP_CHARS. The subsequent
`end.saturating_sub(CHUNK_OVERLAP_CHARS)` produced a value less than the
current `start`, and `ceil_char_boundary` returned a position behind it.
The old safeguard (`if start >= end`) did not fire because new_start <
start < end, so `start` regressed and the loop ran forever.
Fix: guarantee forward progress by taking `end` when `new_start <= start`.
This was the root cause of 100% CPU / GB RAM consumption at startup: the
embed backfill spawned background tasks for unembedded messages (including
an 85 KB tool_result), all of which entered the infinite loop simultaneously,
saturating all tokio worker threads and blocking log flushing.1 parent aa79adf commit aa67ed2
1 file changed
Lines changed: 5 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
49 | 49 | | |
50 | 50 | | |
51 | 51 | | |
52 | | - | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
53 | 55 | | |
54 | | - | |
55 | | - | |
56 | | - | |
57 | | - | |
| 56 | + | |
| 57 | + | |
58 | 58 | | |
59 | 59 | | |
60 | 60 | | |
| |||
0 commit comments