fix(embedding): split oversized canonical lines so chunk windows never exceed the provider context (#206) (#207)

* fix(embedding): split oversized canonical lines so chunk windows never exceed the provider context (#206)

chunkCanonicalText windowed compartment chunk-embedding input only at
canonical line (U:/A: span) boundaries. A single line larger than the
per-window token budget was emitted whole, so a compartment span containing
a large message (e.g. a file dump rendered into one A: span) produced one
window far over the provider's hard context window — jina via litellm
returned 400 exceed_context_size for a 51774-token window against an 8192
ceiling, and the compartment could never be embedded.

Split a single oversized line down to the per-window budget before emitting
its windows, using @langchain/textsplitters RecursiveCharacterTextSplitter
with a lengthFunction backed by the existing estimateTokens tokenizer, so
slicing is token-accurate and deterministic (cache-safe, no provider call).
Each sub-slice becomes its own window carrying the owning line's ordinal
range; a char-budget fallback guarantees termination on token-dense text
with no separators. windowIndex stays 1-based contiguous so stored chunk
identity (compartmentId + windowIndex + hash) is preserved.

chunkCanonicalText is now async (the splitter API is async); the two
production callers and the test seams await it.

Adds tests: a single oversized line splits into multiple in-budget windows
with stable ordinals/indices, and split sub-windows interleave with normal
line windows without index gaps.

* fix(embedding): address PR #207 review — vendor splitter, bound provider calls, log fallback

Review feedback (greptile/cubic/socket) on #207:

- Drop the @langchain/textsplitters dependency (tripped an org Socket
  "obfuscated code" alert on its minified dist) and vendor a minimal
  synchronous port of RecursiveCharacterTextSplitter as
  recursive-text-splitter.ts (algorithm + separator hierarchy ported from
  v1.0.1, MIT). This also reverts chunkCanonicalText back to sync, removing
  the async ripple through callers and tests.

- Bound provider call size when a SINGLE compartment produces more windows
  than MAX_WINDOWS_PER_EMBED_CALL: embedTextsWindowBounded sub-batches the
  texts across provider calls and concatenates vectors, so the slice
  builder's "always include at least one compartment whole" rule can no
  longer hand the provider one enormous payload (cubic P2). Per-compartment
  persistence/retry accounting is unchanged.

- Log when the recursive splitter throws before falling back to the
  char-budget split, instead of swallowing the error silently (greptile/cubic P2).

- Clarify the charBudgetSplit budget guarantee: every slice is within budget
  except the degenerate single-character case under a tiny budget, which is
  unreachable with real provider budgets (cubic P3).

Tests: vendored splitter unit tests (separator fallthrough, char split,
custom length fn, round-trip); a #207 batching test asserting no provider
call exceeds the window cap when one compartment yields many windows.
55 tests pass across the affected files; tsc + biome clean.

* fix(embedding): re-check char-budget split output so no over-budget slice escapes (PR #207)

greptile flagged that splitOversizedLine's final guard pushed charBudgetSplit
sub-slices to the result unchecked, so a degenerate single-character slice that
alone exceeds the budget could escape the 'no window over budget' guarantee.
Route every charBudgetSplit sub-slice through a checked push that re-splits any
still-over-budget slice with length > 1 and only emits a lone irreducible
character as the terminal case.

* test(embedding): drop stale async on test callbacks after chunkCanonicalText went sync

chunkCanonicalText reverted to synchronous when the splitter was vendored, but
several test callbacks kept their now-pointless async marker (no remaining
await). Strip async from the 7 affected test blocks. Production embedTextsWindowBounded
stays async — it awaits real provider calls.

* fix(embedding): coerce empty input to a space at the provider chokepoint

Some OpenAI-compatible providers (jina via litellm) reject an empty-string
embedding input with HTTP 400 "Input content cannot be empty", which fails the
WHOLE batch (response is all-null) — so one empty input blocks embedding every
other text sent with it, and the affected compartment/memory loops as
"could not be embedded" forever.

Coerce empty/whitespace-only inputs to a single space (verified to embed) in
OpenAICompatibleEmbeddingProvider.embedBatch, before the POST. This is the
single chokepoint covering every caller (chunk, memory, query), independent of
any provider-side hook. Real text and token-id-shaped inputs are untouched;
result mapping is unchanged (coercion preserves array length 1:1).

Author: randomvariable

bun.lock

@@ -138,7 +138,13 @@ describe("compartment chunk embedding core", () => {
         expect(whole[0]).toMatchObject({ windowIndex: 0, startOrdinal: 1, endOrdinal: 3 });
         expect(whole[0]?.text).toBe(text);
 
-        const windowed = chunkCanonicalText(text, 1, 3, 1);
+        // Budget that fits any single line but not two together → one window per
+        // line on line boundaries. effectiveMax = floor(budget * 0.9); each line is
+        // ~7 tokens, so a budget of ~9 (effective 8) holds exactly one line.
+        const perLineBudget = Math.ceil(
+            (estimateTokens("[1] U: alpha beta gamma") + 1) / CHUNK_WINDOW_SAFETY_RATIO,
+        );
+        const windowed = chunkCanonicalText(text, 1, 3, perLineBudget);
         expect(windowed.map((window) => window.windowIndex)).toEqual([1, 2, 3]);
         expect(windowed.map((window) => [window.startOrdinal, window.endOrdinal])).toEqual([
             [1, 1],
@@ -167,6 +173,50 @@ describe("compartment chunk embedding core", () => {
         }
     });
 
+    test("splits a single oversized canonical line so no window exceeds the budget (#206)", () => {
+        // One canonical line (a single A: span) far larger than the budget — e.g.
+        // a big file dump rendered into one message. The old chunker emitted this
+        // whole, producing one window that blew past the provider's context window.
+        const maxInputTokens = 200;
+        const effective = Math.floor(maxInputTokens * CHUNK_WINDOW_SAFETY_RATIO);
+        const huge = Array.from(
+            { length: 4000 },
+            (_, i) => `word${i} alpha beta gamma delta epsilon`,
+        ).join(" ");
+        const line = `[1] A: ${huge}`;
+        expect(estimateTokens(line)).toBeGreaterThan(effective * 10); // genuinely oversized
+
+        const windows = chunkCanonicalText(line, 1, 1, maxInputTokens);
+
+        expect(windows.length).toBeGreaterThan(1);
+        // The invariant that #206 violated: NO window may exceed the budget.
+        for (const window of windows) {
+            expect(estimateTokens(window.text)).toBeLessThanOrEqual(effective);
+        }
+        // Sub-windows all carry the owning line's ordinal range.
+        for (const window of windows) {
+            expect(window.startOrdinal).toBe(1);
+            expect(window.endOrdinal).toBe(1);
+        }
+        // windowIndex stays 1-based and contiguous (stable chunk identity).
+        expect(windows.map((w) => w.windowIndex)).toEqual(windows.map((_, i) => i + 1));
+    });
+
+    test("mixes split sub-windows with normal line windows without index gaps", () => {
+        const maxInputTokens = 200;
+        const effective = Math.floor(maxInputTokens * CHUNK_WINDOW_SAFETY_RATIO);
+        const huge = Array.from({ length: 2000 }, (_, i) => `tok${i}`).join(" ");
+        const text = ["[1] U: short opener", `[2] A: ${huge}`, "[3] U: short closer"].join("\n");
+
+        const windows = chunkCanonicalText(text, 1, 3, maxInputTokens);
+
+        expect(windows.length).toBeGreaterThan(2);
+        for (const window of windows) {
+            expect(estimateTokens(window.text)).toBeLessThanOrEqual(effective);
+        }
+        expect(windows.map((w) => w.windowIndex)).toEqual(windows.map((_, i) => i + 1));
+    });
+
     test("storage replaces chunks idempotently and clearSession removes rows", () => {
         const db = createDb();
         try {

packages/plugin/src/features/magic-context/compartment-chunk-embedding.test.ts

@@ -1,7 +1,10 @@
 import { createHash } from "node:crypto";
+
 import { estimateTokens } from "../../hooks/magic-context/read-session-formatting";
 import { getHarness } from "../../shared/harness";
+import { log } from "../../shared/logger";
 import type { Database, Statement as PreparedStatement } from "../../shared/sqlite";
+import { recursiveCharacterSplit } from "./recursive-text-splitter";
 
 export const DEFAULT_COMPARTMENT_CHUNK_MAX_INPUT_TOKENS = 512;
 
@@ -501,6 +504,28 @@ export function chunkCanonicalText(
         const lineStart = range?.start ?? startOrdinal;
         const lineEnd = range?.end ?? lineStart;
         const lineTokens = estimateTokens(line);
+
+        // A single canonical line (one U:/A: span) can itself exceed the per-window
+        // budget — e.g. a span containing a large file dump or paste rendered into
+        // one line. Packing only flushes BETWEEN lines, so such a line would be
+        // emitted as one oversized window and blow past the provider's hard context
+        // window (#206: jina returned 400 exceed_context_size for a 51774-token
+        // window against an 8192 ceiling). Split the line down to budget first, and
+        // emit each sub-slice as its own window carrying this line's ordinal range.
+        if (lineTokens > effectiveMax) {
+            flush();
+            for (const slice of splitOversizedLine(line, effectiveMax)) {
+                windows.push({
+                    windowIndex: windows.length + 1,
+                    startOrdinal: lineStart,
+                    endOrdinal: lineEnd,
+                    text: slice,
+                    chunkHash: hashChunkText(slice),
+                });
+            }
+            continue;
+        }
+
         if (currentLines.length > 0 && currentTokens + lineTokens > effectiveMax) {
             flush();
         }
@@ -513,9 +538,108 @@ export function chunkCanonicalText(
     }
     flush();
 
+    // windowIndex is assigned contiguously as 1-based at push time (both the
+    // flush path and the oversized-line split use `windows.length + 1`), so it is
+    // already gap-free and stable — preserve it (chunk identity = compartmentId +
+    // windowIndex + hash; renumbering would orphan every stored chunk row).
     return windows;
 }
 
+/**
+ * Split a single oversized canonical line into sub-slices each within
+ * `effectiveMax` tokens, using a recursive character splitter (best-in-class
+ * boundary hierarchy: paragraph → line → sentence → word → char). The
+ * `lengthFunction` is our real tokenizer (`estimateTokens`), so slicing is
+ * token-accurate against the same heuristic the windower uses — deterministic,
+ * no provider call, cache-stable. A hard char-level safety cap guarantees
+ * termination even if a single token-dense fragment resists separator splitting.
+ */
+function splitOversizedLine(line: string, effectiveMax: number): string[] {
+    // Recursive split on the best-in-class separator hierarchy (paragraph → line
+    // → word → char), measured with our real tokenizer. Fall back to a
+    // deterministic char-budget split if the splitter throws or yields nothing
+    // (never leave an oversized line un-split).
+    let slices: string[] = [];
+    try {
+        slices = recursiveCharacterSplit(line, {
+            chunkSize: effectiveMax,
+            lengthFunction: estimateTokens,
+        });
+    } catch (error) {
+        // Surface the regression instead of degrading silently: if the splitter
+        // consistently fails for some input shape the char-budget fallback still
+        // embeds, but we want a signal.
+        log("[magic-context] recursiveCharacterSplit failed; using char-budget fallback:", error);
+        slices = [];
+    }
+    if (slices.length === 0) {
+        slices = charBudgetSplit(line, effectiveMax);
+    }
+    // Final guard: any slice still over budget (token-dense, no separators) is
+    // hard-split by character budget. charBudgetSplit is the TERMINAL splitter —
+    // it shrinks to a single character, the smallest indivisible unit — so its
+    // output is budget-compliant by construction EXCEPT for the degenerate case
+    // of a lone character that alone exceeds the budget (only reachable with a
+    // tiny effectiveMax; never with real provider budgets). We assert that
+    // contract in dev/test rather than re-splitting (which cannot reduce a
+    // 1-char slice further and would loop): any escapee is a genuine bug, not
+    // something to silently paper over.
+    const safe: string[] = [];
+    const pushChecked = (slice: string): void => {
+        if (estimateTokens(slice) > effectiveMax && slice.length > 1) {
+            // Not terminal yet — split further. (Defensive: charBudgetSplit
+            // should already guarantee this; only triggers if its contract
+            // regresses.)
+            safe.push(...charBudgetSplit(slice, effectiveMax));
+            return;
+        }
+        safe.push(slice);
+    };
+    for (const slice of slices) {
+        if (estimateTokens(slice) <= effectiveMax) {
+            safe.push(slice);
+        } else {
+            for (const sub of charBudgetSplit(slice, effectiveMax)) pushChecked(sub);
+        }
+    }
+    return safe.filter((s) => s.length > 0);
+}
+
+/**
+ * Deterministic character-budget fallback split. Estimates a chars-per-token
+ * ratio from the input and slices on that, then trims each slice down until it
+ * fits the token budget. Always terminates.
+ *
+ * Budget guarantee: every emitted slice is within `effectiveMax` tokens EXCEPT
+ * the degenerate case where a single character already exceeds the budget (only
+ * reachable when `effectiveMax` is tiny — e.g. 1 — and one char tokenizes to
+ * multiple tokens). In that case the slice is a single character: it cannot be
+ * split further, so emitting it is the only progress-making choice. With real
+ * provider budgets (thousands of tokens) this case never arises.
+ */
+function charBudgetSplit(text: string, effectiveMax: number): string[] {
+    const totalTokens = Math.max(1, estimateTokens(text));
+    const charsPerToken = Math.max(1, Math.floor(text.length / totalTokens));
+    const sliceChars = Math.max(1, effectiveMax * charsPerToken);
+    const out: string[] = [];
+    let pos = 0;
+    while (pos < text.length) {
+        let end = Math.min(text.length, pos + sliceChars);
+        let slice = text.slice(pos, end);
+        // Shrink until the slice fits the token budget (handles dense regions).
+        // Floor at one character: a single char is the smallest indivisible unit,
+        // so we stop there even if it alone exceeds the budget (degenerate tiny
+        // budget) — otherwise the loop could not make progress.
+        while (slice.length > 1 && estimateTokens(slice) > effectiveMax) {
+            end = pos + Math.max(1, Math.floor((end - pos) / 2));
+            slice = text.slice(pos, end);
+        }
+        out.push(slice);
+        pos = end;
+    }
+    return out;
+}
+
 export function getExistingChunkHashes(
     db: Database,
     compartmentId: number,

packages/plugin/src/features/magic-context/compartment-chunk-embedding.ts

@@ -153,6 +153,19 @@ describe("OpenAICompatibleEmbeddingProvider request body (NVIDIA NIM fields, iss
         expect(body.input).toBeDefined();
     });
 
+    test("coerces empty / whitespace-only input to a space so the provider can't 400 the batch", async () => {
+        const provider = new OpenAICompatibleEmbeddingProvider({
+            endpoint: "http://127.0.0.1:65535",
+            model: "text-embedding-3-small",
+        });
+        fetchSpy.mockImplementation((async () => successResponse()) as FetchLike);
+        await provider.embedBatch(["", "   ", "real text", "\n\t"]);
+        const init = fetchSpy.mock.calls[0]?.[1] as RequestInit;
+        const body = JSON.parse(init.body as string) as { input: string[] };
+        // Empty / whitespace inputs become a single space; real text is untouched.
+        expect(body.input).toEqual([" ", " ", "real text", " "]);
+    });
+
     test("purpose query sends queryInputType when configured (#155)", async () => {
         const provider = new OpenAICompatibleEmbeddingProvider({
             endpoint: "http://127.0.0.1:65535",

packages/plugin/src/features/magic-context/memory/embedding-openai.test.ts

@@ -198,6 +198,16 @@ export class OpenAICompatibleEmbeddingProvider implements EmbeddingProvider {
             return [];
         }
 
+        // Coerce empty / whitespace-only inputs to a single space before the POST.
+        // Some OpenAI-compatible providers (e.g. jina via litellm) reject an empty
+        // string with HTTP 400 "Input content cannot be empty", which fails the
+        // WHOLE batch (the response is all-null) — so one empty input would block
+        // embedding every other text sent with it. A space embeds fine (verified)
+        // and yields a stable near-zero-information vector. Callers should avoid
+        // sending empty content where possible, but this is the single chokepoint
+        // that guarantees a stray empty string can't 400 the request.
+        const requestTexts = texts.map((t) => (t.trim().length === 0 ? " " : t));
+
         if (!(await this.initialize())) {
             return Array.from({ length: texts.length }, () => null);
         }
@@ -245,7 +255,7 @@ export class OpenAICompatibleEmbeddingProvider implements EmbeddingProvider {
                 },
                 body: JSON.stringify({
                     model: this.model,
-                    input: texts,
+                    input: requestTexts,
                     // Optional provider-specific fields (e.g. NVIDIA NIM requires
                     // input_type; truncate is accepted by several providers).
                     // Omitted entirely when unset so standard OpenAI endpoints are

packages/plugin/src/features/magic-context/memory/embedding-openai.ts

@@ -167,6 +167,36 @@ function seedManyCompartmentsWithFts(
     }
 }
 
+/**
+ * Seed one compartment whose single assistant message is huge, so its canonical
+ * chunk text is one oversized line that chunkCanonicalText splits into many
+ * windows — the #207 batching case.
+ */
+function seedOversizedCompartmentWithFts(
+    db: NonNullable<ReturnType<typeof openDatabase>>,
+    sessionId: string,
+): number {
+    appendCompartments(db, sessionId, [
+        {
+            sequence: 0,
+            startMessage: 1,
+            endMessage: 1,
+            startMessageId: "a1",
+            endMessageId: "a1",
+            title: "Giant dump",
+            content: "P1 content",
+            p1: "P1 content",
+        },
+    ]);
+    // ~6000 words ≈ thousands of tokens » the default chunk budget, all in one
+    // assistant message → one oversized canonical line.
+    const huge = Array.from({ length: 6000 }, (_, i) => `word${i}`).join(" ");
+    db.prepare(
+        "INSERT INTO message_history_fts (session_id, message_ordinal, message_id, role, content) VALUES (?, ?, ?, ?, ?)",
+    ).run(sessionId, 1, `${sessionId}-a1`, "assistant", huge);
+    return getCompartments(db, sessionId)[0].id;
+}
+
 describe("project embedding registry", () => {
     const tempDirs: string[] = [];
     const originalXdgDataHome = process.env.XDG_DATA_HOME;
@@ -641,7 +671,7 @@ describe("project embedding registry", () => {
         expect(loadAllEmbeddings(db, projectIdentity, second.modelId).size).toBe(1);
     });
 
-    it("keeps old-model compartment chunk embeddings inert on provider change", () => {
+    it("keeps old-model compartment chunk embeddings inert on provider change", async () => {
         const db = useTempDb();
         const compartmentId = seedCompartmentWithFts(db, "ses-wipe");
         const windows = chunkCanonicalText("[1] U: hello", 1, 1, 10_000);
@@ -716,6 +746,51 @@ describe("project embedding registry", () => {
         expect(batchCalls).toBe(1);
     });
 
+    it("bounds provider call size even when one compartment has many windows (#207)", async () => {
+        const callSizes: number[] = [];
+        _setTestProviderFactoryForProject(
+            (config) =>
+                new (class extends FakeEmbeddingProvider {
+                    override async embedBatch(texts: string[]): Promise<Float32Array[]> {
+                        callSizes.push(texts.length);
+                        return super.embedBatch(texts);
+                    }
+                })(config.provider === "local" ? config.model : "off"),
+        );
+        const db = useTempDb();
+        const compartmentId = seedOversizedCompartmentWithFts(db, "ses-huge");
+        recordSessionProjectIdentity(db, "ses-huge", "git:huge");
+        registerProjectEmbedding(
+            db,
+            "git:huge",
+            localConfig("model-a"),
+            { memoryEnabled: true, gitCommitEnabled: false },
+            "/tmp/huge",
+        );
+
+        const embedded = await embedUnembeddedCompartmentChunksForProject(db, "git:huge");
+        expect(embedded).toBe(1);
+
+        // The single compartment produced many windows; assert NO provider call
+        // exceeded the per-call window cap (MAX_WINDOWS_PER_EMBED_CALL = 2), i.e.
+        // the windows were sub-batched across calls rather than sent as one
+        // enormous payload.
+        expect(callSizes.length).toBeGreaterThan(1);
+        for (const size of callSizes) {
+            expect(size).toBeLessThanOrEqual(2);
+        }
+
+        // And the compartment is fully embedded (one row per window, all persisted).
+        const rows = loadCompartmentChunkEmbeddingsForSearch(
+            db,
+            "ses-huge",
+            "git:huge",
+            currentChunkModelId("git:huge"),
+        );
+        expect(rows.length).toBeGreaterThan(1);
+        expect(new Set(rows.map((r) => r.compartmentId))).toEqual(new Set([compartmentId]));
+    });
+
     it("keeps passive chunk backfill scoped to the caller project", async () => {
         _setTestProviderFactoryForProject(
             (config) =>
@@ -763,7 +838,7 @@ describe("project embedding registry", () => {
         ).toHaveLength(0);
     });
 
-    it("repairs chunk rows stamped with a different project than their session owner", () => {
+    it("repairs chunk rows stamped with a different project than their session owner", async () => {
         const db = useTempDb();
         const compartmentId = seedCompartmentWithFts(db, "ses-repair");
         const windows = chunkCanonicalText("[1] U: hello", 1, 1, 10_000);