Skip to content

Fix _strip_code_fences truncating JSON when content contains inner backticks#2563

Merged
benfrank241 merged 2 commits into
vectorize-io:mainfrom
poog26:fix/strip-code-fences-inner-backticks
Jul 7, 2026
Merged

Fix _strip_code_fences truncating JSON when content contains inner backticks#2563
benfrank241 merged 2 commits into
vectorize-io:mainfrom
poog26:fix/strip-code-fences-inner-backticks

Conversation

@poog26

@poog26 poog26 commented Jul 4, 2026

Copy link
Copy Markdown
Contributor

The current implementation uses content.split("```json")[1].split("```")[0] to strip markdown code fences. This finds the first ``` after the opening fence — so when the extracted JSON itself contains literal triple-backtick characters, the split matches those inner backticks and truncates the JSON mid-string.

When this bites: Hindsight's fact-extraction LLM produces output wrapped in ```json ... ``` fences. If one of the extracted facts describes code fence behavior (e.g. "the model wraps output in json fences"), the JSON now contains ` json as a literal string value, andsplit("```")finds that inner occurrence. Result: truncated JSON →json.loads` fails → retry loop (up to 4 retries per chunk).

The fix replaces string-splitting with line-based fence detection. Per the markdown spec, code fences must appear at the start of a line. Inner triple backticks inside JSON string values are preserved because they aren't at line boundaries.

Before (broken):

_strip_code_fences('```json\n{"facts": [{"what": "uses ```json fences"}]}\n```')
# → '{"facts": [{"what": "uses '  (truncated!)

After (fixed):

_strip_code_fences('```json\n{"facts": [{"what": "uses ```json fences"}]}\n```')
# → '{"facts": [{"what": "uses ```json fences"}]}'  (intact)

Edge cases: normal fences ✓, inner backticks preserved ✓, no fences no-op ✓, opening-only (truncated output) returned unchanged ✓, text around fence block handled correctly ✓.

poog26 and others added 2 commits July 4, 2026 17:11
…ckticks

The old implementation used content.split('')[0] to strip
markdown code fences from LLM responses. This finds the FIRST occurrence of '''
after the opening fence — so when the extracted JSON itself contains literal
triple-backtick characters (e.g. facts about code fence formatting), the split
matches those inner backticks and truncates the JSON mid-string.

Replace with line-based fence detection that only matches fences at line
boundaries per the markdown spec. Inner backticks inside JSON string values
are preserved since they aren't at line boundaries.
@benfrank241

Copy link
Copy Markdown
Member

thanks for this! just had to add a test. merging

@benfrank241 benfrank241 merged commit d251fcb into vectorize-io:main Jul 7, 2026
87 checks passed
benfrank241 added a commit that referenced this pull request Jul 7, 2026
…ON (#2610)

Grafts the parse-validated fallback from #2557 onto the line-based fence
stripper merged in #2563: after stripping, if the candidate is not valid
JSON (partial/absent fence, prose-wrapped or truncated output), fall back
to the outermost parseable {..}/[..] span. Never returns a worse candidate
than the raw content.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants