Skip to content

Fix /llm_log endpoint: replace brace-counting parser with one-JSON-per-line format#149

Merged
CyberSecDef merged 7 commits into
mainfrom
copilot/fix-brace-counting-json-parser
Apr 8, 2026
Merged

Fix /llm_log endpoint: replace brace-counting parser with one-JSON-per-line format#149
CyberSecDef merged 7 commits into
mainfrom
copilot/fix-brace-counting-json-parser

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 8, 2026

The /llm_log endpoint reconstructed multi-line indented JSON using {/} counting, which silently corrupts entries containing brace characters in string values or escaped quotes.

Changes

  • novelforge/llm/client.py — Drop indent=2 from the three llm_logger.info(json.dumps(...)) calls (error, request, response). Each log entry is now a single compact line. Authorization headers are redacted before logging.
  • novelforge/llm/image.py — Drop indent=2 from the image request log write so all llm_logger writers emit single-line JSON consistently.
  • novelforge/__init__.py — Replace the brace-counting loop with a plain JSONL line-by-line parser: each non-empty line is parsed with json.loads(stripped), malformed lines are skipped, and a collections.deque(maxlen=10) keeps memory bounded at O(10) entries regardless of log file size. No buffering or resync logic is needed since all writers now emit single-line JSON.
  • tests/test_integration.py — Add three tests to TestLLMLog using a fully isolated _isolated_app fixture (patches all dirs: NOVELS_DIR, LOGS_DIR, SESSION_FILE_DIR, EXPORT_DIR) covering: correct parsing of multi-entry logs, entries with {/} inside string values, and the 10-entry truncation cap.

Copilot AI requested review from Copilot and removed request for Copilot April 8, 2026 18:31
Copilot AI linked an issue Apr 8, 2026 that may be closed by this pull request
Copilot AI requested review from Copilot and removed request for Copilot April 8, 2026 18:38
Copilot AI changed the title [WIP] Fix brace counting in JSON parser for debug endpoint Fix /llm_log endpoint: replace brace-counting parser with one-JSON-per-line format Apr 8, 2026
Copilot AI requested a review from CyberSecDef April 8, 2026 18:39
Comment thread novelforge/llm/client.py Fixed
…f sensitive information'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Copilot AI review requested due to automatic review settings April 8, 2026 18:40
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the /llm_log endpoint to parse logs as one JSON object per line, eliminating corruption caused by the previous brace-counting reconstruction approach.

Changes:

  • Emit compact single-line JSON for LLM request/response/error logs.
  • Simplify /llm_log parsing to json.loads() each non-empty line and return the last 10 entries.
  • Add integration tests covering multi-entry parsing, braces inside JSON strings, and the 10-entry cap.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
novelforge/llm/client.py Switch LLM log writes to single-line JSON (and redact Authorization) so /llm_log can parse reliably.
novelforge/__init__.py Replace brace-counting parser with line-by-line JSON parsing and return last 10 entries.
tests/test_integration.py Add tests validating the new one-JSON-per-line parsing behavior and truncation logic.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread novelforge/__init__.py Outdated
Comment thread novelforge/__init__.py Outdated
Comment thread tests/test_integration.py Outdated
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings April 8, 2026 19:10
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 4 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread novelforge/__init__.py Outdated
Comment thread novelforge/__init__.py Outdated
Comment thread novelforge/__init__.py Outdated
Comment thread tests/test_integration.py Outdated
Copy link
Copy Markdown
Contributor Author

Copilot AI commented Apr 8, 2026

Just as a heads up, I was blocked by some firewall rules while working on your feedback. Expand below for details.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • api.openai.com
    • Triggering command: /usr/bin/python python -m pytest --tb=short -q (dns block)
    • Triggering command: /usr/bin/python python -m pytest --tb=short (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings April 8, 2026 19:30
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread novelforge/__init__.py
Comment thread novelforge/__init__.py
@CyberSecDef CyberSecDef marked this pull request as ready for review April 8, 2026 19:50
Copilot AI review requested due to automatic review settings April 8, 2026 19:50
@CyberSecDef CyberSecDef merged commit 4a92226 into main Apr 8, 2026
10 checks passed
@CyberSecDef CyberSecDef deleted the copilot/fix-brace-counting-json-parser branch April 8, 2026 19:50
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Brace-Counting JSON Parser in Debug Endpoint

4 participants