Skip to content

Commit 4b2a4c6

Browse files
Extract JSONL parser monolith into focused modules (#57)
* Extract JSONL parser monolith into focused modules * fix(session_peek): scan full file when size ≤10KB * refactor(jsonl): address parser-split review nits * fix(jsonl): address parser-split review follow-ups * fix(session_peek): restore quick_session_info parity with monolith
1 parent 75b8174 commit 4b2a4c6

6 files changed

Lines changed: 502 additions & 432 deletions

File tree

docs/architecture.md

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,9 @@
1515
▼ ▼ ▼
1616
┌─────────────────┐ ┌─────────────────────┐ ┌──────────────────┐
1717
│ session_path │ │ jsonl_parser │ │ exclusion_rules │
18-
│ list_projects │ │ parse_session │ │ load + match │
19-
│ list_sessions │ │ quick_session_info │ └────────┬─────────┘
20-
│ safe_join │ │ _parse_tool_result │ │
18+
│ list_projects │ │ session_peek │ │ load + match │
19+
│ list_sessions │ │ tool_dispatch │ └────────┬─────────┘
20+
│ safe_join │ │ jsonl_helpers │ │
2121
└────────┬────────┘ └──────────┬──────────┘ │
2222
│ │ │
2323
└────────────┬───────────┴────────────────────────┘
@@ -48,7 +48,7 @@
4848
| Layer | Responsibility | Key modules |
4949
|-------|----------------|-------------|
5050
| **Data discovery** | Resolve `~/.claude/projects/`, list projects and sessions, prevent path traversal | `utils/session_path.py` |
51-
| **Parsing** | JSONL → session dict (messages, metadata, tool rendering) | `utils/jsonl_parser.py` |
51+
| **Parsing** | JSONL → session dict (messages, metadata, tool rendering) | `utils/jsonl_parser.py`, `utils/tool_dispatch.py`, `utils/session_peek.py`, `utils/jsonl_helpers.py` |
5252
| **Filtering** | Exclude sensitive sessions via rules file | `utils/exclusion_rules.py` |
5353
| **Statistics** | Aggregates for API and exporters | `utils/session_stats.py` |
5454
| **Export — Markdown** | Session → YAML-frontmatter Markdown | `utils/md_exporter.py` |
@@ -71,13 +71,13 @@
7171

7272
## Dispatch table
7373

74-
In `utils/jsonl_parser.py`, tool results are classified through `_parse_tool_result`, a **predicate-ordered dispatch table** (not a simple `if tool_name == ...` chain). **Order is load-bearing**: the first matching predicate wins. Tests in `tests/test_jsonl_parser.py` guard ordering regressions.
74+
In `utils/tool_dispatch.py`, tool results are classified through `_parse_tool_result`, a **predicate-ordered dispatch table** (not a simple `if tool_name == ...` chain). **Order is load-bearing**: the first matching predicate wins. Tests in `tests/test_jsonl_parser.py` and `tests/test_real_session_fixtures.py` guard ordering regressions.
7575

7676
When adding a new tool renderer:
7777

78-
1. Add predicate + builder pair in the dispatch table in the correct order (specific before generic).
79-
2. Add or extend a JSONL fixture under `tests/fixtures/` if needed.
80-
3. Run `pytest tests/test_jsonl_parser.py -v`.
78+
1. Add a `(predicate, builder)` pair to `_TOOL_RESULT_DISPATCH` in `utils/tool_dispatch.py`, preserving existing predicate order unless you also update fixtures and ordering tests (`tests/test_jsonl_parser.py`, `tests/test_real_session_fixtures.py`). Order is **not**specific before generic” in general — the first match wins. `_tool_result_pred_task_message` is the intentional broad-before-narrow exception (`task_id` or `message` before retrieval/completed/async).
79+
2. Add or extend a JSONL fixture under `tests/fixtures/` (especially for overlaps with existing predicates).
80+
3. Run `pytest tests/test_jsonl_parser.py tests/test_real_session_fixtures.py -v`.
8181

8282
## Export state machine
8383

tests/test_jsonl_parser.py

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -234,6 +234,16 @@ def test_plan_result(self):
234234
r = _parse_tool_result({"plan": [], "filePath": "/plan.md"})
235235
assert r["result_type"] == "plan"
236236

237+
def test_plan_with_content_not_classified_as_file_write(self):
238+
"""plan is registered before file_write in _TOOL_RESULT_DISPATCH."""
239+
r = _parse_tool_result({
240+
"plan": [],
241+
"filePath": "/plan.md",
242+
"content": "plan body",
243+
})
244+
assert r["result_type"] == "plan"
245+
assert r["file_path"] == "/plan.md"
246+
237247
def test_unknown_fallback(self):
238248
r = _parse_tool_result({"unexpected": True})
239249
assert r["result_type"] == "unknown"

utils/jsonl_helpers.py

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
"""Shared content helpers for JSONL parsing and session peek."""
2+
3+
import re
4+
from typing import Any
5+
6+
from models.session import MessageDict
7+
8+
9+
def entry_message(entry: dict[str, Any]) -> dict[str, Any]:
10+
m = entry.get("message")
11+
return m if isinstance(m, dict) else {}
12+
13+
14+
def normalize_content(content: Any) -> list[dict[str, Any]]:
15+
"""Content can be a plain string, a list of strings, or a list of typed
16+
blocks. Normalize everything into [{type, text}, ...] form."""
17+
if isinstance(content, str):
18+
return [{"type": "text", "text": content}]
19+
if isinstance(content, list):
20+
result = []
21+
for part in content:
22+
if isinstance(part, str):
23+
result.append({"type": "text", "text": part})
24+
elif isinstance(part, dict):
25+
result.append(part)
26+
return result
27+
return []
28+
29+
30+
def extract_text(content_parts: Any) -> str:
31+
"""Grab just the text blocks out of a content array, ignore tool_use/thinking."""
32+
parts = normalize_content(content_parts)
33+
texts = []
34+
for part in parts:
35+
if part.get("type") == "text":
36+
texts.append(part.get("text", ""))
37+
return "\n".join(texts)
38+
39+
40+
def extract_images(content_parts: Any) -> list[dict[str, Any]]:
41+
"""Pull base64 image blocks out of a content array.
42+
Also looks inside nested tool_result content blocks."""
43+
parts = normalize_content(content_parts)
44+
images = []
45+
for part in parts:
46+
if part.get("type") == "image":
47+
source = part.get("source", {})
48+
if source.get("type") == "base64" and source.get("data"):
49+
images.append({
50+
"media_type": source.get("media_type", "image/png"),
51+
"data": source["data"],
52+
})
53+
elif part.get("type") == "tool_result":
54+
# Nested content is usually a block list; string content is not normalized here.
55+
nested = part.get("content", [])
56+
if isinstance(nested, list):
57+
for sub in nested:
58+
if isinstance(sub, dict) and sub.get("type") == "image":
59+
source = sub.get("source", {})
60+
if source.get("type") == "base64" and source.get("data"):
61+
images.append({
62+
"media_type": source.get("media_type", "image/png"),
63+
"data": source["data"],
64+
})
65+
return images
66+
67+
68+
def first_title_line(text: str, max_chars: int = 100) -> str:
69+
"""First non-empty line after system-tag strip, truncated for session titles."""
70+
return strip_system_tags(text).strip().split("\n")[0][:max_chars]
71+
72+
73+
def infer_title(messages: list[MessageDict]) -> str:
74+
"""Use the first line of the first real user message as the session title."""
75+
for msg in messages:
76+
if msg["role"] == "user" and msg.get("text"):
77+
first_line = first_title_line(msg["text"])
78+
if first_line:
79+
return first_line
80+
return "Untitled Session"
81+
82+
83+
def strip_system_tags(text: str) -> str:
84+
"""Strip out the internal XML tags Claude Code injects (system-reminder,
85+
ide_opened_file, etc.) so exported text is clean."""
86+
# Remove block tags and their content
87+
for tag in (
88+
"system-reminder", "ide_opened_file", "user-prompt-submit-hook",
89+
"claude_background_info", "fast_mode_info", "env",
90+
):
91+
text = re.sub(rf"<{tag}>[\s\S]*?</{tag}>", "", text)
92+
# Strip remaining known opening/closing tags
93+
text = re.sub(
94+
r"</?(?:ide_selection|local-command-stdout|local-command-stderr|"
95+
r"command-name|antml:\w+|function_calls|example\w*)>",
96+
"",
97+
text,
98+
)
99+
return text.strip()

0 commit comments

Comments
 (0)