Skip to content

Commit f9bc84f

Browse files
RtlZeroMemoryclaude
andcommitted
feat(core): add createMarkdownStream for append-only markdown
Streams agent transcripts and live logs cheaply: under append-only input every top-level block except the last is immutable, so the stream re-parses only the volatile tail block per append and caches completed blocks plus their rendered VNodes. Completed blocks keep referential identity (including the block that just stopped being last, restored via structural comparison), so reconciliation and layout stability signatures skip untouched subtrees and appends stay O(tail) instead of O(document). Chunk boundaries never change the result: CRLF pairs split across appends are carried, NUL bytes are sanitized per chunk, and fuzz tests pin document() to deep-equal parseMarkdown(source()) under random chunking, interleaved queries, resets, and unicode soup. Exports createMarkdownStream/MarkdownStream/renderMarkdownBlock from core; parseMarkdownLines stays module-internal with per-block start tracking in the block parser. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent 12eecd6 commit f9bc84f

11 files changed

Lines changed: 515 additions & 16 deletions

File tree

CHANGELOG.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,8 @@ The format is based on Keep a Changelog and the project follows Semantic Version
88

99
### Added
1010

11-
- **core/markdown**: New `ui.markdown(source, props?)` widget (`experimental` tier) rendering a GitHub-Flavored-Markdown subset onto existing widgets: headings, paragraphs with wrapping, fenced/indented code with tokenizer-based monochrome syntax emphasis, nested and task lists, blockquotes, pipe tables with alignment, thematic breaks, and inline strong/em/del/code/links/autolinks/entities. The zero-dependency parser is bounded and fuzz-tested, never throws on untrusted input, and is exported as `parseMarkdown`/`renderMarkdown` with frozen `MarkdownDocument` AST types for caller-side caching (streaming). JSX exposes `<Markdown source="…" />`.
11+
- **core/markdown**: New `ui.markdown(source, props?)` widget (`experimental` tier) rendering a GitHub-Flavored-Markdown subset onto existing widgets: headings, paragraphs with wrapping, fenced/indented code with tokenizer-based monochrome syntax emphasis, nested and task lists, blockquotes, pipe tables with alignment, thematic breaks, and inline strong/em/del/code/links/autolinks/entities. The zero-dependency parser is bounded and fuzz-tested, never throws on untrusted input, and is exported as `parseMarkdown`/`renderMarkdown`/`renderMarkdownBlock` with frozen `MarkdownDocument` AST types. JSX exposes `<Markdown source="…" />`.
12+
- **core/markdown**: `createMarkdownStream()` for append-only sources (agent transcripts, live logs): re-parses only the volatile tail block per append and caches completed blocks plus their rendered VNodes with referential identity, keeping appends O(tail). Chunk boundaries (including split CRLF pairs) never change the result — `document()` always deep-equals `parseMarkdown(source())`, pinned by fuzz tests.
1213

1314
## [0.1.0-beta.1] - 2026-06-11
1415

ROADMAP.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ This roadmap is directional only. It is not a commitment to scope, ordering, or
1414

1515
- Graduate or prune `experimental`-tier widgets so the 1.0 surface is intentional
1616
- Freeze drawlist and event protocol guarantees together with Zireael
17-
- Streaming mode for `ui.markdown` and an agent-console template
17+
- An agent-console template showcasing streamed markdown transcripts
1818
- Developer tooling: layout, focus, and frame-timing inspection overlay
1919
- Code coverage reporting and blocking benchmark regression gates in CI
2020

docs/widgets/markdown.md

Lines changed: 30 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -53,13 +53,39 @@ with flanking checks.
5353
hard-coded.
5454
- Styled paragraphs wrap at word boundaries; unstyled paragraphs use
5555
grapheme-safe `text` wrapping.
56-
- For streamed or frequently re-rendered documents, pre-parse with
57-
`parseMarkdown(source)` and render cached documents with
58-
`renderMarkdown(doc, options)`; completed blocks of an append-only stream
59-
can be cached by the caller. A built-in streaming mode is planned.
56+
- For append-only sources use `createMarkdownStream()` (see below); for
57+
static documents that re-render often, pre-parse with
58+
`parseMarkdown(source)` and render with `renderMarkdown(doc, options)`.
6059
- The parsed `MarkdownDocument` AST (`MarkdownBlock` / `MarkdownInline`) is
6160
exported and deeply frozen.
6261

62+
## Streaming
63+
64+
For append-only sources — agent transcripts, live logs — use
65+
`createMarkdownStream(options?)`. Under append-only input every top-level
66+
block except the last is immutable, so the stream re-parses only the tail
67+
block per append and caches both parsed blocks and their rendered VNodes.
68+
Completed blocks keep referential identity, so reconciliation and layout
69+
stability signatures skip untouched subtrees; appends stay O(tail) instead of
70+
O(document).
71+
72+
```ts
73+
import { createMarkdownStream } from "@rezi-ui/core";
74+
75+
const stream = createMarkdownStream({ blockGap: 1 });
76+
77+
onTokens((chunk) => {
78+
stream.append(chunk); // chunks may split lines, CRLF pairs, or words
79+
app.update((s) => ({ ...s, transcriptRev: s.transcriptRev + 1 }));
80+
});
81+
82+
app.view((state) => ui.box({ border: "none" }, [stream.vnode()]));
83+
```
84+
85+
The stream guarantees that `stream.document()` always deep-equals
86+
`parseMarkdown(stream.source())` regardless of chunk boundaries.
87+
`reset(source?)` clears the buffer; `document()` exposes the parsed AST.
88+
6389
## Related
6490

6591
- [RichText](rich-text.md)

packages/core/etc/core.api.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1194,6 +1194,9 @@ export function createLayerStackState(): LayerStackState;
11941194
// @public
11951195
export function createLoadingState(initial?: readonly string[]): LoadingState;
11961196

1197+
// @public
1198+
export function createMarkdownStream(options?: MarkdownStreamOptions): MarkdownStream;
1199+
11971200
// @public
11981201
export function createReproReplayDriver(opts: ReproReplayDriverOptions): ReproReplayDriver;
11991202

@@ -3954,6 +3957,18 @@ export type MarkdownRenderOptions = Readonly<{
39543957
blockGap?: number;
39553958
}>;
39563959

3960+
// @public
3961+
export type MarkdownStream = Readonly<{
3962+
append: (chunk: string) => void;
3963+
reset: (source?: string) => void;
3964+
source: () => string;
3965+
document: () => MarkdownDocument;
3966+
vnode: () => VNode;
3967+
}>;
3968+
3969+
// @public
3970+
export type MarkdownStreamOptions = MarkdownRenderOptions;
3971+
39573972
// @public
39583973
export type MarkdownTableAlign = "left" | "center" | "right";
39593974

@@ -4531,6 +4546,9 @@ export function renderHorizontalScrollbar(width: number, state: ScrollbarState,
45314546
// @public
45324547
export function renderMarkdown(doc: MarkdownDocument, options?: MarkdownRenderOptions): VNode;
45334548

4549+
// @public
4550+
export function renderMarkdownBlock(block: MarkdownBlock, options?: MarkdownRenderOptions): VNode;
4551+
45344552
// Warning: (ae-forgotten-export) The symbol "ResolvedTextStyle" needs to be exported by the entry point index.d.ts
45354553
//
45364554
// @public

packages/core/src/index.ts

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -339,9 +339,16 @@ export type {
339339
MarkdownInline,
340340
MarkdownListItem,
341341
MarkdownRenderOptions,
342+
MarkdownStream,
343+
MarkdownStreamOptions,
342344
MarkdownTableAlign,
343345
} from "./widgets/markdown/index.js";
344-
export { parseMarkdown, renderMarkdown } from "./widgets/markdown/index.js";
346+
export {
347+
createMarkdownStream,
348+
parseMarkdown,
349+
renderMarkdown,
350+
renderMarkdownBlock,
351+
} from "./widgets/markdown/index.js";
345352
export type {
346353
InspectorOverlayFrameTiming,
347354
InspectorOverlayPosition,
Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
import { assert, describe, test } from "@rezi-ui/testkit";
2+
import { type Rng, chance, pick, randomAsciiString, randomInt, runFuzz } from "@rezi-ui/testkit";
3+
import { createTestRenderer } from "../../../testing/index.js";
4+
import { parseMarkdown } from "../parse.js";
5+
import { renderMarkdown } from "../render.js";
6+
import { createMarkdownStream } from "../stream.js";
7+
8+
const MARKDOWN_ALPHABET = "ab c\n*_~`#>-|[]()!\\<>&;:.0123456789\t\"'=+xhttps/\r";
9+
10+
const SNIPPETS = [
11+
"# heading",
12+
"paragraph **bold** `code` text",
13+
"[link](https://example.com)",
14+
"- one\n- two\n - nested",
15+
"1. a\n2. b",
16+
"- [x] done",
17+
"> quote line",
18+
"```ts\nconst a = 1;\n```",
19+
" indented code",
20+
"| a | b |\n| - | - |\n| 1 | 2 |",
21+
"---",
22+
"hard \nbreak",
23+
] as const;
24+
25+
function buildDoc(rng: Rng): string {
26+
const parts: string[] = [];
27+
const count = randomInt(rng, 1, 6);
28+
for (let i = 0; i < count; i++) {
29+
parts.push(
30+
chance(rng, 70)
31+
? pick(rng, SNIPPETS)
32+
: randomAsciiString(rng, { maxLength: 60, alphabet: MARKDOWN_ALPHABET }),
33+
);
34+
}
35+
return parts.join(chance(rng, 75) ? "\n\n" : "\n");
36+
}
37+
38+
/** Appends `source` to a fresh stream in random-sized chunks. */
39+
function appendInRandomChunks(
40+
rng: Rng,
41+
stream: ReturnType<typeof createMarkdownStream>,
42+
source: string,
43+
): void {
44+
let i = 0;
45+
while (i < source.length) {
46+
const size = randomInt(rng, 1, 24);
47+
stream.append(source.slice(i, i + size));
48+
i += size;
49+
}
50+
}
51+
52+
describe("markdown stream fuzz", () => {
53+
test("randomly chunked appends always match whole-source parsing", async () => {
54+
await runFuzz(
55+
{ label: "markdown-stream-chunk-equivalence", seed: 0x6d73_0001, iterations: 200 },
56+
(ctx) => {
57+
const source = buildDoc(ctx.rng);
58+
const stream = createMarkdownStream();
59+
appendInRandomChunks(ctx.rng, stream, source);
60+
assert.deepEqual(stream.document(), parseMarkdown(stream.source()));
61+
// Querying the document mid-stream must not corrupt later appends.
62+
const extra = buildDoc(ctx.rng);
63+
appendInRandomChunks(ctx.rng, stream, extra);
64+
assert.deepEqual(stream.document(), parseMarkdown(stream.source()));
65+
},
66+
);
67+
});
68+
69+
test("interleaved document() and vnode() queries keep render equality", async () => {
70+
await runFuzz(
71+
{ label: "markdown-stream-render-equality", seed: 0x6d73_0002, iterations: 60 },
72+
(ctx) => {
73+
const stream = createMarkdownStream();
74+
const rounds = randomInt(ctx.rng, 1, 4);
75+
for (let round = 0; round < rounds; round++) {
76+
appendInRandomChunks(ctx.rng, stream, buildDoc(ctx.rng));
77+
if (chance(ctx.rng, 50)) stream.document();
78+
if (chance(ctx.rng, 30)) stream.vnode();
79+
if (chance(ctx.rng, 10)) stream.reset(buildDoc(ctx.rng));
80+
}
81+
const renderer = createTestRenderer({ viewport: { cols: 40, rows: 30 } });
82+
const streamed = renderer.render(stream.vnode()).toText();
83+
const whole = createTestRenderer({ viewport: { cols: 40, rows: 30 } })
84+
.render(renderMarkdown(parseMarkdown(stream.source())))
85+
.toText();
86+
assert.equal(streamed, whole);
87+
},
88+
);
89+
});
90+
91+
test("unicode soup chunked at arbitrary boundaries stays equivalent", async () => {
92+
await runFuzz(
93+
{ label: "markdown-stream-unicode", seed: 0x6d73_0003, iterations: 100 },
94+
(ctx) => {
95+
const length = randomInt(ctx.rng, 0, 160);
96+
let source = "";
97+
for (let i = 0; i < length; i++) {
98+
const code = randomInt(ctx.rng, 1, 0x2fff);
99+
source += code >= 0xd800 && code <= 0xdfff ? "�" : String.fromCodePoint(code);
100+
}
101+
const stream = createMarkdownStream();
102+
appendInRandomChunks(ctx.rng, stream, source);
103+
assert.deepEqual(stream.document(), parseMarkdown(stream.source()));
104+
},
105+
);
106+
});
107+
});
Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
import { assert, describe, test } from "@rezi-ui/testkit";
2+
import { createTestRenderer } from "../../../testing/index.js";
3+
import { ui } from "../../ui.js";
4+
import { parseMarkdown } from "../parse.js";
5+
import { createMarkdownStream } from "../stream.js";
6+
7+
const KITCHEN_SINK = [
8+
"# Title",
9+
"",
10+
"Some **bold** text with `code` and [a link](https://rezitui.dev).",
11+
"",
12+
"- item one",
13+
"- item two",
14+
" - nested",
15+
"",
16+
"> quoted",
17+
"",
18+
"```ts",
19+
"const a = 1; // comment",
20+
"```",
21+
"",
22+
"| a | b |",
23+
"| - | -: |",
24+
"| 1 | 2 |",
25+
"",
26+
"Tail paragraph https://example.com.",
27+
].join("\n");
28+
29+
function renderText(vnode: Parameters<ReturnType<typeof createTestRenderer>["render"]>[0]): string {
30+
return createTestRenderer({ viewport: { cols: 44, rows: 36 } }).render(vnode).toText();
31+
}
32+
33+
describe("createMarkdownStream", () => {
34+
test("char-by-char appends match whole-source parsing", () => {
35+
const stream = createMarkdownStream();
36+
for (const ch of KITCHEN_SINK) stream.append(ch);
37+
assert.deepEqual(stream.document(), parseMarkdown(KITCHEN_SINK));
38+
assert.equal(stream.source(), KITCHEN_SINK);
39+
});
40+
41+
test("chunked appends match whole-source parsing at several widths", () => {
42+
for (const width of [3, 7, 16, 64]) {
43+
const stream = createMarkdownStream();
44+
for (let i = 0; i < KITCHEN_SINK.length; i += width) {
45+
stream.append(KITCHEN_SINK.slice(i, i + width));
46+
}
47+
assert.deepEqual(stream.document(), parseMarkdown(KITCHEN_SINK), `width ${width}`);
48+
}
49+
});
50+
51+
test("CRLF pairs split across chunk boundaries normalize correctly", () => {
52+
const stream = createMarkdownStream();
53+
stream.append("alpha\r");
54+
stream.append("\nbeta\r");
55+
stream.append("\n\r\ngamma");
56+
assert.deepEqual(stream.document(), parseMarkdown("alpha\r\nbeta\r\n\r\ngamma"));
57+
});
58+
59+
test("appending can extend the last block across blank lines", () => {
60+
const stream = createMarkdownStream();
61+
stream.append("- item\n\n");
62+
const before = stream.document();
63+
assert.equal(before.blocks.length, 1);
64+
stream.append(" continuation");
65+
assert.deepEqual(stream.document(), parseMarkdown("- item\n\n continuation"));
66+
});
67+
68+
test("an unclosed fence keeps absorbing appends until it closes", () => {
69+
const stream = createMarkdownStream();
70+
stream.append("```ts\nconst a");
71+
const open = stream.document().blocks[0];
72+
assert.ok(open !== undefined && open.kind === "codeBlock");
73+
assert.equal(open.text, "const a");
74+
stream.append(" = 1;\n```\nafter");
75+
assert.deepEqual(stream.document(), parseMarkdown("```ts\nconst a = 1;\n```\nafter"));
76+
});
77+
78+
test("completed blocks keep referential identity across appends", () => {
79+
const stream = createMarkdownStream();
80+
stream.append("# one\n\ntwo\n\n# three\n\n");
81+
const before = stream.document().blocks;
82+
stream.append("growing tail");
83+
stream.append(" with more words");
84+
const after = stream.document().blocks;
85+
assert.equal(after[0], before[0]);
86+
assert.equal(after[1], before[1]);
87+
assert.equal(after[2], before[2]);
88+
});
89+
90+
test("vnode() reuses cached VNodes for completed blocks", () => {
91+
const stream = createMarkdownStream();
92+
stream.append("# head\n\nstable paragraph\n\ntail");
93+
const first = stream.vnode();
94+
stream.append(" grows");
95+
const second = stream.vnode();
96+
assert.ok(first.kind === "column" && second.kind === "column");
97+
assert.equal(second.children?.[0], first.children?.[0], "heading vnode cached");
98+
assert.equal(second.children?.[1], first.children?.[1], "paragraph vnode cached");
99+
assert.notEqual(second.children?.[2], first.children?.[2], "tail re-rendered");
100+
});
101+
102+
test("vnode() renders identically to ui.markdown over the same source", () => {
103+
const stream = createMarkdownStream({ blockGap: 2 });
104+
for (let i = 0; i < KITCHEN_SINK.length; i += 11) {
105+
stream.append(KITCHEN_SINK.slice(i, i + 11));
106+
}
107+
const streamed = renderText(stream.vnode());
108+
const whole = renderText(ui.markdown(stream.source(), { blockGap: 2 }));
109+
assert.equal(streamed, whole);
110+
});
111+
112+
test("reset clears state and accepts replacement source", () => {
113+
const stream = createMarkdownStream();
114+
stream.append("# old\n\ncontent");
115+
stream.reset("# fresh");
116+
assert.deepEqual(stream.document(), parseMarkdown("# fresh"));
117+
assert.equal(stream.source(), "# fresh");
118+
stream.reset();
119+
assert.equal(stream.source(), "");
120+
assert.equal(stream.document().blocks.length, 0);
121+
});
122+
123+
test("empty appends and empty streams are safe", () => {
124+
const stream = createMarkdownStream();
125+
stream.append("");
126+
assert.equal(stream.document().blocks.length, 0);
127+
assert.equal(renderText(stream.vnode()).trim(), "");
128+
stream.append("text");
129+
stream.append("");
130+
assert.deepEqual(stream.document(), parseMarkdown("text"));
131+
});
132+
133+
test("document results stay deeply frozen", () => {
134+
const stream = createMarkdownStream();
135+
stream.append("# a\n\nb");
136+
const doc = stream.document();
137+
assert.ok(Object.isFrozen(doc));
138+
assert.ok(Object.isFrozen(doc.blocks));
139+
assert.ok(Object.isFrozen(doc.blocks[0]));
140+
});
141+
});

packages/core/src/widgets/markdown/index.ts

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,12 @@ export type {
1414
MarkdownTableAlign,
1515
} from "./ast.js";
1616
export { parseMarkdown } from "./parse.js";
17-
export { type MarkdownRenderOptions, renderMarkdown } from "./render.js";
17+
export { type MarkdownRenderOptions, renderMarkdown, renderMarkdownBlock } from "./render.js";
18+
export {
19+
createMarkdownStream,
20+
type MarkdownStream,
21+
type MarkdownStreamOptions,
22+
} from "./stream.js";
1823

1924
/**
2025
* Renders a GitHub-Flavored-Markdown subset onto existing widgets.

0 commit comments

Comments
 (0)