diff --git a/docs/block-range-annotations-design.html b/docs/block-range-annotations-design.html new file mode 100644 index 000000000..18b42a2f9 --- /dev/null +++ b/docs/block-range-annotations-design.html @@ -0,0 +1,626 @@ + + + + + + Block Range Annotations Design + + + +
+ + +

Block Range Annotations Design

+

A proposed model for Google Docs-style comments and future edit suggestions over selected block text. The design assumes individual blocks are usually small and that the app's durable unit of text ownership is the block, not a single document-wide text stream.

+ +

Goal

+

Let a user select text inside one block and create a comment thread attached to that selection. The selected text should be highlighted in read mode and edit mode where the renderer can map raw offsets safely. The data model should also allow a future block-range selection to store one segment per block. The comment thread should be discoverable from the target block, behave like normal user content, and leave room for later accept/reject edit suggestions.

+ +

Non-goals

+ + +

Current Substrate

+

The app already has most of the useful surfaces, but they should be composed carefully.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
AreaExisting shapeImplication
ReferencesBlockReference stores target id, alias, and optional sourceField; normalization treats references as a set. See blockData.ts.Use references for comment discovery, not for range anchoring.
Backlink indexblock_references is trigger-maintained from references_json with source_field. See references/localSchema.ts.A ref-typed property can make comment threads appear as typed backlinks to target blocks after reference projection updates references_json.
Read renderingMarkdown rendering is extended through markdownExtensionsFacet. See MarkdownContentRenderer.tsx.Read-mode highlights need a raw-offset-aware decoration layer. Markdown component overrides alone are not enough for syntax and alias mapping.
Edit renderingCodeMirror extensions are contributed per block. See codeMirrorExtensions.ts and CodeMirrorContentRenderer.tsx.Edit-mode highlights should be CodeMirror decorations over resolved block-local ranges.
SyncThe generated PowerSync config defines one stream (workspace_data) covering workspaces, workspace_members, and blocks — blocks staged e2ee as blocks_synced and materialized client-side. See powersync/sync-config.yaml.Block-backed threads avoid a first-pass sync/E2EE table migration.
+ +

Core Decisions

+
+
+

Threads are blocks

+

Comment and suggestion threads should be normal block subtrees stored under an annotation-owned container, not as children of the target block. Their visible comment children, replies, authorship, deletion, sync, and E2EE behavior then reuse the existing block model without polluting the outline.

+
+
+

Anchors are payloads

+

The selected range is metadata on the thread root. It is not embedded in markdown and not stored inside BlockReference.

+
+
+

Discovery uses refs

+

A ref-list property on the thread root points at all target blocks. The existing reference projection gives typed backlinks via sourceField, once the property is projected into references_json.

+
+
+

Decorations are derived

+

Highlights, markers, side panels, and suggestion diffs are UI derived from anchors. They are not storage.

+
+
+ +

Proposed Data Model

+

Each comment thread root is a block with a dedicated type, a deterministic annotation-container parent, and a small property payload.

+ + 
parentId: "annotation-container-for-workspace"
+content: ""
+properties: {
+  "types": ["annotation:thread"],   // blocks have no top-level types field; types live in the `types` property (typesProp)
+  "annotation:kind": "comment",
+  "annotation:status": "open",
+  "annotation:targetBlocks": ["block-a"],
+  "annotation:anchor": {
+    "version": 1,
+    "segments": [
+      {
+        "blockId": "block-a",
+        "from": 12,
+        "to": 31,
+        "exact": "selected text",
+        "prefix": "short text before",
+        "suffix": "short text after"
+      }
+    ]
+  }
+}
+children: first visible comment block, then reply blocks
+ +

annotation:targetBlocks should be a registered refList property. That lets thread roots become visible through the existing reference machinery, while annotation:anchor stores the range selectors.

+ +

For v1, the thread root should be a metadata container with empty content; the first visible comment is its first child. This keeps thread metadata, reply rendering, deletion, and future migrations independent of the body text of the first comment.

+ +

annotation:targetBlocks should be a deduped mirror of annotation:anchor.segments[].blockId. If a later feature needs block-level comments with no text segment, it should add an explicit block-level anchor kind instead of silently adding target blocks without resolvable segments.

+ +
+ Projection requirement: block_references indexes references_json, not raw property values. The annotation schema must register annotation:targetBlocks as a refList, and creation code should either write matching references in the same transaction for immediate discovery or rely on the references processor before querying through block_references. Tests that bypass processors must account for this explicitly. +
+ +

Thread Placement And Lifecycle

+

Thread roots need a tree location because they are normal blocks. The recommended v1 shape is one annotation-owned container per workspace, with thread roots as children and replies as descendants of the thread root. The target relationship is expressed only through annotation:targetBlocks, not through the outline parent.

+ +

The container reuses the existing system-page machinery, not a hand-rolled get-or-create:

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ConcernPolicy
Normal outline renderingAnnotation container subtrees are hidden from ordinary page/outline rendering and opened only through annotation surfaces.
Other enumerating surfacesThere is no central hidden-subtree seam today, and comment children are normal blocks with content: quick-find content search (core.searchByContent is workspace-wide FTS with no exclusion hook), recents (core.recentBlocks), breadcrumbs, and backlink group context each enumerate blocks independently. V1 should add one shared ancestor-based query-filter helper — replies carry no marker type, so filtering keys on descent from the deterministic container id, not per-row markers — and route these surfaces through it, filtering before SQL LIMIT (comment children are among the most recently updated rows, so post-limit filtering hollows out recents). Any surface not routed through the seam leaks annotation internals — an ongoing cost of the threads-are-blocks decision (see Storage Options).
Export and copyToday only subtree copy and the debug DB export exist; subtree copy starts from a user-selected block, so the container is naturally excluded. Exclusion of annotation subtrees is a policy for future exporters, not existing machinery to patch. A later explicit "include comments" mode can serialize threads alongside target anchors.
Target deletionDo not cascade-delete threads automatically. If all target blocks are deleted, hide the thread from normal annotation surfaces and expose it through a workspace-level orphaned/resolved-comments affordance that scans annotation threads by type instead of starting from a live target block (position-independent, like the merge-processor lookup in Block merges). Restoring a target un-orphans the thread naturally: property-derived refs are preserved on target deletion, so discovery is liveness-at-read-time.
Thread deletionDeleting a thread deletes its block subtree. No separate ref-clearing write is needed or wanted: the block_references triggers already de-index soft-deleted sources and re-index on restore, so an explicit ref-removal write would only break undo/restore symmetry. Resolving a thread only changes annotation:status.
Comment child deletionDeleting the last visible comment deletes the thread root subtree (the inverse of the create flow). This is a UI-level rule, not a data invariant: sync, agents, and generic block surfaces can still produce an open thread root with zero visible children, so annotation surfaces must tolerate and hide such threads rather than assume at least one comment child exists.
Multi-block commentsThe container model avoids choosing one target block as the structural owner; all target blocks are peers in annotation:targetBlocks.
+ +

Anchor Shape

+

Because blocks are usually small, anchors should be block-local and hybrid: offsets for the normal fast path, exact quote plus context for repair.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + +
FieldMeaningWhy it exists
blockIdThe block this segment decorates.Keeps anchoring local to durable block identity.
from, toOffsets in the same coordinate system used by the block editor: CodeMirror document positions / JavaScript string offsets, not UTF-8 bytes.Cheap resolution and direct CodeMirror decoration.
exactThe selected source text.Lets the anchor survive small edits and detect stale ranges.
prefix, suffixShort surrounding context.Disambiguates repeated text within a small block.
+ +

A contentHash leg was considered and dropped: whenever the hash would match, the offset check below necessarily also passes, and a range-length substring compare is cheaper than hashing the whole content — which is also async under WebCrypto and would force an otherwise synchronous resolver.

+ +

A future multi-block selection should store one segment per selected block. The UI can group those segments as one thread, but resolution stays per block. V1 creation should stay single-block until there is a block-range selection model that can turn a cross-block selection into ordered block-local raw offsets.

+ +

Resolution Algorithm

+

Anchor resolution should be simple and deterministic. Small blocks mean brute-force search is acceptable.

+ +
+
+ 1. Offset check + If content.slice(from, to) === exact, keep the stored offsets. +
+
+ 2. Exact search + Search for exact inside the same block. +
+
+ 3. Context tie-break + If multiple matches exist, rank by prefix and suffix match. +
+
+ 4. Stale fallback + If no unique match exists, return the segment as stale. +
+
+ +

Resolved ranges feed decorations. Stale anchors still keep the thread attached to their target blocks, but no text highlight is shown for the stale segment until it is repaired.

+ +

Anchor Maintenance

+

For a first implementation, anchors can be repaired lazily at render time and persisted only when the user explicitly repairs a stale anchor. Persisting a repair must be snapshot-atomic and intent-validating: inside the persisting transaction, re-locate the user-confirmed exact text in the in-transaction content and compute offsets from that one snapshot; if the confirmed text no longer matches uniquely, abort or mark the segment stale rather than storing whatever now sits at the old offsets. An internally consistent offsets/exact pair is not enough — it must be the text the user actually confirmed.

+ +

V1 resolution state should be derived: resolving an anchor returns resolved or stale, but the app persists only explicit user actions such as repair, resolve, reopen, accept, or reject. Do not persist routine stale resolver output as thread state unless a later design introduces an explicit invalidation field with clear write rules.

+ +

A later pass can add automatic remapping on block content edits; if it is, prefer a bounded same-transaction processor. Same-tx processors run inside the user's write transaction and can amend writes atomically, so an undo step can cover both the content edit and anchor update. See sameTxProcessor.ts.

+ +
+ Latency guard: automatic remapping must stay bounded. If a block has too many annotation segments, or repair is ambiguous, the processor should stop early and leave affected segments resolving as stale rather than making content saves feel slow. +
+ +

Structural edits should have an explicit v1 fallback. Moving, indenting, outdenting, or reordering a block preserves the block id and content, so block-local anchors should survive and only render-surface visibility may change. Splitting can move selected text to a different block id; until split retargeting exists, affected segments resolve as stale. Merging is handled by the annotation merge processor below.

+ +

Block merges

+

Merge handling needs special care because half-measures provably flip-flop. The generic retargeting processor (mergeRetargetProcessor.ts) rewrites stored BlockReference entries pointing at the merged-away block on live sources (tombstoned sources are skipped and stay consistently stale) — including the thread root's projected annotation:targetBlocks ref — but never touches annotation properties or anchor payloads, and it has no opt-out seam. Reference projection is recompute-authoritative for registered property schemas, so the next property write on the thread root (a resolve, a repair) reprojects from the stale property value and flips the projected ref back to the dead block id: the comment silently vanishes from the merged block, arbitrarily long after the merge. Merge invariant: projected refs, annotation:targetBlocks, and anchor segment blockIds must stay aligned.

+ +
+ Decision: an annotation same-transaction merge processor retargets annotation:targetBlocks, the projected refs, and every affected anchor segment's blockId together — preserving the mirror invariant — with retargeted segments then resolving against the merged content normally (stale if the selected text did not survive the merge). It ships in the same slice as comment creation, not as later hardening: every merge between the two shipping corrupts thread targeting. +
+ +

How the processor finds affected threads matters. It must not look them up through block_references by target_id = fromId: the generic retarget processor watches the same merge event and may run first, and the block_references triggers update the index within the transaction — so by the time the annotation processor runs, the row it is looking for can already point at the merged-into block, and the processor silently no-ops, shipping the exact flip-flop it exists to prevent. Instead, scan live annotation:thread blocks by type — via the trigger-maintained block_types index — whose annotation:targetBlocks property value contains the merged-away id (in-transaction SQL over properties_json). That lookup is registration-order-independent, position-independent (a thread root that sync or an agent reparented out of the container is still retargeted; a container-children scan would miss it), and unaffected by the refs/property divergence window on sync-composed rows (see Concurrency And Sync Merge).

+ +

The merge processor is an accepted exception to the minimize-automatic-writers policy below, and it does not make merges fully safe across clients: a merge on one client races any thread-root property write on another (whole-map LWW can drop the retarget wholesale), and a thread created offline targeting a block that another client concurrently merges away is never retargeted by anyone. In both terminal states the merged-away target is tombstoned, so the thread surfaces through the workspace-level orphaned-comments affordance — that affordance, not the processor, is the recovery path for merge races.

+ +

Concurrency And Sync Merge

+

Sync merges concurrent writes column-LWW per row: the server RPC coalesces each changed column independently, and per-key properties_json merge is deliberately out of scope there (see apply_block_patches and the plan it references). Locally, every property write replaces the whole properties map. This design therefore concentrates several independently-mutable concerns — kind, status, target refs, anchor, patch — in one column of one row, and concurrent writers lose updates wholesale: if client A resolves a thread while client B, offline or concurrent, repairs the same thread's anchor, whichever properties_json lands later wins the entire map. The resolution is silently reopened, or the repair silently reverted. A repair racing a suggestion accept can revert annotation:status from accepted while the content replacement stays applied.

+ +

V1 policy: accept and document LWW for annotation metadata, and minimize automatic writers so the races stay rare — lazy repair persisted only on explicit user action is part of this, the merge processor is the one accepted automatic writer (see Block merges), and any automatic remapping processor widens the contention surface and must be weighed against it. Two mitigating facts: because the whole map is one column, the property-side trio — annotation:targetBlocks, anchor, status — can never tear against each other through sync (projected refs live in a separate column; see the caveat below) — the failure mode is lost updates, not torn state; and thread roots are low-traffic compared to content blocks. If annotation contention becomes real, the graduation paths are the per-key properties merge plan referenced by the RPC, or the synced annotations table from Storage Options.

+ +

A related consistency caveat: rows composed from sync are materialized outside repo.tx, so reference projection never runs on remote changes, and the server can merge properties_json from one writer with references_json from another. Projected refs and annotation:targetBlocks can therefore disagree on every client until the next local content/properties write to the thread root triggers reprojection. The refs-mirror-property invariant is eventually consistent — converged only by local write-path reprojection — and query helpers and tests must not assume it holds at every instant.

+ +

UI Model

+

The UI should present comments as a layer over normal blocks, not as extra inline markdown syntax.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
SurfaceBehavior
CreationV1 creation should start from a non-collapsed edit-mode selection, where CodeMirror already gives raw block offsets. Range anchors require from < to and a non-empty exact. Read-mode creation should wait for a reliable rendered-selection to raw-offset mapping. Creation must also mind the editor's debounced commit: at comment time the repo row can lag the buffer, so flush or await the pending editor commit (or write the buffer content in the same transaction as the anchor) rather than validating the anchor against stale repo content. Caret or block-level comments wait for the explicit block-level anchor kind (see the data model).
Read modeResolved segments render as subtle inline highlights only where the raw-offset mapping layer can preserve markdown syntax and reference aliases. Otherwise, show a margin marker or block-level count that opens the relevant thread list.
Edit modeResolved segments render as CodeMirror decorations. Selection and typing stay native to CodeMirror.
Thread panelClicking a highlight or marker opens a side panel or popover showing the thread root and reply children — via the repo's UI channels: openDialog from @/utils/dialogs for a one-shot popover, or a createToggleStore module store for a persistent panel.
Backlinks viewThread-root refs should not pollute the default linked references view. Current backlinks queries include all sourceField values; flat and grouped backlinks can suppress annotation-thread sources with a built-in referencedBy.sourceField exclusion, but inline backlink counts and any mixed-source edge semantics still need query/API work. The comments section separately queries sourceField = "annotation:targetBlocks". Refs typed inside comment bodies are a separate policy — see the note below the table.
Resolved commentsResolved threads remain blocks, but are hidden or collapsed by default in annotation surfaces.
+ +
+ Comment-body refs: the sourceField exclusion covers only the thread root's projected refs. A wikilink or ((ref)) typed inside a comment body is an ordinary content ref, indistinguishable from outline refs, and will surface in the target's linked references — with grouped backlinks rendering its ancestor chain up through the annotation container — and [[New Page]] in a comment mints a real page. V1 policy: comment-body refs are real graph edges and may appear (a comment referencing a block is a genuine mention). Suppression needs no new query machinery — the typed-query layer already expresses it as an exclude predicate with {scope: 'ancestor', id: containerId} — so keeping them visible is a pure policy choice, with that predicate as the cheap fallback if the behavior proves confusing. Group-context rendering of thread ancestors goes through the hiding seam. +
+ +

Render Surface Policy

+

Annotation decorations should be scoped by render surface. Primary document and focused edit surfaces show inline highlights when they can resolve raw offsets correctly. Secondary surfaces such as backlinks, embeds, breadcrumbs, and block references default to compact markers, counts, or no annotation decoration. Any reveal behavior should be render-time policy, not persisted collapse-state mutation.

+ +

Edit Suggestions

+

Edit suggestions can use the same thread and anchor model with a different kind and a patch payload.

+ + 
{
+  "annotation:kind": "suggestion",
+  "annotation:status": "open",
+  "annotation:anchor": { "version": 1, "segments": [...] },
+  "annotation:patch": {
+    "op": "replace",
+    "deletedText": "old text",
+    "insertedText": "new text"
+  }
+}
+ +

Accepting a suggestion should re-resolve the anchor, apply the replacement to the resolved anchor range, verify that the current text still matches the expected deleted text, then perform a normal block content update and mark the suggestion accepted in the same user-visible operation. Accept must also rewrite the affected anchor segment in that same transaction, from the same content snapshot — offsets spanning the inserted text, exact = the inserted text, fresh prefix/suffix — so the accepted suggestion's highlight tracks the inserted text instead of going stale or re-matching a leftover occurrence of the deleted text elsewhere in the block. V1 suggestions require a non-empty insertedText: a deletion-shaped patch has no text to re-anchor — the rewritten segment would be a collapsed range with empty exact, which the anchor rules forbid and which would make "the anchor resolves" trivially true — so pure deletions are deferred until they get an explicit consumed-segment state.

+ +

If the target block is mounted in edit mode, accept must be editor-mediated: dispatch the replacement into the mounted CodeMirror document and coordinate or cancel any stale debounced commit, or introduce an awaited editor commit API that can persist the editor document plus suggestion status in the same undoable operation. A repo-only write is not enough because the uncontrolled editor can later flush stale text over it. The two shapes are not equivalent for undo: the editor's own commit cannot join an undo group (repo.undoGroup covers only writes issued through its facade), so one undo step needs either the awaited commit API, or dispatch-into-editor plus cancel-the-debounce followed by one repo.tx writing the target content and the thread-root properties (status plus rewritten anchor) together — a single tx is one undo step on its own and preserves the same-transaction rule above, which repo.undoGroup alone would not (it composes separate mutator calls into one undo entry, not one transaction). The editor buffer already matches the dispatched text, so a later flush is a no-op. The patch payload should not carry independent coordinates that can drift from the anchor. Rejection only changes suggestion status.

+ +

These mitigations are local-only by construction. If another client has the target block mounted in edit mode, its editor holds pre-accept text, defers external content while its user types, and its next debounced full-content commit overwrites the applied replacement — while annotation:status stays accepted. The same holds for any concurrent offline edit of the target block. V1 accepts this as ordinary content LWW. Because accept rewrites the anchor to the inserted text, "still in effect" is simply "the anchor resolves": annotation surfaces can flag an accepted suggestion whose segment no longer resolves to the inserted text, instead of silently misreporting it — no separate expected-content field is needed. A later legitimate edit of the accepted text makes the suggestion read as superseded; that decay is intentional. Re-verification and repair stay manual.

+ +
+ Scope recommendation: start with single-block replace suggestions. Multi-block structural suggestions should wait until there is a clear operation model for block splitting, merging, and subtree edits. +
+ +

Storage Options Considered

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
OptionProsConsDecision
Thread blocks with anchor propertyUses existing block sync, undo, E2EE, deletion, and child replies. Queryable through refList backlinks once reference projection is handled.Anchor payload is less queryable than a table; thread metadata shares one properties column, so concurrent annotation actions merge LWW (see Concurrency And Sync Merge); every block-enumerating surface needs explicit exclusions via the shared hiding seam (see Thread Placement), and unpatched surfaces leak.Recommended v1.
Target block stores all annotationsSimple to find annotations for one block.Every comment churns the target block row; properties can grow unbounded; undo/sync couples annotation edits to target content.Reject.
Synced annotations tableBest query shape and isolated annotation churn.Requires hosted schema, RLS, upload/apply path, PowerSync stream, E2EE handling, and migration work.Defer until scale demands it.
Inline hidden markersAnchors move with source text.Pollutes markdown, leaks implementation details into export/copy, and makes normal editing harder.Reject.
+ +

Implementation Slices

+

Annotation code lives in a new plugin (e.g. src/plugins/annotations/), following the references/backlinks pattern. The hiding seam needs kernel hooks only for the non-typed-query surfaces (FTS search, recents); typed-query surfaces can use the existing ancestor-scope exclude predicate, and breadcrumbs/backlink group context filter at render level through the shared helper.

+

V1 comments

+
    +
  1. Define annotation schemas (defineProperty + codecs.refList(), as existing plugins do): thread type, annotation container type, target refList property, anchor payload property, status/kind properties.
    2. +
  2. Add the annotation container lifecycle (systemPagesFacet ensure, kernel-page-style deterministic id, no alias, no page type) plus the shared hiding seam, routing the enumerating surfaces from the Thread Placement table through it.
    3. +
  3. Add annotation UI lifecycle rules for first/last visible comment child deletion.
    4. +
  4. Add creation writes that keep target refs discoverable immediately or explicitly wait for reference projection convergence.
    5. +
  5. Add anchor utilities: create segments from CodeMirror selection, resolve segment against block content, return stale resolver results, and repair by exact/context search.
    6. +
  6. Add the annotation merge processor (same-transaction) that retargets annotation:targetBlocks, projected refs, and anchor segment blockIds together — this ships with creation, per Block merges.
    7. +
  7. Add query helpers for "annotation threads targeting block X" via the existing reference index and sourceField.
    8. +
  8. Extend backlinks behavior so default flat backlinks, grouped backlinks, and inline counts can suppress annotation-thread sources.
    9. +
  9. Add read-mode markers and, where rendered text can be mapped back to raw offsets, inline highlights over resolved ranges.
    10. +
  10. Add edit-mode highlights through a CodeMirror decoration extension.
    11. +
  11. Add create-comment flow from an edit-mode selection and render a minimal thread panel.
    12. +
  12. Add resolve/reopen actions, stale-anchor UI, and a workspace-level orphan query over annotation threads by type.
    13. +
+ +

Later suggestions

+
    +
  1. Add single-block replace suggestions once comments are stable.
    2. +
+ +

Testing Plan

+ + +

Open Questions

+ + +

Related Prior Art

+ +
+ +