Feature/editorial mark attributes (#64)

* document editorial mark attributes, fix qmd roundtripping

Author: cscheid

.gitignore

@@ -1 +1,2 @@
-target
+target
+.beads

CLAUDE.md

@@ -3,6 +3,96 @@
 The main documentation for this repository is located at:
 [crates/quarto-markdown-pandoc/CLAUDE.md](crates/quarto-markdown-pandoc/CLAUDE.md)
 
+## **WORK TRACKING**
+
+We use bd (beads) for issue tracking instead of Markdown TODOs or external tools.
+
+### Quick Reference
+
+```bash
+# Find ready work (no blockers)
+bd ready --json
+
+# Create new issue
+bd create "Issue title" -t bug|feature|task -p 0-4 -d "Description" --json
+
+# Create with explicit ID (for parallel workers)
+bd create "Issue title" --id worker1-100 -p 1 --json
+
+# Create with labels
+bd create "Issue title" -t bug -p 1 -l bug,critical --json
+
+# Create multiple issues from markdown file
+bd create -f feature-plan.md --json
+
+# Update issue status
+bd update <id> --status in_progress --json
+
+# Link discovered work (old way)
+bd dep add <discovered-id> <parent-id> --type discovered-from
+
+# Create and link in one command (new way)
+bd create "Issue title" -t bug -p 1 --deps discovered-from:<parent-id> --json
+
+# Label management
+bd label add <id> <label> --json
+bd label remove <id> <label> --json
+bd label list <id> --json
+bd label list-all --json
+
+# Filter issues by label
+bd list --label bug,critical --json
+
+# Complete work
+bd close <id> --reason "Done" --json
+
+# Show dependency tree
+bd dep tree <id>
+
+# Get issue details
+bd show <id> --json
+
+# Import with collision detection
+bd import -i .beads/issues.jsonl --dry-run             # Preview only
+bd import -i .beads/issues.jsonl --resolve-collisions  # Auto-resolve
+```
+
+### Workflow
+
+1. **Check for ready work**: Run `bd ready` to see what's unblocked
+2. **Claim your task**: `bd update <id> --status in_progress`
+3. **Work on it**: Implement, test, document
+4. **Discover new work**: If you find bugs or TODOs, create issues:
+   - Old way (two commands): `bd create "Found bug in auth" -t bug -p 1 --json` then `bd dep add <new-id> <current-id> --type discovered-from`
+   - New way (one command): `bd create "Found bug in auth" -t bug -p 1 --deps discovered-from:<current-id> --json`
+5. **Complete**: `bd close <id> --reason "Implemented"`
+6. **Export**: Changes auto-sync to `.beads/issues.jsonl` (5-second debounce)
+
+### Issue Types
+
+- `bug` - Something broken that needs fixing
+- `feature` - New functionality
+- `task` - Work item (tests, docs, refactoring)
+- `epic` - Large feature composed of multiple issues
+- `chore` - Maintenance work (dependencies, tooling)
+
+### Priorities
+
+- `0` - Critical (security, data loss, broken builds)
+- `1` - High (major features, important bugs)
+- `2` - Medium (nice-to-have features, minor bugs)
+- `3` - Low (polish, optimization)
+- `4` - Backlog (future ideas)
+
+### Dependency Types
+
+- `blocks` - Hard dependency (issue X blocks issue Y)
+- `related` - Soft relationship (issues are connected)
+- `parent-child` - Epic/subtask relationship
+- `discovered-from` - Track issues discovered during work
+
+Only `blocks` dependencies affect the ready work queue.
+
 ## **CRITICAL - TEST-DRIVEN DEVELOPMENT**
 
 When fixing ANY bug:
@@ -11,11 +101,11 @@ When fixing ANY bug:
 3. **THIRD**: Implement the fix
 4. **FOURTH**: Run the test and verify it passes
 
-**This is non-negotiable. Never implement a fix before verifying the test fails.**
+**This is non-negotiable. Never implement a fix before verifying the test fails. Stop and ask the user if you cannot think of a way to mechanically test the bad behavior.**
 
 ## General Instructions
 
-- in this repository, "qmd" means "quarto markdown", the dialect of markdown we are developing. Although we aim to be largely compatible with Pandoc, it is not necessarily the case that a discrepancy in the behavior is a bug.
+- in this repository, "qmd" means "quarto markdown", the dialect of markdown we are developing. Although we aim to be largely compatible with Pandoc, discrepancies in the behavior might not be bugs.
 - the qmd format only supports the inline syntax for a link [link](./target.html), and not the reference-style syntax [link][1].
 - Always strive for test documents as small as possible. Prefer a large number of small test documents instead of small number of large documents.
 - When fixing bugs, always try to isolate and fix one bug at a time.

crates/quarto-markdown-pandoc/resources/error-corpus/_autogen-table.json

@@ -2,15 +2,15 @@
   {
     "column": 17,
     "row": 0,
-    "state": 1133,
+    "state": 1284,
     "sym": "end",
     "errorInfo": {
       "title": "Unclosed Span",
       "message": "I reached the end of the block before finding a closing ']' for the span or link.",
       "captures": [
         {
           "column": 3,
-          "lrState": 241,
+          "lrState": 301,
           "row": 0,
           "size": 1,
           "sym": "[",
@@ -30,23 +30,23 @@
   {
     "column": 20,
     "row": 0,
-    "state": 2557,
+    "state": 2633,
     "sym": "class_specifier",
     "errorInfo": {
       "title": "Key-value Pair Before Class Specifier in Attribute",
       "message": "This class specifier appears after the key-value pair.",
       "captures": [
         {
           "column": 10,
-          "lrState": 2618,
+          "lrState": 2743,
           "row": 0,
           "size": 3,
           "sym": "key_value_key",
           "label": "key-value-begin"
         },
         {
           "column": 14,
-          "lrState": 2464,
+          "lrState": 2548,
           "row": 0,
           "size": 5,
           "sym": "key_value_value_token1",
@@ -67,15 +67,15 @@
   {
     "column": 18,
     "row": 0,
-    "state": 1926,
+    "state": 2050,
     "sym": "_error",
     "errorInfo": {
       "title": "Mismatched Delimiter in Attribute Specifier",
       "message": "I expected a '}', language specifier, an identifier, a class specifier, or a key-value pair.",
       "captures": [
         {
           "column": 17,
-          "lrState": 1926,
+          "lrState": 2050,
           "row": 0,
           "size": 1,
           "sym": "{",

crates/quarto-markdown-pandoc/src/pandoc/inline.rs

@@ -218,24 +218,28 @@ pub enum CitationMode {
 
 #[derive(Debug, Clone, PartialEq)]
 pub struct Insert {
+    pub attr: Attr,
     pub content: Inlines,
     pub source_info: SourceInfo,
 }
 
 #[derive(Debug, Clone, PartialEq)]
 pub struct Delete {
+    pub attr: Attr,
     pub content: Inlines,
     pub source_info: SourceInfo,
 }
 
 #[derive(Debug, Clone, PartialEq)]
 pub struct Highlight {
+    pub attr: Attr,
     pub content: Inlines,
     pub source_info: SourceInfo,
 }
 
 #[derive(Debug, Clone, PartialEq)]
 pub struct EditComment {
+    pub attr: Attr,
     pub content: Inlines,
     pub source_info: SourceInfo,
 }

crates/quarto-markdown-pandoc/src/pandoc/treesitter.rs

@@ -14,6 +14,9 @@ use crate::pandoc::treesitter_utils::code_fence_content::process_code_fence_cont
 use crate::pandoc::treesitter_utils::code_span::process_code_span;
 use crate::pandoc::treesitter_utils::commonmark_attribute::process_commonmark_attribute;
 use crate::pandoc::treesitter_utils::document::process_document;
+use crate::pandoc::treesitter_utils::editorial_marks::{
+    process_delete, process_editcomment, process_highlight, process_insert,
+};
 use crate::pandoc::treesitter_utils::fenced_code_block::process_fenced_code_block;
 use crate::pandoc::treesitter_utils::fenced_div_block::process_fenced_div_block;
 use crate::pandoc::treesitter_utils::indented_code_block::process_indented_code_block;
@@ -50,8 +53,7 @@ use crate::pandoc::treesitter_utils::uri_autolink::process_uri_autolink;
 use crate::pandoc::ast_context::ASTContext;
 use crate::pandoc::block::{Block, Blocks, BulletList, OrderedList, Paragraph, Plain, RawBlock};
 use crate::pandoc::inline::{
-    Delete, EditComment, Emph, Highlight, Inline, Insert, Note, RawInline, Space, Str, Strikeout,
-    Strong, Subscript, Superscript,
+    Emph, Inline, Note, RawInline, Space, Str, Strikeout, Strong, Subscript, Superscript,
 };
 use crate::pandoc::list::{ListAttributes, ListNumberDelim, ListNumberStyle};
 use crate::pandoc::location::{
@@ -675,38 +677,10 @@ fn native_visitor<T: Write>(
             Strikeout,
             context
         ),
-        "insert" => emphasis_inline!(
-            node,
-            children,
-            "insert_delimiter",
-            native_inline,
-            Insert,
-            context
-        ),
-        "delete" => emphasis_inline!(
-            node,
-            children,
-            "delete_delimiter",
-            native_inline,
-            Delete,
-            context
-        ),
-        "highlight" => emphasis_inline!(
-            node,
-            children,
-            "highlight_delimiter",
-            native_inline,
-            Highlight,
-            context
-        ),
-        "edit_comment" => emphasis_inline!(
-            node,
-            children,
-            "edit_comment_delimiter",
-            native_inline,
-            EditComment,
-            context
-        ),
+        "insert" => process_insert(buf, node, children, context),
+        "delete" => process_delete(buf, node, children, context),
+        "highlight" => process_highlight(buf, node, children, context),
+        "edit_comment" => process_editcomment(buf, node, children, context),
 
         "quoted_span" => process_quoted_span(node, children, native_inline, context),
         "code_span" => process_code_span(buf, node, children, context),

crates/quarto-markdown-pandoc/src/pandoc/treesitter_utils/editorial_marks.rs

@@ -0,0 +1,98 @@
+/*
+ * editorial_marks.rs
+ *
+ * Functions for processing editorial mark nodes in the tree-sitter AST.
+ *
+ * Copyright (c) 2025 Posit, PBC
+ */
+
+use crate::pandoc::ast_context::ASTContext;
+use crate::pandoc::inline::{Delete, EditComment, Highlight, Inline, Inlines, Insert, Space, Str};
+use crate::pandoc::location::{SourceInfo, node_source_info_with_context};
+use once_cell::sync::Lazy;
+use regex::Regex;
+use std::collections::HashMap;
+use std::io::Write;
+
+use super::pandocnativeintermediate::PandocNativeIntermediate;
+use super::text_helpers::apply_smart_quotes;
+
+macro_rules! process_editorial_mark {
+    ($struct_name:ident) => {
+        paste::paste! {
+            pub fn [<process_ $struct_name:lower>]<T: Write>(
+                buf: &mut T,
+                node: &tree_sitter::Node,
+                children: Vec<(String, PandocNativeIntermediate)>,
+                context: &ASTContext,
+            ) -> PandocNativeIntermediate {
+                let whitespace_re: Lazy<Regex> = Lazy::new(|| Regex::new(r"\s+").unwrap());
+                let mut attr = ("".to_string(), vec![], HashMap::new());
+                let mut content: Inlines = vec![];
+
+                for (_node_name, child) in children {
+                    match child {
+                        PandocNativeIntermediate::IntermediateAttr(a) => {
+                            attr = a;
+                        }
+                        PandocNativeIntermediate::IntermediateInline(inline) => {
+                            content.push(inline);
+                        }
+                        PandocNativeIntermediate::IntermediateInlines(mut inlines) => {
+                            content.append(&mut inlines);
+                        }
+                        PandocNativeIntermediate::IntermediateBaseText(text, range) => {
+                            if let Some(_) = whitespace_re.find(&text) {
+                                content.push(Inline::Space(Space {
+                                    source_info: SourceInfo::new(
+                                        if context.filenames.is_empty() {
+                                            None
+                                        } else {
+                                            Some(0)
+                                        },
+                                        range,
+                                    ),
+                                }))
+                            } else {
+                                content.push(Inline::Str(Str {
+                                    text: apply_smart_quotes(text),
+                                    source_info: SourceInfo::new(
+                                        if context.filenames.is_empty() {
+                                            None
+                                        } else {
+                                            Some(0)
+                                        },
+                                        range,
+                                    ),
+                                }))
+                            }
+                        }
+                        PandocNativeIntermediate::IntermediateUnknown(_) => {
+                            // Skip unknown nodes (delimiters, etc.)
+                        }
+                        _ => {
+                            writeln!(
+                                buf,
+                                "Warning: Unexpected node type in {}: {:?}",
+                                stringify!($struct_name:lower),
+                                _node_name
+                            )
+                            .unwrap();
+                        }
+                    }
+                }
+
+                PandocNativeIntermediate::IntermediateInline(Inline::$struct_name($struct_name {
+                    attr,
+                    content,
+                    source_info: node_source_info_with_context(node, context),
+                }))
+            }
+        }
+    };
+}
+
+process_editorial_mark!(Insert);
+process_editorial_mark!(Delete);
+process_editorial_mark!(Highlight);
+process_editorial_mark!(EditComment);

crates/quarto-markdown-pandoc/src/pandoc/treesitter_utils/mod.rs

@@ -13,6 +13,7 @@ pub mod code_fence_content;
 pub mod code_span;
 pub mod commonmark_attribute;
 pub mod document;
+pub mod editorial_marks;
 pub mod fenced_code_block;
 pub mod fenced_div_block;
 pub mod image;