Skip to content

fix: keep a hard break before an attention run#78

Open
spokodev wants to merge 2 commits into
syntax-tree:mainfrom
spokodev:fix/break-before-attention
Open

fix: keep a hard break before an attention run#78
spokodev wants to merge 2 commits into
syntax-tree:mainfrom
spokodev:fix/break-before-attention

Conversation

@spokodev

Copy link
Copy Markdown

Problem

When a hard break is immediately followed by an attention run (emphasis/strong) whose inner side is whitespace, the break is destroyed on a round-trip:

toMarkdown({
  type: 'paragraph',
  children: [
    {type: 'break'},
    {type: 'strong', children: [{type: 'text', value: ' x'}]}
  ]
})
// '\\
** x**\n'

The break’s line ending is character-referenced to 
, so fromMarkdown re-parses \
 as an escaped backslash followed by literal text — the break node becomes a text node:

before: paragraph > [ break, strong ]
after:  paragraph > [ text "
", strong ]

Cause

An attention run sets attentionEncodeSurroundingInfo.before to encode the character on its outer side when that character is whitespace (so the run keeps forming, e.g. x* *z). In containerPhrasing, that character is the last character of the previous sibling’s output. When the previous sibling is a hard break, its last character is the \n, which gets character-referenced — splitting the break.

Fix

A line ending can never fuse with the following run, because the run starts on a new line. So skip the surrounding encoding when the preceding character is a line ending (\n / \r). The inner-side encoding (the leading space →  ) is unaffected, so the run still forms:

'\\\n** x**\n'   // break preserved, round-trips to [ break, strong ]

Tests

Added a case under break asserting the serialized output keeps the break (test/index.js) — red before, green after. Full suite green (479/479); prettier, xo, and tsc --build all clean.

When a hard break was immediately followed by an attention run (emphasis or
strong) whose inner side is whitespace, the break was turned into text on a
round-trip:

    {type: 'paragraph', children: [
      {type: 'break'},
      {type: 'strong', children: [{type: 'text', value: ' x'}]}
    ]}

serialized to `\
** x**`, where the break’s line ending was
character-referenced. The attention run asks to encode the character on its
outer side (here whitespace, to keep the run forming), and that character
happened to be the `\n` of the break, so the break was destroyed.

A line ending can never fuse with the following run — the run starts on a
new line — so skip the surrounding encoding when the preceding character is
a line ending.
@github-actions github-actions Bot added the 👋 phase/new Post is being triaged automatically label Jun 18, 2026
@github-actions

Copy link
Copy Markdown

Hi! It seems you removed the template which we require. Here are our templates (pick the one you want to use and click *raw* to see its source):

I won’t send you any further notifications about this, but I’ll keep on updating this comment, and hide it when done!

Thanks,
— bb

@ChristianMurphy ChristianMurphy left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @spokodev and AI assistant 👋
What particular use case in your app or project did you run into where you want/need this?
Have you read #12, the attention run edge cases there, apply here too have you tested or considered trade offs for this?

Comment thread lib/util/container-phrasing.js Outdated
Comment on lines +108 to +109
// A line ending (such as a hard break) is structural: encoding it would
// break the break, and it can’t fuse with the following attention run.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

these comment's aren't needed

@wooorm

wooorm commented Jun 18, 2026

Copy link
Copy Markdown
Member

A line ending can never fuse with the following run, because the run starts on a new line.

Are you sure?

@spokodev

Copy link
Copy Markdown
Author

@wooorm fair to call that out, the wording in the description was loose. The real reason isn't "it starts on a new line" by itself. The surrounding whitespace that needs guarding is on the run's inner side and is already character-referenced, so the line after the break never begins with a literal * / # / *** that could reparse as a block. The outer character being a line ending doesn't need encoding.

Rather than just assert that, I checked it with a differential fuzz over 60k random paragraphs mixing breaks, emphasis/strong (both * and _), nesting, and inner whitespace/markers, comparing round-trips before and after the change:

  • 0 regressions (nothing that round-tripped before stops)
  • 5280 cases fixed

For example, [break, emphasis(' x')] serialises to a backslash, a line feed, then * x*, and parses back to [break, emphasis]. Without the change the line feed becomes 
, which from-markdown reads as an escaped backslash plus text, so the break is lost.

@ChristianMurphy on the use case: this turned up round-tripping rehype/remark content where a hard break sits right before emphasis/strong whose inner edge is whitespace, and toMarkdown then fromMarkdown quietly turned the break into text. On #12: that's the nested-emphasis delimiter-run path, which this doesn't touch. The change only narrows the existing surrounding-whitespace encoding to skip line endings, and its tests are untouched. The other round-trip gaps the fuzz still shows (adjacent same-type runs, trailing breaks) are pre-existing and unrelated.

@spokodev

Copy link
Copy Markdown
Author

Honest answer: no specific app. I hit this while round-trip testing the serializer. It's real data loss though (the break turns into a text node), so it seemed worth fixing.

#12 is a different case (emphasis in emphasis). My change only guards line endings, so it doesn't touch that path and the tests stay green. I didn't try to fix #12.

Removed the comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

👋 phase/new Post is being triaged automatically

Development

Successfully merging this pull request may close these issues.

3 participants