Skip to content

Record two tokenizer defects handed over from the marko backlog - #233

Open
DylanPiercey wants to merge 1 commit into
mainfrom
dpiercey-ws-agent-cleanup-dp8z5v
Open

Record two tokenizer defects handed over from the marko backlog#233
DylanPiercey wants to merge 1 commit into
mainfrom
dpiercey-ws-agent-cleanup-dp8z5v

Conversation

@DylanPiercey

Copy link
Copy Markdown
Contributor

Both were found while working in marko-js/marko and recorded there; triage put the fix in this repo, so they move here rewritten against this repo's source and with checks that run here.

shouldTerminateHtmlAttrValue has a look-behind for => but not >=, so <const/positive=input.delta >= 0/> silently truncates with no onError. Separately, a backslash-escaped quote in a parsed-text body opens a string, so \" in a <style> swallows the rest of the file.

Removed from marko in marko-js/marko#3714. The ambiguous space-separated > form stays there — the tokenizer can't distinguish it from a legitimate tag close.

@changeset-bot

changeset-bot Bot commented Jul 31, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: f8d17c6

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@codecov

codecov Bot commented Jul 31, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.95%. Comparing base (3819faa) to head (f8d17c6).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #233   +/-   ##
=======================================
  Coverage   99.95%   99.95%           
=======================================
  Files          34       34           
  Lines        4270     4270           
  Branches      793      793           
=======================================
  Hits         4268     4268           
  Misses          2        2           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@coderabbitai

coderabbitai Bot commented Jul 31, 2026

Copy link
Copy Markdown

Review Change Stack

Walkthrough

Added documentation for two suspected parser bugs. One report covers silent truncation of unquoted attribute expressions containing >=. The other covers incorrect handling of backslash-escaped quotes in parsed-text bodies and parsed strings. Each report includes reproduction details and regression testing requirements.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly describes the two tokenizer defects recorded by the pull request.
Description check ✅ Passed The description explains both tokenizer defects, their origin, and the repository-specific scope.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch dpiercey-ws-agent-cleanup-dp8z5v

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@agent-feedback/bugs.md`:
- Line 15: Update PARSED_TEXT_CONTENT.parse so a declined CODE.BACK_SLASH
followed by either quote is consumed as text, without relying on quoteCharCode;
update PARSED_STRING.parse to consume a backslash plus the active quoteCharCode
before quote-closing logic. Add fixtures covering escaped quotes in both
text/style content and parsed strings, and assert INVALID_TEMPLATE_STRING is not
emitted.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: dc479495-b464-4787-a20e-ee87c43ad301

📥 Commits

Reviewing files that changed from the base of the PR and between 3819faa and f8d17c6.

📒 Files selected for processing (1)
  • agent-feedback/bugs.md

Comment thread agent-feedback/bugs.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant