Speed up markdown processing#1903
Draft
hadley wants to merge 4 commits into
Draft
Conversation
tokenizeMd() built a std::set of the 49 verbatim tag names on every call (~13us), dominating its runtime on typical tag-sized inputs. A linear scan over the R vector is cheaper because each text contains only a handful of Rd tags. ~2.5x faster per call.
The old gate parsed the text with commonmark + xml2 whenever it contained any backtick or tilde, which is nearly every doc. The new gate looks for the shapes that can actually trigger evaluation (inline `r ...`, fenced blocks with braced info, code blocks whose first line starts with "r "), cutting ~90% of the wasted parses.
The R tree walk over xml2 nodes was the largest remaining cost of markdown processing (~24% of parse_package() on testthat). The walk now happens in C++ (src/mdxmlToRd.cpp), directly over the XML string that commonmark::markdown_xml() returns, so the xml2 parse and the per-node R dispatch disappear entirely. Token restoration happens inline during the walk, with the same final text-mode pass over the assembled Rd. Everything that needs package state stays in R and is supplied to the walk as a callback: link resolution (parse_link(), which now receives the link pieces precomputed instead of xml2 nodes), R code detection for \code vs \verb, and warnings for unsupported constructs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1901; merge that first and this retargets to main.
Profiling
parse_package()on testthat (after #1901 + #1902) showed markdown processing at ~55% of parse time, dominated by three things, fixed in one commit each:tokenizeMd()rebuilt astd::setof the 49 verbatim tag names on every call (~13µs of its ~17µs per call). A linear scan over the R vector is cheaper because each text contains only a handful of Rd tags.markdown_evaluate()did a full commonmark + xml2 parse whenever the text contained any backtick or tilde (~16% of parse time), almost always finding no R code. The gate now looks for the shapes that can actually trigger evaluation (inline`r `, fences with braced info strings, code blocks whose first line starts with"r "). On the testthat corpus this cuts the parses from 199/530 texts to exactly the 23 that contain R code, with byte-identical results on all 530.The markdown → Rd tree walk moves from R to C++ (~24% of parse time).
src/mdxmlToRd.cppparses the XML string thatcommonmark::markdown_xml()returns directly (cmark emits a small, regular XML subset), so the xml2 parse and per-node R dispatch disappear. Token restoration happens inline during the walk, with the same final text-mode pass over the assembled Rd as before. Everything needing package state stays in R as callbacks:parse_link()(now receiving precomputed link pieces instead of xml2 nodes),\codevs\verbdetection, and warnings.restore_tokens()moves to C++ with the walk; its unit tests became end-to-endmarkdown()tests.Results
parse_package()on testthat: 324ms → 227ms (1.43x faster), measured as the median of 8 runs in identical fresh subprocesses.Verification
R CMD checkclean.roxygenise()output on copies of testthat, pkgdown, and usethis is byte-identical between this branch andmd-grammar(man/ compared withdiff -r, plus NAMESPACE).🤖 Generated with Claude Code