Skip to content

Resolve link topics against a per-run alias index#1902

Merged
hadley merged 5 commits into
mainfrom
fast-topic-lookup
Jul 5, 2026
Merged

Resolve link topics against a per-run alias index#1902
hadley merged 5 commits into
mainfrom
fast-topic-lookup

Conversation

@hadley

@hadley hadley commented Jul 5, 2026

Copy link
Copy Markdown
Member

Profiling roxygenise() on a link-heavy package (testthat) showed ~50% of the roxygen work inside link/topic resolution: has_topic() answers every query with a utils::help() call (~1.5ms), and find_package_lookup() maps it over all dependencies plus base packages, so each unique linked topic costs ~50 help-index searches. The per-topic cache added in #1724 only deduplicates repeated topics — every miss still pays the full scan.

This PR replaces the help() calls with a cached topic index per package:

  • For installed packages, read the help/aliases.rds index that ships with the package (once per package per session — installed packages don't change between runs).
  • For packages loaded from source — the package being documented, or a dependency loaded with pkgload::load_all() — reuse pkgload::dev_topic_index(), the alias index that dev_help() already maintains. (pkgload cached this before too, but the old path reached it through help() → shim → dev_topic_parse()/normalizePath() on every call, and the cache never applied to the ~50 installed dependencies, which were the bigger cost.)
  • Topics are stored as a hashed environment so each has_topic() is a single env_has(); %in% on the alias vector rebuilt a hash table per membership test, which the R6 and Rd-inheritance paths (no per-topic cache in front of them) paid heavily.
  • pkg_deps() is also cached per run — it re-read and re-parsed DESCRIPTION for every unresolved topic.
  • At the start of each run, find_package_cache_reset() evicts only the documented package's entry (from our cache and pkgload's) — its topics are the only ones that change between runs. Failed lookups are never cached, so installing a missing dependency mid-session is picked up immediately.

check_topic() (validation of explicitly-qualified links) and the Rd-inheritance and R6 code paths that use has_topic() get the same speedup for free. Re-export attribution (find_source()), the ambiguous-topic warning, and all warning messages are unchanged.

Benchmark on a quiet machine (medians of 4 adjacent steady-state runs, spreads ±0.04s): roxygenise() on testthat goes from 3.00s to 2.15s (1.4x), and to 2.05s (1.5x) combined with the markdown tokenizer rewrite in #1901. Note ~0.9s of every run is fixed load_all() + Rd-writing overhead, so the roxygen-work portion roughly halves. Link resolution falls from ~50% of the profile to under 8%.

🤖 Generated with Claude Code

hadley added 4 commits July 5, 2026 17:19
has_topic() previously called help() for every (topic, package) pair,
and find_package() maps it over every dependency plus the base
packages, so each unique linked topic cost ~50 help() searches
(~1.5ms each). Instead, read each package's help/aliases.rds once per
roxygenize() run (or scan man/ for packages loaded from source, like
the package being documented) and answer lookups from that index.

This also speeds up check_topic() and the Rd inheritance and R6 code
paths that use has_topic(). Documenting testthat drops from 13.2s to
8.7s.
pkgload already maintains a cached alias index for packages loaded
from source, built with tools::parse_Rd() so it handles Rd macros and
multi-line constructs properly. Use it instead of a hand-rolled man/
scan, and restore the dev_topic_index_reset() call so the index is
rebuilt each run.
…ckage

Installed packages don't change between runs, so there's no need to
re-read their alias indexes every run. The only package whose topics
change from run to run is the one being documented, so
find_package_cache_reset() evicts just that entry (from our cache and
pkgload's). Failed lookups are never cached, so installing a missing
dependency mid-session is picked up right away.
Two hotspots visible in profiles after the index change:

* pkg_deps() re-read and re-parsed DESCRIPTION for every unresolved
  topic (~5% of a roxygenise() run); cache it per package dir, reset
  each run.

* has_topic() used %in% on the alias vector, which rebuilds a hash
  table on every call (~8% of a run, mostly via the R6 and Rd
  inheritance paths, which have no per-topic cache in front of them).
  pkg_topics() now caches a hashed environment so each lookup is a
  single env_has().
@hadley

hadley commented Jul 5, 2026

Copy link
Copy Markdown
Member Author

Added the last two low-hanging items the combined-branch profile surfaced (8.2% + 5.0% of samples):

  • pkg_deps() re-read and re-parsed DESCRIPTION for every unresolved topic — now cached per package dir, reset each run.
  • has_topic() used %in% on the alias vector, rebuilding a hash table per membership test — pkg_topics() now caches a hashed environment, so each lookup is a single env_has(). This mostly benefits the R6 and Rd-inheritance paths, which call has_topic() directly without the link path's per-topic cache.

With these, link/topic resolution is essentially free; the remaining profile is the markdown XML walk (~23%, a structural project) and diffuse Rd assembly.

@hadley hadley merged commit 3dd116b into main Jul 5, 2026
13 checks passed
@hadley hadley deleted the fast-topic-lookup branch July 5, 2026 23:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant