feat(xref): add headings lookup API for cross-spec section links by marcoscaceres · Pull Request #469 · speced/respec-web-services

marcoscaceres · 2026-04-13T02:24:58Z

Extends the xref infrastructure to read and serve section heading data from WebRef's ed/headings/ directory. This enables ReSpec's [[[SPEC#id]]] syntax to display actual heading text instead of just the spec title.

New endpoint

POST /xref/search/headings — looks up section headings by spec shortname and fragment id. Request body: { queries: [{ spec, id }] }. Response: { result: [{ spec, id, title, number, href, level, specTitle }] }.

Capped at 1000 queries per request. Empty spec/id values are rejected with 400.

Changes

scraper.ts: reads ed/headings/*.json during update, writes headings.json. Shortname derived from filename (webref doesn't include it in the JSON).
store.ts: loads headings, indexes by spec+id for O(1) lookup. buildSpecTitleMap iterates the nested {current, snapshot} specmap structure to resolve human-readable spec titles. readJsonOptional only catches ENOENT (not all errors).
headings.post.ts: route handler with input validation (array check, type check, empty-string check, 1000-query cap)
index.ts: registers POST /xref/search/headings with CORS + 1MB body limit
update.ts: also triggers on ed/headings/ changes in webref webhook

Context

Requested by @sidvishnoi in feat(core/inlines): support [[[SPEC#id]]] for cross-spec section links respec#5146 review
Data source: w3c/webref headings data
Companion change needed in ReSpec's src/core/inlines.js to consume this API

Copilot

Pull request overview

Adds a headings-lookup capability to the existing xref service so clients (e.g., ReSpec) can resolve [[[SPEC#id]]] links to section heading metadata sourced from WebRef’s ed/headings/ data.

Changes:

Extend the xref scraper to ingest ed/headings/*.json and emit a consolidated headings.json.
Load headings data into the xref store and expose a getHeading(spec, id) lookup helper.
Add a new POST /xref/headings endpoint and update the webhook trigger to refresh on ed/headings/ changes.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
routes/xref/update.ts	Triggers xref update when WebRef headings data changes.
routes/xref/lib/store.ts	Loads headings.json and adds heading lookup API.
routes/xref/lib/scraper.ts	Reads WebRef headings and writes aggregated headings.json output.
routes/xref/index.ts	Registers the new `/xref/headings` POST route with CORS + JSON body parsing.
routes/xref/headings.post.ts	Implements the headings lookup endpoint response shape.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[[[SPEC#id]]] now displays the actual section heading text (e.g., '§ 3.1.1 Cookie header') instead of just the spec title. Heading text is fetched from POST /xref/headings/ (speced/respec-web-services#469). Also adds alias support: [[[SPEC#id|custom text]]] uses the custom text as link text, skipping the heading lookup. Fallback chain: alias → heading text → spec title → raw ref. Changes: - inlines.js: regex updated for pipe alias, inlineRefMatches stores alias as data-lt - data-cite.js: batch-fetches headings for cross-spec links, formats as '§ NUMBER TITLE' with spec title in a cite element

marcoscaceres · 2026-04-14T10:36:08Z

@copilot can you implement the suggestions I’ve agreed to plus the ones you proposed

Copilot · 2026-04-14T10:41:53Z

@copilot can you implement the suggestions I’ve agreed to plus the ones you proposed

All agreed suggestions implemented in a078fdb:

readdir moved to top-level fs/promises import
readJSON<T> is now generic with a HeadingsJSON interface; all call sites use typed calls
specTitleByShortname changed to Map<string, string> with .get() lookups
Per-item validation added in headings.post.ts (returns 400 if any query item is missing string spec/id fields)

[[[SPEC#id]]] now displays the actual section heading text (e.g., '§ 3.1.1 Cookie header') instead of just the spec title. Heading text is fetched from POST /xref/headings/ (speced/respec-web-services#469). Also adds alias support: [[[SPEC#id|custom text]]] uses the custom text as link text, skipping the heading lookup. Fallback chain: alias → heading text → spec title → raw ref. Changes: - inlines.js: regex updated for pipe alias, inlineRefMatches stores alias as data-lt - data-cite.js: batch-fetches headings for cross-spec links, formats as '§ NUMBER TITLE' with spec title in a cite element

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

sidvishnoi

Some small suggestions, but overall good!

Extends the xref infrastructure to read and serve section heading data from WebRef's ed/headings/ directory. This enables ReSpec's [[[SPEC#id]]] syntax to display actual heading text instead of just the spec title. New endpoint: POST /xref/headings Request: { queries: [{ spec: 'fetch', id: 'cookie-header' }] } Response: { result: [{ spec, id, title, number, href, level, specTitle }] } Changes: - scraper.ts: reads ed/headings/*.json during update, writes headings.json - store.ts: loads headings data, adds getHeading(spec, id) lookup method - headings.post.ts: new route handler - index.ts: registers the /xref/headings POST endpoint - update.ts: also triggers on ed/headings/ changes in webref webhook

- store.ts: index headings by id for O(1) lookup instead of linear scan; precompute specTitleByShortname map; split readJson into required/optional variants so missing xref.json still throws - scraper.ts: fix doc comment on readAllHeadings - headings.post.ts: validate queries is an array before mapping

- Move `readdir` to top-level fs/promises import in scraper.ts - Add generic type `readJSON<T>` with HeadingsJSON interface to reduce `any` - Use typed readJSON<T> calls in getAllData and readAllHeadings - Change specTitleByShortname to Map<string, string> in store.ts - Add per-item validation (spec/id are strings) in headings.post.ts Agent-Logs-Url: https://github.com/speced/respec-web-services/sessions/e14c5758-5d6f-4324-bb98-77839d96f660

Co-authored-by: Sid Vishnoi <8426945+sidvishnoi@users.noreply.github.com>

…andling - Fix buildSpecTitleMap to iterate nested specmap structure correctly (was iterating top-level {current, snapshot} objects instead of entries) - Add 1000-query limit and empty-string validation to headings endpoint - Only catch ENOENT in readJsonOptional (was swallowing all errors) - Remove dead spec?.shortname branch from scraper (webref doesn't include it) - Update JSDoc to reflect actual route path (/xref/search/headings)

Per sidvishnoi's review: index headings by ID during scraping rather than at store load time. Removes the redundant indexHeadings() runtime function.

…ading Per Copilot review: - Trim spec and id before lookup (validation checked trim but passed raw) - Add version-stripped and series-shortname fallback in getHeading so both "cssom-view-1" and "cssom-view" resolve headings correctly - Also resolves specTitle via stripped form as fallback

The specmap type previously described a flat map, but the actual data is nested with current/snapshot groups. This caused TS2352 errors after rebasing onto main's stricter type checking.

- Extract heading resolution fallback into resolveHeadings() method - Remove unnecessary generator (specmapEntries) — iterate directly - Use separate res.status()/res.json() calls instead of chaining

[[[SPEC#id]]] now displays the actual section heading text (e.g., '§ 3.1.1 Cookie header') instead of just the spec title. Heading text is fetched from POST /xref/headings/ (speced/respec-web-services#469). Also adds alias support: [[[SPEC#id|custom text]]] uses the custom text as link text, skipping the heading lookup. Fallback chain: alias → heading text → spec title → raw ref. Changes: - inlines.js: regex updated for pipe alias, inlineRefMatches stores alias as data-lt - data-cite.js: batch-fetches headings for cross-spec links, formats as '§ NUMBER TITLE' with spec title in a cite element

@sidvishnoi

* feat(xref): add headings lookup API for cross-spec section links (#469) * feat(xref): add headings lookup API for cross-spec section links Extends the xref infrastructure to read and serve section heading data from WebRef's ed/headings/ directory. This enables ReSpec's [[[SPEC#id]]] syntax to display actual heading text instead of just the spec title. New endpoint: POST /xref/headings Request: { queries: [{ spec: 'fetch', id: 'cookie-header' }] } Response: { result: [{ spec, id, title, number, href, level, specTitle }] } Changes: - scraper.ts: reads ed/headings/*.json during update, writes headings.json - store.ts: loads headings data, adds getHeading(spec, id) lookup method - headings.post.ts: new route handler - index.ts: registers the /xref/headings POST endpoint - update.ts: also triggers on ed/headings/ changes in webref webhook * fix: address Copilot review feedback on headings API - store.ts: index headings by id for O(1) lookup instead of linear scan; precompute specTitleByShortname map; split readJson into required/optional variants so missing xref.json still throws - scraper.ts: fix doc comment on readAllHeadings - headings.post.ts: validate queries is an array before mapping * refactor(xref): address reviewer feedback on headings API - Move `readdir` to top-level fs/promises import in scraper.ts - Add generic type `readJSON<T>` with HeadingsJSON interface to reduce `any` - Use typed readJSON<T> calls in getAllData and readAllHeadings - Change specTitleByShortname to Map<string, string> in store.ts - Add per-item validation (spec/id are strings) in headings.post.ts Agent-Logs-Url: https://github.com/speced/respec-web-services/sessions/e14c5758-5d6f-4324-bb98-77839d96f660 * Apply suggestion from @sidvishnoi Co-authored-by: Sid Vishnoi <8426945+sidvishnoi@users.noreply.github.com> * Update routes/xref/index.ts Co-authored-by: Sid Vishnoi <8426945+sidvishnoi@users.noreply.github.com> * fix(xref/headings): fix specTitleMap, add query limit, harden error handling - Fix buildSpecTitleMap to iterate nested specmap structure correctly (was iterating top-level {current, snapshot} objects instead of entries) - Add 1000-query limit and empty-string validation to headings endpoint - Only catch ENOENT in readJsonOptional (was swallowing all errors) - Remove dead spec?.shortname branch from scraper (webref doesn't include it) - Update JSDoc to reflect actual route path (/xref/search/headings) * refactor(xref/headings): pre-index headings by ID at scrape time Per sidvishnoi's review: index headings by ID during scraping rather than at store load time. Removes the redundant indexHeadings() runtime function. * fix(xref/headings): trim inputs, add version/series fallback in getHeading Per Copilot review: - Trim spec and id before lookup (validation checked trim but passed raw) - Add version-stripped and series-shortname fallback in getHeading so both "cssom-view-1" and "cssom-view" resolve headings correctly - Also resolves specTitle via stripped form as fallback * fix: correct specmap type to match nested JSON structure The specmap type previously described a flat map, but the actual data is nested with current/snapshot groups. This caused TS2352 errors after rebasing onto main's stricter type checking. * fix: address Sid's review comments on PR #469 - Extract heading resolution fallback into resolveHeadings() method - Remove unnecessary generator (specmapEntries) — iterate directly - Use separate res.status()/res.json() calls instead of chaining --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Sid Vishnoi <8426945+sidvishnoi@users.noreply.github.com> * fix(xref): preserve original case for dfn terms in scraper The scraper was lowercasing all dfn-type terms during indexing, destroying camelCase (e.g., processResponseConsumeBody became processresponseconsumebody). Webref has the correct casing. Changes: - Remove dfn lowercasing from normalizeTerm() in scraper - Add byTermLower index in Store for case-insensitive lookups - Use case-insensitive fallback in filterByTerm() when direct lookup misses (preserves fast path for exact-case IDL terms) - Update tests to reflect case-insensitive search behavior * refactor: export buildTermLowerIndex, deduplicate test helper * fix(utils/sh): reject with Error instances, not plain objects The refactored sh.ts was rejecting with plain objects instead of Error instances, breaking instanceof Error narrowing in callers. Restores the original pattern: Object.assign(err, {...}) for spawn errors and new Error(...) for non-zero exit codes. Also removes unrelated files accidentally included in this branch. * fix(xref): scope case-insensitive fallback to non-IDL queries WebIDL identifiers are case-sensitive, so "baseLine" must not match "Baseline" when types include an IDL type. The fallback now only fires when the query has no IDL types. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(xref): include canonical term in case-insensitive fallback results When the case-insensitive fallback fires in filterByTerm, each returned entry is now tagged with the canonical term it was indexed under. This lets the xref UI build correct cite syntax (e.g., [=baseline=] and {{Baseline}}) instead of using the user's potentially miscased input (e.g., [=baseLine=]). Changes: - Add optional `term` field to DataEntry - Clone entries in filterByTerm fallback path with canonical term - Update xref UI to use entry.term for citation building - Add test verifying term presence on fallback hits --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Sid Vishnoi <8426945+sidvishnoi@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

marcoscaceres mentioned this pull request Apr 13, 2026

feat(core/inlines): support [[[SPEC#id]]] for cross-spec section links speced/respec#5146

Merged

2 tasks

marcoscaceres requested a review from Copilot April 13, 2026 02:44

Copilot started reviewing on behalf of marcoscaceres April 13, 2026 02:45 View session

Copilot AI reviewed Apr 13, 2026

View reviewed changes

Comment thread routes/xref/lib/store.ts

Comment thread routes/xref/lib/store.ts Outdated

Comment thread routes/xref/lib/store.ts Outdated

Comment thread routes/xref/lib/scraper.ts Outdated

Comment thread routes/xref/headings.post.ts

sidvishnoi reviewed Apr 14, 2026

View reviewed changes

Comment thread routes/xref/lib/scraper.ts

Comment thread routes/xref/lib/scraper.ts Outdated

Comment thread routes/xref/lib/scraper.ts Outdated

Comment thread routes/xref/lib/store.ts Outdated

Copilot started work on behalf of marcoscaceres April 14, 2026 10:36 View session

Copilot finished work on behalf of marcoscaceres April 14, 2026 10:42

sidvishnoi reviewed Apr 15, 2026

View reviewed changes

Comment thread routes/xref/lib/store.ts Outdated

Comment thread routes/xref/lib/store.ts Outdated

Comment thread routes/xref/index.ts Outdated

Comment thread routes/xref/headings.post.ts

marcoscaceres requested a review from Copilot April 24, 2026 01:06

Copilot started reviewing on behalf of marcoscaceres April 24, 2026 01:06 View session

Copilot AI reviewed Apr 24, 2026

View reviewed changes

Comment thread routes/xref/lib/store.ts

Comment thread routes/xref/headings.post.ts Outdated

marcoscaceres requested a review from sidvishnoi April 24, 2026 05:30

sidvishnoi reviewed Apr 24, 2026

View reviewed changes

Comment thread routes/xref/lib/scraper.ts

sidvishnoi reviewed Apr 24, 2026

View reviewed changes

Comment thread routes/xref/lib/store.ts Outdated

sidvishnoi reviewed Apr 24, 2026

View reviewed changes

Comment thread routes/xref/lib/store.ts Outdated

sidvishnoi reviewed Apr 24, 2026

View reviewed changes

Comment thread routes/xref/headings.post.ts Outdated

sidvishnoi approved these changes Apr 24, 2026

View reviewed changes

marcoscaceres and others added 9 commits April 25, 2026 17:22

Apply suggestion from @sidvishnoi

abb38b3

Co-authored-by: Sid Vishnoi <8426945+sidvishnoi@users.noreply.github.com>

Update routes/xref/index.ts

0881359

Co-authored-by: Sid Vishnoi <8426945+sidvishnoi@users.noreply.github.com>

refactor(xref/headings): pre-index headings by ID at scrape time

db0b1a7

Per sidvishnoi's review: index headings by ID during scraping rather than at store load time. Removes the redundant indexHeadings() runtime function.

fix: correct specmap type to match nested JSON structure

a722cce

The specmap type previously described a flat map, but the actual data is nested with current/snapshot groups. This caused TS2352 errors after rebasing onto main's stricter type checking.

fix: address Sid's review comments on PR #469

a6a9f23

- Extract heading resolution fallback into resolveHeadings() method - Remove unnecessary generator (specmapEntries) — iterate directly - Use separate res.status()/res.json() calls instead of chaining

marcoscaceres force-pushed the feat/headings-api branch from 2a6772c to a6a9f23 Compare April 25, 2026 10:08

marcoscaceres merged commit 39ce491 into main Apr 25, 2026
3 checks passed

marcoscaceres deleted the feat/headings-api branch April 25, 2026 10:24

Uh oh!

Conversation

marcoscaceres commented Apr 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

New endpoint

Changes

Context

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

marcoscaceres commented Apr 14, 2026

Uh oh!

Copilot AI commented Apr 14, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sidvishnoi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

marcoscaceres commented Apr 13, 2026 •

edited

Loading