feat(xref): add headings lookup API for cross-spec section links#469
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a headings-lookup capability to the existing xref service so clients (e.g., ReSpec) can resolve [[[SPEC#id]]] links to section heading metadata sourced from WebRef’s ed/headings/ data.
Changes:
- Extend the xref scraper to ingest
ed/headings/*.jsonand emit a consolidatedheadings.json. - Load headings data into the xref store and expose a
getHeading(spec, id)lookup helper. - Add a new
POST /xref/headingsendpoint and update the webhook trigger to refresh oned/headings/changes.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| routes/xref/update.ts | Triggers xref update when WebRef headings data changes. |
| routes/xref/lib/store.ts | Loads headings.json and adds heading lookup API. |
| routes/xref/lib/scraper.ts | Reads WebRef headings and writes aggregated headings.json output. |
| routes/xref/index.ts | Registers the new /xref/headings POST route with CORS + JSON body parsing. |
| routes/xref/headings.post.ts | Implements the headings lookup endpoint response shape. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
[[[SPEC#id]]] now displays the actual section heading text (e.g., '§ 3.1.1 Cookie header') instead of just the spec title. Heading text is fetched from POST /xref/headings/ (speced/respec-web-services#469). Also adds alias support: [[[SPEC#id|custom text]]] uses the custom text as link text, skipping the heading lookup. Fallback chain: alias → heading text → spec title → raw ref. Changes: - inlines.js: regex updated for pipe alias, inlineRefMatches stores alias as data-lt - data-cite.js: batch-fetches headings for cross-spec links, formats as '§ NUMBER TITLE' with spec title in a cite element
|
@copilot can you implement the suggestions I’ve agreed to plus the ones you proposed |
All agreed suggestions implemented in a078fdb:
|
[[[SPEC#id]]] now displays the actual section heading text (e.g., '§ 3.1.1 Cookie header') instead of just the spec title. Heading text is fetched from POST /xref/headings/ (speced/respec-web-services#469). Also adds alias support: [[[SPEC#id|custom text]]] uses the custom text as link text, skipping the heading lookup. Fallback chain: alias → heading text → spec title → raw ref. Changes: - inlines.js: regex updated for pipe alias, inlineRefMatches stores alias as data-lt - data-cite.js: batch-fetches headings for cross-spec links, formats as '§ NUMBER TITLE' with spec title in a cite element
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
sidvishnoi
left a comment
There was a problem hiding this comment.
Some small suggestions, but overall good!
Extends the xref infrastructure to read and serve section heading data
from WebRef's ed/headings/ directory. This enables ReSpec's [[[SPEC#id]]]
syntax to display actual heading text instead of just the spec title.
New endpoint: POST /xref/headings
Request: { queries: [{ spec: 'fetch', id: 'cookie-header' }] }
Response: { result: [{ spec, id, title, number, href, level, specTitle }] }
Changes:
- scraper.ts: reads ed/headings/*.json during update, writes headings.json
- store.ts: loads headings data, adds getHeading(spec, id) lookup method
- headings.post.ts: new route handler
- index.ts: registers the /xref/headings POST endpoint
- update.ts: also triggers on ed/headings/ changes in webref webhook
- store.ts: index headings by id for O(1) lookup instead of linear scan; precompute specTitleByShortname map; split readJson into required/optional variants so missing xref.json still throws - scraper.ts: fix doc comment on readAllHeadings - headings.post.ts: validate queries is an array before mapping
- Move `readdir` to top-level fs/promises import in scraper.ts - Add generic type `readJSON<T>` with HeadingsJSON interface to reduce `any` - Use typed readJSON<T> calls in getAllData and readAllHeadings - Change specTitleByShortname to Map<string, string> in store.ts - Add per-item validation (spec/id are strings) in headings.post.ts Agent-Logs-Url: https://github.com/speced/respec-web-services/sessions/e14c5758-5d6f-4324-bb98-77839d96f660
Co-authored-by: Sid Vishnoi <8426945+sidvishnoi@users.noreply.github.com>
Co-authored-by: Sid Vishnoi <8426945+sidvishnoi@users.noreply.github.com>
…andling
- Fix buildSpecTitleMap to iterate nested specmap structure correctly
(was iterating top-level {current, snapshot} objects instead of entries)
- Add 1000-query limit and empty-string validation to headings endpoint
- Only catch ENOENT in readJsonOptional (was swallowing all errors)
- Remove dead spec?.shortname branch from scraper (webref doesn't include it)
- Update JSDoc to reflect actual route path (/xref/search/headings)
Per sidvishnoi's review: index headings by ID during scraping rather than at store load time. Removes the redundant indexHeadings() runtime function.
…ading Per Copilot review: - Trim spec and id before lookup (validation checked trim but passed raw) - Add version-stripped and series-shortname fallback in getHeading so both "cssom-view-1" and "cssom-view" resolve headings correctly - Also resolves specTitle via stripped form as fallback
The specmap type previously described a flat map, but the actual data is nested with current/snapshot groups. This caused TS2352 errors after rebasing onto main's stricter type checking.
- Extract heading resolution fallback into resolveHeadings() method - Remove unnecessary generator (specmapEntries) — iterate directly - Use separate res.status()/res.json() calls instead of chaining
2a6772c to
a6a9f23
Compare
[[[SPEC#id]]] now displays the actual section heading text (e.g., '§ 3.1.1 Cookie header') instead of just the spec title. Heading text is fetched from POST /xref/headings/ (speced/respec-web-services#469). Also adds alias support: [[[SPEC#id|custom text]]] uses the custom text as link text, skipping the heading lookup. Fallback chain: alias → heading text → spec title → raw ref. Changes: - inlines.js: regex updated for pipe alias, inlineRefMatches stores alias as data-lt - data-cite.js: batch-fetches headings for cross-spec links, formats as '§ NUMBER TITLE' with spec title in a cite element
* feat(xref): add headings lookup API for cross-spec section links (#469) * feat(xref): add headings lookup API for cross-spec section links Extends the xref infrastructure to read and serve section heading data from WebRef's ed/headings/ directory. This enables ReSpec's [[[SPEC#id]]] syntax to display actual heading text instead of just the spec title. New endpoint: POST /xref/headings Request: { queries: [{ spec: 'fetch', id: 'cookie-header' }] } Response: { result: [{ spec, id, title, number, href, level, specTitle }] } Changes: - scraper.ts: reads ed/headings/*.json during update, writes headings.json - store.ts: loads headings data, adds getHeading(spec, id) lookup method - headings.post.ts: new route handler - index.ts: registers the /xref/headings POST endpoint - update.ts: also triggers on ed/headings/ changes in webref webhook * fix: address Copilot review feedback on headings API - store.ts: index headings by id for O(1) lookup instead of linear scan; precompute specTitleByShortname map; split readJson into required/optional variants so missing xref.json still throws - scraper.ts: fix doc comment on readAllHeadings - headings.post.ts: validate queries is an array before mapping * refactor(xref): address reviewer feedback on headings API - Move `readdir` to top-level fs/promises import in scraper.ts - Add generic type `readJSON<T>` with HeadingsJSON interface to reduce `any` - Use typed readJSON<T> calls in getAllData and readAllHeadings - Change specTitleByShortname to Map<string, string> in store.ts - Add per-item validation (spec/id are strings) in headings.post.ts Agent-Logs-Url: https://github.com/speced/respec-web-services/sessions/e14c5758-5d6f-4324-bb98-77839d96f660 * Apply suggestion from @sidvishnoi Co-authored-by: Sid Vishnoi <8426945+sidvishnoi@users.noreply.github.com> * Update routes/xref/index.ts Co-authored-by: Sid Vishnoi <8426945+sidvishnoi@users.noreply.github.com> * fix(xref/headings): fix specTitleMap, add query limit, harden error handling - Fix buildSpecTitleMap to iterate nested specmap structure correctly (was iterating top-level {current, snapshot} objects instead of entries) - Add 1000-query limit and empty-string validation to headings endpoint - Only catch ENOENT in readJsonOptional (was swallowing all errors) - Remove dead spec?.shortname branch from scraper (webref doesn't include it) - Update JSDoc to reflect actual route path (/xref/search/headings) * refactor(xref/headings): pre-index headings by ID at scrape time Per sidvishnoi's review: index headings by ID during scraping rather than at store load time. Removes the redundant indexHeadings() runtime function. * fix(xref/headings): trim inputs, add version/series fallback in getHeading Per Copilot review: - Trim spec and id before lookup (validation checked trim but passed raw) - Add version-stripped and series-shortname fallback in getHeading so both "cssom-view-1" and "cssom-view" resolve headings correctly - Also resolves specTitle via stripped form as fallback * fix: correct specmap type to match nested JSON structure The specmap type previously described a flat map, but the actual data is nested with current/snapshot groups. This caused TS2352 errors after rebasing onto main's stricter type checking. * fix: address Sid's review comments on PR #469 - Extract heading resolution fallback into resolveHeadings() method - Remove unnecessary generator (specmapEntries) — iterate directly - Use separate res.status()/res.json() calls instead of chaining --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Sid Vishnoi <8426945+sidvishnoi@users.noreply.github.com> * fix(xref): preserve original case for dfn terms in scraper The scraper was lowercasing all dfn-type terms during indexing, destroying camelCase (e.g., processResponseConsumeBody became processresponseconsumebody). Webref has the correct casing. Changes: - Remove dfn lowercasing from normalizeTerm() in scraper - Add byTermLower index in Store for case-insensitive lookups - Use case-insensitive fallback in filterByTerm() when direct lookup misses (preserves fast path for exact-case IDL terms) - Update tests to reflect case-insensitive search behavior * refactor: export buildTermLowerIndex, deduplicate test helper * fix(utils/sh): reject with Error instances, not plain objects The refactored sh.ts was rejecting with plain objects instead of Error instances, breaking instanceof Error narrowing in callers. Restores the original pattern: Object.assign(err, {...}) for spawn errors and new Error(...) for non-zero exit codes. Also removes unrelated files accidentally included in this branch. * fix(xref): scope case-insensitive fallback to non-IDL queries WebIDL identifiers are case-sensitive, so "baseLine" must not match "Baseline" when types include an IDL type. The fallback now only fires when the query has no IDL types. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix(xref): include canonical term in case-insensitive fallback results When the case-insensitive fallback fires in filterByTerm, each returned entry is now tagged with the canonical term it was indexed under. This lets the xref UI build correct cite syntax (e.g., [=baseline=] and {{Baseline}}) instead of using the user's potentially miscased input (e.g., [=baseLine=]). Changes: - Add optional `term` field to DataEntry - Clone entries in filterByTerm fallback path with canonical term - Update xref UI to use entry.term for citation building - Add test verifying term presence on fallback hits --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Sid Vishnoi <8426945+sidvishnoi@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Extends the xref infrastructure to read and serve section heading data from WebRef's
ed/headings/directory. This enables ReSpec's[[[SPEC#id]]]syntax to display actual heading text instead of just the spec title.New endpoint
POST /xref/search/headings— looks up section headings by spec shortname and fragment id. Request body:{ queries: [{ spec, id }] }. Response:{ result: [{ spec, id, title, number, href, level, specTitle }] }.Capped at 1000 queries per request. Empty spec/id values are rejected with 400.
Changes
ed/headings/*.jsonduring update, writesheadings.json. Shortname derived from filename (webref doesn't include it in the JSON).buildSpecTitleMapiterates the nested{current, snapshot}specmap structure to resolve human-readable spec titles.readJsonOptionalonly catches ENOENT (not all errors).POST /xref/search/headingswith CORS + 1MB body limited/headings/changes in webref webhookContext
src/core/inlines.jsto consume this API