feat(api/baseline): add web-features/baseline data endpoint by marcoscaceres · Pull Request #498 · speced/respec-web-services

marcoscaceres · 2026-04-25T10:36:03Z

Summary

GET /api/baseline — full web-features dataset
GET /api/baseline/:feature — single feature lookup (follows moved/split redirects)
POST /api/baseline/search — spec-URL-based feature matching
In-memory store indexed by feature ID and spec URL
Scraper fetches from web-platform-dx/web-features GitHub releases

Closes #490

Copilot

Pull request overview

Adds a new /api/baseline API surface that serves the web-features “Baseline” dataset from a local on-disk cache, with endpoints for full dataset retrieval, single-feature lookup (including moved/split handling), spec-URL search, and a webhook-triggered background refresh flow.

Changes:

Introduces GET /api/baseline and GET /api/baseline/:feature routes backed by an in-memory store loaded from DATA_DIR/baseline/baseline.json.
Adds POST /api/baseline/search for spec-URL-based feature matching using a spec→feature index.
Implements a GitHub-releases-based scraper plus /api/baseline/update background worker integration, and wires the router + update script + env secret.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
scripts/update-data-sources.ts	Runs the baseline scraper as part of the existing data-source refresh script.
routes/api/baseline/update.worker.ts	Worker entrypoint that invokes the baseline scraper in a background thread.
routes/api/baseline/update.ts	Webhook route that enqueues the background update job and reloads the store.
routes/api/baseline/search.post.ts	Implements POST search by spec URLs against the in-memory spec index.
routes/api/baseline/lib/store.ts	Defines dataset types, URL normalization, and builds in-memory indexes from `baseline.json`.
routes/api/baseline/lib/store-init.ts	Initializes a singleton `BaselineStore` used by the handlers.
routes/api/baseline/lib/scraper.ts	Downloads `data.json` from the latest web-features GitHub release into `DATA_DIR`.
routes/api/baseline/index.ts	Express router wiring (CORS, JSON body parsing, webhook auth).
routes/api/baseline/feature.ts	Implements single-feature lookup with moved/split resolution behavior.
routes/api/baseline/all.ts	Implements full dataset retrieval.
app.ts	Mounts the new router at `/api/baseline`.
.env.sample	Adds `WEB_FEATURES_SECRET` for webhook authentication configuration.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Adds endpoints for serving Baseline browser availability data: - GET /api/baseline — full web-features dataset - GET /api/baseline/:feature — single feature lookup - POST /api/baseline/search — spec-URL-based feature search Data sourced from web-platform-dx/web-features GitHub releases. Enables ReSpec's implementationStatus feature (speced/respec#5174) without requiring unpkg.com dependency. Closes #490

Client expects {result: WebFeatureEntry[]}, server was returning Record<string, {id, feature}[]>. Flatten features, deduplicate by ID, and wrap in {result: [...]}.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 5 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

marcoscaceres · 2026-04-27T08:33:11Z

@copilot can you fix any unaddressed comments

- search.post.ts: return 503 when store.data is null (service unavailable) - search.post.ts: reject empty/whitespace-only strings in specs array - store.ts: normalizeUrl() now also clears search params (query string) - scraper.ts: add ETag/If-None-Match support to skip unnecessary re-downloads - scraper.ts: use atomic write (temp file + rename) to prevent corrupt data on interrupted writes - Add tests: normalizeUrl, feature route (moved/split/404), search.post validation Agent-Logs-Url: https://github.com/speced/respec-web-services/sessions/41823caf-5a5e-40fc-8bf7-fca753fb37ab

…dd return type doc Agent-Logs-Url: https://github.com/speced/respec-web-services/sessions/41823caf-5a5e-40fc-8bf7-fca753fb37ab

Copilot · 2026-04-27T08:45:43Z

@copilot can you fix any unaddressed comments

Fixed all unresolved comments in commits 9e87c3a and 1e3cafc:

search.post.ts: returns 503 when store.data is null (instead of 200 with empty result), and now rejects empty/whitespace-only strings in the specs array
store.ts: normalizeUrl() now also clears the query string (search params) in addition to the hash
scraper.ts: added ETag/If-None-Match conditional request support (skips re-download when data is unchanged, returns false); writes are now atomic via temp file + rename; replaced existsSync + readFile with a try/catch to eliminate the race condition
Tests: added Jasmine specs covering normalizeUrl (hash, query, trailing-slash), feature.ts route (moved→target, split→targets, 404, no data), and search.post.ts validation (503 on no data, 400 on empty/whitespace/non-string specs)

marcoscaceres · 2026-04-27T08:58:11Z

@copilot Can you review this and fix any issues that come up?

marcoscaceres · 2026-04-27T20:18:52Z

@copilot Can you confirm that all your comments have been addressed? Please resolve them if you’re happy. Can you fix anything new you see or outstanding from your feedback that hasn’t been addressed?

…tions, fix XSS - Return 400 (not 200) for non-published webhook actions, consistent with caniuse/update.ts and respec/builds/update.ts patterns - Set res.locals.reason = "action-not-published" for structured logging - Use a static message instead of echoing req.body.action to avoid reflected XSS (CodeQL js/reflected-xss) Agent-Logs-Url: https://github.com/speced/respec-web-services/sessions/5af37242-1fe0-4bdd-b54b-c45ab0c4120f

Copilot · 2026-04-27T20:24:49Z

@copilot Can you confirm that all your comments have been addressed? Please resolve them if you’re happy. Can you fix anything new you see or outstanding from your feedback that hasn’t been addres...

All comments are addressed. The one remaining open thread (update.ts webhook action filtering) was already handled in a previous commit, but I also found and fixed two additional issues in f450113:

update.ts status code: Changed the response for non-"published" webhook actions from 200 to 400, and added res.locals.reason = "action-not-published" — matching the caniuse and respec/builds route patterns.
Reflected XSS: The previous message echoed req.body.action back verbatim (Webhook action "${req.body.action}" ignored.), which CodeQL flagged as js/reflected-xss. Replaced with a static string.

All 62 specs pass, CodeQL is clean.

@copilot Can you review this and fix any issues that come up?

Reviewed and fixed in f450113:

update.ts was returning 200 for non-"published" webhook actions — changed to 400 with res.locals.reason = "action-not-published", consistent with caniuse/update.ts and respec/builds/update.ts
The error message was echoing req.body.action (user input), which is a reflected XSS risk — replaced with a static string

Everything else was already clean: 503 on missing data, whitespace-spec rejection, normalizeUrl stripping search params, ETag-based conditional downloads, atomic scraper writes, and full test coverage.

sidvishnoi · 2026-05-02T07:49:53Z

+  }
+
+  // Check if it's a moved or split feature in the raw data
+  const rawFeature = store.data?.features[featureId];


What if featureId is "constructor" here?

sidvishnoi · 2026-05-02T07:52:41Z

+  "baseline_update",
+);
+
+export default async function route(req: Request, res: Response) {


This requires setting up a webhook in that repo. Do we know someone who can do it for us?

sidvishnoi · 2026-05-02T07:54:06Z

+  afterEach(() => {
+    store.data = null;
+    store.byFeature = new Map();
+    store.bySpecUrl = new Map();
+  });


Do we care to cleanup if beforeEach is setting data anyway?

* feat(api/baseline): add web-features/baseline data endpoint Adds endpoints for serving Baseline browser availability data: - GET /api/baseline — full web-features dataset - GET /api/baseline/:feature — single feature lookup - POST /api/baseline/search — spec-URL-based feature search Data sourced from web-platform-dx/web-features GitHub releases. Enables ReSpec's implementationStatus feature (speced/respec#5174) without requiring unpkg.com dependency. Closes #490 * fix(api/baseline): return flat {result: [...]} from search endpoint Client expects {result: WebFeatureEntry[]}, server was returning Record<string, {id, feature}[]>. Flatten features, deduplicate by ID, and wrap in {result: [...]}. * Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix(api/baseline): address unresolved review comments - search.post.ts: return 503 when store.data is null (service unavailable) - search.post.ts: reject empty/whitespace-only strings in specs array - store.ts: normalizeUrl() now also clears search params (query string) - scraper.ts: add ETag/If-None-Match support to skip unnecessary re-downloads - scraper.ts: use atomic write (temp file + rename) to prevent corrupt data on interrupted writes - Add tests: normalizeUrl, feature route (moved/split/404), search.post validation Agent-Logs-Url: https://github.com/speced/respec-web-services/sessions/41823caf-5a5e-40fc-8bf7-fca753fb37ab * fix(api/baseline): replace existsSync with try-catch for ETag read, add return type doc Agent-Logs-Url: https://github.com/speced/respec-web-services/sessions/41823caf-5a5e-40fc-8bf7-fca753fb37ab * fix(api/baseline): update.ts returns 400 for non-published webhook actions, fix XSS - Return 400 (not 200) for non-published webhook actions, consistent with caniuse/update.ts and respec/builds/update.ts patterns - Set res.locals.reason = "action-not-published" for structured logging - Use a static message instead of echoing req.body.action to avoid reflected XSS (CodeQL js/reflected-xss) Agent-Logs-Url: https://github.com/speced/respec-web-services/sessions/5af37242-1fe0-4bdd-b54b-c45ab0c4120f --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>

marcoscaceres requested a review from Copilot April 25, 2026 12:26

Copilot started reviewing on behalf of marcoscaceres April 25, 2026 12:26 View session

Copilot AI reviewed Apr 25, 2026

View reviewed changes

Comment thread routes/api/baseline/search.post.ts

Comment thread routes/api/baseline/search.post.ts Outdated

Comment thread routes/api/baseline/update.ts

Comment thread routes/api/baseline/search.post.ts Outdated

marcoscaceres force-pushed the feat/baseline-endpoint branch from 403ac39 to 8f9a564 Compare April 25, 2026 13:21

marcoscaceres requested a review from Copilot April 25, 2026 13:21

Copilot started reviewing on behalf of marcoscaceres April 25, 2026 13:21 View session

Copilot AI reviewed Apr 25, 2026

View reviewed changes