feat(/w3c/update): add endpoint to trigger groups data refresh by marcoscaceres · Pull Request #505 · speced/respec-web-services

marcoscaceres · 2026-04-26T00:41:26Z

Summary

Replaces the closed PR #496 (which incorrectly tried to use GitHub Actions).

Adds setInterval that calls the W3C groups update script every 24 hours on the running server
Reloads in-memory groups data from disk after each update
Handles missing groups.json gracefully at startup (empty store instead of crash)
Follows existing setInterval pattern from xref cache invalidation

Closes #395

Copilot

Pull request overview

Implements server-side auto-refresh of the W3C groups list so /w3c/groups stays up-to-date without relying on external automation.

Changes:

Schedule a 24-hour refresh that runs the W3C groups update script and reloads in-memory data from disk.
Make the W3C groups route resilient to a missing groups.json at startup (empty store fallback).
Improve sh() by rejecting on child process errors and waiting for "close" instead of "exit".

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
utils/sh.ts	Improves child process lifecycle handling by listening for `"error"` and `"close"`.
routes/w3c/group.ts	Adds periodic refresh + in-memory reload logic; adds startup fallback when `groups.json` is absent.
docs/superpowers/plans/2026-04-25-typescript6-migration.md	Adds internal planning doc (not directly related to W3C groups refresh).
docs/superpowers/plans/2026-04-25-test-coverage-blitz.md	Adds internal planning doc (not directly related to W3C groups refresh).
docs/superpowers/plans/2026-04-25-node24-modernization.md	Adds internal planning doc (not directly related to W3C groups refresh).
.claude/scheduled_tasks.lock	Adds a Claude runtime lockfile artifact.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Adds a setInterval that calls the update script daily and reloads the in-memory groups data from disk. Follows the existing pattern from xref's periodic cache invalidation. Also handles missing groups.json gracefully at startup (empty store instead of crash). Closes #395

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Replace setInterval with self-scheduling setTimeout so the next refresh only fires after the current one completes. Add in-flight guard for safety. On startup JSON parse failure, trigger an immediate refreshGroups() to self-heal.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

marcoscaceres · 2026-04-27T09:04:01Z

@copilot can you fix those?

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

marcoscaceres · 2026-04-27T14:48:22Z

@copilot can you confirm all your feedback has been addressed and resolved everything that has? Can you fix anything that wasn’t addressed and resolve those comments.

Agent-Logs-Url: https://github.com/speced/respec-web-services/sessions/728917e2-18ac-49b2-8ef9-9630dfdc2e9f Co-authored-by: marcoscaceres <870154+marcoscaceres@users.noreply.github.com>

Copilot · 2026-04-27T14:53:14Z

@copilot can you confirm all your feedback has been addressed and resolved everything that has? Can you fix anything that wasn’t addressed and resolve those comments.

All feedback has been addressed in 33c5e5f:

Corrupted file: Restored from the clean 06a077c baseline (previous "Apply suggestion" commits had merged code without removing old code)
reloadGroups() boolean return: Now returns true/false so refreshGroups() only logs success when the reload actually worked
Retry on any failure: nextDelay now uses refreshed ? 24h : 15m — retries sooner on any failure, not just missing-file cases
All other previously-resolved items (lazy import, in-flight guard, EMPTY_GROUPS fallback, self-scheduling setTimeout, immediate refresh on missing/corrupt file) remain in place

Restructured per review feedback: - Use BackgroundTaskQueue with worker thread (matches xref/caniuse) - Only write groups.json on successful scrape - Add POST /w3c/update endpoint for cron-triggered refreshes - Remove inline setTimeout/setInterval timer

sidvishnoi · 2026-04-29T20:20:12Z


 const w3c = Router();
 w3c.get("/groups{/:shortname}{/:type}", cors(), groupsRoute);
+w3c.post("/update", updateRoute);


Should we add the cron call from a GitHub action? Or something on the server (using cron or pm2)

I'd lean toward a GitHub Action with schedule: cron trigger that hits POST /w3c/update, matching how we could trigger xref/caniuse externally. Keeps the cron config in version control and avoids pm2/server config. Happy to set that up as a follow-up PR once this lands.

Remove redundant existsSync guards (readFileSync throws into catch). Add authGithubWebhook middleware to POST /w3c/update to prevent spam. Add W3C_GROUPS_SECRET to .env.sample. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The reloadGroups() return value was silently discarded, so a successful scrape followed by a failed reload would return 200. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

sidvishnoi

LGTM code-wise, but what'll trigger the GitHub webhook??

sidvishnoi · 2026-05-15T09:27:15Z


 const w3c = Router();
 w3c.get("/groups{/:shortname}{/:type}", cors(), groupsRoute);
+w3c.post("/update", authGithubWebhook(env("W3C_GROUPS_SECRET")), updateRoute);


What will trigger this webhook?

it's supposed to be a cron job.

Similar to this:

respec-web-services/.github/workflows/codeql-analysis.yml

Line 20 in d0a0f9e

schedule:

A cron GitHub action in this repo, that sends a request to update endpoint? Will it have same payload format we expect in authGithubWebhook?

Edit: just saw #505 (comment)

Yeah, does that sound ok?

marcoscaceres requested a review from Copilot April 26, 2026 00:41

Copilot started reviewing on behalf of marcoscaceres April 26, 2026 00:42 View session