ci: central orchestrator + auto-generated deps graph#154
Closed
TeoSlayer wants to merge 4 commits into
Closed
Conversation
Drop a TODO at the top of release.yml enumerating which secrets need to be re-created on `pilot-protocol/` before the repo transfer, since GitHub secrets do not survive a repo transfer. Currently the only expected secret is GITHUB_TOKEN (auto-issued). HOMEBREW_TAP_TOKEN was removed in #122 when update-homebrew.yml was dropped, and NPM_TOKEN / PYPI_TOKEN / COSIGN_KEY are placeholders for the auto-publish (PILOT-203) and binary-signing (PILOT-114) work that hasn't landed yet. This is documentation only — no behavior change. The comment block is load-bearing for the org migration; deleting it before the new org has its secrets configured will silently break the next release.
Two new jobs run after the existing release job: publish-manifest builds public/.well-known/latest.json from the tag's checksums.txt, then repository_dispatches it to pilot-protocol/website, which commits the JSON to main and triggers the Cloudflare deploy. The single canonical manifest at pilotprotocol.network/.well-known/latest.json is consumed by install.sh, the Homebrew formula bump workflow, and the SDK release helpers — one shockwave per release. shockwave fans out repository_dispatch(event_type=upstream-release) to homebrew-pilot, sdk-node, sdk-python, and sdk-swift so each consumer can run its own bump workflow. Per-target dispatch is soft-fail with a summary so a missing token on one repo does not block the others. Both jobs require a new SHOCKWAVE_DISPATCH_TOKEN secret with repository_dispatch scope on each downstream repo (prefer a GitHub App token over a PAT). When the secret is absent the steps emit a clear ::warning:: and exit 0 so existing release flow is not broken.
…ver templates
Adds a central release orchestrator that handles bidirectional dependency
propagation: when any package in the constellation ships a release, the
set of downstream nodes that need a bump is computed automatically and
each gets a dispatch in topological order.
Files:
.github/deps.json
Canonical dependency graph. 25 nodes: hub (web4), 15 Go siblings, 1
Go app, 1 FFI fan-in (libpilot), 3 SDKs, 1 brew tap, 1 surface
(website), 3 freeloaders (cosift, pilot-ca, wallet). Each node
declares its depends_on edges and a bump_policy (stable_only,
auto_commit_main, open_pr_instead_of_main). This is the single
source of truth — changes to the graph are PR-reviewed diffs here.
.github/workflows/orchestrator.yml
Receives package-released dispatches from any node. Computes the
reverse-transitive closure of the released package (BFS by depth
in deps.json), filters per-receiver stable_only when the upstream
is a prerelease, then fan-outs bump-upstream dispatches to each
affected repo. Supports workflow_dispatch with a dry_run flag for
rehearsal.
.github/workflows/_template-emit-release.yml
Reusable workflow that any sibling adopts: at the end of its own
release.yml it calls this with the node name, and it dispatches
package-released to the orchestrator. One token per sibling
(ORCHESTRATOR_DISPATCH_TOKEN) covers the hop.
.github/workflows/_template-bump-upstream.yml
Generic receiver template for Go-sibling repos: receives
bump-upstream, runs go mod edit -require=<module>@<version> +
go mod tidy + go build sanity check, then either commits to main
or opens a PR based on BUMP_MODE. SDK repos will need their own
package-manager-specific variants.
Closure smoke-tests pass:
web4 → 22 targets (every depending node, ordered by depth)
policy → 4 targets (libpilot + 3 SDKs)
handshake → 4 targets (libpilot + 3 SDKs)
libpilot → 3 targets (3 SDKs)
sdk-node → 0 targets (leaf)
cosift → 0 targets (freeloader)
This SUPERSEDES the direct shockwave fan-out in PR #151's release.yml
once orchestrator adoption rolls out — the four hardcoded repos there
become entries in deps.json instead. Keeping #151's direct fan-out for
now as the bootstrap path (orchestrator needs SHOCKWAVE_DISPATCH_TOKEN
set before it can do anything).
…eceiver templates
Replaces the hardcoded shockwave fan-out in release.yml with a single
notify-orchestrator step that emits package-released. The orchestrator
workflow reads .github/deps.json (now auto-generated by
scripts/build-deps.py from real go.mod / package.json / Package.swift)
and computes the reverse-transitive closure of the released package,
then dispatches per-target bump-upstream events in topological order.
Why this is better than the hardcoded fan-out:
- Adding/removing a sibling no longer touches release.yml; just create
or delete the repo and the next deps refresh picks it up.
- The graph is always in sync with reality. scripts/build-deps.py
parses live go.mod files; refresh-deps-graph.yml runs hourly and on
repository_dispatch deps-touched. When the regen produces a diff
it opens (or updates) auto/deps-graph-refresh PR.
- Sibling-to-sibling edges that the hardcoded shockwave missed
(rendezvous → handshake → libpilot, eventstream → dataexchange,
etc.) are now caught by including indirect requires.
- Stable-only policy is honored per-receiver inside the orchestrator
(the bump dispatch is skipped when upstream is a prerelease AND
the receiver wants stable only).
Hand-maintained input is now only:
.github/deps.policy.json — bump policy per node + freeloader markings
+ manual_nodes for repos with no manifest
(homebrew-pilot, website, freeloaders).
Files in this commit:
scripts/build-deps.py generator (gh + raw fetch)
.github/deps.json auto-output, 27 nodes
.github/deps.policy.json hand-maintained sidecar
.github/workflows/orchestrator.yml unchanged from prior commit
.github/workflows/refresh-deps-graph.yml NEW — scheduled regen + PR
.github/workflows/release.yml drop shockwave; add
notify-orchestrator step
.github/workflows/_template-emit-release.yml unchanged
.github/workflows/_template-bump-upstream.yml unchanged
Required secret: SHOCKWAVE_DISPATCH_TOKEN (repository_dispatch scope on
every sibling + the orchestrator's own repo).
This was referenced May 28, 2026
Collaborator
Author
|
Superseded by pilot-protocol/release. The orchestrator + deps graph + manifest worker now live in a single dedicated repo. This consolidates the cascade machinery cleanly outside web4. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR consolidates the cascade system into one mechanism: the orchestrator. Closes the chapter on the hardcoded shockwave fan-out.
What changed
Cascade
release.ymlshockwave job → replaced with a singlenotify-orchestratorstep that emitspackage-released. The orchestrator does the rest.orchestrator.yml(from prior commit) receivespackage-releasedevents, reads.github/deps.json, computes reverse-transitive closure by depth, dispatches per-targetbump-upstreamevents in topological order.release.yml— the graph is data, not code.Auto-generated deps graph
scripts/build-deps.pyparses livego.mod(+package.json,Package.swift) from everypilot-protocol/*repo +TeoSlayer/pilotprotocoland produces.github/deps.json(27 nodes, every edge).refresh-deps-graph.ymlruns the generator hourly, on manual dispatch, and onrepository_dispatch deps-touched. If the regen produces a diff against the checked-in file, opens (or updates) a PR. The graph cannot silently drift..github/deps.policy.json— bump policy per node, freeloader markings, manual nodes for repos with no manifest (homebrew, website).Receiver templates (from prior commit)
_template-emit-release.yml— what each sibling installs to notify the orchestrator on its own release._template-bump-upstream.yml— what each sibling installs to receive a bump dispatch (go mod edit+go mod tidy+ commit/PR per policy).Closure tests (passing)
Required setup
SHOCKWAVE_DISPATCH_TOKENsecret on this repo withrepository_dispatchscope on every sibling + this repo itself. Prefer a GitHub App (actions/create-github-app-token@v1) over a PAT.Supersedes / closes
Test plan
workflow_dispatchorchestrator.ymlwithpackage: common, version: v0.2.0, dry_run: trueand verify the closure JSON.dry_run: falseto fire actual dispatches (once token is set).v1.10.6-rc1) and verifynotify-orchestratorstep fires correctly.🤖 Generated with Claude Code