Skip to content

Commit e43535d

Browse files
teovlteovl
committed
Release: v1.9.0 notes + macOS ad-hoc codesign in workflow
Add ad-hoc codesign + xattr -cr step on darwin builds in release.yml. Without this, downloaded macOS tarballs trigger "killed: 9" or Gatekeeper "cannot be opened because Apple cannot check it for malicious software" on first run. Ad-hoc signing (no Apple Developer account, no notarization) is enough to make the binary launch. RELEASE_NOTES_v1.9.0.md: per-theme summary covering registry hardening, P1-010 closure, multi-beacon discovery, peer-IP privacy, network-member-count gate, runtime banner endpoint, pilotctl updates, async IPC, pubsub retry, and integration-suite stability fixes. CONTRIBUTING.md: lock-discipline rule (per task 30 follow-up).
1 parent b17db41 commit e43535d

4 files changed

Lines changed: 280 additions & 0 deletions

File tree

‎.github/workflows/release.yml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,24 @@ jobs:
7272
go build -ldflags "$LDFLAGS" -o dist/$bin ./cmd/$bin
7373
done
7474
75+
# Ad-hoc codesign + drop quarantine on darwin builds so downloaded
76+
# binaries don't trigger "killed: 9" or Gatekeeper "cannot be opened
77+
# because Apple cannot check it for malicious software". Ad-hoc
78+
# signing is enough to make the binary launch — full notarization
79+
# would need APPLE_ID + APP_SPECIFIC_PASSWORD secrets and a paid
80+
# developer cert; out of scope for this release.
81+
- name: macOS ad-hoc codesign + strip quarantine
82+
if: matrix.goos == 'darwin'
83+
run: |
84+
for bin in dist/*; do
85+
echo "codesigning $bin..."
86+
codesign --force --deep --sign - "$bin"
87+
xattr -cr "$bin" || true
88+
done
89+
for bin in dist/*; do
90+
codesign -dv "$bin" 2>&1 | grep -E "Signature|Authority|TeamIdentifier" | head -1
91+
done
92+
7593
- name: Smoke test binaries
7694
if: matrix.goos == 'linux' && matrix.goarch == 'amd64' || matrix.goos == 'darwin'
7795
run: |

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,9 @@ web/.wrangler/
5959
# Personal analytics scripts + their reports - never commit
6060
web/scripts/local/
6161

62+
# Local node-bin staging — never commit (multi-MB binaries)
63+
abpn/
64+
6265
# Root-level npm tooling (not the main module)
6366
/package.json
6467
/package-lock.json

‎CONTRIBUTING.md‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,52 @@ docs/ # Documentation
8181
WHITEPAPER.pdf # Protocol whitepaper (LaTeX source: WHITEPAPER.tex)
8282
```
8383

84+
## Lock discipline (required reading for registry/daemon contributors)
85+
86+
The registry uses several mutexes covering different scopes. The 2026-04-28
87+
production incident (see `done/22-rendezvous-lock-contention-incident` in
88+
the project vault) was caused by signature-verification work running while
89+
a global lock was held; under load the contention queue grew to ~95k
90+
goroutines on a single mutex.
91+
92+
The rule below exists so this class of bug cannot recur:
93+
94+
> **No signature verification, no JSON marshal, no encoding/decoding, no
95+
> network I/O while holding `s.mu.Lock`, `s.mu.RLock`, or any equivalent
96+
> daemon-side mutex. CPU work belongs OUTSIDE the lock.**
97+
98+
When a handler needs to read state, do CPU-bound work, then mutate state,
99+
use the **3-phase pattern**:
100+
101+
```go
102+
// Phase 1 — RLock to snapshot what verify needs:
103+
s.mu.RLock()
104+
node, ok := s.nodes[nodeID]
105+
if !ok { s.mu.RUnlock(); return ... }
106+
pubKey := append([]byte(nil), node.PublicKey...) // copy, don't alias
107+
s.mu.RUnlock()
108+
109+
// Phase 2 — verify / encode / parse OUTSIDE any lock:
110+
if !crypto.Verify(pubKey, challenge, sig) { return ... }
111+
112+
// Phase 3 — Lock to mutate; re-check pubkey hasn't rotated under us:
113+
s.mu.Lock()
114+
defer s.mu.Unlock()
115+
node, ok = s.nodes[nodeID]
116+
if !ok || !bytes.Equal(node.PublicKey, pubKey) {
117+
return fmt.Errorf("concurrent change, retry")
118+
}
119+
// ... commit ...
120+
```
121+
122+
Working examples in the codebase: `handleRequestHandshake`,
123+
`handlePollHandshakes`, `handleRespondHandshake`, `handleRotateKey`,
124+
`handleSetKeyExpiry` (all `pkg/registry/server.go`).
125+
126+
The lock-ordering invariants are documented as a comment block at the top
127+
of `pkg/registry/server.go` and `pkg/daemon/tunnel.go` — never reverse
128+
those tiers.
129+
84130
## Contributing to the Python SDK
85131

86132
See the **[Python SDK Contributing Guide](sdk/python/CONTRIBUTING.md)**.

‎RELEASE_NOTES_v1.9.0.md‎

Lines changed: 213 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,213 @@
1+
# v1.9.0 — Release Notes
2+
3+
**Scope:** 96 commits + uncommitted hardening since `v1.8.0` (2026-04-21 → 2026-04-30).
4+
5+
This is the stable v1.9.0 cut, going directly from `v1.8.0` without an
6+
`rc3` — `rc1` and `rc2` were development-only milestones that
7+
established the P1-010 tunnel-desync fix and the registry lock-contention
8+
recovery; everything in those plus this round's privacy + ops work is
9+
included.
10+
11+
The headline themes are: **registry lock-contention closed end-to-end**,
12+
**P1-010 tunnel desync recovery**, **registry hardening (panic recovery,
13+
WAL replay, 3-phase rotate-key/set-key-expiry)**, **multi-beacon discovery
14+
+ relay autoscale**, and **privacy: peer IPs hidden by default in pilotctl
15+
output and network member counts admin-gated at the registry**.
16+
17+
## What's new
18+
19+
### Registry hardening
20+
21+
Closes follow-ups left open by `v1.9.0-rc1`/`rc2` and the 2026-04-28
22+
lock-contention incident.
23+
24+
- **Panic recovery** in registry handlers (`pkg/registry/panic_recovery.go`).
25+
A panicking handler no longer takes the whole server down; the panic is
26+
logged with a stack trace and the connection is closed.
27+
- **WAL replay + size cap** (`pkg/registry/wal_replay.go`,
28+
`wal_size_cap_test.go`). The write-ahead log now replays cleanly on
29+
startup and self-bounds to a configured size cap so a stuck `flushSave`
30+
cannot let the WAL grow until the volume fills.
31+
- **3-phase pattern** applied to `handleRotateKey` and
32+
`handleSetKeyExpiry` — RLock for verify, unlock, Lock only for the
33+
mutation. Eliminates the lock-hold during signature checks that the
34+
audit team flagged on 2026-04-29.
35+
- **Heartbeat-ack pre-build** to keep the hot path off `json.Marshal`.
36+
- **`list_nodes` cache-invalidation hooks** wired through every membership
37+
mutation (5 per-net + 5 admin sites).
38+
- **`apply_snapshot` outside-lock build** — the registry now constructs
39+
the snapshot deep-copy without holding `s.mu`, dropping the lock-hold
40+
from 1.7–2.5 s peaks (in the prior incident) to ~5 ms.
41+
- **Snapshot lock-discipline tests** (`apply_snapshot_test.go`,
42+
`snapshot_lock_test.go`).
43+
44+
### Tunnel + dataexchange (P1-010 closure)
45+
46+
These four landed in the rc-line and are bundled into v1.9.0:
47+
48+
1. **Salvage replay** — per-peer plaintext ring buffer, re-encrypted with
49+
the new key after rekey. Recovers fire-and-forget RPCs that were sent
50+
under stale crypto during the rekey window. Webhook event:
51+
`tunnel.desync_salvage`.
52+
2. **Rekey retransmit loop** — per-peer `pendingRekey` state and a
53+
bounded retransmit goroutine (every 4 s, capped at 5 retries). Closes
54+
the case where our `key_exchange` or the peer's reply was dropped.
55+
Webhook event: `tunnel.rekey_gave_up`.
56+
3. **Salvage cap of 4 entries** — original 32 caused a replay-storm on
57+
rekey; 4 covers the realistic shape and is ~6 KiB/peer instead of
58+
~48 KiB.
59+
4. **Preserve nonce/replay state across duplicate `key_exchange`** —
60+
replace `tm.crypto[N]` only when the entry doesn't exist or the peer's
61+
pubkey actually changed. Pinned by
62+
`TestHandleKeyExchangeDuplicatePreservesCryptoState`.
63+
64+
### Multi-beacon discovery
65+
66+
`pkg/daemon/beacon_select.go`, `beacon_discovery.go`. Daemons can now be
67+
configured with a comma-separated `-beacon` list (or learn the list from
68+
the registry's beacon-list endpoint when it ships) and FNV-of-pubkey-hash
69+
to one of the live beacons. Foundation for the autoscaling beacon MIG.
70+
71+
Companion infrastructure: a shell+systemd reconciler now runs on the
72+
production rendezvous VM that watches the GCP `beacon-mig` and rebuilds
73+
iptables `PREROUTING` DNAT rules to load-balance UDP `:9001` across
74+
the live MIG instances. Documented in
75+
`07-Procedures/Procedure - Maintenance Banner Update` (vault) and task 34.
76+
This is an ops-side change that doesn't affect the daemon binary, but
77+
unlocks fleet-wide relay autoscale.
78+
79+
### Privacy: peer IPs no longer in pilotctl output by default
80+
81+
`pilotctl peers`, `pilotctl info`, `pilotctl lookup` now strip
82+
endpoint-bearing fields (`endpoint`, `real_addr`, `lan_addrs`,
83+
`public_addr`, `stun_addr`, `observed_addr`) from their output by
84+
default. Pass `--show-endpoints` to bring them back for ops/debug.
85+
Searches that match endpoint substrings only resolve when
86+
`--show-endpoints` is set, so a search prompt cannot leak IP existence.
87+
88+
Pinned by `cmd/pilotctl/redact_test.go::TestRedactPeerEndpointsRemovesIPFields`
89+
(top-level + nested + per-peer redaction; identity/counter fields
90+
preserved).
91+
92+
### Privacy: network member counts now admin-gated
93+
94+
`pkg/registry/server.go::handleListNetworks` now omits the per-network
95+
`members` count from `list_networks` responses unless the request
96+
carries a valid `admin_token`. Network identity (id, name, join_rule,
97+
enterprise) stays visible — daemons can still discover what to join —
98+
but population per network is treated as a privacy-sensitive aggregate.
99+
Pinned by `tests/network_test.go::TestListNetworksMembersAdminGated`.
100+
101+
The pilotctl `network list` view shows `—` in the MEMBERS column when
102+
the count is hidden, so the redaction is visible-by-policy rather than
103+
mistaken for broken.
104+
105+
`pkg/registry/client.go::ListNetworks` now takes a variadic `adminToken`
106+
to opt into seeing the count. **Backwards-compat note:** any caller
107+
that did `nm["members"].(float64)` without a nil check will now panic
108+
when called without the token. The 8 in-repo call sites were updated;
109+
external SDKs need the same fix or to pass the token.
110+
111+
### Operations: runtime banner endpoint
112+
113+
The dashboard's amber maintenance notice (rendered above the release
114+
banner) is now mutable at runtime via an admin-only HTTP endpoint —
115+
`PUT /api/banner` on the rendezvous's HTTP port. Persisted to
116+
`/var/lib/pilot/banner.txt`, survives restart, no daemon reconnect
117+
storm. Both `GET` and `PUT` require the rendezvous admin token; the
118+
public still sees the banner via `/api/stats.maintenance_banner` and
119+
the dashboard HTML.
120+
121+
Procedure: `07-Procedures/Procedure - Maintenance Banner Update` in the
122+
vault. Pinned by `tests/dashboard_test.go::TestDashboardBannerEndpoint`.
123+
124+
### Operations: `pilotctl updates`
125+
126+
New command to fetch the public changelog feed:
127+
128+
```
129+
pilotctl updates [--count N] [--scope <category>] [--refresh]
130+
pilotctl --json updates
131+
```
132+
133+
Reads `https://teoslayer.github.io/pilot-changelog/feed.xml`, caches at
134+
`~/.pilot/updates-cache.xml` for 5 min, falls back to stale cache when
135+
offline. No external deps (RSS parsed with `encoding/xml`).
136+
137+
Pinned by 11 tests in `cmd/pilotctl/updates_test.go` (cold fetch, cache
138+
hit, `--refresh` bypass, stale-cache refetch, offline fallback,
139+
no-fallback-without-cache, scope filter, count truncation, and combined).
140+
141+
### Daemon stability
142+
143+
- **Async IPC write queue** — buffered `sendCh` per ipcConn + single
144+
writer goroutine, done-channel pattern for race-free shutdown,
145+
`ErrIPCBackpressure` on slow clients. Removes 96.7% of daemon
146+
mutex-wait time observed in stress profiles. Pinned by
147+
`pkg/daemon/ipc_async_write_test.go`.
148+
- **Pubsub retry resilience** — failed publish attempts retry once
149+
before raising; per-peer failure counter exposed via webhook.
150+
`pkg/daemon/pubsub_resilience_test.go`.
151+
- **Lock-ordering invariants** documented in `CONTRIBUTING.md`; review
152+
rule pinned.
153+
154+
### Integration suite
155+
156+
Not shipped in the binary, but bundled into the same release because
157+
it's part of the v1.9.0 quality bar:
158+
159+
- **Webhook readiness helper** — `webhook_helpers.sh` waits for the
160+
python:3.12-slim sink to be HTTP-bound before bringing up agents.
161+
Removes the "webhook-sink never came up" flake that hit 4–5 webhook
162+
tests under suite-wide pressure.
163+
- **NAT IPAM retry** — `test_nat_dual_symmetric.sh` and the shared
164+
`nat_test_common.sh` now sweep stale Docker networks on `pool overlaps`
165+
and retry the compose-up.
166+
- **Bench harness** — `bench_relay.sh` + `docker-compose.multi.bench.yml`
167+
characterize relay throughput without the chaos overlay's 4 MiB tmpfs
168+
cap (an artifact of the disk-full tests). Documented bench numbers in
169+
task 34.
170+
- **Lane bump 3 → 5** for parallel NAT tests.
171+
172+
## What's intentionally NOT in v1.9.0
173+
174+
- **`pilot-rendezvous` decommission of the `:9001` beacon role** —
175+
daemons still hit the singleton's `:9001` directly; the iptables DNAT
176+
reconciler forwards to the MIG. Decom waits on the daemon-side
177+
`/api/beacons` discovery roll-out (see task 15 follow-ups).
178+
- **The old `pilot-rendezvous` VM (`35.238.4.185`)** — still receiving
179+
~60 KB/s sustained from clients pointed at the old IP. Cannot be
180+
deleted until those clients are migrated.
181+
182+
## Migration notes
183+
184+
- **External SDKs / dashboards** that call `list_networks` and read
185+
`members` need to either pass `admin_token` in the request or handle
186+
the field being absent. Pre-v1.9.0 daemons keep returning the count
187+
to anyone; after upgrading the rendezvous, they don't.
188+
- **Scripts / tooling** that grepped pilotctl output for IP fragments
189+
need `--show-endpoints` (or, preferably, switch to node IDs).
190+
191+
## Acceptance evidence
192+
193+
- `go vet ./...` clean.
194+
- `go test ./... -count=1` — 14/14 packages pass.
195+
- `go test ./... -race -count=1 -timeout 600s` — pending in CI matrix
196+
(running concurrently with this draft).
197+
- Integration suite (Docker-based, 232 tests, 8 parallel workers) —
198+
pending in CI matrix.
199+
- 8 production binaries cross-build clean for `linux/{amd64,arm64}` and
200+
`darwin/{amd64,arm64}` with `-X main.version=v1.9.0`.
201+
202+
## Production status at tag time
203+
204+
- `pilot-rendezvous-new` (us-central1, e2-standard-16) running v1.9.0
205+
with the runtime banner endpoint. **Deployed 2026-04-30.**
206+
- `beacon-mig` autoscaler at `min=2 / max=10`, 60% CPU target. iptables
207+
reconciler `pilot-mig-reconcile.timer` healthy (30 s tick).
208+
- `pilot-agent-a` / `pilot-agent-b` running v1.9.0 daemon binaries from
209+
the verification benches.
210+
- Remaining production daemons (`pilot-service-agents`, NAT agents,
211+
`pilot-swarm-1`, `pilot-swarm-2b`) on prior binaries — rollout
212+
scheduled post-soak.
213+
- Active fleet at tag time: ~119,500 total / ~100,500 active nodes.

0 commit comments

Comments
 (0)