Skip to content

Commit f24b502

Browse files
teovlteovl
committed
release(v1.10.3): compat mode on single TCP/443
Compat-mode daemons now talk to the network through exactly one outbound port: TCP/443. The registry channel migrates from raw TCP/9000 to TLS on 443 via nginx ssl_preread + SNI routing on the rendezvous box; the existing beacon WSS bridge stays on 443 (different SNI). On the daemon, -registry-trust system lets the registry client trust the OS x509 root store, removing the previous requirement to pin a SHA-256 fingerprint. When -transport=compat is set and -registry is left at the UDP-mode default, the daemon auto-targets registry.pilotprotocol.network:443 with TLS + system trust. Code: - pkg/daemon/daemon.go: Config.RegistryTrust ('pinned'|'system') + branched DialTLSPool path - cmd/daemon/main.go: -registry-trust flag, compat-mode default override for -registry/-registry-tls/-registry-trust - tests/zz_compat_registry_tls_test.go: TLS-pinned register + WSS bridge end-to-end, and a defence-in-depth test that system-trust refuses a self-signed cert Ops: - docs/RUNBOOK-compat-443-only.md walks through the rendezvous-side deployment (DNS, certbot, nginx vhost moves, stream block, smoke + rollback). The forward path was applied to pilot-rendezvous-new today and verified live (registry.pilotprotocol.network:443 serves its own cert, beacon.pilotprotocol.network:443 still serves WSS, a fresh compat daemon registered + fetched list-agents over a single TCP/443 connection pool). Also lifts the dashboard's /api/metrics-history default range from 'all 17,280 samples (~213 MB)' to 720 samples (~9 MB) and bumps responder worker counts on pilot-service-agents — both unrelated to compat mode but landed in the same window (see CHANGELOG).
1 parent dcdfec1 commit f24b502

5 files changed

Lines changed: 563 additions & 4 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,95 @@ project uses [Semantic Versioning](https://semver.org/).
77
Detailed per-release notes are on the
88
[GitHub Releases page](https://github.com/TeoSlayer/pilotprotocol/releases).
99

10+
## [1.10.3] - 2026-05-19
11+
12+
### Added
13+
- **Compat mode is now single-port-443.** A new `-registry-trust system` flag
14+
on `pilot-daemon` lets the registry client validate the production cert via
15+
the OS x509 root store instead of requiring a pinned SHA-256 fingerprint.
16+
When `-transport=compat` is selected and the operator hasn't overridden
17+
`-registry`, the daemon now auto-targets
18+
`registry.pilotprotocol.network:443` with TLS + system trust — same
19+
hostname:port as the existing beacon WSS bridge. The two endpoints are
20+
multiplexed on a single nginx `listen 443;` via the stream module's
21+
`ssl_preread` SNI router (see `docs/RUNBOOK-compat-443-only.md`).
22+
Net effect: a daemon running in Render, Replit Agent, or any other
23+
managed-claw sandbox that allows only outbound TCP/443 can now register +
24+
resolve + tunnel data — **zero TCP/9000, zero UDP**.
25+
26+
### Changed
27+
- `pilot-daemon -registry-tls` no longer requires `-registry-fingerprint` if
28+
`-registry-trust=system` is set. The two modes coexist:
29+
- `pinned` (default): cert pinned by SHA-256 fingerprint — back-compat
30+
with the existing single-VM deploy
31+
- `system`: OS root store — works against any publicly-trusted CA cert,
32+
i.e. the Let's Encrypt cert on `registry.pilotprotocol.network`
33+
34+
### Tests
35+
- `tests/zz_compat_registry_tls_test.go::TestCompatRegistryTLSPinned` —
36+
spins up an in-process TLS registry + beacon, points a compat daemon at
37+
the TLS registry with a pinned cert, and verifies Register + WSS bridge
38+
attach end-to-end.
39+
- `tests/zz_compat_registry_tls_test.go::TestCompatRegistryTrustSystemRejectsBadCert` —
40+
pins that `-registry-trust=system` refuses an untrusted self-signed cert
41+
(MITM defence-in-depth).
42+
43+
### Ops / deployment
44+
45+
Single runbook covers the production rollout:
46+
[`docs/RUNBOOK-compat-443-only.md`](docs/RUNBOOK-compat-443-only.md). Steps:
47+
48+
1. DNS A record `registry.pilotprotocol.network → 34.71.57.205`
49+
2. `certbot certonly --nginx -d registry.pilotprotocol.network`
50+
3. Move existing nginx HTTP-on-443 vhosts (`beacon.*`, `console.*`,
51+
`polo.*`) to internal `listen 127.0.0.1:14443 ssl;`
52+
4. Add nginx `stream {}` block: `ssl_preread` on, SNI map →
53+
`registry.*` to a stream-mode TLS terminator on 14444 (which proxies
54+
plain TCP to the existing registry on `127.0.0.1:9000`),
55+
everything else to 14443.
56+
5. Smoke-test via `openssl s_client` + a real compat daemon.
57+
58+
Rollback is one `nginx -t && systemctl reload nginx` after restoring the
59+
sed-backed-up sites-enabled files.
60+
61+
### Operational changes applied to `pilot-service-agents` (2026-05-19)
62+
63+
These are server-side ops changes that landed alongside the v1.10.3 code work
64+
to make all 435+ specialist agents responsive under high traffic:
65+
66+
- **`/usr/local/bin/responder-wave-restart.sh` was no-op'd**. Original backed
67+
up alongside as `responder-wave-restart.sh.full-backup-1779223695`. The
68+
every-~10-min wave restart was killing in-flight reply queues across all
69+
agents (each restart SIGTERM'd Python responder workers mid-`pilot-send-stdin`
70+
subprocess), so list-agents queries from outside the fleet timed out
71+
even though dispatch worked. Reply latency dropped from 30s+ timeouts to
72+
sub-second after disabling.
73+
- **Per-agent systemd drop-in
74+
`/etc/systemd/system/pilot-responder-*.service.d/high-traffic.conf`** on
75+
all 435 responders:
76+
- `RESPONDER_REPLY_WORKERS=16` (default `2`)
77+
- `RESPONDER_INBOX_WORKERS=6` (default `2`)
78+
- `RESPONDER_REPLY_QUEUE_MAX=2048`, `RESPONDER_INBOX_QUEUE_MAX=2048`
79+
- `RESPONDER_STALE_REPLY_AFTER=60s` (default `60s`),
80+
`RESPONDER_INBOX_MAX_AGE=60s`
81+
16 workers absorb the swarm noise (failing sends to unreachable swarm
82+
nodes each pin a worker for the hardcoded 25s `pilot-send-stdin`
83+
timeout) while leaving slots for legitimate user traffic.
84+
- **`/opt/pilot-dashboard/metrics-snapshot.json` despiked**. The
85+
restart-wave at 21:29:54 UTC inflated cumulative request counters by
86+
2,866,001 over 133s (21,548 req/s artefact). A second pass at 21:34:11
87+
added another 903,719 over 103s (8,740 req/s). Both spikes flattened to
88+
the 300 req/s baseline rate; the excess subtracted from all subsequent
89+
entries to keep cumulative totals consistent. Backup at
90+
`metrics-snapshot.json.pre-despike-1779227221`.
91+
- **`/opt/pilot-dashboard/server.py::/api/metrics-history` default range**
92+
changed from "all 17,280 samples" to **720 samples (1 hour at 5s
93+
intervals)**. The full buffer is ~213 MB JSON which crashed the
94+
network-stats page when fetched with no query params; the default now
95+
returns ~9 MB in ~1.6s. Callers that want the full 24h window can still
96+
pass `?range=17280`. Backup at
97+
`server.py.pre-metricshistory-default-1779227619`.
98+
1099
## [1.10.2] - 2026-05-19
11100

12101
### Fixed

‎cmd/daemon/main.go‎

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -52,7 +52,8 @@ func main() {
5252
fakeListenAddr := flag.String("fake-listen-addr", "", "advertise this listen_addr to the registry instead of the real one (real socket binding unaffected)")
5353
encrypt := flag.Bool("encrypt", true, "enable tunnel-layer encryption (X25519 + AES-256-GCM)")
5454
registryTLS := flag.Bool("registry-tls", false, "use TLS for registry connection")
55-
registryFingerprint := flag.String("registry-fingerprint", "", "hex SHA-256 fingerprint of registry TLS certificate")
55+
registryFingerprint := flag.String("registry-fingerprint", "", "hex SHA-256 fingerprint of registry TLS certificate (required when -registry-trust=pinned)")
56+
registryTrust := flag.String("registry-trust", "pinned", "trust store for -registry-tls: 'pinned' (verify cert against -registry-fingerprint) or 'system' (OS x509 root store — used for compat-mode registry on registry.pilotprotocol.network:443 with Let's Encrypt)")
5657
identityPath := flag.String("identity", "", "path to persist Ed25519 identity (enables stable identity across restarts)")
5758
email := flag.String("email", "", "email address for account identification and key recovery")
5859
owner := flag.String("owner", "", "(deprecated: use -email) owner identifier for key rotation recovery")
@@ -111,6 +112,29 @@ func main() {
111112
config.ApplyToFlags(cfg)
112113
}
113114

115+
// Compat-mode 443-only defaults. When -transport=compat is selected
116+
// and the operator hasn't explicitly overridden -registry/-registry-tls/
117+
// -registry-trust, route the registry to its TLS hostname (TCP/443
118+
// via nginx SNI routing on the production rendezvous box) so the
119+
// daemon really does use a single port. The TCP/9000 fallback is
120+
// still available to anyone who passes -registry explicitly.
121+
if *transportMode == "compat" {
122+
explicit := map[string]bool{}
123+
flag.Visit(func(f *flag.Flag) { explicit[f.Name] = true })
124+
if !explicit["registry"] && os.Getenv("PILOT_REGISTRY") == "" {
125+
v := "registry.pilotprotocol.network:443"
126+
registryAddr = &v
127+
}
128+
if !explicit["registry-tls"] {
129+
v := true
130+
registryTLS = &v
131+
}
132+
if !explicit["registry-trust"] {
133+
v := "system"
134+
registryTrust = &v
135+
}
136+
}
137+
114138
logging.Setup(*logLevel, *logFormat)
115139

116140
d := daemon.New(daemon.Config{
@@ -123,6 +147,7 @@ func main() {
123147
Encrypt: *encrypt,
124148
RegistryTLS: *registryTLS,
125149
RegistryFingerprint: *registryFingerprint,
150+
RegistryTrust: *registryTrust,
126151
IdentityPath: *identityPath,
127152
Email: *email,
128153
Owner: *owner,

‎docs/RUNBOOK-compat-443-only.md‎

Lines changed: 226 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,226 @@
1+
# Runbook — compat mode on a single TCP/443 port (v1.10.3)
2+
3+
Goal: a compat-mode daemon (`-transport=compat`) talks to the network using
4+
**only outbound TCP/443**. Today it also needs TCP/9000 for the registry; this
5+
runbook routes both ports through 443 on the production rendezvous box using
6+
nginx SNI-routing.
7+
8+
## Architecture
9+
10+
```
11+
┌─────────────────────────────────────────┐
12+
│ rendezvous box (34.71.57.205) │
13+
│ │
14+
daemon ──────────┼──── 443 ── nginx stream ── ssl_preread ─┼─┐
15+
(TCP/443 only) │ │ (read SNI) │ │
16+
│ │ │ │
17+
│ ├── if SNI=beacon.* ──> 14443 (nginx http,
18+
│ │ terminates TLS,
19+
│ │ proxies WS to
20+
│ │ 127.0.0.1:18443)
21+
│ │ │
22+
│ ├── if SNI=console.* ─> 14443 (existing vhost)
23+
│ │ │
24+
│ └── if SNI=registry.* ─> 14444 (nginx stream,
25+
│ terminates TLS,
26+
│ proxies plain
27+
│ TCP to 9000)
28+
│ │
29+
│ registry on :9000 ←──── plain TCP ────┘
30+
└─────────────────────────────────────────┘
31+
```
32+
33+
Nothing in `pkg/registry` changes — the registry stays plain-TCP on 9000.
34+
TLS is terminated entirely at nginx; the certificate for
35+
`registry.pilotprotocol.network` lives only on the rendezvous box.
36+
37+
## Pre-conditions
38+
39+
- nginx 1.18+ built with `stream_ssl_module` and `stream_ssl_preread_module`
40+
(the production box has both — confirmed via `nginx -V`).
41+
- DNS A record: `registry.pilotprotocol.network` → `34.71.57.205`.
42+
- certbot installed (already present for beacon/console/polo).
43+
44+
## Step 1 — DNS
45+
46+
Add an A record at the DNS provider (Cloudflare for `pilotprotocol.network`):
47+
48+
```
49+
registry.pilotprotocol.network A 34.71.57.205 proxy=off
50+
```
51+
52+
Wait for propagation (≤5 min for short TTLs). Verify:
53+
54+
```bash
55+
dig +short registry.pilotprotocol.network @1.1.1.1
56+
# → 34.71.57.205
57+
```
58+
59+
## Step 2 — Let's Encrypt certificate
60+
61+
Run on the rendezvous box:
62+
63+
```bash
64+
# Stop nginx briefly so certbot can bind :80 standalone, OR use the
65+
# existing webroot challenge if the certbot/nginx pairing is healthy.
66+
sudo certbot certonly --nginx -d registry.pilotprotocol.network \
67+
--non-interactive --agree-tos -m teodor@vulturelabs.io
68+
69+
# Verify cert files exist
70+
sudo ls -la /etc/letsencrypt/live/registry.pilotprotocol.network/
71+
```
72+
73+
Auto-renewal is already wired (the certbot systemd timer covers all certs in
74+
`/etc/letsencrypt/live/`).
75+
76+
## Step 3 — Move the existing nginx HTTP vhosts off port 443
77+
78+
Each file in `/etc/nginx/sites-enabled/` with `listen 443 ssl;` needs to
79+
change to `listen 127.0.0.1:14443 ssl;`. Affected vhosts:
80+
81+
- `beacon.pilotprotocol.network`
82+
- `console.pilotprotocol.network`
83+
- `polo.pilotprotocol.network`
84+
85+
```bash
86+
cd /etc/nginx/sites-enabled
87+
sudo sed -i.pre-sni-$(date +%s) \
88+
-e 's/listen 443 ssl;/listen 127.0.0.1:14443 ssl;/g' \
89+
-e 's/listen \[::\]:443 ssl;/listen [::1]:14443 ssl;/g' \
90+
beacon.pilotprotocol.network \
91+
console.pilotprotocol.network \
92+
polo.pilotprotocol.network
93+
```
94+
95+
Sanity check the changes:
96+
97+
```bash
98+
grep "listen " /etc/nginx/sites-enabled/{beacon,console,polo}.pilotprotocol.network
99+
# expected: listen 127.0.0.1:14443 ssl;
100+
```
101+
102+
## Step 4 — Add the stream block
103+
104+
Create `/etc/nginx/streams.d/443-sni.conf` (or wherever the existing nginx.conf
105+
sources `stream {}` from — on Debian-style installs it's `/etc/nginx/nginx.conf`
106+
under the `stream` directive). Add:
107+
108+
```nginx
109+
# /etc/nginx/streams.d/443-sni.conf
110+
# Front-end on 443: pre-read TLS ClientHello SNI, route by hostname.
111+
# - registry.pilotprotocol.network -> nginx stream TLS terminator on 14444
112+
# -> plain TCP to local registry on 9000
113+
# - everything else (beacon/console/polo)
114+
# -> nginx http on 14443 (terminates TLS,
115+
# dispatches by Host header to existing vhosts)
116+
map $ssl_preread_server_name $sni_443_upstream {
117+
registry.pilotprotocol.network 127.0.0.1:14444;
118+
default 127.0.0.1:14443;
119+
}
120+
121+
server {
122+
listen 443;
123+
listen [::]:443;
124+
ssl_preread on;
125+
proxy_pass $sni_443_upstream;
126+
proxy_protocol off;
127+
# Standard 443 timeouts — keep connections idle a while since compat
128+
# daemons keep one long-lived WS connection per node.
129+
proxy_timeout 1h;
130+
proxy_connect_timeout 5s;
131+
}
132+
133+
# Internal listener that TERMINATES TLS for registry.pilotprotocol.network
134+
# and proxies plain bytes to the existing plain-TCP registry on 9000.
135+
# The registry's wire protocol (length-prefixed JSON) is non-HTTP, which is
136+
# why this terminates here in the stream module rather than as an HTTP vhost.
137+
server {
138+
listen 127.0.0.1:14444 ssl;
139+
ssl_certificate /etc/letsencrypt/live/registry.pilotprotocol.network/fullchain.pem;
140+
ssl_certificate_key /etc/letsencrypt/live/registry.pilotprotocol.network/privkey.pem;
141+
ssl_protocols TLSv1.2 TLSv1.3;
142+
proxy_pass 127.0.0.1:9000;
143+
proxy_timeout 5m;
144+
proxy_connect_timeout 5s;
145+
}
146+
```
147+
148+
If `/etc/nginx/streams.d/` doesn't exist, instead add a `stream { ... }` block
149+
to `/etc/nginx/nginx.conf` (at the same level as `http { ... }`).
150+
151+
## Step 5 — Test the nginx config + reload
152+
153+
```bash
154+
sudo nginx -t
155+
# → "syntax is ok" and "test is successful"
156+
157+
sudo systemctl reload nginx
158+
159+
# Verify the new listeners
160+
sudo ss -tlnp | grep -E ':443|:14443|:14444'
161+
# → expect nginx on 0.0.0.0:443 (stream) and on 127.0.0.1:14443 + 14444
162+
```
163+
164+
## Step 6 — Smoke-test end-to-end from outside
165+
166+
From any machine with DNS access (the macbook works):
167+
168+
```bash
169+
# Sanity: beacon.* still works
170+
curl -sI https://beacon.pilotprotocol.network/health
171+
# → HTTP/1.1 200 OK
172+
173+
# Registry: TLS handshake should complete to the new SNI
174+
openssl s_client -connect registry.pilotprotocol.network:443 \
175+
-servername registry.pilotprotocol.network -brief </dev/null
176+
# → CONNECTION ESTABLISHED + cert CN=registry.pilotprotocol.network
177+
178+
# End-to-end: dial the registry via the v1.10.3 daemon
179+
PILOT_REGISTRY=registry.pilotprotocol.network:443 \
180+
~/.pilot/bin/pilot-daemon \
181+
-transport compat \
182+
-registry-tls -registry-trust system \
183+
-socket /tmp/443-compat.sock \
184+
-identity /tmp/443-id.json \
185+
-email compat-443-test@vulturelabs.io \
186+
-log-level info &
187+
sleep 5
188+
PILOT_SOCKET=/tmp/443-compat.sock ~/.pilot/bin/pilotctl info | head -5
189+
# → daemon should report a node_id; only TCP/443 connections in lsof
190+
```
191+
192+
## Rollback
193+
194+
If anything goes sideways:
195+
196+
```bash
197+
# Revert sites-enabled changes
198+
cd /etc/nginx/sites-enabled
199+
for f in beacon.pilotprotocol.network console.pilotprotocol.network \
200+
polo.pilotprotocol.network; do
201+
sudo cp -p "$f.pre-sni-<TIMESTAMP>" "$f"
202+
done
203+
204+
# Drop the new stream config
205+
sudo rm /etc/nginx/streams.d/443-sni.conf
206+
sudo nginx -t && sudo systemctl reload nginx
207+
```
208+
209+
`/etc/letsencrypt/live/registry.pilotprotocol.network/` is harmless to leave in
210+
place even after rollback.
211+
212+
## After-the-fact: what the daemon does
213+
214+
In v1.10.3, `pilot-daemon -transport compat` (with no other flags) auto-
215+
configures itself for the 443-only path:
216+
217+
```
218+
-registry registry.pilotprotocol.network:443 (default in compat mode)
219+
-registry-tls true (auto-set in compat mode)
220+
-registry-trust system (auto-set in compat mode)
221+
-compat-beacon wss://beacon.pilotprotocol.network/v1/compat
222+
-tls-trust system
223+
```
224+
225+
Anyone passing `-registry <hostport>` explicitly opts out of the auto-config
226+
and continues to hit the plain-TCP 9000 path.

0 commit comments

Comments
 (0)