Skip to content

Commit bfc8acc

Browse files
AlexgodorojaAlexgodoroja
andauthored
pilotctl: self-heal the next-steps cache so the installed fleet gets graphs (#399)
pilotctl: lazily self-heal the next-steps cache (reach the installed fleet) The graph was cached only at install, so the feature reached only apps installed AFTER their graph shipped — the existing fleet, and any republished graph, never got hints without a manual `install --force`. That is a blocker: a feature that doesn't reach installed apps isn't shipped. ensureNextStepsFresh closes the gap from the call path, off the render: - steady state is a single stat of a freshness marker and returns instantly — the hot-path-is-a-local-read guarantee holds for all but the rare re-check; - when the 12h TTL has elapsed it spends ONE bounded (400ms) catalogue fetch, so an app installed before its graph existed renders on its very NEXT call, and a republished graph propagates within the TTL; - a fetch that can't reach the catalogue drops a short retry-cooldown marker so an offline host backs off instead of re-fetching every call; - an app the catalogue no longer publishes a graph for has its stale cache dropped. Proven live against the production catalogue: an app with no cache rendered on its first call (no --force), and the next call did zero network. Marker gating, failure cooldown, disabled no-op, and traversal-refusal are unit-tested; gosec clean (writes/removes re-confined to the install root via resolveUnder). Co-authored-by: Alexgodoroja <alex@vulturelabs.io>
1 parent 3964b9f commit bfc8acc

13 files changed

Lines changed: 678 additions & 32 deletions

CHANGELOG.md

Lines changed: 34 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,10 @@ Detailed per-release notes are on the
99

1010
## [Unreleased]
1111

12-
Reliable P2P data transfer across NAT. Tag intentionally held for review.
12+
## [1.12.8] - 2026-07-16
13+
14+
Reliable P2P data transfer across NAT, plus cold-start onboarding fixes: agents
15+
now reach the network on their first install instead of stalling before it.
1316

1417
### Added
1518
- **Inbound-path watchdog — the long-uptime NAT wedge now auto-recovers.**
@@ -40,6 +43,16 @@ Reliable P2P data transfer across NAT. Tag intentionally held for review.
4043
full resolve + NAT hole-punch flow and prefers the direct path.
4144
- `send-file` reports `transport`, `sha256`, and `throughput_mbps`; adds
4245
`--timeout`.
46+
- **Goose joins skill injection.** `pilotctl skills` now lists the real
47+
injection targets — Claude Code, OpenClaw, PicoClaw, OpenHands, Hermes, and
48+
Goose — instead of naming Cursor, which was never a target. (The daemon's
49+
runtime inject-manifest adds `~/.config/goose` with its heartbeat in
50+
`.goosehints`.)
51+
- **Installer GET STARTED walkthrough.** The post-install output now walks a new
52+
operator through the send-`--wait` / read-newest-inbox idiom, pilot-director
53+
(live data), list-agents (known specialists), the app store (local
54+
capabilities), and peers/trust — with copy-paste examples — instead of a
55+
four-line hint.
4356

4457
### Changed
4558
- **Message of the day now rides the pilot-changelog pipeline.** The daemon's
@@ -52,6 +65,13 @@ Reliable P2P data transfer across NAT. Tag intentionally held for review.
5265
banner, `important_update` field, and `motd` in `info` work exactly as
5366
before; only the source feed and its shape changed. Override with
5467
`--motd-feed-url` / `$PILOT_MOTD_URL` as before. (motd)
68+
- **`pilotctl skills` and `pilotctl skills paths` are now read-only.** They ran a
69+
mutating reconcile and then reported the *pre-write* state, so the first
70+
`pilotctl skills` on a fresh host both created the skill files and labelled them
71+
"absent — next: create" — a false failure an agent reads as a broken install.
72+
They now use the injector's read-only dry run: nothing is written just by
73+
looking, and the reported state reflects what is actually on disk. The mutating
74+
`skills check` / `skills enable` are unchanged.
5575

5676
### Fixed
5777
- **`pilotctl daemon start` no longer reports a false failure on slow boots.**
@@ -70,6 +90,19 @@ Reliable P2P data transfer across NAT. Tag intentionally held for review.
7090
- **Dual-NAT key-exchange convergence.** Key exchange is now sent over both
7191
the direct and relay paths, so two NAT'd peers reconverge in ~1 RTT
7292
instead of waiting 28 s–3 min for blackhole detection.
93+
- **The installer now reaches the skill-injection step on the hosts agents
94+
actually run on.** Where `systemctl` exists but systemd is not PID 1
95+
(containers, WSL, CI), the systemd setup ran `systemctl daemon-reload`, which
96+
returns non-zero — and under `set -e` aborted the install ~200 lines before the
97+
`pilotctl skills check` first pass, so injection fired on 0 of the tested cold
98+
starts. The systemd block is now gated on a booted system
99+
(`[ -d /run/systemd/system ]`), its calls can no longer abort the script, and a
100+
non-systemd host is told to run `pilotctl daemon start`.
101+
- **Headless installs no longer die at the email prompt.** A non-interactive
102+
install (piped, no controlling terminal) blocked on `read … < /dev/tty` and
103+
exited `rc=2`; email is now prompted only when a TTY is present, otherwise the
104+
daemon auto-synthesizes its `<fingerprint>@nodes.pilotprotocol.network`
105+
identity. `PILOT_EMAIL` is documented for headless installs.
73106

74107
## [1.12.0] - 2026-06-21
75108

cmd/pilotctl/appstore.go

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1281,6 +1281,15 @@ func cmdAppStoreInstall(args []string) {
12811281
// catalogue simply means no hints — never a failed install.
12821282
cacheNextSteps(finalDir, fetchNextStepsForInstall(m.ID))
12831283

1284+
// Record the catalogue bundle sha we just installed so `outdated`/`upgrade`
1285+
// can detect a SAME-VERSION republish — a rebuilt adapter shipped under an
1286+
// unchanged app_version (the aegis argv-fix shape) that a version compare
1287+
// alone silently misses. Catalogue installs only; a sideload has no
1288+
// catalogue bundle to compare against. Best-effort.
1289+
if source == installSourceCatalogue {
1290+
recordInstalledBundleSHA(finalDir, m.ID)
1291+
}
1292+
12841293
// Mirror to the install-root-level pilotctl-audit log too, so
12851294
// the install+uninstall lifecycle pair stays reconstructable
12861295
// after the app dir (and its per-app supervisor.log) is gone.
@@ -2194,6 +2203,13 @@ func cmdAppStoreCall(args []string) {
21942203
fatalHint("invalid_argument", "json-args must be valid JSON", "%v", err)
21952204
}
21962205
}
2206+
// Lazily self-heal the cached graph before the call so it is current for
2207+
// whichever outcome renders — this is what lets an app installed before its
2208+
// graph existed (or a republished graph) reach the fleet without a manual
2209+
// reinstall. Off the render path, bounded, best-effort: steady state is a
2210+
// single stat and returns instantly (see ensureNextStepsFresh).
2211+
ensureNextStepsFresh(appID)
2212+
21972213
var result json.RawMessage
21982214
if err := ipc.Call(conn, method, argsValue, &result); err != nil {
21992215
hint := fmt.Sprintf("the app %q rejected or could not handle %q", appID, method)

cmd/pilotctl/appstore_nextsteps.go

Lines changed: 140 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@ import (
4444
"path/filepath"
4545
"regexp"
4646
"strings"
47+
"time"
4748
)
4849

4950
// nextStepsFileName is the per-app cache written at install, read at call.
@@ -371,27 +372,160 @@ func cacheNextSteps(appDir string, g *nextStepsGraph) {
371372
if err != nil {
372373
return
373374
}
374-
_ = os.WriteFile(filepath.Join(out, nextStepsFileName), data, 0o600) // #nosec G304 -- confined by resolveUnder
375+
_ = os.WriteFile(filepath.Join(out, nextStepsFileName), data, 0o600) // #nosec G304 G703 -- path re-confined to the install root by resolveUnder above
375376
}
376377

377378
// fetchNextStepsForInstall pulls the graph out of the catalogue metadata for an
378379
// app id. Best-effort and quiet: an app with no catalogue entry (a sideload), no
379380
// metadata_url, or an unreachable host simply gets no graph. It is called once
380381
// per install, never on the call path.
381382
func fetchNextStepsForInstall(appID string) *nextStepsGraph {
383+
g, _ := fetchNextStepsAuthoritative(appID)
384+
return g
385+
}
386+
387+
// fetchNextStepsAuthoritative fetches the app's graph from the catalogue and
388+
// reports whether the answer is AUTHORITATIVE — i.e. whether the catalogue and
389+
// (if present) the metadata doc were actually reached. It is the distinction
390+
// ensureNextStepsFresh needs: a nil graph with ok=true means "this app has no
391+
// graph, stop asking for a while", whereas ok=false means "we couldn't reach the
392+
// catalogue, try again soon". Collapsing the two (as a bare *graph return does)
393+
// would either re-fetch every call forever or give up on a transient blip.
394+
func fetchNextStepsAuthoritative(appID string) (graph *nextStepsGraph, ok bool) {
382395
c, err := loadCatalogue()
383396
if err != nil {
384-
return nil
397+
return nil, false // could not reach/verify the catalogue
385398
}
386399
for i := range c.Apps {
387400
if c.Apps[i].ID != appID {
388401
continue
389402
}
390403
m, err := loadAppMetadata(c.Apps[i])
391-
if err != nil || m == nil {
392-
return nil
404+
if err != nil {
405+
return nil, false // entry found but the detail fetch/verify failed
393406
}
394-
return m.NextSteps
407+
if m == nil {
408+
return nil, true // no metadata doc = authoritatively no graph
409+
}
410+
return m.NextSteps, true
411+
}
412+
return nil, true // not in the catalogue (e.g. a sideload) = authoritatively none
413+
}
414+
415+
// ── lazy self-heal ───────────────────────────────────────────────────────────
416+
//
417+
// The graph is cached at install, but the installed FLEET predates most graphs,
418+
// and a republished graph never touches an existing cache. Without healing, the
419+
// feature only reaches apps installed AFTER their graph shipped — a blocker.
420+
//
421+
// ensureNextStepsFresh closes that gap from the call path, off the render:
422+
// - steady state is a single stat of a freshness marker → returns immediately,
423+
// no network (the hot-path-is-local guarantee holds for all but the rare
424+
// re-check);
425+
// - when the TTL has elapsed it spends ONE bounded fetch, so an app installed
426+
// before its graph existed renders on its very next call, and a republished
427+
// graph propagates within the TTL;
428+
// - a fetch that can't reach the catalogue backs off briefly rather than
429+
// re-trying every call.
430+
const (
431+
// nextStepsFreshTTL bounds how often we re-consult the catalogue per app.
432+
// Graphs change rarely; a half-day is plenty fresh and keeps fetches rare.
433+
nextStepsFreshTTL = 12 * time.Hour
434+
// nextStepsRetryCool caps re-tries after a failed fetch so an offline host
435+
// doesn't spend the fetch budget on every call.
436+
nextStepsRetryCool = 2 * time.Minute
437+
// nextStepsFetchBudget bounds the ONE inline fetch on the call path. Generous
438+
// for a ~15 KiB metadata doc from a CDN, tight enough that a slow/hung host
439+
// never noticeably delays the app call. On a miss past this budget the app
440+
// simply renders no hint this once and the next call (past the retry cool)
441+
// tries again.
442+
nextStepsFetchBudget = 400 * time.Millisecond
443+
nsCheckedMarker = ".next-steps.checked" // mtime = last authoritative check
444+
nsRetryMarker = ".next-steps.retry" // mtime = last failed fetch
445+
)
446+
447+
// ensureNextStepsFresh is called once per `appstore call`, before the IPC call,
448+
// so the cache is current for whichever outcome renders. It never blocks longer
449+
// than nextStepsFetchBudget and never errors out of the call.
450+
func ensureNextStepsFresh(appID string) {
451+
if nextStepsDisabled() {
452+
return
453+
}
454+
dir, err := resolveUnder(appStoreRoot(), appID)
455+
if err != nil {
456+
return
457+
}
458+
// Recent authoritative answer (a cached graph, or a confirmed absence) → done.
459+
if markerFresh(filepath.Join(dir, nsCheckedMarker), nextStepsFreshTTL) {
460+
return
461+
}
462+
// Recently failed → back off rather than re-fetch on every call.
463+
if markerFresh(filepath.Join(dir, nsRetryMarker), nextStepsRetryCool) {
464+
return
465+
}
466+
g, ok := fetchNextStepsTimed(appID, nextStepsFetchBudget)
467+
if !ok {
468+
touchMarker(dir, nsRetryMarker) // couldn't reach the catalogue; short cooldown
469+
return
470+
}
471+
if g != nil && len(g.Edges) > 0 {
472+
cacheNextSteps(dir, g)
473+
} else {
474+
// The catalogue authoritatively has no graph for this app now — drop any
475+
// stale cache so we never render a graph the catalogue stopped publishing.
476+
removeUnderAppDir(dir, nextStepsFileName)
477+
}
478+
touchMarker(dir, nsCheckedMarker)
479+
removeUnderAppDir(dir, nsRetryMarker)
480+
}
481+
482+
// fetchNextStepsTimed runs the authoritative fetch under a hard deadline. On
483+
// timeout it reports ok=false (treated as a transient failure) and lets the
484+
// goroutine finish and fall off on its own — it never writes anything, so an
485+
// abandoned fetch can't race the cache.
486+
func fetchNextStepsTimed(appID string, budget time.Duration) (*nextStepsGraph, bool) {
487+
type res struct {
488+
g *nextStepsGraph
489+
ok bool
490+
}
491+
ch := make(chan res, 1)
492+
go func() {
493+
g, ok := fetchNextStepsAuthoritative(appID)
494+
ch <- res{g, ok}
495+
}()
496+
select {
497+
case r := <-ch:
498+
return r.g, r.ok
499+
case <-time.After(budget):
500+
return nil, false
501+
}
502+
}
503+
504+
// markerFresh reports whether a marker file exists and is younger than ttl.
505+
func markerFresh(path string, ttl time.Duration) bool {
506+
fi, err := os.Stat(path)
507+
if err != nil {
508+
return false
509+
}
510+
return time.Since(fi.ModTime()) < ttl
511+
}
512+
513+
// touchMarker writes/updates a freshness marker in the app dir (mtime = now).
514+
func touchMarker(dir, name string) {
515+
out, err := resolveUnder(appStoreRoot(), filepath.Base(dir))
516+
if err != nil {
517+
return
518+
}
519+
_ = os.WriteFile(filepath.Join(out, name), []byte(time.Now().UTC().Format(time.RFC3339)), 0o600) // #nosec G304 G703 -- path re-confined to the install root by resolveUnder above
520+
}
521+
522+
// removeUnderAppDir deletes a file inside an installed app dir, re-confining the
523+
// path to the install root (the same guard the read and write paths use) so a
524+
// crafted app id can never delete outside the tree the app store owns.
525+
func removeUnderAppDir(dir, name string) {
526+
out, err := resolveUnder(appStoreRoot(), filepath.Base(dir))
527+
if err != nil {
528+
return
395529
}
396-
return nil
530+
_ = os.Remove(filepath.Join(out, name)) // #nosec G304 G703 -- path re-confined to the install root by resolveUnder above
397531
}

cmd/pilotctl/appstore_nextsteps_test.go

Lines changed: 86 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ import (
88
"path/filepath"
99
"strings"
1010
"testing"
11+
"time"
1112
)
1213

1314
// The error strings below are VERBATIM, captured from a live daemon on this
@@ -370,3 +371,88 @@ func TestCacheNextStepsRefusesTraversal(t *testing.T) {
370371
t.Fatal("cacheNextSteps wrote outside the install root")
371372
}
372373
}
374+
375+
// ── self-heal (ensureNextStepsFresh) ─────────────────────────────────────────
376+
377+
func TestMarkerFresh(t *testing.T) {
378+
dir := t.TempDir()
379+
p := filepath.Join(dir, "m")
380+
if markerFresh(p, time.Hour) {
381+
t.Fatal("absent marker must not be fresh")
382+
}
383+
if err := os.WriteFile(p, []byte("x"), 0o600); err != nil {
384+
t.Fatal(err)
385+
}
386+
if !markerFresh(p, time.Hour) {
387+
t.Fatal("just-written marker must be fresh")
388+
}
389+
if markerFresh(p, time.Nanosecond) {
390+
t.Fatal("marker older than a nanosecond TTL must be stale")
391+
}
392+
}
393+
394+
// A recent authoritative check must short-circuit — no fetch, no new markers.
395+
// This is the steady-state guarantee: the call path stays a local stat.
396+
func TestEnsureFreshSkipsWhenChecked(t *testing.T) {
397+
root := withAppRoot(t)
398+
dir := filepath.Join(root, "io.pilot.testapp")
399+
if err := os.MkdirAll(dir, 0o755); err != nil {
400+
t.Fatal(err)
401+
}
402+
touchMarker(dir, nsCheckedMarker)
403+
// Point the catalogue at an unreachable URL: if a fetch were attempted it
404+
// would fail and drop a retry marker. A fresh checked marker must prevent that.
405+
t.Setenv("PILOT_APPSTORE_CATALOG_URL", "http://127.0.0.1:1/nope.json")
406+
ensureNextStepsFresh("io.pilot.testapp")
407+
if _, err := os.Stat(filepath.Join(dir, nsRetryMarker)); err == nil {
408+
t.Fatal("a fresh checked marker must short-circuit before any fetch")
409+
}
410+
}
411+
412+
// With no markers and an unreachable catalogue, the fetch fails and a retry
413+
// marker is written so the next call backs off instead of re-fetching.
414+
func TestEnsureFreshWritesRetryOnFailure(t *testing.T) {
415+
root := withAppRoot(t)
416+
dir := filepath.Join(root, "io.pilot.testapp")
417+
if err := os.MkdirAll(dir, 0o755); err != nil {
418+
t.Fatal(err)
419+
}
420+
t.Setenv("PILOT_APPSTORE_CATALOG_URL", "http://127.0.0.1:1/nope.json")
421+
ensureNextStepsFresh("io.pilot.testapp")
422+
if !markerFresh(filepath.Join(dir, nsRetryMarker), time.Minute) {
423+
t.Fatal("a failed fetch must drop a retry marker for back-off")
424+
}
425+
// A fresh retry marker must then short-circuit a second call.
426+
// (Re-running must not clobber into a checked marker.)
427+
ensureNextStepsFresh("io.pilot.testapp")
428+
if markerFresh(filepath.Join(dir, nsCheckedMarker), time.Minute) {
429+
t.Fatal("a still-failing fetch must not produce an authoritative checked marker")
430+
}
431+
}
432+
433+
func TestEnsureFreshDisabledIsNoOp(t *testing.T) {
434+
root := withAppRoot(t)
435+
dir := filepath.Join(root, "io.pilot.testapp")
436+
if err := os.MkdirAll(dir, 0o755); err != nil {
437+
t.Fatal(err)
438+
}
439+
t.Setenv("PILOT_NEXT_STEPS", "off")
440+
t.Setenv("PILOT_APPSTORE_CATALOG_URL", "http://127.0.0.1:1/nope.json")
441+
ensureNextStepsFresh("io.pilot.testapp")
442+
for _, m := range []string{nsCheckedMarker, nsRetryMarker} {
443+
if _, err := os.Stat(filepath.Join(dir, m)); err == nil {
444+
t.Fatalf("disabled self-heal must write no %s marker", m)
445+
}
446+
}
447+
}
448+
449+
// A traversing app id must not let the healer write or delete outside the root.
450+
func TestEnsureFreshRefusesTraversal(t *testing.T) {
451+
withAppRoot(t)
452+
t.Setenv("PILOT_APPSTORE_CATALOG_URL", "http://127.0.0.1:1/nope.json")
453+
ensureNextStepsFresh("../../../tmp") // must resolveUnder-reject and no-op
454+
if _, err := os.Stat("/tmp/" + nsRetryMarker); err == nil {
455+
os.Remove("/tmp/" + nsRetryMarker)
456+
t.Fatal("traversal id must not write a marker outside the install root")
457+
}
458+
}

0 commit comments

Comments
 (0)