|
| 1 | +--- |
| 2 | +name: network-spot-check |
| 3 | +description: Spot-check the health of a live Aztec network deployment by sweeping recent GCP logs for warn/error messages, mapping each to the deployed code, classifying expected vs unexpected, and verifying the hard invariants (no slashing, no attestation timeouts, no unexplained prunes/conflicts). Use when asked to "spot-check", "review logs", or "health check" a network (staging, testnet, devnet, ...) over a recent window. |
| 4 | +argument-hint: <namespace and window, e.g. "staging-internal, last 4 hours"> |
| 5 | +--- |
| 6 | + |
| 7 | +# Network spot-check |
| 8 | + |
| 9 | +A structured review of a live network's recent logs. This goes deeper than a one-off |
| 10 | +`/network-logs` query: it sweeps a whole time window, maps every distinct warn/error to |
| 11 | +the code that emits it, verifies the network's hard invariants with explicit queries, and |
| 12 | +produces a classified report. It was designed from a staging-internal review; the |
| 13 | +procedure generalizes to any namespace. |
| 14 | + |
| 15 | +## Parameters |
| 16 | + |
| 17 | +- **Namespace**: which deployment (e.g. `staging-internal`, `testnet`, `devnet`). Ask if unclear. |
| 18 | +- **Window**: default the last 4 hours. |
| 19 | +- **Deployed branch**: which git branch the network runs. Do NOT assume it is the current |
| 20 | + checkout. As of 2026-07, `staging-internal` runs `v5-next` while the default checkout is |
| 21 | + `next` — a log string that exists on one line may not exist on the other. If unknown, ask, |
| 22 | + or determine it empirically: `git grep "<log message>" origin/<branch> -- yarn-project` and |
| 23 | + `git log -S "<log message>" --all --oneline` to see which line contains a given message. |
| 24 | +- **Invariants / acceptable errors**: unless the operator overrides, use the defaults below. |
| 25 | + |
| 26 | +## Procedure |
| 27 | + |
| 28 | +### 1. Delegate the sweep to a subagent |
| 29 | + |
| 30 | +Spawn a **sonnet** `network-logs` subagent (keeps the token-heavy log dumps out of the main |
| 31 | +context). Its prompt must ask for, over the full window: |
| 32 | + |
| 33 | +1. All warn+error entries from the core block-building/sync modules — **sequencer**, |
| 34 | + **validator**, **archiver**, and **slasher** (`slasher:*` plus the offense watchers: |
| 35 | + `attested-invalid-proposal-watcher`, `checkpoint-equivocation-watcher`, |
| 36 | + `data-withholding-watcher`, `broadcasted-invalid-checkpoint-proposal-watcher`) — and |
| 37 | + from the supporting subsystems where their failures actually surface: `p2p:*` |
| 38 | + (attestation pool/gossip, tx collection, peering), `world-state:*` (sync), |
| 39 | + `node:l1-tx-utils` / `ethereum:publisher` (L1 tx submission — where failed checkpoint |
| 40 | + pushes show up), and `prover-node:*` (epoch-monitor / epoch-proving-job — epochs that |
| 41 | + fail to prove are what eventually cause prunes). Page through the window; do not |
| 42 | + sample only the newest entries. |
| 43 | +2. Grouping by **distinct message template**, with count, first/last timestamp, and emitting pods. |
| 44 | +3. Code mapping: grep the checkout for each template to find the emitting `file:line` and |
| 45 | + understand the condition that fires it. |
| 46 | +4. Surrounding context (same pod, ±2 min, all severities) for anything significant: what led |
| 47 | + to it, and did the node recover. |
| 48 | +5. The explicit invariant checks from step 3 below, with the queries used. |
| 49 | +6. Overall health signals: checkpoint/block heights advancing monotonically with no gaps, |
| 50 | + archiver download lag, pod restarts (a restart marks a redeploy — relevant for classification). |
| 51 | + |
| 52 | +Have it return raw structured data (tables, sample lines with timestamps and pods), not prose. |
| 53 | + |
| 54 | +### 2. Verify the code mapping yourself |
| 55 | + |
| 56 | +The subagent greps the working checkout; re-check anything it could not find against the |
| 57 | +**deployed branch** (`git grep "<msg>" origin/<deployed-branch>`, `git log -S`). Read the |
| 58 | +emitting function before classifying: the surrounding code and comments usually state whether |
| 59 | +the path is expected (e.g. an early-return for empty blocks) or a genuine failure. |
| 60 | + |
| 61 | +### 3. Explicit invariant checks (never skip, even if the sweep looks clean) |
| 62 | + |
| 63 | +Run these as direct queries over the window, all severities, and report each as a |
| 64 | +zero/non-zero with the query used: |
| 65 | + |
| 66 | +- **Slashing** (must be zero): `[Ss]lash` — then distinguish real slashing from bookkeeping. |
| 67 | + `Starting new slashing round N` (slasher_client.ts `handleNewRound`) is routine round |
| 68 | + rotation, NOT slashing. Also check `executed with` (slash execution), `[Oo]ffen[cs]e` |
| 69 | + (offense detection), and `Slashing is disabled`. Any activity from the offense watchers |
| 70 | + listed in step 1 means a node misbehaved (equivocation, invalid proposal, data |
| 71 | + withholding) — investigate it even if no slash was ever executed. |
| 72 | +- **Attestation-collection timeouts** (must be zero): the exact string is |
| 73 | + `Timeout while waiting for attestations` (sequencer-client, checkpoint proposal job, |
| 74 | + fires on `AttestationTimeoutError`). |
| 75 | +- **Prunes**: `[Pp]run`. A short prune following a checkpoint that failed to push to L1 |
| 76 | + under congestion is acceptable; anything else needs a root cause. |
| 77 | +- **Conflicts / reorgs**: `[Cc]onflict`, `[Rr]eorg` — must be zero. |
| 78 | +- **Timeout sweep**: `[Tt]imeout|[Tt]imed out` — classify every hit. |
| 79 | + |
| 80 | +### 4. Classification |
| 81 | + |
| 82 | +**Acceptable (list, don't alarm):** |
| 83 | +- Checkpoint push to L1 failing under L1 congestion, followed by a short prune. |
| 84 | +- Errors clearly caused by a redeploy/restart (correlate with pod restart times). |
| 85 | +- Known noise (verify these are still current before relying on the list): |
| 86 | + - `Uniswap V4 StateView contract not found, skipping fee asset price oracle` — StateView |
| 87 | + only exists on mainnet; expected everywhere else. |
| 88 | + - OTel telemetry noise: `BatchSpanProcessor dropping spans` and |
| 89 | + `PeriodicExportingMetricReader ... Request Timeout` blips (cross-pod, simultaneous = |
| 90 | + shared collector hiccup; confirm block production continued through it). |
| 91 | + - `Provided no txs to ... addTxs` from the prover orchestrator — expected for every empty |
| 92 | + block (demoted to verbose in PR #24593; only relevant on lines without that change). |
| 93 | + |
| 94 | +**Investigate and report prominently (raw lines, timestamps, pods, counts, root cause):** |
| 95 | +any slashing, attestation-collection timeouts, unexplained prunes, conflicts/reorgs, sync |
| 96 | +failures, or unclassified timeouts. |
| 97 | + |
| 98 | +### 5. Correlating L1-wait timeouts with actual L1 blocks |
| 99 | + |
| 100 | +`Timed out waiting for previous L1 block before sending requests, proceeding` |
| 101 | +(sequencer-publisher, from PR #24037) is the designed fallback when the L1 slot right before |
| 102 | +the target slot appears skipped. To confirm each occurrence against ground truth: |
| 103 | + |
| 104 | +1. Pull the entries with `--format=json`: the `jsonPayload` carries `previousL1BlockTs`, |
| 105 | + `waitDeadlineTs` (= previousL1BlockTs + 8s default), `targetSlot`. |
| 106 | +2. For each `previousL1BlockTs`, find the bracketing L1 blocks via an execution RPC |
| 107 | + (`eth_getBlockByNumber`; estimate the block number from a latest-block anchor at 12s/slot, |
| 108 | + then step). L1 slot timestamps are 12s-aligned: if no block carries exactly that timestamp |
| 109 | + (i.e. it falls inside a ≥24s gap), the slot was genuinely missed and the warn is benign. |
| 110 | +3. Ask the operator for an RPC endpoint for the network's L1 if none is configured. Be |
| 111 | + gentle with third-party endpoints: cache every block fetched, sleep between calls, and |
| 112 | + back off on HTTP 429. |
| 113 | + |
| 114 | +If the slot was NOT missed, the node's L1 RPC served the block late — then (and only then) |
| 115 | +dig into what the node's L1 watchers/archiver were doing at that moment. |
| 116 | + |
| 117 | +### 6. Deliverable |
| 118 | + |
| 119 | +Full classified report (distinct-template table, invariant check results with queries, |
| 120 | +context samples, health signals) as a gist; short outcome-first summary in the thread. If a |
| 121 | +log turns out to be mis-leveled (routine control flow logged at warn), propose demoting it — |
| 122 | +but only open a PR when the operator asks, and against the branch they name. |
0 commit comments