fix: swallow lighthouse head 404 noise on transient REST API race

[barnabasbusa]

Summary

Lighthouse occasionally emits SSE head (and sometimes block) events for a block root that is not yet queryable via its own REST API. The block always lands on the REST endpoint a moment later — I verified this on a running kurtosis devnet: every previously-404'd root I checked (slots 53, 54, 57, 63, 70, 71, 78) returns HTTP 200 with canonical: true, finalized: true from the same Lighthouse instance that 404'd it.

But the indexer was surfacing every one of these as:

ERROR  failed processing head 70 (0xb9b4…): GET failed with status 404: NOT_FOUND: beacon block with root 0xb9b4…  client=cl-3-lighthouse-ethrex

which is misleading (looks operational, isn't) and frequent on multi-client devnets with multiple Lighthouse nodes.

What this PR does

• Detects the 404 in GetBlockHeaderByBlockroot using the same strings.HasPrefix(err.Error(), "GET failed with status 404") check that GetBlockHeaderBySlot and GetBlockBodyByBlockroot already use, so the lookup path matches the rest of the 404 swallows in this file.
• Wraps the 404 in a typed rpc.ErrBlockNotFound sentinel so callers can distinguish "transient race, will retry" from a genuine RPC failure.
• In processBlockEvent / processHeadEvent (indexer/beacon/client.go), demotes the log line to Debugf when the error is ErrBlockNotFound. Anything else still logs at Errorf.

Why a typed error and not bare (nil, nil)

The sibling methods return (nil, nil) directly, but their callers explicitly check for nil. EnsureHeader's contract is "non-nil header iff nil error" — returning (nil, nil) would leave the block-cache entry in a corrupt state (header channel closed with a nil header). The sentinel keeps that invariant and just lets the next block or head event for the same root re-trigger the fetch (which always succeeds, your logs prove it).

Test plan

• go build ./... is clean
• Run dora against the current kurtosis devnet with multiple Lighthouse nodes and confirm the failed processing head ...: 404 lines no longer surface at error level (they should appear at debug only)
• Confirm head/block tracking still keeps up after the change (canonical head computation complete events continue arriving in the seconds following a debug-level "not yet queryable" log)

Out of scope

GetBlockBodyByBlockroot already swallows 404 to (nil, nil), and EnsureBlock / setBlockIndex will happily store that nil — almost certainly a latent bug but pre-existing, and worth a separate PR.

[qu0b-reviewer]

🤖 qu0b-reviewer

Summary

The PR introduces a sentinel error ErrBlockNotFound from GetBlockHeaderByBlockroot and demotes it from error to debug log at the SSE event handlers, allowing the event stream to self-heal when Lighthouse's SSE and REST APIs race. The approach is sound for the intended fix, but there are some call-sites with subtle error-handling gaps worth noting.

Issues

• 🟡 indexer/beacon/requests.go:36 — LoadBeaconHeader dereferences header.Header without a nil guard. Before this PR, GetBlockHeaderByBlockroot never returned (nil, non-nil err), so this was safe. After this PR it can: the next Lighthouse race would cause a nil dereference panic. Fix: add if header == nil { return nil, ErrBlockNotFound } (or wrap with sentinel) before line 36, then update callers at client.go:570 and blockcache.go:442,477 accordingly.
• 🟡 indexer/beacon/client.go:570-571 — the parentHead == nil branch calls LoadBeaconHeader and only checks headerRsp == nil after. If the 404-path now returns an error (per the linked issue above), this caller will propagate it upward instead of falling through to the nil path. Confirmed: LoadBeaconHeader currently can't propagate ErrBlockNotFound because GetBlockHeaderByBlockroot never returned it — but once you add it (fixing the nil deref), this caller's error-gate stops working.
• 🟡 clienst/consensus/rpc/beaconapi.go — GetBlockHeaderByBlockroot now returns ErrBlockNotFound wrapped, while the adjacent GetBlockHeaderBySlot still silently returns (nil, nil) on 404. This asymmetry isn't motivated in the PR and makes it hard to reason about which callers need to handle the 404 case. Consider applying the same sentinel to GetBlockHeaderBySlot for consistency.

Suggestions

• 🟢 clients/consensus/rpc/beaconapi.go:364 — strings.HasPrefix on err.Error() is fragile; if the error string format changes upstream (in the HTTP client or eth2-client library), this silently stops working. A better check would validate the response type in the underlying provider or use a status-code-aware wrapper. Acceptable as a pragmatic workaround, but add a test or at minimum a comment listing the expected error string format.

---

_{Reviewed @ 94cbb207
"Cheaper to fly to Vietnam than to fix it here."}

[barnabasbusa]

sigp/lighthouse#9338 should fix this, and then we won't need to deal with this on dora's side.