Skip to content

Commit 873ddf8

Browse files
feat: add attestation aggregate coverage metrics (lambdaclass#386)
## Description / Motivation Ports two upstream changes that together add attestation aggregate coverage observability: - [leanSpec #735](leanEthereum/leanSpec#735): metric registration (mirrors [zeam #898](blockblaz/zeam#898)). - [zeam #876](blockblaz/zeam#876): per-slot coverage computation and emission. After this PR, all 14 coverage series receive real per-slot updates from chain activity. ## What Changed ### Registration (`crates/blockchain/src/metrics.rs`) - Two `pub const &[&str]` label-set constants — single source of truth for sections and directions. - Three `IntGaugeVec` statics: - `lean_attestation_aggregate_coverage_validators{section, subnet}` — `subnet="combined"` is the section total; `subnet="subnet_N"` is per-subnet coverage (lazy). - `lean_attestation_aggregate_coverage_subnets{section}` — covered-subnet count per section. - `lean_attestation_aggregate_coverage_diff_validators{direction}` — symmetric difference between block-included and locally-aggregated pre-merge aggregates. - `init()` forces the statics and seeds 14 default zero-valued series. **Sections:** `timely`, `late`, `block`, `combined`, `agg_start_new`, `proposal_combined`. **Directions:** `block_only`, `timely_only`. ### Emission (4 sites) Every per-slot section is keyed by the attestation `data.slot` (the voting round being reported), so `timely`/`late`/`block`/`combined` describe the **same cohort** (matching zeam #876). - **`accept_new_attestations` (blockchain/store.rs)** — before each promote, snapshots `new_payloads` participant bits, tagged **per entry with their `data.slot`**, stashed on the Store. Read at the next slot boundary to populate the `timely` section ("prev_new" in zeam). - **`on_tick` interval 1 (blockchain/lib.rs)** — emits the post-block report for `slot - 1`. Fired at **interval 1, not 0**, so the block carrying that round's votes (proposed at interval 0 of this slot) has typically been received and processed, letting the `block` section observe the same round. Computes `timely` (pre-merge snapshot filtered to the round), `late` (current `new_payloads` for the round), `block` (attestations in the **canonical head block** filtered to the round — canonical by construction, so a fork block can't poison it), and `combined` (their union); then emits `diff_validators` as the symmetric difference between `block` and `timely`. **Emits nothing** when no section has coverage for the round, so a missed slot doesn't surface as a misleading `block_only=0, timely_only=N`. - **`start_aggregation_session` (blockchain/lib.rs)** — emits `agg_start_new` from current `new_payloads` right before fork-choice aggregation at interval 2. - **`propose_block` (blockchain/lib.rs)** — emits `proposal_combined` (the full set of validators included across the block's aggregated attestations) after the block is built. (A payload-vs-gossip split was considered but dropped: ethlambda builds blocks exclusively from `known_aggregated_payloads`, so every included validator is payload-sourced by construction — `proposal_gossip` would always be 0 and `proposal_payloads` would always equal `proposal_combined`.) ### Supporting changes - The three emitters live as private free `fn`s at the bottom of `blockchain/src/lib.rs`. They use `Vec<bool>` locals (`seen` for validators, `has_subnet` for subnets) plus three small helpers (`cov_add`, `cov_record`, `or_into`). Subnet derivation is `vid % committee_count`, matching the gossip subnet assignment in `crates/net/p2p/src/lib.rs`. - `Store` gets a single `CoverageSnapshot` field (`pre_merge_new_coverage`) holding `(data.slot, AggregationBits)` entries captured before each promote — only participant bits, no proof duplication. The `block` section needs no stored state: it is read from the canonical head block at report time. - `BlockChain::spawn` now takes `attestation_committee_count` (resolved in `main.rs` from CLI > validator-config.yaml > 1; previously only known to P2P for subnet subscriptions). ## Operator interpretation of `diff_validators` The aggregation pipeline produces two pools for the same slot: `timely` (locally aggregated pre-merge) and `block` (what the proposer included). The diff counts validators in the symmetric difference: - `block_only` persistently high → this node was slow to receive/aggregate via gossip; proposer had a better view. - `timely_only` persistently high → proposer omitted attestations the network had time to gossip. - Both near zero → local aggregation tracks proposers; the network is converging. ## Correctness / Behavior Guarantees - **No fork-choice or state-transition change.** Coverage snapshots are pure observability and do not feed back into block processing. - The `diff_validators` help text **intentionally diverges** from upstream's terse phrasing to spell out the symmetric-difference semantics. Metric name, labels, and values are unchanged; dashboards built against any client's schema work as-is. - Cardinality: 14 series at registration; per-subnet series appear lazily when written. Even with full per-subnet instrumentation at 64 subnets, the validators gauge tops out at `6 × 65 = 390` series. ## Tests Added / Run No new unit tests. The earlier draft of this PR included unit tests for an internal `Coverage` bitset wrapper; once the wrapper was inlined as `Vec<bool>` locals, those tests were exercising trivial iterator ops rather than the emission contract, so they were dropped. Commands run: - `cargo fmt --all -- --check` — clean - `make lint` (clippy `-D warnings`) — clean - `cargo test -p ethlambda-blockchain --release --lib --bins` — 21 passed - `cargo test -p ethlambda-storage --release` — 38 passed - `cargo test -p ethlambda-blockchain --release --test forkchoice_spectests` — 62 passed, 8 failed (all `AttestationTooFarInFuture`, pre-existing fixture flakes on `main`, unrelated to this PR). ## Related Issues / PRs - Ports: leanEthereum/leanSpec#735 - Mirrors: blockblaz/zeam#898 + blockblaz/zeam#876 ## ✅ Verification Checklist - [x] Ran `make fmt` — clean - [x] Ran `make lint` (clippy with `-D warnings`) — clean - [x] Ran `cargo test --workspace --release` — passing modulo 8 pre-existing forkchoice fixture flakes documented above (unrelated to this PR) --------- Co-authored-by: Tomás Grüner <47506558+MegaRedHand@users.noreply.github.com>
1 parent e4aa0d1 commit 873ddf8

5 files changed

Lines changed: 373 additions & 2 deletions

File tree

bin/ethlambda/src/main.rs

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -221,7 +221,12 @@ async fn main() -> eyre::Result<()> {
221221
// and the API server (which exposes GET/POST admin endpoints).
222222
let aggregator = AggregatorController::new(options.is_aggregator);
223223

224-
let blockchain = BlockChain::spawn(store.clone(), validator_keys, aggregator.clone());
224+
let blockchain = BlockChain::spawn(
225+
store.clone(),
226+
validator_keys,
227+
aggregator.clone(),
228+
attestation_committee_count,
229+
);
225230

226231
// Note: SwarmConfig.is_aggregator is intentionally a plain bool, not the
227232
// AggregatorController — subnet subscriptions are decided once here and

crates/blockchain/src/coverage.rs

Lines changed: 194 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,194 @@
1+
//! Attestation aggregate coverage emission.
2+
//!
3+
//! Pure observability — nothing here feeds back into fork choice or the state
4+
//! transition. The emitters build `Vec<bool>` locals (`seen` for validators,
5+
//! `has_subnet` for subnets, with subnet = `vid % committee_count`, matching
6+
//! the gossip subnet assignment in `crates/net/p2p/src/lib.rs`) and push the
7+
//! resulting counts to the coverage gauges registered in
8+
//! [`crate::metrics`].
9+
10+
use ethlambda_storage::Store;
11+
use ethlambda_types::attestation::{AggregatedAttestation, AggregationBits, validator_indices};
12+
13+
use crate::metrics;
14+
15+
/// Pre-merge snapshot of `new_payloads` participant bits, used by the
16+
/// attestation aggregate coverage report.
17+
///
18+
/// Each entry is tagged with its attestation `data.slot` (the voting round) so
19+
/// the consumer can filter to a single round at emit time — `new_payloads` may
20+
/// hold entries spanning more than one slot. Holds raw participant bits; the
21+
/// consumer constructs coverage bitsets at emit time using the current
22+
/// validator and committee counts.
23+
#[derive(Debug, Clone)]
24+
pub(crate) struct CoverageSnapshot {
25+
pub(crate) entries: Vec<(u64, AggregationBits)>,
26+
}
27+
28+
/// Capture the participant bits of every entry in `new_payloads` for the
29+
/// attestation aggregate coverage report. Each entry is tagged with its
30+
/// attestation `data.slot` so the post-block report can filter to a single
31+
/// voting round (`new_payloads` may span multiple slots).
32+
///
33+
/// Returns `None` when `new_payloads` is empty so callers can keep their last
34+
/// non-empty snapshot rather than overwriting it with nothing — a node that
35+
/// missed a round still reports the round it last saw.
36+
pub(crate) fn snapshot_new_payloads(store: &Store) -> Option<CoverageSnapshot> {
37+
let entries = store.new_aggregated_payload_participants();
38+
if entries.is_empty() {
39+
return None;
40+
}
41+
Some(CoverageSnapshot { entries })
42+
}
43+
44+
fn cov_add(seen: &mut [bool], has_subnet: &mut [bool], bits: &AggregationBits) {
45+
let cc = has_subnet.len();
46+
if cc == 0 {
47+
return;
48+
}
49+
for vid in validator_indices(bits) {
50+
let vid = vid as usize;
51+
if vid < seen.len() {
52+
seen[vid] = true;
53+
has_subnet[vid % cc] = true;
54+
}
55+
}
56+
}
57+
58+
fn cov_record(section: &str, seen: &[bool], has_subnet: &[bool]) {
59+
metrics::set_attestation_aggregate_coverage_validators(
60+
section,
61+
"combined",
62+
seen.iter().filter(|&&b| b).count() as i64,
63+
);
64+
metrics::set_attestation_aggregate_coverage_subnets(
65+
section,
66+
has_subnet.iter().filter(|&&b| b).count() as i64,
67+
);
68+
}
69+
70+
fn or_into(dst: &mut [bool], src: &[bool]) {
71+
for (d, &s) in dst.iter_mut().zip(src) {
72+
*d |= s;
73+
}
74+
}
75+
76+
/// Post-block coverage report for `reporting_slot`. Emits `timely` / `late` /
77+
/// `block` / `combined` sections plus the `diff_validators` symmetric
78+
/// difference between `block` and `timely`. Called at interval 1 of the
79+
/// next slot.
80+
pub(crate) fn emit_post_block_coverage(
81+
store: &Store,
82+
pre_merge_coverage: Option<&CoverageSnapshot>,
83+
committee_count: u64,
84+
reporting_slot: u64,
85+
) {
86+
let validator_count = store.head_state().validators.len();
87+
if validator_count == 0 || committee_count == 0 {
88+
return;
89+
}
90+
let cc = committee_count as usize;
91+
let (mut timely_v, mut timely_s) = (vec![false; validator_count], vec![false; cc]);
92+
let (mut late_v, mut late_s) = (vec![false; validator_count], vec![false; cc]);
93+
let (mut block_v, mut block_s) = (vec![false; validator_count], vec![false; cc]);
94+
95+
// Every section is the same cohort: validators whose attestations *for*
96+
// `reporting_slot` (`data.slot == reporting_slot`) were seen via that
97+
// channel.
98+
99+
// `timely`: pre-merge snapshot of `new_payloads`, filtered to this round.
100+
if let Some(snap) = pre_merge_coverage {
101+
for (data_slot, bits) in &snap.entries {
102+
if *data_slot == reporting_slot {
103+
cov_add(&mut timely_v, &mut timely_s, bits);
104+
}
105+
}
106+
}
107+
// `late`: current `new_payloads` for this round (arrived after the promote).
108+
for (data_slot, bits) in store.new_aggregated_payload_participants() {
109+
if data_slot == reporting_slot {
110+
cov_add(&mut late_v, &mut late_s, &bits);
111+
}
112+
}
113+
// `block`: attestations included in the canonical head block. At interval 1
114+
// the head is normally the block proposed at `reporting_slot + 1`, which
115+
// carries this round's votes; filter by `data.slot` so we count the same
116+
// cohort even if the head is at a different slot.
117+
if let Some(block) = store.get_block(&store.head()) {
118+
for att in block.body.attestations.iter() {
119+
if att.data.slot == reporting_slot {
120+
cov_add(&mut block_v, &mut block_s, &att.aggregation_bits);
121+
}
122+
}
123+
}
124+
125+
let mut combined_v = timely_v.clone();
126+
let mut combined_s = timely_s.clone();
127+
or_into(&mut combined_v, &late_v);
128+
or_into(&mut combined_s, &late_s);
129+
or_into(&mut combined_v, &block_v);
130+
or_into(&mut combined_s, &block_s);
131+
132+
// Only report a round once the canonical head block actually carries its
133+
// votes (`block_v` non-empty). Gating on `combined` instead would still
134+
// fire on a missed slot — the `timely` snapshot for the round is populated
135+
// while `block_v` is all-false — pushing exactly the misleading
136+
// `block_only=0, timely_only=N` the diff is meant to avoid. When there is
137+
// no block for the round the gauges retain their previous value.
138+
if !block_v.iter().any(|&b| b) {
139+
return;
140+
}
141+
142+
cov_record("timely", &timely_v, &timely_s);
143+
cov_record("late", &late_v, &late_s);
144+
cov_record("block", &block_v, &block_s);
145+
cov_record("combined", &combined_v, &combined_s);
146+
147+
let (block_only, timely_only) =
148+
block_v
149+
.iter()
150+
.zip(timely_v.iter())
151+
.fold((0i64, 0i64), |(b, t), (bv, tv)| match (bv, tv) {
152+
(true, false) => (b + 1, t),
153+
(false, true) => (b, t + 1),
154+
_ => (b, t),
155+
});
156+
metrics::set_attestation_aggregate_coverage_diff_validators("block_only", block_only);
157+
metrics::set_attestation_aggregate_coverage_diff_validators("timely_only", timely_only);
158+
}
159+
160+
/// `agg_start_new` coverage from `new_payloads`, called right before fork-
161+
/// choice aggregation runs at interval 2.
162+
pub(crate) fn emit_agg_start_new_coverage(store: &Store, committee_count: u64) {
163+
let validator_count = store.head_state().validators.len();
164+
if validator_count == 0 || committee_count == 0 {
165+
return;
166+
}
167+
let cc = committee_count as usize;
168+
let mut seen = vec![false; validator_count];
169+
let mut has_subnet = vec![false; cc];
170+
for (_slot, bits) in store.new_aggregated_payload_participants() {
171+
cov_add(&mut seen, &mut has_subnet, &bits);
172+
}
173+
cov_record("agg_start_new", &seen, &has_subnet);
174+
}
175+
176+
/// `proposal_combined` coverage for a block we are about to publish: the full
177+
/// set of validators included across the block's aggregated attestations.
178+
pub(crate) fn emit_proposal_coverage<'a>(
179+
store: &Store,
180+
committee_count: u64,
181+
selected: impl IntoIterator<Item = &'a AggregatedAttestation>,
182+
) {
183+
let validator_count = store.head_state().validators.len();
184+
if validator_count == 0 || committee_count == 0 {
185+
return;
186+
}
187+
let cc = committee_count as usize;
188+
let mut combined_v = vec![false; validator_count];
189+
let mut combined_s = vec![false; cc];
190+
for att in selected {
191+
cov_add(&mut combined_v, &mut combined_s, &att.aggregation_bits);
192+
}
193+
cov_record("proposal_combined", &combined_v, &combined_s);
194+
}

crates/blockchain/src/lib.rs

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ use crate::store::StoreError;
2828

2929
pub mod aggregation;
3030
pub mod block_builder;
31+
pub(crate) mod coverage;
3132
pub(crate) mod fork_choice_tree;
3233
pub mod key_manager;
3334
pub mod metrics;
@@ -70,6 +71,7 @@ impl BlockChain {
7071
store: Store,
7172
validator_keys: HashMap<u64, ValidatorKeyPair>,
7273
aggregator: AggregatorController,
74+
attestation_committee_count: u64,
7375
) -> BlockChain {
7476
metrics::set_is_aggregator(aggregator.is_enabled());
7577
metrics::set_node_sync_status(metrics::SyncStatus::Idle);
@@ -96,6 +98,8 @@ impl BlockChain {
9698
pending_block_parents: HashMap::new(),
9799
current_aggregation: None,
98100
last_tick_instant: None,
101+
attestation_committee_count,
102+
pre_merge_coverage: None,
99103
}
100104
.start();
101105
let time_until_genesis = (SystemTime::UNIX_EPOCH + Duration::from_secs(genesis_time))
@@ -149,6 +153,17 @@ pub struct BlockChainServer {
149153

150154
/// Last tick instant for measuring interval duration.
151155
last_tick_instant: Option<Instant>,
156+
157+
/// Number of attestation committees (= subnet count). Used by the
158+
/// attestation aggregate coverage emission.
159+
attestation_committee_count: u64,
160+
161+
/// Pre-merge `new_payloads` snapshot for the attestation aggregate coverage
162+
/// report. Captured at the end-of-slot promote (interval 4), read at the
163+
/// next slot boundary. Owned solely by the actor and only touched from the
164+
/// single-threaded message loop, so no synchronization is needed.
165+
/// Observability-only.
166+
pre_merge_coverage: Option<coverage::CoverageSnapshot>,
152167
}
153168

154169
impl BlockChainServer {
@@ -190,6 +205,23 @@ impl BlockChainServer {
190205
.then(|| self.get_our_proposer(slot))
191206
.flatten();
192207

208+
// Snapshot the pre-merge `new_payloads` set at the end-of-slot promote
209+
// (interval 4), so the post-block report for this round sees its
210+
// "timely" cohort just before it is promoted out of `new_payloads`.
211+
//
212+
// Only interval 4 — not the proposer's interval-0 promote. By interval 0
213+
// the round's votes have already been promoted at the previous slot's
214+
// interval 4; `new_payloads` then holds only stragglers, and snapshotting
215+
// them here would overwrite the good interval-4 snapshot the report still
216+
// needs (those stragglers surface in the `late` section instead). Skip
217+
// empty snapshots so a missed round keeps the last set we saw. Pure
218+
// observability.
219+
if interval == 4
220+
&& let Some(snapshot) = coverage::snapshot_new_payloads(&self.store)
221+
{
222+
self.pre_merge_coverage = Some(snapshot);
223+
}
224+
193225
// Tick the store first - this accepts attestations at interval 0 if we have a proposal
194226
store::on_tick(
195227
&mut self.store,
@@ -198,6 +230,7 @@ impl BlockChainServer {
198230
);
199231

200232
if interval == 2 && is_aggregator {
233+
coverage::emit_agg_start_new_coverage(&self.store, self.attestation_committee_count);
201234
self.start_aggregation_session(slot, ctx).await;
202235
}
203236

@@ -210,6 +243,18 @@ impl BlockChainServer {
210243
// Reuse the same snapshot so self-delivery decisions match the rest
211244
// of the tick.
212245
if interval == 1 {
246+
// Emit the post-block coverage report for the previous slot. Fired
247+
// at interval 1 (not 0) so the block carrying `slot - 1`'s votes —
248+
// proposed at interval 0 of this slot — has typically been received
249+
// and processed, letting the `block` section see the same round.
250+
if slot > 0 {
251+
coverage::emit_post_block_coverage(
252+
&self.store,
253+
self.pre_merge_coverage.as_ref(),
254+
self.attestation_committee_count,
255+
slot - 1,
256+
);
257+
}
213258
self.produce_attestations(slot, is_aggregator);
214259
}
215260

@@ -351,6 +396,12 @@ impl BlockChainServer {
351396
return;
352397
};
353398

399+
coverage::emit_proposal_coverage(
400+
&self.store,
401+
self.attestation_committee_count,
402+
block.body.attestations.iter(),
403+
);
404+
354405
// Sign the block root with the proposal key
355406
let block_root = block.hash_tree_root();
356407
let Ok(proposer_signature) = self

0 commit comments

Comments
 (0)