Merge pull request #1 from lambdaclass/dora-monitor

Add dora_monitor: Slack alerting tool for ethrex devnet

Author: ilitteri

dora_monitor/.gitignore

@@ -0,0 +1,25 @@
+# Virtual env
+.venv/
+
+# Python bytecode / build
+__pycache__/
+*.py[cod]
+*.egg-info/
+*.egg
+build/
+dist/
+
+# Tooling caches
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.tox/
+.coverage
+coverage.xml
+htmlcov/
+.python-version
+
+# Local config / runtime state
+config.yaml
+dora_monitor_state.json
+*.log

dora_monitor/Makefile

@@ -0,0 +1,23 @@
+.PHONY: all deps run dry-run dry-run-once clean
+
+CONFIG ?= config.yaml
+
+all: deps
+
+.venv:
+	uv venv
+	uv pip install -e .
+
+deps: .venv
+
+run: deps
+	uv run dora-monitor -c $(CONFIG)
+
+dry-run: deps
+	uv run dora-monitor -c $(CONFIG) --dry-run --debug
+
+dry-run-once: deps
+	uv run dora-monitor -c $(CONFIG) --once --dry-run --debug
+
+clean:
+	rm -rf .venv *.egg-info build dist dora_monitor_state.json

dora_monitor/README.md

@@ -0,0 +1,55 @@
+# dora_monitor
+
+Polls a [Dora explorer](https://github.com/ethpandaops/dora) API and posts Slack and/or Discord alerts when a specific client (e.g. `ethrex`) misses a block, orphans a block, drifts onto a fork, falls behind the canonical head, or drops offline.
+
+## Install
+
+```bash
+cd dora_monitor
+python -m venv .venv && source .venv/bin/activate
+pip install -e .
+```
+
+## Configure
+
+Copy `config.example.yaml` and edit it; the mandatory fields are `dora_url`, `client_match`, and at least one webhook (`slack_webhook_url` or `discord_webhook_url`; either can come from the `SLACK_WEBHOOK_URL` / `DISCORD_WEBHOOK_URL` env var instead). Both may be set, in which case every alert and heartbeat is fanned out to each.
+
+```bash
+cp config.example.yaml config.yaml
+$EDITOR config.yaml
+```
+
+`client_match` is a case-insensitive substring matched against Dora's `proposer_name` and client `name` fields (e.g. `lighthouse-ethrex-1`), so `ethrex` catches every CL/EL pair that runs the ethrex execution layer.
+
+## Run
+
+```bash
+# foreground
+dora-monitor -c config.yaml
+
+# one tick, no Slack
+SLACK_WEBHOOK_URL=unused dora-monitor -c config.yaml --once --dry-run
+```
+
+The process holds dedup state in `state_file` so restarts don't re-alert on already-reported missed slots / open conditions.
+
+## Alerts
+
+| Trigger | Source endpoint |
+|---|---|
+| Missed slot by `client_match` proposer | `/api/v1/slots?with_missing=1` |
+| Orphaned block by `client_match` proposer | `/api/v1/slots?with_orphaned=1` |
+| `client_match` node on a non-canonical fork | `/api/v1/network/client_head_forks` |
+| `client_match` node lagging `>= sync_lag_threshold` slots | `/api/v1/network/client_head_forks` |
+| `client_match` node `status != online` | `/api/v1/network/client_head_forks` |
+| `client_match` EL `version` string changes (deploy/rollback) | scraped from `/clients/execution` HTML |
+
+Recoveries (fork resolved, caught up, back online) are posted as well.
+
+The periodic heartbeat digest uses Slack Block Kit (`{"blocks": [...]}`) on Slack and a rich embed on Discord, both built from the same gathered data plus a shared plain-text fallback. Action alerts (offline / fork / lag / version change / missed-block) use plain mrkdwn `text` posts on Slack; the same strings are run through a mrkdwn→Discord-markdown translator (single-asterisk bold → double, `:shortcode:` emoji → unicode) before being posted to Discord. Clients with status `online`, on the canonical fork, and at `distance == 0` from canonical head collapse into a single "online @ canonical" bucket so the digest highlights outliers instead of repeating identical rows; use `heartbeat_other_clients: detailed` (default) to list the healthy names, `summary` for just a count, or `off` to drop the section entirely.
+
+## A note on what "client" means here
+
+`/api/v1/network/client_head_forks` lists Dora's **beacon (CL)** clients; their names embed the paired EL (e.g. `lighthouse-ethrex-1` is the Lighthouse beacon paired with an ethrex EL). So the offline / fork / lag signals are observed on the beacon side. An ethrex-EL crash shows up indirectly: the paired beacon's head stops advancing (sync_lag) or its status flips to non-online (offline).
+
+Dora's `/api/v1/clients/execution` is deliberately NOT used. Its `status` field only reflects whether Dora's devp2p crawler could fetch `admin_nodeInfo` from the node, not whether the EL is healthy; ethrex EL nodes typically show `disconnected` there even when the UI shows them as `Ready` and following the chain. The execution clients page's real status (`Ready`/`Synchronizing`/`Offline`) is not exposed via any JSON API as of Dora master.

dora_monitor/config.example.yaml

@@ -0,0 +1,74 @@
+# Dora explorer base URL (no trailing slash).
+dora_url: "https://dora.bal-devnet-7.ethpandaops.io"
+
+# Substring (case-insensitive) matched against proposer_name / client name
+# in the Dora API. Any name containing this is considered "ours".
+client_match: "ethrex"
+
+# Slack incoming webhook URL. Can also be set via SLACK_WEBHOOK_URL env var.
+# Leave empty to disable Slack delivery.
+slack_webhook_url: ""
+
+# Discord webhook URL. Can also be set via DISCORD_WEBHOOK_URL env var.
+# Leave empty to disable Discord delivery. At least one of slack / discord
+# must be configured (unless --dry-run).
+discord_webhook_url: ""
+
+# Optional label shown in alert headers (e.g. network name).
+network_label: "bal-devnet-7"
+
+# Poll interval in seconds.
+poll_interval: 30
+
+# How many recent slots to scan each tick for missed/orphaned blocks.
+slot_scan_limit: 64
+
+# Sync lag threshold (slots behind the canonical head) before alerting.
+sync_lag_threshold: 16
+
+# Number of consecutive polls a matched client must be on a non-canonical
+# head before we fire a fork alert. Filters propagation-timing jitter
+# (1-2 slot leads/lags that self-resolve in the next poll). With
+# poll_interval=30, fork_confirm_ticks=3 = ~90s of confirmation.
+fork_confirm_ticks: 3
+
+# Path for persisted dedup state. Use null for in-memory only.
+state_file: "./dora_monitor_state.json"
+
+# HTTP timeout in seconds.
+http_timeout: 10
+
+# Set true for verbose logging.
+debug: false
+
+# Periodic digest ("heartbeat") posted to Slack to confirm the monitor
+# is alive and give a network snapshot. Set 0 to disable.
+heartbeat_interval_minutes: 360
+# Slots scanned for missed/orphaned counts in each heartbeat.
+heartbeat_slot_window: 256
+# How to include non-`client_match` clients in the heartbeat:
+#   off       — skip them entirely
+#   summary   — one-line aggregate ("12 total, 1 non-online, 0 off-canonical")
+#   detailed  — per-client list with status / head / distance
+heartbeat_other_clients: "detailed"
+
+# Missed-block storm mode. When a single proposer hits
+# `missed_burst_threshold` misses inside the last `missed_burst_window_minutes`,
+# we stop posting per-slot alerts and switch to one "burst started" alert
+# plus periodic "still bursting" updates whose interval backs off through
+# `missed_burst_update_schedule_minutes` (last value repeats indefinitely).
+# A "burst resolved" alert fires once the window has been clear of misses.
+missed_burst_threshold: 5
+missed_burst_window_minutes: 15
+missed_burst_update_schedule_minutes: [15, 30, 60, 120]
+
+# Which checks to enable.
+checks:
+  missed_blocks: true
+  forks: true
+  offline: true
+  sync_lag: true
+  # Posts an alert when an ethrex EL's reported version string changes
+  # between polls (deploy / rollback detection). Reads from the HTML page
+  # since the v1 JSON API doesn't expose ethrex's EL version.
+  version_drift: true

dora_monitor/dora_monitor/__init__.py

@@ -0,0 +1,4 @@
+from dora_monitor.main import cli
+
+if __name__ == "__main__":
+    cli()