Skip to content

Commit 0a41b8b

Browse files
mrq1911mrq1911
authored
Merge pull request #1470 from galacticcouncil/feat/coretime-renew-alert
feat: coretime renewal watchdog
2 parents 11f685b + 47c406f commit 0a41b8b

7 files changed

Lines changed: 553 additions & 0 deletions

File tree

‎scripts/coretime-alert/.dockerignore‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
node_modules
2+
.state.json
3+
*.log
4+
README.md
5+
lark-stack.yml
6+
.gitignore
7+
Dockerfile
8+
.dockerignore
9+
package-lock.json

‎scripts/coretime-alert/.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
node_modules/
2+
.state.json
3+
*.log

‎scripts/coretime-alert/Dockerfile‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
FROM node:22-alpine
2+
3+
WORKDIR /app
4+
5+
COPY package.json ./
6+
RUN npm install --omit=dev --no-audit --no-fund && npm cache clean --force
7+
8+
COPY check.mjs ./
9+
10+
ENV STATE_FILE=/app/state/.state.json \
11+
ALERT_COOLDOWN_HOURS=12 \
12+
LEADIN_ALERT_DAYS=3 \
13+
TOTAL_ALERT_DAYS=7 \
14+
CHECK_INTERVAL_SECONDS=3600
15+
16+
VOLUME ["/app/state"]
17+
18+
# Run the check, then sleep for the configured interval, forever.
19+
CMD ["sh", "-c", "while true; do node check.mjs; sleep ${CHECK_INTERVAL_SECONDS:-3600}; done"]

‎scripts/coretime-alert/README.md‎

Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
# coretime renewal watchdog
2+
3+
Alerts a Discord webhook when Hydration's (Polkadot, task `2034`) or Basilisk's
4+
(Kusama, task `2090`) bulk coretime cores are at risk of lapsing in the current
5+
sale.
6+
7+
Bulk cores are **not leases** — they must be renewed every ~28-day sale cycle or
8+
they fall back to the open market. This watchdog watches the `broker` pallet on
9+
each coretime chain and pings before the renewal window closes.
10+
11+
## what triggers an alert
12+
13+
An alert fires for a chain only when it is **short of its target core count for
14+
the upcoming region** (i.e. some cores are still un-renewed) **and** the deadline
15+
is near:
16+
17+
| condition | default | severity |
18+
|-----------|---------|----------|
19+
| `< TOTAL_ALERT_DAYS` until the region begins (hard deadline — renewal right is lost after) | 7 days | `URGENT` |
20+
| inside the lead-in period with `< LEADIN_ALERT_DAYS` left in it | 3 days | `WARNING` |
21+
22+
"Short" = `secured cores for next region (workplan) < desired`. The alert lists the
23+
pending `broker.renew(core)` calls (with encoded hex) needed to close the gap.
24+
25+
> Note: the cheapest, priority window to renew is actually the **interlude**
26+
> (before lead-in). These thresholds, as requested, only warn once the lead-in is
27+
> closing / the hard deadline is near — so treat an alert as "renew now, you're
28+
> past the ideal window". Lower the thresholds if you want earlier warnings.
29+
30+
## setup
31+
32+
```sh
33+
cd scripts/coretime-alert
34+
npm install
35+
export DISCORD_WEBHOOK_URL='https://discord.com/api/webhooks/XXX/YYY'
36+
node check.mjs # one-shot check; alerts if within thresholds
37+
```
38+
39+
### useful invocations
40+
41+
```sh
42+
node check.mjs --dry-run # print the payload instead of POSTing (no webhook needed)
43+
node check.mjs --test # POST a "webhook reachable" message and exit
44+
node check.mjs --force # ignore the cooldown and (re)send any active alert
45+
```
46+
47+
## configuration (env vars)
48+
49+
| var | default | meaning |
50+
|-----|---------|---------|
51+
| `DISCORD_WEBHOOK_URL` | — | **required** (unless `--dry-run`) |
52+
| `DISCORD_WEBHOOK_URL_FILE` | — | alternative source: read the webhook from a file (e.g. a mounted secret) |
53+
| `CHECK_INTERVAL_SECONDS` | `3600` | (docker image only) seconds between checks in the built-in loop |
54+
| `LEADIN_ALERT_DAYS` | `3` | warn when this many days are left in the lead-in period |
55+
| `TOTAL_ALERT_DAYS` | `7` | warn when this many days are left until the region begins |
56+
| `ALERT_COOLDOWN_HOURS` | `12` | minimum gap between repeat alerts for the same condition |
57+
| `HYDRATION_DESIRED_CORES` | `3` | target core count for task 2034 |
58+
| `BASILISK_DESIRED_CORES` | `3` | target core count for task 2090 |
59+
| `STATE_FILE` | `./.state.json` | where the throttle state is persisted |
60+
61+
State persists across runs so a standing condition only re-pings every
62+
`ALERT_COOLDOWN_HOURS`; it re-alerts immediately if severity or shortfall changes,
63+
and clears itself once the cores are renewed.
64+
65+
## running it as a service
66+
67+
It's a one-shot check — schedule it. Hourly is plenty (the relevant deadlines move
68+
in days).
69+
70+
### cron
71+
72+
```cron
73+
0 * * * * cd /path/to/hydration-node/scripts/coretime-alert && DISCORD_WEBHOOK_URL='https://discord.com/api/webhooks/XXX/YYY' /usr/bin/node check.mjs >> /var/log/coretime-alert.log 2>&1
74+
```
75+
76+
### systemd timer
77+
78+
`/etc/systemd/system/coretime-alert.service`
79+
80+
```ini
81+
[Unit]
82+
Description=Coretime renewal watchdog
83+
After=network-online.target
84+
85+
[Service]
86+
Type=oneshot
87+
WorkingDirectory=/path/to/hydration-node/scripts/coretime-alert
88+
Environment=DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/XXX/YYY
89+
ExecStart=/usr/bin/node check.mjs
90+
```
91+
92+
`/etc/systemd/system/coretime-alert.timer`
93+
94+
```ini
95+
[Unit]
96+
Description=Run coretime renewal watchdog hourly
97+
98+
[Timer]
99+
OnCalendar=hourly
100+
Persistent=true
101+
102+
[Install]
103+
WantedBy=timers.target
104+
```
105+
106+
```sh
107+
systemctl enable --now coretime-alert.timer
108+
```
109+
110+
## docker image / lark deployment
111+
112+
A `Dockerfile` builds `galacticcouncil/coretime-alert`; the image runs the check
113+
on an hourly loop (`CHECK_INTERVAL_SECONDS`).
114+
115+
```sh
116+
docker build -t galacticcouncil/coretime-alert:latest .
117+
docker push galacticcouncil/coretime-alert:latest
118+
# smoke test: one-shot dry-run inside the image
119+
docker run --rm galacticcouncil/coretime-alert:latest node check.mjs --dry-run
120+
```
121+
122+
On the **lark** Swarm it's deployed as the `coretime-alert` stack from
123+
`lark-stack.yml` (single replica, `state` volume, `autoredeploy` on). The webhook
124+
is injected as `DISCORD_WEBHOOK_URL` via `$env` at deploy time and is never stored
125+
in the repo — see the safe-forwarding options below.
126+
127+
### forwarding the webhook safely
128+
129+
`$env:DISCORD_WEBHOOK_URL` resolves against the **swarmpit MCP server's**
130+
environment (defined in `~/.claude.json`), not your interactive shell. Pick one:
131+
132+
- **local deploy file** — copy `lark-stack.yml` outside the repo with the real URL
133+
inlined (`chmod 600`), and deploy that file. Value never enters the repo or chat.
134+
- **MCP server env** — add `DISCORD_WEBHOOK_URL` to the `swarmpit-lark` `env` block
135+
in `~/.claude.json`, then restart so the server inherits it.
136+
137+
Do **not** paste the URL into a chat/agent prompt — it ends up in transcripts.
138+
Rotate the webhook in Discord if it ever leaks.
139+
140+
## notes
141+
142+
- Endpoints have built-in fallbacks per chain; a failed assessment posts a (throttled)
143+
`🔌 check failed` notice so the watchdog never dies silently.
144+
- Renewed cores get **new core indices** each cycle (the assignment to the task is
145+
preserved), so the listed core numbers change after every renewal — expected.
146+
- Adding another para: append an entry to `CHAINS` in `check.mjs` with its task id,
147+
relay, and coretime endpoints.

0 commit comments

Comments
 (0)