Skip to content

Commit 41cc0b1

Browse files
rsantacrocersantacroce
committed
Add OPERATOR_GUIDE.md — running the pps-classic pool day-to-day
Section-by-section runbook for the pool operator: - Quick reference (URLs, admin credentials, SSH info) - Where things live on the box (dirs, services, config) - Value flow diagram (miner share → BTC coinbase → deposit → Thunder) - Daily operations: health check via admin dashboard, deposit runbook (grpcurl commands for CreateDepositTransaction + GetCtip check), watching payouts land, stuck-payout reconciliation (both crash cases with the exact SQL) - Onboarding a new miner (stratum URL, Thunder-address usernames, rig labels) - Rotating the admin password - Rolling back to solo mode - Troubleshooting: reject storm, permanent 'reserve short', stale DB path, ':3334 in use' (with the pkill -x warning about not matching your own SSH session), payouts never firing - Open items honestly listed: pool_btc_address should point at an enforcer-wallet-owned address for the deposit flow to be self-contained, BMM/Thunder sidechain block production not completing yet, admin 'Deposit to Thunder' form not yet built
1 parent c900dc9 commit 41cc0b1

1 file changed

Lines changed: 347 additions & 0 deletions

File tree

‎OPERATOR_GUIDE.md‎

Lines changed: 347 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,347 @@
1+
# Pool operator guide (pps-thunder-classic)
2+
3+
Everything you need to run this pool day-to-day. Assumes the branch
4+
already deployed (see `PPS_THUNDER.md` and `CLASSIC_PAYOUTS.md` for
5+
background on why the design looks like this).
6+
7+
---
8+
9+
## Quick reference
10+
11+
| what | where | credentials |
12+
| --- | --- | --- |
13+
| Public dashboard | http://45.33.100.24:8081/ | none |
14+
| **Admin dashboard** | **http://45.33.100.24:8081/admin** | **`admin` / `4DqSK1d6jkefposLEw5DEP5luPT`** |
15+
| Admin JSON API | http://45.33.100.24:8081/api/admin/summary | same basic auth |
16+
| Stratum (miner endpoint) | `stratum+tcp://45.33.100.24:3334` | username = Thunder base58 address |
17+
| SSH | `root@45.33.100.24` | `~/.ssh/id_epitetus` |
18+
19+
The admin password is also stashed at `/root/simplepool-admin-cred.txt`
20+
on the box (root-only). To rotate, edit
21+
`/etc/systemd/system/simplepool-dashboard.service.d/pps-thunder.conf`
22+
and `systemctl daemon-reload && systemctl restart simplepool-dashboard`.
23+
24+
---
25+
26+
## Where things live on the box
27+
28+
- **Root**: `/home/forknet/pps-thunder-test/`
29+
- `simplepool/` — this repo, checked out at `pps-thunder-classic`
30+
- `data/shares.db` — pool state (SQLite, WAL mode)
31+
- `data/archive/` — old DB snapshots from mode switches
32+
- `logs/` — pool + payout + Thunder logs (plus stashed Thunder mnemonic)
33+
- `thunder-data/` — Thunder node chain data
34+
35+
- **Services** (all `systemctl status <name>`):
36+
- `simplepool.service` — the stratum proxy (managed manually right now;
37+
launched via `nohup` from the deploy). Systemd unit exists but not
38+
always enabled.
39+
- `simplepool-payout.service` — Node payout worker
40+
- `simplepool-dashboard.service` — Node public + admin dashboards
41+
42+
- **Config**: `/home/forknet/pps-thunder-test/simplepool/proxy.conf`
43+
(gitignored). Current `pool_mode = pps-classic`.
44+
45+
---
46+
47+
## The value flow, end to end
48+
49+
```
50+
┌──────────┐ diff × 1000 sats
51+
│ miner │─────────────────────────┐
52+
│ ASIC │ ▼
53+
└────┬─────┘ ┌────────────────┐
54+
│ stratum share │ pps_credits │
55+
│ │ accrued_sats │
56+
▼ └────────┬───────┘
57+
┌───────────┐ block payout worker
58+
│ simplepool│ found ↓ every 30s
59+
│ :3334 │──────►coinbase ──┘ IF Thunder reserve has funds
60+
└───────────┘ │
61+
│ pays pool_btc_address (BTC P2WPKH)
62+
│
63+
▼
64+
┌───────────────────┐
65+
│ pool BTC wallet │ accumulates BTC over blocks
66+
│ │
67+
└────────┬──────────┘
68+
│ operator triggers deposit
69+
│ (admin dashboard button OR grpcurl)
70+
▼
71+
┌──────────────────────┐
72+
│ BIP300 deposit tx │
73+
│ CreateDepositTx │ mainchain → sidechain
74+
└────────┬─────────────┘
75+
│ Ctip updates on enforcer
76+
▼
77+
┌──────────────────────┐
78+
│ Thunder wallet │
79+
│ available_sats > 0 │
80+
└────────┬─────────────┘
81+
│ payout worker's next tick
82+
│ thunder.transfer(miner_addr, owed, fee)
83+
▼
84+
┌──────────────────────┐
85+
│ miner Thunder wallet │
86+
└──────────────────────┘
87+
```
88+
89+
Everything before `pool_btc_address` runs automatically. Everything
90+
after is either automatic (payout worker) or one operator action
91+
per deposit (moving BTC into Thunder).
92+
93+
---
94+
95+
## Daily operations
96+
97+
### 1. Health check (30 seconds)
98+
99+
Open the admin dashboard. Look at the top three cards:
100+
101+
- **Thunder reserve** — how much you can pay out right now
102+
- **PPS ledger totals** — how much you *owe* (owed = accrued − paid)
103+
- **⚠ Reserve is short** banner — appears when owed > available; means
104+
the payout worker will keep skipping ticks until you deposit more BTC
105+
106+
If you see a `⚠` banner AND owed is significant, do a deposit (below).
107+
108+
Also check the **In-flight payouts** card — should be empty. Any row
109+
with a set `txid` means a payout crashed mid-flight and needs manual
110+
reconciliation.
111+
112+
### 2. Deposit BTC into Thunder (when reserve is short)
113+
114+
**Currently manual — no admin button yet.** The runbook:
115+
116+
```sh
117+
ssh -i ~/.ssh/id_epitetus root@45.33.100.24
118+
119+
# 1. Get a Thunder-owned deposit address (formatted for the mainchain wallet)
120+
TCLI=/home/forknet/forknet-software/thunder-rust/target/debug/thunder_app_cli
121+
POOL_ADDR=$(sudo -u forknet $TCLI get-new-address)
122+
DEP=$(sudo -u forknet $TCLI format-deposit-address "$POOL_ADDR")
123+
echo "deposit target: $DEP"
124+
125+
# 2. Trigger the deposit via the enforcer's wallet. AMOUNT + FEE are yours to pick.
126+
GRPCURL=/home/forknet/forknet-software/grpcurl
127+
AMOUNT_SATS=1000000 # 0.01 BTC = 1_000_000 sats
128+
FEE_SATS=1000
129+
$GRPCURL -plaintext \
130+
-d "{\"sidechain_id\":9,\"address\":\"$DEP\",\"value_sats\":$AMOUNT_SATS,\"fee_sats\":$FEE_SATS}" \
131+
127.0.0.1:50051 cusf.mainchain.v1.WalletService/CreateDepositTransaction
132+
133+
# 3. Wait for a mainchain block. If you want to force one on the test:
134+
$GRPCURL -plaintext -d '{"blocks":1}' \
135+
127.0.0.1:50051 cusf.mainchain.v1.WalletService/GenerateBlocks
136+
137+
# 4. Confirm the Ctip moved
138+
$GRPCURL -plaintext -d '{"sidechain_number":9}' \
139+
127.0.0.1:50051 cusf.mainchain.v1.ValidatorService/GetCtip
140+
```
141+
142+
**About the "enforcer's wallet"**: the enforcer runs its own on-box BTC
143+
wallet, seeded during the setup phase. Its balance is visible via
144+
`WalletService/GetBalance`. Right now it's ~1 BTC because we mined some
145+
funding blocks during activation. **Long-term**, you want the pool
146+
coinbase to accumulate straight into an enforcer-wallet-owned address —
147+
that means changing `pool_btc_address` in `proxy.conf` to an address
148+
generated by `WalletService/CreateNewAddress` on the enforcer. Until
149+
that's done, the operator's own BTC (at the current `pool_btc_address`)
150+
sits in an external wallet the enforcer can't spend directly.
151+
152+
### 3. Watch payouts land
153+
154+
After a deposit completes:
155+
156+
```sh
157+
# tail the payout worker
158+
sudo journalctl -u simplepool-payout.service -f
159+
# or read the log directly:
160+
tail -f /home/forknet/pps-thunder-test/logs/payout.log
161+
```
162+
163+
You should see lines like:
164+
```
165+
payout: 3Z6z1hPySN…rig1 -> 3Z6z1hPySN… 96085 sats txid=…
166+
```
167+
168+
Refresh the admin dashboard — `owed` should drop, `paid` should rise,
169+
`in-flight` should stay empty.
170+
171+
**Known Thunder-side gotcha:** for `payout: reserve short — available=0`
172+
to actually go away, Thunder itself has to mine a sidechain block (BMM)
173+
that recognizes the deposit. Deposits are always visible on the
174+
enforcer's Ctip, but Thunder's own wallet balance stays 0 until BMM
175+
runs. On this box Thunder's `mine` RPC is currently blocking without
176+
producing a sidechain block — likely a BMM-side infrastructure issue
177+
that needs investigation. In the meantime, the pool correctly logs the
178+
short-reserve state and does not double-pay or lose accrual data.
179+
180+
### 4. Reconciling stuck payouts
181+
182+
If the admin dashboard's **In-flight payouts** card is non-empty, each
183+
row means a payout crashed between broadcast and DB finalize. Two cases:
184+
185+
- **`txid = —`** (no txid): the broadcast never happened. Safe to just
186+
delete the row and let the worker retry next tick:
187+
```sh
188+
sudo -u forknet sqlite3 /home/forknet/pps-thunder-test/data/shares.db \
189+
"DELETE FROM payouts_in_flight WHERE id = <ID>;"
190+
```
191+
192+
- **`txid = <something>`**: the Thunder tx went out; the DB finalize
193+
crashed. Verify the tx is on Thunder (via `thunder_app_cli
194+
get-transaction <txid>` or a Thunder wallet viewer). If confirmed,
195+
finalize by hand:
196+
```sh
197+
sudo -u forknet sqlite3 /home/forknet/pps-thunder-test/data/shares.db "
198+
BEGIN;
199+
UPDATE pps_credits SET paid_sats = paid_sats + (SELECT sats FROM payouts_in_flight WHERE id = <ID>)
200+
WHERE worker_id = (SELECT worker_id FROM payouts_in_flight WHERE id = <ID>);
201+
DELETE FROM payouts_in_flight WHERE id = <ID>;
202+
COMMIT;
203+
"
204+
```
205+
206+
The pool never auto-resolves in-flight rows because it can't safely
207+
tell apart "broadcast didn't happen" from "broadcast happened,
208+
finalize crashed" without Thunder-side mempool/chain lookup.
209+
210+
---
211+
212+
## Onboarding a new miner
213+
214+
Nothing on the pool side. The miner points their ASIC firmware at:
215+
216+
```
217+
stratum+tcp://45.33.100.24:3334
218+
username: <their-Thunder-address>[.<rig_label>]
219+
password: (anything)
220+
```
221+
222+
The username **must** be a valid Thunder address — base58 of a 20-byte
223+
hash. If they don't have a Thunder wallet, tell them to run a Thunder
224+
node and do `thunder-cli get-new-address`.
225+
226+
A Bitcoin address as the username produces `invalid thunder address:
227+
thunder address decoded length 25 (expected 20)` on authorize and no
228+
shares accrue. Rejects show up in the admin dashboard's implicit
229+
`rejects` table (surfaced on the per-worker page) so you can help
230+
someone debug their config.
231+
232+
**Same address, multiple rigs:** append `.<rig_label>` (e.g.
233+
`3Z6z1hPySN….basement`, `3Z6z1hPySN….garage`). They show as separate
234+
rows on the workers page but the same payout address on the admin
235+
view.
236+
237+
---
238+
239+
## Rotating the admin password
240+
241+
```sh
242+
ssh -i ~/.ssh/id_epitetus root@45.33.100.24
243+
NEW_PASS=$(openssl rand -base64 21 | tr -d '=+/')
244+
sed -i "s/^Environment=ADMIN_PASSWORD=.*/Environment=ADMIN_PASSWORD=$NEW_PASS/" \
245+
/etc/systemd/system/simplepool-dashboard.service.d/pps-thunder.conf
246+
echo "admin / $NEW_PASS" > /root/simplepool-admin-cred.txt
247+
systemctl daemon-reload
248+
systemctl restart simplepool-dashboard.service
249+
cat /root/simplepool-admin-cred.txt
250+
```
251+
252+
Old password stops working immediately; the dashboard is up in ~1s.
253+
254+
---
255+
256+
## Rolling back / switching modes
257+
258+
To pause and restore the old solo pool exactly as it was:
259+
260+
```sh
261+
# stop pps-classic pool
262+
pkill -9 -x simplepool
263+
# start the pre-existing solo unit (still installed)
264+
systemctl start simplepool.service
265+
```
266+
267+
That points at the OLD proxy.conf at `~/forknet-software/simplepool/`.
268+
The dashboard drop-in stays pointed at the pps-classic DB by default;
269+
edit `/etc/systemd/system/simplepool-dashboard.service.d/pps-thunder.conf`
270+
if you want to swing it back to the solo DB.
271+
272+
---
273+
274+
## Troubleshooting
275+
276+
### "invalid thunder address" reject storm
277+
278+
A miner is using a BTC address. Have them update their firmware config
279+
to use a Thunder address. Until they do, they burn hashrate for zero
280+
credit — the pool has no fallback.
281+
282+
### Payout worker log shows `reserve short — available=0` forever
283+
284+
Either (a) you haven't done a deposit yet (see runbook above), or (b)
285+
you deposited but Thunder hasn't seen the credit yet due to BMM not
286+
running. Check with:
287+
288+
```sh
289+
# enforcer says the deposit was consensus-accepted?
290+
grpcurl -plaintext -d '{"sidechain_number":9}' \
291+
127.0.0.1:50051 cusf.mainchain.v1.ValidatorService/GetCtip
292+
# Thunder wallet sees the balance?
293+
sudo -u forknet /home/forknet/forknet-software/thunder-rust/target/debug/thunder_app_cli balance
294+
```
295+
296+
If Ctip shows the deposit but Thunder wallet balance is 0, it's a
297+
BMM/Thunder-side sync issue. Escalate to Thunder infra.
298+
299+
### Dashboard shows the OLD DB (pre-pps-classic reset)
300+
301+
Check the drop-in override:
302+
```sh
303+
cat /etc/systemd/system/simplepool-dashboard.service.d/pps-thunder.conf
304+
# PROXY_DB_PATH should be /home/forknet/pps-thunder-test/data/shares.db
305+
```
306+
Fix + `systemctl daemon-reload && systemctl restart simplepool-dashboard`.
307+
308+
### Simplepool won't start — `Address already in use`
309+
310+
There's a stale binary still bound. **DO NOT** `pkill -f simplepool`
311+
from an SSH session — that pattern matches your own command line and
312+
kills your SSH. Use exact-name match instead:
313+
314+
```sh
315+
pkill -9 -x simplepool
316+
sleep 2
317+
ss -tlnp | grep :3334 || echo ":3334 free"
318+
```
319+
320+
### Miner shares accrue but payouts never fire
321+
322+
Read the payout log. Common causes:
323+
- Reserve short (fix: deposit)
324+
- Thunder RPC unreachable (fix: `systemctl status`, log tail)
325+
- Worker's Thunder address is `NULL` in the `workers` table (means it
326+
never authorized cleanly — check the `rejects` table for that
327+
worker_name)
328+
329+
---
330+
331+
## Open items on the operator side
332+
333+
1. **Point `pool_btc_address` at an enforcer-wallet-owned address** so
334+
the enforcer can spend accumulated coinbase directly into deposit
335+
txs. Right now it's set to the operator's external wallet — every
336+
deposit currently spends the enforcer's own seed funds.
337+
2. **BMM / Thunder sidechain block production** on this box currently
338+
doesn't complete. Deposits are consensus-recognized (Ctip updates)
339+
but Thunder wallet balance stays 0 until BMM runs. Investigating
340+
this is a Thunder-side task, not a pool task.
341+
3. **Admin "Deposit to Thunder" form** — CLASSIC_PAYOUTS.md sketches
342+
the endpoint (`POST /admin/deposit`) but it's not implemented yet.
343+
Until it lands, deposits are the CLI runbook above.
344+
345+
None of these block the pool from running correctly — they're
346+
improvements to the operator UX and end-to-end value flow, not
347+
correctness gaps.

0 commit comments

Comments
 (0)