Skip to content

Commit f1cb7de

Browse files
rsantacrocersantacroce
committed
OPERATOR_GUIDE: add 'Reserve is short' troubleshooting section
Adds a full runbook at the end of the troubleshooting section for when the admin dashboard shows the '⚠ Reserve is short by N sats' banner. Covers the mental model (where BTC lives at each of the three hops), how much to deposit, and step-by-step commands: Step 1 — bitcoin-cli sendtoaddress from operator wallet to a fresh enforcer-owned receive address Step 2 — CreateDepositTransaction from the enforcer wallet to a bare-base58 Thunder address (with the same wrapper-form warning the deposit-address handling already carries in thunder_address_decode and scripts/log-deposit.sh) Step 3 — poke Thunder 'mine' RPC to include the deposit in a sidechain block (client-side timeout is normal; the block gets produced anyway) Step 4 — log the row via scripts/log-deposit.sh so the admin dashboard's 'Deposits' card shows it with txid + Ctip delta Also covers the two commonly-hit edge cases: - the current all-or-nothing worker policy — partial deposits won't unstick payouts until the reserve covers total owed (with a pointer at the partial-payout-policy open item) - Ctip moves but Thunder balance stays 0: BMM is stalled; the fix is a Thunder restart with the exact command All hostnames / SSH keys / bitcoind RPC creds are placeholders (<pool-host>, <ssh-key>, <rpcuser>, <rpcpassword>, <wallet-name>). No live secrets.
1 parent 411a003 commit f1cb7de

1 file changed

Lines changed: 147 additions & 0 deletions

File tree

‎OPERATOR_GUIDE.md‎

Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -347,6 +347,153 @@ Read the payout log. Common causes:
347347
never authorized cleanly — check the `rejects` table for that
348348
worker_name)
349349

350+
### "⚠ Reserve is short by N sats" banner on /admin
351+
352+
The admin dashboard's PPS ledger card is telling you the payout worker's
353+
next tick will see less BTC in Thunder than the sum of everything the
354+
pool owes. The worker keeps skipping ticks (correctly — its policy is
355+
"pay everyone or nobody" to prevent partial-payout states) until you
356+
close the gap by moving BTC through the two-step deposit flow.
357+
358+
**Where BTC lives at each step:**
359+
360+
```
361+
coinbase (manual: sendtoaddress)
362+
│ │
363+
▼ ▼
364+
┌─────────────────┐ step 1 ┌──────────────────┐ step 2 ┌────────────────┐
365+
│ operator BTC │────────────▶│ enforcer wallet │─────────▶│ Thunder wallet │
366+
│ wallet │ │ (on-box, BIP300 │ (Create-│ (BMM confirms) │
367+
│ pool_btc_addr │ │ aware) │ Deposit-│ │
368+
└─────────────────┘ └──────────────────┘ Tx) └───────┬────────┘
369+
accumulates spends into │ step 3
370+
coinbase drivechain deposits │ payout worker
371+
▼
372+
┌────────────────┐
373+
│ miner Thunder │
374+
│ wallets │
375+
└────────────────┘
376+
```
377+
378+
The banner means "step 1 + step 2 haven't happened recently enough."
379+
Runbook to close it:
380+
381+
**How much to deposit.** Rule of thumb: **1.1× the current owed**,
382+
i.e. enough to cover the debt plus headroom for the fee_sats budget +
383+
new shares landing while you deposit. Read `totals.owed` off
384+
`/api/admin/summary` (admin auth) for the current number.
385+
386+
**Step 1 — fund the enforcer wallet from the operator wallet.**
387+
388+
```sh
389+
ssh -i <ssh-key> root@<pool-host>
390+
GRPCURL=/home/forknet/forknet-software/grpcurl
391+
BCLI="/home/forknet/forknet-software/drivechain-forknet/build/bin/bitcoin-cli \
392+
-datadir=/home/forknet/.drivechain-forknet \
393+
-rpcuser=<rpcuser> -rpcpassword=<rpcpassword> \
394+
-rpcwallet=<wallet-name>"
395+
396+
# Fresh enforcer-owned receive address (one per deposit).
397+
ENF=$($GRPCURL -plaintext 127.0.0.1:50051 \
398+
cusf.mainchain.v1.WalletService/CreateNewAddress \
399+
| python3 -c "import json,sys; print(json.load(sys.stdin)['address'])")
400+
echo "enforcer receive: $ENF"
401+
402+
# Send the target amount from the operator wallet. Wait for the next
403+
# natural mainchain block (30-60s in a healthy miner) to confirm.
404+
$BCLI sendtoaddress "$ENF" <btc_amount>
405+
```
406+
407+
**Step 2 — deposit from the enforcer wallet into Thunder.**
408+
409+
```sh
410+
TCLI=/home/forknet/forknet-software/thunder-rust/target/debug/thunder_app_cli
411+
412+
# BARE Thunder address ONLY. Do NOT use `format-deposit-address` — the
413+
# 's<n>_<base58>_<hex6>' wrapper is rejected by Thunder's OP_RETURN
414+
# parser and any deposit to it ends up at a fallback address the pool
415+
# doesn't own. Empirically verified; the C-side authorize check now
416+
# rejects wrapper-form usernames for the same reason.
417+
POOL_ADDR=$(sudo -u forknet $TCLI get-new-address)
418+
echo "deposit target: $POOL_ADDR"
419+
420+
# Capture Ctip before so we can verify it moves.
421+
CTIP_BEFORE=$($GRPCURL -plaintext -d '{"sidechain_number":9}' \
422+
127.0.0.1:50051 cusf.mainchain.v1.ValidatorService/GetCtip)
423+
echo "$CTIP_BEFORE"
424+
425+
# Deposit — value is in sats. E.g. 5 BTC = 500_000_000, 50 BTC = 5_000_000_000.
426+
DEPOSIT_SATS=<sats>
427+
$GRPCURL -plaintext \
428+
-d "{\"sidechain_id\":9,\"address\":\"$POOL_ADDR\",\
429+
\"value_sats\":$DEPOSIT_SATS,\"fee_sats\":1000}" \
430+
127.0.0.1:50051 cusf.mainchain.v1.WalletService/CreateDepositTransaction
431+
```
432+
433+
Wait for a mainchain block, then confirm the Ctip grew by roughly
434+
`DEPOSIT_SATS`:
435+
436+
```sh
437+
$GRPCURL -plaintext -d '{"sidechain_number":9}' \
438+
127.0.0.1:50051 cusf.mainchain.v1.ValidatorService/GetCtip
439+
```
440+
441+
**Step 3 — poke Thunder to include the deposit in a sidechain block.**
442+
443+
```sh
444+
curl -sS --max-time 15 -H 'content-type: application/json' \
445+
--data '{"jsonrpc":"2.0","id":1,"method":"mine","params":[]}' \
446+
http://127.0.0.1:6009/
447+
448+
# balance should now show the new sats
449+
sudo -u forknet $TCLI balance
450+
```
451+
452+
Client-side `mine` sometimes times out at ~12s but the block gets
453+
produced anyway — the timeout is on the RPC response, not the actual
454+
work. Re-check `balance` after 30s if the first check still shows 0.
455+
456+
**Step 4 — log it so the admin dashboard's "Deposits" card shows the row.**
457+
458+
```sh
459+
scripts/log-deposit.sh \
460+
--db /home/forknet/pps-thunder-test/data/shares.db \
461+
--txid <mainchain txid from step 2> \
462+
--sats $DEPOSIT_SATS --fee 1000 \
463+
--recipient $POOL_ADDR \
464+
--ctip-before <n> --ctip-after <n> \
465+
--note "top-up to cover payout backlog"
466+
```
467+
468+
The payout worker's next tick (max 30s later) then sees
469+
`available_sats >= totals.owed + fees`, drops the warning banner, and
470+
starts firing Thunder transfers. Each one lands in the admin's
471+
"Recent payouts" card with its txid; miners see the same txid on their
472+
public per-worker page.
473+
474+
**Doing it in smaller rounds.** The current all-or-nothing worker policy
475+
(pay everyone or nobody) means even 90% of the owed sum still won't
476+
trip the gate — save yourself the disappointment and either close the
477+
gap completely in one round, or land the partial-payout policy first
478+
(sort due workers by `owed ASC`, pay whoever fits, stop when the next
479+
one won't fit; ~15 LoC in `payout/lib/payout.js`).
480+
481+
**If the Ctip moves but Thunder balance stays at 0.** BMM is stalled.
482+
Restart Thunder to unblock it:
483+
484+
```sh
485+
pkill -x thunder_app
486+
sudo -u forknet -H bash -c "
487+
/home/forknet/forknet-software/thunder-rust/target/debug/thunder_app \
488+
--headless --datadir /home/forknet/pps-thunder-test/thunder-data \
489+
--network forknet --mainchain-grpc-url http://127.0.0.1:50051 \
490+
--net-addr 127.0.0.1:4009 --rpc-addr 127.0.0.1:6009 \
491+
--log-level INFO \
492+
> /home/forknet/pps-thunder-test/logs/thunder.log 2>&1 &"
493+
```
494+
495+
Give it ~30s to sync and re-run step 3.
496+
350497
---
351498

352499
## Open items on the operator side

0 commit comments

Comments
 (0)