fix: handle BROADCAST_DEADLINE gracefully — deadlock fix, retry, configurable daemon attempts

The bot deadlocked after a BROADCAST_DEADLINE credential daemon response,
leaving the grid unreconciled and the process hung. Three interrelated
changes:

## Deadlock fix (fillLockAlreadyHeld:true)
File: modules/dexbot_class.ts:4242
- Problem: _reconcileAfterUncertainBroadcast tried to acquire
  _fillProcessingLock, but it was already held by the fill-processing
  call chain (fill loop → safe-rebalance → COW broadcast → catch).
  AsyncLock is not reentrant, so the second acquire() queued forever.
- Impact: reconcile never ran, no log output after the error.
- Fix: pass fillLockAlreadyHeld:true from the catch block — the lock
  is always held at this call site.

## Bot-level retry for BroadcastUncertainError
File: modules/dexbot_class.ts:2943-2972
- The daemon's broadcastWithRetry runs up to N attempts against a
  single 25s inner deadline (CREDENTIAL_DAEMON_INNER_DEADLINE_MS).
  If all expire, the error reaches the bot immediately.
- Fix: new _executeWithRetryOnUncertain wrapper retries the broadcast
  once with a fresh 25s window before falling through to reconcile.
- Skips retry when err.partialOnChainState is true (pair-mode grouped
  execution) — re-broadcasting would duplicate already-committed creates.

## Configurable daemon broadcast retries
File: modules/constants.ts:271, credential-daemon.ts:407,421
- Hardcoded `attempt <= 2` replaced by
  CREDENTIAL_DAEMON_BROADCAST_RETRIES (default 3).
- Daemon reads constant via TIMING lookup with ??2 fallback.

## Test coverage
File: tests/test_uncertain_broadcast.ts
- UNC-012: drives _updateOrdersOnChainBatchCOW through the
  BroadcastUncertainError path, asserts reconcile receives
  fillLockAlreadyHeld:true (deadlock regression guard).
- UNC-013: asserts _executeWithRetryOnUncertain retries exactly
  once then re-throws on second BroadcastUncertainError.
- UNC-014: asserts retry is skipped when partialOnChainState is
  set (pair-mode double-publish guard).
- Also added missing stubs (lockOrders, unlockOrders, etc.) to
  makeBot() to support the deeper method path.

## Risk/Edge-case Notes
- Bot-level retry adds up to ~25s per retry. Current config = 1 retry,
  so worst-case broadcast stall is ~50s (2 × 25s) before reconcile.
- partialOnChainState skip only matters for pair-mode (outside-in
  create groups). Non-pair batches (updates/cancels) always set it
  false, so they always get the retry.
- Daemon's broadcastWithRetry now reads TIMING at call time, not
  module-load time — consistent with the existing pattern used by
  CREDENTIAL_DAEMON_INNER_DEADLINE_MS.

## Testing Notes
- npm test (uncertain broadcast suite) — all 24 tests pass.
- Also verified: cow_orchestration_fixes, cow_commit_guards,
  main_loop_sync_fill_rebalance.

Author: froooze

credential-daemon.ts

@@ -404,8 +404,9 @@ async function broadcastWithRetry(accountName: any, privateKey: any, broadcastFn
         }, innerDeadlineMs);
     });
 
+    const maxRetries = TIMING?.CREDENTIAL_DAEMON_BROADCAST_RETRIES ?? 2;
     const work = (async () => {
-        for (let attempt = 1; attempt <= 2; attempt++) {
+        for (let attempt = 1; attempt <= maxRetries; attempt++) {
             try {
                 if (_nativeChainClient.getStatus() !== 'connected') {
                     _nativeChainClient.setNodes(_nativeNodeList.length > 0 ? _nativeNodeList : NODE_MANAGEMENT.DEFAULT_NODES);
@@ -417,7 +418,7 @@ async function broadcastWithRetry(accountName: any, privateKey: any, broadcastFn
                 await client.initPromise;
                 return await broadcastFn(client);
             } catch (err: any) {
-                if (attempt === 2) throw err;
+                if (attempt === maxRetries) throw err;
                 debugLog(`Broadcast failed (attempt ${attempt}), reconnecting: ${err.message}`);
                 try { _nativeChainClient.disconnect(); } catch (_) {}
             }

modules/constants.ts

@@ -267,6 +267,13 @@ let TIMING = {
     // path handles whatever takes longer.
     CREDENTIAL_DAEMON_INNER_DEADLINE_MS: 25000,
 
+    // CREDENTIAL_DAEMON_BROADCAST_RETRIES: Number of broadcast attempts inside the
+    // credential daemon's broadcastWithRetry. Each attempt connects, signs, and pushes
+    // the transaction. Total wall time across all attempts is capped by
+    // CREDENTIAL_DAEMON_INNER_DEADLINE_MS above. 2 is a pragmatic balance: try once,
+    // retry once on disconnect, while leaving room for a slow block before the deadline.
+    CREDENTIAL_DAEMON_BROADCAST_RETRIES: 3,
+
     // SAFETY_NET_SYNC_TIMEOUT_MS: Cap on the post-reconnect safety-net sync
     // in dexbot_class.ts. Must stay below the 20s shutdown lock timeout so
     // it never holds _fillProcessingLock longer than the shutdown deadline.

modules/dexbot_class.ts

@@ -2940,6 +2940,37 @@ class DEXBot {
         return false;
     }
 
+    /**
+     * Execute operations with retry on BroadcastUncertainError.
+     * The daemon already retries internally against a 25s deadline.
+     * If all expire, a fresh bot-level attempt buys a new 25s window.
+     *
+     * Skips retry when partialOnChainState is true (pair-mode grouped
+     * execution where earlier groups already committed). Re-broadcasting
+     * the full operations array would duplicate those creates on chain.
+     */
+    async _executeWithRetryOnUncertain(operations, opContexts) {
+        const MAX_RETRIES = 1;
+        for (let attempt = 1; ; attempt++) {
+            try {
+                return await this._executeOperationsWithStrategy(operations, opContexts);
+            } catch (err) {
+                const isRetriable = err instanceof BroadcastUncertainError
+                    && !err.partialOnChainState
+                    && attempt <= MAX_RETRIES;
+                if (isRetriable) {
+                    this.manager.logger.log(
+                        `[COW] Broadcast uncertain (attempt ${attempt}/${MAX_RETRIES + 1}), retrying...`,
+                        'warn'
+                    );
+                    await this._ensureCredentialDaemonWritable('COW batch retry');
+                    continue;
+                }
+                throw err;
+            }
+        }
+    }
+
     /**
      * Execute blockchain operations with appropriate strategy (single batch or pair mode).
      * @param {Array<import('./types').CreatedOperation>} operations - Array of operation objects
@@ -4077,7 +4108,7 @@ class DEXBot {
             // Execute batch
             this.manager.logger.log(`[COW] Broadcasting batch with ${operations.length} operations...`, 'info');
             this._lastBroadcastHeartbeatAt = Date.now();
-            const execution = await this._executeOperationsWithStrategy(operations, opContexts);
+            const execution = await this._executeWithRetryOnUncertain(operations, opContexts);
             const result = execution.result;
             const executedContexts = execution.opContexts;
 
@@ -4211,7 +4242,7 @@ class DEXBot {
             // throwing would re-enter the catch loop on the next attempt and
             // potentially double-publish.
             if (err instanceof BroadcastUncertainError) {
-                return await this._reconcileAfterUncertainBroadcast(err, opContexts);
+                return await this._reconcileAfterUncertainBroadcast(err, opContexts, { fillLockAlreadyHeld: true });
             }
 
             // Handle hard abort

tests/test_uncertain_broadcast.ts

@@ -882,14 +882,136 @@ function makeBot() {
         },
         requestStructuralGridResync: async () => {},
         _lastUnmatchedChainOrders: [],
-        _pendingBroadcasts: new Map()
+        _pendingBroadcasts: new Map(),
+        lockOrders: () => {},
+        unlockOrders: () => {},
+        startBroadcasting: () => {},
+        stopBroadcasting: () => {},
+        _setRebalanceState: () => {},
+        pauseFundRecalc: () => {},
+        resumeFundRecalc: async () => {},
+        _clearWorkingGridRef: () => {},
+        _throwOnIllegalState: false
     };
     bot.account = 'test-account';
     bot.privateKey = 'test-private-key';
     bot._currentCycleId = 1;
     return bot;
 }
 
+async function testCowCatchBlockPassesFillLockAlreadyHeld() {
+    console.log('\n[UNC-012] _updateOrdersOnChainBatchCOW passes fillLockAlreadyHeld:true to reconcile...');
+    const bot = makeBot();
+    const slotId = 'slot-unc-012';
+    const actionOrderId = '1.7.999999';
+    const origBuildUpdate = chainOrders.buildUpdateOrderOp;
+    const origExecuteStrategy = bot._executeOperationsWithStrategy;
+    const origReconcile = bot._reconcileAfterUncertainBroadcast;
+    const origValidateFunds = bot._validateOperationFunds;
+    let reconcileOptions = null;
+
+    bot.manager.orders.set(slotId, { id: slotId, type: 'sell', price: 0.05, size: 100, orderId: actionOrderId });
+    bot.manager.getChainFundsSnapshot = () => ({});
+
+    chainOrders.buildUpdateOrderOp = async (account, orderId, delta, rawOnChain) => ({
+        op: { fee: { amount: 0, asset_id: '1.3.0' }, op_data: {} },
+        finalInts: { sell: 100, receive: 1 }
+    });
+
+    bot._validateOperationFunds = () => ({ isValid: true, summary: '', violations: [] });
+
+    bot._executeOperationsWithStrategy = async (_ops, _ctxs) => {
+        throw new BroadcastUncertainError('uncertain', { batchId: 'unc-012', timeoutMs: 30000 });
+    };
+
+    bot._reconcileAfterUncertainBroadcast = async (_err, _ctxs, opts) => {
+        reconcileOptions = opts;
+        return { executed: false, uncertain: true };
+    };
+
+    const cowResult = {
+        workingGrid: {},
+        workingIndexes: {},
+        workingBoundary: {},
+        actions: [{
+            type: COW_ACTIONS.UPDATE,
+            id: slotId,
+            orderId: actionOrderId,
+            newSize: 100,
+            order: { type: 'sell', price: 0.05, size: 100 }
+        }]
+    };
+
+    try {
+        await bot._updateOrdersOnChainBatchCOW(cowResult);
+        assert(reconcileOptions, '_reconcileAfterUncertainBroadcast must be called');
+        assert.strictEqual(reconcileOptions.fillLockAlreadyHeld, true,
+            'reconcile must receive fillLockAlreadyHeld:true to avoid reentrant deadlock');
+    } finally {
+        chainOrders.buildUpdateOrderOp = origBuildUpdate;
+        bot._executeOperationsWithStrategy = origExecuteStrategy;
+        bot._reconcileAfterUncertainBroadcast = origReconcile;
+    }
+    console.log('✓ UNC-012 passed');
+}
+
+async function testExecuteWithRetryOnUncertainRetriesOnce() {
+    console.log('\n[UNC-013] _executeWithRetryOnUncertain retries once on BroadcastUncertainError then throws...');
+    const bot = makeBot();
+    const origExecuteStrategy = bot._executeOperationsWithStrategy;
+    let callCount = 0;
+
+    bot._executeOperationsWithStrategy = async (_ops, _ctxs) => {
+        callCount++;
+        throw new BroadcastUncertainError('uncertain', { batchId: 'unc-013', timeoutMs: 30000 });
+    };
+
+    try {
+        await assert.rejects(
+            () => bot._executeWithRetryOnUncertain([], []),
+            (err) => {
+                assert(err instanceof BroadcastUncertainError, 'must throw BroadcastUncertainError');
+                assert.strictEqual(callCount, 2, 'must try exactly 2 times (1 initial + 1 retry)');
+                return true;
+            }
+        );
+    } finally {
+        bot._executeOperationsWithStrategy = origExecuteStrategy;
+    }
+    console.log('✓ UNC-013 passed');
+}
+
+async function testExecuteWithRetrySkipsPartialOnChainState() {
+    console.log('\n[UNC-014] _executeWithRetryOnUncertain skips retry when partialOnChainState is set...');
+    const bot = makeBot();
+    const origExecuteStrategy = bot._executeOperationsWithStrategy;
+    let callCount = 0;
+
+    bot._executeOperationsWithStrategy = async (_ops, _ctxs) => {
+        callCount++;
+        const err = new BroadcastUncertainError('uncertain', { batchId: 'unc-014', timeoutMs: 30000 });
+        err.partialOnChainState = true;
+        err.groupsBroadcast = 1;
+        err.groupsTotal = 2;
+        throw err;
+    };
+
+    try {
+        await assert.rejects(
+            () => bot._executeWithRetryOnUncertain([], []),
+            (err) => {
+                assert(err instanceof BroadcastUncertainError, 'must throw BroadcastUncertainError');
+                assert.strictEqual(callCount, 1, 'must NOT retry when partialOnChainState is true');
+                assert.strictEqual(err.partialOnChainState, true);
+                return true;
+            }
+        );
+    } finally {
+        bot._executeOperationsWithStrategy = origExecuteStrategy;
+    }
+    console.log('✓ UNC-014 passed');
+}
+
 async function main() {
     console.log('Running uncertain-broadcast recovery tests...');
     await testFingerprintDeterministic();
@@ -915,8 +1037,11 @@ async function main() {
     await testAutoCancelUsesSyncEngineChainOrderIdShape();
     await testAutoCancelSkipsWhenPendingBroadcasts();
     await testAutoCancelSkipsFingerprinted();
+    await testCowCatchBlockPassesFillLockAlreadyHeld();
+    await testExecuteWithRetryOnUncertainRetriesOnce();
+    await testExecuteWithRetrySkipsPartialOnChainState();
     testsComplete = true;
-    console.log('\nAll uncertain-broadcast tests passed.');
+    console.log('\nAll uncertain-broadcast tests passed (incl. retry + deadlock regression guards).');
 }
 
 main().catch((err) => {