Skip to content

Commit 7adf99a

Browse files
frooozefroooze
committed
docs: comprehensive system invariants expansion with categorized prefixes
Rewrite COW_INVARIANTS.md to cover the full system: - Sync engine invariants (drift refetch, fill back-pressure, orphans) - Maintenance runtime (pipeline gating, illegal-state, cooldown, dust) - Grid structure (one-to-one order mapping, price-level uniqueness) - Reconcile (duplicate cancel, rebalance, extreme placement) - Batch/pipeline (illegal-state abort, stale-only cancel, recovery) - Fund registry, subscriptions across all subsystems All invariants tagged with categorized prefixes: INV-COW, INV-REC, INV-PROJ, INV-ID, INV-ACC, INV-DUST, INV-SYNC, INV-MAINT, INV-GRID, INV-RECON, INV-BATCH, INV-PIPE, INV-STATE, INV-REG, INV-SUB, INV-BOOT, INV-UPDATE
1 parent 7eee1ac commit 7adf99a

1 file changed

Lines changed: 340 additions & 18 deletions

File tree

‎docs/COW_INVARIANTS.md‎

Lines changed: 340 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,42 @@
1-
# COW Invariants (Stable Theory Contract)
1+
# System Invariants (Stable Theory Contract)
22

3-
This document defines the non-negotiable behavioral invariants for the Copy-on-Write (COW) pipeline. It is a contract for code review and release safety, not a design tutorial.
3+
This document defines the non-negotiable behavioral invariants for the DEXBot2 system. It is a contract for code review and release safety, not a design tutorial.
44

55
## Scope
66

7-
- Applies to COW planning, projection, reconciliation, commit, and fund accounting flows.
8-
- Primary modules: `modules/order/utils/validate.ts`, `modules/order/manager.ts`, `modules/order/accounting.ts`, `modules/order/working_grid.ts`.
7+
- **COW Pipeline**: planning, projection, reconciliation, commit, and fund accounting flows.
8+
- **Sync Engine**: fill history sync, open-order sync, orphan detection, cache vs chain consistency.
9+
- **Maintenance Runtime**: pipeline gating, illegal-state handling, cooldown semantics, dust ordering.
10+
- **Grid & Reconcile**: price-level uniqueness, dust detection scope, reconcile cancel semantics.
11+
- **Batch & Pipeline**: stale handling, retry gating, stale-flag cleanup.
12+
- **Fund Registry**: cross-bot allocation invariants.
13+
- **Subscriptions**: health watchdog, silent-death detection.
914

10-
## Invariants
15+
## Invariant Prefixes
16+
17+
| Prefix | Subsystem |
18+
|--------|-----------|
19+
| `INV-COW` | COW pipeline |
20+
| `INV-REC` | Reconcile |
21+
| `INV-PROJ` | Projection |
22+
| `INV-ID` | Order identity |
23+
| `INV-ACC` | Accounting / fund tracking |
24+
| `INV-DUST` | Dust health |
25+
| `INV-SYNC` | Sync engine |
26+
| `INV-MAINT` | Maintenance runtime |
27+
| `INV-GRID` | Grid structure |
28+
| `INV-RECON` | Reconcile layer |
29+
| `INV-BATCH` | Batch / pipeline |
30+
| `INV-PIPE` | Pipeline signals |
31+
| `INV-STATE` | State / lifecycle |
32+
| `INV-REG` | Fund registry |
33+
| `INV-SUB` | Subscriptions |
34+
| `INV-BOOT` | Bootstrap / resync |
35+
| `INV-UPDATE` | On-chain updates |
36+
37+
---
38+
39+
## COW Pipeline
1140

1241
- `INV-COW-001` Master immutability until commit
1342
- The master grid must not be mutated during planning/execution prep.
@@ -42,43 +71,336 @@ This document defines the non-negotiable behavioral invariants for the Copy-on-W
4271
- Committed chain/grid totals derive only from on-chain orders (`ACTIVE`/`PARTIAL` with `orderId`) and their projected sizes.
4372
- Virtual orders contribute only to virtual pools, not committed chain totals.
4473

45-
- `INV-ACC-002` Fund invariant consistency
46-
- Tracked totals must remain consistent with blockchain totals within configured tolerance.
74+
- `INV-ACC-002` Fund invariant consistency (INVARIANT 1)
75+
- Tracked totals must remain consistent with blockchain totals within `FUND_INVARIANT_PERCENT_TOLERANCE`.
76+
- `Total = Free + Committed` must hold per side.
4777
- False violations due to ideal-size projection overstatement are prohibited.
4878

79+
- `INV-ACC-003` Cross-bot fund registry invariant (INVARIANT 3)
80+
- Shared-account per-bot commitment must not exceed the bot's proportional share of chain balance.
81+
- Checked with widened tolerance `max(PERCENT_TOLERANCE * 3, 0.15)`.
82+
- Registry failure logs a warning, not a silent skip.
83+
4984
- `INV-DUST-001` Dust health gating parity
5085
- Dust health thresholding applies consistently to both CREATE and rotation destination holes.
5186

87+
---
88+
89+
## Sync Engine
90+
91+
- `INV-SYNC-001` Drift-refetch for partial fill correctness
92+
- `isEffectivelyFull` must not unconditionally trust cached `rawOnChain.for_sale`.
93+
- When cached `for_sale` is smaller than the grid's own size (drift signal), refetch from chain via `readSingleOrder`.
94+
- `isEffectivelyFull` requires either: chain-confirmed empty (refetch returned null), grid also at 0, or the other side rounding to 0.
95+
- The legacy `newSizeInt <= 0` fast-path is forbidden.
96+
97+
- `INV-SYNC-002` No TTL-based chain refetch
98+
- Chain is only consulted on a drift signal (cache < grid size).
99+
- Time-based TTL refetches are prohibited — a stale-but-consistent cache is still correct.
100+
101+
- `INV-SYNC-003` Drift refetch null = chain-confirmed empty
102+
- When drift refetch returns null from `readSingleOrder`, the order is authoritatively gone.
103+
- The sync engine must treat this as a chain-confirmed empty, not a connection failure.
104+
105+
- `INV-SYNC-004` Orphan adoption rejects duplicate price levels
106+
- Pass-2 fallback adoption must check whether an active grid order of the same type already exists at the same price (within `calculatePriceTolerance` using `Math.max(size)`).
107+
- If a duplicate exists, the orphan is NOT adopted; it is pushed to `unmatchedChainOrders` for chain cancel.
108+
- Size is irrelevant — any duplicate violates the one-order-per-price-level invariant.
109+
110+
- `INV-SYNC-005` Fill queue back-pressure
111+
- Subscription callbacks must not push fills beyond `MAX_INCOMING_FILL_QUEUE`.
112+
- Rejected fills must leave subscription cursors retryable; the queue must remain unchanged.
113+
- Back-pressure must not start a consumer for rejected fills.
114+
115+
- `INV-SYNC-006` syncFromOpenOrders acquires fill-processing lock
116+
- `syncFromOpenOrders` acquires `_fillProcessingLock` by default.
117+
- Callers already inside the lock must pass `fillLockAlreadyHeld=true`.
118+
- Direct call sites without the lock contract are prohibited.
119+
120+
- `INV-SYNC-007` Authoritative sync preserves fetched free balances
121+
- `synchronizeWithChain` must not double-deduct already-locked funds from fetched `buyFree`/`sellFree`.
122+
- After authoritative open-order sync, `checkFundDriftAfterFills` must return `isValid=true`.
123+
124+
---
125+
126+
## Maintenance Runtime
127+
128+
- `INV-MAINT-001` Pipeline in-flight defers maintenance
129+
- When `isPipelineEmpty` returns `isEmpty=false` (batch in-flight, recovery in-flight, or broadcasting active), `checkSpreadCondition` and `checkGridHealth` must be skipped.
130+
- Pipeline signals (`batchInFlight`, `recoveryInFlight`, `broadcasting`) must be passed to `isPipelineEmpty`.
131+
132+
- `INV-MAINT-002` Illegal state abort
133+
- When `consumeIllegalStateSignal` returns a non-null signal, the maintenance cycle must:
134+
- Trigger immediate recovery sync (`_triggerStateRecoverySync`).
135+
- Skip persistence (`_persistAndRecoverIfNeeded`) in the same cycle.
136+
- Skip remaining maintenance steps (spread checks).
137+
- Arm one maintenance cooldown cycle (`_maintenanceCooldownCycles = 1`).
138+
139+
- `INV-MAINT-003` Maintenance cooldown
140+
- After a hard-abort (illegal state), the next maintenance cycle must skip all checks (spread, health, etc.).
141+
- Cooldown is exactly one cycle; normal checks resume on the subsequent cycle.
142+
- Cooldown must not stack or compound.
143+
144+
- `INV-MAINT-004` Maintenance idle gate
145+
- Maintenance must wait for a quiet window with no fill queue, sync, or batch activity (`BLOCKCHAIN_SETTLE_DELAY_MS`).
146+
- Recent activity tracking covers: fill queueing, fill processing completion, COW batch start/end, open-order sync, periodic fetches.
147+
148+
- `INV-MAINT-005` Dust-first ordering
149+
- Dust partials default to `DUST_CANCEL_DELAY_SEC` (30s).
150+
- Grid resync and structural maintenance must be deferred until dust timers complete, plus an additional blockchain settle delay before retrying reset work.
151+
152+
---
153+
154+
## Grid Structure
155+
156+
- `INV-GRID-001` One-to-one order mapping
157+
- One grid slot = at most one on-chain order. No two chain orders may map to the same grid slot.
158+
- Sync engine tracks `matchedGridOrderIds` through both sync passes and skips already-matched slots.
159+
- Surplus orders (matched count > `activeOrders.buy/sell` targets) are flagged for cancellation.
160+
161+
- `INV-GRID-002` One order per price level
162+
- The active grid must have at most one on-chain order per (type, price) pair.
163+
- Duplicate price levels are a structural violation — the sync engine must reject orphan adoption at a duplicate price, and the reconcile layer must cancel offenders on chain.
164+
- Size is irrelevant; any duplicate violates this invariant.
165+
166+
- `INV-GRID-003` Interior dust detection at duplicate price levels
167+
- Interior partials (not top-of-window) are eligible for dust detection if they share a price level with an ACTIVE sibling.
168+
- Top-of-window partials remain always eligible.
169+
- Two PARTIALs sharing a price with no active sibling do not qualify (left to rebalancer).
170+
171+
---
172+
173+
## Reconcile
174+
175+
- `INV-RECON-001` Reconcile cancels duplicate chain orders unconditionally
176+
- When an unmatched order is within `looseTolerance` of an active grid order, it must be cancelled on chain via `_cancelChainOrder` with `releaseUntrackedFunds: true`.
177+
- Cancelled IDs are filtered out of `unmatchedParsed` to prevent reprocessing.
178+
- No size guard — any duplicate at the same price is a violation.
179+
- `SUSPECTED_DUPLICATE_TOLERANCE_FLOOR` (absolute price floor) is removed — only `tolerance * SUSPECTED_DUPLICATE_TOLERANCE_MULTIPLIER` is used.
180+
181+
- `INV-RECON-002` Rebalance must not convert on-chain slots to SPREAD via CREATE
182+
- `performSafeRebalance` must not emit `CREATE` actions that convert existing on-chain slots into SPREAD orders.
183+
- On-chain mid-slot must keep its BUY/SELL type before commit.
184+
185+
- `INV-RECON-003` Extreme placement ordering
186+
- BUY placements must use nearest available free slots first (ascending price).
187+
- SELL placements must use nearest available free slots first (descending price).
188+
189+
---
190+
191+
## Batch / Pipeline
192+
193+
- `INV-BATCH-001` Illegal state batch abort
194+
- When `executeBatch` throws `ILLEGAL_ORDER_STATE`, the caller must:
195+
- Return `abortedForIllegalState: true`.
196+
- Trigger one immediate recovery sync.
197+
- Arm one maintenance cooldown cycle.
198+
199+
- `INV-BATCH-002` Stale-only cancel fast path
200+
- When a cancel-only batch fails with "order does not exist" (stale), the handler must:
201+
- Virtualize the slot (`state = VIRTUAL`, `orderId = null`, `size = 0`, `rawOnChain = null`).
202+
- Return `stale: true`.
203+
- NOT trigger a recovery sync.
204+
- Track the stale order id in `_staleCleanedOrderIds` to prevent double-credit.
205+
- Preserve manager index validity.
206+
207+
- `INV-BATCH-003` "Cannot deduct" → recovery sync
208+
- When `executeBatch` throws "Cannot deduct all or more from order than order contains", the handler must:
209+
- Return `recoveredBySync: true`, `reason: 'ORDER_SIZE_DRIFT'`.
210+
- Trigger one recovery sync.
211+
- NOT virtualize the slot.
212+
- Preserve `orderId` until sync reconciles it.
213+
- NOT mark the order as stale-cleaned.
214+
215+
---
216+
217+
## Pipeline Signals
218+
219+
- `INV-PIPE-001` Stale correction entry removal
220+
- `correctOrderPriceOnChain` must remove the entry from `ordersNeedingPriceCorrection` in ALL exit paths: success, skip (updateOrder returns null), and error.
221+
- Use `try`/`catch`/`finally` with `.filter()` removal only in `finally`.
222+
223+
- `INV-PIPE-002` Throw-safe grid divergence corrections
224+
- `updateGridFromBlockchainSnapshot` called from `applyGridDivergenceCorrections` must be wrapped in try/catch that clears `_gridSidesUpdated` on failure.
225+
- A throw must not leave `_gridSidesUpdated` permanently set, which would block the next tick.
226+
227+
---
228+
229+
## State / Lifecycle
230+
231+
- `INV-STATE-001` Bootstrap suppresses invariant checks
232+
- While `isBootstrapping()` is true, `_verifyFundInvariants` must be suppressed.
233+
- Suppression covers `recalculateFunds` and `_updateOrder` trigger paths.
234+
235+
- `INV-STATE-002` Recovery validation not masked by bootstrap
236+
- `_performStateRecovery` must detect drift even when `isBootstrapping()` is true.
237+
- Bootstrap suppression of invariant checks must not also suppress recovery validation.
238+
239+
- `INV-STATE-003` Grid resize respects budget after capping
240+
- BUY allocation must not exceed calculated budget after `_recalculateGridOrderSizesFromBlockchain` capping.
241+
242+
---
243+
244+
## Fund Registry
245+
246+
- `INV-REG-001` Cross-bot allocation ≤ proportional share
247+
- Per-bot committed amounts (sum of on-chain orders) must not exceed `totalChainBalance × allocatedPercent`.
248+
- Violation triggers a warning-level log entry (not silent), with tolerance `max(PERCENT_TOLERANCE * 3, 0.15)`.
249+
- Registry registration is pre-flight + atomic; all bots register before any starts.
250+
- Release happens in `DEXBot.shutdown`.
251+
252+
- `INV-REG-002` Async-locked registry writes
253+
- All `fund_registry` mutations (register, update, release) must hold an async lock.
254+
- Concurrent writes from multiple bots sharing the same account must be serialized.
255+
256+
---
257+
258+
## Subscriptions
259+
260+
- `INV-SUB-001` Subscription health watchdog
261+
- A periodic timer (`SUBSCRIPTION_HEALTH_CHECK_INTERVAL_MS`) must check `lastNoticeAt` per subscription.
262+
- If silence exceeds `SUBSCRIPTION_SILENT_THRESHOLD_MS`, trigger `resubscribeEntry('healthcheck')`.
263+
- `reconnecting` flag must guard against concurrent resubscribe calls.
264+
265+
---
266+
267+
## Broadcast
268+
269+
- `INV-BROADCAST-001` BROADCAST_DEADLINE graceful recovery
270+
- On `BroadcastUncertainError`, the bot must:
271+
- Retry once with a fresh deadline window (`_executeWithRetryOnUncertain`).
272+
- Skip retry when `err.partialOnChainState` is true (pair-mode grouped execution).
273+
- After retry expiry, reconcile with `fillLockAlreadyHeld=true` to avoid AsyncLock deadlock.
274+
- Daemon broadcast retries are configurable via `CREDENTIAL_DAEMON_BROADCAST_RETRIES`.
275+
276+
- `INV-BROADCAST-002` Deadlock-free reconcile after uncertain broadcast
277+
- `_reconcileAfterUncertainBroadcast` must pass `fillLockAlreadyHeld=true` because `_fillProcessingLock` is already held by the fill-processing call chain.
278+
- AsyncLock is not reentrant; a second `acquire()` would queue forever.
279+
280+
---
281+
282+
## Constants & Precision
283+
284+
- `INV-CONST-001` All magic numbers centralized in `constants.ts`
285+
- Hardcoded fallbacks in runtime code are prohibited.
286+
- Every timing value, limit, and threshold must have a named constant.
287+
288+
- `INV-PREC-001` Precision helpers throw on invalid precision
289+
- `formatAmountByPrecision` and `formatSizeByOrderType` must throw when precision is undefined.
290+
- Silent fallback to `DEFAULT_ASSET_PRECISION (8)` is prohibited.
291+
- `floatToBlockchainInt` throws on undefined precision — callers must guarantee precision is available before calling.
292+
293+
---
294+
52295
## Test Mapping
53296

297+
### COW Pipeline
54298
- `INV-COW-001`, `INV-COW-002`
55299
- `tests/test_cow_master_plan.ts` (`COW-001`, `COW-002`)
56300
- `tests/test_cow_commit_guards.ts`
57-
58301
- `INV-REC-001`
59302
- `tests/test_cow_master_plan.ts` (`COW-016`)
60-
61303
- `INV-PROJ-001`
62304
- `tests/test_cow_master_plan.ts` (`COW-012`, `COW-013`, `COW-014`)
63-
64305
- `INV-PROJ-002`
65306
- `tests/test_cow_master_plan.ts` (`COW-018`, `COW-018c`)
66-
67307
- `INV-PROJ-003`
68308
- `tests/test_cow_master_plan.ts` (`COW-018b`)
69-
70309
- `INV-DUST-001`
71310
- `tests/test_cow_master_plan.ts` (`COW-017`)
72311

312+
### Accounting / Fund Tracking
313+
- `INV-ACC-001`, `INV-ACC-002`
314+
- `tests/test_funds.ts`
315+
- `modules/order/accounting.ts:472-594` (`_verifyFundInvariants`)
316+
- `INV-ACC-003`
317+
- `modules/order/accounting.ts:534-583` (cross-bot INVARIANT 3)
318+
319+
### Sync Engine
320+
- `INV-SYNC-001`, `INV-SYNC-002`, `INV-SYNC-003`
321+
- `tests/test_sync_fill_drift_refetch.ts` (5 sub-cases)
322+
- `tests/test_ghost_order_fix.ts`
323+
- `INV-SYNC-004`
324+
- `tests/test_sync_logic.ts` (`testOrphanAtDuplicatePriceLevelIsNotAdopted`)
325+
- `INV-SYNC-005`
326+
- `tests/test_patch17_invariants.ts` (`testFillCallbackAppliesQueueBackPressure`)
327+
- `INV-SYNC-006`
328+
- `tests/test_sync_lock_routing.ts`
329+
- `INV-SYNC-007`
330+
- `tests/test_resync_invariants.ts` (Case 5)
331+
332+
### Maintenance Runtime
333+
- `INV-MAINT-001`
334+
- `tests/test_patch17_invariants.ts` (`testPipelineInFlightDefersMaintenance`)
335+
- `INV-MAINT-002`, `INV-MAINT-003`
336+
- `tests/test_patch17_invariants.ts` (`testIllegalStateAbortResyncAndCooldown`)
337+
- `INV-MAINT-004`, `INV-MAINT-005`
338+
- `tests/test_dust_rebalance_logic.ts`
339+
340+
### Grid Structure
341+
- `INV-GRID-001`
342+
- `modules/order/sync_engine.ts` (one-to-one mapping via `matchedGridOrderIds`)
343+
- `tests/test_sync_logic.ts` (surplus order cancellation)
344+
- `INV-GRID-002`
345+
- `modules/order/sync_engine.ts:804` (duplicate price level rejection)
346+
- `modules/order/grid_reconcile.ts` (duplicate chain order cancel)
347+
- `INV-GRID-003`
348+
- `tests/test_dust_rebalance_logic.ts` (`testInteriorDustWithDuplicatePriceLevel`, `testInteriorDustAdjacentGridLevelNotEligible`)
349+
350+
### Reconcile
351+
- `INV-RECON-001`
352+
- `modules/order/grid_reconcile.ts` (unconditional duplicate cancel)
353+
- `INV-RECON-002`
354+
- `tests/test_patch17_invariants.ts` (`testRoleAssignmentBlocksOnChainSpreadConversion`)
355+
- `INV-RECON-003`
356+
- `tests/test_patch17_invariants.ts` (`testExtremePlacementOrdering`)
357+
358+
### Batch / Pipeline
359+
- `INV-BATCH-001`
360+
- `tests/test_patch17_invariants.ts` (`testIllegalBatchAbortArmsMaintenanceCooldown`)
361+
- `INV-BATCH-002`
362+
- `tests/test_patch17_invariants.ts` (`testSingleStaleCancelBatchUsesStaleOnlyFastPath`)
363+
- `INV-BATCH-003`
364+
- `tests/test_patch17_invariants.ts` (`testCannotDeductTriggersRecoverySyncInsteadOfVirtualizing`)
365+
366+
### Pipeline Signals
367+
- `INV-PIPE-001`
368+
- `modules/order/utils/order.ts` (correction entry removal via finally)
369+
- `INV-PIPE-002`
370+
- `modules/order/utils/system.ts` (try/catch wrapping grid divergence)
371+
372+
### State / Lifecycle
373+
- `INV-STATE-001`, `INV-STATE-002`
374+
- `tests/test_resync_invariants.ts` (Cases 1-4)
375+
- `INV-STATE-003`
376+
- `tests/test_patch17_invariants.ts` (`testGridResizeRespectsBudgetAfterCap`)
377+
378+
### Fund Registry
379+
- `INV-REG-001`, `INV-REG-002`
380+
- `modules/fund_registry.ts`
381+
- `modules/order/accounting.ts:534-583`
382+
383+
### Subscriptions
384+
- `INV-SUB-001`
385+
- `modules/bitshares-native/subscriptions.ts` (health watchdog)
386+
- `tests/test_native_subscriptions.ts`
387+
388+
### Broadcast
389+
- `INV-BROADCAST-001`, `INV-BROADCAST-002`
390+
- `modules/dexbot_class.ts` (`_executeWithRetryOnUncertain`, `_reconcileAfterUncertainBroadcast`)
391+
392+
---
393+
73394
## Review Checklist (Quick Use)
74395

75-
For any COW/accounting change, reviewers should verify:
396+
For any change touching these subsystems, reviewers should verify:
76397

77-
- Does it preserve `INV-PROJ-002` for on-chain PARTIAL orders?
78-
- Does it avoid non-rotation size UPDATE leakage in reconcile (`INV-REC-001`)?
79-
- Does it keep virtual/on-chain accounting separation (`INV-ACC-001`)?
80-
- Does it preserve atomic commit semantics (`INV-COW-001`, `INV-COW-002`)?
81-
- Are corresponding regression tests added/updated?
398+
- **COW/Accounting**: preserves `INV-PROJ-002` for on-chain PARTIAL orders; avoids non-rotation size UPDATE leakage (`INV-REC-001`); keeps virtual/on-chain separation (`INV-ACC-001`); preserves atomic commit (`INV-COW-001`, `INV-COW-002`).
399+
- **Sync Engine**: does not reintroduce `newSizeInt <= 0` fast-path (`INV-SYNC-001`); does not add TTL-based refetch (`INV-SYNC-002`); orphan adoption checks price-level uniqueness (`INV-SYNC-004`).
400+
- **Maintenance**: pipeline signals are passed to `isPipelineEmpty` (`INV-MAINT-001`); illegal-state abort arms cooldown (`INV-MAINT-002/003`).
401+
- **Reconcile**: duplicate chain orders at same price are cancelled unconditionally (`INV-RECON-001`); no CREATE-based spread conversion of on-chain slots (`INV-RECON-002`).
402+
- **Batch**: stale-only cancel uses fast path without recovery sync (`INV-BATCH-002`); "cannot deduct" triggers sync, not virtualization (`INV-BATCH-003`).
403+
- **Are corresponding regression tests added/updated?**
82404

83405
## Change Policy
84406

0 commit comments

Comments
 (0)