feat: two-step candle gap repair (auto-fill ≤24h, then Kibana) + v1.0.3 bump

froooze · froooze · commit 4f496ed77a6e · 2026-07-01T14:55:14.000+02:00
Complete the v1.0.3 version bump and implement the two-step candle gap
repair in the market adapter.

## Two-Step Candle Gap Repair
File: market_adapter/core/market_adapter_service.ts
- Problem: Every missing candle queried Kibana, even when the native
  incremental fetch already confirmed no trades for short gaps. Empty
  Kibana responses also left writes suppressed indefinitely.
- Impact: Unnecessary Kibana load; unresolved gap state could stall
  trigger/dynamic-grid writes permanently.
- Solution: Step 1 auto-fills gaps ≤24 candles (trusted no-trade
  threshold) directly by synthesizing zero-volume candles — Kibana is
  redundant here since native fetch already verified no trades. Step 2
  queries Kibana only for gaps beyond 24 candles. If Kibana returns
  empty, that IS verification of no trades, so all gaps in the queried
  window are synthesized.

## Version Bump 1.0.2 → 1.0.3
- Bumped version across all manifests: package.json, package-lock.json,
  claw/package.json, analysis/ama_fitting/package.json,
  claw/runtimes/openclaw-plugin/package.json,
  claw/runtimes/openclaw-plugin/openclaw.plugin.json,
  claw/tests/test_claw_mcp_transport.ts

## Test Updates
File: tests/test_market_adapter_service.ts
- Refactored testInternalNoTradeGapsAreSynthesizedWhenKibanaReturnsNoPatchData
  → testInternalNoTradeGapsAreAutoFilledWithinTrustedThreshold
- Refactored testEmptyKibanaRepairOnlySynthesizesGapsInsideCappedRepairWindow
  → testEmptyKibanaResponseResolvesAllGapsInWindow
- Adjusted testNativeIncrementalMergesKibanaActivityInsteadOfSilence and
  testKibanaGapRepairPatchesMissingCandles for new two-step flow
- Added maxNativeGapFillCandles: 0 to config objects in gap-repair tests

## Documentation
- market_adapter/README.md: documented STALE_TAIL_THRESHOLD_CANDLES
  constant and full two-step gap repair flow with auto-fill and
  Kibana-verified synthesis
- docs/README.md, docs/EVOLUTION.md, docs/DEXBOT_COMPARISON.md,
  docs/FUND_MOVEMENT_AND_ACCOUNTING.md: version reference bumps

## Testing Notes
- npm test should pass with refactored gap repair assertions
- Key scenarios: within-threshold auto-fill, beyond-threshold Kibana
  query, empty Kibana → verified no-trade synthesis
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -2,6 +2,16 @@
 
 All notable changes to this project will be documented in this file.
 
+## [1.0.3] - 2026-07-01 - Two-Step Candle Gap Repair
+
+### 2026-07-01
+
+- **Fix**: stop market adapter logs from duplicating into dexbot.log — watchdog spawned the adapter with stdout piped to dexbot.log, but the adapter already writes its own log file. Changed stdio to `'ignore'` and removed dead `outLog` parameter (`modules/launcher/market_adapter_watchdog.ts`, `unlock.ts`).
+- **Feat**: two-step candle gap repair in market adapter — auto-fill gaps ≤24 candles (trusted no-trade threshold) directly without Kibana, then query Kibana only for larger gaps. Empty Kibana response is now treated as verified no-trade instead of leaving gaps "unresolved" (`market_adapter/core/market_adapter_service.ts`).
+- **Chore**: version bumped to 1.0.3 across all manifests (`package.json`, `package-lock.json`, `claw/package.json`, `analysis/ama_fitting/package.json`, `claw/runtimes/openclaw-plugin/package.json`, `claw/runtimes/openclaw-plugin/openclaw.plugin.json`).
+- **Test**: refactored gap repair tests for two-step auto-fill flow (`tests/test_market_adapter_service.ts`).
+- **Docs**: corrected `disallowedDealIds` behavior (blocks reborrow only, not repay) in `docs/MPA_CREDIT_USAGE.md` and `docs/developer_guide.md`; documented `STALE_TAIL_THRESHOLD_CANDLES` constant and two-step gap repair in `market_adapter/README.md`; bumped version/commit references in `docs/README.md`, `docs/EVOLUTION.md`, `docs/DEXBOT_COMPARISON.md`, `docs/FUND_MOVEMENT_AND_ACCOUNTING.md`.
+
 ## [1.0.2] - 2026-06-25 - Auto-Update Default & Update Script Hardening
 
 ### 2026-06-25
diff --git a/analysis/ama_fitting/package.json b/analysis/ama_fitting/package.json
@@ -1,6 +1,6 @@
 {
   "name": "ama_fitting",
-  "version": "1.0.2",
+  "version": "1.0.3",
   "description": "Tools for fitting AMA parameters to market data",
   "main": "../../dist/analysis/ama_fitting/optimizer_high_resolution.js",
   "scripts": {
diff --git a/claw/package.json b/claw/package.json
@@ -1,6 +1,6 @@
 {
   "name": "dexbot2-claw",
-  "version": "1.0.2",
+  "version": "1.0.3",
   "private": true,
   "description": "Claw bridge and runtime integration layer for BitShares and DEXBot2.",
   "type": "commonjs",
diff --git a/claw/runtimes/openclaw-plugin/openclaw.plugin.json b/claw/runtimes/openclaw-plugin/openclaw.plugin.json
@@ -2,7 +2,7 @@
   "id": "bitshares-claw",
   "name": "BitShares Claw",
   "description": "Native BitShares and DEXBot2 tools from the DEXBot2 claw integration",
-  "version": "1.0.2",
+  "version": "1.0.3",
   "skills": [
     "skills"
   ],
diff --git a/claw/runtimes/openclaw-plugin/package.json b/claw/runtimes/openclaw-plugin/package.json
@@ -1,7 +1,7 @@
 {
   "name": "bitshares-claw-openclaw-plugin",
   "private": true,
-  "version": "1.0.2",
+  "version": "1.0.3",
   "type": "commonjs",
   "main": "index.js"
 }
diff --git a/claw/tests/test_claw_mcp_transport.ts b/claw/tests/test_claw_mcp_transport.ts
@@ -30,7 +30,7 @@ async function testParserAcceptsJsonlAcrossChunksAndSingleBuffer() {
       capabilities: {},
       clientInfo: {
         name: 'claw-mcp-transport-test',
-        version: '1.0.2'
+        version: '1.0.3'
       },
       protocolVersion: '2024-11-05'
     }
@@ -181,7 +181,7 @@ async function testMainEntrypointHandlesRealProcessInitialize() {
       capabilities: {},
       clientInfo: {
         name: 'claw-mcp-transport-test',
-        version: '1.0.2'
+        version: '1.0.3'
       },
       protocolVersion: '2024-11-05'
     }
diff --git a/docs/DEXBOT_COMPARISON.md b/docs/DEXBOT_COMPARISON.md
@@ -1,7 +1,7 @@
 # DEXBot vs DEXBot2 — Detailed Comparison Report
 
 > **Date:** 2026-06-16 *(metrics refreshed against local source trees)*
-> **Scope:** Full architectural, functional, and operational comparison between the original [DEXBot](https://github.com/Codaone/DEXBot) (Python, v1.0.0) and DEXBot2 (TypeScript, v1.0.2).
+> **Scope:** Full architectural, functional, and operational comparison between the original [DEXBot](https://github.com/Codaone/DEXBot) (Python, v1.0.0) and DEXBot2 (TypeScript, v1.0.3).
 > **Audience:** Developers, contributors, and operators evaluating or migrating between the two projects.
 
 ---
@@ -36,7 +36,7 @@
 
 | Attribute | DEXBot (original) | DEXBot2 |
 |---|---|---|
-| **Release Track** | 1.0.0 | v1.0.2 |
+| **Release Track** | 1.0.0 | v1.0.3 |
 | **Language** | Python 3.6+ | TypeScript 5.x |
 | **Status** | Released 1.0.0, unmaintained | Active development |
 | **Last Repo Activity** | May 23, 2020 | 2026-06-25 |
@@ -774,7 +774,7 @@ Where:
 
 | Metric | DEXBot | DEXBot2 |
 |---|---|---|
-| **Release Track** | 1.0.0 | v1.0.2 |
+| **Release Track** | 1.0.0 | v1.0.3 |
 | **Active Since** | ~2018 | December 2025 |
 | **Last Commit** | May 23, 2020 | 2026-06-25 |
 | **Total Commits** | 2281 | 1659 at current HEAD |
diff --git a/docs/EVOLUTION.md b/docs/EVOLUTION.md
@@ -2,14 +2,14 @@
 
 ## Executive Summary
 
-DEXBot2 is a sophisticated decentralized exchange trading bot for the BitShares blockchain. This report documents the complete evolution of the project from its inception in December 2025 through the current 1.0.2 stable release.
+DEXBot2 is a sophisticated decentralized exchange trading bot for the BitShares blockchain. This report documents the complete evolution of the project from its inception in December 2025 through the current 1.0.3 stable release.
 
 ### Key Milestones
 - **Project Inception**: December 2, 2025
 - **Growth Phase**: 1,600+ commits over ~6 active months
 - **Code Maturity**: Evolution from basic utilities to a ~58,000+ LoC intelligent TypeScript system
 - **Stability**: Progression from manual testing to a suite of 200+ automated test files
-- **Releases**: 37 release entries (v0.1.0 to v1.0.2)
+- **Releases**: 38 release entries (v0.1.0 to v1.0.3)
 
 ---
 
@@ -212,7 +212,7 @@ DEXBot2 has matured from a basic grid bot into a signal-intelligent, production-
 ---
 
 **Report Originally Generated**: February 19, 2026
-**Last Updated**: June 25, 2026
-**Total Commits**: 1659
-**Date Range**: December 2, 2025 - June 25, 2026 (ongoing)
+**Last Updated**: July 1, 2026
+**Total Commits**: 1660
+**Date Range**: December 2, 2025 - July 1, 2026 (ongoing)
 **Repository**: DEXBot2 (BitShares DEX Trading Bot)
diff --git a/docs/FUND_MOVEMENT_AND_ACCOUNTING.md b/docs/FUND_MOVEMENT_AND_ACCOUNTING.md
@@ -825,4 +825,4 @@ To prevent "Time-of-Check to Time-of-Use" errors:
 3.  **Bootstrapping:** Fills arriving during startup (`isBootstrapping=true`) are queued until the grid is fully reconciled.
 
 ---
-*Technical Reference for DEXBot2 v1.0.2 release*
+*Technical Reference for DEXBot2 v1.0.3 release*
diff --git a/docs/README.md b/docs/README.md
@@ -2,7 +2,7 @@
 
 This directory contains the comprehensive technical documentation for the DEXBot2 trading bot. It is designed to guide developers from high-level architecture down to the nuances of fund accounting and state management.
 
-**Version context:** v1.0.2 is the current release.
+**Version context:** v1.0.3 is the current release.
 
 ---
 
@@ -108,7 +108,7 @@ This directory contains the comprehensive technical documentation for the DEXBot
 
 ### 🧭 [Evolution Report](EVOLUTION.md)
 *Project timeline and major architecture phases.*
-- **Coverage**: Historical milestones from the initial December 2025 bootstrap through the v1.0.2 stable release, including credit maintenance hardening
+- **Coverage**: Historical milestones from the initial December 2025 bootstrap through the v1.0.3 stable release, including credit maintenance hardening
 - **Focus**: Architecture evolution, release history, test growth, and documentation changes
 
 ### 🗒️ [Changelog](../CHANGELOG.md)
diff --git a/market_adapter/README.md b/market_adapter/README.md
@@ -415,7 +415,7 @@ Per cycle, per processed bot, the adapter can produce:
 3. Resolve AMA settings from `profiles/market_profiles.json`.
 4. Read `startPrice` to choose the candle source: `pool`, `book`, or fixed.
 5. Sync candle data from Kibana or native BitShares data, using the selected source.
-6. Repair missing candle gaps when possible.
+6. Repair missing candle gaps: auto-fill gaps ≤24 candles directly (no Kibana), query Kibana only for gaps beyond that threshold. Kibana returning empty is treated as verified no-trade. Writes are suppressed while gaps remain unresolved.
 7. Ignore still-forming 1h candles.
 8. Compute AMA center, trend, ATR, weights, and collateral hint.
 9. Persist the first accepted center, compare center delta, and compare whitelisted range-scaling slope delta.
@@ -726,9 +726,15 @@ See also [research guide](../analysis/trend_detection/DYNAMIC_WEIGHT_RESEARCH.md
 ### Candle and Staleness Handling
 
 The adapter keeps candle caches current using Kibana bootstrap plus native
-incremental updates. It repairs gaps through targeted Kibana fetches when
-possible, prunes old candles to the required AMA window, and acts only on closed
-1h candles.
+incremental updates. Missing candle gaps are repaired in two steps: (1)
+**auto-fill** gaps ≤24 candles (trusted threshold) by carrying the preceding
+close forward with zero volume — Kibana is redundant since native fetch already
+confirmed no trades; (2) **Kibana query** for gaps >24 candles — real candles
+are merged in, or an empty response is treated as verified no-trade (all gaps
+in the queried window are synthesized). Gaps remaining after both steps
+suppress writes via `unresolved_candle_gaps` until repaired on a future cycle.
+The adapter prunes old candles to the required AMA window and acts only on
+closed 1h candles.
 
 #### AMA Warmup Window — Why Candle Length Matters
 
@@ -809,6 +815,7 @@ The two **calibration constants** live in `modules/constants.ts` under
 |----------|-------|---------|
 | `AMA_CONVERGENCE_ER_AVG` | `0.151` | Typical-market Efficiency Ratio. Lower = more conservative (assumes more noise, slower convergence, more candles needed). Calibrated against the fetched 3-year pool 133 1h dataset (`2023-05-07` -> `2026-05-06`); corrects for Jensen's inequality — `E[f(ER)] ≠ f(E[ER])` when `f` is the squaring function. |
 | `AMA_CONVERGENCE_EPSILON` | `0.01` | Target remaining bias fraction. `0.01` means 99 % of the initial bias has decayed by the end of the convergence window. |
+| `STALE_TAIL_THRESHOLD_CANDLES` | `24` | Trusted no-trade gap threshold. Gaps ≤24 candles are auto-filled without Kibana; gaps beyond query Kibana. Also used as stale-tail pruning threshold. |
 
 To recalibrate `AMA_CONVERGENCE_ER_AVG` against new market data, use the
 research script:
@@ -880,3 +887,5 @@ Important fields in `market_adapter/state/market_adapter_state.json`:
 | `weightVariance` | Normalized volatility ratio |
 | `weights` | Current dynamic buy/sell weights |
 | `collateralRecommendation` | Advisory collateral-ratio hint |
+| `kibanaGapRepairCount` | Gaps patched this cycle (auto-fill or Kibana-verified) |
+| `unresolvedGapCount` | Gaps still missing after all repair attempts; writes suppressed while > 0 |
diff --git a/market_adapter/core/market_adapter_service.ts b/market_adapter/core/market_adapter_service.ts
@@ -1620,6 +1620,30 @@ class MarketAdapterService {
                 ? deps.detectMissingCandleTimestamps(nextCandles, cfg.intervalSeconds)
                 : { gapCount: 0, missingTimestamps: [] };
 
+            // Step 1: Fill gaps within the trusted no-trade threshold directly (no Kibana needed).
+            // The native incremental fetch already confirmed no trades for these timestamps.
+            if (gapAnalysis.gapCount > 0) {
+                const filledNoTrade = this.fillVerifiedInternalNoTradeGaps(
+                    nextCandles,
+                    gapAnalysis.missingTimestamps,
+                    cfg.intervalSeconds,
+                    trustedNoTradeGapThresholdCandles
+                );
+                if (filledNoTrade.filledTimestamps.length > 0) {
+                    nextCandles = filledNoTrade.candles;
+                    kibanaGapRepairTimestamps = filledNoTrade.filledTimestamps.slice();
+                    logGapRepairEvent(
+                        `[market_adapter] ${bot.botKey}: synthesized ${filledNoTrade.filledTimestamps.length} no-trade candle(s) within trusted threshold `
+                        + `[${filledNoTrade.filledTimestamps.map((ts: any) => new Date(ts).toISOString()).join(', ')}]`
+                    );
+                }
+                // Re-detect gaps — only beyond-threshold gaps remain
+                gapAnalysis = typeof deps.detectMissingCandleTimestamps === 'function'
+                    ? deps.detectMissingCandleTimestamps(nextCandles, cfg.intervalSeconds)
+                    : { gapCount: 0, missingTimestamps: [] };
+            }
+
+            // Step 2: Only query Kibana for gaps beyond the trusted threshold
             if (gapAnalysis.gapCount > 0 && hasKibanaSource) {
                 const timeRange = this.buildGapRepairTimeRange(
                     gapAnalysis.missingTimestamps,
@@ -1628,12 +1652,8 @@ class MarketAdapterService {
                 );
                 if (timeRange) {
                     kibanaGapRepairAttempted = true;
-                    const verifiedGapTimestamps = this.getMissingTimestampsWithinTimeRange(
-                        gapAnalysis.missingTimestamps,
-                        timeRange
-                    );
                     logGapRepairEvent(
-                        `[market_adapter] ${bot.botKey}: detected ${gapAnalysis.gapCount} unresolved candle gap(s); `
+                        `[market_adapter] ${bot.botKey}: ${gapAnalysis.gapCount} candle gap(s) beyond trusted threshold; `
                         + `requesting Kibana repair for ${timeRange.gte} -> ${timeRange.lte} `
                         + `missing=[${gapAnalysis.missingTimestamps.map((ts: any) => new Date(ts).toISOString()).join(', ')}]`
                     );
@@ -1656,30 +1676,38 @@ class MarketAdapterService {
                             const beforeTimestamps = new Set(nextCandles.map((c: any) => c[0]));
                             nextCandles = deps.mergeCandles(nextCandles, kibanaGapCandles);
                             const afterTimestamps = new Set(nextCandles.map((c: any) => c[0]));
-                            kibanaGapRepairTimestamps = gapAnalysis.missingTimestamps.filter((ts: any) => !beforeTimestamps.has(ts) && afterTimestamps.has(ts));
+                            const kibanaPatchedTs = gapAnalysis.missingTimestamps.filter((ts: any) => !beforeTimestamps.has(ts) && afterTimestamps.has(ts));
+                            kibanaGapRepairTimestamps = [...kibanaGapRepairTimestamps, ...kibanaPatchedTs];
                             logGapRepairEvent(
-                                `[market_adapter] ${bot.botKey}: Kibana gap repair patched ${kibanaGapRepairTimestamps.length}/${gapAnalysis.gapCount} gap(s)`
-                                + (kibanaGapRepairTimestamps.length > 0
-                                    ? ` [${kibanaGapRepairTimestamps.map((ts: any) => new Date(ts).toISOString()).join(', ')}]`
+                                `[market_adapter] ${bot.botKey}: Kibana gap repair patched ${kibanaPatchedTs.length}/${gapAnalysis.gapCount} gap(s)`
+                                + (kibanaPatchedTs.length > 0
+                                    ? ` [${kibanaPatchedTs.map((ts: any) => new Date(ts).toISOString()).join(', ')}]`
                                     : '')
                             );
                         } else {
-                            logGapRepairEvent(
-                                `[market_adapter] ${bot.botKey}: Kibana gap repair returned no candles for the requested gap window`,
-                                'warn'
+                            // Kibana returned no data — that IS verification: no trades happened.
+                            // Synthesize no-trade candles for all gaps within the queried window.
+                            const verifiedGapTimestamps = this.getMissingTimestampsWithinTimeRange(
+                                gapAnalysis.missingTimestamps,
+                                timeRange
                             );
                             const verifiedNoTrade = this.fillVerifiedInternalNoTradeGaps(
                                 nextCandles,
                                 verifiedGapTimestamps,
                                 cfg.intervalSeconds,
-                                trustedNoTradeGapThresholdCandles
+                                verifiedGapTimestamps.length
                             );
                             if (verifiedNoTrade.filledTimestamps.length > 0) {
                                 nextCandles = verifiedNoTrade.candles;
-                                kibanaGapRepairTimestamps = verifiedNoTrade.filledTimestamps.slice();
+                                kibanaGapRepairTimestamps = [...kibanaGapRepairTimestamps, ...verifiedNoTrade.filledTimestamps];
                                 logGapRepairEvent(
                                     `[market_adapter] ${bot.botKey}: synthesized ${verifiedNoTrade.filledTimestamps.length} no-trade candle(s) after empty Kibana repair `
-                                    + `[${verifiedNoTrade.filledTimestamps.map((ts) => new Date(ts).toISOString()).join(', ')}]`
+                                    + `[${verifiedNoTrade.filledTimestamps.map((ts: any) => new Date(ts).toISOString()).join(', ')}]`
+                                );
+                            } else {
+                                logGapRepairEvent(
+                                    `[market_adapter] ${bot.botKey}: Kibana empty but could not synthesize gaps (missing neighbors)`,
+                                    'warn'
                                 );
                             }
                         }
diff --git a/package-lock.json b/package-lock.json
diff --git a/package.json b/package.json
@@ -1,6 +1,6 @@
 {
   "name": "dexbot2",
-  "version": "1.0.2",
+  "version": "1.0.3",
   "description": "The first open source trading bot with zero runtime dependencies and a fully adaptive market making strategy.",
   "main": "dist/modules/chain_orders.js",
   "bin": {
diff --git a/tests/test_market_adapter_service.ts b/tests/test_market_adapter_service.ts

Original file line number	Diff line number	Diff line change
`@@ -1,6 +1,6 @@`
`1`	`1`	`{`
`2`	`2`	`"name": "ama_fitting",`
`3`		`- "version": "1.0.2",`
	`3`	`+ "version": "1.0.3",`
`4`	`4`	`"description": "Tools for fitting AMA parameters to market data",`
`5`	`5`	`"main": "../../dist/analysis/ama_fitting/optimizer_high_resolution.js",`
`6`	`6`	`"scripts": {`
Original file line number	Diff line number	Diff line change
`@@ -1,6 +1,6 @@`
`1`	`1`	`{`
`2`	`2`	`"name": "dexbot2-claw",`
`3`		`- "version": "1.0.2",`
	`3`	`+ "version": "1.0.3",`
`4`	`4`	`"private": true,`
`5`	`5`	`"description": "Claw bridge and runtime integration layer for BitShares and DEXBot2.",`
`6`	`6`	`"type": "commonjs",`
Original file line number	Diff line number	Diff line change
`@@ -1,7 +1,7 @@`
`1`	`1`	`{`
`2`	`2`	`"name": "bitshares-claw-openclaw-plugin",`
`3`	`3`	`"private": true,`
`4`		`- "version": "1.0.2",`
	`4`	`+ "version": "1.0.3",`
`5`	`5`	`"type": "commonjs",`
`6`	`6`	`"main": "index.js"`
`7`	`7`	`}`