diff --git a/.gitignore b/.gitignore index 39c7d64a6..843715bce 100644 --- a/.gitignore +++ b/.gitignore @@ -45,6 +45,9 @@ packages/ibkr/ref/CMakeLists.txt # Turborepo .turbo/ +# Local implementation worktrees +.worktrees/ + # Vercel CLI local config (project id, scope) — generated by `vercel link` .vercel diff --git a/docs/PRD.md b/docs/PRD.md new file mode 100644 index 000000000..81e2bf211 --- /dev/null +++ b/docs/PRD.md @@ -0,0 +1,298 @@ +# JMB Goldmine Product Requirements Document + +**Product:** JMB Goldmine +**Version:** 1.0 +**Status:** Working baseline - pending explicit approval of promotion thresholds and MT5 EA design +**Last updated:** 2026-06-26 +**Product strategy:** Research-first, paper-readiness before automated execution + +## 1. Purpose + +JMB Goldmine is a local-first AI trading research and decision-support workspace. It brings a user's broker-connected market context, research, evidence, and proposed trade actions into one private application while keeping the user in control of any execution. + +**Branding note:** JMB Goldmine is the product name. Legacy technical identifiers such as `OPENALICE_*`, `.openalice`, workspace CLI names, and MT5 Common Files folders remain unchanged for compatibility with existing data, workspace configuration, and read-only MT5 bridge installations. + +This PRD is the planning boundary for the JMB Goldmine research milestone, not a rewrite of every OpenAlice platform capability. The README may still describe OpenAlice as a broad end-to-end trading platform; this milestone is narrower and safety-first. A request that does not map to a goal, workflow, requirement, or explicitly approved change in this document should be clarified before implementation. The goal is not to suppress useful ideas; it is to stop the product becoming an expensive grab-bag of half-finished features. + +## 2. Product decision and scope boundary + +The repository contains two different maturity levels: + +- The existing JMB Goldmine platform supports multi-broker Unified Trading Accounts (UTAs), staged Trading-as-Git orders, guarded execution, agent workspaces, and automation. +- The MT5 Research Desk and bridge remain **research-only** evidence surfaces for HFM and IC Markets instruments. They cannot receive broker credentials, expose an execution endpoint, or submit orders. +- The current app implementation includes a local `/research` dashboard, artifact-backed baseline and walk-forward summaries, a fixed-matrix experiment ledger, and a read-only MT5 demo bridge heartbeat. Separately, the locally installed and independently approved Plan 3 EA may execute Gold only on the exact allowlisted HFM and IC Markets demo accounts after local operator enablement; it exposes no remote order command. + +To resolve that conflict, this PRD defines the next product milestone as **Research-First / Paper-Readiness**. Existing UTA and workspace functionality remains supported platform capability, but no new live-trading or autonomous-execution promise is implied by this milestone. + +The broader platform remains experimental and its cross-broker execution/automation paths should be treated as beta capabilities until they have their own release criteria. This PRD does not require completing, marketing, or expanding those paths as a prerequisite for the research milestone. + +### Current architecture constraints + +- Alice and UTA are split into separate processes. Alice owns workspaces, research, market/news/analysis surfaces, and the user-facing web app; UTA owns broker credentials, broker adapters, order state, guard checks, FX, and snapshots. +- Alice reaches UTA through a loopback HTTP SDK. The current v1 safety boundary relies on local binding and same-host trust; a public or remote UTA carrier needs its own auth and deployment acceptance criteria before it is in scope. +- The in-process model loop is gone. AI work runs inside native workspace CLIs and scheduled jobs spawn headless Workspaces that report through the Inbox. +- Research Desk evidence must stay outside execution authority even though the wider platform still supports staged UTA orders with explicit user approval. + +### In scope for this milestone + +1. A local Research Desk that truthfully shows broker-specific data availability, data quality, backtest evidence, validation status, latest completed-bar observations, and relevant news. +2. A repeatable, auditable research workflow for the initial instruments: + - Gold / USD: HFM research/live `XAUUSDb` (current demo `XAUUSD`), IC Markets `XAUUSD` + - Euro / USD: HFM `EURUSDb`, IC Markets `EURUSD` +3. Controlled promotion from raw data to a paper-trading candidate, with explicit gates and durable evidence. +4. Clear separation between research, paper trading, and live trading. +5. Preservation of the existing UTA safety model for any supported broker execution: staging, guard checks, explicit approval, audit history, and account isolation. + +### Explicitly out of scope for this milestone + +- A profitable strategy claim, signal-selling service, or investment advice. +- Autonomous MT5 live trading, self-modifying live strategies, or an AI agent in the tick-by-tick execution path. +- Enabling an Expert Advisor (EA) to trade solely because one backtest was positive. +- Adding new brokers, asset classes, social/community features, mobile apps, or a multi-tenant cloud service. +- Replacing broker-native risk controls with an LLM. +- Using data from one broker to represent fills, costs, or performance at another broker without conspicuous labelling. + +## 3. Users and jobs to be done + +### Primary user: single, technically capable retail trader/researcher + +The user runs JMB Goldmine on their own machine, controls their broker accounts and AI-provider access, and wants a disciplined way to investigate a strategy without trusting a black-box bot. + +| Job | Desired outcome | +| --- | --- | +| Gather broker data | Know exactly which data was exported, from which broker and symbol, and whether it is usable. | +| Evaluate an idea | Run reproducible research with train/validation/holdout separation and realistic costs. | +| Understand current evidence | See what passed, failed, is incomplete, and the next required step without interpreting raw files. | +| Use AI for research | Ask an agent to analyse information while keeping research distinct from authority to trade. | +| Trade only with control | If using existing UTA capability, review a staged order, see guard results, and explicitly approve execution. | +| Recover or audit | Reconstruct why a decision or order occurred from persistent artifacts and history. | + +### Secondary user: developer/operator + +The operator needs stable local installation, diagnosable failures, isolated credentials, and enough written constraints to extend the application without accidentally broadening trading authority. + +## 4. User experience and core workflows + +### A. Evidence-led research workflow (MVP) + +1. The user connects or exports history from an MT5 terminal using the exact configured broker symbol. +2. OpenAlice stores/reads the local export and records source, coverage, contract metadata, gaps, and validation outcome. +3. The user runs a versioned baseline study. The study records data range, strategy version, parameters, cost assumptions, training selection, validation, untouched holdout, and decision. +4. The Research Desk displays only facts supported by those artifacts: availability, data quality, baseline and walk-forward results, evidence grade, latest **completed** bar observation, read-only bridge state where available, experiment ledger entries, and outstanding gates. +5. A failed holdout is shown as rejected; it may not be silently promoted or presented as a signal. +6. An evidence candidate can advance only after its required validation gates are complete. It remains research-only until a separately configured demo/paper execution integration exists. + +### B. Existing UTA trade-control workflow (platform capability) + +1. The user configures an isolated UTA for one broker/account. +2. An agent or user researches and constructs a proposed order. +3. The order is staged, committed with a human-readable reason, and evaluated by account guards. +4. The user explicitly approves the push to the broker. +5. OpenAlice records broker response, order lifecycle, account snapshots, and execution history. +6. The user can inspect, cancel, amend, or reject work where broker capabilities allow. + +**Rule:** Research Desk evidence must never directly create, stage, approve, or push a broker order. + +For an existing manual order form that combines stage, commit, and push, the final clearly labelled confirmation is the user's explicit approval. The UI must make that consequence unambiguous before submission; it must never be mistaken for a draft or research action. + +### C. Promotion workflow + +```mermaid +flowchart LR + A[Broker-specific export] --> B[Data quality checks] + B --> C[Versioned baseline study] + C --> D[Walk-forward + cost model] + D --> E[Demo forward test] + E --> F[Human review] + F --> G[Separately configured paper EA] + G --> H[Separate decision for limited live use] +``` + +Every arrow requires a saved artifact and an explicit recorded decision. A failure returns the candidate to research or rejects it; it never skips forward. + +## 5. Functional requirements + +### R1. Local-first and single-user operation + +- The application must run locally by default and bind sensitive services to localhost by default. +- Broker credentials must remain outside the agent runtime wherever the architecture permits; the UTA/carrier owns broker connectivity. +- The app is single-user. Multi-user sharing, role management, and tenancy are not implied. + +### R2. Research artifact integrity + +- Research artifacts must be stored outside deployable application files, in a persistent local data directory. +- Each artifact must identify broker, exact broker symbol, canonical instrument, time range, creation time, and schema/version. +- A study must include strategy/version identifier, parameters, cost assumptions, data source, training/validation/holdout windows, performance measures, and a human-readable conclusion. +- Missing, unreadable, stale, or malformed artifacts must produce a visible unavailable/failed state, never fabricated data. + +### R3. Data quality and provenance + +- MT5 export validation must detect and report coverage, duplicates, timestamp ordering, malformed rows, and material gaps. +- Contract metadata must include at least digits, point, contract size, minimum/maximum/step volume, stop distance, and trading mode when supplied by the terminal. +- The actual broker symbol must travel with all exports, reports, displayed results, and later orders. +- Data with incorrect resolution or known fallback history must be excluded from calculations and labelled as excluded. + +### R4. Research Desk + +- Provide a read-only `/research` view that clearly says research-only and trading off. +- Per configured instrument, show export availability, date/file coverage, artifact availability, baseline/holdout results, walk-forward status, evidence grade, latest completed-bar observation, read-only bridge state when available, and the next validation requirement. +- Display recent news only as contextual research material, not as a trade command or profitability forecast. +- State that evidence grades reflect completed validation, not probability of profit. +- Refreshing must not mutate data, broker state, or trading state. + +### R5. Strategy evaluation discipline + +- Time-series research must preserve chronology; no random shuffling of future observations into training. +- Parameter selection must be isolated from the final untouched holdout period. +- Walk-forward evaluation and broker-specific cost modelling are required before promotion to a demo candidate. +- The system must record a rejection when acceptance criteria are not met. A rejected candidate cannot appear as approved or recommended. +- Current observed research status is broker-specific and must be displayed as evidence, not endorsement: + +| Broker / symbol | Current status | +| --- | --- | +| HFM `EURUSDb` | Rejected by historical holdout / walk-forward evidence. | +| HFM `XAUUSDb` | Early historical research candidate only; broker-specific cost model, review, and demo forward test remain required. | +| IC Markets `EURUSD` | Rejected when its own artifacts are present; otherwise waiting for export/artifact evidence. | +| IC Markets `XAUUSD` | Early historical research candidate when its own artifacts are present; otherwise waiting for export/artifact evidence. Broker-specific cost model, review, and demo forward test remain required. | + +These are records of evidence, not product endorsements. + +### R6. Safety boundary for the approved broker-local MT5 demo EA + +- The [approved Plan 3 design](superpowers/specs/2026-07-13-jmb-goldmine-demo-canary-execution-design.md) governs the separately installed broker-local EA. The EA owns tick processing, position sizing, and order submission locally; OpenAlice/AI cannot be on the tick-time execution path and exposes no remote order command. +- Plan 3 is HFM-first. IC Markets may be enabled only after reviewed HFM request, broker-result, stop-protection, durable-log, and restart-reconciliation evidence. +- `XAUUSD` is the only execution-eligible instrument. EURUSD remains shadow-only, all non-demo accounts are blocked, and Plan 3 has no live-mode input or live-account eligibility. +- Status-only mode, execution disabled, and the persistent kill switch are the defaults. Demo execution requires deliberate local operator policy and EA-input changes on the exact account binding. +- The EA must enforce an instrument allowlist, risk-per-trade, total open risk, daily-loss, consecutive-loss, spread, slippage, session/news, and one-position-per-symbol limits. +- A persistent kill switch must block new entries; closing positions requires separately explicit configuration. +- Broker/account-specific magic numbers and logs are required for auditability. +- These requirements require automated tests plus an operator-only MetaEditor compile, harness, status-only observation, and broker-evidence ceremony; automation cannot mark those human gates complete. + +### R7. Existing UTA execution protections + +- App-, API-, UI-, workspace-, and AI-managed broker orders must go through a UTA, not directly from AI tools or the UI. +- The independently approved broker-local MT5 Gold demo EA is governed by R6 rather than the app-managed UTA order flow. It reads local artifacts, binds exact demo accounts inside MT5, and exposes no remote order command to the app, API, Research Desk, scheduler, Codex, or AI. +- Orders must pass configured guards before broker submission. +- User approval is required before execution. Agent text, schedules, research results, or news alone cannot constitute approval. +- A manual one-step order form may use its final confirmation as approval only when it explicitly names the affected account, instrument, order details, estimated exposure, and that submission will send the order to the broker. +- The system must persist staged work, commits/rejections, guard outcomes, broker results, and snapshots. +- Guard or broker failures must be actionable and must not leave the UI reporting a successful order unless the broker state supports it. + +### R8. AI workspace and automation boundaries + +- AI workspaces may research, create reports, and surface results through the Inbox. +- Scheduled headless workspaces may report findings but may not bypass the execution approval boundary. +- Prompt/tool context should be scoped to the task; persistent research summaries and artifacts should be referenced instead of repeatedly rediscovering the same context. +- Credentials and capability exposure must be explicit; a workspace receives only what it needs. + +### R9. Observability and recovery + +- Important operations must produce a durable, inspectable record with timestamps and account/instrument attribution. +- The user must be able to distinguish local cached data, broker data, hosted Hub data, and data with unknown or stale provenance. +- Errors must explain the action, cause when known, safe next step, and whether broker state may need external verification. +- Any data or execution path that cannot be verified must fail visibly rather than infer success. + +## 6. Non-functional requirements + +| Area | Requirement | +| --- | --- | +| Safety | Favor refusal and explicit user action over silent fallback in any trading-related path. | +| Privacy | Treat broker keys, positions, trade history, and AI credentials as private local data. | +| Reliability | Preserve research and trade evidence across app upgrades and restarts. | +| Maintainability | Keep domain logic outside UI components; use typed schemas at boundaries; avoid duplicated broker logic. | +| Explainability | Results, guard blocks, data quality states, and promotion decisions must be understandable without reading source code. | +| Testability | Risk gates, promotion rules, data validators, and broker adapters need automated tests plus broker/paper acceptance scenarios where relevant. | +| Performance | Research view should remain responsive with local artifact summaries; large raw data must be processed in bounded/streamed jobs, not loaded wholesale into the UI. | +| Accessibility | State is conveyed with text as well as color; empty, error, and loading states are clear. | + +## 7. Acceptance criteria for the milestone + +The milestone is complete only when all of the following are true: + +- [ ] The Research Desk has a clear research-only, no-trading boundary and no route/tool that can submit an order from that view. +- [ ] HFM `XAUUSDb` and `EURUSDb` exports show broker-specific availability and quality status; excluded fallback data is not treated as eligible M1 data. +- [ ] IC Markets instruments remain visibly waiting until their own exports and reports exist; when their own artifacts are present, the Research Desk displays their broker-specific rejected/candidate status without borrowing HFM evidence. +- [ ] Each displayed baseline result has a persisted artifact with provenance and a separate training/holdout record. +- [ ] Failed holdouts are visibly rejected; positive early evidence is described as a candidate, not a recommendation. +- [ ] The documented promotion gates are displayed and cannot be represented as complete without their evidence. +- [ ] Regression tests cover artifact parsing/error states, evidence grading, no-order research route behavior, and the existing UTA approval/guard boundary. +- [ ] Any manual one-step order form has an explicit, test-covered final execution confirmation; a staged order is not displayed as filled until broker reconciliation/sync supports that state. +- [ ] There are zero Research Desk, scheduler, workspace, Codex, or AI execution endpoints for the Plan 3 EA; app/API/AI-managed orders continue to require UTA approval and guards. +- [ ] Stage 0 records both exact demo-account bindings locally, with execution disabled and kill switches on, without exposing either account login in repository or Research API artifacts. +- [ ] Each canary broker submission has a durable pre-request event and broker-confirmed stop protection; unknown results fail closed into reconciliation rather than resend. +- [ ] Reviewed HFM canary evidence exists before any IC Markets execution enablement or `ic_canary` promotion. +- [ ] The current MT5 data and training protocol remains accurate and links to this PRD. + +## 8. Success measures + +These measures describe confidence and product discipline, not investment return. + +| Measure | Target for this milestone | +| --- | --- | +| Provenance | 100% of displayed studies identify broker, exact symbol, source range, and artifact version. | +| Truthfulness | 0 cases where missing/unverified evidence is displayed as a completed result or live signal. | +| Research reproducibility | A developer can recreate a displayed result from its documented artifact inputs and study version. | +| Safety | 0 paths from Research Desk or scheduled research output to unapproved order execution. | +| Data quality | 100% of included data is marked eligible by the validator; exclusions are retained and explained. | +| Operator clarity | A user can identify the current stage and next blocking gate for every configured instrument from one page. | + +## 9. Prioritized roadmap + +### Now — baseline and research truthfulness + +1. Finalize the PRD and make it the planning entry point. +2. Harden research artifact schemas, validation, storage, and test coverage. +3. Complete the read-only Research Desk and document its data contracts. +4. Keep IC Markets evidence broker-specific: display observed artifacts when present, and do not infer equivalence from HFM data. + +### Next — paper-readiness evidence + +1. Add walk-forward reports and broker-specific spread, commission, financing, and slippage models. +2. Add a decision log for candidate/rejected/paused studies. +3. Define and test a paper-only MT5 EA interface with the risk gates in R6. +4. Run meaningful demo forward tests and review exceptions before any promotion discussion. + +### Later — only after an explicit PRD revision + +1. Limited live MT5 deployment with a separately approved risk policy and operator ceremony. +2. Additional broker/instrument support. +3. Remote/separated UTA carrier deployment, public mode, or richer automation capabilities. + +## 10. Decisions that require explicit approval before implementation + +The following are deliberately undecided. Do not silently choose one during feature work: + +| Decision | Why it matters | +| --- | --- | +| Exact promotion thresholds | A return or Sharpe threshold alone is insufficient; acceptable drawdown, sample size, robustness, and operational risk need a written policy. | +| Paper-test duration and sample size | This controls when a candidate may progress and cannot be guessed from a backtest. | +| MT5 EA technology/design | It affects deployment, logging, risk enforcement, and how the app communicates with MT5. | +| Live-trading eligibility | Requires separate risk, legal, operational, and user-confirmation requirements. | +| Any broker credentials available to AI | Changes the threat model and should be a conscious, minimized exception. | +| Product positioning | Decide whether OpenAlice is primarily a general trading platform, a research workstation, or both with separately versioned milestones. | + +## 11. Documentation contract + +- `README.md` remains the platform overview and must not present research candidates as trade recommendations. +- `docs/mt5-data-and-training-protocol.md` is the operational evidence and EA-risk protocol for this milestone. +- `docs/project-structure.md` must be refreshed before or alongside architecture work that changes the Alice/UTA split, workspace execution model, routes, connectors, or package layout. +- This PRD is the source of truth for product scope, priorities, acceptance criteria, and out-of-scope boundaries. +- If implementation changes any requirement, safety boundary, workflow, or promotion gate, update this PRD and the relevant operational document in the same change. + +## 12. Assumptions and open risks + +- The intended operator is a single, technically capable user running software locally; this is not a regulated advisory or managed-trading product. +- Broker APIs and MT5 history can be incomplete, delayed, or venue-specific; external/broker-side verification remains necessary. +- AI can assist with analysis and documentation but is not a source of market truth or authority to trade. +- Financial, operational, and security risks remain material even with all guards present. The product must communicate this plainly. + +## 13. Change-intake rule + +Before implementation, each feature request or bug fix should state: + +1. The PRD section/requirement it serves (or an explicit proposal to change this PRD). +2. The user problem and smallest acceptable outcome. +3. Any safety, data-provenance, migration, or documentation impact. +4. Testable acceptance criteria and what is intentionally not being changed. + +If a request spans both the Research Desk and execution, split it into separate changes. Research work must retain its read-only boundary; execution work needs an independently approved scope and safety review. diff --git a/docs/mt5-data-and-training-protocol.md b/docs/mt5-data-and-training-protocol.md new file mode 100644 index 000000000..01130dd4b --- /dev/null +++ b/docs/mt5-data-and-training-protocol.md @@ -0,0 +1,149 @@ +# MT5 Data And Training Protocol + +> Product scope, priorities, and acceptance criteria are defined in the [JMB Goldmine PRD](PRD.md). This document is the operational protocol for the MT5 research track and must stay consistent with that PRD. + +## Purpose + +This protocol defines the evidence required for research and for the separately approved Plan 3 broker-local Gold demo canary. It is deliberately narrower than an autonomous live-trading system: the goal is repeatable evidence and execution-plumbing validation, not a claim of profitability or live eligibility. + +## Instruments And Broker Mapping + +| Canonical instrument | HFM research/live | HFM demo | IC Markets MT5 | +| --- | --- | --- | --- | +| Gold / USD | `XAUUSDb` | `XAUUSD` | `XAUUSD` | +| Euro / USD | `EURUSDb` | `EURUSD` | `EURUSD` | + +Every export, backtest, bridge heartbeat, and order must retain both the canonical instrument and the exact broker symbol. The current HFM demo symbols (`XAUUSD`, `EURUSD`) are distinct from the HFM research/live export symbols (`XAUUSDb`, `EURUSDb`); the bridge must label those differences and a future EA must separately validate fills and costs on the demo contracts. Never use one broker's prices to model another broker's fills without labelling the result as a cross-broker experiment. + +## Data Standard + +1. Use the connected MT5 broker terminal as the primary source. +2. Export ten years of M1 OHLC, tick volume, real volume, and spread points. +3. Capture contract metadata: digits, point, contract size, volume limits, stop distance, and trading mode. +4. Validate coverage, duplicates, out-of-order rows, and large gaps. +5. Keep public or third-party data only as a labelled supplementary research source. + +The exporter requests one month at a time to avoid the terminal's chart-bar cap. Its M1 request is therefore small enough for a normal terminal configuration; do not use a single ten-year `CopyRates` request, because it can be silently truncated. + +Raw tick history is optional and must be requested in bounded windows. Ten years of ticks can consume tens or hundreds of gigabytes, may not be available from the broker, and is unnecessary for an initial bar-based strategy. + +### Observed HFM Export (2026-06-23) + +The HFM monthly export contained 3,877,671 parseable rows with no duplicate or malformed timestamps. The broker supplied a mixture of true M1 history and older daily-resolution fallback bars. The latter must not be used as M1 training data. + +| Symbol | Eligible M1 range | Excluded historical range | +| --- | --- | --- | +| `EURUSDb` | June 2020 (partial) through June 2026 | June 2016 through May 2020 daily-resolution fallback | +| `XAUUSDb` | September 2021 (partial) through June 2026 | June 2016 through August 2021 daily-resolution fallback | + +Current HFM contract metadata captured with the export: `XAUUSDb` has a `0.01` minimum volume and contract size `100`; `EURUSDb` has a `0.01` minimum volume and contract size `100000`. A future EA must re-read these values from the terminal at startup rather than trusting this snapshot. + +## Training And Backtest Discipline + +"Self-training" means model selection under controlled historical tests. It does not mean changing a live strategy because of its most recent winning or losing trades. + +1. Define a small, understandable baseline strategy and its risk model before optimisation. +2. Split time in order: training, validation, and a final untouched out-of-sample test. Do not shuffle time series. +3. Use walk-forward windows. Parameters may be selected on past data, then frozen for the following unseen period. +4. Include broker spread, commission, swap, minimum lot, stop distance, slippage assumptions, and trading-hour restrictions. +5. Compare performance across both brokers when possible. A model that survives only one data feed is not robust enough. +6. Reject strategies that depend on a tiny number of trades, have unstable parameters, fail after realistic costs, or only work in one market regime. +7. Keep a signed experiment record: data source, dates, strategy version, parameters, costs, result, and decision. + +### Rolling Walk-Forward Study + +The local `walk_forward_daily_trend.py` study uses a rolling 60-month training window and sequential six-month unseen windows by default. In each window it selects from the fixed broad lookbacks on training data only, freezes that choice, and records the resulting out-of-sample performance separately. Its 10-basis-point cost is a placeholder, not a broker-specific cost model. The report is evidence for review only and cannot promote a strategy to a demo EA on its own. + +### Fixed-Matrix Experiment Ledger + +`run_daily_trend_experiments.py` appends repeated research runs to `~/.openalice/data/research/daily-trend-experiment-ledger.json`. It compares a declared matrix of lookback sets and fixed cost assumptions, then logs every out-of-sample result for the Research Desk visual ledger. It is intentionally not a genetic optimiser and must not keep mining historical data until a desired win rate appears. + +HFM Gold must use the validated eligible M1 range beginning in 2021; if a 2019 start is requested, the system must reject or shorten the HFM study rather than blend in fallback-resolution history. IC Markets Gold can be evaluated from 2019 when its own exported files and validation support that range. + +The Steward's learning loop is journal-based: read past experiment runs, daily reviews, bridge status, and future demo logs; record lessons, suspected failure causes, protected-risk events, and the next bounded hypothesis. It may improve review discipline and test design, but it may not silently change live or paper risk rules. + +### Initial Trend-Following Baseline + +The first baseline is daily time-series momentum, evaluated across broad 60, 120, 180, and 252 trading-day lookbacks. It selects one lookback using data through 2023 and evaluates that frozen choice on 2024 onward. This is a research hypothesis, not a live signal or a claim of suitability. + +The choice is informed by Moskowitz, Ooi, and Pedersen's study of time-series momentum across liquid currency and commodity futures, which found persistence over one- to twelve-month horizons in its historical sample. That evidence is not a guarantee for HFM's retail `EURUSDb` or `XAUUSDb`, so the broker-specific holdout test remains the relevant gate. [Paper](https://pages.stern.nyu.edu/~lpederse/papers/TimeSeriesMomentum.pdf) + +### Initial Baseline Result (2026-06-23) + +Using a conservative illustrative 10-basis-point one-way cost and a 2018–2023 selection window, the daily trend baseline selected a 120-day lookback for `EURUSDb` and a 60-day lookback for `XAUUSDb`. The EUR/USD choice lost 11.3% in the untouched 2024–2026 holdout, so it is rejected. The gold choice gained 68.4% in that holdout but had a 21.5% maximum drawdown. It is historical research only; the rolling study, broker-specific cost and financing model, and a demo forward test remain mandatory before any EA may use it. + +### Observed Rolling Walk-Forward Result (2026-06-23) + +The initial rolling study used a five-year training window, then selected a lookback and tested it on the following six months. It produced seven sequential unseen test windows through 2026-06-23. These figures retain the illustrative 10-basis-point one-way cost and therefore are not broker-cost-adjusted or a promotion decision. + +| Broker / symbol | Aggregate unseen return | Sharpe | Maximum drawdown | Research status | +| --- | ---: | ---: | ---: | --- | +| HFM `EURUSDb` | -18.05% | -0.75 | -23.14% | Rejected | +| HFM `XAUUSDb` | 81.41% | 0.98 | -29.25% | Early candidate; cost model required | +| IC Markets `EURUSD` | -16.94% | -0.70 | -22.29% | Rejected | +| IC Markets `XAUUSD` | 79.60% | 0.97 | -30.23% | Early candidate; cost model required | + +The two EUR/USD studies are not eligible for an EA under this protocol. The two Gold/USD studies remain historical research candidates only: their drawdowns are material, and commission, spread, swap, slippage, and a meaningful paper forward test remain mandatory. + +## Research Dashboard + +The local OpenAlice **Research** page (`/research`) is a read-only evidence ledger. It combines MT5 export presence, backtest artifacts, validation stages, the latest completed daily trend observation, and OpenAlice's recent-news archive. It never receives broker credentials, cannot submit orders, and uses an evidence grade instead of an invented probability of profit. The installed Windows app reads persistent reports from `~/.openalice/data/research/`, so they are not lost when the application is upgraded. + +The dashboard's “latest trend” is derived from the last completed daily bar in the local backtest artifact. It is not a live quote or a trading instruction. IC Markets remains in a waiting state until its own `XAUUSD` and `EURUSD` history is exported and tested. + +## Approved Plan 3 Demo Canary Boundary + +The [approved Plan 3 design](superpowers/specs/2026-07-13-jmb-goldmine-demo-canary-execution-design.md) permits one narrow exception to the normal UTA execution path. The Research Desk and MT5 bridge remain read-only. A separately installed broker-local EA may submit protected `XAUUSD` orders only on the exact HFM and IC Markets demo accounts after local operator enablement. It exposes no app, API, Research, workspace, Codex, or AI order command. App/API/AI-managed orders remain subject to UTA guards and user approval. + +Plan 3 is HFM-first and IC-second. EURUSD remains shadow-only. The EA has no live-mode input or live-account bypass, and no Plan 3 evidence can promote a live account. Its safe defaults are `status_only`, `InpDemoExecutionEnabled=false`, and `InpKillSwitch=true`. + +## Required EA Risk Gates + +The Plan 3 EA must enforce these locally in MT5, independent of an AI service: + +- Only the configured broker symbols may be traded. +- Maximum risk per trade, maximum open risk, maximum daily loss, and maximum consecutive losses. +- Maximum spread and slippage thresholds. +- One-position-per-symbol rule until the strategy is proven. +- News/session blackout windows where relevant. +- A persistent kill switch that blocks new entries and can close positions only on explicit configuration. +- Unique EA magic numbers per broker and account. +- Status-only demo mode as the default; non-demo accounts and live execution are ineligible with no enabling setting. + +## Latency Boundary + +The MT5 EA owns tick processing, sizing, and order submission. OpenAlice or an AI agent may generate research and review reports, but must not sit in the tick-by-tick order path. This keeps the execution path local and deterministic; it does not guarantee a particular broker or network latency. + +## Stage 0 Readiness And Promotion Gates + +The exact copy layout, MetaEditor steps, inputs, Common Files paths, rollback, pause, and emergency checks are in the [MT5 operator guide](../tools/mt5/README.md#plan-3-broker-local-gold-demo-canary). Codex does not launch MetaEditor, run the MQL harness, attach an EA, enable Algo Trading, or submit a broker request. + +### Stage 0: status-only dry run + +1. The human operator recompiles the completed-D1 read-only bridge, policy script, `JmbGoldmineDemoCanary`, and no-order harness in both broker terminals. Every compile must report `0 errors, 0 warnings`, and the harness must end with `JMB_CANARY_HARNESS PASS` and zero failures. +2. The operator writes a `status_only`, candidate-unapproved policy on each exact demo `XAUUSD` chart, enters the exact expected demo account login locally, and attaches HFM and IC with `InpDemoExecutionEnabled=false` and `InpKillSwitch=true`. Algo Trading remains disabled. +3. The bridge stays on a duplicate Gold chart. Both `OpenAliceMt5ExecutionV1//XAUUSD/latest_status.csv` files must advance for at least two ten-second cycles; both bridge heartbeats must also remain current. The operator confirms zero new orders and zero new positions in both terminals. +4. Both `completed_d1.csv` files must contain a `bar_as_of` later than the stale `2026-06-23` artifact. File modification time alone is not evidence. The recompiled read-only bridge also refreshes the Gold `OpenAliceMt5TradeLedgerV1//XAUUSD/deals.csv` file on a timer so the learning gate does not depend on a forgotten one-shot exporter run. +5. Every new canary gate is compared with the existing `JmbGoldmineDemoRiskShell` output. A mismatch, missing artifact, wrong identity, or uncertain broker state leaves both brokers execution-disabled and triggers rollback to the status-only shell. + +This two-terminal observation is a pending human gate, not an automated test result. + +### HFM canary + +After Stage 0 evidence is reviewed, the operator builds an HFM-only `canary_ready` cost model, writes an operator-approved `hfm_canary` policy, reconfirms the local HFM login/server/symbol/magic binding, and deliberately enables HFM demo execution and turns off its kill switch. IC Markets remains `status_only`. + +Promotion evidence requires one HFM decision with a durable pre-request event, broker result, broker-confirmed protective stop, append-only lifecycle evidence, and successful restart reconciliation. Any unprotected fill, unknown or partial result, reconciliation mismatch, non-demo identity, missing cost evidence, or duplicate uncertainty returns HFM to `status_only`. No automatic resend is allowed. + +### IC Markets canary + +Only after the HFM evidence is accepted may the operator build an IC-only `canary_ready` cost model, write an `ic_canary` policy, and locally enable the IC Markets demo EA. The same evidence chain is required, including IC-specific spread, deviation, volume, server, costs, identifiers, protective stop, and restart behavior. HFM must remain protected and reconciled or be paused. + +Only after both named ceremonies pass may the operator write `both_demo` policies. Each broker remains independently bound and pausable under its own gates. EURUSD remains shadow-only, and live execution remains absent. + +### Pause, rollback, and recovery evidence + +- Pause with the local kill switch on, execution disabled, and a new `status_only` policy. The kill switch blocks new entries but does not close a correctly protected position. +- Roll back by capturing execution and broker evidence, detaching the canary from the Gold status chart, and reattaching `JmbGoldmineDemoRiskShell` with its kill switch on. Leave the read-only bridge attached to its duplicate chart and preserve all event, processed-observation, and reconciliation files. +- Treat an unknown broker result as reconciliation-required, never as a rejection or permission to resend. Verify orders, deals, positions, and stops in MT5 before trusting local status. +- If protection is absent, the EA may attempt its narrowly defined emergency close of the EA-owned position and must persist a protection-error pause. The operator verifies the broker is protected or flat and leaves the broker at `status_only`; a persistent latch is never deleted merely to resume. +- A paused or restarted EA returns with execution disabled and kill switch on for at least two stable status cycles before any human review of re-enablement. diff --git a/docs/superpowers/plans/2026-07-13-jmb-goldmine-demo-canary-execution.md b/docs/superpowers/plans/2026-07-13-jmb-goldmine-demo-canary-execution.md new file mode 100644 index 000000000..0549249de --- /dev/null +++ b/docs/superpowers/plans/2026-07-13-jmb-goldmine-demo-canary-execution.md @@ -0,0 +1,1421 @@ +# JMB Goldmine Demo Canary Execution Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add deterministic, broker-local, Gold-only demo canary execution for HFM first and IC Markets second, with fresh completed-D1 observations, stable decision identity, fail-closed risk gates, reconciliation, read-only UI status, and no live execution path. + +**Architecture:** The existing read-only MT5 bridge exports fresh completed D1 bars. A deterministic TypeScript service derives one stable execution-decision lease per completed observation and publishes it every five minutes through the internal `Pump`, without an LLM, shell, or order API. A new modular MT5 EA independently validates local policy and broker state, submits at most one protected demo order, reconciles authoritative MT5 state, and writes append-only execution evidence for the Research Desk. + +**Tech Stack:** TypeScript 5.9, Node.js file APIs, Vitest 4, Hono, React 19, MQL5, MT5 Common Files CSV/JSONL. + +## Operational Note — 2026-07-14 + +- HFM and IC Markets XAUUSD read-only bridges are publishing current Common Files status. +- HFM and IC Markets canary EAs compile cleanly and publish `OpenAliceMt5ExecutionV1//XAUUSD/latest_status.csv`. +- The no-order harness passed, then its compiled `.ex5` copies were disabled in the operator MT5 folders so it does not run during forward monitoring. +- A false `reconciliation_required` state was traced to MQL5 `ZeroMemory` leaving latch string fields as `(null)`; `InitializeCanarySafetyLatch` now explicitly clears every latch field. +- Current broker-local state is safe-disabled: `state=disabled`, `execution_enabled=0`, `kill_switch=1`; enabling demo evaluation/trading still requires a separate approved demo canary ceremony. + +## Global Constraints + +- Source design: `docs/superpowers/specs/2026-07-13-jmb-goldmine-demo-canary-execution-design.md`. +- `PRD.txt` is absent; the project source of truth is `docs/PRD.md` per the approved design. +- Documentation conflict to resolve in Task 10: `docs/PRD.md` R6 describes a broker-local MT5 EA, while R7 says app-managed broker execution uses UTA approval. The approved Plan 3 design governs this separate demo-only EA; the PRD must explicitly distinguish it from app/AI-controlled UTA orders before Stage 1. +- Demo accounts only; the EA must have no live-mode input or live-account bypass. +- Execution instrument is exactly `XAUUSD`; EURUSD remains shadow-only. +- Strategy allowlist is exactly `daily-trend-v1`. +- Maximum volume is exactly `0.01` lot. +- HFM server is exactly `HFMarketsGlobal-Demo4`; IC Markets server is exactly `ICMarketsSC-Demo`. +- HFM magic number is `880101`; IC Markets magic number is `880201`. +- Expected demo account login is an operator-entered local EA input; it is never committed or returned by the Research API. +- Maximum per-trade demo risk is `10.00` account-currency units. +- Maximum daily realized demo loss is `40.00` account-currency units. +- Four losing EA-owned Gold trades pause new entries until the next broker day. +- HFM maximum spread/deviation is `0.75`/`0.50` price units; IC Markets is `0.30`/`0.30`. +- Entry session is Monday-Thursday `06:00-20:00 UTC` and Friday `06:00-16:00 UTC`. +- High-impact USD news blocks entry from 30 minutes before through 30 minutes after the event. +- Estimated post-order free margin must retain at least a ten-times margin buffer. +- Execution defaults are `InpDemoExecutionEnabled=false` and `InpKillSwitch=true`. +- No martingale, grid, recovery sizing, pyramiding, automatic lot growth, fixed take-profit, or AI-controlled risk change. +- AI journal/import code may summarize reconciled evidence but cannot write policy, strategy, EA-input, processed-ID, or broker-state roots. +- Research Desk remains read-only and must never expose account login or add an execution control endpoint. +- Do not launch MetaEditor through PowerShell; the operator compiles manually because Kaspersky previously blocked that automation path. +- Preserve unrelated dirty-worktree changes and commit only files listed by each task. + +--- + +## File Structure + +### TypeScript domain and service files + +- Create `src/domain/mt5/completed-d1.ts`: strict completed-D1 CSV parser and trend observation derivation. +- Create `src/domain/mt5/completed-d1.spec.ts`: malformed, stale, unsafe, ordering, and lookback tests. +- Create `src/domain/mt5/demo-execution-policy.ts`: strict read-only policy parser and hard-ceiling validation. +- Create `src/domain/mt5/demo-execution-policy.spec.ts`: rollout, allowlist, and ceiling tests. +- Create `src/domain/mt5/broker-cost-model.ts`: deterministic observed cost-model builder/parser. +- Create `src/domain/mt5/broker-cost-model.spec.ts`: spread sample and ledger evidence tests. +- Create `src/domain/mt5/local-paths.ts`: one environment-aware resolver for every JMB MT5 Common Files root. +- Create `src/domain/mt5/local-paths.spec.ts`: explicit-root, default-root, and missing-home tests. +- Create `src/domain/mt5/execution-decision.ts`: isolated execution-decision schema, stable IDs, atomic latest lease, append-once journal. +- Create `src/domain/mt5/execution-decision.spec.ts`: schema, identity, atomicity, and regression tests. +- Create `src/domain/mt5/demo-decision-engine.ts`: pure fail-closed execution-decision builder. +- Create `src/domain/mt5/demo-decision-engine.spec.ts`: Gold eligibility and EURUSD block tests. +- Create `src/domain/mt5/demo-decision-service.ts`: one deterministic four-instrument decision cycle. +- Create `src/domain/mt5/demo-decision-service.spec.ts`: per-broker isolation, deduplication, and stale-observation tests. +- Create `src/task/mt5-decision-scheduler.ts`: private five-minute `Pump` lifecycle. +- Create `src/task/mt5-decision-scheduler.spec.ts`: immediate catch-up, serial cadence, backoff, and stop tests. +- Create `src/domain/mt5/execution-status.ts`: fail-closed EA latest-status read model. +- Create `src/domain/mt5/execution-status.spec.ts`: lifecycle and privacy tests. +- Create `src/domain/mt5/execution-outcomes.ts`: append-only execution-event parser and reconciled learning import. +- Create `src/domain/mt5/execution-outcomes.spec.ts`: duplicate, incomplete, closed, stopped, and privacy tests. +- Create `src/task/mt5-outcome-importer.ts`: deterministic broker-outcome import cycle with no policy write access. +- Create `src/task/mt5-outcome-importer.spec.ts`: restart idempotency and per-broker isolation tests. +- Modify `src/main.ts`: start and stop the internal MT5 decision scheduler. +- Modify `src/webui/routes/research.ts`: add read-only execution projection. +- Create `src/webui/routes/research.spec.ts`: route state and account-login non-disclosure tests. + +### Operator and MT5 files + +- Modify `tools/mt5/OpenAliceMt5ReadOnlyBridge.mq5`: atomically export last 300 completed D1 bars and append bounded spread samples. +- Create `tools/mt5/ConfigureJmbGoldmineDemoPolicy.mq5`: operator-only policy writer with status-only default. +- Create `tools/mt5/run_demo_canary_decisions.ts`: manual diagnostic wrapper around the deterministic service. +- Create `tools/mt5/JmbGoldmineDemoCanary/JmbGoldmineDemoCanary.mq5`: thin EA lifecycle orchestration. +- Create `tools/mt5/JmbGoldmineDemoCanary/JmbCanaryTypes.mqh`: protocol types and lifecycle enums. +- Create `tools/mt5/JmbGoldmineDemoCanary/JmbCanaryCsv.mqh`: strict parsing, escaping, atomic status, and flushed JSONL. +- Create `tools/mt5/JmbGoldmineDemoCanary/JmbCanaryPolicy.mqh`: hard ceilings and identity/policy validation. +- Create `tools/mt5/JmbGoldmineDemoCanary/JmbCanaryGates.mqh`: pure gate evaluation. +- Create `tools/mt5/JmbGoldmineDemoCanary/JmbCanaryState.mqh`: processed observations, instance lock, and transition reducer. +- Create `tools/mt5/JmbGoldmineDemoCanary/JmbCanaryTradeGateway.mqh`: the only file allowed to contain `OrderCheck` or `OrderSend`. +- Create `tools/mt5/JmbGoldmineDemoCanary/JmbCanaryReconcile.mqh`: authoritative order/position/deal and daily-loss reconciliation. +- Create `tools/mt5/tests/JmbGoldmineDemoCanaryHarness.mq5`: no-order table-driven MQL harness. +- Create `tools/mt5/tests/README.md`: manual compile/run expectations. +- Modify `tools/mt5/README.md`: installation, policy, dry-run, rollout, rollback, and recovery. + +### UI files + +- Modify `ui/src/api/research.ts`: execution lifecycle contract. +- Create `ui/src/components/research/Mt5ExecutionStatusCard.tsx`: focused read-only status card. +- Create `ui/src/components/research/__tests__/Mt5ExecutionStatusCard.spec.tsx`: lifecycle rendering and privacy tests. +- Modify `ui/src/pages/ResearchDashboardPage.tsx`: integrate the status card without adding controls. +- Create `ui/src/demo/handlers/research.ts`: typed demo Research response. +- Modify `ui/src/demo/handlers/index.ts`: register Research handler before catch-all. + +--- + +### Task 1: Export and parse fresh completed D1 broker bars + +**Files:** +- Modify: `tools/mt5/OpenAliceMt5ReadOnlyBridge.mq5` +- Create: `src/domain/mt5/completed-d1.ts` +- Create: `src/domain/mt5/completed-d1.spec.ts` +- Test: `src/domain/mt5/completed-d1.spec.ts` + +**Interfaces:** +- Produces: `readMt5CompletedD1(root, broker, symbol, options: { now?: Date; maxAgeHours: number }) => Promise`. +- Produces: `deriveCompletedTrendObservation(parsed, lookbackDays) => CompletedTrendObservation`. +- MT5 file: `OpenAliceMt5BridgeV1///completed_d1.csv`. +- Spread file: `OpenAliceMt5BridgeV1///spread_samples_YYYYMMDD.csv`. + +- [ ] **Step 1: Write failing parser and observation tests** + +```ts +import { describe, expect, it } from 'vitest' +import { deriveCompletedTrendObservation, parseCompletedD1Csv } from './completed-d1.js' + +function csv(closes: number[]): string { + const rows = closes.map((close, index) => + `1,2026-07-13T09:00:00.000Z,hfmarkets,HFMarketsGlobal-Demo4,demo,XAUUSD,2026-05-${String(index + 1).padStart(2, '0')},${index + 1},${close},${close},${close},${close}`, + ) + return ['schema_version,captured_at,broker,server,account_mode,symbol,bar_as_of,bar_open_epoch,open,high,low,close', ...rows].join('\n') +} + +describe('completed D1 broker bars', () => { + it('derives the signal from completed bars only', () => { + const parsed = parseCompletedD1Csv(csv([100, 101, 103])) + expect(deriveCompletedTrendObservation(parsed, 2)).toMatchObject({ + direction: 'uptrend', + lookbackDays: 2, + latestClose: 103, + referenceClose: 100, + }) + }) + + it('rejects duplicate or descending bar epochs', () => { + expect(() => parseCompletedD1Csv(csv([100, 101]).replace(',2,101,', ',1,101,'))).toThrow(/ascending/) + }) +}) +``` + +- [ ] **Step 2: Run the test and verify RED** + +Run: `pnpm vitest run src/domain/mt5/completed-d1.spec.ts` + +Expected: FAIL because `completed-d1.ts` does not exist. + +- [ ] **Step 3: Implement the strict TypeScript parser** + +```ts +export type CompletedD1State = 'ready' | 'missing' | 'stale' | 'unsafe' | 'malformed' +export type TrendDirection = 'uptrend' | 'downtrend' | 'flat' + +export interface CompletedD1Bar { + asOf: string + openEpoch: number + open: number + high: number + low: number + close: number +} + +export interface ParsedCompletedD1 { + capturedAt: string + broker: string + server: string + accountMode: string + symbol: string + bars: CompletedD1Bar[] +} + +export interface Mt5CompletedD1Summary { + state: CompletedD1State + detail: string + ageHours: number | null + parsed: ParsedCompletedD1 | null +} + +export interface CompletedTrendObservation { + asOf: string + direction: TrendDirection + lookbackReturn: number + lookbackDays: number + latestClose: number + referenceClose: number +} + +export function deriveCompletedTrendObservation(input: ParsedCompletedD1, lookbackDays: number): CompletedTrendObservation { + if (!Number.isInteger(lookbackDays) || lookbackDays <= 0 || input.bars.length < lookbackDays + 1) { + throw new Error('Completed D1 history is insufficient for the selected lookback') + } + const latest = input.bars.at(-1)! + const reference = input.bars.at(-(lookbackDays + 1))! + const lookbackReturn = latest.close / reference.close - 1 + return { + asOf: latest.asOf, + direction: lookbackReturn > 0 ? 'uptrend' : lookbackReturn < 0 ? 'downtrend' : 'flat', + lookbackReturn, + lookbackDays, + latestClose: latest.close, + referenceClose: reference.close, + } +} +``` + +Implement `parseCompletedD1Csv` with the exact header from the test, numeric finite checks, identical metadata across rows, demo-only validation, unique `bar_as_of`, and strictly increasing `bar_open_epoch`. Implement `readMt5CompletedD1` using file modification time and a policy-supplied maximum observation age rather than wall-clock text from the broker server. + +- [ ] **Step 4: Extend the read-only bridge exporter** + +Add a status-only function that calls `CopyRates(InpSymbol, PERIOD_D1, 1, 300, rates)`. Shift `1` excludes the forming daily bar. Write a temporary Common Files CSV, flush and close it, then atomically replace the destination: + +```cpp +bool ReplaceCommonFile(const string temp_path,const string final_path) +{ + return FileMove(temp_path,FILE_COMMON,final_path,FILE_COMMON|FILE_REWRITE); +} + +bool WriteCompletedD1() +{ + MqlRates rates[]; + ArraySetAsSeries(rates,false); + int copied=CopyRates(InpSymbol,PERIOD_D1,1,300,rates); + if(copied<2) return false; + string final_path=OUTPUT_ROOT+"\\"+InpBrokerId+"\\"+InpSymbol+"\\completed_d1.csv"; + string temp_path=final_path+".tmp"; + int handle=FileOpen(temp_path,FILE_WRITE|FILE_CSV|FILE_ANSI|FILE_COMMON,','); + if(handle==INVALID_HANDLE) return false; + FileWrite(handle,"schema_version","captured_at","broker","server","account_mode","symbol","bar_as_of","bar_open_epoch","open","high","low","close"); + for(int i=0;i Promise`. +- Produces: `buildBrokerCostModel(input) => BrokerCostModel`. +- Produces: `writeBrokerCostModel(root, model) => Promise` using unique-temp-file plus atomic rename. +- Policy file: `OpenAliceMt5DemoPolicyV1//XAUUSD/policy.csv`. +- Cost file: `OpenAliceMt5CostModelV1//XAUUSD/cost_model.csv`. + +- [ ] **Step 1: Write failing hard-ceiling policy tests** + +```ts +import { describe, expect, it } from 'vitest' +import { validateDemoExecutionPolicy } from './demo-execution-policy.js' + +const policy = { + schemaVersion: 1 as const, + policyVersion: 'hfm-canary-v1', + broker: 'hfmarkets', + server: 'HFMarketsGlobal-Demo4', + symbol: 'XAUUSD', + strategyVersion: 'daily-trend-v1', + rolloutStage: 'hfm_canary' as const, + candidateApproved: true, + completedObservationMaxAgeHours: 72, + maxSpread: 0.75, + maxDeviation: 0.5, + maxRiskAmount: 10, + maxDailyLoss: 40, + maxDailyLosingTrades: 4, + maxVolume: 0.01, + magicNumber: 880101, +} + +describe('demo execution policy', () => { + it('accepts the exact HFM canary ceiling', () => expect(validateDemoExecutionPolicy(policy).state).toBe('ready')) + it('blocks a policy that loosens max volume', () => expect(validateDemoExecutionPolicy({ ...policy, maxVolume: 0.02 }).state).toBe('blocked')) + it('blocks EURUSD regardless of candidate flag', () => expect(validateDemoExecutionPolicy({ ...policy, symbol: 'EURUSD' }).state).toBe('blocked')) +}) +``` + +- [ ] **Step 2: Run the policy test and verify RED** + +Run: `pnpm vitest run src/domain/mt5/demo-execution-policy.spec.ts` + +Expected: FAIL because the module does not exist. + +- [ ] **Step 3: Implement the exact policy contract** + +```ts +export type DemoRolloutStage = 'status_only' | 'hfm_canary' | 'ic_canary' | 'both_demo' + +export interface DemoExecutionPolicy { + schemaVersion: 1 + policyVersion: string + broker: 'hfmarkets' | 'icmarkets' + server: 'HFMarketsGlobal-Demo4' | 'ICMarketsSC-Demo' + symbol: 'XAUUSD' + strategyVersion: 'daily-trend-v1' + rolloutStage: DemoRolloutStage + candidateApproved: boolean + completedObservationMaxAgeHours: number + maxSpread: number + maxDeviation: number + maxRiskAmount: number + maxDailyLoss: number + maxDailyLosingTrades: number + maxVolume: number + magicNumber: 880101 | 880201 +} + +export interface DemoExecutionPolicySummary { + state: 'ready' | 'blocked' | 'missing' | 'malformed' + detail: string + policy: DemoExecutionPolicy | null +} +``` + +`validateDemoExecutionPolicy` must compare every policy limit against immutable hard ceilings. A policy may tighten a limit but cannot loosen it. `status_only` is always non-executable. The TypeScript app receives no write function for the policy root. + +The strict `policy.csv` header is exactly: + +```text +schema_version,policy_version,broker,server,symbol,strategy_version,rollout_stage,candidate_approved,completed_observation_max_age_hours,max_spread,max_deviation,max_risk_amount,max_daily_loss,max_daily_losing_trades,max_volume,magic_number +``` + +- [ ] **Step 4: Write and implement broker-cost evidence tests** + +Test that `buildBrokerCostModel` returns `canary_ready` only when all conditions hold: fresh bridge, at least 100 recent spread samples, parseable commission/swap fields from at least one closed demo Gold deal, matching contract fingerprint, and configured conservative deviation ceiling. Test missing evidence as `blocked`, not `warn`. + +```ts +export interface BrokerCostModel { + schemaVersion: 1 + version: string + broker: 'hfmarkets' | 'icmarkets' + server: string + symbol: 'XAUUSD' + state: 'canary_ready' | 'blocked' + observedFrom: string + observedTo: string + expiresAt: string + spreadSampleCount: number + observedMaxSpread: number + configuredMaxSpread: number + configuredMaxDeviation: number + commissionObserved: boolean + swapObserved: boolean + contractFingerprint: string + evidence: string[] +} + +export async function writeBrokerCostModel(root: string, model: BrokerCostModel): Promise { + const destination = join(root, model.broker, model.symbol, 'cost_model.csv') + await mkdir(dirname(destination), { recursive: true }) + const temporary = `${destination}.${process.pid}.${randomUUID()}.tmp` + await writeFile(temporary, serializeBrokerCostModelCsv(model), { encoding: 'utf8', flag: 'wx' }) + await renameReplacing(temporary, destination) +} +``` + +The strict `cost_model.csv` header is exactly: + +```text +schema_version,version,broker,server,symbol,state,observed_from,observed_to,expires_at,spread_sample_count,observed_max_spread,configured_max_spread,configured_max_deviation,commission_observed,swap_observed,contract_fingerprint,evidence_json +``` + +Run: `pnpm vitest run src/domain/mt5/broker-cost-model.spec.ts` + +Expected after implementation: PASS. + +- [ ] **Step 5: Add the operator-only MQL policy script** + +The script must refuse non-demo accounts, derive broker/server/magic from exact allowlists, support only `XAUUSD`, and default `InpRolloutStage="status_only"`. It writes the exact CSV schema atomically under `OpenAliceMt5DemoPolicyV1`. It must not contain `OrderSend`, `OrderCheck`, or position-close code. + +- [ ] **Step 6: Run Task 2 verification** + +Run: `pnpm vitest run src/domain/mt5/demo-execution-policy.spec.ts src/domain/mt5/broker-cost-model.spec.ts` + +Expected: PASS. + +Run: `rg -n "OrderSend|OrderCheck|PositionClose" tools/mt5/ConfigureJmbGoldmineDemoPolicy.mq5` + +Expected: no matches. + +- [ ] **Step 7: Commit Task 2** + +```powershell +git add -- src/domain/mt5/demo-execution-policy.ts src/domain/mt5/demo-execution-policy.spec.ts src/domain/mt5/broker-cost-model.ts src/domain/mt5/broker-cost-model.spec.ts tools/mt5/ConfigureJmbGoldmineDemoPolicy.mq5 +git commit -m "feat: add mt5 demo policy and cost gates" +``` + +--- + +### Task 3: Publish isolated, stable execution-decision leases + +**Files:** +- Create: `src/domain/mt5/local-paths.ts` +- Create: `src/domain/mt5/local-paths.spec.ts` +- Create: `src/domain/mt5/execution-decision.ts` +- Create: `src/domain/mt5/execution-decision.spec.ts` +- Create: `src/domain/mt5/demo-decision-engine.ts` +- Create: `src/domain/mt5/demo-decision-engine.spec.ts` +- Create: `src/domain/mt5/demo-decision-service.ts` +- Create: `src/domain/mt5/demo-decision-service.spec.ts` +- Create: `tools/mt5/run_demo_canary_decisions.ts` + +**Interfaces:** +- Produces: `createObservationId`, `createExecutionDecisionId`, `serializeExecutionDecisionCsv`, `parseExecutionDecisionCsv`. +- Produces: `buildDemoExecutionDecision(input) => JmbExecutionDecision`. +- Produces: `runDemoDecisionCycle(options) => Promise`. +- Produces: `resolveJmbMt5Roots(options?) => JmbMt5Roots`. +- Output root: `OpenAliceMt5ExecutionDecisionV1//XAUUSD/`. + +- [ ] **Step 1: Write failing stable-identity tests** + +```ts +it('keeps ids stable when only the five-minute lease changes', () => { + const first = sampleDecision({ leaseIssuedAt: '2026-07-13T09:00:00Z', leaseExpiresAt: '2026-07-13T09:10:00Z' }) + const second = sampleDecision({ leaseIssuedAt: '2026-07-13T09:05:00Z', leaseExpiresAt: '2026-07-13T09:15:00Z' }) + expect(createObservationId(first)).toBe(createObservationId(second)) + expect(createExecutionDecisionId(first)).toBe(createExecutionDecisionId(second)) +}) + +it('changes identity for a newer completed D1 date', () => { + expect(createObservationId(sampleDecision({ observationAsOf: '2026-07-12' }))) + .not.toBe(createObservationId(sampleDecision({ observationAsOf: '2026-07-11' }))) +}) + +it('does not re-identify a consumed observation after policy or cost refresh', () => { + const first = sampleDecision({ candidatePolicyVersion: 'hfm-v1', costModelVersion: 'cost-0900' }) + const refreshed = sampleDecision({ candidatePolicyVersion: 'hfm-v2', costModelVersion: 'cost-0905' }) + expect(createExecutionDecisionId(first)).toBe(createExecutionDecisionId(refreshed)) +}) +``` + +- [ ] **Step 2: Run the identity test and verify RED** + +Run: `pnpm vitest run src/domain/mt5/execution-decision.spec.ts` + +Expected: FAIL because the module does not exist. + +- [ ] **Step 3: Implement the isolated protocol and hashes** + +```ts +export interface JmbGateResult { + name: string + state: 'pass' | 'block' + detail: string +} + +export interface JmbExecutionDecision { + schemaVersion: 1 + decisionId: string + observationId: string + observationAsOf: string + createdAt: string + leaseIssuedAt: string + leaseExpiresAt: string + broker: 'hfmarkets' | 'icmarkets' + server: 'HFMarketsGlobal-Demo4' | 'ICMarketsSC-Demo' + accountMode: 'demo' + symbol: 'XAUUSD' + strategyVersion: 'daily-trend-v1' + direction: 'buy' | 'sell' | 'flat' + entryReferencePrice: number | null + volume: 0.01 + stopLoss: number | null + maxRiskAmount: number + candidatePolicyVersion: string + costModelVersion: string + gateResults: JmbGateResult[] +} + +export function createObservationId(input: Pick): string { + return hash([input.broker, input.symbol, input.strategyVersion, input.observationAsOf]) +} + +export function createExecutionDecisionId(input: Pick): string { + return hash(['daily-trend-v1', input.observationId]) +} +``` + +The decision ID is permanently bound to the observation ID; policy, cost, quote, direction, and lease refreshes cannot create a second identity for a consumed observation. Before the first entry attempt, a renewed lease may refresh evidence fields under the same ID. After any attempt, the EA's processed-observation store is authoritative and forbids another send. The CSV parser must use an exact allowlisted header and semantic enum validation. The JSONL journal appends only when decision evidence materially changes, while retaining the same decision ID. `latest_decision.csv` uses unique temporary write plus `rename`; a regressed `observationAsOf` must not overwrite a newer lease. + +The strict `latest_decision.csv` header is exactly: + +```text +schema_version,decision_id,observation_id,observation_as_of,created_at,lease_issued_at,lease_expires_at,broker,server,account_mode,symbol,strategy_version,direction,entry_reference_price,volume,stop_loss,max_risk_amount,candidate_policy_version,cost_model_version,gate_results_json +``` + +- [ ] **Step 4: Write failing pure-engine tests** + +Test HFM Gold `ready`, IC Gold blocked during `hfm_canary`, EURUSD always blocked, stale completed observation blocked, blocked cost model, missing stop, spread breach, and flat direction. The engine must never return executable when any hard gate fails. + +- [ ] **Step 5: Implement the pure engine** + +`buildDemoExecutionDecision` receives already-parsed bridge, observation, policy, cost model, learning, and quote values. It creates a `buy`, `sell`, or `flat` lease and an ordered `gateResults` array. The function has no file, network, time, LLM, or broker dependencies. + +- [ ] **Step 6: Implement the deterministic cycle service** + +```ts +export interface JmbMt5Roots { + bridgeRoot: string + ledgerRoot: string + policyRoot: string + costModelRoot: string + executionDecisionRoot: string + executionRoot: string + researchRoot: string +} + +export interface JmbDemoInstrumentConfig { + broker: 'hfmarkets' | 'icmarkets' + server: 'HFMarketsGlobal-Demo4' | 'ICMarketsSC-Demo' + symbol: 'XAUUSD' | 'EURUSD' + researchArtifactSymbol: 'XAUUSDb' | 'EURUSDb' | 'XAUUSD' | 'EURUSD' + maxSpread: number + maxDeviation: number +} + +export interface DemoDecisionCycleOptions { + roots: JmbMt5Roots + now?: () => Date + instruments?: readonly JmbDemoInstrumentConfig[] +} + +export interface DemoDecisionCycleResult { + broker: string + symbol: string + state: 'published' | 'blocked' | 'error' + observationId: string | null + decisionId: string | null + detail: string +} +``` + +`resolveJmbMt5Roots` accepts explicit overrides first, then `OPENALICE_MT5_COMMON_FILES_ROOT`, then `%APPDATA%\MetaQuotes\Terminal\Common\Files`; it throws when no absolute Common Files root can be resolved. It derives `OpenAliceMt5BridgeV1`, `OpenAliceMt5TradeLedgerV1`, `OpenAliceMt5DemoPolicyV1`, `OpenAliceMt5CostModelV1`, `OpenAliceMt5ExecutionDecisionV1`, and `OpenAliceMt5ExecutionV1` beneath that root. `researchRoot` resolves from `OPENALICE_RESEARCH_ROOT`, otherwise `~/.openalice/data/research`. Process broker/symbol pairs independently so one malformed broker does not suppress the other three. Read the frozen selected lookback from research artifacts but derive direction from fresh completed-D1 broker bars. Build and atomically persist the current cost model before the engine evaluates it. Publish execution leases only to the isolated execution-decision root. EURUSD returns `blocked`, publishes no execution lease, and continues using the existing shadow-decision journal as its durable record. + +- [ ] **Step 7: Add the diagnostic CLI** + +`tools/mt5/run_demo_canary_decisions.ts` resolves local roots, calls `runDemoDecisionCycle`, prints one concise line per broker/symbol, and sets a nonzero exit code only for thrown infrastructure errors. It contains no duplicated strategy rules and no child-process calls. + +- [ ] **Step 8: Run Task 3 verification** + +Run: `pnpm vitest run src/domain/mt5/execution-decision.spec.ts src/domain/mt5/demo-decision-engine.spec.ts src/domain/mt5/demo-decision-service.spec.ts` + +Expected: PASS. + +Run: `rg -n "child_process|powershell|MetaEditor|OrderSend|OrderCheck" src/domain/mt5/execution-decision.ts src/domain/mt5/demo-decision-engine.ts src/domain/mt5/demo-decision-service.ts tools/mt5/run_demo_canary_decisions.ts` + +Expected: no matches. + +- [ ] **Step 9: Commit Task 3** + +```powershell +git add -- src/domain/mt5/local-paths.ts src/domain/mt5/local-paths.spec.ts src/domain/mt5/execution-decision.ts src/domain/mt5/execution-decision.spec.ts src/domain/mt5/demo-decision-engine.ts src/domain/mt5/demo-decision-engine.spec.ts src/domain/mt5/demo-decision-service.ts src/domain/mt5/demo-decision-service.spec.ts tools/mt5/run_demo_canary_decisions.ts +git commit -m "feat: add deterministic mt5 execution decisions" +``` + +--- + +### Task 4: Run the decision cycle through the internal Pump + +**Files:** +- Create: `src/task/mt5-decision-scheduler.ts` +- Create: `src/task/mt5-decision-scheduler.spec.ts` +- Modify: `src/main.ts` + +**Interfaces:** +- Consumes: `runDemoDecisionCycle(options)` from Task 3. +- Produces: `createJmbMt5DecisionScheduler(options) => JmbMt5DecisionScheduler`. + +- [ ] **Step 1: Write failing lifecycle tests** + +```ts +it('runs one catch-up cycle before arming the five-minute pump', async () => { + const runCycle = vi.fn(async () => []) + const scheduler = createJmbMt5DecisionScheduler({ runCycle }) + await scheduler.start() + expect(runCycle).toHaveBeenCalledTimes(1) + scheduler.stop() +}) + +it('does not overlap a slow cycle', async () => { + let release!: () => void + const slowCycle = new Promise((resolve) => { release = resolve }) + const runCycle = vi.fn() + .mockResolvedValueOnce([]) + .mockImplementation(() => slowCycle) + const scheduler = createJmbMt5DecisionScheduler({ runCycle, every: '5m' }) + await scheduler.start() + await vi.advanceTimersByTimeAsync(15 * 60_000) + expect(runCycle).toHaveBeenCalledTimes(2) + release() + await Promise.resolve() + scheduler.stop() +}) +``` + +- [ ] **Step 2: Run the scheduler test and verify RED** + +Run: `pnpm vitest run src/task/mt5-decision-scheduler.spec.ts` + +Expected: FAIL because the module does not exist. + +- [ ] **Step 3: Implement using `createPump`** + +```ts +import { createPump, type Pump } from '../core/pump.js' + +export interface JmbMt5DecisionScheduler { + start(): Promise + stop(): void + runNow(): Promise +} + +export function createJmbMt5DecisionScheduler(options: { + runCycle: () => Promise + every?: string + logger?: Pick +}): JmbMt5DecisionScheduler { + const pump: Pump = createPump({ + name: 'jmb-mt5-decision-cycle', + every: options.every ?? '5m', + serial: true, + onTick: async () => { await options.runCycle() }, + logger: options.logger, + }) + return { + async start() { await pump.runNow(); pump.start() }, + stop() { pump.stop() }, + runNow() { return pump.runNow() }, + } +} +``` + +- [ ] **Step 4: Wire lifecycle in `src/main.ts`** + +Construct the scheduler after data paths are available, call `await scheduler.start()` before the engine-ready log, and call `scheduler.stop()` in shutdown before listeners stop. Do not register it with user Cron, ToolCenter, workspaces, or EngineContext. + +- [ ] **Step 5: Run scheduler and source-boundary verification** + +Run: `pnpm vitest run src/task/mt5-decision-scheduler.spec.ts src/core/pump.spec.ts` + +Expected: PASS. + +Run: `rg -n "cron|workspace|child_process|powershell|spawn\(|exec\(" src/task/mt5-decision-scheduler.ts` + +Expected: no matches. + +- [ ] **Step 6: Commit Task 4** + +```powershell +git add -- src/task/mt5-decision-scheduler.ts src/task/mt5-decision-scheduler.spec.ts src/main.ts +git commit -m "feat: schedule mt5 demo decisions internally" +``` + +--- + +### Task 5: Add execution status read model and Research Desk projection + +**Files:** +- Create: `src/domain/mt5/execution-status.ts` +- Create: `src/domain/mt5/execution-status.spec.ts` +- Modify: `src/webui/routes/research.ts` +- Create: `src/webui/routes/research.spec.ts` +- Modify: `ui/src/api/research.ts` +- Create: `ui/src/components/research/Mt5ExecutionStatusCard.tsx` +- Create: `ui/src/components/research/__tests__/Mt5ExecutionStatusCard.spec.tsx` +- Modify: `ui/src/pages/ResearchDashboardPage.tsx` +- Create: `ui/src/demo/handlers/research.ts` +- Modify: `ui/src/demo/handlers/index.ts` + +**Interfaces:** +- Consumes: `OpenAliceMt5ExecutionV1//XAUUSD/latest_status.csv` from Tasks 6-8. +- Produces: `summarizeLatestJmbExecutionStatus(root, broker, symbol, now?) => Promise`. +- Produces: read-only `execution` on each Research instrument. + +- [ ] **Step 1: Write failing strict-status parser tests** + +```ts +it('summarizes a protected demo fill without exposing account login', async () => { + const summary = await summarizeLatestJmbExecutionStatus(root, 'hfmarkets', 'XAUUSD') + expect(summary.state).toBe('filled_protected') + expect(JSON.stringify(summary)).not.toMatch(/account.?login/i) +}) + +it('rejects an unexpected account_login column', () => { + expect(() => parseExecutionStatusCsv(validCsv.replace('symbol,', 'account_login,symbol,'))).toThrow(/schema/) +}) +``` + +- [ ] **Step 2: Run the parser test and verify RED** + +Run: `pnpm vitest run src/domain/mt5/execution-status.spec.ts` + +Expected: FAIL because the module does not exist. + +- [ ] **Step 3: Implement the safe lifecycle contract** + +```ts +export type JmbExecutionLifecycleState = + | 'disabled' | 'paused' | 'blocked' | 'ready' | 'order_requesting' + | 'order_rejected' | 'reconciliation_required' | 'filled_protected' + | 'close_requesting' | 'closed' | 'stopped' | 'emergency_close' | 'error' + +export type JmbExecutionRolloutStage = 'status_only' | 'hfm_canary' | 'ic_canary' | 'both_demo' + +export type JmbResearchExecutionState = JmbExecutionLifecycleState | 'demo_blocked' | 'missing' | 'malformed' | 'stale' + +export interface JmbExecutionStatusSummary { + state: JmbResearchExecutionState + label: string + detail: string + capturedAt: string | null + broker: 'hfmarkets' | 'icmarkets' + server: string | null + accountMode: 'demo' | null + symbol: 'XAUUSD' | 'EURUSD' + rolloutStage: JmbExecutionRolloutStage + executionEnabled: boolean + killSwitch: boolean + decisionId: string | null + observationId: string | null + latestEvent: { id: string; type: string; at: string; resultCode: string; detail: string } | null + stopProtectionConfirmed: boolean + position: { direction: 'buy' | 'sell'; volume: number; openPrice: number; stopLoss: number; id: string } | null + reconciliationState: string + dailyLossCount: number + dailyRealizedLoss: number + blockingGate: string | null + nextSafeAction: string +} +``` + +The summary exposes broker, symbol, server, account mode, captured/update times, rollout stage, execution/kill-switch flags, decision/observation IDs, latest event, stop confirmation, EA-only position, reconciliation state, daily losses, blocking gate, and next safe action. It never includes raw or masked login. + +The strict `latest_status.csv` header is exactly: + +```text +schema_version,captured_at,broker,server,account_mode,symbol,state,detail,rollout_stage,execution_enabled,kill_switch,decision_id,observation_id,event_id,event_type,event_time,result_code,result_detail,stop_protection_confirmed,position_direction,position_volume,position_open_price,position_stop_loss,position_id,reconciliation_state,daily_loss_count,daily_realized_loss,blocking_gate,next_safe_action +``` + +Any missing, duplicate, reordered, or extra column is malformed and maps to a fail-closed Research summary. Position, order, and deal IDs are opaque strings; no account identifier is accepted by this read model. + +- [ ] **Step 4: Extend the Research route with a testability seam** + +Change the factory to `createResearchRoutes(ctx, overrides?: { executionRoot?: string })`. Use `bridgeSymbol ?? symbol` for execution paths. Gold receives the parsed execution summary; EURUSD receives an explicit read-only `DEMO BLOCKED` projection. Keep top-level `mode: 'research_only'` and `tradingEnabled: false` because those describe Research Desk authority, not broker-local EA state. + +- [ ] **Step 5: Write and pass the route privacy test** + +Mount the Hono route with a temporary execution root, request `/`, and assert Gold lifecycle state, EURUSD block, and absence of `accountLogin`, `account_login`, or a sentinel login value. + +Run: `pnpm vitest run src/webui/routes/research.spec.ts` + +Expected: PASS. + +- [ ] **Step 6: Add the focused status card and tests** + +The component maps approved labels, always shows `DEMO ONLY`, displays latest event/stop/exposure/daily loss/blocking gate/next safe action, and contains no button, toggle, or order action. + +```tsx +export function Mt5ExecutionStatusCard({ execution }: { execution: JmbExecutionStatusSummary }) { + return
+ DEMO ONLY + {execution.label} +

{execution.detail}

+ {execution.blockingGate ?

Blocked by: {execution.blockingGate}

: null} +

Next: {execution.nextSafeAction}

+
+} +``` + +Test `filled_protected`, `paused`, `reconciliation_required`, and EUR `demo_blocked`; assert no login text and no interactive execution control. + +- [ ] **Step 7: Integrate UI and demo handler** + +Render the new card after the Plan 2 decision card. Update the hero to explain that the page is a read-only monitor and execution remains broker-local in MT5. Add a typed demo handler before catch-all so demo builds do not return `{}`. + +- [ ] **Step 8: Run UI/API verification** + +Run: `pnpm vitest run src/domain/mt5/execution-status.spec.ts src/webui/routes/research.spec.ts ui/src/components/research/__tests__/Mt5ExecutionStatusCard.spec.tsx` + +Expected: PASS. + +Run: `pnpm -F open-alice-ui build` + +Expected: PASS. + +Run: `pnpm -F open-alice-ui build:demo` + +Expected: PASS. + +- [ ] **Step 9: Commit Task 5** + +```powershell +git add -- src/domain/mt5/execution-status.ts src/domain/mt5/execution-status.spec.ts src/webui/routes/research.ts src/webui/routes/research.spec.ts ui/src/api/research.ts ui/src/components/research/Mt5ExecutionStatusCard.tsx ui/src/components/research/__tests__/Mt5ExecutionStatusCard.spec.tsx ui/src/pages/ResearchDashboardPage.tsx ui/src/demo/handlers/research.ts ui/src/demo/handlers/index.ts +git commit -m "feat: show mt5 demo execution lifecycle" +``` + +--- + +### Task 6: Build the modular dry-run EA gates and no-order harness + +**Files:** +- Create: `tools/mt5/JmbGoldmineDemoCanary/JmbGoldmineDemoCanary.mq5` +- Create: `tools/mt5/JmbGoldmineDemoCanary/JmbCanaryTypes.mqh` +- Create: `tools/mt5/JmbGoldmineDemoCanary/JmbCanaryCsv.mqh` +- Create: `tools/mt5/JmbGoldmineDemoCanary/JmbCanaryPolicy.mqh` +- Create: `tools/mt5/JmbGoldmineDemoCanary/JmbCanaryGates.mqh` +- Create: `tools/mt5/JmbGoldmineDemoCanary/JmbCanaryState.mqh` +- Create: `tools/mt5/tests/JmbGoldmineDemoCanaryHarness.mq5` +- Create: `tools/mt5/tests/README.md` +- Create: `src/domain/mt5/demo-canary-source.spec.ts` + +**Interfaces:** +- Consumes: Task 2 policy CSV and Task 3 execution-decision lease CSV. +- Produces: `OpenAliceMt5ExecutionV1//XAUUSD/latest_status.csv`. +- No order API is permitted in this task. + +- [ ] **Step 1: Write the failing TypeScript source acceptance test** + +```ts +it('keeps the dry-run bundle order-free and safe by default', async () => { + const source = await readCanaryBundle() + expect(source).not.toMatch(/OrderSend|OrderCheck|CTrade|PositionClose/) + expect(source).toContain('input bool InpDemoExecutionEnabled = false;') + expect(source).toContain('input bool InpKillSwitch = true;') + expect(source).not.toMatch(/live.?mode/i) +}) +``` + +- [ ] **Step 2: Run the source test and verify RED** + +Run: `pnpm vitest run src/domain/mt5/demo-canary-source.spec.ts` + +Expected: FAIL because the bundle does not exist. + +- [ ] **Step 3: Implement types, policy hard ceilings, and pure gates** + +Define `CanaryDecision`, `CanaryPolicy`, `CanaryEnvironment`, `CanaryGateResult`, `CanaryLifecycleState`, and `CanaryEvaluation`. `EvaluateCanaryGates` accepts structs and returns a result without reading files or calling MT5 order APIs. + +```cpp +CanaryGateResult Gate(const string name,const bool passed,const string detail) +{ + CanaryGateResult result; + result.name=name; + result.passed=passed; + result.detail=detail; + return result; +} +``` + +Apply the exact gate order from the design: demo/identity, switches, rollout, allowlists, freshness, volume, stop/risk, daily loss/count, exposure, margin, spread/deviation, session, news, log preflight, reconciliation. + +The stop-risk gate calls `OrderCalcProfit` from the current entry quote to the proposed stop and requires the absolute loss to be at most the tighter of policy and hard ceiling. The margin gate calls `OrderCalcMargin` and requires free margin after the estimate to retain the ten-times buffer. The news gate queries the MT5 economic calendar for high-impact USD events; unavailable, incomplete, or failed calendar reads block entry. + +- [ ] **Step 4: Implement strict CSV and durable status helpers** + +The parser locates fields by exact header name, rejects duplicate/missing fields, forbids physical multiline values, validates enums/numbers, and rejects leases outside `leaseIssuedAt <= TimeGMT() <= leaseExpiresAt`. Status writes use a temporary Common Files path, `FileFlush`, `FileClose`, and `FileMove(temp, FILE_COMMON, destination, FILE_COMMON|FILE_REWRITE)`. + +- [ ] **Step 5: Implement thin dry-run EA orchestration** + +Use `#property version "0.200"`. `OnInit` validates inputs, sets a ten-second timer, exports no credentials, and evaluates once. `OnTick`/`OnTimer` call one serial `Evaluate()` function that writes `disabled`, `paused`, `blocked`, or `ready`. Even if `InpDemoExecutionEnabled=true`, this task remains order-free and reports `ready` only. + +```cpp +input string InpBrokerId = ""; +input string InpExpectedServer = ""; +input long InpExpectedAccountLogin = 0; +input string InpSymbol = "XAUUSD"; +input long InpMagicNumber = 0; +input bool InpDemoExecutionEnabled = false; +input bool InpKillSwitch = true; + +int OnInit() +{ + if(InpExpectedAccountLogin<=0 || InpSymbol!="XAUUSD") return INIT_PARAMETERS_INCORRECT; + EventSetTimer(10); + Evaluate(); + return INIT_SUCCEEDED; +} + +void OnTimer() { Evaluate(); } +void OnTick() { Evaluate(); } +void OnDeinit(const int reason) { EventKillTimer(); } +``` + +`InpExpectedAccountLogin` is checked in memory against `ACCOUNT_LOGIN` but is never written to status, events, logs, comments, or API artifacts. + +- [ ] **Step 6: Add table-driven MQL harness cases** + +The harness includes production types/gates/state but no trade gateway. It writes PASS/FAIL for demo/server/symbol/magic, switches, volume/stop/risk, spread/session/news, exposure, duplicate observation, four-loss reset, and log failure. A nonzero failure count returns `INIT_FAILED`. + +- [ ] **Step 7: Run TypeScript source checks** + +Run: `pnpm vitest run src/domain/mt5/demo-canary-source.spec.ts` + +Expected: PASS. + +Run: `rg -n "OrderSend|OrderCheck|CTrade|PositionClose" tools/mt5/JmbGoldmineDemoCanary tools/mt5/tests/JmbGoldmineDemoCanaryHarness.mq5` + +Expected: no matches. + +- [ ] **Step 8: Ask the operator to compile manually** + +The operator compiles both `JmbGoldmineDemoCanary.mq5` and `JmbGoldmineDemoCanaryHarness.mq5` in MetaEditor. Acceptance is `0 errors, 0 warnings`; Codex must not launch MetaEditor. + +- [ ] **Step 9: Commit Task 6** + +```powershell +git add -- tools/mt5/JmbGoldmineDemoCanary tools/mt5/tests/JmbGoldmineDemoCanaryHarness.mq5 tools/mt5/tests/README.md src/domain/mt5/demo-canary-source.spec.ts +git commit -m "feat: add mt5 demo canary dry run" +``` + +--- + +### Task 7: Add the single protected-order gateway + +**Files:** +- Create: `tools/mt5/JmbGoldmineDemoCanary/JmbCanaryTradeGateway.mqh` +- Modify: `tools/mt5/JmbGoldmineDemoCanary/JmbGoldmineDemoCanary.mq5` +- Modify: `src/domain/mt5/demo-canary-source.spec.ts` + +**Interfaces:** +- Consumes: `CanaryEvaluation` from Task 6. +- Produces: one `TradeSubmitResult SubmitProtectedMarketOrder(...)` function. +- This is the only repository file in Plan 3 allowed to contain `OrderCheck` or `OrderSend`. + +- [ ] **Step 1: Strengthen the source test before adding the gateway** + +Assert every MQL/MQH file except `JmbCanaryTradeGateway.mqh` contains neither `OrderSend` nor `OrderCheck`, the gateway contains exactly one `OrderSend(` call, and no source contains a live-mode input, EURUSD execution allowlist, martingale/grid/recovery, or volume above `0.01`. + +- [ ] **Step 2: Run the source test and verify RED** + +Run: `pnpm vitest run src/domain/mt5/demo-canary-source.spec.ts` + +Expected: FAIL because the gateway does not exist. + +- [ ] **Step 3: Implement the only order gateway** + +```cpp +struct TradeSubmitResult +{ + bool sent; + uint retcode; + ulong order_ticket; + ulong deal_ticket; + string detail; +}; + +bool ResolveMarketFilling(const string symbol,ENUM_ORDER_TYPE_FILLING &resolved) +{ + long flags=SymbolInfoInteger(symbol,SYMBOL_FILLING_MODE); + long execution=SymbolInfoInteger(symbol,SYMBOL_TRADE_EXEMODE); + if((flags&SYMBOL_FILLING_FOK)==SYMBOL_FILLING_FOK) { resolved=ORDER_FILLING_FOK; return true; } + if((flags&SYMBOL_FILLING_IOC)==SYMBOL_FILLING_IOC) { resolved=ORDER_FILLING_IOC; return true; } + if(execution!=SYMBOL_TRADE_EXECUTION_MARKET) { resolved=ORDER_FILLING_RETURN; return true; } + return false; +} + +TradeSubmitResult SubmitProtectedMarketOrder(const CanaryDecision &decision,const CanaryPolicy &policy) +{ + TradeSubmitResult result; + ZeroMemory(result); + if(AccountInfoInteger(ACCOUNT_TRADE_MODE)!=ACCOUNT_TRADE_MODE_DEMO) + { + result.detail="Account is not demo"; + return result; + } + MqlTradeRequest request={}; + MqlTradeCheckResult check={}; + MqlTradeResult broker={}; + request.action=TRADE_ACTION_DEAL; + request.magic=policy.magic_number; + request.symbol=decision.symbol; + request.volume=0.01; + request.type=decision.direction=="buy" ? ORDER_TYPE_BUY : ORDER_TYPE_SELL; + request.price=decision.direction=="buy" ? SymbolInfoDouble(decision.symbol,SYMBOL_ASK) : SymbolInfoDouble(decision.symbol,SYMBOL_BID); + request.sl=decision.stop_loss; + if(!ResolveMarketFilling(decision.symbol,request.type_filling)) + { + result.detail="No supported market filling mode"; + return result; + } + request.deviation=(ulong)MathFloor(policy.max_deviation_price/SymbolInfoDouble(decision.symbol,SYMBOL_POINT)); + request.comment="JMB:"+StringSubstr(decision.decision_id,0,20); + if(!OrderCheck(request,check) || check.retcode!=0) + { + result.retcode=check.retcode; + result.detail=check.comment; + return result; + } + result.sent=OrderSend(request,broker); + result.retcode=broker.retcode; + result.order_ticket=broker.order; + result.deal_ticket=broker.deal; + result.detail=broker.comment; + return result; +} +``` + +Do not interpret `OrderSend=true` as a fill. The caller records `order_requesting` before the call, flushes the event, invokes the gateway once, and always moves to broker reconciliation. + +- [ ] **Step 4: Wire the gateway behind all dry-run gates** + +The EA calls the gateway only when `InpDemoExecutionEnabled=true`, kill switch is off, lifecycle is `ready`, event journal flush succeeded, and the observation has no prior attempt. Store the attempted observation before any possible retry path. `OnTradeTransaction` only marks reconciliation dirty and returns. + +- [ ] **Step 5: Run source and TypeScript regression tests** + +Run: `pnpm vitest run src/domain/mt5/demo-canary-source.spec.ts src/domain/mt5/execution-status.spec.ts` + +Expected: PASS. + +Run: `rg -n "OrderSend|OrderCheck" tools/mt5/JmbGoldmineDemoCanary` + +Expected: matches only `JmbCanaryTradeGateway.mqh`. + +- [ ] **Step 6: Ask the operator to compile manually with execution disabled** + +Acceptance is `0 errors, 0 warnings`. The operator attaches it to an HFM demo Gold duplicate chart with `InpDemoExecutionEnabled=false` and `InpKillSwitch=true`; status must remain `disabled` and zero orders must be created. + +- [ ] **Step 7: Commit Task 7** + +```powershell +git add -- tools/mt5/JmbGoldmineDemoCanary/JmbCanaryTradeGateway.mqh tools/mt5/JmbGoldmineDemoCanary/JmbGoldmineDemoCanary.mq5 src/domain/mt5/demo-canary-source.spec.ts +git commit -m "feat: add protected mt5 demo order gateway" +``` + +--- + +### Task 8: Reconcile fills, protection, closures, and daily loss + +**Files:** +- Create: `tools/mt5/JmbGoldmineDemoCanary/JmbCanaryReconcile.mqh` +- Modify: `tools/mt5/JmbGoldmineDemoCanary/JmbCanaryState.mqh` +- Modify: `tools/mt5/JmbGoldmineDemoCanary/JmbGoldmineDemoCanary.mq5` +- Modify: `tools/mt5/tests/JmbGoldmineDemoCanaryHarness.mq5` +- Modify: `src/domain/mt5/execution-status.spec.ts` + +**Interfaces:** +- Consumes: magic, symbol, decision/observation IDs, broker orders/deals/positions. +- Produces: authoritative lifecycle events and daily-loss state. +- Produces: append-only `OpenAliceMt5ExecutionV1//XAUUSD/events.jsonl` with this stable event shape: + +```ts +interface JmbExecutionEventV1 { + schema_version: 1 + event_id: string + event_type: JmbExecutionLifecycleState + event_time: string + broker: 'hfmarkets' | 'icmarkets' + server: 'HFMarketsGlobal-Demo4' | 'ICMarketsSC-Demo' + account_mode: 'demo' + account_identity_masked: string + symbol: 'XAUUSD' + strategy_version: 'daily-trend-v1' + magic_number: 880101 | 880201 + decision_id: string + observation_id: string + gate_results: JmbGateResult[] + calculated_risk: number | null + requested_volume: number | null + requested_price: number | null + requested_stop_loss: number | null + accepted_volume: number | null + accepted_price: number | null + accepted_stop_loss: number | null + result_code: string + result_detail: string + order_ticket: string + deal_ticket: string + position_id: string + reconciliation_state: string + daily_loss_count: number + daily_realized_loss: number + commission: number | null + swap: number | null + fee: number | null + net_result: number | null + max_adverse_excursion: number | null + max_favorable_excursion: number | null +} +``` + +`account_identity_masked` is derived locally and must never equal the raw login. Ticket and position identifiers remain strings to avoid numeric precision loss. + +- [ ] **Step 1: Add failing transition and daily-reset harness scenarios** + +Add table cases for rejected request, unknown result, partial fill, filled-with-stop, filled-without-stop, stopped observation, opposite-signal close, four losing positions, server-day reset, restart with protected position, and restart with foreign exposure. Expected next states must exactly match the design lifecycle union. + +- [ ] **Step 2: Add failing TypeScript fixtures for reconciliation states** + +`execution-status.spec.ts` must parse `reconciliation_required`, `filled_protected`, `stopped`, and `emergency_close`, and must reject a status that claims `filled_protected` while `stop_protection_confirmed=0`. + +- [ ] **Step 3: Implement authoritative reconciliation** + +Scan orders/positions by symbol and classify JMB versus foreign using magic number. Do not use account-wide `PositionsTotal()>0`. Query history by server-day range, group fully closed results by `DEAL_POSITION_ID`, and calculate net as `DEAL_PROFIT + DEAL_COMMISSION + DEAL_SWAP + DEAL_FEE`. Attribute loss to the server day of final closure. + +```cpp +double DealNet(const ulong deal_ticket) +{ + return HistoryDealGetDouble(deal_ticket,DEAL_PROFIT) + +HistoryDealGetDouble(deal_ticket,DEAL_COMMISSION) + +HistoryDealGetDouble(deal_ticket,DEAL_SWAP) + +HistoryDealGetDouble(deal_ticket,DEAL_FEE); +} +``` + +- [ ] **Step 4: Implement protection and unknown-result rules** + +Only broker-confirmed exposure with a nonzero valid stop may enter `filled_protected`. Unknown/timeout/partial results enter `reconciliation_required` and forbid another send. An unprotected EA-owned fill triggers the explicit emergency protective close, logs every attempt, and pauses the broker; it never opens replacement exposure. + +- [ ] **Step 5: Implement same-direction, reversal, and consumed-observation behavior** + +Same-direction observation is a durable no-op. An opposite observation closes the existing EA-owned position, waits for confirmed closure, flushes the close event, re-evaluates every entry gate, and only then may submit the new side once. A stopped observation is consumed and cannot re-enter until `observationAsOf` advances. + +- [ ] **Step 6: Run all source/parser tests and manual harness** + +Run: `pnpm vitest run src/domain/mt5/demo-canary-source.spec.ts src/domain/mt5/execution-status.spec.ts` + +Expected: PASS. + +The operator compiles and runs the harness. Expected Experts output ends with `JMB_CANARY_HARNESS PASS` and MetaEditor reports `0 errors, 0 warnings`. + +- [ ] **Step 7: Commit Task 8** + +```powershell +git add -- tools/mt5/JmbGoldmineDemoCanary/JmbCanaryReconcile.mqh tools/mt5/JmbGoldmineDemoCanary/JmbCanaryState.mqh tools/mt5/JmbGoldmineDemoCanary/JmbGoldmineDemoCanary.mq5 tools/mt5/tests/JmbGoldmineDemoCanaryHarness.mq5 src/domain/mt5/execution-status.spec.ts +git commit -m "feat: reconcile mt5 demo canary outcomes" +``` + +--- + +### Task 9: Import reconciled EA outcomes into the learning ledger + +**Files:** +- Create: `src/domain/mt5/execution-outcomes.ts` +- Create: `src/domain/mt5/execution-outcomes.spec.ts` +- Create: `src/task/mt5-outcome-importer.ts` +- Create: `src/task/mt5-outcome-importer.spec.ts` +- Modify: `src/main.ts` + +**Interfaces:** +- Consumes: `OpenAliceMt5ExecutionV1//XAUUSD/events.jsonl` written by Tasks 6-8. +- Produces: `importReconciledExecutionOutcomes(options) => Promise`. +- Learning root: `~/.openalice/data/research/mt5-execution-learning//XAUUSD/`. +- The importer receives only execution and learning roots; it has no policy, decision, EA-input, or broker write interface. + +- [ ] **Step 1: Write failing terminal-outcome and idempotency tests** + +```ts +it('imports one fully reconciled close exactly once', async () => { + await writeExecutionEvents(executionRoot, [requestEvent, fillEvent, closedEvent]) + await importReconciledExecutionOutcomes({ executionRoot, learningRoot, instruments: [hfmGold] }) + await importReconciledExecutionOutcomes({ executionRoot, learningRoot, instruments: [hfmGold] }) + const records = await readExecutionLearningRecords(learningRoot, 'hfmarkets', 'XAUUSD') + expect(records).toHaveLength(1) + expect(records[0]).toMatchObject({ + decisionId: closedEvent.decision_id, + outcomeEventId: closedEvent.event_id, + result: 'loss', + netResult: -6.25, + source: 'ea_demo', + }) +}) + +it('does not import an unresolved or unprotected exposure', async () => { + await writeExecutionEvents(executionRoot, [requestEvent, reconciliationRequiredEvent]) + const result = await importReconciledExecutionOutcomes({ executionRoot, learningRoot, instruments: [hfmGold] }) + expect(result[0].imported).toBe(0) +}) +``` + +- [ ] **Step 2: Run the outcome tests and verify RED** + +Run: `pnpm vitest run src/domain/mt5/execution-outcomes.spec.ts src/task/mt5-outcome-importer.spec.ts` + +Expected: FAIL because both modules do not exist. + +- [ ] **Step 3: Implement the strict immutable outcome contract** + +```ts +export interface JmbExecutionOutcomeRecord { + schemaVersion: 1 + outcomeEventId: string + outcomeAt: string + broker: 'hfmarkets' | 'icmarkets' + server: 'HFMarketsGlobal-Demo4' | 'ICMarketsSC-Demo' + accountMode: 'demo' + symbol: 'XAUUSD' + strategyVersion: 'daily-trend-v1' + decisionId: string + observationId: string + positionId: string + result: 'win' | 'loss' | 'breakeven' + netResult: number + commission: number + swap: number + fee: number + requestedPrice: number | null + acceptedPrice: number | null + slippage: number | null + maxAdverseExcursion: number | null + maxFavorableExcursion: number | null + source: 'ea_demo' +} + +export interface ExecutionOutcomeImportOptions { + executionRoot: string + learningRoot: string + instruments: readonly Pick[] +} + +export interface ExecutionOutcomeImportResult { + broker: 'hfmarkets' | 'icmarkets' + symbol: 'XAUUSD' + state: 'imported' | 'no_new_outcome' | 'blocked' | 'error' + imported: number + detail: string +} +``` + +Parse physical JSONL lines independently and fail closed on malformed schema, non-demo mode, non-Gold symbol, unknown lifecycle, non-finite money fields, or missing identifiers. Only a fully reconciled terminal `closed` or `stopped` event may become a learning outcome. Derive `result` from the reconciled net result after commission, swap, and fee. Never mutate the original decision or execution event. + +- [ ] **Step 4: Implement append-once learning persistence** + +```ts +export async function appendOutcomeOnce(root: string, record: JmbExecutionOutcomeRecord): Promise { + const records = await readExecutionLearningRecords(root, record.broker, record.symbol) + if (records.some((item) => item.outcomeEventId === record.outcomeEventId)) return false + await appendDurableJsonLine(outcomeJournalPath(root, record.broker, record.symbol), record) + await writeJsonAtomically(outcomeSummaryPath(root, record.broker, record.symbol), summarizeOutcomes([...records, record])) + return true +} +``` + +The summary contains counts, total net, win/loss/breakeven counts, cost totals, average slippage, and latest outcome time. It is evidence only and contains no approval flag, new risk limit, strategy parameter, or profit prediction. + +- [ ] **Step 5: Add the isolated five-minute importer** + +```ts +export interface JmbMt5OutcomeImporter { + start(): Promise + stop(): void + runNow(): Promise +} + +export function createJmbMt5OutcomeImporter(options: { + runCycle: () => Promise + every?: string +}): JmbMt5OutcomeImporter { + const pump = createPump({ + name: 'jmb-mt5-outcome-import', + every: options.every ?? '5m', + serial: true, + onTick: async () => { await options.runCycle() }, + }) + return { + async start() { await pump.runNow(); pump.start() }, + stop() { pump.stop() }, + runNow() { return pump.runNow() }, + } +} +``` + +Wire it in `src/main.ts` after the decision scheduler. Start with one catch-up import, run serially, isolate broker failures, and stop it during shutdown. It may write only the learning root. + +- [ ] **Step 6: Run outcome and boundary verification** + +Run: `pnpm vitest run src/domain/mt5/execution-outcomes.spec.ts src/task/mt5-outcome-importer.spec.ts` + +Expected: PASS. + +Run: `rg -n "policyRoot|OrderSend|OrderCheck|child_process|openai|anthropic|llm" src/domain/mt5/execution-outcomes.ts src/task/mt5-outcome-importer.ts` + +Expected: no matches. + +- [ ] **Step 7: Commit Task 9** + +```powershell +git add -- src/domain/mt5/execution-outcomes.ts src/domain/mt5/execution-outcomes.spec.ts src/task/mt5-outcome-importer.ts src/task/mt5-outcome-importer.spec.ts src/main.ts +git commit -m "feat: import reconciled mt5 demo outcomes" +``` + +--- + +### Task 10: Document and verify Stage 0 readiness + +**Files:** +- Modify: `tools/mt5/README.md` +- Modify: `docs/mt5-data-and-training-protocol.md` +- Modify: `docs/PRD.md` + +**Interfaces:** +- Produces: operator ceremony for installation, dry-run, HFM canary, IC canary, rollback, pause, and recovery. + +- [ ] **Step 1: Document exact operator steps** + +Document bridge recompilation for completed-D1 export, policy-script use, folder paths, EA copy layout, MetaEditor compile, harness run, status-only attach, HFM inputs, IC inputs, rollout stages, kill switch, expected status files, rollback to `JmbGoldmineDemoRiskShell`, and emergency verification. Name the staged ceremonies exactly “HFM canary” and “IC Markets canary.” Explicitly state that Codex does not launch MetaEditor and that live/EURUSD execution is absent. + +Update `docs/PRD.md` without rewriting unrelated sections: + +- In Current State, replace the claim that all MT5 work is research-only with the truthful split: the Research Desk and bridge are read-only; the separately installed Plan 3 EA may execute Gold on exact demo accounts only after local operator enablement. +- In R6, link the approved Plan 3 design and restate HFM-first, IC-second, EURUSD-shadow-only, no-live scope. +- In R7, clarify that UTA approval governs app/API/AI-managed orders, while the independently approved broker-local MT5 demo EA is governed by R6 and exposes no remote order command. +- In Success Criteria, require zero Research/AI execution endpoints, both demo-account bindings, broker-confirmed stop protection, and HFM evidence before IC promotion. + +- [ ] **Step 2: Run the complete TypeScript test slice** + +Run: + +```powershell +pnpm vitest run src/domain/mt5 src/task/mt5-decision-scheduler.spec.ts src/task/mt5-outcome-importer.spec.ts src/webui/routes/research.spec.ts ui/src/components/research/__tests__/Mt5ExecutionStatusCard.spec.tsx +``` + +Expected: all tests PASS with zero failures. + +- [ ] **Step 3: Run builds** + +Run: `pnpm -F open-alice-ui build` + +Expected: PASS. + +Run: `pnpm -F open-alice-ui build:demo` + +Expected: PASS. + +Run: `pnpm build` + +Expected: PASS. + +- [ ] **Step 4: Run mandatory safety scans** + +Run: + +```powershell +rg -n "live.?mode|ACCOUNT_TRADE_MODE_REAL|EURUSD|martingale|grid|recovery|lot.?growth" tools/mt5/JmbGoldmineDemoCanary src/domain/mt5/demo-canary-source.spec.ts +``` + +Expected: no live-mode input/bypass, no EURUSD allowlist, and no martingale/grid/recovery/lot-growth implementation. A negative `ACCOUNT_TRADE_MODE_REAL` rejection test is allowed. + +Run: + +```powershell +rg -n "OrderSend|OrderCheck" tools/mt5/JmbGoldmineDemoCanary +``` + +Expected: matches only `JmbCanaryTradeGateway.mqh`. + +Run: + +```powershell +rg -n "accountLogin|account_login|expectedAccountLogin|expected_account_login" src/webui/routes/research.ts ui/src/api/research.ts ui/src/pages/ResearchDashboardPage.tsx ui/src/components/research ui/src/demo/handlers/research.ts +``` + +Expected: no matches. + +- [ ] **Step 5: Verify Stage 0 on both demo terminals** + +The operator compiles with `0 errors, 0 warnings`, attaches HFM and IC Gold with execution disabled and kill switch on, and confirms current status files update for at least two timer cycles with zero orders and zero new positions. Fresh completed-D1 files must use a date later than the stale `2026-06-23` artifact before any canary stage can be considered. + +- [ ] **Step 6: Commit Task 10** + +```powershell +git add -- tools/mt5/README.md docs/mt5-data-and-training-protocol.md docs/PRD.md +git commit -m "docs: add mt5 demo canary operations" +``` + +--- + +## Post-Implementation Human Gates + +These are operational promotion gates, not code tasks and not automatic scheduler actions. + +1. Review Stage 0 evidence and leave both EAs execution-disabled if any gate differs from the existing status-only shell. +2. Build an HFM `canary_ready` cost model and write HFM `hfm_canary` policy using the operator-only script. +3. Explicitly set the HFM expected demo account login locally, set `InpDemoExecutionEnabled=true`, and turn off the HFM kill switch. +4. Verify one HFM decision through durable pre-request event, broker result, stop protection, and restart reconciliation. +5. If HFM evidence passes, write IC `ic_canary` policy and repeat the ceremony on IC Markets. +6. Only after both canaries pass, write `both_demo` policies. EURUSD remains shadow-only. +7. Any unprotected fill, unknown result, reconciliation mismatch, non-demo identity, or missing cost evidence returns the affected broker to `status_only`. diff --git a/docs/superpowers/plans/2026-07-13-jmb-goldmine-learning-foundation.md b/docs/superpowers/plans/2026-07-13-jmb-goldmine-learning-foundation.md new file mode 100644 index 000000000..c08f57117 --- /dev/null +++ b/docs/superpowers/plans/2026-07-13-jmb-goldmine-learning-foundation.md @@ -0,0 +1,875 @@ +# JMB Goldmine Learning Foundation Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build the first safe learning foundation for JMB Goldmine Demo/Paper Autopilot V1: account-scoped MT5 trade-history ingestion, origin labelling, per-broker/symbol learning states, and Research Desk visibility for HFM and IC Markets demo Gold/EURUSD. + +**Architecture:** This plan does not enable order execution. It adds a typed TypeScript domain layer that reads an append-only MT5 trade ledger exported under MetaTrader Common Files, summarizes learning state per configured broker/symbol, and exposes the status through the existing Research Desk API/UI. MQL5 export support is specified as a separate read-only exporter file that writes CSV rows; the app parser is testable without a terminal. + +**Tech Stack:** TypeScript, Vitest, Hono routes, React Research Dashboard, MQL5 CSV exporter, local files under MetaTrader Common Files and `~/.openalice/data/research`. + +## Global Constraints + +- V1 is demo/paper only. +- No live account trading. +- No LLM or workspace agent may submit an order, click buy/sell, change live settings, or sit in the tick-by-tick execution path. +- The MT5 EA is the only component allowed to place demo orders, and order execution is outside this plan. +- AI agents may research, score, review, journal, recommend, or veto. +- Risk rules are fixed configuration, not self-modified by AI. +- A persistent kill switch must block new entries in execution plans. +- If broker/account mode cannot be proven to be demo, the system must refuse progression toward demo automation. +- Gold/XAUUSD and EURUSD are included for HFM demo and IC Markets demo. +- EURUSD learns immediately but cannot become demo-trade eligible until its own candidate gate passes. +- Do not change unrelated dirty working-tree files. + +--- + +## Scope split + +The approved design spans several independent subsystems: trade-history import, broker cost modelling, daily learning jobs, Research Desk status, shadow signals, demo EA risk gates, and demo execution. This plan implements the first independently testable slice only: + +1. Read and validate MT5 trade-history CSV exports. +2. Summarize learning state for HFM/IC Gold and EURUSD. +3. Add Research Desk API/UI evidence that the system is learning or blocked. +4. Add a read-only MQL5 trade-history exporter source file for the user to compile/attach. + +Broker-cost modelling, scheduler automation, EA risk shell, and demo order execution must be covered by follow-up plans after this foundation is passing. + +## File structure + +- Create `src/domain/mt5/trade-ledger.ts` + - Parses and summarizes MT5 order/deal CSV files. + - Has no UI, no Hono dependency, and no order execution behavior. +- Create `src/domain/mt5/trade-ledger.spec.ts` + - Covers parser, origin labelling, account-mode blocking, symbol filtering, and learning-state summaries. +- Modify `src/webui/routes/research.ts` + - Adds trade ledger root config, ledger summary read, and per-instrument learning state in the JSON response. +- Modify `ui/src/api/research.ts` + - Adds typed fields returned by the research route. +- Modify `ui/src/pages/ResearchDashboardPage.tsx` + - Displays learning state and trade-history summary without showing it as trading approval. +- Create `tools/mt5/ExportMt5TradeLedger.mq5` + - Read-only MQL5 script/EA source that exports MT5 history to CSV in account-scoped folders. +- Modify `tools/mt5/README.md` + - Documents how to compile and run the exporter, and repeats the demo-only/no-execution boundary. + +--- + +### Task 1: Add MT5 trade-ledger parser and learning summary + +**Files:** +- Create: `C:\Users\mwbri\Documents\Open Alice Trading bot\OpenAlice\src\domain\mt5\trade-ledger.ts` +- Create: `C:\Users\mwbri\Documents\Open Alice Trading bot\OpenAlice\src\domain\mt5\trade-ledger.spec.ts` + +**Interfaces:** +- Consumes: CSV files under a root shaped like `///deals.csv`. +- Produces: + - `parseMt5TradeLedgerCsv(text: string): Mt5TradeLedgerRow[]` + - `deriveMt5TradeOrigin(row: Pick): Mt5TradeOrigin` + - `summarizeMt5TradeLedger(root: string, broker: string, symbol: string, now?: Date): Promise` + +- [ ] **Step 1: Write the failing parser and summary tests** + +Create `src/domain/mt5/trade-ledger.spec.ts` with: + +```ts +import { afterEach, describe, expect, it } from 'vitest' +import { mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { deriveMt5TradeOrigin, parseMt5TradeLedgerCsv, summarizeMt5TradeLedger } from './trade-ledger.js' + +const directories: string[] = [] + +afterEach(async () => { + await Promise.all(directories.splice(0).map((directory) => rm(directory, { recursive: true, force: true }))) +}) + +describe('parseMt5TradeLedgerCsv', () => { + it('parses deal rows and keeps tickets as strings', () => { + const rows = parseMt5TradeLedgerCsv([ + 'account_mode,server,login,broker,symbol,deal_ticket,order_ticket,position_id,time,entry,type,reason,volume,price,commission,fee,swap,profit,magic,comment', + 'demo,HFM-Demo,123456,hfmarkets,XAUUSD,987654321012345,123456789012345,555,2026-07-13T01:02:03.000Z,out,buy,client,0.01,2410.25,-0.07,0,-0.01,4.25,0,manual close', + ].join('\n')) + + expect(rows).toHaveLength(1) + expect(rows[0]).toMatchObject({ + accountMode: 'demo', + broker: 'hfmarkets', + symbol: 'XAUUSD', + dealTicket: '987654321012345', + orderTicket: '123456789012345', + positionId: '555', + volume: 0.01, + profit: 4.25, + }) + }) + + it('rejects malformed rows with a clear error', () => { + expect(() => parseMt5TradeLedgerCsv('account_mode,server\nonly-one-column')).toThrow('Malformed MT5 trade ledger row 2') + }) +}) + +describe('deriveMt5TradeOrigin', () => { + it('labels client zero-magic trades as manual', () => { + expect(deriveMt5TradeOrigin({ magic: 0, reason: 'client', comment: 'closed from terminal' })).toBe('manual') + }) + + it('labels non-zero magic trades as ea', () => { + expect(deriveMt5TradeOrigin({ magic: 880001, reason: 'expert', comment: 'JMB Goldmine' })).toBe('ea') + }) + + it('labels balance operations and unknown reasons separately', () => { + expect(deriveMt5TradeOrigin({ magic: 0, reason: 'balance', comment: 'deposit' })).toBe('other') + expect(deriveMt5TradeOrigin({ magic: 0, reason: '', comment: '' })).toBe('unknown') + }) +}) + +describe('summarizeMt5TradeLedger', () => { + it('summarizes fresh demo trade history for a broker symbol', async () => { + const root = await mkdtemp(join(tmpdir(), 'openalice-ledger-')) + directories.push(root) + const directory = join(root, 'hfmarkets', 'XAUUSD') + await mkdir(directory, { recursive: true }) + await writeFile(join(directory, 'deals.csv'), [ + 'account_mode,server,login,broker,symbol,deal_ticket,order_ticket,position_id,time,entry,type,reason,volume,price,commission,fee,swap,profit,magic,comment', + 'demo,HFM-Demo,123456,hfmarkets,XAUUSD,1,11,101,2026-07-13T01:00:00.000Z,out,buy,client,0.01,2410.25,-0.07,0,-0.01,4.25,0,manual close', + 'demo,HFM-Demo,123456,hfmarkets,XAUUSD,2,12,102,2026-07-13T02:00:00.000Z,out,sell,expert,0.01,2408.25,-0.07,0,-0.01,-1.25,880001,JMB Goldmine demo', + ].join('\n')) + + const summary = await summarizeMt5TradeLedger(root, 'hfmarkets', 'XAUUSD', new Date('2026-07-13T02:01:00.000Z')) + + expect(summary.state).toBe('learning') + expect(summary.totalDeals).toBe(2) + expect(summary.manualDeals).toBe(1) + expect(summary.eaDeals).toBe(1) + expect(summary.netProfit).toBeCloseTo(2.84) + expect(summary.accountMode).toBe('demo') + }) + + it('blocks non-demo trade history from progression', async () => { + const root = await mkdtemp(join(tmpdir(), 'openalice-ledger-')) + directories.push(root) + const directory = join(root, 'icmarkets', 'EURUSD') + await mkdir(directory, { recursive: true }) + await writeFile(join(directory, 'deals.csv'), [ + 'account_mode,server,login,broker,symbol,deal_ticket,order_ticket,position_id,time,entry,type,reason,volume,price,commission,fee,swap,profit,magic,comment', + 'real,IC-Live,123456,icmarkets,EURUSD,1,11,101,2026-07-13T01:00:00.000Z,out,buy,client,0.01,1.17000,-0.07,0,0,1.25,0,manual close', + ].join('\n')) + + const summary = await summarizeMt5TradeLedger(root, 'icmarkets', 'EURUSD', new Date('2026-07-13T02:01:00.000Z')) + + expect(summary.state).toBe('blocked') + expect(summary.detail).toContain('non-demo') + }) +}) +``` + +- [ ] **Step 2: Run the focused test and verify it fails** + +Run: + +```powershell +pnpm vitest run src/domain/mt5/trade-ledger.spec.ts +``` + +Expected: FAIL because `src/domain/mt5/trade-ledger.ts` does not exist. + +- [ ] **Step 3: Add the minimal parser and summary implementation** + +Create `src/domain/mt5/trade-ledger.ts` with: + +```ts +import { readFile, stat } from 'node:fs/promises' +import { join } from 'node:path' + +export type Mt5TradeOrigin = 'manual' | 'ea' | 'other' | 'unknown' +export type Mt5TradeLedgerState = 'no_data' | 'learning' | 'blocked' | 'stale' + +export interface Mt5TradeLedgerRow { + accountMode: string + server: string + login: string + broker: string + symbol: string + dealTicket: string + orderTicket: string + positionId: string + time: string + entry: string + type: string + reason: string + volume: number + price: number + commission: number + fee: number + swap: number + profit: number + magic: number + comment: string + origin: Mt5TradeOrigin +} + +export interface Mt5TradeLedgerSummary { + state: Mt5TradeLedgerState + label: string + detail: string + broker: string + symbol: string + accountMode: string | null + server: string | null + lastDealTime: string | null + lastUpdated: string | null + totalDeals: number + manualDeals: number + eaDeals: number + otherDeals: number + unknownDeals: number + netProfit: number +} + +const STALE_AFTER_MS = 24 * 60 * 60_000 + +function parseCsvLine(line: string): string[] { + const cells: string[] = [] + let current = '' + let quoted = false + for (let index = 0; index < line.length; index += 1) { + const character = line[index] + if (character === '"') { + if (quoted && line[index + 1] === '"') { + current += '"' + index += 1 + } else { + quoted = !quoted + } + } else if (character === ',' && !quoted) { + cells.push(current) + current = '' + } else { + current += character + } + } + cells.push(current) + return cells +} + +function numberField(row: Record, key: string, lineNumber: number): number { + const value = row[key] ?? '' + const parsed = Number(value) + if (!Number.isFinite(parsed)) throw new Error(`Invalid numeric MT5 trade ledger field "${key}" on row ${lineNumber}`) + return parsed +} + +export function deriveMt5TradeOrigin(row: Pick): Mt5TradeOrigin { + const reason = row.reason.toLowerCase() + const comment = row.comment.toLowerCase() + if (row.magic !== 0 || reason === 'expert' || comment.includes('jmb goldmine')) return 'ea' + if (reason === 'client' || reason === 'mobile' || reason === 'web') return 'manual' + if (reason === 'balance' || reason === 'correction' || reason === 'charge') return 'other' + return 'unknown' +} + +export function parseMt5TradeLedgerCsv(text: string): Mt5TradeLedgerRow[] { + const lines = text.trim().split(/\r?\n/).filter(Boolean) + if (lines.length === 0) return [] + const headers = parseCsvLine(lines[0]!).map((header) => header.trim()) + return lines.slice(1).map((line, index) => { + const lineNumber = index + 2 + const values = parseCsvLine(line) + if (values.length !== headers.length) throw new Error(`Malformed MT5 trade ledger row ${lineNumber}`) + const raw = Object.fromEntries(headers.map((header, headerIndex) => [header, values[headerIndex]!.trim()])) + const base = { + accountMode: raw['account_mode'] ?? '', + server: raw['server'] ?? '', + login: raw['login'] ?? '', + broker: raw['broker'] ?? '', + symbol: raw['symbol'] ?? '', + dealTicket: raw['deal_ticket'] ?? '', + orderTicket: raw['order_ticket'] ?? '', + positionId: raw['position_id'] ?? '', + time: raw['time'] ?? '', + entry: raw['entry'] ?? '', + type: raw['type'] ?? '', + reason: raw['reason'] ?? '', + volume: numberField(raw, 'volume', lineNumber), + price: numberField(raw, 'price', lineNumber), + commission: numberField(raw, 'commission', lineNumber), + fee: numberField(raw, 'fee', lineNumber), + swap: numberField(raw, 'swap', lineNumber), + profit: numberField(raw, 'profit', lineNumber), + magic: numberField(raw, 'magic', lineNumber), + comment: raw['comment'] ?? '', + } + return { ...base, origin: deriveMt5TradeOrigin(base) } + }) +} + +export async function summarizeMt5TradeLedger( + root: string, + broker: string, + symbol: string, + now = new Date(), +): Promise { + const path = join(root, broker, symbol, 'deals.csv') + let text: string + let modified: Date + try { + const result = await Promise.all([readFile(path, 'utf8'), stat(path).then((entry) => entry.mtime)]) + text = result[0] + modified = result[1] + } catch { + return { + state: 'no_data', + label: 'Awaiting trade history', + detail: 'Run the read-only MT5 trade ledger exporter for this demo account and symbol.', + broker, + symbol, + accountMode: null, + server: null, + lastDealTime: null, + lastUpdated: null, + totalDeals: 0, + manualDeals: 0, + eaDeals: 0, + otherDeals: 0, + unknownDeals: 0, + netProfit: 0, + } + } + + const rows = parseMt5TradeLedgerCsv(text).filter((row) => row.broker === broker && row.symbol === symbol) + const first = rows[0] + const lastDealTime = rows.map((row) => row.time).sort().at(-1) ?? null + const totalMoney = rows.reduce((total, row) => total + row.profit + row.commission + row.fee + row.swap, 0) + const base = { + broker, + symbol, + accountMode: first?.accountMode ?? null, + server: first?.server ?? null, + lastDealTime, + lastUpdated: modified.toISOString(), + totalDeals: rows.length, + manualDeals: rows.filter((row) => row.origin === 'manual').length, + eaDeals: rows.filter((row) => row.origin === 'ea').length, + otherDeals: rows.filter((row) => row.origin === 'other').length, + unknownDeals: rows.filter((row) => row.origin === 'unknown').length, + netProfit: Number(totalMoney.toFixed(2)), + } + if (first && first.accountMode !== 'demo') { + return { ...base, state: 'blocked', label: 'Trade history blocked', detail: 'The ledger contains non-demo account history, so it cannot unlock demo automation.' } + } + if (now.getTime() - modified.getTime() > STALE_AFTER_MS) { + return { ...base, state: 'stale', label: 'Trade history stale', detail: 'The trade ledger has not been refreshed in the last 24 hours.' } + } + return { ...base, state: 'learning', label: 'Learning from demo history', detail: 'Manual and EA demo trades are available for review and journaling.' } +} +``` + +- [ ] **Step 4: Run the focused test and verify it passes** + +Run: + +```powershell +pnpm vitest run src/domain/mt5/trade-ledger.spec.ts +``` + +Expected: PASS for all tests in `trade-ledger.spec.ts`. + +- [ ] **Step 5: Commit Task 1** + +Run: + +```powershell +git add src/domain/mt5/trade-ledger.ts src/domain/mt5/trade-ledger.spec.ts +git commit -m "feat: add mt5 trade ledger summary" +``` + +Expected: one commit containing only the parser and tests. + +--- + +### Task 2: Add trade-ledger learning state to the Research API + +**Files:** +- Modify: `C:\Users\mwbri\Documents\Open Alice Trading bot\OpenAlice\src\webui\routes\research.ts` +- Test: `C:\Users\mwbri\Documents\Open Alice Trading bot\OpenAlice\src\domain\mt5\trade-ledger.spec.ts` + +**Interfaces:** +- Consumes: `summarizeMt5TradeLedger(root, broker, symbol, now?)`. +- Produces: Each research instrument JSON object includes `learning`. + +- [ ] **Step 1: Add the import and trade ledger root constant** + +Modify the imports in `src/webui/routes/research.ts`: + +```ts +import { readMt5ReadOnlyBridge } from '../../domain/mt5/read-only-bridge.js' +import { summarizeMt5TradeLedger } from '../../domain/mt5/trade-ledger.js' +``` + +Add this constant after `MT5_BRIDGE_ROOT`: + +```ts +const MT5_TRADE_LEDGER_ROOT = process.env['OPENALICE_MT5_TRADE_LEDGER_ROOT'] ?? join( + process.env['APPDATA'] ?? join(homedir(), 'AppData', 'Roaming'), + 'MetaQuotes', 'Terminal', 'Common', 'Files', 'OpenAliceMt5TradeLedgerV1', +) +``` + +- [ ] **Step 2: Add ledger summary to each instrument** + +Replace the per-instrument Promise block: + +```ts +const [exportData, report, walkForward, bridge] = await Promise.all([ + inspectExport(instrument.broker, instrument.symbol), + readReport(instrument.artifact), + readReport(instrument.walkForwardArtifact), + readMt5ReadOnlyBridge(MT5_BRIDGE_ROOT, instrument.broker, instrument.bridgeSymbol ?? instrument.symbol), +]) +``` + +with: + +```ts +const bridgeSymbol = instrument.bridgeSymbol ?? instrument.symbol +const [exportData, report, walkForward, bridge, learning] = await Promise.all([ + inspectExport(instrument.broker, instrument.symbol), + readReport(instrument.artifact), + readReport(instrument.walkForwardArtifact), + readMt5ReadOnlyBridge(MT5_BRIDGE_ROOT, instrument.broker, bridgeSymbol), + summarizeMt5TradeLedger(MT5_TRADE_LEDGER_ROOT, instrument.broker, bridgeSymbol), +]) +``` + +Then add `learning` to the returned instrument object: + +```ts +return { + ...instrument, + export: exportData, + report, + walkForward, + bridge, + learning, + quality: qualityFor(validationReport, instrument.broker, instrument.symbol, exportData.available), + evidence: evidenceFor(report), +} +``` + +- [ ] **Step 3: Add summary counts** + +Add this line next to `readyDemoBridges`: + +```ts +const learningInstruments = instruments.filter((instrument) => instrument.learning.state === 'learning').length +``` + +Add `tradeLedgerRoot` and `learningInstruments` to `summary`: + +```ts +summary: { + exportRoot: MT5_EXPORT_ROOT, + tradeLedgerRoot: MT5_TRADE_LEDGER_ROOT, + instrumentsWithData: instruments.filter((instrument) => instrument.export.available).length, + completedBaselines, + completedWalkForwards, + readyDemoBridges, + learningInstruments, + validatedInstruments, + hfmReady, + experimentRuns: experimentLedger?.runs.length ?? 0, +}, +``` + +- [ ] **Step 4: Add learning stage** + +Add this stage between `bridge` and `demo`: + +```ts +{ key: 'learning', label: 'Trade-history learning', state: learningInstruments === INSTRUMENTS.length ? 'complete' : learningInstruments > 0 ? 'next' : 'waiting', detail: learningInstruments > 0 ? `${learningInstruments}/${INSTRUMENTS.length} broker-symbol ledgers are fresh and demo-only.` : 'Run the MT5 trade ledger exporter so manual and demo trades can be reviewed.' }, +``` + +- [ ] **Step 5: Run route-adjacent tests** + +Run: + +```powershell +pnpm vitest run src/domain/mt5/read-only-bridge.spec.ts src/domain/mt5/trade-ledger.spec.ts +``` + +Expected: PASS. + +- [ ] **Step 6: Commit Task 2** + +Run: + +```powershell +git add src/webui/routes/research.ts +git commit -m "feat: expose mt5 learning state in research api" +``` + +Expected: one commit modifying the research route only. + +--- + +### Task 3: Display learning status in the Research Dashboard UI + +**Files:** +- Modify: `C:\Users\mwbri\Documents\Open Alice Trading bot\OpenAlice\ui\src\api\research.ts` +- Modify: `C:\Users\mwbri\Documents\Open Alice Trading bot\OpenAlice\ui\src\pages\ResearchDashboardPage.tsx` + +**Interfaces:** +- Consumes: `learning` object returned per instrument by `/api/research`. +- Produces: UI copy that says learning history is evidence, not trading approval. + +- [ ] **Step 1: Extend the Research API types** + +In `ui/src/api/research.ts`, add: + +```ts +export type Mt5LearningState = 'no_data' | 'learning' | 'blocked' | 'stale' + +export interface Mt5TradeLedgerSummary { + state: Mt5LearningState + label: string + detail: string + broker: string + symbol: string + accountMode: string | null + server: string | null + lastDealTime: string | null + lastUpdated: string | null + totalDeals: number + manualDeals: number + eaDeals: number + otherDeals: number + unknownDeals: number + netProfit: number +} +``` + +Find the instrument response interface and add: + +```ts +learning: Mt5TradeLedgerSummary +``` + +Find the summary interface and add: + +```ts +tradeLedgerRoot: string +learningInstruments: number +``` + +- [ ] **Step 2: Add UI tone helper** + +In `ui/src/pages/ResearchDashboardPage.tsx`, add this helper near the existing tone helpers: + +```ts +function learningTone(state: string) { + if (state === 'learning') return 'green' + if (state === 'blocked') return 'red' + if (state === 'stale') return 'amber' + return 'muted' +} +``` + +- [ ] **Step 3: Render learning state per instrument** + +In each instrument card, add a learning block near bridge status: + +```tsx +
+ Trade-history learning + {instrument.learning.label} + {instrument.learning.detail} + + Deals: {instrument.learning.totalDeals} · Manual: {instrument.learning.manualDeals} · EA: {instrument.learning.eaDeals} · Net: {instrument.learning.netProfit.toFixed(2)} + +
+``` + +If the page uses a different metric class pattern, keep the same markup content and use the existing card/metric wrapper. + +- [ ] **Step 4: Add the safety copy** + +Add this copy near the Research Dashboard disclaimer: + +```tsx +

+ Trade-history learning imports manual and demo outcomes for review. It is not approval for live trading and it cannot submit orders. +

+``` + +- [ ] **Step 5: Run TypeScript/UI checks** + +Run: + +```powershell +pnpm vitest run ui/src +``` + +Expected: PASS, or if the repo has no UI-specific tests, Vitest should report no matching test files without TypeScript compile errors. If the command fails because the project does not define UI tests, run: + +```powershell +pnpm test -- --runInBand +``` + +Expected: existing unrelated failures are documented; new TypeScript errors from this task are fixed before commit. + +- [ ] **Step 6: Commit Task 3** + +Run: + +```powershell +git add ui/src/api/research.ts ui/src/pages/ResearchDashboardPage.tsx +git commit -m "feat: show mt5 learning status in research dashboard" +``` + +Expected: one commit containing only UI/API type changes. + +--- + +### Task 4: Add read-only MQL5 trade ledger exporter source + +**Files:** +- Create: `C:\Users\mwbri\Documents\Open Alice Trading bot\OpenAlice\tools\mt5\ExportMt5TradeLedger.mq5` +- Modify: `C:\Users\mwbri\Documents\Open Alice Trading bot\OpenAlice\tools\mt5\README.md` + +**Interfaces:** +- Consumes: MT5 account history through `HistorySelect`, `HistoryDealsTotal`, and `HistoryDealGet*`. +- Produces: `Common Files/OpenAliceMt5TradeLedgerV1///deals.csv`. + +- [ ] **Step 1: Create the exporter source** + +Create `tools/mt5/ExportMt5TradeLedger.mq5` with: + +```mql5 +#property strict +#property script_show_inputs + +input string InpBrokerId = "hfmarkets"; +input string InpSymbol = "XAUUSD"; +input int InpHistoryDays = 30; + +string AccountModeLabel() +{ + long mode = AccountInfoInteger(ACCOUNT_TRADE_MODE); + if(mode == ACCOUNT_TRADE_MODE_DEMO) return "demo"; + if(mode == ACCOUNT_TRADE_MODE_REAL) return "real"; + return "contest"; +} + +string DealEntryLabel(long value) +{ + if(value == DEAL_ENTRY_IN) return "in"; + if(value == DEAL_ENTRY_OUT) return "out"; + if(value == DEAL_ENTRY_INOUT) return "inout"; + if(value == DEAL_ENTRY_OUT_BY) return "out_by"; + return IntegerToString((int)value); +} + +string DealTypeLabel(long value) +{ + if(value == DEAL_TYPE_BUY) return "buy"; + if(value == DEAL_TYPE_SELL) return "sell"; + if(value == DEAL_TYPE_BALANCE) return "balance"; + if(value == DEAL_TYPE_CREDIT) return "credit"; + if(value == DEAL_TYPE_CHARGE) return "charge"; + if(value == DEAL_TYPE_CORRECTION) return "correction"; + return IntegerToString((int)value); +} + +string DealReasonLabel(long value) +{ + if(value == DEAL_REASON_CLIENT) return "client"; + if(value == DEAL_REASON_MOBILE) return "mobile"; + if(value == DEAL_REASON_WEB) return "web"; + if(value == DEAL_REASON_EXPERT) return "expert"; + return IntegerToString((int)value); +} + +string CsvEscape(string value) +{ + StringReplace(value, "\"", "\"\""); + if(StringFind(value, ",") >= 0 || StringFind(value, "\"") >= 0) + return "\"" + value + "\""; + return value; +} + +void OnStart() +{ + string symbol = InpSymbol == "" ? _Symbol : InpSymbol; + datetime toTime = TimeCurrent(); + datetime fromTime = toTime - (InpHistoryDays * 86400); + if(!HistorySelect(fromTime, toTime)) + { + Print("HistorySelect failed: ", GetLastError()); + return; + } + + string directory = "OpenAliceMt5TradeLedgerV1\\" + InpBrokerId + "\\" + symbol; + FolderCreate("OpenAliceMt5TradeLedgerV1", FILE_COMMON); + FolderCreate("OpenAliceMt5TradeLedgerV1\\" + InpBrokerId, FILE_COMMON); + FolderCreate(directory, FILE_COMMON); + + string path = directory + "\\deals.csv"; + int handle = FileOpen(path, FILE_WRITE | FILE_CSV | FILE_COMMON | FILE_ANSI, ','); + if(handle == INVALID_HANDLE) + { + Print("FileOpen failed for ", path, ": ", GetLastError()); + return; + } + + FileWrite(handle, "account_mode", "server", "login", "broker", "symbol", "deal_ticket", "order_ticket", "position_id", "time", "entry", "type", "reason", "volume", "price", "commission", "fee", "swap", "profit", "magic", "comment"); + + int total = HistoryDealsTotal(); + for(int index = 0; index < total; index++) + { + ulong ticket = HistoryDealGetTicket(index); + string dealSymbol = HistoryDealGetString(ticket, DEAL_SYMBOL); + if(dealSymbol != symbol) continue; + + datetime dealTime = (datetime)HistoryDealGetInteger(ticket, DEAL_TIME); + string isoTime = TimeToString(dealTime, TIME_DATE | TIME_SECONDS); + StringReplace(isoTime, ".", "-"); + StringReplace(isoTime, " ", "T"); + isoTime = isoTime + ".000Z"; + + FileWrite( + handle, + AccountModeLabel(), + AccountInfoString(ACCOUNT_SERVER), + IntegerToString((int)AccountInfoInteger(ACCOUNT_LOGIN)), + InpBrokerId, + symbol, + IntegerToString((long)ticket), + IntegerToString((long)HistoryDealGetInteger(ticket, DEAL_ORDER)), + IntegerToString((long)HistoryDealGetInteger(ticket, DEAL_POSITION_ID)), + isoTime, + DealEntryLabel(HistoryDealGetInteger(ticket, DEAL_ENTRY)), + DealTypeLabel(HistoryDealGetInteger(ticket, DEAL_TYPE)), + DealReasonLabel(HistoryDealGetInteger(ticket, DEAL_REASON)), + DoubleToString(HistoryDealGetDouble(ticket, DEAL_VOLUME), 2), + DoubleToString(HistoryDealGetDouble(ticket, DEAL_PRICE), _Digits), + DoubleToString(HistoryDealGetDouble(ticket, DEAL_COMMISSION), 2), + DoubleToString(HistoryDealGetDouble(ticket, DEAL_FEE), 2), + DoubleToString(HistoryDealGetDouble(ticket, DEAL_SWAP), 2), + DoubleToString(HistoryDealGetDouble(ticket, DEAL_PROFIT), 2), + IntegerToString((long)HistoryDealGetInteger(ticket, DEAL_MAGIC)), + CsvEscape(HistoryDealGetString(ticket, DEAL_COMMENT)) + ); + } + + FileClose(handle); + Print("JMB Goldmine trade ledger exported: ", path); +} +``` + +- [ ] **Step 2: Document compile and run instructions** + +Append this section to `tools/mt5/README.md`: + +```md +## Read-only trade ledger exporter + +`ExportMt5TradeLedger.mq5` exports MT5 deal history for one broker/symbol into MetaTrader Common Files: + +`OpenAliceMt5TradeLedgerV1///deals.csv` + +Use it on demo accounts first: + +1. Open MetaEditor from the target MT5 terminal. +2. Copy or open `tools/mt5/ExportMt5TradeLedger.mq5`. +3. Compile it. +4. Run it once per broker/symbol with: + - HFM demo Gold: `InpBrokerId=hfmarkets`, `InpSymbol=XAUUSD` + - HFM demo EURUSD: `InpBrokerId=hfmarkets`, `InpSymbol=EURUSD` + - IC Markets demo Gold: `InpBrokerId=icmarkets`, `InpSymbol=XAUUSD` + - IC Markets demo EURUSD: `InpBrokerId=icmarkets`, `InpSymbol=EURUSD` + +The exporter is read-only. It does not submit, modify, or close orders. If the exported account mode is not `demo`, JMB Goldmine must show the ledger as blocked for demo-autopilot progression. +``` + +- [ ] **Step 3: Run text-level verification** + +Run: + +```powershell +Select-String -Path tools/mt5/ExportMt5TradeLedger.mq5 -Pattern 'OrderSend|PositionClose|trade.Buy|trade.Sell' +``` + +Expected: no matches. + +- [ ] **Step 4: Commit Task 4** + +Run: + +```powershell +git add tools/mt5/ExportMt5TradeLedger.mq5 tools/mt5/README.md +git commit -m "feat: add mt5 trade ledger exporter" +``` + +Expected: one commit containing the read-only exporter and documentation. + +--- + +### Task 5: Add learning-foundation verification run + +**Files:** +- Modify only files from Tasks 1-4 if verification reveals a direct defect. + +**Interfaces:** +- Consumes: all previous task outputs. +- Produces: verified learning foundation with no execution authority. + +- [ ] **Step 1: Run all MT5 domain tests** + +Run: + +```powershell +pnpm vitest run src/domain/mt5 +``` + +Expected: PASS. + +- [ ] **Step 2: Verify no order API was introduced in app/domain files** + +Run: + +```powershell +rg "OrderSend|trade\\.Buy|trade\\.Sell|PositionClose|CTrade" src tools/mt5 +``` + +Expected: either no matches, or matches only in files that are explicitly documented execution-capable from prior work. `tools/mt5/ExportMt5TradeLedger.mq5` must not appear in the results. + +- [ ] **Step 3: Verify git only includes intended learning-foundation files** + +Run: + +```powershell +git status --short +``` + +Expected: existing unrelated dirty files may remain, but newly staged/committed changes from this plan should be limited to: + +```text +src/domain/mt5/trade-ledger.ts +src/domain/mt5/trade-ledger.spec.ts +src/webui/routes/research.ts +ui/src/api/research.ts +ui/src/pages/ResearchDashboardPage.tsx +tools/mt5/ExportMt5TradeLedger.mq5 +tools/mt5/README.md +``` + +- [ ] **Step 4: Record completion note** + +Append this line to the final handoff message, not to a source file: + +```text +Learning foundation is complete: Gold and EURUSD on HFM demo and IC Markets demo can be imported, labelled, summarized, and displayed. Demo order execution remains locked for the next plan. +``` + diff --git a/docs/superpowers/plans/2026-07-13-jmb-goldmine-shadow-risk-shell.md b/docs/superpowers/plans/2026-07-13-jmb-goldmine-shadow-risk-shell.md new file mode 100644 index 000000000..618fe34ef --- /dev/null +++ b/docs/superpowers/plans/2026-07-13-jmb-goldmine-shadow-risk-shell.md @@ -0,0 +1,1166 @@ +# JMB Goldmine Shadow Risk Shell Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build the shadow-decision and demo-risk-shell layer so JMB Goldmine starts learning from its own logged decisions before any demo order execution is enabled. + +**Architecture:** The app creates append-only JMB decision records and a latest-decision CSV that an MT5 risk-shell EA can read. The MT5 risk shell validates demo account, symbol, lot, stop, spread, freshness, kill switch, and manual/foreign exposure, then writes gate status only. This plan deliberately does not submit demo orders; it prepares and verifies the deterministic safety shell for a later execution plan. + +**Tech Stack:** TypeScript, Vitest, Hono Research API, React Research Dashboard, MQL5 read/gate/log EA, MetaTrader Common Files CSV/JSONL. + +## Global Constraints + +- Manual trades are supporting broker/cost evidence, not primary strategy labels. +- JMB-generated decisions are the primary learning dataset. +- Shadow mode exists before demo execution. +- Demo execution remains EA-only and deterministic. +- Gold and EURUSD are both included. +- EURUSD remains demo-blocked until its own candidate gate passes. +- Live trading remains out of scope. +- Every decision and skip is logged with reason and gate results. +- This plan must not introduce `OrderSend`, `CTrade`, `trade.Buy`, `trade.Sell`, order modification, or position close logic. +- The MT5 risk shell must log gate status only; actual demo order submission is a later plan. +- If any gate cannot be evaluated, it fails closed. +- Do not change unrelated dirty working-tree files. + +--- + +## Scope split + +This design has three layers: + +1. **Plan 2, this plan:** shadow decisions, latest-decision export, risk-shell gate logging, and UI visibility. +2. **Plan 3, later:** demo order submission from the EA after the risk shell has been verified. +3. **Future live spec, later:** separate live pilot policy, account binding, approval ceremony, and loss cap. + +This plan stops at layer 1. A successful result means the system can say, "JMB would trade / would skip / would be blocked, and here are the exact gates," but it still cannot place orders. + +## File structure + +- Create `src/domain/mt5/decision-record.ts` + - Owns decision record types, CSV serialization for MT5, JSONL persistence, latest decision read/write, and summaries. +- Create `src/domain/mt5/decision-record.spec.ts` + - Tests decision IDs, CSV escaping, append/read behavior, malformed fail-closed behavior, and latest decision summary. +- Create `src/domain/mt5/shadow-decision-engine.ts` + - Pure strategy/risk pre-gate function that turns bridge/learning/trend inputs into `shadow`, `skipped`, or `demo_blocked` decision records. +- Create `src/domain/mt5/shadow-decision-engine.spec.ts` + - Tests Gold shadow decisions, EURUSD demo-blocking, stale bridge skips, and stop-loss-required blocking. +- Create `tools/mt5/run_shadow_decisions.ts` + - Local runner that reads Research artifacts plus bridge/ledger summaries and writes decision JSONL + latest-decision CSV for all four broker/symbol pairs. +- Create `tools/mt5/JmbGoldmineDemoRiskShell.mq5` + - MQL5 EA that reads latest-decision CSV, evaluates gates, and writes `gate_status.csv`; no order submission. +- Modify `tools/mt5/README.md` + - Documents shadow runner and risk shell setup. +- Modify `src/webui/routes/research.ts` + - Adds latest JMB decision summaries per instrument. +- Modify `ui/src/api/research.ts` + - Adds decision summary types. +- Modify `ui/src/pages/ResearchDashboardPage.tsx` + - Displays JMB decision mode and risk-shell status as non-trading approval evidence. + +--- + +### Task 1: Add decision record persistence + +**Files:** +- Create: `C:\Users\mwbri\Documents\Open Alice Trading bot\OpenAlice\src\domain\mt5\decision-record.ts` +- Create: `C:\Users\mwbri\Documents\Open Alice Trading bot\OpenAlice\src\domain\mt5\decision-record.spec.ts` + +**Interfaces:** +- Produces: + - `type JmbDecisionMode = 'shadow' | 'demo_blocked' | 'demo_order_requested' | 'demo_filled' | 'demo_closed' | 'skipped'` + - `type JmbDecisionDirection = 'buy' | 'sell' | 'flat'` + - `interface JmbDecisionRecord` + - `function createJmbDecisionId(input: Pick): string` + - `function serializeLatestDecisionCsv(record: JmbDecisionRecord): string` + - `function parseLatestDecisionCsv(text: string): JmbDecisionRecord` + - `async function appendJmbDecisionRecord(root: string, record: JmbDecisionRecord): Promise` + - `async function writeLatestJmbDecision(root: string, record: JmbDecisionRecord): Promise` + - `async function summarizeLatestJmbDecision(root: string, broker: string, symbol: string, now?: Date): Promise` + +- [ ] **Step 1: Write failing tests** + +Create `src/domain/mt5/decision-record.spec.ts` with: + +```ts +import { afterEach, describe, expect, it } from 'vitest' +import { mkdtemp, readFile, rm } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { + appendJmbDecisionRecord, + createJmbDecisionId, + parseLatestDecisionCsv, + serializeLatestDecisionCsv, + summarizeLatestJmbDecision, + writeLatestJmbDecision, + type JmbDecisionRecord, +} from './decision-record.js' + +const directories: string[] = [] + +afterEach(async () => { + await Promise.all(directories.splice(0).map((directory) => rm(directory, { recursive: true, force: true }))) +}) + +function sampleDecision(overrides: Partial = {}): JmbDecisionRecord { + const base: JmbDecisionRecord = { + schemaVersion: 1, + decisionId: 'decision-1', + createdAt: '2026-07-13T09:00:00.000Z', + broker: 'hfmarkets', + server: 'HFMarketsGlobal-Demo4', + accountMode: 'demo', + symbol: 'XAUUSD', + canonicalInstrument: 'Gold / USD', + strategyVersion: 'daily-trend-v1', + mode: 'shadow', + direction: 'buy', + reasonCode: 'daily_trend_shadow', + reasonDetail: 'Completed daily trend filter is positive.', + entryReferencePrice: 2410.25, + stopLoss: 2402.25, + takeProfit: null, + volume: 0.01, + spread: 0.36, + riskAmount: 0.8, + maxAllowedRisk: 1, + gateResults: [ + { gate: 'account_demo', state: 'pass', detail: 'MT5 reports demo mode' }, + { gate: 'shadow_only', state: 'pass', detail: 'No order submission in Plan 2' }, + ], + orderTicket: null, + positionId: null, + outcome: null, + } + return { ...base, ...overrides } +} + +describe('JMB decision records', () => { + it('creates stable ids from deterministic fields', () => { + const id = createJmbDecisionId(sampleDecision()) + expect(id).toBe(createJmbDecisionId(sampleDecision())) + expect(id).not.toBe(createJmbDecisionId(sampleDecision({ direction: 'sell' }))) + }) + + it('round-trips latest-decision CSV without losing gate results', () => { + const decision = sampleDecision({ reasonDetail: 'Spread, trend, and stop checked' }) + const parsed = parseLatestDecisionCsv(serializeLatestDecisionCsv(decision)) + expect(parsed).toMatchObject({ + broker: 'hfmarkets', + symbol: 'XAUUSD', + mode: 'shadow', + direction: 'buy', + volume: 0.01, + stopLoss: 2402.25, + }) + expect(parsed.gateResults).toHaveLength(2) + }) + + it('writes append-only JSONL and latest CSV under broker symbol folders', async () => { + const root = await mkdtemp(join(tmpdir(), 'jmb-decisions-')) + directories.push(root) + const decision = sampleDecision() + + await appendJmbDecisionRecord(root, decision) + await writeLatestJmbDecision(root, decision) + + const jsonl = await readFile(join(root, 'hfmarkets', 'XAUUSD', 'decisions.jsonl'), 'utf8') + const latest = await readFile(join(root, 'hfmarkets', 'XAUUSD', 'latest_decision.csv'), 'utf8') + expect(jsonl.trim().split('\n')).toHaveLength(1) + expect(parseLatestDecisionCsv(latest).decisionId).toBe('decision-1') + }) + + it('summarizes unreadable latest CSV as blocked instead of throwing', async () => { + const root = await mkdtemp(join(tmpdir(), 'jmb-decisions-')) + directories.push(root) + await writeLatestJmbDecision(root, sampleDecision()) + await import('node:fs/promises').then(({ writeFile }) => writeFile(join(root, 'hfmarkets', 'XAUUSD', 'latest_decision.csv'), 'broken,csv\n1')) + + const summary = await summarizeLatestJmbDecision(root, 'hfmarkets', 'XAUUSD') + + expect(summary.state).toBe('error') + expect(summary.label).toBe('Decision unreadable') + }) +}) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: + +```powershell +pnpm vitest run src/domain/mt5/decision-record.spec.ts +``` + +Expected: FAIL with module not found for `./decision-record.js`. + +- [ ] **Step 3: Implement decision record persistence** + +Create `src/domain/mt5/decision-record.ts` with: + +```ts +import { createHash } from 'node:crypto' +import { mkdir, readFile, stat, writeFile, appendFile } from 'node:fs/promises' +import { join } from 'node:path' + +export type JmbDecisionMode = 'shadow' | 'demo_blocked' | 'demo_order_requested' | 'demo_filled' | 'demo_closed' | 'skipped' +export type JmbDecisionDirection = 'buy' | 'sell' | 'flat' +export type JmbGateState = 'pass' | 'fail' | 'warn' + +export interface JmbGateResult { + gate: string + state: JmbGateState + detail: string +} + +export interface JmbDecisionRecord { + schemaVersion: 1 + decisionId: string + createdAt: string + broker: string + server: string | null + accountMode: string | null + symbol: string + canonicalInstrument: string + strategyVersion: string + mode: JmbDecisionMode + direction: JmbDecisionDirection + reasonCode: string + reasonDetail: string + entryReferencePrice: number | null + stopLoss: number | null + takeProfit: number | null + volume: number + spread: number | null + riskAmount: number | null + maxAllowedRisk: number + gateResults: JmbGateResult[] + orderTicket: string | null + positionId: string | null + outcome: string | null +} + +export interface JmbDecisionSummary { + state: 'no_decision' | 'shadow' | 'demo_blocked' | 'error' + label: string + detail: string + broker: string + symbol: string + lastUpdated: string | null + decision: JmbDecisionRecord | null +} + +const HEADER = [ + 'schema_version', 'decision_id', 'created_at', 'broker', 'server', 'account_mode', 'symbol', + 'canonical_instrument', 'strategy_version', 'mode', 'direction', 'reason_code', 'reason_detail', + 'entry_reference_price', 'stop_loss', 'take_profit', 'volume', 'spread', 'risk_amount', + 'max_allowed_risk', 'gate_results_json', 'order_ticket', 'position_id', 'outcome', +] + +function csvEscape(value: string): string { + if (/[",\r\n]/.test(value)) return `"${value.replace(/"/g, '""')}"` + return value +} + +function parseCsvLine(line: string): string[] { + const cells: string[] = [] + let current = '' + let quoted = false + for (let index = 0; index < line.length; index += 1) { + const character = line[index] + if (character === '"') { + if (quoted && line[index + 1] === '"') { + current += '"' + index += 1 + } else { + quoted = !quoted + } + } else if (character === ',' && !quoted) { + cells.push(current) + current = '' + } else { + current += character + } + } + cells.push(current) + return cells +} + +function numberOrNull(value: string): number | null { + if (value === '') return null + const parsed = Number(value) + if (!Number.isFinite(parsed)) throw new Error(`Invalid numeric decision field: ${value}`) + return parsed +} + +export function createJmbDecisionId(input: Pick): string { + return createHash('sha256') + .update([input.createdAt, input.broker, input.symbol, input.strategyVersion, input.mode, input.direction].join('|')) + .digest('hex') + .slice(0, 24) +} + +export function serializeLatestDecisionCsv(record: JmbDecisionRecord): string { + const values = [ + String(record.schemaVersion), + record.decisionId, + record.createdAt, + record.broker, + record.server ?? '', + record.accountMode ?? '', + record.symbol, + record.canonicalInstrument, + record.strategyVersion, + record.mode, + record.direction, + record.reasonCode, + record.reasonDetail, + record.entryReferencePrice == null ? '' : String(record.entryReferencePrice), + record.stopLoss == null ? '' : String(record.stopLoss), + record.takeProfit == null ? '' : String(record.takeProfit), + String(record.volume), + record.spread == null ? '' : String(record.spread), + record.riskAmount == null ? '' : String(record.riskAmount), + String(record.maxAllowedRisk), + JSON.stringify(record.gateResults), + record.orderTicket ?? '', + record.positionId ?? '', + record.outcome ?? '', + ] + return `${HEADER.join(',')}\n${values.map(csvEscape).join(',')}\n` +} + +export function parseLatestDecisionCsv(text: string): JmbDecisionRecord { + const [headerLine, valueLine] = text.trim().split(/\r?\n/, 2) + if (!headerLine || !valueLine) throw new Error('Decision CSV is missing header or value row') + const headers = parseCsvLine(headerLine) + const values = parseCsvLine(valueLine) + if (headers.join(',') !== HEADER.join(',') || values.length !== HEADER.length) throw new Error('Decision CSV schema mismatch') + const row = Object.fromEntries(headers.map((header, index) => [header, values[index] ?? ''])) + return { + schemaVersion: 1, + decisionId: row['decision_id']!, + createdAt: row['created_at']!, + broker: row['broker']!, + server: row['server'] || null, + accountMode: row['account_mode'] || null, + symbol: row['symbol']!, + canonicalInstrument: row['canonical_instrument']!, + strategyVersion: row['strategy_version']!, + mode: row['mode'] as JmbDecisionMode, + direction: row['direction'] as JmbDecisionDirection, + reasonCode: row['reason_code']!, + reasonDetail: row['reason_detail']!, + entryReferencePrice: numberOrNull(row['entry_reference_price']!), + stopLoss: numberOrNull(row['stop_loss']!), + takeProfit: numberOrNull(row['take_profit']!), + volume: Number(row['volume']), + spread: numberOrNull(row['spread']!), + riskAmount: numberOrNull(row['risk_amount']!), + maxAllowedRisk: Number(row['max_allowed_risk']), + gateResults: JSON.parse(row['gate_results_json']!) as JmbGateResult[], + orderTicket: row['order_ticket'] || null, + positionId: row['position_id'] || null, + outcome: row['outcome'] || null, + } +} + +function decisionDirectory(root: string, broker: string, symbol: string): string { + return join(root, broker, symbol) +} + +export async function appendJmbDecisionRecord(root: string, record: JmbDecisionRecord): Promise { + const directory = decisionDirectory(root, record.broker, record.symbol) + await mkdir(directory, { recursive: true }) + await appendFile(join(directory, 'decisions.jsonl'), `${JSON.stringify(record)}\n`, 'utf8') +} + +export async function writeLatestJmbDecision(root: string, record: JmbDecisionRecord): Promise { + const directory = decisionDirectory(root, record.broker, record.symbol) + await mkdir(directory, { recursive: true }) + await writeFile(join(directory, 'latest_decision.csv'), serializeLatestDecisionCsv(record), 'utf8') +} + +export async function summarizeLatestJmbDecision(root: string, broker: string, symbol: string, now = new Date()): Promise { + const path = join(root, broker, symbol, 'latest_decision.csv') + let text: string + let modified: Date + try { + const result = await Promise.all([readFile(path, 'utf8'), stat(path).then((entry) => entry.mtime)]) + text = result[0] + modified = result[1] + } catch { + return { state: 'no_decision', label: 'No JMB decision yet', detail: 'Run the shadow decision runner before enabling any demo risk shell.', broker, symbol, lastUpdated: null, decision: null } + } + try { + const decision = parseLatestDecisionCsv(text) + const ageMinutes = Math.round((now.getTime() - modified.getTime()) / 60_000) + const state = decision.mode === 'demo_blocked' ? 'demo_blocked' : decision.mode === 'shadow' || decision.mode === 'skipped' ? 'shadow' : 'error' + return { state, label: decision.mode === 'demo_blocked' ? 'Demo blocked by gates' : 'Shadow decision logged', detail: `Latest ${decision.mode} decision is ${ageMinutes} minutes old.`, broker, symbol, lastUpdated: modified.toISOString(), decision } + } catch { + return { state: 'error', label: 'Decision unreadable', detail: 'The latest decision CSV is malformed. The risk shell must fail closed.', broker, symbol, lastUpdated: modified.toISOString(), decision: null } + } +} +``` + +- [ ] **Step 4: Run focused tests** + +Run: + +```powershell +pnpm vitest run src/domain/mt5/decision-record.spec.ts +``` + +Expected: PASS. + +- [ ] **Step 5: Commit Task 1** + +Run: + +```powershell +git add src/domain/mt5/decision-record.ts src/domain/mt5/decision-record.spec.ts +git commit -m "feat: add jmb mt5 decision records" +``` + +Expected: one commit containing only Task 1 files. + +--- + +### Task 2: Add pure shadow decision engine + +**Files:** +- Create: `C:\Users\mwbri\Documents\Open Alice Trading bot\OpenAlice\src\domain\mt5\shadow-decision-engine.ts` +- Create: `C:\Users\mwbri\Documents\Open Alice Trading bot\OpenAlice\src\domain\mt5\shadow-decision-engine.spec.ts` + +**Interfaces:** +- Consumes: `JmbDecisionRecord`, `createJmbDecisionId` from `decision-record.ts`. +- Produces: + - `interface BuildShadowDecisionInput` + - `function buildShadowDecision(input: BuildShadowDecisionInput): JmbDecisionRecord` + +- [ ] **Step 1: Write failing tests** + +Create `src/domain/mt5/shadow-decision-engine.spec.ts` with: + +```ts +import { describe, expect, it } from 'vitest' +import { buildShadowDecision } from './shadow-decision-engine.js' + +const baseInput = { + createdAt: '2026-07-13T10:00:00.000Z', + broker: 'hfmarkets', + server: 'HFMarketsGlobal-Demo4', + accountMode: 'demo', + symbol: 'XAUUSD', + canonicalInstrument: 'Gold / USD', + strategyVersion: 'daily-trend-v1', + bridgeState: 'ready', + learningState: 'learning', + latestDirection: 'uptrend', + bid: 2410, + ask: 2410.36, + spread: 0.36, + maxSpread: 0.75, + volume: 0.01, + maxVolume: 0.01, + stopLoss: 2402, + riskAmount: 0.8, + maxAllowedRisk: 1, + demoCandidateApproved: true, +} as const + +describe('buildShadowDecision', () => { + it('logs a Gold buy shadow decision when gates pass', () => { + const decision = buildShadowDecision(baseInput) + + expect(decision.mode).toBe('shadow') + expect(decision.direction).toBe('buy') + expect(decision.reasonCode).toBe('daily_trend_shadow') + expect(decision.gateResults.every((gate) => gate.state === 'pass')).toBe(true) + }) + + it('keeps EURUSD demo-blocked when its candidate gate is not approved', () => { + const decision = buildShadowDecision({ + ...baseInput, + symbol: 'EURUSD', + canonicalInstrument: 'Euro / USD', + demoCandidateApproved: false, + }) + + expect(decision.mode).toBe('demo_blocked') + expect(decision.direction).toBe('buy') + expect(decision.gateResults.some((gate) => gate.gate === 'candidate_gate' && gate.state === 'fail')).toBe(true) + }) + + it('skips flat when bridge is stale', () => { + const decision = buildShadowDecision({ ...baseInput, bridgeState: 'stale' }) + + expect(decision.mode).toBe('skipped') + expect(decision.direction).toBe('flat') + expect(decision.reasonCode).toBe('gate_blocked') + }) + + it('blocks when stop loss is missing', () => { + const decision = buildShadowDecision({ ...baseInput, stopLoss: null }) + + expect(decision.mode).toBe('skipped') + expect(decision.direction).toBe('flat') + expect(decision.gateResults.some((gate) => gate.gate === 'stop_loss' && gate.state === 'fail')).toBe(true) + }) +}) +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: + +```powershell +pnpm vitest run src/domain/mt5/shadow-decision-engine.spec.ts +``` + +Expected: FAIL with module not found for `./shadow-decision-engine.js`. + +- [ ] **Step 3: Implement the pure decision engine** + +Create `src/domain/mt5/shadow-decision-engine.ts` with: + +```ts +import { createJmbDecisionId, type JmbDecisionDirection, type JmbDecisionRecord, type JmbGateResult } from './decision-record.js' + +export interface BuildShadowDecisionInput { + createdAt: string + broker: string + server: string | null + accountMode: string | null + symbol: string + canonicalInstrument: string + strategyVersion: string + bridgeState: string + learningState: string + latestDirection: 'uptrend' | 'downtrend' | 'flat' + bid: number | null + ask: number | null + spread: number | null + maxSpread: number + volume: number + maxVolume: number + stopLoss: number | null + riskAmount: number | null + maxAllowedRisk: number + demoCandidateApproved: boolean +} + +function gate(gateName: string, state: JmbGateResult['state'], detail: string): JmbGateResult { + return { gate: gateName, state, detail } +} + +function directionFor(latestDirection: BuildShadowDecisionInput['latestDirection']): JmbDecisionDirection { + if (latestDirection === 'uptrend') return 'buy' + if (latestDirection === 'downtrend') return 'sell' + return 'flat' +} + +export function buildShadowDecision(input: BuildShadowDecisionInput): JmbDecisionRecord { + const direction = directionFor(input.latestDirection) + const gateResults: JmbGateResult[] = [ + gate('account_demo', input.accountMode === 'demo' ? 'pass' : 'fail', input.accountMode === 'demo' ? 'MT5 reports demo mode' : 'Account mode is not confirmed demo'), + gate('bridge_ready', input.bridgeState === 'ready' ? 'pass' : 'fail', `Bridge state is ${input.bridgeState}`), + gate('learning_ready', input.learningState === 'learning' ? 'pass' : 'warn', `Learning state is ${input.learningState}`), + gate('spread', input.spread != null && input.spread <= input.maxSpread ? 'pass' : 'fail', input.spread == null ? 'Spread is unavailable' : `${input.spread} <= ${input.maxSpread}`), + gate('volume', input.volume > 0 && input.volume <= input.maxVolume ? 'pass' : 'fail', `${input.volume} <= ${input.maxVolume}`), + gate('stop_loss', input.stopLoss != null ? 'pass' : 'fail', input.stopLoss == null ? 'Stop loss is required' : `Stop loss ${input.stopLoss}`), + gate('risk_amount', input.riskAmount != null && input.riskAmount <= input.maxAllowedRisk ? 'pass' : 'fail', input.riskAmount == null ? 'Risk amount unavailable' : `${input.riskAmount} <= ${input.maxAllowedRisk}`), + gate('candidate_gate', input.demoCandidateApproved ? 'pass' : 'fail', input.demoCandidateApproved ? 'Candidate gate approved for shadow review' : 'Broker/symbol is not approved for demo execution'), + gate('shadow_only', 'pass', 'Plan 2 logs decisions only and submits no orders'), + ] + const hardFailure = gateResults.some((item) => item.state === 'fail' && item.gate !== 'candidate_gate') + const mode = hardFailure ? 'skipped' : input.demoCandidateApproved ? 'shadow' : 'demo_blocked' + const finalDirection = hardFailure ? 'flat' : direction + const record: JmbDecisionRecord = { + schemaVersion: 1, + decisionId: 'pending', + createdAt: input.createdAt, + broker: input.broker, + server: input.server, + accountMode: input.accountMode, + symbol: input.symbol, + canonicalInstrument: input.canonicalInstrument, + strategyVersion: input.strategyVersion, + mode, + direction: finalDirection, + reasonCode: hardFailure ? 'gate_blocked' : 'daily_trend_shadow', + reasonDetail: hardFailure ? 'One or more hard gates failed; no order can be requested.' : `Completed trend state is ${input.latestDirection}; decision logged for learning only.`, + entryReferencePrice: finalDirection === 'buy' ? input.ask : finalDirection === 'sell' ? input.bid : null, + stopLoss: input.stopLoss, + takeProfit: null, + volume: input.volume, + spread: input.spread, + riskAmount: input.riskAmount, + maxAllowedRisk: input.maxAllowedRisk, + gateResults, + orderTicket: null, + positionId: null, + outcome: null, + } + return { ...record, decisionId: createJmbDecisionId(record) } +} +``` + +- [ ] **Step 4: Run focused tests** + +Run: + +```powershell +pnpm vitest run src/domain/mt5/shadow-decision-engine.spec.ts +``` + +Expected: PASS. + +- [ ] **Step 5: Commit Task 2** + +Run: + +```powershell +git add src/domain/mt5/shadow-decision-engine.ts src/domain/mt5/shadow-decision-engine.spec.ts +git commit -m "feat: add mt5 shadow decision engine" +``` + +Expected: one commit containing only Task 2 files. + +--- + +### Task 3: Add local shadow decision runner + +**Files:** +- Create: `C:\Users\mwbri\Documents\Open Alice Trading bot\OpenAlice\tools\mt5\run_shadow_decisions.ts` +- Test: `C:\Users\mwbri\Documents\Open Alice Trading bot\OpenAlice\src\domain\mt5\decision-record.spec.ts` +- Test: `C:\Users\mwbri\Documents\Open Alice Trading bot\OpenAlice\src\domain\mt5\shadow-decision-engine.spec.ts` + +**Interfaces:** +- Consumes: + - `buildShadowDecision(input)` + - `appendJmbDecisionRecord(root, record)` + - `writeLatestJmbDecision(root, record)` + - `readMt5ReadOnlyBridge(root, broker, symbol)` + - `summarizeMt5TradeLedger(root, broker, symbol)` +- Produces: `OpenAliceMt5DecisionLogV1///decisions.jsonl` and `latest_decision.csv`. + +- [ ] **Step 1: Create the runner** + +Create `tools/mt5/run_shadow_decisions.ts` with: + +```ts +import { homedir } from 'node:os' +import { join } from 'node:path' +import { readFile } from 'node:fs/promises' +import { buildShadowDecision } from '../../src/domain/mt5/shadow-decision-engine.js' +import { appendJmbDecisionRecord, writeLatestJmbDecision } from '../../src/domain/mt5/decision-record.js' +import { readMt5ReadOnlyBridge } from '../../src/domain/mt5/read-only-bridge.js' +import { summarizeMt5TradeLedger } from '../../src/domain/mt5/trade-ledger.js' + +type TrendReport = { + latest_observation?: { direction: 'uptrend' | 'downtrend' | 'flat' } +} + +const appData = process.env['APPDATA'] ?? join(homedir(), 'AppData', 'Roaming') +const commonFiles = join(appData, 'MetaQuotes', 'Terminal', 'Common', 'Files') +const bridgeRoot = process.env['OPENALICE_MT5_BRIDGE_ROOT'] ?? join(commonFiles, 'OpenAliceMt5BridgeV1') +const tradeLedgerRoot = process.env['OPENALICE_MT5_TRADE_LEDGER_ROOT'] ?? join(commonFiles, 'OpenAliceMt5TradeLedgerV1') +const decisionRoot = process.env['OPENALICE_MT5_DECISION_ROOT'] ?? join(commonFiles, 'OpenAliceMt5DecisionLogV1') +const researchRoot = process.env['OPENALICE_RESEARCH_ARTIFACTS_DIR'] ?? join(homedir(), '.openalice', 'data', 'research') + +const instruments = [ + { broker: 'hfmarkets', symbol: 'XAUUSD', report: 'xauusd-trend-baseline.json', canonical: 'Gold / USD', demoCandidateApproved: true, maxSpread: 0.75, stopDistance: 8, maxRisk: 1 }, + { broker: 'hfmarkets', symbol: 'EURUSD', report: 'eurusd-trend-baseline.json', canonical: 'Euro / USD', demoCandidateApproved: false, maxSpread: 0.00025, stopDistance: 0.0020, maxRisk: 1 }, + { broker: 'icmarkets', symbol: 'XAUUSD', report: 'icmarkets-xauusd-trend-baseline.json', canonical: 'Gold / USD', demoCandidateApproved: true, maxSpread: 0.30, stopDistance: 8, maxRisk: 1 }, + { broker: 'icmarkets', symbol: 'EURUSD', report: 'icmarkets-eurusd-trend-baseline.json', canonical: 'Euro / USD', demoCandidateApproved: false, maxSpread: 0.00015, stopDistance: 0.0020, maxRisk: 1 }, +] as const + +async function readTrendDirection(fileName: string): Promise<'uptrend' | 'downtrend' | 'flat'> { + try { + const report = JSON.parse(await readFile(join(researchRoot, fileName), 'utf8')) as TrendReport + return report.latest_observation?.direction ?? 'flat' + } catch { + return 'flat' + } +} + +for (const instrument of instruments) { + const [bridge, learning, latestDirection] = await Promise.all([ + readMt5ReadOnlyBridge(bridgeRoot, instrument.broker, instrument.symbol), + summarizeMt5TradeLedger(tradeLedgerRoot, instrument.broker, instrument.symbol), + readTrendDirection(instrument.report), + ]) + const referencePrice = latestDirection === 'downtrend' ? bridge.bid : bridge.ask + const stopLoss = referencePrice == null + ? null + : latestDirection === 'downtrend' + ? Number((referencePrice + instrument.stopDistance).toFixed(instrument.symbol === 'XAUUSD' ? 2 : 5)) + : latestDirection === 'uptrend' + ? Number((referencePrice - instrument.stopDistance).toFixed(instrument.symbol === 'XAUUSD' ? 2 : 5)) + : null + const decision = buildShadowDecision({ + createdAt: new Date().toISOString(), + broker: instrument.broker, + server: bridge.server, + accountMode: bridge.state === 'ready' ? 'demo' : null, + symbol: instrument.symbol, + canonicalInstrument: instrument.canonical, + strategyVersion: 'daily-trend-v1', + bridgeState: bridge.state, + learningState: learning.state, + latestDirection, + bid: bridge.bid, + ask: bridge.ask, + spread: bridge.spread, + maxSpread: instrument.maxSpread, + volume: 0.01, + maxVolume: 0.01, + stopLoss, + riskAmount: instrument.maxRisk, + maxAllowedRisk: instrument.maxRisk, + demoCandidateApproved: instrument.demoCandidateApproved, + }) + await appendJmbDecisionRecord(decisionRoot, decision) + await writeLatestJmbDecision(decisionRoot, decision) + console.log(`${decision.broker} ${decision.symbol}: ${decision.mode} ${decision.direction} ${decision.reasonCode}`) +} +``` + +- [ ] **Step 2: Run domain tests before runner smoke** + +Run: + +```powershell +pnpm vitest run src/domain/mt5/decision-record.spec.ts src/domain/mt5/shadow-decision-engine.spec.ts +``` + +Expected: PASS. + +- [ ] **Step 3: Run runner smoke** + +Run: + +```powershell +pnpm exec tsx tools/mt5/run_shadow_decisions.ts +``` + +Expected: four console lines, one for each broker/symbol. It may produce `skipped flat gate_blocked` if bridge/trend data is missing; that is acceptable because it is fail-closed. + +- [ ] **Step 4: Verify files were produced** + +Run: + +```powershell +$root = Join-Path $env:APPDATA 'MetaQuotes\Terminal\Common\Files\OpenAliceMt5DecisionLogV1' +Get-ChildItem -Path $root -Recurse -Filter latest_decision.csv | Select-Object FullName,Length,LastWriteTime +``` + +Expected: four `latest_decision.csv` files under HFM/IC Gold/EURUSD folders. + +- [ ] **Step 5: Commit Task 3** + +Run: + +```powershell +git add tools/mt5/run_shadow_decisions.ts +git commit -m "feat: add mt5 shadow decision runner" +``` + +Expected: one commit containing only Task 3 runner. + +--- + +### Task 4: Add Research API and UI decision status + +**Files:** +- Modify: `C:\Users\mwbri\Documents\Open Alice Trading bot\OpenAlice\src\webui\routes\research.ts` +- Modify: `C:\Users\mwbri\Documents\Open Alice Trading bot\OpenAlice\ui\src\api\research.ts` +- Modify: `C:\Users\mwbri\Documents\Open Alice Trading bot\OpenAlice\ui\src\pages\ResearchDashboardPage.tsx` + +**Interfaces:** +- Consumes: `summarizeLatestJmbDecision(root, broker, symbol)`. +- Produces: Research API includes `decision`; Research Dashboard displays latest JMB mode as evidence only. + +- [ ] **Step 1: Add Research API decision root and summaries** + +In `src/webui/routes/research.ts`, add: + +```ts +import { summarizeLatestJmbDecision } from '../../domain/mt5/decision-record.js' +``` + +Add this constant near the bridge/trade-ledger roots: + +```ts +const MT5_DECISION_ROOT = process.env['OPENALICE_MT5_DECISION_ROOT'] ?? join( + process.env['APPDATA'] ?? join(homedir(), 'AppData', 'Roaming'), + 'MetaQuotes', 'Terminal', 'Common', 'Files', 'OpenAliceMt5DecisionLogV1', +) +``` + +In the per-instrument `Promise.all`, add: + +```ts +summarizeLatestJmbDecision(MT5_DECISION_ROOT, instrument.broker, bridgeSymbol), +``` + +Return it as: + +```ts +decision, +``` + +Add summary count: + +```ts +const shadowDecisions = instruments.filter((instrument) => instrument.decision.state === 'shadow' || instrument.decision.state === 'demo_blocked').length +``` + +Add to summary: + +```ts +decisionRoot: MT5_DECISION_ROOT, +shadowDecisions, +``` + +Add a stage before demo: + +```ts +{ key: 'shadow', label: 'JMB shadow decisions', state: shadowDecisions === INSTRUMENTS.length ? 'complete' : shadowDecisions > 0 ? 'next' : 'waiting', detail: shadowDecisions > 0 ? `${shadowDecisions}/${INSTRUMENTS.length} broker-symbol pairs have logged JMB decisions.` : 'Run the shadow decision runner before enabling the demo risk shell.' }, +``` + +- [ ] **Step 2: Add UI API types** + +In `ui/src/api/research.ts`, add: + +```ts +export type JmbDecisionState = 'no_decision' | 'shadow' | 'demo_blocked' | 'error' + +export interface JmbDecisionSummary { + state: JmbDecisionState + label: string + detail: string + broker: string + symbol: string + lastUpdated: string | null + decision: null | { + decisionId: string + createdAt: string + strategyVersion: string + mode: string + direction: string + reasonCode: string + reasonDetail: string + spread: number | null + volume: number + stopLoss: number | null + gateResults: Array<{ gate: string; state: string; detail: string }> + } +} +``` + +Add `decision: JmbDecisionSummary` to the research instrument type and add `decisionRoot: string` plus `shadowDecisions: number` to the summary type. + +- [ ] **Step 3: Display decision status in Research Dashboard** + +In `ui/src/pages/ResearchDashboardPage.tsx`, add this helper: + +```ts +function decisionTone(state: string) { + if (state === 'shadow') return 'green' + if (state === 'demo_blocked') return 'amber' + if (state === 'error') return 'red' + return 'muted' +} +``` + +In each instrument card near the learning block, render: + +```tsx +
+ JMB decision learning + {instrument.decision.label} + {instrument.decision.detail} + {instrument.decision.decision ? ( + + {instrument.decision.decision.mode} • {instrument.decision.decision.direction} • {instrument.decision.decision.reasonCode} + + ) : null} +
+``` + +Add safety copy near the dashboard disclaimer: + +```tsx +

+ Shadow decisions are JMB learning records only. They are not live-trading approval and they do not submit orders. +

+``` + +- [ ] **Step 4: Run UI/domain verification** + +Run: + +```powershell +pnpm vitest run src/domain/mt5 ui/src +pnpm -F open-alice-ui build +``` + +Expected: tests and UI build pass. Existing Vite chunk-size warnings are acceptable. + +- [ ] **Step 5: Commit Task 4** + +Run: + +```powershell +git add src/webui/routes/research.ts ui/src/api/research.ts ui/src/pages/ResearchDashboardPage.tsx +git commit -m "feat: show jmb shadow decisions in research" +``` + +Expected: one commit containing only Task 4 files. + +--- + +### Task 5: Add MT5 demo risk shell EA with no order submission + +**Files:** +- Create: `C:\Users\mwbri\Documents\Open Alice Trading bot\OpenAlice\tools\mt5\JmbGoldmineDemoRiskShell.mq5` +- Modify: `C:\Users\mwbri\Documents\Open Alice Trading bot\OpenAlice\tools\mt5\README.md` + +**Interfaces:** +- Consumes: `Common Files/OpenAliceMt5DecisionLogV1///latest_decision.csv`. +- Produces: `Common Files/OpenAliceMt5RiskShellV1///gate_status.csv`. +- Must not contain order submission APIs. + +- [ ] **Step 1: Create the risk shell EA** + +Create `tools/mt5/JmbGoldmineDemoRiskShell.mq5` with: + +```mql5 +#property strict + +input string InpBrokerId = "hfmarkets"; +input string InpSymbol = "XAUUSD"; +input double InpMaxLot = 0.01; +input double InpMaxSpread = 0.75; +input int InpDecisionMaxAgeSeconds = 300; +input bool InpKillSwitch = true; + +string AccountModeLabel() +{ + long mode = AccountInfoInteger(ACCOUNT_TRADE_MODE); + if(mode == ACCOUNT_TRADE_MODE_DEMO) return "demo"; + if(mode == ACCOUNT_TRADE_MODE_REAL) return "real"; + return "contest"; +} + +string NowIso() +{ + string value = TimeToString(TimeCurrent(), TIME_DATE | TIME_SECONDS); + StringReplace(value, ".", "-"); + StringReplace(value, " ", "T"); + return value + ".000Z"; +} + +void EnsureFolders(string root, string broker, string symbol) +{ + FolderCreate(root, FILE_COMMON); + FolderCreate(root + "\\" + broker, FILE_COMMON); + FolderCreate(root + "\\" + broker + "\\" + symbol, FILE_COMMON); +} + +string ReadLatestDecision() +{ + string path = "OpenAliceMt5DecisionLogV1\\" + InpBrokerId + "\\" + InpSymbol + "\\latest_decision.csv"; + int handle = FileOpen(path, FILE_READ | FILE_TXT | FILE_COMMON | FILE_ANSI); + if(handle == INVALID_HANDLE) return ""; + string text = ""; + while(!FileIsEnding(handle)) + text += FileReadString(handle) + "\n"; + FileClose(handle); + return text; +} + +void WriteGateStatus(string state, string detail) +{ + string root = "OpenAliceMt5RiskShellV1"; + EnsureFolders(root, InpBrokerId, InpSymbol); + string path = root + "\\" + InpBrokerId + "\\" + InpSymbol + "\\gate_status.csv"; + int handle = FileOpen(path, FILE_WRITE | FILE_CSV | FILE_COMMON | FILE_ANSI, ','); + if(handle == INVALID_HANDLE) + { + Print("Unable to write risk shell status: ", GetLastError()); + return; + } + FileWrite(handle, "captured_at", "broker", "symbol", "account_mode", "state", "detail", "bid", "ask", "spread", "positions"); + double bid = SymbolInfoDouble(InpSymbol, SYMBOL_BID); + double ask = SymbolInfoDouble(InpSymbol, SYMBOL_ASK); + double spread = ask - bid; + FileWrite(handle, NowIso(), InpBrokerId, InpSymbol, AccountModeLabel(), state, detail, DoubleToString(bid, _Digits), DoubleToString(ask, _Digits), DoubleToString(spread, _Digits), IntegerToString(PositionsTotal())); + FileClose(handle); +} + +void Evaluate() +{ + if(InpKillSwitch) + { + WriteGateStatus("paused", "Kill switch is on; new entries blocked."); + return; + } + if(AccountModeLabel() != "demo") + { + WriteGateStatus("blocked", "Account is not demo."); + return; + } + if(_Symbol != InpSymbol) + { + WriteGateStatus("blocked", "EA chart symbol does not match configured symbol."); + return; + } + string decision = ReadLatestDecision(); + if(decision == "") + { + WriteGateStatus("blocked", "No latest JMB decision file found."); + return; + } + double bid = SymbolInfoDouble(InpSymbol, SYMBOL_BID); + double ask = SymbolInfoDouble(InpSymbol, SYMBOL_ASK); + double spread = ask - bid; + if(spread > InpMaxSpread) + { + WriteGateStatus("blocked", "Spread exceeds configured maximum."); + return; + } + WriteGateStatus("shadow_ready", "Decision file exists and local shell gates passed. No orders are submitted in this version."); +} + +int OnInit() +{ + EventSetTimer(10); + Evaluate(); + return INIT_SUCCEEDED; +} + +void OnDeinit(const int reason) +{ + EventKillTimer(); +} + +void OnTimer() +{ + Evaluate(); +} + +void OnTick() +{ + Evaluate(); +} +``` + +- [ ] **Step 2: Document shell setup** + +Append to `tools/mt5/README.md`: + +```md +## Demo risk shell, no order submission + +`JmbGoldmineDemoRiskShell.mq5` is an EA that reads the latest JMB shadow decision and writes gate status to: + +`OpenAliceMt5RiskShellV1///gate_status.csv` + +This shell does not submit, modify, or close orders. It is for validating demo account, symbol, spread, kill-switch, and decision-file readiness before any future demo execution plan. + +Recommended first run: + +- Keep `InpKillSwitch=true`. +- Attach to HFM demo `XAUUSD` and `EURUSD`. +- Attach to IC Markets demo `XAUUSD` and `EURUSD`. +- Use `InpBrokerId=hfmarkets` or `icmarkets`. +- Use the exact chart symbol in `InpSymbol`. + +Only after the gate status is stable should a separate plan add demo order submission. +``` + +- [ ] **Step 3: Verify no order APIs exist** + +Run: + +```powershell +Select-String -Path tools/mt5/JmbGoldmineDemoRiskShell.mq5 -Pattern 'OrderSend|PositionClose|CTrade|trade.Buy|trade.Sell|OrderModify' +``` + +Expected: no matches. + +- [ ] **Step 4: Commit Task 5** + +Run: + +```powershell +git add tools/mt5/JmbGoldmineDemoRiskShell.mq5 tools/mt5/README.md +git commit -m "feat: add mt5 demo risk shell" +``` + +Expected: one commit containing only Task 5 files. + +--- + +### Task 6: Final verification + +**Files:** +- Modify only files from Tasks 1-5 if verification reveals a direct defect. + +**Interfaces:** +- Consumes all previous task outputs. +- Produces a verified shadow-risk shell with no order execution. + +- [ ] **Step 1: Run MT5 domain tests** + +Run: + +```powershell +pnpm vitest run src/domain/mt5 +``` + +Expected: PASS. + +- [ ] **Step 2: Run UI tests and build** + +Run: + +```powershell +pnpm vitest run ui/src +pnpm -F open-alice-ui build +``` + +Expected: PASS. Existing Vite warnings about chunk size or missing local connector config are acceptable if exit code is `0`. + +- [ ] **Step 3: Run shadow runner smoke** + +Run: + +```powershell +pnpm exec tsx tools/mt5/run_shadow_decisions.ts +``` + +Expected: four lines for HFM/IC Gold/EURUSD. `skipped` or `demo_blocked` is acceptable; thrown exceptions are not. + +- [ ] **Step 4: Scan for order execution APIs in this plan's files** + +Run: + +```powershell +rg "OrderSend|PositionClose|CTrade|trade\\.Buy|trade\\.Sell|OrderModify" src/domain/mt5 tools/mt5/JmbGoldmineDemoRiskShell.mq5 tools/mt5/run_shadow_decisions.ts src/webui/routes/research.ts ui/src/api/research.ts ui/src/pages/ResearchDashboardPage.tsx +``` + +Expected: no matches. + +- [ ] **Step 5: Record handoff** + +Final handoff must say: + +```text +Shadow risk shell is complete: JMB can log its own Gold/EURUSD shadow decisions for HFM and IC Markets, show them in Research, and let MT5 validate gates without submitting orders. Demo order execution remains locked for Plan 3. +``` + diff --git a/docs/superpowers/specs/2026-07-13-jmb-goldmine-demo-canary-execution-design.md b/docs/superpowers/specs/2026-07-13-jmb-goldmine-demo-canary-execution-design.md new file mode 100644 index 000000000..7546cbbbf --- /dev/null +++ b/docs/superpowers/specs/2026-07-13-jmb-goldmine-demo-canary-execution-design.md @@ -0,0 +1,348 @@ +# JMB Goldmine Demo Canary Execution Design + +**Date:** 2026-07-13 + +**Status:** Approved design, pending implementation plan + +**Scope:** HFM and IC Markets demo Gold execution; EURUSD remains shadow-only + +**Safety boundary:** This design contains no live-account eligibility or live-mode switch + +## 1. Purpose + +Plan 3 adds the smallest execution-capable slice to JMB Goldmine: deterministic, broker-local demo orders for Gold after the existing shadow-decision and MT5 risk-shell gates pass. + +The purpose is to validate execution plumbing, broker costs, protection, reconciliation, and learning records. It is not a profitability claim, live-trading approval, or permission for an AI agent to submit orders. + +The repository does not contain the `PRD.txt` named by the workspace instructions. This design therefore uses `docs/PRD.md`, the existing JMB Goldmine PRD, as the product source of truth. + +## 2. Approved product decisions + +- Gold/XAUUSD is the only execution-eligible instrument in Plan 3. +- HFM demo is enabled first as the initial canary. +- IC Markets demo is enabled only after the HFM canary proves order submission, broker-side stop protection, durable logging, and restart reconciliation. +- After the IC canary passes, both Gold demo executors may remain enabled. +- EURUSD continues to produce shadow decisions but cannot submit orders. +- Maximum order volume is `0.01` lot. +- Each broker pauses new entries after four losing trades in the broker day. +- The daily loss-count pause resets automatically at the next broker day. +- A separate account-currency daily-loss gate also applies and resets at the next broker day. +- The current daily strategy may create at most one new entry per broker per completed daily observation. The four-loss rule is an upper safety boundary, not a target trade frequency. +- No martingale, grid, recovery sizing, pyramiding, or automatic lot growth is permitted. +- AI agents may review and journal outcomes but may not submit orders, change risk limits, override pauses, or change the strategy automatically. + +## 3. Architecture choice + +Plan 3 evolves the verified status-only risk shell into a new demo-canary EA while retaining the status-only EA as a rollback and diagnostic tool. + +The new EA replaces `JmbGoldmineDemoRiskShell` on the existing Gold risk-shell chart after dry-run acceptance. The separate read-only bridge remains attached to its duplicate Gold chart. No third Gold chart is required. + +Direct order submission from TypeScript, the Research Desk, Codex, an LLM, or a workspace agent is prohibited. The MT5 EA is the only execution authority and independently re-evaluates all gates using current broker state. + +### Components + +1. **Completed-observation decision cycle** + - Reads broker-specific research, bridge, cost, and learning artifacts. + - Writes an append-only decision record and atomic latest-decision CSV. + - Uses a stable observation ID based on broker, symbol, strategy version, and completed-bar `as_of` date. + - Repeated scheduler runs for the same completed observation do not create a new execution identity. + +2. **Demo execution policy** + - Stores broker-specific candidate approval, exact server allowlist, strategy allowlist, maximum spread, maximum slippage, risk limits, session rules, news blackout, and rollout stage. + - Lives in persistent local data, not deployable source files. + - Cannot be changed by the AI review loop. + +3. **JMB Goldmine demo-canary EA** + - Reads the latest decision and local policy. + - Reconciles MT5 orders, positions, deals, and processed decision IDs. + - Evaluates gates and either blocks, closes/reverses, or submits one protected demo order. + - Writes append-only execution events plus an atomic latest-status CSV. + +4. **Outcome reconciler** + - Imports EA-owned order/deal outcomes using broker, account, symbol, magic number, decision ID, order ticket, deal ticket, and position ID. + - Updates learning summaries without rewriting the original decision or execution event. + +5. **Research Desk status** + - Displays rollout stage, execution switch, gate state, latest decision, latest execution event, open EA-owned exposure, daily loss count, daily realized loss, and reconciliation errors. + - Clearly labels all Plan 3 activity as demo-only. + +## 4. Deployment and identity binding + +The EA has no live-mode input. It refuses to initialize as execution-capable unless all of these identities match: + +- `ACCOUNT_TRADE_MODE_DEMO`. +- Exact configured account login entered locally by the operator. +- Exact server allowlist: + - HFM: `HFMarketsGlobal-Demo4`. + - IC Markets: `ICMarketsSC-Demo`. +- Exact chart and configured symbol: `XAUUSD`. +- Exact broker ID: `hfmarkets` or `icmarkets`. +- Strategy allowlist: `daily-trend-v1`. +- Broker-specific magic number: + - HFM Gold: `880101`. + - IC Markets Gold: `880201`. + +Account logins are local MT5 inputs and must never be committed to the repository or written to the Research Desk API. + +The EA defaults remain: + +- `InpDemoExecutionEnabled=false`. +- `InpKillSwitch=true`. + +Both must be changed deliberately before the initial HFM canary. Enabling IC Markets is a separate later operator action. + +## 5. Decision idempotency + +Every execution-eligible record includes: + +- `decision_id`. +- `observation_id`. +- `observation_as_of`. +- Broker, server, account mode, symbol, and strategy version. +- Direction, reference price, volume, stop loss, and calculated risk ceiling. +- Cost-model version and candidate-policy version. +- All pre-decision gate results. + +The EA persists processed decision and observation IDs. It must not send another entry when any of these are true: + +- The decision ID was already accepted, rejected, filled, closed, stopped, or reconciled. +- The observation ID already produced an entry attempt. +- An EA-owned position already matches the decision direction. +- The current observation is not newer than the position's opening observation. +- A position stopped out and the decision still refers to the same completed observation. + +Atomic latest-file replacement and append-only journals prevent a partially written decision from becoming executable. + +## 6. Entry gates + +All gates fail closed when their inputs are missing, malformed, stale, or unverifiable. + +Before every entry the EA enforces: + +1. Demo account, account login, server, broker, chart symbol, and magic-number binding. +2. `InpDemoExecutionEnabled=true` and persistent kill switch off. +3. HFM-first or IC-enabled rollout stage from the local policy. +4. Gold and `daily-trend-v1` allowlists. +5. Fresh decision, bridge heartbeat, candidate policy, broker cost model, and completed observation. +6. `0.01` requested volume and broker volume-minimum/step compatibility. +7. Required stop loss accepted by broker constraints before submission. +8. Real account-currency loss to the stop calculated with MT5 `OrderCalcProfit` and not greater than `10.00` demo account-currency units by default. +9. Daily realized loss not greater than `40.00` demo account-currency units by default. +10. Fewer than four losing EA-owned Gold trades in the broker day. +11. No EA-owned position, no pending EA-owned order, and no manual or foreign Gold exposure. +12. Sufficient margin: estimated order margin is available and post-order free margin retains at least a ten-times margin buffer. +13. Current spread at or below: + - HFM Gold: `0.75` price units. + - IC Markets Gold: `0.30` price units. +14. Maximum requested price deviation at or below: + - HFM Gold: `0.50` price units. + - IC Markets Gold: `0.30` price units. +15. Entry session: + - Monday through Thursday, `06:00` to `20:00` UTC. + - Friday, `06:00` to `16:00` UTC. + - No weekend or rollover-window entries. +16. No high-impact USD economic event from 30 minutes before through 30 minutes after the event, using the MT5 economic calendar. Unavailable calendar data blocks new entries. +17. Durable event-log path is writable before order submission. +18. Restart reconciliation and unknown-order recovery are complete. + +The `10.00` per-trade and `40.00` daily currency defaults are demo-canary plumbing limits. They are not approved settings for a future `100 USD` live account. Live-capital eligibility requires a separate design and a minimum-lot risk feasibility check. + +## 7. Order submission and protection + +- The EA sends one market request with the decision ID in the comment and the required broker-side stop loss in the original request. +- It records an `order_requesting` event before submission and flushes the journal. +- It records the full broker result code, accepted volume, accepted price, stop, order ticket, deal ticket, and position ID after the call. +- A broker rejection is terminal for that decision. The EA does not retry blindly. +- A timeout or unknown result enters `reconciliation_required`; the EA searches broker orders, deals, and positions by magic number, symbol, and decision comment before deciding whether an order exists. +- Partial or unexpected volume enters `reconciliation_required` and blocks further entries. +- If MT5 reports a fill without confirmed stop protection, the EA performs the explicitly designed emergency protective close, logs every attempt, and pauses the broker. It does not open replacement exposure. + +## 8. Position lifecycle + +- A same-direction completed observation while a JMB position is open is a no-op and is logged. +- An opposite completed observation requests closure of the existing JMB position. +- A reversal entry is considered only after the close is confirmed, the close event is durable, and all entry gates are re-evaluated. +- A stop-loss close marks the observation as consumed. The EA cannot re-enter from the same observation. +- No fixed take-profit is added in Plan 3 because the research baseline is daily time-series momentum. Adding a take-profit would create an untested strategy variant. +- The kill switch blocks new entries. It does not close a correctly protected position. +- Emergency closing is limited to an unprotected EA-owned position or a separately confirmed opposite-signal close. No hidden AI or remote close command exists. + +## 9. Daily loss policy + +The loss day is the broker server's calendar day and is recorded with the server identifier and UTC conversion. + +An EA-owned Gold trade counts as a losing trade when its fully reconciled net result is below zero after commission, swap, and fees. + +New entries pause for the remainder of the broker day when either condition is true: + +- Four losing EA-owned Gold trades have closed during that broker day. +- Reconciled realized EA-owned Gold loss reaches `40.00` demo account-currency units. + +The count and realized-loss gate reset automatically when the broker day changes. The reset does not clear unresolved orders, unprotected exposure, reconciliation errors, kill switches, or identity mismatches. + +The current daily strategy is limited to one new entry per completed daily observation per broker, so it normally cannot accumulate four new stopped trades in one day. The four-loss rule remains defense in depth for later separately approved strategies and unexpected lifecycle conditions. + +## 10. Scheduler and learning loop + +- A local JMB scheduler evaluates artifacts every five minutes while the desktop service is running. +- It only emits a new execution identity when a newer completed daily observation exists. +- Scheduler downtime cannot remove broker-side stop protection. On restart it catches up once, then resumes the five-minute cadence. +- It never shells through PowerShell or launches MetaEditor. +- The execution cycle is deterministic and does not call an LLM. + +After reconciliation, AI workspaces may create daily and weekly journals that: + +- Compare decision context with the eventual net result. +- Summarize spread, slippage, commission, swap, adverse excursion, favorable excursion, blocks, and operational errors. +- Recommend `NO ACTION`, `MONITOR`, `SHADOW`, `DEMO CANDIDATE`, or `PAUSE` for human review. + +AI output cannot alter policy files, strategy parameters, EA inputs, processed IDs, or broker state. + +## 11. Execution and outcome records + +Execution events are append-only JSONL records with an atomic latest-status CSV for MT5 and UI interoperability. Each record includes: + +- Schema version and event ID. +- Event type and UTC timestamp. +- Broker, server, account mode, symbol, strategy, magic number, and masked account identity. +- Decision and observation IDs. +- Gate results and risk calculations. +- Requested and accepted order fields. +- MT5 return code and human-readable result. +- Order, deal, and position identifiers stored as strings. +- Reconciliation state. +- Daily loss count and realized loss. +- Outcome fields when closed. + +Permitted lifecycle states are: + +- `disabled`. +- `paused`. +- `blocked`. +- `ready`. +- `order_requesting`. +- `order_rejected`. +- `reconciliation_required`. +- `filled_protected`. +- `close_requesting`. +- `closed`. +- `stopped`. +- `emergency_close`. +- `error`. + +No local state may claim `filled_protected` unless broker state confirms both exposure and stop protection. + +## 12. Error handling + +- Missing, stale, or malformed inputs produce a durable `blocked` event with a safe next action. +- Unwritable logs block entry before the broker call. +- Unknown broker results never trigger an automatic resend. +- Restart with unknown exposure enters reconciliation before processing decisions. +- Foreign or manual Gold exposure blocks entry in both netting and hedging account modes. +- Broker/account mismatch remains blocked even if the decision file otherwise passes. +- Economic-calendar failure blocks entry but does not interfere with management of an existing protected position. +- An exception in the app scheduler cannot grant MT5 authority or bypass EA gates. + +## 13. Research Desk behavior + +For HFM and IC Markets Gold, the Research Desk displays: + +- `EXECUTION DISABLED`, `CANARY READY`, `DEMO ENABLED`, `PAUSED`, `BLOCKED`, or `RECONCILIATION REQUIRED`. +- Latest completed observation and decision age. +- Latest broker execution event. +- Whether broker-side stop protection is confirmed. +- Current EA-owned position summary without exposing account login. +- Daily losing-trade count and realized loss. +- Current blocking gate and next safe action. +- HFM-first / IC-pending / both-enabled rollout stage. + +EURUSD continues to display `DEMO BLOCKED` and shadow decisions. The UI must state that demo performance is not live approval or evidence of future profit. + +## 14. Rollout + +### Stage 0: status-only dry run + +- Compile with zero errors and zero warnings. +- Attach the new EA with execution disabled and kill switch on. +- Compare every new EA gate with the existing risk-shell output. +- Run restart, stale-file, foreign-position, spread, session, news, and disk-failure checks without submitting an order. + +### Stage 1: HFM canary + +- Explicitly set the expected HFM demo account login locally. +- Enable demo execution and turn off the kill switch on HFM Gold only. +- Observe one eligible decision through request, fill/rejection, stop confirmation, event persistence, and restart reconciliation. +- Any unprotected fill, unknown result, or reconciliation mismatch returns the rollout to Stage 0. + +### Stage 2: IC Markets canary + +- Keep HFM protected and reconciled. +- Repeat the same ceremony for IC Markets Gold. +- Verify IC-specific spread, slippage, volume, server, cost, and identifier behavior. + +### Stage 3: both demo brokers + +- Both Gold demo EAs may remain enabled with independent gates, magic numbers, loss days, and pauses. +- EURUSD remains shadow-only. +- Live execution remains absent. + +## 15. Testing strategy + +### TypeScript tests + +- Stable observation and decision IDs. +- No duplicate execution identity for repeated scheduler runs. +- New identity only for a newer completed daily observation. +- Demo-candidate and rollout-policy validation. +- Broker-specific cost-model validation. +- Execution-event parsing, append-only persistence, and fail-closed malformed-state handling. +- Outcome reconciliation and daily net-loss summaries. +- Research API/UI states without account-login exposure. + +### MQL5 tests and harness scenarios + +- Demo-only, account, server, symbol, strategy, and magic binding. +- Execution-disabled and kill-switch defaults. +- `0.01` lot enforcement and broker step compatibility. +- `OrderCalcProfit` stop-risk and `OrderCalcMargin` buffer gates. +- Spread, deviation, session, Friday, rollover, and economic-calendar gates. +- One-position, foreign-exposure, and pending-order conflicts. +- Decision/observation idempotency. +- Four-loss and account-currency daily pause with next-day reset. +- Same-direction no-op, opposite-direction close/re-evaluate, and stopped-observation consumption. +- Broker rejection, timeout, unknown result, partial fill, unprotected fill, emergency close, and restart reconciliation. +- Log-write failure before submission. + +### Acceptance checks + +- Source scan confirms no live-mode input or live-account bypass. +- Source scan confirms no martingale, grid, recovery, or automatic lot-growth path. +- Dry-run produces no broker orders. +- HFM canary satisfies the rollout evidence before IC can be enabled. +- IC canary satisfies the same evidence before both-enabled status. +- Every broker submission has a durable pre-request event and reconciled broker result. +- No repeated scheduler run can duplicate a broker entry. + +## 16. Explicitly out of scope + +- Any live-account order. +- EURUSD demo execution. +- Intraday or high-frequency strategies. +- Self-modifying strategy logic or autonomous risk-limit changes. +- Profit, win-rate, or drawdown guarantees. +- Automatic promotion based only on demo profit. +- Remote order commands from AI, Codex, Research Desk, chat, or workspaces. +- Take-profit, trailing-stop, martingale, grid, recovery, or portfolio allocation features. + +## 17. Completion criteria + +Plan 3 is complete only when: + +- The approved automated tests pass. +- The new EA compiles with zero errors and zero warnings. +- Status-only dry-run evidence is captured for both brokers. +- HFM canary evidence is reviewed before IC enablement. +- IC canary evidence is reviewed before both-enabled status. +- Research Desk and journals truthfully display broker-confirmed lifecycle states. +- EURUSD and all non-demo accounts remain unable to execute. +- Documentation explains installation, inputs, enablement, rollback, pause, recovery, and evidence review. diff --git a/docs/superpowers/specs/2026-07-13-jmb-goldmine-demo-paper-autopilot-design.md b/docs/superpowers/specs/2026-07-13-jmb-goldmine-demo-paper-autopilot-design.md new file mode 100644 index 000000000..66ade03bb --- /dev/null +++ b/docs/superpowers/specs/2026-07-13-jmb-goldmine-demo-paper-autopilot-design.md @@ -0,0 +1,294 @@ +# JMB Goldmine Demo/Paper Autopilot V1 Design + +**Date:** 2026-07-13 +**Status:** Written for user review +**Product:** JMB Goldmine +**Related docs:** `docs/PRD.md`, `docs/mt5-data-and-training-protocol.md` +**Approved high-level scope:** Learn from and demo-test Gold/XAUUSD and EURUSD on both HFM demo and IC Markets demo accounts. + +## 1. Purpose + +JMB Goldmine Demo/Paper Autopilot V1 turns the current research-only MT5 bridge into a controlled demo/paper learning and execution pipeline. It is designed to learn faster from broker-specific data, manual/demo trade history, repeated walk-forward tests, and daily AI review while keeping the live-money boundary locked. + +This design does not promise profit, win rate, or live-trading readiness. Its job is to create clean evidence and safe demo execution plumbing so future promotion decisions are based on auditable facts instead of excitement after a few winning trades. + +## 2. Safety boundary + +V1 is demo/paper only. + +- No live account trading. +- No LLM or workspace agent may submit an order, click buy/sell, change live settings, or sit in the tick-by-tick execution path. +- The MT5 EA is the only component allowed to place demo orders, and only when deterministic local gates pass. +- AI agents may research, score, review, journal, recommend, or veto. +- Risk rules are fixed configuration, not self-modified by AI. +- A persistent kill switch must block new entries. +- Closing positions requires explicit deterministic EA logic; agents cannot manually flatten through hidden tools. + +If broker/account mode cannot be proven to be demo, the EA must refuse to open new trades. + +## 3. Instruments and broker mapping + +V1 covers these exact demo instruments: + +| Broker | Canonical instrument | Demo symbol | Initial mode | +| --- | --- | --- | --- | +| HFM demo | Gold / USD | `XAUUSD` | Demo automation after gates pass | +| HFM demo | Euro / USD | `EURUSD` | Learn immediately; demo entries only after EUR candidate gate passes | +| IC Markets demo | Gold / USD | `XAUUSD` | Demo automation after gates pass | +| IC Markets demo | Euro / USD | `EURUSD` | Learn immediately; demo entries only after EUR candidate gate passes | + +Gold starts closer to demo automation because existing research showed early historical candidate behavior. EURUSD starts stricter because current trend evidence was rejected. EURUSD is still imported, backtested, reviewed, and shadow-scored daily from day one; it simply cannot open demo trades until its own broker-specific validation gate passes. + +## 4. Roles and responsibilities + +### 4.1 MT5 EA Executor + +The MT5 EA runs inside each MetaTrader 5 terminal. It owns: + +- Tick and completed-bar observation. +- Broker/account/symbol validation. +- Position sizing. +- Stop-loss and take-profit placement. +- Order submission for demo accounts only. +- Deterministic risk gates. +- Local decision and execution logs. +- Kill-switch enforcement. + +### 4.2 Learning Agent + +The Learning Agent imports evidence and updates the learning ledger. It owns: + +- MT5 orders/deals import. +- Manual, EA, other, and unknown trade-origin labels. +- Broker-specific cost observations. +- Daily learning summaries. +- Detection of stale or missing bridge/history data. + +### 4.3 Strategy Steward + +The Strategy Steward reviews backtests, forward demo evidence, manual trade outcomes, and market context. It owns: + +- Daily strategy journal. +- Candidate status recommendation: `NO ACTION`, `MONITOR`, or `DEMO CANDIDATE`. +- Lessons learned from losing and winning trades. +- Overfitting warnings. + +### 4.4 Risk Governor + +The Risk Governor has veto power but no execution authority. It owns: + +- Risk-policy review. +- Gate failure summaries. +- Drawdown and consecutive-loss monitoring. +- Spread, slippage, cost, stale-data, and broker-mismatch warnings. +- Promotion/demotion recommendation. + +### 4.5 Execution Auditor + +The Execution Auditor reconciles what the EA intended with what MT5 reports. It owns: + +- Deal/order/fill reconciliation. +- Commission, swap, slippage, and spread checks. +- Detection of foreign/manual positions that conflict with EA rules. +- Ledger freshness checks. + +## 5. Data flow + +1. Each MT5 terminal exports account-scoped bridge and trade-history data. +2. The importer normalizes the data by broker, account, symbol, order, deal, position, magic number, and trade origin. +3. The research runner uses broker-specific market data and cost assumptions to run fixed backtests and walk-forward studies. +4. The learning ledger records each run, result, gate decision, and daily journal. +5. AI agents review the ledger and produce recommendations or vetoes. +6. The EA reads only approved deterministic configuration and local gate state. +7. The EA places demo orders only if every local gate passes. +8. The auditor reconciles the resulting broker history back into the ledger. + +The Research Desk may display this evidence, but it must not submit orders. + +## 6. Trade-history import requirements + +The importer must preserve: + +- Broker company, server, account login or local account key, and account trade mode. +- Exact broker symbol and canonical instrument. +- Order tickets, deal tickets, and position identifiers as strings. +- Open/close times with millisecond precision when available. +- Order type, deal type, entry/exit side, reason, state, filling mode, and expiration. +- Volume, requested price, fill price, stop loss, take profit, commission, fee, swap, profit, and balance operations. +- Magic number, comment, and external ID when present. +- Derived origin label: `manual`, `ea`, `other`, or `unknown`. + +Manual winning trades are useful evidence, but they are not automatic training labels. The system may study them for context, timing, cost, and broker behavior; it must not blindly clone them as a strategy. + +## 7. Broker cost model + +V1 must keep separate cost models for HFM demo and IC Markets demo. It must not treat demo costs as live costs without a future explicit live-cost validation step. + +Each cost model records: + +- Spread observations by symbol and session. +- Commission and fee behavior. +- Swap/financing behavior. +- Slippage between requested and fill prices when available. +- Contract size, digits, point, tick value, minimum volume, volume step, stop distance, freeze level, filling mode, and trading mode. +- Rollover and high-cost time windows when observed. + +Backtests that still use placeholder costs must be labelled as research-only and cannot unlock demo automation by themselves. + +## 8. Strategy and learning loop + +The daily learning loop runs in this order: + +1. Import new MT5 history from both demo terminals. +2. Import or refresh bridge/spread observations. +3. Validate account, broker, symbol, and data freshness. +4. Run fixed backtests and walk-forward tests using eligible broker-specific data. +5. Compare simulated outcomes with manual and EA demo outcomes. +6. Update the learning ledger and daily journal. +7. Produce AI recommendations and vetoes. +8. Refresh Research Desk status. + +The learning loop may improve hypotheses, review discipline, and strategy selection. It may not mine history until it finds a desired 90% win rate. Acceptance is based on robustness, realistic costs, drawdown, repeatability, and forward-demo behavior. + +## 9. Candidate gates + +### 9.1 Gold/XAUUSD gate + +Gold may become demo-trade eligible when all of these are true: + +- Broker account is confirmed demo. +- Exact demo symbol is allowlisted. +- Trade-history import is fresh. +- Bridge/spread telemetry is fresh. +- Broker cost model is not placeholder-only. +- Strategy version is fixed for the test window. +- No open manual or foreign Gold position conflicts with EA rules. +- Risk gates pass locally in the EA. + +### 9.2 EURUSD gate + +EURUSD starts as learn-and-shadow mode. It may become demo-trade eligible only when all Gold-style gates pass and EURUSD has its own broker-specific candidate decision recorded as `DEMO CANDIDATE`. + +The EURUSD gate must fail closed if: + +- The latest broker-specific walk-forward result remains rejected. +- The result depends on excluded fallback data. +- Cost model evidence is missing. +- The strategy only works on one broker while failing materially on the other. +- The AI recommendation is based mainly on a small set of recent manual wins. + +## 10. Demo EA risk policy + +Initial V1 EA settings: + +- Account mode: demo only. +- Symbols: HFM `XAUUSD`, HFM `EURUSD`, IC Markets `XAUUSD`, IC Markets `EURUSD`. +- Lot size: `0.01` maximum for all symbols. +- Position limit: one EA-owned open position per broker/symbol. +- Pyramiding: disabled. +- Grid: disabled. +- Martingale/recovery sizing: disabled. +- Stop loss: required before entry. +- Take profit: optional only if strategy spec defines it; otherwise exit must be deterministic. +- Spread gate: required per broker/symbol. +- Slippage gate: required per broker/symbol. +- Daily loss gate: required. +- Consecutive-loss gate: required. +- Stale-data gate: required. +- Foreign/manual exposure gate: required. +- Persistent kill switch: required. +- Restart behavior: default to block new entries until state is reconciled. + +If computed risk cannot be represented safely at the broker minimum lot, the EA must skip the trade instead of rounding risk upward. + +## 11. User-visible states + +Each broker/symbol should show one of these states: + +- `NO DATA`: required data is missing. +- `LEARNING`: data exists and is being imported/reviewed. +- `SHADOW`: strategy produces would-trade decisions but EA does not submit orders. +- `DEMO ENABLED`: EA may place demo orders when every local gate passes. +- `BLOCKED`: a safety, data, cost, or validation gate failed. +- `PAUSED`: user or kill switch disabled new entries. + +The UI must distinguish learning progress from permission to trade. A good journal or recent win must not appear as approval. + +## 12. Error handling and fail-closed rules + +The system blocks new entries when any of these occur: + +- Account is not demo. +- Broker, server, account key, or symbol does not match the approved config. +- Trading is disabled in MT5 or Algo Trading is off. +- Bridge or trade-history data is stale. +- Spread or slippage exceeds limits. +- Stop loss cannot be placed. +- Minimum lot would exceed allowed risk. +- Manual or foreign exposure conflicts with one-position rules. +- Commission, swap, or fill history cannot be reconciled. +- Daily loss or consecutive-loss limit is hit. +- Disk logging fails. +- EA restarts and cannot reconcile open positions. + +All skips and blocks must be logged with a human-readable reason. + +## 13. Testing and verification + +The implementation plan must include tests for: + +- Trade-history parsing and origin labelling. +- Account/symbol allowlist enforcement. +- Demo-only refusal on non-demo account mode. +- Risk gate pass/fail behavior. +- Stop-loss-required behavior. +- One-position-per-symbol behavior. +- Kill-switch persistence. +- Ledger updates for taken and skipped trades. +- EURUSD remaining blocked until its candidate gate passes. +- Research Desk displaying evidence states without order-submission authority. + +Broker-connected acceptance checks must be run on demo accounts only. + +## 14. Out of scope for V1 + +- Live trading. +- Autonomous risk-rule changes by AI. +- LLM-driven order submission. +- Broker credential access inside AI agents. +- Martingale, grid, recovery, or averaging down. +- New brokers or new instruments beyond HFM/IC demo Gold and EURUSD. +- Profit guarantees or fixed win-rate promises. +- Copying manual trades without strategy validation. + +## 15. Implementation sequencing + +V1 should be implemented in small, reviewable slices: + +1. Trade-history exporter/importer and account-scoped ledger. +2. Broker cost model ingestion. +3. Daily learning-loop runner and journal writer. +4. Research Desk status for learning, shadow, blocked, and demo-enabled states. +5. Paper/shadow signal generation. +6. Demo-only EA risk gate shell. +7. Demo order execution for Gold after gates pass. +8. EURUSD promotion from shadow to demo-enabled only after its candidate gate passes. +9. Execution reconciliation and auditor reports. + +Each slice must preserve the live-trading lock. + +## 16. Open user decision before any future live pilot + +This spec does not approve live trading. Before any $100 live-account pilot is designed, the user must explicitly approve a live risk budget and pilot-loss cap. The recommended first cap is a hard total pilot-loss cap near `$2`, not 8-10%, because minimum-lot Gold risk can move quickly. + +## 17. Acceptance criteria for this design + +The design is accepted when: + +- Gold and EURUSD are both included for HFM demo and IC Markets demo. +- EURUSD learns immediately but cannot demo-trade until its own candidate gate passes. +- AI agents have review/veto/journal roles only. +- MT5 EA is the only demo execution component. +- Live trading remains locked. +- Risk gates are deterministic and fail closed. +- The implementation can be broken into small, testable slices. diff --git a/docs/superpowers/specs/2026-07-13-jmb-goldmine-shadow-to-demo-self-trading-design.md b/docs/superpowers/specs/2026-07-13-jmb-goldmine-shadow-to-demo-self-trading-design.md new file mode 100644 index 000000000..9a162cda3 --- /dev/null +++ b/docs/superpowers/specs/2026-07-13-jmb-goldmine-shadow-to-demo-self-trading-design.md @@ -0,0 +1,255 @@ +# JMB Goldmine Shadow-to-Demo Self-Trading Risk Shell Design + +**Date:** 2026-07-13 +**Status:** Written for user review +**Product:** JMB Goldmine +**Builds on:** `2026-07-13-jmb-goldmine-demo-paper-autopilot-design.md`, `2026-07-13-jmb-goldmine-learning-foundation.md` +**Approved direction:** Manual Gold trades are supporting broker/cost evidence. JMB Goldmine should learn primarily from its own logged shadow decisions and demo EA trades. + +## 1. Purpose + +This design defines the next safe layer after the learning foundation: a shadow-to-demo risk shell that lets JMB Goldmine generate its own trade decisions, record the reason for every decision, and later allow a deterministic MT5 EA to place tiny demo trades only when all local risk gates pass. + +The main learning dataset becomes JMB Goldmine's own decisions, not manual trades. Manual trades remain useful for broker cost, spread, commission, swap, fill behavior, and comparison, but they are not treated as strategy labels. + +## 2. Core principle + +The system must know why it would enter a trade before it is allowed to learn from the trade result. + +Every JMB-generated decision must have: + +- Strategy version. +- Broker and exact symbol. +- Decision mode: `shadow`, `demo_blocked`, `demo_order_requested`, `demo_filled`, `demo_closed`, or `skipped`. +- Direction: `buy`, `sell`, or `flat`. +- Entry reason code. +- Invalidation reason or stop-loss basis. +- Spread at decision time. +- Expected risk and maximum allowed risk. +- Gate results. +- Outcome after close, when available. + +If a trade has no JMB decision record, it may be audited but not used as a primary self-learning example. + +## 3. Scope + +### In scope + +- HFM demo `XAUUSD`. +- HFM demo `EURUSD`. +- IC Markets demo `XAUUSD`. +- IC Markets demo `EURUSD`. +- Shadow decisions for all four broker/symbol pairs. +- Demo EA risk shell for all four broker/symbol pairs. +- Demo order requests only after deterministic gates pass. +- Manual trades used only as supporting broker/cost/context evidence. +- AI daily review of JMB's own shadow/demo decisions. + +### Out of scope + +- Live trading. +- AI clicking buy/sell. +- LLM-generated orders without EA risk gates. +- Martingale, grid, averaging down, recovery sizing, pyramiding. +- Copying manual trades as a strategy. +- Letting recent wins automatically loosen risk limits. +- EURUSD demo entries before its candidate gate passes. + +## 4. Operating modes + +### 4.1 Learning-only + +The system imports broker data, bridge status, trade history, and manual/demo outcomes. It does not generate trade decisions. + +### 4.2 Shadow mode + +The strategy engine generates would-trade decisions and logs them, but no order request is sent to MT5. Shadow mode is the default first step for both Gold and EURUSD. + +Shadow logs answer: "If JMB had been allowed to trade, what would it have done, why, and what would the result have been?" + +### 4.3 Demo-blocked mode + +The strategy engine produces a candidate decision, but one or more gates block it. The system logs the block reason. Blocked decisions are valuable learning data because they show discipline. + +### 4.4 Demo-enabled mode + +The MT5 EA may place a demo order only when: + +- The account is confirmed demo. +- The symbol is allowlisted. +- The strategy version is approved for demo mode. +- The broker/symbol candidate gate is satisfied. +- Every local EA risk gate passes. +- No manual/foreign position conflict exists. +- The persistent kill switch is off. + +Demo-enabled does not mean every signal trades. It means the EA may trade only when all gates pass. + +## 5. Manual trades policy + +Manual trades are not strategy labels unless the setup was recorded at decision time with a structured reason. The existing Gold manual wins are therefore treated as: + +- Broker cost evidence. +- Spread and fill evidence. +- Commission/swap evidence. +- Risk and exposure context. +- Human benchmark for comparison. + +They must not be used to claim that the system learned a profitable entry strategy. + +## 6. Strategy candidate gates + +### 6.1 Gold/XAUUSD + +Gold may begin in shadow mode immediately and may progress to demo-enabled mode after the risk shell is implemented and these are true: + +- Fresh bridge telemetry exists for the broker/symbol. +- Fresh trade ledger import exists. +- Account mode is demo. +- Cost model is not placeholder-only. +- Strategy version is fixed. +- Spread, stop, lot, daily-loss, and position gates are configured. +- The EA can log decisions and blocked decisions before order execution. + +### 6.2 EURUSD + +EURUSD may begin in shadow mode immediately. It must remain demo-blocked until its own broker-specific candidate record is approved. + +EURUSD cannot become demo-enabled merely because Gold performs well. + +## 7. Decision record schema + +Each JMB decision should be append-only and include: + +```json +{ + "schema_version": 1, + "decision_id": "stable unique id", + "created_at": "ISO-8601 UTC", + "broker": "hfmarkets", + "server": "broker server", + "account_mode": "demo", + "symbol": "XAUUSD", + "canonical_instrument": "Gold / USD", + "strategy_version": "daily-trend-v1", + "mode": "shadow", + "direction": "buy", + "reason_code": "daily_trend_breakout", + "reason_detail": "Completed daily trend filter is positive and spread gate passed.", + "entry_reference_price": 2410.25, + "stop_loss": 2402.25, + "take_profit": null, + "volume": 0.01, + "spread": 0.36, + "risk_amount": 0.80, + "max_allowed_risk": 1.00, + "gate_results": [ + { "gate": "account_demo", "state": "pass", "detail": "MT5 reports demo mode" }, + { "gate": "spread", "state": "pass", "detail": "0.36 <= configured limit" } + ], + "order_ticket": null, + "position_id": null, + "outcome": null +} +``` + +The exact field names may be implemented in TypeScript/MQL5-friendly CSV or JSONL, but the persisted meaning must remain intact. + +## 8. EA risk shell requirements + +The MT5 EA risk shell must enforce: + +- Demo account only. +- Broker/server/account binding. +- Exact symbol allowlist. +- Strategy version allowlist. +- `0.01` maximum lot for V1. +- One EA-owned open position per broker/symbol. +- No pyramiding. +- No martingale or recovery sizing. +- Stop loss required before order submission. +- Maximum spread. +- Maximum slippage. +- Maximum daily loss. +- Maximum consecutive losses. +- Minimum free margin. +- No manual/foreign position conflict on the same symbol. +- Fresh decision file and bridge state. +- Persistent kill switch. +- Restart reconciliation before new entries. +- Disk logging must work before new entries. + +If any gate cannot be evaluated, it fails closed. + +## 9. AI role + +AI agents may: + +- Review shadow decisions. +- Review demo trades after close. +- Compare JMB decisions against manual trades. +- Identify repeated mistakes. +- Recommend `NO ACTION`, `MONITOR`, `SHADOW`, `DEMO CANDIDATE`, or `PAUSE`. +- Draft daily and weekly learning journals. + +AI agents may not: + +- Submit orders. +- Change EA risk limits. +- Switch live mode on. +- Override kill switches. +- Reclassify manual trades as system strategy labels without recorded setup reasons. + +## 10. Learning loop + +Daily loop: + +1. Import bridge status and trade ledger. +2. Import JMB decision records. +3. Reconcile demo orders/deals with decision records. +4. Mark outcomes for closed demo trades. +5. Score gate discipline and strategy behavior. +6. Compare shadow decisions with actual market movement. +7. Write daily journal with mistakes, protected-risk skips, and next bounded hypothesis. + +The system learns from discipline as much as from profit. A correctly skipped bad trade is a positive learning event. + +## 11. User-visible states + +Each broker/symbol should display: + +- `LEARNING`: trade history and bridge evidence are being collected. +- `SHADOW`: JMB is logging would-trade decisions, no orders. +- `DEMO BLOCKED`: a candidate exists but a gate blocks execution. +- `DEMO ENABLED`: EA may place demo trades if all gates pass. +- `PAUSED`: kill switch or user pause blocks new entries. +- `ERROR`: required evidence/logging is malformed or stale. + +The UI must say that shadow/demo status is not live-trading approval. + +## 12. Acceptance criteria + +This design is ready for implementation planning when: + +- Manual trades are explicitly supporting evidence, not primary strategy labels. +- JMB-generated decisions are the primary learning dataset. +- Shadow mode exists before demo execution. +- Demo execution remains EA-only and deterministic. +- Gold and EURUSD are both included. +- EURUSD remains demo-blocked until its own candidate gate passes. +- Live trading remains out of scope. +- Every decision and skip is logged with reason and gate results. + +## 13. Future live boundary + +This design does not approve live trading. A future live pilot requires a separate PRD/spec with: + +- Explicit account. +- Explicit broker/server/account binding. +- Fixed strategy version. +- Human approval ceremony. +- Hard pilot-loss cap. +- Live cost validation. +- Emergency stop procedure. + +Until that exists, live remains blocked. diff --git a/src/domain/mt5/broker-cost-model.spec.ts b/src/domain/mt5/broker-cost-model.spec.ts new file mode 100644 index 000000000..4c7c15856 --- /dev/null +++ b/src/domain/mt5/broker-cost-model.spec.ts @@ -0,0 +1,132 @@ +import { afterEach, describe, expect, it } from 'vitest' +import { mkdtemp, readFile, readdir, rm } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { buildBrokerCostModel, writeBrokerCostModel } from './broker-cost-model.js' + +const now = '2026-07-13T10:00:00.000Z' +const fingerprint = 'XAUUSD|100|0.01|0.01|10|0' + +function readyInput() { + return { + version: 'hfm-observed-v1', + broker: 'hfmarkets' as const, + server: 'HFMarketsGlobal-Demo4', + symbol: 'XAUUSD' as const, + now, + bridge: { + state: 'ready' as const, + capturedAt: '2026-07-13T09:59:30.000Z', + contractFingerprint: fingerprint, + }, + spreadSamples: Array.from({ length: 100 }, (_, index) => ({ + capturedAt: new Date(Date.parse(now) - (100 - index) * 60_000).toISOString(), + spread: 0.25 + index / 1_000, + contractFingerprint: fingerprint, + })), + closedDeals: [{ + accountMode: 'demo', + symbol: 'XAUUSD', + closed: true, + commission: '-0.10', + swap: '0.00', + }], + expectedContractFingerprint: fingerprint, + configuredMaxSpread: 0.75, + configuredMaxDeviation: 0.5, + } +} + +const directories: string[] = [] + +afterEach(async () => { + await Promise.all(directories.splice(0).map((directory) => rm(directory, { recursive: true, force: true }))) +}) + +describe('broker cost model', () => { + it('becomes canary-ready only with complete fresh observed evidence', () => { + expect(buildBrokerCostModel(readyInput())).toMatchObject({ + state: 'canary_ready', + spreadSampleCount: 100, + observedMaxSpread: 0.349, + configuredMaxSpread: 0.75, + configuredMaxDeviation: 0.5, + commissionObserved: true, + swapObserved: true, + contractFingerprint: fingerprint, + }) + }) + + it('blocks missing evidence instead of degrading to a warning', () => { + const input = readyInput() + input.spreadSamples = input.spreadSamples.slice(0, 99) + input.closedDeals = [] + + const model = buildBrokerCostModel(input) + expect(model.state).toBe('blocked') + expect(model.evidence).toEqual(expect.arrayContaining([ + expect.stringMatching(/100 recent spread samples/), + expect.stringMatching(/closed demo Gold deal/), + ])) + }) + + it('blocks stale bridge, stale samples, fingerprint mismatch, and loose deviation independently', () => { + const staleBridge = readyInput() + staleBridge.bridge.capturedAt = '2026-07-13T09:57:00.000Z' + expect(buildBrokerCostModel(staleBridge).state).toBe('blocked') + + const staleSamples = readyInput() + staleSamples.spreadSamples = staleSamples.spreadSamples.map((sample) => ({ ...sample, capturedAt: '2026-07-11T10:00:00.000Z' })) + expect(buildBrokerCostModel(staleSamples).state).toBe('blocked') + + const mismatch = readyInput() + mismatch.bridge.contractFingerprint = 'different' + expect(buildBrokerCostModel(mismatch).state).toBe('blocked') + + const looseDeviation = readyInput() + looseDeviation.configuredMaxDeviation = 0.51 + expect(buildBrokerCostModel(looseDeviation).state).toBe('blocked') + }) + + it('blocks contradictory recent contract evidence even when 100 samples match', () => { + const input = readyInput() + input.spreadSamples.push({ + capturedAt: '2026-07-13T09:59:45.000Z', + spread: 0.25, + contractFingerprint: 'different', + }) + + expect(buildBrokerCostModel(input).state).toBe('blocked') + }) + + it('requires finite commission and swap fields on the same closed demo Gold deal', () => { + const input = readyInput() + input.closedDeals = [{ + accountMode: 'demo', + symbol: 'XAUUSD', + closed: true, + commission: '-0.10', + swap: 'not-a-number', + }] + + expect(buildBrokerCostModel(input)).toMatchObject({ + state: 'blocked', + commissionObserved: false, + swapObserved: false, + }) + }) + + it('writes the exact CSV schema through a replaced temporary file', async () => { + const root = await mkdtemp(join(tmpdir(), 'openalice-cost-model-')) + directories.push(root) + const model = buildBrokerCostModel(readyInput()) + await writeBrokerCostModel(root, model) + await writeBrokerCostModel(root, { ...model, version: 'hfm-observed-v2' }) + + const directory = join(root, 'hfmarkets', 'XAUUSD') + const text = await readFile(join(directory, 'cost_model.csv'), 'utf8') + expect(text.split(/\r?\n/, 1)[0]).toBe('schema_version,version,broker,server,symbol,state,observed_from,observed_to,expires_at,spread_sample_count,observed_max_spread,configured_max_spread,configured_max_deviation,commission_observed,swap_observed,contract_fingerprint,evidence_json') + expect(text.split(/\r?\n/)[1]).toContain('hfm-observed-v2') + expect(await readdir(directory)).toEqual(['cost_model.csv']) + }) +}) diff --git a/src/domain/mt5/broker-cost-model.ts b/src/domain/mt5/broker-cost-model.ts new file mode 100644 index 000000000..5014a6440 --- /dev/null +++ b/src/domain/mt5/broker-cost-model.ts @@ -0,0 +1,223 @@ +import { randomUUID } from 'node:crypto' +import { mkdir, rename, rm, writeFile } from 'node:fs/promises' +import { dirname, join } from 'node:path' + +export interface BrokerCostModel { + schemaVersion: 1 + version: string + broker: 'hfmarkets' | 'icmarkets' + server: string + symbol: 'XAUUSD' + state: 'canary_ready' | 'blocked' + observedFrom: string + observedTo: string + expiresAt: string + spreadSampleCount: number + observedMaxSpread: number + configuredMaxSpread: number + configuredMaxDeviation: number + commissionObserved: boolean + swapObserved: boolean + contractFingerprint: string + evidence: string[] +} + +export interface BrokerCostSpreadSample { + capturedAt: string + spread: number + contractFingerprint: string +} + +export interface BrokerCostClosedDeal { + accountMode: string + symbol: string + closed: boolean + commission: string | number + swap: string | number +} + +export interface BrokerCostModelInput { + version: string + broker: 'hfmarkets' | 'icmarkets' + server: string + symbol: 'XAUUSD' + now: string + bridge: { + state: string + capturedAt: string + contractFingerprint: string + } + spreadSamples: BrokerCostSpreadSample[] + closedDeals: BrokerCostClosedDeal[] + expectedContractFingerprint: string + configuredMaxSpread: number + configuredMaxDeviation: number +} + +const COST_MODEL_HEADER = 'schema_version,version,broker,server,symbol,state,observed_from,observed_to,expires_at,spread_sample_count,observed_max_spread,configured_max_spread,configured_max_deviation,commission_observed,swap_observed,contract_fingerprint,evidence_json' +const BRIDGE_MAX_AGE_MS = 2 * 60_000 +const OBSERVATION_MAX_AGE_MS = 24 * 60 * 60_000 + +const BROKER_CEILINGS = { + hfmarkets: { server: 'HFMarketsGlobal-Demo4', maxSpread: 0.75, maxDeviation: 0.5 }, + icmarkets: { server: 'ICMarketsSC-Demo', maxSpread: 0.3, maxDeviation: 0.3 }, +} as const + +function finiteAtLeastZero(value: unknown): value is number { + return typeof value === 'number' && Number.isFinite(value) && value >= 0 +} + +function parseFiniteCost(value: string | number): number | null { + if ((typeof value === 'string' && value.trim() === '') || (typeof value !== 'string' && typeof value !== 'number')) return null + const parsed = Number(value) + return Number.isFinite(parsed) ? parsed : null +} + +function timestamp(value: string): number | null { + const parsed = Date.parse(value) + return Number.isFinite(parsed) ? parsed : null +} + +export function buildBrokerCostModel(input: BrokerCostModelInput): BrokerCostModel { + const failures: string[] = [] + const nowMs = timestamp(input.now) + const ceilings = BROKER_CEILINGS[input.broker] + if (nowMs === null) failures.push('The cost-model build time is invalid.') + if (input.version.trim() === '') failures.push('The cost-model version is missing.') + if (input.symbol !== 'XAUUSD' || input.server !== ceilings.server) { + failures.push('The broker, server, and Gold symbol identity must match the exact demo allowlist.') + } + + const bridgeCapturedAt = timestamp(input.bridge.capturedAt) + const bridgeAgeMs = nowMs === null || bridgeCapturedAt === null ? null : nowMs - bridgeCapturedAt + if (input.bridge.state !== 'ready' || bridgeAgeMs === null || bridgeAgeMs < 0 || bridgeAgeMs > BRIDGE_MAX_AGE_MS) { + failures.push('A fresh ready bridge observation from the last two minutes is required.') + } + + const expectedFingerprint = input.expectedContractFingerprint + if (expectedFingerprint.trim() === '' || input.bridge.contractFingerprint !== expectedFingerprint) { + failures.push('The bridge contract fingerprint does not match the configured fingerprint.') + } + + const recentCandidates = input.spreadSamples.filter((sample) => { + const capturedAt = timestamp(sample.capturedAt) + const ageMs = nowMs === null || capturedAt === null ? null : nowMs - capturedAt + return ageMs !== null && ageMs >= 0 && ageMs <= OBSERVATION_MAX_AGE_MS + }) + if (recentCandidates.some((sample) => sample.contractFingerprint !== expectedFingerprint)) { + failures.push('Recent spread evidence contains a mismatched contract fingerprint.') + } + if (recentCandidates.some((sample) => !finiteAtLeastZero(sample.spread))) { + failures.push('Recent spread evidence contains an invalid spread value.') + } + const recentSamples = recentCandidates.filter((sample) => { + return finiteAtLeastZero(sample.spread) + && sample.contractFingerprint === expectedFingerprint + }) + if (recentSamples.length < 100) failures.push('At least 100 recent spread samples with the matching contract fingerprint are required.') + + const orderedSampleTimes = recentSamples + .map((sample) => sample.capturedAt) + .sort((left, right) => Date.parse(left) - Date.parse(right)) + const observedFrom = orderedSampleTimes[0] ?? input.now + const observedTo = orderedSampleTimes.at(-1) ?? input.now + const observedMaxSpread = recentSamples.length === 0 + ? 0 + : Math.max(...recentSamples.map((sample) => sample.spread)) + + if (!finiteAtLeastZero(input.configuredMaxSpread) + || input.configuredMaxSpread === 0 + || input.configuredMaxSpread > ceilings.maxSpread + || observedMaxSpread > input.configuredMaxSpread) { + failures.push('The configured spread ceiling is not conservative for the broker observations.') + } + if (!finiteAtLeastZero(input.configuredMaxDeviation) + || input.configuredMaxDeviation === 0 + || input.configuredMaxDeviation > ceilings.maxDeviation) { + failures.push('The configured deviation ceiling exceeds the immutable broker ceiling.') + } + + const observedDeal = input.closedDeals.find((deal) => deal.accountMode === 'demo' + && deal.symbol === 'XAUUSD' + && deal.closed + && parseFiniteCost(deal.commission) !== null + && parseFiniteCost(deal.swap) !== null) + const commissionObserved = observedDeal !== undefined + const swapObserved = observedDeal !== undefined + if (!observedDeal) failures.push('A closed demo Gold deal with parseable commission and swap fields is required.') + + const expiresAt = nowMs === null + ? input.now + : new Date(nowMs + OBSERVATION_MAX_AGE_MS).toISOString() + const evidence = failures.length > 0 + ? failures + : [ + `Fresh bridge observed at ${input.bridge.capturedAt}.`, + `${recentSamples.length} recent spread samples matched the contract fingerprint.`, + 'Commission and swap fields were observed on a closed demo Gold deal.', + ] + + return { + schemaVersion: 1, + version: input.version, + broker: input.broker, + server: input.server, + symbol: input.symbol, + state: failures.length === 0 ? 'canary_ready' : 'blocked', + observedFrom, + observedTo, + expiresAt, + spreadSampleCount: recentSamples.length, + observedMaxSpread, + configuredMaxSpread: input.configuredMaxSpread, + configuredMaxDeviation: input.configuredMaxDeviation, + commissionObserved, + swapObserved, + contractFingerprint: expectedFingerprint, + evidence, + } +} + +function csvCell(value: string): string { + return `"${value.replaceAll('"', '""')}"` +} + +export function serializeBrokerCostModelCsv(model: BrokerCostModel): string { + const row = [ + model.schemaVersion, + model.version, + model.broker, + model.server, + model.symbol, + model.state, + model.observedFrom, + model.observedTo, + model.expiresAt, + model.spreadSampleCount, + model.observedMaxSpread, + model.configuredMaxSpread, + model.configuredMaxDeviation, + model.commissionObserved ? 1 : 0, + model.swapObserved ? 1 : 0, + model.contractFingerprint, + csvCell(JSON.stringify(model.evidence)), + ].join(',') + return `${COST_MODEL_HEADER}\n${row}\n` +} + +async function renameReplacing(temporary: string, destination: string): Promise { + await rename(temporary, destination) +} + +export async function writeBrokerCostModel(root: string, model: BrokerCostModel): Promise { + const destination = join(root, model.broker, model.symbol, 'cost_model.csv') + await mkdir(dirname(destination), { recursive: true }) + const temporary = `${destination}.${process.pid}.${randomUUID()}.tmp` + await writeFile(temporary, serializeBrokerCostModelCsv(model), { encoding: 'utf8', flag: 'wx' }) + try { + await renameReplacing(temporary, destination) + } catch (error) { + await rm(temporary, { force: true }) + throw error + } +} diff --git a/src/domain/mt5/completed-d1.spec.ts b/src/domain/mt5/completed-d1.spec.ts new file mode 100644 index 000000000..a011fd972 --- /dev/null +++ b/src/domain/mt5/completed-d1.spec.ts @@ -0,0 +1,145 @@ +import { afterEach, describe, expect, it, vi } from 'vitest' +import { mkdtemp, mkdir, rm, utimes, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { deriveCompletedTrendObservation, parseCompletedD1Csv, readMt5CompletedD1 } from './completed-d1.js' + +const fsMocks = vi.hoisted(() => ({ + open: vi.fn(), + readFile: vi.fn(), + stat: vi.fn(), +})) + +vi.mock('node:fs/promises', async (importOriginal) => { + const actual = await importOriginal() + fsMocks.open.mockImplementation(actual.open) + fsMocks.readFile.mockImplementation(actual.readFile) + fsMocks.stat.mockImplementation(actual.stat) + return { ...actual, ...fsMocks } +}) + +const directories: string[] = [] + +afterEach(async () => { + await Promise.all(directories.splice(0).map((directory) => rm(directory, { recursive: true, force: true }))) + const actual = await vi.importActual('node:fs/promises') + fsMocks.open.mockReset().mockImplementation(actual.open) + fsMocks.readFile.mockReset().mockImplementation(actual.readFile) + fsMocks.stat.mockReset().mockImplementation(actual.stat) +}) + +function csv(closes: number[]): string { + const rows = closes.map((close, index) => + `1,2026-07-13T09:00:00.000Z,hfmarkets,HFMarketsGlobal-Demo4,demo,XAUUSD,2026-05-${String(index + 1).padStart(2, '0')},${index + 1},${close},${close},${close},${close}`, + ) + return ['schema_version,captured_at,broker,server,account_mode,symbol,bar_as_of,bar_open_epoch,open,high,low,close', ...rows].join('\n') +} + +describe('completed D1 broker bars', () => { + it('derives the signal from completed bars only', () => { + const parsed = parseCompletedD1Csv(csv([100, 101, 103])) + expect(deriveCompletedTrendObservation(parsed, 2)).toMatchObject({ + direction: 'uptrend', + lookbackDays: 2, + latestClose: 103, + referenceClose: 100, + }) + }) + + it('rejects duplicate or descending bar epochs', () => { + expect(() => parseCompletedD1Csv(csv([100, 101]).replace(',2,101,', ',1,101,'))).toThrow(/ascending/) + }) + + it('rejects non-finite prices and malformed row widths', () => { + expect(() => parseCompletedD1Csv(csv([100, 101]).replace(',101,101,101,101', ',NaN,101,101,101'))).toThrow(/finite/) + expect(() => parseCompletedD1Csv(`${csv([100])},extra`)).toThrow(/columns/) + }) + + it('requires consistent demo metadata and unique bar dates', () => { + expect(() => parseCompletedD1Csv(csv([100, 101]).replace('HFMarketsGlobal-Demo4,demo,XAUUSD,2026-05-02', 'Other-Demo,demo,XAUUSD,2026-05-02'))).toThrow(/metadata/) + expect(() => parseCompletedD1Csv(csv([100]).replace(',demo,', ',real,'))).toThrow(/demo/) + expect(() => parseCompletedD1Csv(csv([100, 101]).replace('2026-05-02,2', '2026-05-01,2'))).toThrow(/unique/) + }) + + it('reads a fresh file using its modification time', async () => { + const root = await mkdtemp(join(tmpdir(), 'openalice-mt5-d1-')) + directories.push(root) + const directory = join(root, 'hfmarkets', 'XAUUSD') + const path = join(directory, 'completed_d1.csv') + await mkdir(directory, { recursive: true }) + await writeFile(path, csv([100, 101, 103])) + await utimes(path, new Date('2026-07-13T08:00:00.000Z'), new Date('2026-07-13T08:00:00.000Z')) + + await expect(readMt5CompletedD1(root, 'hfmarkets', 'XAUUSD', { + now: new Date('2026-07-13T09:30:00.000Z'), + maxAgeHours: 2, + expectedServer: 'HFMarketsGlobal-Demo4', + })).resolves.toMatchObject({ + state: 'ready', + ageHours: 1.5, + parsed: { broker: 'hfmarkets', symbol: 'XAUUSD' }, + }) + }) + + it('fails closed for missing, stale, unsafe, and malformed files', async () => { + const root = await mkdtemp(join(tmpdir(), 'openalice-mt5-d1-')) + directories.push(root) + const directory = join(root, 'hfmarkets', 'XAUUSD') + const path = join(directory, 'completed_d1.csv') + + await expect(readMt5CompletedD1(root, 'hfmarkets', 'XAUUSD', { maxAgeHours: 2, expectedServer: 'HFMarketsGlobal-Demo4' })).resolves.toMatchObject({ state: 'missing', parsed: null }) + + await mkdir(directory, { recursive: true }) + await writeFile(path, csv([100, 101])) + await utimes(path, new Date('2026-07-13T06:00:00.000Z'), new Date('2026-07-13T06:00:00.000Z')) + await expect(readMt5CompletedD1(root, 'hfmarkets', 'XAUUSD', { + now: new Date('2026-07-13T09:00:00.000Z'), + maxAgeHours: 2, + expectedServer: 'HFMarketsGlobal-Demo4', + })).resolves.toMatchObject({ state: 'stale', ageHours: 3, parsed: expect.any(Object) }) + + await writeFile(path, csv([100, 101]).replace(',demo,', ',real,')) + await expect(readMt5CompletedD1(root, 'hfmarkets', 'XAUUSD', { maxAgeHours: 2, expectedServer: 'HFMarketsGlobal-Demo4' })).resolves.toMatchObject({ state: 'unsafe', parsed: null }) + + await writeFile(path, 'not,a,completed,d1,file') + await expect(readMt5CompletedD1(root, 'hfmarkets', 'XAUUSD', { maxAgeHours: 2, expectedServer: 'HFMarketsGlobal-Demo4' })).resolves.toMatchObject({ state: 'malformed', parsed: null }) + }) + + it('fails closed when fresh completed D1 evidence comes from the wrong server', async () => { + const root = await mkdtemp(join(tmpdir(), 'openalice-mt5-d1-')) + directories.push(root) + const directory = join(root, 'hfmarkets', 'XAUUSD') + const path = join(directory, 'completed_d1.csv') + await mkdir(directory, { recursive: true }) + await writeFile(path, csv([100, 101, 103]).replaceAll('HFMarketsGlobal-Demo4', 'Other-Demo')) + await utimes(path, new Date('2026-07-13T09:00:00.000Z'), new Date('2026-07-13T09:00:00.000Z')) + + await expect(readMt5CompletedD1(root, 'hfmarkets', 'XAUUSD', { + now: new Date('2026-07-13T09:30:00.000Z'), + maxAgeHours: 2, + expectedServer: 'HFMarketsGlobal-Demo4', + })).resolves.toMatchObject({ state: 'unsafe', parsed: null }) + }) + + it('reads contents and modification time from the same file identity', async () => { + const staleModifiedAt = new Date('2026-07-13T06:00:00.000Z') + const freshModifiedAt = new Date('2026-07-13T09:00:00.000Z') + const staleText = csv([100, 101]) + const close = vi.fn().mockResolvedValue(undefined) + + fsMocks.readFile.mockResolvedValueOnce(staleText) + fsMocks.stat.mockResolvedValueOnce({ mtime: freshModifiedAt }) + fsMocks.open.mockResolvedValueOnce({ + readFile: vi.fn().mockResolvedValue(staleText), + stat: vi.fn().mockResolvedValue({ mtime: staleModifiedAt }), + close, + }) + + await expect(readMt5CompletedD1('root', 'hfmarkets', 'XAUUSD', { + now: freshModifiedAt, + maxAgeHours: 2, + expectedServer: 'HFMarketsGlobal-Demo4', + })).resolves.toMatchObject({ state: 'stale', ageHours: 3 }) + expect(close).toHaveBeenCalledOnce() + }) +}) diff --git a/src/domain/mt5/completed-d1.ts b/src/domain/mt5/completed-d1.ts new file mode 100644 index 000000000..0b487a8a4 --- /dev/null +++ b/src/domain/mt5/completed-d1.ts @@ -0,0 +1,211 @@ +import { open } from 'node:fs/promises' +import { join } from 'node:path' + +export type CompletedD1State = 'ready' | 'missing' | 'stale' | 'unsafe' | 'malformed' +export type TrendDirection = 'uptrend' | 'downtrend' | 'flat' + +export interface CompletedD1Bar { + asOf: string + openEpoch: number + open: number + high: number + low: number + close: number +} + +export interface ParsedCompletedD1 { + capturedAt: string + broker: string + server: string + accountMode: string + symbol: string + bars: CompletedD1Bar[] +} + +export interface Mt5CompletedD1Summary { + state: CompletedD1State + detail: string + ageHours: number | null + parsed: ParsedCompletedD1 | null +} + +export interface CompletedTrendObservation { + asOf: string + direction: TrendDirection + lookbackReturn: number + lookbackDays: number + latestClose: number + referenceClose: number +} + +const COMPLETED_D1_HEADER = 'schema_version,captured_at,broker,server,account_mode,symbol,bar_as_of,bar_open_epoch,open,high,low,close' +const COMPLETED_D1_COLUMN_COUNT = 12 + +class CompletedD1ParseError extends Error { + constructor( + message: string, + readonly state: Extract = 'malformed', + ) { + super(message) + } +} + +function malformed(message: string): never { + throw new CompletedD1ParseError(message) +} + +function parseFinite(value: string | undefined, field: string): number { + const parsed = Number(value) + if (value === undefined || value.trim() === '' || !Number.isFinite(parsed)) { + malformed(`Completed D1 ${field} must be finite`) + } + return parsed +} + +export function parseCompletedD1Csv(text: string): ParsedCompletedD1 { + const [header, ...rows] = text.trim().split(/\r?\n/) + if (header !== COMPLETED_D1_HEADER || rows.length === 0) { + malformed('Completed D1 CSV has an invalid header or no bars') + } + + let previousEpoch = Number.NEGATIVE_INFINITY + let expectedMetadata: string[] | undefined + const barDates = new Set() + const bars = rows.map((row) => { + const values = row.split(',') + if (values.length !== COMPLETED_D1_COLUMN_COUNT) malformed('Completed D1 rows must contain exactly 12 columns') + if (values[0] !== '1') malformed('Completed D1 schema version must be 1') + + const metadata = values.slice(1, 6) + if (metadata.some((value) => value.trim() === '')) malformed('Completed D1 metadata cannot be empty') + if (expectedMetadata === undefined) { + expectedMetadata = metadata + } else if (metadata.some((value, index) => value !== expectedMetadata![index])) { + malformed('Completed D1 metadata must be identical across rows') + } + if (values[4] !== 'demo') { + throw new CompletedD1ParseError('Completed D1 observations must come from a demo account', 'unsafe') + } + + const asOf = values[6]! + if (asOf === '') malformed('Completed D1 bar_as_of cannot be empty') + if (barDates.has(asOf)) malformed('Completed D1 bar_as_of values must be unique') + barDates.add(asOf) + + const openEpoch = parseFinite(values[7], 'bar_open_epoch') + if (!Number.isInteger(openEpoch)) malformed('Completed D1 bar_open_epoch must be an integer') + if (openEpoch <= previousEpoch) malformed('Completed D1 bar epochs must be strictly ascending') + previousEpoch = openEpoch + return { + asOf, + openEpoch, + open: parseFinite(values[8], 'open'), + high: parseFinite(values[9], 'high'), + low: parseFinite(values[10], 'low'), + close: parseFinite(values[11], 'close'), + } + }) + return { + capturedAt: expectedMetadata![0]!, + broker: expectedMetadata![1]!, + server: expectedMetadata![2]!, + accountMode: expectedMetadata![3]!, + symbol: expectedMetadata![4]!, + bars, + } +} + +export async function readMt5CompletedD1( + root: string, + broker: string, + symbol: string, + options: { now?: Date; maxAgeHours: number; expectedServer: string }, +): Promise { + if (!Number.isFinite(options.maxAgeHours) || options.maxAgeHours <= 0) { + throw new Error('Completed D1 maximum age must be a positive finite number of hours') + } + + const path = join(root, broker, symbol, 'completed_d1.csv') + let text: string + let modifiedAt: Date + try { + const handle = await open(path, 'r') + try { + const [contents, metadata] = await Promise.all([ + handle.readFile('utf8'), + handle.stat(), + ]) + text = contents + modifiedAt = metadata.mtime + } finally { + await handle.close() + } + } catch { + return { + state: 'missing', + detail: 'The completed D1 broker export is missing or unreadable.', + ageHours: null, + parsed: null, + } + } + + let parsed: ParsedCompletedD1 + try { + parsed = parseCompletedD1Csv(text) + } catch (error) { + const state = error instanceof CompletedD1ParseError ? error.state : 'malformed' + return { + state, + detail: error instanceof Error ? error.message : 'The completed D1 broker export is malformed.', + ageHours: null, + parsed: null, + } + } + + if (parsed.broker !== broker + || parsed.server !== options.expectedServer + || parsed.symbol !== symbol + || parsed.accountMode !== 'demo') { + return { + state: 'unsafe', + detail: 'The completed D1 export identity does not match the requested demo broker, server, and symbol.', + ageHours: null, + parsed: null, + } + } + + const now = options.now ?? new Date() + const ageHours = Math.max(0, (now.getTime() - modifiedAt.getTime()) / 3_600_000) + if (ageHours > options.maxAgeHours) { + return { + state: 'stale', + detail: 'The completed D1 broker export is older than the policy maximum observation age.', + ageHours, + parsed, + } + } + + return { + state: 'ready', + detail: 'Completed D1 broker bars are current and demo-only.', + ageHours, + parsed, + } +} + +export function deriveCompletedTrendObservation(input: ParsedCompletedD1, lookbackDays: number): CompletedTrendObservation { + if (!Number.isInteger(lookbackDays) || lookbackDays <= 0 || input.bars.length < lookbackDays + 1) { + throw new Error('Completed D1 history is insufficient for the selected lookback') + } + const latest = input.bars.at(-1)! + const reference = input.bars.at(-(lookbackDays + 1))! + const lookbackReturn = latest.close / reference.close - 1 + return { + asOf: latest.asOf, + direction: lookbackReturn > 0 ? 'uptrend' : lookbackReturn < 0 ? 'downtrend' : 'flat', + lookbackReturn, + lookbackDays, + latestClose: latest.close, + referenceClose: reference.close, + } +} diff --git a/src/domain/mt5/decision-record.spec.ts b/src/domain/mt5/decision-record.spec.ts new file mode 100644 index 000000000..9cf9dbf6c --- /dev/null +++ b/src/domain/mt5/decision-record.spec.ts @@ -0,0 +1,146 @@ +import { afterEach, describe, expect, it } from 'vitest' +import { mkdir, mkdtemp, readFile, rm } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { + appendJmbDecisionRecord, + createJmbDecisionId, + parseLatestDecisionCsv, + serializeLatestDecisionCsv, + summarizeLatestJmbDecision, + writeLatestJmbDecision, + type JmbDecisionRecord, +} from './decision-record.js' + +const directories: string[] = [] + +afterEach(async () => { + await Promise.all(directories.splice(0).map((directory) => rm(directory, { recursive: true, force: true }))) +}) + +function sampleDecision(overrides: Partial = {}): JmbDecisionRecord { + const base: JmbDecisionRecord = { + schemaVersion: 1, + decisionId: 'decision-1', + createdAt: '2026-07-13T09:00:00.000Z', + broker: 'hfmarkets', + server: 'HFMarketsGlobal-Demo4', + accountMode: 'demo', + symbol: 'XAUUSD', + canonicalInstrument: 'Gold / USD', + strategyVersion: 'daily-trend-v1', + mode: 'shadow', + direction: 'buy', + reasonCode: 'daily_trend_shadow', + reasonDetail: 'Completed daily trend filter is positive.', + entryReferencePrice: 2410.25, + stopLoss: 2402.25, + takeProfit: null, + volume: 0.01, + spread: 0.36, + riskAmount: 0.8, + maxAllowedRisk: 1, + gateResults: [ + { gate: 'account_demo', state: 'pass', detail: 'MT5 reports demo mode' }, + { gate: 'shadow_only', state: 'pass', detail: 'No order submission in Plan 2' }, + ], + orderTicket: null, + positionId: null, + outcome: null, + } + return { ...base, ...overrides } +} + +describe('JMB decision records', () => { + it('creates stable ids from deterministic fields', () => { + const id = createJmbDecisionId(sampleDecision()) + expect(id).toBe(createJmbDecisionId(sampleDecision())) + expect(id).not.toBe(createJmbDecisionId(sampleDecision({ direction: 'sell' }))) + }) + + it('round-trips latest-decision CSV without losing gate results', () => { + const decision = sampleDecision({ reasonDetail: 'Spread, trend, and stop checked' }) + const parsed = parseLatestDecisionCsv(serializeLatestDecisionCsv(decision)) + expect(parsed).toMatchObject({ + broker: 'hfmarkets', + symbol: 'XAUUSD', + mode: 'shadow', + direction: 'buy', + volume: 0.01, + stopLoss: 2402.25, + }) + expect(parsed.gateResults).toHaveLength(2) + }) + + it('round-trips quoted CSV fields containing commas, quotes, and newlines', () => { + const decision = sampleDecision({ + reasonDetail: 'Spread, "trend", and stop checked\nwith broker evidence', + gateResults: [ + { gate: 'broker_note', state: 'warn', detail: 'Value had comma, "quote", and newline\nfrom MT5' }, + ], + }) + + const parsed = parseLatestDecisionCsv(serializeLatestDecisionCsv(decision)) + + expect(parsed.reasonDetail).toBe('Spread, "trend", and stop checked\nwith broker evidence') + expect(parsed.gateResults).toEqual([ + { gate: 'broker_note', state: 'warn', detail: 'Value had comma, "quote", and newline\nfrom MT5' }, + ]) + }) + + it('writes append-only JSONL and latest CSV under broker symbol folders', async () => { + const root = await mkdtemp(join(tmpdir(), 'jmb-decisions-')) + directories.push(root) + const decision = sampleDecision() + + await appendJmbDecisionRecord(root, decision) + await writeLatestJmbDecision(root, decision) + + const jsonl = await readFile(join(root, 'hfmarkets', 'XAUUSD', 'decisions.jsonl'), 'utf8') + const latest = await readFile(join(root, 'hfmarkets', 'XAUUSD', 'latest_decision.csv'), 'utf8') + expect(jsonl.trim().split('\n')).toHaveLength(1) + expect(parseLatestDecisionCsv(latest).decisionId).toBe('decision-1') + }) + + it('summarizes unreadable latest CSV as blocked instead of throwing', async () => { + const root = await mkdtemp(join(tmpdir(), 'jmb-decisions-')) + directories.push(root) + await writeLatestJmbDecision(root, sampleDecision()) + await import('node:fs/promises').then(({ writeFile }) => + writeFile(join(root, 'hfmarkets', 'XAUUSD', 'latest_decision.csv'), 'broken,csv\n1'), + ) + + const summary = await summarizeLatestJmbDecision(root, 'hfmarkets', 'XAUUSD') + + expect(summary.state).toBe('error') + expect(summary.label).toBe('Decision unreadable') + }) + + it('summarizes malformed required numeric fields as unreadable', async () => { + const root = await mkdtemp(join(tmpdir(), 'jmb-decisions-')) + directories.push(root) + await writeLatestJmbDecision(root, sampleDecision()) + await import('node:fs/promises').then(({ writeFile }) => + writeFile( + join(root, 'hfmarkets', 'XAUUSD', 'latest_decision.csv'), + serializeLatestDecisionCsv(sampleDecision()).replace(',0.01,0.36,0.8,1,', ',not-a-number,0.36,0.8,also-bad,'), + ), + ) + + const summary = await summarizeLatestJmbDecision(root, 'hfmarkets', 'XAUUSD') + + expect(summary.state).toBe('error') + expect(summary.label).toBe('Decision unreadable') + }) + + it('summarizes read errors as unreadable instead of missing decision', async () => { + const root = await mkdtemp(join(tmpdir(), 'jmb-decisions-')) + directories.push(root) + await mkdir(join(root, 'hfmarkets', 'XAUUSD', 'latest_decision.csv'), { recursive: true }) + + const summary = await summarizeLatestJmbDecision(root, 'hfmarkets', 'XAUUSD') + + expect(summary.state).toBe('error') + expect(summary.label).toBe('Decision unreadable') + }) +}) diff --git a/src/domain/mt5/decision-record.ts b/src/domain/mt5/decision-record.ts new file mode 100644 index 000000000..468629030 --- /dev/null +++ b/src/domain/mt5/decision-record.ts @@ -0,0 +1,297 @@ +import { createHash } from 'node:crypto' +import { appendFile, mkdir, readFile, stat, writeFile } from 'node:fs/promises' +import { join } from 'node:path' + +export type JmbDecisionMode = 'shadow' | 'demo_blocked' | 'demo_order_requested' | 'demo_filled' | 'demo_closed' | 'skipped' +export type JmbDecisionDirection = 'buy' | 'sell' | 'flat' +export type JmbGateState = 'pass' | 'fail' | 'warn' + +export interface JmbGateResult { + gate: string + state: JmbGateState + detail: string +} + +export interface JmbDecisionRecord { + schemaVersion: 1 + decisionId: string + createdAt: string + broker: string + server: string | null + accountMode: string | null + symbol: string + canonicalInstrument: string + strategyVersion: string + mode: JmbDecisionMode + direction: JmbDecisionDirection + reasonCode: string + reasonDetail: string + entryReferencePrice: number | null + stopLoss: number | null + takeProfit: number | null + volume: number + spread: number | null + riskAmount: number | null + maxAllowedRisk: number + gateResults: JmbGateResult[] + orderTicket: string | null + positionId: string | null + outcome: string | null +} + +export interface JmbDecisionSummary { + state: 'no_decision' | 'shadow' | 'demo_blocked' | 'error' + label: string + detail: string + broker: string + symbol: string + lastUpdated: string | null + decision: JmbDecisionRecord | null +} + +const HEADER = [ + 'schema_version', + 'decision_id', + 'created_at', + 'broker', + 'server', + 'account_mode', + 'symbol', + 'canonical_instrument', + 'strategy_version', + 'mode', + 'direction', + 'reason_code', + 'reason_detail', + 'entry_reference_price', + 'stop_loss', + 'take_profit', + 'volume', + 'spread', + 'risk_amount', + 'max_allowed_risk', + 'gate_results_json', + 'order_ticket', + 'position_id', + 'outcome', +] + +function csvEscape(value: string): string { + if (/[",\r\n]/.test(value)) return `"${value.replace(/"/g, '""')}"` + return value +} + +function parseCsvRecords(text: string): string[][] { + const records: string[][] = [] + let record: string[] = [] + let current = '' + let quoted = false + + for (let index = 0; index < text.length; index += 1) { + const character = text[index] + + if (character === '"') { + if (quoted && text[index + 1] === '"') { + current += '"' + index += 1 + } else { + quoted = !quoted + } + } else if (character === ',' && !quoted) { + record.push(current) + current = '' + } else if ((character === '\n' || character === '\r') && !quoted) { + record.push(current) + records.push(record) + record = [] + current = '' + if (character === '\r' && text[index + 1] === '\n') index += 1 + } else { + current += character + } + } + + if (quoted) throw new Error('Decision CSV has an unterminated quoted field') + if (current !== '' || record.length > 0) { + record.push(current) + records.push(record) + } + + return records +} + +function numberOrNull(value: string): number | null { + if (value === '') return null + const parsed = Number(value) + if (!Number.isFinite(parsed)) throw new Error(`Invalid numeric decision field: ${value}`) + return parsed +} + +function requiredNumber(value: string, field: string): number { + if (value === '') throw new Error(`Missing numeric decision field: ${field}`) + const parsed = Number(value) + if (!Number.isFinite(parsed)) throw new Error(`Invalid numeric decision field: ${field}`) + return parsed +} + +export function createJmbDecisionId( + input: Pick, +): string { + return createHash('sha256') + .update([input.createdAt, input.broker, input.symbol, input.strategyVersion, input.mode, input.direction].join('|')) + .digest('hex') + .slice(0, 24) +} + +export function serializeLatestDecisionCsv(record: JmbDecisionRecord): string { + const values = [ + String(record.schemaVersion), + record.decisionId, + record.createdAt, + record.broker, + record.server ?? '', + record.accountMode ?? '', + record.symbol, + record.canonicalInstrument, + record.strategyVersion, + record.mode, + record.direction, + record.reasonCode, + record.reasonDetail, + record.entryReferencePrice == null ? '' : String(record.entryReferencePrice), + record.stopLoss == null ? '' : String(record.stopLoss), + record.takeProfit == null ? '' : String(record.takeProfit), + String(record.volume), + record.spread == null ? '' : String(record.spread), + record.riskAmount == null ? '' : String(record.riskAmount), + String(record.maxAllowedRisk), + JSON.stringify(record.gateResults), + record.orderTicket ?? '', + record.positionId ?? '', + record.outcome ?? '', + ] + + return `${HEADER.join(',')}\n${values.map(csvEscape).join(',')}\n` +} + +export function parseLatestDecisionCsv(text: string): JmbDecisionRecord { + const records = parseCsvRecords(text) + if (records.length !== 2) throw new Error('Decision CSV is missing header or value row') + + const [headers, values] = records + if (headers.join(',') !== HEADER.join(',') || values.length !== HEADER.length) throw new Error('Decision CSV schema mismatch') + + const row = Object.fromEntries(headers.map((header, index) => [header, values[index] ?? ''])) + + return { + schemaVersion: 1, + decisionId: row['decision_id']!, + createdAt: row['created_at']!, + broker: row['broker']!, + server: row['server'] || null, + accountMode: row['account_mode'] || null, + symbol: row['symbol']!, + canonicalInstrument: row['canonical_instrument']!, + strategyVersion: row['strategy_version']!, + mode: row['mode'] as JmbDecisionMode, + direction: row['direction'] as JmbDecisionDirection, + reasonCode: row['reason_code']!, + reasonDetail: row['reason_detail']!, + entryReferencePrice: numberOrNull(row['entry_reference_price']!), + stopLoss: numberOrNull(row['stop_loss']!), + takeProfit: numberOrNull(row['take_profit']!), + volume: requiredNumber(row['volume']!, 'volume'), + spread: numberOrNull(row['spread']!), + riskAmount: numberOrNull(row['risk_amount']!), + maxAllowedRisk: requiredNumber(row['max_allowed_risk']!, 'max_allowed_risk'), + gateResults: JSON.parse(row['gate_results_json']!) as JmbGateResult[], + orderTicket: row['order_ticket'] || null, + positionId: row['position_id'] || null, + outcome: row['outcome'] || null, + } +} + +function decisionDirectory(root: string, broker: string, symbol: string): string { + return join(root, broker, symbol) +} + +export async function appendJmbDecisionRecord(root: string, record: JmbDecisionRecord): Promise { + const directory = decisionDirectory(root, record.broker, record.symbol) + await mkdir(directory, { recursive: true }) + await appendFile(join(directory, 'decisions.jsonl'), `${JSON.stringify(record)}\n`, 'utf8') +} + +export async function writeLatestJmbDecision(root: string, record: JmbDecisionRecord): Promise { + const directory = decisionDirectory(root, record.broker, record.symbol) + await mkdir(directory, { recursive: true }) + await writeFile(join(directory, 'latest_decision.csv'), serializeLatestDecisionCsv(record), 'utf8') +} + +function fileErrorCode(error: unknown): string | null { + return typeof error === 'object' && error !== null && 'code' in error ? String((error as { code: unknown }).code) : null +} + +function noDecisionSummary(broker: string, symbol: string): JmbDecisionSummary { + return { + state: 'no_decision', + label: 'No JMB decision yet', + detail: 'Run the shadow decision runner before enabling any demo risk shell.', + broker, + symbol, + lastUpdated: null, + decision: null, + } +} + +function unreadableSummary(broker: string, symbol: string, lastUpdated: string | null): JmbDecisionSummary { + return { + state: 'error', + label: 'Decision unreadable', + detail: 'The latest decision CSV is malformed. The risk shell must fail closed.', + broker, + symbol, + lastUpdated, + decision: null, + } +} + +export async function summarizeLatestJmbDecision( + root: string, + broker: string, + symbol: string, + now = new Date(), +): Promise { + const path = join(root, broker, symbol, 'latest_decision.csv') + let text: string + let modified: Date + + try { + modified = (await stat(path)).mtime + } catch (error) { + return fileErrorCode(error) === 'ENOENT' ? noDecisionSummary(broker, symbol) : unreadableSummary(broker, symbol, null) + } + + try { + text = await readFile(path, 'utf8') + } catch (error) { + return fileErrorCode(error) === 'ENOENT' ? noDecisionSummary(broker, symbol) : unreadableSummary(broker, symbol, modified.toISOString()) + } + + try { + const decision = parseLatestDecisionCsv(text) + const ageMinutes = Math.round((now.getTime() - modified.getTime()) / 60_000) + const state = decision.mode === 'demo_blocked' ? 'demo_blocked' : decision.mode === 'shadow' || decision.mode === 'skipped' ? 'shadow' : 'error' + + return { + state, + label: decision.mode === 'demo_blocked' ? 'Demo blocked by gates' : 'Shadow decision logged', + detail: `Latest ${decision.mode} decision is ${ageMinutes} minutes old.`, + broker, + symbol, + lastUpdated: modified.toISOString(), + decision, + } + } catch { + return unreadableSummary(broker, symbol, modified.toISOString()) + } +} diff --git a/src/domain/mt5/demo-canary-source.spec.ts b/src/domain/mt5/demo-canary-source.spec.ts new file mode 100644 index 000000000..459b1a241 --- /dev/null +++ b/src/domain/mt5/demo-canary-source.spec.ts @@ -0,0 +1,561 @@ +import { readFile, readdir } from 'node:fs/promises' +import { join } from 'node:path' +import { describe, expect, it } from 'vitest' + +const CANARY_DIRECTORY = join('tools', 'mt5', 'JmbGoldmineDemoCanary') + +async function readCanarySource(file: string): Promise { + return readFile(join(CANARY_DIRECTORY, file), 'utf8') +} + +async function readMqlSources(directory: string): Promise> { + const entries = await readdir(directory, { withFileTypes: true }) + const nested = await Promise.all(entries.map(async (entry) => { + const path = join(directory, entry.name) + if (entry.isDirectory()) return readMqlSources(path) + if (!/\.mq[5h]$/i.test(entry.name)) return [] + return [{ path, source: await readFile(path, 'utf8') }] + })) + return nested.flat() +} + +async function readCanarySources(): Promise> { + return readMqlSources(CANARY_DIRECTORY) +} + +async function readCanaryBundle(): Promise { + return (await readCanarySources()).map(({ source }) => source).join('\n') +} + +describe('MT5 demo canary source contract', () => { + it('has one protected gateway and keeps every other MQL source order-API-free', async () => { + const sources = await readMqlSources(join('tools', 'mt5')) + const gatewayPath = join(CANARY_DIRECTORY, 'JmbCanaryTradeGateway.mqh') + const gateway = sources.find(({ path }) => path === gatewayPath) + + expect(gateway, 'the protected-order gateway must exist').toBeDefined() + expect(gateway?.source.match(/\bOrderCheck\s*\(/g)).toHaveLength(1) + expect(gateway?.source.match(/\bOrderSend\s*\(/g)).toHaveLength(1) + expect(gateway?.source).toContain('TradeSubmitResult CheckedSendCanaryRequest(') + expect(gateway?.source).toMatch(/SubmitProtectedMarketOrder[\s\S]*CheckedSendCanaryRequest\(request\)/) + expect(gateway?.source).toMatch(/SubmitCanaryReversalClose[\s\S]*CheckedSendCanaryRequest\(request\)/) + expect(gateway?.source).toMatch(/SubmitCanaryEmergencyClose[\s\S]*CheckedSendCanaryRequest\(request\)/) + expect(gateway?.source).not.toMatch(/\bCTrade\b/) + for (const file of sources.filter(({ path }) => path !== gatewayPath)) { + expect(file.source, `${file.path} must not call an order API`).not.toMatch(/\bOrder(?:Check|Send)\s*\(/) + expect(file.source, `${file.path} must not use CTrade`).not.toMatch(/\bCTrade\b/) + } + }) + + it('revalidates the exact local account binding inside every mutating gateway', async () => { + const [gateway, main, harness] = await Promise.all([ + readCanarySource('JmbCanaryTradeGateway.mqh'), + readCanarySource('JmbGoldmineDemoCanary.mq5'), + readFile(join('tools', 'mt5', 'tests', 'JmbGoldmineDemoCanaryHarness.mq5'), 'utf8'), + ]) + const identity = gateway.match(/bool CanaryGatewayIdentityValid\([\s\S]*?\n\}/)?.[0] ?? '' + const binding = gateway.match(/bool CanaryGatewayBindingMatches\([\s\S]*?\n\}/)?.[0] ?? '' + + expect(identity).toContain('ACCOUNT_LOGIN') + expect(identity).toContain('ACCOUNT_SERVER') + expect(identity).toContain('_Symbol') + expect(binding).toContain('actual_mode==ACCOUNT_TRADE_MODE_DEMO') + expect(binding).toContain('CanaryAllowedServer(broker)') + expect(binding).toContain('CanaryAllowedMagic(broker)') + expect(gateway.match(/CanaryGatewayIdentityValid\(/g)).toHaveLength(4) + expect(main).toContain('mutation_identity_valid') + expect(main).toContain('identity_mismatch') + expect(harness).toContain('wrong demo account blocks emergency close') + expect(harness).toContain('account switch blocks reversal close') + }) + + it('reconciles authoritative Gold state by exact magic before any decision or resend', async () => { + const [main, reconcile] = await Promise.all([ + readCanarySource('JmbGoldmineDemoCanary.mq5'), + readCanarySource('JmbCanaryReconcile.mqh'), + ]) + + expect(main).toContain('#include "JmbCanaryReconcile.mqh"') + expect(main).toContain('g_reconciliation_dirty=true;') + expect(main).toMatch(/void OnTradeTransaction\([\s\S]*g_reconciliation_dirty=true;\s*return;\s*\}/) + expect(main).toMatch(/void Evaluate\([\s\S]*ReconcileCanaryBrokerState\([\s\S]*EvaluateCanaryGates\(/) + expect(reconcile).toContain('PositionGetString(POSITION_SYMBOL)!=symbol') + expect(reconcile).toContain('PositionGetInteger(POSITION_MAGIC)==magic_number') + expect(reconcile).toContain('OrderGetString(ORDER_SYMBOL)!=symbol') + expect(reconcile).toContain('OrderGetInteger(ORDER_MAGIC)==magic_number') + expect(reconcile).not.toMatch(/if\s*\(\s*PositionsTotal\(\)\s*>\s*0\s*\)/) + expect(main).not.toMatch(/\b(?:PositionsTotal|OrdersTotal|HistorySelect|HistoryDealGet)\s*\(/) + expect(main).not.toMatch(/\b(?:retry|resend)\b/i) + }) + + it('groups fully closed positions by broker day and complete net result', async () => { + const reconcile = await readCanarySource('JmbCanaryReconcile.mqh') + + expect(reconcile).toContain('DEAL_POSITION_ID') + expect(reconcile).toContain('HistorySelect(day_start,now)') + expect(reconcile).toContain('HistorySelectByPosition(position_id)') + expect(reconcile).toMatch(/double DealNet\(const ulong deal_ticket\)[\s\S]*DEAL_PROFIT[\s\S]*DEAL_COMMISSION[\s\S]*DEAL_SWAP[\s\S]*DEAL_FEE/) + expect(reconcile).toContain('DEAL_ENTRY_OUT_BY') + expect(reconcile).toContain('final_close_time') + expect(reconcile).toContain('daily.lossCount++') + expect(reconcile).toContain('daily.realizedLoss+=MathAbs(net_result)') + }) + + it('publishes the stable flushed event schema without exposing the raw login', async () => { + const [state, main] = await Promise.all([ + readCanarySource('JmbCanaryState.mqh'), + readCanarySource('JmbGoldmineDemoCanary.mq5'), + ]) + const stableFields = [ + 'schema_version', 'event_id', 'event_type', 'event_time', 'broker', 'server', 'account_mode', + 'account_identity_masked', 'symbol', 'strategy_version', 'magic_number', 'decision_id', + 'observation_id', 'gate_results', 'calculated_risk', 'requested_volume', 'requested_price', + 'requested_stop_loss', 'accepted_volume', 'accepted_price', 'accepted_stop_loss', 'result_code', + 'result_detail', 'order_ticket', 'deal_ticket', 'position_id', 'reconciliation_state', + 'daily_loss_count', 'daily_realized_loss', 'commission', 'swap', 'fee', 'net_result', + 'max_adverse_excursion', 'max_favorable_excursion', + ] + + for (const field of stableFields) expect(state).toContain(`\\"${field}\\"`) + expect(state).toContain('events.jsonl') + expect(state).toContain('FileFlush(handle)') + expect(state).toContain('CanaryMaskedAccountIdentity(') + expect(state).not.toMatch(/FileWrite(?:String)?\([^;]*ACCOUNT_LOGIN/s) + expect(main).not.toMatch(/(?:Print|Comment)\([^;]*ACCOUNT_LOGIN/s) + }) + + it('models protection, closures, durable no-ops, and persistent broker safety latches', async () => { + const [main, reconcile, state, harness] = await Promise.all([ + readCanarySource('JmbGoldmineDemoCanary.mq5'), + readCanarySource('JmbCanaryReconcile.mqh'), + readCanarySource('JmbCanaryState.mqh'), + readFile(join('tools', 'mt5', 'tests', 'JmbGoldmineDemoCanaryHarness.mq5'), 'utf8'), + ]) + + expect(reconcile).toContain('CANARY_LIFECYCLE_RECONCILIATION_REQUIRED') + expect(reconcile).toContain('CANARY_LIFECYCLE_FILLED_PROTECTED') + expect(reconcile).toContain('CANARY_LIFECYCLE_EMERGENCY_CLOSE') + expect(reconcile).toContain('CANARY_LIFECYCLE_STOPPED') + expect(main).toContain('SubmitCanaryReversalClose(') + expect(main).toContain('SubmitCanaryEmergencyClose(') + expect(main).toContain('PersistCanarySafetyLatch(') + expect(main).toContain('PersistCanaryAttempt(') + expect(state).toContain('reconciliation_latch.csv') + expect(state).toContain('protection_error') + expect(state).toContain('pending_close_decision_id') + expect(harness).toContain('rejected request') + expect(harness).toContain('unknown result') + expect(harness).toContain('partial fill') + expect(harness).toContain('filled with stop') + expect(harness).toContain('filled without stop') + expect(harness).toContain('stopped observation') + expect(harness).toContain('opposite signal close') + expect(harness).toContain('four losing positions') + expect(harness).toContain('server day reset') + expect(harness).toContain('restart with protected position') + expect(harness).toContain('restart with foreign exposure') + }) + + it('requires an exact actionable opposite and complete gate authorization before reversal close', async () => { + const [main, reconcile, harness] = await Promise.all([ + readCanarySource('JmbGoldmineDemoCanary.mq5'), + readCanarySource('JmbCanaryReconcile.mqh'), + readFile(join('tools', 'mt5', 'tests', 'JmbGoldmineDemoCanaryHarness.mq5'), 'utf8'), + ]) + + expect(reconcile).toMatch(/bool IsCanaryActionableOpposite\([\s\S]*decision_direction=="buy"[\s\S]*position_direction=="sell"[\s\S]*decision_direction=="sell"[\s\S]*position_direction=="buy"/) + expect(reconcile).toContain('facts.oppositeDirection=!observation_used') + expect(reconcile).toContain('IsCanaryActionableOpposite(decision.direction,reconciliation.position.direction)') + expect(main).toMatch(/CanaryEnvironment reversal_environment;[\s\S]*reversal_environment=environment;[\s\S]*reversal_environment\.hasEaPosition=false;[\s\S]*CanaryEvaluation reversal_evaluation=EvaluateCanaryGates\(decision,policy,reversal_environment\)/) + expect(main).toMatch(/reversal_evaluation\.ready[\s\S]*HandleCanaryOppositeClose\(/) + expect(harness).toContain('flat decision cannot close') + }) + + it('blocks every close request when authoritative ownership is foreign or unavailable', async () => { + const [main, reconcile, harness] = await Promise.all([ + readCanarySource('JmbGoldmineDemoCanary.mq5'), + readCanarySource('JmbCanaryReconcile.mqh'), + readFile(join('tools', 'mt5', 'tests', 'JmbGoldmineDemoCanaryHarness.mq5'), 'utf8'), + ]) + const reducer = reconcile.match(/CanaryLifecycleState ReduceCanaryLifecycle\([\s\S]*?\n\}/)?.[0] ?? '' + + expect(reducer.indexOf('!facts.brokerStateAvailable')).toBeGreaterThan(-1) + expect(reducer.indexOf('facts.hasForeignGoldExposure')).toBeGreaterThan(reducer.indexOf('!facts.brokerStateAvailable')) + expect(reducer.indexOf('facts.hasEaPosition && !facts.eaPositionProtected')).toBeGreaterThan(reducer.indexOf('facts.hasForeignGoldExposure')) + expect(main).toMatch(/reconciliation\.state==CANARY_LIFECYCLE_EMERGENCY_CLOSE[\s\S]*reconciliation\.available[\s\S]*!reconciliation\.hasForeignGoldExposure[\s\S]*HandleCanaryEmergencyClose/) + expect(harness).toContain('mixed magic unprotected position blocks close') + expect(harness).toContain('multiple positions block close') + }) + + it('counts EA-origin positions closed by a nonmagic final deal without claiming foreign-origin trades', async () => { + const [reconcile, harness] = await Promise.all([ + readCanarySource('JmbCanaryReconcile.mqh'), + readFile(join('tools', 'mt5', 'tests', 'JmbGoldmineDemoCanaryHarness.mq5'), 'utf8'), + ]) + + expect(reconcile).toContain('CanaryClosedOwnershipClass ClassifyCanaryClosedPositionOwnership(') + expect(reconcile).toContain('HistoryDealGetString(deal_ticket,DEAL_SYMBOL)!=symbol) continue;') + expect(reconcile).toMatch(/if\(entry!=DEAL_ENTRY_OUT && entry!=DEAL_ENTRY_OUT_BY\) continue;[\s\S]*DEAL_POSITION_ID/) + expect(reconcile).toContain('origin_magic==magic_number') + expect(reconcile).toContain('CANARY_CLOSED_OWNERSHIP_EA') + expect(reconcile).toContain('CANARY_CLOSED_OWNERSHIP_FOREIGN') + expect(reconcile).toContain('CANARY_CLOSED_OWNERSHIP_UNSAFE') + expect(harness).toContain('nonmagic final closure ownership') + }) + + it('resolves authoritative immediate stops and emergency closures without allowing resend', async () => { + const [main, reconcile, state, harness] = await Promise.all([ + readCanarySource('JmbGoldmineDemoCanary.mq5'), + readCanarySource('JmbCanaryReconcile.mqh'), + readCanarySource('JmbCanaryState.mqh'), + readFile(join('tools', 'mt5', 'tests', 'JmbGoldmineDemoCanaryHarness.mq5'), 'utf8'), + ]) + const reducer = reconcile.match(/CanaryLifecycleState ReduceCanaryLifecycle\([\s\S]*?\n\}/)?.[0] ?? '' + + expect(reducer.indexOf('facts.stoppedObservation')).toBeLessThan(reducer.indexOf('facts.resultClass==CANARY_RESULT_UNKNOWN')) + expect(reconcile).toContain('authoritative_stop_closure') + expect(reconcile).toContain('authoritative_emergency_closure') + expect(state).toContain('emergency_position_id') + expect(main).toContain('latch.emergencyPositionId=CanaryTicketString(reconciliation.position.identifier)') + expect(main).toMatch(/CANARY_LIFECYCLE_STOPPED[\s\S]*latch\.unresolved=false;[\s\S]*PersistCanarySafetyLatch/) + expect(main).toMatch(/authoritative_emergency_closure[\s\S]*ConfirmCanaryEmergencyClosure/) + expect(state).toMatch(/FinalizeCanaryEmergencyTerminalCorrelation[\s\S]*latch\.unresolved=false;/) + expect(main).toContain('latch.protectionError=true;') + expect(harness).toContain('correlated stopped observation') + expect(harness).toContain('unrelated stop remains unresolved') + expect(harness).toContain('protection error pauses after emergency closure') + }) + + it('publishes the in-memory latest event only after durable append verification succeeds', async () => { + const main = await readCanarySource('JmbGoldmineDemoCanary.mq5') + const managed = main.match(/bool AppendManagedCanaryEvent\([\s\S]*?\n\}/)?.[0] ?? '' + + expect(managed).toContain('CanaryExecutionEvent candidate;') + expect(managed).toContain('AppendCanaryExecutionEvent(candidate,detail)') + expect(managed.indexOf('g_latest_event=candidate;')).toBeGreaterThan(managed.indexOf('AppendCanaryExecutionEvent(candidate,detail)')) + expect(main).not.toMatch(/BuildCanaryLifecycleEvent\([^;]*g_latest_event\)/s) + expect(main).not.toMatch(/AppendCanaryExecutionEvent\(g_latest_event,/) + }) + + it('correlates an unresolved entry decision to its authoritative position before accepting a stop', async () => { + const [main, reconcile, state, harness] = await Promise.all([ + readCanarySource('JmbGoldmineDemoCanary.mq5'), + readCanarySource('JmbCanaryReconcile.mqh'), + readCanarySource('JmbCanaryState.mqh'), + readFile(join('tools', 'mt5', 'tests', 'JmbGoldmineDemoCanaryHarness.mq5'), 'utf8'), + ]) + + expect(state).toContain('pending_entry_decision_id') + expect(state).toContain('pending_entry_observation_id') + expect(state).toContain('pending_entry_attempted_at') + expect(main).toMatch(/latch\.pendingEntryDecisionId=decision\.decisionId[\s\S]*PersistCanarySafetyLatch\([\s\S]*SubmitProtectedMarketOrder/) + expect(reconcile).toContain('CanaryEntryCorrelationComment(latch.pendingEntryDecisionId)') + expect(reconcile).toContain('IsCanaryCorrelatedLifecyclePosition(') + expect(reconcile).toMatch(/facts\.stoppedObservation=[^;]*correlated_entry_position[^;]*lastCloseWasStop/s) + expect(harness).toContain('correlated stopped observation') + expect(harness).toContain('unrelated stop remains unresolved') + }) + + it('persists opening correlation through protected fill, restart, stop, and reversal closure', async () => { + const [main, reconcile, state, harness] = await Promise.all([ + readCanarySource('JmbGoldmineDemoCanary.mq5'), + readCanarySource('JmbCanaryReconcile.mqh'), + readCanarySource('JmbCanaryState.mqh'), + readFile(join('tools', 'mt5', 'tests', 'JmbGoldmineDemoCanaryHarness.mq5'), 'utf8'), + ]) + + for (const field of [ + 'active_position_decision_id', 'active_position_observation_id', 'active_position_id', + 'active_requested_volume', 'active_requested_price', 'active_requested_stop_loss', + 'active_calculated_risk', 'active_entry_comment', + ]) expect(state).toContain(field) + expect(main).toContain('ActivateCanaryPositionCorrelation(') + expect(reconcile).toContain('latch.activePositionDecisionId') + expect(main).toContain('AppendManagedCanaryPositionEvent(') + expect(main).toMatch(/ConfirmCanaryOppositeClosure[\s\S]*AppendManagedCanaryPositionEvent[\s\S]*ClearCanaryActivePositionCorrelation/) + expect(main).toMatch(/CANARY_LIFECYCLE_STOPPED[\s\S]*AppendManagedCanaryPositionEvent[\s\S]*ClearCanaryActivePositionCorrelation/) + expect(harness).toContain('protected fill then later stop after restart') + expect(harness).toContain('opening decision differs from reversal decision') + }) + + it('durably publishes one opening-correlated emergency terminal before clearing its restart latch', async () => { + const [main, state, harness] = await Promise.all([ + readCanarySource('JmbGoldmineDemoCanary.mq5'), + readCanarySource('JmbCanaryState.mqh'), + readFile(join('tools', 'mt5', 'tests', 'JmbGoldmineDemoCanaryHarness.mq5'), 'utf8'), + ]) + const emergencyConfirmation = main.match(/bool ConfirmCanaryEmergencyClosure\([\s\S]*?\n\}/)?.[0] ?? '' + const terminalRecorded = emergencyConfirmation.indexOf('CanaryCorrelatedTerminalEventRecorded(') + const terminalAppend = emergencyConfirmation.indexOf('AppendManagedCanaryPositionEvent(') + const terminalFinalize = emergencyConfirmation.indexOf('FinalizeCanaryEmergencyTerminalCorrelation(') + + expect(emergencyConfirmation).toContain('terminal_reconciliation.reconciliationState="reconciled"') + expect(emergencyConfirmation).toContain('CANARY_LIFECYCLE_CLOSED') + expect(terminalRecorded).toBeGreaterThan(-1) + expect(terminalAppend).toBeGreaterThan(terminalRecorded) + expect(terminalFinalize).toBeGreaterThan(terminalAppend) + expect(emergencyConfirmation).toContain('if(any_terminal_event_durable && !terminal_event_durable) return false;') + expect(state).toMatch(/FinalizeCanaryEmergencyTerminalCorrelation[\s\S]*if\(!terminal_event_durable\) return false;[\s\S]*ClearCanaryActivePositionCorrelation/) + for (const field of [ + 'decision_id', 'observation_id', 'position_id', 'calculated_risk', + 'requested_volume', 'requested_price', 'requested_stop_loss', + 'commission', 'swap', 'fee', 'net_result', + ]) expect(state).toMatch(new RegExp(`CanaryCorrelatedTerminalEventRecorded[\\s\\S]*${field}`)) + expect(main).toMatch(/authoritative_emergency_closure && \(latch\.unresolved[\s\S]*latch\.activePositionDecisionId!=""[\s\S]*ConfirmCanaryEmergencyClosure/) + expect(main).toMatch(/CANARY_LIFECYCLE_PAUSED[\s\S]*!authoritative_emergency_closure/) + expect(harness).toContain('emergency terminal waits for durable event') + expect(harness).toContain('emergency terminal clears active correlation after restart') + expect(harness).toContain('emergency protection pause remains') + }) + + it('recovers a correlated stop or emergency closure independently of broker-day loss selection', async () => { + const [reconcile, harness] = await Promise.all([ + readCanarySource('JmbCanaryReconcile.mqh'), + readFile(join('tools', 'mt5', 'tests', 'JmbGoldmineDemoCanaryHarness.mq5'), 'utf8'), + ]) + const lifecycleRecovery = reconcile.match(/bool ReadCanaryLifecyclePositionById\([\s\S]*?\n\}/)?.[0] ?? '' + + expect(lifecycleRecovery).toContain('HistorySelectByPosition(position_id)') + expect(lifecycleRecovery).not.toContain('day_start') + expect(reconcile).toContain('latch.emergencyPositionId') + expect(reconcile).toContain('latch.pendingEntryAttemptedAt') + expect(harness).toContain('position-specific rollover recovery') + }) + + it('projects persistent protection pauses distinctly from broker-day loss pauses', async () => { + const [main, reconcile, state, harness] = await Promise.all([ + readCanarySource('JmbGoldmineDemoCanary.mq5'), + readCanarySource('JmbCanaryReconcile.mqh'), + readCanarySource('JmbCanaryState.mqh'), + readFile(join('tools', 'mt5', 'tests', 'JmbGoldmineDemoCanaryHarness.mq5'), 'utf8'), + ]) + + expect(reconcile).toMatch(/CANARY_LIFECYCLE_PAUSED[\s\S]*facts\.persistentSafetyPause[\s\S]*reconciliationState="protection_error"/) + expect(state).toMatch(/reconciliation\.reconciliationState=="protection_error"[\s\S]*"protection"/) + expect(state).toContain('Resolve the persistent broker protection error before operator clearance.') + expect(main).toContain('Persistent broker protection error keeps this canary paused pending operator clearance.') + expect(harness).toContain('persistent protection status semantics') + }) + + it('keeps execution disabled, demo-only, Gold-only, fixed-volume, and non-expanding', async () => { + const [sources, gateway, types] = await Promise.all([ + readCanarySources(), + readCanarySource('JmbCanaryTradeGateway.mqh'), + readCanarySource('JmbCanaryTypes.mqh'), + ]) + const source = sources.map((file) => file.source).join('\n') + + expect(source).toContain('input bool InpDemoExecutionEnabled = false;') + expect(source).toContain('input bool InpKillSwitch = true;') + for (const file of sources) { + expect(file.source, `${file.path} must not add a live-mode input`).not.toMatch(/input[^;]*(?:live|real)/i) + expect(file.source, `${file.path} must remain Gold-only and non-expanding`).not.toMatch( + /EURUSD|martingale|grid|recovery|take.?profit|\b(?:tp|take_profit)\s*=|lot.?growth|volume.?growth|scale.?in|pyramid/i, + ) + for (const match of file.source.matchAll(/\b(?:volume|maxVolume|max_volume)\s*=\s*(\d+(?:\.\d+)?)/g)) { + expect(Number(match[1]), `${file.path} must not assign volume above 0.01`).toBeLessThanOrEqual(0.01) + } + for (const match of file.source.matchAll(/\b(?:volume|maxVolume|max_volume)\s*=(?!=)\s*([^;\r\n]+)/g)) { + expect(match[1], `${file.path} must not scale a volume assignment`).not.toMatch(/[*/+-]/) + } + } + expect(types).toMatch(/const double CANARY_HARD_MAX_VOLUME\s*=\s*0\.01;/) + expect(gateway).toContain('ACCOUNT_TRADE_MODE_DEMO') + expect(gateway).toContain('decision.symbol!="XAUUSD"') + expect(gateway).toContain('decision.volume!=CANARY_HARD_MAX_VOLUME') + expect([...gateway.matchAll(/request\.volume\s*=\s*([^;]+);/g)].map((match) => match[1].trim())) + .toEqual(['CANARY_HARD_MAX_VOLUME']) + expect(gateway).not.toMatch(/(?:request\.volume\s*(?:\*=|\/=|\+=|-=)|(?:\+\+|--)request\.volume|request\.volume(?:\+\+|--))/) + expect(gateway).toContain('request.sl=decision.stopLoss;') + expect(gateway).toMatch(/policy\.magicNumber!=880101\s*&&\s*policy\.magicNumber!=880201/) + }) + + it('persists requesting and attempt barriers before one submit then defers every result to reconciliation', async () => { + const [main, state] = await Promise.all([ + readCanarySource('JmbGoldmineDemoCanary.mq5'), + readCanarySource('JmbCanaryState.mqh'), + ]) + const submissionFlow = main.match(/void SubmitReadyCanaryDecision\([\s\S]*?\n\}/)?.[0] ?? '' + const eventBarrier = submissionFlow.indexOf('AppendCanaryOrderRequestingEvent(') + const attemptBarrier = submissionFlow.indexOf('PersistCanaryAttempt(') + const submit = submissionFlow.indexOf('SubmitProtectedMarketOrder(') + + expect(main).toContain('#include "JmbCanaryTradeGateway.mqh"') + expect(main).toContain('evaluation.state!=CANARY_LIFECYCLE_READY') + expect(main).toContain('!InpDemoExecutionEnabled || InpKillSwitch') + expect(main).toContain('if(!status_persisted)') + expect(main).toContain('CanaryProcessedStateContains(processed_state,decision.decisionId,decision.observationId)') + expect(eventBarrier).toBeGreaterThan(-1) + expect(attemptBarrier).toBeGreaterThan(eventBarrier) + expect(submit).toBeGreaterThan(attemptBarrier) + expect(main).toContain('AppendCanaryOrderRequestingEvent(') + expect(state).toMatch(/AppendCanaryExecutionEvent\([\s\S]*FileFlush\(handle\)[\s\S]*ReadCanaryCommonText/) + expect(main).toMatch(/PersistCanaryAttempt\([\s\S]*FileFlush\(handle\)[\s\S]*LoadCanaryProcessedState/) + expect(main).toContain('processed_observations.lock') + expect(main).toMatch(/PersistCanaryAttempt\([\s\S]*FileOpen\(lock_path[\s\S]*LoadCanaryProcessedState\(path,locked_state/) + expect(main.match(/SubmitProtectedMarketOrder\s*\(/g)).toHaveLength(1) + expect(main).toContain('g_reconciliation_dirty=true;') + expect(main).toContain('environment.reconciliationComplete=!g_reconciliation_dirty') + expect(main).toContain('void OnTradeTransaction(') + expect(main).not.toMatch(/filled_protected/) + expect(main).not.toMatch(/\b(?:retry|resend)\b/i) + }) + + it('rejects durable replacements that drop, mutate, reorder, duplicate, or truncate prior attempts', async () => { + const main = await readCanarySource('JmbGoldmineDemoCanary.mq5') + const submissionFlow = main.match(/void SubmitReadyCanaryDecision\([\s\S]*?\n\}/)?.[0] ?? '' + + expect(main).toContain('bool CanaryProcessedStateIsExactAppend(') + expect(main).toMatch(/CanaryProcessedStateIsExactAppend\([\s\S]*candidate_count!=prior_count\+1/) + expect(main).toMatch(/CanaryProcessedStateIsExactAppend\([\s\S]*candidate\.decisionIds\[index\]!=prior\.decisionIds\[index\][\s\S]*candidate\.observationIds\[index\]!=prior\.observationIds\[index\][\s\S]*candidate\.attemptedAt\[index\]!=prior\.attemptedAt\[index\]/) + expect(main).toMatch(/candidate\.decisionIds\[prior_count\]==decision\.decisionId[\s\S]*candidate\.observationIds\[prior_count\]==decision\.observationId[\s\S]*candidate\.attemptedAt\[prior_count\]==attempted_at/) + expect(main).toContain('CanaryProcessedStateIsExactAppend(locked_state,temporary_state,decision,attempted_at)') + expect(main).toContain('CanaryProcessedStateIsExactAppend(locked_state,durable_state,decision,attempted_at)') + expect(submissionFlow).toMatch(/if\(!PersistCanaryAttempt\([\s\S]*?\n\s*return;\n\s*\}[\s\S]*TradeSubmitResult submission=SubmitProtectedMarketOrder/) + }) + + it('keeps risk evaluation pure while retaining broker-side calculation evidence', async () => { + const source = await readCanaryBundle() + + expect(source).toContain('CanaryEvaluation EvaluateCanaryGates(') + expect(source).toContain('OrderCalcProfit(') + expect(source).toContain('environment.calculatedStopRisk') + expect(source).toContain('environment.dailyRealizedLoss { + const source = await readCanaryBundle() + + expect(source).toContain('HFMarketsGlobal-Demo4') + expect(source).toContain('ICMarketsSC-Demo') + expect(source).toContain('daily-trend-v1') + expect(source).toContain('880101') + expect(source).toContain('880201') + expect(source).toContain('schema_version,decision_id,observation_id,observation_as_of,created_at,lease_issued_at,lease_expires_at,broker,server,account_mode,symbol,strategy_version,direction,entry_reference_price,volume,stop_loss,max_risk_amount,candidate_policy_version,cost_model_version,gate_results_json') + expect(source).toContain('schema_version,captured_at,broker,server,account_mode,symbol,state,detail,rollout_stage,execution_enabled,kill_switch,decision_id,observation_id,event_id,event_type,event_time,result_code,result_detail,stop_protection_confirmed,position_direction,position_volume,position_open_price,position_stop_loss,position_id,reconciliation_state,daily_loss_count,daily_realized_loss,blocking_gate,next_safe_action') + expect(source).not.toMatch(/FileWrite\([^;]*InpExpectedAccountLogin/s) + expect(source).not.toMatch(/(?:Print|Comment)\([^;]*InpExpectedAccountLogin/s) + }) + + it('strictly validates policy grammar, gate JSON, and Task 3 hash identities', async () => { + const [csv, policy, harness] = await Promise.all([ + readCanarySource('JmbCanaryCsv.mqh'), + readCanarySource('JmbCanaryPolicy.mqh'), + readFile(join('tools', 'mt5', 'tests', 'JmbGoldmineDemoCanaryHarness.mq5'), 'utf8'), + ]) + + expect(policy).toContain('bool IsCanonicalCanaryPolicyVersion(') + expect(policy).toContain('StringFind(value,",")') + expect(policy).toContain('StringFind(value,"\\\"")') + expect(csv).toContain('bool ParseCanaryGateResultsJson(') + expect(csv).toContain('CRYPT_HASH_SHA256') + expect(csv).toContain('CreateCanaryObservationId(') + expect(csv).toContain('CreateCanaryDecisionId(') + expect(csv).toContain('decision.observationId!=CreateCanaryObservationId(decision)') + expect(csv).toContain('decision.decisionId!=CreateCanaryDecisionId(decision)') + expect(harness).toContain('arbitrary decision ids') + expect(harness).toContain('malformed gate JSON') + expect(harness).toContain('unknown gate name') + expect(harness).toContain('unknown gate state') + expect(harness).toContain('unknown gate field') + expect(harness).toContain('duplicate gate field') + expect(harness).toContain('missing gate field') + expect(harness).toContain('noncanonical gate evidence') + expect(harness).toContain('policy version grammar') + }) + + it('exports Task 5-compatible effective enablement and loads durable attempt state', async () => { + const [main, state, harness] = await Promise.all([ + readCanarySource('JmbGoldmineDemoCanary.mq5'), + readCanarySource('JmbCanaryState.mqh'), + readFile(join('tools', 'mt5', 'tests', 'JmbGoldmineDemoCanaryHarness.mq5'), 'utf8'), + ]) + + expect(state).toContain('schema_version,decision_id,observation_id,attempted_at') + expect(state).toContain('processed_observations.csv') + expect(state).toContain('bool LoadCanaryProcessedState(') + expect(state).toContain('CanarySha256Identity("daily-trend-v1|"+values[2])') + expect(main).toContain('LoadCanaryProcessedState(') + expect(main).toContain('bool effective_execution_enabled=') + expect(state).toContain('policy.rolloutStage!="status_only"') + expect(state).toContain('if(policy.rolloutStage=="status_only" && execution_enabled) return false;') + expect(main).not.toMatch(/WriteCanaryLatestStatus\([^;]*InpDemoExecutionEnabled/s) + expect(harness).toContain('loaded duplicate state') + expect(harness).toContain('malformed processed state') + }) + + it('requires broker-valid stop metadata, checked writes, and a working timer', async () => { + const [main, state, harness] = await Promise.all([ + readCanarySource('JmbGoldmineDemoCanary.mq5'), + readCanarySource('JmbCanaryState.mqh'), + readFile(join('tools', 'mt5', 'tests', 'JmbGoldmineDemoCanaryHarness.mq5'), 'utf8'), + ]) + + expect(main).toContain('SYMBOL_ORDER_MODE') + expect(main).toContain('SYMBOL_ORDER_SL') + expect(main).toContain('SYMBOL_TRADE_TICK_SIZE') + expect(main).toContain('if(!EventSetTimer(10))') + expect(state).toContain('uint preflight_written=FileWriteString(handle,payload)') + expect(state).toContain('if(preflight_written!=StringLen(payload))') + expect(state).toContain('uint header_written=FileWrite(') + expect(state).toContain('uint row_written=FileWrite(') + expect(state).toContain('ReadStrictCanaryCsv(temporary_path') + expect(harness).toContain('stop mode unsupported') + expect(harness).toContain('stop tick unavailable') + expect(harness).toContain('stop tick misaligned') + }) + + it('rejects physical policy quotes without weakening quoted decision CSV', async () => { + const [csv, policy, harness] = await Promise.all([ + readCanarySource('JmbCanaryCsv.mqh'), + readCanarySource('JmbCanaryPolicy.mqh'), + readFile(join('tools', 'mt5', 'tests', 'JmbGoldmineDemoCanaryHarness.mq5'), 'utf8'), + ]) + + expect(csv).toContain('bool ReadStrictCanaryCsvText(') + expect(csv).toMatch(/bool ReadCanaryDecision\([\s\S]*ReadStrictCanaryCsv\(path,expected,values,detail\)/) + expect(policy).toContain('bool ParseCanaryPolicyCsvText(') + expect(policy).toContain('StringFind(policy_text,"\\\"")>=0') + expect(policy).toContain('ReadStrictCanaryCsvText(policy_text') + expect(harness).toContain('fully quoted policy row') + expect(harness).toContain('partially quoted policy row') + }) + + it('verifies exact reopened preflight and all intended status fields', async () => { + const [state, harness] = await Promise.all([ + readCanarySource('JmbCanaryState.mqh'), + readFile(join('tools', 'mt5', 'tests', 'JmbGoldmineDemoCanaryHarness.mq5'), 'utf8'), + ]) + + expect(state).toContain('const string payload="openalice-canary-preflight\\r\\n";') + expect(state).toContain('ReadCanaryCommonText(path,reopened,read_detail)') + expect(state).toContain('reopened!=payload') + expect(state).toContain('bool CanaryExactValuesMatch(') + expect(state).toContain('CanaryExactValuesMatch(intended_values,verified_values)') + expect(harness).toContain('truncated next safe action') + }) + + it('decodes canonical JSON escapes and rejects malformed Unicode sequences', async () => { + const [csv, harness] = await Promise.all([ + readCanarySource('JmbCanaryCsv.mqh'), + readFile(join('tools', 'mt5', 'tests', 'JmbGoldmineDemoCanaryHarness.mq5'), 'utf8'), + ]) + + expect(csv).toContain('bool ParseCanaryUnicodeEscape(') + expect(csv).toContain('escaped=="b"') + expect(csv).toContain('escaped=="f"') + expect(csv).toContain('escaped=="n"') + expect(csv).toContain('escaped=="r"') + expect(csv).toContain('escaped=="t"') + expect(csv).toContain('escaped=="/"') + expect(harness).toContain('canonical JSON escapes') + expect(harness).toContain('invalid JSON escapes') + expect(harness).toContain('invalid surrogate pairs') + }) +}) diff --git a/src/domain/mt5/demo-decision-engine.spec.ts b/src/domain/mt5/demo-decision-engine.spec.ts new file mode 100644 index 000000000..c6cbd0a85 --- /dev/null +++ b/src/domain/mt5/demo-decision-engine.spec.ts @@ -0,0 +1,160 @@ +import { describe, expect, it } from 'vitest' + +import type { BrokerCostModel } from './broker-cost-model.js' +import type { DemoExecutionPolicy, DemoExecutionPolicySummary } from './demo-execution-policy.js' +import { buildDemoExecutionDecision, type BuildDemoExecutionDecisionInput } from './demo-decision-engine.js' + +const policy: DemoExecutionPolicy = { + schemaVersion: 1, + policyVersion: 'hfm-canary-v1', + broker: 'hfmarkets', + server: 'HFMarketsGlobal-Demo4', + symbol: 'XAUUSD', + strategyVersion: 'daily-trend-v1', + rolloutStage: 'hfm_canary', + candidateApproved: true, + completedObservationMaxAgeHours: 72, + maxSpread: 0.75, + maxDeviation: 0.5, + maxRiskAmount: 10, + maxDailyLoss: 40, + maxDailyLosingTrades: 4, + maxVolume: 0.01, + magicNumber: 880101, +} + +const costModel: BrokerCostModel = { + schemaVersion: 1, + version: 'hfm-cost-0900', + broker: 'hfmarkets', + server: 'HFMarketsGlobal-Demo4', + symbol: 'XAUUSD', + state: 'canary_ready', + observedFrom: '2026-07-12T10:00:00.000Z', + observedTo: '2026-07-13T09:59:00.000Z', + expiresAt: '2026-07-14T10:00:00.000Z', + spreadSampleCount: 100, + observedMaxSpread: 0.3, + configuredMaxSpread: 0.75, + configuredMaxDeviation: 0.5, + commissionObserved: true, + swapObserved: true, + contractFingerprint: 'contract-fingerprint', + evidence: ['complete'], +} + +function readyInput(overrides: Partial = {}): BuildDemoExecutionDecisionInput { + return { + createdAt: '2026-07-13T10:00:00.000Z', + leaseIssuedAt: '2026-07-13T10:00:00.000Z', + leaseExpiresAt: '2026-07-13T10:10:00.000Z', + broker: 'hfmarkets', + server: 'HFMarketsGlobal-Demo4', + symbol: 'XAUUSD', + bridge: { + state: 'ready', + broker: 'hfmarkets', + server: 'HFMarketsGlobal-Demo4', + symbol: 'XAUUSD', + }, + completedObservation: { + state: 'ready', + detail: 'Fresh completed D1 evidence.', + observation: { + asOf: '2026-07-12', + direction: 'uptrend', + lookbackReturn: 0.02, + lookbackDays: 120, + latestClose: 2400, + referenceClose: 2352.94, + }, + }, + policy: { state: 'ready', detail: 'Policy approved.', policy }, + costModel, + learning: { + state: 'learning', + accountMode: 'demo', + server: 'HFMarketsGlobal-Demo4', + }, + quote: { bid: 2399.9, ask: 2400.1, spread: 0.2 }, + stopLoss: 2392.1, + ...overrides, + } +} + +describe('demo execution decision engine', () => { + it('creates an executable HFM Gold lease when every hard gate passes', () => { + const result = buildDemoExecutionDecision(readyInput()) + expect(result.state).toBe('ready') + expect(result.decision).toMatchObject({ + broker: 'hfmarkets', + symbol: 'XAUUSD', + direction: 'buy', + entryReferencePrice: 2400.1, + stopLoss: 2392.1, + volume: 0.01, + maxRiskAmount: 10, + }) + expect(result.decision?.gateResults.map((gate) => gate.name)).toEqual([ + 'bridge', + 'completed_observation', + 'candidate_policy', + 'cost_model', + 'learning', + 'quote', + 'spread', + 'direction', + 'stop_loss', + ]) + expect(result.decision?.gateResults.every((gate) => gate.state === 'pass')).toBe(true) + }) + + it('blocks IC Gold while policy remains at the HFM canary stage', () => { + const icPolicy = { + ...policy, + policyVersion: 'ic-hfm-stage-v1', + broker: 'icmarkets', + server: 'ICMarketsSC-Demo', + rolloutStage: 'hfm_canary', + maxSpread: 0.3, + maxDeviation: 0.3, + magicNumber: 880201, + } as DemoExecutionPolicy + const policySummary: DemoExecutionPolicySummary = { + state: 'blocked', + detail: 'The rollout stage does not authorize this broker.', + policy: icPolicy, + } + const result = buildDemoExecutionDecision(readyInput({ + broker: 'icmarkets', + server: 'ICMarketsSC-Demo', + bridge: { state: 'ready', broker: 'icmarkets', server: 'ICMarketsSC-Demo', symbol: 'XAUUSD' }, + policy: policySummary, + costModel: { ...costModel, version: 'ic-cost-0900', broker: 'icmarkets', server: 'ICMarketsSC-Demo', configuredMaxSpread: 0.3, configuredMaxDeviation: 0.3 }, + learning: { state: 'learning', accountMode: 'demo', server: 'ICMarketsSC-Demo' }, + })) + expect(result.state).toBe('blocked') + expect(result.decision?.gateResults).toContainEqual(expect.objectContaining({ name: 'candidate_policy', state: 'block' })) + }) + + it('blocks EURUSD without creating a persistable execution decision', () => { + const result = buildDemoExecutionDecision(readyInput({ + symbol: 'EURUSD', + bridge: { state: 'ready', broker: 'hfmarkets', server: 'HFMarketsGlobal-Demo4', symbol: 'EURUSD' }, + })) + expect(result).toMatchObject({ state: 'blocked', decision: null }) + expect(result.detail).toMatch(/EURUSD.*shadow/i) + }) + + it.each([ + ['stale completed observation', { completedObservation: { ...readyInput().completedObservation, state: 'stale' } }, 'completed_observation'], + ['blocked cost model', { costModel: { ...costModel, state: 'blocked' } }, 'cost_model'], + ['missing stop', { stopLoss: null }, 'stop_loss'], + ['spread breach', { quote: { bid: 2399, ask: 2400, spread: 1 } }, 'spread'], + ['flat direction', { completedObservation: { ...readyInput().completedObservation, observation: { ...readyInput().completedObservation.observation!, direction: 'flat' } } }, 'direction'], + ] as const)('never returns ready for %s', (_name, override, blockedGate) => { + const result = buildDemoExecutionDecision(readyInput(override as Partial)) + expect(result.state).toBe('blocked') + expect(result.decision?.gateResults).toContainEqual(expect.objectContaining({ name: blockedGate, state: 'block' })) + }) +}) diff --git a/src/domain/mt5/demo-decision-engine.ts b/src/domain/mt5/demo-decision-engine.ts new file mode 100644 index 000000000..f5a7fcb8a --- /dev/null +++ b/src/domain/mt5/demo-decision-engine.ts @@ -0,0 +1,168 @@ +import type { BrokerCostModel } from './broker-cost-model.js' +import type { CompletedD1State, CompletedTrendObservation } from './completed-d1.js' +import type { DemoExecutionPolicySummary } from './demo-execution-policy.js' +import { + createExecutionDecisionId, + createObservationId, + type JmbExecutionDecision, + type JmbGateResult, +} from './execution-decision.js' +import type { Mt5BridgeState } from './read-only-bridge.js' +import type { Mt5TradeLedgerState } from './trade-ledger.js' + +type JmbBroker = JmbExecutionDecision['broker'] +type JmbServer = JmbExecutionDecision['server'] + +export interface BuildDemoExecutionDecisionInput { + createdAt: string + leaseIssuedAt: string + leaseExpiresAt: string + broker: JmbBroker + server: JmbServer + symbol: 'XAUUSD' | 'EURUSD' + bridge: { + state: Mt5BridgeState + broker: string + server: string | null + symbol: string + } + completedObservation: { + state: CompletedD1State + detail: string + observation: CompletedTrendObservation | null + } + policy: DemoExecutionPolicySummary + costModel: BrokerCostModel + learning: { + state: Mt5TradeLedgerState + accountMode: string | null + server: string | null + } + quote: { + bid: number | null + ask: number | null + spread: number | null + } + stopLoss: number | null +} + +export interface DemoExecutionDecisionBuildResult { + state: 'ready' | 'blocked' + decision: JmbExecutionDecision | null + detail: string +} + +function gate(name: string, passed: boolean, passDetail: string, blockDetail: string): JmbGateResult { + return { name, state: passed ? 'pass' : 'block', detail: passed ? passDetail : blockDetail } +} + +function finitePositive(value: number | null): value is number { + return value !== null && Number.isFinite(value) && value > 0 +} + +function finiteAtLeastZero(value: number | null): value is number { + return value !== null && Number.isFinite(value) && value >= 0 +} + +function expectedDirection(observation: CompletedTrendObservation): JmbExecutionDecision['direction'] { + return observation.direction === 'uptrend' ? 'buy' : observation.direction === 'downtrend' ? 'sell' : 'flat' +} + +export function buildDemoExecutionDecision(input: BuildDemoExecutionDecisionInput): DemoExecutionDecisionBuildResult { + if (input.symbol === 'EURUSD') { + return { + state: 'blocked', + decision: null, + detail: 'EURUSD remains shadow-only and cannot produce an execution-decision lease.', + } + } + + const observation = input.completedObservation.observation + const policy = input.policy.policy + if (observation === null || policy === null) { + return { + state: 'blocked', + decision: null, + detail: observation === null + ? `No persistable Gold decision was built: ${input.completedObservation.detail}` + : `No persistable Gold decision was built: ${input.policy.detail}`, + } + } + + const direction = expectedDirection(observation) + const entryReferencePrice = direction === 'buy' ? input.quote.ask : direction === 'sell' ? input.quote.bid : null + const bridgeReady = input.bridge.state === 'ready' + && input.bridge.broker === input.broker + && input.bridge.server === input.server + && input.bridge.symbol === 'XAUUSD' + const completedReady = input.completedObservation.state === 'ready' + const policyReady = input.policy.state === 'ready' + && policy.broker === input.broker + && policy.server === input.server + && policy.symbol === 'XAUUSD' + && policy.strategyVersion === 'daily-trend-v1' + && policy.candidateApproved + const costReady = input.costModel.state === 'canary_ready' + && input.costModel.broker === input.broker + && input.costModel.server === input.server + && input.costModel.symbol === 'XAUUSD' + && Date.parse(input.costModel.expiresAt) >= Date.parse(input.leaseIssuedAt) + const learningReady = input.learning.state === 'learning' + && input.learning.accountMode === 'demo' + && input.learning.server === input.server + const quoteReady = finitePositive(input.quote.bid) + && finitePositive(input.quote.ask) + && input.quote.ask >= input.quote.bid + && finiteAtLeastZero(input.quote.spread) + const spreadReady = finiteAtLeastZero(input.quote.spread) && input.quote.spread <= policy.maxSpread + const directionReady = direction !== 'flat' + const stopReady = finitePositive(input.stopLoss) + && finitePositive(entryReferencePrice) + && (direction === 'buy' ? input.stopLoss < entryReferencePrice : direction === 'sell' && input.stopLoss > entryReferencePrice) + + const gateResults: JmbGateResult[] = [ + gate('bridge', bridgeReady, 'The read-only demo bridge identity is current.', 'The read-only demo bridge is not ready or its identity does not match.'), + gate('completed_observation', completedReady, input.completedObservation.detail, input.completedObservation.detail), + gate('candidate_policy', policyReady, 'The operator policy approves this broker at the current rollout stage.', input.policy.detail), + gate('cost_model', costReady, 'The observed broker cost model is canary-ready.', 'The observed broker cost model is blocked, expired, or mismatched.'), + gate('learning', learningReady, 'Fresh demo trade history is available for learning.', 'Demo trade history is missing, stale, blocked, or mismatched.'), + gate('quote', quoteReady, 'The broker quote is finite and internally ordered.', 'The broker quote is missing or invalid.'), + gate('spread', spreadReady, 'The current spread is within the operator policy ceiling.', 'The current spread exceeds the operator policy ceiling.'), + gate('direction', directionReady, 'The completed D1 observation has an actionable trend.', 'A flat completed D1 observation cannot produce an entry.'), + gate('stop_loss', stopReady, 'The stop loss is present on the protective side of the quote.', 'The stop loss is missing, invalid, or on the wrong side of the quote.'), + ] + + const observationIdentity = { + broker: input.broker, + symbol: 'XAUUSD' as const, + strategyVersion: 'daily-trend-v1' as const, + observationAsOf: observation.asOf, + } + const observationId = createObservationId(observationIdentity) + const decision: JmbExecutionDecision = { + schemaVersion: 1, + decisionId: createExecutionDecisionId({ observationId }), + observationId, + observationAsOf: observation.asOf, + createdAt: input.createdAt, + leaseIssuedAt: input.leaseIssuedAt, + leaseExpiresAt: input.leaseExpiresAt, + broker: input.broker, + server: input.server, + accountMode: 'demo', + symbol: 'XAUUSD', + strategyVersion: 'daily-trend-v1', + direction, + entryReferencePrice: finitePositive(entryReferencePrice) ? entryReferencePrice : null, + volume: 0.01, + stopLoss: input.stopLoss, + maxRiskAmount: policy.maxRiskAmount, + candidatePolicyVersion: policy.policyVersion, + costModelVersion: input.costModel.version, + gateResults, + } + const blockingGate = gateResults.find((result) => result.state === 'block') + return blockingGate === undefined + ? { state: 'ready', decision, detail: 'All application-side demo execution gates passed.' } + : { state: 'blocked', decision, detail: `${blockingGate.name}: ${blockingGate.detail}` } +} diff --git a/src/domain/mt5/demo-decision-service.spec.ts b/src/domain/mt5/demo-decision-service.spec.ts new file mode 100644 index 000000000..114cf4d01 --- /dev/null +++ b/src/domain/mt5/demo-decision-service.spec.ts @@ -0,0 +1,196 @@ +import { afterEach, describe, expect, it } from 'vitest' +import { mkdir, mkdtemp, readFile, rm, stat, utimes, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' + +import { parseExecutionDecisionCsv } from './execution-decision.js' +import { + DEFAULT_JMB_DEMO_INSTRUMENTS, + runDemoDecisionCycle, + type JmbDemoInstrumentConfig, +} from './demo-decision-service.js' +import type { JmbMt5Roots } from './local-paths.js' + +const now = new Date('2026-07-13T10:00:00.000Z') +const directories: string[] = [] + +afterEach(async () => { + await Promise.all(directories.splice(0).map((directory) => rm(directory, { recursive: true, force: true }))) +}) + +async function roots(): Promise { + const root = await mkdtemp(join(tmpdir(), 'openalice-demo-cycle-')) + directories.push(root) + return { + bridgeRoot: join(root, 'OpenAliceMt5BridgeV1'), + ledgerRoot: join(root, 'OpenAliceMt5TradeLedgerV1'), + policyRoot: join(root, 'OpenAliceMt5DemoPolicyV1'), + costModelRoot: join(root, 'OpenAliceMt5CostModelV1'), + executionDecisionRoot: join(root, 'OpenAliceMt5ExecutionDecisionV1'), + executionRoot: join(root, 'OpenAliceMt5ExecutionV1'), + researchRoot: join(root, 'research'), + } +} + +function statusCsv(broker: string, server: string, symbol = 'XAUUSD'): string { + return [ + 'captured_at,broker,symbol,bridge_mode,account_mode,server,terminal_connected,trade_allowed,trade_expert,symbol_trade_mode,bid,ask,spread_price,tick_time,contract_size,volume_min,volume_max,volume_step,stops_level,open_positions,open_orders', + `2026-07-13T09:59:30.000Z,${broker},${symbol},read_only,demo,${server},1,1,1,4,2399.9,2400.1,0.2,2026-07-13T09:59:30.000Z,100,0.01,100,0.01,10,0,0`, + ].join('\n') +} + +function completedD1Csv(broker: string, server: string, closes = [2300, 2350, 2400]): string { + return [ + 'schema_version,captured_at,broker,server,account_mode,symbol,bar_as_of,bar_open_epoch,open,high,low,close', + ...closes.map((close, index) => `1,2026-07-13T09:59:30.000Z,${broker},${server},demo,XAUUSD,2026-07-${String(10 + index).padStart(2, '0')},${index + 1},${close},${close},${close},${close}`), + ].join('\n') +} + +function spreadCsv(broker: string, server: string): string { + return [ + 'schema_version,captured_at,broker,server,account_mode,symbol,bid,ask,spread,point,digits,contract_size,volume_min,volume_step,stops_level,freeze_level', + ...Array.from({ length: 100 }, (_, index) => { + const capturedAt = new Date(now.getTime() - (100 - index) * 60_000).toISOString() + return `1,${capturedAt},${broker},${server},demo,XAUUSD,2399.9,2400.1,0.2,0.01,2,100,0.01,0.01,10,0` + }), + ].join('\n') +} + +function ledgerCsv(broker: string, server: string): string { + return [ + 'account_mode,server,login,broker,symbol,deal_ticket,order_ticket,position_id,time,entry,type,reason,volume,price,commission,fee,swap,profit,magic,comment', + `demo,${server},redacted,${broker},XAUUSD,1,1,1,2026-07-13T09:00:00.000Z,out,buy,expert,0.01,2390,-0.1,0,0,1,880101,JMB Goldmine`, + ].join('\n') +} + +function policyCsv(broker: 'hfmarkets' | 'icmarkets', server: string): string { + const hfm = broker === 'hfmarkets' + return [ + 'schema_version,policy_version,broker,server,symbol,strategy_version,rollout_stage,candidate_approved,completed_observation_max_age_hours,max_spread,max_deviation,max_risk_amount,max_daily_loss,max_daily_losing_trades,max_volume,magic_number', + `1,${broker}-canary-v1,${broker},${server},XAUUSD,daily-trend-v1,${hfm ? 'hfm_canary' : 'ic_canary'},1,72,${hfm ? '0.75' : '0.3'},${hfm ? '0.5' : '0.3'},10,40,4,0.01,${hfm ? '880101' : '880201'}`, + ].join('\n') +} + +async function writeReadyGoldFiles( + rootsValue: JmbMt5Roots, + instrument: JmbDemoInstrumentConfig, + closes?: number[], +): Promise { + const bridgeDirectory = join(rootsValue.bridgeRoot, instrument.broker, 'XAUUSD') + const ledgerDirectory = join(rootsValue.ledgerRoot, instrument.broker, 'XAUUSD') + const policyDirectory = join(rootsValue.policyRoot, instrument.broker, 'XAUUSD') + await Promise.all([bridgeDirectory, ledgerDirectory, policyDirectory, rootsValue.researchRoot].map((directory) => mkdir(directory, { recursive: true }))) + const statusPath = join(bridgeDirectory, 'status.csv') + const completedPath = join(bridgeDirectory, 'completed_d1.csv') + const ledgerPath = join(ledgerDirectory, 'deals.csv') + await writeFile(statusPath, statusCsv(instrument.broker, instrument.server)) + await writeFile(completedPath, completedD1Csv(instrument.broker, instrument.server, closes)) + await writeFile(join(bridgeDirectory, 'spread_samples_20260713.csv'), spreadCsv(instrument.broker, instrument.server)) + await writeFile(ledgerPath, ledgerCsv(instrument.broker, instrument.server)) + await writeFile(join(policyDirectory, 'policy.csv'), policyCsv(instrument.broker, instrument.server)) + const researchName = instrument.broker === 'hfmarkets' ? 'xauusd-trend-baseline.json' : 'icmarkets-xauusd-trend-baseline.json' + await writeFile(join(rootsValue.researchRoot, researchName), JSON.stringify({ + symbol: instrument.researchArtifactSymbol, + selected_on_training_sharpe: { lookback_days: 2 }, + latest_observation: { as_of: '2026-07-12', direction: 'downtrend', lookback_days: 2 }, + })) + await Promise.all([statusPath, completedPath, ledgerPath].map((path) => utimes(path, now, now))) +} + +describe('demo decision cycle', () => { + it('keeps the diagnostic CLI delegation-only', async () => { + const source = await readFile(join(process.cwd(), 'tools', 'mt5', 'run_demo_canary_decisions.ts'), 'utf8') + expect(source).toContain('resolveJmbMt5Roots') + expect(source).toContain('runDemoDecisionCycle') + expect(source).not.toMatch(/child_process|powershell|MetaEditor|OrderSend|OrderCheck|lookback|stopDistance/) + }) + + it('inherits the approved eight-unit protective Gold stop for both brokers', () => { + expect(DEFAULT_JMB_DEMO_INSTRUMENTS.filter((item) => item.symbol === 'XAUUSD').map((item) => [item.broker, item.stopDistance])) + .toEqual([['hfmarkets', 8], ['icmarkets', 8]]) + }) + + it('publishes HFM from fresh completed D1 bars, persists cost first, and isolates a malformed IC cycle', async () => { + const rootsValue = await roots() + const hfm = DEFAULT_JMB_DEMO_INSTRUMENTS.find((item) => item.broker === 'hfmarkets' && item.symbol === 'XAUUSD')! + const ic = DEFAULT_JMB_DEMO_INSTRUMENTS.find((item) => item.broker === 'icmarkets' && item.symbol === 'XAUUSD')! + await writeReadyGoldFiles(rootsValue, hfm) + await writeReadyGoldFiles(rootsValue, ic) + await writeFile(join(rootsValue.researchRoot, 'icmarkets-xauusd-trend-baseline.json'), '{malformed') + + const results = await runDemoDecisionCycle({ roots: rootsValue, now: () => now, instruments: [ic, hfm] }) + + expect(results).toEqual([ + expect.objectContaining({ broker: 'icmarkets', symbol: 'XAUUSD', state: 'error', observationId: null, decisionId: null }), + expect.objectContaining({ broker: 'hfmarkets', symbol: 'XAUUSD', state: 'published', observationId: expect.any(String), decisionId: expect.any(String) }), + ]) + const decision = parseExecutionDecisionCsv(await readFile(join(rootsValue.executionDecisionRoot, 'hfmarkets', 'XAUUSD', 'latest_decision.csv'), 'utf8')) + expect(decision).toMatchObject({ direction: 'buy', entryReferencePrice: 2400.1, stopLoss: 2392.1 }) + expect(decision.observationAsOf).toBe('2026-07-12') + expect((await readFile(join(rootsValue.costModelRoot, 'hfmarkets', 'XAUUSD', 'cost_model.csv'), 'utf8'))).toContain(',canary_ready,') + expect((await stat(join(rootsValue.costModelRoot, 'hfmarkets', 'XAUUSD', 'cost_model.csv'))).mtimeMs).toBeLessThanOrEqual( + (await stat(join(rootsValue.executionDecisionRoot, 'hfmarkets', 'XAUUSD', 'latest_decision.csv'))).mtimeMs, + ) + }) + + it('uses the IC Gold default on the protective side of a sell reference', async () => { + const rootsValue = await roots() + const ic = DEFAULT_JMB_DEMO_INSTRUMENTS.find((item) => item.broker === 'icmarkets' && item.symbol === 'XAUUSD')! + await writeReadyGoldFiles(rootsValue, ic, [2500, 2450, 2400]) + + const [result] = await runDemoDecisionCycle({ roots: rootsValue, now: () => now, instruments: [ic] }) + + expect(result).toMatchObject({ state: 'published' }) + const decision = parseExecutionDecisionCsv(await readFile(join(rootsValue.executionDecisionRoot, 'icmarkets', 'XAUUSD', 'latest_decision.csv'), 'utf8')) + expect(decision).toMatchObject({ direction: 'sell', entryReferencePrice: 2399.9, stopLoss: 2407.9 }) + }) + + it('does not derive an executable observation from fresh D1 evidence on the wrong server', async () => { + const rootsValue = await roots() + const hfm = DEFAULT_JMB_DEMO_INSTRUMENTS.find((item) => item.broker === 'hfmarkets' && item.symbol === 'XAUUSD')! + await writeReadyGoldFiles(rootsValue, hfm) + const completedPath = join(rootsValue.bridgeRoot, 'hfmarkets', 'XAUUSD', 'completed_d1.csv') + await writeFile(completedPath, completedD1Csv('hfmarkets', 'ICMarketsSC-Demo')) + await utimes(completedPath, now, now) + + const [result] = await runDemoDecisionCycle({ roots: rootsValue, now: () => now, instruments: [hfm] }) + + expect(result).toMatchObject({ state: 'blocked', observationId: null, decisionId: null }) + expect(result.detail).toMatch(/server|identity/i) + }) + + it('rejects spread samples that would be parsed in the workstation local timezone', async () => { + const rootsValue = await roots() + const hfm = DEFAULT_JMB_DEMO_INSTRUMENTS.find((item) => item.broker === 'hfmarkets' && item.symbol === 'XAUUSD')! + await writeReadyGoldFiles(rootsValue, hfm) + const spreadPath = join(rootsValue.bridgeRoot, 'hfmarkets', 'XAUUSD', 'spread_samples_20260713.csv') + await writeFile(spreadPath, spreadCsv('hfmarkets', 'HFMarketsGlobal-Demo4').replaceAll('.000Z', '')) + + const [result] = await runDemoDecisionCycle({ roots: rootsValue, now: () => now, instruments: [hfm] }) + + expect(result).toMatchObject({ state: 'error', observationId: null, decisionId: null }) + expect(result.detail).toMatch(/captured_at.*UTC|timestamp.*UTC/i) + }) + + it('rejects an injected Gold stop distance other than the immutable eight units', async () => { + const rootsValue = await roots() + const defaultHfm = DEFAULT_JMB_DEMO_INSTRUMENTS.find((item) => item.broker === 'hfmarkets' && item.symbol === 'XAUUSD')! + const injected = { ...defaultHfm, stopDistance: 80 } + await writeReadyGoldFiles(rootsValue, injected) + + const [result] = await runDemoDecisionCycle({ roots: rootsValue, now: () => now, instruments: [injected] }) + + expect(result).toMatchObject({ state: 'error', observationId: null, decisionId: null }) + expect(result.detail).toMatch(/stop distance.*exactly 8/i) + await expect(stat(join(rootsValue.executionDecisionRoot, 'hfmarkets', 'XAUUSD'))).rejects.toMatchObject({ code: 'ENOENT' }) + }) + + it('returns EURUSD blocked and never creates an execution lease directory', async () => { + const rootsValue = await roots() + const eurusd = DEFAULT_JMB_DEMO_INSTRUMENTS.find((item) => item.broker === 'hfmarkets' && item.symbol === 'EURUSD')! + const [result] = await runDemoDecisionCycle({ roots: rootsValue, now: () => now, instruments: [eurusd] }) + + expect(result).toMatchObject({ broker: 'hfmarkets', symbol: 'EURUSD', state: 'blocked', observationId: null, decisionId: null }) + await expect(stat(join(rootsValue.executionDecisionRoot, 'hfmarkets', 'EURUSD'))).rejects.toMatchObject({ code: 'ENOENT' }) + }) +}) diff --git a/src/domain/mt5/demo-decision-service.ts b/src/domain/mt5/demo-decision-service.ts new file mode 100644 index 000000000..dc7102ce5 --- /dev/null +++ b/src/domain/mt5/demo-decision-service.ts @@ -0,0 +1,321 @@ +import { createHash } from 'node:crypto' +import { readFile, readdir } from 'node:fs/promises' +import { join } from 'node:path' + +import { + buildBrokerCostModel, + writeBrokerCostModel, + type BrokerCostClosedDeal, + type BrokerCostSpreadSample, +} from './broker-cost-model.js' +import { deriveCompletedTrendObservation, readMt5CompletedD1 } from './completed-d1.js' +import { buildDemoExecutionDecision } from './demo-decision-engine.js' +import { readDemoExecutionPolicy } from './demo-execution-policy.js' +import { writeExecutionDecision } from './execution-decision.js' +import type { JmbMt5Roots } from './local-paths.js' +import { readMt5ReadOnlyBridge } from './read-only-bridge.js' +import { parseMt5TradeLedgerCsv, summarizeMt5TradeLedger, type Mt5TradeLedgerRow } from './trade-ledger.js' + +export interface JmbDemoInstrumentConfig { + broker: 'hfmarkets' | 'icmarkets' + server: 'HFMarketsGlobal-Demo4' | 'ICMarketsSC-Demo' + symbol: 'XAUUSD' | 'EURUSD' + researchArtifactSymbol: 'XAUUSDb' | 'EURUSDb' | 'XAUUSD' | 'EURUSD' + maxSpread: number + maxDeviation: number + /** Required for Gold; null for non-persistable shadow-only instruments. */ + stopDistance: number | null +} + +export interface DemoDecisionCycleOptions { + roots: JmbMt5Roots + now?: () => Date + instruments?: readonly JmbDemoInstrumentConfig[] +} + +export interface DemoDecisionCycleResult { + broker: string + symbol: string + state: 'published' | 'blocked' | 'error' + observationId: string | null + decisionId: string | null + detail: string +} + +export const JMB_GOLD_STOP_DISTANCE = 8 as const + +export const DEFAULT_JMB_DEMO_INSTRUMENTS = [ + { broker: 'hfmarkets', server: 'HFMarketsGlobal-Demo4', symbol: 'XAUUSD', researchArtifactSymbol: 'XAUUSDb', maxSpread: 0.75, maxDeviation: 0.5, stopDistance: JMB_GOLD_STOP_DISTANCE }, + { broker: 'hfmarkets', server: 'HFMarketsGlobal-Demo4', symbol: 'EURUSD', researchArtifactSymbol: 'EURUSDb', maxSpread: 0.00025, maxDeviation: 0.0002, stopDistance: null }, + { broker: 'icmarkets', server: 'ICMarketsSC-Demo', symbol: 'XAUUSD', researchArtifactSymbol: 'XAUUSD', maxSpread: 0.3, maxDeviation: 0.3, stopDistance: JMB_GOLD_STOP_DISTANCE }, + { broker: 'icmarkets', server: 'ICMarketsSC-Demo', symbol: 'EURUSD', researchArtifactSymbol: 'EURUSD', maxSpread: 0.00015, maxDeviation: 0.0001, stopDistance: null }, +] as const satisfies readonly JmbDemoInstrumentConfig[] + +const SPREAD_HEADER = 'schema_version,captured_at,broker,server,account_mode,symbol,bid,ask,spread,point,digits,contract_size,volume_min,volume_step,stops_level,freeze_level' +const RFC3339_UTC = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{3})?Z$/ + +interface ResearchTrendReport { + symbol: string + selected_on_training_sharpe: { lookback_days: number } +} + +interface ParsedSpreadEvidence { + samples: BrokerCostSpreadSample[] + expectedContractFingerprint: string +} + +function fileCode(error: unknown): string | null { + return typeof error === 'object' && error !== null && 'code' in error ? String((error as { code: unknown }).code) : null +} + +function finite(value: string | undefined, field: string): number { + const parsed = Number(value) + if (value === undefined || value.trim() === '' || !Number.isFinite(parsed)) throw new Error(`Spread evidence ${field} must be finite.`) + return parsed +} + +function contractFingerprint(values: readonly string[]): string { + return createHash('sha256').update(values.join('|')).digest('hex') +} + +function parseSpreadEvidenceCsv( + text: string, + instrument: JmbDemoInstrumentConfig, +): BrokerCostSpreadSample[] { + const [header, ...rows] = text.trim().split(/\r?\n/) + if (header !== SPREAD_HEADER || rows.length === 0) throw new Error('Spread evidence CSV has an invalid header or no samples.') + return rows.map((row) => { + const values = row.split(',') + if (values.length !== 16 || values[0] !== '1') throw new Error('Spread evidence row does not match schema version 1.') + if (values[2] !== instrument.broker + || values[3] !== instrument.server + || values[4] !== 'demo' + || values[5] !== 'XAUUSD') { + throw new Error('Spread evidence identity does not match the demo Gold instrument.') + } + if (!RFC3339_UTC.test(values[1]!) || !Number.isFinite(Date.parse(values[1]!))) { + throw new Error('Spread evidence captured_at must be canonical RFC 3339 UTC.') + } + finite(values[6], 'bid') + finite(values[7], 'ask') + const spread = finite(values[8], 'spread') + const fingerprintValues = values.slice(9, 16) + fingerprintValues.forEach((value, index) => finite(value, `contract field ${index + 1}`)) + return { + capturedAt: values[1]!, + spread, + contractFingerprint: contractFingerprint(fingerprintValues), + } + }) +} + +async function readSpreadEvidence(root: string, instrument: JmbDemoInstrumentConfig): Promise { + const directory = join(root, instrument.broker, instrument.symbol) + let names: string[] + try { + names = (await readdir(directory)).filter((name) => /^spread_samples_\d{8}\.csv$/.test(name)).sort() + } catch (error) { + if (fileCode(error) === 'ENOENT') return { samples: [], expectedContractFingerprint: '' } + throw error + } + const samples = (await Promise.all(names.map(async (name) => { + return parseSpreadEvidenceCsv(await readFile(join(directory, name), 'utf8'), instrument) + }))).flat().sort((left, right) => Date.parse(left.capturedAt) - Date.parse(right.capturedAt)) + return { + samples, + expectedContractFingerprint: samples.at(-1)?.contractFingerprint ?? '', + } +} + +async function readLedgerRows(root: string, instrument: JmbDemoInstrumentConfig): Promise { + try { + return parseMt5TradeLedgerCsv(await readFile(join(root, instrument.broker, instrument.symbol, 'deals.csv'), 'utf8')) + .filter((row) => row.broker === instrument.broker && row.symbol === instrument.symbol) + } catch (error) { + if (fileCode(error) === 'ENOENT') return [] + throw error + } +} + +function closedDeals(rows: readonly Mt5TradeLedgerRow[]): BrokerCostClosedDeal[] { + return rows.map((row) => ({ + accountMode: row.accountMode, + symbol: row.symbol, + closed: row.entry === 'out' || row.entry === 'out_by', + commission: row.commission, + swap: row.swap, + })) +} + +function costModelVersion( + instrument: JmbDemoInstrumentConfig, + spreadEvidence: ParsedSpreadEvidence, + rows: readonly Mt5TradeLedgerRow[], +): string { + const digest = createHash('sha256') + .update(JSON.stringify({ spreadEvidence, deals: closedDeals(rows) })) + .digest('hex') + .slice(0, 16) + return `${instrument.broker}-observed-${digest}` +} + +function researchFileName(instrument: JmbDemoInstrumentConfig): string { + const canonical = instrument.researchArtifactSymbol.replace(/b$/, '').toLowerCase() + return `${instrument.broker === 'icmarkets' ? 'icmarkets-' : ''}${canonical}-trend-baseline.json` +} + +async function readSelectedLookback(root: string, instrument: JmbDemoInstrumentConfig): Promise { + const path = join(root, researchFileName(instrument)) + let parsed: unknown + try { + parsed = JSON.parse(await readFile(path, 'utf8')) + } catch (error) { + throw new Error(`Research artifact is missing or malformed: ${error instanceof Error ? error.message : 'unknown error'}`) + } + if (typeof parsed !== 'object' || parsed === null) throw new Error('Research artifact must be an object.') + const report = parsed as Partial + const lookback = report.selected_on_training_sharpe?.lookback_days + if (report.symbol !== instrument.researchArtifactSymbol || !Number.isInteger(lookback) || (lookback ?? 0) <= 0) { + throw new Error('Research artifact identity or frozen selected lookback is invalid.') + } + return lookback! +} + +function protectiveStop( + direction: 'uptrend' | 'downtrend' | 'flat', + bid: number | null, + ask: number | null, + distance: number | null, +): number | null { + if (distance === null || !Number.isFinite(distance) || distance <= 0) return null + if (direction === 'uptrend' && ask !== null && Number.isFinite(ask)) return Number((ask - distance).toFixed(2)) + if (direction === 'downtrend' && bid !== null && Number.isFinite(bid)) return Number((bid + distance).toFixed(2)) + return null +} + +function blockedEurUsd(instrument: JmbDemoInstrumentConfig): DemoDecisionCycleResult { + return { + broker: instrument.broker, + symbol: instrument.symbol, + state: 'blocked', + observationId: null, + decisionId: null, + detail: 'EURUSD remains shadow-only; its existing shadow journal remains the durable decision record.', + } +} + +async function runInstrumentCycle( + roots: JmbMt5Roots, + instrument: JmbDemoInstrumentConfig, + now: Date, +): Promise { + if (instrument.symbol === 'EURUSD') return blockedEurUsd(instrument) + if (instrument.stopDistance !== JMB_GOLD_STOP_DISTANCE) { + throw new Error(`Gold execution stop distance must be exactly ${JMB_GOLD_STOP_DISTANCE}; caller overrides are not permitted.`) + } + + const policy = await readDemoExecutionPolicy(roots.policyRoot, instrument.broker, instrument.symbol) + const [bridge, completed, learning, rows, spreadEvidence, lookback] = await Promise.all([ + readMt5ReadOnlyBridge(roots.bridgeRoot, instrument.broker, instrument.symbol, now), + readMt5CompletedD1(roots.bridgeRoot, instrument.broker, instrument.symbol, { + now, + maxAgeHours: policy.policy?.completedObservationMaxAgeHours ?? 72, + expectedServer: instrument.server, + }), + summarizeMt5TradeLedger(roots.ledgerRoot, instrument.broker, instrument.symbol, now), + readLedgerRows(roots.ledgerRoot, instrument), + readSpreadEvidence(roots.bridgeRoot, instrument), + readSelectedLookback(roots.researchRoot, instrument), + ]) + + const nowIso = now.toISOString() + const model = buildBrokerCostModel({ + version: costModelVersion(instrument, spreadEvidence, rows), + broker: instrument.broker, + server: instrument.server, + symbol: 'XAUUSD', + now: nowIso, + bridge: { + state: bridge.state, + capturedAt: bridge.lastUpdated ?? '', + contractFingerprint: spreadEvidence.expectedContractFingerprint, + }, + spreadSamples: spreadEvidence.samples, + closedDeals: closedDeals(rows), + expectedContractFingerprint: spreadEvidence.expectedContractFingerprint, + configuredMaxSpread: instrument.maxSpread, + configuredMaxDeviation: instrument.maxDeviation, + }) + await writeBrokerCostModel(roots.costModelRoot, model) + + const observation = completed.parsed === null ? null : deriveCompletedTrendObservation(completed.parsed, lookback) + const result = buildDemoExecutionDecision({ + createdAt: nowIso, + leaseIssuedAt: nowIso, + leaseExpiresAt: new Date(now.getTime() + 10 * 60_000).toISOString(), + broker: instrument.broker, + server: instrument.server, + symbol: instrument.symbol, + bridge: { + state: bridge.state, + broker: bridge.broker, + server: bridge.server, + symbol: bridge.symbol, + }, + completedObservation: { state: completed.state, detail: completed.detail, observation }, + policy, + costModel: model, + learning: { state: learning.state, accountMode: learning.accountMode, server: learning.server }, + quote: { bid: bridge.bid, ask: bridge.ask, spread: bridge.spread }, + stopLoss: observation === null ? null : protectiveStop(observation.direction, bridge.bid, bridge.ask, JMB_GOLD_STOP_DISTANCE), + }) + + if (result.decision === null) { + return { + broker: instrument.broker, + symbol: instrument.symbol, + state: 'blocked', + observationId: null, + decisionId: null, + detail: result.detail, + } + } + const persisted = await writeExecutionDecision(roots.executionDecisionRoot, result.decision) + if (persisted.state === 'regressed') { + return { + broker: instrument.broker, + symbol: instrument.symbol, + state: 'blocked', + observationId: result.decision.observationId, + decisionId: result.decision.decisionId, + detail: 'A newer completed observation is already published; the regressed lease was ignored.', + } + } + return { + broker: instrument.broker, + symbol: instrument.symbol, + state: result.state === 'ready' ? 'published' : 'blocked', + observationId: result.decision.observationId, + decisionId: result.decision.decisionId, + detail: result.detail, + } +} + +export async function runDemoDecisionCycle(options: DemoDecisionCycleOptions): Promise { + const now = (options.now ?? (() => new Date()))() + const instruments = options.instruments ?? DEFAULT_JMB_DEMO_INSTRUMENTS + return Promise.all(instruments.map(async (instrument) => { + try { + return await runInstrumentCycle(options.roots, instrument, now) + } catch (error) { + return { + broker: instrument.broker, + symbol: instrument.symbol, + state: 'error', + observationId: null, + decisionId: null, + detail: error instanceof Error ? error.message : 'Unknown demo decision cycle error.', + } + } + })) +} diff --git a/src/domain/mt5/demo-execution-policy.spec.ts b/src/domain/mt5/demo-execution-policy.spec.ts new file mode 100644 index 000000000..b2fd22f1b --- /dev/null +++ b/src/domain/mt5/demo-execution-policy.spec.ts @@ -0,0 +1,180 @@ +import { afterEach, describe, expect, it } from 'vitest' +import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { readDemoExecutionPolicy, validateDemoExecutionPolicy } from './demo-execution-policy.js' + +const policy = { + schemaVersion: 1 as const, + policyVersion: 'hfm-canary-v1', + broker: 'hfmarkets', + server: 'HFMarketsGlobal-Demo4', + symbol: 'XAUUSD', + strategyVersion: 'daily-trend-v1', + rolloutStage: 'hfm_canary' as const, + candidateApproved: true, + completedObservationMaxAgeHours: 72, + maxSpread: 0.75, + maxDeviation: 0.5, + maxRiskAmount: 10, + maxDailyLoss: 40, + maxDailyLosingTrades: 4, + maxVolume: 0.01, + magicNumber: 880101, +} + +const header = 'schema_version,policy_version,broker,server,symbol,strategy_version,rollout_stage,candidate_approved,completed_observation_max_age_hours,max_spread,max_deviation,max_risk_amount,max_daily_loss,max_daily_losing_trades,max_volume,magic_number' +const policyRow = '1,hfm-canary-v1,hfmarkets,HFMarketsGlobal-Demo4,XAUUSD,daily-trend-v1,hfm_canary,1,72,0.75,0.5,10,40,4,0.01,880101' +const policyCsv = `${header}\n${policyRow}` +const directories: string[] = [] + +afterEach(async () => { + await Promise.all(directories.splice(0).map((directory) => rm(directory, { recursive: true, force: true }))) +}) + +describe('demo execution policy', () => { + it('accepts the exact HFM canary ceiling', () => expect(validateDemoExecutionPolicy(policy).state).toBe('ready')) + + it('blocks a policy that loosens max volume', () => { + expect(validateDemoExecutionPolicy({ ...policy, maxVolume: 0.02 }).state).toBe('blocked') + }) + + it('blocks EURUSD regardless of candidate flag', () => { + expect(validateDemoExecutionPolicy({ ...policy, symbol: 'EURUSD' }).state).toBe('blocked') + }) + + it('allows tightened limits but blocks every loosened hard ceiling', () => { + expect(validateDemoExecutionPolicy({ + ...policy, + completedObservationMaxAgeHours: 48, + maxSpread: 0.5, + maxDeviation: 0.25, + maxRiskAmount: 5, + maxDailyLoss: 20, + maxDailyLosingTrades: 2, + maxVolume: 0.005, + }).state).toBe('ready') + + for (const [field, value] of [ + ['completedObservationMaxAgeHours', 73], + ['maxSpread', 0.76], + ['maxDeviation', 0.51], + ['maxRiskAmount', 10.01], + ['maxDailyLoss', 40.01], + ['maxDailyLosingTrades', 5], + ['maxVolume', 0.02], + ] as const) { + expect(validateDemoExecutionPolicy({ ...policy, [field]: value }).state, field).toBe('blocked') + } + }) + + it('blocks status-only, unapproved, mismatched, and non-finite policies', () => { + expect(validateDemoExecutionPolicy({ ...policy, rolloutStage: 'status_only' }).state).toBe('blocked') + expect(validateDemoExecutionPolicy({ ...policy, candidateApproved: false }).state).toBe('blocked') + expect(validateDemoExecutionPolicy({ ...policy, server: 'ICMarketsSC-Demo' }).state).toBe('blocked') + expect(validateDemoExecutionPolicy({ ...policy, maxSpread: Number.NaN }).state).toBe('blocked') + }) + + it('reads only the exact strict CSV contract and requested identity', async () => { + const root = await mkdtemp(join(tmpdir(), 'openalice-policy-')) + directories.push(root) + const directory = join(root, 'hfmarkets', 'XAUUSD') + await mkdir(directory, { recursive: true }) + await writeFile(join(directory, 'policy.csv'), policyCsv) + + await expect(readDemoExecutionPolicy(root, 'hfmarkets', 'XAUUSD')).resolves.toMatchObject({ + state: 'ready', + policy: { broker: 'hfmarkets', symbol: 'XAUUSD', candidateApproved: true }, + }) + + await writeFile(join(directory, 'policy.csv'), `${header},unexpected\n1,hfm-canary-v1,hfmarkets,HFMarketsGlobal-Demo4,XAUUSD,daily-trend-v1,hfm_canary,1,72,0.75,0.5,10,40,4,0.01,880101,extra`) + await expect(readDemoExecutionPolicy(root, 'hfmarkets', 'XAUUSD')).resolves.toMatchObject({ state: 'malformed', policy: null }) + }) + + it('returns missing for an unreadable policy file', async () => { + await expect(readDemoExecutionPolicy('missing-root', 'hfmarkets', 'XAUUSD')).resolves.toMatchObject({ state: 'missing', policy: null }) + }) + + it('accepts exactly two lines with only an optional LF or CRLF terminator', async () => { + const root = await mkdtemp(join(tmpdir(), 'openalice-policy-boundary-')) + directories.push(root) + const directory = join(root, 'hfmarkets', 'XAUUSD') + const path = join(directory, 'policy.csv') + await mkdir(directory, { recursive: true }) + + const crlfPolicyCsv = `${header}\r\n${policyRow}` + for (const text of [policyCsv, `${policyCsv}\n`, crlfPolicyCsv, `${crlfPolicyCsv}\r\n`]) { + await writeFile(path, text) + await expect(readDemoExecutionPolicy(root, 'hfmarkets', 'XAUUSD')).resolves.toMatchObject({ state: 'ready' }) + } + }) + + it('rejects whitespace and blank lines outside the exact two-line CSV contract', async () => { + const root = await mkdtemp(join(tmpdir(), 'openalice-policy-boundary-')) + directories.push(root) + const directory = join(root, 'hfmarkets', 'XAUUSD') + const path = join(directory, 'policy.csv') + await mkdir(directory, { recursive: true }) + + for (const text of [ + ` ${policyCsv}`, + `\n${policyCsv}`, + `${policyCsv}\n\n`, + `${policyCsv} `, + `${policyCsv}\n `, + ]) { + await writeFile(path, text) + await expect(readDemoExecutionPolicy(root, 'hfmarkets', 'XAUUSD'), JSON.stringify(text)).resolves.toMatchObject({ state: 'malformed', policy: null }) + } + }) + + it('keeps the MQL policy-version writer grammar aligned with the reader', async () => { + const source = await readFile(join(process.cwd(), 'tools', 'mt5', 'ConfigureJmbGoldmineDemoPolicy.mq5'), 'utf8') + expect(source).toContain('string canonical = InpPolicyVersion;') + expect(source).toContain('StringTrimLeft(canonical);') + expect(source).toContain('StringTrimRight(canonical);') + expect(source).toContain('canonical == InpPolicyVersion') + expect(source).toContain('StringFind(canonical, ",") < 0') + expect(source).toContain('StringFind(canonical, "\\\"") < 0') + expect(source).toContain('StringFind(canonical, "\\r") < 0') + expect(source).toContain('StringFind(canonical, "\\n") < 0') + + const invalidVersions = [ + ' hfm-canary-v1', + 'hfm-canary-v1 ', + 'hfm,canary-v1', + 'hfm"canary-v1', + 'hfm\rcanary-v1', + 'hfm\ncanary-v1', + ] + for (const version of invalidVersions) { + expect(validateDemoExecutionPolicy({ ...policy, policyVersion: version }).state, JSON.stringify(version)).toBe('blocked') + } + + const root = await mkdtemp(join(tmpdir(), 'openalice-policy-version-')) + directories.push(root) + const directory = join(root, 'hfmarkets', 'XAUUSD') + const path = join(directory, 'policy.csv') + await mkdir(directory, { recursive: true }) + + await writeFile(path, policyCsv) + await expect(readDemoExecutionPolicy(root, 'hfmarkets', 'XAUUSD')).resolves.toMatchObject({ state: 'ready' }) + for (const version of invalidVersions) { + await writeFile(path, `${header}\n${policyRow.replace('hfm-canary-v1', version)}`) + await expect(readDemoExecutionPolicy(root, 'hfmarkets', 'XAUUSD')).resolves.toMatchObject({ state: 'malformed', policy: null }) + } + }) + + it('keeps policy writes operator-only and demo-Gold allowlisted in MQL', async () => { + const source = await readFile(join(process.cwd(), 'tools', 'mt5', 'ConfigureJmbGoldmineDemoPolicy.mq5'), 'utf8') + expect(source).toContain('input string InpRolloutStage = "status_only";') + expect(source).toContain('ACCOUNT_TRADE_MODE_DEMO') + expect(source).toContain('HFMarketsGlobal-Demo4') + expect(source).toContain('ICMarketsSC-Demo') + expect(source).toContain('880101') + expect(source).toContain('880201') + expect(source).toContain('OpenAliceMt5DemoPolicyV1') + expect(source).toContain('"schema_version","policy_version","broker","server","symbol","strategy_version","rollout_stage","candidate_approved","completed_observation_max_age_hours","max_spread","max_deviation","max_risk_amount","max_daily_loss","max_daily_losing_trades","max_volume","magic_number"') + expect(source).not.toMatch(/OrderSend|OrderCheck|PositionClose/) + }) +}) diff --git a/src/domain/mt5/demo-execution-policy.ts b/src/domain/mt5/demo-execution-policy.ts new file mode 100644 index 000000000..8cf3f9285 --- /dev/null +++ b/src/domain/mt5/demo-execution-policy.ts @@ -0,0 +1,186 @@ +import { readFile } from 'node:fs/promises' +import { join } from 'node:path' + +export type DemoRolloutStage = 'status_only' | 'hfm_canary' | 'ic_canary' | 'both_demo' + +export interface DemoExecutionPolicy { + schemaVersion: 1 + policyVersion: string + broker: 'hfmarkets' | 'icmarkets' + server: 'HFMarketsGlobal-Demo4' | 'ICMarketsSC-Demo' + symbol: 'XAUUSD' + strategyVersion: 'daily-trend-v1' + rolloutStage: DemoRolloutStage + candidateApproved: boolean + completedObservationMaxAgeHours: number + maxSpread: number + maxDeviation: number + maxRiskAmount: number + maxDailyLoss: number + maxDailyLosingTrades: number + maxVolume: number + magicNumber: 880101 | 880201 +} + +export interface DemoExecutionPolicySummary { + state: 'ready' | 'blocked' | 'missing' | 'malformed' + detail: string + policy: DemoExecutionPolicy | null +} + +const POLICY_HEADER = 'schema_version,policy_version,broker,server,symbol,strategy_version,rollout_stage,candidate_approved,completed_observation_max_age_hours,max_spread,max_deviation,max_risk_amount,max_daily_loss,max_daily_losing_trades,max_volume,magic_number' +const POLICY_VERSION_FORBIDDEN = /[,"\r\n]/ + +const BROKER_RULES = { + hfmarkets: { + server: 'HFMarketsGlobal-Demo4', + magicNumber: 880101, + maxSpread: 0.75, + maxDeviation: 0.5, + stages: ['hfm_canary', 'both_demo'], + }, + icmarkets: { + server: 'ICMarketsSC-Demo', + magicNumber: 880201, + maxSpread: 0.3, + maxDeviation: 0.3, + stages: ['ic_canary', 'both_demo'], + }, +} as const + +const SHARED_CEILINGS = { + completedObservationMaxAgeHours: 72, + maxRiskAmount: 10, + maxDailyLoss: 40, + maxDailyLosingTrades: 4, + maxVolume: 0.01, +} as const + +function blocked(detail: string, policy: DemoExecutionPolicy | null = null): DemoExecutionPolicySummary { + return { state: 'blocked', detail, policy } +} + +function isPositiveFinite(value: unknown): value is number { + return typeof value === 'number' && Number.isFinite(value) && value > 0 +} + +function isCanonicalPolicyVersion(value: unknown): value is string { + return typeof value === 'string' + && value !== '' + && value.trim() === value + && !POLICY_VERSION_FORBIDDEN.test(value) +} + +export function validateDemoExecutionPolicy(input: unknown): DemoExecutionPolicySummary { + if (typeof input !== 'object' || input === null) return blocked('The demo execution policy is not an object.') + const candidate = input as Record + const broker = candidate['broker'] + if (broker !== 'hfmarkets' && broker !== 'icmarkets') return blocked('The policy broker is not allowlisted.') + const rules = BROKER_RULES[broker] + + const exactIdentity = candidate['schemaVersion'] === 1 + && isCanonicalPolicyVersion(candidate['policyVersion']) + && candidate['server'] === rules.server + && candidate['symbol'] === 'XAUUSD' + && candidate['strategyVersion'] === 'daily-trend-v1' + && candidate['magicNumber'] === rules.magicNumber + if (!exactIdentity) return blocked('The policy identity does not match the immutable demo allowlist.') + + const rolloutStage = candidate['rolloutStage'] + if (rolloutStage === 'status_only') return blocked('The status-only rollout stage is never executable.', input as DemoExecutionPolicy) + if (!rules.stages.some((stage) => stage === rolloutStage)) { + return blocked('The rollout stage does not authorize this broker.', input as DemoExecutionPolicy) + } + if (candidate['candidateApproved'] !== true) { + return blocked('The demo candidate is not operator-approved.', input as DemoExecutionPolicy) + } + + const limits = { + completedObservationMaxAgeHours: SHARED_CEILINGS.completedObservationMaxAgeHours, + maxSpread: rules.maxSpread, + maxDeviation: rules.maxDeviation, + maxRiskAmount: SHARED_CEILINGS.maxRiskAmount, + maxDailyLoss: SHARED_CEILINGS.maxDailyLoss, + maxDailyLosingTrades: SHARED_CEILINGS.maxDailyLosingTrades, + maxVolume: SHARED_CEILINGS.maxVolume, + } + for (const [field, ceiling] of Object.entries(limits)) { + const value = candidate[field] + if (!isPositiveFinite(value) || value > ceiling) { + return blocked(`The policy ${field} exceeds or invalidates its immutable hard ceiling.`, input as DemoExecutionPolicy) + } + } + if (!Number.isInteger(candidate['maxDailyLosingTrades'])) { + return blocked('The policy maxDailyLosingTrades must be a positive integer.', input as DemoExecutionPolicy) + } + + return { + state: 'ready', + detail: 'The operator policy matches the immutable demo execution ceilings.', + policy: input as DemoExecutionPolicy, + } +} + +function parseNumber(value: string | undefined, field: string): number { + if (value === undefined || value.trim() === '') throw new Error(`Policy ${field} is empty.`) + const parsed = Number(value) + if (!Number.isFinite(parsed)) throw new Error(`Policy ${field} must be finite.`) + return parsed +} + +function parseDemoExecutionPolicyCsv(text: string): DemoExecutionPolicy { + const contents = text.endsWith('\r\n') + ? text.slice(0, -2) + : text.endsWith('\n') ? text.slice(0, -1) : text + const lines = contents.split(/\r?\n/) + if (lines.length !== 2 || lines[0] !== POLICY_HEADER) throw new Error('The policy CSV header or row count is invalid.') + const values = lines[1]!.split(',') + if (values.length !== 16) throw new Error('The policy CSV row must contain exactly 16 columns.') + if (values.some((value) => value.trim() !== value)) throw new Error('The policy CSV contains unexpected whitespace.') + if (!isCanonicalPolicyVersion(values[1])) throw new Error('Policy policy_version contains a forbidden character.') + if (values[7] !== '0' && values[7] !== '1') throw new Error('Policy candidate_approved must be 0 or 1.') + + return { + schemaVersion: parseNumber(values[0], 'schema_version') as 1, + policyVersion: values[1]!, + broker: values[2] as DemoExecutionPolicy['broker'], + server: values[3] as DemoExecutionPolicy['server'], + symbol: values[4] as 'XAUUSD', + strategyVersion: values[5] as 'daily-trend-v1', + rolloutStage: values[6] as DemoRolloutStage, + candidateApproved: values[7] === '1', + completedObservationMaxAgeHours: parseNumber(values[8], 'completed_observation_max_age_hours'), + maxSpread: parseNumber(values[9], 'max_spread'), + maxDeviation: parseNumber(values[10], 'max_deviation'), + maxRiskAmount: parseNumber(values[11], 'max_risk_amount'), + maxDailyLoss: parseNumber(values[12], 'max_daily_loss'), + maxDailyLosingTrades: parseNumber(values[13], 'max_daily_losing_trades'), + maxVolume: parseNumber(values[14], 'max_volume'), + magicNumber: parseNumber(values[15], 'magic_number') as DemoExecutionPolicy['magicNumber'], + } +} + +export async function readDemoExecutionPolicy(root: string, broker: string, symbol: string): Promise { + let text: string + try { + text = await readFile(join(root, broker, symbol, 'policy.csv'), 'utf8') + } catch { + return { state: 'missing', detail: 'The operator-managed demo execution policy is missing or unreadable.', policy: null } + } + + let policy: DemoExecutionPolicy + try { + policy = parseDemoExecutionPolicyCsv(text) + } catch (error) { + return { + state: 'malformed', + detail: error instanceof Error ? error.message : 'The demo execution policy is malformed.', + policy: null, + } + } + + if (policy.broker !== broker || policy.symbol !== symbol) { + return blocked('The policy identity does not match the requested broker and symbol.', policy) + } + return validateDemoExecutionPolicy(policy) +} diff --git a/src/domain/mt5/execution-decision.spec.ts b/src/domain/mt5/execution-decision.spec.ts new file mode 100644 index 000000000..e95bbac25 --- /dev/null +++ b/src/domain/mt5/execution-decision.spec.ts @@ -0,0 +1,248 @@ +import { afterEach, describe, expect, it, vi } from 'vitest' +import { mkdir, mkdtemp, readFile, readdir, rm } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' + +import { + createExecutionDecisionId, + createObservationId, + parseExecutionDecisionCsv, + serializeExecutionDecisionCsv, + writeExecutionDecision, + type JmbExecutionDecision, +} from './execution-decision.js' + +const fsMocks = vi.hoisted(() => ({ readFile: vi.fn() })) + +vi.mock('node:fs/promises', async (importOriginal) => { + const actual = await importOriginal() + fsMocks.readFile.mockImplementation(actual.readFile) + return { ...actual, readFile: fsMocks.readFile } +}) + +const directories: string[] = [] + +afterEach(async () => { + await Promise.all(directories.splice(0).map((directory) => rm(directory, { recursive: true, force: true }))) + const actual = await vi.importActual('node:fs/promises') + fsMocks.readFile.mockReset().mockImplementation(actual.readFile) +}) + +async function forceConcurrentLatestSnapshots(firstReturnDelayMs = 0): Promise { + const actual = await vi.importActual('node:fs/promises') + let latestReads = 0 + let notifySecond!: () => void + const secondSeen = new Promise((resolve) => { notifySecond = resolve }) + fsMocks.readFile.mockImplementation(async (path: string, encoding: BufferEncoding) => { + if (path.endsWith('latest_decision.csv') && latestReads < 2) { + latestReads += 1 + const readNumber = latestReads + if (readNumber === 1) { + await Promise.race([ + secondSeen, + new Promise((resolve) => setTimeout(resolve, 40)), + ]) + } else { + notifySecond() + } + + let snapshot: string | Error + try { + snapshot = await actual.readFile(path, encoding) + } catch (error) { + snapshot = error as Error + } + if (readNumber === 1 && firstReturnDelayMs > 0) { + await new Promise((resolve) => setTimeout(resolve, firstReturnDelayMs)) + } + if (snapshot instanceof Error) throw snapshot + return snapshot + } + return actual.readFile(path, encoding) + }) +} + +function sampleDecision(overrides: Partial = {}): JmbExecutionDecision { + const identity = { + broker: 'hfmarkets' as const, + symbol: 'XAUUSD' as const, + strategyVersion: 'daily-trend-v1' as const, + observationAsOf: '2026-07-12', + ...overrides, + } + const observationId = overrides.observationId ?? createObservationId(identity) + return { + schemaVersion: 1, + decisionId: overrides.decisionId ?? createExecutionDecisionId({ observationId }), + observationId, + observationAsOf: identity.observationAsOf, + createdAt: '2026-07-13T09:00:00.000Z', + leaseIssuedAt: '2026-07-13T09:00:00.000Z', + leaseExpiresAt: '2026-07-13T09:10:00.000Z', + broker: identity.broker, + server: 'HFMarketsGlobal-Demo4', + accountMode: 'demo', + symbol: identity.symbol, + strategyVersion: identity.strategyVersion, + direction: 'buy', + entryReferencePrice: 2400.25, + volume: 0.01, + stopLoss: 2392.25, + maxRiskAmount: 10, + candidatePolicyVersion: 'hfm-v1', + costModelVersion: 'cost-0900', + gateResults: [{ name: 'candidate_policy', state: 'pass', detail: 'HFM canary approved.' }], + ...overrides, + } +} + +describe('execution decision identity', () => { + it('keeps ids stable when only the five-minute lease changes', () => { + const first = sampleDecision({ leaseIssuedAt: '2026-07-13T09:00:00Z', leaseExpiresAt: '2026-07-13T09:10:00Z' }) + const second = sampleDecision({ leaseIssuedAt: '2026-07-13T09:05:00Z', leaseExpiresAt: '2026-07-13T09:15:00Z' }) + expect(createObservationId(first)).toBe(createObservationId(second)) + expect(createExecutionDecisionId(first)).toBe(createExecutionDecisionId(second)) + }) + + it('changes identity for a newer completed D1 date', () => { + expect(createObservationId(sampleDecision({ observationAsOf: '2026-07-12' }))) + .not.toBe(createObservationId(sampleDecision({ observationAsOf: '2026-07-11' }))) + }) + + it('does not re-identify a consumed observation after policy or cost refresh', () => { + const first = sampleDecision({ candidatePolicyVersion: 'hfm-v1', costModelVersion: 'cost-0900' }) + const refreshed = sampleDecision({ candidatePolicyVersion: 'hfm-v2', costModelVersion: 'cost-0905' }) + expect(createExecutionDecisionId(first)).toBe(createExecutionDecisionId(refreshed)) + }) +}) + +describe('execution decision CSV', () => { + it('round-trips the exact allowlisted schema', () => { + const decision = sampleDecision() + const csv = serializeExecutionDecisionCsv(decision) + expect(csv.split(/\r?\n/, 1)[0]).toBe('schema_version,decision_id,observation_id,observation_as_of,created_at,lease_issued_at,lease_expires_at,broker,server,account_mode,symbol,strategy_version,direction,entry_reference_price,volume,stop_loss,max_risk_amount,candidate_policy_version,cost_model_version,gate_results_json') + expect(parseExecutionDecisionCsv(csv)).toEqual(decision) + }) + + it('rejects extra headers and invalid semantic enum values', () => { + const csv = serializeExecutionDecisionCsv(sampleDecision()) + expect(() => parseExecutionDecisionCsv(csv.replace('gate_results_json', 'gate_results_json,unexpected').replace('\n', '\nextra,'))).toThrow(/schema/i) + expect(() => parseExecutionDecisionCsv(csv.replace(',buy,', ',hold,'))).toThrow(/direction/i) + expect(() => parseExecutionDecisionCsv(csv.replace('HFMarketsGlobal-Demo4', 'ICMarketsSC-Demo'))).toThrow(/server/i) + expect(() => parseExecutionDecisionCsv(csv.replace('"pass"', '"warn"'))).toThrow(/gate/i) + }) + + it('rejects forged identities and invalid lease ordering', () => { + const csv = serializeExecutionDecisionCsv(sampleDecision()) + expect(() => parseExecutionDecisionCsv(csv.replace(/\n1,[^,]+/, '\n1,forged'))).toThrow(/decision id/i) + expect(() => parseExecutionDecisionCsv(csv.replace('2026-07-13T09:10:00.000Z', '2026-07-13T08:59:00.000Z'))).toThrow(/lease/i) + }) +}) + +describe('execution decision persistence', () => { + it('serializes identical concurrent publications without duplicating the journal', async () => { + const root = await mkdtemp(join(tmpdir(), 'openalice-execution-decision-')) + directories.push(root) + const decision = sampleDecision() + await forceConcurrentLatestSnapshots() + + await Promise.all([writeExecutionDecision(root, decision), writeExecutionDecision(root, decision)]) + + const journal = await readFile(join(root, 'hfmarkets', 'XAUUSD', 'decisions.jsonl'), 'utf8') + expect(journal.trim().split(/\r?\n/)).toHaveLength(1) + }) + + it('serializes concurrent observations so an older publication cannot win last', async () => { + const root = await mkdtemp(join(tmpdir(), 'openalice-execution-decision-')) + directories.push(root) + const olderIdentity = { + broker: 'hfmarkets' as const, + symbol: 'XAUUSD' as const, + strategyVersion: 'daily-trend-v1' as const, + observationAsOf: '2026-07-11', + } + const olderObservationId = createObservationId(olderIdentity) + const older = sampleDecision({ + observationAsOf: olderIdentity.observationAsOf, + observationId: olderObservationId, + decisionId: createExecutionDecisionId({ observationId: olderObservationId }), + }) + const newer = sampleDecision({ observationAsOf: '2026-07-12' }) + await forceConcurrentLatestSnapshots(75) + + await Promise.all([writeExecutionDecision(root, older), writeExecutionDecision(root, newer)]) + + const latest = parseExecutionDecisionCsv(await readFile(join(root, 'hfmarkets', 'XAUUSD', 'latest_decision.csv'), 'utf8')) + expect(latest.observationAsOf).toBe('2026-07-12') + }) + + it('fails closed when another process owns the publication lock', async () => { + const root = await mkdtemp(join(tmpdir(), 'openalice-execution-decision-')) + directories.push(root) + const directory = join(root, 'hfmarkets', 'XAUUSD') + await mkdir(join(directory, '.publication.lock'), { recursive: true }) + + await expect(writeExecutionDecision(root, sampleDecision())).rejects.toThrow(/publication lock.*manual/i) + await expect(readdir(directory)).resolves.toEqual(['.publication.lock']) + }) + + it('validates the complete decision before writing any artifact', async () => { + const root = await mkdtemp(join(tmpdir(), 'openalice-execution-decision-')) + directories.push(root) + const invalid = { ...sampleDecision(), volume: 0.02 } as unknown as JmbExecutionDecision + + await expect(writeExecutionDecision(root, invalid)).rejects.toThrow(/volume/i) + await expect(readdir(join(root, 'hfmarkets', 'XAUUSD'))).resolves.toEqual([]) + }) + + it('atomically replaces latest and appends only materially changed evidence', async () => { + const root = await mkdtemp(join(tmpdir(), 'openalice-execution-decision-')) + directories.push(root) + const first = sampleDecision() + const renewed = sampleDecision({ + createdAt: '2026-07-13T09:05:00.000Z', + leaseIssuedAt: '2026-07-13T09:05:00.000Z', + leaseExpiresAt: '2026-07-13T09:15:00.000Z', + }) + const refreshedEvidence = sampleDecision({ + createdAt: '2026-07-13T09:05:00.000Z', + leaseIssuedAt: '2026-07-13T09:05:00.000Z', + leaseExpiresAt: '2026-07-13T09:15:00.000Z', + candidatePolicyVersion: 'hfm-v2', + }) + + await writeExecutionDecision(root, first) + await writeExecutionDecision(root, renewed) + await writeExecutionDecision(root, refreshedEvidence) + + const directory = join(root, 'hfmarkets', 'XAUUSD') + expect(parseExecutionDecisionCsv(await readFile(join(directory, 'latest_decision.csv'), 'utf8'))).toEqual(refreshedEvidence) + expect((await readFile(join(directory, 'decisions.jsonl'), 'utf8')).trim().split(/\r?\n/)).toHaveLength(2) + expect(await readdir(directory)).toEqual(['decisions.jsonl', 'latest_decision.csv']) + }) + + it('does not let an older observation replace or append after a newer lease', async () => { + const root = await mkdtemp(join(tmpdir(), 'openalice-execution-decision-')) + directories.push(root) + const newer = sampleDecision({ observationAsOf: '2026-07-12' }) + const olderIdentity = { + broker: newer.broker, + symbol: newer.symbol, + strategyVersion: newer.strategyVersion, + observationAsOf: '2026-07-11', + } + const olderObservationId = createObservationId(olderIdentity) + const older = sampleDecision({ + observationAsOf: olderIdentity.observationAsOf, + observationId: olderObservationId, + decisionId: createExecutionDecisionId({ observationId: olderObservationId }), + }) + + await writeExecutionDecision(root, newer) + await expect(writeExecutionDecision(root, older)).resolves.toMatchObject({ state: 'regressed' }) + + const directory = join(root, 'hfmarkets', 'XAUUSD') + expect(parseExecutionDecisionCsv(await readFile(join(directory, 'latest_decision.csv'), 'utf8'))).toEqual(newer) + expect((await readFile(join(directory, 'decisions.jsonl'), 'utf8')).trim().split(/\r?\n/)).toHaveLength(1) + }) +}) diff --git a/src/domain/mt5/execution-decision.ts b/src/domain/mt5/execution-decision.ts new file mode 100644 index 000000000..80f49eb0f --- /dev/null +++ b/src/domain/mt5/execution-decision.ts @@ -0,0 +1,330 @@ +import { createHash, randomUUID } from 'node:crypto' +import { appendFile, mkdir, readFile, rename, rm, rmdir, writeFile } from 'node:fs/promises' +import { dirname, join } from 'node:path' + +export interface JmbGateResult { + name: string + state: 'pass' | 'block' + detail: string +} + +export interface JmbExecutionDecision { + schemaVersion: 1 + decisionId: string + observationId: string + observationAsOf: string + createdAt: string + leaseIssuedAt: string + leaseExpiresAt: string + broker: 'hfmarkets' | 'icmarkets' + server: 'HFMarketsGlobal-Demo4' | 'ICMarketsSC-Demo' + accountMode: 'demo' + symbol: 'XAUUSD' + strategyVersion: 'daily-trend-v1' + direction: 'buy' | 'sell' | 'flat' + entryReferencePrice: number | null + volume: 0.01 + stopLoss: number | null + maxRiskAmount: number + candidatePolicyVersion: string + costModelVersion: string + gateResults: JmbGateResult[] +} + +export interface WriteExecutionDecisionResult { + state: 'published' | 'regressed' + journalAppended: boolean +} + +const HEADER = [ + 'schema_version', + 'decision_id', + 'observation_id', + 'observation_as_of', + 'created_at', + 'lease_issued_at', + 'lease_expires_at', + 'broker', + 'server', + 'account_mode', + 'symbol', + 'strategy_version', + 'direction', + 'entry_reference_price', + 'volume', + 'stop_loss', + 'max_risk_amount', + 'candidate_policy_version', + 'cost_model_version', + 'gate_results_json', +] as const + +const ISO_DATE = /^\d{4}-\d{2}-\d{2}$/ +const FORBIDDEN_TEXT = /[\r\n]/ +const PUBLICATION_LOCK_DIRECTORY = '.publication.lock' +const PUBLICATION_LOCK_TIMEOUT_MS = 1_000 +const PUBLICATION_LOCK_RETRY_MS = 20 + +function hash(parts: readonly string[]): string { + return createHash('sha256').update(parts.join('|')).digest('hex').slice(0, 24) +} + +export function createObservationId( + input: Pick, +): string { + return hash([input.broker, input.symbol, input.strategyVersion, input.observationAsOf]) +} + +export function createExecutionDecisionId(input: Pick): string { + return hash(['daily-trend-v1', input.observationId]) +} + +function csvEscape(value: string): string { + return /[",\r\n]/.test(value) ? `"${value.replaceAll('"', '""')}"` : value +} + +function parseCsvRecords(text: string): string[][] { + const records: string[][] = [] + let record: string[] = [] + let current = '' + let quoted = false + for (let index = 0; index < text.length; index += 1) { + const character = text[index] + if (character === '"') { + if (quoted && text[index + 1] === '"') { + current += '"' + index += 1 + } else { + quoted = !quoted + } + } else if (character === ',' && !quoted) { + record.push(current) + current = '' + } else if ((character === '\n' || character === '\r') && !quoted) { + record.push(current) + records.push(record) + record = [] + current = '' + if (character === '\r' && text[index + 1] === '\n') index += 1 + } else { + current += character + } + } + if (quoted) throw new Error('Execution decision CSV has an unterminated quoted field.') + if (current !== '' || record.length > 0) { + record.push(current) + records.push(record) + } + return records +} + +function requiredFinite(value: string, field: string): number { + const parsed = Number(value) + if (value.trim() === '' || !Number.isFinite(parsed)) throw new Error(`Execution decision ${field} must be finite.`) + return parsed +} + +function optionalFinite(value: string, field: string): number | null { + return value === '' ? null : requiredFinite(value, field) +} + +function requireTimestamp(value: string, field: string): number { + const parsed = Date.parse(value) + if (!Number.isFinite(parsed)) throw new Error(`Execution decision ${field} must be a valid timestamp.`) + return parsed +} + +function requireCanonicalText(value: string, field: string): void { + if (value === '' || value.trim() !== value || FORBIDDEN_TEXT.test(value)) { + throw new Error(`Execution decision ${field} must be non-empty canonical text.`) + } +} + +function validateGateResults(value: unknown): asserts value is JmbGateResult[] { + if (!Array.isArray(value) || value.length === 0) throw new Error('Execution decision gate results must be a non-empty array.') + for (const gate of value) { + if (typeof gate !== 'object' || gate === null) throw new Error('Execution decision gate result is malformed.') + const candidate = gate as Record + if (Object.keys(candidate).sort().join(',') !== 'detail,name,state' + || typeof candidate['name'] !== 'string' + || typeof candidate['detail'] !== 'string' + || (candidate['state'] !== 'pass' && candidate['state'] !== 'block')) { + throw new Error('Execution decision gate result has an invalid semantic value.') + } + requireCanonicalText(candidate['name'], 'gate name') + requireCanonicalText(candidate['detail'], 'gate detail') + } +} + +function validateDecision(decision: JmbExecutionDecision): void { + if (decision.schemaVersion !== 1) throw new Error('Execution decision schema version must be 1.') + if (!ISO_DATE.test(decision.observationAsOf) || !Number.isFinite(Date.parse(`${decision.observationAsOf}T00:00:00Z`))) { + throw new Error('Execution decision observation date is invalid.') + } + const createdAt = requireTimestamp(decision.createdAt, 'created_at') + const leaseIssuedAt = requireTimestamp(decision.leaseIssuedAt, 'lease_issued_at') + const leaseExpiresAt = requireTimestamp(decision.leaseExpiresAt, 'lease_expires_at') + if (createdAt > leaseIssuedAt || leaseIssuedAt >= leaseExpiresAt) throw new Error('Execution decision lease ordering is invalid.') + if (decision.broker !== 'hfmarkets' && decision.broker !== 'icmarkets') throw new Error('Execution decision broker is not allowlisted.') + const expectedServer = decision.broker === 'hfmarkets' ? 'HFMarketsGlobal-Demo4' : 'ICMarketsSC-Demo' + if (decision.server !== expectedServer) throw new Error('Execution decision server does not match the broker allowlist.') + if (decision.accountMode !== 'demo') throw new Error('Execution decision account mode must be demo.') + if (decision.symbol !== 'XAUUSD') throw new Error('Execution decision symbol must be XAUUSD.') + if (decision.strategyVersion !== 'daily-trend-v1') throw new Error('Execution decision strategy is not allowlisted.') + if (!['buy', 'sell', 'flat'].includes(decision.direction)) throw new Error('Execution decision direction is invalid.') + if (decision.volume !== 0.01) throw new Error('Execution decision volume must be exactly 0.01.') + if (decision.entryReferencePrice !== null && (!Number.isFinite(decision.entryReferencePrice) || decision.entryReferencePrice <= 0)) { + throw new Error('Execution decision entry reference price must be positive and finite.') + } + if (decision.stopLoss !== null && (!Number.isFinite(decision.stopLoss) || decision.stopLoss <= 0)) { + throw new Error('Execution decision stop loss must be positive and finite.') + } + if (!Number.isFinite(decision.maxRiskAmount) || decision.maxRiskAmount <= 0) throw new Error('Execution decision maximum risk must be positive and finite.') + requireCanonicalText(decision.candidatePolicyVersion, 'candidate policy version') + requireCanonicalText(decision.costModelVersion, 'cost model version') + validateGateResults(decision.gateResults) + const expectedObservationId = createObservationId(decision) + if (decision.observationId !== expectedObservationId) throw new Error('Execution decision observation ID does not match its immutable identity.') + if (decision.decisionId !== createExecutionDecisionId(decision)) throw new Error('Execution decision ID does not match its observation ID.') +} + +export function serializeExecutionDecisionCsv(decision: JmbExecutionDecision): string { + validateDecision(decision) + const values = [ + String(decision.schemaVersion), + decision.decisionId, + decision.observationId, + decision.observationAsOf, + decision.createdAt, + decision.leaseIssuedAt, + decision.leaseExpiresAt, + decision.broker, + decision.server, + decision.accountMode, + decision.symbol, + decision.strategyVersion, + decision.direction, + decision.entryReferencePrice === null ? '' : String(decision.entryReferencePrice), + String(decision.volume), + decision.stopLoss === null ? '' : String(decision.stopLoss), + String(decision.maxRiskAmount), + decision.candidatePolicyVersion, + decision.costModelVersion, + JSON.stringify(decision.gateResults), + ] + return `${HEADER.join(',')}\n${values.map(csvEscape).join(',')}\n` +} + +export function parseExecutionDecisionCsv(text: string): JmbExecutionDecision { + const records = parseCsvRecords(text) + if (records.length !== 2) throw new Error('Execution decision CSV must contain exactly one header and one row.') + const [headers, values] = records as [string[], string[]] + if (headers.join(',') !== HEADER.join(',') || values.length !== HEADER.length) throw new Error('Execution decision CSV schema mismatch.') + const row = Object.fromEntries(headers.map((header, index) => [header, values[index] ?? ''])) + let gateResults: unknown + try { + gateResults = JSON.parse(row['gate_results_json']!) + } catch { + throw new Error('Execution decision gate results JSON is malformed.') + } + validateGateResults(gateResults) + const decision: JmbExecutionDecision = { + schemaVersion: row['schema_version'] === '1' ? 1 : (Number(row['schema_version']) as 1), + decisionId: row['decision_id']!, + observationId: row['observation_id']!, + observationAsOf: row['observation_as_of']!, + createdAt: row['created_at']!, + leaseIssuedAt: row['lease_issued_at']!, + leaseExpiresAt: row['lease_expires_at']!, + broker: row['broker'] as JmbExecutionDecision['broker'], + server: row['server'] as JmbExecutionDecision['server'], + accountMode: row['account_mode'] as 'demo', + symbol: row['symbol'] as 'XAUUSD', + strategyVersion: row['strategy_version'] as 'daily-trend-v1', + direction: row['direction'] as JmbExecutionDecision['direction'], + entryReferencePrice: optionalFinite(row['entry_reference_price']!, 'entry_reference_price'), + volume: requiredFinite(row['volume']!, 'volume') as 0.01, + stopLoss: optionalFinite(row['stop_loss']!, 'stop_loss'), + maxRiskAmount: requiredFinite(row['max_risk_amount']!, 'max_risk_amount'), + candidatePolicyVersion: row['candidate_policy_version']!, + costModelVersion: row['cost_model_version']!, + gateResults, + } + validateDecision(decision) + return decision +} + +function materialEvidence(decision: JmbExecutionDecision): string { + const { createdAt: _createdAt, leaseIssuedAt: _leaseIssuedAt, leaseExpiresAt: _leaseExpiresAt, ...evidence } = decision + return JSON.stringify(evidence) +} + +function fileCode(error: unknown): string | null { + return typeof error === 'object' && error !== null && 'code' in error ? String((error as { code: unknown }).code) : null +} + +async function readLatest(path: string): Promise { + try { + return parseExecutionDecisionCsv(await readFile(path, 'utf8')) + } catch (error) { + if (fileCode(error) === 'ENOENT') return null + throw error + } +} + +async function replaceAtomically(destination: string, contents: string): Promise { + const temporary = `${destination}.${process.pid}.${randomUUID()}.tmp` + await writeFile(temporary, contents, { encoding: 'utf8', flag: 'wx' }) + try { + await rename(temporary, destination) + } catch (error) { + await rm(temporary, { force: true }) + throw error + } +} + +function wait(milliseconds: number): Promise { + return new Promise((resolve) => setTimeout(resolve, milliseconds)) +} + +async function acquirePublicationLock(directory: string): Promise<() => Promise> { + const lockPath = join(directory, PUBLICATION_LOCK_DIRECTORY) + const deadline = Date.now() + PUBLICATION_LOCK_TIMEOUT_MS + while (true) { + try { + await mkdir(lockPath) + return async () => rmdir(lockPath) + } catch (error) { + if (fileCode(error) !== 'EEXIST') { + throw new Error(`Execution decision publication lock could not be acquired at ${lockPath}: ${error instanceof Error ? error.message : 'unknown filesystem error'}`) + } + if (Date.now() >= deadline) { + throw new Error(`Execution decision publication lock timed out at ${lockPath}. Another process may still own it; stop the owner and remove the lock directory manually before retrying.`) + } + await wait(PUBLICATION_LOCK_RETRY_MS) + } + } +} + +export async function writeExecutionDecision(root: string, decision: JmbExecutionDecision): Promise { + const destination = join(root, decision.broker, decision.symbol, 'latest_decision.csv') + await mkdir(dirname(destination), { recursive: true }) + const serialized = serializeExecutionDecisionCsv(decision) + const releaseLock = await acquirePublicationLock(dirname(destination)) + try { + const latest = await readLatest(destination) + if (latest !== null && (decision.observationAsOf < latest.observationAsOf + || (decision.observationAsOf === latest.observationAsOf && Date.parse(decision.leaseIssuedAt) < Date.parse(latest.leaseIssuedAt)))) { + return { state: 'regressed', journalAppended: false } + } + + const journalAppended = latest === null || materialEvidence(latest) !== materialEvidence(decision) + if (journalAppended) { + await appendFile(join(dirname(destination), 'decisions.jsonl'), `${JSON.stringify(decision)}\n`, 'utf8') + } + await replaceAtomically(destination, serialized) + return { state: 'published', journalAppended } + } finally { + await releaseLock() + } +} diff --git a/src/domain/mt5/execution-outcomes.spec.ts b/src/domain/mt5/execution-outcomes.spec.ts new file mode 100644 index 000000000..dd1ecd66b --- /dev/null +++ b/src/domain/mt5/execution-outcomes.spec.ts @@ -0,0 +1,209 @@ +import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' + +import { afterEach, beforeEach, describe, expect, it } from 'vitest' + +import { + appendOutcomeOnce, + executionEventToOutcome, + parseExecutionEventJsonLine, + readExecutionLearningRecords, + type JmbExecutionOutcomeRecord, +} from './execution-outcomes.js' + +const terminalEvent = { + schema_version: 1, + event_id: 'event-closed-1', + event_type: 'closed', + event_time: '2026-07-13T09:10:00.000Z', + broker: 'hfmarkets', + server: 'HFMarketsGlobal-Demo4', + account_mode: 'demo', + account_identity_masked: 'masked-opaque', + symbol: 'XAUUSD', + strategy_version: 'daily-trend-v1', + magic_number: 880101, + decision_id: 'decision-1', + observation_id: 'observation-1', + gate_results: [], + calculated_risk: 10, + requested_volume: 0.01, + requested_price: 3334.25, + requested_stop_loss: 3324.25, + accepted_volume: 0.01, + accepted_price: 3334.5, + accepted_stop_loss: 3324.25, + result_code: '10009', + result_detail: 'Position closed and reconciled.', + order_ticket: 'order-1', + deal_ticket: 'deal-1', + position_id: 'position-1', + reconciliation_state: 'reconciled', + daily_loss_count: 1, + daily_realized_loss: 6.25, + commission: -0.5, + swap: -0.25, + fee: 0, + net_result: -6.25, + max_adverse_excursion: null, + max_favorable_excursion: 2.5, +} as const + +const outcome: JmbExecutionOutcomeRecord = { + schemaVersion: 1, + outcomeEventId: 'event-closed-1', + outcomeAt: '2026-07-13T09:10:00.000Z', + broker: 'hfmarkets', + server: 'HFMarketsGlobal-Demo4', + accountMode: 'demo', + symbol: 'XAUUSD', + strategyVersion: 'daily-trend-v1', + decisionId: 'decision-1', + observationId: 'observation-1', + positionId: 'position-1', + result: 'loss', + netResult: -6.25, + commission: -0.5, + swap: -0.25, + fee: 0, + requestedPrice: 3334.25, + acceptedPrice: 3334.5, + slippage: 0.25, + maxAdverseExcursion: null, + maxFavorableExcursion: 2.5, + source: 'ea_demo', +} + +describe('MT5 execution outcomes', () => { + let root: string + + beforeEach(async () => { + root = await mkdtemp(join(tmpdir(), 'openalice-outcomes-')) + }) + + afterEach(async () => { + await rm(root, { recursive: true, force: true }) + }) + + it('converts a fully reconciled terminal event into immutable evidence', () => { + const source = structuredClone(terminalEvent) + const parsed = parseExecutionEventJsonLine(JSON.stringify(source)) + + expect(executionEventToOutcome(parsed)).toEqual(outcome) + expect(source).toEqual(terminalEvent) + }) + + it.each([ + ['a non-demo account', { account_mode: 'real' }], + ['a non-Gold symbol', { symbol: 'EURUSD' }], + ['an unknown lifecycle', { event_type: 'profitable' }], + ['a missing event identifier', { event_id: '' }], + ['a missing decision identifier', { decision_id: '' }], + ['a missing observation identifier', { observation_id: '' }], + ['a missing position identifier', { position_id: '' }], + ['a non-finite terminal money field', { net_result: null }], + ])('fails closed on %s', (_case, replacement) => { + expect(() => parseExecutionEventJsonLine(JSON.stringify({ ...terminalEvent, ...replacement }))).toThrow() + }) + + it('fails closed when a nonterminal physical line lacks correlation identifiers', () => { + expect(() => parseExecutionEventJsonLine(JSON.stringify({ + ...terminalEvent, + event_type: 'order_requesting', + decision_id: '', + }))).toThrow(/decision_id|identifier/i) + }) + + it.each([ + ['null', null], + ['a scalar', 'pass'], + ['a missing key', { name: 'identity', state: 'pass' }], + ['an unknown key', { name: 'identity', state: 'pass', detail: 'Matched.', approval: true }], + ['an unknown state', { name: 'identity', state: 'warn', detail: 'Matched.' }], + ['an empty name', { name: '', state: 'pass', detail: 'Matched.' }], + ['an empty detail', { name: 'identity', state: 'pass', detail: '' }], + ])('rejects gate_results containing %s', (_case, gate) => { + expect(() => parseExecutionEventJsonLine(JSON.stringify({ + ...terminalEvent, + gate_results: [gate], + }))).toThrow(/gate/i) + }) + + it('does not convert nonterminal or unreconciled events', () => { + const unresolved = parseExecutionEventJsonLine(JSON.stringify({ + ...terminalEvent, + event_type: 'reconciliation_required', + reconciliation_state: 'required', + commission: null, + swap: null, + fee: null, + net_result: null, + })) + const unreconciled = parseExecutionEventJsonLine(JSON.stringify({ + ...terminalEvent, + reconciliation_state: 'required', + })) + + expect(executionEventToOutcome(unresolved)).toBeNull() + expect(executionEventToOutcome(unreconciled)).toBeNull() + }) + + it('appends an outcome once and writes an evidence-only summary', async () => { + await expect(appendOutcomeOnce(root, outcome)).resolves.toBe(true) + await expect(appendOutcomeOnce(root, outcome)).resolves.toBe(false) + + const records = await readExecutionLearningRecords(root, 'hfmarkets', 'XAUUSD') + const summary = JSON.parse(await readFile(join(root, 'hfmarkets', 'XAUUSD', 'summary.json'), 'utf8')) as Record + + expect(records).toEqual([outcome]) + expect(summary).toMatchObject({ + schemaVersion: 1, + count: 1, + totalNetResult: -6.25, + winCount: 0, + lossCount: 1, + breakevenCount: 0, + totalCommission: -0.5, + totalSwap: -0.25, + totalFee: 0, + averageSlippage: 0.25, + latestOutcomeAt: outcome.outcomeAt, + }) + expect(JSON.stringify(summary)).not.toMatch(/approval|risk.?limit|strategy.?parameter|profit.?prediction/i) + }) + + it.each(['unexpected', 'approval', 'riskLimit', 'strategyParameter', 'profitPrediction'])( + 'rejects the extra or prohibited outcome key %s', + async (key) => { + const contaminated = { ...outcome, [key]: true } + + await expect(appendOutcomeOnce(root, contaminated)).rejects.toThrow(/contract|field|key/i) + await expect(readExecutionLearningRecords(root, 'hfmarkets', 'XAUUSD')).resolves.toEqual([]) + }, + ) + + it('rejects a mismatched outcome broker and server pair', async () => { + const mismatched = { ...outcome, server: 'ICMarketsSC-Demo' } as JmbExecutionOutcomeRecord + + await expect(appendOutcomeOnce(root, mismatched)).rejects.toThrow(/server|broker/i) + }) + + it('rejects a result that is inconsistent with netResult', async () => { + const inconsistent = { ...outcome, result: 'win' } as JmbExecutionOutcomeRecord + + await expect(appendOutcomeOnce(root, inconsistent)).rejects.toThrow(/result/i) + }) + + it.each(['missing', 'stale'])('repairs a %s summary when the outcome is already journaled', async (state) => { + await appendOutcomeOnce(root, outcome) + const summaryPath = join(root, 'hfmarkets', 'XAUUSD', 'summary.json') + if (state === 'missing') await rm(summaryPath) + else await writeFile(summaryPath, '{"schemaVersion":1,"count":999}\n', 'utf8') + + await expect(appendOutcomeOnce(root, outcome)).resolves.toBe(false) + + const summary = JSON.parse(await readFile(summaryPath, 'utf8')) as Record + expect(summary).toMatchObject({ count: 1, totalNetResult: -6.25, lossCount: 1 }) + }) +}) diff --git a/src/domain/mt5/execution-outcomes.ts b/src/domain/mt5/execution-outcomes.ts new file mode 100644 index 000000000..225b972a0 --- /dev/null +++ b/src/domain/mt5/execution-outcomes.ts @@ -0,0 +1,494 @@ +import { open, mkdir, readFile, rename, rm, writeFile } from 'node:fs/promises' +import { dirname, join } from 'node:path' +import { randomUUID } from 'node:crypto' + +import type { JmbDemoInstrumentConfig } from './demo-decision-service.js' + +export interface JmbExecutionOutcomeRecord { + schemaVersion: 1 + outcomeEventId: string + outcomeAt: string + broker: 'hfmarkets' | 'icmarkets' + server: 'HFMarketsGlobal-Demo4' | 'ICMarketsSC-Demo' + accountMode: 'demo' + symbol: 'XAUUSD' + strategyVersion: 'daily-trend-v1' + decisionId: string + observationId: string + positionId: string + result: 'win' | 'loss' | 'breakeven' + netResult: number + commission: number + swap: number + fee: number + requestedPrice: number | null + acceptedPrice: number | null + slippage: number | null + maxAdverseExcursion: number | null + maxFavorableExcursion: number | null + source: 'ea_demo' +} + +export interface ExecutionOutcomeImportOptions { + executionRoot: string + learningRoot: string + instruments: readonly Pick[] +} + +export interface ExecutionOutcomeImportResult { + broker: 'hfmarkets' | 'icmarkets' + symbol: 'XAUUSD' + state: 'imported' | 'no_new_outcome' | 'blocked' | 'error' + imported: number + detail: string +} + +const EXECUTION_EVENT_FIELDS = [ + 'schema_version', 'event_id', 'event_type', 'event_time', 'broker', 'server', 'account_mode', + 'account_identity_masked', 'symbol', 'strategy_version', 'magic_number', 'decision_id', 'observation_id', + 'gate_results', 'calculated_risk', 'requested_volume', 'requested_price', 'requested_stop_loss', + 'accepted_volume', 'accepted_price', 'accepted_stop_loss', 'result_code', 'result_detail', 'order_ticket', + 'deal_ticket', 'position_id', 'reconciliation_state', 'daily_loss_count', 'daily_realized_loss', + 'commission', 'swap', 'fee', 'net_result', 'max_adverse_excursion', 'max_favorable_excursion', +] as const + +const OUTCOME_RECORD_FIELDS = [ + 'schemaVersion', 'outcomeEventId', 'outcomeAt', 'broker', 'server', 'accountMode', 'symbol', + 'strategyVersion', 'decisionId', 'observationId', 'positionId', 'result', 'netResult', 'commission', + 'swap', 'fee', 'requestedPrice', 'acceptedPrice', 'slippage', 'maxAdverseExcursion', + 'maxFavorableExcursion', 'source', +] as const + +const LIFECYCLE_STATES = new Set([ + 'disabled', 'paused', 'blocked', 'ready', 'order_requesting', 'order_rejected', + 'reconciliation_required', 'filled_protected', 'close_requesting', 'closed', 'stopped', + 'emergency_close', 'error', +]) + +type Broker = JmbExecutionOutcomeRecord['broker'] +type Server = JmbExecutionOutcomeRecord['server'] + +export interface JmbExecutionEvent { + schemaVersion: 1 + eventId: string + eventType: string + eventTime: string + broker: Broker + server: Server + accountMode: 'demo' + symbol: 'XAUUSD' + strategyVersion: 'daily-trend-v1' + decisionId: string + observationId: string + positionId: string + reconciliationState: string + requestedPrice: number | null + acceptedPrice: number | null + commission: number | null + swap: number | null + fee: number | null + netResult: number | null + maxAdverseExcursion: number | null + maxFavorableExcursion: number | null +} + +export class ExecutionOutcomeValidationError extends Error { + constructor(message: string) { + super(message) + this.name = 'ExecutionOutcomeValidationError' + } +} + +function validationError(message: string): never { + throw new ExecutionOutcomeValidationError(message) +} + +function objectValue(value: unknown): Record { + if (typeof value !== 'object' || value === null || Array.isArray(value)) { + validationError('Execution event must be a JSON object.') + } + return value as Record +} + +function stringValue(value: unknown, field: string, allowEmpty = false): string { + if (typeof value !== 'string' || (!allowEmpty && value.trim() === '')) { + validationError(`Execution event ${field} must be a ${allowEmpty ? '' : 'nonempty '}string.`) + } + return value +} + +function finiteValue(value: unknown, field: string): number { + if (typeof value !== 'number' || !Number.isFinite(value)) { + validationError(`Execution event ${field} must be finite.`) + } + return value +} + +function nullableFiniteValue(value: unknown, field: string): number | null { + if (value === null) return null + return finiteValue(value, field) +} + +function assertExactFields(value: Record): void { + const actual = Object.keys(value).sort() + const expected = [...EXECUTION_EVENT_FIELDS].sort() + if (actual.length !== expected.length || !actual.every((field, index) => field === expected[index])) { + validationError('Execution event schema fields do not match version 1.') + } +} + +function validateGateResults(value: unknown): void { + if (!Array.isArray(value)) validationError('Execution event gate_results must be an array.') + for (const gate of value) { + if (typeof gate !== 'object' || gate === null || Array.isArray(gate)) { + validationError('Execution event gate result must be an object.') + } + const candidate = gate as Record + if (Object.keys(candidate).sort().join(',') !== 'detail,name,state') { + validationError('Execution event gate result fields do not match the Task 8 contract.') + } + const name = stringValue(candidate['name'], 'gate name') + const detail = stringValue(candidate['detail'], 'gate detail') + if (name.trim() !== name || detail.trim() !== detail + || (candidate['state'] !== 'pass' && candidate['state'] !== 'block')) { + validationError('Execution event gate result has an invalid semantic value.') + } + } +} + +export function parseExecutionEventJsonLine(line: string): JmbExecutionEvent { + let decoded: unknown + try { + decoded = JSON.parse(line) + } catch { + validationError('Execution event line is not valid JSON.') + } + const value = objectValue(decoded) + assertExactFields(value) + + if (value['schema_version'] !== 1) validationError('Execution event schema_version must be 1.') + const eventId = stringValue(value['event_id'], 'event_id') + const eventType = stringValue(value['event_type'], 'event_type') + if (!LIFECYCLE_STATES.has(eventType)) validationError('Execution event lifecycle is not allowlisted.') + const eventTime = stringValue(value['event_time'], 'event_time') + if (!Number.isFinite(Date.parse(eventTime))) validationError('Execution event event_time is invalid.') + + const broker = stringValue(value['broker'], 'broker') + if (broker !== 'hfmarkets' && broker !== 'icmarkets') validationError('Execution event broker is not allowlisted.') + const server = stringValue(value['server'], 'server') + const expectedServer = broker === 'hfmarkets' ? 'HFMarketsGlobal-Demo4' : 'ICMarketsSC-Demo' + if (server !== expectedServer) validationError('Execution event server does not match the broker demo server.') + if (value['account_mode'] !== 'demo') validationError('Execution event must come from a demo account.') + stringValue(value['account_identity_masked'], 'account_identity_masked') + if (value['symbol'] !== 'XAUUSD') validationError('Execution event must describe XAUUSD.') + if (value['strategy_version'] !== 'daily-trend-v1') validationError('Execution event strategy_version is invalid.') + const expectedMagicNumber = broker === 'hfmarkets' ? 880101 : 880201 + if (value['magic_number'] !== expectedMagicNumber) validationError('Execution event magic_number does not match the broker.') + + const decisionId = stringValue(value['decision_id'], 'decision_id') + const observationId = stringValue(value['observation_id'], 'observation_id') + validateGateResults(value['gate_results']) + nullableFiniteValue(value['calculated_risk'], 'calculated_risk') + nullableFiniteValue(value['requested_volume'], 'requested_volume') + const requestedPrice = nullableFiniteValue(value['requested_price'], 'requested_price') + nullableFiniteValue(value['requested_stop_loss'], 'requested_stop_loss') + nullableFiniteValue(value['accepted_volume'], 'accepted_volume') + const acceptedPrice = nullableFiniteValue(value['accepted_price'], 'accepted_price') + nullableFiniteValue(value['accepted_stop_loss'], 'accepted_stop_loss') + stringValue(value['result_code'], 'result_code', true) + stringValue(value['result_detail'], 'result_detail', true) + stringValue(value['order_ticket'], 'order_ticket', true) + stringValue(value['deal_ticket'], 'deal_ticket', true) + const positionId = stringValue(value['position_id'], 'position_id', true) + const reconciliationState = stringValue(value['reconciliation_state'], 'reconciliation_state') + const dailyLossCount = finiteValue(value['daily_loss_count'], 'daily_loss_count') + if (!Number.isInteger(dailyLossCount) || dailyLossCount < 0) { + validationError('Execution event daily_loss_count must be a non-negative integer.') + } + finiteValue(value['daily_realized_loss'], 'daily_realized_loss') + const commission = nullableFiniteValue(value['commission'], 'commission') + const swap = nullableFiniteValue(value['swap'], 'swap') + const fee = nullableFiniteValue(value['fee'], 'fee') + const netResult = nullableFiniteValue(value['net_result'], 'net_result') + const maxAdverseExcursion = nullableFiniteValue(value['max_adverse_excursion'], 'max_adverse_excursion') + const maxFavorableExcursion = nullableFiniteValue(value['max_favorable_excursion'], 'max_favorable_excursion') + + if (eventType === 'closed' || eventType === 'stopped') { + if (!positionId) { + validationError('Terminal execution event correlation identifiers are required.') + } + if (commission === null || swap === null || fee === null || netResult === null) { + validationError('Terminal execution event money fields must be finite.') + } + } + + return { + schemaVersion: 1, + eventId, + eventType, + eventTime, + broker, + server, + accountMode: 'demo', + symbol: 'XAUUSD', + strategyVersion: 'daily-trend-v1', + decisionId, + observationId, + positionId, + reconciliationState, + requestedPrice, + acceptedPrice, + commission, + swap, + fee, + netResult, + maxAdverseExcursion, + maxFavorableExcursion, + } +} + +export function executionEventToOutcome(event: JmbExecutionEvent): JmbExecutionOutcomeRecord | null { + if ((event.eventType !== 'closed' && event.eventType !== 'stopped') || event.reconciliationState !== 'reconciled') { + return null + } + if (event.commission === null || event.swap === null || event.fee === null || event.netResult === null) { + validationError('Reconciled terminal execution event is missing finite outcome money fields.') + } + + return { + schemaVersion: 1, + outcomeEventId: event.eventId, + outcomeAt: event.eventTime, + broker: event.broker, + server: event.server, + accountMode: 'demo', + symbol: 'XAUUSD', + strategyVersion: 'daily-trend-v1', + decisionId: event.decisionId, + observationId: event.observationId, + positionId: event.positionId, + result: event.netResult > 0 ? 'win' : event.netResult < 0 ? 'loss' : 'breakeven', + netResult: event.netResult, + commission: event.commission, + swap: event.swap, + fee: event.fee, + requestedPrice: event.requestedPrice, + acceptedPrice: event.acceptedPrice, + slippage: event.requestedPrice === null || event.acceptedPrice === null + ? null + : event.acceptedPrice - event.requestedPrice, + maxAdverseExcursion: event.maxAdverseExcursion, + maxFavorableExcursion: event.maxFavorableExcursion, + source: 'ea_demo', + } +} + +function physicalJsonLines(text: string, label: string): string[] { + if (text === '') return [] + const lines = text.split('\n') + if (lines.at(-1) === '') lines.pop() + return lines.map((physicalLine, index) => { + const line = physicalLine.endsWith('\r') ? physicalLine.slice(0, -1) : physicalLine + if (line === '') validationError(`${label} physical line ${index + 1} is empty.`) + return line + }) +} + +export async function readExecutionEvents( + executionRoot: string, + broker: Broker, + symbol: 'XAUUSD', +): Promise { + const text = await readFile(join(executionRoot, broker, symbol, 'events.jsonl'), 'utf8') + return physicalJsonLines(text, 'Execution event').map(parseExecutionEventJsonLine) +} + +function learningDirectory(root: string, broker: Broker, symbol: 'XAUUSD'): string { + return join(root, broker, symbol) +} + +export function outcomeJournalPath(root: string, broker: Broker, symbol: 'XAUUSD'): string { + return join(learningDirectory(root, broker, symbol), 'outcomes.jsonl') +} + +export function outcomeSummaryPath(root: string, broker: Broker, symbol: 'XAUUSD'): string { + return join(learningDirectory(root, broker, symbol), 'summary.json') +} + +function validateOutcomeRecord(value: unknown): JmbExecutionOutcomeRecord { + const candidate = objectValue(value) + const actualFields = Reflect.ownKeys(candidate) + if (actualFields.some((field) => typeof field !== 'string') + || actualFields.map(String).sort().join(',') !== [...OUTCOME_RECORD_FIELDS].sort().join(',')) { + validationError('Execution learning record fields do not match the exact outcome contract.') + } + + const broker = candidate['broker'] + const server = candidate['server'] + const result = candidate['result'] + if (candidate['schemaVersion'] !== 1 + || (broker !== 'hfmarkets' && broker !== 'icmarkets') + || (server !== 'HFMarketsGlobal-Demo4' && server !== 'ICMarketsSC-Demo') + || candidate['accountMode'] !== 'demo' + || candidate['symbol'] !== 'XAUUSD' + || candidate['strategyVersion'] !== 'daily-trend-v1' + || candidate['source'] !== 'ea_demo' + || (result !== 'win' && result !== 'loss' && result !== 'breakeven')) { + validationError('Execution learning record contract is invalid.') + } + if (server !== (broker === 'hfmarkets' ? 'HFMarketsGlobal-Demo4' : 'ICMarketsSC-Demo')) { + validationError('Execution learning broker and server pair is inconsistent.') + } + const outcomeEventId = stringValue(candidate['outcomeEventId'], 'outcomeEventId') + const outcomeAt = stringValue(candidate['outcomeAt'], 'outcomeAt') + const decisionId = stringValue(candidate['decisionId'], 'decisionId') + const observationId = stringValue(candidate['observationId'], 'observationId') + const positionId = stringValue(candidate['positionId'], 'positionId') + for (const [field, item] of [ + ['outcomeEventId', outcomeEventId], ['outcomeAt', outcomeAt], ['decisionId', decisionId], + ['observationId', observationId], ['positionId', positionId], + ] as const) { + if (item.trim() !== item) validationError(`Execution learning ${field} must be canonical text.`) + } + if (!Number.isFinite(Date.parse(outcomeAt))) validationError('Execution learning outcomeAt is invalid.') + const netResult = finiteValue(candidate['netResult'], 'netResult') + const commission = finiteValue(candidate['commission'], 'commission') + const swap = finiteValue(candidate['swap'], 'swap') + const fee = finiteValue(candidate['fee'], 'fee') + const requestedPrice = nullableFiniteValue(candidate['requestedPrice'], 'requestedPrice') + const acceptedPrice = nullableFiniteValue(candidate['acceptedPrice'], 'acceptedPrice') + const slippage = nullableFiniteValue(candidate['slippage'], 'slippage') + const maxAdverseExcursion = nullableFiniteValue(candidate['maxAdverseExcursion'], 'maxAdverseExcursion') + const maxFavorableExcursion = nullableFiniteValue(candidate['maxFavorableExcursion'], 'maxFavorableExcursion') + const expectedResult = netResult > 0 ? 'win' : netResult < 0 ? 'loss' : 'breakeven' + if (result !== expectedResult) validationError('Execution learning result is inconsistent with netResult.') + const expectedSlippage = requestedPrice === null || acceptedPrice === null + ? null + : acceptedPrice - requestedPrice + if (slippage !== expectedSlippage) { + validationError('Execution learning slippage is inconsistent with requested and accepted prices.') + } + + return { + schemaVersion: 1, + outcomeEventId, + outcomeAt, + broker, + server, + accountMode: 'demo', + symbol: 'XAUUSD', + strategyVersion: 'daily-trend-v1', + decisionId, + observationId, + positionId, + result, + netResult, + commission, + swap, + fee, + requestedPrice, + acceptedPrice, + slippage, + maxAdverseExcursion, + maxFavorableExcursion, + source: 'ea_demo', + } +} + +export async function readExecutionLearningRecords( + root: string, + broker: Broker, + symbol: 'XAUUSD', +): Promise { + let text: string + try { + text = await readFile(outcomeJournalPath(root, broker, symbol), 'utf8') + } catch (error) { + if (typeof error === 'object' && error !== null && 'code' in error && error.code === 'ENOENT') return [] + throw error + } + return physicalJsonLines(text, 'Execution learning record').map((line) => { + try { + return validateOutcomeRecord(JSON.parse(line)) + } catch (error) { + if (error instanceof ExecutionOutcomeValidationError) throw error + validationError('Execution learning record line is not valid JSON.') + } + }) +} + +function summarizeOutcomes(records: readonly JmbExecutionOutcomeRecord[]): Record { + const slippages = records.flatMap((record) => record.slippage === null ? [] : [record.slippage]) + const latest = records.reduce((current, record) => ( + current === null || Date.parse(record.outcomeAt) > Date.parse(current) ? record.outcomeAt : current + ), null) + return { + schemaVersion: 1, + evidenceOnly: true, + count: records.length, + totalNetResult: records.reduce((total, record) => total + record.netResult, 0), + winCount: records.filter((record) => record.result === 'win').length, + lossCount: records.filter((record) => record.result === 'loss').length, + breakevenCount: records.filter((record) => record.result === 'breakeven').length, + totalCommission: records.reduce((total, record) => total + record.commission, 0), + totalSwap: records.reduce((total, record) => total + record.swap, 0), + totalFee: records.reduce((total, record) => total + record.fee, 0), + averageSlippage: slippages.length === 0 ? null : slippages.reduce((total, value) => total + value, 0) / slippages.length, + latestOutcomeAt: latest, + } +} + +async function appendDurableJsonLine(path: string, value: unknown): Promise { + await mkdir(dirname(path), { recursive: true }) + const handle = await open(path, 'a') + try { + await handle.appendFile(`${JSON.stringify(value)}\n`, 'utf8') + await handle.sync() + } finally { + await handle.close() + } +} + +async function writeJsonAtomically(path: string, value: unknown): Promise { + await mkdir(dirname(path), { recursive: true }) + const temporaryPath = `${path}.${process.pid}.${randomUUID()}.tmp` + try { + await writeFile(temporaryPath, `${JSON.stringify(value, null, 2)}\n`, { encoding: 'utf8', flag: 'wx' }) + await rename(temporaryPath, path) + } finally { + await rm(temporaryPath, { force: true }) + } +} + +const appendLocks = new Map>() + +async function withAppendLock(key: string, operation: () => Promise): Promise { + const previous = appendLocks.get(key) ?? Promise.resolve() + let release!: () => void + const gate = new Promise((resolve) => { release = resolve }) + const current = previous.then(() => gate) + appendLocks.set(key, current) + await previous + try { + return await operation() + } finally { + release() + if (appendLocks.get(key) === current) appendLocks.delete(key) + } +} + +export async function appendOutcomeOnce(root: string, record: JmbExecutionOutcomeRecord): Promise { + const normalizedRecord = validateOutcomeRecord(record) + const journalPath = outcomeJournalPath(root, normalizedRecord.broker, normalizedRecord.symbol) + return withAppendLock(journalPath, async () => { + const records = await readExecutionLearningRecords(root, normalizedRecord.broker, normalizedRecord.symbol) + const alreadyJournaled = records.some((item) => item.outcomeEventId === normalizedRecord.outcomeEventId) + const authoritativeRecords = alreadyJournaled ? records : [...records, normalizedRecord] + if (!alreadyJournaled) await appendDurableJsonLine(journalPath, normalizedRecord) + await writeJsonAtomically( + outcomeSummaryPath(root, normalizedRecord.broker, normalizedRecord.symbol), + summarizeOutcomes(authoritativeRecords), + ) + return !alreadyJournaled + }) +} diff --git a/src/domain/mt5/execution-status.spec.ts b/src/domain/mt5/execution-status.spec.ts new file mode 100644 index 000000000..d345fe981 --- /dev/null +++ b/src/domain/mt5/execution-status.spec.ts @@ -0,0 +1,243 @@ +import { mkdir, mkdtemp, rm, utimes, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { afterEach, beforeEach, describe, expect, it } from 'vitest' +import { parseExecutionStatusCsv, summarizeLatestJmbExecutionStatus } from './execution-status.js' + +const HEADER = 'schema_version,captured_at,broker,server,account_mode,symbol,state,detail,rollout_stage,execution_enabled,kill_switch,decision_id,observation_id,event_id,event_type,event_time,result_code,result_detail,stop_protection_confirmed,position_direction,position_volume,position_open_price,position_stop_loss,position_id,reconciliation_state,daily_loss_count,daily_realized_loss,blocking_gate,next_safe_action' + +const VALID_ROW = [ + '1', + '2026-07-13T09:10:00.000Z', + 'hfmarkets', + 'HFMarketsGlobal-Demo4', + 'demo', + 'XAUUSD', + 'filled_protected', + 'Broker confirms the EA-owned position and protective stop.', + 'hfm_canary', + '1', + '0', + 'decision-opaque', + 'observation-opaque', + 'event-opaque', + 'fill_confirmed', + '2026-07-13T09:09:58.000Z', + '10009', + 'Request completed', + '1', + 'buy', + '0.01', + '3334.25', + '3324.25', + 'position-opaque', + 'reconciled', + '1', + '-8.75', + '', + 'Monitor broker-side protection and reconciliation.', +].join(',') + +const VALID_CSV = `${HEADER}\n${VALID_ROW}\n` + +describe('parseExecutionStatusCsv', () => { + it('parses the exact execution status contract without an account identifier', () => { + const status = parseExecutionStatusCsv(VALID_CSV) + + expect(status.state).toBe('filled_protected') + expect(status.position).toEqual({ + direction: 'buy', + volume: 0.01, + openPrice: 3334.25, + stopLoss: 3324.25, + id: 'position-opaque', + }) + expect(status.latestEvent).toEqual({ + id: 'event-opaque', + type: 'fill_confirmed', + at: '2026-07-13T09:09:58.000Z', + resultCode: '10009', + detail: 'Request completed', + }) + expect(JSON.stringify(status)).not.toMatch(/account.?login/i) + }) + + it.each([ + ['extra', VALID_CSV.replace('symbol,', 'account_login,symbol,')], + ['missing', VALID_CSV.replace(',result_detail', '')], + ['duplicate', VALID_CSV.replace('result_code,result_detail', 'result_code,result_code,result_detail')], + ['reordered', VALID_CSV.replace('broker,server', 'server,broker')], + ])('rejects a %s column contract', (_case, csv) => { + expect(() => parseExecutionStatusCsv(csv)).toThrow(/schema/i) + }) + + it.each([ + ['a live account', VALID_CSV.replace(',demo,XAUUSD,', ',real,XAUUSD,')], + ['an unknown lifecycle', VALID_CSV.replace(',filled_protected,', ',profitable,')], + ['a non-binary switch', VALID_CSV.replace(',hfm_canary,1,0,', ',hfm_canary,true,0,')], + ['an incomplete position', VALID_CSV.replace(',buy,0.01,', ',,0.01,')], + ])('rejects %s', (_case, csv) => { + expect(() => parseExecutionStatusCsv(csv)).toThrow() + }) + + it('accepts an in-progress event before broker result fields exist', () => { + const csv = VALID_CSV + .replace(',filled_protected,', ',order_requesting,') + .replace( + ',2026-07-13T09:09:58.000Z,10009,Request completed,1,buy,0.01,3334.25,3324.25,position-opaque,reconciled,', + ',2026-07-13T09:09:58.000Z,,,0,,,,,,reconciliation_pending,', + ) + + expect(parseExecutionStatusCsv(csv)).toMatchObject({ + state: 'order_requesting', + latestEvent: { type: 'fill_confirmed', resultCode: '', detail: '' }, + position: null, + }) + }) + + it.each([ + ['stop confirmation off', VALID_CSV.replace(',Request completed,1,buy,', ',Request completed,0,buy,')], + ['no position', VALID_CSV.replace(',1,buy,0.01,3334.25,3324.25,position-opaque,reconciled,', ',1,,,,,,reconciled,')], + ['zero volume', VALID_CSV.replace(',buy,0.01,', ',buy,0,')], + ['negative volume', VALID_CSV.replace(',buy,0.01,', ',buy,-0.01,')], + ['zero open price', VALID_CSV.replace(',0.01,3334.25,', ',0.01,0,')], + ['zero stop', VALID_CSV.replace(',3334.25,3324.25,', ',3334.25,0,')], + ['a nonprotective buy stop', VALID_CSV.replace(',3334.25,3324.25,', ',3334.25,3344.25,')], + ['a nonprotective sell stop', VALID_CSV.replace(',buy,0.01,3334.25,3324.25,', ',sell,0.01,3334.25,3324.25,')], + ['an empty position id', VALID_CSV.replace(',position-opaque,reconciled,', ',,reconciled,')], + ])('rejects filled_protected with %s', (_case, csv) => { + expect(() => parseExecutionStatusCsv(csv)).toThrow(/position/i) + }) + + it('rejects stop confirmation without a complete valid protective position', () => { + const csv = VALID_CSV + .replace(',filled_protected,', ',ready,') + .replace(',1,buy,0.01,3334.25,3324.25,position-opaque,reconciled,', ',1,,,,,,reconciled,') + + expect(() => parseExecutionStatusCsv(csv)).toThrow(/stop protection/i) + }) + + it('rejects execution enabled in the status-only rollout stage', () => { + expect(() => parseExecutionStatusCsv(VALID_CSV.replace(',hfm_canary,1,', ',status_only,1,'))).toThrow(/status.only/i) + }) + + it('rejects a broker paired with the other allowlisted demo server', () => { + expect(() => parseExecutionStatusCsv(VALID_CSV.replace('HFMarketsGlobal-Demo4', 'ICMarketsSC-Demo'))).toThrow(/broker.*server|server.*broker/i) + }) + + it('rejects workstation-local timestamps without a canonical UTC suffix', () => { + expect(() => parseExecutionStatusCsv(VALID_CSV.replaceAll('.000Z', ''))).toThrow(/timestamp|captured_at/i) + }) + + it('accepts a protected fill while the kill switch blocks new entries', () => { + expect(parseExecutionStatusCsv(VALID_CSV.replace(',hfm_canary,1,0,', ',hfm_canary,1,1,'))).toMatchObject({ + state: 'filled_protected', + killSwitch: true, + stopProtectionConfirmed: true, + }) + }) + + it.each([ + [ + 'reconciliation_required', + VALID_CSV + .replace(',filled_protected,', ',reconciliation_required,') + .replace(',Request completed,1,buy,0.01,3334.25,3324.25,position-opaque,reconciled,', ',Request completed,0,,,,,,required,'), + ], + ['filled_protected', VALID_CSV], + [ + 'stopped', + VALID_CSV + .replace(',filled_protected,', ',stopped,') + .replace(',Request completed,1,buy,0.01,3334.25,3324.25,position-opaque,reconciled,', ',Request completed,0,,,,,,reconciled,'), + ], + [ + 'emergency_close', + VALID_CSV + .replace(',filled_protected,', ',emergency_close,') + .replace(',Request completed,1,buy,0.01,3334.25,3324.25,position-opaque,reconciled,', ',Request completed,0,buy,0.01,3334.25,0,position-opaque,protection_error,'), + ], + ])('parses the Task 8 %s reconciliation lifecycle', (state, csv) => { + expect(parseExecutionStatusCsv(csv)).toMatchObject({ state }) + }) + + it('rejects filled_protected when broker stop confirmation is absent', () => { + const csv = VALID_CSV.replace(',Request completed,1,buy,', ',Request completed,0,buy,') + + expect(() => parseExecutionStatusCsv(csv)).toThrow(/filled_protected|position/i) + }) +}) + +describe('summarizeLatestJmbExecutionStatus', () => { + let root: string + + beforeEach(async () => { + root = await mkdtemp(join(tmpdir(), 'openalice-execution-status-')) + }) + + afterEach(async () => { + await rm(root, { recursive: true, force: true }) + }) + + it('summarizes a protected demo fill without exposing account login', async () => { + const directory = join(root, 'hfmarkets', 'XAUUSD') + await mkdir(directory, { recursive: true }) + const path = join(directory, 'latest_status.csv') + await writeFile(path, VALID_CSV, 'utf8') + await utimes(path, new Date('2026-07-13T09:10:00.000Z'), new Date('2026-07-13T09:10:00.000Z')) + + const summary = await summarizeLatestJmbExecutionStatus(root, 'hfmarkets', 'XAUUSD', new Date('2026-07-13T09:10:30.000Z')) + + expect(summary.state).toBe('filled_protected') + expect(summary.label).toBe('DEMO ENABLED') + expect(JSON.stringify(summary)).not.toMatch(/account.?login/i) + }) + + it('fails closed when the status is missing, malformed, or stale', async () => { + await expect(summarizeLatestJmbExecutionStatus(root, 'hfmarkets', 'XAUUSD')).resolves.toMatchObject({ + state: 'missing', + executionEnabled: false, + killSwitch: true, + }) + + const directory = join(root, 'hfmarkets', 'XAUUSD') + await mkdir(directory, { recursive: true }) + await writeFile(join(directory, 'latest_status.csv'), VALID_CSV.replace('symbol,', 'account_login,symbol,'), 'utf8') + await expect(summarizeLatestJmbExecutionStatus(root, 'hfmarkets', 'XAUUSD')).resolves.toMatchObject({ + state: 'malformed', + executionEnabled: false, + killSwitch: true, + }) + + await writeFile(join(directory, 'latest_status.csv'), VALID_CSV, 'utf8') + await utimes(join(directory, 'latest_status.csv'), new Date('2026-07-13T09:10:00.000Z'), new Date('2026-07-13T09:10:00.000Z')) + await expect(summarizeLatestJmbExecutionStatus(root, 'hfmarkets', 'XAUUSD', new Date('2026-07-13T09:15:01.000Z'))).resolves.toMatchObject({ + state: 'stale', + executionEnabled: false, + killSwitch: true, + }) + }) + + it('fails closed when the row identity does not match the requested path', async () => { + const directory = join(root, 'hfmarkets', 'XAUUSD') + await mkdir(directory, { recursive: true }) + await writeFile(join(directory, 'latest_status.csv'), VALID_CSV.replace(',hfmarkets,', ',icmarkets,'), 'utf8') + + await expect(summarizeLatestJmbExecutionStatus(root, 'hfmarkets', 'XAUUSD')).resolves.toMatchObject({ + state: 'malformed', + executionEnabled: false, + killSwitch: true, + }) + }) + + it('uses the trusted file modification time for freshness on non-UTC workstations', async () => { + const directory = join(root, 'hfmarkets', 'XAUUSD') + const path = join(directory, 'latest_status.csv') + await mkdir(directory, { recursive: true }) + await writeFile(path, VALID_CSV.replaceAll('2026-07-13T09:10:00.000Z', '2026-01-01T00:00:00.000Z'), 'utf8') + await utimes(path, new Date('2026-07-13T09:10:00.000Z'), new Date('2026-07-13T09:10:00.000Z')) + + await expect(summarizeLatestJmbExecutionStatus(root, 'hfmarkets', 'XAUUSD', new Date('2026-07-13T09:10:30.000Z'))) + .resolves.toMatchObject({ state: 'filled_protected' }) + }) +}) diff --git a/src/domain/mt5/execution-status.ts b/src/domain/mt5/execution-status.ts new file mode 100644 index 000000000..20ae9b539 --- /dev/null +++ b/src/domain/mt5/execution-status.ts @@ -0,0 +1,367 @@ +import { open } from 'node:fs/promises' +import { join } from 'node:path' + +export type JmbExecutionLifecycleState = + | 'disabled' + | 'paused' + | 'blocked' + | 'ready' + | 'order_requesting' + | 'order_rejected' + | 'reconciliation_required' + | 'filled_protected' + | 'close_requesting' + | 'closed' + | 'stopped' + | 'emergency_close' + | 'error' + +export type JmbExecutionRolloutStage = 'status_only' | 'hfm_canary' | 'ic_canary' | 'both_demo' +export type JmbResearchExecutionState = JmbExecutionLifecycleState | 'demo_blocked' | 'missing' | 'malformed' | 'stale' +export type JmbExecutionBroker = 'hfmarkets' | 'icmarkets' +export type JmbExecutionSymbol = 'XAUUSD' | 'EURUSD' + +export interface JmbExecutionStatusSummary { + state: JmbResearchExecutionState + label: string + detail: string + capturedAt: string | null + broker: JmbExecutionBroker + server: string | null + accountMode: 'demo' | null + symbol: JmbExecutionSymbol + rolloutStage: JmbExecutionRolloutStage + executionEnabled: boolean + killSwitch: boolean + decisionId: string | null + observationId: string | null + latestEvent: { id: string; type: string; at: string; resultCode: string; detail: string } | null + stopProtectionConfirmed: boolean + position: { direction: 'buy' | 'sell'; volume: number; openPrice: number; stopLoss: number; id: string } | null + reconciliationState: string + dailyLossCount: number + dailyRealizedLoss: number + blockingGate: string | null + nextSafeAction: string +} + +const STATUS_HEADER = [ + 'schema_version', + 'captured_at', + 'broker', + 'server', + 'account_mode', + 'symbol', + 'state', + 'detail', + 'rollout_stage', + 'execution_enabled', + 'kill_switch', + 'decision_id', + 'observation_id', + 'event_id', + 'event_type', + 'event_time', + 'result_code', + 'result_detail', + 'stop_protection_confirmed', + 'position_direction', + 'position_volume', + 'position_open_price', + 'position_stop_loss', + 'position_id', + 'reconciliation_state', + 'daily_loss_count', + 'daily_realized_loss', + 'blocking_gate', + 'next_safe_action', +] as const + +const LIFECYCLE_STATES = new Set([ + 'disabled', + 'paused', + 'blocked', + 'ready', + 'order_requesting', + 'order_rejected', + 'reconciliation_required', + 'filled_protected', + 'close_requesting', + 'closed', + 'stopped', + 'emergency_close', + 'error', +]) +const ROLLOUT_STAGES = new Set(['status_only', 'hfm_canary', 'ic_canary', 'both_demo']) +const MAX_STATUS_AGE_MS = 2 * 60_000 +const MAX_CLOCK_SKEW_MS = 60_000 +const RFC3339_UTC = /^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d{3})?Z$/ +const EXPECTED_SERVERS: Record = { + hfmarkets: 'HFMarketsGlobal-Demo4', + icmarkets: 'ICMarketsSC-Demo', +} + +function isCanonicalUtcTimestamp(value: string): boolean { + return RFC3339_UTC.test(value) && Number.isFinite(Date.parse(value)) +} + +function parseCsvRecords(text: string): string[][] { + const records: string[][] = [] + let record: string[] = [] + let field = '' + let quoted = false + + for (let index = 0; index < text.length; index += 1) { + const character = text[index] + if (character === '"') { + if (quoted && text[index + 1] === '"') { + field += '"' + index += 1 + } else { + quoted = !quoted + } + } else if (character === ',' && !quoted) { + record.push(field) + field = '' + } else if ((character === '\n' || character === '\r') && !quoted) { + record.push(field) + records.push(record) + field = '' + record = [] + if (character === '\r' && text[index + 1] === '\n') index += 1 + } else { + field += character + } + } + + if (quoted) throw new Error('Execution status CSV contains an unterminated quoted field.') + if (field !== '' || record.length > 0) { + record.push(field) + records.push(record) + } + return records +} + +function requiredFinite(value: string, field: string): number { + if (value === '') throw new Error(`Execution status ${field} is required.`) + const parsed = Number(value) + if (!Number.isFinite(parsed)) throw new Error(`Execution status ${field} must be finite.`) + return parsed +} + +function binary(value: string, field: string): boolean { + if (value !== '0' && value !== '1') throw new Error(`Execution status ${field} must be 0 or 1.`) + return value === '1' +} + +function nullable(value: string): string | null { + return value === '' ? null : value +} + +function assertExactGroup(values: string[], label: string): boolean { + const populated = values.filter((value) => value !== '').length + if (populated !== 0 && populated !== values.length) throw new Error(`Execution status ${label} is incomplete.`) + return populated === values.length +} + +function labelFor(state: JmbResearchExecutionState, executionEnabled: boolean): string { + if (state === 'disabled') return 'EXECUTION DISABLED' + if (state === 'ready') return 'CANARY READY' + if (state === 'filled_protected') return executionEnabled ? 'DEMO ENABLED' : 'EXECUTION DISABLED' + if (state === 'paused') return 'PAUSED' + if (state === 'reconciliation_required') return 'RECONCILIATION REQUIRED' + if (state === 'demo_blocked') return 'DEMO BLOCKED' + if (state === 'missing') return 'STATUS MISSING' + if (state === 'malformed') return 'STATUS MALFORMED' + if (state === 'stale') return 'STATUS STALE' + if (state === 'order_requesting' || state === 'close_requesting') return 'DEMO ENABLED' + return 'BLOCKED' +} + +export function parseExecutionStatusCsv(text: string): JmbExecutionStatusSummary { + const records = parseCsvRecords(text) + if (records.length !== 2) throw new Error('Execution status CSV schema requires one header and one row.') + const [header, values] = records + if (header.length !== STATUS_HEADER.length + || values.length !== STATUS_HEADER.length + || !header.every((field, index) => field === STATUS_HEADER[index])) { + throw new Error('Execution status CSV schema mismatch.') + } + + const row = Object.fromEntries(STATUS_HEADER.map((field, index) => [field, values[index] ?? ''])) + if (row['schema_version'] !== '1') throw new Error('Execution status schema_version must be 1.') + if (!isCanonicalUtcTimestamp(row['captured_at']!)) throw new Error('Execution status captured_at must be canonical RFC 3339 UTC.') + if (row['broker'] !== 'hfmarkets' && row['broker'] !== 'icmarkets') throw new Error('Execution status broker is not allowlisted.') + if (row['server'] !== EXPECTED_SERVERS[row['broker']]) throw new Error('Execution status broker and server pairing is invalid.') + if (row['account_mode'] !== 'demo') throw new Error('Execution status must describe a demo account.') + if (row['symbol'] !== 'XAUUSD' && row['symbol'] !== 'EURUSD') throw new Error('Execution status symbol is not allowlisted.') + if (!LIFECYCLE_STATES.has(row['state'] as JmbExecutionLifecycleState)) throw new Error('Execution status lifecycle state is invalid.') + if (!ROLLOUT_STAGES.has(row['rollout_stage'] as JmbExecutionRolloutStage)) throw new Error('Execution status rollout stage is invalid.') + if (!row['server'] || !row['detail'] || !row['reconciliation_state'] || !row['next_safe_action']) { + throw new Error('Execution status required operational detail is missing.') + } + + const hasEvent = assertExactGroup([row['event_id']!, row['event_type']!, row['event_time']!], 'latest event identity') + if (!hasEvent && (row['result_code'] !== '' || row['result_detail'] !== '')) { + throw new Error('Execution status event result has no event identity.') + } + if (hasEvent && !isCanonicalUtcTimestamp(row['event_time']!)) throw new Error('Execution status event_time must be canonical RFC 3339 UTC.') + + const hasPosition = assertExactGroup([ + row['position_direction']!, row['position_volume']!, row['position_open_price']!, row['position_stop_loss']!, row['position_id']!, + ], 'position') + if (hasPosition && row['position_direction'] !== 'buy' && row['position_direction'] !== 'sell') { + throw new Error('Execution status position direction is invalid.') + } + + const dailyLossCount = requiredFinite(row['daily_loss_count']!, 'daily_loss_count') + if (!Number.isInteger(dailyLossCount) || dailyLossCount < 0) throw new Error('Execution status daily_loss_count must be a non-negative integer.') + const state = row['state'] as JmbExecutionLifecycleState + const rolloutStage = row['rollout_stage'] as JmbExecutionRolloutStage + const executionEnabled = binary(row['execution_enabled']!, 'execution_enabled') + const killSwitch = binary(row['kill_switch']!, 'kill_switch') + const stopProtectionConfirmed = binary(row['stop_protection_confirmed']!, 'stop_protection_confirmed') + const position = hasPosition ? { + direction: row['position_direction'] as 'buy' | 'sell', + volume: requiredFinite(row['position_volume']!, 'position_volume'), + openPrice: requiredFinite(row['position_open_price']!, 'position_open_price'), + stopLoss: requiredFinite(row['position_stop_loss']!, 'position_stop_loss'), + id: row['position_id']!, + } : null + + if (position && (position.volume <= 0 || position.openPrice <= 0 || position.id.trim() === '')) { + throw new Error('Execution status position must have a positive volume and open price with a nonempty opaque id.') + } + const hasProtectiveStop = position !== null + && position.stopLoss > 0 + && (position.direction === 'buy' ? position.stopLoss < position.openPrice : position.stopLoss > position.openPrice) + if (stopProtectionConfirmed && !hasProtectiveStop) { + throw new Error('Execution status stop protection requires a complete position with a valid protective stop.') + } + if (state === 'filled_protected' && (!stopProtectionConfirmed || !hasProtectiveStop)) { + throw new Error('Execution status filled_protected requires a confirmed protected position.') + } + if (rolloutStage === 'status_only' && executionEnabled) { + throw new Error('Execution status status_only rollout cannot enable execution.') + } + + return { + state, + label: labelFor(state, executionEnabled), + detail: row['detail']!, + capturedAt: row['captured_at']!, + broker: row['broker'] as JmbExecutionBroker, + server: row['server']!, + accountMode: 'demo', + symbol: row['symbol'] as JmbExecutionSymbol, + rolloutStage, + executionEnabled, + killSwitch, + decisionId: nullable(row['decision_id']!), + observationId: nullable(row['observation_id']!), + latestEvent: hasEvent ? { + id: row['event_id']!, + type: row['event_type']!, + at: row['event_time']!, + resultCode: row['result_code']!, + detail: row['result_detail']!, + } : null, + stopProtectionConfirmed, + position, + reconciliationState: row['reconciliation_state']!, + dailyLossCount, + dailyRealizedLoss: requiredFinite(row['daily_realized_loss']!, 'daily_realized_loss'), + blockingGate: nullable(row['blocking_gate']!), + nextSafeAction: row['next_safe_action']!, + } +} + +function safeSummary( + state: 'missing' | 'malformed' | 'stale', + broker: JmbExecutionBroker, + symbol: JmbExecutionSymbol, + capturedAt: string | null = null, +): JmbExecutionStatusSummary { + const detail = state === 'missing' + ? 'No broker-local MT5 demo execution status is available.' + : state === 'malformed' + ? 'The broker-local execution status failed strict validation and is treated as blocked.' + : 'The broker-local execution status is stale and cannot be trusted for operational monitoring.' + return { + state, + label: labelFor(state, false), + detail, + capturedAt, + broker, + server: null, + accountMode: null, + symbol, + rolloutStage: 'status_only', + executionEnabled: false, + killSwitch: true, + decisionId: null, + observationId: null, + latestEvent: null, + stopProtectionConfirmed: false, + position: null, + reconciliationState: 'unverified', + dailyLossCount: 0, + dailyRealizedLoss: 0, + blockingGate: 'execution_status', + nextSafeAction: 'Inspect the broker-local MT5 demo EA and restore a fresh, valid status artifact.', + } +} + +export function createDemoBlockedExecutionSummary( + broker: JmbExecutionBroker, + symbol: 'EURUSD', +): JmbExecutionStatusSummary { + return { + ...safeSummary('missing', broker, symbol), + state: 'demo_blocked', + label: labelFor('demo_blocked', false), + detail: 'EURUSD remains shadow-only and is not eligible for Plan 3 demo execution.', + blockingGate: 'instrument_allowlist', + nextSafeAction: 'Continue read-only shadow observation; no execution action is available.', + } +} + +function fileErrorCode(error: unknown): string | null { + return typeof error === 'object' && error !== null && 'code' in error ? String((error as { code: unknown }).code) : null +} + +export async function summarizeLatestJmbExecutionStatus( + root: string, + broker: JmbExecutionBroker, + symbol: JmbExecutionSymbol, + now = new Date(), +): Promise { + let text: string + let modifiedAt: Date + try { + const handle = await open(join(root, broker, symbol, 'latest_status.csv'), 'r') + try { + const [contents, metadata] = await Promise.all([handle.readFile('utf8'), handle.stat()]) + text = contents + modifiedAt = metadata.mtime + } finally { + await handle.close() + } + } catch (error) { + return safeSummary(fileErrorCode(error) === 'ENOENT' ? 'missing' : 'malformed', broker, symbol) + } + + let summary: JmbExecutionStatusSummary + try { + summary = parseExecutionStatusCsv(text) + } catch { + return safeSummary('malformed', broker, symbol) + } + if (summary.broker !== broker || summary.symbol !== symbol) return safeSummary('malformed', broker, symbol) + + const ageMs = now.getTime() - modifiedAt.getTime() + if (ageMs > MAX_STATUS_AGE_MS || ageMs < -MAX_CLOCK_SKEW_MS) { + return safeSummary('stale', broker, symbol, summary.capturedAt) + } + return summary +} diff --git a/src/domain/mt5/local-paths.spec.ts b/src/domain/mt5/local-paths.spec.ts new file mode 100644 index 000000000..baa5df271 --- /dev/null +++ b/src/domain/mt5/local-paths.spec.ts @@ -0,0 +1,47 @@ +import { describe, expect, it } from 'vitest' +import { join } from 'node:path' + +import { resolveJmbMt5Roots } from './local-paths.js' + +describe('JMB MT5 local roots', () => { + it('prefers explicit roots and derives missing protocol roots from explicit Common Files', () => { + const commonFilesRoot = 'C:\\MT5\\Common\\Files' + const roots = resolveJmbMt5Roots({ + commonFilesRoot, + bridgeRoot: 'D:\\bridge', + researchRoot: 'D:\\research', + env: {}, + homeDirectory: 'C:\\Users\\tester', + }) + + expect(roots).toEqual({ + bridgeRoot: 'D:\\bridge', + ledgerRoot: join(commonFilesRoot, 'OpenAliceMt5TradeLedgerV1'), + policyRoot: join(commonFilesRoot, 'OpenAliceMt5DemoPolicyV1'), + costModelRoot: join(commonFilesRoot, 'OpenAliceMt5CostModelV1'), + executionDecisionRoot: join(commonFilesRoot, 'OpenAliceMt5ExecutionDecisionV1'), + executionRoot: join(commonFilesRoot, 'OpenAliceMt5ExecutionV1'), + researchRoot: 'D:\\research', + }) + }) + + it('uses environment Common Files before APPDATA and the documented research environment variable', () => { + const roots = resolveJmbMt5Roots({ + env: { + OPENALICE_MT5_COMMON_FILES_ROOT: 'D:\\common-files', + OPENALICE_RESEARCH_ROOT: 'D:\\research-env', + APPDATA: 'C:\\ignored-appdata', + }, + homeDirectory: 'C:\\Users\\tester', + }) + expect(roots.bridgeRoot).toBe(join('D:\\common-files', 'OpenAliceMt5BridgeV1')) + expect(roots.researchRoot).toBe('D:\\research-env') + }) + + it('derives Common Files from APPDATA and rejects unresolved or relative roots', () => { + expect(resolveJmbMt5Roots({ env: { APPDATA: 'C:\\Users\\tester\\AppData\\Roaming' }, homeDirectory: 'C:\\Users\\tester' }).bridgeRoot) + .toBe(join('C:\\Users\\tester\\AppData\\Roaming', 'MetaQuotes', 'Terminal', 'Common', 'Files', 'OpenAliceMt5BridgeV1')) + expect(() => resolveJmbMt5Roots({ env: {}, homeDirectory: 'C:\\Users\\tester' })).toThrow(/absolute Common Files root/i) + expect(() => resolveJmbMt5Roots({ commonFilesRoot: 'relative', env: {}, homeDirectory: 'C:\\Users\\tester' })).toThrow(/absolute Common Files root/i) + }) +}) diff --git a/src/domain/mt5/local-paths.ts b/src/domain/mt5/local-paths.ts new file mode 100644 index 000000000..638eeb119 --- /dev/null +++ b/src/domain/mt5/local-paths.ts @@ -0,0 +1,53 @@ +import { homedir } from 'node:os' +import { isAbsolute, join } from 'node:path' + +export interface JmbMt5Roots { + bridgeRoot: string + ledgerRoot: string + policyRoot: string + costModelRoot: string + executionDecisionRoot: string + executionRoot: string + researchRoot: string +} + +export interface ResolveJmbMt5RootsOptions extends Partial { + commonFilesRoot?: string + env?: NodeJS.ProcessEnv + homeDirectory?: string +} + +function requireAbsoluteCommonFilesRoot(root: string | undefined): string { + if (root === undefined || !isAbsolute(root)) { + throw new Error('An absolute Common Files root is required for JMB MT5 local artifacts.') + } + return root +} + +function explicitOrDerived(explicit: string | undefined, commonFilesRoot: string, directory: string): string { + return explicit ?? join(commonFilesRoot, directory) +} + +export function resolveJmbMt5Roots(options: ResolveJmbMt5RootsOptions = {}): JmbMt5Roots { + const env = options.env ?? process.env + const commonFilesRoot = requireAbsoluteCommonFilesRoot( + options.commonFilesRoot + ?? env['OPENALICE_MT5_COMMON_FILES_ROOT'] + ?? (env['APPDATA'] === undefined + ? undefined + : join(env['APPDATA'], 'MetaQuotes', 'Terminal', 'Common', 'Files')), + ) + const homeDirectory = options.homeDirectory ?? homedir() + + return { + bridgeRoot: explicitOrDerived(options.bridgeRoot, commonFilesRoot, 'OpenAliceMt5BridgeV1'), + ledgerRoot: explicitOrDerived(options.ledgerRoot, commonFilesRoot, 'OpenAliceMt5TradeLedgerV1'), + policyRoot: explicitOrDerived(options.policyRoot, commonFilesRoot, 'OpenAliceMt5DemoPolicyV1'), + costModelRoot: explicitOrDerived(options.costModelRoot, commonFilesRoot, 'OpenAliceMt5CostModelV1'), + executionDecisionRoot: explicitOrDerived(options.executionDecisionRoot, commonFilesRoot, 'OpenAliceMt5ExecutionDecisionV1'), + executionRoot: explicitOrDerived(options.executionRoot, commonFilesRoot, 'OpenAliceMt5ExecutionV1'), + researchRoot: options.researchRoot + ?? env['OPENALICE_RESEARCH_ROOT'] + ?? join(homeDirectory, '.openalice', 'data', 'research'), + } +} diff --git a/src/domain/mt5/read-only-bridge.spec.ts b/src/domain/mt5/read-only-bridge.spec.ts new file mode 100644 index 000000000..c0df4a6d5 --- /dev/null +++ b/src/domain/mt5/read-only-bridge.spec.ts @@ -0,0 +1,69 @@ +import { afterEach, describe, expect, it } from 'vitest' +import { mkdtemp, mkdir, readFile, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { readMt5ReadOnlyBridge } from './read-only-bridge.js' + +const directories: string[] = [] + +afterEach(async () => { + await Promise.all(directories.splice(0).map((directory) => rm(directory, { recursive: true, force: true }))) +}) + +describe('readMt5ReadOnlyBridge', () => { + it('defaults bridge exports to the exact Gold demo symbol', async () => { + const source = await readFile(join(process.cwd(), 'tools', 'mt5', 'OpenAliceMt5ReadOnlyBridge.mq5'), 'utf8') + + expect(source).toContain('input string InpSymbol = "XAUUSD";') + expect(source).not.toContain('input string InpSymbol = "XAUUSDb";') + expect(source).toContain('OUTPUT_ROOT+"\\\\"+InpBrokerId+"\\\\"+InpSymbol+"\\\\completed_d1.csv"') + expect(source).toContain('const string TRADE_LEDGER_ROOT = "OpenAliceMt5TradeLedgerV1";') + expect(source).toContain('bool WriteTradeLedger(const datetime captured_at)') + expect(source).toContain('HistorySelect(from_time, captured_at)') + expect(source).toContain('%02d:%02d:%02d.000Z') + expect(source).toContain('datetime captured_at = TimeGMT()') + expect(source).toContain('IsoTime(captured_at)') + expect(source).not.toContain('IsoTime(TimeCurrent())') + expect(source).not.toContain('IsoTime(tick.time)') + }) + + it('keeps the bridge read-only with no order-capable calls', async () => { + const source = await readFile(join(process.cwd(), 'tools', 'mt5', 'OpenAliceMt5ReadOnlyBridge.mq5'), 'utf8') + + expect(source).not.toMatch(/\bOrderSend\s*\(/) + expect(source).not.toMatch(/\bOrderCheck\s*\(/) + expect(source).not.toMatch(/\bCTrade\b/) + expect(source).not.toMatch(/\btrade\.(Buy|Sell)\s*\(/) + expect(source).not.toMatch(/\bPositionClose\s*\(/) + }) + + it('accepts a fresh demo-only read-only heartbeat', async () => { + const root = await mkdtemp(join(tmpdir(), 'openalice-mt5-')) + directories.push(root) + const directory = join(root, 'hfmarkets', 'XAUUSDb') + await mkdir(directory, { recursive: true }) + await writeFile(join(directory, 'status.csv'), [ + 'captured_at,broker,symbol,bridge_mode,account_mode,server,terminal_connected,bid,ask,spread_price,open_positions,open_orders', + '2026-06-23T12:00:00,hfmarkets,XAUUSDb,read_only,demo,HFM-Demo,1,2345.10,2345.30,0.20,0,0', + ].join('\n')) + + const status = await readMt5ReadOnlyBridge(root, 'hfmarkets', 'XAUUSDb') + + expect(status.state).toBe('ready') + expect(status.spread).toBeCloseTo(0.2) + expect(status.server).toBe('HFM-Demo') + }) + + it('blocks a heartbeat that is not a demo account', async () => { + const root = await mkdtemp(join(tmpdir(), 'openalice-mt5-')) + directories.push(root) + const directory = join(root, 'icmarkets', 'XAUUSD') + await mkdir(directory, { recursive: true }) + await writeFile(join(directory, 'status.csv'), [ + 'captured_at,broker,symbol,bridge_mode,account_mode,server,terminal_connected,bid,ask,spread_price,open_positions,open_orders', + '2026-06-23T12:00:00,icmarkets,XAUUSD,read_only,real,IC-Demo,1,2345.10,2345.30,0.20,0,0', + ].join('\n')) + + await expect(readMt5ReadOnlyBridge(root, 'icmarkets', 'XAUUSD')).resolves.toMatchObject({ state: 'unsafe_account' }) + }) +}) diff --git a/src/domain/mt5/read-only-bridge.ts b/src/domain/mt5/read-only-bridge.ts new file mode 100644 index 000000000..a90be5a95 --- /dev/null +++ b/src/domain/mt5/read-only-bridge.ts @@ -0,0 +1,100 @@ +import { readFile, stat } from 'node:fs/promises' +import { join } from 'node:path' + +export type Mt5BridgeState = 'awaiting_bridge' | 'stale' | 'unsafe_account' | 'disconnected' | 'ready' + +export interface Mt5ReadOnlyBridgeStatus { + state: Mt5BridgeState + label: string + detail: string + broker: string + symbol: string + server: string | null + capturedAt: string | null + lastUpdated: string | null + bid: number | null + ask: number | null + spread: number | null + openPositions: number | null + openOrders: number | null +} + +const STALE_AFTER_MS = 2 * 60_000 + +type BridgeRow = Record + +function parseCsv(text: string): BridgeRow | null { + const [header, value] = text.trim().split(/\r?\n/, 2) + if (!header || !value) return null + const keys = header.split(',') + const values = value.split(',') + if (keys.length !== values.length) return null + return Object.fromEntries(keys.map((key, index) => [key.trim(), values[index]!.trim()])) +} + +function numberOrNull(value: string | undefined): number | null { + if (!value) return null + const parsed = Number(value) + return Number.isFinite(parsed) ? parsed : null +} + +function waiting(broker: string, symbol: string, detail: string): Mt5ReadOnlyBridgeStatus { + return { + state: 'awaiting_bridge', + label: 'Bridge not attached', + detail, + broker, + symbol, + server: null, + capturedAt: null, + lastUpdated: null, + bid: null, + ask: null, + spread: null, + openPositions: null, + openOrders: null, + } +} + +/** Reads a status file produced by the local MT5 EA. This path has no order API. */ +export async function readMt5ReadOnlyBridge( + root: string, + broker: string, + symbol: string, + now = new Date(), +): Promise { + const path = join(root, broker, symbol, 'status.csv') + let text: string + let modified: Date + try { + [text, modified] = await Promise.all([readFile(path, 'utf8'), stat(path).then((entry) => entry.mtime)]) + } catch { + return waiting(broker, symbol, 'Attach OpenAliceMt5ReadOnlyBridge to this demo terminal and symbol.') + } + + const row = parseCsv(text) + if (!row) return waiting(broker, symbol, 'The bridge status file is unreadable; restart the EA and check the MT5 Experts log.') + + const base = { + broker, + symbol, + server: row['server'] || null, + capturedAt: row['captured_at'] || null, + lastUpdated: modified.toISOString(), + bid: numberOrNull(row['bid']), + ask: numberOrNull(row['ask']), + spread: numberOrNull(row['spread_price']), + openPositions: numberOrNull(row['open_positions']), + openOrders: numberOrNull(row['open_orders']), + } + if (row['bridge_mode'] !== 'read_only' || row['account_mode'] !== 'demo') { + return { ...base, state: 'unsafe_account', label: 'Demo-only check failed', detail: 'The bridge is not reporting a demo, read-only terminal. It is blocked from progression.' } + } + if (row['terminal_connected'] !== '1') { + return { ...base, state: 'disconnected', label: 'Terminal disconnected', detail: 'MT5 reported no broker connection. No execution capability is available.' } + } + if (now.getTime() - modified.getTime() > STALE_AFTER_MS) { + return { ...base, state: 'stale', label: 'Bridge heartbeat stale', detail: 'The last terminal update is older than two minutes. Check that the EA is attached and Algo Trading settings permit it to run.' } + } + return { ...base, state: 'ready', label: 'Demo bridge connected', detail: 'Read-only telemetry is current. This bridge has no order-submission code.' } +} diff --git a/src/domain/mt5/shadow-decision-engine.spec.ts b/src/domain/mt5/shadow-decision-engine.spec.ts new file mode 100644 index 000000000..cbf7b03c1 --- /dev/null +++ b/src/domain/mt5/shadow-decision-engine.spec.ts @@ -0,0 +1,65 @@ +import { describe, expect, it } from 'vitest' +import { buildShadowDecision } from './shadow-decision-engine.js' + +const baseInput = { + createdAt: '2026-07-13T10:00:00.000Z', + broker: 'hfmarkets', + server: 'HFMarketsGlobal-Demo4', + accountMode: 'demo', + symbol: 'XAUUSD', + canonicalInstrument: 'Gold / USD', + strategyVersion: 'daily-trend-v1', + bridgeState: 'ready', + learningState: 'learning', + latestDirection: 'uptrend', + bid: 2410, + ask: 2410.36, + spread: 0.36, + maxSpread: 0.75, + volume: 0.01, + maxVolume: 0.01, + stopLoss: 2402, + riskAmount: 0.8, + maxAllowedRisk: 1, + demoCandidateApproved: true, +} as const + +describe('buildShadowDecision', () => { + it('logs a Gold buy shadow decision when gates pass', () => { + const decision = buildShadowDecision(baseInput) + + expect(decision.mode).toBe('shadow') + expect(decision.direction).toBe('buy') + expect(decision.reasonCode).toBe('daily_trend_shadow') + expect(decision.gateResults.every((gate) => gate.state === 'pass')).toBe(true) + }) + + it('keeps EURUSD demo-blocked when its candidate gate is not approved', () => { + const decision = buildShadowDecision({ + ...baseInput, + symbol: 'EURUSD', + canonicalInstrument: 'Euro / USD', + demoCandidateApproved: false, + }) + + expect(decision.mode).toBe('demo_blocked') + expect(decision.direction).toBe('buy') + expect(decision.gateResults.some((gate) => gate.gate === 'candidate_gate' && gate.state === 'fail')).toBe(true) + }) + + it('skips flat when bridge is stale', () => { + const decision = buildShadowDecision({ ...baseInput, bridgeState: 'stale' }) + + expect(decision.mode).toBe('skipped') + expect(decision.direction).toBe('flat') + expect(decision.reasonCode).toBe('gate_blocked') + }) + + it('blocks when stop loss is missing', () => { + const decision = buildShadowDecision({ ...baseInput, stopLoss: null }) + + expect(decision.mode).toBe('skipped') + expect(decision.direction).toBe('flat') + expect(decision.gateResults.some((gate) => gate.gate === 'stop_loss' && gate.state === 'fail')).toBe(true) + }) +}) diff --git a/src/domain/mt5/shadow-decision-engine.ts b/src/domain/mt5/shadow-decision-engine.ts new file mode 100644 index 000000000..8d7ba215b --- /dev/null +++ b/src/domain/mt5/shadow-decision-engine.ts @@ -0,0 +1,97 @@ +import { createJmbDecisionId, type JmbDecisionDirection, type JmbDecisionRecord, type JmbGateResult } from './decision-record.js' + +export interface BuildShadowDecisionInput { + createdAt: string + broker: string + server: string | null + accountMode: string | null + symbol: string + canonicalInstrument: string + strategyVersion: string + bridgeState: string + learningState: string + latestDirection: 'uptrend' | 'downtrend' | 'flat' + bid: number | null + ask: number | null + spread: number | null + maxSpread: number + volume: number + maxVolume: number + stopLoss: number | null + riskAmount: number | null + maxAllowedRisk: number + demoCandidateApproved: boolean +} + +function gate(gateName: string, state: JmbGateResult['state'], detail: string): JmbGateResult { + return { gate: gateName, state, detail } +} + +function directionFor(latestDirection: BuildShadowDecisionInput['latestDirection']): JmbDecisionDirection { + if (latestDirection === 'uptrend') return 'buy' + if (latestDirection === 'downtrend') return 'sell' + return 'flat' +} + +export function buildShadowDecision(input: BuildShadowDecisionInput): JmbDecisionRecord { + const direction = directionFor(input.latestDirection) + const gateResults: JmbGateResult[] = [ + gate( + 'account_demo', + input.accountMode === 'demo' ? 'pass' : 'fail', + input.accountMode === 'demo' ? 'MT5 reports demo mode' : 'Account mode is not confirmed demo', + ), + gate('bridge_ready', input.bridgeState === 'ready' ? 'pass' : 'fail', `Bridge state is ${input.bridgeState}`), + gate('learning_ready', input.learningState === 'learning' ? 'pass' : 'warn', `Learning state is ${input.learningState}`), + gate( + 'spread', + input.spread != null && input.spread <= input.maxSpread ? 'pass' : 'fail', + input.spread == null ? 'Spread is unavailable' : `${input.spread} <= ${input.maxSpread}`, + ), + gate('volume', input.volume > 0 && input.volume <= input.maxVolume ? 'pass' : 'fail', `${input.volume} <= ${input.maxVolume}`), + gate('stop_loss', input.stopLoss != null ? 'pass' : 'fail', input.stopLoss == null ? 'Stop loss is required' : `Stop loss ${input.stopLoss}`), + gate( + 'risk_amount', + input.riskAmount != null && input.riskAmount <= input.maxAllowedRisk ? 'pass' : 'fail', + input.riskAmount == null ? 'Risk amount unavailable' : `${input.riskAmount} <= ${input.maxAllowedRisk}`, + ), + gate( + 'candidate_gate', + input.demoCandidateApproved ? 'pass' : 'fail', + input.demoCandidateApproved ? 'Candidate gate approved for shadow review' : 'Broker/symbol is not approved for demo execution', + ), + gate('shadow_only', 'pass', 'Plan 2 logs decisions only and submits no orders'), + ] + const hardFailure = gateResults.some((item) => item.state === 'fail' && item.gate !== 'candidate_gate') + const mode = hardFailure ? 'skipped' : input.demoCandidateApproved ? 'shadow' : 'demo_blocked' + const finalDirection = hardFailure ? 'flat' : direction + const record: JmbDecisionRecord = { + schemaVersion: 1, + decisionId: 'pending', + createdAt: input.createdAt, + broker: input.broker, + server: input.server, + accountMode: input.accountMode, + symbol: input.symbol, + canonicalInstrument: input.canonicalInstrument, + strategyVersion: input.strategyVersion, + mode, + direction: finalDirection, + reasonCode: hardFailure ? 'gate_blocked' : 'daily_trend_shadow', + reasonDetail: hardFailure + ? 'One or more hard gates failed; no order can be requested.' + : `Completed trend state is ${input.latestDirection}; decision logged for learning only.`, + entryReferencePrice: finalDirection === 'buy' ? input.ask : finalDirection === 'sell' ? input.bid : null, + stopLoss: input.stopLoss, + takeProfit: null, + volume: input.volume, + spread: input.spread, + riskAmount: input.riskAmount, + maxAllowedRisk: input.maxAllowedRisk, + gateResults, + orderTicket: null, + positionId: null, + outcome: null, + } + return { ...record, decisionId: createJmbDecisionId(record) } +} diff --git a/src/domain/mt5/trade-ledger.spec.ts b/src/domain/mt5/trade-ledger.spec.ts new file mode 100644 index 000000000..dc4116e4b --- /dev/null +++ b/src/domain/mt5/trade-ledger.spec.ts @@ -0,0 +1,152 @@ +import { afterEach, describe, expect, it } from 'vitest' +import { mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { deriveMt5TradeOrigin, parseMt5TradeLedgerCsv, summarizeMt5TradeLedger } from './trade-ledger.js' + +const directories: string[] = [] + +afterEach(async () => { + await Promise.all(directories.splice(0).map((directory) => rm(directory, { recursive: true, force: true }))) +}) + +describe('parseMt5TradeLedgerCsv', () => { + it('parses deal rows and keeps tickets as strings', () => { + const rows = parseMt5TradeLedgerCsv([ + 'account_mode,server,login,broker,symbol,deal_ticket,order_ticket,position_id,time,entry,type,reason,volume,price,commission,fee,swap,profit,magic,comment', + 'demo,HFM-Demo,123456,hfmarkets,XAUUSD,987654321012345,123456789012345,555,2026-07-13T01:02:03.000Z,out,buy,client,0.01,2410.25,-0.07,0,-0.01,4.25,0,manual close', + ].join('\n')) + + expect(rows).toHaveLength(1) + expect(rows[0]).toMatchObject({ + accountMode: 'demo', + broker: 'hfmarkets', + symbol: 'XAUUSD', + dealTicket: '987654321012345', + orderTicket: '123456789012345', + positionId: '555', + volume: 0.01, + profit: 4.25, + }) + }) + + it('rejects malformed rows with a clear error', () => { + expect(() => parseMt5TradeLedgerCsv('account_mode,server\nonly-one-column')).toThrow('Malformed MT5 trade ledger row 2') + }) +}) + +describe('deriveMt5TradeOrigin', () => { + it('labels client zero-magic trades as manual', () => { + expect(deriveMt5TradeOrigin({ magic: 0, reason: 'client', comment: 'closed from terminal' })).toBe('manual') + }) + + it('labels non-zero magic trades as ea', () => { + expect(deriveMt5TradeOrigin({ magic: 880001, reason: 'expert', comment: 'JMB Goldmine' })).toBe('ea') + }) + + it('labels balance operations and unknown reasons separately', () => { + expect(deriveMt5TradeOrigin({ magic: 0, reason: 'balance', comment: 'deposit' })).toBe('other') + expect(deriveMt5TradeOrigin({ magic: 0, reason: '', comment: '' })).toBe('unknown') + }) +}) + +describe('summarizeMt5TradeLedger', () => { + it('summarizes fresh demo trade history for a broker symbol', async () => { + const root = await mkdtemp(join(tmpdir(), 'openalice-ledger-')) + directories.push(root) + const directory = join(root, 'hfmarkets', 'XAUUSD') + await mkdir(directory, { recursive: true }) + await writeFile(join(directory, 'deals.csv'), [ + 'account_mode,server,login,broker,symbol,deal_ticket,order_ticket,position_id,time,entry,type,reason,volume,price,commission,fee,swap,profit,magic,comment', + 'demo,HFM-Demo,123456,hfmarkets,XAUUSD,1,11,101,2026-07-13T01:00:00.000Z,out,buy,client,0.01,2410.25,-0.07,0,-0.01,4.25,0,manual close', + 'demo,HFM-Demo,123456,hfmarkets,XAUUSD,2,12,102,2026-07-13T02:00:00.000Z,out,sell,expert,0.01,2408.25,-0.07,0,-0.01,-1.25,880001,JMB Goldmine demo', + ].join('\n')) + + const summary = await summarizeMt5TradeLedger(root, 'hfmarkets', 'XAUUSD', new Date('2026-07-13T02:01:00.000Z')) + + expect(summary.state).toBe('learning') + expect(summary.totalDeals).toBe(2) + expect(summary.manualDeals).toBe(1) + expect(summary.eaDeals).toBe(1) + expect(summary.netProfit).toBeCloseTo(2.84) + expect(summary.accountMode).toBe('demo') + }) + + it('blocks non-demo trade history from progression', async () => { + const root = await mkdtemp(join(tmpdir(), 'openalice-ledger-')) + directories.push(root) + const directory = join(root, 'icmarkets', 'EURUSD') + await mkdir(directory, { recursive: true }) + await writeFile(join(directory, 'deals.csv'), [ + 'account_mode,server,login,broker,symbol,deal_ticket,order_ticket,position_id,time,entry,type,reason,volume,price,commission,fee,swap,profit,magic,comment', + 'real,IC-Live,123456,icmarkets,EURUSD,1,11,101,2026-07-13T01:00:00.000Z,out,buy,client,0.01,1.17000,-0.07,0,0,1.25,0,manual close', + ].join('\n')) + + const summary = await summarizeMt5TradeLedger(root, 'icmarkets', 'EURUSD', new Date('2026-07-13T02:01:00.000Z')) + + expect(summary.state).toBe('blocked') + expect(summary.detail).toContain('non-demo') + }) + + it('does not learn from a ledger that has no rows for the requested broker symbol', async () => { + const root = await mkdtemp(join(tmpdir(), 'openalice-ledger-')) + directories.push(root) + const directory = join(root, 'hfmarkets', 'XAUUSD') + await mkdir(directory, { recursive: true }) + await writeFile(join(directory, 'deals.csv'), [ + 'account_mode,server,login,broker,symbol,deal_ticket,order_ticket,position_id,time,entry,type,reason,volume,price,commission,fee,swap,profit,magic,comment', + 'demo,HFM-Demo,123456,hfmarkets,EURUSD,1,11,101,2026-07-13T01:00:00.000Z,out,buy,client,0.01,1.17000,-0.07,0,0,1.25,0,manual close', + 'demo,IC-Demo,123456,icmarkets,XAUUSD,2,12,102,2026-07-13T01:05:00.000Z,out,sell,client,0.01,2410.25,-0.07,0,-0.01,4.25,0,manual close', + ].join('\n')) + + const summary = await summarizeMt5TradeLedger(root, 'hfmarkets', 'XAUUSD', new Date('2026-07-13T02:01:00.000Z')) + + expect(summary.state).toBe('no_data') + expect(summary.totalDeals).toBe(0) + expect(summary.accountMode).toBeNull() + }) + + it('blocks mixed demo and real rows for the requested broker symbol', async () => { + const root = await mkdtemp(join(tmpdir(), 'openalice-ledger-')) + directories.push(root) + const directory = join(root, 'icmarkets', 'EURUSD') + await mkdir(directory, { recursive: true }) + await writeFile(join(directory, 'deals.csv'), [ + 'account_mode,server,login,broker,symbol,deal_ticket,order_ticket,position_id,time,entry,type,reason,volume,price,commission,fee,swap,profit,magic,comment', + 'demo,IC-Demo,123456,icmarkets,EURUSD,1,11,101,2026-07-13T01:00:00.000Z,out,buy,client,0.01,1.17000,-0.07,0,0,1.25,0,manual close', + 'real,IC-Live,123456,icmarkets,EURUSD,2,12,102,2026-07-13T01:05:00.000Z,out,sell,client,0.01,1.17100,-0.07,0,0,-0.75,0,manual close', + ].join('\n')) + + const summary = await summarizeMt5TradeLedger(root, 'icmarkets', 'EURUSD', new Date('2026-07-13T02:01:00.000Z')) + + expect(summary.state).toBe('blocked') + expect(summary.detail).toContain('non-demo') + expect(summary.totalDeals).toBe(2) + }) + + it('blocks unreadable trade history instead of rejecting malformed ledger rows', async () => { + const root = await mkdtemp(join(tmpdir(), 'openalice-ledger-')) + directories.push(root) + const directory = join(root, 'hfmarkets', 'XAUUSD') + await mkdir(directory, { recursive: true }) + await writeFile(join(directory, 'deals.csv'), [ + 'account_mode,server,login,broker,symbol,deal_ticket,order_ticket,position_id,time,entry,type,reason,volume,price,commission,fee,swap,profit,magic,comment', + 'demo,HFM-Demo,123456,hfmarkets,XAUUSD,1', + ].join('\n')) + + const summary = await summarizeMt5TradeLedger(root, 'hfmarkets', 'XAUUSD', new Date('2026-07-13T02:01:00.000Z')) + + expect(summary.state).toBe('blocked') + expect(summary.label).toBe('Trade history unreadable') + expect(summary.detail).toContain('Malformed MT5 trade ledger row 2') + expect(summary.broker).toBe('hfmarkets') + expect(summary.symbol).toBe('XAUUSD') + expect(summary.lastUpdated).toEqual(expect.any(String)) + expect(summary.totalDeals).toBe(0) + expect(summary.manualDeals).toBe(0) + expect(summary.eaDeals).toBe(0) + expect(summary.otherDeals).toBe(0) + expect(summary.unknownDeals).toBe(0) + expect(summary.netProfit).toBe(0) + }) +}) diff --git a/src/domain/mt5/trade-ledger.ts b/src/domain/mt5/trade-ledger.ts new file mode 100644 index 000000000..7baddc497 --- /dev/null +++ b/src/domain/mt5/trade-ledger.ts @@ -0,0 +1,209 @@ +import { readFile, stat } from 'node:fs/promises' +import { join } from 'node:path' + +export type Mt5TradeOrigin = 'manual' | 'ea' | 'other' | 'unknown' +export type Mt5TradeLedgerState = 'no_data' | 'learning' | 'blocked' | 'stale' + +export interface Mt5TradeLedgerRow { + accountMode: string + server: string + login: string + broker: string + symbol: string + dealTicket: string + orderTicket: string + positionId: string + time: string + entry: string + type: string + reason: string + volume: number + price: number + commission: number + fee: number + swap: number + profit: number + magic: number + comment: string + origin: Mt5TradeOrigin +} + +export interface Mt5TradeLedgerSummary { + state: Mt5TradeLedgerState + label: string + detail: string + broker: string + symbol: string + accountMode: string | null + server: string | null + lastDealTime: string | null + lastUpdated: string | null + totalDeals: number + manualDeals: number + eaDeals: number + otherDeals: number + unknownDeals: number + netProfit: number +} + +const STALE_AFTER_MS = 24 * 60 * 60_000 + +function parseCsvLine(line: string): string[] { + const cells: string[] = [] + let current = '' + let quoted = false + for (let index = 0; index < line.length; index += 1) { + const character = line[index] + if (character === '"') { + if (quoted && line[index + 1] === '"') { + current += '"' + index += 1 + } else { + quoted = !quoted + } + } else if (character === ',' && !quoted) { + cells.push(current) + current = '' + } else { + current += character + } + } + cells.push(current) + return cells +} + +function numberField(row: Record, key: string, lineNumber: number): number { + const value = row[key] ?? '' + const parsed = Number(value) + if (!Number.isFinite(parsed)) throw new Error(`Invalid numeric MT5 trade ledger field "${key}" on row ${lineNumber}`) + return parsed +} + +export function deriveMt5TradeOrigin(row: Pick): Mt5TradeOrigin { + const reason = row.reason.toLowerCase() + const comment = row.comment.toLowerCase() + if (row.magic !== 0 || reason === 'expert' || comment.includes('jmb goldmine')) return 'ea' + if (reason === 'client' || reason === 'mobile' || reason === 'web') return 'manual' + if (reason === 'balance' || reason === 'correction' || reason === 'charge') return 'other' + return 'unknown' +} + +export function parseMt5TradeLedgerCsv(text: string): Mt5TradeLedgerRow[] { + const lines = text.trim().split(/\r?\n/).filter(Boolean) + if (lines.length === 0) return [] + const headers = parseCsvLine(lines[0]!).map((header) => header.trim()) + return lines.slice(1).map((line, index) => { + const lineNumber = index + 2 + const values = parseCsvLine(line) + if (values.length !== headers.length) throw new Error(`Malformed MT5 trade ledger row ${lineNumber}`) + const raw = Object.fromEntries(headers.map((header, headerIndex) => [header, values[headerIndex]!.trim()])) + const base = { + accountMode: raw['account_mode'] ?? '', + server: raw['server'] ?? '', + login: raw['login'] ?? '', + broker: raw['broker'] ?? '', + symbol: raw['symbol'] ?? '', + dealTicket: raw['deal_ticket'] ?? '', + orderTicket: raw['order_ticket'] ?? '', + positionId: raw['position_id'] ?? '', + time: raw['time'] ?? '', + entry: raw['entry'] ?? '', + type: raw['type'] ?? '', + reason: raw['reason'] ?? '', + volume: numberField(raw, 'volume', lineNumber), + price: numberField(raw, 'price', lineNumber), + commission: numberField(raw, 'commission', lineNumber), + fee: numberField(raw, 'fee', lineNumber), + swap: numberField(raw, 'swap', lineNumber), + profit: numberField(raw, 'profit', lineNumber), + magic: numberField(raw, 'magic', lineNumber), + comment: raw['comment'] ?? '', + } + return { ...base, origin: deriveMt5TradeOrigin(base) } + }) +} + +export async function summarizeMt5TradeLedger( + root: string, + broker: string, + symbol: string, + now = new Date(), +): Promise { + const path = join(root, broker, symbol, 'deals.csv') + let text: string + let modified: Date + try { + const result = await Promise.all([readFile(path, 'utf8'), stat(path).then((entry) => entry.mtime)]) + text = result[0] + modified = result[1] + } catch { + return { + state: 'no_data', + label: 'Awaiting trade history', + detail: 'Run the read-only MT5 trade ledger exporter for this demo account and symbol.', + broker, + symbol, + accountMode: null, + server: null, + lastDealTime: null, + lastUpdated: null, + totalDeals: 0, + manualDeals: 0, + eaDeals: 0, + otherDeals: 0, + unknownDeals: 0, + netProfit: 0, + } + } + + let rows: Mt5TradeLedgerRow[] + try { + rows = parseMt5TradeLedgerCsv(text).filter((row) => row.broker === broker && row.symbol === symbol) + } catch (error) { + const detail = error instanceof Error ? error.message : 'Unknown MT5 trade ledger parse error' + return { + state: 'blocked', + label: 'Trade history unreadable', + detail: `The MT5 trade ledger exists but could not be read safely: ${detail}. Refresh the read-only export before learning can continue.`, + broker, + symbol, + accountMode: null, + server: null, + lastDealTime: null, + lastUpdated: modified.toISOString(), + totalDeals: 0, + manualDeals: 0, + eaDeals: 0, + otherDeals: 0, + unknownDeals: 0, + netProfit: 0, + } + } + const first = rows[0] + const lastDealTime = rows.map((row) => row.time).sort().at(-1) ?? null + const totalMoney = rows.reduce((total, row) => total + row.profit + row.commission + row.fee + row.swap, 0) + const base = { + broker, + symbol, + accountMode: first?.accountMode ?? null, + server: first?.server ?? null, + lastDealTime, + lastUpdated: modified.toISOString(), + totalDeals: rows.length, + manualDeals: rows.filter((row) => row.origin === 'manual').length, + eaDeals: rows.filter((row) => row.origin === 'ea').length, + otherDeals: rows.filter((row) => row.origin === 'other').length, + unknownDeals: rows.filter((row) => row.origin === 'unknown').length, + netProfit: Number(totalMoney.toFixed(2)), + } + if (rows.length === 0) { + return { ...base, state: 'no_data', label: 'Awaiting trade history', detail: 'No matching demo trade history was found for this broker and symbol.' } + } + if (rows.some((row) => row.accountMode !== 'demo')) { + return { ...base, state: 'blocked', label: 'Trade history blocked', detail: 'The ledger contains non-demo account history, so it cannot unlock demo automation.' } + } + if (now.getTime() - modified.getTime() > STALE_AFTER_MS) { + return { ...base, state: 'stale', label: 'Trade history stale', detail: 'The trade ledger has not been refreshed in the last 24 hours.' } + } + return { ...base, state: 'learning', label: 'Learning from demo history', detail: 'Manual and EA demo trades are available for review and journaling.' } +} diff --git a/src/main.ts b/src/main.ts index d4d3d1ffb..70f5a6d3e 100644 --- a/src/main.ts +++ b/src/main.ts @@ -1,5 +1,5 @@ import { readFile, writeFile, mkdir } from 'fs/promises' -import { dirname } from 'path' +import { dirname, join } from 'path' import { acquireOpenAliceRuntimeLocks, takeoverRequested, @@ -66,6 +66,10 @@ import { provenanceShowFactory } from './tool/provenance-show.js' import { conversationToolFactories } from './tool/conversation.js' import { artifactConversationToolFactories } from './tool/conversation-artifacts.js' import { createToolCallLog } from './core/tool-call-log.js' +import { createJmbMt5DecisionScheduler } from './task/mt5-decision-scheduler.js' +import { createJmbMt5OutcomeImporter, importReconciledExecutionOutcomes } from './task/mt5-outcome-importer.js' +import { DEFAULT_JMB_DEMO_INSTRUMENTS, runDemoDecisionCycle } from './domain/mt5/demo-decision-service.js' +import { resolveJmbMt5Roots } from './domain/mt5/local-paths.js' import { NewsCollectorStore, NewsCollector } from './domain/news/index.js' import { createNewsArchiveTools } from './tool/news.js' @@ -405,6 +409,23 @@ async function main() { console.log(`plugin started: ${plugin.name}`) } + const mt5DecisionScheduler = createJmbMt5DecisionScheduler({ + runCycle: async () => runDemoDecisionCycle({ roots: resolveJmbMt5Roots() }), + }) + await mt5DecisionScheduler.start() + + const mt5Roots = resolveJmbMt5Roots() + const mt5OutcomeImporter = createJmbMt5OutcomeImporter({ + runCycle: async () => importReconciledExecutionOutcomes({ + executionRoot: mt5Roots.executionRoot, + learningRoot: join(mt5Roots.researchRoot, 'mt5-execution-learning'), + instruments: DEFAULT_JMB_DEMO_INSTRUMENTS + .filter((instrument) => instrument.symbol === 'XAUUSD') + .map(({ broker, server, symbol }) => ({ broker, server, symbol })), + }), + }) + await mt5OutcomeImporter.start() + console.log('engine: started') // Broker catalog refresh, snapshot scheduling, and broker close-on- @@ -415,6 +436,8 @@ async function main() { let stopped = false const shutdown = async () => { stopped = true + mt5OutcomeImporter.stop() + mt5DecisionScheduler.stop() newsCollector?.stop() for (const plugin of [...corePlugins, ...optionalPlugins.values()]) { await plugin.stop() diff --git a/src/task/mt5-decision-scheduler.spec.ts b/src/task/mt5-decision-scheduler.spec.ts new file mode 100644 index 000000000..97930a8e4 --- /dev/null +++ b/src/task/mt5-decision-scheduler.spec.ts @@ -0,0 +1,44 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' + +import { createJmbMt5DecisionScheduler } from './mt5-decision-scheduler.js' + +describe('JMB MT5 decision scheduler', () => { + beforeEach(() => { + vi.useFakeTimers() + }) + + afterEach(() => { + vi.clearAllTimers() + vi.useRealTimers() + }) + + it('runs one catch-up cycle before arming the five-minute pump', async () => { + const runCycle = vi.fn(async () => []) + const scheduler = createJmbMt5DecisionScheduler({ runCycle }) + + await scheduler.start() + + expect(runCycle).toHaveBeenCalledTimes(1) + scheduler.stop() + }) + + it('does not overlap a slow cycle', async () => { + let release!: () => void + const slowCycle = new Promise((resolve) => { release = resolve }) + const runCycle = vi.fn() + .mockResolvedValueOnce([]) + .mockImplementation(() => slowCycle) + const scheduler = createJmbMt5DecisionScheduler({ runCycle, every: '5m' }) + + await scheduler.start() + await vi.advanceTimersByTimeAsync(15 * 60_000) + + expect(runCycle).toHaveBeenCalledTimes(2) + + release() + await slowCycle + await vi.advanceTimersByTimeAsync(0) + scheduler.stop() + expect(vi.getTimerCount()).toBe(0) + }) +}) diff --git a/src/task/mt5-decision-scheduler.ts b/src/task/mt5-decision-scheduler.ts new file mode 100644 index 000000000..3df8b79e2 --- /dev/null +++ b/src/task/mt5-decision-scheduler.ts @@ -0,0 +1,34 @@ +import { createPump, type Pump } from '../core/pump.js' + +export interface JmbMt5DecisionScheduler { + start(): Promise + stop(): void + runNow(): Promise +} + +export function createJmbMt5DecisionScheduler(options: { + runCycle: () => Promise + every?: string + logger?: Pick +}): JmbMt5DecisionScheduler { + const pump: Pump = createPump({ + name: 'jmb-mt5-decision-cycle', + every: options.every ?? '5m', + serial: true, + onTick: async () => { await options.runCycle() }, + logger: options.logger, + }) + + return { + async start() { + await pump.runNow() + pump.start() + }, + stop() { + pump.stop() + }, + runNow() { + return pump.runNow() + }, + } +} diff --git a/src/task/mt5-outcome-importer.spec.ts b/src/task/mt5-outcome-importer.spec.ts new file mode 100644 index 000000000..146958998 --- /dev/null +++ b/src/task/mt5-outcome-importer.spec.ts @@ -0,0 +1,289 @@ +import { mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' + +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' + +import { readExecutionLearningRecords } from '../domain/mt5/execution-outcomes.js' +import { + createJmbMt5OutcomeImporter, + importReconciledExecutionOutcomes, +} from './mt5-outcome-importer.js' + +const hfmGold = { broker: 'hfmarkets', server: 'HFMarketsGlobal-Demo4', symbol: 'XAUUSD' } as const +const icGold = { broker: 'icmarkets', server: 'ICMarketsSC-Demo', symbol: 'XAUUSD' } as const + +function executionEvent(overrides: Record = {}): Record { + return { + schema_version: 1, + event_id: 'event-request-1', + event_type: 'order_requesting', + event_time: '2026-07-13T09:00:00.000Z', + broker: 'hfmarkets', + server: 'HFMarketsGlobal-Demo4', + account_mode: 'demo', + account_identity_masked: 'masked-opaque', + symbol: 'XAUUSD', + strategy_version: 'daily-trend-v1', + magic_number: 880101, + decision_id: 'decision-1', + observation_id: 'observation-1', + gate_results: [], + calculated_risk: 10, + requested_volume: 0.01, + requested_price: 3334.25, + requested_stop_loss: 3324.25, + accepted_volume: null, + accepted_price: null, + accepted_stop_loss: null, + result_code: '', + result_detail: 'Request persisted.', + order_ticket: '', + deal_ticket: '', + position_id: '', + reconciliation_state: 'pending', + daily_loss_count: 0, + daily_realized_loss: 0, + commission: null, + swap: null, + fee: null, + net_result: null, + max_adverse_excursion: null, + max_favorable_excursion: null, + ...overrides, + } +} + +const requestEvent = executionEvent() +const fillEvent = executionEvent({ + event_id: 'event-fill-1', + event_type: 'filled_protected', + event_time: '2026-07-13T09:01:00.000Z', + accepted_volume: 0.01, + accepted_price: 3334.5, + accepted_stop_loss: 3324.25, + order_ticket: 'order-1', + deal_ticket: 'deal-open-1', + position_id: 'position-1', + reconciliation_state: 'reconciled', +}) +const closedEvent = executionEvent({ + event_id: 'event-closed-1', + event_type: 'closed', + event_time: '2026-07-13T10:00:00.000Z', + accepted_volume: 0.01, + accepted_price: 3334.5, + accepted_stop_loss: null, + order_ticket: 'order-1', + deal_ticket: 'deal-close-1', + position_id: 'position-1', + reconciliation_state: 'reconciled', + daily_loss_count: 1, + daily_realized_loss: 6.25, + commission: -0.5, + swap: -0.25, + fee: 0, + net_result: -6.25, +}) +const reconciliationRequiredEvent = executionEvent({ + event_id: 'event-reconcile-1', + event_type: 'reconciliation_required', + event_time: '2026-07-13T09:01:00.000Z', + accepted_volume: 0.01, + accepted_price: 3334.5, + order_ticket: 'order-1', + deal_ticket: 'deal-open-1', + position_id: 'position-1', + reconciliation_state: 'required', +}) + +async function writeExecutionEvents(root: string, events: readonly Record[]): Promise { + const directory = join(root, String(events[0]?.['broker']), 'XAUUSD') + await mkdir(directory, { recursive: true }) + await writeFile(join(directory, 'events.jsonl'), `${events.map((event) => JSON.stringify(event)).join('\n')}\n`, 'utf8') +} + +describe('MT5 outcome importer', () => { + let executionRoot: string + let learningRoot: string + + beforeEach(async () => { + executionRoot = await mkdtemp(join(tmpdir(), 'openalice-execution-')) + learningRoot = await mkdtemp(join(tmpdir(), 'openalice-learning-')) + }) + + afterEach(async () => { + vi.useRealTimers() + await Promise.all([ + rm(executionRoot, { recursive: true, force: true }), + rm(learningRoot, { recursive: true, force: true }), + ]) + }) + + it('imports one fully reconciled close exactly once', async () => { + await writeExecutionEvents(executionRoot, [requestEvent, fillEvent, closedEvent]) + await importReconciledExecutionOutcomes({ executionRoot, learningRoot, instruments: [hfmGold] }) + await importReconciledExecutionOutcomes({ executionRoot, learningRoot, instruments: [hfmGold] }) + + const records = await readExecutionLearningRecords(learningRoot, 'hfmarkets', 'XAUUSD') + expect(records).toHaveLength(1) + expect(records[0]).toMatchObject({ + decisionId: closedEvent['decision_id'], + outcomeEventId: closedEvent['event_id'], + result: 'loss', + netResult: -6.25, + source: 'ea_demo', + }) + }) + + it('imports a restart-recovered emergency terminal once under its opening identity', async () => { + const openingDecisionId = 'bc0bc128a9155065dda0b5bc' + const openingObservationId = '53f2bd057c1ee3608a02d1f2' + const emergencyRequest = executionEvent({ + event_id: 'event-emergency-request-1', + event_type: 'emergency_close', + event_time: '2026-07-13T09:02:00.000Z', + decision_id: openingDecisionId, + observation_id: openingObservationId, + calculated_risk: 7.5, + position_id: 'position-4321', + reconciliation_state: 'protection_error', + }) + const pausedAfterRestart = executionEvent({ + event_id: 'event-emergency-pause-1', + event_type: 'paused', + event_time: '2026-07-13T09:03:00.000Z', + decision_id: 'unrelated-current-decision', + observation_id: 'unrelated-current-observation', + position_id: 'position-4321', + reconciliation_state: 'protection_error', + }) + const emergencyTerminal = executionEvent({ + ...closedEvent, + event_id: 'event-emergency-closed-1', + event_time: '2026-07-13T09:04:00.000Z', + decision_id: openingDecisionId, + observation_id: openingObservationId, + calculated_risk: 7.5, + requested_price: 2400, + accepted_price: 2400.5, + requested_stop_loss: 2392, + position_id: 'position-4321', + commission: -0.5, + swap: -0.25, + fee: -0.1, + net_result: -6.25, + }) + await writeExecutionEvents(executionRoot, [emergencyRequest, pausedAfterRestart, emergencyTerminal]) + + const first = await importReconciledExecutionOutcomes({ executionRoot, learningRoot, instruments: [hfmGold] }) + const second = await importReconciledExecutionOutcomes({ executionRoot, learningRoot, instruments: [hfmGold] }) + + expect(first[0]).toMatchObject({ state: 'imported', imported: 1 }) + expect(second[0]).toMatchObject({ state: 'no_new_outcome', imported: 0 }) + await expect(readExecutionLearningRecords(learningRoot, 'hfmarkets', 'XAUUSD')).resolves.toEqual([ + expect.objectContaining({ + outcomeEventId: 'event-emergency-closed-1', + decisionId: openingDecisionId, + observationId: openingObservationId, + positionId: 'position-4321', + netResult: -6.25, + commission: -0.5, + swap: -0.25, + fee: -0.1, + }), + ]) + }) + + it('imports a fully reconciled stopped outcome', async () => { + const stoppedEvent = { + ...closedEvent, + event_id: 'event-stopped-1', + event_type: 'stopped', + event_time: '2026-07-13T09:30:00.000Z', + } + await writeExecutionEvents(executionRoot, [requestEvent, fillEvent, stoppedEvent]) + + const result = await importReconciledExecutionOutcomes({ executionRoot, learningRoot, instruments: [hfmGold] }) + + expect(result[0]).toMatchObject({ state: 'imported', imported: 1 }) + await expect(readExecutionLearningRecords(learningRoot, 'hfmarkets', 'XAUUSD')).resolves.toEqual([ + expect.objectContaining({ outcomeEventId: 'event-stopped-1', result: 'loss' }), + ]) + }) + + it('does not import an unresolved or unprotected exposure', async () => { + await writeExecutionEvents(executionRoot, [requestEvent, reconciliationRequiredEvent]) + + const result = await importReconciledExecutionOutcomes({ executionRoot, learningRoot, instruments: [hfmGold] }) + + expect(result[0]).toMatchObject({ state: 'no_new_outcome', imported: 0 }) + }) + + it('fails the broker closed when any physical JSONL line is malformed', async () => { + const directory = join(executionRoot, 'hfmarkets', 'XAUUSD') + await mkdir(directory, { recursive: true }) + await writeFile(join(directory, 'events.jsonl'), `${JSON.stringify(closedEvent)}\n{"schema_version":1\n`, 'utf8') + + const result = await importReconciledExecutionOutcomes({ executionRoot, learningRoot, instruments: [hfmGold] }) + + expect(result[0]).toMatchObject({ state: 'blocked', imported: 0 }) + await expect(readExecutionLearningRecords(learningRoot, 'hfmarkets', 'XAUUSD')).resolves.toEqual([]) + }) + + it('isolates a malformed broker journal from another broker', async () => { + const hfmDirectory = join(executionRoot, 'hfmarkets', 'XAUUSD') + await mkdir(hfmDirectory, { recursive: true }) + await writeFile(join(hfmDirectory, 'events.jsonl'), '{bad json}\n', 'utf8') + await writeExecutionEvents(executionRoot, [{ + ...closedEvent, + event_id: 'ic-event-closed-1', + broker: 'icmarkets', + server: 'ICMarketsSC-Demo', + magic_number: 880201, + }]) + + const result = await importReconciledExecutionOutcomes({ + executionRoot, + learningRoot, + instruments: [hfmGold, icGold], + }) + + expect(result).toEqual(expect.arrayContaining([ + expect.objectContaining({ broker: 'hfmarkets', state: 'blocked', imported: 0 }), + expect.objectContaining({ broker: 'icmarkets', state: 'imported', imported: 1 }), + ])) + }) + + it('runs one catch-up cycle then uses a serial five-minute Pump', async () => { + vi.useFakeTimers() + let release: (() => void) | undefined + const runCycle = vi.fn(async () => { + if (runCycle.mock.calls.length === 2) await new Promise((resolve) => { release = resolve }) + return [] + }) + const importer = createJmbMt5OutcomeImporter({ runCycle }) + + await importer.start() + expect(runCycle).toHaveBeenCalledTimes(1) + + const scheduled = vi.advanceTimersByTimeAsync(5 * 60_000) + await vi.advanceTimersByTimeAsync(5 * 60_000) + expect(runCycle).toHaveBeenCalledTimes(2) + release?.() + await scheduled + await importer.runNow() + expect(runCycle).toHaveBeenCalledTimes(3) + importer.stop() + }) + + it('starts after the decision scheduler and stops during shutdown', async () => { + const mainSource = await readFile(join(import.meta.dirname, '..', 'main.ts'), 'utf8') + const schedulerStart = mainSource.indexOf('await mt5DecisionScheduler.start()') + const importerStart = mainSource.indexOf('await mt5OutcomeImporter.start()') + + expect(mainSource).toContain("join(mt5Roots.researchRoot, 'mt5-execution-learning')") + expect(importerStart).toBeGreaterThan(schedulerStart) + expect(mainSource).toContain('mt5OutcomeImporter.stop()') + }) +}) diff --git a/src/task/mt5-outcome-importer.ts b/src/task/mt5-outcome-importer.ts new file mode 100644 index 000000000..c1c202703 --- /dev/null +++ b/src/task/mt5-outcome-importer.ts @@ -0,0 +1,112 @@ +import { createPump } from '../core/pump.js' +import { + appendOutcomeOnce, + ExecutionOutcomeValidationError, + executionEventToOutcome, + readExecutionEvents, + type ExecutionOutcomeImportOptions, + type ExecutionOutcomeImportResult, +} from '../domain/mt5/execution-outcomes.js' + +export interface JmbMt5OutcomeImporter { + start(): Promise + stop(): void + runNow(): Promise +} + +function fileErrorCode(error: unknown): string | null { + return typeof error === 'object' && error !== null && 'code' in error ? String(error.code) : null +} + +async function importInstrument( + options: ExecutionOutcomeImportOptions, + instrument: ExecutionOutcomeImportOptions['instruments'][number], +): Promise { + if (instrument.symbol !== 'XAUUSD') { + return { + broker: instrument.broker, + symbol: 'XAUUSD', + state: 'blocked', + imported: 0, + detail: 'Only XAUUSD demo execution outcomes are eligible for learning import.', + } + } + const expectedServer = instrument.broker === 'hfmarkets' ? 'HFMarketsGlobal-Demo4' : 'ICMarketsSC-Demo' + if (instrument.server !== expectedServer) { + return { + broker: instrument.broker, + symbol: 'XAUUSD', + state: 'blocked', + imported: 0, + detail: 'The configured demo server does not match the broker allowlist.', + } + } + + try { + const events = await readExecutionEvents(options.executionRoot, instrument.broker, 'XAUUSD') + const outcomes = events.flatMap((event) => { + if (event.broker !== instrument.broker || event.server !== instrument.server) { + throw new ExecutionOutcomeValidationError('Execution event identity does not match the configured instrument.') + } + const outcome = executionEventToOutcome(event) + return outcome === null ? [] : [outcome] + }) + + let imported = 0 + for (const outcome of outcomes) { + if (await appendOutcomeOnce(options.learningRoot, outcome)) imported += 1 + } + return { + broker: instrument.broker, + symbol: 'XAUUSD', + state: imported > 0 ? 'imported' : 'no_new_outcome', + imported, + detail: imported > 0 + ? `Imported ${imported} reconciled demo execution outcome${imported === 1 ? '' : 's'}.` + : 'No new reconciled terminal demo execution outcome is available.', + } + } catch (error) { + if (fileErrorCode(error) === 'ENOENT') { + return { + broker: instrument.broker, + symbol: 'XAUUSD', + state: 'no_new_outcome', + imported: 0, + detail: 'No demo execution event journal is available.', + } + } + const blocked = error instanceof ExecutionOutcomeValidationError + return { + broker: instrument.broker, + symbol: 'XAUUSD', + state: blocked ? 'blocked' : 'error', + imported: 0, + detail: blocked + ? `The demo execution journal failed strict validation: ${error.message}` + : 'The demo execution outcome import failed at its isolated broker boundary.', + } + } +} + +export async function importReconciledExecutionOutcomes( + options: ExecutionOutcomeImportOptions, +): Promise { + return Promise.all(options.instruments.map((instrument) => importInstrument(options, instrument))) +} + +export function createJmbMt5OutcomeImporter(options: { + runCycle: () => Promise + every?: string +}): JmbMt5OutcomeImporter { + const pump = createPump({ + name: 'jmb-mt5-outcome-import', + every: options.every ?? '5m', + serial: true, + onTick: async () => { await options.runCycle() }, + }) + return { + async start() { await pump.runNow(); pump.start() }, + stop() { pump.stop() }, + runNow() { return pump.runNow() }, + } +} diff --git a/src/webui/plugin.ts b/src/webui/plugin.ts index 4aa5bbe6d..65a069ed4 100644 --- a/src/webui/plugin.ts +++ b/src/webui/plugin.ts @@ -19,6 +19,7 @@ import { createToolsRoutes } from './routes/tools.js' import { createAgentStatusRoutes } from './routes/agent-status.js' import { createPersonaRoutes } from './routes/persona.js' import { createNewsRoutes } from './routes/news.js' +import { createResearchRoutes } from './routes/research.js' import { createMarketRoutes } from './routes/market.js' import { createBarsRoutes } from './routes/bars.js' import { createReferenceRoutes } from './routes/reference.js' @@ -232,6 +233,7 @@ export class WebPlugin implements Plugin { app.route('/api/tools', createToolsRoutes(ctx.toolCenter)) app.route('/api/agent-status', createAgentStatusRoutes(ctx)) app.route('/api/news', createNewsRoutes(ctx)) + app.route('/api/research', createResearchRoutes(ctx)) app.route('/api/market', createMarketRoutes(ctx)) app.route('/api/bars', createBarsRoutes(ctx)) app.route('/api/reference', createReferenceRoutes(ctx)) diff --git a/src/webui/routes/research.spec.ts b/src/webui/routes/research.spec.ts new file mode 100644 index 000000000..b578ae8d4 --- /dev/null +++ b/src/webui/routes/research.spec.ts @@ -0,0 +1,91 @@ +import { mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises' +import { tmpdir } from 'node:os' +import { join } from 'node:path' +import { afterEach, beforeEach, describe, expect, it } from 'vitest' +import type { EngineContext } from '../../core/types.js' +import type { JmbExecutionStatusSummary } from '../../domain/mt5/execution-status.js' +import { createResearchRoutes } from './research.js' + +const HEADER = 'schema_version,captured_at,broker,server,account_mode,symbol,state,detail,rollout_stage,execution_enabled,kill_switch,decision_id,observation_id,event_id,event_type,event_time,result_code,result_detail,stop_protection_confirmed,position_direction,position_volume,position_open_price,position_stop_loss,position_id,reconciliation_state,daily_loss_count,daily_realized_loss,blocking_gate,next_safe_action' + +function statusCsv(capturedAt: string): string { + return `${HEADER}\n${[ + '1', capturedAt, 'hfmarkets', 'HFMarketsGlobal-Demo4', 'demo', 'XAUUSD', 'filled_protected', + 'Broker confirms protected demo exposure.', 'hfm_canary', '1', '0', 'decision-1', 'observation-1', + 'event-1', 'fill_confirmed', capturedAt, '10009', 'Request completed', '1', 'buy', '0.01', + '3334.25', '3324.25', 'position-1', 'reconciled', '1', '-8.75', '', 'Monitor protection.', + ].join(',')}\n` +} + +type ResearchResponse = { + mode: string + tradingEnabled: boolean + stages: Array<{ key: string; state: string; detail: string }> + instruments: Array<{ broker: string; symbol: string; execution: JmbExecutionStatusSummary }> +} + +describe('GET /', () => { + let executionRoot: string + + beforeEach(async () => { + executionRoot = await mkdtemp(join(tmpdir(), 'openalice-research-route-')) + }) + + afterEach(async () => { + await rm(executionRoot, { recursive: true, force: true }) + }) + + it('projects Gold status and blocks EURUSD without exposing account login', async () => { + const directory = join(executionRoot, 'hfmarkets', 'XAUUSD') + await mkdir(directory, { recursive: true }) + await writeFile(join(directory, 'latest_status.csv'), statusCsv(new Date().toISOString()), 'utf8') + const app = createResearchRoutes({} as EngineContext, { executionRoot }) + + const response = await app.request('/') + const body = await response.json() as ResearchResponse + const hfmGold = body.instruments.find((instrument) => instrument.broker === 'hfmarkets' && instrument.symbol === 'XAUUSDb') + const hfmEuro = body.instruments.find((instrument) => instrument.broker === 'hfmarkets' && instrument.symbol === 'EURUSDb') + + expect(response.status).toBe(200) + expect(body.mode).toBe('research_only') + expect(body.tradingEnabled).toBe(false) + expect(body.stages.find((stage) => stage.key === 'demo')?.detail).toMatch(/broker-local/i) + expect(hfmGold?.execution).toMatchObject({ state: 'filled_protected', symbol: 'XAUUSD' }) + expect(hfmEuro?.execution).toMatchObject({ state: 'demo_blocked', label: 'DEMO BLOCKED', symbol: 'EURUSD' }) + expect(JSON.stringify(body)).not.toMatch(/accountLogin|account_login/i) + }) + + it('fails closed and drops a sentinel login from a malformed status artifact', async () => { + const directory = join(executionRoot, 'hfmarkets', 'XAUUSD') + await mkdir(directory, { recursive: true }) + const malformed = statusCsv(new Date().toISOString()) + .replace('symbol,state', 'account_login,symbol,state') + .replace(',hfmarkets,HFMarkets', ',hfmarkets,SENTINEL-LOGIN-884422,HFMarkets') + await writeFile(join(directory, 'latest_status.csv'), malformed, 'utf8') + const app = createResearchRoutes({} as EngineContext, { executionRoot }) + + const response = await app.request('/') + const serialized = JSON.stringify(await response.json()) + + expect(serialized).toContain('malformed') + expect(serialized).not.toContain('SENTINEL-LOGIN-884422') + expect(serialized).not.toMatch(/accountLogin|account_login/i) + }) + + it('does not count a contradictory protected fill as protected or complete', async () => { + const directory = join(executionRoot, 'hfmarkets', 'XAUUSD') + await mkdir(directory, { recursive: true }) + const contradictory = statusCsv(new Date().toISOString()).replace(',Request completed,1,buy,', ',Request completed,0,buy,') + await writeFile(join(directory, 'latest_status.csv'), contradictory, 'utf8') + const app = createResearchRoutes({} as EngineContext, { executionRoot }) + + const response = await app.request('/') + const body = await response.json() as ResearchResponse + const hfmGold = body.instruments.find((instrument) => instrument.broker === 'hfmarkets' && instrument.symbol === 'XAUUSDb') + const demoStage = body.stages.find((stage) => stage.key === 'demo') + + expect(hfmGold?.execution).toMatchObject({ state: 'malformed', stopProtectionConfirmed: false, position: null }) + expect(demoStage?.state).not.toBe('complete') + expect(demoStage?.detail).toContain('0/2') + }) +}) diff --git a/src/webui/routes/research.ts b/src/webui/routes/research.ts new file mode 100644 index 000000000..9c496469b --- /dev/null +++ b/src/webui/routes/research.ts @@ -0,0 +1,302 @@ +import { readdir, readFile, stat } from 'node:fs/promises' +import { homedir } from 'node:os' +import { join } from 'node:path' +import { Hono } from 'hono' +import { dataPath } from '../../core/paths.js' +import { summarizeLatestJmbDecision } from '../../domain/mt5/decision-record.js' +import { + createDemoBlockedExecutionSummary, + summarizeLatestJmbExecutionStatus, + type JmbExecutionBroker, + type JmbExecutionSymbol, +} from '../../domain/mt5/execution-status.js' +import { readMt5ReadOnlyBridge } from '../../domain/mt5/read-only-bridge.js' +import { summarizeMt5TradeLedger } from '../../domain/mt5/trade-ledger.js' +import type { EngineContext } from '../../core/types.js' + +type TrendReport = { + symbol: string + data: { daily_bars: number; first_day: string; last_day: string } + selected_on_training_sharpe: { lookback_days: number; sharpe: number | null; max_drawdown: number | null } + untouched_holdout: { total_return: number | null; sharpe: number | null; max_drawdown: number | null } + latest_observation?: { as_of: string; direction: 'uptrend' | 'downtrend' | 'flat'; lookback_return: number; lookback_days: number } +} + +type WalkForwardReport = { + method: { training_months: number; test_months: number } + windows: Array + out_of_sample_aggregate: { total_return: number | null; sharpe: number | null; max_drawdown: number | null } +} + +type ExperimentScenario = { + id: string + lookback_set: string + lookbacks: number[] + one_way_cost_bps: number + unseen_windows: number + out_of_sample: { total_return: number | null; sharpe: number | null; max_drawdown: number | null; win_rate: number | null } + review_flags: string[] +} + +type ExperimentRun = { + id: string + created_at: string + broker: string + symbol: string + data: { first_eligible_day: string; last_day: string; daily_bars: number; effective_train_start: string } + method: { training_months: number; test_months: number; drawdown_review_alert: number } + scenarios: ExperimentScenario[] + warning: string +} + +type ExperimentLedger = { runs: ExperimentRun[] } + +type ValidationFile = { + file: string + rows: number + bad_rows: number + duplicates: number + gaps_over_three_minutes: number + likely_m1: boolean +} + +type ValidationReport = { + files: ValidationFile[] +} + +type DataQuality = { + label: string + tone: 'muted' | 'green' | 'amber' | 'red' + inspectedFiles: number + likelyM1Files: number + fallbackFiles: number + badRows: number + duplicateRows: number +} + +const ARTIFACTS_DIR = process.env['OPENALICE_RESEARCH_ARTIFACTS_DIR'] ?? dataPath('research') +const LEGACY_ARTIFACTS_DIR = join(process.cwd(), '.codex-run') +const MT5_EXPORT_ROOT = process.env['OPENALICE_MT5_EXPORT_ROOT'] ?? join( + process.env['APPDATA'] ?? join(homedir(), 'AppData', 'Roaming'), + 'MetaQuotes', 'Terminal', 'Common', 'Files', 'OpenAliceMt5HistoryV2', +) +const MT5_BRIDGE_ROOT = process.env['OPENALICE_MT5_BRIDGE_ROOT'] ?? join( + process.env['APPDATA'] ?? join(homedir(), 'AppData', 'Roaming'), + 'MetaQuotes', 'Terminal', 'Common', 'Files', 'OpenAliceMt5BridgeV1', +) +const MT5_TRADE_LEDGER_ROOT = process.env['OPENALICE_MT5_TRADE_LEDGER_ROOT'] ?? join( + process.env['APPDATA'] ?? join(homedir(), 'AppData', 'Roaming'), + 'MetaQuotes', 'Terminal', 'Common', 'Files', 'OpenAliceMt5TradeLedgerV1', +) +const MT5_DECISION_ROOT = process.env['OPENALICE_MT5_DECISION_ROOT'] ?? join( + process.env['APPDATA'] ?? join(homedir(), 'AppData', 'Roaming'), + 'MetaQuotes', 'Terminal', 'Common', 'Files', 'OpenAliceMt5DecisionLogV1', +) +const MT5_EXECUTION_ROOT = process.env['OPENALICE_MT5_EXECUTION_ROOT'] ?? join( + process.env['APPDATA'] ?? join(homedir(), 'AppData', 'Roaming'), + 'MetaQuotes', 'Terminal', 'Common', 'Files', 'OpenAliceMt5ExecutionV1', +) + +type ResearchInstrumentConfig = { + broker: JmbExecutionBroker + symbol: string + bridgeSymbol?: JmbExecutionSymbol + label: string + artifact: string + walkForwardArtifact: string +} + +const INSTRUMENTS: readonly ResearchInstrumentConfig[] = [ + { broker: 'hfmarkets', symbol: 'XAUUSDb', bridgeSymbol: 'XAUUSD', label: 'Gold / USD', artifact: 'xauusd-trend-baseline.json', walkForwardArtifact: 'xauusd-walk-forward.json' }, + { broker: 'hfmarkets', symbol: 'EURUSDb', bridgeSymbol: 'EURUSD', label: 'Euro / USD', artifact: 'eurusd-trend-baseline.json', walkForwardArtifact: 'eurusd-walk-forward.json' }, + { broker: 'icmarkets', symbol: 'XAUUSD', label: 'Gold / USD', artifact: 'icmarkets-xauusd-trend-baseline.json', walkForwardArtifact: 'icmarkets-xauusd-walk-forward.json' }, + { broker: 'icmarkets', symbol: 'EURUSD', label: 'Euro / USD', artifact: 'icmarkets-eurusd-trend-baseline.json', walkForwardArtifact: 'icmarkets-eurusd-walk-forward.json' }, +] as const + +function localMt5Symbol(instrument: ResearchInstrumentConfig): JmbExecutionSymbol { + const symbol = instrument.bridgeSymbol ?? instrument.symbol + if (symbol !== 'XAUUSD' && symbol !== 'EURUSD') { + throw new Error(`Unsupported local MT5 symbol mapping: ${instrument.broker}/${symbol}`) + } + return symbol +} + +async function readJson(path: string): Promise { + try { + return JSON.parse(await readFile(path, 'utf-8')) as T + } catch { + return null + } +} + +async function readReport(fileName: string): Promise { + const primary = await readJson(join(ARTIFACTS_DIR, fileName)) + if (primary || ARTIFACTS_DIR === LEGACY_ARTIFACTS_DIR) return primary + return readJson(join(LEGACY_ARTIFACTS_DIR, fileName)) +} + +async function inspectExport(broker: string, symbol: string) { + const directory = join(MT5_EXPORT_ROOT, broker, symbol) + try { + const files = (await readdir(directory, { withFileTypes: true })) + .filter((entry) => entry.isFile() && /^m1_\d{4}_\d{2}\.csv$/.test(entry.name)) + .map((entry) => entry.name) + .sort() + const sizes = await Promise.all(files.map((file) => stat(join(directory, file)))) + return { + available: files.length > 0, + files: files.length, + firstFile: files[0] ?? null, + lastFile: files.at(-1) ?? null, + totalBytes: sizes.reduce((total, item) => total + item.size, 0), + lastUpdated: sizes.reduce((latest, item) => !latest || item.mtime > latest ? item.mtime : latest, null)?.toISOString() ?? null, + } + } catch { + return { available: false, files: 0, firstFile: null, lastFile: null, totalBytes: 0, lastUpdated: null } + } +} + +function qualityFor( + report: ValidationReport | null, + broker: string, + symbol: string, + hasExport: boolean, +): DataQuality { + if (!report) { + return { + label: hasExport ? 'Needs validation' : 'Awaiting export', + tone: hasExport ? 'amber' : 'muted', + inspectedFiles: 0, + likelyM1Files: 0, + fallbackFiles: 0, + badRows: 0, + duplicateRows: 0, + } + } + + const marker = `/${broker}/${symbol}/`.toLowerCase() + const files = report.files.filter((item) => item.file.replace(/\\/g, '/').toLowerCase().includes(marker)) + const likelyM1Files = files.filter((item) => item.likely_m1).length + const fallbackFiles = files.length - likelyM1Files + const badRows = files.reduce((total, item) => total + item.bad_rows, 0) + const duplicateRows = files.reduce((total, item) => total + item.duplicates, 0) + + if (files.length === 0) { + return { label: hasExport ? 'Not included in validation report' : 'Awaiting export', tone: hasExport ? 'amber' : 'muted', inspectedFiles: 0, likelyM1Files: 0, fallbackFiles: 0, badRows: 0, duplicateRows: 0 } + } + if (badRows > 0 || duplicateRows > 0) { + return { label: 'Data issue needs review', tone: 'red', inspectedFiles: files.length, likelyM1Files, fallbackFiles, badRows, duplicateRows } + } + if (fallbackFiles > 0) { + return { label: 'Mixed M1 and fallback history', tone: 'amber', inspectedFiles: files.length, likelyM1Files, fallbackFiles, badRows, duplicateRows } + } + return { label: 'M1 files validated', tone: 'green', inspectedFiles: files.length, likelyM1Files, fallbackFiles, badRows, duplicateRows } +} + +function evidenceFor(report: TrendReport | null) { + if (!report) return { label: 'Waiting for baseline', tone: 'muted' as const, score: 0 } + const holdout = report.untouched_holdout + if (holdout.total_return == null || holdout.total_return <= 0 || (holdout.sharpe ?? 0) <= 0) { + return { label: 'Rejected by holdout', tone: 'red' as const, score: 1 } + } + return { label: 'Early research candidate', tone: 'amber' as const, score: 2 } +} + +export function createResearchRoutes(ctx: EngineContext, overrides: { executionRoot?: string } = {}) { + const app = new Hono() + const executionRoot = overrides.executionRoot ?? MT5_EXECUTION_ROOT + + app.get('/', async (c) => { + const [validationReport, experimentLedger] = await Promise.all([ + readReport('mt5-history-report.json'), + readReport('daily-trend-experiment-ledger.json'), + ]) + const instruments = await Promise.all(INSTRUMENTS.map(async (instrument) => { + const bridgeSymbol = localMt5Symbol(instrument) + const [exportData, report, walkForward, bridge, learning, decision, execution] = await Promise.all([ + inspectExport(instrument.broker, instrument.symbol), + readReport(instrument.artifact), + readReport(instrument.walkForwardArtifact), + readMt5ReadOnlyBridge(MT5_BRIDGE_ROOT, instrument.broker, bridgeSymbol), + summarizeMt5TradeLedger(MT5_TRADE_LEDGER_ROOT, instrument.broker, bridgeSymbol), + summarizeLatestJmbDecision(MT5_DECISION_ROOT, instrument.broker, bridgeSymbol), + bridgeSymbol === 'XAUUSD' + ? summarizeLatestJmbExecutionStatus(executionRoot, instrument.broker, bridgeSymbol) + : Promise.resolve(createDemoBlockedExecutionSummary(instrument.broker, bridgeSymbol)), + ]) + return { + ...instrument, + export: exportData, + report, + walkForward, + bridge, + learning, + decision, + execution, + quality: qualityFor(validationReport, instrument.broker, instrument.symbol, exportData.available), + evidence: evidenceFor(report), + } + })) + + const news = ctx.newsProvider + ? (await ctx.newsProvider.getNewsV2({ endTime: new Date(), lookback: '24h', limit: 8 })).slice(-8).reverse().map((item) => ({ + time: item.time.toISOString(), + title: item.title, + source: item.metadata.source ?? null, + link: item.metadata.link ?? null, + })) + : [] + + const hfmReady = instruments.some((instrument) => instrument.broker === 'hfmarkets' && instrument.quality.tone !== 'muted') + const completedBaselines = instruments.filter((instrument) => instrument.report != null).length + const completedWalkForwards = instruments.filter((instrument) => instrument.walkForward != null).length + const readyDemoBridges = instruments.filter((instrument) => instrument.bridge.state === 'ready').length + const learningInstruments = instruments.filter((instrument) => instrument.learning.state === 'learning').length + const shadowDecisions = instruments.filter((instrument) => instrument.decision.state === 'shadow' || instrument.decision.state === 'demo_blocked').length + const goldExecutions = instruments.filter((instrument) => instrument.execution.symbol === 'XAUUSD') + const brokerLocalExecutionStatuses = goldExecutions.filter((instrument) => !['missing', 'malformed', 'stale'].includes(instrument.execution.state)).length + const protectedDemoPositions = goldExecutions.filter((instrument) => instrument.execution.state === 'filled_protected').length + const validatedInstruments = instruments.filter((instrument) => instrument.quality.tone === 'green' || instrument.quality.tone === 'amber').length + return c.json({ + asOf: new Date().toISOString(), + mode: 'research_only', + tradingEnabled: false, + summary: { + exportRoot: MT5_EXPORT_ROOT, + tradeLedgerRoot: MT5_TRADE_LEDGER_ROOT, + decisionRoot: MT5_DECISION_ROOT, + executionRoot, + instrumentsWithData: instruments.filter((instrument) => instrument.export.available).length, + completedBaselines, + completedWalkForwards, + readyDemoBridges, + learningInstruments, + shadowDecisions, + validatedInstruments, + hfmReady, + experimentRuns: experimentLedger?.runs.length ?? 0, + }, + stages: [ + { key: 'data', label: 'Broker data inspected', state: validatedInstruments === INSTRUMENTS.length ? 'complete' : hfmReady ? 'next' : 'waiting', detail: validationReport ? 'All current exports have a validation record; fallback-resolution files remain visibly labelled.' : 'Run the local MT5 validator before trusting a backtest.' }, + { key: 'baseline', label: 'Baseline trend study', state: completedBaselines > 0 ? 'complete' : 'waiting', detail: 'Holdout results are recorded separately from training.' }, + { key: 'walkforward', label: 'Rolling walk-forward', state: completedWalkForwards === INSTRUMENTS.length ? 'complete' : completedWalkForwards > 0 ? 'next' : 'waiting', detail: completedWalkForwards === INSTRUMENTS.length ? 'Sequential unseen windows were recorded for every configured instrument; review is still required.' : 'Required before a candidate can advance.' }, + { key: 'costs', label: 'Broker cost model', state: 'next', detail: 'Commission, financing, spread, and slippage still need verification.' }, + { key: 'bridge', label: 'MT5 demo bridge', state: readyDemoBridges === INSTRUMENTS.length ? 'complete' : readyDemoBridges > 0 ? 'next' : 'waiting', detail: readyDemoBridges > 0 ? `${readyDemoBridges}/${INSTRUMENTS.length} terminals report a fresh, demo-only read-only heartbeat.` : 'Attach the read-only MT5 bridge before any paper-execution work.' }, + { key: 'learning', label: 'Trade-history learning', state: learningInstruments === INSTRUMENTS.length ? 'complete' : learningInstruments > 0 ? 'next' : 'waiting', detail: learningInstruments > 0 ? `${learningInstruments}/${INSTRUMENTS.length} broker-symbol ledgers are fresh and demo-only.` : 'Run the MT5 trade ledger exporter so manual and demo trades can be reviewed.' }, + { key: 'shadow', label: 'JMB shadow decisions', state: shadowDecisions === INSTRUMENTS.length ? 'complete' : shadowDecisions > 0 ? 'next' : 'waiting', detail: shadowDecisions > 0 ? `${shadowDecisions}/${INSTRUMENTS.length} broker-symbol pairs have logged JMB decisions.` : 'Run the shadow decision runner before enabling the demo risk shell.' }, + { + key: 'demo', + label: 'Demo forward test', + state: protectedDemoPositions === goldExecutions.length ? 'complete' : brokerLocalExecutionStatuses > 0 ? 'next' : 'blocked', + detail: `${brokerLocalExecutionStatuses}/${goldExecutions.length} Gold broker-local MT5 lifecycle statuses are valid. Research Desk remains read-only; demo results are not live approval.`, + }, + ], + instruments, + experiments: [...(experimentLedger?.runs ?? [])].sort((a, b) => b.created_at.localeCompare(a.created_at)).slice(0, 24), + news, + disclaimer: 'Evidence grades measure completed validation steps, not a probability of profit or a trade recommendation.', + }) + }) + + return app +} diff --git a/tools/mt5/ConfigureJmbGoldmineDemoPolicy.mq5 b/tools/mt5/ConfigureJmbGoldmineDemoPolicy.mq5 new file mode 100644 index 000000000..394ddea77 --- /dev/null +++ b/tools/mt5/ConfigureJmbGoldmineDemoPolicy.mq5 @@ -0,0 +1,167 @@ +#property strict +#property script_show_inputs + +input string InpPolicyVersion = "operator-policy-v1"; +input string InpRolloutStage = "status_only"; +input bool InpCandidateApproved = false; +input int InpCompletedObservationMaxAgeHours = 72; +input double InpMaxSpread = 0.75; +input double InpMaxDeviation = 0.50; +input double InpMaxRiskAmount = 10.00; +input double InpMaxDailyLoss = 40.00; +input int InpMaxDailyLosingTrades = 4; +input double InpMaxVolume = 0.01; + +const string POLICY_ROOT = "OpenAliceMt5DemoPolicyV1"; +const string POLICY_SYMBOL = "XAUUSD"; +const string STRATEGY_VERSION = "daily-trend-v1"; + +struct BrokerPolicyBinding +{ + string broker; + string server; + long magic_number; + double max_spread; + double max_deviation; +}; + +bool ResolveBrokerBinding(BrokerPolicyBinding &binding) +{ + string server = AccountInfoString(ACCOUNT_SERVER); + if(server == "HFMarketsGlobal-Demo4") + { + binding.broker = "hfmarkets"; + binding.server = server; + binding.magic_number = 880101; + binding.max_spread = 0.75; + binding.max_deviation = 0.50; + return true; + } + if(server == "ICMarketsSC-Demo") + { + binding.broker = "icmarkets"; + binding.server = server; + binding.magic_number = 880201; + binding.max_spread = 0.30; + binding.max_deviation = 0.30; + return true; + } + return false; +} + +bool IsSafePolicyVersion() +{ + string canonical = InpPolicyVersion; + StringTrimLeft(canonical); + StringTrimRight(canonical); + return canonical != "" + && canonical == InpPolicyVersion + && StringFind(canonical, ",") < 0 + && StringFind(canonical, "\"") < 0 + && StringFind(canonical, "\r") < 0 + && StringFind(canonical, "\n") < 0; +} + +bool IsAllowedRolloutStage(const BrokerPolicyBinding &binding) +{ + if(InpRolloutStage == "status_only") return true; + if(InpRolloutStage == "both_demo") return true; + if(binding.broker == "hfmarkets" && InpRolloutStage == "hfm_canary") return true; + if(binding.broker == "icmarkets" && InpRolloutStage == "ic_canary") return true; + return false; +} + +bool IsPositiveFinite(const double value) +{ + return MathIsValidNumber(value) && value > 0.0; +} + +bool ValidateLimits(const BrokerPolicyBinding &binding) +{ + return InpCompletedObservationMaxAgeHours > 0 + && InpCompletedObservationMaxAgeHours <= 72 + && IsPositiveFinite(InpMaxSpread) + && InpMaxSpread <= binding.max_spread + && IsPositiveFinite(InpMaxDeviation) + && InpMaxDeviation <= binding.max_deviation + && IsPositiveFinite(InpMaxRiskAmount) + && InpMaxRiskAmount <= 10.00 + && IsPositiveFinite(InpMaxDailyLoss) + && InpMaxDailyLoss <= 40.00 + && InpMaxDailyLosingTrades > 0 + && InpMaxDailyLosingTrades <= 4 + && IsPositiveFinite(InpMaxVolume) + && InpMaxVolume <= 0.01; +} + +bool WritePolicy(const BrokerPolicyBinding &binding) +{ + string directory = POLICY_ROOT + "\\" + binding.broker + "\\" + POLICY_SYMBOL; + FolderCreate(POLICY_ROOT, FILE_COMMON); + FolderCreate(POLICY_ROOT + "\\" + binding.broker, FILE_COMMON); + FolderCreate(directory, FILE_COMMON); + + string final_path = directory + "\\policy.csv"; + string temporary_path = final_path + "." + IntegerToString((int)GetTickCount()) + ".tmp"; + int handle = FileOpen(temporary_path, FILE_WRITE | FILE_CSV | FILE_ANSI | FILE_COMMON, ','); + if(handle == INVALID_HANDLE) + { + PrintFormat("Policy temporary file could not be opened. Error %d", GetLastError()); + return false; + } + + FileWrite(handle,"schema_version","policy_version","broker","server","symbol","strategy_version","rollout_stage","candidate_approved","completed_observation_max_age_hours","max_spread","max_deviation","max_risk_amount","max_daily_loss","max_daily_losing_trades","max_volume","magic_number"); + FileWrite(handle, + 1, + InpPolicyVersion, + binding.broker, + binding.server, + POLICY_SYMBOL, + STRATEGY_VERSION, + InpRolloutStage, + InpCandidateApproved ? 1 : 0, + InpCompletedObservationMaxAgeHours, + InpMaxSpread, + InpMaxDeviation, + InpMaxRiskAmount, + InpMaxDailyLoss, + InpMaxDailyLosingTrades, + InpMaxVolume, + binding.magic_number); + FileFlush(handle); + FileClose(handle); + + if(FileMove(temporary_path, FILE_COMMON, final_path, FILE_COMMON | FILE_REWRITE)) return true; + PrintFormat("Policy file could not be replaced. Error %d", GetLastError()); + FileDelete(temporary_path, FILE_COMMON); + return false; +} + +void OnStart() +{ + if((ENUM_ACCOUNT_TRADE_MODE)AccountInfoInteger(ACCOUNT_TRADE_MODE) != ACCOUNT_TRADE_MODE_DEMO) + { + Print("Policy configuration refused: the current account is not demo."); + return; + } + if(_Symbol != POLICY_SYMBOL) + { + Print("Policy configuration refused: only the exact XAUUSD symbol is supported."); + return; + } + + BrokerPolicyBinding binding; + if(!ResolveBrokerBinding(binding)) + { + Print("Policy configuration refused: the current demo server is not allowlisted."); + return; + } + if(!IsSafePolicyVersion() || !IsAllowedRolloutStage(binding) || !ValidateLimits(binding)) + { + Print("Policy configuration refused: inputs violate the immutable policy contract."); + return; + } + + if(WritePolicy(binding)) + PrintFormat("Operator demo policy written for %s / %s at stage %s.", binding.broker, POLICY_SYMBOL, InpRolloutStage); +} diff --git a/tools/mt5/ExportMt5TradeLedger.mq5 b/tools/mt5/ExportMt5TradeLedger.mq5 new file mode 100644 index 000000000..511d85f81 --- /dev/null +++ b/tools/mt5/ExportMt5TradeLedger.mq5 @@ -0,0 +1,119 @@ +#property strict +#property script_show_inputs + +input string InpBrokerId = "hfmarkets"; +input string InpSymbol = "XAUUSD"; +input int InpHistoryDays = 30; + +string AccountModeLabel() +{ + long mode = AccountInfoInteger(ACCOUNT_TRADE_MODE); + if(mode == ACCOUNT_TRADE_MODE_DEMO) return "demo"; + if(mode == ACCOUNT_TRADE_MODE_REAL) return "real"; + return "contest"; +} + +string DealEntryLabel(long value) +{ + if(value == DEAL_ENTRY_IN) return "in"; + if(value == DEAL_ENTRY_OUT) return "out"; + if(value == DEAL_ENTRY_INOUT) return "inout"; + if(value == DEAL_ENTRY_OUT_BY) return "out_by"; + return IntegerToString((int)value); +} + +string DealTypeLabel(long value) +{ + if(value == DEAL_TYPE_BUY) return "buy"; + if(value == DEAL_TYPE_SELL) return "sell"; + if(value == DEAL_TYPE_BALANCE) return "balance"; + if(value == DEAL_TYPE_CREDIT) return "credit"; + if(value == DEAL_TYPE_CHARGE) return "charge"; + if(value == DEAL_TYPE_CORRECTION) return "correction"; + return IntegerToString((int)value); +} + +string DealReasonLabel(long value) +{ + if(value == DEAL_REASON_CLIENT) return "client"; + if(value == DEAL_REASON_MOBILE) return "mobile"; + if(value == DEAL_REASON_WEB) return "web"; + if(value == DEAL_REASON_EXPERT) return "expert"; + return IntegerToString((int)value); +} + +string CsvEscape(string value) +{ + StringReplace(value, "\"", "\"\""); + if(StringFind(value, ",") >= 0 || StringFind(value, "\"") >= 0) + return "\"" + value + "\""; + return value; +} + +void OnStart() +{ + string symbol = InpSymbol == "" ? _Symbol : InpSymbol; + datetime toTime = TimeCurrent(); + datetime fromTime = toTime - (InpHistoryDays * 86400); + if(!HistorySelect(fromTime, toTime)) + { + Print("HistorySelect failed: ", GetLastError()); + return; + } + + string directory = "OpenAliceMt5TradeLedgerV1\\" + InpBrokerId + "\\" + symbol; + FolderCreate("OpenAliceMt5TradeLedgerV1", FILE_COMMON); + FolderCreate("OpenAliceMt5TradeLedgerV1\\" + InpBrokerId, FILE_COMMON); + FolderCreate(directory, FILE_COMMON); + + string path = directory + "\\deals.csv"; + int handle = FileOpen(path, FILE_WRITE | FILE_CSV | FILE_COMMON | FILE_ANSI, ','); + if(handle == INVALID_HANDLE) + { + Print("FileOpen failed for ", path, ": ", GetLastError()); + return; + } + + FileWrite(handle, "account_mode", "server", "login", "broker", "symbol", "deal_ticket", "order_ticket", "position_id", "time", "entry", "type", "reason", "volume", "price", "commission", "fee", "swap", "profit", "magic", "comment"); + + int total = HistoryDealsTotal(); + for(int index = 0; index < total; index++) + { + ulong ticket = HistoryDealGetTicket(index); + string dealSymbol = HistoryDealGetString(ticket, DEAL_SYMBOL); + if(dealSymbol != symbol) continue; + + datetime dealTime = (datetime)HistoryDealGetInteger(ticket, DEAL_TIME); + string isoTime = TimeToString(dealTime, TIME_DATE | TIME_SECONDS); + StringReplace(isoTime, ".", "-"); + StringReplace(isoTime, " ", "T"); + isoTime = isoTime + ".000Z"; + + FileWrite( + handle, + AccountModeLabel(), + AccountInfoString(ACCOUNT_SERVER), + IntegerToString((int)AccountInfoInteger(ACCOUNT_LOGIN)), + InpBrokerId, + symbol, + IntegerToString((long)ticket), + IntegerToString((long)HistoryDealGetInteger(ticket, DEAL_ORDER)), + IntegerToString((long)HistoryDealGetInteger(ticket, DEAL_POSITION_ID)), + isoTime, + DealEntryLabel(HistoryDealGetInteger(ticket, DEAL_ENTRY)), + DealTypeLabel(HistoryDealGetInteger(ticket, DEAL_TYPE)), + DealReasonLabel(HistoryDealGetInteger(ticket, DEAL_REASON)), + DoubleToString(HistoryDealGetDouble(ticket, DEAL_VOLUME), 2), + DoubleToString(HistoryDealGetDouble(ticket, DEAL_PRICE), _Digits), + DoubleToString(HistoryDealGetDouble(ticket, DEAL_COMMISSION), 2), + DoubleToString(HistoryDealGetDouble(ticket, DEAL_FEE), 2), + DoubleToString(HistoryDealGetDouble(ticket, DEAL_SWAP), 2), + DoubleToString(HistoryDealGetDouble(ticket, DEAL_PROFIT), 2), + IntegerToString((long)HistoryDealGetInteger(ticket, DEAL_MAGIC)), + CsvEscape(HistoryDealGetString(ticket, DEAL_COMMENT)) + ); + } + + FileClose(handle); + Print("JMB Goldmine trade ledger exported: ", path); +} diff --git a/tools/mt5/JmbGoldmineDemoCanary/JmbCanaryCsv.mqh b/tools/mt5/JmbGoldmineDemoCanary/JmbCanaryCsv.mqh new file mode 100644 index 000000000..030a5d00c --- /dev/null +++ b/tools/mt5/JmbGoldmineDemoCanary/JmbCanaryCsv.mqh @@ -0,0 +1,562 @@ +#ifndef OPENALICE_JMB_CANARY_CSV_MQH +#define OPENALICE_JMB_CANARY_CSV_MQH + +#include "JmbCanaryTypes.mqh" + +const string CANARY_DECISION_HEADER="schema_version,decision_id,observation_id,observation_as_of,created_at,lease_issued_at,lease_expires_at,broker,server,account_mode,symbol,strategy_version,direction,entry_reference_price,volume,stop_loss,max_risk_amount,candidate_policy_version,cost_model_version,gate_results_json"; + +void AddCanaryCsvCell(string &cells[],const string value) +{ + int size=ArraySize(cells); + ArrayResize(cells,size+1); + cells[size]=value; +} + +bool ParseCanaryCsvRecord(const string line,string &cells[],string &detail) +{ + ArrayResize(cells,0); + string current=""; + bool quoted=false; + bool quote_closed=false; + + for(int index=0;index0 && StringSubstr(line,length-1,1)=="\r") + line=StringSubstr(line,0,length-1); + if(StringFind(line,"\r")>=0) + { + detail="CSV physical multiline values are forbidden."; + return false; + } + return true; +} + +bool SplitCanaryCsvDocument(const string text,string &header_line,string &value_line,string &detail) +{ + int first_line_end=StringFind(text,"\n"); + if(first_line_end<0) + { + detail="CSV must contain exactly one header and one data row."; + return false; + } + + header_line=StringSubstr(text,0,first_line_end); + string remainder=StringSubstr(text,first_line_end+1); + int second_line_end=StringFind(remainder,"\n"); + if(second_line_end>=0) + { + if(second_line_end!=StringLen(remainder)-1) + { + detail="CSV physical multiline values or extra rows are forbidden."; + return false; + } + value_line=StringSubstr(remainder,0,second_line_end); + } + else + { + value_line=remainder; + } + + if(!StripCanaryLineEnding(header_line,detail) || !StripCanaryLineEnding(value_line,detail)) return false; + if(header_line=="" || value_line=="") + { + detail="CSV header and data row must both be present."; + return false; + } + return true; +} + +bool ReadCanaryCommonText(const string path,string &text,string &detail) +{ + int handle=FileOpen(path,FILE_READ|FILE_BIN|FILE_COMMON|FILE_ANSI); + if(handle==INVALID_HANDLE) + { + detail="Required Common Files input is missing or unreadable."; + return false; + } + ulong size=FileSize(handle); + if(size<=0 || size>1024*1024) + { + FileClose(handle); + detail="Required Common Files input has an invalid size."; + return false; + } + text=FileReadString(handle,(int)size); + FileClose(handle); + if(StringLen(text)==0) + { + detail="Required Common Files input is empty."; + return false; + } + return true; +} + +int FindCanaryHeader(const string &headers[],const string name) +{ + int found=-1; + for(int index=0;index=0) return -2; + found=index; + } + return found; +} + +bool ReadStrictCanaryCsvText(const string text,const string &expected_fields[],string &ordered_values[],string &detail) +{ + string header_line=""; + string value_line=""; + if(!SplitCanaryCsvDocument(text,header_line,value_line,detail)) return false; + + string headers[]; + string values[]; + if(!ParseCanaryCsvRecord(header_line,headers,detail) || !ParseCanaryCsvRecord(value_line,values,detail)) return false; + if(ArraySize(headers)!=ArraySize(expected_fields) || ArraySize(values)!=ArraySize(headers)) + { + detail="CSV schema field count does not match the exact contract."; + return false; + } + + for(int left=0;left=0 || StringFind(value,"\n")>=0) return false; + string trimmed=value; + StringTrimLeft(trimmed); + StringTrimRight(trimmed); + return trimmed==value; +} + +bool TryCanaryDouble(const string value,double &result) +{ + if(value=="") return false; + bool has_digit=false; + bool has_decimal=false; + bool has_exponent=false; + bool exponent_needs_digit=false; + + for(int index=0;index=48 && character<=57) + { + has_digit=true; + exponent_needs_digit=false; + continue; + } + if((character==43 || character==45) && (index==0 || (index>0 && (StringSubstr(value,index-1,1)=="e" || StringSubstr(value,index-1,1)=="E")))) + continue; + if(character==46 && !has_decimal && !has_exponent) + { + has_decimal=true; + continue; + } + if((character==101 || character==69) && has_digit && !has_exponent) + { + has_exponent=true; + exponent_needs_digit=true; + continue; + } + return false; + } + if(!has_digit || exponent_needs_digit) return false; + result=StringToDouble(value); + return MathIsValidNumber(result); +} + +bool TryCanaryLong(const string value,long &result) +{ + if(value=="") return false; + int start=0; + if(StringSubstr(value,0,1)=="-") start=1; + if(start>=StringLen(value)) return false; + for(int index=start;index57) return false; + } + result=StringToInteger(value); + return true; +} + +bool TryCanaryIsoUtc(const string value,datetime &result) +{ + int length=StringLen(value); + if(length<20 || StringSubstr(value,4,1)!="-" || StringSubstr(value,7,1)!="-" + || StringSubstr(value,10,1)!="T" || StringSubstr(value,length-1,1)!="Z") return false; + if(length>20) + { + if(length<22 || StringSubstr(value,19,1)!=".") return false; + for(int index=20;index57) return false; + } + } + string normalized=StringSubstr(value,0,19); + StringReplace(normalized,"T"," "); + StringReplace(normalized,"-","."); + result=StringToTime(normalized); + if(result<=0) return false; + MqlDateTime parts; + TimeToStruct(result,parts); + string round_trip=StringFormat("%04d-%02d-%02dT%02d:%02d:%02d", + parts.year,parts.mon,parts.day,parts.hour,parts.min,parts.sec); + return round_trip==StringSubstr(value,0,19); +} + +bool IsCanaryIsoDate(const string value) +{ + if(StringLen(value)!=10 || StringSubstr(value,4,1)!="-" || StringSubstr(value,7,1)!="-") return false; + datetime parsed=StringToTime(StringSubstr(value,0,4)+"."+StringSubstr(value,5,2)+"."+StringSubstr(value,8,2)); + if(parsed<=0) return false; + MqlDateTime parts; + TimeToStruct(parsed,parts); + return StringFormat("%04d-%02d-%02d",parts.year,parts.mon,parts.day)==value; +} + +bool ConsumeCanaryJsonToken(const string json,int &cursor,const string token) +{ + if(StringSubstr(json,cursor,StringLen(token))!=token) return false; + cursor+=StringLen(token); + return true; +} + +int CanaryHexDigit(const int character) +{ + if(character>=48 && character<=57) return character-48; + if(character>=65 && character<=70) return character-65+10; + if(character>=97 && character<=102) return character-97+10; + return -1; +} + +bool ParseCanaryJsonCodeUnit(const string json,int &cursor,int &code_unit) +{ + if(cursor+4>StringLen(json)) return false; + code_unit=0; + for(int index=0;index<4;index++) + { + int digit=CanaryHexDigit((int)StringGetCharacter(json,cursor+index)); + if(digit<0) return false; + code_unit=code_unit*16+digit; + } + cursor+=4; + return true; +} + +bool ParseCanaryUnicodeEscape(const string json,int &cursor,string &decoded) +{ + decoded=""; + int first=0; + if(!ParseCanaryJsonCodeUnit(json,cursor,first)) return false; + if(first>=0xDC00 && first<=0xDFFF) return false; + if(first>=0xD800 && first<=0xDBFF) + { + if(!ConsumeCanaryJsonToken(json,cursor,"\\u")) return false; + int second=0; + if(!ParseCanaryJsonCodeUnit(json,cursor,second) || second<0xDC00 || second>0xDFFF) return false; + decoded=ShortToString((ushort)first)+ShortToString((ushort)second); + return true; + } + decoded=ShortToString((ushort)first); + return true; +} + +bool ParseCanonicalCanaryJsonString(const string json,int &cursor,string &value) +{ + value=""; + if(!ConsumeCanaryJsonToken(json,cursor,"\"")) return false; + while(cursor=StringLen(json)) return false; + string escaped=StringSubstr(json,cursor,1); + cursor++; + if(escaped=="\"" || escaped=="\\" || escaped=="/") value+=escaped; + else if(escaped=="b") value+=ShortToString((ushort)8); + else if(escaped=="f") value+=ShortToString((ushort)12); + else if(escaped=="n") value+=ShortToString((ushort)10); + else if(escaped=="r") value+=ShortToString((ushort)13); + else if(escaped=="t") value+=ShortToString((ushort)9); + else if(escaped=="u") + { + string decoded=""; + if(!ParseCanaryUnicodeEscape(json,cursor,decoded)) return false; + value+=decoded; + } + else return false; + continue; + } + int code=(int)StringGetCharacter(character,0); + if(code<32) return false; + value+=character; + } + return false; +} + +bool ParseCanaryGateResultsJson(const string json,bool &all_passed,string &detail) +{ + all_passed=true; + if(StringFind(json,"\r")>=0 || StringFind(json,"\n")>=0) + { + detail="Execution decision gate JSON contains physical multiline evidence."; + return false; + } + string expected_names[]={"bridge","completed_observation","candidate_policy","cost_model", + "learning","quote","spread","direction","stop_loss"}; + int cursor=0; + if(!ConsumeCanaryJsonToken(json,cursor,"[")) return false; + for(int index=0;index0 && !ConsumeCanaryJsonToken(json,cursor,",")) return false; + if(!ConsumeCanaryJsonToken(json,cursor,"{")) return false; + string key=""; + string name=""; + string state=""; + string evidence_detail=""; + if(!ParseCanonicalCanaryJsonString(json,cursor,key) || key!="name" + || !ConsumeCanaryJsonToken(json,cursor,":") + || !ParseCanonicalCanaryJsonString(json,cursor,name) || name!=expected_names[index] + || !ConsumeCanaryJsonToken(json,cursor,",") + || !ParseCanonicalCanaryJsonString(json,cursor,key) || key!="state" + || !ConsumeCanaryJsonToken(json,cursor,":") + || !ParseCanonicalCanaryJsonString(json,cursor,state) + || (state!="pass" && state!="block") + || !ConsumeCanaryJsonToken(json,cursor,",") + || !ParseCanonicalCanaryJsonString(json,cursor,key) || key!="detail" + || !ConsumeCanaryJsonToken(json,cursor,":") + || !ParseCanonicalCanaryJsonString(json,cursor,evidence_detail) + || !IsCanonicalCanaryText(evidence_detail) + || !ConsumeCanaryJsonToken(json,cursor,"}")) + { + detail="Execution decision gate JSON violates the exact object contract."; + return false; + } + if(state=="block") all_passed=false; + } + if(!ConsumeCanaryJsonToken(json,cursor,"]") || cursor!=StringLen(json)) + { + detail="Execution decision gate JSON contains unknown, duplicate, missing, or trailing evidence."; + return false; + } + return true; +} + +string CanarySha256Identity(const string source_text) +{ + uchar source[]; + uchar key[]; + uchar digest[]; + int copied=StringToCharArray(source_text,source,0,StringLen(source_text),CP_UTF8); + if(copied!=StringLen(source_text)) return ""; + ArrayResize(key,0); + if(CryptEncode(CRYPT_HASH_SHA256,source,key,digest)!=32) return ""; + string hex=""; + for(int index=0;index<12;index++) hex+=StringFormat("%02x",(int)digest[index]); + return hex; +} + +string CreateCanaryObservationId(const CanaryDecision &decision) +{ + return CanarySha256Identity(decision.broker+"|"+decision.symbol+"|" + +decision.strategyVersion+"|"+decision.observationAsOf); +} + +string CreateCanaryDecisionId(const CanaryDecision &decision) +{ + return CanarySha256Identity("daily-trend-v1|"+decision.observationId); +} + +bool ReadCanaryDecision(const string path,const datetime now,CanaryDecision &decision,string &detail) +{ + string expected[]; + string header_copy=CANARY_DECISION_HEADER; + if(StringSplit(header_copy,(ushort)StringGetCharacter(",",0),expected)!=20) + { + detail="Internal decision schema definition is invalid."; + return false; + } + string values[]; + if(!ReadStrictCanaryCsv(path,expected,values,detail)) return false; + + long schema=0; + double entry=0.0; + double volume=0.0; + double stop=0.0; + double max_risk=0.0; + bool all_application_gates_passed=false; + datetime created=0; + datetime issued=0; + datetime expires=0; + bool has_entry=values[13]!=""; + bool has_stop=values[15]!=""; + if(!TryCanaryLong(values[0],schema) || schema!=1 + || !TryCanaryIsoUtc(values[4],created) + || !TryCanaryIsoUtc(values[5],issued) + || !TryCanaryIsoUtc(values[6],expires) + || (has_entry && !TryCanaryDouble(values[13],entry)) + || !TryCanaryDouble(values[14],volume) + || (has_stop && !TryCanaryDouble(values[15],stop)) + || !TryCanaryDouble(values[16],max_risk)) + { + detail="Execution decision contains a malformed number or timestamp."; + return false; + } + if(!IsCanonicalCanaryText(values[1]) || !IsCanonicalCanaryText(values[2]) || !IsCanaryIsoDate(values[3]) + || created>issued || issued>=expires || nowexpires) + { + detail="Execution decision identity or lease interval is invalid or expired."; + return false; + } + string expected_server=values[7]=="hfmarkets" ? "HFMarketsGlobal-Demo4" + : values[7]=="icmarkets" ? "ICMarketsSC-Demo" : ""; + if(expected_server=="" || values[8]!=expected_server || values[9]!="demo" || values[10]!="XAUUSD" + || values[11]!="daily-trend-v1" || (values[12]!="buy" && values[12]!="sell" && values[12]!="flat")) + { + detail="Execution decision contains an enum or immutable allowlist violation."; + return false; + } + bool actionable=values[12]=="buy" || values[12]=="sell"; + if((actionable && (!has_entry || entry<=0.0 || !has_stop || stop<=0.0)) + || (!actionable && (has_entry || has_stop)) + || volume<=0.0 || max_risk<=0.0 || max_risk>CANARY_HARD_MAX_RISK + || !IsCanonicalCanaryText(values[17]) || !IsCanonicalCanaryText(values[18]) + || !ParseCanaryGateResultsJson(values[19],all_application_gates_passed,detail)) + { + detail="Execution decision contains invalid entry, risk, version, or gate evidence."; + return false; + } + + decision.loaded=true; + decision.schemaVersion=(int)schema; + decision.decisionId=values[1]; + decision.observationId=values[2]; + decision.observationAsOf=values[3]; + decision.createdAt=created; + decision.leaseIssuedAt=issued; + decision.leaseExpiresAt=expires; + decision.broker=values[7]; + decision.server=values[8]; + decision.accountMode=values[9]; + decision.symbol=values[10]; + decision.strategyVersion=values[11]; + decision.direction=values[12]; + decision.entryReferencePrice=entry; + decision.volume=volume; + decision.stopLoss=stop; + decision.maxRiskAmount=max_risk; + decision.candidatePolicyVersion=values[17]; + decision.costModelVersion=values[18]; + decision.gateResultsJson=values[19]; + decision.preDecisionGatesPassed=all_application_gates_passed; + if(decision.observationId!=CreateCanaryObservationId(decision) + || decision.decisionId!=CreateCanaryDecisionId(decision)) + { + decision.loaded=false; + detail="Execution decision IDs do not match the immutable Task 3 hash identity."; + return false; + } + detail="Execution decision lease matches the strict schema and current UTC lease window."; + return true; +} + +#endif diff --git a/tools/mt5/JmbGoldmineDemoCanary/JmbCanaryGates.mqh b/tools/mt5/JmbGoldmineDemoCanary/JmbCanaryGates.mqh new file mode 100644 index 000000000..fe351de15 --- /dev/null +++ b/tools/mt5/JmbGoldmineDemoCanary/JmbCanaryGates.mqh @@ -0,0 +1,216 @@ +#ifndef OPENALICE_JMB_CANARY_GATES_MQH +#define OPENALICE_JMB_CANARY_GATES_MQH + +#include "JmbCanaryTypes.mqh" + +CanaryGateResult Gate(const string name,const bool passed,const string detail) +{ + CanaryGateResult result; + result.name=name; + result.passed=passed; + result.detail=detail; + return result; +} + +void AppendCanaryGate(CanaryEvaluation &evaluation,CanaryGateResult &gate) +{ + int index=ArraySize(evaluation.gates); + ArrayResize(evaluation.gates,index+1); + evaluation.gates[index]=gate; +} + +string CanaryLifecycleLabel(const CanaryLifecycleState state) +{ + if(state==CANARY_LIFECYCLE_DISABLED) return "disabled"; + if(state==CANARY_LIFECYCLE_PAUSED) return "paused"; + if(state==CANARY_LIFECYCLE_READY) return "ready"; + return "blocked"; +} + +bool CanaryNearlyEqual(const double left,const double right) +{ + return MathAbs(left-right)<=0.00000001; +} + +CanaryEvaluation EvaluateCanaryGates(const CanaryDecision &decision, + const CanaryPolicy &policy, + const CanaryEnvironment &environment) +{ + CanaryEvaluation evaluation; + evaluation.state=CANARY_LIFECYCLE_BLOCKED; + evaluation.ready=false; + evaluation.detail="One or more dry-run safety gates blocked readiness."; + evaluation.blockingGate=""; + evaluation.nextSafeAction="Resolve the first blocking gate and evaluate again."; + ArrayResize(evaluation.gates,0); + + bool identity_passed=decision.loaded + && policy.loaded + && environment.accountIsDemo + && environment.loginMatches + && environment.serverMatches + && environment.brokerMatches + && environment.chartSymbolMatches + && environment.magicMatches; + CanaryGateResult identity=Gate("demo_identity",identity_passed, + identity_passed ? "Demo account and immutable identity bindings match." + : "Demo account or an immutable identity binding is missing or mismatched."); + AppendCanaryGate(evaluation,identity); + + bool switches_passed=environment.executionEnabled && !environment.killSwitch; + CanaryGateResult switches=Gate("switches",switches_passed, + switches_passed ? "The local execution switch is enabled and the kill switch is off." + : "The local execution switch is disabled or the kill switch is on."); + AppendCanaryGate(evaluation,switches); + + bool rollout_passed=environment.rolloutAuthorized && environment.candidateApproved; + CanaryGateResult rollout=Gate("rollout",rollout_passed, + rollout_passed ? "The operator policy authorizes this broker rollout." + : "The rollout stage or operator candidate approval blocks this broker."); + AppendCanaryGate(evaluation,rollout); + + bool allowlists_passed=environment.allowlistsMatch; + CanaryGateResult allowlists=Gate("allowlists",allowlists_passed, + allowlists_passed ? "Gold and daily-trend-v1 match the immutable allowlists." + : "The broker, server, symbol, strategy, direction, or magic allowlist failed."); + AppendCanaryGate(evaluation,allowlists); + + bool freshness_passed=environment.decisionFresh + && environment.bridgeFresh + && environment.policyFresh + && environment.costModelFresh + && environment.observationFresh + && environment.processedStateAvailable + && environment.observationUnused; + CanaryGateResult freshness=Gate("freshness",freshness_passed, + freshness_passed ? "Decision, bridge, policy, cost, and observation evidence is current and unused." + : "Freshness, broker evidence, or unique-observation evidence is missing or stale."); + AppendCanaryGate(evaluation,freshness); + + bool volume_passed=environment.volumeEvidenceAvailable + && environment.volumeCompatible + && CanaryNearlyEqual(decision.volume,CANARY_HARD_MAX_VOLUME) + && decision.volume<=policy.maxVolume + && decision.volume<=CANARY_HARD_MAX_VOLUME; + CanaryGateResult volume=Gate("volume",volume_passed, + volume_passed ? "The requested 0.01 volume matches policy and broker constraints." + : "Volume evidence is missing or the requested volume violates policy or broker constraints."); + AppendCanaryGate(evaluation,volume); + + double risk_ceiling=MathMin(CANARY_HARD_MAX_RISK,MathMin(policy.maxRiskAmount,decision.maxRiskAmount)); + bool stop_risk_passed=environment.stopEvidenceAvailable + && environment.stopModeSupportsSl + && environment.stopTickSizeAvailable + && environment.stopTickAligned + && environment.stopBrokerValid + && environment.riskCalculationAvailable + && decision.stopLoss>0.0 + && environment.calculatedStopRisk>0.0 + && environment.calculatedStopRisk<=risk_ceiling; + CanaryGateResult stop_risk=Gate("stop_risk",stop_risk_passed, + stop_risk_passed ? "The broker-valid stop and calculated account-currency risk are within the tighter ceiling." + : "Stop evidence or the account-currency risk calculation is unavailable or outside its ceiling."); + AppendCanaryGate(evaluation,stop_risk); + + double daily_ceiling=MathMin(CANARY_HARD_MAX_DAILY_LOSS,policy.maxDailyLoss); + int count_ceiling=MathMin(CANARY_HARD_MAX_DAILY_LOSSES,policy.maxDailyLosingTrades); + bool daily_passed=environment.dailyStateAvailable + && environment.dailyRealizedLoss>=0.0 + && environment.dailyRealizedLoss=0 + && environment.dailyLossCount0.0 + && environment.freeMargin>=environment.estimatedMargin + && environment.freeMargin-environment.estimatedMargin + >=environment.estimatedMargin*CANARY_MARGIN_BUFFER_MULTIPLIER; + CanaryGateResult margin=Gate("margin",margin_passed, + margin_passed ? "Estimated margin is available with the required ten-times post-estimate buffer." + : "Margin evidence is missing or the ten-times free-margin buffer would not remain."); + AppendCanaryGate(evaluation,margin); + + bool spread_deviation_passed=environment.spreadAvailable + && environment.deviationAvailable + && environment.currentSpread>=0.0 + && environment.currentSpread<=policy.maxSpread + && environment.requestedDeviation>=0.0 + && environment.requestedDeviation<=policy.maxDeviation; + CanaryGateResult spread_deviation=Gate("spread_deviation",spread_deviation_passed, + spread_deviation_passed ? "Current spread and requested deviation are within broker policy." + : "Spread or deviation evidence is missing or outside broker policy."); + AppendCanaryGate(evaluation,spread_deviation); + + bool session_passed=environment.sessionEvidenceAvailable && environment.sessionOpen; + CanaryGateResult session=Gate("session",session_passed, + session_passed ? "The current UTC time is inside the approved weekday session." + : "Session evidence is unavailable or the UTC entry session is closed."); + AppendCanaryGate(evaluation,session); + + bool news_passed=environment.newsEvidenceAvailable && !environment.newsBlackout; + CanaryGateResult news=Gate("news",news_passed, + news_passed ? "Calendar evidence is complete and no high-impact USD blackout is active." + : "Calendar evidence is unavailable or a high-impact USD blackout is active."); + AppendCanaryGate(evaluation,news); + + bool log_passed=environment.logPreflightReady; + CanaryGateResult log_preflight=Gate("log_preflight",log_passed, + log_passed ? "The durable Common Files path passed its write preflight." + : "The durable Common Files path is missing or failed its write preflight."); + AppendCanaryGate(evaluation,log_preflight); + + bool reconciliation_passed=environment.reconciliationComplete; + CanaryGateResult reconciliation=Gate("reconciliation",reconciliation_passed, + reconciliation_passed ? "Restart and unknown-state reconciliation evidence is complete." + : "Restart or unknown-state reconciliation evidence is incomplete."); + AppendCanaryGate(evaluation,reconciliation); + + for(int index=0;index=0 || StringFind(value,"\"")>=0 + || StringFind(value,"\r")>=0 || StringFind(value,"\n")>=0) return false; + string trimmed=value; + StringTrimLeft(trimmed); + StringTrimRight(trimmed); + return trimmed==value; +} + +bool ValidateCanaryPolicy(const CanaryPolicy &policy,string &detail) +{ + string allowed_server=CanaryAllowedServer(policy.broker); + long allowed_magic=CanaryAllowedMagic(policy.broker); + if(policy.schemaVersion!=1 || !IsCanonicalCanaryPolicyVersion(policy.policyVersion) + || allowed_server=="" || policy.server!=allowed_server || policy.symbol!="XAUUSD" + || policy.strategyVersion!="daily-trend-v1" || policy.magicNumber!=allowed_magic) + { + detail="Policy identity does not match the immutable demo allowlist."; + return false; + } + if(policy.rolloutStage!="status_only" && policy.rolloutStage!="hfm_canary" + && policy.rolloutStage!="ic_canary" && policy.rolloutStage!="both_demo") + { + detail="Policy rollout_stage enum is invalid."; + return false; + } + if(policy.completedObservationMaxAgeHours<=0.0 || policy.completedObservationMaxAgeHours>72.0 + || policy.maxSpread<=0.0 || policy.maxSpread>CanaryHardSpread(policy.broker) + || policy.maxDeviation<=0.0 || policy.maxDeviation>CanaryHardDeviation(policy.broker) + || policy.maxRiskAmount<=0.0 || policy.maxRiskAmount>CANARY_HARD_MAX_RISK + || policy.maxDailyLoss<=0.0 || policy.maxDailyLoss>CANARY_HARD_MAX_DAILY_LOSS + || policy.maxDailyLosingTrades<=0 || policy.maxDailyLosingTrades>CANARY_HARD_MAX_DAILY_LOSSES + || policy.maxVolume<=0.0 || policy.maxVolume>CANARY_HARD_MAX_VOLUME) + { + detail="Policy limit exceeds or invalidates an immutable hard ceiling."; + return false; + } + detail="Policy matches the exact schema and immutable hard ceilings."; + return true; +} + +bool ParseCanaryPolicyCsvText(const string policy_text,CanaryPolicy &policy,string &detail) +{ + if(StringFind(policy_text,"\"")>=0) + { + detail="Policy CSV physical quotes are forbidden."; + return false; + } + string expected[]; + string header_copy=CANARY_POLICY_HEADER; + if(StringSplit(header_copy,(ushort)StringGetCharacter(",",0),expected)!=16) + { + detail="Internal policy schema definition is invalid."; + return false; + } + string values[]; + if(!ReadStrictCanaryCsvText(policy_text,expected,values,detail)) return false; + + long schema=0; + long loss_count=0; + long magic=0; + double observation_age=0.0; + double spread=0.0; + double deviation=0.0; + double risk=0.0; + double daily_loss=0.0; + double volume=0.0; + if(!TryCanaryLong(values[0],schema) || schema!=1 + || !TryCanaryDouble(values[8],observation_age) + || !TryCanaryDouble(values[9],spread) + || !TryCanaryDouble(values[10],deviation) + || !TryCanaryDouble(values[11],risk) + || !TryCanaryDouble(values[12],daily_loss) + || !TryCanaryLong(values[13],loss_count) + || !TryCanaryDouble(values[14],volume) + || !TryCanaryLong(values[15],magic) + || (values[7]!="0" && values[7]!="1")) + { + detail="Policy contains a malformed number, boolean, or schema version."; + return false; + } + + policy.loaded=true; + policy.schemaVersion=(int)schema; + policy.policyVersion=values[1]; + policy.broker=values[2]; + policy.server=values[3]; + policy.symbol=values[4]; + policy.strategyVersion=values[5]; + policy.rolloutStage=values[6]; + policy.candidateApproved=values[7]=="1"; + policy.completedObservationMaxAgeHours=observation_age; + policy.maxSpread=spread; + policy.maxDeviation=deviation; + policy.maxRiskAmount=risk; + policy.maxDailyLoss=daily_loss; + policy.maxDailyLosingTrades=(int)loss_count; + policy.maxVolume=volume; + policy.magicNumber=magic; + + if(!ValidateCanaryPolicy(policy,detail)) + { + policy.loaded=false; + return false; + } + return true; +} + +bool ReadCanaryPolicy(const string path,CanaryPolicy &policy,string &detail) +{ + string policy_text=""; + if(!ReadCanaryCommonText(path,policy_text,detail)) return false; + return ParseCanaryPolicyCsvText(policy_text,policy,detail); +} + +#endif diff --git a/tools/mt5/JmbGoldmineDemoCanary/JmbCanaryReconcile.mqh b/tools/mt5/JmbGoldmineDemoCanary/JmbCanaryReconcile.mqh new file mode 100644 index 000000000..076851db5 --- /dev/null +++ b/tools/mt5/JmbGoldmineDemoCanary/JmbCanaryReconcile.mqh @@ -0,0 +1,694 @@ +#ifndef OPENALICE_JMB_CANARY_RECONCILE_MQH +#define OPENALICE_JMB_CANARY_RECONCILE_MQH + +#include "JmbCanaryState.mqh" + +CanaryLifecycleState ReduceCanaryLifecycle(const CanaryReconciliationFacts &facts) +{ + if(!facts.brokerStateAvailable) + return CANARY_LIFECYCLE_RECONCILIATION_REQUIRED; + if(facts.hasForeignGoldExposure) + return CANARY_LIFECYCLE_BLOCKED; + if(facts.hasEaPosition && !facts.eaPositionProtected) + return CANARY_LIFECYCLE_EMERGENCY_CLOSE; + if(facts.stoppedObservation) + return CANARY_LIFECYCLE_STOPPED; + if(facts.persistentSafetyPause) + return CANARY_LIFECYCLE_PAUSED; + if(facts.resultClass==CANARY_RESULT_UNKNOWN || facts.resultClass==CANARY_RESULT_PARTIAL + || facts.hasEaPendingOrder) + return CANARY_LIFECYCLE_RECONCILIATION_REQUIRED; + if(facts.oppositeDirection && facts.hasEaPosition) + return CANARY_LIFECYCLE_CLOSE_REQUESTING; + if(facts.hasEaPosition && facts.eaPositionProtected) + return CANARY_LIFECYCLE_FILLED_PROTECTED; + if(facts.closeConfirmed) + return CANARY_LIFECYCLE_CLOSED; + if(facts.dailyLimitReached) + return CANARY_LIFECYCLE_PAUSED; + if(facts.resultClass==CANARY_RESULT_REJECTED) + return CANARY_LIFECYCLE_ORDER_REJECTED; + return CANARY_LIFECYCLE_READY; +} + +bool IsCanaryActionableOpposite(const string decision_direction,const string position_direction) +{ + return (decision_direction=="buy" && position_direction=="sell") + || (decision_direction=="sell" && position_direction=="buy"); +} + +CanaryClosedOwnershipClass ClassifyCanaryClosedPositionOwnership( + const bool origin_is_ea,const bool has_foreign_entry,const bool has_nonmagic_close) +{ + if(!origin_is_ea) return CANARY_CLOSED_OWNERSHIP_FOREIGN; + if(has_foreign_entry) return CANARY_CLOSED_OWNERSHIP_UNSAFE; + return CANARY_CLOSED_OWNERSHIP_EA; +} + +bool IsCanaryCorrelatedLifecyclePosition(const string decision_id, + const ulong expected_position_id, + const string origin_comment, + const ulong candidate_position_id) +{ + return expected_position_id>0 && candidate_position_id==expected_position_id + && origin_comment==CanaryEntryCorrelationComment(decision_id); +} + +CanaryBrokerResultClass ClassifyCanaryBrokerResult(const bool sent,const uint retcode, + const double requested_volume, + const double accepted_volume) +{ + if(retcode==TRADE_RETCODE_DONE_PARTIAL + || (accepted_volume>0.0 && MathAbs(accepted_volume-requested_volume)>0.00000001)) + return CANARY_RESULT_PARTIAL; + if(retcode==TRADE_RETCODE_REQUOTE || retcode==TRADE_RETCODE_REJECT || retcode==TRADE_RETCODE_CANCEL + || retcode==TRADE_RETCODE_INVALID || retcode==TRADE_RETCODE_INVALID_VOLUME + || retcode==TRADE_RETCODE_INVALID_PRICE || retcode==TRADE_RETCODE_INVALID_STOPS + || retcode==TRADE_RETCODE_TRADE_DISABLED || retcode==TRADE_RETCODE_MARKET_CLOSED + || retcode==TRADE_RETCODE_NO_MONEY || retcode==TRADE_RETCODE_PRICE_CHANGED + || retcode==TRADE_RETCODE_PRICE_OFF || retcode==TRADE_RETCODE_INVALID_EXPIRATION + || retcode==TRADE_RETCODE_NO_CHANGES || retcode==TRADE_RETCODE_SERVER_DISABLES_AT + || retcode==TRADE_RETCODE_CLIENT_DISABLES_AT || retcode==TRADE_RETCODE_INVALID_FILL + || retcode==TRADE_RETCODE_ONLY_REAL || retcode==TRADE_RETCODE_LIMIT_ORDERS + || retcode==TRADE_RETCODE_LIMIT_VOLUME || retcode==TRADE_RETCODE_INVALID_ORDER + || retcode==TRADE_RETCODE_POSITION_CLOSED || retcode==TRADE_RETCODE_INVALID_CLOSE_VOLUME + || retcode==TRADE_RETCODE_LIMIT_POSITIONS || retcode==TRADE_RETCODE_REJECT_CANCEL + || retcode==TRADE_RETCODE_LONG_ONLY || retcode==TRADE_RETCODE_SHORT_ONLY + || retcode==TRADE_RETCODE_CLOSE_ONLY || retcode==TRADE_RETCODE_FIFO_CLOSE + || retcode==TRADE_RETCODE_HEDGE_PROHIBITED) + return CANARY_RESULT_REJECTED; + if(!sent || retcode==0 || retcode==TRADE_RETCODE_TIMEOUT + || retcode==TRADE_RETCODE_CONNECTION || retcode==TRADE_RETCODE_TOO_MANY_REQUESTS + || retcode==TRADE_RETCODE_LOCKED || retcode==TRADE_RETCODE_FROZEN + || retcode==TRADE_RETCODE_PLACED || retcode==TRADE_RETCODE_DONE) + return CANARY_RESULT_UNKNOWN; + return CANARY_RESULT_UNKNOWN; +} + +void InitializeCanaryPositionSnapshot(CanaryPositionSnapshot &position) +{ + ZeroMemory(position); + position.direction=""; +} + +void InitializeCanaryReconciliation(CanaryReconciliation &reconciliation) +{ + ZeroMemory(reconciliation); + reconciliation.state=CANARY_LIFECYCLE_RECONCILIATION_REQUIRED; + reconciliation.reconciliationState="required"; + InitializeCanaryPositionSnapshot(reconciliation.position); +} + +bool CanaryPositionHasValidStop(const string direction,const double open_price,const double stop_loss) +{ + if(!MathIsValidNumber(open_price) || open_price<=0.0 + || !MathIsValidNumber(stop_loss) || stop_loss<=0.0) return false; + return direction=="buy" ? stop_lossopen_price : false; +} + +bool CanaryOpenPositionOwnershipExact(const string symbol,const long magic_number, + const ulong position_id,bool &exact_owned) +{ + exact_owned=false; + if(position_id==0 || !HistorySelectByPosition(position_id)) return false; + bool found_expected=false; + int total=HistoryDealsTotal(); + for(int index=0;index0) + { + found=true; + position_id=candidate; + return true; + } + } + } + + datetime search_start=latch.pendingEntryAttemptedAt>300 + ? latch.pendingEntryAttemptedAt-300 : latch.pendingEntryAttemptedAt; + if(!HistorySelect(search_start,now)) return false; + int total=HistoryDealsTotal(); + for(int index=0;index=final_close_time) + { + final_close_time=deal_time; + final_deal_ticket=deal_ticket; + final_reason=HistoryDealGetInteger(deal_ticket,DEAL_REASON); + accepted_price=HistoryDealGetDouble(deal_ticket,DEAL_PRICE); + } + } + if(final_close_time<=0) return false; + closed=true; + reconciliation.hasClosedPosition=true; + reconciliation.finalCloseTime=final_close_time; + reconciliation.dealTicket=final_deal_ticket; + reconciliation.closedPositionId=position_id; + reconciliation.lastCloseWasStop=final_reason==DEAL_REASON_SL; + reconciliation.acceptedPrice=accepted_price; + reconciliation.commission=commission; + reconciliation.swap=swap; + reconciliation.fee=fee; + reconciliation.netResult=net_result; + return true; +} + +bool ReadCanaryClosedPositionResults(const string symbol,const long magic_number, + const datetime day_start,const datetime now, + CanaryDailyState &daily,CanaryReconciliation &reconciliation) +{ + daily.brokerDay=""; + daily.lossCount=0; + daily.realizedLoss=0.0; + if(day_start<=0 || nownow) continue; + int found=FindCanaryPositionId(position_ids,position_id); + if(found<0) + { + int size=ArraySize(position_ids); + ArrayResize(position_ids,size+1); + ArrayResize(final_close_times,size+1); + position_ids[size]=position_id; + final_close_times[size]=close_time; + } + else if(close_time>final_close_times[found]) final_close_times[found]=close_time; + } + + for(int position_index=0;position_index=final_close_time) + { + final_close_time=deal_time; + final_deal_ticket=deal_ticket; + final_reason=HistoryDealGetInteger(deal_ticket,DEAL_REASON); + accepted_price=HistoryDealGetDouble(deal_ticket,DEAL_PRICE); + } + } + if(final_close_timenow) continue; + if(net_result<0.0) + { + daily.lossCount++; + daily.realizedLoss+=MathAbs(net_result); + } + if(!reconciliation.hasClosedPosition || final_close_time>reconciliation.finalCloseTime) + { + reconciliation.hasClosedPosition=true; + reconciliation.finalCloseTime=final_close_time; + reconciliation.dealTicket=final_deal_ticket; + reconciliation.closedPositionId=position_id; + reconciliation.lastCloseWasStop=final_reason==DEAL_REASON_SL; + reconciliation.acceptedPrice=accepted_price; + reconciliation.commission=commission; + reconciliation.swap=swap; + reconciliation.fee=fee; + reconciliation.netResult=net_result; + } + } + daily.brokerDay=CanaryDayKey(day_start); + return HistorySelect(day_start,now); +} + +datetime CanaryServerDayStart(const datetime server_now) +{ + MqlDateTime parts; + if(server_now<=0 || !TimeToStruct(server_now,parts)) return 0; + parts.hour=0; + parts.min=0; + parts.sec=0; + return StructToTime(parts); +} + +bool ReconcileCanaryBrokerState(const string symbol,const long magic_number, + const CanaryDecision &decision,const bool observation_used, + const CanaryBrokerResultClass result_class, + const CanarySafetyLatch &latch, + CanaryReconciliation &reconciliation) +{ + InitializeCanaryReconciliation(reconciliation); + int ea_position_count=0; + if(symbol!="XAUUSD" || (magic_number!=880101 && magic_number!=880201) + || !ScanCanaryGoldExposure(symbol,magic_number,reconciliation.position, + ea_position_count,reconciliation.hasEaPendingOrder,reconciliation.hasForeignGoldExposure)) + { + reconciliation.detail="Authoritative Gold exposure could not be classified."; + return false; + } + + datetime now=TimeTradeServer(); + datetime day_start=CanaryServerDayStart(now); + if(!ReadCanaryClosedPositionResults(symbol,magic_number,day_start,now, + reconciliation.daily,reconciliation)) + { + reconciliation.detail="Authoritative broker-day deal history could not be reconciled."; + return false; + } + + bool correlated_entry_position=false; + bool correlated_entry_closed=false; + ulong correlated_entry_position_id=0; + string correlated_entry_decision_id=""; + if(latch.activePositionDecisionId!="") + { + correlated_entry_position=true; + correlated_entry_decision_id=latch.activePositionDecisionId; + correlated_entry_position_id=(ulong)StringToInteger(latch.activePositionId); + if(correlated_entry_position_id==0 + || !ReadCanaryLifecyclePositionById(symbol,magic_number,correlated_entry_position_id, + correlated_entry_decision_id,reconciliation,correlated_entry_closed) + || (!correlated_entry_closed && (!reconciliation.position.present + || reconciliation.position.identifier!=correlated_entry_position_id))) + { + reconciliation.detail="The active opening correlation could not recover its exact position lifecycle."; + return false; + } + } + else if(latch.pendingEntryDecisionId!="") + { + correlated_entry_decision_id=latch.pendingEntryDecisionId; + if(!FindCanaryPendingEntryPosition(latch,symbol,magic_number,now, + correlated_entry_position,correlated_entry_position_id)) + { + reconciliation.detail="The pending entry identity could not be authoritatively correlated."; + return false; + } + if(correlated_entry_position + && !ReadCanaryLifecyclePositionById(symbol,magic_number,correlated_entry_position_id, + correlated_entry_decision_id,reconciliation,correlated_entry_closed)) + { + reconciliation.detail="The correlated entry position lifecycle could not be recovered."; + return false; + } + } + bool emergency_position_closed=false; + if(latch.emergencyPositionId!="") + { + ulong emergency_position_id=(ulong)StringToInteger(latch.emergencyPositionId); + if(!ReadCanaryLifecyclePositionById(symbol,magic_number,emergency_position_id,"", + reconciliation,emergency_position_closed)) + { + reconciliation.detail="The emergency position lifecycle could not be recovered."; + return false; + } + } + + CanaryReconciliationFacts facts; + ZeroMemory(facts); + facts.brokerStateAvailable=ea_position_count<=1; + facts.resultClass=result_class; + facts.hasEaPosition=reconciliation.position.present; + facts.eaPositionProtected=reconciliation.position.stopProtected; + if(reconciliation.position.present + && MathAbs(reconciliation.position.volume-CANARY_HARD_MAX_VOLUME)>0.00000001) + facts.resultClass=CANARY_RESULT_PARTIAL; + facts.hasEaPendingOrder=reconciliation.hasEaPendingOrder; + facts.hasForeignGoldExposure=reconciliation.hasForeignGoldExposure; + facts.sameDirection=decision.loaded && !observation_used && reconciliation.position.present + && (decision.direction=="buy" || decision.direction=="sell") + && decision.direction==reconciliation.position.direction; + facts.oppositeDirection=!observation_used && decision.loaded && reconciliation.position.present + && IsCanaryActionableOpposite(decision.direction,reconciliation.position.direction); + facts.closeConfirmed=latch.pendingCloseDecisionId!="" && correlated_entry_position + && correlated_entry_closed && !reconciliation.position.present && !reconciliation.hasEaPendingOrder; + facts.stoppedObservation=!latch.protectionError + && (latch.unresolved || latch.activePositionDecisionId!="") + && correlated_entry_position && correlated_entry_closed + && reconciliation.lastCloseWasStop && !reconciliation.position.present + && !reconciliation.hasEaPendingOrder; + bool authoritative_stop_closure=facts.stoppedObservation; + bool authoritative_emergency_closure=latch.protectionError && latch.emergencyCloseAttempted + && latch.emergencyPositionId!="" && !reconciliation.position.present + && emergency_position_closed + && CanaryTicketString(reconciliation.closedPositionId)==latch.emergencyPositionId; + reconciliation.authoritativeStopClosure=authoritative_stop_closure; + reconciliation.authoritativeEmergencyClosure=authoritative_emergency_closure; + facts.persistentSafetyPause=latch.protectionError && !reconciliation.position.present; + facts.dailyLimitReached=reconciliation.daily.lossCount>=CANARY_HARD_MAX_DAILY_LOSSES + || reconciliation.daily.realizedLoss>=CANARY_HARD_MAX_DAILY_LOSS; + if(latch.protectionError && reconciliation.position.present) + facts.eaPositionProtected=false; + if(authoritative_stop_closure || authoritative_emergency_closure) + facts.resultClass=CANARY_RESULT_NONE; + else if(facts.resultClass==CANARY_RESULT_UNKNOWN + && ((correlated_entry_position && reconciliation.position.present + && reconciliation.position.identifier==correlated_entry_position_id + && reconciliation.position.stopProtected + && MathAbs(reconciliation.position.volume-CANARY_HARD_MAX_VOLUME)<=0.00000001) + || facts.closeConfirmed)) + facts.resultClass=CANARY_RESULT_NONE; + if(latch.unresolved && result_class==CANARY_RESULT_NONE + && !reconciliation.position.present && !facts.closeConfirmed + && !authoritative_stop_closure && !authoritative_emergency_closure) + facts.resultClass=CANARY_RESULT_UNKNOWN; + + reconciliation.state=ReduceCanaryLifecycle(facts); + reconciliation.available=facts.brokerStateAvailable; + if(reconciliation.state==CANARY_LIFECYCLE_FILLED_PROTECTED) + { + reconciliation.reconciliationState="reconciled"; + reconciliation.detail=facts.sameDirection + ? "Broker confirms the same-direction EA-owned Gold position and protective stop; no new order is allowed." + : "Broker confirms the EA-owned Gold position and valid protective stop."; + } + else if(reconciliation.state==CANARY_LIFECYCLE_EMERGENCY_CLOSE) + { + reconciliation.reconciliationState="protection_error"; + reconciliation.detail="Broker confirms EA-owned Gold exposure without a valid protective stop."; + } + else if(reconciliation.state==CANARY_LIFECYCLE_RECONCILIATION_REQUIRED) + { + reconciliation.reconciliationState="required"; + reconciliation.detail="The broker outcome is unknown, partial, pending, or otherwise unresolved; submission remains blocked."; + } + else if(reconciliation.state==CANARY_LIFECYCLE_BLOCKED) + { + reconciliation.reconciliationState="foreign_exposure"; + reconciliation.detail="Manual or foreign Gold exposure blocks this broker."; + } + else if(reconciliation.state==CANARY_LIFECYCLE_STOPPED) + { + reconciliation.reconciliationState="reconciled"; + reconciliation.detail="The broker confirms a stop-loss closure and the attempted observation remains consumed."; + } + else if(reconciliation.state==CANARY_LIFECYCLE_CLOSED) + { + reconciliation.reconciliationState="reconciled"; + reconciliation.detail="The broker confirms the opposite-signal position closure."; + } + else if(reconciliation.state==CANARY_LIFECYCLE_ORDER_REJECTED) + { + reconciliation.reconciliationState="terminal"; + reconciliation.detail="The broker definitively rejected the request; this decision remains consumed."; + } + else if(reconciliation.state==CANARY_LIFECYCLE_PAUSED) + { + if(facts.persistentSafetyPause) + { + reconciliation.reconciliationState="protection_error"; + reconciliation.detail="A persistent broker protection error pauses new entries until operator clearance."; + } + else + { + reconciliation.reconciliationState="reconciled"; + reconciliation.detail="The broker-day realized-loss ceiling pauses new entries until the next server day."; + } + } + else + { + reconciliation.reconciliationState="reconciled"; + reconciliation.detail="Authoritative broker state is reconciled and contains no blocking Gold exposure."; + } + return reconciliation.available; +} + +#endif diff --git a/tools/mt5/JmbGoldmineDemoCanary/JmbCanaryState.mqh b/tools/mt5/JmbGoldmineDemoCanary/JmbCanaryState.mqh new file mode 100644 index 000000000..953de0e13 --- /dev/null +++ b/tools/mt5/JmbGoldmineDemoCanary/JmbCanaryState.mqh @@ -0,0 +1,1163 @@ +#ifndef OPENALICE_JMB_CANARY_STATE_MQH +#define OPENALICE_JMB_CANARY_STATE_MQH + +#include "JmbCanaryTypes.mqh" +#include "JmbCanaryGates.mqh" +#include "JmbCanaryCsv.mqh" +#include "JmbCanaryPolicy.mqh" + +const string CANARY_EXECUTION_ROOT="OpenAliceMt5ExecutionV1"; +const string CANARY_STATUS_HEADER="schema_version,captured_at,broker,server,account_mode,symbol,state,detail,rollout_stage,execution_enabled,kill_switch,decision_id,observation_id,event_id,event_type,event_time,result_code,result_detail,stop_protection_confirmed,position_direction,position_volume,position_open_price,position_stop_loss,position_id,reconciliation_state,daily_loss_count,daily_realized_loss,blocking_gate,next_safe_action"; +const string CANARY_PROCESSED_HEADER="schema_version,decision_id,observation_id,attempted_at"; + +const CanaryLifecycleState CANARY_LIFECYCLE_ORDER_REQUESTING=(CanaryLifecycleState)4; +const CanaryLifecycleState CANARY_LIFECYCLE_ORDER_REJECTED=(CanaryLifecycleState)5; +const CanaryLifecycleState CANARY_LIFECYCLE_RECONCILIATION_REQUIRED=(CanaryLifecycleState)6; +const CanaryLifecycleState CANARY_LIFECYCLE_FILLED_PROTECTED=(CanaryLifecycleState)7; +const CanaryLifecycleState CANARY_LIFECYCLE_CLOSE_REQUESTING=(CanaryLifecycleState)8; +const CanaryLifecycleState CANARY_LIFECYCLE_CLOSED=(CanaryLifecycleState)9; +const CanaryLifecycleState CANARY_LIFECYCLE_STOPPED=(CanaryLifecycleState)10; +const CanaryLifecycleState CANARY_LIFECYCLE_EMERGENCY_CLOSE=(CanaryLifecycleState)11; +const CanaryLifecycleState CANARY_LIFECYCLE_ERROR=(CanaryLifecycleState)12; + +enum CanaryBrokerResultClass +{ + CANARY_RESULT_NONE=0, + CANARY_RESULT_REJECTED=1, + CANARY_RESULT_UNKNOWN=2, + CANARY_RESULT_PARTIAL=3 +}; + +enum CanaryClosedOwnershipClass +{ + CANARY_CLOSED_OWNERSHIP_FOREIGN=0, + CANARY_CLOSED_OWNERSHIP_EA=1, + CANARY_CLOSED_OWNERSHIP_UNSAFE=2 +}; + +struct CanaryPositionSnapshot +{ + bool present; + ulong ticket; + ulong identifier; + string direction; + double volume; + double openPrice; + double stopLoss; + bool stopProtected; +}; + +struct CanarySafetyLatch +{ + bool valid; + bool unresolved; + bool protectionError; + string pendingCloseDecisionId; + string pendingCloseObservationId; + bool emergencyCloseAttempted; + string emergencyPositionId; + string pendingEntryDecisionId; + string pendingEntryObservationId; + datetime pendingEntryAttemptedAt; + string pendingEntryOrderId; + string pendingEntryDealId; + double pendingRequestedVolume; + double pendingRequestedPrice; + double pendingRequestedStopLoss; + double pendingCalculatedRisk; + string pendingEntryComment; + string activePositionDecisionId; + string activePositionObservationId; + string activePositionId; + double activeRequestedVolume; + double activeRequestedPrice; + double activeRequestedStopLoss; + double activeCalculatedRisk; + string activeEntryComment; +}; + +struct CanaryReconciliationFacts +{ + bool brokerStateAvailable; + CanaryBrokerResultClass resultClass; + bool hasEaPosition; + bool eaPositionProtected; + bool hasEaPendingOrder; + bool hasForeignGoldExposure; + bool sameDirection; + bool oppositeDirection; + bool closeConfirmed; + bool stoppedObservation; + bool persistentSafetyPause; + bool dailyLimitReached; +}; + +struct CanaryReconciliation +{ + bool available; + CanaryLifecycleState state; + string detail; + string reconciliationState; + CanaryPositionSnapshot position; + bool hasEaPendingOrder; + bool hasForeignGoldExposure; + CanaryDailyState daily; + bool hasClosedPosition; + bool lastCloseWasStop; + datetime finalCloseTime; + ulong dealTicket; + ulong closedPositionId; + double acceptedPrice; + double commission; + double swap; + double fee; + double netResult; + bool authoritativeStopClosure; + bool authoritativeEmergencyClosure; +}; + +struct CanaryExecutionEvent +{ + CanaryLifecycleState eventType; + string eventId; + datetime eventTime; + string broker; + string server; + long accountLogin; + string decisionId; + string observationId; + string gateResultsJson; + bool hasCalculatedRisk; + double calculatedRisk; + bool hasRequestedOrder; + double requestedVolume; + double requestedPrice; + double requestedStopLoss; + bool hasAcceptedOrder; + bool hasAcceptedStopLoss; + double acceptedVolume; + double acceptedPrice; + double acceptedStopLoss; + string resultCode; + string resultDetail; + ulong orderTicket; + ulong dealTicket; + ulong positionId; + string reconciliationState; + int dailyLossCount; + double dailyRealizedLoss; + bool hasOutcome; + double commission; + double swap; + double fee; + double netResult; +}; + +string CanaryAuthoritativeLifecycleLabel(const CanaryLifecycleState state) +{ + if(state==CANARY_LIFECYCLE_ORDER_REQUESTING) return "order_requesting"; + if(state==CANARY_LIFECYCLE_ORDER_REJECTED) return "order_rejected"; + if(state==CANARY_LIFECYCLE_RECONCILIATION_REQUIRED) return "reconciliation_required"; + if(state==CANARY_LIFECYCLE_FILLED_PROTECTED) return "filled_protected"; + if(state==CANARY_LIFECYCLE_CLOSE_REQUESTING) return "close_requesting"; + if(state==CANARY_LIFECYCLE_CLOSED) return "closed"; + if(state==CANARY_LIFECYCLE_STOPPED) return "stopped"; + if(state==CANARY_LIFECYCLE_EMERGENCY_CLOSE) return "emergency_close"; + if(state==CANARY_LIFECYCLE_ERROR) return "error"; + return CanaryLifecycleLabel(state); +} + +string CanaryStatusDirectory(const string broker,const string symbol) +{ + return CANARY_EXECUTION_ROOT+"\\"+broker+"\\"+symbol; +} + +string CanaryIsoTime(const datetime value) +{ + MqlDateTime parts; + TimeToStruct(value,parts); + return StringFormat("%04d-%02d-%02dT%02d:%02d:%02d.000Z", + parts.year,parts.mon,parts.day,parts.hour,parts.min,parts.sec); +} + +string CanaryDayKey(const datetime value) +{ + MqlDateTime parts; + TimeToStruct(value,parts); + return StringFormat("%04d-%02d-%02d",parts.year,parts.mon,parts.day); +} + +string CanaryProcessedStatePath(const string broker,const string symbol) +{ + return CanaryStatusDirectory(broker,symbol)+"\\processed_observations.csv"; +} + +void InitializeCanaryProcessedState(CanaryProcessedState &state) +{ + state.valid=false; + state.filePresent=false; + ArrayResize(state.decisionIds,0); + ArrayResize(state.observationIds,0); + ArrayResize(state.attemptedAt,0); +} + +bool IsCanaryHashIdentity(const string value) +{ + if(StringLen(value)!=24) return false; + for(int index=0;index=48 && character<=57) || (character>=97 && character<=102))) return false; + } + return true; +} + +string CanaryEntryCorrelationComment(const string decision_id) +{ + return IsCanaryHashIdentity(decision_id) ? "JMB:"+StringSubstr(decision_id,0,20) : ""; +} + +bool IsCanaryTicketText(const string value) +{ + if(value=="") return true; + long parsed=StringToInteger(value); + return parsed>0 && StringFormat("%I64u",(ulong)parsed)==value; +} + +void ClearCanaryPendingEntryLatch(CanarySafetyLatch &latch) +{ + latch.pendingEntryDecisionId=""; + latch.pendingEntryObservationId=""; + latch.pendingEntryAttemptedAt=0; + latch.pendingEntryOrderId=""; + latch.pendingEntryDealId=""; + latch.pendingRequestedVolume=0.0; + latch.pendingRequestedPrice=0.0; + latch.pendingRequestedStopLoss=0.0; + latch.pendingCalculatedRisk=0.0; + latch.pendingEntryComment=""; +} + +void ClearCanaryActivePositionCorrelation(CanarySafetyLatch &latch) +{ + latch.activePositionDecisionId=""; + latch.activePositionObservationId=""; + latch.activePositionId=""; + latch.activeRequestedVolume=0.0; + latch.activeRequestedPrice=0.0; + latch.activeRequestedStopLoss=0.0; + latch.activeCalculatedRisk=0.0; + latch.activeEntryComment=""; +} + +bool FinalizeCanaryEmergencyTerminalCorrelation(CanarySafetyLatch &latch, + const bool terminal_event_durable) +{ + if(!terminal_event_durable) return false; + latch.unresolved=false; + ClearCanaryPendingEntryLatch(latch); + ClearCanaryActivePositionCorrelation(latch); + return true; +} + +bool CanaryRequestedEvidenceValid(const string decision_id,const string observation_id, + const double volume,const double price,const double stop_loss, + const double calculated_risk,const string entry_comment) +{ + return IsCanaryHashIdentity(decision_id) && IsCanaryHashIdentity(observation_id) + && MathIsValidNumber(volume) && CanaryNearlyEqual(volume,CANARY_HARD_MAX_VOLUME) + && MathIsValidNumber(price) && price>0.0 + && MathIsValidNumber(stop_loss) && stop_loss>0.0 + && MathIsValidNumber(calculated_risk) && calculated_risk>0.0 + && entry_comment==CanaryEntryCorrelationComment(decision_id); +} + +bool ActivateCanaryPositionCorrelation(CanarySafetyLatch &latch,const ulong position_id) +{ + if(position_id==0 || !CanaryRequestedEvidenceValid(latch.pendingEntryDecisionId, + latch.pendingEntryObservationId,latch.pendingRequestedVolume,latch.pendingRequestedPrice, + latch.pendingRequestedStopLoss,latch.pendingCalculatedRisk,latch.pendingEntryComment)) return false; + latch.activePositionDecisionId=latch.pendingEntryDecisionId; + latch.activePositionObservationId=latch.pendingEntryObservationId; + latch.activePositionId=CanaryTicketString(position_id); + latch.activeRequestedVolume=latch.pendingRequestedVolume; + latch.activeRequestedPrice=latch.pendingRequestedPrice; + latch.activeRequestedStopLoss=latch.pendingRequestedStopLoss; + latch.activeCalculatedRisk=latch.pendingCalculatedRisk; + latch.activeEntryComment=latch.pendingEntryComment; + return true; +} + +bool ApplyCanaryPositionCorrelation(CanaryExecutionEvent &event,const CanarySafetyLatch &latch) +{ + string decision_id=latch.activePositionDecisionId!="" + ? latch.activePositionDecisionId : latch.pendingEntryDecisionId; + string observation_id=latch.activePositionDecisionId!="" + ? latch.activePositionObservationId : latch.pendingEntryObservationId; + double volume=latch.activePositionDecisionId!="" + ? latch.activeRequestedVolume : latch.pendingRequestedVolume; + double price=latch.activePositionDecisionId!="" + ? latch.activeRequestedPrice : latch.pendingRequestedPrice; + double stop_loss=latch.activePositionDecisionId!="" + ? latch.activeRequestedStopLoss : latch.pendingRequestedStopLoss; + double calculated_risk=latch.activePositionDecisionId!="" + ? latch.activeCalculatedRisk : latch.pendingCalculatedRisk; + string entry_comment=latch.activePositionDecisionId!="" + ? latch.activeEntryComment : latch.pendingEntryComment; + if(!CanaryRequestedEvidenceValid(decision_id,observation_id,volume,price,stop_loss, + calculated_risk,entry_comment)) return false; + event.decisionId=decision_id; + event.observationId=observation_id; + event.gateResultsJson="[]"; + event.hasCalculatedRisk=true; + event.calculatedRisk=calculated_risk; + event.hasRequestedOrder=true; + event.requestedVolume=volume; + event.requestedPrice=price; + event.requestedStopLoss=stop_loss; + return true; +} + +bool CanaryProcessedStateContains(const CanaryProcessedState &state, + const string decision_id, + const string observation_id) +{ + if(!state.valid) return true; + for(int index=0;index0 && lines[line_count-1]=="") record_count--; + for(int line_index=1;line_index<=record_count;line_index++) + { + if(!StripCanaryLineEnding(lines[line_index],detail) || lines[line_index]=="") + { + detail="Processed-observation state contains a blank or multiline record."; + return false; + } + string values[]; + if(StringFind(lines[line_index],"\"")>=0 + || !ParseCanaryCsvRecord(lines[line_index],values,detail) || ArraySize(values)!=4 + || values[0]!="1" || !IsCanaryHashIdentity(values[1]) || !IsCanaryHashIdentity(values[2]) + || values[1]!=CanarySha256Identity("daily-trend-v1|"+values[2])) + { + detail="Processed-observation state contains a malformed record."; + return false; + } + datetime attempted_at=0; + if(!TryCanaryIsoUtc(values[3],attempted_at)) + { + detail="Processed-observation state contains an invalid attempted_at timestamp."; + return false; + } + for(int existing=0;existing0) record+=","; + record+=CanaryCsvEscapedCell(values[index]); + } + return record; +} + +bool CanaryStatusProjectionIsPossible(const CanaryEvaluation &evaluation, + const CanaryPolicy &policy, + const bool execution_enabled, + const bool kill_switch) +{ + if(policy.rolloutStage=="status_only" && execution_enabled) return false; + if(evaluation.state==CANARY_LIFECYCLE_DISABLED) return !execution_enabled && !evaluation.ready; + if(evaluation.state==CANARY_LIFECYCLE_PAUSED) return kill_switch && !evaluation.ready; + if(evaluation.state==CANARY_LIFECYCLE_READY) + return execution_enabled && !kill_switch && evaluation.ready + && policy.rolloutStage!="status_only" && evaluation.blockingGate==""; + return evaluation.state==CANARY_LIFECYCLE_BLOCKED && !evaluation.ready && evaluation.blockingGate!=""; +} + +bool CanaryEffectiveExecutionEnabled(const bool input_enabled,const CanaryPolicy &policy) +{ + return input_enabled && policy.loaded && policy.candidateApproved + && policy.rolloutStage!="status_only" + && CanaryRolloutAuthorized(policy.broker,policy.rolloutStage); +} + +bool WriteCanaryLatestStatus(const string broker, + const string server, + const string symbol, + const CanaryPolicy &policy, + const CanaryDecision &decision, + const CanaryEvaluation &evaluation, + const bool execution_enabled, + const bool kill_switch, + const int daily_loss_count, + const double daily_realized_loss, + string &detail) +{ + if(!CanaryStatusProjectionIsPossible(evaluation,policy,execution_enabled,kill_switch)) + { + detail="Refused to publish an impossible dry-run lifecycle projection."; + return false; + } + + EnsureCanaryDirectory(broker,symbol); + string destination_path=CanaryStatusDirectory(broker,symbol)+"\\latest_status.csv"; + string temporary_path=destination_path+"."+IntegerToString((int)GetTickCount())+".tmp"; + int handle=FileOpen(temporary_path,FILE_WRITE|FILE_BIN|FILE_ANSI|FILE_COMMON); + if(handle==INVALID_HANDLE) + { + detail="The temporary status file could not be opened."; + return false; + } + + string intended_values[]; + ArrayResize(intended_values,29); + intended_values[0]="1"; + intended_values[1]=CanaryIsoTime(TimeGMT()); + intended_values[2]=broker; + intended_values[3]=server; + intended_values[4]="demo"; + intended_values[5]=symbol; + intended_values[6]=CanaryLifecycleLabel(evaluation.state); + intended_values[7]=evaluation.detail; + intended_values[8]=policy.rolloutStage; + intended_values[9]=execution_enabled ? "1" : "0"; + intended_values[10]=kill_switch ? "1" : "0"; + intended_values[11]=decision.loaded ? decision.decisionId : ""; + intended_values[12]=decision.loaded ? decision.observationId : ""; + intended_values[13]=""; + intended_values[14]=""; + intended_values[15]=""; + intended_values[16]=""; + intended_values[17]=""; + intended_values[18]="0"; + intended_values[19]=""; + intended_values[20]=""; + intended_values[21]=""; + intended_values[22]=""; + intended_values[23]=""; + intended_values[24]=evaluation.state==CANARY_LIFECYCLE_READY ? "complete" : "required"; + intended_values[25]=IntegerToString(daily_loss_count); + intended_values[26]=DoubleToString(daily_realized_loss,2); + intended_values[27]=evaluation.blockingGate; + intended_values[28]=evaluation.nextSafeAction; + + string payload=CANARY_STATUS_HEADER+"\r\n"+CanaryCsvRecord(intended_values)+"\r\n"; + uint written=FileWriteString(handle,payload); + if(written!=StringLen(payload)) + { + FileClose(handle); + FileDelete(temporary_path,FILE_COMMON); + detail="The strict status payload could not be written completely."; + return false; + } + FileFlush(handle); + FileClose(handle); + + string verified_payload=""; + string verification_detail=""; + if(!ReadCanaryCommonText(temporary_path,verified_payload,verification_detail) + || verified_payload!=payload) + { + FileDelete(temporary_path,FILE_COMMON); + detail="The temporary status file did not preserve the exact payload."; + return false; + } + + if(FileMove(temporary_path, FILE_COMMON, destination_path, FILE_COMMON | FILE_REWRITE)) + { + detail="The strict latest_status.csv projection was replaced atomically."; + return true; + } + FileDelete(temporary_path,FILE_COMMON); + detail="The temporary status file could not replace latest_status.csv."; + return false; +} + +string CanaryJsonEscape(const string value) +{ + string escaped=""; + for(int index=0;index=0 && StringFind(lines[index],position_token)>=0) return true; + return false; +} + +bool CanaryCorrelatedTerminalEventRecorded(const string broker, + const CanaryExecutionEvent &event) +{ + if((event.eventType!=CANARY_LIFECYCLE_CLOSED && event.eventType!=CANARY_LIFECYCLE_STOPPED) + || event.positionId==0 || event.reconciliationState!="reconciled" + || event.decisionId=="" || event.observationId=="" + || !event.hasCalculatedRisk || !event.hasRequestedOrder || !event.hasOutcome) return false; + string path=CanaryStatusDirectory(broker,"XAUUSD")+"\\events.jsonl"; + string text=""; + string detail=""; + if(!ReadCanaryCommonText(path,text,detail)) return false; + string tokens[]={ + "\"event_type\":"+CanaryJsonString(CanaryAuthoritativeLifecycleLabel(event.eventType)), + "\"decision_id\":"+CanaryJsonString(event.decisionId), + "\"observation_id\":"+CanaryJsonString(event.observationId), + "\"calculated_risk\":"+CanaryJsonNumberOrNull(true,event.calculatedRisk,8), + "\"requested_volume\":"+CanaryJsonNumberOrNull(true,event.requestedVolume,8), + "\"requested_price\":"+CanaryJsonNumberOrNull(true,event.requestedPrice,8), + "\"requested_stop_loss\":"+CanaryJsonNumberOrNull(true,event.requestedStopLoss,8), + "\"deal_ticket\":"+CanaryJsonString(CanaryTicketString(event.dealTicket)), + "\"position_id\":"+CanaryJsonString(CanaryTicketString(event.positionId)), + "\"reconciliation_state\":\"reconciled\"", + "\"commission\":"+CanaryJsonNumberOrNull(true,event.commission,8), + "\"swap\":"+CanaryJsonNumberOrNull(true,event.swap,8), + "\"fee\":"+CanaryJsonNumberOrNull(true,event.fee,8), + "\"net_result\":"+CanaryJsonNumberOrNull(true,event.netResult,8) + }; + string lines[]; + int count=StringSplit(text,(ushort)StringGetCharacter("\n",0),lines); + for(int line_index=0;line_index0 ? CanaryIsoTime(latch.pendingEntryAttemptedAt) : "", + latch.pendingEntryOrderId,latch.pendingEntryDealId, + latch.pendingEntryDecisionId!="" ? DoubleToString(latch.pendingRequestedVolume,8) : "", + latch.pendingEntryDecisionId!="" ? DoubleToString(latch.pendingRequestedPrice,8) : "", + latch.pendingEntryDecisionId!="" ? DoubleToString(latch.pendingRequestedStopLoss,8) : "", + latch.pendingEntryDecisionId!="" ? DoubleToString(latch.pendingCalculatedRisk,8) : "", + latch.pendingEntryComment,latch.activePositionDecisionId,latch.activePositionObservationId, + latch.activePositionId,latch.activePositionDecisionId!="" ? DoubleToString(latch.activeRequestedVolume,8) : "", + latch.activePositionDecisionId!="" ? DoubleToString(latch.activeRequestedPrice,8) : "", + latch.activePositionDecisionId!="" ? DoubleToString(latch.activeRequestedStopLoss,8) : "", + latch.activePositionDecisionId!="" ? DoubleToString(latch.activeCalculatedRisk,8) : "", + latch.activeEntryComment); + if(header_written==0 || row_written==0) + { + FileClose(handle); + FileDelete(temporary_path,FILE_COMMON); + detail="The complete reconciliation latch could not be written."; + return false; + } + FileFlush(handle); + FileClose(handle); + string expected[]={"schema_version","unresolved","protection_error", + "pending_close_decision_id","pending_close_observation_id","emergency_close_attempted", + "emergency_position_id","pending_entry_decision_id","pending_entry_observation_id", + "pending_entry_attempted_at","pending_entry_order_id","pending_entry_deal_id", + "pending_requested_volume","pending_requested_price","pending_requested_stop_loss", + "pending_calculated_risk","pending_entry_comment","active_position_decision_id", + "active_position_observation_id","active_position_id","active_requested_volume", + "active_requested_price","active_requested_stop_loss","active_calculated_risk", + "active_entry_comment"}; + string intended[]={"1",latch.unresolved ? "1" : "0",latch.protectionError ? "1" : "0", + latch.pendingCloseDecisionId,latch.pendingCloseObservationId, + latch.emergencyCloseAttempted ? "1" : "0",latch.emergencyPositionId, + latch.pendingEntryDecisionId,latch.pendingEntryObservationId, + latch.pendingEntryAttemptedAt>0 ? CanaryIsoTime(latch.pendingEntryAttemptedAt) : "", + latch.pendingEntryOrderId,latch.pendingEntryDealId, + latch.pendingEntryDecisionId!="" ? DoubleToString(latch.pendingRequestedVolume,8) : "", + latch.pendingEntryDecisionId!="" ? DoubleToString(latch.pendingRequestedPrice,8) : "", + latch.pendingEntryDecisionId!="" ? DoubleToString(latch.pendingRequestedStopLoss,8) : "", + latch.pendingEntryDecisionId!="" ? DoubleToString(latch.pendingCalculatedRisk,8) : "", + latch.pendingEntryComment,latch.activePositionDecisionId,latch.activePositionObservationId, + latch.activePositionId,latch.activePositionDecisionId!="" ? DoubleToString(latch.activeRequestedVolume,8) : "", + latch.activePositionDecisionId!="" ? DoubleToString(latch.activeRequestedPrice,8) : "", + latch.activePositionDecisionId!="" ? DoubleToString(latch.activeRequestedStopLoss,8) : "", + latch.activePositionDecisionId!="" ? DoubleToString(latch.activeCalculatedRisk,8) : "", + latch.activeEntryComment}; + string verified_values[]; + string verify_detail=""; + if(!ReadStrictCanaryCsv(temporary_path,expected,verified_values,verify_detail) + || !CanaryExactValuesMatch(intended,verified_values)) + { + FileDelete(temporary_path,FILE_COMMON); + detail="The flushed reconciliation latch failed exact verification."; + return false; + } + if(!FileMove(temporary_path,FILE_COMMON,path,FILE_COMMON|FILE_REWRITE)) + { + FileDelete(temporary_path,FILE_COMMON); + detail="The reconciliation latch could not be replaced atomically."; + return false; + } + CanarySafetyLatch durable; + if(!LoadCanarySafetyLatch(broker,symbol,durable,verify_detail) + || durable.unresolved!=latch.unresolved || durable.protectionError!=latch.protectionError + || durable.pendingCloseDecisionId!=latch.pendingCloseDecisionId + || durable.pendingCloseObservationId!=latch.pendingCloseObservationId + || durable.emergencyCloseAttempted!=latch.emergencyCloseAttempted + || durable.emergencyPositionId!=latch.emergencyPositionId + || durable.pendingEntryDecisionId!=latch.pendingEntryDecisionId + || durable.pendingEntryObservationId!=latch.pendingEntryObservationId + || durable.pendingEntryAttemptedAt!=latch.pendingEntryAttemptedAt + || durable.pendingEntryOrderId!=latch.pendingEntryOrderId + || durable.pendingEntryDealId!=latch.pendingEntryDealId + || !CanaryNearlyEqual(durable.pendingRequestedVolume,latch.pendingRequestedVolume) + || !CanaryNearlyEqual(durable.pendingRequestedPrice,latch.pendingRequestedPrice) + || !CanaryNearlyEqual(durable.pendingRequestedStopLoss,latch.pendingRequestedStopLoss) + || !CanaryNearlyEqual(durable.pendingCalculatedRisk,latch.pendingCalculatedRisk) + || durable.pendingEntryComment!=latch.pendingEntryComment + || durable.activePositionDecisionId!=latch.activePositionDecisionId + || durable.activePositionObservationId!=latch.activePositionObservationId + || durable.activePositionId!=latch.activePositionId + || !CanaryNearlyEqual(durable.activeRequestedVolume,latch.activeRequestedVolume) + || !CanaryNearlyEqual(durable.activeRequestedPrice,latch.activeRequestedPrice) + || !CanaryNearlyEqual(durable.activeRequestedStopLoss,latch.activeRequestedStopLoss) + || !CanaryNearlyEqual(durable.activeCalculatedRisk,latch.activeCalculatedRisk) + || durable.activeEntryComment!=latch.activeEntryComment) + { + detail="The durable reconciliation latch failed exact reopen verification."; + return false; + } + detail="The reconciliation and protection latch was persisted atomically."; + return true; +} + +bool PersistCanaryStatusValues(const string broker,const string symbol, + const string &intended_values[],string &detail) +{ + if(ArraySize(intended_values)!=29) return false; + EnsureCanaryDirectory(broker,symbol); + string destination_path=CanaryStatusDirectory(broker,symbol)+"\\latest_status.csv"; + string temporary_path=destination_path+"."+IntegerToString((long)GetTickCount())+".tmp"; + int handle=FileOpen(temporary_path,FILE_WRITE|FILE_BIN|FILE_ANSI|FILE_COMMON); + if(handle==INVALID_HANDLE) return false; + string payload=CANARY_STATUS_HEADER+"\r\n"+CanaryCsvRecord(intended_values)+"\r\n"; + uint written=FileWriteString(handle,payload); + if(written!=StringLen(payload)) + { + FileClose(handle); + FileDelete(temporary_path,FILE_COMMON); + return false; + } + FileFlush(handle); + FileClose(handle); + string verified_payload=""; + string verify_detail=""; + if(!ReadCanaryCommonText(temporary_path,verified_payload,verify_detail) + || verified_payload!=payload + || !FileMove(temporary_path,FILE_COMMON,destination_path,FILE_COMMON|FILE_REWRITE)) + { + FileDelete(temporary_path,FILE_COMMON); + detail="The authoritative status failed exact payload verification or atomic replacement."; + return false; + } + detail="The authoritative Task 5-compatible latest status was replaced atomically."; + return true; +} + +bool WriteCanaryReconciledStatus(const string broker,const string server,const string symbol, + const CanaryPolicy &policy,const CanaryDecision &decision, + const CanaryReconciliation &reconciliation, + const CanaryExecutionEvent &latest_event, + const bool execution_enabled,const bool kill_switch, + string &detail) +{ + if(reconciliation.state==CANARY_LIFECYCLE_FILLED_PROTECTED + && (!reconciliation.position.present || !reconciliation.position.stopProtected + || reconciliation.position.stopLoss<=0.0)) + { + detail="Refused to publish filled_protected without broker-confirmed stop protection."; + return false; + } + string values[]; + ArrayResize(values,29); + values[0]="1"; + values[1]=CanaryIsoTime(TimeGMT()); + values[2]=broker; + values[3]=server; + values[4]="demo"; + values[5]=symbol; + values[6]=CanaryAuthoritativeLifecycleLabel(reconciliation.state); + values[7]=reconciliation.detail; + values[8]=policy.rolloutStage; + values[9]=execution_enabled ? "1" : "0"; + values[10]=kill_switch ? "1" : "0"; + values[11]=decision.loaded ? decision.decisionId : ""; + values[12]=decision.loaded ? decision.observationId : ""; + values[13]=latest_event.eventId; + values[14]=latest_event.eventId=="" ? "" : CanaryAuthoritativeLifecycleLabel(latest_event.eventType); + values[15]=latest_event.eventId=="" ? "" : CanaryIsoTime(latest_event.eventTime); + values[16]=latest_event.resultCode; + values[17]=latest_event.resultDetail; + values[18]=reconciliation.position.present && reconciliation.position.stopProtected ? "1" : "0"; + values[19]=reconciliation.position.present ? reconciliation.position.direction : ""; + values[20]=reconciliation.position.present ? DoubleToString(reconciliation.position.volume,8) : ""; + values[21]=reconciliation.position.present ? DoubleToString(reconciliation.position.openPrice,8) : ""; + values[22]=reconciliation.position.present ? DoubleToString(reconciliation.position.stopLoss,8) : ""; + values[23]=reconciliation.position.present ? CanaryTicketString(reconciliation.position.identifier) : ""; + values[24]=reconciliation.reconciliationState; + values[25]=IntegerToString(reconciliation.daily.lossCount); + values[26]=DoubleToString(reconciliation.daily.realizedLoss,2); + values[27]=reconciliation.reconciliationState=="identity_mismatch" ? "demo_identity" + : reconciliation.reconciliationState=="protection_error" ? "protection" + : reconciliation.state==CANARY_LIFECYCLE_RECONCILIATION_REQUIRED ? "reconciliation" + : reconciliation.state==CANARY_LIFECYCLE_EMERGENCY_CLOSE ? "protection" + : reconciliation.state==CANARY_LIFECYCLE_BLOCKED ? "exposure" + : reconciliation.state==CANARY_LIFECYCLE_PAUSED ? "daily_loss_count" : ""; + values[28]=reconciliation.state==CANARY_LIFECYCLE_FILLED_PROTECTED + ? "Monitor broker-side protection and reconciliation." + : reconciliation.state==CANARY_LIFECYCLE_EMERGENCY_CLOSE + ? "Keep the broker paused and confirm the protective close before operator review." + : reconciliation.reconciliationState=="identity_mismatch" + ? "Restore the exact configured demo account, server, broker, symbol, and magic binding." + : reconciliation.state==CANARY_LIFECYCLE_RECONCILIATION_REQUIRED + ? "Do not submit again; inspect authoritative broker orders, deals, and positions." + : reconciliation.reconciliationState=="protection_error" + ? "Resolve the persistent broker protection error before operator clearance." + : reconciliation.state==CANARY_LIFECYCLE_PAUSED + ? "Wait for the next broker server day; unresolved safety latches remain active." + : "Re-evaluate every entry gate before any new protected demo request."; + return PersistCanaryStatusValues(broker,symbol,values,detail); +} + +bool IsCanaryPersistentProtectionPause(const CanaryReconciliation &reconciliation) +{ + return reconciliation.state==CANARY_LIFECYCLE_PAUSED + && reconciliation.reconciliationState=="protection_error"; +} + +#endif diff --git a/tools/mt5/JmbGoldmineDemoCanary/JmbCanaryTradeGateway.mqh b/tools/mt5/JmbGoldmineDemoCanary/JmbCanaryTradeGateway.mqh new file mode 100644 index 000000000..31b49ae34 --- /dev/null +++ b/tools/mt5/JmbGoldmineDemoCanary/JmbCanaryTradeGateway.mqh @@ -0,0 +1,266 @@ +#ifndef OPENALICE_JMB_CANARY_TRADE_GATEWAY_MQH +#define OPENALICE_JMB_CANARY_TRADE_GATEWAY_MQH + +#include "JmbCanaryState.mqh" + +struct TradeSubmitResult +{ + bool sent; + uint retcode; + ulong order_ticket; + ulong deal_ticket; + double accepted_volume; + double accepted_price; + string detail; +}; + +bool ResolveMarketFilling(const string symbol,ENUM_ORDER_TYPE_FILLING &resolved) +{ + long flags=0; + long execution=0; + if(!SymbolInfoInteger(symbol,SYMBOL_FILLING_MODE,flags) + || !SymbolInfoInteger(symbol,SYMBOL_TRADE_EXEMODE,execution)) return false; + if((flags&SYMBOL_FILLING_FOK)==SYMBOL_FILLING_FOK) + { + resolved=ORDER_FILLING_FOK; + return true; + } + if((flags&SYMBOL_FILLING_IOC)==SYMBOL_FILLING_IOC) + { + resolved=ORDER_FILLING_IOC; + return true; + } + if(execution!=SYMBOL_TRADE_EXECUTION_MARKET) + { + resolved=ORDER_FILLING_RETURN; + return true; + } + return false; +} + +TradeSubmitResult CheckedSendCanaryRequest(MqlTradeRequest &request) +{ + TradeSubmitResult result; + ZeroMemory(result); + MqlTradeCheckResult check={}; + MqlTradeResult broker={}; + if(!OrderCheck(request,check) || check.retcode!=0) + { + result.retcode=check.retcode; + result.detail=check.comment; + return result; + } + result.sent=OrderSend(request,broker); + result.retcode=broker.retcode; + result.order_ticket=broker.order; + result.deal_ticket=broker.deal; + result.accepted_volume=broker.volume; + result.accepted_price=broker.price; + result.detail=broker.comment; + return result; +} + +bool IsGatewayStopProtective(const CanaryDecision &decision,const MqlTick &tick) +{ + long order_mode=0; + long stops_level=0; + double point=0.0; + double tick_size=0.0; + if(!SymbolInfoInteger("XAUUSD",SYMBOL_ORDER_MODE,order_mode) + || !SymbolInfoInteger("XAUUSD",SYMBOL_TRADE_STOPS_LEVEL,stops_level) + || !SymbolInfoDouble("XAUUSD",SYMBOL_POINT,point) + || !SymbolInfoDouble("XAUUSD",SYMBOL_TRADE_TICK_SIZE,tick_size)) return false; + if((order_mode&SYMBOL_ORDER_SL)!=SYMBOL_ORDER_SL || stops_level<0 + || !MathIsValidNumber(point) || point<=0.0 + || !MathIsValidNumber(tick_size) || tick_size<=0.0 + || !MathIsValidNumber(decision.stopLoss) || decision.stopLoss<=0.0) return false; + + double aligned_stop=MathRound(decision.stopLoss/tick_size)*tick_size; + double tolerance=MathMax(0.00000001,tick_size*0.0000001); + if(MathAbs(decision.stopLoss-aligned_stop)>tolerance) return false; + double entry=decision.direction=="buy" ? tick.ask : tick.bid; + double minimum_distance=(double)stops_level*point; + if(!MathIsValidNumber(entry) || entry<=0.0) return false; + return decision.direction=="buy" + ? decision.stopLoss=minimum_distance + : decision.stopLoss>entry && decision.stopLoss-entry+tolerance>=minimum_distance; +} + +bool CanaryGatewayBindingMatches(const string broker,const string expected_server, + const long expected_login,const string symbol, + const long magic_number,const ENUM_ACCOUNT_TRADE_MODE actual_mode, + const string actual_server,const long actual_login, + const string actual_symbol) +{ + return actual_mode==ACCOUNT_TRADE_MODE_DEMO + && expected_login>0 && actual_login==expected_login + && actual_server==expected_server + && expected_server==CanaryAllowedServer(broker) + && symbol=="XAUUSD" && actual_symbol==symbol + && magic_number==CanaryAllowedMagic(broker); +} + +bool CanaryGatewayIdentityValid(const string broker,const string expected_server, + const long expected_login,const string symbol, + const long magic_number) +{ + return CanaryGatewayBindingMatches(broker,expected_server,expected_login,symbol,magic_number, + (ENUM_ACCOUNT_TRADE_MODE)AccountInfoInteger(ACCOUNT_TRADE_MODE), + AccountInfoString(ACCOUNT_SERVER),AccountInfoInteger(ACCOUNT_LOGIN),_Symbol); +} + +bool SetCanaryCloseVolume(MqlTradeRequest &close,const double close_volume) +{ + if(!MathIsValidNumber(close_volume) || close_volume<=0.0 + || close_volume>CANARY_HARD_MAX_VOLUME) return false; + close.volume=close_volume; + return true; +} + +bool BuildCanaryCloseRequest(const CanaryPositionSnapshot &position, + const long magic_number, + const double max_deviation, + const string comment, + MqlTradeRequest &close) +{ + if(!position.present || position.ticket==0 + || (position.direction!="buy" && position.direction!="sell")) return false; + MqlTick tick; + double point=0.0; + if(!SymbolInfoTick("XAUUSD",tick) || !SymbolInfoDouble("XAUUSD",SYMBOL_POINT,point) + || !MathIsValidNumber(point) || point<=0.0 || !MathIsValidNumber(max_deviation) + || max_deviation<0.0) return false; + double deviation_points=MathFloor(max_deviation/point); + if(!MathIsValidNumber(deviation_points) || deviation_points<0.0 + || deviation_points>(double)ULONG_MAX) return false; + + ZeroMemory(close); + close.action=TRADE_ACTION_DEAL; + close.magic=(ulong)magic_number; + close.symbol="XAUUSD"; + if(!SetCanaryCloseVolume(close,position.volume)) return false; + close.type=position.direction=="buy" ? ORDER_TYPE_SELL : ORDER_TYPE_BUY; + close.price=position.direction=="buy" ? tick.bid : tick.ask; + close.position=position.ticket; + close.deviation=(ulong)deviation_points; + close.comment=comment; + return ResolveMarketFilling("XAUUSD",close.type_filling); +} + +TradeSubmitResult SubmitProtectedMarketOrder(const CanaryDecision &decision,const CanaryPolicy &policy, + const string broker,const string expected_server, + const long expected_login,const string symbol, + const long magic_number) +{ + TradeSubmitResult result; + ZeroMemory(result); + if(!CanaryGatewayIdentityValid(broker,expected_server,expected_login,symbol,magic_number)) + { + result.detail="Account is not demo or broker magic is not allowlisted."; + return result; + } + if(!decision.loaded || !policy.loaded || decision.accountMode!="demo" + || decision.symbol!="XAUUSD" || policy.symbol!="XAUUSD" + || decision.broker!=broker || policy.broker!=broker + || decision.server!=expected_server || policy.server!=expected_server + || policy.magicNumber!=magic_number + || decision.volume!=CANARY_HARD_MAX_VOLUME || policy.maxVolume!=CANARY_HARD_MAX_VOLUME) + { + result.detail="The immutable Gold or 0.01-volume binding failed."; + return result; + } + long expected_magic=decision.broker=="hfmarkets" ? 880101 + : decision.broker=="icmarkets" ? 880201 : 0; + if(expected_magic==0 || policy.broker!=decision.broker || policy.magicNumber!=expected_magic + || (policy.magicNumber!=880101 && policy.magicNumber!=880201)) + { + result.detail="The immutable broker magic binding failed."; + return result; + } + if(decision.direction!="buy" && decision.direction!="sell") + { + result.detail="The direction is not allowlisted."; + return result; + } + + MqlTick tick; + double point=0.0; + if(!SymbolInfoTick("XAUUSD",tick) || !IsGatewayStopProtective(decision,tick) + || !SymbolInfoDouble("XAUUSD",SYMBOL_POINT,point) || !MathIsValidNumber(point) || point<=0.0 + || !MathIsValidNumber(policy.maxDeviation) || policy.maxDeviation<0.0) + { + result.detail="The original request cannot carry a broker-valid protective stop."; + return result; + } + double deviation_points=MathFloor(policy.maxDeviation/point); + if(!MathIsValidNumber(deviation_points) || deviation_points<0.0 + || deviation_points>(double)ULONG_MAX) + { + result.detail="The deviation cannot be represented safely in points."; + return result; + } + + MqlTradeRequest request={}; + request.action=TRADE_ACTION_DEAL; + request.magic=(ulong)policy.magicNumber; + request.symbol="XAUUSD"; + request.volume=CANARY_HARD_MAX_VOLUME; + request.type=decision.direction=="buy" ? ORDER_TYPE_BUY : ORDER_TYPE_SELL; + request.price=decision.direction=="buy" ? tick.ask : tick.bid; + request.sl=decision.stopLoss; + request.deviation=(ulong)deviation_points; + request.comment=CanaryEntryCorrelationComment(decision.decisionId); + if(!ResolveMarketFilling("XAUUSD",request.type_filling)) + { + result.detail="No supported market filling mode is available."; + return result; + } + return CheckedSendCanaryRequest(request); +} + +TradeSubmitResult SubmitCanaryReversalClose(const CanaryPositionSnapshot &position, + const CanaryDecision &decision, + const CanaryPolicy &policy, + const string broker,const string expected_server, + const long expected_login,const string symbol, + const long magic_number) +{ + MqlTradeRequest request={}; + TradeSubmitResult result; + ZeroMemory(result); + if(!CanaryGatewayIdentityValid(broker,expected_server,expected_login,symbol,magic_number) + || !decision.loaded || !policy.loaded || decision.symbol!="XAUUSD" + || decision.broker!=broker || policy.broker!=broker + || decision.server!=expected_server || policy.server!=expected_server + || policy.magicNumber!=magic_number + || policy.symbol!="XAUUSD" || !BuildCanaryCloseRequest(position,policy.magicNumber, + policy.maxDeviation,"JMB-REV:"+StringSubstr(decision.decisionId,0,16),request)) + { + result.detail="The reversal close request failed immutable validation."; + return result; + } + return CheckedSendCanaryRequest(request); +} + +TradeSubmitResult SubmitCanaryEmergencyClose(const CanaryPositionSnapshot &position, + const CanaryPolicy &policy, + const string broker,const string expected_server, + const long expected_login,const string symbol, + const long magic_number) +{ + MqlTradeRequest request={}; + TradeSubmitResult result; + ZeroMemory(result); + if(!CanaryGatewayIdentityValid(broker,expected_server,expected_login,symbol,magic_number) + || !policy.loaded || policy.symbol!="XAUUSD" || policy.broker!=broker + || policy.server!=expected_server || policy.magicNumber!=magic_number + || !BuildCanaryCloseRequest(position,policy.magicNumber,policy.maxDeviation, + "JMB-EMERGENCY-CLOSE",request)) + { + result.detail="The emergency close request failed immutable validation."; + return result; + } + return CheckedSendCanaryRequest(request); +} + +#endif diff --git a/tools/mt5/JmbGoldmineDemoCanary/JmbCanaryTypes.mqh b/tools/mt5/JmbGoldmineDemoCanary/JmbCanaryTypes.mqh new file mode 100644 index 000000000..77da8b5db --- /dev/null +++ b/tools/mt5/JmbGoldmineDemoCanary/JmbCanaryTypes.mqh @@ -0,0 +1,162 @@ +#ifndef OPENALICE_JMB_CANARY_TYPES_MQH +#define OPENALICE_JMB_CANARY_TYPES_MQH + +const double CANARY_HARD_MAX_RISK = 10.00; +const double CANARY_HARD_MAX_DAILY_LOSS = 40.00; +const int CANARY_HARD_MAX_DAILY_LOSSES = 4; +const double CANARY_HARD_MAX_VOLUME = 0.01; +const int CANARY_MARGIN_BUFFER_MULTIPLIER = 10; +const int CANARY_NEWS_WINDOW_SECONDS = 30 * 60; + +enum CanaryLifecycleState +{ + CANARY_LIFECYCLE_DISABLED = 0, + CANARY_LIFECYCLE_PAUSED = 1, + CANARY_LIFECYCLE_BLOCKED = 2, + CANARY_LIFECYCLE_READY = 3 +}; + +struct CanaryDecision +{ + bool loaded; + int schemaVersion; + string decisionId; + string observationId; + string observationAsOf; + datetime createdAt; + datetime leaseIssuedAt; + datetime leaseExpiresAt; + string broker; + string server; + string accountMode; + string symbol; + string strategyVersion; + string direction; + double entryReferencePrice; + double volume; + double stopLoss; + double maxRiskAmount; + string candidatePolicyVersion; + string costModelVersion; + string gateResultsJson; + bool preDecisionGatesPassed; +}; + +struct CanaryPolicy +{ + bool loaded; + int schemaVersion; + string policyVersion; + string broker; + string server; + string symbol; + string strategyVersion; + string rolloutStage; + bool candidateApproved; + double completedObservationMaxAgeHours; + double maxSpread; + double maxDeviation; + double maxRiskAmount; + double maxDailyLoss; + int maxDailyLosingTrades; + double maxVolume; + long magicNumber; +}; + +struct CanaryEnvironment +{ + bool accountIsDemo; + bool loginMatches; + bool serverMatches; + bool brokerMatches; + bool chartSymbolMatches; + bool magicMatches; + + bool executionEnabled; + bool killSwitch; + + bool rolloutAuthorized; + bool candidateApproved; + bool allowlistsMatch; + + bool decisionFresh; + bool bridgeFresh; + bool policyFresh; + bool costModelFresh; + bool observationFresh; + bool processedStateAvailable; + bool observationUnused; + + bool volumeEvidenceAvailable; + bool volumeCompatible; + + bool stopEvidenceAvailable; + bool stopModeSupportsSl; + bool stopTickSizeAvailable; + bool stopTickAligned; + bool stopBrokerValid; + bool riskCalculationAvailable; + double calculatedStopRisk; + + bool dailyStateAvailable; + double dailyRealizedLoss; + int dailyLossCount; + + bool exposureStateAvailable; + bool hasEaPosition; + bool hasEaPendingOrder; + bool hasForeignGoldExposure; + + bool marginCalculationAvailable; + double estimatedMargin; + double freeMargin; + + bool spreadAvailable; + double currentSpread; + bool deviationAvailable; + double requestedDeviation; + + bool sessionEvidenceAvailable; + bool sessionOpen; + + bool newsEvidenceAvailable; + bool newsBlackout; + + bool logPreflightReady; + bool reconciliationComplete; +}; + +struct CanaryGateResult +{ + string name; + bool passed; + string detail; +}; + +struct CanaryEvaluation +{ + CanaryLifecycleState state; + bool ready; + string detail; + string blockingGate; + string nextSafeAction; + CanaryGateResult gates[]; +}; + +struct CanaryDailyState +{ + string brokerDay; + int lossCount; + double realizedLoss; +}; + +struct CanaryProcessedState +{ + bool valid; + bool filePresent; + string decisionIds[]; + string observationIds[]; + datetime attemptedAt[]; +}; + +#endif diff --git a/tools/mt5/JmbGoldmineDemoCanary/JmbGoldmineDemoCanary.mq5 b/tools/mt5/JmbGoldmineDemoCanary/JmbGoldmineDemoCanary.mq5 new file mode 100644 index 000000000..858a44420 --- /dev/null +++ b/tools/mt5/JmbGoldmineDemoCanary/JmbGoldmineDemoCanary.mq5 @@ -0,0 +1,1035 @@ +// JMB Goldmine broker-local demo canary with one protected demo-order gateway. +#property strict +#property version "1.300" + +#include "JmbCanaryTypes.mqh" +#include "JmbCanaryCsv.mqh" +#include "JmbCanaryPolicy.mqh" +#include "JmbCanaryGates.mqh" +#include "JmbCanaryState.mqh" +#include "JmbCanaryReconcile.mqh" +#include "JmbCanaryTradeGateway.mqh" + +input string InpBrokerId = ""; +input string InpExpectedServer = ""; +input long InpExpectedAccountLogin = 0; +input string InpSymbol = "XAUUSD"; +input long InpMagicNumber = 0; +input bool InpDemoExecutionEnabled = false; +input bool InpKillSwitch = true; + +const string CANARY_POLICY_ROOT="OpenAliceMt5DemoPolicyV1"; +const string CANARY_DECISION_ROOT="OpenAliceMt5ExecutionDecisionV1"; + +bool g_evaluating=false; +bool g_reconciliation_dirty=true; +bool g_last_submit_sent=false; +uint g_last_submit_retcode=0; +ulong g_last_submit_order_ticket=0; +ulong g_last_submit_deal_ticket=0; +double g_last_submit_accepted_volume=0.0; +double g_last_submit_accepted_price=0.0; +string g_last_submit_detail=""; +CanaryBrokerResultClass g_last_result_class=CANARY_RESULT_NONE; +CanaryExecutionEvent g_latest_event; + +void BuildCanaryLifecycleEvent(const CanaryDecision &decision, + const CanaryReconciliation &reconciliation, + const CanaryLifecycleState state, + const string result_code,const string result_detail, + const double calculated_risk, + CanaryExecutionEvent &event) +{ + InitializeCanaryExecutionEvent(event); + event.eventType=state; + event.eventTime=TimeGMT(); + event.broker=InpBrokerId; + event.server=InpExpectedServer; + event.accountLogin=AccountInfoInteger(ACCOUNT_LOGIN); + event.decisionId=decision.loaded ? decision.decisionId : ""; + event.observationId=decision.loaded ? decision.observationId : ""; + event.gateResultsJson=decision.loaded ? decision.gateResultsJson : "[]"; + event.hasCalculatedRisk=decision.loaded && calculated_risk>0.0; + event.calculatedRisk=calculated_risk; + event.hasRequestedOrder=decision.loaded; + event.requestedVolume=decision.volume; + event.requestedPrice=decision.entryReferencePrice; + event.requestedStopLoss=decision.stopLoss; + event.hasAcceptedOrder=reconciliation.position.present; + event.hasAcceptedStopLoss=reconciliation.position.present && reconciliation.position.stopProtected; + event.acceptedVolume=reconciliation.position.present ? reconciliation.position.volume : g_last_submit_accepted_volume; + event.acceptedPrice=reconciliation.position.present ? reconciliation.position.openPrice : g_last_submit_accepted_price; + event.acceptedStopLoss=reconciliation.position.present ? reconciliation.position.stopLoss : 0.0; + event.resultCode=result_code; + event.resultDetail=result_detail; + event.orderTicket=g_last_submit_order_ticket; + event.dealTicket=reconciliation.dealTicket!=0 ? reconciliation.dealTicket : g_last_submit_deal_ticket; + event.positionId=reconciliation.position.present ? reconciliation.position.identifier : reconciliation.closedPositionId; + event.reconciliationState=reconciliation.reconciliationState; + event.dailyLossCount=reconciliation.daily.lossCount; + event.dailyRealizedLoss=reconciliation.daily.realizedLoss; + event.hasOutcome=reconciliation.hasClosedPosition; + event.commission=reconciliation.commission; + event.swap=reconciliation.swap; + event.fee=reconciliation.fee; + event.netResult=reconciliation.netResult; +} + +bool AppendCanaryOrderRequestingEvent(const CanaryDecision &decision, + const double calculated_risk,string &detail) +{ + CanaryReconciliation reconciliation; + InitializeCanaryReconciliation(reconciliation); + reconciliation.reconciliationState="pending"; + CanaryExecutionEvent candidate; + BuildCanaryLifecycleEvent(decision,reconciliation,CANARY_LIFECYCLE_ORDER_REQUESTING,"", + "Request persistence completed before submission.",calculated_risk,candidate); + if(!AppendCanaryExecutionEvent(candidate,detail)) return false; + g_latest_event=candidate; + return true; +} + +bool CanaryProcessedStateIsExactAppend(const CanaryProcessedState &prior, + const CanaryProcessedState &candidate, + const CanaryDecision &decision, + const datetime attempted_at) +{ + if(!prior.valid || !candidate.valid) return false; + int prior_count=ArraySize(prior.observationIds); + int candidate_count=ArraySize(candidate.observationIds); + if(ArraySize(prior.decisionIds)!=prior_count || ArraySize(prior.attemptedAt)!=prior_count + || candidate_count!=prior_count+1 + || ArraySize(candidate.decisionIds)!=candidate_count + || ArraySize(candidate.attemptedAt)!=candidate_count) return false; + for(int index=0;index0.0; + candidate.acceptedVolume=submission.accepted_volume; + candidate.acceptedPrice=submission.accepted_price; + candidate.orderTicket=submission.order_ticket; + candidate.dealTicket=submission.deal_ticket; + if(!AppendCanaryExecutionEvent(candidate,persistence_detail)) + { + latch.unresolved=true; + g_last_result_class=CANARY_RESULT_UNKNOWN; + } + else g_latest_event=candidate; + if(!PersistCanarySafetyLatch(InpBrokerId,InpSymbol,latch,latch_detail)) + Print("JMB demo canary reconciliation latch persistence failed: ",latch_detail); + g_reconciliation_dirty=true; +} + +void InitializeCanaryPolicy(CanaryPolicy &policy) +{ + policy.loaded=false; + policy.schemaVersion=0; + policy.policyVersion=""; + policy.broker=InpBrokerId; + policy.server=InpExpectedServer; + policy.symbol=InpSymbol; + policy.strategyVersion="daily-trend-v1"; + policy.rolloutStage="status_only"; + policy.candidateApproved=false; + policy.completedObservationMaxAgeHours=0.0; + policy.maxSpread=0.0; + policy.maxDeviation=0.0; + policy.maxRiskAmount=0.0; + policy.maxDailyLoss=0.0; + policy.maxDailyLosingTrades=0; + policy.maxVolume=0.0; + policy.magicNumber=0; +} + +void InitializeCanaryDecision(CanaryDecision &decision) +{ + decision.loaded=false; + decision.schemaVersion=0; + decision.decisionId=""; + decision.observationId=""; + decision.observationAsOf=""; + decision.createdAt=0; + decision.leaseIssuedAt=0; + decision.leaseExpiresAt=0; + decision.broker=""; + decision.server=""; + decision.accountMode=""; + decision.symbol=""; + decision.strategyVersion=""; + decision.direction=""; + decision.entryReferencePrice=0.0; + decision.volume=0.0; + decision.stopLoss=0.0; + decision.maxRiskAmount=0.0; + decision.candidatePolicyVersion=""; + decision.costModelVersion=""; + decision.gateResultsJson=""; + decision.preDecisionGatesPassed=false; +} + +void InitializeCanaryEnvironment(CanaryEnvironment &environment) +{ + ZeroMemory(environment); + environment.executionEnabled=InpDemoExecutionEnabled; + environment.killSwitch=InpKillSwitch; +} + +bool CanaryObservationIsFresh(const CanaryDecision &decision,const CanaryPolicy &policy,const datetime now) +{ + if(!decision.loaded || !policy.loaded || !IsCanaryIsoDate(decision.observationAsOf)) return false; + string normalized=decision.observationAsOf; + StringReplace(normalized,"-","."); + datetime observation_time=StringToTime(normalized); + double age_seconds=(double)(now-observation_time); + return observation_time>0 && age_seconds>=0.0 + && age_seconds<=policy.completedObservationMaxAgeHours*3600.0; +} + +bool CanaryVolumeEvidence(const CanaryDecision &decision,bool &compatible) +{ + compatible=false; + double minimum=0.0; + double maximum=0.0; + double step=0.0; + if(!SymbolInfoDouble(InpSymbol,SYMBOL_VOLUME_MIN,minimum) + || !SymbolInfoDouble(InpSymbol,SYMBOL_VOLUME_MAX,maximum) + || !SymbolInfoDouble(InpSymbol,SYMBOL_VOLUME_STEP,step) + || minimum<=0.0 || maximum=minimum && decision.volume<=maximum + && MathAbs(steps*step-decision.volume)<=0.00000001; + return true; +} + +bool CalculateCanaryStopRisk(const CanaryDecision &decision, + const MqlTick &tick, + bool &evidence_available, + bool &mode_supports_sl, + bool &tick_size_available, + bool &tick_aligned, + bool &stop_valid, + double &calculated_risk) +{ + evidence_available=false; + mode_supports_sl=false; + tick_size_available=false; + tick_aligned=false; + stop_valid=false; + calculated_risk=0.0; + if(decision.direction!="buy" && decision.direction!="sell") return false; + long order_mode=0; + long stops_level=0; + double point=0.0; + double tick_size=0.0; + if(!SymbolInfoInteger(InpSymbol,SYMBOL_ORDER_MODE,order_mode) + || !SymbolInfoInteger(InpSymbol,SYMBOL_TRADE_STOPS_LEVEL,stops_level) + || !SymbolInfoDouble(InpSymbol,SYMBOL_POINT,point) + || !SymbolInfoDouble(InpSymbol,SYMBOL_TRADE_TICK_SIZE,tick_size)) return false; + evidence_available=true; + mode_supports_sl=(order_mode&SYMBOL_ORDER_SL)==SYMBOL_ORDER_SL; + tick_size_available=MathIsValidNumber(tick_size) && tick_size>0.0; + if(!mode_supports_sl || !tick_size_available || !MathIsValidNumber(point) || point<=0.0 || stops_level<0) + return false; + double aligned_stop=MathRound(decision.stopLoss/tick_size)*tick_size; + double alignment_tolerance=MathMax(0.00000001,tick_size*0.0000001); + tick_aligned=MathAbs(decision.stopLoss-aligned_stop)<=alignment_tolerance; + if(!tick_aligned) return false; + double entry=decision.direction=="buy" ? tick.ask : tick.bid; + double minimum_distance=(double)stops_level*point; + stop_valid=decision.direction=="buy" + ? decision.stopLoss=minimum_distance + : decision.stopLoss>entry && decision.stopLoss-entry+alignment_tolerance>=minimum_distance; + if(!stop_valid) return false; + + ENUM_ORDER_TYPE calculation_type=decision.direction=="buy" ? ORDER_TYPE_BUY : ORDER_TYPE_SELL; + double projected_result=0.0; + if(!OrderCalcProfit(calculation_type,InpSymbol,decision.volume,entry,decision.stopLoss,projected_result)) + return false; + calculated_risk=MathAbs(projected_result); + return MathIsValidNumber(calculated_risk) && calculated_risk>0.0; +} + +bool CalculateCanaryMargin(const CanaryDecision &decision,const MqlTick &tick,double &estimated_margin) +{ + estimated_margin=0.0; + if(decision.direction!="buy" && decision.direction!="sell") return false; + ENUM_ORDER_TYPE calculation_type=decision.direction=="buy" ? ORDER_TYPE_BUY : ORDER_TYPE_SELL; + double entry=decision.direction=="buy" ? tick.ask : tick.bid; + if(!OrderCalcMargin(calculation_type,InpSymbol,decision.volume,entry,estimated_margin)) return false; + return MathIsValidNumber(estimated_margin) && estimated_margin>0.0; +} + +bool CanarySessionOpen(const datetime now) +{ + MqlDateTime parts; + if(!TimeToStruct(now,parts)) return false; + if(parts.day_of_week==0 || parts.day_of_week==6) return false; + if(parts.day_of_week>=1 && parts.day_of_week<=4) return parts.hour>=6 && parts.hour<20; + return parts.day_of_week==5 && parts.hour>=6 && parts.hour<16; +} + +bool ReadCanaryNewsBlackout(const datetime now,bool &blackout) +{ + blackout=false; + MqlCalendarValue values[]; + ResetLastError(); + int count=CalendarValueHistory(values,now-CANARY_NEWS_WINDOW_SECONDS, + now+CANARY_NEWS_WINDOW_SECONDS,NULL,"USD"); + if(count<0) return false; + for(int index=0;index=decision.leaseIssuedAt && now<=decision.leaseExpiresAt; + environment.bridgeFresh=decision.loaded && decision.preDecisionGatesPassed; + environment.policyFresh=policy.loaded && decision.loaded + && decision.candidatePolicyVersion==policy.policyVersion; + environment.costModelFresh=decision.loaded && decision.preDecisionGatesPassed + && IsCanonicalCanaryText(decision.costModelVersion); + environment.observationFresh=CanaryObservationIsFresh(decision,policy,now); + environment.processedStateAvailable=processed_state.valid; + environment.observationUnused=decision.loaded && processed_state.valid + && !CanaryProcessedStateContains(processed_state,decision.decisionId,decision.observationId); + + environment.volumeEvidenceAvailable=decision.loaded && CanaryVolumeEvidence(decision,environment.volumeCompatible); + + MqlTick tick; + bool has_tick=decision.loaded && SymbolSelect(InpSymbol,true) && SymbolInfoTick(InpSymbol,tick); + if(has_tick) + { + environment.riskCalculationAvailable=CalculateCanaryStopRisk(decision,tick, + environment.stopEvidenceAvailable,environment.stopModeSupportsSl, + environment.stopTickSizeAvailable,environment.stopTickAligned, + environment.stopBrokerValid,environment.calculatedStopRisk); + environment.marginCalculationAvailable=CalculateCanaryMargin(decision,tick,environment.estimatedMargin); + environment.freeMargin=AccountInfoDouble(ACCOUNT_MARGIN_FREE); + environment.spreadAvailable=tick.ask>=tick.bid && tick.bid>0.0; + environment.currentSpread=tick.ask-tick.bid; + } + + environment.dailyStateAvailable=reconciliation.available; + environment.dailyLossCount=reconciliation.daily.lossCount; + environment.dailyRealizedLoss=reconciliation.daily.realizedLoss; + environment.exposureStateAvailable=reconciliation.available; + environment.hasEaPosition=reconciliation.position.present; + environment.hasEaPendingOrder=reconciliation.hasEaPendingOrder; + environment.hasForeignGoldExposure=reconciliation.hasForeignGoldExposure; + environment.deviationAvailable=policy.loaded; + environment.requestedDeviation=policy.loaded ? policy.maxDeviation : 0.0; + environment.sessionEvidenceAvailable=now>0; + environment.sessionOpen=now>0 && CanarySessionOpen(now); + datetime calendar_now=TimeTradeServer(); + environment.newsEvidenceAvailable=calendar_now>0 + && ReadCanaryNewsBlackout(calendar_now,environment.newsBlackout); + environment.logPreflightReady=PreflightCanaryLog(InpBrokerId,InpSymbol); + environment.reconciliationComplete=!g_reconciliation_dirty && reconciliation.available + && reconciliation.state!=CANARY_LIFECYCLE_RECONCILIATION_REQUIRED + && reconciliation.state!=CANARY_LIFECYCLE_EMERGENCY_CLOSE + && !latch.unresolved && !latch.protectionError; +} + +bool ValidateCanaryInputs() +{ + return InpExpectedAccountLogin>0 + && InpSymbol=="XAUUSD" + && (InpBrokerId=="hfmarkets" || InpBrokerId=="icmarkets") + && InpExpectedServer==CanaryAllowedServer(InpBrokerId) + && InpMagicNumber==CanaryAllowedMagic(InpBrokerId); +} + +bool AppendManagedCanaryEvent(const CanaryDecision &decision, + const CanaryReconciliation &reconciliation, + const CanaryLifecycleState state, + const string result_code,const string result_detail) +{ + CanaryExecutionEvent candidate; + BuildCanaryLifecycleEvent(decision,reconciliation,state,result_code,result_detail,0.0,candidate); + string detail=""; + if(AppendCanaryExecutionEvent(candidate,detail)) + { + g_latest_event=candidate; + return true; + } + Print("JMB demo canary lifecycle event failed: ",detail); + return false; +} + +bool AppendManagedCanaryPositionEvent(const CanaryDecision &decision, + const CanaryReconciliation &reconciliation, + const CanaryLifecycleState state, + const string result_code,const string result_detail, + const CanarySafetyLatch &latch) +{ + CanaryExecutionEvent candidate; + BuildCanaryLifecycleEvent(decision,reconciliation,state,result_code,result_detail,0.0,candidate); + if(!ApplyCanaryPositionCorrelation(candidate,latch)) return false; + string detail=""; + if(AppendCanaryExecutionEvent(candidate,detail)) + { + g_latest_event=candidate; + return true; + } + Print("JMB demo canary correlated lifecycle event failed: ",detail); + return false; +} + +bool HandleCanaryEmergencyClose(const CanaryDecision &decision,const CanaryPolicy &policy, + const CanaryReconciliation &reconciliation, + CanarySafetyLatch &latch) +{ + latch.valid=true; + latch.protectionError=true; + latch.unresolved=true; + if(latch.emergencyCloseAttempted) return false; + if(!AppendManagedCanaryPositionEvent(decision,reconciliation,CANARY_LIFECYCLE_EMERGENCY_CLOSE,"", + "Emergency protective close is durably requesting.",latch)) return false; + latch.emergencyCloseAttempted=true; + latch.emergencyPositionId=CanaryTicketString(reconciliation.position.identifier); + string detail=""; + if(!PersistCanarySafetyLatch(InpBrokerId,InpSymbol,latch,detail)) + { + Print("JMB demo canary emergency latch failed: ",detail); + return false; + } + + TradeSubmitResult submission=SubmitCanaryEmergencyClose(reconciliation.position,policy, + InpBrokerId,InpExpectedServer,InpExpectedAccountLogin,InpSymbol,InpMagicNumber); + g_last_submit_sent=submission.sent; + g_last_submit_retcode=submission.retcode; + g_last_submit_order_ticket=submission.order_ticket; + g_last_submit_deal_ticket=submission.deal_ticket; + g_last_submit_accepted_volume=submission.accepted_volume; + g_last_submit_accepted_price=submission.accepted_price; + g_last_submit_detail=submission.detail; + g_last_result_class=ClassifyCanaryBrokerResult(submission.sent,submission.retcode, + reconciliation.position.volume,submission.accepted_volume); + AppendManagedCanaryPositionEvent(decision,reconciliation,CANARY_LIFECYCLE_EMERGENCY_CLOSE, + IntegerToString((long)submission.retcode),submission.detail,latch); + PersistCanarySafetyLatch(InpBrokerId,InpSymbol,latch,detail); + g_reconciliation_dirty=true; + return true; +} + +bool HandleCanaryOppositeClose(const CanaryDecision &decision,const CanaryPolicy &policy, + const CanaryProcessedState &processed_state, + const string processed_path, + const CanaryReconciliation &reconciliation, + CanarySafetyLatch &latch) +{ + if(latch.pendingCloseDecisionId!="") return false; + if(!AppendManagedCanaryEvent(decision,reconciliation,CANARY_LIFECYCLE_CLOSE_REQUESTING,"", + "Opposite-signal close is durably requesting before its sole broker call.")) return false; + latch.valid=true; + latch.unresolved=true; + latch.pendingCloseDecisionId=decision.decisionId; + latch.pendingCloseObservationId=decision.observationId; + string detail=""; + if(!PersistCanarySafetyLatch(InpBrokerId,InpSymbol,latch,detail)) + { + Print("JMB demo canary opposite-close latch failed: ",detail); + return false; + } + + TradeSubmitResult submission=SubmitCanaryReversalClose(reconciliation.position,decision,policy, + InpBrokerId,InpExpectedServer,InpExpectedAccountLogin,InpSymbol,InpMagicNumber); + g_last_submit_sent=submission.sent; + g_last_submit_retcode=submission.retcode; + g_last_submit_order_ticket=submission.order_ticket; + g_last_submit_deal_ticket=submission.deal_ticket; + g_last_submit_accepted_volume=submission.accepted_volume; + g_last_submit_accepted_price=submission.accepted_price; + g_last_submit_detail=submission.detail; + g_last_result_class=ClassifyCanaryBrokerResult(submission.sent,submission.retcode, + reconciliation.position.volume,submission.accepted_volume); + CanaryLifecycleState result_state=g_last_result_class==CANARY_RESULT_REJECTED + ? CANARY_LIFECYCLE_ORDER_REJECTED : CANARY_LIFECYCLE_RECONCILIATION_REQUIRED; + AppendManagedCanaryEvent(decision,reconciliation,result_state, + IntegerToString((long)submission.retcode),submission.detail); + if(g_last_result_class==CANARY_RESULT_REJECTED) + { + string attempt_detail=""; + if(PersistCanaryAttempt(processed_path,decision,processed_state,attempt_detail)) + { + latch.unresolved=false; + latch.pendingCloseDecisionId=""; + latch.pendingCloseObservationId=""; + } + } + PersistCanarySafetyLatch(InpBrokerId,InpSymbol,latch,detail); + g_reconciliation_dirty=true; + return true; +} + +bool PersistCanarySameDirectionNoOp(const CanaryDecision &decision, + const CanaryProcessedState &processed_state, + const string processed_path, + const CanaryReconciliation &reconciliation, + CanarySafetyLatch &latch) +{ + if(!AppendManagedCanaryEvent(decision,reconciliation,CANARY_LIFECYCLE_FILLED_PROTECTED,"", + "Same-direction observation is a durable no-op; no broker request was made.")) return false; + string detail=""; + if(PersistCanaryAttempt(processed_path,decision,processed_state,detail)) return true; + latch.valid=true; + latch.unresolved=true; + PersistCanarySafetyLatch(InpBrokerId,InpSymbol,latch,detail); + return false; +} + +bool ConfirmCanaryOppositeClosure(const CanaryDecision &decision, + const CanaryReconciliation &reconciliation, + CanarySafetyLatch &latch) +{ + bool closed_event_recorded=CanaryLifecycleEventRecorded(InpBrokerId,CANARY_LIFECYCLE_CLOSED, + reconciliation.closedPositionId); + if(!closed_event_recorded + && !AppendManagedCanaryPositionEvent(decision,reconciliation,CANARY_LIFECYCLE_CLOSED,"", + "Opposite-signal closure is broker-confirmed and durable before gate re-evaluation.",latch)) return false; + latch.unresolved=false; + latch.pendingCloseDecisionId=""; + latch.pendingCloseObservationId=""; + ClearCanaryActivePositionCorrelation(latch); + string detail=""; + if(!PersistCanarySafetyLatch(InpBrokerId,InpSymbol,latch,detail)) + { + Print("JMB demo canary confirmed-close latch failed: ",detail); + return false; + } + return true; +} + +bool ConfirmCanaryEmergencyClosure(const CanaryDecision &decision, + const CanaryReconciliation &reconciliation, + CanarySafetyLatch &latch) +{ + if(!reconciliation.authoritativeEmergencyClosure || reconciliation.closedPositionId==0) + return false; + string closed_position_id=CanaryTicketString(reconciliation.closedPositionId); + if(latch.activePositionDecisionId!="" && latch.activePositionId!=closed_position_id) + return false; + + if(latch.activePositionDecisionId=="" + && !ActivateCanaryPositionCorrelation(latch,reconciliation.closedPositionId)) return false; + CanaryReconciliation terminal_reconciliation; + terminal_reconciliation=reconciliation; + terminal_reconciliation.state=CANARY_LIFECYCLE_CLOSED; + terminal_reconciliation.reconciliationState="reconciled"; + CanaryExecutionEvent expected_terminal; + BuildCanaryLifecycleEvent(decision,terminal_reconciliation,CANARY_LIFECYCLE_CLOSED,"", + "Emergency protection closure is broker-confirmed with its opening evidence.",0.0, + expected_terminal); + if(!ApplyCanaryPositionCorrelation(expected_terminal,latch)) return false; + bool any_terminal_event_durable=CanaryLifecycleEventRecorded(InpBrokerId, + CANARY_LIFECYCLE_CLOSED,reconciliation.closedPositionId); + bool terminal_event_durable=CanaryCorrelatedTerminalEventRecorded(InpBrokerId, + expected_terminal); + if(any_terminal_event_durable && !terminal_event_durable) return false; + if(!terminal_event_durable) + { + terminal_event_durable=AppendManagedCanaryPositionEvent(decision,terminal_reconciliation, + CANARY_LIFECYCLE_CLOSED,"", + "Emergency protection closure is broker-confirmed with its opening evidence.",latch); + } + + CanarySafetyLatch finalized_latch; + finalized_latch=latch; + if(!FinalizeCanaryEmergencyTerminalCorrelation(finalized_latch,terminal_event_durable)) return false; + string detail=""; + if(!PersistCanarySafetyLatch(InpBrokerId,InpSymbol,finalized_latch,detail)) + { + Print("JMB demo canary emergency-terminal latch failed: ",detail); + return false; + } + latch=finalized_latch; + return true; +} + +void Evaluate() +{ + if(g_evaluating) return; + g_evaluating=true; + + CanaryPolicy policy; + InitializeCanaryPolicy(policy); + CanaryDecision decision; + InitializeCanaryDecision(decision); + string read_detail=""; + string policy_path=CANARY_POLICY_ROOT+"\\"+InpBrokerId+"\\"+InpSymbol+"\\policy.csv"; + if(!ReadCanaryPolicy(policy_path,policy,read_detail)) InitializeCanaryPolicy(policy); + string decision_path=CANARY_DECISION_ROOT+"\\"+InpBrokerId+"\\"+InpSymbol+"\\latest_decision.csv"; + ReadCanaryDecision(decision_path,TimeGMT(),decision,read_detail); + + CanaryProcessedState processed_state; + InitializeCanaryProcessedState(processed_state); + string processed_path=CanaryProcessedStatePath(InpBrokerId,InpSymbol); + LoadCanaryProcessedState(processed_path,processed_state,read_detail); + + CanarySafetyLatch latch; + bool latch_loaded=LoadCanarySafetyLatch(InpBrokerId,InpSymbol,latch,read_detail); + if(!latch_loaded) + { + InitializeCanarySafetyLatch(latch); + latch.valid=true; + latch.unresolved=true; + } + bool observation_used=decision.loaded && processed_state.valid + && CanaryProcessedStateContains(processed_state,decision.decisionId,decision.observationId); + CanaryReconciliation reconciliation; + bool reconciled=ReconcileCanaryBrokerState(InpSymbol,InpMagicNumber,decision,observation_used, + g_last_result_class,latch,reconciliation); + g_reconciliation_dirty=!reconciled; + if(!reconciled) + { + latch.valid=true; + latch.unresolved=true; + string unresolved_detail=""; + PersistCanarySafetyLatch(InpBrokerId,InpSymbol,latch,unresolved_detail); + } + + if(reconciled && reconciliation.state==CANARY_LIFECYCLE_FILLED_PROTECTED + && latch.unresolved && !latch.protectionError) + { + string latch_detail=""; + bool correlation_activated=ActivateCanaryPositionCorrelation(latch, + reconciliation.position.identifier); + if(correlation_activated) + { + latch.unresolved=false; + ClearCanaryPendingEntryLatch(latch); + if(PersistCanarySafetyLatch(InpBrokerId,InpSymbol,latch,latch_detail)) + { + g_last_result_class=CANARY_RESULT_NONE; + reconciled=ReconcileCanaryBrokerState(InpSymbol,InpMagicNumber,decision,observation_used, + CANARY_RESULT_NONE,latch,reconciliation); + g_reconciliation_dirty=!reconciled; + } + } + else g_reconciliation_dirty=true; + } + + if(reconciled && reconciliation.state==CANARY_LIFECYCLE_STOPPED + && reconciliation.authoritativeStopClosure && latch.unresolved) + { + string stopped_latch_detail=""; + if((latch.activePositionDecisionId!="" || ActivateCanaryPositionCorrelation(latch, + reconciliation.closedPositionId))) + { + latch.unresolved=false; + ClearCanaryPendingEntryLatch(latch); + g_last_result_class=CANARY_RESULT_NONE; + if(!PersistCanarySafetyLatch(InpBrokerId,InpSymbol,latch,stopped_latch_detail)) + Print("JMB demo canary stopped-position latch failed: ",stopped_latch_detail); + } + } + bool authoritative_emergency_closure=reconciled + && reconciliation.authoritativeEmergencyClosure; + if(authoritative_emergency_closure && (latch.unresolved + || latch.activePositionDecisionId!="" || latch.pendingEntryDecisionId!="")) + { + if(ConfirmCanaryEmergencyClosure(decision,reconciliation,latch)) + g_last_result_class=CANARY_RESULT_NONE; + else g_reconciliation_dirty=true; + } + + CanaryEnvironment environment; + BuildCanaryEnvironment(decision,policy,processed_state,reconciliation,latch,environment); + CanaryEvaluation evaluation=EvaluateCanaryGates(decision,policy,environment); + bool mutation_identity_valid=CanaryGatewayIdentityValid(InpBrokerId,InpExpectedServer, + InpExpectedAccountLogin,InpSymbol,InpMagicNumber); + bool mutation_requested=reconciliation.state==CANARY_LIFECYCLE_EMERGENCY_CLOSE + || reconciliation.state==CANARY_LIFECYCLE_CLOSE_REQUESTING; + if(mutation_requested && !mutation_identity_valid) + { + reconciliation.state=CANARY_LIFECYCLE_BLOCKED; + reconciliation.available=false; + reconciliation.reconciliationState="identity_mismatch"; + reconciliation.detail="The actual terminal identity changed; every broker mutation is refused."; + latch.valid=true; + latch.unresolved=true; + string identity_latch_detail=""; + PersistCanarySafetyLatch(InpBrokerId,InpSymbol,latch,identity_latch_detail); + } + + if(reconciliation.state==CANARY_LIFECYCLE_EMERGENCY_CLOSE + && mutation_identity_valid && reconciliation.available && !reconciliation.hasForeignGoldExposure) + HandleCanaryEmergencyClose(decision,policy,reconciliation,latch); + else if(reconciliation.state==CANARY_LIFECYCLE_CLOSE_REQUESTING && decision.loaded + && !observation_used && latch.pendingCloseDecisionId=="") + { + CanaryEnvironment reversal_environment; + reversal_environment=environment; + reversal_environment.hasEaPosition=false; + CanaryEvaluation reversal_evaluation=EvaluateCanaryGates(decision,policy,reversal_environment); + if(mutation_identity_valid && reconciliation.available && !reconciliation.hasForeignGoldExposure + && reversal_evaluation.ready) + HandleCanaryOppositeClose(decision,policy,processed_state,processed_path,reconciliation,latch); + } + else if(reconciliation.state==CANARY_LIFECYCLE_CLOSED) + { + if(ConfirmCanaryOppositeClosure(decision,reconciliation,latch)) + { + g_last_result_class=CANARY_RESULT_NONE; + reconciled=ReconcileCanaryBrokerState(InpSymbol,InpMagicNumber,decision,false, + CANARY_RESULT_NONE,latch,reconciliation); + g_reconciliation_dirty=!reconciled; + } + } + else if(reconciliation.state==CANARY_LIFECYCLE_FILLED_PROTECTED && decision.loaded + && !observation_used && decision.direction==reconciliation.position.direction) + PersistCanarySameDirectionNoOp(decision,processed_state,processed_path,reconciliation,latch); + + if(reconciliation.state==CANARY_LIFECYCLE_FILLED_PROTECTED + && !CanaryLifecycleEventRecorded(InpBrokerId,CANARY_LIFECYCLE_FILLED_PROTECTED, + reconciliation.position.identifier)) + AppendManagedCanaryPositionEvent(decision,reconciliation,CANARY_LIFECYCLE_FILLED_PROTECTED,"", + "Broker-confirmed EA-owned exposure has a valid protective stop.",latch); + if(reconciliation.state==CANARY_LIFECYCLE_STOPPED + && !CanaryLifecycleEventRecorded(InpBrokerId,CANARY_LIFECYCLE_STOPPED, + reconciliation.closedPositionId)) + { + if(AppendManagedCanaryPositionEvent(decision,reconciliation,CANARY_LIFECYCLE_STOPPED,"", + "The fully reconciled stop-loss closure consumes its observation.",latch)) + { + latch.unresolved=false; + latch.pendingCloseDecisionId=""; + latch.pendingCloseObservationId=""; + ClearCanaryActivePositionCorrelation(latch); + string terminal_latch_detail=""; + PersistCanarySafetyLatch(InpBrokerId,InpSymbol,latch,terminal_latch_detail); + } + } + else if(reconciliation.state==CANARY_LIFECYCLE_STOPPED + && CanaryLifecycleEventRecorded(InpBrokerId,CANARY_LIFECYCLE_STOPPED, + reconciliation.closedPositionId) && latch.activePositionDecisionId!="") + { + latch.unresolved=false; + latch.pendingCloseDecisionId=""; + latch.pendingCloseObservationId=""; + ClearCanaryActivePositionCorrelation(latch); + string recovered_terminal_detail=""; + PersistCanarySafetyLatch(InpBrokerId,InpSymbol,latch,recovered_terminal_detail); + } + if(reconciliation.state==CANARY_LIFECYCLE_PAUSED && !authoritative_emergency_closure + && reconciliation.hasClosedPosition + && !CanaryLifecycleEventRecorded(InpBrokerId,CANARY_LIFECYCLE_PAUSED, + reconciliation.closedPositionId)) + AppendManagedCanaryEvent(decision,reconciliation,CANARY_LIFECYCLE_PAUSED,"", + IsCanaryPersistentProtectionPause(reconciliation) + ? "Persistent broker protection error keeps this canary paused pending operator clearance." + : "The broker-day losing-trade or realized-loss ceiling pauses new entries."); + + BuildCanaryEnvironment(decision,policy,processed_state,reconciliation,latch,environment); + evaluation=EvaluateCanaryGates(decision,policy,environment); + bool effective_execution_enabled=CanaryEffectiveExecutionEnabled(InpDemoExecutionEnabled,policy); + + string write_detail=""; + bool broker_lifecycle_active=reconciliation.state!=CANARY_LIFECYCLE_READY; + bool status_persisted=broker_lifecycle_active + ? WriteCanaryReconciledStatus(InpBrokerId,AccountInfoString(ACCOUNT_SERVER),InpSymbol, + policy,decision,reconciliation,g_latest_event,effective_execution_enabled,InpKillSwitch,write_detail) + : WriteCanaryLatestStatus(InpBrokerId,AccountInfoString(ACCOUNT_SERVER),InpSymbol, + policy,decision,evaluation,effective_execution_enabled,InpKillSwitch, + environment.dailyLossCount,environment.dailyRealizedLoss,write_detail); + if(!status_persisted) + Print("JMB demo canary status publication failed: ",write_detail); + if(reconciliation.state==CANARY_LIFECYCLE_READY) + SubmitReadyCanaryDecision(decision,policy,evaluation,processed_state,processed_path, + effective_execution_enabled,status_persisted,environment.calculatedStopRisk); + + g_evaluating=false; +} + +void PublishCanarySchedulerFailure() +{ + CanaryPolicy policy; + InitializeCanaryPolicy(policy); + CanaryDecision decision; + InitializeCanaryDecision(decision); + CanaryEvaluation evaluation; + evaluation.state=CANARY_LIFECYCLE_BLOCKED; + evaluation.ready=false; + evaluation.detail="The ten-second evaluation timer could not be started."; + evaluation.blockingGate="scheduler"; + evaluation.nextSafeAction="Keep the canary disabled and resolve the terminal timer failure."; + ArrayResize(evaluation.gates,0); + string write_detail=""; + if(!WriteCanaryLatestStatus(InpBrokerId,AccountInfoString(ACCOUNT_SERVER),InpSymbol, + policy,decision,evaluation,false,InpKillSwitch,0,0.0,write_detail)) + Print("JMB demo canary scheduler failure status could not be published: ",write_detail); +} + +int OnInit() +{ + if(InpExpectedAccountLogin<=0 || InpSymbol!="XAUUSD") return INIT_PARAMETERS_INCORRECT; + if(!ValidateCanaryInputs()) return INIT_PARAMETERS_INCORRECT; + if(!EventSetTimer(10)) + { + PublishCanarySchedulerFailure(); + Print("JMB demo canary initialization failed because its evaluation timer is unavailable."); + return INIT_FAILED; + } + Evaluate(); + return INIT_SUCCEEDED; +} + +void OnTimer() { Evaluate(); } +void OnTick() { Evaluate(); } +void OnTradeTransaction(const MqlTradeTransaction &transaction, + const MqlTradeRequest &request, + const MqlTradeResult &result) +{ + g_reconciliation_dirty=true; + return; +} +void OnDeinit(const int reason) { EventKillTimer(); } diff --git a/tools/mt5/JmbGoldmineDemoRiskShell.mq5 b/tools/mt5/JmbGoldmineDemoRiskShell.mq5 new file mode 100644 index 000000000..7e4bcf946 --- /dev/null +++ b/tools/mt5/JmbGoldmineDemoRiskShell.mq5 @@ -0,0 +1,378 @@ +// JMB Goldmine demo risk shell. This Expert Advisor validates local gates and +// writes status only. It deliberately has no trade-submit, amend, or close path. +#property strict +#property version "1.100" + +input string InpBrokerId = "hfmarkets"; +input string InpSymbol = "XAUUSD"; +input double InpMaxLot = 0.01; +input double InpMaxSpread = 0.75; +input int InpDecisionMaxAgeSeconds = 300; +input bool InpKillSwitch = true; + +const string DECISION_ROOT = "OpenAliceMt5DecisionLogV1"; +const string STATUS_ROOT = "OpenAliceMt5RiskShellV1"; + +struct DecisionSnapshot +{ + bool parsed; + string decisionId; + datetime createdAt; + int ageSeconds; + string symbol; + string mode; + string direction; + double volume; + double stopLoss; +}; + +string AccountModeLabel() +{ + long mode = AccountInfoInteger(ACCOUNT_TRADE_MODE); + if(mode == ACCOUNT_TRADE_MODE_DEMO) return "demo"; + if(mode == ACCOUNT_TRADE_MODE_REAL) return "real"; + if(mode == ACCOUNT_TRADE_MODE_CONTEST) return "contest"; + return "unknown"; +} + +string IsoTime(datetime value) +{ + MqlDateTime parts; + TimeToStruct(value, parts); + return StringFormat( + "%04d-%02d-%02dT%02d:%02d:%02d.000Z", + parts.year, + parts.mon, + parts.day, + parts.hour, + parts.min, + parts.sec + ); +} + +void AddCsvCell(string &cells[], string value) +{ + int size = ArraySize(cells); + ArrayResize(cells, size + 1); + cells[size] = value; +} + +void ParseCsvLine(string line, string &cells[]) +{ + ArrayResize(cells, 0); + string current = ""; + bool quoted = false; + + for(int index = 0; index < StringLen(line); index++) + { + string character = StringSubstr(line, index, 1); + if(character == "\"") + { + if(quoted && index + 1 < StringLen(line) && StringSubstr(line, index + 1, 1) == "\"") + { + current += "\""; + index++; + } + else + { + quoted = !quoted; + } + } + else if(character == "," && !quoted) + { + AddCsvCell(cells, current); + current = ""; + } + else + { + current += character; + } + } + + AddCsvCell(cells, current); +} + +bool TryCsvField(string headerLine, string valueLine, string fieldName, string &value) +{ + string headers[]; + string values[]; + ParseCsvLine(headerLine, headers); + ParseCsvLine(valueLine, values); + + if(ArraySize(headers) != ArraySize(values)) return false; + + for(int index = 0; index < ArraySize(headers); index++) + { + if(headers[index] == fieldName) + { + value = values[index]; + return true; + } + } + + return false; +} + +bool TryIsoTime(string value, datetime &result) +{ + if(StringLen(value) < 19) return false; + + string normalized = StringSubstr(value, 0, 19); + StringReplace(normalized, "T", " "); + StringReplace(normalized, "-", "."); + result = StringToTime(normalized); + return result > 0; +} + +bool EnsureStatusDirectory() +{ + if(!FolderCreate(STATUS_ROOT, FILE_COMMON)) return false; + if(!FolderCreate(STATUS_ROOT + "\\" + InpBrokerId, FILE_COMMON)) return false; + return FolderCreate(STATUS_ROOT + "\\" + InpBrokerId + "\\" + InpSymbol, FILE_COMMON); +} + +bool ReadDecisionLines(string &headerLine, string &valueLine) +{ + string path = DECISION_ROOT + "\\" + InpBrokerId + "\\" + InpSymbol + "\\latest_decision.csv"; + int handle = FileOpen(path, FILE_READ | FILE_BIN | FILE_COMMON | FILE_ANSI); + if(handle == INVALID_HANDLE) return false; + + int size = (int)FileSize(handle); + string text = FileReadString(handle, size); + FileClose(handle); + + int firstLineEnd = StringFind(text, "\n"); + if(firstLineEnd < 0) return false; + + headerLine = StringSubstr(text, 0, firstLineEnd); + string remainder = StringSubstr(text, firstLineEnd + 1); + int secondLineEnd = StringFind(remainder, "\n"); + valueLine = secondLineEnd >= 0 ? StringSubstr(remainder, 0, secondLineEnd) : remainder; + StringReplace(headerLine, "\r", ""); + StringReplace(valueLine, "\r", ""); + + return headerLine != "" && valueLine != ""; +} + +DecisionSnapshot ReadLatestDecision() +{ + DecisionSnapshot snapshot; + snapshot.parsed = false; + snapshot.decisionId = ""; + snapshot.createdAt = 0; + snapshot.ageSeconds = -1; + snapshot.symbol = ""; + snapshot.mode = ""; + snapshot.direction = ""; + snapshot.volume = 0.0; + snapshot.stopLoss = 0.0; + + string headerLine = ""; + string valueLine = ""; + if(!ReadDecisionLines(headerLine, valueLine)) return snapshot; + + string createdAt = ""; + string volume = ""; + string stopLoss = ""; + datetime parsedTime = 0; + + if(!TryCsvField(headerLine, valueLine, "decision_id", snapshot.decisionId)) return snapshot; + if(!TryCsvField(headerLine, valueLine, "created_at", createdAt)) return snapshot; + if(!TryCsvField(headerLine, valueLine, "symbol", snapshot.symbol)) return snapshot; + if(!TryCsvField(headerLine, valueLine, "mode", snapshot.mode)) return snapshot; + if(!TryCsvField(headerLine, valueLine, "direction", snapshot.direction)) return snapshot; + if(!TryCsvField(headerLine, valueLine, "volume", volume)) return snapshot; + if(!TryCsvField(headerLine, valueLine, "stop_loss", stopLoss)) return snapshot; + if(!TryIsoTime(createdAt, parsedTime)) return snapshot; + + snapshot.createdAt = parsedTime; + snapshot.ageSeconds = (int)(TimeGMT() - parsedTime); + snapshot.volume = StringToDouble(volume); + snapshot.stopLoss = StringToDouble(stopLoss); + snapshot.parsed = true; + return snapshot; +} + +void WriteGateStatus(string state, string detail, DecisionSnapshot &decision) +{ + if(!EnsureStatusDirectory()) + { + Print("Unable to create risk shell status directory: ", GetLastError()); + return; + } + + MqlTick tick; + bool hasTick = SymbolSelect(InpSymbol, true) && SymbolInfoTick(InpSymbol, tick); + int digits = (int)SymbolInfoInteger(InpSymbol, SYMBOL_DIGITS); + double spread = hasTick ? tick.ask - tick.bid : 0.0; + + string path = STATUS_ROOT + "\\" + InpBrokerId + "\\" + InpSymbol + "\\gate_status.csv"; + FileDelete(path, FILE_COMMON); + int handle = FileOpen(path, FILE_WRITE | FILE_CSV | FILE_COMMON | FILE_ANSI, ','); + if(handle == INVALID_HANDLE) + { + Print("Unable to write risk shell status: ", GetLastError()); + return; + } + + FileWrite( + handle, + "captured_at", + "broker", + "symbol", + "account_mode", + "state", + "detail", + "decision_id", + "decision_mode", + "direction", + "decision_age_seconds", + "decision_volume", + "decision_stop_loss", + "bid", + "ask", + "spread", + "positions", + "orders" + ); + FileWrite( + handle, + IsoTime(TimeGMT()), + InpBrokerId, + InpSymbol, + AccountModeLabel(), + state, + detail, + decision.decisionId, + decision.mode, + decision.direction, + IntegerToString(decision.ageSeconds), + DoubleToString(decision.volume, 2), + DoubleToString(decision.stopLoss, digits), + hasTick ? DoubleToString(tick.bid, digits) : "", + hasTick ? DoubleToString(tick.ask, digits) : "", + hasTick ? DoubleToString(spread, digits) : "", + IntegerToString(PositionsTotal()), + IntegerToString(OrdersTotal()) + ); + FileClose(handle); +} + +void Evaluate() +{ + DecisionSnapshot decision = ReadLatestDecision(); + + if(InpKillSwitch) + { + WriteGateStatus("paused", "Kill switch is on; new entries are blocked.", decision); + return; + } + + if(AccountModeLabel() != "demo") + { + WriteGateStatus("blocked", "Account is not demo.", decision); + return; + } + + if(_Symbol != InpSymbol) + { + WriteGateStatus("blocked", "EA chart symbol does not match configured symbol.", decision); + return; + } + + if(!decision.parsed) + { + WriteGateStatus("blocked", "Latest JMB decision file is missing or unreadable.", decision); + return; + } + + if(decision.symbol != InpSymbol) + { + WriteGateStatus("blocked", "Decision symbol does not match configured symbol.", decision); + return; + } + + if(decision.ageSeconds < 0 || decision.ageSeconds > InpDecisionMaxAgeSeconds) + { + WriteGateStatus("blocked", "Decision is stale or has an invalid timestamp.", decision); + return; + } + + if(decision.mode != "shadow") + { + WriteGateStatus("blocked", "Latest decision is not a shadow-ready decision.", decision); + return; + } + + if(decision.direction != "buy" && decision.direction != "sell") + { + WriteGateStatus("blocked", "Latest decision has no actionable shadow direction.", decision); + return; + } + + if(decision.volume <= 0.0 || decision.volume > InpMaxLot) + { + WriteGateStatus("blocked", "Decision lot size is outside the configured shell limit.", decision); + return; + } + + if(decision.stopLoss <= 0.0) + { + WriteGateStatus("blocked", "Decision stop loss is missing or invalid.", decision); + return; + } + + if(PositionsTotal() > 0 || OrdersTotal() > 0) + { + WriteGateStatus("blocked", "Existing manual or foreign exposure is present.", decision); + return; + } + + if(!SymbolSelect(InpSymbol, true)) + { + WriteGateStatus("blocked", "Configured symbol is unavailable.", decision); + return; + } + + MqlTick tick; + if(!SymbolInfoTick(InpSymbol, tick)) + { + WriteGateStatus("blocked", "Current quote is unavailable.", decision); + return; + } + + double spread = tick.ask - tick.bid; + if(spread > InpMaxSpread) + { + WriteGateStatus("blocked", "Spread exceeds configured maximum.", decision); + return; + } + + WriteGateStatus("shadow_ready", "All local shell gates passed. This version writes status only.", decision); +} + +int OnInit() +{ + if(InpBrokerId == "" || InpSymbol == "" || InpMaxLot <= 0.0 || InpMaxSpread <= 0.0 || InpDecisionMaxAgeSeconds <= 0) + return INIT_PARAMETERS_INCORRECT; + + EventSetTimer(10); + Evaluate(); + Print("JMB Goldmine demo risk shell started in status-only mode."); + return INIT_SUCCEEDED; +} + +void OnDeinit(const int reason) +{ + EventKillTimer(); +} + +void OnTimer() +{ + Evaluate(); +} + +void OnTick() +{ + Evaluate(); +} diff --git a/tools/mt5/OpenAliceMt5ReadOnlyBridge.mq5 b/tools/mt5/OpenAliceMt5ReadOnlyBridge.mq5 new file mode 100644 index 000000000..b6789aab2 --- /dev/null +++ b/tools/mt5/OpenAliceMt5ReadOnlyBridge.mq5 @@ -0,0 +1,275 @@ +// OpenAlice MT5 read-only bridge. This Expert Advisor does not include any +// trade function and cannot place, change, or cancel orders. +#property strict +#property version "1.100" + +input string InpBrokerId = "hfmarkets"; +input string InpSymbol = "XAUUSD"; +input int InpHeartbeatSeconds = 30; +input int InpTradeLedgerHistoryDays = 30; +input int InpTradeLedgerRefreshMinutes = 15; + +const string OUTPUT_ROOT = "OpenAliceMt5BridgeV1"; +const string TRADE_LEDGER_ROOT = "OpenAliceMt5TradeLedgerV1"; + +datetime g_last_trade_ledger_refresh = 0; + +string IsoTime(datetime value) +{ + MqlDateTime parts; + TimeToStruct(value, parts); + return StringFormat("%04d-%02d-%02dT%02d:%02d:%02d.000Z", parts.year, parts.mon, parts.day, parts.hour, parts.min, parts.sec); +} + +string IsoDate(datetime value) +{ + MqlDateTime parts; + TimeToStruct(value, parts); + return StringFormat("%04d-%02d-%02d", parts.year, parts.mon, parts.day); +} + +string AccountModeLabel() +{ + ENUM_ACCOUNT_TRADE_MODE mode = (ENUM_ACCOUNT_TRADE_MODE)AccountInfoInteger(ACCOUNT_TRADE_MODE); + if(mode == ACCOUNT_TRADE_MODE_DEMO) return "demo"; + if(mode == ACCOUNT_TRADE_MODE_CONTEST) return "contest"; + if(mode == ACCOUNT_TRADE_MODE_REAL) return "real"; + return "unknown"; +} + +bool EnsureOutputDirectory() +{ + if(!FolderCreate(OUTPUT_ROOT, FILE_COMMON)) return false; + if(!FolderCreate(OUTPUT_ROOT + "\\" + InpBrokerId, FILE_COMMON)) return false; + return FolderCreate(OUTPUT_ROOT + "\\" + InpBrokerId + "\\" + InpSymbol, FILE_COMMON); +} + +bool EnsureTradeLedgerDirectory() +{ + if(!FolderCreate(TRADE_LEDGER_ROOT, FILE_COMMON)) return false; + if(!FolderCreate(TRADE_LEDGER_ROOT + "\\" + InpBrokerId, FILE_COMMON)) return false; + return FolderCreate(TRADE_LEDGER_ROOT + "\\" + InpBrokerId + "\\" + InpSymbol, FILE_COMMON); +} + +bool ReplaceCommonFile(const string temp_path,const string final_path) +{ + return FileMove(temp_path,FILE_COMMON,final_path,FILE_COMMON|FILE_REWRITE); +} + +bool WriteCompletedD1() +{ + MqlRates rates[]; + ArraySetAsSeries(rates,false); + int copied=CopyRates(InpSymbol,PERIOD_D1,1,300,rates); + if(copied<2) return false; + string final_path=OUTPUT_ROOT+"\\"+InpBrokerId+"\\"+InpSymbol+"\\completed_d1.csv"; + string temp_path=final_path+".tmp"; + int handle=FileOpen(temp_path,FILE_WRITE|FILE_CSV|FILE_ANSI|FILE_COMMON,','); + if(handle==INVALID_HANDLE) return false; + datetime captured_at=TimeGMT(); + FileWrite(handle,"schema_version","captured_at","broker","server","account_mode","symbol","bar_as_of","bar_open_epoch","open","high","low","close"); + for(int i=0;i= 0 || StringFind(value, "\"") >= 0) + return "\"" + value + "\""; + return value; +} + +bool ShouldRefreshTradeLedger(const datetime captured_at) +{ + if(InpTradeLedgerHistoryDays <= 0 || InpTradeLedgerRefreshMinutes <= 0) + return false; + if(g_last_trade_ledger_refresh == 0) + return true; + return captured_at - g_last_trade_ledger_refresh >= InpTradeLedgerRefreshMinutes * 60; +} + +bool WriteTradeLedger(const datetime captured_at) +{ + if(!EnsureTradeLedgerDirectory()) return false; + + datetime from_time = captured_at - (InpTradeLedgerHistoryDays * 86400); + if(!HistorySelect(from_time, captured_at)) return false; + + string final_path = TRADE_LEDGER_ROOT + "\\" + InpBrokerId + "\\" + InpSymbol + "\\deals.csv"; + string temp_path = final_path + ".tmp"; + int handle = FileOpen(temp_path, FILE_WRITE | FILE_CSV | FILE_COMMON | FILE_ANSI, ','); + if(handle == INVALID_HANDLE) return false; + + FileWrite(handle, "account_mode", "server", "login", "broker", "symbol", "deal_ticket", "order_ticket", "position_id", "time", "entry", "type", "reason", "volume", "price", "commission", "fee", "swap", "profit", "magic", "comment"); + + int total = HistoryDealsTotal(); + for(int index = 0; index < total; index++) + { + ulong ticket = HistoryDealGetTicket(index); + string deal_symbol = HistoryDealGetString(ticket, DEAL_SYMBOL); + if(deal_symbol != InpSymbol) continue; + + datetime deal_time = (datetime)HistoryDealGetInteger(ticket, DEAL_TIME); + FileWrite( + handle, + AccountModeLabel(), + AccountInfoString(ACCOUNT_SERVER), + IntegerToString((int)AccountInfoInteger(ACCOUNT_LOGIN)), + InpBrokerId, + InpSymbol, + IntegerToString((long)ticket), + IntegerToString((long)HistoryDealGetInteger(ticket, DEAL_ORDER)), + IntegerToString((long)HistoryDealGetInteger(ticket, DEAL_POSITION_ID)), + IsoTime(deal_time), + DealEntryLabel(HistoryDealGetInteger(ticket, DEAL_ENTRY)), + DealTypeLabel(HistoryDealGetInteger(ticket, DEAL_TYPE)), + DealReasonLabel(HistoryDealGetInteger(ticket, DEAL_REASON)), + DoubleToString(HistoryDealGetDouble(ticket, DEAL_VOLUME), 2), + DoubleToString(HistoryDealGetDouble(ticket, DEAL_PRICE), _Digits), + DoubleToString(HistoryDealGetDouble(ticket, DEAL_COMMISSION), 2), + DoubleToString(HistoryDealGetDouble(ticket, DEAL_FEE), 2), + DoubleToString(HistoryDealGetDouble(ticket, DEAL_SWAP), 2), + DoubleToString(HistoryDealGetDouble(ticket, DEAL_PROFIT), 2), + IntegerToString((long)HistoryDealGetInteger(ticket, DEAL_MAGIC)), + CsvEscape(HistoryDealGetString(ticket, DEAL_COMMENT)) + ); + } + + FileFlush(handle); + FileClose(handle); + if(!ReplaceCommonFile(temp_path, final_path)) + return false; + g_last_trade_ledger_refresh = captured_at; + return true; +} + +void WriteHeartbeat() +{ + if(!SymbolSelect(InpSymbol, true) || !EnsureOutputDirectory()) return; + + MqlTick tick; + if(!SymbolInfoTick(InpSymbol, tick)) return; + + string relativePath = OUTPUT_ROOT + "\\" + InpBrokerId + "\\" + InpSymbol + "\\status.csv"; + FileDelete(relativePath, FILE_COMMON); + int handle = FileOpen(relativePath, FILE_WRITE | FILE_CSV | FILE_ANSI | FILE_COMMON, ','); + if(handle == INVALID_HANDLE) + { + PrintFormat("OpenAlice bridge could not write %s. Error %d", relativePath, GetLastError()); + return; + } + + FileWrite(handle, + "captured_at", "broker", "symbol", "bridge_mode", "account_mode", "server", "terminal_connected", + "trade_allowed", "trade_expert", "symbol_trade_mode", "bid", "ask", "spread_price", "tick_time", + "contract_size", "volume_min", "volume_max", "volume_step", "stops_level", "open_positions", "open_orders"); + datetime captured_at = TimeGMT(); + FileWrite(handle, + IsoTime(captured_at), InpBrokerId, InpSymbol, "read_only", AccountModeLabel(), AccountInfoString(ACCOUNT_SERVER), + (int)TerminalInfoInteger(TERMINAL_CONNECTED), (int)AccountInfoInteger(ACCOUNT_TRADE_ALLOWED), + (int)AccountInfoInteger(ACCOUNT_TRADE_EXPERT), (int)SymbolInfoInteger(InpSymbol, SYMBOL_TRADE_MODE), + tick.bid, tick.ask, tick.ask - tick.bid, IsoTime(captured_at), SymbolInfoDouble(InpSymbol, SYMBOL_TRADE_CONTRACT_SIZE), + SymbolInfoDouble(InpSymbol, SYMBOL_VOLUME_MIN), SymbolInfoDouble(InpSymbol, SYMBOL_VOLUME_MAX), + SymbolInfoDouble(InpSymbol, SYMBOL_VOLUME_STEP), (int)SymbolInfoInteger(InpSymbol, SYMBOL_TRADE_STOPS_LEVEL), + PositionsTotal(), OrdersTotal()); + FileClose(handle); + + if(!WriteCompletedD1()) + PrintFormat("OpenAlice bridge could not refresh completed D1 bars. Error %d",GetLastError()); + if(!WriteSpreadSample(tick)) + PrintFormat("OpenAlice bridge could not append the spread sample. Error %d",GetLastError()); + if(ShouldRefreshTradeLedger(captured_at) && !WriteTradeLedger(captured_at)) + PrintFormat("OpenAlice bridge could not refresh trade ledger. Error %d",GetLastError()); +} + +int OnInit() +{ + if(InpBrokerId == "" || InpSymbol == "" || InpHeartbeatSeconds < 5 || InpTradeLedgerHistoryDays < 0 || InpTradeLedgerRefreshMinutes < 0) + return INIT_PARAMETERS_INCORRECT; + EventSetTimer(InpHeartbeatSeconds); + WriteHeartbeat(); + Print("OpenAlice read-only bridge started. No trade functions are present."); + return INIT_SUCCEEDED; +} + +void OnTimer() { WriteHeartbeat(); } + +void OnDeinit(const int reason) { EventKillTimer(); } diff --git a/tools/mt5/README.md b/tools/mt5/README.md new file mode 100644 index 000000000..a6aa63e32 --- /dev/null +++ b/tools/mt5/README.md @@ -0,0 +1,271 @@ +# MT5 Broker History Tools + +Most tools in this directory collect and validate broker-native data without submitting orders. The separately installed Plan 3 demo-canary EA described below is the only exception: after explicit local operator enablement, it may submit protected Gold orders on the exact allowlisted demo accounts. It has no live-account mode, EURUSD execution path, or remote order command. + +## Broker mappings + +| Broker terminal | Gold research/live | Gold demo | EUR/USD | Export `InpBrokerId` | +| --- | --- | --- | --- | --- | +| HFM MT5 | `XAUUSDb` | `XAUUSD` | research/live `EURUSDb`; demo `EURUSD` | `hfmarkets` | +| IC Markets MT5 | `XAUUSD` | `XAUUSD` | `EURUSD` | `icmarkets` | + +## Export + +1. Open the MT5 terminal connected to the intended broker. +2. The exporter requests data one month at a time, so its M1 request stays below the terminal chart-bar cap. A normal `100,000`-bar limit is sufficient; no terminal restart is required solely for this exporter. +3. Copy `ExportBrokerM1History.mq5` into that terminal's `MQL5\\Scripts` directory and compile it in MetaEditor. +4. Run it from **Navigator -> Scripts** with the broker-specific settings below. + +HFM: + +```text +InpBrokerId: hfmarkets +InpSymbols: XAUUSDb,EURUSDb +InpStart: 2016.06.22 00:00 +InpOverwrite: false +``` + +IC Markets: + +```text +InpBrokerId: icmarkets +InpSymbols: XAUUSD,EURUSD +InpStart: 2016.06.22 00:00 +InpOverwrite: false +``` + +The script requests M1 history from the connected MT5 broker in monthly chunks and writes one CSV per month under the shared MT5 directory: + +```text +%APPDATA%\\MetaQuotes\\Terminal\\Common\\Files\\OpenAliceMt5HistoryV2\\\\\\ +``` + +Each symbol also gets a `contract.csv` file containing the broker's current contract and volume rules, plus a `manifest.csv` containing export coverage. `InpOverwrite=false` is intentional: it skips completed monthly files rather than duplicating them. Set it to `true` only when deliberately refreshing a period. + +## Validate + +After exporting, run: + +```powershell +python tools/mt5/validate_history.py "$env:APPDATA\\MetaQuotes\\Terminal\\Common\\Files\\OpenAliceMt5HistoryV2" --output .codex-run/mt5-history-report.json +``` + +The report is streamed, so it works with large data sets. A multi-minute gap is a review signal, not automatically bad: forex weekends, broker maintenance, and market holidays are expected. It must be classified before any strategy is trusted. + +## Scope boundary + +Ten years of M1 bars are the first research layer. Do not silently substitute generic public data for broker-native testing. Tick history is much larger and may not be retained by a broker for the entire period; collect it only for specific strategy windows after an M1 strategy survives walk-forward testing. + +## Research Baseline + +`backtest_daily_trend.py` is a research-only daily time-series-momentum baseline. It aggregates the broker export into daily bars, evaluates a small set of broad lookbacks on a training period, then reports the chosen lookback on an untouched holdout period. It does not connect to MT5 or generate an order. + +```powershell +python tools/mt5/backtest_daily_trend.py "$env:APPDATA\MetaQuotes\Terminal\Common\Files\OpenAliceMt5HistoryV2" --symbol EURUSDb --output .codex-run/eurusd-trend-baseline.json +python tools/mt5/backtest_daily_trend.py "$env:APPDATA\MetaQuotes\Terminal\Common\Files\OpenAliceMt5HistoryV2" --symbol XAUUSDb --output .codex-run/xauusd-trend-baseline.json +``` + +For the installed Windows app, write the reports to its persistent local data +folder so they survive application upgrades: + +```powershell +$research = Join-Path $env:USERPROFILE '.openalice\data\research' +New-Item -ItemType Directory -Force $research | Out-Null +python tools/mt5/backtest_daily_trend.py "$env:APPDATA\MetaQuotes\Terminal\Common\Files\OpenAliceMt5HistoryV2" --broker icmarkets --symbol EURUSD --output "$research\icmarkets-eurusd-trend-baseline.json" +python tools/mt5/backtest_daily_trend.py "$env:APPDATA\MetaQuotes\Terminal\Common\Files\OpenAliceMt5HistoryV2" --broker icmarkets --symbol XAUUSD --output "$research\icmarkets-xauusd-trend-baseline.json" +``` + +The default cost assumption is deliberately conservative and not a substitute for HFM's actual commissions, financing, and spread history. A passing result is research input only, never permission to enable an EA. + +## Walk-Forward Evaluation + +The baseline is one historical split. Before a study can be considered for a demo EA, run a rolling walk-forward study: each window selects its lookback from the preceding 60 months, freezes that choice, and evaluates the next unseen six months. The default 10 bps cost is only a placeholder and must be replaced by broker-specific costs before promotion. + +```powershell +python tools/mt5/walk_forward_daily_trend.py "$env:APPDATA\MetaQuotes\Terminal\Common\Files\OpenAliceMt5HistoryV2" --broker hfmarkets --symbol XAUUSDb --output "$research\xauusd-walk-forward.json" +python tools/mt5/walk_forward_daily_trend.py "$env:APPDATA\MetaQuotes\Terminal\Common\Files\OpenAliceMt5HistoryV2" --broker hfmarkets --symbol EURUSDb --output "$research\eurusd-walk-forward.json" +python tools/mt5/walk_forward_daily_trend.py "$env:APPDATA\MetaQuotes\Terminal\Common\Files\OpenAliceMt5HistoryV2" --broker icmarkets --symbol XAUUSD --output "$research\icmarkets-xauusd-walk-forward.json" +python tools/mt5/walk_forward_daily_trend.py "$env:APPDATA\MetaQuotes\Terminal\Common\Files\OpenAliceMt5HistoryV2" --broker icmarkets --symbol EURUSD --output "$research\icmarkets-eurusd-walk-forward.json" +``` + +The report shows every selection/test window and an aggregate of sequential unseen-period returns. It is historical evidence, not a forecast or a permission to trade. + +## Demo Terminal Readiness Check + +`CheckDemoReadiness.mq5` is the first live-terminal check. It is read-only: it refuses a non-demo account, checks the terminal connection and configured symbol, then writes the current demo server, contract, trade-permission, bid/ask, and spread information to a shared CSV. It cannot place, modify, or cancel an order. + +1. Copy the script into the relevant MT5 terminal's `MQL5\\Scripts` folder and compile it in MetaEditor. +2. Run it in the HFM demo terminal with `InpBrokerId=hfmarkets`, `InpSymbol=XAUUSD`. +3. Run it in the IC Markets terminal with `InpBrokerId=icmarkets`, `InpSymbol=XAUUSD`. +4. Confirm the terminal log says **"No order was sent"**. + +The generated reports appear at: + +```text +%APPDATA%\\MetaQuotes\\Terminal\\Common\\Files\\OpenAliceMt5DiagnosticsV1\\\\\\demo-readiness.csv +``` + +Run this check at normal market conditions, around rollover, and during a major-news window. It captures observed demo spreads for the eventual cost model; it does not claim those conditions will equal live execution. + +## Read-Only MT5 Connector (Phase 1) + +`OpenAliceMt5ReadOnlyBridge.mq5` is the first connector component. Attach it as an Expert Advisor to the Gold chart in each **demo** terminal; it writes a fresh local heartbeat every 30 seconds. OpenAlice reads that heartbeat to show terminal mode, server, quote/spread, and open position/order counts in the Research Desk. The bridge also refreshes Gold completed-D1 bars, spread samples, and a read-only Gold trade ledger for learning evidence. + +It has no `CTrade` dependency and no order-placement, amendment, or cancellation function. A heartbeat from a non-demo account is displayed as unsafe and does not advance the connector. + +| Terminal | EA inputs | +| --- | --- | +| HFM demo | `InpBrokerId=hfmarkets`, `InpSymbol=XAUUSD` | +| IC Markets demo | `InpBrokerId=icmarkets`, `InpSymbol=XAUUSD` | + +The bridge writes to: + +```text +%APPDATA%\\MetaQuotes\\Terminal\\Common\\Files\\OpenAliceMt5BridgeV1\\\\\\status.csv +``` + +The bridge remains read-only. Plan 3 adds a separate broker-local demo-canary EA; the bridge itself never gains order authority. + +## Read-only trade ledger exporter + +`OpenAliceMt5ReadOnlyBridge.mq5` now refreshes the Gold `XAUUSD` trade ledger automatically every `InpTradeLedgerRefreshMinutes` minutes after the bridge is recompiled and reattached. It writes MT5 deal history for the configured broker/symbol into MetaTrader Common Files: + +`OpenAliceMt5TradeLedgerV1///deals.csv` + +`ExportMt5TradeLedger.mq5` remains available as a one-shot manual exporter, especially for EURUSD shadow-learning ledgers or for an immediate manual catch-up before the next bridge refresh. + +Use it on demo accounts first: + +1. Open MetaEditor from the target MT5 terminal. +2. Copy or open `tools/mt5/ExportMt5TradeLedger.mq5`. +3. Compile it. +4. Run it once per broker/symbol with: + - HFM demo Gold: `InpBrokerId=hfmarkets`, `InpSymbol=XAUUSD` + - HFM demo EURUSD: `InpBrokerId=hfmarkets`, `InpSymbol=EURUSD` + - IC Markets demo Gold: `InpBrokerId=icmarkets`, `InpSymbol=XAUUSD` + - IC Markets demo EURUSD: `InpBrokerId=icmarkets`, `InpSymbol=EURUSD` + +Both the bridge ledger refresh and the exporter are read-only. They do not submit, modify, or close orders. If the exported account mode is not `demo`, JMB Goldmine must show the ledger as blocked for demo-autopilot progression. + +## Demo risk shell, no order submission + +`JmbGoldmineDemoRiskShell.mq5` is an EA that reads the latest JMB shadow decision and writes gate status to: + +```text +OpenAliceMt5RiskShellV1///gate_status.csv +``` + +This shell does not submit, modify, or close orders. It validates demo account mode, chart symbol, decision freshness, shadow mode, lot size, stop loss, spread, kill switch, and existing manual or foreign exposure before reporting `shadow_ready`. + +Recommended first run: + +- Keep `InpKillSwitch=true`. +- Attach to HFM demo `XAUUSD` and `EURUSD`. +- Attach to IC Markets demo `XAUUSD` and `EURUSD`. +- Use `InpBrokerId=hfmarkets` or `icmarkets`. +- Use the exact chart symbol in `InpSymbol`. + +Keep this EA available as the rollback and diagnostic shell for the Plan 3 ceremony below. + +## Plan 3 broker-local Gold demo canary + +The [approved Plan 3 design](../../docs/superpowers/specs/2026-07-13-jmb-goldmine-demo-canary-execution-design.md) governs this separately installed EA. UTA remains mandatory for app-, API-, UI-, workspace-, or AI-managed broker orders. `JmbGoldmineDemoCanary` is a narrower broker-local exception governed by the PRD's R6: it reads local artifacts, exposes no remote order command, and independently enforces its demo-account and Gold-only bindings inside MT5. + +Codex does not launch MetaEditor, compile MQL, attach an EA, enable Algo Trading, change terminal inputs, or submit a broker request. Every MetaEditor and terminal action below is an operator-only human gate. Keep terminal Algo Trading disabled throughout Stage 0. + +### Install and compile manually + +Perform these steps separately in the HFM demo terminal and the IC Markets demo terminal. Use each terminal's **File -> Open Data Folder** so files are copied into the correct installation. + +1. Re-copy `OpenAliceMt5ReadOnlyBridge.mq5` to `MQL5\Experts\OpenAliceMt5ReadOnlyBridge.mq5`. This version exports the latest 300 completed D1 bars, bounded spread samples, and the read-only Gold trade ledger in addition to its read-only heartbeat; recompile it even if an older bridge is already attached. +2. Copy `ConfigureJmbGoldmineDemoPolicy.mq5` to `MQL5\Scripts\ConfigureJmbGoldmineDemoPolicy.mq5`. +3. Copy the complete repository folder `JmbGoldmineDemoCanary\` to `MQL5\Experts\JmbGoldmineDemoCanary\`. Preserve `JmbGoldmineDemoCanary.mq5` and every sibling `.mqh` file in that layout. +4. Copy `tests\JmbGoldmineDemoCanaryHarness.mq5` to `MQL5\Experts\tests\JmbGoldmineDemoCanaryHarness.mq5`, preserving its relative path to `..\JmbGoldmineDemoCanary\`. +5. In MetaEditor opened from that terminal, compile the bridge, policy script, canary EA, and harness. Accept only `0 errors, 0 warnings` for every file. +6. With Algo Trading still disabled, attach the harness to an unused demo chart. Its Experts log must end with `JMB_CANARY_HARNESS PASS`, every case must print `PASS`, and the failure count must be zero. Remove the harness after the run. The harness includes no trade gateway. + +Compilation and harness results are pending until a human records them for both terminals. A source-code build or TypeScript test cannot substitute for this gate. + +### Shared Common Files layout + +OpenAlice and the terminals exchange local artifacts only through MetaTrader Common Files under `%APPDATA%\MetaQuotes\Terminal\Common\Files`. The operator should expect these broker-specific artifacts: + +```text +OpenAliceMt5BridgeV1\\XAUUSD\status.csv +OpenAliceMt5BridgeV1\\XAUUSD\completed_d1.csv +OpenAliceMt5BridgeV1\\XAUUSD\spread_samples_YYYYMMDD.csv +OpenAliceMt5DemoPolicyV1\\XAUUSD\policy.csv +OpenAliceMt5CostModelV1\\XAUUSD\cost_model.csv +OpenAliceMt5ExecutionDecisionV1\\XAUUSD\latest_decision.csv +OpenAliceMt5ExecutionDecisionV1\\XAUUSD\decisions.jsonl +OpenAliceMt5ExecutionV1\\XAUUSD\latest_status.csv +OpenAliceMt5ExecutionV1\\XAUUSD\events.jsonl +OpenAliceMt5ExecutionV1\\XAUUSD\processed_observations.csv +OpenAliceMt5ExecutionV1\\XAUUSD\reconciliation_latch.csv +``` + +The bridge owns its first three read-only files, the operator-only script owns `policy.csv`, OpenAlice's deterministic decision cycle owns the cost-model and decision files, and the EA owns the execution files. Do not hand-edit, truncate, copy between brokers, or delete the append-only event, decision, processed-observation, or reconciliation files during rollback or recovery. The local expected account login is an EA input only and must never be put in a policy, status file, screenshot, commit, or Research API response. + +### Stage 0: status-only dry run + +Stage 0 is a two-terminal human ceremony. It does not authorize any order. + +1. In each demo terminal, keep two `XAUUSD` charts: the recompiled read-only bridge stays on its existing duplicate Gold chart; the status/execution chart is reserved for the risk shell or canary. Do not attach the canary to EURUSD. +2. Reattach the bridge with `InpBrokerId=hfmarkets`, `InpSymbol=XAUUSD` in HFM and `InpBrokerId=icmarkets`, `InpSymbol=XAUUSD` in IC Markets. Confirm `status.csv`, `completed_d1.csv`, and the current monthly spread-sample file update. +3. Save the stable `JmbGoldmineDemoRiskShell` gate result for comparison. Remove that shell from the Gold status chart, then attach `JmbGoldmineDemoCanary` to the same chart with the exact status-only inputs below. +4. Run `ConfigureJmbGoldmineDemoPolicy` on each terminal's `XAUUSD` chart with `InpRolloutStage=status_only` and `InpCandidateApproved=false`. Use a unique immutable version such as `hfm-status-only-v1` or `ic-status-only-v1`. HFM may use the script's `0.75` spread and `0.50` deviation defaults. IC Markets must set both to `0.30`; the HFM defaults are deliberately refused on IC. Keep the remaining limits at or below 72 hours, `10.00` risk, `40.00` daily loss, 4 losses, and `0.01` lot. +5. Leave terminal Algo Trading disabled. Confirm the EA's `latest_status.csv` advances for at least two ten-second timer cycles on both terminals. Also observe at least two 30-second bridge cycles so its heartbeat and completed-D1 export remain current. +6. Compare every reported canary gate with the saved risk-shell result. Confirm the MT5 **Trade** and **History** tabs show zero new canary orders and zero new positions across the observation window. `execution_enabled` must be `0`, `kill_switch` must be `1`, and the rollout stage must be `status_only`. +7. Inspect the newest row in each `completed_d1.csv`. Its `bar_as_of` must be later than the stale `2026-06-23` artifact. A merely recent file modification time is insufficient. If either broker lacks fresh completed-D1 evidence, both canary promotions remain blocked. +8. Record terminal/server, compile and harness results, before/after order and position counts, two-cycle timestamps, risk-shell comparison, and completed-D1 `bar_as_of` values without recording account logins. + +Use these exact EA inputs; replace `` only in the terminal input dialog: + +| Input | HFM | IC Markets | +| --- | --- | --- | +| `InpBrokerId` | `hfmarkets` | `icmarkets` | +| `InpExpectedServer` | `HFMarketsGlobal-Demo4` | `ICMarketsSC-Demo` | +| `InpExpectedAccountLogin` | `` | `` | +| `InpSymbol` | `XAUUSD` | `XAUUSD` | +| `InpMagicNumber` | `880101` | `880201` | +| `InpDemoExecutionEnabled` | `false` | `false` | +| `InpKillSwitch` | `true` | `true` | + +Any non-demo account, login/server mismatch, missing cost or policy evidence, stale decision or completed observation, foreign Gold exposure, reconciliation error, or unwritable log remains blocked. Do not weaken an input to make Stage 0 appear ready. + +### HFM canary + +This is the Stage 1 ceremony and remains pending until the human Stage 0 evidence above is reviewed. + +1. Leave IC Markets at `status_only`, `InpDemoExecutionEnabled=false`, and `InpKillSwitch=true`. +2. Build and review an HFM `canary_ready` broker-cost model from fresh HFM bridge spread evidence. Confirm HFM has a fresh completed-D1 observation and an eligible `daily-trend-v1` Gold decision. +3. On the HFM demo `XAUUSD` chart only, run the operator policy script with a new policy version, `InpRolloutStage=hfm_canary`, `InpCandidateApproved=true`, and limits no looser than HFM's immutable ceilings. +4. Reconfirm the locally entered HFM demo login, server, symbol, and magic. Only the operator then sets `InpDemoExecutionEnabled=true`, sets `InpKillSwitch=false`, and enables terminal Algo Trading for the HFM canary. +5. Observe one eligible decision from its durable `order_requesting` event through broker result, accepted volume/price, broker-confirmed stop protection, append-only evidence, and restart reconciliation. The broker's Trade/History tabs are authoritative. +6. An unprotected fill, unknown result, partial result, reconciliation mismatch, non-demo identity, missing cost evidence, or duplicate/uncertain observation immediately returns HFM to `status_only` using the pause procedure below. Do not resend an unknown request. + +HFM evidence must be reviewed and accepted before any IC execution setting changes. + +### IC Markets canary + +This is the Stage 2 ceremony and remains pending until the HFM canary has broker-confirmed stop protection, durable request/result evidence, and successful restart reconciliation. + +1. Keep the HFM position, if any, protected and reconciled; otherwise pause HFM before proceeding. +2. Build and review an IC `canary_ready` cost model from IC-only evidence. Never reuse the HFM model. +3. On the IC Markets demo `XAUUSD` chart only, run the operator policy script with a new policy version, `InpRolloutStage=ic_canary`, `InpCandidateApproved=true`, `InpMaxSpread<=0.30`, `InpMaxDeviation<=0.30`, and the remaining immutable ceilings unchanged or tighter. +4. Reconfirm the locally entered IC demo login, `ICMarketsSC-Demo`, `XAUUSD`, and magic `880201`. Only the operator then sets `InpDemoExecutionEnabled=true`, sets `InpKillSwitch=false`, and enables Algo Trading for the IC Markets canary. +5. Repeat the HFM evidence chain and specifically verify IC spread, deviation, volume, server, cost, order/deal/position identifiers, broker-side stop, and restart behavior. + +Only after both ceremonies pass may the operator write `both_demo` policies. Each broker keeps independent gates, magic numbers, loss days, pauses, and evidence. EURUSD remains shadow-only and live execution remains absent at every stage. + +### Kill switch, pause, rollback, and recovery + +To pause a broker, set `InpKillSwitch=true` first, set `InpDemoExecutionEnabled=false`, write a new `status_only` policy with the operator script, and disable terminal Algo Trading if no other approved EA needs it. Confirm two `latest_status.csv` cycles with execution disabled. The kill switch blocks new entries; it does not close a correctly protected existing position. + +To roll back, keep the broker paused, capture `latest_status.csv`, `events.jsonl`, `processed_observations.csv`, `reconciliation_latch.csv`, and the broker Trade/History state, then detach `JmbGoldmineDemoCanary` from the Gold status chart and reattach `JmbGoldmineDemoRiskShell` with its kill switch on. Leave the read-only bridge on its duplicate Gold chart. The rollback shell submits no orders. Do not delete canary evidence or treat rollback as clearance of an unresolved broker result. + +For an unknown result, restart, or reconciliation failure, keep execution disabled and use the broker terminal to correlate symbol, magic, decision comment, order, deal, position, and stop with the durable event record. The EA never blindly resends. Restart the canary only with execution disabled and the kill switch on, then require two stable status cycles before review. + +If a fill lacks broker-confirmed stop protection, the EA's sole emergency path attempts to close only that EA-owned unprotected position and persists a protection-error pause. The operator must immediately verify the actual position and stop in the broker terminal, correlate the close result with `events.jsonl` and `reconciliation_latch.csv`, and leave the broker at `status_only`. A local `filled_protected` label is not evidence unless the broker confirms both exposure and stop. Never clear or delete the persistent latch merely to resume; promotion requires an explicit human incident review and approved recovery decision. + +There is no live-mode input, live-account bypass, EURUSD demo allowlist, AI risk override, or remote close/order command. App/API/AI-managed orders continue to require UTA approval and guards. diff --git a/tools/mt5/run_demo_canary_decisions.ts b/tools/mt5/run_demo_canary_decisions.ts new file mode 100644 index 000000000..f7759d449 --- /dev/null +++ b/tools/mt5/run_demo_canary_decisions.ts @@ -0,0 +1,12 @@ +import { runDemoDecisionCycle } from '../../src/domain/mt5/demo-decision-service.js' +import { resolveJmbMt5Roots } from '../../src/domain/mt5/local-paths.js' + +try { + const results = await runDemoDecisionCycle({ roots: resolveJmbMt5Roots() }) + for (const result of results) { + console.log(`${result.broker} ${result.symbol}: ${result.state} ${result.detail}`) + } +} catch (error) { + console.error(error instanceof Error ? error.message : 'Demo canary decision cycle failed.') + process.exitCode = 1 +} diff --git a/tools/mt5/run_shadow_decisions.ts b/tools/mt5/run_shadow_decisions.ts new file mode 100644 index 000000000..a6715f3cf --- /dev/null +++ b/tools/mt5/run_shadow_decisions.ts @@ -0,0 +1,114 @@ +import { readFile } from 'node:fs/promises' +import { homedir } from 'node:os' +import { join } from 'node:path' + +import { appendJmbDecisionRecord, writeLatestJmbDecision } from '../../src/domain/mt5/decision-record.js' +import { readMt5ReadOnlyBridge } from '../../src/domain/mt5/read-only-bridge.js' +import { buildShadowDecision } from '../../src/domain/mt5/shadow-decision-engine.js' +import { summarizeMt5TradeLedger } from '../../src/domain/mt5/trade-ledger.js' + +type TrendReport = { + latest_observation?: { direction: 'uptrend' | 'downtrend' | 'flat' } +} + +const appData = process.env['APPDATA'] ?? join(homedir(), 'AppData', 'Roaming') +const commonFiles = join(appData, 'MetaQuotes', 'Terminal', 'Common', 'Files') +const bridgeRoot = process.env['OPENALICE_MT5_BRIDGE_ROOT'] ?? join(commonFiles, 'OpenAliceMt5BridgeV1') +const tradeLedgerRoot = process.env['OPENALICE_MT5_TRADE_LEDGER_ROOT'] ?? join(commonFiles, 'OpenAliceMt5TradeLedgerV1') +const decisionRoot = process.env['OPENALICE_MT5_DECISION_ROOT'] ?? join(commonFiles, 'OpenAliceMt5DecisionLogV1') +const researchRoot = process.env['OPENALICE_RESEARCH_ARTIFACTS_DIR'] ?? join(homedir(), '.openalice', 'data', 'research') + +const instruments = [ + { + broker: 'hfmarkets', + symbol: 'XAUUSD', + report: 'xauusd-trend-baseline.json', + canonical: 'Gold / USD', + demoCandidateApproved: true, + maxSpread: 0.75, + stopDistance: 8, + maxRisk: 1, + }, + { + broker: 'hfmarkets', + symbol: 'EURUSD', + report: 'eurusd-trend-baseline.json', + canonical: 'Euro / USD', + demoCandidateApproved: false, + maxSpread: 0.00025, + stopDistance: 0.002, + maxRisk: 1, + }, + { + broker: 'icmarkets', + symbol: 'XAUUSD', + report: 'icmarkets-xauusd-trend-baseline.json', + canonical: 'Gold / USD', + demoCandidateApproved: true, + maxSpread: 0.3, + stopDistance: 8, + maxRisk: 1, + }, + { + broker: 'icmarkets', + symbol: 'EURUSD', + report: 'icmarkets-eurusd-trend-baseline.json', + canonical: 'Euro / USD', + demoCandidateApproved: false, + maxSpread: 0.00015, + stopDistance: 0.002, + maxRisk: 1, + }, +] as const + +async function readTrendDirection(fileName: string): Promise<'uptrend' | 'downtrend' | 'flat'> { + try { + const report = JSON.parse(await readFile(join(researchRoot, fileName), 'utf8')) as TrendReport + return report.latest_observation?.direction ?? 'flat' + } catch { + return 'flat' + } +} + +for (const instrument of instruments) { + const [bridge, learning, latestDirection] = await Promise.all([ + readMt5ReadOnlyBridge(bridgeRoot, instrument.broker, instrument.symbol), + summarizeMt5TradeLedger(tradeLedgerRoot, instrument.broker, instrument.symbol), + readTrendDirection(instrument.report), + ]) + const referencePrice = latestDirection === 'downtrend' ? bridge.bid : bridge.ask + const stopLoss = + referencePrice == null + ? null + : latestDirection === 'downtrend' + ? Number((referencePrice + instrument.stopDistance).toFixed(instrument.symbol === 'XAUUSD' ? 2 : 5)) + : latestDirection === 'uptrend' + ? Number((referencePrice - instrument.stopDistance).toFixed(instrument.symbol === 'XAUUSD' ? 2 : 5)) + : null + const decision = buildShadowDecision({ + createdAt: new Date().toISOString(), + broker: instrument.broker, + server: bridge.server, + accountMode: bridge.state === 'ready' ? 'demo' : null, + symbol: instrument.symbol, + canonicalInstrument: instrument.canonical, + strategyVersion: 'daily-trend-v1', + bridgeState: bridge.state, + learningState: learning.state, + latestDirection, + bid: bridge.bid, + ask: bridge.ask, + spread: bridge.spread, + maxSpread: instrument.maxSpread, + volume: 0.01, + maxVolume: 0.01, + stopLoss, + riskAmount: instrument.maxRisk, + maxAllowedRisk: instrument.maxRisk, + demoCandidateApproved: instrument.demoCandidateApproved, + }) + + await appendJmbDecisionRecord(decisionRoot, decision) + await writeLatestJmbDecision(decisionRoot, decision) + console.log(`${decision.broker} ${decision.symbol}: ${decision.mode} ${decision.direction} ${decision.reasonCode}`) +} diff --git a/tools/mt5/tests/JmbGoldmineDemoCanaryHarness.mq5 b/tools/mt5/tests/JmbGoldmineDemoCanaryHarness.mq5 new file mode 100644 index 000000000..755d19963 --- /dev/null +++ b/tools/mt5/tests/JmbGoldmineDemoCanaryHarness.mq5 @@ -0,0 +1,687 @@ +// Table-driven, submission-free harness for the Task 6 demo-canary pure contracts. +#property strict +#property version "1.200" + +#include "..\JmbGoldmineDemoCanary\JmbCanaryTypes.mqh" +#include "..\JmbGoldmineDemoCanary\JmbCanaryGates.mqh" +#include "..\JmbGoldmineDemoCanary\JmbCanaryState.mqh" +#include "..\JmbGoldmineDemoCanary\JmbCanaryReconcile.mqh" +#include "..\JmbGoldmineDemoCanary\JmbCanaryTradeGateway.mqh" + +enum HarnessMutation +{ + HARNESS_READY=0, + HARNESS_NON_DEMO, + HARNESS_SERVER_MISMATCH, + HARNESS_SYMBOL_MISMATCH, + HARNESS_MAGIC_MISMATCH, + HARNESS_EXECUTION_DISABLED, + HARNESS_KILL_SWITCH, + HARNESS_ROLLOUT_BLOCKED, + HARNESS_VOLUME_INVALID, + HARNESS_STOP_INVALID, + HARNESS_STOP_MODE_UNSUPPORTED, + HARNESS_STOP_TICK_UNAVAILABLE, + HARNESS_STOP_TICK_MISALIGNED, + HARNESS_RISK_EXCESSIVE, + HARNESS_TIGHT_DAILY_LOSS_EQUALITY, + HARNESS_HARD_DAILY_LOSS_EQUALITY, + HARNESS_SPREAD_EXCESSIVE, + HARNESS_SESSION_CLOSED, + HARNESS_NEWS_BLOCKED, + HARNESS_EXPOSURE_PRESENT, + HARNESS_DUPLICATE_OBSERVATION, + HARNESS_MALFORMED_PROCESSED_STATE, + HARNESS_LOG_FAILURE, + HARNESS_RECONCILIATION_MISSING +}; + +struct HarnessCase +{ + string name; + HarnessMutation mutation; + CanaryLifecycleState expectedState; + string expectedGate; +}; + +enum HarnessReconciliationMutation +{ + HARNESS_RECONCILE_REJECTED=0, + HARNESS_RECONCILE_UNKNOWN, + HARNESS_RECONCILE_PARTIAL, + HARNESS_RECONCILE_FILLED_WITH_STOP, + HARNESS_RECONCILE_FILLED_WITHOUT_STOP, + HARNESS_RECONCILE_STOPPED, + HARNESS_RECONCILE_SAME_DIRECTION, + HARNESS_RECONCILE_OPPOSITE_DIRECTION, + HARNESS_RECONCILE_FOUR_LOSSES, + HARNESS_RECONCILE_SERVER_DAY_RESET, + HARNESS_RECONCILE_RESTART_PROTECTED, + HARNESS_RECONCILE_RESTART_FOREIGN, + HARNESS_RECONCILE_MIXED_UNPROTECTED, + HARNESS_RECONCILE_MULTIPLE_POSITIONS, + HARNESS_RECONCILE_PROTECTION_PAUSE_AFTER_CLOSE +}; + +struct HarnessReconciliationCase +{ + string name; + HarnessReconciliationMutation mutation; + CanaryLifecycleState expectedState; +}; + +void AddHarnessCase(HarnessCase &cases[],const string name,const HarnessMutation mutation, + const CanaryLifecycleState expected_state,const string expected_gate) +{ + int index=ArraySize(cases); + ArrayResize(cases,index+1); + cases[index].name=name; + cases[index].mutation=mutation; + cases[index].expectedState=expected_state; + cases[index].expectedGate=expected_gate; +} + +void BuildHarnessPolicy(CanaryPolicy &policy) +{ + policy.loaded=true; + policy.schemaVersion=1; + policy.policyVersion="harness-policy-v1"; + policy.broker="hfmarkets"; + policy.server="HFMarketsGlobal-Demo4"; + policy.symbol="XAUUSD"; + policy.strategyVersion="daily-trend-v1"; + policy.rolloutStage="hfm_canary"; + policy.candidateApproved=true; + policy.completedObservationMaxAgeHours=72.0; + policy.maxSpread=0.75; + policy.maxDeviation=0.50; + policy.maxRiskAmount=10.0; + policy.maxDailyLoss=40.0; + policy.maxDailyLosingTrades=4; + policy.maxVolume=0.01; + policy.magicNumber=880101; +} + +void BuildHarnessDecision(CanaryDecision &decision) +{ + decision.loaded=true; + decision.schemaVersion=1; + decision.decisionId="decision-harness"; + decision.observationId="observation-harness"; + decision.observationAsOf="2026-07-12"; + decision.createdAt=1; + decision.leaseIssuedAt=1; + decision.leaseExpiresAt=2; + decision.broker="hfmarkets"; + decision.server="HFMarketsGlobal-Demo4"; + decision.accountMode="demo"; + decision.symbol="XAUUSD"; + decision.strategyVersion="daily-trend-v1"; + decision.direction="buy"; + decision.entryReferencePrice=2400.0; + decision.volume=0.01; + decision.stopLoss=2399.0; + decision.maxRiskAmount=10.0; + decision.candidatePolicyVersion="harness-policy-v1"; + decision.costModelVersion="harness-cost-v1"; + decision.gateResultsJson="[]"; + decision.preDecisionGatesPassed=true; +} + +void BuildHarnessEnvironment(CanaryEnvironment &environment) +{ + ZeroMemory(environment); + environment.accountIsDemo=true; + environment.loginMatches=true; + environment.serverMatches=true; + environment.brokerMatches=true; + environment.chartSymbolMatches=true; + environment.magicMatches=true; + environment.executionEnabled=true; + environment.killSwitch=false; + environment.rolloutAuthorized=true; + environment.candidateApproved=true; + environment.allowlistsMatch=true; + environment.decisionFresh=true; + environment.bridgeFresh=true; + environment.policyFresh=true; + environment.costModelFresh=true; + environment.observationFresh=true; + environment.processedStateAvailable=true; + environment.observationUnused=true; + environment.volumeEvidenceAvailable=true; + environment.volumeCompatible=true; + environment.stopEvidenceAvailable=true; + environment.stopModeSupportsSl=true; + environment.stopTickSizeAvailable=true; + environment.stopTickAligned=true; + environment.stopBrokerValid=true; + environment.riskCalculationAvailable=true; + environment.calculatedStopRisk=5.0; + environment.dailyStateAvailable=true; + environment.dailyRealizedLoss=0.0; + environment.dailyLossCount=0; + environment.exposureStateAvailable=true; + environment.hasEaPosition=false; + environment.hasEaPendingOrder=false; + environment.hasForeignGoldExposure=false; + environment.marginCalculationAvailable=true; + environment.estimatedMargin=10.0; + environment.freeMargin=120.0; + environment.spreadAvailable=true; + environment.currentSpread=0.20; + environment.deviationAvailable=true; + environment.requestedDeviation=0.20; + environment.sessionEvidenceAvailable=true; + environment.sessionOpen=true; + environment.newsEvidenceAvailable=true; + environment.newsBlackout=false; + environment.logPreflightReady=true; + environment.reconciliationComplete=true; +} + +void ApplyHarnessMutation(const HarnessMutation mutation,CanaryPolicy &policy, + CanaryDecision &decision,CanaryEnvironment &environment) +{ + if(mutation==HARNESS_NON_DEMO) environment.accountIsDemo=false; + else if(mutation==HARNESS_SERVER_MISMATCH) environment.serverMatches=false; + else if(mutation==HARNESS_SYMBOL_MISMATCH) environment.chartSymbolMatches=false; + else if(mutation==HARNESS_MAGIC_MISMATCH) environment.magicMatches=false; + else if(mutation==HARNESS_EXECUTION_DISABLED) environment.executionEnabled=false; + else if(mutation==HARNESS_KILL_SWITCH) environment.killSwitch=true; + else if(mutation==HARNESS_ROLLOUT_BLOCKED) environment.rolloutAuthorized=false; + else if(mutation==HARNESS_VOLUME_INVALID) decision.volume=0.02; + else if(mutation==HARNESS_STOP_INVALID) environment.stopBrokerValid=false; + else if(mutation==HARNESS_STOP_MODE_UNSUPPORTED) environment.stopModeSupportsSl=false; + else if(mutation==HARNESS_STOP_TICK_UNAVAILABLE) environment.stopTickSizeAvailable=false; + else if(mutation==HARNESS_STOP_TICK_MISALIGNED) environment.stopTickAligned=false; + else if(mutation==HARNESS_RISK_EXCESSIVE) environment.calculatedStopRisk=10.01; + else if(mutation==HARNESS_TIGHT_DAILY_LOSS_EQUALITY) + { + policy.maxDailyLoss=5.0; + environment.dailyRealizedLoss=5.0; + } + else if(mutation==HARNESS_HARD_DAILY_LOSS_EQUALITY) environment.dailyRealizedLoss=40.0; + else if(mutation==HARNESS_SPREAD_EXCESSIVE) environment.currentSpread=0.76; + else if(mutation==HARNESS_SESSION_CLOSED) environment.sessionOpen=false; + else if(mutation==HARNESS_NEWS_BLOCKED) environment.newsBlackout=true; + else if(mutation==HARNESS_EXPOSURE_PRESENT) environment.hasForeignGoldExposure=true; + else if(mutation==HARNESS_DUPLICATE_OBSERVATION) environment.observationUnused=false; + else if(mutation==HARNESS_MALFORMED_PROCESSED_STATE) + { + environment.processedStateAvailable=false; + environment.observationUnused=false; + } + else if(mutation==HARNESS_LOG_FAILURE) environment.logPreflightReady=false; + else if(mutation==HARNESS_RECONCILIATION_MISSING) environment.reconciliationComplete=false; +} + +bool RunHarnessCase(const HarnessCase &test_case) +{ + CanaryPolicy policy; + CanaryDecision decision; + CanaryEnvironment environment; + BuildHarnessPolicy(policy); + BuildHarnessDecision(decision); + BuildHarnessEnvironment(environment); + ApplyHarnessMutation(test_case.mutation,policy,decision,environment); + CanaryEvaluation evaluation=EvaluateCanaryGates(decision,policy,environment); + bool passed=evaluation.state==test_case.expectedState && evaluation.blockingGate==test_case.expectedGate; + PrintFormat("%s %s state=%s gate=%s",passed ? "PASS" : "FAIL",test_case.name, + CanaryLifecycleLabel(evaluation.state),evaluation.blockingGate); + return passed; +} + +void AddHarnessReconciliationCase(HarnessReconciliationCase &cases[],const string name, + const HarnessReconciliationMutation mutation, + const CanaryLifecycleState expected_state) +{ + int index=ArraySize(cases); + ArrayResize(cases,index+1); + cases[index].name=name; + cases[index].mutation=mutation; + cases[index].expectedState=expected_state; +} + +void BuildHarnessReconciliationFacts(CanaryReconciliationFacts &facts) +{ + ZeroMemory(facts); + facts.brokerStateAvailable=true; + facts.resultClass=CANARY_RESULT_NONE; +} + +void ApplyHarnessReconciliationMutation(const HarnessReconciliationMutation mutation, + CanaryReconciliationFacts &facts) +{ + if(mutation==HARNESS_RECONCILE_REJECTED) facts.resultClass=CANARY_RESULT_REJECTED; + else if(mutation==HARNESS_RECONCILE_UNKNOWN) facts.resultClass=CANARY_RESULT_UNKNOWN; + else if(mutation==HARNESS_RECONCILE_PARTIAL) facts.resultClass=CANARY_RESULT_PARTIAL; + else if(mutation==HARNESS_RECONCILE_FILLED_WITH_STOP + || mutation==HARNESS_RECONCILE_RESTART_PROTECTED) + { + facts.hasEaPosition=true; + facts.eaPositionProtected=true; + } + else if(mutation==HARNESS_RECONCILE_FILLED_WITHOUT_STOP) + facts.hasEaPosition=true; + else if(mutation==HARNESS_RECONCILE_STOPPED) facts.stoppedObservation=true; + else if(mutation==HARNESS_RECONCILE_SAME_DIRECTION) + { + facts.hasEaPosition=true; + facts.eaPositionProtected=true; + facts.sameDirection=true; + } + else if(mutation==HARNESS_RECONCILE_OPPOSITE_DIRECTION) + { + facts.hasEaPosition=true; + facts.eaPositionProtected=true; + facts.oppositeDirection=true; + } + else if(mutation==HARNESS_RECONCILE_FOUR_LOSSES) facts.dailyLimitReached=true; + else if(mutation==HARNESS_RECONCILE_RESTART_FOREIGN) facts.hasForeignGoldExposure=true; + else if(mutation==HARNESS_RECONCILE_MIXED_UNPROTECTED) + { + facts.hasEaPosition=true; + facts.hasForeignGoldExposure=true; + } + else if(mutation==HARNESS_RECONCILE_MULTIPLE_POSITIONS) + { + facts.brokerStateAvailable=false; + facts.hasEaPosition=true; + } + else if(mutation==HARNESS_RECONCILE_PROTECTION_PAUSE_AFTER_CLOSE) + { + facts.resultClass=CANARY_RESULT_NONE; + facts.persistentSafetyPause=true; + } +} + +bool RunHarnessReconciliationCase(const HarnessReconciliationCase &test_case) +{ + CanaryReconciliationFacts facts; + BuildHarnessReconciliationFacts(facts); + ApplyHarnessReconciliationMutation(test_case.mutation,facts); + CanaryLifecycleState actual=ReduceCanaryLifecycle(facts); + bool passed=actual==test_case.expectedState; + PrintFormat("%s %s state=%s",passed ? "PASS" : "FAIL",test_case.name, + CanaryAuthoritativeLifecycleLabel(actual)); + return passed; +} + +bool RunActionableOppositeCase() +{ + bool passed=!IsCanaryActionableOpposite("flat","buy") + && !IsCanaryActionableOpposite("flat","sell") + && !IsCanaryActionableOpposite("buy","buy") + && IsCanaryActionableOpposite("buy","sell") + && IsCanaryActionableOpposite("sell","buy"); + PrintFormat("%s flat decision cannot close",passed ? "PASS" : "FAIL"); + return passed; +} + +bool RunClosedOwnershipCase() +{ + bool passed=ClassifyCanaryClosedPositionOwnership(true,false,true)==CANARY_CLOSED_OWNERSHIP_EA + && ClassifyCanaryClosedPositionOwnership(false,false,true)==CANARY_CLOSED_OWNERSHIP_FOREIGN + && ClassifyCanaryClosedPositionOwnership(true,true,true)==CANARY_CLOSED_OWNERSHIP_UNSAFE; + PrintFormat("%s nonmagic final closure ownership",passed ? "PASS" : "FAIL"); + return passed; +} + +bool RunLifecycleCorrelationCase() +{ + string decision_id="0123456789abcdef01234567"; + string correlation=CanaryEntryCorrelationComment(decision_id); + bool correlated=IsCanaryCorrelatedLifecyclePosition(decision_id,4321,correlation,4321); + bool unrelated=!IsCanaryCorrelatedLifecyclePosition(decision_id,4321, + CanaryEntryCorrelationComment("89abcdef0123456701234567"),9876); + CanaryReconciliationFacts facts; + BuildHarnessReconciliationFacts(facts); + facts.resultClass=CANARY_RESULT_UNKNOWN; + facts.stoppedObservation=correlated; + bool stopped=ReduceCanaryLifecycle(facts)==CANARY_LIFECYCLE_STOPPED; + facts.stoppedObservation=!unrelated; + bool blocked=ReduceCanaryLifecycle(facts)==CANARY_LIFECYCLE_RECONCILIATION_REQUIRED; + PrintFormat("%s correlated stopped observation",stopped ? "PASS" : "FAIL"); + PrintFormat("%s unrelated stop remains unresolved",blocked ? "PASS" : "FAIL"); + PrintFormat("%s position-specific rollover recovery",correlated ? "PASS" : "FAIL"); + return correlated && unrelated && stopped && blocked; +} + +bool RunProtectionPauseSemanticsCase() +{ + CanaryReconciliation reconciliation; + InitializeCanaryReconciliation(reconciliation); + reconciliation.state=CANARY_LIFECYCLE_PAUSED; + reconciliation.reconciliationState="protection_error"; + bool passed=IsCanaryPersistentProtectionPause(reconciliation); + PrintFormat("%s persistent protection status semantics",passed ? "PASS" : "FAIL"); + return passed; +} + +bool RunDuplicateObservationStateCase() +{ + CanaryProcessedState state; + InitializeCanaryProcessedState(state); + state.valid=true; + ArrayResize(state.decisionIds,1); + ArrayResize(state.observationIds,1); + ArrayResize(state.attemptedAt,1); + state.decisionIds[0]="bc0bc128a9155065dda0b5bc"; + state.observationIds[0]="53f2bd057c1ee3608a02d1f2"; + state.attemptedAt[0]=1; + bool passed=CanaryProcessedStateContains(state,"bc0bc128a9155065dda0b5bc","new-observation") + && CanaryProcessedStateContains(state,"new-decision","53f2bd057c1ee3608a02d1f2") + && !CanaryProcessedStateContains(state,"new-decision","new-observation"); + PrintFormat("%s loaded duplicate state",passed ? "PASS" : "FAIL"); + return passed; +} + +bool RunTask3IdentityCase() +{ + CanaryDecision decision; + BuildHarnessDecision(decision); + string observation_id=CreateCanaryObservationId(decision); + decision.observationId=observation_id; + string decision_id=CreateCanaryDecisionId(decision); + bool passed=observation_id=="53f2bd057c1ee3608a02d1f2" + && decision_id=="bc0bc128a9155065dda0b5bc" + && "arbitrary-observation"!=observation_id + && "arbitrary-decision"!=decision_id; + PrintFormat("%s arbitrary decision ids",passed ? "PASS" : "FAIL"); + return passed; +} + +bool RunGateJsonContractCase() +{ + string valid="[{\"name\":\"bridge\",\"state\":\"pass\",\"detail\":\"ok\"}," + +"{\"name\":\"completed_observation\",\"state\":\"pass\",\"detail\":\"ok\"}," + +"{\"name\":\"candidate_policy\",\"state\":\"pass\",\"detail\":\"ok\"}," + +"{\"name\":\"cost_model\",\"state\":\"pass\",\"detail\":\"ok\"}," + +"{\"name\":\"learning\",\"state\":\"pass\",\"detail\":\"ok\"}," + +"{\"name\":\"quote\",\"state\":\"pass\",\"detail\":\"ok\"}," + +"{\"name\":\"spread\",\"state\":\"pass\",\"detail\":\"ok\"}," + +"{\"name\":\"direction\",\"state\":\"pass\",\"detail\":\"ok\"}," + +"{\"name\":\"stop_loss\",\"state\":\"pass\",\"detail\":\"ok\"}]"; + string malformed=valid+" trailing"; + string unknown_name=valid; + StringReplace(unknown_name,"\"name\":\"bridge\"","\"name\":\"unknown\""); + string unknown_state=valid; + StringReplace(unknown_state,"\"state\":\"pass\"","\"state\":\"warn\""); + string unknown_field=valid; + StringReplace(unknown_field,"\"detail\":\"ok\"","\"extra\":\"x\",\"detail\":\"ok\""); + string duplicate_field=valid; + StringReplace(duplicate_field,"\"state\":\"pass\"","\"name\":\"bridge\",\"state\":\"pass\""); + string missing_field=valid; + StringReplace(missing_field,",\"detail\":\"ok\"",""); + string noncanonical=valid; + StringReplace(noncanonical,"[{","[ {"); + bool all_passed=false; + string detail=""; + bool passed=ParseCanaryGateResultsJson(valid,all_passed,detail) && all_passed + && !ParseCanaryGateResultsJson(malformed,all_passed,detail) + && !ParseCanaryGateResultsJson(unknown_name,all_passed,detail) + && !ParseCanaryGateResultsJson(unknown_state,all_passed,detail) + && !ParseCanaryGateResultsJson(unknown_field,all_passed,detail) + && !ParseCanaryGateResultsJson(duplicate_field,all_passed,detail) + && !ParseCanaryGateResultsJson(missing_field,all_passed,detail) + && !ParseCanaryGateResultsJson(noncanonical,all_passed,detail); + PrintFormat("%s malformed gate JSON: unknown gate name, unknown gate state, unknown gate field, duplicate gate field, missing gate field, noncanonical gate evidence", + passed ? "PASS" : "FAIL"); + return passed; +} + +bool RunJsonEscapeCase() +{ + string canonical="\"quote\\\" reverse\\\\ solidus\\/ backspace\\b formfeed\\f newline\\n return\\r tab\\t bmp\\u263A pair\\uD83D\\uDE00\""; + string invalid_escape="\"bad\\x\""; + string invalid_hex="\"bad\\uZZZZ\""; + string lone_high="\"bad\\uD83D\""; + string lone_low="\"bad\\uDE00\""; + string invalid_pair="\"bad\\uD83D\\u263A\""; + int cursor=0; + string decoded=""; + string expected="quote\" reverse\\ solidus/ backspace"+ShortToString((ushort)8) + +" formfeed"+ShortToString((ushort)12)+" newline"+ShortToString((ushort)10) + +" return"+ShortToString((ushort)13)+" tab"+ShortToString((ushort)9)+" bmp"+ShortToString((ushort)0x263A) + +" pair"+ShortToString((ushort)0xD83D)+ShortToString((ushort)0xDE00); + bool canonical_passed=ParseCanonicalCanaryJsonString(canonical,cursor,decoded) + && cursor==StringLen(canonical) && decoded==expected; + PrintFormat("%s canonical JSON escapes",canonical_passed ? "PASS" : "FAIL"); + + cursor=0; + bool invalid_escape_blocked=!ParseCanonicalCanaryJsonString(invalid_escape,cursor,decoded); + cursor=0; + invalid_escape_blocked=invalid_escape_blocked + && !ParseCanonicalCanaryJsonString(invalid_hex,cursor,decoded); + PrintFormat("%s invalid JSON escapes",invalid_escape_blocked ? "PASS" : "FAIL"); + + cursor=0; + bool invalid_surrogates_blocked=!ParseCanonicalCanaryJsonString(lone_high,cursor,decoded); + cursor=0; + invalid_surrogates_blocked=invalid_surrogates_blocked + && !ParseCanonicalCanaryJsonString(lone_low,cursor,decoded); + cursor=0; + invalid_surrogates_blocked=invalid_surrogates_blocked + && !ParseCanonicalCanaryJsonString(invalid_pair,cursor,decoded); + PrintFormat("%s invalid surrogate pairs",invalid_surrogates_blocked ? "PASS" : "FAIL"); + return canonical_passed && invalid_escape_blocked && invalid_surrogates_blocked; +} + +bool RunQuotedPolicyCsvCase() +{ + string row="1,harness-policy-v1,hfmarkets,HFMarketsGlobal-Demo4,XAUUSD,daily-trend-v1,hfm_canary,1,72,0.75,0.50,10,40,4,0.01,880101"; + string valid=CANARY_POLICY_HEADER+"\n"+row+"\n"; + string fully_quoted=CANARY_POLICY_HEADER+"\n" + +"\"1\",\"harness-policy-v1\",\"hfmarkets\",\"HFMarketsGlobal-Demo4\",\"XAUUSD\",\"daily-trend-v1\",\"hfm_canary\",\"1\",\"72\",\"0.75\",\"0.50\",\"10\",\"40\",\"4\",\"0.01\",\"880101\"\n"; + string partially_quoted=CANARY_POLICY_HEADER+"\n1,\"harness-policy-v1\",hfmarkets,HFMarketsGlobal-Demo4,XAUUSD,daily-trend-v1,hfm_canary,1,72,0.75,0.50,10,40,4,0.01,880101\n"; + CanaryPolicy policy; + string detail=""; + bool valid_passed=ParseCanaryPolicyCsvText(valid,policy,detail); + bool fully_blocked=!ParseCanaryPolicyCsvText(fully_quoted,policy,detail); + bool partially_blocked=!ParseCanaryPolicyCsvText(partially_quoted,policy,detail); + PrintFormat("%s fully quoted policy row",fully_blocked ? "PASS" : "FAIL"); + PrintFormat("%s partially quoted policy row",partially_blocked ? "PASS" : "FAIL"); + return valid_passed && fully_blocked && partially_blocked; +} + +bool RunStatusExactComparisonCase() +{ + string intended_values[]; + string verified_values[]; + ArrayResize(intended_values,29); + ArrayResize(verified_values,29); + for(int index=0;index<29;index++) + { + intended_values[index]="value-"+IntegerToString(index); + verified_values[index]=intended_values[index]; + } + bool exact_passed=CanaryExactValuesMatch(intended_values,verified_values); + verified_values[28]="truncated"; + bool truncation_blocked=!CanaryExactValuesMatch(intended_values,verified_values); + PrintFormat("%s truncated next safe action",truncation_blocked ? "PASS" : "FAIL"); + return exact_passed && truncation_blocked; +} + +bool RunPolicyVersionGrammarCase() +{ + bool passed=IsCanonicalCanaryPolicyVersion("operator-policy-v1") + && !IsCanonicalCanaryPolicyVersion("") + && !IsCanonicalCanaryPolicyVersion(" policy") + && !IsCanonicalCanaryPolicyVersion("policy ") + && !IsCanonicalCanaryPolicyVersion("policy,version") + && !IsCanonicalCanaryPolicyVersion("policy\"version") + && !IsCanonicalCanaryPolicyVersion("policy\nversion"); + PrintFormat("%s policy version grammar",passed ? "PASS" : "FAIL"); + return passed; +} + +bool RunMalformedProcessedStateCase() +{ + CanaryProcessedState state; + InitializeCanaryProcessedState(state); + bool passed=!state.valid && CanaryProcessedStateContains(state,"new-decision","new-observation"); + PrintFormat("%s malformed processed state",passed ? "PASS" : "FAIL"); + return passed; +} + +bool RunFourLossResetCase() +{ + CanaryDailyState state; + state.brokerDay="2026-07-12"; + state.lossCount=4; + state.realizedLoss=40.0; + ResetCanaryDailyStateForDay(state,"2026-07-12"); + bool retained=state.lossCount==4 && CanaryNearlyEqual(state.realizedLoss,40.0); + ResetCanaryDailyStateForDay(state,"2026-07-13"); + bool reset=state.brokerDay=="2026-07-13" && state.lossCount==0 + && CanaryNearlyEqual(state.realizedLoss,0.0); + bool passed=retained && reset; + PrintFormat("%s server day reset",passed ? "PASS" : "FAIL"); + return passed; +} + +bool RunGatewayBindingCase() +{ + bool wrong_account_blocked=!CanaryGatewayBindingMatches("hfmarkets","HFMarketsGlobal-Demo4", + 123456,"XAUUSD",880101,ACCOUNT_TRADE_MODE_DEMO,"HFMarketsGlobal-Demo4",654321,"XAUUSD"); + bool account_switch_blocked=!CanaryGatewayBindingMatches("hfmarkets","HFMarketsGlobal-Demo4", + 123456,"XAUUSD",880101,ACCOUNT_TRADE_MODE_DEMO,"ICMarketsSC-Demo",123456,"XAUUSD"); + PrintFormat("%s wrong demo account blocks emergency close",wrong_account_blocked ? "PASS" : "FAIL"); + PrintFormat("%s account switch blocks reversal close",account_switch_blocked ? "PASS" : "FAIL"); + return wrong_account_blocked && account_switch_blocked; +} + +bool RunActivePositionCorrelationCase() +{ + CanarySafetyLatch latch; + InitializeCanarySafetyLatch(latch); + latch.valid=true; + latch.pendingEntryDecisionId="bc0bc128a9155065dda0b5bc"; + latch.pendingEntryObservationId="53f2bd057c1ee3608a02d1f2"; + latch.pendingEntryAttemptedAt=1; + latch.pendingRequestedVolume=0.01; + latch.pendingRequestedPrice=2400.0; + latch.pendingRequestedStopLoss=2392.0; + latch.pendingCalculatedRisk=7.5; + latch.pendingEntryComment=CanaryEntryCorrelationComment(latch.pendingEntryDecisionId); + bool activated=ActivateCanaryPositionCorrelation(latch,4321); + ClearCanaryPendingEntryLatch(latch); + CanaryExecutionEvent stopped; + InitializeCanaryExecutionEvent(stopped); + bool restart_stop_correlated=activated && ApplyCanaryPositionCorrelation(stopped,latch) + && stopped.decisionId=="bc0bc128a9155065dda0b5bc" + && stopped.observationId=="53f2bd057c1ee3608a02d1f2" + && CanaryNearlyEqual(stopped.calculatedRisk,7.5); + string reversal_decision="0123456789abcdef01234567"; + bool decisions_separate=restart_stop_correlated && stopped.decisionId!=reversal_decision; + PrintFormat("%s protected fill then later stop after restart",restart_stop_correlated ? "PASS" : "FAIL"); + PrintFormat("%s opening decision differs from reversal decision",decisions_separate ? "PASS" : "FAIL"); + return restart_stop_correlated && decisions_separate; +} + +bool RunEmergencyTerminalCorrelationCase() +{ + CanarySafetyLatch latch; + InitializeCanarySafetyLatch(latch); + latch.valid=true; + latch.unresolved=false; + latch.protectionError=true; + latch.emergencyCloseAttempted=true; + latch.emergencyPositionId="4321"; + latch.activePositionDecisionId="bc0bc128a9155065dda0b5bc"; + latch.activePositionObservationId="53f2bd057c1ee3608a02d1f2"; + latch.activePositionId="4321"; + latch.activeRequestedVolume=0.01; + latch.activeRequestedPrice=2400.0; + latch.activeRequestedStopLoss=2392.0; + latch.activeCalculatedRisk=7.5; + latch.activeEntryComment=CanaryEntryCorrelationComment(latch.activePositionDecisionId); + + bool rejected=FinalizeCanaryEmergencyTerminalCorrelation(latch,false); + bool waited=!rejected && !latch.unresolved && latch.activePositionDecisionId!="" + && CanaryNearlyEqual(latch.activeCalculatedRisk,7.5); + bool finalized=FinalizeCanaryEmergencyTerminalCorrelation(latch,true); + bool cleared=finalized && !latch.unresolved && latch.activePositionDecisionId=="" + && latch.activePositionObservationId=="" && latch.activePositionId==""; + bool pause_retained=latch.protectionError && latch.emergencyCloseAttempted + && latch.emergencyPositionId=="4321"; + PrintFormat("%s emergency terminal waits for durable event",waited ? "PASS" : "FAIL"); + PrintFormat("%s emergency terminal clears active correlation after restart",cleared ? "PASS" : "FAIL"); + PrintFormat("%s emergency protection pause remains",pause_retained ? "PASS" : "FAIL"); + return waited && cleared && pause_retained; +} + +int OnInit() +{ + HarnessCase cases[]; + AddHarnessCase(cases,"all gates ready",HARNESS_READY,CANARY_LIFECYCLE_READY,""); + AddHarnessCase(cases,"demo binding",HARNESS_NON_DEMO,CANARY_LIFECYCLE_BLOCKED,"demo_identity"); + AddHarnessCase(cases,"server binding",HARNESS_SERVER_MISMATCH,CANARY_LIFECYCLE_BLOCKED,"demo_identity"); + AddHarnessCase(cases,"symbol binding",HARNESS_SYMBOL_MISMATCH,CANARY_LIFECYCLE_BLOCKED,"demo_identity"); + AddHarnessCase(cases,"magic binding",HARNESS_MAGIC_MISMATCH,CANARY_LIFECYCLE_BLOCKED,"demo_identity"); + AddHarnessCase(cases,"execution switch",HARNESS_EXECUTION_DISABLED,CANARY_LIFECYCLE_DISABLED,"switches"); + AddHarnessCase(cases,"kill switch",HARNESS_KILL_SWITCH,CANARY_LIFECYCLE_PAUSED,"switches"); + AddHarnessCase(cases,"rollout",HARNESS_ROLLOUT_BLOCKED,CANARY_LIFECYCLE_BLOCKED,"rollout"); + AddHarnessCase(cases,"volume",HARNESS_VOLUME_INVALID,CANARY_LIFECYCLE_BLOCKED,"volume"); + AddHarnessCase(cases,"stop",HARNESS_STOP_INVALID,CANARY_LIFECYCLE_BLOCKED,"stop_risk"); + AddHarnessCase(cases,"stop mode unsupported",HARNESS_STOP_MODE_UNSUPPORTED,CANARY_LIFECYCLE_BLOCKED,"stop_risk"); + AddHarnessCase(cases,"stop tick unavailable",HARNESS_STOP_TICK_UNAVAILABLE,CANARY_LIFECYCLE_BLOCKED,"stop_risk"); + AddHarnessCase(cases,"stop tick misaligned",HARNESS_STOP_TICK_MISALIGNED,CANARY_LIFECYCLE_BLOCKED,"stop_risk"); + AddHarnessCase(cases,"risk",HARNESS_RISK_EXCESSIVE,CANARY_LIFECYCLE_BLOCKED,"stop_risk"); + AddHarnessCase(cases,"tighter daily loss equality",HARNESS_TIGHT_DAILY_LOSS_EQUALITY,CANARY_LIFECYCLE_BLOCKED,"daily_loss_count"); + AddHarnessCase(cases,"hard daily loss equality",HARNESS_HARD_DAILY_LOSS_EQUALITY,CANARY_LIFECYCLE_BLOCKED,"daily_loss_count"); + AddHarnessCase(cases,"spread",HARNESS_SPREAD_EXCESSIVE,CANARY_LIFECYCLE_BLOCKED,"spread_deviation"); + AddHarnessCase(cases,"session",HARNESS_SESSION_CLOSED,CANARY_LIFECYCLE_BLOCKED,"session"); + AddHarnessCase(cases,"news",HARNESS_NEWS_BLOCKED,CANARY_LIFECYCLE_BLOCKED,"news"); + AddHarnessCase(cases,"exposure",HARNESS_EXPOSURE_PRESENT,CANARY_LIFECYCLE_BLOCKED,"exposure"); + AddHarnessCase(cases,"loaded duplicate state",HARNESS_DUPLICATE_OBSERVATION,CANARY_LIFECYCLE_BLOCKED,"freshness"); + AddHarnessCase(cases,"malformed processed state",HARNESS_MALFORMED_PROCESSED_STATE,CANARY_LIFECYCLE_BLOCKED,"freshness"); + AddHarnessCase(cases,"log failure",HARNESS_LOG_FAILURE,CANARY_LIFECYCLE_BLOCKED,"log_preflight"); + AddHarnessCase(cases,"reconciliation",HARNESS_RECONCILIATION_MISSING,CANARY_LIFECYCLE_BLOCKED,"reconciliation"); + + HarnessReconciliationCase reconciliation_cases[]; + AddHarnessReconciliationCase(reconciliation_cases,"rejected request",HARNESS_RECONCILE_REJECTED,CANARY_LIFECYCLE_ORDER_REJECTED); + AddHarnessReconciliationCase(reconciliation_cases,"unknown result",HARNESS_RECONCILE_UNKNOWN,CANARY_LIFECYCLE_RECONCILIATION_REQUIRED); + AddHarnessReconciliationCase(reconciliation_cases,"partial fill",HARNESS_RECONCILE_PARTIAL,CANARY_LIFECYCLE_RECONCILIATION_REQUIRED); + AddHarnessReconciliationCase(reconciliation_cases,"filled with stop",HARNESS_RECONCILE_FILLED_WITH_STOP,CANARY_LIFECYCLE_FILLED_PROTECTED); + AddHarnessReconciliationCase(reconciliation_cases,"filled without stop",HARNESS_RECONCILE_FILLED_WITHOUT_STOP,CANARY_LIFECYCLE_EMERGENCY_CLOSE); + AddHarnessReconciliationCase(reconciliation_cases,"stopped observation",HARNESS_RECONCILE_STOPPED,CANARY_LIFECYCLE_STOPPED); + AddHarnessReconciliationCase(reconciliation_cases,"same direction durable no-op",HARNESS_RECONCILE_SAME_DIRECTION,CANARY_LIFECYCLE_FILLED_PROTECTED); + AddHarnessReconciliationCase(reconciliation_cases,"opposite signal close",HARNESS_RECONCILE_OPPOSITE_DIRECTION,CANARY_LIFECYCLE_CLOSE_REQUESTING); + AddHarnessReconciliationCase(reconciliation_cases,"four losing positions",HARNESS_RECONCILE_FOUR_LOSSES,CANARY_LIFECYCLE_PAUSED); + AddHarnessReconciliationCase(reconciliation_cases,"server day reset",HARNESS_RECONCILE_SERVER_DAY_RESET,CANARY_LIFECYCLE_READY); + AddHarnessReconciliationCase(reconciliation_cases,"restart with protected position",HARNESS_RECONCILE_RESTART_PROTECTED,CANARY_LIFECYCLE_FILLED_PROTECTED); + AddHarnessReconciliationCase(reconciliation_cases,"restart with foreign exposure",HARNESS_RECONCILE_RESTART_FOREIGN,CANARY_LIFECYCLE_BLOCKED); + AddHarnessReconciliationCase(reconciliation_cases,"mixed magic unprotected position blocks close",HARNESS_RECONCILE_MIXED_UNPROTECTED,CANARY_LIFECYCLE_BLOCKED); + AddHarnessReconciliationCase(reconciliation_cases,"multiple positions block close",HARNESS_RECONCILE_MULTIPLE_POSITIONS,CANARY_LIFECYCLE_RECONCILIATION_REQUIRED); + AddHarnessReconciliationCase(reconciliation_cases,"protection error pauses after emergency closure",HARNESS_RECONCILE_PROTECTION_PAUSE_AFTER_CLOSE,CANARY_LIFECYCLE_PAUSED); + + int failures=0; + for(int index=0;index/XAUUSD/processed_observations.csv` in Common Files, with exact header `schema_version,decision_id,observation_id,attempted_at`. A missing file is an empty pre-attempt state. Any present unreadable, malformed, duplicated, noncanonical, or identity-inconsistent record blocks readiness. Task 6 only reads this file; the later gateway/reconciliation tasks own durable attempt writes. + +If the ten-second terminal timer cannot start, initialization returns `INIT_FAILED` after attempting a safe blocked status projection. The EA never silently falls back to tick-only scheduling. diff --git a/ui/src/api/research.ts b/ui/src/api/research.ts new file mode 100644 index 000000000..e0765b2bd --- /dev/null +++ b/ui/src/api/research.ts @@ -0,0 +1,164 @@ +import { fetchJson } from './client' + +export type EvidenceTone = 'muted' | 'red' | 'amber' +export type QualityTone = EvidenceTone | 'green' +export type BridgeState = 'awaiting_bridge' | 'stale' | 'unsafe_account' | 'disconnected' | 'ready' +export type Mt5LearningState = 'no_data' | 'learning' | 'blocked' | 'stale' +export type JmbDecisionState = 'no_decision' | 'shadow' | 'demo_blocked' | 'error' +export type JmbExecutionLifecycleState = + | 'disabled' | 'paused' | 'blocked' | 'ready' | 'order_requesting' + | 'order_rejected' | 'reconciliation_required' | 'filled_protected' + | 'close_requesting' | 'closed' | 'stopped' | 'emergency_close' | 'error' +export type JmbResearchExecutionState = JmbExecutionLifecycleState | 'demo_blocked' | 'missing' | 'malformed' | 'stale' +export type JmbExecutionRolloutStage = 'status_only' | 'hfm_canary' | 'ic_canary' | 'both_demo' + +export interface JmbExecutionStatusSummary { + state: JmbResearchExecutionState + label: string + detail: string + capturedAt: string | null + broker: 'hfmarkets' | 'icmarkets' + server: string | null + accountMode: 'demo' | null + symbol: 'XAUUSD' | 'EURUSD' + rolloutStage: JmbExecutionRolloutStage + executionEnabled: boolean + killSwitch: boolean + decisionId: string | null + observationId: string | null + latestEvent: { id: string; type: string; at: string; resultCode: string; detail: string } | null + stopProtectionConfirmed: boolean + position: { direction: 'buy' | 'sell'; volume: number; openPrice: number; stopLoss: number; id: string } | null + reconciliationState: string + dailyLossCount: number + dailyRealizedLoss: number + blockingGate: string | null + nextSafeAction: string +} + +export interface Mt5TradeLedgerSummary { + state: Mt5LearningState + label: string + detail: string + broker: string + symbol: string + accountMode: string | null + server: string | null + lastDealTime: string | null + lastUpdated: string | null + totalDeals: number + manualDeals: number + eaDeals: number + otherDeals: number + unknownDeals: number + netProfit: number +} + +export interface JmbDecisionSummary { + state: JmbDecisionState + label: string + detail: string + broker: string + symbol: string + lastUpdated: string | null + decision: null | { + decisionId: string + createdAt: string + strategyVersion: string + mode: string + direction: string + reasonCode: string + reasonDetail: string + spread: number | null + volume: number + stopLoss: number | null + gateResults: Array<{ gate: string; state: string; detail: string }> + } +} + +export interface ResearchInstrument { + broker: string + symbol: string + bridgeSymbol?: string + label: string + export: { + available: boolean + files: number + firstFile: string | null + lastFile: string | null + totalBytes: number + lastUpdated: string | null + } + quality: { + label: string + tone: QualityTone + inspectedFiles: number + likelyM1Files: number + fallbackFiles: number + badRows: number + duplicateRows: number + } + report: null | { + data: { daily_bars: number; first_day: string; last_day: string } + selected_on_training_sharpe: { lookback_days: number; sharpe: number | null; max_drawdown: number | null } + untouched_holdout: { total_return: number | null; sharpe: number | null; max_drawdown: number | null } + latest_observation?: { as_of: string; direction: 'uptrend' | 'downtrend' | 'flat'; lookback_return: number; lookback_days: number } + } + walkForward: null | { + method: { training_months: number; test_months: number } + windows: unknown[] + out_of_sample_aggregate: { total_return: number | null; sharpe: number | null; max_drawdown: number | null } + } + bridge: { + state: BridgeState + label: string + detail: string + broker: string + symbol: string + server: string | null + capturedAt: string | null + lastUpdated: string | null + bid: number | null + ask: number | null + spread: number | null + openPositions: number | null + openOrders: number | null + } + learning: Mt5TradeLedgerSummary + decision: JmbDecisionSummary + execution: JmbExecutionStatusSummary + evidence: { label: string; tone: EvidenceTone; score: number } +} + +export interface ResearchDashboard { + asOf: string + mode: 'research_only' + tradingEnabled: boolean + summary: { exportRoot: string; tradeLedgerRoot: string; decisionRoot: string; executionRoot: string; instrumentsWithData: number; completedBaselines: number; completedWalkForwards: number; readyDemoBridges: number; learningInstruments: number; shadowDecisions: number; validatedInstruments: number; hfmReady: boolean; experimentRuns: number } + stages: Array<{ key: string; label: string; state: 'complete' | 'waiting' | 'next' | 'blocked'; detail: string }> + instruments: ResearchInstrument[] + experiments: Array<{ + id: string + created_at: string + broker: string + symbol: string + data: { first_eligible_day: string; last_day: string; daily_bars: number; effective_train_start: string } + method: { training_months: number; test_months: number; drawdown_review_alert: number } + scenarios: Array<{ + id: string + lookback_set: string + lookbacks: number[] + one_way_cost_bps: number + unseen_windows: number + out_of_sample: { total_return: number | null; sharpe: number | null; max_drawdown: number | null; win_rate: number | null } + review_flags: string[] + }> + warning: string + }> + news: Array<{ time: string; title: string; source: string | null; link: string | null }> + disclaimer: string +} + +export const researchApi = { + get: () => fetchJson('/api/research'), +} diff --git a/ui/src/components/research/Mt5ExecutionStatusCard.tsx b/ui/src/components/research/Mt5ExecutionStatusCard.tsx new file mode 100644 index 000000000..3a32719dc --- /dev/null +++ b/ui/src/components/research/Mt5ExecutionStatusCard.tsx @@ -0,0 +1,85 @@ +import { Radio, ShieldCheck, ShieldX, TriangleAlert } from 'lucide-react' +import type { JmbExecutionStatusSummary, JmbResearchExecutionState } from '../../api/research' + +function readable(value: string): string { + return value.replaceAll('_', ' ') +} + +function stateTone(state: JmbResearchExecutionState): string { + if (state === 'filled_protected') return 'border-green/30 bg-green/10 text-green' + if (state === 'ready') return 'border-accent/30 bg-accent/10 text-accent' + if (state === 'disabled' || state === 'demo_blocked' || state === 'missing') return 'border-border bg-bg-tertiary text-text-muted' + if (state === 'paused' || state === 'stale') return 'border-yellow-500/30 bg-yellow-500/10 text-yellow-300' + return 'border-red/30 bg-red/10 text-red' +} + +function price(value: number): string { + return value.toFixed(value >= 100 ? 2 : 5) +} + +export function Mt5ExecutionStatusCard({ execution }: { execution: JmbExecutionStatusSummary }) { + const hasProtectedExposure = execution.position && execution.stopProtectionConfirmed + + return ( +
+
+
+
+ + {execution.label} + +
+ +
+

{execution.detail}

+
+ {readable(execution.rolloutStage).toUpperCase()} + + EXECUTION {execution.executionEnabled ? 'ON' : 'OFF'} + + KILL {execution.killSwitch ? 'ON' : 'OFF'} + {execution.server ? <>{execution.server} : null} +
+ +
+
+
Latest event
+
+ {execution.latestEvent ? `${readable(execution.latestEvent.type)} · ${execution.latestEvent.resultCode}` : 'none recorded'} +
+
+
+
Stop protection
+
+ {execution.stopProtectionConfirmed ?
+
+
+
EA exposure
+
+ {execution.position + ? `${execution.position.direction.toUpperCase()} ${execution.position.volume} @ ${price(execution.position.openPrice)} · SL ${price(execution.position.stopLoss)}` + : 'none'} +
+
+
+
Broker-day loss
+
{execution.dailyLossCount} losing / {execution.dailyRealizedLoss.toFixed(2)}
+
+
+ +
+

+ {execution.blockingGate ?

+

Next: {execution.nextSafeAction}

+
+
+
+ ) +} diff --git a/ui/src/components/research/__tests__/Mt5ExecutionStatusCard.spec.tsx b/ui/src/components/research/__tests__/Mt5ExecutionStatusCard.spec.tsx new file mode 100644 index 000000000..bd89b0115 --- /dev/null +++ b/ui/src/components/research/__tests__/Mt5ExecutionStatusCard.spec.tsx @@ -0,0 +1,82 @@ +import { cleanup, render, screen } from '@testing-library/react' +import { afterEach, describe, expect, it } from 'vitest' +import type { JmbExecutionStatusSummary } from '../../../api/research' +import { Mt5ExecutionStatusCard } from '../Mt5ExecutionStatusCard' + +function execution(overrides: Partial = {}): JmbExecutionStatusSummary { + return { + state: 'filled_protected', + label: 'untrusted upstream label', + detail: 'Broker confirms protected demo exposure.', + capturedAt: '2026-07-13T09:10:00.000Z', + broker: 'hfmarkets', + server: 'HFMarketsGlobal-Demo4', + accountMode: 'demo', + symbol: 'XAUUSD', + rolloutStage: 'hfm_canary', + executionEnabled: true, + killSwitch: false, + decisionId: 'decision-1', + observationId: 'observation-1', + latestEvent: { + id: 'event-1', + type: 'fill_confirmed', + at: '2026-07-13T09:09:58.000Z', + resultCode: '10009', + detail: 'Request completed', + }, + stopProtectionConfirmed: true, + position: { direction: 'buy', volume: 0.01, openPrice: 3334.25, stopLoss: 3324.25, id: 'position-1' }, + reconciliationState: 'reconciled', + dailyLossCount: 1, + dailyRealizedLoss: -8.75, + blockingGate: null, + nextSafeAction: 'Monitor broker-side protection.', + ...overrides, + } +} + +afterEach(cleanup) + +describe('Mt5ExecutionStatusCard', () => { + it.each([ + ['filled_protected', 'DEMO ENABLED'], + ['paused', 'PAUSED'], + ['reconciliation_required', 'RECONCILIATION REQUIRED'], + ['demo_blocked', 'DEMO BLOCKED'], + ['missing', 'STATUS MISSING'], + ['malformed', 'STATUS MALFORMED'], + ['stale', 'STATUS STALE'], + ] as const)('maps %s to the approved operational label', (state, label) => { + render() + + expect(screen.getByRole('region', { name: 'MT5 demo execution status' }).textContent).toContain(label) + expect(screen.queryByText('untrusted upstream label')).toBeNull() + }) + + it('shows the operational demo evidence compactly', () => { + render() + + const card = screen.getByRole('region', { name: 'MT5 demo execution status' }) + expect(card.textContent).toContain('DEMO ONLY') + expect(card.textContent).toContain('HFM CANARY') + expect(card.textContent).toContain('EXECUTION ON') + expect(card.textContent).toContain('KILL OFF') + expect(card.textContent).toContain('HFMarketsGlobal-Demo4') + expect(card.textContent).toContain('fill confirmed') + expect(card.textContent).toContain('10009') + expect(card.textContent).toContain('STOP CONFIRMED') + expect(card.textContent).toContain('BUY 0.01') + expect(card.textContent).toContain('1 losing / -8.75') + expect(card.textContent).toContain('Blocked by: daily loss limit') + expect(card.textContent).toContain('Next: Monitor broker-side protection.') + }) + + it('contains no account identity or interactive execution control', () => { + const { container } = render() + + expect(container.textContent).not.toMatch(/account.?login/i) + expect(container.querySelector('button, input, select, textarea, [role="switch"]')).toBeNull() + expect(container.textContent).not.toMatch(/place order|buy now|sell now|enable trading/i) + }) +}) diff --git a/ui/src/demo/handlers/index.ts b/ui/src/demo/handlers/index.ts index 10d85710f..a5f947700 100644 --- a/ui/src/demo/handlers/index.ts +++ b/ui/src/demo/handlers/index.ts @@ -14,6 +14,7 @@ import { agentStatusHandlers } from './agentStatus' import { newsListHandlers } from './newsList' import { devMiscHandlers } from './devMisc' import { headlessHandlers } from './headless' +import { researchHandlers } from './research' import { preferencesHandlers } from './preferences' import { inquiryHandlers } from './inquiries' import { catchAllHandlers } from './catchAll' @@ -38,6 +39,7 @@ export const handlers = [ ...newsListHandlers, ...devMiscHandlers, ...headlessHandlers, + ...researchHandlers, ...preferencesHandlers, ...inquiryHandlers, ...catchAllHandlers, diff --git a/ui/src/demo/handlers/research.ts b/ui/src/demo/handlers/research.ts new file mode 100644 index 000000000..45dc84a73 --- /dev/null +++ b/ui/src/demo/handlers/research.ts @@ -0,0 +1,139 @@ +import { http, HttpResponse } from 'msw' +import type { JmbExecutionStatusSummary, ResearchDashboard, ResearchInstrument } from '../../api/research' + +const DEMO_NOW = '2026-07-13T09:10:00.000Z' + +function execution( + broker: 'hfmarkets' | 'icmarkets', + symbol: 'XAUUSD' | 'EURUSD', + state: JmbExecutionStatusSummary['state'], +): JmbExecutionStatusSummary { + const blocked = symbol === 'EURUSD' + const protectedFill = state === 'filled_protected' + return { + state, + label: blocked ? 'DEMO BLOCKED' : protectedFill ? 'DEMO ENABLED' : 'CANARY READY', + detail: blocked + ? 'EURUSD remains shadow-only and is not eligible for Plan 3 demo execution.' + : protectedFill + ? 'Broker confirms the EA-owned demo position and protective stop.' + : 'The broker-local EA reports that canary gates are ready for operator review.', + capturedAt: DEMO_NOW, + broker, + server: broker === 'hfmarkets' ? 'HFMarketsGlobal-Demo4' : 'ICMarketsSC-Demo', + accountMode: 'demo', + symbol, + rolloutStage: broker === 'hfmarkets' ? 'hfm_canary' : 'status_only', + executionEnabled: protectedFill, + killSwitch: !protectedFill, + decisionId: protectedFill ? 'demo-decision-7f31' : null, + observationId: protectedFill ? 'demo-observation-2026-07-12' : null, + latestEvent: protectedFill ? { + id: 'demo-event-a92c', + type: 'fill_confirmed', + at: DEMO_NOW, + resultCode: '10009', + detail: 'Demo request completed; stop confirmed.', + } : null, + stopProtectionConfirmed: protectedFill, + position: protectedFill ? { direction: 'sell', volume: 0.01, openPrice: 3334.25, stopLoss: 3344.25, id: 'demo-position-4d18' } : null, + reconciliationState: protectedFill ? 'reconciled' : 'not_required', + dailyLossCount: 1, + dailyRealizedLoss: -8.75, + blockingGate: blocked ? 'instrument_allowlist' : null, + nextSafeAction: blocked ? 'Continue read-only shadow observation; no execution action is available.' : 'Monitor broker-side protection and reconciliation.', + } +} + +function instrument( + broker: 'hfmarkets' | 'icmarkets', + symbol: 'XAUUSDb' | 'EURUSDb' | 'XAUUSD' | 'EURUSD', + localSymbol: 'XAUUSD' | 'EURUSD', + executionStatus: JmbExecutionStatusSummary, +): ResearchInstrument { + return { + broker, + symbol, + ...(symbol !== localSymbol ? { bridgeSymbol: localSymbol } : {}), + label: localSymbol === 'XAUUSD' ? 'Gold / USD' : 'Euro / USD', + export: { available: false, files: 0, firstFile: null, lastFile: null, totalBytes: 0, lastUpdated: null }, + quality: { label: 'Awaiting export', tone: 'muted', inspectedFiles: 0, likelyM1Files: 0, fallbackFiles: 0, badRows: 0, duplicateRows: 0 }, + report: null, + walkForward: null, + bridge: { + state: 'ready', + label: 'Demo bridge connected', + detail: 'Read-only telemetry is current. This bridge has no order-submission code.', + broker, + symbol: localSymbol, + server: broker === 'hfmarkets' ? 'HFMarketsGlobal-Demo4' : 'ICMarketsSC-Demo', + capturedAt: DEMO_NOW, + lastUpdated: DEMO_NOW, + bid: localSymbol === 'XAUUSD' ? 3334.2 : 1.14267, + ask: localSymbol === 'XAUUSD' ? 3334.3 : 1.14284, + spread: localSymbol === 'XAUUSD' ? 0.1 : 0.00017, + openPositions: executionStatus.position ? 1 : 0, + openOrders: 0, + }, + learning: { + state: 'no_data', label: 'Awaiting trade history', detail: 'No demo trade history is included in this static preview.', + broker, symbol: localSymbol, accountMode: 'demo', server: executionStatus.server, lastDealTime: null, lastUpdated: null, + totalDeals: 0, manualDeals: 0, eaDeals: 0, otherDeals: 0, unknownDeals: 0, netProfit: 0, + }, + decision: { + state: localSymbol === 'EURUSD' ? 'demo_blocked' : 'shadow', + label: localSymbol === 'EURUSD' ? 'Demo blocked by gates' : 'Shadow decision logged', + detail: 'Static demo decision evidence for the read-only Research Desk preview.', + broker, + symbol: localSymbol, + lastUpdated: DEMO_NOW, + decision: null, + }, + execution: executionStatus, + evidence: { label: 'Waiting for baseline', tone: 'muted', score: 0 }, + } +} + +const dashboard: ResearchDashboard = { + asOf: DEMO_NOW, + mode: 'research_only', + tradingEnabled: false, + summary: { + exportRoot: 'Local MT5 demo history', + tradeLedgerRoot: 'Local MT5 demo trade ledger', + decisionRoot: 'Local JMB decision log', + executionRoot: 'Local MT5 broker execution status', + instrumentsWithData: 0, + completedBaselines: 0, + completedWalkForwards: 0, + readyDemoBridges: 4, + learningInstruments: 0, + shadowDecisions: 4, + validatedInstruments: 0, + hfmReady: false, + experimentRuns: 0, + }, + stages: [ + { key: 'data', label: 'Broker data inspected', state: 'waiting', detail: 'Static preview: local export evidence is not included.' }, + { key: 'baseline', label: 'Baseline trend study', state: 'waiting', detail: 'Holdout results remain separate from training.' }, + { key: 'walkforward', label: 'Rolling walk-forward', state: 'waiting', detail: 'Required before a candidate can advance.' }, + { key: 'costs', label: 'Broker cost model', state: 'next', detail: 'Broker-local costs require operator review.' }, + { key: 'bridge', label: 'MT5 demo bridge', state: 'complete', detail: 'Static read-only demo telemetry is shown.' }, + { key: 'learning', label: 'Trade-history learning', state: 'waiting', detail: 'No trade history is included in this preview.' }, + { key: 'shadow', label: 'JMB shadow decisions', state: 'complete', detail: 'EURUSD remains shadow-only.' }, + { key: 'demo', label: 'Demo forward test', state: 'next', detail: 'Broker-local demo lifecycle status is visible; Research Desk remains read-only.' }, + ], + instruments: [ + instrument('hfmarkets', 'XAUUSDb', 'XAUUSD', execution('hfmarkets', 'XAUUSD', 'filled_protected')), + instrument('hfmarkets', 'EURUSDb', 'EURUSD', execution('hfmarkets', 'EURUSD', 'demo_blocked')), + instrument('icmarkets', 'XAUUSD', 'XAUUSD', execution('icmarkets', 'XAUUSD', 'ready')), + instrument('icmarkets', 'EURUSD', 'EURUSD', execution('icmarkets', 'EURUSD', 'demo_blocked')), + ], + experiments: [], + news: [], + disclaimer: 'Demo status is operational evidence, not live approval, a probability of profit, or a trade recommendation.', +} + +export const researchHandlers = [ + http.get('/api/research', () => HttpResponse.json(dashboard)), +] diff --git a/ui/src/pages/ResearchDashboardPage.tsx b/ui/src/pages/ResearchDashboardPage.tsx new file mode 100644 index 000000000..250724789 --- /dev/null +++ b/ui/src/pages/ResearchDashboardPage.tsx @@ -0,0 +1,350 @@ +import { useCallback, useEffect, useMemo, useState } from 'react' +import { Activity, ArrowDownRight, ArrowUpRight, BarChart3, Database, FlaskConical, Radio, RefreshCw, ShieldCheck } from 'lucide-react' +import { PageHeader } from '../components/PageHeader' +import { Mt5ExecutionStatusCard } from '../components/research/Mt5ExecutionStatusCard' +import { researchApi, type ResearchDashboard, type ResearchInstrument } from '../api/research' + +function percentage(value: number | null | undefined): string { + return value == null ? 'n/a' : `${(value * 100).toFixed(1)}%` +} + +function compactBytes(value: number): string { + if (value < 1024 * 1024) return `${Math.round(value / 1024)} KB` + return `${(value / (1024 * 1024)).toFixed(1)} MB` +} + +function dateTime(value: string | null): string { + if (!value) return 'not exported' + return new Date(value).toLocaleString(undefined, { month: 'short', day: 'numeric', hour: '2-digit', minute: '2-digit' }) +} + +function bridgePrice(value: number | null): string { + if (value == null) return 'n/a' + return value.toFixed(value >= 100 ? 2 : 5) +} + +function toneClass(tone: ResearchInstrument['evidence']['tone']): string { + if (tone === 'red') return 'border-red/30 bg-red/10 text-red' + if (tone === 'amber') return 'border-yellow-500/30 bg-yellow-500/10 text-yellow-300' + return 'border-border bg-bg-tertiary text-text-muted' +} + +function qualityToneClass(tone: ResearchInstrument['quality']['tone']): string { + if (tone === 'green') return 'border-green/30 bg-green/10 text-green' + return toneClass(tone) +} + +function bridgeToneClass(state: ResearchInstrument['bridge']['state']): string { + if (state === 'ready') return 'border-green/30 bg-green/10 text-green' + if (state === 'awaiting_bridge') return 'border-border bg-bg-tertiary text-text-muted' + return 'border-red/30 bg-red/10 text-red' +} + +function learningTone(state: string): ResearchInstrument['quality']['tone'] { + if (state === 'learning') return 'green' + if (state === 'blocked') return 'red' + if (state === 'stale') return 'amber' + return 'muted' +} + +function decisionTone(state: string): ResearchInstrument['quality']['tone'] { + if (state === 'shadow') return 'green' + if (state === 'demo_blocked') return 'amber' + if (state === 'error') return 'red' + return 'muted' +} + +function stageClass(state: ResearchDashboard['stages'][number]['state']): string { + if (state === 'complete') return 'bg-green text-bg' + if (state === 'next') return 'bg-accent text-bg' + if (state === 'blocked') return 'bg-red text-white' + return 'bg-border text-text-muted' +} + +function InstrumentStudy({ instrument }: { instrument: ResearchInstrument }) { + const report = instrument.report + const walkForward = instrument.walkForward + const bridge = instrument.bridge + const observation = report?.latest_observation + const isUp = observation?.direction === 'uptrend' + const isDown = observation?.direction === 'downtrend' + + return ( +
+
+
+
{instrument.broker}
+

{instrument.symbol}

+

{instrument.label}

+
+ + {instrument.evidence.label} + +
+ +
+ {observation ? ( +
+
+
Latest completed trend
+
+ {isUp ? : isDown ? : } + {observation.direction} +
+
+
+
{percentage(observation.lookback_return)}
+
{observation.lookback_days}d lookback · {observation.as_of}
+
+
+ ) : ( +
No completed baseline is available for this broker yet.
+ )} + +
+ + 0 ? 'green' : 'red'} /> + + +
+ +
+
+ Walk-forward evidence + {walkForward ? `${walkForward.windows.length} unseen windows` : 'not run'} +
+ {walkForward ? ( +
+ {walkForward.method.training_months}m train / {walkForward.method.test_months}m test + {percentage(walkForward.out_of_sample_aggregate.total_return)} + Sharpe {walkForward.out_of_sample_aggregate.sharpe?.toFixed(2) ?? 'n/a'} +
+ ) :

Sequential unseen-period testing has not been recorded yet.

} +
+ +
+
+ Data quality + {instrument.quality.label} +
+

+ {instrument.quality.inspectedFiles} monthly files inspected · {instrument.quality.likelyM1Files} likely M1 + {instrument.quality.fallbackFiles > 0 ? ` · ${instrument.quality.fallbackFiles} fallback-resolution files excluded from M1-only work` : ''} + {instrument.quality.badRows > 0 || instrument.quality.duplicateRows > 0 ? ` · ${instrument.quality.badRows} malformed, ${instrument.quality.duplicateRows} duplicates` : ''} +

+
+ +
+
+ MT5 demo bridge + {bridge.label} +
+ {bridge.state === 'ready' ? ( +

+ {bridge.server} · terminal symbol {bridge.symbol} · bid {bridgePrice(bridge.bid)} / ask {bridgePrice(bridge.ask)} · spread {bridgePrice(bridge.spread)} · {bridge.openPositions ?? 0} positions +

+ ) :

{bridge.detail}

} +
+ +
+
+ Trade-history learning + {instrument.learning.label} +
+

{instrument.learning.detail}

+

+ Deals: {instrument.learning.totalDeals} • Manual: {instrument.learning.manualDeals} • EA: {instrument.learning.eaDeals} • Net: {instrument.learning.netProfit.toFixed(2)} +

+
+ +
+
+ JMB decision learning + {instrument.decision.label} +
+

{instrument.decision.detail}

+ {instrument.decision.decision ? ( +

+ {instrument.decision.decision.mode} / {instrument.decision.decision.direction} / {instrument.decision.decision.reasonCode} +

+ ) : null} +
+ + + +
+ {instrument.export.available ? `${instrument.export.files} monthly files · ${compactBytes(instrument.export.totalBytes)}` : 'No export found'} + {dateTime(instrument.export.lastUpdated)} +
+
+
+ ) +} + +function Metric({ label, value, emphasis }: { label: string; value: string; emphasis?: 'green' | 'red' }) { + return ( +
+
{label}
+
{value}
+
+ ) +} + +function ExperimentLedger({ experiments }: { experiments: ResearchDashboard['experiments'] }) { + if (experiments.length === 0) { + return ( +
+
Fixed-matrix experiment ledger
+

No experiment run has been recorded yet. Runs compare the same declared parameter and cost cases over sequential unseen windows; they do not search for a target win rate.

+
+ ) + } + + return ( +
+
+
+
Fixed-matrix experiment ledger
+

Historical research only · return bars are out-of-sample · red drawdown alerts require review, not another optimisation cycle.

+
+ {experiments.length} logged run{experiments.length === 1 ? '' : 's'} +
+
+ {experiments.map((run) => { + const maximum = Math.max(0.01, ...run.scenarios.map((scenario) => Math.abs(scenario.out_of_sample.total_return ?? 0))) + return ( +
+
+
{run.broker} · {run.symbol} {run.data.first_eligible_day} → {run.data.last_day}
+
{dateTime(run.created_at)} · {run.data.daily_bars} daily bars
+
+
+ {run.scenarios.map((scenario) => { + const totalReturn = scenario.out_of_sample.total_return ?? 0 + const drawdown = scenario.out_of_sample.max_drawdown + const width = `${Math.max(3, Math.min(100, Math.abs(totalReturn) / maximum * 100))}%` + const positive = totalReturn >= 0 + const drawdownExceeded = drawdown != null && Math.abs(drawdown) > run.method.drawdown_review_alert + return ( +
+
{scenario.id}
+
+
+ {percentage(totalReturn)} · Sharpe {scenario.out_of_sample.sharpe?.toFixed(2) ?? 'n/a'} · {scenario.unseen_windows} windows +
+
DD {percentage(drawdown)}
+
+ ) + })} +
+

{run.warning}

+
+ ) + })} +
+
+ ) +} + +export function ResearchDashboardPage() { + const [dashboard, setDashboard] = useState(null) + const [error, setError] = useState(null) + const [loading, setLoading] = useState(true) + + const load = useCallback(async () => { + try { + setError(null) + const next = await researchApi.get() + setDashboard(next) + } catch (err) { + setError(err instanceof Error ? err.message : 'Could not load research status') + } finally { + setLoading(false) + } + }, []) + + useEffect(() => { + void load() + const timer = window.setInterval(() => void load(), 30_000) + return () => window.clearInterval(timer) + }, [load]) + + const hfm = useMemo(() => dashboard?.instruments.filter((instrument) => instrument.broker === 'hfmarkets') ?? [], [dashboard]) + const icMarkets = useMemo(() => dashboard?.instruments.filter((instrument) => instrument.broker === 'icmarkets') ?? [], [dashboard]) + + return ( +
+ void load()} className="btn-secondary-sm inline-flex items-center gap-1.5" title="Refresh research data"> Refresh} + /> + +
+ {error ?
{error}
: null} + {!dashboard && loading ?
Loading local research artifacts…
: null} + {dashboard ? ( +
+
+
+
Research-only mode
+

Evidence before automation.

+

This read-only desk monitors what has been exported, tested, rejected, or broker-confirmed. Any demo execution authority remains inside the locally configured MT5 EA; this page cannot send or manage orders.

+
+
+ + + + +
+
+ + + +
+

Validation path

+
+ {dashboard.stages.map((stage, index) => ( +
+
{index + 1}{stage.label}
+

{stage.detail}

+
+ ))} +
+
+ +
+

HFM studies

+
{hfm.map((instrument) => )}
+
+ +
+

IC Markets comparison

+
{icMarkets.map((instrument) => )}
+
+ +
+
+
Evidence meaning
+

{dashboard.disclaimer}

+

Trade-history learning imports manual and demo outcomes for review. It is not approval for live trading and it cannot submit orders.

+

Shadow decisions are JMB learning records only. They are not live-trading approval and they do not submit orders.

+

Demo execution status is operational evidence only. Demo performance is not live approval and does not imply future profit.

+
Data source: local MT5 exports. Last analysed candles, not streaming broker quotes.
+
+
+

Recent news

last 24 hours
+ {dashboard.news.length === 0 ?
No collected news yet. Configure feeds in Settings → News Sources.
: } +
+
+
+ ) : null} +
+
+ ) +} + +function SummaryStat({ label, value, red }: { label: string; value: string; red?: boolean }) { + return
{label}
{value}
+}