Skip to content

Commit 09142b1

Browse files
quantDIYquantDIY
committed
chore: save work on execution-node and databento (excluding large data files)
1 parent c638a7f commit 09142b1

28 files changed

Lines changed: 1859 additions & 100 deletions

‎.agent/skills/universal_runner/SKILL.md‎

Lines changed: 85 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,85 @@
1+
---
2+
name: universal_runner
3+
description: Instructions for operating the QuanuX Universal Runner (quanux_runner) for Live Trading and Offline Backtesting.
4+
---
5+
6+
# Universal Runner Operator Guide
7+
8+
The `quanux_runner` is the core C++ execution engine for QuanuX. Use this skill to execute strategies, run backtests, or deploy to live markets.
9+
10+
## 1. Capabilities
11+
* **Offline Replay**: High-performance replay of Databento `.dbn` files (MBO/L3 supported).
12+
* **Rithmic Replay**: Historical tick replay via RApi+ (Trade info).
13+
* **Live Trading**: Direct connection to Databento Live API via `libdatabento` or Rithmic RApi+.
14+
* **Zero-Copy Execution**: Loads strategies as shared objects (`.dylib`/`.so`) for maximum speed.
15+
16+
## 2. Usage
17+
18+
### Location
19+
Binary is located at: `execution-node/cpp/build/quanux_runner`
20+
21+
### CLI Arguments
22+
| Argument | Required | Description |
23+
| :--- | :--- | :--- |
24+
| `--strategy <path>` | **Yes** | Path to the compiled strategy shared object (`.dylib`/`.so`). |
25+
| `--file <path>` | No* | Path to input `.dbn` file (Offline Mode). *Required for Backtesting.* |
26+
| `--key <key>` | No* | Databento API Key. *Required for Live Mode.* |
27+
| `--symbol <sym>` | No | Symbol to subscribe to (default: `ES`). |
28+
| `--dataset <ds>` | No | Dataset (default: `GLBX.MDP3`). |
29+
| `--ruser <u>` | No* | Rithmic Username. *Required for Rithmic.* |
30+
| `--rpass <p>` | No* | Rithmic Password. *Required for Rithmic.* |
31+
| `--rexch <e>` | No | Rithmic Exchange (e.g., `CME`). *Required for Rithmic.* |
32+
| `--start <ts>` | No | Start Timestamp (Unix Epoch). *Required for Replay.* |
33+
| `--end <ts>` | No | End Timestamp (Unix Epoch). *Required for Replay.* |
34+
35+
### Examples
36+
37+
#### A. Run Offline Backtest (Recommended)
38+
Use this to verify strategy logic using historical data. Zstd decompression is handled automatically if the runner was linked correctly.
39+
```bash
40+
./execution-node/cpp/build/quanux_runner \
41+
--strategy execution-node/cpp/build/demo_strategy.dylib \
42+
--file data/raw/glbx-mdp3-20260122.mbo.dbn
43+
```
44+
45+
#### B. Run Live Mode
46+
Requires `DATABENTO_API_KEY` (env var or arg).
47+
```bash
48+
export DATABENTO_API_KEY=db-xc...
49+
./execution-node/cpp/build/quanux_runner \
50+
--strategy execution-node/cpp/build/demo_strategy.dylib \
51+
--symbol ESH6
52+
```
53+
54+
#### C. Run Rithmic Replay
55+
Replay historical trades from Rithmic servers.
56+
```bash
57+
./execution-node/cpp/build/quanux_runner \
58+
--strategy execution-node/cpp/build/demo_strategy.dylib \
59+
--ruser "DEMO_USER" \
60+
--rpass "DEMO_PASS" \
61+
--symbol "ES.M24" \
62+
--rexch "CME" \
63+
--start 1715000000 \
64+
--end 1715003600
65+
```
66+
67+
## 3. Workflow Checklist
68+
69+
Before running, ensure:
70+
1. **Strategy Built**: The strategy must be compiled freshly.
71+
* `cmake --build execution-node/cpp/build --target <strategy_name>`
72+
2. **Runner Built**: The host binary must be up to date.
73+
* `cmake --build execution-node/cpp/build --target quanux_runner`
74+
3. **Data Ready**: For offline mode, ensure the `.dbn` file exists and is readable.
75+
76+
## 4. Troubleshooting
77+
78+
* **Exit Code 139 (Segfault)**:
79+
* Usually means the strategy is crashing (e.g., indicator out of bounds).
80+
* **Fix**: Add `std::cout` logs to `on_market_data` in the strategy and rebuild to isolate the crash.
81+
* **"Symbol not found"**:
82+
* The strategy `.dylib` is missing symbols or ABI mismatch.
83+
* **Fix**: Ensure `extern "C" { Strategy* create_strategy() ... }` is strictly defined in the strategy source.
84+
* **"Invalid argument 'key'"**:
85+
* API Key is missing or invalid in Live Mode.

‎.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,3 +110,7 @@ client/desktop/tauri-app/src-tauri/src/*.bak.*
110110
# ---- Project Management ----
111111
# project_status.md
112112
# docs/technical_whitepaper.md
113+
114+
# Data
115+
*.dbn
116+
*.zst

‎COMMIT_MSG.txt‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
1-
docs(control): Document Python Control Plane and Bridge Architecture
1+
docs(runtime): Document Supervisor Command Dispatcher & Activation
22

3-
Finalized documentation for the "Best of Both Worlds" architecture.
3+
Documented the Phase 2 Activation of the C++ Supervisor.
44

5-
## Updates
6-
* **server/control/SKILL.md**: New Agent Skill defining `quanux.runtime` as a remote control client, not a wrapper.
7-
* **server/control/README.md**: Developer guide explaining the C++ Speed / Python Agility bridge.
8-
* **server/agents.md**: Updated Central Brain to include the Control Plane skill.
5+
## Documentation Updates
6+
* **server/runtime/SKILL.md**: Added "Command Processing (The Brain)" section detailing the JSON protocol (`sys.cmd.spawn`) and process spawning capability.
7+
* **server/agents.md**: Updated Central Brain to reflect the Supervisor's new ability to *Act* (spawn/kill) rather than just manage the bus.
8+
* **walkthrough.md**: Documented the activation of the Action Loop.
99

10-
This completes the architectural pivot to a C++ Nucleus with a Python Nervous System interface.
10+
This confirms the system is no longer just "Architecture" but "Active Logic".

‎build_debug.txt‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
[100%] Building CXX object CMakeFiles/quanux_runner.dir/src/runner.cpp.o
2+
/Users/Duncan/Antigravity/QuanuX/QuanuX/execution-node/cpp/src/runner.cpp:103:29: error: use of undeclared identifier 'RithmicFeed'
3+
103 | feed = std::make_unique<RithmicFeed>(r_user, r_pass, symbol, r_exch);
4+
| ^
5+
1 error generated.
6+
make[3]: *** [CMakeFiles/quanux_runner.dir/src/runner.cpp.o] Error 1
7+
make[2]: *** [CMakeFiles/quanux_runner.dir/all] Error 2
8+
make[1]: *** [CMakeFiles/quanux_runner.dir/rule] Error 2
9+
make: *** [quanux_runner] Error 2

‎build_errors_2.txt‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
[100%] Building CXX object CMakeFiles/quanux_runner.dir/src/runner.cpp.o
2+
In file included from /Users/Duncan/Antigravity/QuanuX/QuanuX/execution-node/cpp/src/runner.cpp:11:
3+
/Users/Duncan/Antigravity/QuanuX/QuanuX/execution-node/cpp/src/rithmic_feed.hpp:105:15: error: 'Trade' marked 'override' but does not override any member functions
4+
105 | virtual int Trade(RApi::TradeInfo *pInfo, void *pContext,
5+
| ^
6+
/Users/Duncan/Antigravity/QuanuX/QuanuX/execution-node/cpp/src/rithmic_feed.hpp:21:12: error: no member named 'pLogger' in 'RApi::REngineParams'
7+
21 | params.pLogger = &logger_; // Use internal logger if needed
8+
| ~~~~~~ ^
9+
/Users/Duncan/Antigravity/QuanuX/QuanuX/execution-node/cpp/src/rithmic_feed.hpp:38:17: error: no member named 'sUser' in 'RApi::LoginParams'
10+
38 | loginParams.sUser = to_tsNCharcb(user_);
11+
| ~~~~~~~~~~~ ^
12+
/Users/Duncan/Antigravity/QuanuX/QuanuX/execution-node/cpp/src/rithmic_feed.hpp:39:17: error: no member named 'sPassword' in 'RApi::LoginParams'
13+
39 | loginParams.sPassword = to_tsNCharcb(pass_);
14+
| ~~~~~~~~~~~ ^
15+
/Users/Duncan/Antigravity/QuanuX/QuanuX/execution-node/cpp/src/rithmic_feed.hpp:40:17: error: no member named 'sSystemName' in 'RApi::LoginParams'
16+
40 | loginParams.sSystemName =
17+
| ~~~~~~~~~~~ ^
18+
/Users/Duncan/Antigravity/QuanuX/QuanuX/execution-node/cpp/src/rithmic_feed.hpp:42:17: error: no member named 'sInfraType' in 'RApi::LoginParams'
19+
42 | loginParams.sInfraType = to_tsNCharcb("ticker plant");
20+
| ~~~~~~~~~~~ ^
21+
/Users/Duncan/Antigravity/QuanuX/QuanuX/execution-node/cpp/src/rithmic_feed.hpp:43:17: error: no member named 'sServerName' in 'RApi::LoginParams'
22+
43 | loginParams.sServerName = to_tsNCharcb(
23+
| ~~~~~~~~~~~ ^
24+
/Users/Duncan/Antigravity/QuanuX/QuanuX/execution-node/cpp/src/rithmic_feed.hpp:54:36: error: call to implicitly-deleted copy constructor of 'std::atomic<bool>'
25+
54 | cv_.wait(lock, [this] { return connected_; });
26+
| ^~~~~~~~~~
27+
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__atomic/atomic.h:230:17: note: copy constructor of 'atomic<bool>' is implicitly deleted because base class '__atomic_base<bool>' has a deleted copy constructor
28+
230 | struct atomic : public __atomic_base<_Tp> {
29+
| ^
30+
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__atomic/atomic.h:139:3: note: '__atomic_base' has been explicitly marked deleted here
31+
139 | __atomic_base(const __atomic_base&) = delete;
32+
| ^
33+
8 errors generated.
34+
make[3]: *** [CMakeFiles/quanux_runner.dir/src/runner.cpp.o] Error 1
35+
make[2]: *** [CMakeFiles/quanux_runner.dir/all] Error 2
36+
make[1]: *** [CMakeFiles/quanux_runner.dir/rule] Error 2
37+
make: *** [quanux_runner] Error 2

‎build_log.txt‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
[100%] Building CXX object CMakeFiles/quanux_runner.dir/src/runner.cpp.o
2+
/Users/Duncan/Antigravity/QuanuX/QuanuX/execution-node/cpp/src/runner.cpp:103:29: error: use of undeclared identifier 'RithmicFeed'
3+
103 | feed = std::make_unique<RithmicFeed>(r_user, r_pass, symbol, r_exch);
4+
| ^
5+
1 error generated.
6+
make[3]: *** [CMakeFiles/quanux_runner.dir/src/runner.cpp.o] Error 1
7+
make[2]: *** [CMakeFiles/quanux_runner.dir/all] Error 2
8+
make[1]: *** [CMakeFiles/quanux_runner.dir/rule] Error 2
9+
make: *** [quanux_runner] Error 2

‎build_log_final.txt‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
Building CXX object CMakeFiles/quanux_runner.dir/src/runner.cpp.o
2+
In file included from /Users/Duncan/Antigravity/QuanuX/QuanuX/execution-node/cpp/src/runner.cpp:11:
3+
/Users/Duncan/Antigravity/QuanuX/QuanuX/execution-node/cpp/src/rithmic_feed.hpp:117:15: error: 'Trade' marked 'override' but does not override any member functions
4+
117 | virtual int Trade(RApi::TradeInfo *pInfo, void *pContext,
5+
| ^
6+
/Users/Duncan/Antigravity/QuanuX/QuanuX/execution-node/cpp/src/rithmic_feed.hpp:23:12: error: no member named 'pLogger' in 'RApi::REngineParams'
7+
23 | params.pLogger = &logger_; // Use internal logger if needed
8+
| ~~~~~~ ^
9+
/Users/Duncan/Antigravity/QuanuX/QuanuX/execution-node/cpp/src/rithmic_feed.hpp:40:17: error: no member named 'sUser' in 'RApi::LoginParams'
10+
40 | loginParams.sUser = to_tsNCharcb(user_);
11+
| ~~~~~~~~~~~ ^
12+
/Users/Duncan/Antigravity/QuanuX/QuanuX/execution-node/cpp/src/rithmic_feed.hpp:41:17: error: no member named 'sPassword' in 'RApi::LoginParams'
13+
41 | loginParams.sPassword = to_tsNCharcb(pass_);
14+
| ~~~~~~~~~~~ ^
15+
/Users/Duncan/Antigravity/QuanuX/QuanuX/execution-node/cpp/src/rithmic_feed.hpp:42:17: error: no member named 'sSystemName' in 'RApi::LoginParams'
16+
42 | loginParams.sSystemName =
17+
| ~~~~~~~~~~~ ^
18+
/Users/Duncan/Antigravity/QuanuX/QuanuX/execution-node/cpp/src/rithmic_feed.hpp:44:17: error: no member named 'sInfraType' in 'RApi::LoginParams'
19+
44 | loginParams.sInfraType = to_tsNCharcb("ticker plant");
20+
| ~~~~~~~~~~~ ^
21+
/Users/Duncan/Antigravity/QuanuX/QuanuX/execution-node/cpp/src/rithmic_feed.hpp:45:17: error: no member named 'sServerName' in 'RApi::LoginParams'
22+
45 | loginParams.sServerName = to_tsNCharcb(
23+
| ~~~~~~~~~~~ ^
24+
/Users/Duncan/Antigravity/QuanuX/QuanuX/execution-node/cpp/src/rithmic_feed.hpp:56:36: error: call to implicitly-deleted copy constructor of 'std::atomic<bool>'
25+
56 | cv_.wait(lock, [this] { return connected_; });
26+
| ^~~~~~~~~~
27+
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__atomic/atomic.h:230:17: note: copy constructor of 'atomic<bool>' is implicitly deleted because base class '__atomic_base<bool>' has a deleted copy constructor
28+
230 | struct atomic : public __atomic_base<_Tp> {
29+
| ^
30+
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/__atomic/atomic.h:139:3: note: '__atomic_base' has been explicitly marked deleted here
31+
139 | __atomic_base(const __atomic_base&) = delete;
32+
| ^
33+
8 errors generated.
34+
make[3]: *** [CMakeFiles/quanux_runner.dir/src/runner.cpp.o] Error 1
35+
make[2]: *** [CMakeFiles/quanux_runner.dir/all] Error 2
36+
make[1]: *** [CMakeFiles/quanux_runner.dir/rule] Error 2
37+
make: *** [quanux_runner] Error 2

‎docs/drafts/strategy_development_ideas.md‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
# Skill: C++ Strategy Development
2+
3+
**Goal**: Create a high-performance trading strategy compatible with the QuanuX Universal Runner.
4+
5+
## 1. The Strategy Template
6+
7+
Use the following boilerplate for any new strategy (`execution-node/cpp/strategies/my_strategy.cpp`):
8+
9+
```cpp
10+
#include "strategy_interface.h"
11+
#include <iostream>
12+
#include <memory>
13+
// Include your indicators
14+
#include "quanux/indicators/market_profile.hpp"
15+
16+
// 1. Definition
17+
class MyStrategy {
18+
// State goes here
19+
double vwap_ = 0.0;
20+
public:
21+
void init(const OrderService* service) {
22+
std::cout << "[MyStrat] Ready.\n";
23+
}
24+
25+
void on_market_data(const MarketUpdate* update) {
26+
if (!update->is_trade) return;
27+
28+
// Logic
29+
if (update->price > vwap_) {
30+
// ...
31+
}
32+
}
33+
};
34+
35+
// 2. ABI Wiring (Boilerplate)
36+
extern "C" {
37+
StrategyContext* my_create() { return (StrategyContext*)new MyStrategy(); }
38+
void my_destroy(StrategyContext* ctx) { delete (MyStrategy*)ctx; }
39+
void my_init(StrategyContext* ctx, const OrderService* s) { ((MyStrategy*)ctx)->init(s); }
40+
void my_md(StrategyContext* ctx, const MarketUpdate* u) { ((MyStrategy*)ctx)->on_market_data(u); }
41+
// ... Map other signals
42+
43+
Strategy* create_strategy() {
44+
static Strategy s = {
45+
.name = "MyStrategy",
46+
.create_context = my_create,
47+
.destroy_context = my_destroy,
48+
.on_init = my_init,
49+
.on_market_data = my_md,
50+
};
51+
return &s;
52+
}
53+
}
54+
```
55+
56+
## 2. CMake Integration
57+
58+
Add your strategy to `execution-node/cpp/CMakeLists.txt`:
59+
60+
```cmake
61+
add_library(my_strategy SHARED strategies/my_strategy.cpp)
62+
target_include_directories(my_strategy PRIVATE include)
63+
target_link_libraries(my_strategy PRIVATE quanux_indicators_lib) // Optional: Link indicators
64+
set_target_properties(my_strategy PROPERTIES PREFIX "") // Ensures my_strategy.so / .dylib
65+
```
66+
67+
## 3. Best Practices (HFT)
68+
69+
* **Avoid Heap Allocation**: `new` / `malloc` in `on_market_data` is forbidden. Pre-allocate in `init`.
70+
* **Fast Path**: `if (!is_trade) return;` should be the first line if you only trade ticks.
71+
* **Logging**: `std::cout` is slow. In production, use the `Logger` service (coming in Phase 4).
72+
* **Indicators**: Use `quanux_indicators` library which is designed for incremental updates (Zero-History needed).
73+
* Example: `MarketProfile<DenseStorage>` is O(1) for updates.
74+
75+
## 4. Testing
76+
77+
Always verify your strategy offline first!
78+
79+
1. **Download Sample Data**: Fetch a small `.dbn` file (MBO or Trades) from Databento.
80+
2. **Run Offline**:
81+
```bash
82+
./quanux_runner --strategy my_strategy.dylib --file sample.dbn
83+
```
84+
3. **Validate**: Check logs for "Initialized Strategy" and ensuring logic triggers as expected.

‎docs/research/UNIVERSAL_RUNNER_VISION.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# QuanuX Architecture: The Universal Runner (Phase 3)
2+
3+
## The "Eureka" Discovery
4+
We have built the **Brain** (Indicators) and the **Nervous System** (Runtime), but we lack the **Cardiovascular System** (Data Pump).
5+
6+
Currently, strategies are passive shared libraries (`.so`). They need a host to "pump" life (ticks) into them.
7+
8+
## The Solution: `quanux_runner`
9+
One optimized C++ binary to rule them all.
10+
11+
### 1. Architecture
12+
* **Role**: A "Strategy Container" optimized for <10µs latency.
13+
* **Mechanism**:
14+
1. **Loader**: `dlopen`s the strategy shared object.
15+
2. **Feed**: Connects to the data source (Databento Live, NATS Replay, or PCAP).
16+
3. **Hot Loop**: Pumps data directly into the strategy's `on_tick()` method.
17+
18+
```mermaid
19+
graph TD
20+
Data[Databento Live] --> |Kernel Bypass| Runner(quanux_runner)
21+
Runner --> |dlopen| Strat[Strategy.so]
22+
Strat --> |Signal| Runner
23+
Runner --> |Order| Exchange
24+
Runner --> |Audit| NATS[NATS JetStream]
25+
```
26+
27+
### 2. The Universal Interface
28+
Every strategy (C++ or Python Proxy) implements this specialized ABI:
29+
30+
```cpp
31+
// strategy_interface.h
32+
class Strategy {
33+
public:
34+
virtual void on_tick(const Tick& tick) = 0;
35+
virtual void on_order_update(const Order& order) = 0;
36+
};
37+
38+
// Exported Factory
39+
extern "C" Strategy* create_strategy();
40+
```
41+
42+
### 3. "Zero-Code" Backtesting
43+
Because the *Runner* abstracts the feed, the **Same Strategy Binary** can be backtested without recompilation.
44+
* **Live**: `./quanux_runner --strat=demo.so --feed=databento --live`
45+
* **Replay**: `./quanux_runner --strat=demo.so --feed=pcap --file=2023_data.pcap`
46+
47+
### 4. Implementation Plan
48+
1. **Feed Handler**: Integrate `databento` C++ client.
49+
2. **Plugin Loader**: Implement `dlopen` wrapper in `execution-node/cpp`.
50+
3. **The Runner**: Create the `main()` loop that bridges Feed -> Plugin.

0 commit comments

Comments
 (0)