Merge main into feat/clear-street-cpp-adapter

Author: quantDIY

.agent/skills/hft_engine/SKILL.md

@@ -0,0 +1,65 @@
+---
+name: HFT Stats Engine
+description: Usage and architecture of the QuanuX High-Frequency Trading Statistics Engine.
+---
+
+# HFT Stats Engine Skill
+
+## Overview
+The **QuanuX Stats Engine** is a C++20 microservice designed for sub-microsecond signal generation. It ingests binary market data, computes rolling statistics (Welford's algorithm), and emits trading signals via a lock-free queue.
+
+## Capabilities
+1.  **Binary Ingestion**: Consumes `MARKET.BIN` NATS subjects. Payload must be a 64-byte `quanux::MarketTick` struct.
+2.  **Persistence**: High-throughput storage to DuckDB via Appender API (no SQL overhead).
+3.  **Signaling**: Emits `Signal` structs to an internal SPSC queue when Z-Score > 2.0.
+4.  **Telemetry**: Logs correlation matrices to stdout every 10 seconds.
+
+## Architecture
+-   **Thread A (Producer)**:
+    -   Ingest NATS message (Zero-Copy cast).
+    -   Append to DuckDB.
+    -   Update Rolling Stats (RingBuffer).
+    -   Push to SPSC Queue (Lock-Free).
+-   **Thread B (Consumer)**:
+    -   Spin-wait on SPSC Queue using `_mm_pause()`.
+    -   Execute Strategy Logic (currently simulated).
+
+## Data Structure (`quanux::MarketTick`)
+Must be exactly **64 bytes** (Cache Line Aligned).
+```cpp
+struct alignas(64) MarketTick {
+    uint64_t local_rec_ts;       // 8 bytes
+    uint64_t exchange_ts;        // 8 bytes
+    double price;                // 8 bytes
+    uint32_t size;               // 4 bytes
+    uint32_t flags;              // 4 bytes
+    uint32_t instrument_id;      // 4 bytes
+    // Implicit Padding: 4 bytes here
+    uint64_t internal_arrival_ts;// 8 bytes
+    uint64_t processing_start_ts;// 8 bytes
+    uint8_t _pad[8];             // 8 bytes Explicit Padding
+};
+```
+
+## Performance Benchmarks (macOS M1/M2)
+-   **Min Latency**: 59 nanoseconds
+-   **Avg Latency**: ~250 microseconds (OS scheduling noise)
+-   **Throughput**: >3.2 Million msg/sec
+
+## Usage
+### Running the Engine
+```bash
+./quanux_stats
+```
+
+### Verifying
+Use the provided Python script to send compliant binary packets:
+```bash
+python3 verify_hft_engine.py
+```
+
+### Benchmarking
+Run the micro-benchmark to measure internal SPSC latency:
+```bash
+./benchmark_hft_engine
+```

.agent/skills/python_packaging/SKILL.md

@@ -0,0 +1,59 @@
+---
+name: python_packaging
+description: Guidelines for creating robust Python extension setup.py files that survive CI environments.
+---
+# Python Packaging in QuanuX
+
+When creating or modifying `setup.py` files for Python extensions (especially those using Cython, PyBind11, or Numpy), you must ensure they are robust against missing build-time dependencies.
+
+## The Problem
+CI/CD systems (like GitHub's Dependency Submission Action) often run in environments where build dependencies like `Cython` or `numpy` are NOT installed. If `setup.py` unconditionally imports them, the dependency graph generation fails.
+
+## The Solution: Robust Setup Pattern
+Always wrap build-time imports in `try-except` blocks. If they are missing, the script should still run successfully but yield an empty list of extensions.
+
+### Template
+
+```python
+import os
+from setuptools import setup, Extension
+
+# 1. Wrap build dependencies
+try:
+    from Cython.Build import cythonize
+    import numpy 
+except ImportError:
+    cythonize = None
+    numpy = None
+
+ext_modules = []
+
+# 2. Conditional Build Logic
+if cythonize and numpy:
+    # Define extensions here
+    extensions = [
+        Extension(
+            "my_extension",
+            sources=["src/my_extension.pyx"],
+            include_dirs=[numpy.get_include()],
+            # ...
+        )
+    ]
+    ext_modules = cythonize(extensions, language_level=3)
+else:
+    print("Build dependencies (Cython/Numpy) not found. Skipping extension build.")
+    # This print is important for debugging but allows the script to exit with code 0
+
+# 3. Setup Call
+setup(
+    name="my_package",
+    ext_modules=ext_modules, # Will be empty if deps missing
+    # ... standard metadata ...
+)
+```
+
+## Checklist
+- [ ] Import `Cython.Build.cythonize` inside `try-except ImportError`.
+- [ ] Import `numpy` inside `try-except ImportError` (if used).
+- [ ] Check for existence of required directories (e.g. `_deps`, `sdk`) before adding them to `include_dirs`.
+- [ ] Ensure `python setup.py --name` runs successfully in a bare environment.

QuanuX-Common/cpp/include/quanux/MarketTick.hpp

@@ -0,0 +1,34 @@
+#pragma once
+
+#include <cstdint>
+
+namespace quanux {
+
+/**
+ * @brief Cache-aligned Market Tick Structure for HFT
+ *
+ * Designed to fit exactly into a 64-byte cache line to prevent false sharing
+ * and optimize memory access patterns during high-frequency updates.
+ */
+struct alignas(64) MarketTick {
+  uint64_t local_rec_ts;  // 8 bytes: Nanoseconds since epoch (Packet receipt)
+  uint64_t exchange_ts;   // 8 bytes: Nanoseconds since epoch (Matching engine)
+  double price;           // 8 bytes: Execution price
+  uint32_t size;          // 4 bytes: Execution size
+  uint32_t flags;         // 4 bytes: Trade flags
+  uint32_t instrument_id; // 4 bytes: Internal instrument mapping
+
+  // Internal Latency Tracking
+  uint64_t internal_arrival_ts; // 8 bytes: Time tick entered process
+  uint64_t processing_start_ts; // 8 bytes: Time stats engine picked it up
+
+  // Padding to reach 64 bytes
+  // Offset is 56 bytes (including 4 bytes implicit padding after instrument_id)
+  // Remaining: 8 bytes
+  uint8_t _pad[8];
+};
+
+static_assert(sizeof(MarketTick) == 64, "MarketTick must be exactly 64 bytes");
+static_assert(alignof(MarketTick) == 64, "MarketTick must be 64-byte aligned");
+
+} // namespace quanux

QuanuX-Common/cpp/include/quanux/SPSCQueue.hpp

@@ -0,0 +1,64 @@
+#pragma once
+
+#include <atomic>
+#include <cstddef>
+#include <optional>
+#include <vector>
+
+namespace quanux {
+
+/**
+ * @brief Lock-Free Single-Producer Single-Consumer Queue
+ *
+ * Uses a ring buffer with atomic head/tail indices.
+ * Optimized for cache line separation to prevent false sharing between producer
+ * and consumer.
+ */
+template <typename T> class SPSCQueue {
+public:
+  explicit SPSCQueue(size_t capacity)
+      : capacity_(capacity), data_(capacity + 1) {}
+
+  /**
+   * @brief Push an item into the queue.
+   * @return true if successful, false if full.
+   */
+  bool push(const T &item) {
+    const size_t current_tail = tail_.load(std::memory_order_relaxed);
+    const size_t next_tail = (current_tail + 1) % data_.size();
+
+    if (next_tail != head_.load(std::memory_order_acquire)) {
+      data_[current_tail] = item;
+      tail_.store(next_tail, std::memory_order_release);
+      return true;
+    }
+    return false;
+  }
+
+  /**
+   * @brief Pop an item from the queue.
+   * @param item Reference to store the popped item.
+   * @return true if successful, false if empty.
+   */
+  bool pop(T &item) {
+    const size_t current_head = head_.load(std::memory_order_relaxed);
+
+    if (current_head == tail_.load(std::memory_order_acquire)) {
+      return false; // Empty
+    }
+
+    item = data_[current_head];
+    head_.store((current_head + 1) % data_.size(), std::memory_order_release);
+    return true;
+  }
+
+private:
+  size_t capacity_;
+  std::vector<T> data_;
+
+  // Align head and tail to separate cache lines to prevent false sharing
+  alignas(64) std::atomic<size_t> head_{0};
+  alignas(64) std::atomic<size_t> tail_{0};
+};
+
+} // namespace quanux

QuanuX-Statistics/cpp/CMakeLists.txt

@@ -59,6 +59,7 @@ include_directories(../../QuanuX-Common/cpp/include)
 # Core Library
 add_library(stats_core STATIC
     src/stats_engine.cpp
+    src/core/StatsEngineCore.cpp
     src/time_series_store.cpp
 )
 target_link_libraries(stats_core PUBLIC