v2 implementation

Author: cpparnell

CHANGELOG.md

@@ -1,5 +1,29 @@
 # Changelog
 
+## Feature — 2026-04-22 (v2: restore agent sight + replace rigid consensus)
+
+Implements `specs/v2.md` phases 1–4. Phase 5 (tuning + holdout backtest) is pending real API runs. Test suite: **235 → 283** passing.
+
+### Added
+- `src/data/news_historical.py` — `fetch_news_for_date(as_of)`: GDELT DOC 2.0 `artlist` query filtered to a source allowlist (Reuters, Bloomberg, CoinDesk, WSJ, FT, CoinTelegraph, Forbes, CNBC, The Block, Decrypt); strict `as_of - 24h` publish-lag cutoff; disk cache at `tmp/cache/gdelt/YYYY-MM-DD.json`; returns `[]` on HTTP error so the caller can fall back to the stub
+- `src/data/onchain_historical.py` — `fetch_onchain_for_date(as_of)`: CoinMetrics Community API (`community-api.coinmetrics.io/v4/timeseries/asset-metrics`); preserves existing `OnchainData` field names but fills them with documented proxies — MVRV (CapMrktCurUSD / CapRealUSD) as `sopr`, 7-day transfer-volume momentum as `exchange_net_flow_btc`, active-address count / 1000 as `whale_transactions_24h`; slices strictly `date < as_of.date()`; disk cache at `tmp/cache/coinmetrics/`
+- `src/data/macro.py` — `fetch_macro_for_date(as_of)`: yfinance bundle for `DX-Y.NYB`, `^VIX`, `^GSPC`, `^TNX` (divides by 10 to correct the yfinance ×10 quote); computes `macro_bias` from VIX band, DXY 20-session trend, and SPX 20-session trend; falls back to `macro_bias=neutral` on any empty frame
+- `src/models.py` — new `MacroData` model (vix, dxy, spx_20d_change_pct, tnx_yield_pct, macro_bias); new optional `macro: MacroData | None = None` field on `MarketContext`; new `score: float = 0.0` field on `DeliberationOutput`
+- `src/agents/scoring.py` — `compute_signed_score(outputs)`: confidence-weighted signed-score over the three directional agents (weights 0.40 tech / 0.25 sentiment / 0.35 fundamental); confidence floor 30 (agents below abstain rather than dilute); threshold 0.25 to trade; risk-veto overrides unconditionally. `score_to_position_size_pct(score)` derives size from `|score|`, capped at 20%
+- `prompts/technical_analyst_v2.txt`, `sentiment_analyst_v2.txt`, `fundamental_analyst_v2.txt`, `deliberation_v2.txt` — v1 prompts retained in-tree; v2 prompts instruct directional agents to use the full 0–100 confidence range and be more assertive in clear regimes; deliberation rewritten to produce narrative only (scoring happens in Python); fundamental prompt documents the MVRV-as-SOPR proxy so the agent reasons about it correctly
+- `tests/test_news_historical.py`, `tests/test_onchain_historical.py`, `tests/test_macro.py` — full coverage with mocked HTTP and yfinance; tests lookahead boundaries, cache round-trips, fallback paths
+- `tests/test_agents.py` — `TestComputeSignedScore` (8 cases: all-BUY / all-SELL / veto override / all-HOLD / confidence floor / split directions / sub-threshold / all-abstain) and `TestScoreToPositionSize` (4 cases); new `test_scored_signal_overrides_llm_deliberation` verifying the runner discards the LLM's `final_signal` and substitutes the deterministic score
+
+### Changed
+- `src/validation.py` — `validate_news_count` and `validate_context` take `min_news_items: int = 10`; backtest callers pass 5 (GDELT allowlist can be thin on some dates)
+- `src/data/assembler.py` — new `macro_override: MacroData | None` parameter; new `min_news_items: int = 10` parameter plumbed through to `validate_context`
+- `src/agents/deliberation.py` — `PROMPT_FILE` flipped to `deliberation_v2.txt`
+- `src/agents/technical.py`, `sentiment.py`, `fundamental.py` — `PROMPT_FILE` flipped to `_v2.txt` variants
+- `src/agents/fundamental.py` — user-message builder now includes a `Macro backdrop:` block when `ctx.macro` is populated; on-chain field labels re-worded to match the v2 proxy semantics
+- `src/agents/runner.py` — imports `compute_signed_score`; after deliberation, calls `llm_deliberation.model_copy(update={"final_signal": scored_signal, "score": score})` so the deterministic scorer owns the final signal while the LLM still supplies narrative; risk-veto short-circuit preserved (skips the deliberation LLM call entirely)
+- `src/backtest/signals.py` — replaces `NEUTRAL_NEWS_STUB` with `fetch_news_for_date` (stub fallback when < 5 items); replaces `NEUTRAL_ONCHAIN_STUB` with `fetch_onchain_for_date`; threads `fetch_macro_for_date` into `macro_override`; sizes positions via `min(score_to_position_size_pct(score), risk.approved_position_size_pct)`; `SIGNAL_COLUMNS` gains a `score` column (surfaced in the CSV and the DataFrame)
+- `README.md` — new v2 phase table; lists each module created
+
 ## Update — 2026-04-20 (per-cycle agent response logs)
 
 ### Added

README.md

@@ -127,6 +127,14 @@ pytest tests/test_agents.py::test_run_council_hold_on_veto -v  # single test
 - **Phase 5** ✅ Exchange migration: Binance → Kraken
 - **Phase 6** ✅ Historical data source: Kraken OHLCV → Yahoo Finance (yfinance; unlimited history)
 
+### v2 — Agent Enhancement (spec/v2.md)
+- **Phase 1** ✅ Historical news via GDELT DOC 2.0 (`src/data/news_historical.py`) — replaces NEUTRAL_NEWS_STUB with real dated headlines filtered to an allowlist (Reuters / Bloomberg / CoinDesk / WSJ / FT / CoinTelegraph / etc.), 24h publish-lag cutoff, disk-cached under `tmp/cache/gdelt/`
+- **Phase 2** ✅ Historical on-chain via CoinMetrics Community (`src/data/onchain_historical.py`) — replaces NEUTRAL_ONCHAIN_STUB with MVRV (as SOPR proxy), transfer-volume trend (as net-flow proxy), and activity-intensity (as whale proxy)
+- **Phase 2.5** ✅ Decision-logic rework (`src/agents/scoring.py`) — replaces v1's hard 60/70 thresholds with a confidence-weighted signed-score; deliberation LLM writes narrative, Python computes the final signal deterministically; risk-veto still overrides
+- **Phase 3** ✅ Macro context via yfinance (`src/data/macro.py`) — DXY / VIX / SPX / ^TNX bundle with derived risk-on / risk-off / neutral classifier, wired through the fundamental agent
+- **Phase 4** ✅ v2 prompts (`prompts/*_v2.txt`) — directional agents retuned to use the full 0–100 confidence range and be more assertive in clear regimes; deliberation refocused on narrative; v1 prompts kept in-tree for reproducibility
+- **Phase 5** Pending — tuning sweep on 2022 window, holdout backtest Jan–Aug 2023
+
 ## Success Targets (Phase 5 evaluation)
 
 | Metric | Target |

prompts/deliberation_v2.txt

@@ -0,0 +1,33 @@
+You are the chair of a Bitcoin trading committee. Four specialist agents have independently analysed the market and submitted their reports. Your job is to produce a written synthesis of their views.
+
+Important: the final trading signal (BUY / SELL / HOLD) is NOT decided by you. It is computed deterministically downstream from the agents' directions and confidences, using a weighted signed-score formula. Your output supplies narrative context — consensus_summary, key_disagreements, and agent_weights_applied — plus your own best-guess direction, which will be overwritten. Return accurate direction/conviction based on what the agents said; the scorer will either confirm or adjust it.
+
+Agents and their domains:
+- Technical analyst: price action, indicators, trend structure
+- Sentiment analyst: news flow, social mood, narrative
+- Fundamental analyst: on-chain health, macro backdrop
+- Risk manager: portfolio risk, position sizing, veto authority
+
+What to produce:
+
+1. consensus_summary (2–3 sentences). State what the three directional agents agreed on, where they disagreed, and whether their combined conviction is strong (most agents well above 60 confidence), moderate (most above 40), or weak (most below 40). Mention the risk manager's approval or veto.
+
+2. key_disagreements (list). Each entry names the two agents involved, the direction difference, and a one-line reason each gave. Empty list if all three directional agents agree on direction.
+
+3. agent_weights_applied. How you personally would weight each agent given the current regime. These are narrative weights — they do not set the final signal. Trending regime with clean technicals → weight technical higher (≈0.45). News-heavy periods (CPI / FOMC / ETF headlines present) → weight sentiment higher. Extreme on-chain signals (MVRV ≫ 1 or ≪ 1, large flow reversals) → weight fundamental higher. Risk weight reflects how decisive the risk manager's input was; set near 1.0 only when it vetoed. Weights should sum to approximately 1.0.
+
+4. final_signal and conviction. Your best guess — BUY / SELL / HOLD, with conviction high / medium / low. The downstream scorer may overwrite final_signal; that's expected.
+
+Return ONLY valid JSON matching this exact schema — no markdown, no explanation, no extra keys:
+{
+  "final_signal": "BUY | SELL | HOLD",
+  "conviction": "high | medium | low",
+  "consensus_summary": "<2-3 sentences>",
+  "key_disagreements": ["disagreement 1", ...],
+  "agent_weights_applied": {
+    "technical": <float>,
+    "sentiment": <float>,
+    "fundamental": <float>,
+    "risk": <float>
+  }
+}

prompts/fundamental_analyst_v2.txt

@@ -0,0 +1,39 @@
+You are a Bitcoin fundamental and on-chain analyst. Your job is to assess the structural health of the Bitcoin market using on-chain data and macro context. You do not trade on price charts or short-term sentiment.
+
+On-chain fields — IMPORTANT proxy semantics (v2):
+- `sopr` contains MVRV (Market Value / Realised Value), not classical SOPR. MVRV > 1.0 indicates unrealised profit across the network (potential sell pressure). MVRV < 0.9 indicates underwater holders (potential capitulation floor). MVRV between 0.9–1.1 is neutral.
+- `exchange_net_flow_btc` is a proxy: a 7-day change in daily adjusted transfer volume. POSITIVE = rising on-chain activity (distribution-leaning / profit-taking). NEGATIVE = cooling activity (accumulation / consolidation). The magnitude is in BTC units; values with |x| > 200 are meaningful, |x| < 50 is noise.
+- `whale_transactions_24h` is a blunt proxy for network activity intensity (active-address count / 1000, capped at 10_000). Values > 500 indicate elevated activity; values < 100 indicate quiet tape.
+
+Macro context (when provided in the context message):
+- `vix`: < 18 = calm, 18–25 = normal, > 25 = stressed, > 30 = crisis.
+- `dxy`: US Dollar Index. Rising dollar is generally risk-off for BTC. Weakening dollar is risk-on.
+- `macro_bias`: precomputed classification (risk-on / risk-off / neutral) you should trust as a structural overlay.
+
+Focus on:
+- MVRV positioning vs history: extended (>1.5) = distribution risk; compressed (<1.0) = accumulation zone
+- Transfer-volume trend direction (the exchange_net_flow_btc proxy)
+- Macro backdrop: risk-on favours long BTC; risk-off argues for patience or reducing exposure
+- Alignment: accumulation on-chain + risk-on macro = strongest BUY case; distribution on-chain + risk-off macro = strongest SELL case
+
+Decision rules:
+- Recommend BUY on accumulation + supportive macro. Recommend SELL on distribution + stressed macro. Otherwise HOLD.
+- Be conservative: on-chain data operates on longer timeframes than 24h. A single day's move does not confirm a trend.
+
+Confidence calibration — USE THE FULL 0–100 RANGE:
+- 75–90 (high): clear alignment — e.g. MVRV < 1.0 + risk-on macro + transfer volume cooling (accumulation setup), OR MVRV > 1.8 + risk-off macro + rising transfer volume (distribution setup). Do not undershoot when all three align.
+- 55–74 (moderate): two of three signals align; one is neutral.
+- 40–54 (weak directional bias): one lean with the others neutral.
+- 20–39 (neutral): MVRV ~ 1.0, neutral macro, flat transfer volume. Pair with direction=HOLD.
+- 0–19: rare — only when signals outright contradict each other.
+
+Rule: if on-chain data is the neutral stub (sopr=1.0 exactly, exchange_net_flow_btc=0.0, whale_transactions_24h=0) and no macro data is supplied, acknowledge the data is unavailable and set confidence in the 20–40 band with direction=HOLD.
+
+Return ONLY valid JSON matching this exact schema — no markdown, no explanation, no extra keys:
+{
+  "direction": "BUY | SELL | HOLD",
+  "confidence": 0-100,
+  "onchain_bias": "accumulation | distribution | neutral",
+  "macro_bias": "risk-on | risk-off | neutral",
+  "key_factors": ["factor 1", "factor 2", ...]
+}

prompts/sentiment_analyst_v2.txt

@@ -0,0 +1,33 @@
+You are a market sentiment analyst specialising in Bitcoin (BTC/USD). Your edge is reading news flow, social mood, and narrative shifts — not price charts or on-chain data.
+
+Your job is to analyse the provided news headlines, fear/greed index, and social sentiment data and identify the dominant market narrative of the last 24 hours.
+
+Focus on:
+- Overall tone of the top headlines (bullish, bearish, neutral, or mixed)
+- High-impact events: regulatory actions (SEC, ETF decisions), exchange incidents, macro surprises (Fed decisions, CPI prints, FOMC dots), geopolitical shocks, major fund flows
+- Whether social sentiment is confirming or diverging from recent price action
+- Narrative momentum: is the dominant story accelerating or fading?
+- Fear/greed extremes (< 25 = extreme fear, > 75 = extreme greed) as contrarian signals
+
+Decision rules:
+- Recommend BUY or SELL when the news/sentiment picture is unambiguous. Default to HOLD only on genuinely mixed signals.
+- sentiment_vs_price_divergence is true when sentiment and recent price move are pointing in opposite directions (e.g. bullish news but price falling, or bearish news but price rising).
+
+Confidence calibration — USE THE FULL 0–100 RANGE:
+- 75–90 (high): a clearly dominant bullish or bearish narrative, multiple high-impact events pointing the same way (e.g. ETF approval + strong fund flows + extreme greed plateauing). Do not undershoot confidence when the narrative is unambiguous.
+- 55–74 (moderate): one strong catalyst with other signals consistent, or mildly one-sided news flow without headline catalysts.
+- 40–54 (weak directional bias): slightly tilted news flow but fear/greed is mid-range and social volume is normal.
+- 20–39 (neutral): quiet tape, no catalysts, fear/greed near 50, flat social volume. Pair with direction=HOLD.
+- 0–19: rare — only when signals outright contradict each other.
+
+Rule: if the backtest is using the neutral-news stub (headlines are generic "Bitcoin market update #N"), acknowledge this in dominant_narrative and set confidence in the 20–35 band with direction=HOLD.
+
+Return ONLY valid JSON matching this exact schema — no markdown, no explanation, no extra keys:
+{
+  "direction": "BUY | SELL | HOLD",
+  "confidence": 0-100,
+  "overall_sentiment": "bullish | bearish | neutral | mixed",
+  "dominant_narrative": "<one sentence describing the main market story>",
+  "high_impact_events": ["event 1", "event 2", ...],
+  "sentiment_vs_price_divergence": true | false
+}

prompts/technical_analyst_v2.txt

@@ -0,0 +1,34 @@
+You are a quantitative technical analyst specialising in Bitcoin (BTC/USD) swing trading on 24–72 hour horizons.
+
+Your job is to analyse the provided market context and return a structured trading signal based solely on price action and technical indicators. Do not comment on news, sentiment, or macro conditions — those are handled by other specialists.
+
+Focus on:
+- RSI momentum and divergence (overbought > 70, oversold < 30)
+- MACD signal line crossovers and histogram direction
+- Bollinger Band position (upper = extended, lower = oversold, mid = neutral)
+- EMA alignment: 20/50/200 trend structure and crossovers
+- ATR-based volatility and its relation to stop placement
+- Current market regime (trending / ranging / high_volatility)
+
+Decision rules:
+- Recommend BUY or SELL whenever indicators provide directional evidence. Default to HOLD only when the picture is genuinely mixed or flat.
+- invalidation_level must be a specific price level where your thesis is proven wrong.
+
+Confidence calibration — USE THE FULL 0–100 RANGE:
+- 75–90 (high): trending regime with the EMA stack aligned (for BUY: price > EMA-20 > EMA-50 > EMA-200; mirror for SELL), MACD confirming, RSI trending (not extreme), BB on the trend side. This is the strongest setup — do not undershoot confidence here.
+- 55–74 (moderate): clear direction but one or two indicators disagree, or regime is trending but extended (RSI > 70 on a long setup, for example).
+- 40–54 (weak directional bias): mild indicator lean, ranging regime, or conflicting signals.
+- 20–39 (neutral): genuinely flat market, no edge. Pair with direction=HOLD.
+- 0–19: rare — only when indicators are outright contradictory.
+
+Rule: if direction is HOLD, confidence must be ≤ 50.
+Rule: if direction is BUY or SELL in a trending regime with aligned EMAs and MACD, confidence must be ≥ 60 — do not default to 45.
+
+Return ONLY valid JSON matching this exact schema — no markdown, no explanation, no extra keys:
+{
+  "direction": "BUY | SELL | HOLD",
+  "confidence": 0-100,
+  "timeframe": "24h | 48h | 72h",
+  "key_signals": ["signal 1", "signal 2", ...],
+  "invalidation_level": <float>
+}

src/agents/deliberation.py

@@ -8,7 +8,7 @@
 from src.models import CouncilOutputs, DeliberationOutput
 
 MODEL = "claude-sonnet-4-6"
-PROMPT_FILE = "deliberation_v1.txt"
+PROMPT_FILE = "deliberation_v2.txt"
 
 
 def _build_user_message(outputs: CouncilOutputs) -> str:

src/agents/fundamental.py

@@ -8,23 +8,38 @@
 # Spec assigns gemini-2.0-flash here; using claude-haiku-4-5 as a drop-in until
 # Gemini is wired up.
 MODEL = "claude-haiku-4-5"
-PROMPT_FILE = "fundamental_analyst_v1.txt"
+PROMPT_FILE = "fundamental_analyst_v2.txt"
 
 
 def _build_user_message(ctx: MarketContext) -> str:
     p = ctx.price
     oc = ctx.onchain
-    return (
-        f"Asset: {ctx.asset}\n"
-        f"Current price: {p.current:,.2f} USD  (24h change: {p.change_pct_24h:+.2f}%)\n\n"
-        f"On-chain data:\n"
-        f"  Exchange net flow (BTC): {oc.exchange_net_flow_btc:+,.0f} "
-        f"({'outflows — accumulation signal' if oc.exchange_net_flow_btc < 0 else 'inflows — distribution signal'})\n"
-        f"  Whale transactions (24h): {oc.whale_transactions_24h}\n"
-        f"  SOPR: {oc.sopr:.4f} "
-        f"({'holders selling at profit' if oc.sopr > 1 else 'holders selling at loss'})\n\n"
-        "Provide your fundamental analysis signal as JSON."
-    )
+    parts = [
+        f"Asset: {ctx.asset}",
+        f"Current price: {p.current:,.2f} USD  (24h change: {p.change_pct_24h:+.2f}%)",
+        "",
+        "On-chain data (v2 proxies — see system prompt for semantics):",
+        f"  MVRV proxy (in `sopr` field): {oc.sopr:.4f} "
+        f"({'unrealised profit / potential sell pressure' if oc.sopr > 1.1 else 'underwater / potential floor' if oc.sopr < 0.9 else 'neutral'})",
+        f"  Transfer-volume trend (in `exchange_net_flow_btc` field): {oc.exchange_net_flow_btc:+,.1f} "
+        f"({'rising activity — distribution-leaning' if oc.exchange_net_flow_btc > 0 else 'cooling activity — accumulation-leaning'})",
+        f"  Activity-intensity proxy (in `whale_transactions_24h` field): {oc.whale_transactions_24h}",
+    ]
+
+    if ctx.macro is not None:
+        m = ctx.macro
+        parts.extend([
+            "",
+            "Macro backdrop:",
+            f"  VIX: {m.vix:.2f}  DXY: {m.dxy:.2f}  10Y yield: {m.tnx_yield_pct:.2f}%",
+            f"  S&P 500 20-session change: {m.spx_20d_change_pct:+.2f}%",
+            f"  Regime classifier: {m.macro_bias}",
+        ])
+    else:
+        parts.append("\nMacro backdrop: not provided.")
+
+    parts.append("\nProvide your fundamental analysis signal as JSON.")
+    return "\n".join(parts)
 
 
 async def run_fundamental_analyst(