Feature/expansion e2e docs

.gitignore

@@ -7,3 +7,6 @@ __pycache__
 **/recommended_trades.xlsx
 **/momentum_strategy.xlsx
 venv/
+output/
+reports/
+.pytest_cache/

EXPANSION_TODO.md

@@ -0,0 +1,64 @@
+# Expansion To-Do List (All Items)
+
+This file lists every expansion item and where it is implemented.
+
+## Backtesting & Robustness
+
+| # | Item | Status | Location |
+|---|------|--------|----------|
+| 1 | Backtest engine (returns-based) for all 3 strategies | Done | `src/backtest/engine.py`, `scripts/run_backtest.py` |
+| 2 | Walk-forward / out-of-sample testing | Done | `src/walk_forward/testing.py`, `scripts/run_backtest.py` |
+| 3 | Monte Carlo / bootstrap for strategy uncertainty | Done | `src/monte_carlo/bootstrap.py`, `scripts/run_backtest.py` |
+
+## Factor & Strategy Depth
+
+| # | Item | Status | Location |
+|---|------|--------|----------|
+| 4 | Multi-factor combined strategy (value + momentum + quality) | Done | `src/scoring/*.py`, `scripts/run_all_expansions.py`, config |
+| 5 | Sector-neutral or sector tilt constraints | Done | `src/portfolio/sector.py`, `scripts/run_all_expansions.py` |
+| 6 | Quality / low-vol overlay filter | Done | `src/scoring/quality.py`, `scripts/run_all_expansions.py` |
+| 7 | Alternative momentum definitions (52w high, risk-adj) | Done | `src/scoring/momentum.py`, `scripts/run_all_expansions.py` |
+| 8 | Alternative value metrics (FCF yield, E/P, div yield) | Done | `src/scoring/value.py` `compute_alternative_value_metrics`, `scripts/run_all_expansions.py` |
+
+## Portfolio Construction & Risk
+
+| # | Item | Status | Location |
+|---|------|--------|----------|
+| 9 | Risk parity or volatility-weighted sizing | Done | `src/portfolio/weighting.py` `weights_risk_parity`, `weights_volatility_scaled` |
+| 10 | Minimum variance or max Sharpe optimization | Done | `src/portfolio/weighting.py` `weights_min_variance`, `weights_max_sharpe` |
+| 11 | Position sizing by conviction (score-based) | Done | `src/portfolio/weighting.py` `weights_conviction`, config `conviction_power` |
+| 12 | Correlation and diversification report | Done | `src/reports/correlation.py`, `scripts/run_all_expansions.py` |
+
+## Data & Infrastructure
+
+| # | Item | Status | Location |
+|---|------|--------|----------|
+| 13 | Multiple data sources (e.g. yfinance fallback) | Done | `src/data/sources.py` `get_prices`, `get_prices_iex`, `get_prices_yfinance` |
+| 14 | Survivorship-bias-free universe handling | Done | `src/data/universe.py` `load_universe_survivorship_aware` |
+| 15 | Config-driven pipeline (YAML/JSON) | Done | `config/strategy_config.yaml`, `src/config_loader.py` |
+| 16 | Shared Python package (src/ modules) | Done | `src/` (config_loader, data, scoring, portfolio, backtest, walk_forward, monte_carlo, reports) |
+
+## Reporting & Monitoring
+
+| # | Item | Status | Location |
+|---|------|--------|----------|
+| 17 | Strategy dashboard (HTML/PDF report) | Done | `src/reports/dashboard.py`, `scripts/run_all_expansions.py` (HTML); PDF via weasyprint/pdfkit if installed |
+| 18 | Periodic rebalance script (trade list output) | Done | `scripts/rebalance.py` |
+
+## Theory & Education
+
+| # | Item | Status | Location |
+|---|------|--------|----------|
+| 19 | Factor primer document (theory/education) | Done | `docs/FACTOR_PRIMER.md` |
+| 20 | Comparison to ETFs (MTUM, IWD, RPV) | Done | `scripts/compare_etfs.py` |
+
+---
+
+## How to Run
+
+- **Config:** Edit `config/strategy_config.yaml` and load in code via `from src.config_loader import load_config`.
+- **Backtest + Walk-forward + Monte Carlo:** `python scripts/run_backtest.py`
+- **All expansion features (multi-factor, sector, quality, weighting, correlation, dashboard):** `python scripts/run_all_expansions.py`
+- **Rebalance (placeholder trade list):** `python scripts/rebalance.py --value 100000 --out output/rebalance_trades.csv`
+- **ETF comparison:** `python scripts/compare_etfs.py --etfs MTUM IWD RPV`
+- **Factor primer:** Read `docs/FACTOR_PRIMER.md`.

PULL_REQUEST.md

@@ -0,0 +1,65 @@
+# Pull Request: Repository Expansion + E2E Testing + Documentation
+
+## Title
+**feat: Expansion modules, config-driven pipeline, E2E tests, and documentation**
+
+## Description
+
+This PR extends the algorithmic-trading-python repository with **20 expansion features** (backtest, walk-forward, Monte Carlo, multi-factor, sector/quality, alternative momentum/value, risk parity, min-var, max-Sharpe, conviction sizing, correlation report, multi-source data, survivorship awareness, config pipeline, shared package, dashboard, rebalance script, factor primer, ETF comparison), adds **comprehensive E2E tests** for both original notebooks and new code, and includes **documentation** for running and reviewing everything.
+
+### Summary of changes
+
+| Area | Changes |
+|------|--------|
+| **Config** | `config/strategy_config.yaml` – single source for universe, data, strategies, portfolio, backtest, output |
+| **Shared package** | `src/` – config_loader, data (sources, universe), scoring (momentum, value, quality), portfolio (weighting, sector), backtest, walk_forward, monte_carlo, reports (dashboard, correlation) |
+| **Scripts** | `scripts/run_backtest.py`, `run_all_expansions.py`, `rebalance.py`, `compare_etfs.py` |
+| **Notebook** | `finished_files/004_expansion_all_features.ipynb` – runs expansion features from config + src |
+| **Tests** | `tests/` – fixtures, `test_old_notebooks_logic.py` (001/002/003), `test_new_expansion.py` (src + scripts), `run_e2e_tests.py`, `E2E_TEST_REPORT.md` |
+| **Docs** | `docs/FACTOR_PRIMER.md` (factor investing primer), `EXPANSION_TODO.md` (checklist), updated `README.md` |
+| **Dependencies** | `requirements.txt` – added PyYAML, yfinance, openpyxl |
+
+### Breaking changes
+
+- **None.** Original notebooks (001, 002, 003) and starter files are unchanged. All new code lives under `config/`, `src/`, `scripts/`, `tests/`, `docs/`, and one new notebook (004).
+
+### How to review
+
+1. **Run tests**
+   ```bash
+   pip install -r requirements.txt
+   python -m pytest tests/ -v --tb=short
+   python tests/run_e2e_tests.py
+   ```
+2. **Run scripts**
+   ```bash
+   python scripts/run_backtest.py
+   python scripts/run_all_expansions.py
+   python scripts/rebalance.py --out output/rebalance_trades.csv
+   python scripts/compare_etfs.py
+   ```
+3. **Read docs**
+   - `README.md` – quick start and expansion overview
+   - `docs/FACTOR_PRIMER.md` – value, momentum, quality theory
+   - `EXPANSION_TODO.md` – list of 20 expansion items and locations
+   - `tests/E2E_TEST_REPORT.md` – what was tested and output checks
+
+### Checklist
+
+- [x] New code under `src/`, `config/`, `scripts/`, `tests/`, `docs/`
+- [x] Original notebooks and logic unchanged; E2E tests verify 001/002/003 behavior
+- [x] All 26 pytest tests pass; scripts run with exit 0
+- [x] Documentation added (README, factor primer, expansion checklist, test report)
+- [x] `.gitignore` updated (output/, reports/, .pytest_cache/)
+
+---
+
+## Branch
+
+- **Branch name:** `feature/expansion-e2e-docs`
+- **Base:** `master`
+
+## Related
+
+- Expands on the course outline in README (Sections 3–5) with production-style tooling and tests.
+- No issues linked; can be linked to a tracking issue if one exists.

README.md

@@ -1,6 +1,22 @@
 # Algorithmic Trading in Python
 
-This repository
+This repository contains course materials and code for building algorithmic trading strategies in Python: equal-weight S&P 500, quantitative momentum (HQM), and quantitative value (RV), plus a full set of **expansion modules** for backtesting, multi-factor, sector/quality, and reporting.
+
+## Quick Start
+
+```bash
+pip install -r requirements.txt
+# Optional: add secrets.py with IEX_CLOUD_API_TOKEN for IEX Cloud
+python scripts/run_backtest.py          # Backtest, walk-forward, Monte Carlo
+python scripts/run_all_expansions.py     # Multi-factor, sector, weighting, dashboard
+python scripts/rebalance.py --value 100000
+python scripts/compare_etfs.py --etfs MTUM IWD RPV
+```
+
+- **Config:** `config/strategy_config.yaml`  
+- **Shared package:** `src/` (data, scoring, portfolio, backtest, walk_forward, monte_carlo, reports)  
+- **Factor primer:** `docs/FACTOR_PRIMER.md`  
+- **Expansion checklist:** `EXPANSION_TODO.md`
 
 ## Course Outline
 

config/strategy_config.yaml

@@ -0,0 +1,80 @@
+# Algorithmic Trading Strategy Configuration
+# All pipelines read from this file for reproducibility.
+
+universe:
+  name: "S&P 500"
+  constituents_file: "sp_500_stocks.csv"
+  use_survivorship_free: false  # Set true when point-in-time constituent data available
+
+data:
+  primary: "iex"       # iex | yfinance
+  fallback: "yfinance"
+  batch_size: 100
+
+strategies:
+  equal_weight:
+    enabled: true
+    top_n: null       # null = all constituents
+
+  momentum:
+    enabled: true
+    top_n: 50
+    time_periods: ["One-Year", "Six-Month", "Three-Month", "One-Month"]
+    # Alternative momentum options
+    use_52w_high_proximity: false
+    use_risk_adjusted_momentum: false
+
+  value:
+    enabled: true
+    top_n: 50
+    metrics: ["PE", "PB", "PS", "EV_EBITDA", "EV_GP"]
+    # Alternative value options
+    use_fcf_yield: false
+    use_earnings_yield: false
+    use_dividend_yield: false
+
+  multi_factor:
+    enabled: true
+    top_n: 50
+    factors:
+      value: 0.33
+      momentum: 0.33
+      quality: 0.34
+    quality_metrics: ["ROE", "earnings_stability"]
+
+  sector:
+    neutral: false    # true = sector-neutral weights
+    tilt: null        # e.g. {"Technology": 1.2, "Energy": 0.8}
+
+portfolio:
+  weighting: "equal"  # equal | risk_parity | volatility | conviction | min_variance | max_sharpe
+  conviction_power: 1.0  # For conviction weighting: weight ~ score^power
+  min_weight: 0.005
+  max_weight: 0.10
+
+filters:
+  quality_overlay: false
+  min_roe: 0.10
+  low_vol_overlay: false
+  max_volatility_pct: 0.50
+
+backtest:
+  start_date: "2010-01-01"
+  end_date: null      # null = latest
+  rebalance_freq: "M" # M = monthly, W = weekly
+  initial_capital: 100000
+
+walk_forward:
+  train_months: 60
+  test_months: 12
+  step_months: 12
+
+monte_carlo:
+  n_simulations: 1000
+  block_bootstrap: true
+  block_size: 12      # months
+
+output:
+  excel_dir: "output"
+  report_dir: "reports"
+  dashboard_format: "html"  # html | pdf

docs/FACTOR_PRIMER.md

@@ -0,0 +1,86 @@
+# Factor Investing Primer
+
+## Overview
+
+This document summarizes the theory and practice behind the factors implemented in this repository: **value**, **momentum**, and **quality**. It is intended as educational context for the strategies in the course.
+
+---
+
+## Value
+
+**Idea:** Stocks that are "cheap" relative to fundamentals (earnings, assets, sales, cash flow) tend to outperform over the long run.
+
+**Theoretical basis:**  
+- Fama–French: value (e.g. high book-to-market) earns a risk premium.  
+- Behavioral: investors overreact to bad news, so value stocks are underpriced.  
+- Risk-based: value firms are riskier (distress, cyclicality), so higher expected return.
+
+**Common metrics (lower = cheaper = more value):**
+- **P/E** (price-to-earnings): price per dollar of earnings. Fails for negative earnings.
+- **P/B** (price-to-book): market value vs book value. Less meaningful for intangibles-heavy firms.
+- **P/S** (price-to-sales): useful when earnings are negative or volatile.
+- **EV/EBITDA**: enterprise value to operating cash flow; good for cross-sector comparison.
+- **EV/GP** (enterprise value to gross profit): less affected by leverage and accounting choices.
+
+**In this repo:** The **RV (Robust Value)** score is the average of percentiles across P/E, P/B, P/S, EV/EBITDA, and EV/GP (lower raw metric → higher percentile → higher RV). We select the top 50 by RV for the value strategy.
+
+---
+
+## Momentum
+
+**Idea:** Stocks that have performed well over the recent past (e.g. 6–12 months) tend to continue performing well in the short term.
+
+**Theoretical basis:**  
+- Underreaction to information (earnings, news) so trends persist.  
+- Behavioral: herding and slow information diffusion.  
+- Risk: momentum as compensation for time-varying risk exposure.
+
+**Common implementations:**
+- **Price momentum:** past 2–12 month return (skip most recent month to avoid short-term reversal).
+- **High-quality momentum:** require strength across multiple horizons (e.g. 1m, 3m, 6m, 12m) to avoid one-off spikes (e.g. FDA approval).
+
+**In this repo:** **HQM (High-Quality Momentum)** uses 1m, 3m, 6m, and 1y price return percentiles and averages them. Top 50 by HQM form the momentum portfolio. Alternatives implemented: 52-week high proximity, risk-adjusted momentum (return/volatility).
+
+---
+
+## Quality
+
+**Idea:** Profitable, stable, low-leverage firms (high "quality") tend to have higher risk-adjusted returns.
+
+**Common metrics:**
+- **ROE** (return on equity): profitability.
+- **Earnings stability:** low volatility of earnings over time.
+- **Low leverage:** debt/equity or interest coverage.
+
+**In this repo:** Quality is used as an overlay (filter or score) and in the **multi-factor** strategy (value + momentum + quality weights). Quality score can be based on ROE and optional earnings stability.
+
+---
+
+## Multi-Factor Combination
+
+Combining value, momentum, and quality can improve diversification of signals and reduce regime dependence. In this repo, the multi-factor strategy uses configurable weights (e.g. 1/3 value, 1/3 momentum, 1/3 quality), ranks by composite score, and selects the top 50.
+
+---
+
+## Portfolio Construction
+
+- **Equal weight:** simplest; each name has same weight.  
+- **Risk parity / volatility weighting:** weight inversely to volatility so each position contributes similarly to risk.  
+- **Conviction:** weight by score (e.g. higher HQM or lower RV → higher weight).  
+- **Optimization:** minimum variance (minimize portfolio vol) or max Sharpe (maximize risk-adjusted return) subject to constraints.
+
+---
+
+## Backtesting Caveats
+
+- **Survivorship bias:** using only current index members overstates historical returns; use point-in-time constituents when possible.  
+- **Look-ahead bias:** ensure all inputs (prices, fundamentals) are available at each rebalance date.  
+- **Transaction costs:** rebalancing and turnover reduce net returns; model explicitly for realism.
+
+---
+
+## References (Further Reading)
+
+- Fama, E. F., & French, K. R. (1992). The cross‐section of expected stock returns. *Journal of Finance*.  
+- Jegadeesh, N., & Titman, S. (1993). Returns to buying winners and selling losers. *Journal of Finance*.  
+- Asness, C. S., Frazzini, A., & Pedersen, L. H. (2019). Quality minus junk. *Review of Accounting Studies*.

docs/README.md

@@ -0,0 +1,5 @@
+# Documentation
+
+- **[FACTOR_PRIMER.md](FACTOR_PRIMER.md)** – Factor investing primer (value, momentum, quality) and how it relates to this repo’s strategies.
+
+For project quick start and expansion overview, see the root [README.md](../README.md). For the full expansion checklist, see [EXPANSION_TODO.md](../EXPANSION_TODO.md). For test coverage, see [tests/E2E_TEST_REPORT.md](../tests/E2E_TEST_REPORT.md).