Skip to content

README.md

Latest commit

 

History

History
76 lines (61 loc) · 3.72 KB

README.md

File metadata and controls

76 lines (61 loc) · 3.72 KB
Raw

Data module

Goal

Provide reproducible, point-in-time market, macro, membership, sector, and text inputs without silently replacing missing real observations with synthetic data. Network calls are cached so historical data is reused and only missing tails are requested.

How it works

  1. market_data.py and macro_data.py load yfinance data into incremental Parquet caches.
  2. point_in_time_universe.py and liquid_universe.py apply dated membership and lagged 63-session dollar-volume eligibility.
  3. fama_french.py downloads the official daily research-factor ZIP once and reuses a decimal- return Parquet cache for beta estimation and attribution.
  4. fnspid.py, finmultitime.py, and remote_zip.py import historical text selectively and resumably. FinMultiTime ZIP members are fetched with HTTP byte ranges rather than downloading the full archive.
  5. text_store.py normalizes and deduplicates all sources in data/raw/nlp_posts.parquet.
  6. confirmation_inputs.py audits prices, sectors, and news before confirmation can run.
  7. feature_store.py aligns the audited inputs for feature engineering.
  8. sector_history.py performs backward as-of joins and incrementally extends trailing sector-ETF classifications without copying future sectors into earlier rows.
  9. corporate_actions.py caches yfinance split dates/ratios and supports a pre-activation 1/5/20-session event study.

Synthetic data is allowed only for an explicit demo workflow. The public-confirmation manifest sets market.use_demo_fallback: false, and both market and macro loaders fail closed.

Current achievement

  • Public 2024 through 2025 Q1 confirmation requested 527 point-in-time stocks.
  • Non-synthetic price and inferred-sector coverage: 505 stocks, or 95.8%.
  • The 22 unavailable symbols were excluded before scoring with no substitutions.
  • FinMultiTime import: 79,364 ticker-linked news rows.
  • News coverage: 94.8% of confirmation business days.
  • Incremental price caches and resumable news caches avoid repeated network and model work.
  • Official daily Fama-French factors are cached locally with source URL, retrieval time, date range, row count, and a warning that the current research file is not a vintage archive.

Limitations

  • FinMultiTime market data ends on 2025-03-28, so the original full 2024-2025 confirmation contract remains unavailable.
  • Inferred sectors use trailing relationships with sector ETFs rather than official historical GICS classifications.
  • Split history is cached for the current live universe from 2024 onward; older delisted-symbol corporate actions remain incomplete.
  • Most historical news provides only a publication date, not a tradable intraday timestamp.
  • The public Fama-French file can be revised as source data and construction methods change; it is not a perfect point-in-time factor vintage.

Main commands

python -m src.cli collect-news
python -m src.cli import-fnspid --help
python -m src.cli import-finmultitime --help
python -m src.cli prepare-confirmation-inputs --help
python -m src.cli download-fama-french --help
python -m src.cli run-split-event-study --help
python -m src.cli build-dataset --help

Improvement backlog

  • Add licensed or reproducible delisted-security prices and official effective-dated sectors.
  • Record source, retrieval time, checksum, row count, and coverage with every dataset artifact.
  • Acquire genuine publication timestamps before evaluating event-triggered execution.
  • Keep all real-data paths fail closed and retain synthetic data only for tests and demos.

Related documentation