Skip to content

feat: add CSV, JSON, and Parquet URL data providers#472

Merged
MDUYN merged 2 commits into
devfrom
feature/external-data-providers
Apr 23, 2026
Merged

feat: add CSV, JSON, and Parquet URL data providers#472
MDUYN merged 2 commits into
devfrom
feature/external-data-providers

Conversation

@MDUYN
Copy link
Copy Markdown
Collaborator

@MDUYN MDUYN commented Apr 22, 2026

Summary

Add support for loading external CSV, JSON, and Parquet data from any URL during strategy execution.

Changes

  • BaseURLDataProvider — shared base class with caching (in-memory + file), refresh intervals, date parsing, and pre/post-processing callbacks
  • CSVURLDataProvider, JSONURLDataProvider, ParquetURLDataProvider — format-specific implementations
  • DataSource.from_csv(), from_json(), from_parquet() — factory methods for pre-declared data sources
  • context.fetch_csv(), fetch_json(), fetch_parquet() — on-demand methods for dynamic fetching
  • Cache path fix — cache files now stored inside resources/data/ (synced by state handlers in cloud deployments) instead of a detached .data_cache/ directory
  • 43 tests covering all three providers
  • Documentation — new external-data.md docs page, sidebar registration, README updates

Cloud deployment support

Cache files live inside the resource directory, so AWSS3StorageStateHandler and AzureBlobStorageStateHandler automatically persist them. refresh_interval works correctly across cold starts using file modification times.

Closes #458

MDUYN added 2 commits April 23, 2026 00:47
- Add BaseURLDataProvider with shared caching, refresh intervals,
  date parsing, and pre/post-processing logic
- Add CSVURLDataProvider, JSONURLDataProvider, ParquetURLDataProvider
- Add DataSource.from_csv(), from_json(), from_parquet() factory methods
- Add context.fetch_csv(), fetch_json(), fetch_parquet() on-demand methods
- Fix cache path resolution for cloud deployments (AWS Lambda / Azure Functions)
- Add 43 tests covering all three providers
- Add external-data.md documentation page
- Update README with external data capabilities

Closes #458
@MDUYN MDUYN merged commit c5fdea2 into dev Apr 23, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant