|
1 | | -# :material-language-typescript:{.icon} IEX Download |
2 | 1 |
|
3 | | -**Automated Access to IEX Historical Market Data** |
| 2 | +# **IEX-DOWNLOAD: Rust-Powered Access to IEX Historical Data** |
4 | 3 |
|
5 | | -The Investors Exchange (IEX) provides free access to its historical datasets — including **TOPS** (top-of-book quotes and trades) and **DEEP** (detailed order book with depth-of-market snapshots). While valuable, the official interface forces users to manually click through each file, making large-scale downloads inefficient and error-prone. |
| 4 | +**Project Goal** |
| 5 | +IEX-DOWNLOAD is a robust command-line utility, written in pure Rust, for retrieving **IEX TOPS, DEEP, and DEEP+ historical datasets**. It provides fast, reliable downloads from the official IEX historical feed, making it trivial to bootstrap your research pipeline with **over 13 TB of market data (4,600+ files) spanning 2016–2025**. |
6 | 6 |
|
7 | | -The IEX Download project was created to address the challenge of retrieving large volumes of IEX historical data efficiently. It automates the bulk acquisition of packet capture (pcap) files provided by IEX and, when paired with the companion project IEX2H5, transforms these raw datasets into research-ready HDF5 arrays — enabling high-speed time-series analysis and rigorous backtesting at scale. |
| 7 | +It succeeds the original **Node.js IEX Download project**, preserving its automation features while adding the **performance, safety, and reliability of Rust**. |
8 | 8 |
|
9 | | -## Problems & Solutions |
| 9 | +**The Problem** |
| 10 | +Obtaining historical tick-level data from exchanges is often messy and error-prone: |
10 | 11 |
|
11 | | -* :material-bottle-tonic-plus:{.icon} **Manual bottlenecks** → Automated, scriptable downloads across date ranges. |
12 | | -* :material-select-drag:{.icon} **File selection overhead** → Support for choosing **TOPS** or **DEEP** datasets on demand. |
13 | | -* :material-run-fast:{.icon} **Uncertainty in runs** → Dry-run mode to test queries without downloading. |
14 | | -* :material-approximately-equal-box:{.icon} **Stalled or failed transfers** → Built-in retry logic and progress timeout handling. |
15 | | -* :material-format-paragraph-spacing:{.icon} **Usability gap** → Visual progress bars and flexible download directory settings. |
| 12 | +* :material-cloud-download:{.icon} Manual: the official interface forces tedious, one-by-one downloads. |
| 13 | +* :material-cancel:{.icon} Unreliable: network hiccups and partial transfers force restarts. |
| 14 | +* :material-file-question:{.icon} Unstructured: managing thousands of files across years of trading days is cumbersome. |
16 | 15 |
|
17 | | -## Features |
18 | 16 |
|
19 | | -* **:material-vector-arrange-above:{.icon} Bulk Automation** — Retrieve entire ranges of historical datasets without manual effort. |
20 | | -* **:material-calendar-edit:{.icon} Flexible Date Selection** — Download data across any custom time window. |
21 | | -* **:material-select-multiple:{.icon} Dataset Choice** — Access either **TOPS** (top-of-book) or **DEEP** (depth-of-book) feeds depending on your analysis needs. |
22 | | -* **:material-magnify:{.icon} Preview Mode** — Run queries in dry-run mode to confirm parameters before downloading. |
23 | | -* **:material-format-header-5:{.icon} Custom Storage** — Choose exactly where your datasets are saved. |
24 | | -* **:material-transfer-right:{.icon} Reliable Transfers** — Automatic safeguards against stalled or incomplete downloads. |
25 | | -* **:material-check-all:{.icon} Ready for Analysis** — Designed to work hand-in-hand with **IEX2H5** for seamless conversion to HDF5 arrays. |
| 17 | +**The Solution** |
| 18 | +**IEX-DOWNLOAD** brings together the **automation of the original Node.js version** with the **performance of Rust**: |
26 | 19 |
|
| 20 | +* :material-lightning-bolt:{.icon} **Performs**: Rust delivers speed, safety, and reliability at scale. |
| 21 | +* :material-calendar-edit:{.icon} **Flexible date ranges**: download a single day or entire multi-year spans in one command. |
| 22 | +* :material-select-multiple:{.icon} **Dataset choice**: selectively fetch **TOPS**, **DEEP**, or **DEEP+** feeds. |
| 23 | +* :material-magnify:{.icon} **Dry-run mode**: preview download size and files before committing. |
| 24 | +* :material-transfer-right:{.icon} **Reliable transfers**: built-in retries, resumable downloads, and graceful handling of stalled connections. |
| 25 | +* :material-folder-multiple:{.icon} **Organized output**: automatically stores datasets by feed and date, consistent with IEX conventions. |
| 26 | +* :fontawesome-solid-terminal:{.icon} **Simple CLI**: no fragile scripts, just a single command. |
| 27 | +* :material-history:{.icon} **Complete coverage**: supports all available feeds since December 2016. |
27 | 28 |
|
28 | | -Do you want me to also rewrite the **Usage section** in the same "feature-first" style, so it reads more like a product capability page rather than a developer tool manual? |
| 29 | +**Why It Matters** |
| 30 | +Market researchers, quants, and engineers need **raw truth data** to test ideas and power analytics. With IEX-DOWNLOAD, you don’t just get a handful of samples—you unlock the **entire IEX historical feed, 13+ terabytes strong**, reliably and reproducibly, without wrestling with brittle tooling. |
29 | 31 |
|
| 32 | +Paired with **IEX2H5**, the Rust downloader + C++ converter form a complete pipeline: |
| 33 | +:material-vector-arrange-above:{.icon} **download terabytes of PCAPs** → :material-database:{.icon} **convert into compressed HDF5 arrays** → :material-brain:{.icon} **analyze at scale**. |
30 | 34 |
|
31 | | -## Usage |
| 35 | +> :fontawesome-brands-github:{.icon} **Explore the Project** |
| 36 | +> GitHub: [github.com/vargaconsulting/iex-download](https://github.com/vargaconsulting/iex-download) |
| 37 | +> Docs: [vargaconsulting.github.io/iex-download](https://vargaconsulting.github.io/iex-download) |
32 | 38 |
|
33 | | -```bash |
34 | | -After running the script, the `download` directory will be populated with |
35 | | -TOPS or DEEP gzip-compressed datasets, named according to the corresponding |
36 | | -trading day. For additional details on processing the data, see `iex2h5`. |
37 | | - |
38 | | -The data is provided free of charge by IEX. By accessing or using IEX |
39 | | -Historical Data, you agree to their Terms of Use. For more information, |
40 | | -visit: https://iextrading.com/iex-historical-data-terms/ |
41 | | - |
42 | | -INSTALLATION: |
43 | | - npm install ./iex-download-x.y.z.tgz |
44 | | - |
45 | | - |
46 | | -Options: |
47 | | - --version Show version number [boolean] |
48 | | - --deep Download IEX DEEP datasets |
49 | | - --tops Download IEX TOPS datasets |
50 | | - --directory Location to save downloaded files [default: "./"] |
51 | | - --progress_stall_timeout Timeout duration (in seconds) to detect and handle stalled downloads [default: 30] |
52 | | - --max_retry Number of times to retry downloading the same file before giving up [default: 5] |
53 | | - --from Date you start downloading from [default: "2025-08-19"] |
54 | | - --to Last day to download [default: "2025-08-21"] |
55 | | - --dry-run Skips downloading |
56 | | - --help Show help [boolean] |
57 | | - |
58 | | -Copyright © <2017-2025> Varga Consulting, Toronto, ON. info@vargaconsulting.ca |
59 | | - |
60 | | -``` |
61 | | - |
62 | | -## Licensing & Credits |
63 | | - |
64 | | -* Data is provided free of charge by **IEX Exchange**, subject to their [Terms of Use](https://iextrading.com/iex-historical-data-terms/). |
65 | | -* Project © 2017–2025 **Varga Consulting**, Toronto, ON. |
| 39 | +[repo]: https://github.com/vargaconsulting/iex-download |
| 40 | +[docs]: https://vargaconsulting.github.io/iex-download |
0 commit comments