@@ -9,6 +9,9 @@ kernelspec:
99 display_name : Python 3 (ipykernel)
1010 language : python
1111 name : python3
12+ myst :
13+ substitutions :
14+ extra_dependencies : pyarrow
1215---
1316
1417(streaming_dataset)=
@@ -47,6 +50,11 @@ bounded buffer.
4750` N ` is small here so the notebook runs in seconds. The streaming code is the same at
4851any size, and the last section shows what changes at scale.
4952
53+ +++
54+
55+ :::{include} ../extra_installs.md
56+ :::
57+
5058``` {code-cell} ipython3
5159import glob
5260import tempfile
@@ -68,9 +76,9 @@ az.style.use("arviz-variat")
6876## Put a dataset on disk and forget the array
6977
7078We build a logistic-regression dataset, write it to Parquet shards, and delete the
71- in-memory table. From here the features only exist on disk. We keep ` X ` and ` y `
72- around only to build the in-RAM ` pm.Minibatch ` baseline later; the streaming fit
73- never reads them.
79+ in-memory table. From here the streaming path reads only the disk copy . We keep ` X `
80+ and ` y ` around only to build the in-RAM ` pm.Minibatch ` baseline later; the streaming
81+ fit never reads them.
7482
7583``` {code-cell} ipython3
7684N = 30_000
@@ -130,8 +138,7 @@ with pm.Model() as model:
130138 idata_stream = approx.sample(1000)
131139```
132140
133- The negative-ELBO trace shows the fit converging while only ever holding a
134- ` batch_size ` buffer in memory:
141+ The negative-ELBO trace shows the fit converging on minibatches read off disk:
135142
136143``` {code-cell} ipython3
137144fig, ax = plt.subplots(figsize=(9, 3))
@@ -164,15 +171,14 @@ bs_stream = idata_stream.posterior["b"].values.reshape(-1, 4)
164171bs_inram = idata_inram.posterior["b"].values.reshape(-1, 4)
165172names = ["intercept", "slope x1", "slope x2", "slope x3"]
166173
167- fig, axes = plt.subplots(1, 4, figsize=(13, 3))
174+ fig, axes = plt.subplots(1, 4, figsize=(13, 3), layout="tight" )
168175for k, ax in enumerate(axes):
169176 ax.hist(bs_stream[:, k], bins=40, density=True, alpha=0.5, label="streaming")
170177 ax.hist(bs_inram[:, k], bins=40, density=True, alpha=0.5, label="in-RAM")
171178 ax.axvline(b_true[k], color="k", ls="--", lw=1)
172179 ax.set(title=names[k], yticks=[])
173180axes[0].legend(fontsize=8)
174- fig.suptitle("Posterior of b: streaming vs in-RAM (dashed = ground truth)", y=1.04)
175- fig.tight_layout();
181+ fig.suptitle("Posterior of b: streaming vs in-RAM (dashed = ground truth)", y=1.04);
176182```
177183
178184## Memory usage
@@ -186,31 +192,30 @@ cost. The line below is its lower bound (`N * ncols * 8` bytes), not a measureme
186192ncols = 4 # 3 features + observed
187193n_grid = np.logspace(5, 9, 50)
188194inram_gb = n_grid * ncols * 8 / 1e9 # whole dataset resident (array lower bound)
189- stream_gb = np.full_like(n_grid, batch_size * ncols * 8 / 1e9) # just the buffer
195+ stream_gb = np.full_like(n_grid, batch_size * ncols * 8 / 1e9) # one resident batch
190196
191- fig, ax = plt.subplots(figsize=(8, 5))
197+ fig, ax = plt.subplots(figsize=(8, 5), layout="tight" )
192198ax.loglog(n_grid, inram_gb, lw=2.5, label="in-RAM pm.Minibatch (O(N))")
193199ax.loglog(n_grid, stream_gb, lw=2.5, label="streaming DataLoader (O(batch))")
194200ax.axhline(26, color="0.5", ls="--", lw=1)
195201ax.text(n_grid[-1], 30, "26 GB RAM", color="0.5", ha="right", va="bottom")
196202ax.set_xlabel("dataset size N")
197203ax.set_ylabel("array footprint (GB, lower bound)")
198204ax.set_title("Memory is flat in N when streaming")
199- ax.legend(loc="lower right", framealpha=0.95)
200- fig.tight_layout();
205+ ax.legend(loc="lower right", framealpha=0.95);
201206```
202207
203208That line is only the bare array. Actual peak RSS is higher, because of the
204- framework and PyTensor's resident copy, and it hits the RAM ceiling sooner. To get
205- the real number on public data, we measured peak memory on the
209+ framework and PyTensor's resident copy, and it hits the RAM ceiling sooner. As a
210+ real-data check, outside this notebook, we ran the same logistic model (13 numeric
211+ features plus the click label) on the
206212[ Criteo 1TB Click Logs] ( https://huggingface.co/datasets/criteo/CriteoClickLogs ) , a
207- standard out-of-core learning benchmark, with the same logistic model (13 numeric
208- features plus the click label). Streaming through the ` DataLoader ` stayed flat at
209- about 0.7 GB across a sweep from 1M to 150M rows. The in-RAM ` pm.Minibatch ` baseline
210- rose linearly to 15.7 GB at 150M rows, about 21 times more, and extrapolates to
211- out-of-memory near 238M rows on a 26 GB machine. The streaming and in-RAM posteriors agree coefficient for coefficient; the
212- largest gap is about 0.1, on the intercept. Criteo is
213- public, so anyone can rerun this.
213+ standard, publicly available out-of-core learning benchmark. Peak memory for the
214+ streaming ` DataLoader ` stayed flat at about 0.7 GB across a sweep from 1M to 150M
215+ rows, while the in-RAM ` pm.Minibatch ` baseline rose linearly to 15.7 GB at 150M
216+ rows, which extrapolates to out-of-memory around 250M rows on the same 26 GB
217+ machine. The streaming and in-RAM posteriors agreed coefficient for coefficient;
218+ the largest gap was about 0.1, on the intercept.
214219
215220## When to use it
216221
0 commit comments