|
400 | 400 | "source": [ |
401 | 401 | "That line is only the bare array. Actual peak RSS is higher, because of the\n", |
402 | 402 | "framework and PyTensor's resident copy, and it hits the RAM ceiling sooner. To get\n", |
403 | | - "the real number on public data, I measured peak memory on the\n", |
| 403 | + "the real number on public data, we measured peak memory on the\n", |
404 | 404 | "[Criteo 1TB Click Logs](https://huggingface.co/datasets/criteo/CriteoClickLogs), a\n", |
405 | 405 | "standard out-of-core learning benchmark, with the same logistic model (13 numeric\n", |
406 | 406 | "features plus the click label). Streaming through the `DataLoader` stayed flat at\n", |
407 | 407 | "about 0.7 GB across a sweep from 1M to 150M rows. The in-RAM `pm.Minibatch` baseline\n", |
408 | 408 | "rose linearly to 15.7 GB at 150M rows, about 21 times more, and extrapolates to\n", |
409 | | - "out-of-memory near 238M rows on a 26 GB machine. The two posteriors agree to\n", |
410 | | - "within ADVI noise (correlation 0.999 across all 14 coefficients). Criteo is\n", |
| 409 | + "out-of-memory near 238M rows on a 26 GB machine. The streaming and in-RAM posteriors agree coefficient for coefficient; the\n", |
| 410 | + "largest gap is about 0.1, on the intercept. Criteo is\n", |
411 | 411 | "public, so anyone can rerun this.\n", |
412 | 412 | "\n", |
413 | 413 | "## When to use it\n", |
|
0 commit comments