Skip to content

Commit 22aafbf

Browse files
FrancescAltedclaude
andcommitted
utf8 plan: express speedups as Nx multipliers instead of percentages
Consistent with how these numbers are now reported elsewhere: 6.05x total (3622ms -> 598.6ms), 1.22x slower than string(), 2.16x faster than vlstring(). Proportions that aren't speedups (time-share breakdown, a data-loss rate) are left as percentages. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
1 parent 45b0b4c commit 22aafbf

1 file changed

Lines changed: 16 additions & 15 deletions

File tree

plans/utf8-write-ingest-optim.md

Lines changed: 16 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
# utf8 columns: closing the write/ingest gap (incremental plan)
22

33
**Status:** I1 and I2 both LANDED (2026-07-17, branch `enhancing-ctable3`).
4-
Taxi ingest: 3622 ms → 870.5 ms after I1 → 598.6 ms after I2, now within
5-
~20% of `string()` (490.4 ms) and clearly ahead of `vlstring()`
6-
(1292.2 ms). See "Honest assessment" below for the full before/after
7-
table. Successor work to `plans/utf8-reads-filter-optim.md` (U1 + U2,
4+
Taxi ingest: 3622 ms → 870.5 ms after I1 (4.16x) → 598.6 ms after I2
5+
(6.05x total), now 1.22x slower than `string()` (490.4 ms) and 2.16x
6+
faster than `vlstring()` (1292.2 ms). See "Honest assessment" below for
7+
the full before/after table. Successor work to `plans/utf8-reads-filter-optim.md` (U1 + U2,
88
landed on branch `enhancing-ctable3`, commits `3a7d9a3d`, `45ec7906`).
99
Read that plan first for background and for the exact working rebuild
1010
commands referenced below; this plan assumes it.
@@ -197,7 +197,7 @@ Key properties:
197197
outside this loop body.
198198

199199
**Measured** (extend-loop only, `_rewrite_from` unchanged, taxi-like ASCII
200-
workload, 1e7 rows): 1557ms → 943ms (**-39%**).
200+
workload, 1e7 rows): 1557ms → 943ms (**1.65x**).
201201

202202
**`append()`/`_flush_if_needed()`:** leave essentially untouched — already
203203
minimal single-row overhead, no measurable win from rewriting them.
@@ -244,9 +244,9 @@ decision 3).
244244
unchanged path, ~3ms `isascii()`-scan overhead, matches expectation).
245245

246246
**Combined I1.a + I1.c** (full extend+flush pipeline, 1e7 rows,
247-
Python-level cost only): 1557ms → 630ms (**-60%**). Projected onto the
247+
Python-level cost only): 1557ms → 630ms (**2.47x**). Projected onto the
248248
full 3622ms end-to-end benchmark (NDArray-side cost assumed unaffected,
249-
flush count unchanged): **≈2695ms** (-26%).
249+
flush count unchanged): **≈2695ms** (1.34x).
250250

251251
### I1.d — Raising `_FLUSH_ROWS` (folded into I1)
252252

@@ -313,9 +313,10 @@ regression elsewhere in that script, or in `sort_by`/groupby/`to_arrow`
313313
(which reuse `_rewrite_from` via `set_all`/`copy`).
314314

315315
**ACTUAL (2026-07-17, `enhancing-ctable3`, Apple-silicon Mac, I1.a + I1.c +
316-
I1.d combined):** taxi ingest **870.5 ms** (from 3622 ms, **-76%**) —
316+
I1.d combined):** taxi ingest **870.5 ms** (from 3622 ms, **4.16x**) —
317317
gate passed with a huge margin, and utf8 ingest is now *faster* than both
318-
`string()` (1011.1 ms) and `vlstring()` (1106.1 ms) on this workload. No
318+
`string()` (1011.1 ms, 1.16x) and `vlstring()` (1106.1 ms, 1.27x) on this
319+
workload. No
319320
regression in the rest of `bench_string_kinds.py`'s output (full read,
320321
filter, groupby, sort, `to_arrow`, both workloads, all three column
321322
kinds) beyond ordinary run-to-run noise. Full `tests/` suite: 7513
@@ -404,9 +405,9 @@ adding cleverness — ~2065 ms of the original 3622 ms is inherent NDArray
404405
write/compression cost outside I1/I2's scope, so there is a hard floor.
405406

406407
**ACTUAL (2026-07-17, `enhancing-ctable3`, Apple-silicon Mac):** taxi
407-
ingest **598.6 ms** (from 870.5 ms after I1 alone, **-31%** further;
408-
**-83.5%** from the original 3622 ms) — gate passed with a large margin,
409-
now within ~22% of `string()` (490.4 ms) on this workload. No regression
408+
ingest **598.6 ms** (from 870.5 ms after I1 alone, **1.45x** further;
409+
**6.05x** from the original 3622 ms) — gate passed with a large margin,
410+
now 1.22x slower than `string()` (490.4 ms) on this workload. No regression
410411
elsewhere in `bench_string_kinds.py` (full read, filter, groupby, sort,
411412
`to_arrow`, both workloads, all three column kinds) beyond ordinary
412413
run-to-run noise.
@@ -455,13 +456,13 @@ text, kept for the record:
455456
**Update after I2 landed:** the recommendation above (re-evaluate before
456457
building I2, since the realistic ceiling looked like only ~150ms) turned
457458
out to be pessimistic — I2 delivered another **272ms** (870.5ms →
458-
598.6ms, -31%), more than the ~150ms estimated from the I1.d
459+
598.6ms, **1.45x**), more than the ~150ms estimated from the I1.d
459460
`_FLUSH_ROWS` sweep's asymptote. The compiled kernel avoids not just the
460461
per-row `.encode()` calls but also the intermediate `list` of `bytes`
461462
objects and the `b"".join()` allocation that I1.c's Python fast path
462463
still pays; those turned out to matter more than the sweep alone
463-
suggested. Final state: utf8 ingest 3622ms → 598.6ms (**-83.5%**),
464-
within ~22% of `string()` (490.4ms) and clearly ahead of `vlstring()`
464+
suggested. Final state: utf8 ingest 3622ms → 598.6ms (**6.05x**),
465+
1.22x slower than `string()` (490.4ms) and 2.16x faster than `vlstring()`
465466
(1292.2ms). Remaining gap to `string()` (~108ms) is presumably the last
466467
of the inherent NDArray resize/write/compression cost referenced above —
467468
not investigated further; not gated by this plan.

0 commit comments

Comments
 (0)