Commit 9320fbf
committed
perf(decompress): Batch column drops in decompress_columns()
Individual df.drop() per column caused O(N_cols × N_rows) pandas
reindex on each call. For 7 columns × 82M rows, this was 12s out
of 17s total (72% of runtime).
Fix: Collect all compressed columns, drop once after loop.
Expected: 17s → ~5s for 7-column decompression on 82M rows.
Profile-driven: cProfile showed df.drop → _slice_take_blocks_ax0
→ numpy.take consuming 12.1s across 7 calls.1 parent 650a97e commit 9320fbf
1 file changed
Lines changed: 10 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6692 | 6692 | | |
6693 | 6693 | | |
6694 | 6694 | | |
| 6695 | + | |
| 6696 | + | |
| 6697 | + | |
| 6698 | + | |
6695 | 6699 | | |
6696 | 6700 | | |
6697 | 6701 | | |
| |||
6741 | 6745 | | |
6742 | 6746 | | |
6743 | 6747 | | |
6744 | | - | |
| 6748 | + | |
6745 | 6749 | | |
6746 | | - | |
| 6750 | + | |
6747 | 6751 | | |
6748 | 6752 | | |
6749 | 6753 | | |
| |||
6753 | 6757 | | |
6754 | 6758 | | |
6755 | 6759 | | |
| 6760 | + | |
| 6761 | + | |
| 6762 | + | |
| 6763 | + | |
6756 | 6764 | | |
6757 | 6765 | | |
6758 | 6766 | | |
| |||
0 commit comments