Use async_fs file for java and python writes to avoid object_store buffering on local disks#7920
Conversation
Signed-off-by: Robert Kruszewski <github@robertk.io>
a72e75d to
214dfb1
Compare
|
This is recreation of #7904 with small addition of having python also go through local file for writes. This already happens in C/C++ and is not necessary in rust since we don't expose api to write to a path so the user has to resolve the path to file handle |
Merging this PR will improve performance by 20.27%
|
| Mode | Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|---|
| ⚡ | Simulation | new_bp_prim_test_between[i16, 32768] |
134.1 µs | 120.2 µs | +11.57% |
| ⚡ | Simulation | new_bp_prim_test_between[i32, 32768] |
169.9 µs | 141.1 µs | +20.44% |
| ⚡ | Simulation | new_bp_prim_test_between[i32, 16384] |
109.1 µs | 94.8 µs | +15.11% |
| ⚡ | Simulation | new_bp_prim_test_between[i64, 16384] |
144.4 µs | 115.1 µs | +25.43% |
| ⚡ | Simulation | new_bp_prim_test_between[i64, 32768] |
236.7 µs | 178 µs | +32.96% |
| ⚡ | Simulation | new_alp_prim_test_between[f64, 16384] |
148.8 µs | 126.9 µs | +17.29% |
Tip
Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.
Comparing rk/asyncfscreate (214dfb1) with develop (7349cd6)
Footnotes
-
24 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
Object store writes are tuned for multipart uploads, it's simpler to use std
file for local writes