dd: write complete blocks without buffering them first#13471
Open
relative23 wants to merge 1 commit into
Open
Conversation
Merging this PR will improve performance by ×7.4
Performance Changes
Tip Curious why this is faster? Comment Comparing 
Footnotes
|
|
GNU testsuite comparison: |
BufferedOutput::write_blocks copies the incoming bytes into its internal buffer, hands that buffer to the inner writer, clears it, and then re-buffers the trailing partial block -- even when nothing was pending and the incoming bytes already form complete blocks. That is the common case: the copy loop hands over a multiple of the output block size on every iteration, so each one paid for a full memcpy of the block. Take that case directly: with an empty internal buffer, write the complete blocks straight from the caller's slice and buffer only the remainder. The existing path still handles a pending partial block. Instruction counts (cachegrind, same toolchain and base): dd_copy_separate_blocks 55,916,646 -> 5,447,251 -90.26% dd_copy_default 36,615,676 -> 30,718,154 -16.11% dd_copy_with_seek 8,511,906 -> 8,488,175 -0.28% dd_copy_with_skip 8,412,120 -> 8,389,437 -0.27% dd_copy_4k_blocks 4,742,612 -> 4,730,524 -0.25% dd_copy_8k_blocks 4,408,987 -> 4,401,037 -0.18% dd_copy_partial 3,222,128 -> 3,220,704 -0.04% dd_copy_64k_blocks 5,107,029 -> 5,105,557 -0.03% dd_copy_1m_blocks 9,410,822 -> 9,410,932 +0.00% Peak heap is unchanged (DHAT, 236,549 -> 236,546 bytes).
relative23
force-pushed
the
dd-buffered-output-fast-path
branch
from
July 21, 2026 09:05
e895791 to
3ff85c0
Compare
Contributor
|
Wahou, impressive wins! Bravo |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BufferedOutput::write_blockscopies the incoming bytes into its internal buffer, hands that buffer to the inner writer, clears it, and then re-buffers the trailing partial block — even when nothing was pending and the incoming bytes already form complete blocks. That is the common case: the copy loop hands over a multiple of the output block size on every iteration, so each iteration paid for a full memcpy of the block.Change
With an empty internal buffer, write the complete blocks straight from the caller's slice and buffer only the remainder. The existing path is untouched and still handles a pending partial block. Pure optimization — no behavioural change.
Measurements
Instruction counts via cachegrind on the CodSpeed simulation binaries (
cargo codspeed build -m simulation -p uu_dd), same toolchain, both built from this PR's base:maindd_copy_separate_blocksdd_copy_defaultdd_copy_with_seekdd_copy_with_skipdd_copy_4k_blocksdd_copy_8k_blocksdd_copy_partialdd_copy_64k_blocksdd_copy_1m_blocksThe two large wins are the configurations that actually use the buffered output path:
dd_copy_separate_blocks(ibs=8K obs=16K) anddd_copy_default(nobs, so output is buffered). The measurement is deterministic — repeated runs of the same binary vary by ~2 instructions out of 36M.Peak heap is unchanged: 236,549 → 236,546 bytes (DHAT,
ibs=8K obs=16K, both binaries built in the same target directory).Testing
cargo test -p uu_ddand the dd integration suite pass.mainover 17 configurations (oddibs/obs,ibs>obsandobs>ibs,bs,conv=sync/block/unblock/swab,count,count_bytes,skip/seek,obs=1M,ibs=1M obs=7): output and therecords in/records outlines are identical in every one.cargo fmt, the repository clippy invocation (util/run-clippy.py --features all --workspace) and cspell are clean.Note
This touches the same function as #13459, which fixes a correctness bug in the pending-partial-block path. The two are independent — this PR only adds the empty-buffer fast path and leaves the existing branch alone — but they overlap textually, so whichever lands second needs a trivial rebase.