feat: range aware file write

[XciD]

Summary

APIs for range-aware file writes: instead of re-uploading an entire file when only part of it changed, compose a new CAS file from stable segments + re-chunked dirty windows. Supports resize edits (insert / delete / arbitrary replace) in addition to in-place rewrites.

API: upload_ranges

pub async fn upload_ranges(
    config: Arc<TranslatorConfig>,
    cas_client: Arc<dyn Client>,
    original_hash: MerkleHash,
    original_size: u64,
    dirty_inputs: Vec<DirtyInput>,
) -> Result<XetFileInfo>

/// A single edit applied to the original file: replace `original_range` with
/// `new_length` bytes from `reader`. Edits are expressed in original-file coordinates.
pub struct DirtyInput {
    pub original_range: Range<u64>,
    pub reader: Pin<Box<dyn AsyncRead + Send>>,
    pub new_length: u64,
}

The output file size is derived from the inputs (no total_size parameter): original_size - removed + added.

Edit shapes (all expressible with the same struct)

Operation	original_range	new_length
In-place edit	a..b	b - a
Resize replace	a..b	any
Pure insert	p..p	> 0
Pure delete	a..b	0
Append	original_size..original_size	> 0
Truncate to N	N..original_size	0
No-op	empty dirty_inputs	—

Motivating example:

abc + upload_ranges([0..1), "foo", 3) = foobc
abc + upload_ranges([0..0), "foo", 3) = fooabc
abc + upload_ranges([0..1), "",    0) = bc

Per-range AsyncRead instead of ReadSeek over the staging file. The earlier prototype took dirty_ranges: &[(u64, u64)] + dirty_source: &mut dyn ReadSeek. That had a subtle bug: for truncation we silently extended the dirty set with a boundary chunk and read those bytes from the staging file, but if the file was never opened for write the staging file contains zeros at those positions (real bytes are in CAS) → silent corruption on the truncation boundary chunk. Pairing each edit with its own reader makes that structurally impossible: any byte not provided by the caller is fetched from CAS.

How it works

High level

                           upload_ranges
   +----------------------+   |   +----------------------+
   |  original file (CAS) |---+-->|  composed file (CAS) |
   +----------------------+       +----------------------+
   only the dirty windows are re-uploaded; everything else
   is reused as whole CAS segments.

Step 1 — coalesce + snap edits to segment boundaries

Edits are user-coordinates (byte ranges). We snap each edit's original_range to the enclosing CAS segments so composition can swap whole segments instead of truncating one mid-chunk. Adjacent / overlapping snapped ranges are then coalesced.

Pure inserts (start == end) snap to the segment that owns start; an insert at original_size snaps to the last segment.

Step 2 — server returns windows + gap subtrees

Single CAS call: GET /v2/file-chunk-hashes/{file_id} with the segment-aligned ranges in an X-Range-Dirty: bytes=A-B,C-D header. Response shape (xetcas#987):

struct FileChunkHashesResponse {
    windows:      Vec<ChunkWindow>,         // one per dirty range
    hash_ranges:  Vec<Option<MerkleHashSubtree>>, // N+1 entries: [gap0, gap1, ..., gapN]
}

windows[i].chunks carries the chunk hashes the server actually owns for that window (we re-upload these bytes). hash_ranges[i] is the MerkleHashSubtree for the i-th unmodified gap, or None when there is no gap there. This is the key to composing the final file hash without touching unmodified bytes.

Step 3 — for each window, stream [CAS prefix | edits | CAS suffix] through a fresh cleaner

window = [w_start ............................................. w_end]
edits in this window:        [edit_a]    [edit_b]
                                ^           ^
streamed input to the cleaner:
  CAS bytes [w_start, edit_a.start)
  reader bytes for edit_a (new_length bytes)
  CAS bytes [edit_a.end, edit_b.start)
  reader bytes for edit_b
  CAS bytes [edit_b.end, w_end)

Pure inserts contribute zero original bytes but still emit new_length reader bytes. Pure deletes contribute zero reader bytes. The cleaner produces a new MDBFileInfo per window and a ChunkHashList.

Step 4 — compose the file hash via MerkleHashSubtree::merge

merge_seq = [gap0, w0, gap1, w1, ..., wN, gapN]   // skip None gaps

merged          = MerkleHashSubtree::merge(merge_seq)
aggregated_hash = merged.final_hash()
combined_hash   = aggregated_hash.hmac(zero)      // matches cleaner's file_hash

Special-case: if total_size == 0 (e.g. truncate to empty) the result is MerkleHash::default() without HMAC, mirroring file_hash([]).

Step 5 — splice segments + register

Walk the original MDBFileInfo.segments and replace any segment that falls inside a window with that window's freshly-uploaded segments. Verification entries follow segment-for-segment when present. metadata_ext = None (no SHA-256, see Limitations). Then register_composed_file + finalize.

Multi-window example

Two edits: replace [50MB, 51MB) and [150MB, 151MB) on a 200MB file:

+-----------+-------+------------+-------+-----------+
|  GAP 0    |  W0   |   GAP 1    |  W1   |  GAP 2    |
|  reused   |upload |  reused    |upload |  reused   |
| (subtree) | ~1MB  | (subtree)  | ~1MB  | (subtree) |
+-----------+-------+------------+-------+-----------+

Wire transfer: ~2MB upload + a few hundred KB of CAS reads for window
boundary chunks. Old approach: 200MB download + 200MB upload.

Empty original short-circuit

When original_size == 0 there is nothing to compose against — every edit's original_range must be 0..0 (validated). We just stream the new bytes through a fresh cleaner (upload_fresh_file).

Reviewer note: chunk_window_builder is a re-implementation of xetcas

xet_client/src/cas_client/chunk_window_builder.rs is a port of the same window-building state machine that already lives in xetcas — it's only used by the local / in-memory simulation clients (local_client, memory_client) so the mock CAS server returns the same shape as the real one in tests. No need to re-review it as part of this PR: it mirrors logic already reviewed and merged in xetcas#987. A follow-up xetcas PR will deduplicate by removing the server-side copy and pulling this one in (or vice versa); the duplication is intentional and temporary.

Limitations

• No SHA-256 metadata: composed files have metadata_ext = None since recomputing SHA-256 would require reading the full file. Only suitable for contexts that don't require SHA-256 verification (HF buckets, xet-native repos), not for Git LFS-backed repos.
• Memory: for very large files, the per-window in-memory state (chunk hash list + composed segments) is bounded by the dirty regions, not the whole file. The chunk-hashes response is paginated by the server-defined window granularity.

Tests (27)

Covering all edit shapes + edge cases. Notable:

Test	Purpose
test_resize_edits_abc	The 3 motivating FUSE examples
test_resize_large_replace_grows_file	Replace [a..b) with much more data
test_resize_large_replace_shrinks_file	Replace [a..b) with much less data
test_resize_mid_file_insert	Pure insert in the middle
test_resize_mid_file_delete	Pure delete in the middle
test_resize_multi_edit_mix	Insert + replace + delete in one call
test_resize_insert_at_segment_boundary	Snapping correctness for inserts
test_upload_ranges_mid_file_edit	In-place edit
test_upload_ranges_truncation	Pure truncate (sub-segment)
test_upload_ranges_truncation_empty_staging	Truncate when staging is all-zero (boundary read from CAS)
test_upload_ranges_truncation_with_overlapping_dirty	Truncate + dirty range overlapping the boundary
test_truncate_to_empty_matches_clean_empty	Truncating to 0 hashes to MerkleHash::default() (matches a fresh empty cleaner)
test_upload_ranges_append	Pure append
test_append_with_gap_before_dirty_range	Append where reader covers a sparse gap too
test_append_sparse_staging_file	Append on a sparse staging file
test_mid_edit_plus_append	Mid-file edit and append in one call (P1 codex regression)
test_empty_original_append	original_size == 0 + append falls into the fresh-file path (P2 codex regression)
test_empty_original_validates_ranges	original_size == 0 still runs validation (reviewer regression)
test_upload_ranges_at_file_start	Edit at offset 0 (no stable prefix)
test_upload_ranges_multiple_regions	Two non-adjacent dirty windows with stable gap
test_single_input_spanning_many_chunks	One edit covering many CDC chunks
test_data_integrity_scenarios	5 sub-scenarios covering composition correctness
test_noop_returns_original_hash	Empty dirty_inputs → no CAS call, original hash returned
test_rejects_dirty_range_past_total_size	Validation: range past original_size
test_rejects_overlapping_dirty_ranges	Validation: overlapping edits
test_rejects_unsorted_dirty_ranges	Validation: unsorted edits
test_upload_ranges_small_file_mid_edit	Small files (single segment)

Dependencies

• xetcas: GET /v2/file-chunk-hashes/{file_id} with windows[] + hash_ranges[] response shape — huggingface-internal/xetcas#987 (merged).
• Consumer: feat: sparse writes with range_upload, zero-download write path hf-mount#41.

---

Note

High Risk
High risk because it adds a new partial-upload composition path that splices CAS segments and recomputes file hashes from window subtrees, touching core data integrity and client/server chunk-boundary logic.

Overview
Adds range-aware file writes via new upload_ranges, letting callers apply insert/delete/replace edits and upload only re-chunked dirty windows while reusing stable CAS segments.

Introduces a new CAS API get_file_chunk_hashes (GET /v2/file-chunk-hashes/{file_id} with X-Range-Dirty) plus response types (FileChunkHashesResponse, ChunkWindow) and simulation support (chunk_window_builder) that extends dirty ranges to stable chunk boundaries and returns gap MerkleHashSubtree summaries + stable-segment verification.

Refactors dedup/cleaning plumbing to expose per-chunk hash lists (ChunkHashList), adds detached cleaner/session completion and register_composed_file to avoid orphan shard entries, and moves/re-exports next_stable_chunk_boundary into xet_core_structures for shared stable-window computations.

^{Reviewed by Cursor Bugbot for commit 2f4cee4. Bugbot is set up for automated code reviews on this repo. Configure here.}

[rajatarya]

Re-review (post xetcas#987 merge)

This PR has come a long way since my approval on March 27. I went back and re-read the whole thing end-to-end against the new wire shape, and the rework is genuinely good — what shipped is cleaner than what I approved.

What's new since last pass

1. Wire shape pivot. Composition is now driven by windows[] + hash_ranges[] of opaque MerkleHashSubtree summaries (xetcas#987), not flat per-chunk hashes. Per-chunk hashes never cross the wire, so the response stays O(windows + gaps) regardless of file size. Final hash is rebuilt with MerkleHashSubtree::merge — no ChunkHashList-glue layer, no duplicated tree-folding logic on the client.
2. Resize edits. DirtyInput { original_range, reader, new_length } collapses in-place edit / replace / pure-insert / pure-delete / append / truncate into a single shape. The original_range.len() != new_length branch falls out cleanly thanks to per-edit readers (no ReadSeek over a sparse staging file — that bug class is structurally gone).
3. My GC concern is addressed at the client layer. The new register_single_file_clean_completion_detached + register_composed_file split means window-uploads no longer pollute the session shard with orphan MDBFileInfo entries — only the final composed file is registered. Combined with sirahd's GC tracking issue (huggingface-internal/xet-garbage-collection#2), the design now does the right thing: window xorbs are findable through the composed file, and there are no dangling shard references.
4. Defensive runtime invariants (commit 87b9061): server-contract violations (windows.len() != n_windows, hash_ranges.len() != n_windows + 1, leftover edits not assigned to any window, gap_verification length mismatch with stable-segment count) are now hard runtime errors instead of silent corruption. Exactly the right call for a composition algorithm where zip would otherwise drop windows quietly.

What upload_ranges offers, visually

End-to-end flow:

sequenceDiagram
    autonumber
    participant C as Caller (e.g. hf-mount)
    participant U as upload_ranges
    participant S as CAS server
    participant D as Cleaner / dedup

    C->>U: dirty_inputs[]<br/>(original_range, reader, new_length)
    Note over U: validate + coalesce<br/>snap to segment boundaries
    U->>S: GET /v2/file-chunk-hashes/{id}<br/>X-Range-Dirty: bytes=A-B,C-D
    S-->>U: windows[N] + hash_ranges[N+1]<br/>+ gap_verification[stable_segs]
    loop one cleaner per window
        U->>S: stream CAS prefix (boundary chunks)
        U->>D: feed prefix bytes
        U->>D: feed reader bytes (the edit)
        U->>S: stream CAS suffix
        U->>D: feed suffix bytes
        D-->>U: chunks + new MDBFileInfo (detached)
    end
    Note over U: merge_seq = [gap0, w0, gap1, ..., gapN]<br/>combined_hash = merge(seq).hmac(0)
    U->>S: register_composed_file(MDBFileInfo)
    U->>S: finalize session
    U-->>C: XetFileInfo { hash, size }

Loading

Why this is a big deal — the wire savings:

flowchart LR
    subgraph old ["Old: full re-upload"]
        OF["200MB original"] --> OU["download 200MB"]
        OU --> OE["edit 1MB"]
        OE --> OUP["upload 200MB"]
    end
    subgraph new ["upload_ranges: 2MB edit"]
        NF["200MB original"] --> NS["snap edits to<br/>segment boundaries"]
        NS --> NW["server returns<br/>2 windows + 3 gap subtrees"]
        NW --> NB["download ~few KB<br/>boundary chunks"]
        NB --> NU["upload ~2MB<br/>(2 dirty windows)"]
    end
    style old fill:#ffe5e5
    style new fill:#e5ffe5

Loading

Hash composition without re-reading the file:

flowchart TB
    F["Original file: chunks 0..M"] --> G0["gap0 subtree<br/>(server-stored)"] & G1["gap1 subtree<br/>(server-stored)"] & G2["gap2 subtree<br/>(server-stored)"]
    R["DirtyInput readers"] --> CL1["cleaner: window 0"] --> W0["w0 subtree<br/>(client-computed)"]
    R --> CL2["cleaner: window 1"] --> W1["w1 subtree<br/>(client-computed)"]
    G0 --> M{{"MerkleHashSubtree::merge<br/>[gap0, w0, gap1, w1, gap2]"}}
    W0 --> M
    G1 --> M
    W1 --> M
    G2 --> M
    M --> H["final_hash().hmac(0)<br/>== file_hash of edited file"]

Loading

Edit-shape coverage (one struct, six operations):

flowchart LR
    DI["DirtyInput<br/>{ original_range: a..b, new_length: n }"] --> Q1{"a == b?"}
    Q1 -->|yes| Q2{"n > 0?"}
    Q1 -->|no| Q3{"n == 0?"}
    Q2 -->|yes| INS["pure insert at a"]
    Q2 -->|no| NOOP["degenerate no-op"]
    Q3 -->|yes| DEL["pure delete a..b"]
    Q3 -->|no| Q4{"b - a == n?"}
    Q4 -->|yes| IPE["in-place edit"]
    Q4 -->|no| RES["resize replace"]
    INS -.->|special: a == original_size| APP["append"]
    DEL -.->|special: b == original_size| TRU["truncate to a"]

Loading

Items to think about (none blocking)

1. chunk_window_builder.rs duplication. PR description flags this as intentional and temporary, mirroring xetcas#987 server logic so simulation clients answer get_file_chunk_hashes without HTTP. Worth filing a follow-up issue (or noting #717 in xetcas) to track the eventual dedup so the duplication doesn't quietly stick around as duplications always do without a forcing function.
2. Empty gap_verification on partial-verification shards. When file_info.verification.len() != file_info.segments.len(), the simulation builder silently emits an empty gap_verification, and then compose_mdb errors out at runtime with a less-targeted message. Inline below — either error up-front or document the silent path.
3. One readability nit + one defensive question. Inline.

Nothing here blocks the merge — most of these are conversation starters. Approving again on the strength of the current state. Nice work.

[cursor]

Cursor Bugbot has reviewed your changes using default mode and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 2f4cee4. Configure here.}

[cursor]

Shared read buffer across windows prevents concurrent processing

Low Severity

A single buf allocation is hoisted outside the window loop and reused across all windows. While this avoids repeated allocation, it also makes the loop inherently sequential — each window must finish reading before the next can begin. For multi-window edits on large files, allocating a per-window buffer (or using a pool) would enable future parallelization of window processing without a redesign.

 

^{Reviewed by Cursor Bugbot for commit 2f4cee4. Configure here.}