Skip to content

Commit d53de48

Browse files
MagicalTuxclaude
andcommitted
Use single-stream download: presigned URLs are GET-only
Segmented/parallel mode probes the size with a HEAD request, but the blob URLs are SigV4-presigned for GET only, so the HEAD returns 403 Forbidden. Drop segment_size/parallelism and use single-stream mode (GET, with Range-based resume) — still streamed to disk with no size cap, just not parallelized. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent a7b260e commit d53de48

1 file changed

Lines changed: 7 additions & 4 deletions

File tree

src/main.rs

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -178,8 +178,13 @@ fn parse_sha256_hex(hex: &str) -> Option<[u8; 32]> {
178178
/// These blobs can be hundreds of MiB, so the body is *streamed to a file*
179179
/// rather than buffered in memory. rsurl's resumable downloader continues a
180180
/// partial `.part` across retries and process restarts (so a dropped
181-
/// connection resumes instead of restarting), fetches segments in parallel,
182-
/// and — given the content hash — verifies the finished file end-to-end.
181+
/// connection resumes instead of restarting) and — given the content hash —
182+
/// verifies the finished file end-to-end.
183+
///
184+
/// Single-stream (not segmented/parallel): segmented mode probes the size with
185+
/// a `HEAD`, but these blob URLs are SigV4-presigned for `GET` only, so a
186+
/// `HEAD` returns 403. A plain `GET` stream still gets the essentials — no
187+
/// size cap, and `Range`-based resume on a drop.
183188
fn fetch_blob(args: &RunArgs, d: &DataRef) -> Result<Vec<u8>> {
184189
let url = d
185190
.url
@@ -206,8 +211,6 @@ fn fetch_blob(args: &RunArgs, d: &DataRef) -> Result<Vec<u8>> {
206211
None => (cache.join("pending-download.tmp"), true),
207212
};
208213
let opts = rsurl::DownloadOptions {
209-
segment_size: Some(16 * 1024 * 1024),
210-
parallelism: 4,
211214
max_time: Some(Duration::from_secs(300)),
212215
expected_sha256: parse_sha256_hex(&d.hash),
213216
..Default::default()

0 commit comments

Comments
 (0)