Skip to content

Commit a682589

Browse files
joeywangzroz-agent
andcommitted
Harden checkpoint mechanics: missing upload targets and unsafe filenames
Review follow-ups on the checkpoint upload/commit pipeline. 1. Withhold the commit when the server omits a blob's upload target. `upload_gathered_snapshot` already anticipates a short `upload-snapshot` response and only warns; the target-less blob then reached `upload_entry` and was recorded as `EntryStatus::Skipped`. The checkpoint gate rejected only `Failed`, so the attempt committed a silently smaller object set and made it the selected checkpoint — discarding a previously complete one. `Skipped` conflated two very different causes. Split out `EntryStatus::NoTarget` for "server returned no presigned target" and treat it as fatal for a checkpoint attempt, leaving `Skipped` to mean only the deliberate `MAX_SNAPSHOT_FILES_PER_RUN` cap. Both still surface in the manifest as `skipped` so rehydration consumers keep a stable status vocabulary; the distinguishing detail stays in `error`. The legacy end-of-run path is unchanged in behavior. 2. Sanitize agent-controlled filenames before they reach storage names. `gather_repo` sanitized its filename component but `gather_file` used the raw basename, which flows into `checkpoint_<generation>__<logical_name>`. `__` is documented as the reserved separator and the charset as `[A-Za-z0-9._-]`, but nothing enforced that for the filename half — and basenames come from agent-created files. `a__b.txt`, or `checkpoint_1700000000000-0__evil.txt`, produced an ambiguous storage name that either fails the server's existence check at commit time (losing the whole checkpoint) or lands under a different generation. `sanitize_name_component` now collapses to the server charset and squashes `_` runs so `__` can never appear; `gather_file` and `sanitize_filename_component` both route through it, and `storage_name` debug-asserts the invariant. 3. Smaller fixes. - `CheckpointGeneration::from_validated` now actually validates (debug assertion) instead of only claiming to in its name and docs. - Move the "pending commit" log after the outcome check so it no longer fires for attempts that failed to allocate targets. - Correct the `CheckpointResult::Failed { generation: None }` doc: an external timeout also reports `None` after a generation was minted, so `None` must not be read as "nothing landed in storage". Tests: commit is withheld when the server omits a blob target (chunked so the truncation hits a blob rather than the always-last manifest); sanitization never yields `__` or out-of-charset bytes; and an end-to-end guard that a hostile basename cannot produce an ambiguous storage name. Co-Authored-By: Oz <oz-agent@warp.dev>
1 parent 1e98b54 commit a682589

3 files changed

Lines changed: 277 additions & 36 deletions

File tree

app/src/ai/agent_sdk/driver/snapshot.rs

Lines changed: 96 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -584,7 +584,18 @@ struct SnapshotUploadFile {
584584
enum EntryStatus {
585585
Uploaded,
586586
Failed,
587+
/// Deliberately dropped from the upload plan to honor [`MAX_SNAPSHOT_FILES_PER_RUN`].
588+
/// This is a policy decision rather than a failure, so a checkpoint attempt may still
589+
/// commit the kept subset.
587590
Skipped,
591+
/// The server returned no presigned target for this blob — a contract violation of
592+
/// `upload-snapshot`'s positional alignment (see the length-mismatch warning in
593+
/// [`upload_gathered_snapshot`]).
594+
///
595+
/// Deliberately distinct from [`EntryStatus::Skipped`]: nothing intentional happened
596+
/// here, so a checkpoint attempt that hits this must be withheld rather than committing
597+
/// a silently smaller object set over a previously complete selected checkpoint.
598+
NoTarget,
588599
GatherFailed,
589600
ReadFailed,
590601
}
@@ -595,6 +606,7 @@ impl EntryStatus {
595606
Self::Uploaded => "uploaded",
596607
Self::Failed => "failed",
597608
Self::Skipped => "skipped",
609+
Self::NoTarget => "no_target",
598610
Self::GatherFailed => "gather_failed",
599611
Self::ReadFailed => "read_failed",
600612
}
@@ -613,6 +625,7 @@ struct SnapshotSummary {
613625
uploaded: usize,
614626
failed: usize,
615627
skipped: usize,
628+
no_target: usize,
616629
gather_failed: usize,
617630
read_failed: usize,
618631
total: usize,
@@ -625,6 +638,7 @@ impl SnapshotSummary {
625638
uploaded: 0,
626639
failed: 0,
627640
skipped: 0,
641+
no_target: 0,
628642
gather_failed: 0,
629643
read_failed: 0,
630644
total: entries.len(),
@@ -635,6 +649,7 @@ impl SnapshotSummary {
635649
EntryStatus::Uploaded => s.uploaded += 1,
636650
EntryStatus::Failed => s.failed += 1,
637651
EntryStatus::Skipped => s.skipped += 1,
652+
EntryStatus::NoTarget => s.no_target += 1,
638653
EntryStatus::GatherFailed => s.gather_failed += 1,
639654
EntryStatus::ReadFailed => s.read_failed += 1,
640655
}
@@ -671,10 +686,15 @@ pub(super) enum CheckpointResult {
671686
/// beyond reading local state were made.
672687
Skipped,
673688
/// A required upload (a non-cap-skipped blob, or the manifest), the upload-target
674-
/// allocation, or the commit call itself failed. `generation` is `None` only when the
675-
/// attempt was cut off before a generation was even minted (e.g. an external timeout
676-
/// wrapping the whole attempt). Any minted generation's objects (if uploaded) are left
677-
/// as uncommitted debris in storage; the server's existing marker (if any) is untouched.
689+
/// allocation, or the commit call itself failed. Any minted generation's objects (if
690+
/// uploaded) are left as uncommitted debris in storage; the server's existing marker
691+
/// (if any) is untouched.
692+
///
693+
/// `generation` is `None` when the attempt never reported one back to the caller. That
694+
/// covers both "cut off before a generation was minted" and "cut off by an external
695+
/// timeout wrapping the whole attempt" — in the latter case a generation may well have
696+
/// been minted and objects uploaded, so `None` must not be read as "nothing landed in
697+
/// storage".
678698
Failed {
679699
generation: Option<CheckpointGeneration>,
680700
reason: String,
@@ -731,6 +751,10 @@ pub(super) fn mint_generation() -> CheckpointGeneration {
731751
/// field, so no client-side renaming is needed before that point.
732752
#[allow(dead_code)]
733753
fn storage_name(generation: &CheckpointGeneration, logical: &str) -> String {
754+
debug_assert!(
755+
!logical.contains("__"),
756+
"logical snapshot filename must not contain the reserved `__` separator: {logical}"
757+
);
734758
format!("checkpoint_{}__{logical}", generation.as_str())
735759
}
736760

@@ -1036,32 +1060,37 @@ async fn run_checkpoint_pipeline(
10361060
pre_upload_entries,
10371061
)
10381062
.await;
1039-
log::info!(
1040-
"Checkpoint attempt generation={generation} pending commit",
1041-
generation = generation.as_str()
1042-
);
10431063
let Some(outcome) = outcome else {
10441064
return CheckpointResult::Failed {
10451065
generation: Some(generation),
10461066
reason: "failed to allocate upload targets or serialize manifest".to_string(),
10471067
};
10481068
};
10491069
log_snapshot_outcome(&outcome);
1070+
log::info!(
1071+
"Checkpoint attempt generation={generation} pending commit",
1072+
generation = generation.as_str()
1073+
);
10501074

10511075
if !outcome.manifest_uploaded {
10521076
return CheckpointResult::Failed {
10531077
generation: Some(generation),
10541078
reason: "manifest failed to upload".to_string(),
10551079
};
10561080
}
1081+
// `NoTarget` is fatal alongside `Failed`: the server owes us a presigned target for
1082+
// every requested filename, so a missing one means this attempt would otherwise commit
1083+
// a silently smaller object set and make it the selected checkpoint, discarding a
1084+
// previously complete one. Only `Skipped` (the deliberate per-run cap) is tolerated.
10571085
if outcome
10581086
.entries
10591087
.iter()
1060-
.any(|e| e.status == EntryStatus::Failed)
1088+
.any(|e| matches!(e.status, EntryStatus::Failed | EntryStatus::NoTarget))
10611089
{
10621090
return CheckpointResult::Failed {
10631091
generation: Some(generation),
1064-
reason: "one or more required blobs failed to upload".to_string(),
1092+
reason: "one or more required blobs failed to upload or had no upload target"
1093+
.to_string(),
10651094
};
10661095
}
10671096

@@ -1388,10 +1417,16 @@ async fn gather_file(
13881417
let path = Path::new(file_path);
13891418
match tokio::fs::read(path).await {
13901419
Ok(content) => {
1391-
let preferred = path
1392-
.file_name()
1393-
.map(|n| n.to_string_lossy().to_string())
1394-
.unwrap_or_else(|| file_path.to_string());
1420+
// Sanitize before uniquifying: the basename comes from an agent-created file, and
1421+
// it ends up inside the `checkpoint_<generation>__<logical_name>` storage name
1422+
// that the exact-set commit has to reproduce byte for byte.
1423+
let preferred = sanitize_name_component(
1424+
&path
1425+
.file_name()
1426+
.map(|n| n.to_string_lossy().to_string())
1427+
.unwrap_or_else(|| file_path.to_string()),
1428+
"snapshot_artifact",
1429+
);
13951430
let filename = unique_filename(&preferred, used_filenames);
13961431
let mime = mime_guess::from_path(path)
13971432
.first_or_octet_stream()
@@ -1430,18 +1465,21 @@ async fn gather_file(
14301465
}
14311466

14321467
/// Upload a single prepared file through the retry helper.
1433-
/// Produces an [`EntryResult`] labelled with the file's filename, or marked `skipped` if the
1434-
/// server did not return a target for it.
1468+
/// Produces an [`EntryResult`] labelled with the file's filename, or marked
1469+
/// [`EntryStatus::NoTarget`] if the server did not return a target for it.
14351470
async fn upload_entry(
14361471
http: &http_client::Client,
14371472
file: &SnapshotUploadFile,
14381473
target_map: &HashMap<String, UploadTarget>,
14391474
) -> EntryResult {
14401475
let Some(target) = target_map.get(&file.filename) else {
1441-
log::warn!("No upload target for file '{}', skipping", file.filename);
1476+
log::warn!(
1477+
"No upload target returned by the server for file '{}'; it will not be uploaded",
1478+
file.filename
1479+
);
14421480
return EntryResult {
14431481
label: file.filename.clone(),
1444-
status: EntryStatus::Skipped,
1482+
status: EntryStatus::NoTarget,
14451483
error: Some("no upload target returned by server".to_string()),
14461484
};
14471485
};
@@ -1488,7 +1526,11 @@ fn fold_upload_results(
14881526
repo_entry.status = "failed";
14891527
repo_entry.error = entry.error.clone();
14901528
}
1491-
EntryStatus::Skipped => {
1529+
// Both surface in the manifest as `skipped` so downstream rehydration
1530+
// consumers keep seeing a stable status vocabulary; the distinguishing
1531+
// detail lives in `error` (and in the checkpoint gate, which treats
1532+
// `NoTarget` as fatal).
1533+
EntryStatus::Skipped | EntryStatus::NoTarget => {
14921534
repo_entry.uploaded = Some(false);
14931535
repo_entry.status = "skipped";
14941536
repo_entry.error = entry.error.clone();
@@ -1514,7 +1556,7 @@ fn fold_upload_results(
15141556
file_entry.status = "failed";
15151557
file_entry.error = entry.error.clone();
15161558
}
1517-
EntryStatus::Skipped => {
1559+
EntryStatus::Skipped | EntryStatus::NoTarget => {
15181560
file_entry.uploaded = Some(false);
15191561
file_entry.status = "skipped";
15201562
file_entry.error = entry.error.clone();
@@ -1615,11 +1657,13 @@ fn log_snapshot_outcome(outcome: &SnapshotOutcome) {
16151657
"manifest: failed"
16161658
};
16171659
let header = format!(
1618-
"Snapshot upload: {}/{} uploaded (failed: {}, skipped: {}, gather_failed: {}, read_failed: {}; {manifest_bit})",
1660+
"Snapshot upload: {}/{} uploaded (failed: {}, skipped: {}, no_target: {}, \
1661+
gather_failed: {}, read_failed: {}; {manifest_bit})",
16191662
summary.uploaded,
16201663
summary.total,
16211664
summary.failed,
16221665
summary.skipped,
1666+
summary.no_target,
16231667
summary.gather_failed,
16241668
summary.read_failed,
16251669
);
@@ -1715,25 +1759,44 @@ async fn git_output_string(repo_dir: &Path, args: &[&str]) -> Option<String> {
17151759
if value.is_empty() { None } else { Some(value) }
17161760
}
17171761

1718-
fn sanitize_filename_component(value: &str) -> String {
1719-
let sanitized = value
1720-
.chars()
1721-
.map(|c| {
1722-
if c.is_ascii_alphanumeric() || matches!(c, '.' | '-' | '_') {
1723-
c
1724-
} else {
1725-
'_'
1726-
}
1727-
})
1728-
.collect::<String>();
1762+
/// Collapse `value` into the server's `[A-Za-z0-9._-]` charset, squashing runs of `_` so the
1763+
/// result can never contain `__`.
1764+
///
1765+
/// `__` is reserved as the separator in `checkpoint_<generation>__<logical_name>` storage
1766+
/// object names (see [`storage_name`] and [`CheckpointGeneration`]), and the logical name half
1767+
/// is derived from **agent-controlled** input: workspace file basenames and repo directory
1768+
/// names. Leaving it unsanitized lets an agent-created file such as `a__b.txt` (or, worse,
1769+
/// `checkpoint_1700000000000-0__x.txt`) produce an ambiguous storage name, which either fails
1770+
/// the server's existence check at commit time — losing the whole checkpoint — or lands under
1771+
/// a different generation than intended.
1772+
///
1773+
/// Returns `fallback` when nothing usable survives sanitization.
1774+
fn sanitize_name_component(value: &str, fallback: &str) -> String {
1775+
let mut sanitized = String::with_capacity(value.len());
1776+
for c in value.chars() {
1777+
let c = if c.is_ascii_alphanumeric() || matches!(c, '.' | '-' | '_') {
1778+
c
1779+
} else {
1780+
'_'
1781+
};
1782+
// Squash runs so `__` can never appear in the output.
1783+
if c == '_' && sanitized.ends_with('_') {
1784+
continue;
1785+
}
1786+
sanitized.push(c);
1787+
}
17291788
let trimmed = sanitized.trim_matches('_');
17301789
if trimmed.is_empty() {
1731-
"repo".to_string()
1790+
fallback.to_string()
17321791
} else {
17331792
trimmed.to_string()
17341793
}
17351794
}
17361795

1796+
fn sanitize_filename_component(value: &str) -> String {
1797+
sanitize_name_component(value, "repo")
1798+
}
1799+
17371800
fn unique_filename(preferred: &str, used: &mut HashSet<String>) -> String {
17381801
let preferred = Path::new(preferred)
17391802
.file_name()

0 commit comments

Comments
 (0)