fix(ecr,sqs,elasticache): data-durability + portability gaps (bug-hunt 4.2/4.3/4.4)#2290
Merged
Conversation
…t 4.2/4.3/4.4) - 4.2 ECR — in-progress layer-upload spools live under the system temp dir and never survive a restart, but `layer_uploads` was persisted, leaving orphan entries whose next `UploadLayerPart` opened a missing spool and 500'd. Mark `layer_uploads` `#[serde(skip)]` (ephemeral, consistent with the gone spools) and add `.create(true)` to the append opens as defense-in-depth. - 4.3 SQS — when a redrive target ARN is unresolvable the message is returned to the source queue (never destroyed), but it re-entered with receive_count still past maxReceiveCount and visible_at=None, so the next ReceiveMessage re-redrove it into the same dead target and spun in a hot loop. Reset the receive history on return-to-source (mirroring the DLQ-landing reset) so it is reprocessed normally. - 4.4 ElastiCache — snapshot RDB dump path hardcoded `/tmp`, absent on Windows and sometimes non-writable in containers. Use `std::env::temp_dir()` via a new `snapshot_rdb_temp_path` helper. Tests: ECR serde-skip round-trip; SQS unresolvable-DLQ message re-receivable (loop-prevention) added to the existing test; ElastiCache rdb path roots under the platform temp dir.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three Tier-2 durability / portability findings from the 2026-07-15 bug hunt, each in a restart or error path CI never exercised.
spool_path) and never survive a restart, butlayer_uploadswas persisted, so a restart left orphan entries whose nextUploadLayerPartopened a missing spool and returned 500.layer_uploadsis now#[serde(skip)](ephemeral, consistent with the gone spools — a stale client simply re-initiates), and the append opens gained.create(true)as defense-in-depth.receive_countstill pastmaxReceiveCountandvisible_at = None, so the very nextReceiveMessagere-redrove it into the same dead target and spun. The receive history is now reset on return-to-source (mirroring the existing DLQ-landing reset), so the message is reprocessed normally./tmp. The snapshot RDB dump path hardcoded/tmp/fakecloud-ec-*.rdb, absent on Windows and sometimes non-writable in containers. Now roots understd::env::temp_dir()via a newsnapshot_rdb_temp_pathhelper.Test plan
cargo test -p fakecloud-ecr --lib layer_uploads_are_not_persisted_across_snapshot(new).cargo test -p fakecloud-sqs --lib redrive_with_unresolvable_dlq_arn_keeps_message_on_source_queue— extended to assert the returned message is re-receivable (loop-prevention).cargo test -p fakecloud-elasticache --lib rdb_temp_path_roots_under_platform_temp_dir(new).cargo clippy -p fakecloud-ecr -p fakecloud-sqs -p fakecloud-elasticache --tests -- -D warningsclean.No API surface / SDK / docs change (persistence + portability internals only).
Summary by cubic
Fixes three durability and portability bugs across
fakecloud-ecr,fakecloud-sqs, andfakecloud-elasticache. Prevents ECR 500s after restarts, SQS redrive hot loops on missing DLQs, and snapshot path failures on non-Unix systems.layer_uploads(#[serde(skip)]) to avoid orphaned state after restart; append opens now use.create(true)to recreate missing spools.receive_countandfirst_received_atbefore returning the message to the source queue so it is reprocessed, not redriven in a loop.std::env::temp_dir()viasnapshot_rdb_temp_pathinstead of hardcoded/tmpfor cross-platform portability.Written for commit 367b47c. Summary will update on new commits.