Skip to content

fix(ecr,sqs,elasticache): data-durability + portability gaps (bug-hunt 4.2/4.3/4.4)#2290

Merged
vieiralucas merged 1 commit into
mainfrom
bh-dura
Jul 16, 2026
Merged

fix(ecr,sqs,elasticache): data-durability + portability gaps (bug-hunt 4.2/4.3/4.4)#2290
vieiralucas merged 1 commit into
mainfrom
bh-dura

Conversation

@vieiralucas

@vieiralucas vieiralucas commented Jul 16, 2026

Copy link
Copy Markdown
Member

Summary

Three Tier-2 durability / portability findings from the 2026-07-15 bug hunt, each in a restart or error path CI never exercised.

  • 4.2 ECR — orphaned layer uploads 500 on restart. In-progress layer-upload spools live under the system temp dir (spool_path) and never survive a restart, but layer_uploads was persisted, so a restart left orphan entries whose next UploadLayerPart opened a missing spool and returned 500. layer_uploads is now #[serde(skip)] (ephemeral, consistent with the gone spools — a stale client simply re-initiates), and the append opens gained .create(true) as defense-in-depth.
  • 4.3 SQS — unresolvable-DLQ redrive hot loop. When a redrive target ARN doesn't resolve (DLQ deleted / typo) the message is correctly returned to the source queue instead of destroyed, but it re-entered with receive_count still past maxReceiveCount and visible_at = None, so the very next ReceiveMessage re-redrove it into the same dead target and spun. The receive history is now reset on return-to-source (mirroring the existing DLQ-landing reset), so the message is reprocessed normally.
  • 4.4 ElastiCache — hardcoded /tmp. The snapshot RDB dump path hardcoded /tmp/fakecloud-ec-*.rdb, absent on Windows and sometimes non-writable in containers. Now roots under std::env::temp_dir() via a new snapshot_rdb_temp_path helper.

Test plan

  • cargo test -p fakecloud-ecr --lib layer_uploads_are_not_persisted_across_snapshot (new).
  • cargo test -p fakecloud-sqs --lib redrive_with_unresolvable_dlq_arn_keeps_message_on_source_queue — extended to assert the returned message is re-receivable (loop-prevention).
  • cargo test -p fakecloud-elasticache --lib rdb_temp_path_roots_under_platform_temp_dir (new).
  • cargo clippy -p fakecloud-ecr -p fakecloud-sqs -p fakecloud-elasticache --tests -- -D warnings clean.

No API surface / SDK / docs change (persistence + portability internals only).


Summary by cubic

Fixes three durability and portability bugs across fakecloud-ecr, fakecloud-sqs, and fakecloud-elasticache. Prevents ECR 500s after restarts, SQS redrive hot loops on missing DLQs, and snapshot path failures on non-Unix systems.

  • Bug Fixes
    • ECR: Stop persisting in-progress layer_uploads (#[serde(skip)]) to avoid orphaned state after restart; append opens now use .create(true) to recreate missing spools.
    • SQS: When the DLQ target ARN is unresolvable, reset receive_count and first_received_at before returning the message to the source queue so it is reprocessed, not redriven in a loop.
    • ElastiCache: Write snapshot RDB files under std::env::temp_dir() via snapshot_rdb_temp_path instead of hardcoded /tmp for cross-platform portability.

Written for commit 367b47c. Summary will update on new commits.

Review in cubic

…t 4.2/4.3/4.4)

- 4.2 ECR — in-progress layer-upload spools live under the system temp dir and
  never survive a restart, but `layer_uploads` was persisted, leaving orphan
  entries whose next `UploadLayerPart` opened a missing spool and 500'd. Mark
  `layer_uploads` `#[serde(skip)]` (ephemeral, consistent with the gone spools)
  and add `.create(true)` to the append opens as defense-in-depth.

- 4.3 SQS — when a redrive target ARN is unresolvable the message is returned
  to the source queue (never destroyed), but it re-entered with receive_count
  still past maxReceiveCount and visible_at=None, so the next ReceiveMessage
  re-redrove it into the same dead target and spun in a hot loop. Reset the
  receive history on return-to-source (mirroring the DLQ-landing reset) so it
  is reprocessed normally.

- 4.4 ElastiCache — snapshot RDB dump path hardcoded `/tmp`, absent on Windows
  and sometimes non-writable in containers. Use `std::env::temp_dir()` via a
  new `snapshot_rdb_temp_path` helper.

Tests: ECR serde-skip round-trip; SQS unresolvable-DLQ message re-receivable
(loop-prevention) added to the existing test; ElastiCache rdb path roots under
the platform temp dir.
@vieiralucas
vieiralucas requested a review from Copilot July 16, 2026 00:49

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@vieiralucas
vieiralucas merged commit b347c33 into main Jul 16, 2026
156 checks passed
@vieiralucas
vieiralucas deleted the bh-dura branch July 16, 2026 04:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants