fix(autoscaling,ec2,kinesis): persist CFN ASG + scheduler/pipes deliveries and guard EC2 restart reconcile (bug-hunt)

[vieiralucas]

Summary

Fixes a cluster of four CONFIRMED restart data-loss / persistence bugs that share one root cause: a write path mutates in-memory state through a service built WITHOUT a snapshot store, so it never persists and never reconciles on restart.

1. CFN Auto Scaling Group loses ALL instances on restart. CreateStack/UpdateStack insert the ASG with instances=[], fire cfn_reconcile_capacity DETACHED, then persist_touched_services serializes the group empty. The detached reconcile pushed launched instances into g.instances but never re-persisted, so the ASG reloaded with zero instances. The detached reconcile now fires the autoscaling snapshot hook after it finishes mutating, so the launched instances survive restart. Threaded the autoscaling + EC2 snapshot hooks through ContainerBackingHandles::with_snapshot_hooks at every stack path (CreateStack, UpdateStack, ExecuteChangeSet, Cloud Control).

2. ASG-launched EC2 instances had no snapshot store -> container leak on restart. run_ec2_instances drives a bare Ec2Service::with_state(...) (no snapshot store), so the EC2 records lived only in memory; on restart EC2 boot-recovery had no persisted pending/running row to re-drive and the containers leaked with no owning state. apply_capacity now fires an EC2 snapshot hook after launching/terminating instances. Wired for BOTH the CFN reconcile path and the direct-API path (autoscaling_service.with_ec2_snapshot_hook(...) in the server), and fired before the ASG write lock so a concurrent group delete can't skip it.

3. EC2 recover_persisted_containers incomplete reconcile guard (restart lifecycle race). During the restart recovery window the Ok arm guarded only state_code == 48 and the Err arm guarded neither terminal code, so a concurrent StopInstances (code 80) was clobbered back to running with a fresh container, and a concurrent TerminateInstances (code 48) on a boot failure was overwritten to stopped. Both arms now use the same both-terminal guard (recovery_terminal_wins) that the normal-path reconcile_started uses, so a concurrent stop/terminate wins.

4. Scheduler/Pipes Kinesis delivery never persisted. The EventBridge Scheduler and EventBridge-Pipes delivery buses built their Kinesis sender with KinesisDeliveryImpl::new(...) (which sets dirty: None, making mark_dirty() a no-op), so records delivered to a Kinesis stream target by a schedule or a pipe landed in memory but never flipped the shared kinesis_delivery_dirty flag the flusher persists on. Both sites now use with_dirty_flag(kinesis_state.clone(), kinesis_delivery_dirty.clone()), matching the main EventBridge-rule path.

Test plan

• cargo test -p fakecloud-autoscaling -p fakecloud-ec2 -p fakecloud-kinesis — all pass, including new regressions:
  • cfn_provision::tests::cfn_reconcile_persists_asg_and_ec2_instances — runs the CFN ASG reconcile with capturing snapshot stores and asserts the autoscaling snapshot (ci: run examples in CI #1) AND the EC2 snapshot (ci: add Moto compatibility test suite to CI #2) both contain the launched instances after reconcile.
  • service::recover_guard_tests::* — assert the recovery reconcile does NOT overwrite a concurrent terminate (48) / stop (80) in either the Ok or Err arm, and still recovers a still-pending instance.
  • delivery::tests::new_does_not_mark_dirty_but_with_dirty_flag_does — pins the ::new (no-op flag) vs with_dirty_flag (flips the shared flag) contract the scheduler/pipes wiring depends on.
• cargo build -p fakecloud --tests — server-bin wiring compiles.
• cargo clippy -p fakecloud-autoscaling -p fakecloud-ec2 -p fakecloud-kinesis -p fakecloud-cloudformation --all-targets -- -D warnings and cargo clippy -p fakecloud --bin fakecloud -- -D warnings — clean.
• cargo fmt — clean.
• cargo test -p fakecloud-cloudformation — 270 pass (no regression from the ContainerBackingHandles plumbing).

No Docker is required by the new tests; they assert the persist/serialize path is invoked with capturing snapshot stores and metadata-only EC2 launches.

Surface sync

Internal persistence/reconcile correctness only. No new API surface, field, operation, or response shape; no behavior visible to clients changed. No SDK / docs / website / metadata change is needed.

---

Summary by cubic

Fixes four restart data-loss bugs by persisting ASG/EC2 state and Kinesis deliveries, and by guarding EC2 restart recovery. ASG instances and scheduler/pipe records now survive restarts, and EC2 state is no longer clobbered during recovery.

• Bug Fixes
  • CFN ASG instances lost on restart: the detached capacity reconcile now fires autoscaling and EC2 snapshot hooks after it mutates state; hooks are threaded via ContainerBackingHandles.with_snapshot_hooks and passed to cfn_reconcile_capacity.
  • ASG-launched EC2 not persisted: apply_capacity triggers an EC2 snapshot after launch/terminate; server wires with_ec2_snapshot_hook(...), and the hook runs before the ASG write to avoid a delete race.
  • EC2 restart recovery clobber: both success and error arms use a recovery_terminal_wins guard (codes 48/80) so concurrent Stop/Terminate wins and is not overwritten.
  • Scheduler/Pipes Kinesis deliveries not persisted: deliveries now use with_dirty_flag(kinesis_state, kinesis_delivery_dirty) instead of ::new so the background flusher persists them.

^{Written for commit 05886de. Summary will update on new commits.}

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.