ts-redis-cluster, upgrade failed today during pulumi destroy. All the other steps succeeded — we just couldn't destroy the stack. Pulumi reported the following error:
error deleting MemoryDB Cluster (ts-redis-cluster-u-3990398698-redis): InvalidClusterStateFault: Cluster ts-redis-cluster-u-3990398698-redis has status snapshotting which is not valid for deletion.
When I manually tried to delete the stack later, it succeeded. It looks like Pulumi just isn't correctly waiting for the state to change to non-snapshotting.
If this happens again, we should consider adding a between-destroy-retries.sh hook that finds the cluster and waits for its status to be something valid.
I'm not sure exactly what that looks like, because I just took down the cluster (🤦). But next time this happens, we should poke around with aws memorydb describe-clusters and figure out how to script the wait.
ts-redis-cluster, upgradefailed today duringpulumi destroy. All the other steps succeeded — we just couldn't destroy the stack. Pulumi reported the following error:When I manually tried to delete the stack later, it succeeded. It looks like Pulumi just isn't correctly waiting for the state to change to non-snapshotting.
If this happens again, we should consider adding a
between-destroy-retries.shhook that finds the cluster and waits for its status to be something valid.I'm not sure exactly what that looks like, because I just took down the cluster (🤦). But next time this happens, we should poke around with
aws memorydb describe-clustersand figure out how to script the wait.