test(spanner): make samples instance and backup cleanup robust against FailedPrecondition and quota exhaustion (googleapis#17141)

Thank you for opening a Pull Request! Before submitting your PR, there
are a few things you can do to make sure it goes smoothly:
- [ ] Make sure to open an issue as a
[bug/issue](https://github.com/googleapis/google-cloud-python/issues)
before writing your code! That way we can discuss the change, evaluate
designs, and agree on the general idea
- [ ] Ensure the tests and linter pass
- [ ] Code coverage does not decrease (if any source code was changed)
- [ ] Appropriate docs were updated (if necessary)

### Root Cause
During Kokoro continuous test runs for Cloud Spanner samples, tests
restore databases from backups. The restored databases hold active
backend references to the backups (`backup.referencing_databases`) while
optimizing in the background.
During module teardown, `conftest.py` calls `database.drop()` and
immediately iterates to `backup.delete()`. Because `database.drop()` is
asynchronous on the Spanner backend, `backup.delete()` frequently fails
with a `FailedPrecondition` error (backup in use). This unhandled
exception crashes the fixture teardown generator, skipping
`instance.delete()` and leaking instances and backups.
As these leaked resources accumulate across frequent Kokoro runs in the
shared test project, subsequent runs fail due to `ResourceExhausted`
(quota exceeded) errors.
### Solution
- Created a robust retry helper (`retry_cleanup`) that retries on both
`ResourceExhausted` and `FailedPrecondition`.
- Isolated individual database drops and backup deletions within
`try...except exceptions.GoogleAPICallError` blocks so that an error on
a single resource never aborts the scrubbing generator.
- Unified this robust scrubbing logic across `cleanup_old_instances`,
`sample_instance`, and `multi_region_instance` teardown fixtures.

Author: sakthivelmanii

packages/google-cloud-spanner/samples/samples/conftest.py

@@ -29,6 +29,9 @@
 OPERATION_TIMEOUT_SECONDS = 120  # seconds
 
 retry_429 = retry.RetryErrors(exceptions.ResourceExhausted, delay=15)
+retry_cleanup = retry.RetryErrors(
+    (exceptions.ResourceExhausted, exceptions.FailedPrecondition), max_tries=6, delay=15
+)
 
 
 @pytest.fixture(scope="module")
@@ -60,12 +63,22 @@ def spanner_client():
 
 
 def scrub_instance_ignore_not_found(to_scrub):
-    """Helper for func:`cleanup_old_instances`"""
+    """Robustly delete an instance, its databases, and backups."""
     try:
+        for database_pb in to_scrub.list_databases():
+            try:
+                database.Database.from_pb(database_pb, to_scrub).drop()
+            except exceptions.GoogleAPICallError:
+                pass
+
         for backup_pb in to_scrub.list_backups():
-            backup.Backup.from_pb(backup_pb, to_scrub).delete()
+            try:
+                b = backup.Backup.from_pb(backup_pb, to_scrub)
+                retry_cleanup(b.delete)()
+            except exceptions.GoogleAPICallError:
+                pass
 
-        retry_429(to_scrub.delete)()
+        retry_cleanup(to_scrub.delete)()
     except exceptions.NotFound:
         pass
 
@@ -154,13 +167,7 @@ def sample_instance(
 
     yield sample_instance
 
-    for database_pb in sample_instance.list_databases():
-        database.Database.from_pb(database_pb, sample_instance).drop()
-
-    for backup_pb in sample_instance.list_backups():
-        backup.Backup.from_pb(backup_pb, sample_instance).delete()
-
-    sample_instance.delete()
+    scrub_instance_ignore_not_found(sample_instance)
 
 
 @pytest.fixture(scope="module")
@@ -189,13 +196,7 @@ def multi_region_instance(
 
     yield multi_region_instance
 
-    for database_pb in multi_region_instance.list_databases():
-        database.Database.from_pb(database_pb, multi_region_instance).drop()
-
-    for backup_pb in multi_region_instance.list_backups():
-        backup.Backup.from_pb(backup_pb, multi_region_instance).delete()
-
-    multi_region_instance.delete()
+    scrub_instance_ignore_not_found(multi_region_instance)
 
 
 @pytest.fixture(scope="module")