Sleep 0.5s between component new and component delete

We seem to have a race condition between the component Git repo getting
initialized by `component new` and deleted by `component delete`.

Without the sleep, the test fails at least one in ten times (initial
local numbers are between 9/100 and 24/100 for separate runs). The
failure seems to be caused because something apparently is still
creating files in the bare component checkout while `component delete`
is deleting the bare checkout directory.

We observe a slightly increased number of test failures increases
(locally: approx 1-2 / 100) when we assert that the bare checkout is
fully deleted after `component delete`, in some cases `shutil.rmtree()`
apparently succeeds but the directory remains.

We've landed on 0.5s sleep after trying a couple numbers. We've seen no
failures in ~8000 consecutive test runs with 0.5s sleep. For CI, a
single 0.5s delay in one test (that already takes 1.7-1.8s in isolation
without the sleep) out of >1000 should be acceptable.

Note that we don't actually understand the root cause here, but the
failing test is a synthetic sequence of commands that should almost
never happen in real usage without at least 1s delay between them.

Additionally note that the flakiness disappears when we switch the test
to `subprocess.call()` instead of click's `CliRunner` even without the
sleep( or at least is rare enough that it doesn't show up in >1000
runs).

Author: simu

tests/test_component_template.py

@@ -9,6 +9,7 @@
 
 import click
 import pytest
+import time
 import shutil
 import yaml
 from pathlib import Path as P
@@ -858,6 +859,14 @@ def test_run_component_new_then_delete(tmp_path: P, cli_runner: RunnerFunc):
         / f".repos/github.com/projectsyn/component-{component_name}.git"
     ).exists()
 
+    # NOTE(sg): This test is flaky without this sleep because something continues to create files in
+    # the component's Git repo directory after the `cli_runner()` call for `component create`
+    # returns. After some testing, we've settled on 0.5s sleep which is enough to not see a single
+    # failure in approx 8000 back-to-back runs. Using `subprocess.call()` instead of the click
+    # `CliRunner` seems to also fix (or hide) the issue, but two `subprocess.call()` are actually
+    # slower than sleeping here.
+    time.sleep(0.5)
+
     result = cli_runner(
         ["-d", tmp_path, "-vvv", "component", "delete", "--force", component_name]
     )