fix: workflows use 'onExit: stop' handler to catch all errors during create

[tommartensen]

Major changes:

• change all workflows to run their destroy template in the onExit handler, reducing leakages
• enhance e2e simulate workflow to upload a file to GCS in create and delete it in destroy, ensuring that destroy runs as intended in onExit handler.

Other changes:

• use a bigger cluster in PRs, because the testing with GCS showed performance bottlenecks for workflows :(
• cleanup smoke test list
• new default: skip create-delay-seconds and delete-delay-seconds in e2e simulate workflow, saving 20 seconds for many e2e tests

[rhacs-bot]

A single node development cluster (infra-pr-1543) was allocated in production infra for this PR.

CI will attempt to deploy quay.io/rhacs-eng/infra-server:0.11.17-13-g22d1263c55 to it.

🔌 You can connect to this cluster with:

gcloud container clusters get-credentials infra-pr-1543 --zone us-central1-a --project acs-team-temp-dev

🛠️ And pull infractl from the deployed dev infra-server with:

nohup kubectl -n infra port-forward svc/infra-server-service 8443:8443 &
make pull-infractl-from-dev-server

🚲 You can then use the dev infra instance e.g.:

bin/infractl -k -e localhost:8443 whoami

⚠️ Any clusters that you start using your dev infra instance should have a lifespan shorter then the development cluster instance. Otherwise they will not be destroyed when the dev infra instance ceases to exist when the development cluster is deleted. ⚠️

Further Development

☕ If you make changes, you can commit and push and CI will take care of updating the development cluster.

🚀 If you only modify configuration (chart/infra-server/configuration) or templates (chart/infra-server/{static,templates}), you can get a faster update with:

make helm-deploy

Logs

Logs for the development infra depending on your @redhat.com authuser:

• authuser=0
• authuser=1
• authuser=2

Or:

kubectl -n infra logs -l app=infra-server --tail=1 -f

[davdhacs]

+1

[davdhacs]

I think using an external semaphore, in gcs, is a great idea. 👍

[tommartensen]

I have smoke-tested all flavors manually, going to merge now.