ROX-15718: Prepare usage of WorkflowTemplates in flavors

[tommartensen]

With reusable WorkflowTemplates, we can refactor duplicated code between flavors, for example the wait step, or the openshift-perf-scale workflow CR.
It is also very useful if we want to make the infra flavors more flexible, e.g. by combining an empty cluster with a demo-ifyer + data seeder, as such a part could be implemented through a WorkflowTemplate with DRY principles in mind.

Other things this PR achieves:

• Cleaning the output of helm-{deploy, template, diff} by sending output to stderr
• installing Argo Workflows CRDs outside of the Helm lifecycle, see https://artifacthub.io/packages/helm/argo/argo-workflows#custom-resource-definitions
• update test-qa-demo flavor to not have ImagePullBackOff, which occurs because the docker/whalesay:latest image is not compatible with our modern GKE version

test-simulate is an example workflow that uses templates. During e2e testing for infractl logs, I observed that pod names are different for template referenced steps, which I fixed in the determinePodName method.

Validation

besides e2e tests, I deployed 5965462 to https://dev.infra.rox.systems/. Feel free to play around with it.

[rhacs-bot]

A single node development cluster (infra-pr-1553) was allocated in production infra for this PR.

CI will attempt to deploy quay.io/rhacs-eng/infra-server: to it.

🔌 You can connect to this cluster with:

gcloud container clusters get-credentials infra-pr-1553 --zone us-central1-a --project acs-team-temp-dev

🛠️ And pull infractl from the deployed dev infra-server with:

nohup kubectl -n infra port-forward svc/infra-server-service 8443:8443 &
make pull-infractl-from-dev-server

🚲 You can then use the dev infra instance e.g.:

bin/infractl -k -e localhost:8443 whoami

⚠️ Any clusters that you start using your dev infra instance should have a lifespan shorter then the development cluster instance. Otherwise they will not be destroyed when the dev infra instance ceases to exist when the development cluster is deleted. ⚠️

Further Development

☕ If you make changes, you can commit and push and CI will take care of updating the development cluster.

🚀 If you only modify configuration (chart/infra-server/configuration) or templates (chart/infra-server/{static,templates}), you can get a faster update with:

make helm-deploy

Logs

Logs for the development infra depending on your @redhat.com authuser:

• authuser=0
• authuser=1
• authuser=2

Or:

kubectl -n infra logs -l app=infra-server --tail=1 -f

[davdhacs]

+1 this looks great

[davdhacs]

• successful simulate workflow ran, and aks and gke clusters created+destroyed.