Move E2E test workflow to Elastx

[HaoruiPeng]

Warning

This is a public repository, ensure not to disclose:

• personal data beyond what is necessary for interacting with this pull request, nor
• business confidential information, such as customer names.

What kind of PR is this?

Required: Mark one of the following that is applicable:

• kind/feature
• [] kind/improvement
• kind/deprecation
• kind/documentation
• kind/clean-up
• kind/bug
• kind/other

Optional: Mark one or more of the following that are applicable:

Important

Breaking changes should be marked kind/admin-change or kind/dev-change depending on type
Critical security fixes should be marked with kind/security

• kind/admin-change
• kind/dev-change
• kind/security
• [kind/adr](set-me)

What does this PR do / why do we need this PR?

Fixes #2010

This new workflow will run a full Welkin cluster installation with Kubespray on Elastx and run E2E tests will our end-to-end test suite.

Note:

• I'm running the tests on my own fork repository right now, so some configurations are specified to my forked repo. Thoes will be change back to elastisys apps repo before merging. Some other changes will also need to be made in the workflow before merging, they are noted in the code comments and marked as TODO
• I had a discussion with @aarnq , and decided to create the clusters without using kube-oidc-login, so only admin.config is used as KUBECONFIG for both sc and wc. This caused some tests failures since They are configured to use google connector in dex. We need to adjust the e2d tests a bit to allow static dex login for platform admin as well. The will need another task to do this and to adjust the RBAC configuration in for the pipeline CK8S_CONFIG_PATH.
• Despite of failing the test steps, the pipeline works for creating the whole cluster with existing configuration. However the cluster creation may fail due to issues on Elastx (See discussion here) . I'm now using v2-k1 volumes and v2-c4-m8-d120 image flavor, but sometimes still got timeout error when running Kubespray (when joining new CPs) and installing apps. The volume and flavor types can be adjust accordingly before merging.
• The workflow is configured to run in an environment named test (we can change the name later). The following environment secrets and variables should be configured:
  • Secrets:

  # For creating DNS records
  AWS_DNS_ACCESS_KEY_ID
  AWS_DNS_SECRET_ACCESS_KEY
  #For creating S3 buckets on Elastx 
  ELASTX_API_ACCESS_KEY
  ELASTX_API_SECRET_KEY
  # For configuring dex, but we don't really need to if we remove google connector at all
  GCP_CLIENT_ID
  GCP_CLIENT_SECRET
  GOOGLE_SA
  
  # Elastx Openstack OS_password 
  OS_PASSWORD
  
  # The workflow will create new GPG key each run, this is input passphrase for the new GPG key
  PGP_PASSPHRASE

  • Variables:

  # Elastx Openstack 
  OS_AUTH_URL
  OS_IDENTITY_API_VERSION
  OS_INTERFACE
  OS_PROJECT_ID
  OS_PROJECT_NAME
  OS_REGION_NAME
  OS_USERNAME
  OS_USER_DOMAIN_NAME
  
  # Email for GPG creation
  PGP_EMAIL

Information to reviewers

Checklist

• Proper commit message prefix on all commits
• Change checks:
  • The change is transparent
  • The change is disruptive
  • The change requires no migration steps
  • The change requires migration steps
  • The change updates CRDs
  • The change updates the config and the schema
• Documentation checks:
  • The public documentation required no updates
  • The public documentation required an update - [link to change](set-me)
• Metrics checks:
  • The metrics are still exposed and present in Grafana after the change
  • The metrics names didn't change (Grafana dashboards and Prometheus alerts required no updates)
  • The metrics names did change (Grafana dashboards and Prometheus alerts required an update)
• Logs checks:
  • The logs do not show any errors after the change
• PodSecurityPolicy checks:
  • Any changed Pod is covered by Kubernetes Pod Security Standards
  • Any changed Pod is covered by Gatekeeper Pod Security Policies
  • The change does not cause any Pods to be blocked by Pod Security Standards or Policies
• NetworkPolicy checks:
  • Any changed Pod is covered by Network Policies
  • The change does not cause any dropped packets in the NetworkPolicy Dashboard
• Audit checks:
  • The change does not cause any unnecessary Kubernetes audit events
  • The change requires changes to Kubernetes audit policy
• Falco checks:
  • The change does not cause any alerts to be generated by Falco
• Bug checks:
  • The bug fix is covered by regression tests