Skip to content

fix: workflows use 'onExit: stop' handler to catch all errors during create #1543

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tommartensen merged 13 commits into from
Apr 24, 2025
Merged

fix: workflows use 'onExit: stop' handler to catch all errors during create #1543

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
dbc005f
simulate flavor should have 'onExit: stop' handler
tommartensen Mar 28, 2025
2c26a76
update test workflows
tommartensen Mar 28, 2025
31ef9f2
update remaining workflows and smoke-test
tommartensen Mar 28, 2025
37e0a8e
fix input to gke workflows
tommartensen Mar 28, 2025
da979eb
upload and delete a file in the simulate workflow
tommartensen Apr 9, 2025
f7aad2f
update remaining tests
tommartensen Apr 9, 2025
ac8b082
add comments
tommartensen Apr 9, 2025
5a22235
skip GCS test if not required
tommartensen Apr 16, 2025
78cb2e7
Merge branch 'master' into tm/onExit-workflows
tommartensen Apr 16, 2025
e14d256
make it conditionally in GCS
tommartensen Apr 16, 2025
a61ccab
skipping sleep by default, making tests 20s faster by default
tommartensen Apr 16, 2025
eb44384
fix consequential typo
tommartensen Apr 16, 2025
22d1263
more oomph
tommartensen Apr 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/PR.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ jobs:
with:
flavor: gke-default
name: infra-pr-${{ github.event.pull_request.number }}
args: machine-type=e2-medium,nodes=3,gcp-image-type=ubuntu_containerd
args: machine-type=e2-standard-4,nodes=3,gcp-image-type=ubuntu_containerd
lifespan: ${{ github.actor == 'dependabot[bot]' && '1h' || '24h' }}
wait: true
token: ${{ secrets.INFRA_TOKEN }}
Expand Down
3 changes: 1 addition & 2 deletions .github/workflows/smoke-test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,10 +46,9 @@ jobs:
{name: "ocp-4", "args": "", "uniqueness": "g"},
{name: "ocp-4-demo", "args": "", "uniqueness": "h"},
{name: "ocp-4-perf-scale", "args": "", "uniqueness": "i"},
{name: "ocp-3", "args": "", "uniqueness": "j"},
{name: "osd-on-aws", "args": "", "uniqueness": "k"},
{name: "osd-on-gcp", "args": "", "uniqueness": "l"},
{name: "qa-demo", "args": "main-image=quay.io/rhacs-eng/main:4.3.4", "uniqueness": "m"},
{name: "qa-demo", "args": "main-image=quay.io/rhacs-eng/main:4.7.0", "uniqueness": "m"},
{name: "rosa", "args": "", "uniqueness": "n"},
{name: "rosahcp", "args": "", "uniqueness": "o"},
]
Expand Down
8 changes: 6 additions & 2 deletions chart/infra-server/static/flavors.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1243,20 +1243,24 @@
value: ""
- name: create-delay-seconds
description: how long to spend in create
value: "10"
value: "0"
kind: optional
- name: create-outcome
description: success or fail
value: success
kind: optional
- name: destroy-delay-seconds
description: how long to spend in destroy
value: "10"
value: "0"
kind: optional
- name: destroy-outcome
description: success or fail
value: success
kind: optional
- name: test-gcs
description: whether to upload or delete a test artifact in GCS
value: "false"
kind: optional

################
# Test Janitor #
Expand Down
3 changes: 3 additions & 0 deletions chart/infra-server/static/test-connect-artifact.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ metadata:
generateName: test-connect-artifact-
spec:
entrypoint: start
onExit: stop
arguments:
parameters:
- name: name
Expand All @@ -16,6 +17,8 @@ spec:
template: create
- - name: wait
template: wait
- name: stop
steps:
- - name: destroy
template: destroy

Expand Down
5 changes: 4 additions & 1 deletion chart/infra-server/static/test-gke-lite.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ metadata:
generateName: gke-lite-
spec:
entrypoint: start
onExit: stop
arguments:
parameters:
- name: name
Expand All @@ -30,12 +31,14 @@ spec:
template: create
- - name: wait
template: wait
- name: stop
steps:
- - name: destroy
template: destroy
arguments:
parameters:
- name: name
value: '{{ "{{" }}steps.create.outputs.parameters.cluster_name{{ "}}" }}'
value: '{{ "{{" }}workflow.parameters.name{{ "}}" }}'

- name: create
activeDeadlineSeconds: 3600
Expand Down
3 changes: 3 additions & 0 deletions chart/infra-server/static/test-qa-demo.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ metadata:
generateName: test-qa-demo-
spec:
entrypoint: start
onExit: stop
arguments:
parameters:
- name: name
Expand All @@ -15,6 +16,8 @@ spec:
template: whalesay
- - name: wait
template: wait
- name: stop
steps:
- - name: destroy
template: whalesay

Expand Down
62 changes: 56 additions & 6 deletions chart/infra-server/static/test-simulate.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ metadata:
generateName: simulate-
spec:
entrypoint: start
onExit: stop
arguments:
parameters:
- name: name
Expand All @@ -15,6 +16,13 @@ spec:
value: ""
- name: destroy-outcome
value: ""
- name: test-gcs
value: ""

volumes:
- name: credentials
secret:
secretName: google-credentials

templates:
- name: start
Expand All @@ -27,8 +35,12 @@ spec:
value: '{{ "{{" }}workflow.parameters.create-delay-seconds{{ "}}" }}'
- name: outcome
value: '{{ "{{" }}workflow.parameters.create-outcome{{ "}}" }}'
- name: test-gcs
value: '{{ "{{"}}workflow.parameters.test-gcs{{ "}}" }}'
- - name: wait
template: wait
- name: stop
steps:
- - name: destroy
template: simulate
arguments:
Expand All @@ -37,24 +49,62 @@ spec:
value: '{{ "{{" }}workflow.parameters.destroy-delay-seconds{{ "}}" }}'
- name: outcome
value: '{{ "{{" }}workflow.parameters.destroy-outcome{{ "}}" }}'
- name: test-gcs
value: '{{ "{{"}}workflow.parameters.test-gcs{{ "}}" }}'

- name: simulate
inputs:
parameters:
- name: delay-seconds
- name: outcome
- name: test-gcs
script:
image: debian:9.4
image: gcr.io/google.com/cloudsdktool/google-cloud-cli:stable
command: [bash]
source: |
set -x
start=0
while sleep 1; do
if [[ $((start++)) -ge {{ "{{" }}inputs.parameters.delay-seconds{{ "}}" }} ]]; then
break

delay() {
start=0
while sleep 1; do
if [[ $((start++)) -ge {{ "{{" }}inputs.parameters.delay-seconds{{ "}}" }} ]]; then
break
fi
done
}

upload_or_delete_gcs_object() {
gcloud auth activate-service-account --key-file /tmp/google-credentials.json
gcloud config set core/disable_prompts True

BUCKET_NAME="infra-e2e-upload-test"
FILE="{{ "{{" }}workflow.name{{ "}}" }}"
touch "${FILE}"

DESTINATION="gs://${BUCKET_NAME}/${FILE}"

if gsutil -q stat "${DESTINATION}"; then
echo "File exists. Deleting..."
gsutil rm "${DESTINATION}"
else
echo "File does not exist. Proceeding to upload."
gsutil cp "${FILE}" "${DESTINATION}"
fi
done
}

if [[ "{{ "{{" }}inputs.parameters.test-gcs{{ "}}" }}" == "true" ]]; then
upload_or_delete_gcs_object

@davdhacs davdhacs Apr 17, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think using an external semaphore, in gcs, is a great idea. 👍

fi

if [[ {{ "{{" }}inputs.parameters.delay-seconds{{ "}}" }} -gt 0 ]]; then
delay
fi

[[ "{{ "{{" }}inputs.parameters.outcome{{ "}}" }}" == "success" ]] || exit 1

volumeMounts:
- name: credentials
mountPath: /tmp

- name: wait
suspend: {}
3 changes: 3 additions & 0 deletions chart/infra-server/static/test-url-artifact.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ metadata:
generateName: test-url-artifact-
spec:
entrypoint: start
onExit: stop
arguments:
parameters:
- name: name
Expand All @@ -16,6 +17,8 @@ spec:
template: create
- - name: wait
template: wait
- name: stop
steps:
- - name: destroy
template: destroy

Expand Down
3 changes: 3 additions & 0 deletions chart/infra-server/static/workflow-aks.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ metadata:
generateName: aks-
spec:
entrypoint: start
onExit: stop
arguments:
parameters:
- name: name
Expand All @@ -27,6 +28,8 @@ spec:
- - name: wait
template: wait

- name: stop
steps:
- - name: destroy
template: destroy

Expand Down
3 changes: 3 additions & 0 deletions chart/infra-server/static/workflow-eks.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ metadata:
generateName: eks-
spec:
entrypoint: start
onExit: stop
arguments:
parameters:
- name: name
Expand All @@ -27,6 +28,8 @@ spec:
- - name: wait
template: wait

- name: stop
steps:
- - name: destroy
template: destroy

Expand Down
6 changes: 5 additions & 1 deletion chart/infra-server/static/workflow-gke-default.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ metadata:
generateName: gke-default-
spec:
entrypoint: start
onExit: stop
arguments:
parameters:
- name: name
Expand All @@ -30,12 +31,15 @@ spec:
template: create
- - name: wait
template: wait

- name: stop
steps:
- - name: destroy
template: destroy
arguments:
parameters:
- name: name
value: '{{ "{{" }}steps.create.outputs.parameters.cluster_name{{ "}}" }}'
value: '{{ "{{" }}workflow.parameters.name{{ "}}" }}'

- name: create
activeDeadlineSeconds: 3600
Expand Down
3 changes: 3 additions & 0 deletions chart/infra-server/static/workflow-openshift-4-demo.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ metadata:
generateName: openshift-4-demo-
spec:
entrypoint: start
onExit: stop
arguments:
parameters:
- name: name
Expand Down Expand Up @@ -74,6 +75,8 @@ spec:
- - name: wait
template: wait

- name: stop
steps:
- - name: destroy
template: destroy

Expand Down
3 changes: 3 additions & 0 deletions chart/infra-server/static/workflow-openshift-4-perf-scale.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ metadata:
generateName: openshift-4-perf-scale-
spec:
entrypoint: start
onExit: stop
arguments:
parameters:
- name: name
Expand Down Expand Up @@ -50,6 +51,8 @@ spec:
- - name: wait
template: wait

- name: stop
steps:
- - name: destroy
template: destroy

Expand Down
3 changes: 3 additions & 0 deletions chart/infra-server/static/workflow-openshift-4.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ metadata:
generateName: openshift-4-
spec:
entrypoint: start
onExit: stop
arguments:
parameters:
- name: name
Expand Down Expand Up @@ -51,6 +52,8 @@ spec:
- - name: wait
template: wait

- name: stop
steps:
- - name: destroy
template: destroy

Expand Down
3 changes: 3 additions & 0 deletions chart/infra-server/static/workflow-openshift-aro.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ metadata:
generateName: aro-
spec:
entrypoint: start
onExit: stop
arguments:
parameters:
- name: name
Expand Down Expand Up @@ -32,6 +33,8 @@ spec:
- - name: wait
template: wait

- name: stop
steps:
- - name: destroy
template: destroy

Expand Down
3 changes: 3 additions & 0 deletions chart/infra-server/static/workflow-openshift-ibmroks.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ metadata:
generateName: roks-
spec:
entrypoint: start
onExit: stop
arguments:
parameters:
- name: name
Expand Down Expand Up @@ -39,6 +40,8 @@ spec:
- - name: wait
template: wait

- name: stop
steps:
- - name: destroy
template: destroy

Expand Down
3 changes: 3 additions & 0 deletions chart/infra-server/static/workflow-openshift-rosa-hcp.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ metadata:
generateName: rosa-
spec:
entrypoint: start
onExit: stop
arguments:
parameters:
- name: name
Expand Down Expand Up @@ -40,6 +41,8 @@ spec:
- - name: wait
template: wait

- name: stop
steps:
- - name: destroy
template: destroy

Expand Down
Loading