E2E: Rerun Failswitch from failure point failing in nightly

[subhashkhileri]

Classification

fix_category: product_bug

Failed Tests

Test	Error
Rerun Failswitch from failure point	expect(locator).toBeVisible() failed — getByText('Completed', { exact: true }) not found after 120s

Root Cause

The SonataFlow "rerun from failure point" feature does not re-read environment variables from the pod. The test:

1. Patches the failswitch SonataFlow CR's HTTPBIN env var to https://foobar.org/ (an invalid URL)
2. Runs the workflow → it fails as expected (SSL cert mismatch on httpbinGet step)
3. Patches HTTPBIN back to https://httpbin.org/ and waits for the pod to pick it up
4. Clicks "Run again → From failure point"
5. Expects the workflow to complete

But the rerun still fails with: javax.net.ssl.SSLPeerUnverifiedException - Certificate for <foobar.org> doesn't match any of the subject alternative names: [www.foobar.org]

The "from failure point" rerun replays the workflow using the process context saved at the time of the original failure, which still contains the old foobar.org URL. The updated env var is not picked up because the workflow engine restores saved process variables rather than re-reading from the environment.

This is a bug in the orchestrator/SonataFlow "rerun from failure point" feature — when a workflow is retried from the failure point, environment variables that changed since the original execution should be re-evaluated.

Remediation

The orchestrator plugin's "rerun from failure point" implementation needs to re-resolve env-var-backed workflow inputs when retrying, rather than using the cached process context from the failed execution. This fix belongs in the orchestrator plugin source code, not in this repo.

Artifacts

https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-redhat-developer-rhdh-plugin-export-overlays-main-e2e-ocp-helm-nightly/2067457860433874944

[subhashkhileri]

Attempt 1 — Corrected Analysis

Classification: fix_category: test_fix (corrected from product_bug)

The previous analysis concluded that SonataFlow's "rerun from failure point" caches old env vars. This is incorrect. The workflow log from this run disproves it:

The failswitch pod restarted at 05:01:47 with the correct HTTPBIN=https://httpbin.org/ env var. The retrigger happened at 05:02:04 — only 17 seconds after startup. The error was 503 Service Temporarily Unavailable (an nginx/proxy 503 HTML page), NOT the foobar.org SSL certificate error. The SSL error visible in the UI Results section is from the original failure, not the retrigger.

Root cause: After patchHttpbin restarts the failswitch deployment, the pod reaches Running state and the env var is confirmed, but the Quarkus/SonataFlow application inside the pod hasn't fully initialized its HTTP endpoints. The retrigger call hits this window and gets 503.

Fix: Add a health endpoint poll (/q/health/ready) after waitForRunningPodHttpbin in patchHttpbin() to ensure the SonataFlow management API is fully ready before returning.

Artifacts: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-redhat-developer-rhdh-plugin-export-overlays-main-e2e-ocp-helm-nightly/2067457860433874944

[subhashkhileri]

Fix Attempt 1

PR: #2639
Approach: Add health endpoint polling after pod restart in patchHttpbin()

After the failswitch deployment restarts, the pod may report Running to K8s before the Quarkus/SonataFlow application has fully initialized its HTTP endpoints. The new waitForHealthReady() function polls /q/health/ready (up to 60s) to ensure the management API is ready before the test proceeds with the retrigger action.

This directly addresses the 503 error seen in the workflow log — the retrigger was hitting the management API during the application startup window.