[KFLUXINFRA-3003] Replace DR sleep-pipeline with nightly sanity test pipeline#13074
[KFLUXINFRA-3003] Replace DR sleep-pipeline with nightly sanity test pipeline#13074vsokolen-redhat wants to merge 1 commit into
Conversation
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: vsokolen-redhat The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Kustomize Render DiffComparing
Total: 3 components, +2607 -69 lines 📋 Full diff available in the workflow summary and as a downloadable artifact. |
PR Summary by QodoReplace DR sleep pipeline with nightly DR sanity test Tekton pipeline
AI Description
Diagram
High-Level Assessment
Files changed (5)
|
Code Review by Qodo
Context used 1. 2h deadline too short
|
![qodo-for-redhat-appstudio[bot]](https://avatars.githubusercontent.com/u/89927105?s=60&v=4){kind=small}
| taskRunTemplate: | ||
| serviceAccountName: dr-sanity-runner | ||
| timeouts: | ||
| pipeline: 2h |
There was a problem hiding this comment.
1. 2h deadline too short 🐞 Bug ☼ Reliability
The CronJob activeDeadlineSeconds: 7200 and PipelineRun timeouts.pipeline: 2h are shorter than the pipeline’s own worst-case sequential polling loops, so runs can be terminated mid-drill and may not reach cleanup. This can cause frequent false failures and leave disruptive resources behind (e.g., the image-controller NetworkPolicy).
Agent Prompt
### Issue description
CronJob and PipelineRun timeouts are set to 2 hours, but the pipeline tasks contain polling loops and run sequentially with a worst-case duration significantly greater than 2 hours. This can terminate the run before completion and skip/interrupt cleanup.
### Issue Context
- CronJob caps the Job runtime to 7200s.
- TriggerTemplate caps the PipelineRun pipeline timeout to 2h.
- Baseline + backup + restore + post-restore verification loops can exceed 2h in aggregate.
### Fix Focus Areas
- components/disaster-recovery/development/cronjob.yaml[8-15]
- components/disaster-recovery/development/triggertemplate.yaml[12-18]
- components/disaster-recovery/development/pipeline.yaml[97-200]
- components/disaster-recovery/development/pipeline.yaml[258-401]
- components/disaster-recovery/development/pipeline.yaml[466-552]
### What to change
- Either (A) increase `spec.jobTemplate.spec.activeDeadlineSeconds` and `PipelineRun.spec.timeouts.pipeline` to cover the worst-case sequential duration with headroom (e.g., 5–6h), or (B) reduce the polling limits so the total worst-case fits within 2h.
- If keeping a strict overall cap, ensure cleanup is handled out-of-band (or guaranteed to run before termination) so the NetworkPolicy and Velero objects aren’t left behind.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
There was a problem hiding this comment.
The e2e tests also set the timeout at 3h, so this is a valid point
| # NetworkPolicies — create/delete in image-controller namespace | ||
| - apiGroups: ["networking.k8s.io"] | ||
| resources: ["networkpolicies"] | ||
| verbs: ["create", "delete", "get"] |
There was a problem hiding this comment.
3. Networkpolicy rbac too narrow 🐞 Bug ≡ Correctness
The pipeline uses oc apply on a fixed NetworkPolicy name in the image-controller namespace, but the dr-sanity-runner ClusterRole does not grant patch/update on networkpolicies. If the policy already exists (e.g., leftover from a prior interrupted run), oc apply will fail and abort the drill.
Agent Prompt
### Issue description
`dr-disaster-and-restore` creates the `dr-block-image-controller` NetworkPolicy using `oc apply`, which requires update-style permissions when the object already exists. The ClusterRole currently grants only create/delete/get for NetworkPolicies.
### Issue Context
A previous run may leave the NetworkPolicy behind (especially under timeout/deadline termination), making reruns hit the existing-object path.
### Fix Focus Areas
- components/disaster-recovery/development/pipeline.yaml[301-316]
- components/disaster-recovery/development/rbac.yaml[91-95]
### What to change
- Add `patch` (and/or `update`) to the `networkpolicies` rule for the `dr-sanity-runner` ClusterRole.
- Alternatively, change the script to handle existing policies explicitly (e.g., `oc delete ... --ignore-not-found` then `oc create -f -`) if you want to avoid patch/update permissions.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
| finally: | ||
| - name: cleanup | ||
| taskRef: | ||
| kind: Task | ||
| name: sleep | ||
| name: dr-cleanup | ||
| params: | ||
| - name: TENANT_NAMESPACE | ||
| value: $(params.TENANT_NAMESPACE) | ||
| - name: BACKUP_NAME | ||
| value: $(tasks.backup.results.backup-name) |
There was a problem hiding this comment.
4. Backup cleanup can be skipped 🐞 Bug ☼ Reliability
Cleanup only deletes the Velero Backup when BACKUP_NAME is non-empty, but BACKUP_NAME is sourced from tasks.backup.results.backup-name, which won’t be available if the run fails before the result is written or if backup never ran. This can leak Backup CRs and their stored backups over time.
Agent Prompt
### Issue description
The finally cleanup deletes the Backup only if it receives `BACKUP_NAME`, but the pipeline passes this from a task result. If the result isn’t available (task not executed, or failed before writing results), cleanup won’t delete the Backup.
### Issue Context
Cleanup script explicitly skips backup deletion on empty BACKUP_NAME.
### Fix Focus Areas
- components/disaster-recovery/development/pipeline.yaml[558-595]
- components/disaster-recovery/development/pipeline.yaml[658-701]
### What to change
- Remove the dependency on a task result for identifying the Backup.
- Option A (recommended): define a deterministic backup name in the Pipeline (e.g., `dr-sanity-backup-$(context.pipelineRun.uid)`), pass it as a param to both the `backup` and `disaster-and-restore` tasks, and pass the same value to `cleanup`.
- Option B: have cleanup discover and delete backups/restores created by this run via labels/annotations (add labels when creating Backup/Restore), instead of relying on a result.
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
🤖 Pipeline Failure AnalysisCategory: Infrastructure The pipeline failed due to a timeout when deploying Konflux ArgoCD applications, primarily caused by missing Kubernetes resources and a degraded 📋 Technical DetailsImmediate CauseMultiple critical Konflux ArgoCD applications ( Contributing FactorsThe widespread failure of these applications to deploy was due to numerous missing Kubernetes resources required for their operation. A key contributing factor was the ImpactThis failure prevented the successful bootstrapping of the cluster for disaster recovery tests, blocking the pipeline from proceeding and leading to the overall job failure. 🔍 Evidenceappstudio-konflux-disaster-recovery/redhat-appstudio-konflux-disaster-recoveryCategory: Logs:
|
|
@flacatus @enkeefe00 @p8r-the-gr8 do you think this test pipeline should be integrated with ring deployments? |
…h, add patch verb to NetworkPolicy RBAC for oc apply idempotency, make cleanup discover backups by prefix pattern instead of relying on task result
cfa0c57 to
cbc1443
Compare
In the next fixVersion, yes. This is out of scope for our current one. |
|
/retest pull-ci-redhat-appstudio-infra-deployments-main-appstudio-konflux-disaster-recovery |
|
/test appstudio-konflux-disaster-recovery |
|
Unsupported PR languages |
|
/retest |
|
@vsokolen-redhat: The following tests failed, say
Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
5 participants
Summary
Replaces the placeholder
sleep-pipelinein thekonflux-disaster-recoverynamespace with a real DR sanity test pipeline that runs the proven manual drill againstreleng-test-product-tenanton a nightly schedule.Changes
dr-sanity-pipelineimplementing: preflight checks → baseline build/integration/release→ Velero backup → disaster simulation (NetworkPolicy + namespace deletion) → restore → post-restore build/integration/release verification → cleanup (openshift-adp), tenant namespace operations, release pipeline polling (rhtap-releng-tenant), and NetworkPolicy management (image-controller)activeDeadlineSeconds: 7200andstartingDeadlineSeconds: 300dr-sanity-runnerSA, 2h timeoutdurationparamContext
The full-chain DR drill was validated manually on 2026-07-15 (KFLUXINFRA-3914). This PR automates the same procedure. The Go e2e test suite (Meirav/Manish) will complement this from CI when ready ([Option C, confirmed in KFLUXINFRA-3003 comments](https://redhat.atlassian.net/browse/KFLUXINFRA-3003?focusedCommentId=17616945)).
Test plan
kustomize build components/disaster-recovery/staging/stone-stg-rh01/)oc get pipelineruns -n konflux-disaster-recovery)Jira
https://redhat.atlassian.net/browse/KFLUXINFRA-3003