Skip to content

[ISV-7093] guard hosted merge on base branch change#940

Open
mantomas wants to merge 3 commits intomainfrom
ISV-7093
Open

[ISV-7093] guard hosted merge on base branch change#940
mantomas wants to merge 3 commits intomainfrom
ISV-7093

Conversation

@mantomas
Copy link
Copy Markdown
Contributor

@mantomas mantomas commented Apr 28, 2026

Closes ISV-7093

Handles situation when multiple parallel PRs (and pipelines) are touching the same operator or catalog and merging one of them can make the other PR(s) invalid (e.g. multiple channel heads).

The solution creates fingerprint (hash) of the updated operator/catalog tree-ish in a new task right after detect-changes and is then used as a guard in merge-pr:

  • uses target branch tip (or reuse clone-repository-base if OID match) to create fingerprint
  • pass if the target tip oid does not change (nothing was merged during the pipeline run)
  • calculate and compare new fingerprint if base tip moved
    • pass if fingerprint match (PR related catalog or operator is unchanged)
    • guards the merge on fingerprint mismatch

Known limitations: as a naive solution, there is no actual opm validate, leaving some space for false positives (stated in the PR comment if guarded, pipeline re-run should fix it). This trade off is time effective, regular validation with catalog/index build will introduce another time-window for race condition.

Example guarded merge is here.

  • fixed by pipeline rerun as there was no actual conflict in this case

Merge Request Checklists

  • Development is done in feature branches
  • Code changes are submitted as pull request into a primary branch [Provide reason for non-primary branch submissions]
  • Code changes are covered with unit and integration tests.
  • Code passes all automated code tests:
    • Linting
    • Code formatter - Black
    • Security scanners
    • Unit tests
    • Integration tests
  • Code is reviewed by at least 1 team member
  • Pull request is tagged with "risk/good-to-go" label for minor changes

@qodo-code-review
Copy link
Copy Markdown

qodo-code-review Bot commented Apr 28, 2026

Review Summary by Qodo

(Agentic_describe updated until commit e386835)

Add merge-base lane fingerprint guard for parallel PR race condition detection

✨ Enhancement

Grey Divider

Walkthroughs

Description
• Implement merge-base lane fingerprint guard to prevent race conditions
  when multiple parallel PRs modify the same operator/catalog paths
• Create snapshot of operator/catalog file-level fingerprints at pipeline start
  using live target branch tip OID
• Verify fingerprint at merge time; block merge if base branch advanced and
  lane paths changed, with exit code 10 for pipeline re-run
• Add CLI tool and Tekton tasks for recording and verifying merge-base snapshots
  with SSH/HTTPS credential support
Diagram
flowchart LR
  detect["detect-changes"] --> record["record-merge-base-lane-snapshot"]
  record --> snapshot["snapshot.json<br/>base_oid + lane_fp"]
  snapshot --> merge["merge-pr task"]
  merge --> verify["merge-base-lane-fingerprint verify"]
  verify -->|tip unchanged| allow["Allow merge"]
  verify -->|tip changed, fp match| allow
  verify -->|tip changed, fp mismatch| block["Block merge<br/>exit 10"]
Loading

Grey Divider

File Changes

1. operatorcert/merge_base_lane.py ✨ Enhancement +326/-0

Core fingerprinting and git operations for merge guard

operatorcert/merge_base_lane.py


2. operatorcert/entrypoints/merge_base_lane_fingerprint.py ✨ Enhancement +118/-0

CLI entrypoint for record and verify subcommands

operatorcert/entrypoints/merge_base_lane_fingerprint.py


3. tests/operatorcert/test_merge_base_lane.py 🧪 Tests +421/-0

Comprehensive unit tests for fingerprinting logic

tests/operatorcert/test_merge_base_lane.py


View more (6)
4. tests/entrypoints/test_merge_base_lane_fingerprint.py 🧪 Tests +127/-0

CLI entrypoint smoke and error handling tests

tests/entrypoints/test_merge_base_lane_fingerprint.py


5. ansible/roles/operator-pipeline/templates/openshift/tasks/record-merge-base-lane-snapshot.yml ✨ Enhancement +144/-0

Tekton task to record snapshot at pipeline start

ansible/roles/operator-pipeline/templates/openshift/tasks/record-merge-base-lane-snapshot.yml


6. ansible/roles/operator-pipeline/templates/openshift/tasks/merge-pr.yml ✨ Enhancement +83/-2

Integrate merge-base guard verification before PR merge

ansible/roles/operator-pipeline/templates/openshift/tasks/merge-pr.yml


7. ansible/roles/operator-pipeline/templates/openshift/pipelines/operator-hosted-pipeline.yml ✨ Enhancement +44/-0

Add record-merge-base-lane-snapshot task to pipeline

ansible/roles/operator-pipeline/templates/openshift/pipelines/operator-hosted-pipeline.yml


8. pyproject.toml ⚙️ Configuration changes +1/-0

Register merge-base-lane-fingerprint CLI entrypoint

pyproject.toml


9. README.md 📝 Documentation +21/-0

Document merge base lane guard feature and limitations

README.md


Grey Divider

Qodo Logo

@qodo-code-review
Copy link
Copy Markdown

qodo-code-review Bot commented Apr 28, 2026

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (1)

Context used

Grey Divider


Action required

1. Guard misses pre-snapshot drift 📎 Requirement gap ☼ Reliability ⭐ New
Description
The merge-base guard snapshots the live target-branch tip even when it already differs from
git_commit_base, and the merge check only guards further drift after that snapshot. This can let
the hosted pipeline pass and merge even though another concurrent bundle merge already changed the
operator/catalog state the PR was validated against, risking post-merge incompatibility and multiple
channel heads.
Code

ansible/roles/operator-pipeline/templates/openshift/tasks/record-merge-base-lane-snapshot.yml[R110-134]

+        TIP_REF="refs/heads/$(params.git_base_branch)"
+        TIP_OID_REMOTE="$(git ls-remote "${REMOTE_URL}" "${TIP_REF}" | awk '{print $1; exit}')"
+        if [[ -z "${TIP_OID_REMOTE}" ]]; then
+          echo "ERROR: could not resolve ${TIP_REF} on remote"
+          exit 1
+        fi
+
+        # shared git config no matter which base used
+        export GIT_CONFIG_COUNT=1
+        export GIT_CONFIG_KEY_0=safe.directory
+
+        if [[ "${TIP_OID_REMOTE}" == "$(params.git_commit_base)" ]]; then
+          echo "Live target tip matches git_commit_base; reusing base workspace clone."
+          rm -rf "${TIP_CLONE}"
+          TIP_CLONE="${BASE_CLONE}"
+          TIP_OID="${TIP_OID_REMOTE}"
+          # safe.directory for base workspace (merge-base-lane-fingerprint); env only, no $HOME writes.
+          export GIT_CONFIG_VALUE_0="${TIP_CLONE}"
+        else
+          rm -rf "${TIP_CLONE}"
+          # safe.directory before git -C on the new clone (ownership vs step user).
+          export GIT_CONFIG_VALUE_0="${TIP_CLONE}"
+          git clone --depth 1 -b "$(params.git_base_branch)" "${REMOTE_URL}" "${TIP_CLONE}"
+          TIP_OID="$(git -C "${TIP_CLONE}" rev-parse HEAD)"
+        fi
Evidence
README clarifies that detect-changes still compares PR head to the clone at git_commit_base even
if the live base tip differs, while the snapshot baseline is taken from the live target-branch tip.
The snapshot task implements this by cloning/fingerprinting the live tip when it differs from
git_commit_base without failing or forcing re-validation, so a concurrent merge that happens
before snapshot time is not treated as a blocking incompatibility condition—contrary to the
checklist goals for concurrent-PR safety and preventing multiple channel heads.

Hosted pipeline validates compatibility with concurrent PRs for the same operator
Hosted pipeline prevents multiple channel heads caused by concurrent updates to the same operator
README.md[148-154]
ansible/roles/operator-pipeline/templates/openshift/tasks/record-merge-base-lane-snapshot.yml[110-134]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The merge-base lane guard only blocks merges when the base tip changes *after* the snapshot is recorded; if the base tip already advanced away from `git_commit_base` before the snapshot task runs, the pipeline can still proceed/merge without re-validating against the new base state.

## Issue Context
`detect-changes` uses a clone at `git_commit_base`, but the snapshot task intentionally fingerprints the live target-branch tip when it differs from `git_commit_base`. This means concurrent merges that occur earlier in the pipeline run may not be treated as a compatibility risk per the checklist objectives.

## Fix Focus Areas
- README.md[148-164]
- ansible/roles/operator-pipeline/templates/openshift/tasks/record-merge-base-lane-snapshot.yml[79-137]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

2. Read-only gitconfig breaks guard🐞 Bug ☼ Reliability
Description
In the merge-pr task, when the basic-auth workspace is bound, the script chmods $HOME/.gitconfig to
400 and then runs git config --global --add safe.directory ..., which requires writing to that
file and can fail. This can cause the merge-base guard (and subsequent git operations) to error out
and block merges unexpectedly.
Code

ansible/roles/operator-pipeline/templates/openshift/tasks/merge-pr.yml[R109-126]

+            if [[ "${WORKSPACE_BASIC_AUTH_DIRECTORY_BOUND}" == "true" ]]; then
+              cp "${WORKSPACE_BASIC_AUTH_DIRECTORY_PATH}/.git-credentials" "${HOME}/.git-credentials"
+              cp "${WORKSPACE_BASIC_AUTH_DIRECTORY_PATH}/.gitconfig" "${HOME}/.gitconfig"
+              chmod 400 "${HOME}/.git-credentials" "${HOME}/.gitconfig"
+            fi
+            if [[ "${WORKSPACE_SSH_DIRECTORY_BOUND}" == "true" ]]; then
+              cp -R "${WORKSPACE_SSH_DIRECTORY_PATH}" "${HOME}/.ssh"
+              chmod 700 "${HOME}/.ssh"
+              chmod -R 400 "${HOME}/.ssh"/*
+              eval "$(ssh-agent -s)"
+              for key in "${HOME}/.ssh/id_rsa" "${HOME}/.ssh/id_ed25519"; do
+                if [[ -f "${key}" ]]; then
+                  ssh-add "${key}"
+                fi
+              done
+            fi
+            git config --global --add safe.directory "$(workspaces.source.path)"
+            VERIFY_RC=0
Evidence
The task makes the copied global git config read-only before attempting to modify global git
configuration. git config --global writes to $HOME/.gitconfig, so the safe.directory addition can
fail when permissions are 400, and the guard then may fail due to missing safe.directory in
environments where Git enforces dubious-ownership protection.

ansible/roles/operator-pipeline/templates/openshift/tasks/merge-pr.yml[109-113]
ansible/roles/operator-pipeline/templates/openshift/tasks/merge-pr.yml[125-132]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`merge-pr` copies `.gitconfig` from the optional `basic-auth` workspace and immediately sets it to mode `400`, then runs `git config --global --add safe.directory ...`. This can fail because Git must write to `$HOME/.gitconfig`.

### Issue Context
This breaks the merge-base lane guard (and potentially other git operations) in environments where the `basic-auth` workspace is used (HTTPS auth) and where Git requires `safe.directory` to be set.

### Fix Focus Areas
- ansible/roles/operator-pipeline/templates/openshift/tasks/merge-pr.yml[109-126]

### Suggested changes
- Do not make `$HOME/.gitconfig` read-only before running `git config --global ...`.
- Options:
 - Move the `chmod` for `.gitconfig` to *after* the `git config --global --add safe.directory ...` call, or
 - Use a writable mode (e.g., `chmod 600 "$HOME/.gitconfig"`) so `git config --global` can update it.
- (Optional hardening) Check the return code of `git config --global ...` and fail fast with a clear message if it cannot be applied.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Previous review results

Review updated until commit e386835

Results up to commit 4a9513e


🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0)


Remediation recommended
1. Read-only gitconfig breaks guard🐞 Bug ☼ Reliability
Description
In the merge-pr task, when the basic-auth workspace is bound, the script chmods $HOME/.gitconfig to
400 and then runs git config --global --add safe.directory ..., which requires writing to that
file and can fail. This can cause the merge-base guard (and subsequent git operations) to error out
and block merges unexpectedly.
Code

ansible/roles/operator-pipeline/templates/openshift/tasks/merge-pr.yml[R109-126]

+            if [[ "${WORKSPACE_BASIC_AUTH_DIRECTORY_BOUND}" == "true" ]]; then
+              cp "${WORKSPACE_BASIC_AUTH_DIRECTORY_PATH}/.git-credentials" "${HOME}/.git-credentials"
+              cp "${WORKSPACE_BASIC_AUTH_DIRECTORY_PATH}/.gitconfig" "${HOME}/.gitconfig"
+              chmod 400 "${HOME}/.git-credentials" "${HOME}/.gitconfig"
+            fi
+            if [[ "${WORKSPACE_SSH_DIRECTORY_BOUND}" == "true" ]]; then
+              cp -R "${WORKSPACE_SSH_DIRECTORY_PATH}" "${HOME}/.ssh"
+              chmod 700 "${HOME}/.ssh"
+              chmod -R 400 "${HOME}/.ssh"/*
+              eval "$(ssh-agent -s)"
+              for key in "${HOME}/.ssh/id_rsa" "${HOME}/.ssh/id_ed25519"; do
+                if [[ -f "${key}" ]]; then
+                  ssh-add "${key}"
+                fi
+              done
+            fi
+            git config --global --add safe.directory "$(workspaces.source.path)"
+            VERIFY_RC=0
Evidence
The task makes the copied global git config read-only before attempting to modify global git
configuration. git config --global writes to $HOME/.gitconfig, so the safe.directory addition can
fail when permissions are 400, and the guard then may fail due to missing safe.directory in
environments where Git enforces dubious-ownership protection.

ansible/roles/operator-pipeline/templates/openshift/tasks/merge-pr.yml[109-113]
ansible/roles/operator-pipeline/templates/openshift/tasks/merge-pr.yml[125-132]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`merge-pr` copies `.gitconfig` from the optional `basic-auth` workspace and immediately sets it to mode `400`, then runs `git config --global --add safe.directory ...`. This can fail because Git must write to `$HOME/.gitconfig`.

### Issue Context
This breaks the merge-base lane guard (and potentially other git operations) in environments where the `basic-auth` workspace is used (HTTPS auth) and where Git requires `safe.directory` to be set.

### Fix Focus Areas
- ansible/roles/operator-pipeline/templates/openshift/tasks/merge-pr.yml[109-126]

### Suggested changes
- Do not make `$HOME/.gitconfig` read-only before running `git config --global ...`.
- Options:
 - Move the `chmod` for `.gitconfig` to *after* the `git config --global --add safe.directory ...` call, or
 - Use a writable mode (e.g., `chmod 600 "$HOME/.gitconfig"`) so `git config --global` can update it.
- (Optional hardening) Check the return code of `git config --global ...` and fail fast with a clear message if it cannot be applied.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Qodo Logo

@mantomas mantomas marked this pull request as draft April 29, 2026 10:33
@openshift-ci openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 29, 2026
@mantomas mantomas marked this pull request as ready for review April 30, 2026 08:19
@openshift-ci openshift-ci Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 30, 2026
@qodo-code-review
Copy link
Copy Markdown

qodo-code-review Bot commented Apr 30, 2026

Persistent review updated to latest commit e386835

Comment on lines +110 to +134
TIP_REF="refs/heads/$(params.git_base_branch)"
TIP_OID_REMOTE="$(git ls-remote "${REMOTE_URL}" "${TIP_REF}" | awk '{print $1; exit}')"
if [[ -z "${TIP_OID_REMOTE}" ]]; then
echo "ERROR: could not resolve ${TIP_REF} on remote"
exit 1
fi

# shared git config no matter which base used
export GIT_CONFIG_COUNT=1
export GIT_CONFIG_KEY_0=safe.directory

if [[ "${TIP_OID_REMOTE}" == "$(params.git_commit_base)" ]]; then
echo "Live target tip matches git_commit_base; reusing base workspace clone."
rm -rf "${TIP_CLONE}"
TIP_CLONE="${BASE_CLONE}"
TIP_OID="${TIP_OID_REMOTE}"
# safe.directory for base workspace (merge-base-lane-fingerprint); env only, no $HOME writes.
export GIT_CONFIG_VALUE_0="${TIP_CLONE}"
else
rm -rf "${TIP_CLONE}"
# safe.directory before git -C on the new clone (ownership vs step user).
export GIT_CONFIG_VALUE_0="${TIP_CLONE}"
git clone --depth 1 -b "$(params.git_base_branch)" "${REMOTE_URL}" "${TIP_CLONE}"
TIP_OID="$(git -C "${TIP_CLONE}" rev-parse HEAD)"
fi
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. Guard misses pre-snapshot drift 📎 Requirement gap ☼ Reliability

The merge-base guard snapshots the live target-branch tip even when it already differs from
git_commit_base, and the merge check only guards further drift after that snapshot. This can let
the hosted pipeline pass and merge even though another concurrent bundle merge already changed the
operator/catalog state the PR was validated against, risking post-merge incompatibility and multiple
channel heads.
Agent Prompt
## Issue description
The merge-base lane guard only blocks merges when the base tip changes *after* the snapshot is recorded; if the base tip already advanced away from `git_commit_base` before the snapshot task runs, the pipeline can still proceed/merge without re-validating against the new base state.

## Issue Context
`detect-changes` uses a clone at `git_commit_base`, but the snapshot task intentionally fingerprints the live target-branch tip when it differs from `git_commit_base`. This means concurrent merges that occur earlier in the pipeline run may not be treated as a compatibility risk per the checklist objectives.

## Fix Focus Areas
- README.md[148-164]
- ansible/roles/operator-pipeline/templates/openshift/tasks/record-merge-base-lane-snapshot.yml[79-137]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The snapshot step always resolves the live refs/heads/<git_base_branch> tip via ls-remote and records base_oid from that tip (reusing the base workspace only when that OID equals git_commit_base; otherwise it shallow-clones that tip and fingerprints HEAD). merge-base-lane-fingerprint verify compares merge-time ls-remote to base_oid from the snapshot, so drift after the snapshot is detected relative to the tip we actually fingerprinted, including the case where the branch was already ahead of git_commit_base before the snapshot ran.
What stays at git_commit_base is detect-changes and the base workspace used there—that’s the intentional scope called out in the README: lane-level fingerprinting, not a second full validation pass on the newer tip.

@mantomas mantomas self-assigned this Apr 30, 2026
@mantomas mantomas requested review from bclindner and tomasbakk April 30, 2026 09:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant