Skip to content

fix(ci): compare coverage against merge-base commit, not latest main#2117

Open
jordanstephens wants to merge 1 commit into
mainfrom
fix/coverage-regression-merge-base
Open

fix(ci): compare coverage against merge-base commit, not latest main#2117
jordanstephens wants to merge 1 commit into
mainfrom
fix/coverage-regression-merge-base

Conversation

@jordanstephens
Copy link
Copy Markdown
Member

@jordanstephens jordanstephens commented May 15, 2026

Summary

  • Dependabot PRs (and any branch that diverges from main) were failing the coverage regression check because the baseline was always the most recent main artifact. If new tested code landed on main after the branch was created, main's coverage would rise while the branch stayed flat — a false regression.
  • Now the check finds the GitHub Actions run whose head_sha matches the git merge-base of the PR branch and main, and downloads the coverage artifact from that specific run.
  • Falls back gracefully (continue-on-error) when no matching run exists (first run, expired artifact, API error) — identical behavior to before in those cases.

Test plan

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Chores
    • Enhanced the continuous integration workflow's coverage baseline retrieval mechanism with improved reliability and error handling for more accurate coverage comparisons.

Review Change Stack

Dependabot PRs (and any branch that diverges from main) were failing the
coverage regression check because the baseline was always the most recent
main artifact. If new tested code landed on main after the branch was
created, coverage on main would rise while the branch stayed flat.

Instead, find the GitHub Actions run whose head_sha matches the git
merge-base of the PR branch and main, and download the coverage artifact
from that specific run. This ensures the comparison is always
branch-point vs. branch, not latest-main vs. branch.

Falls back gracefully (continue-on-error) when no matching run exists
(first run, expired artifact, API error).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@jordanstephens jordanstephens requested a review from raphaeltm as a code owner May 15, 2026 18:34
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 15, 2026

📝 Walkthrough

Walkthrough

The workflow now uses merge-base commit detection to identify the baseline coverage run instead of always pulling from the main branch. It adds actions: read permission to query workflow runs and implements shell logic to find the matching successful run and download its artifact.

Changes

CI Coverage Baseline Detection

Layer / File(s) Summary
Coverage baseline detection via merge-base and workflow API
.github/workflows/go.yml
Adds actions: read permission and implements merge-base–based baseline coverage detection. The workflow fetches origin/main, computes the merge-base commit, queries GitHub Actions API to locate the successful go.yml run whose head_sha matches that commit, and conditionally downloads the coverage-main artifact from that run (with graceful fallback if artifact is missing or expired).

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~15 minutes

Possibly related PRs

  • DefangLabs/defang#1988: Both PRs modify the same .github/workflows/go.yml coverage-gating logic—specifically the step that downloads the coverage-main baseline from the main branch (with #1988 adding it conditionally and the main PR changing that baseline download to use a merge-base–selected workflow run).

Suggested reviewers

  • raphaeltm
  • lionello

Poem

🐰 A workflow hops through git's merge-base trail,
Finds baseline coverage without fail,
Queries runs with actions' newfound sight,
Downloads artifacts—perfectly right!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely summarizes the main change: switching from comparing coverage against the latest main to using the merge-base commit.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/coverage-regression-merge-base

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
.github/workflows/go.yml (1)

62-63: 💤 Low value

Shallow clone depth may prevent merge-base detection for long-lived branches.

With actions/checkout defaulting to fetch-depth: 1 and main fetched with --depth=50, the merge-base may not be reachable if the branch diverged more than ~50 commits ago. This would silently trigger the fallback (skipping coverage comparison) rather than the intended cases (first run, expired artifact, API error).

For Dependabot PRs this should be fine since they fork from recent main commits. Consider increasing the depth or using --deepen if you want more reliable merge-base detection for older feature branches.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.github/workflows/go.yml around lines 62 - 63, The shallow fetch depth (git
fetch origin main --depth=50) can make MERGE_BASE=$(git merge-base HEAD
origin/main) fail for long-lived branches; update the workflow to fetch more
history (e.g., remove --depth, use a larger --depth, or run git fetch --deepen
<n> until merge-base is found) so MERGE_BASE detection in the step that uses
MERGE_BASE=$(git merge-base HEAD origin/main) is reliable for older branches
while keeping the fallback behavior for true first runs or errors.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In @.github/workflows/go.yml:
- Around line 62-63: The shallow fetch depth (git fetch origin main --depth=50)
can make MERGE_BASE=$(git merge-base HEAD origin/main) fail for long-lived
branches; update the workflow to fetch more history (e.g., remove --depth, use a
larger --depth, or run git fetch --deepen <n> until merge-base is found) so
MERGE_BASE detection in the step that uses MERGE_BASE=$(git merge-base HEAD
origin/main) is reliable for older branches while keeping the fallback behavior
for true first runs or errors.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: bf2fff6a-789c-4544-aab8-99f0ece84d04

📥 Commits

Reviewing files that changed from the base of the PR and between 5c42faa and 18f0c92.

📒 Files selected for processing (1)
  • .github/workflows/go.yml

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant