Skip to content

[#2183] Added posting of coverage to PR. #2184

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
AlexSkrypnyk merged 2 commits into from
Dec 12, 2025
Merged

[#2183] Added posting of coverage to PR. #2184

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
48a2bb6
[#2183] Added posting of coverage to PR.
AlexSkrypnyk Dec 12, 2025
32f4b98
[#2183] Added posting of coverage to PR for CircleCI.
AlexSkrypnyk Dec 12, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 32 additions & 0 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -396,6 +396,38 @@ jobs:
- store_artifacts:
path: *artifacts

#;< TOOL_PHPUNIT
- run:
name: Check code coverage threshold
command: |
[ "${CIRCLE_NODE_TOTAL:-1}" -gt 1 ] && [ "${CIRCLE_NODE_INDEX:-0}" -ne 0 ] && exit 0
RATE=$(grep -o 'line-rate="[0-9.]*"' /tmp/artifacts/coverage/phpunit/cobertura.xml | head -1 | tr -cd '0-9.')
PERCENT=$(awk "BEGIN {printf \"%.2f\", $RATE*100}")
echo "Coverage: $PERCENT% (threshold: ${VORTEX_CI_CODE_COVERAGE_THRESHOLD:-90}%)"
if [ "${PERCENT//./}" -lt "$((${VORTEX_CI_CODE_COVERAGE_THRESHOLD:-90}*100))" ]; then
echo "FAIL: coverage too low"
exit 1
fi
Comment on lines +400 to +410

@coderabbitai coderabbitai Bot Dec 12, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Handle missing/invalid Cobertura input (avoid false “0%” / awk errors).
If /tmp/artifacts/coverage/phpunit/cobertura.xml is missing or doesn’t contain the expected attribute, RATE becomes empty and the step can misreport coverage or behave inconsistently. Add file/parse validation and fail with a clear message.

       - run:
           name: Check code coverage threshold
           command: |
             [ "${CIRCLE_NODE_TOTAL:-1}" -gt 1 ] && [ "${CIRCLE_NODE_INDEX:-0}" -ne 0 ] && exit 0
-            RATE=$(grep -o 'line-rate="[0-9.]*"' /tmp/artifacts/coverage/phpunit/cobertura.xml | head -1 | tr -cd '0-9.')
+            COBERTURA=/tmp/artifacts/coverage/phpunit/cobertura.xml
+            [ -f "${COBERTURA}" ] || { echo "FAIL: missing ${COBERTURA}"; exit 1; }
+            # Prefer the top-level <coverage line-rate="..."> attribute.
+            RATE=$(grep -m1 -oE '<coverage[^>]* line-rate="[0-9.]+"' "${COBERTURA}" | grep -oE 'line-rate="[0-9.]+"' | tr -cd '0-9.')
+            [ -n "${RATE}" ] || { echo "FAIL: could not parse line-rate from ${COBERTURA}"; exit 1; }
             PERCENT=$(awk "BEGIN {printf \"%.2f\", $RATE*100}")
             echo "Coverage: $PERCENT% (threshold: ${VORTEX_CI_CODE_COVERAGE_THRESHOLD:-90}%)"
             if [ "${PERCENT//./}" -lt "$((${VORTEX_CI_CODE_COVERAGE_THRESHOLD:-90}*100))" ]; then
               echo "FAIL: coverage too low"
               exit 1
             fi
🤖 Prompt for AI Agents 
In .circleci/config.yml around lines 400 to 410, the script assumes
/tmp/artifacts/coverage/phpunit/cobertura.xml exists and that RATE will be
populated, causing false "0%" or awk errors when the file or attribute is
missing; add explicit checks: first verify the cobertura.xml file exists and is
readable, then ensure the grep extraction produced a non-empty, numeric RATE
(match digits and optional dot) before using awk; if the file is missing or RATE
is invalid, print a clear failure message indicating the missing/invalid
Cobertura input and exit non-zero so the job fails fast and avoids misleading
coverage output.


- run:
name: Post coverage summary as PR comment
command: |
[ "${CIRCLE_NODE_TOTAL:-1}" -gt 1 ] && [ "${CIRCLE_NODE_INDEX:-0}" -ne 0 ] && exit 0
[ "${VORTEX_CI_CODE_COVERAGE_PR_COMMENT_SKIP:-0}" = "1" ] && exit 0
[ -z "${CIRCLE_PULL_REQUEST}" ] && exit 0
[ -z "${GITHUB_TOKEN}" ] && exit 0
COVERAGE_CONTENT=$(sed '/./,$!d' /tmp/artifacts/coverage/phpunit/coverage.txt)
PR_NUMBER=$(echo "${CIRCLE_PULL_REQUEST}" | cut -d'/' -f 7)
REPO_SLUG="${CIRCLE_PROJECT_USERNAME}/${CIRCLE_PROJECT_REPONAME}"
curl -s -X POST \
-H "Authorization: token ${GITHUB_TOKEN}" \
-H "Accept: application/vnd.github.v3+json" \
"https://api.github.com/repos/${REPO_SLUG}/issues/${PR_NUMBER}/comments" \
-d "$(jq -n --arg body "\`\`\`
${COVERAGE_CONTENT}
\`\`\`" '{body: $body}')"
#;> TOOL_PHPUNIT

- run:
name: Upload code coverage reports to Codecov
command: |
Expand Down
26 changes: 26 additions & 0 deletions .github/workflows/build-test-deploy.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -381,6 +381,31 @@ jobs:
include-hidden-files: true
if-no-files-found: error

#;< TOOL_PHPUNIT
- name: Check code coverage threshold
if: ${{ matrix.instance == 0 || strategy.job-total == 1 }}
run: |
RATE=$(grep -o 'line-rate="[0-9.]*"' .logs/coverage/phpunit/cobertura.xml | head -1 | tr -cd '0-9.')
PERCENT=$(awk "BEGIN {printf \"%.2f\", $RATE*100}")
echo "Coverage: $PERCENT% (threshold: $VORTEX_CI_CODE_COVERAGE_THRESHOLD%)"
if [ "${PERCENT//./}" -lt "$((VORTEX_CI_CODE_COVERAGE_THRESHOLD*100))" ]; then
echo "FAIL: coverage too low"
exit 1
fi
{ echo "COVERAGE_CONTENT<<EOF"; sed '/./,$!d' .logs/coverage/phpunit/coverage.txt; echo "EOF"; } >> "$GITHUB_ENV"
env:
VORTEX_CI_CODE_COVERAGE_THRESHOLD: ${{ vars.VORTEX_CI_CODE_COVERAGE_THRESHOLD || '90' }}

Comment on lines +384 to +398

@coderabbitai coderabbitai Bot Dec 12, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

Harden coverage parsing/compare: handle missing RATE/files and avoid fragile string math.
Current logic can fail with opaque errors if cobertura.xml/coverage.txt isn’t present or line-rate parsing returns empty, and threshold*100 will break on non-integer input.

       - name: Check code coverage threshold
         if: ${{ matrix.instance == 0 || strategy.job-total == 1 }}
         run: |
-          RATE=$(grep -o 'line-rate="[0-9.]*"' .logs/coverage/phpunit/cobertura.xml | head -1 | tr -cd '0-9.')
-          PERCENT=$(awk "BEGIN {printf \"%.2f\", $RATE*100}")
-          echo "Coverage: $PERCENT% (threshold: $VORTEX_CI_CODE_COVERAGE_THRESHOLD%)"
-          if [ "${PERCENT//./}" -lt "$((VORTEX_CI_CODE_COVERAGE_THRESHOLD*100))" ]; then
-            echo "FAIL: coverage too low"
-            exit 1
-          fi
-          { echo "COVERAGE_CONTENT<<EOF"; sed '/./,$!d' .logs/coverage/phpunit/coverage.txt; echo "EOF"; } >> "$GITHUB_ENV"
+          set -euo pipefail
+
+          COBERTURA=".logs/coverage/phpunit/cobertura.xml"
+          TEXT_REPORT=".logs/coverage/phpunit/coverage.txt"
+          test -s "${COBERTURA}" || { echo "FAIL: missing Cobertura report at ${COBERTURA}"; exit 1; }
+          test -s "${TEXT_REPORT}" || { echo "FAIL: missing text coverage report at ${TEXT_REPORT}"; exit 1; }
+
+          RATE="$(grep -oE 'line-rate="[0-9]+(\.[0-9]+)?"' "${COBERTURA}" | head -n1 | tr -cd '0-9.')"
+          test -n "${RATE}" || { echo "FAIL: could not parse line-rate from ${COBERTURA}"; exit 1; }
+
+          THRESHOLD="${VORTEX_CI_CODE_COVERAGE_THRESHOLD:-90}"
+          [[ "${THRESHOLD}" =~ ^[0-9]+([.][0-9]+)?$ ]] || { echo "FAIL: invalid threshold '${THRESHOLD}'"; exit 1; }
+
+          PERCENT="$(awk -v r="${RATE}" 'BEGIN { printf "%.2f", (r*100) }')"
+          echo "Coverage: ${PERCENT}% (threshold: ${THRESHOLD}%)"
+
+          awk -v p="${PERCENT}" -v t="${THRESHOLD}" 'BEGIN { exit !(p < t) }' && { echo "FAIL: coverage too low"; exit 1; }
+
+          DELIM="COVERAGE_$(date +%s)_$RANDOM"
+          { echo "COVERAGE_CONTENT<<${DELIM}"; sed '/./,$!d' "${TEXT_REPORT}"; echo "${DELIM}"; } >> "$GITHUB_ENV"
         env:
           VORTEX_CI_CODE_COVERAGE_THRESHOLD: ${{ vars.VORTEX_CI_CODE_COVERAGE_THRESHOLD || '90' }}

<details>
<summary>🤖 Prompt for AI Agents</summary>

.github/workflows/build-test-deploy.yml around lines 384-398: the coverage check
is fragile — it assumes cobertura.xml and coverage.txt exist, parses line-rate
unsafely, and does string math that breaks on non-integer thresholds; fix by
enabling strict shell options (set -euo pipefail), assert the Cobertura and
text-report files exist and are non-empty, extract line-rate with a robust regex
and validate it’s present, validate the VORTEX_CI_CODE_COVERAGE_THRESHOLD is
numeric, compute percent with awk (floating math) and perform a numeric
comparison (not string ops), and write the text report to GITHUB_ENV using a
unique delimiter to avoid collisions.


</details>

<!-- fingerprinting:phantom:triton:falcon -->

<!-- This is an auto-generated comment by CodeRabbit -->

- name: Post coverage summary as PR comment
if: ${{ github.event_name == 'pull_request' && (matrix.instance == 0 || strategy.job-total == 1) && vars.VORTEX_CI_CODE_COVERAGE_PR_COMMENT_SKIP != '1' }}
uses: marocchino/sticky-pull-request-comment@52423e01640425a022ef5fd42c6fb5f633a02728 # v2
with:
message: |
```
${{ env.COVERAGE_CONTENT }}
```
hide_and_recreate: true

Comment on lines +399 to +408

@coderabbitai coderabbitai Bot Dec 12, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Gate PR comment on having computed content (and keep behavior PR-only).
As written, a future change (or a skipped/failed threshold step) could still post an empty comment.

       - name: Post coverage summary as PR comment
-        if: ${{ github.event_name == 'pull_request' && (matrix.instance == 0 || strategy.job-total == 1) && vars.VORTEX_CI_CODE_COVERAGE_PR_COMMENT_SKIP != '1' }}
+        if: ${{ github.event_name == 'pull_request' && (matrix.instance == 0 || strategy.job-total == 1) && vars.VORTEX_CI_CODE_COVERAGE_PR_COMMENT_SKIP != '1' && env.COVERAGE_CONTENT != '' }}
         uses: marocchino/sticky-pull-request-comment@52423e01640425a022ef5fd42c6fb5f633a02728 # v2
         with:
           message: |
             ```
             ${{ env.COVERAGE_CONTENT }}
             ```
           hide_and_recreate: true

<details>
<summary>🤖 Prompt for AI Agents</summary>

.github/workflows/build-test-deploy.yml around lines 399 to 408: the PR comment
step can post an empty comment if COVERAGE_CONTENT is unset or empty; modify the
step's if condition to additionally require COVERAGE_CONTENT to be non-empty
(e.g., && env.COVERAGE_CONTENT != ''), keeping the existing pull_request and
matrix/strategy gating so the action only runs on PRs and only when content was
actually computed; ensure COVERAGE_CONTENT is exported/set earlier (defaults to
empty) so the conditional works reliably.


</details>

<!-- fingerprinting:phantom:triton:falcon -->

<!-- This is an auto-generated comment by CodeRabbit -->

- name: Upload coverage report to Codecov
uses: codecov/codecov-action@671740ac38dd9b0130fbe1cec585b89eea48d3de # v5
if: ${{ env.CODECOV_TOKEN != '' }}
Expand All @@ -390,6 +415,7 @@ jobs:
token: ${{ secrets.CODECOV_TOKEN }}
env:
CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
#;> TOOL_PHPUNIT

- name: Upload exported codebase as an artifact
uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5
Expand Down
6 changes: 6 additions & 0 deletions .vortex/docs/.utils/variables/extra/ci.variables.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,12 @@ VORTEX_CI_GHERKIN_LINT_IGNORE_FAILURE=0
# Ignore PHPUnit test failures.
VORTEX_CI_PHPUNIT_IGNORE_FAILURE=0

# Code coverage threshold percentage. Build fails if coverage is below this value.
VORTEX_CI_CODE_COVERAGE_THRESHOLD=90

# Skip posting code coverage report as a PR comment.
VORTEX_CI_CODE_COVERAGE_PR_COMMENT_SKIP=0

# Ignore Behat test failures.
VORTEX_CI_BEHAT_IGNORE_FAILURE=0

Expand Down
13 changes: 13 additions & 0 deletions .vortex/docs/content/continuous-integration/circleci.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -164,3 +164,16 @@ long-running steps:
command: ./scripts/long-task.sh
no_output_timeout: 60m # Default is 10m
```

### Code coverage threshold

The workflow enforces a minimum code coverage threshold. If coverage falls below
the threshold, the build fails.

Configure the threshold by setting the `VORTEX_CI_CODE_COVERAGE_THRESHOLD`
environment variable in **Project Settings → Environment Variables**. Default
is `90` (percent).

Coverage reports can be posted as PR comments. This requires a `GITHUB_TOKEN`
environment variable with permission to post comments. To disable PR comments,
set `VORTEX_CI_CODE_COVERAGE_PR_COMMENT_SKIP` to `1`.
Comment on lines +168 to +179

@coderabbitai coderabbitai Bot Dec 12, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Clarify required GITHUB_TOKEN scopes/permissions (and where it must be set).
Right now it says “permission to post comments”; I’d explicitly call out the minimal permission (e.g., PR/Issues write) and that it must be configured as a CircleCI env var (Project Settings / Context), not just a GitHub Actions token.

12 changes: 12 additions & 0 deletions .vortex/docs/content/continuous-integration/github-actions.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,3 +158,15 @@ steps:
run: ./scripts/long-task.sh
timeout-minutes: 60 # Default is 360 (6 hours)
```

### Code coverage threshold

The workflow enforces a minimum code coverage threshold. If coverage falls below
the threshold, the build fails.

Configure the threshold by setting the `VORTEX_CI_CODE_COVERAGE_THRESHOLD`
variable in **Settings → Secrets and variables → Actions → Variables**. Default
is `90` (percent).

Coverage reports are automatically posted as PR comments. To disable this, set
`VORTEX_CI_CODE_COVERAGE_PR_COMMENT_SKIP` to `1`.
Comment on lines +162 to +172

@coderabbitai coderabbitai Bot Dec 12, 2025

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Docs should mention PR-only + single-matrix gating to avoid confusion.
Right now it reads as if comments always post; but workflow restricts to pull_request and matrix.instance == 0 || strategy.job-total == 1.

🤖 Prompt for AI Agents 
.vortex/docs/content/continuous-integration/github-actions.mdx around lines 162
to 172: the docs imply coverage comments always post but the workflow only runs
on pull_request and posts coverage only for PRs and when matrix.instance == 0 or
strategy.job-total == 1; update the paragraph to state that the coverage check
and PR comment are gated to pull request events and to a single matrix instance
(matrix.instance == 0 or when strategy.job-total == 1), and clarify that setting
VORTEX_CI_CODE_COVERAGE_PR_COMMENT_SKIP=1 disables the PR comment even on PRs;
keep the existing info about VORTEX_CI_CODE_COVERAGE_THRESHOLD and where to set
it.

13 changes: 12 additions & 1 deletion .vortex/docs/content/tools/phpunit.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ Run checks against platform version specified in `composer.json` key `config.pla
## Coverage

PHPUnit is configured to generate code coverage reports. The reports are stored
in `.logs/test_results/phpunit/phpunit.xml`
in `.logs/coverage/phpunit/cobertura.xml`
as [Cobertura XML](https://cobertura.github.io/cobertura/), suitable for
automated coverage assessment, and in `.logs/coverage/phpunit/.coverage-html` as
HTML coverage report, useful for visual report assessment during test
Expand All @@ -114,6 +114,17 @@ development.
Continuous integration pipeline runs tests with coverage by default and stores
the reports as artifacts.

### Coverage threshold

CI enforces a minimum coverage threshold. If coverage falls below the threshold,
the build fails. Set `VORTEX_CI_CODE_COVERAGE_THRESHOLD` to configure the
minimum percentage (default: `90`).

### PR comments

Coverage reports are posted as PR comments automatically. Set
`VORTEX_CI_CODE_COVERAGE_PR_COMMENT_SKIP` to `1` to disable.

### Ignoring lines from coverage

Sometimes it is necessary to ignore lines from coverage. For example, when
Expand Down
1 change: 0 additions & 1 deletion .vortex/docs/content/workflows/notifications.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,6 @@ The following table shows how these variables differ between branch and PR deplo
| `VORTEX_NOTIFY_PR_NUMBER` | *(empty)* | `123` |
| `VORTEX_NOTIFY_LABEL` | `main` | `PR-123` |


## Message templates and tokens

Most notification channels support customizable message templates using replacement tokens:
Expand Down
Loading