Coverage that cannot report a green it did not earn - #24
Merged
thc1006 merged 1 commit intoJul 29, 2026
Merged
Conversation
Two parts of the coverage path report success while doing nothing. Both are in the logs of any recent run. The six matrix legs all uploaded an artifact named `coverage` with `overwrite: true`, which deletes the previous artifact rather than merging into it. Run 30461091931 shows `Artifact 'coverage' (ID: 8727950028) deleted` and then finalized again by a later leg, so five of the six reports were thrown away and the surviving one was whichever leg happened to finish last. Each leg now writes and uploads a report named after itself, and the upload job downloads all of them. The headline number was not far off, since all six legs land on 80%, but which platform's report reached Codecov was decided by a race, and code that only runs on one platform was counted or not depending on it. `files:` was a YAML block literal, so the name arrived as "coverage.xml\n" and was not found. The upload only worked because the uploader falls back to searching. That fallback is now unnecessary: the reports are in one directory, and the directory is what is passed. `fail_ci_if_error` is tied to whether a token was there to use. A pull request from a fork gets no secrets, and a contributor should not see red for that. What this does not fix: `CODECOV_TOKEN` resolves to empty on this repository, so the upload is still rejected with "Token required - not valid tokenless upload", and with no token the run stays green. Setting that secret is the last step and it needs repository admin. Once it is set, this workflow fails when an upload fails, without another change. Upstream RocketPy carries both of these unchanged. Happy to send the same patch there so it comes back through the usual sync. Verified: actionlint clean, `--cov-report=xml:<name>` writes the named file, and `pattern` / `merge-multiple` / `directory` / `fail_ci_if_error` all exist as inputs on the pinned actions. Signed-off-by: thc1006 <84045975+thc1006@users.noreply.github.com>
thc1006
deleted the
ci/coverage-that-cannot-report-a-green-it-did-not-earn
branch
July 29, 2026 15:42
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #22.
Two parts of the coverage path report success while doing nothing.
Five of six reports were being thrown away
The six matrix legs all uploaded an artifact named
coveragewithoverwrite: true, which deletes the previous artifact rather than merging into it. From run 30461091931:Whichever leg finished last was the whole picture. Each leg now writes and uploads a report named after itself, and the upload job downloads all of them.
Worth saying plainly: the headline number was not far off. All six legs land on 80%. What the race decided was which platform's report reached Codecov, so code that only runs on one platform was counted or not depending on timing.
The upload was finding the file by accident
files:was a YAML block literal, so the name arrived as"coverage.xml\n"and was not found. The upload worked only because the uploader falls back to searching. The reports now sit in one directory and the directory is what is passed, so nothing depends on that fallback.A new step refuses to run the upload when the directory is empty, since an upload with nothing in it reporting success is the shape of the problem being fixed.
What this does not fix
CODECOV_TOKENresolves to empty here, so the upload is still rejected withToken required - not valid tokenless upload, and with no token the run stays green. Setting that secret needs repository admin.fail_ci_if_erroris tied to whether a token was there to use, so the moment the secret exists this workflow starts failing on a failed upload with no further change, and pull requests from forks stay unaffected.Verified
--cov-report=xml:<name>writes the named file (ran it)pattern,merge-multiple,directoryandfail_ci_if_errorall exist as inputs on the pinned actions (read theiraction.yml)coverageUpstream RocketPy carries both of these unchanged. Happy to send the same patch there so it comes back through the usual sync.