fix(ci): Stop silencing test and security scan failures#2
Conversation
- Test step now checks for test files before running pytest, properly failing if tests exist and fail - Security scan (bandit) now fails on medium+ severity findings instead of silently passing with `|| true` Found by CodeRabbit review. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
📝 WalkthroughWalkthroughUpdated GitHub Actions CI workflow to conditionally execute pytest based on test file existence and modified security scanning severity level configuration in bandit command. Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~7 minutes Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches🧪 Generate unit tests (beta)
Comment |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
1 issue found across 1 file
Prompt for AI agents (all issues)
Check if these issues are valid — if so, understand the root cause of each and fix them.
<file name=".github/workflows/ci.yml">
<violation number="1" location=".github/workflows/ci.yml:42">
P2: The `find` expression uses `-o` without grouping, so the implicit `-print` only applies to the right side. As a result, `test_*.py` files aren’t detected and tests can be skipped even though they exist. Group the name expressions so both patterns are evaluated before printing.</violation>
</file>
Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.
| - name: Run tests | ||
| run: | | ||
| pytest tests/ -v --tb=short || echo "No tests yet" | ||
| if find tests -name 'test_*.py' -o -name '*_test.py' 2>/dev/null | grep -q .; then |
There was a problem hiding this comment.
P2: The find expression uses -o without grouping, so the implicit -print only applies to the right side. As a result, test_*.py files aren’t detected and tests can be skipped even though they exist. Group the name expressions so both patterns are evaluated before printing.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At .github/workflows/ci.yml, line 42:
<comment>The `find` expression uses `-o` without grouping, so the implicit `-print` only applies to the right side. As a result, `test_*.py` files aren’t detected and tests can be skipped even though they exist. Group the name expressions so both patterns are evaluated before printing.</comment>
<file context>
@@ -39,7 +39,11 @@ jobs:
- name: Run tests
run: |
- pytest tests/ -v --tb=short || echo "No tests yet"
+ if find tests -name 'test_*.py' -o -name '*_test.py' 2>/dev/null | grep -q .; then
+ pytest tests/ -v --tb=short
+ else
</file context>
| if find tests -name 'test_*.py' -o -name '*_test.py' 2>/dev/null | grep -q .; then | |
| if find tests \( -name 'test_*.py' -o -name '*_test.py' \) 2>/dev/null | grep -q .; then |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Fix all issues with AI agents
In @.github/workflows/ci.yml:
- Around line 42-43: The find expression "find tests -name 'test_*.py' -o -name
'*_test.py' ..." fails to print matches for the first branch because -o is
ungrouped; update the find invocation used in the CI step so the two -name
predicates are grouped (e.g., wrap them in escaped parentheses) and ensure
-print (or an explicit action) is applied to the grouped predicate so files
matching 'test_*.py' and '*_test.py' are both detected by the grep check and
pytest step.
- Line 67: The CI line invoking "bandit -r src/aiana --severity-level medium"
uses a flag added in Bandit 1.8.0; update the pyproject.toml Bandit constraint
to "bandit>=1.8.0" so the CI uses a compatible version, or alternatively change
the CI invocation to the legacy syntax ("bandit -r src/aiana -ll") which is
compatible with bandit>=1.7.5—make the change either in the CI workflow entry
referencing that command or in pyproject.toml's dependency spec to resolve the
mismatch.
| if find tests -name 'test_*.py' -o -name '*_test.py' 2>/dev/null | grep -q .; then | ||
| pytest tests/ -v --tb=short |
There was a problem hiding this comment.
Fix find expression so test_*.py files are detected.
Line 42’s find uses -o without grouping, so implicit -print only applies to the second branch; files matching test_*.py won’t be printed and tests may be skipped.
✅ Proposed fix
- if find tests -name 'test_*.py' -o -name '*_test.py' 2>/dev/null | grep -q .; then
+ if find tests -type f \( -name 'test_*.py' -o -name '*_test.py' \) -print -quit 2>/dev/null | grep -q .; then🤖 Prompt for AI Agents
In @.github/workflows/ci.yml around lines 42 - 43, The find expression "find
tests -name 'test_*.py' -o -name '*_test.py' ..." fails to print matches for the
first branch because -o is ungrouped; update the find invocation used in the CI
step so the two -name predicates are grouped (e.g., wrap them in escaped
parentheses) and ensure -print (or an explicit action) is applied to the grouped
predicate so files matching 'test_*.py' and '*_test.py' are both detected by the
grep check and pytest step.
| - name: Security scan with bandit | ||
| run: | | ||
| bandit -r src/aiana -ll || true | ||
| bandit -r src/aiana --severity-level medium |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
fd -a -t f 'pyproject\.toml|requirements.*\.txt|setup\.py|setup\.cfg' | head -20Repository: ry-ops/aiana
Length of output: 89
🏁 Script executed:
# Search for bandit version specifications
rg -n "bandit" pyproject.toml requirements*.txt setup.py setup.cfg 2>/dev/null || trueRepository: ry-ops/aiana
Length of output: 94
🌐 Web query:
Bandit --severity-level flag documentation 1.7.5
💡 Result:
Bandit 1.7.5 does not have a --severity-level CLI flag. Severity filtering is done with -l/--level (repeat the flag to raise the minimum reported severity) [1]:
-l/--level→ report LOW and higher-ll→ report MEDIUM and higher-lll→ report HIGH only [1]
Example (only high-severity issues):
bandit -r . -lllThis matches the 1.7.5 docs example usage (“only reporting on the high-severity issues”) [2].
Related: confidence filtering is -i/--confidence (-i, -ii, -iii) [1].
References: [1] [2]
🌐 Web query:
Bandit exit code behavior severity level findings
💡 Result:
Bandit’s severity/confidence “level” flags control both (a) what gets reported and (b) whether Bandit exits non‑zero.
What counts as a “finding” for exit-code purposes
Bandit filters issues by:
- Severity:
-l/-ll/-lllor--severity-level {low,medium,high,all}(reports issues at that severity or higher) [1] - Confidence:
-i/-ii/-iiior--confidence-level {low,medium,high,all}(reports issues at that confidence or higher) [1]
Its CLI then computes results_count(sev_filter=..., conf_filter=...) and:
- exits 0 if the filtered count is 0
- exits 1 if the filtered count is > 0 [2]
This “exit code respects the filters” behavior was explicitly fixed after a historical bug where Bandit could exit 1 due to lower-severity issues even when none were displayed under -ll/-lll [3].
Exit codes you’ll commonly see
- 0: no issues at/above your chosen severity/confidence thresholds (after filtering) [2]
- 1: at least one issue at/above your chosen thresholds [2]
- 0 even with issues: if you pass
--exit-zero[1] - 2: used for “can’t run correctly” situations (e.g., no tests would run / config problems), where Bandit calls
sys.exit(2)[2]
Practical CI implication (“severity level” vs “fail the build”)
Because the same thresholds drive both reporting and exit code, Bandit can’t (natively) “report everything but only fail on high” in a single run—this is a common feature request [4]. Typical workaround: run twice (once for a full report with --exit-zero, once with strict levels to gate CI) [4].
Sources: [1] Bandit CLI docs, options for severity/confidence and --exit-zero; [2] Bandit CLI code path showing exit 1 iff filtered results_count(...) > 0, else 0, and sys.exit(2) on certain failures; [3] Launchpad bug describing and fixing “exit code for filtered results”; [4] GitHub issue describing coupled reporting and exit behavior.
🌐 Web query:
Bandit --severity-level flag when was it introduced version history
💡 Result:
bandit --severity-level (and the matching --confidence-level) is not present in the Bandit 1.7.8 man page (it only documents -l/--level with -l/-ll/-lll). [1]
In the current (“latest”) Bandit man page, --severity-level={all,high,medium,low} is documented (alongside the older -l/--level). [2]
So, based on the published docs, --severity-level was introduced sometime after 1.7.8 and by the time modern docs were updated—Bandit’s 1.8.0 release was Nov 27, 2024 [3], and a PR to update documentation to cover --severity-level / --confidence-level was merged Apr 7, 2025. [4]
Best-supported version window: between 1.7.8 and 1.8.x (at latest, it existed by early 2025; docs were explicitly updated in April 2025).
References: [1] https://bandit.readthedocs.io/en/1.7.8/man/bandit.html [2] https://bandit.readthedocs.io/en/latest/man/bandit.html [3] https://github.com/PyCQA/bandit/releases?page=2 [4] PyCQA/bandit#1254
🏁 Script executed:
cat -n .github/workflows/ci.yml | sed -n '60,75p'Repository: ry-ops/aiana
Length of output: 318
Update Bandit version constraint to bandit>=1.8.0.
The --severity-level flag is not supported in Bandit 1.7.5–1.7.8 (only in 1.8.0+). The repository's pyproject.toml currently specifies bandit>=1.7.5, which allows versions that don't support this flag and will cause the CI to fail. Either update the constraint to bandit>=1.8.0 or use the legacy syntax (-ll for medium severity) compatible with 1.7.5+.
🤖 Prompt for AI Agents
In @.github/workflows/ci.yml at line 67, The CI line invoking "bandit -r
src/aiana --severity-level medium" uses a flag added in Bandit 1.8.0; update the
pyproject.toml Bandit constraint to "bandit>=1.8.0" so the CI uses a compatible
version, or alternatively change the CI invocation to the legacy syntax ("bandit
-r src/aiana -ll") which is compatible with bandit>=1.7.5—make the change either
in the CI workflow entry referencing that command or in pyproject.toml's
dependency spec to resolve the mismatch.
1 participant
Summary
Changes
pytest ... || echo "No tests yet"with conditional check for test filesbandit ... || truewithbandit --severity-level mediumto fail on real issuesTest plan
🐰 Found by CodeRabbit review
🤖 Generated with Claude Code
Summary by cubic
CI now surfaces real failures: test jobs run only when test files exist and fail on test errors; Bandit now fails on medium+ severity findings.
Written for commit d4ff61e. Summary will update on new commits.
Summary by CodeRabbit