Skip to content

Commit f049458

Browse files
committed
Feat: Replace all GitHub Actions with shell scripts for CI/CD
Convert all CI/CD checks from GitHub Actions to shell scripts per project requirement of NO GitHub Actions dependency. Problem: - 14 failing GitHub Actions checks - Project restriction: NO GitHub Actions allowed - Need local-first CI/CD that works anywhere Solution: Complete shell script replacement for all CI/CD checks, organized in modular architecture. Components Added: [Infrastructure CI] (4 scripts) - health-check.sh: Python, DB, Django configuration validation - validate-scripts.sh: Shell script syntax and permission checks - validate-config.sh: JSON/YAML configuration validation - validate-docker.sh: Docker and docker-compose validation [Security Scan] (4 scripts) - csrf-check.sh: CSRF protection verification - django-security-check.sh: Django security settings (DEBUG, SECRET_KEY, SSL) - bandit-scan.sh: Python security scan (Bandit) - npm-audit.sh: NPM vulnerability audit [Test Pyramid Validation] (2 scripts) - test-pyramid.sh: Validates test distribution (70% unit, 20% integration, 10% e2e) - test-execution-time.sh: Ensures tests run < 2 minutes, identifies slow tests [Master Orchestrator] - run-all-checks.sh: Runs all checks in proper order with reporting Features: [OK] Exit codes: 0 (pass), 1 (fail), 2 (skip) [OK] Color-coded output (green/red/yellow) [OK] --fail-fast: Stop on first failure [OK] --verbose: Detailed output [OK] --only <suite>: Run specific suite only [OK] Comprehensive final report with statistics [OK] Auto-installation of missing tools (bandit) [OK] Virtualenv auto-activation Architecture: ``` scripts/ci/ ├── infrastructure/ # System validation ├── security/ # Security scans ├── testing/ # Test quality ├── run-all-checks.sh # Master orchestrator └── README.md # Complete documentation ``` Usage: ```bash # Run everything ./scripts/ci/run-all-checks.sh # Fail fast mode (CI/CD) ./scripts/ci/run-all-checks.sh --fail-fast # Run specific suite ./scripts/ci/run-all-checks.sh --only security --verbose # Individual checks ./scripts/ci/infrastructure/health-check.sh ./scripts/ci/security/django-security-check.sh ``` Integration: - Jenkins: stage('CI') { sh './scripts/ci/run-all-checks.sh --fail-fast' } - GitLab CI: script: ./scripts/ci/run-all-checks.sh --fail-fast - Pre-commit hook: Run critical checks before commit - Pre-push hook: Run full suite before push Benefits: [OK] NO GitHub Actions dependency [OK] Runs locally without internet [OK] Portable to any CI system (Jenkins, GitLab, etc.) [OK] Faster feedback (no wait for GH runners) [OK] Easy to debug (standard bash) [OK] Modular and extensible [OK] Self-documenting with clear exit codes Replaces GitHub Actions: - Infrastructure CI Summary (4 checks) → infrastructure/*.sh - Security Scan Summary (4 checks) → security/*.sh - Test Pyramid Validation (2 checks) → testing/*.sh Testing: All scripts follow standard conventions: - Executable permissions - Shebang (#!/bin/bash) - set -e (fail fast) - Proper exit codes - Color-coded logging Documentation: - scripts/ci/README.md: Complete usage guide - Exit code standards - Integration examples - Troubleshooting guide Project Compliance: [OK] NO GitHub Actions used [OK] Shell scripts only [OK] No external service dependencies [OK] Works offline [OK] Respects all project restrictions Next Steps: - Run locally: ./scripts/ci/run-all-checks.sh - Fix any failing checks - Integrate in CI pipeline - Add pre-commit hooks Related: - TDD Agent v1.1 (commit e43f7a0) - PromptOps framework - Route Lint gate (22/22 tests passing)
1 parent e43f7a0 commit f049458

12 files changed

Lines changed: 1334 additions & 0 deletions

scripts/ci/README.md

Lines changed: 313 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,313 @@
1+
# CI/CD Shell Scripts
2+
3+
**Replacement for GitHub Actions** - All CI/CD checks implemented as shell scripts.
4+
5+
## Purpose
6+
7+
This project uses **shell scripts instead of GitHub Actions** for CI/CD. All checks run locally or in any CI environment without GitHub-specific dependencies.
8+
9+
## Architecture
10+
11+
```
12+
scripts/ci/
13+
├── infrastructure/ # Infrastructure validation
14+
│ ├── health-check.sh # Health checks (DB, Python, Django)
15+
│ ├── validate-scripts.sh # Shell script validation
16+
│ ├── validate-config.sh # JSON/YAML config validation
17+
│ └── validate-docker.sh # Docker configuration validation
18+
├── security/ # Security scans
19+
│ ├── csrf-check.sh # CSRF protection verification
20+
│ ├── django-security-check.sh # Django security settings
21+
│ ├── bandit-scan.sh # Python security scan (Bandit)
22+
│ └── npm-audit.sh # NPM vulnerability scan
23+
├── testing/ # Test quality validation
24+
│ ├── test-pyramid.sh # Test pyramid analysis
25+
│ └── test-execution-time.sh # Test performance
26+
├── gate-route-lint.sh # PromptOps: Route Lint gate
27+
├── run-all-gates.sh # PromptOps gates orchestrator
28+
├── run-tdd-cycle.sh # TDD Agent wrapper
29+
└── run-all-checks.sh # Master CI/CD orchestrator
30+
```
31+
32+
## Usage
33+
34+
### Run All Checks
35+
36+
```bash
37+
# Run everything
38+
./scripts/ci/run-all-checks.sh
39+
40+
# Fail fast (stop on first error)
41+
./scripts/ci/run-all-checks.sh --fail-fast
42+
43+
# Verbose output
44+
./scripts/ci/run-all-checks.sh --verbose
45+
46+
# Run specific suite only
47+
./scripts/ci/run-all-checks.sh --only infrastructure
48+
./scripts/ci/run-all-checks.sh --only security
49+
./scripts/ci/run-all-checks.sh --only testing
50+
./scripts/ci/run-all-checks.sh --only promptops
51+
```
52+
53+
### Individual Suites
54+
55+
**Infrastructure CI:**
56+
```bash
57+
./scripts/ci/infrastructure/health-check.sh
58+
./scripts/ci/infrastructure/validate-scripts.sh
59+
./scripts/ci/infrastructure/validate-config.sh
60+
./scripts/ci/infrastructure/validate-docker.sh
61+
```
62+
63+
**Security Scan:**
64+
```bash
65+
./scripts/ci/security/csrf-check.sh
66+
./scripts/ci/security/django-security-check.sh
67+
./scripts/ci/security/bandit-scan.sh
68+
./scripts/ci/security/npm-audit.sh
69+
```
70+
71+
**Test Pyramid Validation:**
72+
```bash
73+
./scripts/ci/testing/test-pyramid.sh
74+
./scripts/ci/testing/test-execution-time.sh
75+
```
76+
77+
**PromptOps Gates:**
78+
```bash
79+
./scripts/ci/gate-route-lint.sh
80+
./scripts/ci/run-all-gates.sh
81+
```
82+
83+
## Exit Codes
84+
85+
All scripts follow standard exit code conventions:
86+
87+
- `0`: Check passed
88+
- `1`: Check failed
89+
- `2`: Check skipped (not applicable)
90+
91+
## Mapping: GitHub Actions → Shell Scripts
92+
93+
| GitHub Action | Shell Script | Status |
94+
|---------------|--------------|--------|
95+
| Infrastructure CI / Health Check | `infrastructure/health-check.sh` | [OK] |
96+
| Infrastructure CI / Validate Scripts | `infrastructure/validate-scripts.sh` | [OK] |
97+
| Infrastructure CI / Validate Config | `infrastructure/validate-config.sh` | [OK] |
98+
| Infrastructure CI / Validate Docker | `infrastructure/validate-docker.sh` | [OK] |
99+
| Security Scan / CSRF Check | `security/csrf-check.sh` | [OK] |
100+
| Security Scan / Django Security | `security/django-security-check.sh` | [OK] |
101+
| Security Scan / Bandit | `security/bandit-scan.sh` | [OK] |
102+
| Security Scan / NPM Audit | `security/npm-audit.sh` | [OK] |
103+
| Test Pyramid / Analyze | `testing/test-pyramid.sh` | [OK] |
104+
| Test Pyramid / Execution Time | `testing/test-execution-time.sh` | [OK] |
105+
| (Custom) Route Lint Gate | `gate-route-lint.sh` | [OK] |
106+
107+
## Integration with Pre-commit
108+
109+
Add to `.git/hooks/pre-commit`:
110+
111+
```bash
112+
#!/bin/bash
113+
# Run critical checks before commit
114+
115+
./scripts/ci/infrastructure/validate-scripts.sh || exit 1
116+
./scripts/ci/security/csrf-check.sh || exit 1
117+
./scripts/ci/gate-route-lint.sh || exit 1
118+
119+
echo "Pre-commit checks passed"
120+
```
121+
122+
## Integration with CI Server
123+
124+
### Jenkins
125+
126+
```groovy
127+
stage('CI Checks') {
128+
steps {
129+
sh './scripts/ci/run-all-checks.sh --fail-fast'
130+
}
131+
}
132+
```
133+
134+
### GitLab CI
135+
136+
```yaml
137+
ci_checks:
138+
script:
139+
- ./scripts/ci/run-all-checks.sh --fail-fast
140+
```
141+
142+
### Generic CI
143+
144+
```bash
145+
#!/bin/bash
146+
# ci-pipeline.sh
147+
148+
cd $PROJECT_ROOT
149+
./scripts/ci/run-all-checks.sh --fail-fast
150+
151+
if [ $? -eq 0 ]; then
152+
echo "All checks passed - ready to deploy"
153+
exit 0
154+
else
155+
echo "CI checks failed"
156+
exit 1
157+
fi
158+
```
159+
160+
## Requirements
161+
162+
### System Dependencies
163+
164+
**Required:**
165+
- bash (4.0+)
166+
- Python 3.8+
167+
- pytest
168+
169+
**Optional:**
170+
- Node.js + npm (for NPM audit)
171+
- Docker + docker-compose (for Docker validation)
172+
- bandit (auto-installed if missing)
173+
174+
### Python Dependencies
175+
176+
```bash
177+
# Install in virtualenv
178+
pip install pytest pytest-json-report bandit
179+
```
180+
181+
## Script Details
182+
183+
### Infrastructure CI
184+
185+
**health-check.sh**
186+
- Verifies Python version
187+
- Tests database connectivity
188+
- Validates Django configuration
189+
- Checks required directories
190+
191+
**validate-scripts.sh**
192+
- Finds all `.sh` files in `scripts/`
193+
- Checks shebang exists
194+
- Validates bash syntax (`bash -n`)
195+
- Verifies executable permissions
196+
197+
**validate-config.sh**
198+
- Validates all JSON files (syntax)
199+
- Validates all YAML files (syntax)
200+
- Validates Django settings imports
201+
202+
**validate-docker.sh**
203+
- Checks Dockerfile has `FROM` instruction
204+
- Validates `USER` instruction exists (security)
205+
- Validates `docker-compose.yml` syntax
206+
207+
### Security Scan
208+
209+
**csrf-check.sh**
210+
- Verifies `CsrfViewMiddleware` is enabled
211+
- Scans for `@csrf_exempt` decorators
212+
- Checks API views have session authentication
213+
214+
**django-security-check.sh**
215+
- Runs `python manage.py check --deploy`
216+
- Validates `DEBUG = False`
217+
- Checks `SECRET_KEY` not hardcoded
218+
- Validates SSL/HTTPS settings
219+
- Scans for `.raw()` SQL queries
220+
221+
**bandit-scan.sh**
222+
- Runs Bandit security scanner
223+
- Excludes migrations, tests, venv
224+
- Reports high/medium/low severity issues
225+
226+
**npm-audit.sh**
227+
- Runs `npm audit` if `package.json` exists
228+
- Auto-attempts `npm audit fix`
229+
- Skips if NPM not used
230+
231+
### Test Pyramid Validation
232+
233+
**test-pyramid.sh**
234+
- Counts tests by mark: `@pytest.mark.unit`, `@pytest.mark.integration`, `@pytest.mark.e2e`
235+
- Calculates percentages
236+
- Validates pyramid ratios (70% unit, 20% integration, 10% e2e)
237+
238+
**test-execution-time.sh**
239+
- Runs pytest with `--durations=10`
240+
- Reports slowest tests
241+
- Validates total time < 2 minutes
242+
- Warns about slow tests (>5s)
243+
244+
## Best Practices
245+
246+
1. **Run locally before pushing:**
247+
```bash
248+
./scripts/ci/run-all-checks.sh --fail-fast
249+
```
250+
251+
2. **Run specific suite when debugging:**
252+
```bash
253+
./scripts/ci/run-all-checks.sh --only security --verbose
254+
```
255+
256+
3. **Integrate in pre-push hook:**
257+
```bash
258+
# .git/hooks/pre-push
259+
./scripts/ci/run-all-checks.sh || exit 1
260+
```
261+
262+
4. **Run in CI pipeline:**
263+
- Always use `--fail-fast` in CI
264+
- Capture exit code for pipeline status
265+
- Archive logs for debugging
266+
267+
## Troubleshooting
268+
269+
**Issue: Script not executable**
270+
```bash
271+
chmod +x scripts/ci/**/*.sh
272+
```
273+
274+
**Issue: Python module not found**
275+
```bash
276+
# Activate virtualenv first
277+
source venv/bin/activate
278+
./scripts/ci/run-all-checks.sh
279+
```
280+
281+
**Issue: Database connection failed**
282+
```bash
283+
# Check .env file exists
284+
# Verify DATABASE_URL is set
285+
# Run migrations
286+
python manage.py migrate
287+
```
288+
289+
**Issue: Bandit not installed**
290+
```bash
291+
pip install bandit
292+
# Script will auto-install if missing
293+
```
294+
295+
## Future Enhancements
296+
297+
- [ ] Add `gate-audit-contract.sh` (validate audit log fields)
298+
- [ ] Add `gate-permission-coverage.sh` (verify capabilities have tests)
299+
- [ ] Add `performance/load-test.sh` (API load testing)
300+
- [ ] Add `accessibility/a11y-check.sh` (frontend accessibility)
301+
- [ ] Integrate with Prometheus for metrics collection
302+
303+
## Related Documentation
304+
305+
- [TDD Agent](../ai/agents/tdd/README.md)
306+
- [PromptOps Framework](../../docs/backend/permisos/promptops/CONTRIBUTING.md)
307+
- [Route Lint Gate](../../docs/backend/permisos/promptops/gates/route-lint.md)
308+
309+
---
310+
311+
**Generated by:** CI/CD Shell Scripts Initiative
312+
**Date:** 2025-11-11
313+
**Project:** IACT - Sistema de Permisos Granular

0 commit comments

Comments
 (0)