Skip to content
This repository was archived by the owner on Jun 23, 2026. It is now read-only.

Commit 6421f09

Browse files
kanywstkanywst
committed
ci: harden — tests, lint, type-check, CodeQL, audit, pre-commit
What changed: - Replace pr-checks.yml with ci.yml — 6 parallel jobs: lint-python (ruff check + format), lint-web (astro check), lint-meta (markdownlint + actionlint), test-python (pytest + coverage XML artefact), build-python (compileall), build-web (astro build). Runs on PR and on push to main (paths-ignore for data-only commits). - Add codeql.yml for python + javascript-typescript with the security-and-quality query suite. Weekly cron + on PR/push to main. - Add audit.yml: pip-audit, npm audit --audit-level=high, license reports for both, and a copyleft gate (fails on GPL / AGPL in the Python tree, allows LGPL). Weekly cron + on deps-file PRs. - Add pyproject.toml with ruff and pytest config, pinning py312 + enabling rules E/W/F/I/B/UP/SIM. Coverage source = awsdd. - Add requirements-dev.txt (pytest, pytest-cov, ruff, pip-audit, pip-licenses). - Add .markdownlint-cli2.jsonc mirroring kt's global config so CI and local agree on the same disabled rules. - Add .pre-commit-config.yaml (ruff, markdownlint, actionlint, basic hygiene hooks) for opt-in local enforcement. - Add pytest suite at tests/ (24 tests, 71% line coverage): score, normalize, report, collect_rss, collect_github, plus a full normalize → score → report pipeline smoke test against fixtures (no network). - Refactor scripts/awsdd/collect_rss.py and collect_github.py to expose pure entry_to_item / release_to_item conversion functions so the collectors can be unit-tested without hitting RSS or the GitHub API. The collect() entry points are unchanged. - Apply ruff format + auto-fixes across the awsdd package (datetime.UTC alias, import ordering, no behavior change). - Makefile: add dev-install / test / lint / format / audit targets. - README: add a CI table and a Local commands block listing the new make targets. PR gating relies on branch protection (Settings → Branches) requiring ci.yml's job statuses to pass before merge into main. That's a repo config step, not in this commit.
1 parent 238f885 commit 6421f09

27 files changed

Lines changed: 894 additions & 83 deletions

‎.coverage‎

68 KB
Binary file not shown.

‎.github/workflows/audit.yml‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
name: audit
2+
3+
on:
4+
schedule:
5+
- cron: '0 6 * * 1'
6+
workflow_dispatch:
7+
pull_request:
8+
paths:
9+
- 'requirements*.txt'
10+
- 'web/package*.json'
11+
- '.github/workflows/audit.yml'
12+
13+
permissions:
14+
contents: read
15+
16+
jobs:
17+
python-vuln:
18+
runs-on: ubuntu-latest
19+
steps:
20+
- uses: actions/checkout@v4
21+
- uses: actions/setup-python@v5
22+
with:
23+
python-version: '3.12'
24+
- run: pip install pip-audit
25+
- name: pip-audit (runtime + dev)
26+
run: pip-audit -r requirements.txt -r requirements-dev.txt
27+
28+
python-license:
29+
runs-on: ubuntu-latest
30+
steps:
31+
- uses: actions/checkout@v4
32+
- uses: actions/setup-python@v5
33+
with:
34+
python-version: '3.12'
35+
- run: pip install -r requirements.txt -r requirements-dev.txt pip-licenses
36+
- name: report licenses
37+
run: pip-licenses --format=markdown --output-file=python-licenses.md
38+
- name: fail on copyleft (GPL family except LGPL)
39+
run: |
40+
if pip-licenses --format=plain-vertical --with-license-file=false \
41+
| grep -E '^(License: )?(GNU )?(Affero )?General Public License' \
42+
| grep -v 'Lesser'; then
43+
echo "::error::Copyleft license found in dependency tree."
44+
exit 1
45+
fi
46+
- uses: actions/upload-artifact@v4
47+
with:
48+
name: python-licenses
49+
path: python-licenses.md
50+
retention-days: 30
51+
52+
npm-vuln:
53+
runs-on: ubuntu-latest
54+
steps:
55+
- uses: actions/checkout@v4
56+
- uses: actions/setup-node@v4
57+
with:
58+
node-version: '22'
59+
- run: npm install --prefix web
60+
- name: npm audit (production, high+)
61+
run: npm audit --prefix web --omit=dev --audit-level=high
62+
63+
npm-license:
64+
runs-on: ubuntu-latest
65+
steps:
66+
- uses: actions/checkout@v4
67+
- uses: actions/setup-node@v4
68+
with:
69+
node-version: '22'
70+
- run: npm install --prefix web
71+
- name: report licenses
72+
run: npx --yes license-checker --start web --production --summary > npm-licenses.txt
73+
- uses: actions/upload-artifact@v4
74+
with:
75+
name: npm-licenses
76+
path: npm-licenses.txt
77+
retention-days: 30

‎.github/workflows/ci.yml‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
name: ci
2+
3+
on:
4+
pull_request:
5+
push:
6+
branches: [main]
7+
paths-ignore:
8+
- 'tracks/*/data/**'
9+
- 'tracks/*/reports/**'
10+
- '_notes/**'
11+
workflow_dispatch:
12+
13+
permissions:
14+
contents: read
15+
16+
concurrency:
17+
group: ci-${{ github.ref }}
18+
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
19+
20+
jobs:
21+
lint-python:
22+
runs-on: ubuntu-latest
23+
steps:
24+
- uses: actions/checkout@v4
25+
- uses: actions/setup-python@v5
26+
with:
27+
python-version: '3.12'
28+
cache: pip
29+
cache-dependency-path: |
30+
requirements.txt
31+
requirements-dev.txt
32+
- run: pip install -r requirements.txt -r requirements-dev.txt
33+
- name: ruff check
34+
run: ruff check .
35+
- name: ruff format --check
36+
run: ruff format --check .
37+
38+
lint-web:
39+
runs-on: ubuntu-latest
40+
steps:
41+
- uses: actions/checkout@v4
42+
- uses: actions/setup-node@v4
43+
with:
44+
node-version: '22'
45+
- run: npm install --prefix web
46+
- name: astro check (type-check Astro + TS)
47+
run: npm run check --prefix web
48+
49+
lint-meta:
50+
runs-on: ubuntu-latest
51+
steps:
52+
- uses: actions/checkout@v4
53+
- uses: actions/setup-node@v4
54+
with:
55+
node-version: '22'
56+
- name: markdownlint-cli2
57+
run: npx --yes markdownlint-cli2 "**/*.md"
58+
- name: actionlint
59+
uses: raven-actions/actionlint@v2
60+
with:
61+
fail_on_error: true
62+
63+
test-python:
64+
runs-on: ubuntu-latest
65+
steps:
66+
- uses: actions/checkout@v4
67+
- uses: actions/setup-python@v5
68+
with:
69+
python-version: '3.12'
70+
cache: pip
71+
cache-dependency-path: |
72+
requirements.txt
73+
requirements-dev.txt
74+
- run: pip install -r requirements.txt -r requirements-dev.txt
75+
- name: pytest
76+
run: pytest --cov=awsdd --cov-report=term --cov-report=xml
77+
- uses: actions/upload-artifact@v4
78+
if: always()
79+
with:
80+
name: coverage
81+
path: coverage.xml
82+
retention-days: 14
83+
84+
build-python:
85+
runs-on: ubuntu-latest
86+
steps:
87+
- uses: actions/checkout@v4
88+
- uses: actions/setup-python@v5
89+
with:
90+
python-version: '3.12'
91+
- run: python -m compileall -q scripts/awsdd
92+
93+
build-web:
94+
runs-on: ubuntu-latest
95+
steps:
96+
- uses: actions/checkout@v4
97+
- uses: actions/setup-node@v4
98+
with:
99+
node-version: '22'
100+
- run: npm install --prefix web
101+
- run: npm run build --prefix web

‎.github/workflows/codeql.yml‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
name: codeql
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
branches: [main]
8+
schedule:
9+
- cron: '0 12 * * 1'
10+
11+
permissions:
12+
contents: read
13+
security-events: write
14+
actions: read
15+
16+
jobs:
17+
analyze:
18+
runs-on: ubuntu-latest
19+
strategy:
20+
fail-fast: false
21+
matrix:
22+
language: [python, javascript-typescript]
23+
steps:
24+
- uses: actions/checkout@v4
25+
- uses: github/codeql-action/init@v3
26+
with:
27+
languages: ${{ matrix.language }}
28+
queries: security-and-quality
29+
- uses: github/codeql-action/autobuild@v3
30+
- uses: github/codeql-action/analyze@v3
31+
with:
32+
category: /language:${{ matrix.language }}

‎.github/workflows/pr-checks.yml‎

Lines changed: 0 additions & 27 deletions
This file was deleted.

‎.markdownlint-cli2.jsonc‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
{
2+
"config": {
3+
"default": true,
4+
"MD013": false,
5+
"MD024": false,
6+
"MD033": { "allowed_elements": ["picture", "source", "img"] },
7+
"MD036": false,
8+
"MD041": false,
9+
"MD060": false
10+
},
11+
"ignores": [
12+
"**/node_modules/**",
13+
"web/dist/**",
14+
"web/.astro/**",
15+
".venv/**",
16+
"_notes/**",
17+
"tracks/*/reports/**",
18+
"tracks/*/data/**"
19+
]
20+
}

‎.pre-commit-config.yaml‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
repos:
2+
- repo: https://github.com/astral-sh/ruff-pre-commit
3+
rev: v0.8.4
4+
hooks:
5+
- id: ruff
6+
args: [--fix]
7+
- id: ruff-format
8+
9+
- repo: https://github.com/DavidAnson/markdownlint-cli2
10+
rev: v0.18.1
11+
hooks:
12+
- id: markdownlint-cli2
13+
14+
- repo: https://github.com/rhysd/actionlint
15+
rev: v1.7.7
16+
hooks:
17+
- id: actionlint
18+
19+
- repo: https://github.com/pre-commit/pre-commit-hooks
20+
rev: v5.0.0
21+
hooks:
22+
- id: end-of-file-fixer
23+
- id: trailing-whitespace
24+
- id: check-yaml
25+
- id: check-json
26+
- id: check-added-large-files
27+
args: ["--maxkb=10000"]

‎Makefile‎

Lines changed: 17 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
TRACKS := iam security whats-new releases
22

3-
.PHONY: install update weekly new-track new-deep-dive
3+
.PHONY: install update weekly new-track new-deep-dive test lint format audit dev-install
44

55
install:
66
@for t in $(TRACKS); do echo "=== install: $$t ==="; $(MAKE) -C tracks/$$t install || exit $$?; done
@@ -19,3 +19,19 @@ new-deep-dive:
1919
@test -n "$(TRACK)" || (echo "TRACK=<track> required" >&2; exit 1)
2020
@test -n "$(TOPIC)" || (echo "TOPIC=<topic> required" >&2; exit 1)
2121
bash scripts/new-deep-dive.sh "$(TRACK)" "$(TOPIC)"
22+
23+
dev-install:
24+
pip install -r requirements.txt -r requirements-dev.txt
25+
26+
test:
27+
pytest --cov=awsdd --cov-report=term
28+
29+
lint:
30+
ruff check .
31+
32+
format:
33+
ruff format .
34+
35+
audit:
36+
pip-audit -r requirements.txt -r requirements-dev.txt
37+
cd web && npm audit --omit=dev --audit-level=high

‎README.md‎

Lines changed: 22 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -54,14 +54,34 @@ web/ # Astro 6 + Tailwind v4 + recharts (React isl
5454
## Local
5555

5656
```bash
57-
pip install -r requirements.txt
58-
make update # daily pipeline for all tracks
57+
make dev-install # runtime + dev deps
58+
make test # pytest + coverage
59+
make lint # ruff check
60+
make format # ruff format
61+
make audit # pip-audit + npm audit (production deps)
62+
63+
make update # daily pipeline for all tracks (hits network)
5964
make -C tracks/iam weekly # single track, weekly mode
6065
cd web && npm install && npm run build
6166
```
6267

6368
Python 3.12 / Node 22+ (Astro 6 requirement).
6469

70+
## CI
71+
72+
| Workflow | Trigger | Purpose |
73+
| -------------------- | -------------------------------------- | ------------------------------------------------------------------------ |
74+
| `ci.yml` | PR + push to `main` (code paths only) | ruff, astro check, markdownlint, actionlint, pytest+coverage, builds |
75+
| `codeql.yml` | PR + push to `main` + weekly cron | CodeQL SAST for Python and TS/JS |
76+
| `audit.yml` | weekly cron + manual + deps PR | `pip-audit`, `npm audit --audit-level=high`, license report (copyleft gate) |
77+
| `daily-update.yml` | 06:00 UTC cron | per-track collect / score / report; commits artefacts to `main` |
78+
| `weekly-digest.yml` | Mon 08:00 UTC cron | per-track weekly digest; commits artefact to `main` |
79+
| `deploy-pages.yml` | push to `main` (web/, tracks/ changes) | builds `web/dist` and publishes to GitHub Pages |
80+
81+
PRs into `main` should be gated by branch protection requiring `ci.yml` jobs to pass. Configure once in repo Settings → Branches.
82+
83+
`pre-commit` config at [`.pre-commit-config.yaml`](./.pre-commit-config.yaml) covers ruff, markdownlint, actionlint, and basic hygiene hooks. Opt-in for contributors (`pre-commit install`).
84+
6585
## Add a track or a deep-dive
6686

6787
```bash

‎pyproject.toml‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
[project]
2+
name = "awsdd"
3+
version = "0.1.0"
4+
description = "aws-deepdive shared collection / scoring / reporting package"
5+
requires-python = ">=3.12"
6+
7+
[tool.ruff]
8+
line-length = 100
9+
target-version = "py312"
10+
extend-exclude = [".venv", "web/node_modules", "web/dist", "web/.astro", "_notes"]
11+
12+
[tool.ruff.lint]
13+
select = ["E", "W", "F", "I", "B", "UP", "SIM"]
14+
ignore = [
15+
"E501", # line length handled by formatter
16+
"SIM108", # ternary nags
17+
]
18+
19+
[tool.ruff.lint.per-file-ignores]
20+
"tests/**" = ["B011"]
21+
22+
[tool.pytest.ini_options]
23+
testpaths = ["tests"]
24+
addopts = "-ra --strict-markers"
25+
pythonpath = ["scripts"]
26+
27+
[tool.coverage.run]
28+
source = ["awsdd"]
29+
branch = true
30+
31+
[tool.coverage.report]
32+
show_missing = true
33+
skip_covered = false
34+
exclude_lines = [
35+
"pragma: no cover",
36+
"if __name__ == .__main__.:",
37+
]

0 commit comments

Comments
 (0)