Skip to content

Commit d3c5844

Browse files
nervghnervgh
committed
chore(dev): add implementation plan, spec, and tasks for Python package
ecosystems Document the new Python package ecosystems feature for the packages directive, including types, alias resolution, command generation, and SBOM integration. Signed-off-by: Alexandr Zaytsev <alexandr.zaytsev@flant.com>
1 parent b7d648f commit d3c5844

3 files changed

Lines changed: 372 additions & 0 deletions

File tree

‎spec/python-packages/plan.md‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
---
2+
status: migrated
3+
feature: python-packages
4+
created: 2026-07-10
5+
source: branch feat-sbom-get-python-sbom
6+
---
7+
8+
# Implementation Plan: Python Package Ecosystems
9+
10+
## Technical Context
11+
12+
This feature adds Python package manager support to werf's `packages` config directive and its SBOM integration. It extends the existing Go-module pattern into a general `FileBasedSpec` model.
13+
14+
### Frameworks and Libraries
15+
16+
- **Go standard library** (`fmt`, `maps`, `sort`, `slices`, `strings`) — used in the ecosystem registry and command generation
17+
- **gopkg.in/yaml.v2** — YAML unmarshaling of raw config directives
18+
- **syft** (via `pkg/sbom/scanner`) — SBOM component cataloging using `python-package-cataloger`
19+
- **ginkgo/gomega** (test) — BDD-style tests for config parsing and SBOM filtering
20+
- **CycloneDX** (via `github.com/CycloneDX/cyclonedx-go`) — BOM data structures
21+
22+
### Ecosystem Registry Design
23+
24+
A central `ecosystems` map (`map[PackagesDirectiveType]PackageEcosystem`) registers all file-based package types. Each entry holds:
25+
26+
| Field | Purpose |
27+
|-------|---------|
28+
| `Type` | Canonical type constant |
29+
| `Aliases` | Short name variants (e.g., `uv` → `python-uv`) |
30+
| `DefaultSpec` | Default manifest file name |
31+
| `DefaultLock` | Default lock file name (empty if not applicable) |
32+
| `InstallCmd` | Function returning the package manager install command |
33+
| `CatalogerName` | syft cataloger for SBOM generation |
34+
35+
The alias index (`aliasToType`) is built at package init time from the ecosystem registry.
36+
37+
### Architecture Changes
38+
39+
1. **`GoModSpec` → `FileBasedSpec`** — Struct renamed and generalized with same fields (`Workdir`, `Spec`, `Lock`)
40+
2. **`fillGoModSpec` → `fillFileBasedSpec`** — Reads defaults from ecosystem registry; validates lock availability; returns error on invalid spec type
41+
3. **`GeneratePackagesCommands`** — Refactored from `switch` to registry-based dispatch: OSPM has a dedicated `if` branch, then the rest fall through to ecosystem lookup
42+
4. **`buildResolvers`** in `managedinput.go` — Dynamically constructs input resolvers from `config.Ecosystems()`, sorted alphabetically for deterministic ordering
43+
5. **`rawPackagesDirective.toDirective`** — Added alias resolution before type canonicalization
44+
45+
## Project Structure
46+
47+
```
48+
pkg/config/
49+
packages_directive.go — Ecosystem registry, types, FileBasedSpec
50+
packages_commands.go — Command generation
51+
raw_packages_directive.go — YAML parsing and validation
52+
helpers_test.go — Shared test helper (directivesFromYaml)
53+
packages_directive_go_mod_test.go — Go-mod specific tests (refactored)
54+
packages_directive_python_test.go — Python-specific tests (new)
55+
raw_packages_directive_test.go — Validation and normalization tests (extended)
56+
57+
pkg/sbom/managedinput/
58+
managedinput.go — buildResolvers, ToCatalogers, FilterBOMBySourcePaths
59+
managedinput_test.go — SBOM tests for python catalogers (extended)
60+
61+
test/e2e/sbom/
62+
pip_test.go — E2E: pip install + SBOM
63+
poetry_test.go — E2E: poetry install + SBOM
64+
uv_test.go — E2E: uv sync + SBOM
65+
_fixtures/inject/pip_simple/ — Fixture: requirements.txt
66+
_fixtures/inject/poetry_simple/ — Fixture: poetry project
67+
_fixtures/inject/uv_simple/ — Fixture: uv project
68+
69+
docs/_data/
70+
werf_yaml.yml — Updated type descriptions for Python types
71+
```
72+
73+
## Complexity Assessment
74+
75+
- **File count**: ~12 files changed (6 source, 5 unit test, 4 e2e)
76+
- **Source LOC added**: ~250 (net) across ecosystem registry, command dispatch, SBOM resolvers
77+
- **Test LOC added**: ~400 (unit) + ~130 (e2e) + fixture data
78+
- **Dependency changes**: None — all new types use existing packages and tools
79+
- **Risk**: Low — the refactoring maintains backward compatibility for existing `go-mod` and `os-pm` usage; the `GoModSpec` → `FileBasedSpec` rename is internal only
80+
81+
## Design Decisions
82+
83+
1. **Ecosystem registry over switch** — A map-based registry makes adding new package types purely declarative (add a constant + entry). The old switch on `PackagesDirectiveType` is replaced entirely for file-based types.
84+
2. **Aliases resolved at parse time** — `rawPackagesDirective.toDirective()` canonicalizes aliases before storing, so downstream code only sees canonical types.
85+
3. **Separate OSPM handling** — OS package manager (`os-pm`) remains distinct because its spec structure (`PackagesSpec.Packages` list) is incompatible with the file-based model.
86+
4. **Deterministic resolver ordering** — `buildResolvers()` sorts types alphabetically so that SBOM cataloger order is stable across invocations and builds.
87+
5. **Quoted `workdir` in shell commands** — All generated `cd` commands now use `%q` formatting (`cd "/app" && ...`) to handle paths with spaces or special characters.

‎spec/python-packages/spec.md‎

Lines changed: 154 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,154 @@
1+
---
2+
status: migrated
3+
feature: python-packages
4+
created: 2026-07-10
5+
source: branch feat-sbom-get-python-sbom
6+
---
7+
8+
# Python Package Ecosystems for werf Packages Directive
9+
10+
## User Scenarios
11+
12+
### Scenario: Declare uv-managed Python dependencies
13+
14+
A user has a Python project managed by **uv** with a `pyproject.toml` and `uv.lock`. They add a `packages` directive in `werf.yaml`:
15+
16+
```yaml
17+
packages:
18+
- type: python-uv
19+
workdir: /app
20+
```
21+
22+
- **WHEN** the build runs
23+
- **THEN** werf runs `cd "/app" && uv sync --frozen` inside the build container, installing locked dependencies
24+
- **AND** `--frozen` ensures the build fails if `uv.lock` is missing or outdated
25+
- **AND** syft's `python-package-cataloger` scans `/app/pyproject.toml` and `/app/uv.lock` to produce the SBOM
26+
- **AND** SBOM filtering keeps only components found by those declared paths
27+
28+
### Scenario: Declare pip-managed Python dependencies
29+
30+
A user has a `requirements.txt` file without a lock mechanism. They use:
31+
32+
```yaml
33+
packages:
34+
- type: python-pip
35+
workdir: /app
36+
```
37+
38+
- **WHEN** the build runs
39+
- **THEN** werf runs `cd "/app" && pip install --no-cache-dir -r "requirements.txt"` inside the build container
40+
- **AND** the `lock` field is rejected by config validation (pip has no lock semantics)
41+
- **AND** syft's `python-package-cataloger` scans `/app/requirements.txt` for the SBOM
42+
43+
### Scenario: Declare poetry-managed Python dependencies
44+
45+
A user has a project managed by **poetry** with `pyproject.toml` and `poetry.lock`:
46+
47+
```yaml
48+
packages:
49+
- type: python-poetry
50+
workdir: /app
51+
```
52+
53+
- **WHEN** the build runs
54+
- **THEN** werf runs `cd "/app" && poetry install --no-root` inside the build container
55+
- **AND** syft's `python-package-cataloger` scans `/app/pyproject.toml` and `/app/poetry.lock` for the SBOM
56+
57+
### Scenario: Use short aliases
58+
59+
A user prefers concise YAML:
60+
61+
```yaml
62+
packages:
63+
- type: uv
64+
workdir: /app
65+
```
66+
67+
- **WHEN** werf parses the config
68+
- **THEN** the alias is canonicalized to `python-uv` via the `aliasToType` index
69+
- **AND** all defaults (`pyproject.toml`, `uv.lock`, `uv sync --frozen`, `python-package-cataloger`) apply identically
70+
71+
### Scenario: Mix Python ecosystems with other package types
72+
73+
A project uses both Go modules and Python dependencies:
74+
75+
```yaml
76+
packages:
77+
- type: go-mod
78+
workdir: /app
79+
- type: python-uv
80+
workdir: /lib
81+
- type: os-pm
82+
packages:
83+
- curl
84+
```
85+
86+
- **WHEN** the build runs
87+
- **THEN** all three directives produce commands that are run inside the build container: `go mod download`, `uv sync --frozen`, `pm install curl`
88+
- **AND** each directive contributes its own cataloger for SBOM scanning
89+
90+
## Requirements
91+
92+
### R1: Three Python ecosystem types
93+
94+
The `packages` directive SHALL support types `python-uv`, `python-pip`, and `python-poetry`, each with its own package manager command, default manifest file, and cataloger.
95+
96+
### R2: Short aliases
97+
98+
Types `uv`, `pip`, and `poetry` SHALL be accepted as aliases and canonicalized to their full form at parse time.
99+
100+
### R3: File-based ecosystem abstraction
101+
102+
All file-based package ecosystems (Go modules, Python types) SHALL use a shared `FileBasedSpec` with `Workdir`, `Spec`, and `Lock` fields. The old `GoModSpec` SHALL be removed.
103+
104+
### R4: Ecosystem registry
105+
106+
All file-based ecosystems SHALL be registered in a single `ecosystems` map keyed by `PackagesDirectiveType`. The map SHALL be exposed read-only via `Ecosystems()`.
107+
108+
### R5: Command generation from registry
109+
110+
`GeneratePackagesCommands` SHALL dispatch commands via the ecosystem registry rather than a hardcoded switch. The `PackagesDirectiveTypeOSPM` case remains special because it has a different structure.
111+
112+
### R6: SBOM catalogers from ecosystem registry
113+
114+
`ToCatalogers` and `buildResolvers` in `pkg/sbom/managedinput` SHALL derive cataloger entries dynamically from `config.Ecosystems()` instead of hardcoding them per type.
115+
116+
### R7: Lock validation (determinism)
117+
118+
- For `python-uv`: `uv sync --frozen` SHALL be used, ensuring `uv.lock` exists
119+
- For `python-poetry`: `poetry.lock` SHALL be the default lock file
120+
- For `python-pip`: the `lock` field in YAML SHALL be rejected at config validation (pip has no lock semantics)
121+
122+
### R8: Default file names
123+
124+
| Type | Default spec | Default lock |
125+
|------|-------------|--------------|
126+
| `python-uv` | `pyproject.toml` | `uv.lock` |
127+
| `python-pip` | `requirements.txt` | (empty) |
128+
| `python-poetry` | `pyproject.toml` | `poetry.lock` |
129+
| `go-mod` | `go.mod` | `go.sum` |
130+
131+
### R9: Install commands
132+
133+
| Type | Command |
134+
|------|---------|
135+
| `python-uv` | `uv sync --frozen` |
136+
| `python-pip` | `pip install --no-cache-dir -r <spec>` |
137+
| `python-poetry` | `poetry install --no-root` |
138+
| `go-mod` | `go mod download` |
139+
140+
## Success Criteria
141+
142+
- SC1: A `packages` entry with `type: python-uv` (or alias `uv`) and `workdir: /app` successfully installs dependencies and generates an SBOM containing `requests@2.32.3` when `uv.lock` lists that dependency.
143+
- SC2: A `packages` entry with `type: python-pip` (or alias `pip`) and `workdir: /app` successfully installs dependencies and generates an SBOM containing `requests@2.32.3` when `requirements.txt` lists that dependency.
144+
- SC3: A `packages` entry with `type: python-poetry` (or alias `poetry`) and `workdir: /app` successfully installs dependencies and generates an SBOM containing `requests@2.32.3` when `poetry.lock` lists that dependency.
145+
- SC4: `python-pip` rejects the `lock` field at config validation.
146+
- SC5: An unknown package type (e.g., `pythonn-uv`) is rejected at config validation.
147+
- SC6: A `python-uv` entry without `workdir` is rejected at config validation.
148+
- SC7: A mixed configuration (`go-mod` + `python-uv` + `os-pm`) generates all expected commands and catalogers correctly.
149+
150+
## Assumptions
151+
152+
- Python package managers (`uv`, `pip`, `poetry`) must be pre-installed in the builder image; werf does not install them.
153+
- The lock file is expected to be present in the build context by the respective package manager. For `uv`, the `--frozen` flag enforces this. For `poetry`, the lock file is expected but not enforced by a flag (the user must ensure it exists).
154+
- `pip` has no lock semantics; users are expected to pin versions in `requirements.txt` directly.

‎spec/python-packages/tasks.md‎

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
---
2+
status: migrated
3+
feature: python-packages
4+
created: 2026-07-10
5+
source: branch feat-sbom-get-python-sbom
6+
---
7+
8+
# Tasks: Python Package Ecosystems
9+
10+
All tasks are completed — this feature was built incrementally on the branch and is now fully implemented.
11+
12+
## Task Group 1: Core Types and Ecosystem Registry
13+
14+
- [x] **T1.1** Add `PythonPackageEcosystem` types: `PackagesDirectiveTypePythonUV`, `PackagesDirectiveTypePythonPip`, `PackagesDirectiveTypePythonPoetry`
15+
- Files: `pkg/config/packages_directive.go`
16+
- Status: Constants added with canonical naming `python-{uv,pip,poetry}`
17+
18+
- [x] **T1.2** Rename `GoModSpec` to `FileBasedSpec` and update all references
19+
- Files: `pkg/config/packages_directive.go` + all callers
20+
- Status: Struct used for Go-mod and all Python ecosystems
21+
22+
- [x] **T1.3** Implement `PackageEcosystem` struct and `ecosystems` map registry
23+
- Files: `pkg/config/packages_directive.go`
24+
- Status: Registry with Type, Aliases, DefaultSpec, DefaultLock, InstallCmd, CatalogerName
25+
26+
- [x] **T1.4** Implement alias resolution (`aliasToType` index) and `Ecosystems()` accessor
27+
- Files: `pkg/config/packages_directive.go`
28+
- Status: Aliases `uv/pip/poetry` canonicalized at parse time
29+
30+
## Task Group 2: Config Parsing and Validation
31+
32+
- [x] **T2.1** Refactor `rawPackagesDirective.toDirective()` with alias resolution and ecosystem dispatch
33+
- Files: `pkg/config/raw_packages_directive.go`
34+
- Status: OSPM handled via `if`; file-based types delegated to `fillFileBasedSpec`
35+
36+
- [x] **T2.2** Implement `fillFileBasedSpec` (replaces `fillGoModSpec`)
37+
- Files: `pkg/config/raw_packages_directive.go`
38+
- Status: Reads defaults from ecosystem registry; validates spec type (must be string); rejects lock for lockless ecosystems (pip)
39+
40+
- [x] **T2.3** Update `PackagesDirective.validate()` for ecosystem-based validation
41+
- Files: `pkg/config/packages_directive.go`
42+
- Status: OSPM checks packages non-empty; file-based checks workdir non-empty; unknown types rejected
43+
44+
## Task Group 3: Command Generation
45+
46+
- [x] **T3.1** Refactor `GeneratePackagesCommands` to use ecosystem registry
47+
- Files: `pkg/config/packages_commands.go`
48+
- Status: OSPM has dedicated `if`; other types dispatch via `ecosystems[pkg.Type].InstallCmd`
49+
50+
- [x] **T3.2** Add Python install commands with `--frozen` determinism for uv and `--no-root` for poetry
51+
- Files: `pkg/config/packages_directive.go` (ecosystem entries)
52+
- Status: `uv sync --frozen`, `pip install --no-cache-dir -r <spec>`, `poetry install --no-root`
53+
54+
## Task Group 4: SBOM Integration
55+
56+
- [x] **T4.1** Refactor `buildResolvers` to construct cataloger resolvers from `config.Ecosystems()`
57+
- Files: `pkg/sbom/managedinput/managedinput.go`
58+
- Status: Dynamic, deterministic sorting by type name
59+
60+
- [x] **T4.2** Update `ToCatalogers` to handle python-package-cataloger
61+
- Files: `pkg/sbom/managedinput/managedinput.go`
62+
- Status: Python directives map to `python-package-cataloger` with correct source paths (spec + lock when available)
63+
64+
- [x] **T4.3** Ensure `FilterBOMBySourcePaths` works with python property paths (syft location keys)
65+
- Files: `pkg/sbom/managedinput/managedinput.go`
66+
- Status: Uses prefix matching against `syft:location:*:path` properties; works identically for go and python components
67+
68+
## Task Group 5: Unit Tests
69+
70+
- [x] **T5.1** Extract `directivesFromYaml` shared test helper
71+
- Files: `pkg/config/helpers_test.go`
72+
- Status: Shared between go-mod and python test files
73+
74+
- [x] **T5.2** Write `packages_directive_python_test.go` — unmarshal, defaults, aliases, error cases
75+
- Files: `pkg/config/packages_directive_python_test.go`
76+
- Status: 14 entries covering all types, aliases, explicit overrides, missing workdir, invalid spec type, unknown type, lock rejection for pip
77+
78+
- [x] **T5.3** Refactor `packages_directive_go_mod_test.go` to use `FileBasedSpec` and shared helper
79+
- Files: `pkg/config/packages_directive_go_mod_test.go`
80+
- Status: All existing tests pass with new struct
81+
82+
- [x] **T5.4** Refactor `raw_packages_directive_test.go` to use DescribeTable for normalizePackages
83+
- Files: `pkg/config/raw_packages_directive_test.go`
84+
- Status: 6 entries replacing individual It blocks; python smoke test added
85+
86+
- [x] **T5.5** Add python-specific `ToCatalogers` and `FilterBOMBySourcePaths` tests
87+
- Files: `pkg/sbom/managedinput/managedinput_test.go`
88+
- Status: 5 entries testing python-package-cataloger mapping, pip lockless, uv+poetry lock paths, path filtering, go-mod regression
89+
90+
- [x] **T5.6** Add `buildResolvers` determinism test
91+
- Files: `pkg/sbom/managedinput/managedinput_test.go`
92+
- Status: Verifies deterministic ordering across 20 invocations and alphabetical sort
93+
94+
## Task Group 6: E2E Tests
95+
96+
- [x] **T6.1** Create pip e2e test with fixture (`pip_simple`)
97+
- Files: `test/e2e/sbom/pip_test.go` + `_fixtures/inject/pip_simple/`
98+
- Status: Validates `requests@2.32.3` in BOM after `pip install -r requirements.txt`
99+
100+
- [x] **T6.2** Create poetry e2e test with fixture (`poetry_simple`)
101+
- Files: `test/e2e/sbom/poetry_test.go` + `_fixtures/inject/poetry_simple/`
102+
- Status: Validates `requests@2.32.3` in BOM after `poetry install --no-root`
103+
104+
- [x] **T6.3** Create uv e2e test with fixture (`uv_simple`)
105+
- Files: `test/e2e/sbom/uv_test.go` + `_fixtures/inject/uv_simple/`
106+
- Status: Validates `requests@2.32.3` in BOM after `uv sync --frozen`
107+
108+
## Task Group 7: Documentation
109+
110+
- [x] **T7.1** Update `docs/_data/werf_yaml.yml` with Python type descriptions, aliases, defaults
111+
- Files: `docs/_data/werf_yaml.yml`
112+
- Status: `type`, `workdir`, `spec`, `lock` fields updated for all three Python ecosystems
113+
114+
## Task Group 8: Incidental Cleanup (extracted from PR)
115+
116+
- [x] **T8.1** Remove `serviceLabelsConfigMutation` from `verity_annotation.go` and simplify `SignStage.MutateImage`
117+
- Files: `pkg/build/stage/verity_annotation.go`, `pkg/build/stage/sign.go`
118+
- Status: Service labels propagation moved into the registry mutation flow
119+
120+
- [x] **T8.2** Clean up `mutateImage` in docker_registry (remove stale cf snapshot logic)
121+
- Files: `pkg/docker_registry/api/mutate.go`
122+
- Status: Removed `cfBeforeLayerMutation` read that was no longer needed
123+
124+
- [x] **T8.3** Delete obsolete mutate tests
125+
- Files: `pkg/docker_registry/api/mutate_test.go`, `pkg/docker_registry/api/suite_test.go`
126+
- Status: Tests removed as they tested removed behavior
127+
128+
## Gaps Identified
129+
130+
1. ⚠️ **poetry lock enforcement** — Unlike `uv --frozen`, poetry's `install --no-root` doesn't have a built-in flag to reject a missing or outdated `poetry.lock`. The lock file existence is expected but not enforced at the package manager level.
131+
2. ⚠️ **E2E tests with native Buildah** — The e2e tests have `XEntry` (pending) for `native-chroot` and `native-rootless` container backends, indicating these aren't yet validated for Python SBOM scenarios.

0 commit comments

Comments
 (0)