Skip to content

Commit 320ec49

Browse files
ruromeroclaude
ruromero
and
claude
authored
feat(pip): add pyproject.toml support via pip --dry-run --report (guacsec#465)
* feat(pip): add pyproject.toml support via pip --dry-run --report Add Python_pip_pyproject provider that resolves PEP 621 pyproject.toml dependencies using `pip install --dry-run --ignore-installed --report`. This is the fallback provider when no lock file (uv.lock/poetry.lock) is found, enabling analysis of standard pyproject.toml projects without requiring uv or poetry. The pip report JSON provides resolved versions, requires_dist for building the dependency tree, and requested flags for direct vs transitive classification. Extras-only deps are filtered out. Supports TRUSTIFY_DA_PIP_REPORT env var for testing without pip. Implements TC-4065 Assisted-by: Claude Code * fix(python): align pip pyproject _getDependencyData signature with base class Python_pip_pyproject._getDependencyData() had only 3 parameters (manifestDir, parsed, opts) instead of the 4 defined by the base class (manifestDir, workspaceDir, parsed, opts). When called from _createSbom() with 4 arguments, workspaceDir was silently received as parsed, and parsed was received as opts, causing argument misalignment. Add the missing _workspaceDir parameter to match the base class contract and sibling providers (python_uv, python_poetry). Implements TC-4098 Assisted-by: Claude Code * fix(python): only exclude root entry from pip report graph, not all dir_info entries _parsePipReport() filtered nonRootPackages by checking p.download_info?.dir_info === undefined, which excluded ALL packages with dir_info (including local/path dependencies). Changed to filter by identity (p !== rootEntry) so only the specific root project entry is excluded, matching the Java implementation's behavior. Implements TC-4099 Assisted-by: Claude Code * refactor(test): parameterize pyproject SBOM tests and add doc comments Extract shared SBOM_CASES constant and use forEach loops to eliminate repetitive stack/component test pairs across uv, poetry, and pip suites. Parameterize validateLockFile and workspace tests similarly. Add JSDoc comments to all 39 test functions. Implements TC-4065 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test(pip): add extras package to pip fixture for realistic extras filtering Add requests[socks] to the test pyproject.toml and include PySocks in the pip_report.json install list. This makes the extras-filtering test exercise the actual code path: PySocks is now present in the report but correctly excluded from the SBOM via _hasExtraMarker. Previously the test trivially passed because PySocks was never in the install list. Implements TC-4065 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test(pip): call pip directly instead of using env var overrides Remove TRUSTIFY_DA_PIP_REPORT setup/teardown from pip tests and let them invoke pip directly. Pin fixture pyproject.toml versions to get deterministic output, add --quiet flag to suppress non-JSON stdout, and update expected SBOMs with real resolved versions. Add testing convention to CONVENTIONS.md and ignore *.egg-info build artifacts. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(test): lower requires-python to >=3.9 to match CI Python version CI uses Python 3.9 (setup-python in test.yml), so pip rejects pyproject.toml fixtures that require >=3.12. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(pip): clean up .egg-info directories created by pip --dry-run pip creates .egg-info as a side effect of --dry-run --report. Snapshot existing .egg-info dirs before invoking pip and remove only newly created ones afterward to avoid polluting the user's project. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test(pip): replace manual ignore assertions with golden SBOM tests The pip_pep621_ignore fixture was missing expected_stack_sbom.json and expected_component_sbom.json, relying on partial manual assertions instead of the standard SBOM_CASES golden-file pattern. Add the golden files and switch to deep.equal comparison. Document the golden SBOM convention in CONVENTIONS.md to prevent this gap in future implementations. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix(python): build true dependency graph instead of tree The pyproject base class built an intermediate tree with a visited set that prevented a dependency from appearing under multiple parents. This caused PySocks to be missing from requests' dependsOn when both requests and urllib3 declared it (via extras). All other providers (npm, cargo, go, maven) already produce true CycloneDX graphs. Replace _buildDependencyTree/_collectTransitive/_addAllDependencies with graph-based BFS reachability + flat edge iteration. No recursion prevention needed — Python resolvers reject circular dependencies. Regenerate all pyproject golden SBOM files (pip, uv, poetry, workspace). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: use GraphEntry typedef instead of inline type in JSDoc Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent 3621bb7 commit 320ec49

24 files changed

Lines changed: 1282 additions & 863 deletions

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,3 +10,4 @@ build
1010
target
1111
.npmrc
1212
src/providers/*.wasm
13+
*.egg-info

‎CONVENTIONS.md‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
# Coding Conventions
2+
3+
<!-- This file documents project-specific coding standards for exhort-javascript-api. -->
4+
5+
## Language and Framework
6+
7+
- **Primary Language**: JavaScript (ES modules, `"type": "module"` in package.json)
8+
- **TypeScript**: Configuration present but code is primarily JavaScript with JSDoc
9+
- **Node.js**: Requires Node >= 20.0.0, npm >= 11.5.1
10+
- **CLI**: `yargs` for command-line argument parsing
11+
- **Parsing Libraries**: `fast-xml-parser`, `fast-toml`, `smol-toml`, `tree-sitter-requirements`
12+
13+
## Code Style
14+
15+
- **Linter**: ESLint with recommended config + editorconfig + import plugins
16+
- **Indentation**: Tabs (4 spaces for YAML/Markdown)
17+
- **Line endings**: LF
18+
- **Max line length**: 100 (120 for Markdown)
19+
- **Charset**: UTF-8, final newline, trim trailing whitespace
20+
- **Import ordering** (ESLint enforced): builtin, external, internal, parent, sibling, index — alphabetical within groups
21+
- **Strict equality**: `eqeqeq: ["warn", "always", {"null": "never"}]`
22+
- **Curly braces**: Required (`curly: "warn"`)
23+
- **No throw literals**: `no-throw-literal: "warn"`
24+
- **No Prettier** — ESLint + EditorConfig handle formatting
25+
26+
## Naming Conventions
27+
28+
- **Classes**: PascalCase with underscore-separated language names (`Java_maven`, `Base_java`, `Javascript_npm`)
29+
- **Files**: snake_case for providers (`base_java.js`, `javascript_npm.js`, `python_pip.js`)
30+
- **Test files**: `*.test.js` suffix (`analysis.test.js`, `provider.test.js`)
31+
- **Functions/Methods**: camelCase (`provideComponent()`, `provideStack()`, `validateLockFile()`)
32+
- **Variables**: camelCase (`manifestPath`, `backendUrl`)
33+
- **Constants**: UPPER_SNAKE_CASE (`ecosystem_maven`, `DEFAULT_WORKSPACE_DISCOVERY_IGNORE`)
34+
- **Private class fields**: `#` prefix (`#manifest`, `#cmd`, `#ecosystem`)
35+
- **Protected methods**: `_` prefix (`_lockFileName()`, `_cmdName()`, `_listCmdArgs()`)
36+
37+
## File Organization
38+
39+
```
40+
src/
41+
├── index.js # Main export
42+
├── cli.js # CLI entry point
43+
├── analysis.js # API request handling
44+
├── provider.js # Provider matching logic
45+
├── workspace.js # Workspace discovery
46+
├── tools.js # Utilities
47+
├── sbom.js # SBOM handling
48+
├── cyclone_dx_sbom.js # CycloneDX SBOM generation
49+
├── providers/ # Ecosystem providers
50+
│ ├── base_java.js
51+
│ ├── base_javascript.js
52+
│ ├── java_maven.js
53+
│ ├── javascript_npm.js
54+
│ ├── python_pip.js
55+
│ ├── rust_cargo.js
56+
│ └── processors/ # Specialized processors
57+
├── license/ # License detection
58+
└── oci_image/ # OCI image analysis
59+
60+
test/
61+
├── analysis.test.js
62+
├── provider.test.js
63+
├── tools.test.js
64+
└── providers/ # Provider-specific tests
65+
```
66+
67+
## Error Handling
68+
69+
- **Throw Error objects**: `throw new Error("message")`, `throw new TypeError("message")`
70+
- **No custom error classes** — uses built-in `Error` and `TypeError`
71+
- **HTTP errors**: Check `resp.status`, throw with status code and response text
72+
- **Async errors**: Bubble up naturally via async/await (no blanket try-catch)
73+
- **Validation errors**: Thrown early with descriptive context (manifest type, lock file)
74+
75+
## Testing Conventions
76+
77+
- **Framework**: Mocha with TDD UI (`suite()` / `test()`)
78+
- **Assertions**: Chai with `expect()` syntax
79+
- **Mocking**: Sinon for stubs; MSW (Mock Service Worker) for HTTP mocking
80+
- **Module mocking**: `esmock` with experimental loader
81+
- **Coverage**: C8 with 82% line coverage requirement
82+
- **Test patterns**: `expect(res).to.deep.equal(...)`, `expect(() => ...).to.throw('message')`
83+
- **Higher-order setup**: Functions like `interceptAndRun()` for test setup/teardown
84+
- **Prefer real tool invocations over env var overrides**: Tests should call the actual ecosystem tools (pip, uv, poetry, mvn, npm, etc.) rather than injecting pre-recorded output via `TRUSTIFY_DA_*` environment variables. The CI environment has these tools available. Env var overrides (`TRUSTIFY_DA_PIP_REPORT`, `TRUSTIFY_DA_UV_EXPORT`, etc.) exist for users who lack the tool locally, but tests should exercise the real tool path to catch integration issues.
85+
- **Golden SBOM files for every test fixture**: Every provider test fixture directory must include `expected_stack_sbom.json` and `expected_component_sbom.json` golden files. Tests must use the `SBOM_CASES` pattern to do a full `deep.equal` comparison of the provider output against these golden files. Manual partial assertions (e.g. checking a single component name) are not a substitute — they may be added as supplementary tests but never as the only verification for a fixture.
86+
87+
## Commit Messages
88+
89+
- Likely Conventional Commits format
90+
- DCO (Developer Certificate of Origin) required
91+
- Semantic versioning (`0.3.0` in package.json)
92+
93+
## Dependencies
94+
95+
- **Package manager**: npm with `package-lock.json`
96+
- **Module system**: ES modules with explicit `.js` extensions in relative imports
97+
- **Import convention**: `import fs from 'node:fs'` (node: protocol for built-ins)
98+
- **Environment variables**: Prefixed with `TRUSTIFY_DA_` (e.g., `TRUSTIFY_DA_MVN_PATH`, `TRUSTIFY_DA_TOKEN`, `TRUSTIFY_DA_DEBUG`)
99+
- **Multi-ecosystem support**: npm, pnpm, yarn, Maven, Gradle, pip, cargo, Go modules, Docker/Podman

‎src/provider.js‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@ import Javascript_npm from './providers/javascript_npm.js';
88
import Javascript_pnpm from './providers/javascript_pnpm.js';
99
import Javascript_yarn from './providers/javascript_yarn.js';
1010
import pythonPipProvider from './providers/python_pip.js'
11+
import Python_pip_pyproject from './providers/python_pip_pyproject.js'
1112
import Python_poetry from './providers/python_poetry.js'
1213
import Python_uv from './providers/python_uv.js'
1314
import rustCargoProvider from './providers/rust_cargo.js'
@@ -30,6 +31,7 @@ export const availableProviders = [
3031
pythonPipProvider,
3132
new Python_poetry(),
3233
new Python_uv(),
34+
new Python_pip_pyproject(),
3335
rustCargoProvider]
3436

3537
/**

‎src/providers/base_pyproject.js‎

Lines changed: 38 additions & 75 deletions
Original file line numberDiff line numberDiff line change
@@ -250,64 +250,27 @@ export default class Base_pyproject {
250250
}
251251

252252
/**
253-
* Build dependency tree from graph, starting from direct deps.
253+
* Compute the set of graph nodes reachable from direct deps, excluding ignored.
254254
* @param {Map<string, GraphEntry>} graph
255-
* @param {string[]} directDeps - canonical names of direct deps
255+
* @param {string[]} directDeps
256256
* @param {Set<string>} ignoredDeps
257-
* @param {boolean} includeTransitive
258-
* @returns {DepTreeEntry[]}
257+
* @returns {Set<string>}
259258
* @protected
260259
*/
261-
_buildDependencyTree(graph, directDeps, ignoredDeps, includeTransitive) {
262-
let result = []
263-
264-
for (let key of directDeps) {
265-
if (ignoredDeps.has(key)) { continue }
266-
267-
let entry = graph.get(key)
268-
if (!entry) { continue }
269-
270-
let depTree = []
271-
if (includeTransitive) {
272-
let visited = new Set()
273-
visited.add(key)
274-
this._collectTransitive(graph, entry.children, depTree, ignoredDeps, visited)
260+
_reachableNodes(graph, directDeps, ignoredDeps) {
261+
let reachable = new Set()
262+
let queue = directDeps.filter(k => !ignoredDeps.has(k) && graph.has(k))
263+
while (queue.length > 0) {
264+
let key = queue.shift()
265+
if (reachable.has(key)) { continue }
266+
reachable.add(key)
267+
for (let child of graph.get(key).children) {
268+
if (!ignoredDeps.has(child) && graph.has(child) && !reachable.has(child)) {
269+
queue.push(child)
270+
}
275271
}
276-
277-
result.push({ name: entry.name, version: entry.version, dependencies: depTree })
278-
}
279-
280-
result.sort((a, b) => a.name.toLowerCase().localeCompare(b.name.toLowerCase()))
281-
return result
282-
}
283-
284-
/**
285-
* Recursively collect transitive dependencies.
286-
* @param {Map<string, GraphEntry>} graph
287-
* @param {string[]} childKeys
288-
* @param {DepTreeEntry[]} result - mutated in place
289-
* @param {Set<string>} ignoredDeps
290-
* @param {Set<string>} visited
291-
* @returns {void}
292-
* @protected
293-
*/
294-
_collectTransitive(graph, childKeys, result, ignoredDeps, visited) {
295-
for (let childKey of childKeys) {
296-
let canonKey = this._canonicalize(childKey)
297-
if (ignoredDeps.has(canonKey)) { continue }
298-
if (visited.has(canonKey)) { continue }
299-
visited.add(canonKey)
300-
301-
let entry = graph.get(canonKey)
302-
if (!entry) { continue }
303-
304-
let childDeps = []
305-
this._collectTransitive(graph, entry.children, childDeps, ignoredDeps, visited)
306-
307-
result.push({ name: entry.name, version: entry.version, dependencies: childDeps })
308272
}
309-
310-
result.sort((a, b) => a.name.toLowerCase().localeCompare(b.name.toLowerCase()))
273+
return reachable
311274
}
312275

313276
/**
@@ -320,22 +283,6 @@ export default class Base_pyproject {
320283
return new PackageURL('pypi', undefined, name, version, undefined, undefined)
321284
}
322285

323-
/**
324-
* Recursively add a dependency and its transitive deps to the SBOM.
325-
* @param {PackageURL} source
326-
* @param {DepTreeEntry} dep
327-
* @param {Sbom} sbom
328-
* @returns {void}
329-
* @private
330-
*/
331-
_addAllDependencies(source, dep, sbom) {
332-
let targetPurl = this._toPurl(dep.name, dep.version)
333-
sbom.addDependency(source, targetPurl)
334-
if (dep.dependencies && dep.dependencies.length > 0) {
335-
dep.dependencies.forEach(child => this._addAllDependencies(this._toPurl(dep.name, dep.version), child, sbom))
336-
}
337-
}
338-
339286
/**
340287
* Create SBOM json string for a pyproject.toml project.
341288
* @param {string} manifest - path to pyproject.toml
@@ -353,7 +300,6 @@ export default class Base_pyproject {
353300
let { directDeps, graph } = await this._getDependencyData(manifestDir, workspaceDir, parsed, opts)
354301

355302
let ignoredDeps = this._getIgnoredDeps(manifest)
356-
let dependencies = this._buildDependencyTree(graph, directDeps, ignoredDeps, includeTransitive)
357303

358304
let sbom = new Sbom()
359305
let rootName = this._getProjectName(parsed) || DEFAULT_ROOT_NAME
@@ -362,13 +308,30 @@ export default class Base_pyproject {
362308
let license = this.readLicenseFromManifest(manifest)
363309
sbom.addRoot(rootPurl, license)
364310

365-
dependencies.forEach(dep => {
366-
if (includeTransitive) {
367-
this._addAllDependencies(rootPurl, dep, sbom)
368-
} else {
369-
sbom.addDependency(rootPurl, this._toPurl(dep.name, dep.version))
311+
if (includeTransitive) {
312+
let reachable = this._reachableNodes(graph, directDeps, ignoredDeps)
313+
for (let key of directDeps) {
314+
if (!reachable.has(key)) { continue }
315+
let entry = graph.get(key)
316+
sbom.addDependency(rootPurl, this._toPurl(entry.name, entry.version))
370317
}
371-
})
318+
for (let [key, entry] of graph) {
319+
if (!reachable.has(key)) { continue }
320+
let parentPurl = this._toPurl(entry.name, entry.version)
321+
for (let child of entry.children) {
322+
if (!reachable.has(child)) { continue }
323+
let childEntry = graph.get(child)
324+
sbom.addDependency(parentPurl, this._toPurl(childEntry.name, childEntry.version))
325+
}
326+
}
327+
} else {
328+
for (let key of directDeps) {
329+
if (ignoredDeps.has(key)) { continue }
330+
let entry = graph.get(key)
331+
if (!entry) { continue }
332+
sbom.addDependency(rootPurl, this._toPurl(entry.name, entry.version))
333+
}
334+
}
372335

373336
return sbom.getAsJsonString(opts)
374337
}

0 commit comments

Comments
 (0)