Improve PURL creation logic for SBoM by spoorcc · Pull Request #782 · dfetch-org/dfetch

spoorcc · 2025-09-13T22:05:27Z

Move out the purl logic and add tests for it. Also add support for Bitbucket PURLs

To be tested on dependency-track

Description by Korbit AI

What change is being made?

Improve PURL creation logic for SBoM by centralizing remote URL to PURL conversion, enriching SBOM components with accurate VCS references, and expanding support for various VCS styles (GitHub, Bitbucket, SVN, SSH paths, and generic VCS URLs); add tests and minor workflow/readme updates to reflect the changes.

Why are these changes being made?

Provide a robust, consistent mapping from remote URLs to PURLs, ensuring accurate type, namespace, name, and external VCS references across common VCS providers and edge cases; align SBOM generation with the new PURL logic and expand test coverage.

Is this description stale? Ask me to generate a new description by commenting /korbit-generate-pr-description

korbit-ai

Review by Korbit AI

Korbit automatically attempts to detect when you fix issues in new commits.

Category	Issue	Status
	Hardcoded domain list without explanation ▹ view
	Hardcoded fallback string 'unknown' ▹ view
	Invalid empty VCS reference creation ▹ view
	Credential leakage via VCS URL in SBOM ▹ view
	Lack of error handling around PURL creation ▹ view
	Rigid handler registry (Open-Closed violation) ▹ view
	Internal helper docstrings lack return value details ▹ view

Files scanned

File Path	Reviewed
dfetch/reporting/sbom_reporter.py	✅
dfetch/util/purl.py	✅

Explore our documentation to understand the languages and file types we support and the files we ignore.

Check out our docs on how you can make Korbit work best for you and your team.

Loving Korbit!? Share us on LinkedIn Reddit and X

spoorcc · 2025-09-15T19:46:47Z

/korbit-review

korbit-ai

Review by Korbit AI

Korbit automatically attempts to detect when you fix issues in new commits.

Category	Issue	Status
	Undocumented Complex Regex Pattern ▹ view	✅ Fix detected

Files scanned

File Path	Reviewed
dfetch/util/purl.py	✅
dfetch/reporting/sbom_reporter.py	✅

Explore our documentation to understand the languages and file types we support and the files we ignore.

Check out our docs on how you can make Korbit work best for you and your team.

Loving Korbit!? Share us on LinkedIn Reddit and X

ben-edna · 2025-09-16T07:09:59Z

Using https://github.com/CycloneDX/sbom-utility with

$ sbom-utility validate --input-file my-sbom.json

shows that PURL seems ok, but externalReferences url still uses ssh

 {
        "type": "format",
        "field": "components.0.externalReferences.0.url",
        "context": "(root).components.0.externalReferences.0.url",
        "description": "Does not match format 'iri-reference'",
        "value": "git@git.mycompany.com:mycompany/some-repo"
  }

spoorcc · 2025-09-19T20:58:24Z

Verified SBoM with:

 wget https://github.com/CycloneDX/sbom-utility/releases/download/v0.18.1/sbom-utility-v0.18.1-linux-amd64.tar.gz
tar -xvzf sbom-utility-v0.18.1-linux-amd64.tar.gz 
./sbom-utility validate --input-file report.json

spoorcc · 2025-09-19T21:02:36Z

/korbit-review

korbit-ai

Review by Korbit AI

Korbit automatically attempts to detect when you fix issues in new commits.

Category	Issue	Status
	Inefficient TLD extraction without caching ▹ view
	Unexplained SSH URL exclusion ▹ view
	Python version compatibility issue with removeprefix ▹ view
	Incorrect unpacking of actual_content in assertion message ▹ view

Files scanned

File Path	Reviewed
dfetch/reporting/sbom_reporter.py	✅
dfetch/util/purl.py	✅
features/steps/generic_steps.py	✅

Explore our documentation to understand the languages and file types we support and the files we ignore.

Check out our docs on how you can make Korbit work best for you and your team.

Loving Korbit!? Share us on LinkedIn Reddit and X

korbit-ai · 2025-09-19T21:04:43Z

+from packageurl import PackageURL
+from tldextract import TLDExtract
+
+NO_FETCH_EXTRACT = TLDExtract(suffix_list_urls=(), extra_suffixes=("local",))


Inefficient TLD extraction without caching

Tell me more

What is the issue?

TLDExtract is instantiated at module level with network-disabled configuration, creating a global object that performs domain parsing on every call.

Why this matters

This creates unnecessary overhead when the same domains are parsed repeatedly, and the object could be reused more efficiently or cached results could be stored.

Suggested change ∙ Feature Preview

Consider caching the results of domain extraction or using a lazy initialization pattern if the TLDExtract object is expensive to create. Alternatively, cache the results of NO_FETCH_EXTRACT(domain).domain calls within the _namespace_and_name_from_domain_and_path function using functools.lru_cache.

Provide feedback to improve future suggestions

_{💬 Looking for more details? Reply to this comment to chat with Korbit.}

korbit-ai · 2025-09-19T21:04:43Z

+            component.group = purl.namespace
+
+            vcs_url = purl.qualifiers.get("vcs_url", "")
+            if vcs_url and "ssh://" not in vcs_url:


Unexplained SSH URL exclusion

Tell me more

What is the issue?

The string literal 'ssh://' is hardcoded in the condition without explanation of why SSH URLs are excluded.

Why this matters

Without context, developers need to guess why SSH URLs are being filtered out, making the code harder to maintain.

Suggested change ∙ Feature Preview

# Define a constant at module level SSH_URL_PREFIX = 'ssh://' # Add a comment explaining the exclusion # Skip SSH URLs as they are not publicly accessible if vcs_url and SSH_URL_PREFIX not in vcs_url:

Provide feedback to improve future suggestions

_{💬 Looking for more details? Reply to this comment to chat with Korbit.}

korbit-ai · 2025-09-19T21:04:43Z

+
+    if "svn" in parsed.scheme or "svn." in parsed.netloc:
+        namespace, name = _namespace_and_name_from_domain_and_path(parsed.netloc, path)
+        namespace = namespace.replace("/svn/", "/").removeprefix("p/")


Python version compatibility issue with removeprefix

Tell me more

What is the issue?

The removeprefix method is only available in Python 3.9+, but the code doesn't specify this requirement.

Why this matters

This will cause AttributeError on Python versions below 3.9, making the code incompatible with older Python versions.

Suggested change ∙ Feature Preview

Use compatible string slicing or add version requirement:

# Option 1: Compatible approach namespace = namespace.replace("/svn/", "/") if namespace.startswith("p/"): namespace = namespace[2:] # Option 2: Or ensure Python 3.9+ requirement is documented

Provide feedback to improve future suggestions

_{💬 Looking for more details? Reply to this comment to chat with Korbit.}

ben-edna · 2025-09-23T11:14:48Z

SBoM is now accepted by Dependency track. A thing to document or be aware of is that the name used is the project name in the manifest and not necessarily the authoritative name

Move out the purl logic and add tests for it. Also add support for Bitbucket PURLs

korbit-ai Bot reviewed Sep 13, 2025

View reviewed changes

spoorcc force-pushed the spoorcc/issue780 branch from 6d1873a to 088fbac Compare September 13, 2025 22:12

korbit-ai Bot reviewed Sep 15, 2025

View reviewed changes

Comment thread dfetch/util/purl.py

spoorcc linked an issue Sep 15, 2025 that may be closed by this pull request

Invalid SBoM created when projects uses ssh #780

Closed

korbit-ai Bot reviewed Sep 19, 2025

View reviewed changes

spoorcc added 12 commits September 29, 2025 20:34

Fixes Invalid SBoM created when projects uses ssh

7e90e80

Move out the purl logic and add tests for it. Also add support for Bitbucket PURLs

Use separate list of exlcuded domains

50483f2

Add separate default name

8a1f213

Fix buildin python <=3.8

6ced392

Don't add vcs url if there is no url

e1aaf7b

Simplify code

9be6b01

Use proper path in example

f07dafa

Cleanup PURLs a little more

0e35af4

Reduce complexity

8f87a63

Make sure ssh vcs_url has a scheme

05e8463

Show more output when comparing files fails

6086bf9

Add External VCS reference to SBoM if possible

0e39681

spoorcc force-pushed the spoorcc/issue780 branch from 174d9fc to 0e39681 Compare September 29, 2025 20:34

Fix review comments

9b03794

spoorcc force-pushed the spoorcc/issue780 branch 4 times, most recently from 629c622 to 6bf70cd Compare September 29, 2025 21:07

Use svn instead of subversion

3224239

spoorcc force-pushed the spoorcc/issue780 branch from 6bf70cd to 3224239 Compare September 29, 2025 21:10

Add OpenSSF best practices

815cb43

spoorcc merged commit 33638c1 into main Sep 29, 2025
25 checks passed

spoorcc deleted the spoorcc/issue780 branch September 29, 2025 21:26

Conversation

spoorcc commented Sep 13, 2025 • edited by korbit-ai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description by Korbit AI

What change is being made?

Why are these changes being made?

Uh oh!

korbit-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Review by Korbit AI

Korbit automatically attempts to detect when you fix issues in new commits.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

spoorcc commented Sep 15, 2025

Uh oh!

korbit-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Review by Korbit AI

Korbit automatically attempts to detect when you fix issues in new commits.

Uh oh!

Uh oh!

ben-edna commented Sep 16, 2025

Uh oh!

spoorcc commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

spoorcc commented Sep 19, 2025

Uh oh!

korbit-ai Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Review by Korbit AI

Korbit automatically attempts to detect when you fix issues in new commits.

Uh oh!

korbit-ai Bot Sep 19, 2025

Choose a reason for hiding this comment

Inefficient TLD extraction without caching

What is the issue?

Why this matters

Suggested change ∙ Feature Preview

Provide feedback to improve future suggestions

Uh oh!

korbit-ai Bot Sep 19, 2025

Choose a reason for hiding this comment

Unexplained SSH URL exclusion

What is the issue?

Why this matters

Suggested change ∙ Feature Preview

Provide feedback to improve future suggestions

Uh oh!

korbit-ai Bot Sep 19, 2025

Choose a reason for hiding this comment

Python version compatibility issue with removeprefix

What is the issue?

Why this matters

Suggested change ∙ Feature Preview

Provide feedback to improve future suggestions

Uh oh!

Uh oh!

ben-edna commented Sep 23, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

spoorcc commented Sep 13, 2025 •

edited by korbit-ai Bot

Loading

korbit-ai Bot left a comment •

edited

Loading

korbit-ai Bot left a comment •

edited

Loading

spoorcc commented Sep 19, 2025 •

edited

Loading

korbit-ai Bot left a comment •

edited

Loading