Publish licensedcode data/index as seperate pypi wheels and improve arm installation support by AyanSinhaMahapatra · Pull Request #4847 · aboutcode-org/scancode-toolkit

AyanSinhaMahapatra · 2026-03-17T12:30:17Z

This is basically an extension of the work done in:

Tasks

Reviewed contribution guidelines
PR is descriptively titled 📑 and links the original issue above 🔗
Tests pass -- look for a green checkbox ✔️ a few minutes after opening your PR
Run tests locally to check for errors.
Commits are in uniquely-named feature branch and has no merge conflicts 📁
Updated documentation pages (if applicable)
Updated CHANGELOG.rst (if applicable)

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Reference: https://pypi.org/project/flot/0.7.3/ Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Fix release archive smoke tests. Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Also add a default of `persist-credentials` as False. These are fixes from running zizmorcore/zizmor Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

github-advanced-security · 2026-04-16T16:41:46Z

You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool.

What Enabling Code Scanning Means:

The 'Security' tab will display more code scanning analysis results (e.g., for the default branch).
Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results.
You will be able to see the analysis results for the pull request's branch on this overview once the scans have completed and the checks have passed.

For more information about GitHub Code Scanning, check out the documentation.

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Reference: aboutcode-org/scancode.io#1646 Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

AyanSinhaMahapatra · 2026-05-01T16:48:05Z

Release tests all green at https://github.com/AyanSinhaMahapatra/scancode-toolkit/actions/runs/25222214153 and scancode tests also passing now.

Note that the failure to load index and rebuilding of index at https://github.com/AyanSinhaMahapatra/scancode-toolkit/actions/runs/25222214153/job/73957559207#step:6:351 is caused because the latest licensedcode index wheels are not released yet so older test wheels are being installed.

JonoYang

@AyanSinhaMahapatra lgtm, thanks for handling this!

pombredanne

There are a few minor point that you should fix before merging, like double checking URLS to GH repos and missing references to AboutCode in copyrights, otherwise this is all good to go. 🙇

Reference: #4959 Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Reference: actions/runner-images#14017 Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

gotmax23

I left some comments on the new packaging from the perspective of a distribution packager of scancode. Happy to discuss further!

gotmax23 · 2026-06-04T22:48:21Z

Having multiple pyproject-*.toml files in a single directory with different suffixes is not really in the spirit of Python standards. The usual structure for monorepos is to have each package in a separate subdirectory with its own pyproject.toml file. uv has a workspaces feature that supports this usecase pretty well, for example.

Also, introducing an entirely new PEP 517/518 build backend is nonideal, since we already have so many, including hatchling and setuptools that are more configurable and pluggable than flit is and keep up-to-date with Python packaging standards. flot also does not support the PEP 639 standard for improved license metadata that upstream flit and other build backends do, for example. This will become more of an issue as Python packaging continues to evolve.

Would you consider a different approach here? At least, I'd suggest properly documenting how the project can be built from source with a standard build frontend like python -m build.

At least, I'd suggest properly documenting how the project can be built from source with a standard build frontend like python -m build.

From reading the buildapi code, I see that flot supports --config-settings=--pyproject=FILE, but that didn't seem to be documented anywhere else and also seems like a worse interface than just having one pyproject.toml per directory.

gotmax23 · 2026-06-04T22:54:32Z

+name = "licensedcode-data"
+version = "32.5.0"
+description = "A packaging of the ScanCode licensedb license and license rules database."
+long_description_content_type = "text/x-rst"


[project] does not support long_description_content_type, and this results in a warning.

The other pyproject-* files have this same issue.

gotmax23 · 2026-06-04T22:54:54Z

+description = "A packaging of the ScanCode licensedb license and license rules database."
+long_description_content_type = "text/x-rst"
+readme = "src/licensedcode/data/README.rst"
+license = { text = "Apache-2.0 AND CC-BY-4.0 AND LicenseRef-scancode-other-permissive AND LicenseRef-scancode-other-copyleft" }


This format is outdated. See my previous comment about PEP 639.

gotmax23 · 2026-06-04T22:56:16Z

+build-backend = "flot.buildapi"
+
+[project]
+name = "scancode-toolkit-mini"


Why does there need to be an entirely separate -mini package? Why not add extras/optional dependencies to the main scancode-toolkit package?

gotmax23 · 2026-06-04T22:59:54Z

Why is there a vendored wheel here?

gotmax23 · 2026-06-04T23:01:11Z

+]
+
+dependencies = [
+    "licensedcode-index",


Should this have a version constraint? Otherwise, the dependencies could get out-of-sync when users run pip install -U scancode-toolkit.

AyanSinhaMahapatra added 7 commits March 4, 2026 12:02

Add smaller wheels for license data and index

b458334

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Update to flot v0.7.3

5856d9b

Reference: https://pypi.org/project/flot/0.7.3/ Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Update Configure and release scripts

6c0e96d

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Update pyproject files for releasing multiple wheels

a500061

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Update github action release scripts for multiple wheels

9dfc115

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Grant execute permission to licensedcode wheel script

850318e

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Also build wheels for release app archives

574d4a4

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

AyanSinhaMahapatra force-pushed the smaller-wheels branch from 31e9c41 to 574d4a4 Compare March 18, 2026 13:54

Update configure scripts with licensedcode index

ead87ba

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

AyanSinhaMahapatra requested a review from pombredanne March 18, 2026 14:24

AyanSinhaMahapatra added 2 commits March 18, 2026 20:35

Create license index for release app archives

30a6184

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Include licensedcode_index wheel as thirdparty

97d0909

Fix release archive smoke tests. Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

AyanSinhaMahapatra force-pushed the smaller-wheels branch from 702e93f to 474c9b5 Compare March 20, 2026 12:33

AyanSinhaMahapatra changed the title ~~Publish licensedcode data/index as seperate pypi wheels~~ Publish licensedcode data/index as seperate pypi wheels and improve installation support Mar 24, 2026

AyanSinhaMahapatra added 2 commits April 16, 2026 21:13

Replace tagged versions with commit hashes in actions

b1afd4a

Also add a default of `persist-credentials` as False. These are fixes from running zizmorcore/zizmor Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Add zizmor as github actions run

4cf1cb3

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

AyanSinhaMahapatra force-pushed the smaller-wheels branch from 474c9b5 to 4cf1cb3 Compare April 16, 2026 16:41

github-advanced-security AI found potential problems Apr 16, 2026

View reviewed changes

Comment thread .github/workflows/licensedcode-data-index-release.yml Dismissed

Comment thread .github/workflows/scancode-release.yml Dismissed

AyanSinhaMahapatra added 5 commits April 17, 2026 15:38

Fix windows configure script

2e065ce

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Update typecode to use system-provided libmagic

85cc3f4

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Test scancode releases in arm/x86 for linux/macos

fc1e53f

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Ensure libmagic is installed by homebrew for macos

76b142d

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Use dummy wheels for apple silicon

686f43c

Reference: aboutcode-org/scancode.io#1646 Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

github-advanced-security AI found potential problems Apr 17, 2026

View reviewed changes

Comment thread .github/workflows/scancode-release.yml Fixed

AyanSinhaMahapatra force-pushed the smaller-wheels branch from 47834e6 to 182e246 Compare April 17, 2026 10:30

Only test scancode-toolkit mini wheel on macos

e174dec

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

AyanSinhaMahapatra force-pushed the smaller-wheels branch from 182e246 to e174dec Compare April 17, 2026 12:49

AyanSinhaMahapatra added 2 commits April 18, 2026 01:47

Add linux arm dummy wheels

4c99203

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Only test macos ARM support

00bb832

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

AyanSinhaMahapatra changed the title ~~Publish licensedcode data/index as seperate pypi wheels and improve installation support~~ Publish licensedcode data/index as seperate pypi wheels and improve arm installation support Apr 24, 2026

AyanSinhaMahapatra mentioned this pull request Apr 24, 2026

installation failing on arm aboutcode-org/vulnerablecode#2277

Open

AyanSinhaMahapatra added 3 commits April 29, 2026 21:56

Update dependencies for linux arm support

144cf2f

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Test release apps and wheels without dummy dists

1668805

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Fix configure script failures

2cbef80

Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

AyanSinhaMahapatra requested a review from JonoYang May 1, 2026 16:40

JonoYang approved these changes May 1, 2026

View reviewed changes

pombredanne approved these changes May 7, 2026

View reviewed changes

AyanSinhaMahapatra added 2 commits May 12, 2026 16:46

Do not include sdists in release tarballs

fa9081a

Reference: #4959 Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

Update windows runner image

99217ee

Reference: actions/runner-images#14017 Signed-off-by: Ayan Sinha Mahapatra <asmahapatra@aboutcode.org>

AyanSinhaMahapatra merged commit bd47fa8 into develop May 15, 2026
40 checks passed

gotmax23 reviewed Jun 4, 2026

View reviewed changes

gotmax23 mentioned this pull request Jun 5, 2026

Make scancode.api.get_licenses more widely useful by reducing dependencies #5115

Open

7 tasks

Uh oh!

Uh oh!

Conversation

AyanSinhaMahapatra commented Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Tasks

Uh oh!

github-advanced-security AI commented Apr 16, 2026

What Enabling Code Scanning Means:

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AyanSinhaMahapatra commented May 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JonoYang left a comment

Choose a reason for hiding this comment

Uh oh!

pombredanne left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gotmax23 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

AyanSinhaMahapatra commented Mar 17, 2026 •

edited

Loading

AyanSinhaMahapatra commented May 1, 2026 •

edited

Loading