Merge pull request #3104 from nexB/prep-release-31-2

Prepare release 31.2

Author: pombredanne

.github/workflows/about-files-ci.yml

@@ -2,6 +2,9 @@ name: CI About Files
 
 on: [push, pull_request]
 
+permissions:
+  contents: read # to fetch code (actions/checkout)
+
 jobs:
   build:
     runs-on: ubuntu-20.04

.github/workflows/docs-ci.yml

@@ -2,6 +2,9 @@ name: CI Documentation
 
 on: [push, pull_request]
 
+permissions:
+  contents: read # to fetch code (actions/checkout)
+
 jobs:
   build:
     runs-on: ubuntu-20.04

.github/workflows/scancode-release.yml

@@ -17,9 +17,13 @@ on:
     tags:
       - "v*.*.*"
 
+permissions: {}
 jobs:
 
   build_scancode_for_pypi:
+    permissions:
+      contents: read # to fetch code (actions/checkout)
+
     name: Build PyPI archives
     runs-on: ubuntu-20.04
 
@@ -67,6 +71,9 @@ jobs:
 
 
   build_scancode_for_release_linux:
+    permissions:
+      contents: read # to fetch code (actions/checkout)
+
     name: Build Release for linux
     runs-on: ubuntu-20.04
     needs:
@@ -106,6 +113,9 @@ jobs:
 
 
   build_scancode_for_release_macos:
+    permissions:
+      contents: read # to fetch code (actions/checkout)
+
     name: Build Release for mac
     runs-on: ubuntu-20.04
     needs:
@@ -145,6 +155,9 @@ jobs:
 
 
   build_scancode_for_release_windows:
+    permissions:
+      contents: read # to fetch code (actions/checkout)
+
     name: Build Release for windows
     runs-on: ubuntu-20.04
     needs:
@@ -183,6 +196,9 @@ jobs:
 
 
   build_scancode_for_release_source:
+    permissions:
+      contents: read # to fetch code (actions/checkout)
+
     name: Build source
     runs-on: ubuntu-20.04
     needs:
@@ -221,6 +237,9 @@ jobs:
 
 
   smoke_test_install_and_run_pypi_dists_posix:
+    permissions:
+      contents: read # to fetch code (actions/checkout)
+
     name: Test POSIX PyPI wheels
     needs:
       - build_scancode_for_pypi
@@ -267,6 +286,9 @@ jobs:
 
 
   smoke_test_install_and_run_pypi_dists_windows:
+    permissions:
+      contents: read # to fetch code (actions/checkout)
+
     name: Test Windows PyPI wheels
     needs:
       - build_scancode_for_pypi
@@ -312,6 +334,9 @@ jobs:
 
 
   smoke_test_install_and_run_app_archives_on_linux:
+    permissions:
+      contents: read # to fetch code (actions/checkout)
+
     name: Test app on ${{ matrix.os }}
     needs:
       - build_scancode_for_release_linux
@@ -350,6 +375,9 @@ jobs:
 
 
   smoke_test_install_and_run_app_archives_on_macos:
+    permissions:
+      contents: read # to fetch code (actions/checkout)
+
     name: Test app on ${{ matrix.os }}
     needs:
       - build_scancode_for_release_macos
@@ -388,6 +416,9 @@ jobs:
 
 
   smoke_test_install_and_run_app_archives_on_windows:
+    permissions:
+      contents: read # to fetch code (actions/checkout)
+
     name: Test app on ${{ matrix.os }}
     needs:
       - build_scancode_for_release_windows
@@ -422,6 +453,9 @@ jobs:
           for %%F in (dist/*.zip) do python etc/release/scancode_release_tests.py dist/%%F
 
   publish_to_gh_release:
+    permissions:
+      contents: write # to create GitHub release (softprops/action-gh-release)
+
     name: Publish to GH Release
     needs:
       - smoke_test_install_and_run_app_archives_on_linux

CHANGELOG.rst

@@ -55,6 +55,14 @@ License detection:
   as an option.
 
 
+v31.2.0 - 2022-09-29
+----------------------------------
+
+This is a minor release with small bug fixes and minor feature updates.
+
+- Update SPDX license list to 3.18
+- Improve how we discard license matches that are "gibberish"
+- And new and improve existing license and license detection rules
 
 
 v31.1.1 - 2022-09-02

ROADMAP.rst

@@ -18,19 +18,27 @@ even if it is correct and it is technically correct.
 The goal of this improvement is to:
 
 - combine multiple related license matches in a single license detection
+
 - in a license detection, expose a primary license expression in addition
   to the complete, full license expression.
+
 - make the logic of selection of the primary license visible, at the minimum
   with a log of combination and primary license selection operations
 
 This is for SCTK first.
 
+Status: This has been completed in SCTK and also included in SCIO. We use
+an updated --summary option and a new license clarity score for this.
+Some work is still in progress as part of 3.) "detections"
+
 
 2. Package files.
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 Reporting the set of package files for each package instance is important because
-it allows for natural grouping of these in one unit. 
+it allows for natural grouping of these in one unit.
+
+This has been completed in SCTK and also included in SCIO.
 
 
 3. Go to two-level reporting of detections to provide more effective detections

docs/scripts/sphinx_build_link_check.sh

@@ -68,7 +68,7 @@ Can I install ScanCode in a Unicode path?
 -----------------------------------------
 
 Yes but but it is best to void this for now. See
-https://github.com/nexB/scancode- toolkit/issues/867
+https://github.com/nexB/scancode-toolkit/issues/867
 
 There is a bug in virtualenv https://github.com/pypa/virtualenv/issues/457 that
 is now fixed but has not been extensively tested for ScanCode.

docs/source/misc/faq.rst

@@ -3,14 +3,71 @@
 Add A Post-Scan Plugin
 ======================
 
+Scan plugins in ``scancode-toolkit``
+------------------------------------
+
+A lot of scancode features are built-in plugins which are present with scancode-toolkit source code
+and are usually enabled via the different scancode-toolkit CLI options and are grouped by the types
+of plugins.
+
+Here are the major types of plugins:
+
+1. Pre-scan plugins (`scancode_pre_scan` in entry points)
+
+   These plugins are run before the main scanning steps and are usually
+   filtering of input files, or file classification steps, on whose results
+   the main scan plugins depend on. The base plugin class to be extended is ``PreScanPlugin`` at
+   `/src/plugincode/pre_scan.py <https://github.com/nexB/plugincode/blob/main/src/plugincode/pre_scan.py>`_.
+
+2. Scan plugins (`scancode_scan` in entry points)
+
+   The are the scancode plugins which does the file scanning for useful
+   information like license, copyrights, packages and others. These are
+   run on multiprocessing for speed as they are done on a per-file basis,
+   but there can also be post-processing steps on these which are run afterwards
+   and have access to all the per-file scan results. The base plugin class to be extended is
+   ``ScanPlugin`` at `/src/plugincode/scan.py <https://github.com/nexB/plugincode/blob/main/src/plugincode/scan.py>`_.
+
+3. Post-scan plugins (`scancode_post_scan` in entry points)
+
+   These are mainly data processing, summerizing and reporting plugins which
+   depend on all the results for the scan plugins. These add new codebase level
+   or file-level attributes, and even removes/modifies data as required
+   for consolidation or summarization. The base plugin class to be extended is ``PostScanPlugin``
+   at `/src/plugincode/post_scan.py <https://github.com/nexB/plugincode/blob/main/src/plugincode/post_scan.py>`_.
+
+4. Output plugins (`scancode_output` in entry points)
+
+   Supported output options in scancode-toolkit are all plugins and
+   these can also be multiple output options selected. These convert, process
+   and writes the data in the specific file format as the output of the scanning
+   procedures. The base plugin class to be extended is ``OutputPlugin`` at
+   `/src/plugincode/output.py <https://github.com/nexB/plugincode/blob/main/src/plugincode/output.py>`_.
+
+5. Output Filter Plugins (`scancode_output_filter` in entry points)
+
+   There are also output filter plugins which apply filters to the outputs
+   and is modified. These filters can be based on whether resources had any
+   detections, ignorables present in licenses and others.
+   The base plugin class to be extended is ``OutputFilterPlugin`` at
+   `/src/plugincode/output_filter.py <https://github.com/nexB/plugincode/blob/main/src/plugincode/output_filter.py>`_.
+
+6. Location Provider Plugins
+
+   These plugins provide pre-built binary libraries and utilities and their locations which
+   are packaged to be used in scancode-toolkit. The base plugin class to be extended is
+   ``LocationProviderPlugin`` at `/src/plugincode/location_provider.py <https://github.com/nexB/plugincode/blob/main/src/plugincode/location_provider.py>`_.
+
+
 Built-In vs. Optional Installation
 ----------------------------------
 
 Built-In
 ^^^^^^^^
 
-Some post-scan plugins are installed when ScanCode itself is installed, e.g., the
-:ref:`license_policy_plugin`, whose code is located here::
+Some post-scan plugins are installed when ScanCode itself is installed, and they are specified at
+``[options.entry_points]`` in the `setup.cfg <https://github.com/nexB/scancode-toolkit/blob/develop/setup.cfg>`_ file.
+For example, the :ref:`license_policy_plugin` is a built-in plugin, whose code is located here::
 
     https://github.com/nexB/scancode-toolkit/blob/develop/src/licensedcode/plugin_license_policy.py
 
@@ -23,7 +80,7 @@ Optional
 ScanCode is also designed to use post-scan plugins that must be installed separately from the
 installation of ScanCode. The code for this sort of plugin is located here::
 
-    https://github.com/nexB/scancode-toolkit/tree/develop/plugins/
+    https://github.com/nexB/scancode-plugins
 
 This wiki page will focus on optional post-scan plugins.
 
@@ -45,26 +102,25 @@ We'll start by creating three folders:
 1. Top-level folder -- ``/scancode-hello/``
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-- In the ``/scancode-toolkit/plugins/`` directory, add a folder with a relevant name, e.g.,
-  ``scancode-hello``. This folder will hold all of your plugin code.
+- In the ``scancode-plugins`` repository, in the ``misc`` directory, add a folder with
+  a relevant name, e.g., ``scancode-hello``. This folder will hold all of your plugin code.
 
 - Inside the ``/scancode-hello/`` folder you'll need to add a folder named ``src`` and 7 files.
-
-1. ``/src/`` -- This folder will contain your primary Python code and is discussed in more detail
-   in the following section.
+  ``/src/`` -- This folder will contain your primary Python code and is discussed in more detail
+  in the following section.
 
 The 7 Files are:
 
 1. ``.gitignore`` -- See, e.g.,
-   `/plugins/scancode-ignore-binaries/.gitignore <https://github.com/nexB/scancode-toolkit/blob/develop/plugins/scancode-ignore-binaries/.gitignore>`_
+   `/scancode-ignore-binaries/.gitignore <https://github.com/nexB/scancode-plugins/blob/main/misc/scancode-ignore-binaries/.gitignore>`_
 
 ::
 
     /build/
     /dist/
 
 2. ``apache-2.0.LICENSE`` -- See, e.g.,
-   `/plugins/scancode-ignore-binaries/apache-2.0.LICENSE <https://github.com/nexB/scancode-toolkit/blob/develop/plugins/scancode-ignore-binaries/apache-2.0.LICENSE>`_
+   `/scancode-ignore-binaries/apache-2.0.LICENSE <https://github.com/nexB/scancode-plugins/blob/main/misc/scancode-ignore-binaries/apache-2.0.LICENSE>`_
 
 3. ``MANIFEST.in``
 
@@ -83,7 +139,7 @@ The 7 Files are:
     global-exclude *.py[co] __pycache__ *.*~
 
 4. ``NOTICE`` -- See, e.g.,
-   `/plugins/scancode-ignore-binaries/NOTICE <https://github.com/nexB/scancode-toolkit/blob/develop/plugins/scancode-ignore-binaries/NOTICE>`__
+   `/scancode-ignore-binaries/NOTICE <https://github.com/nexB/scancode-plugins/blob/main/misc/scancode-ignore-binaries/NOTICE>`__
 
 5. ``README.md``
 
@@ -129,7 +185,7 @@ The 7 Files are:
         long_description=desc,
         author='nexB',
         author_email='info@aboutcode.org',
-        url='https://github.com/nexB/scancode-toolkit/plugins/scancode-categories',
+        url='https://github.com/nexB/scancode-plugins/blob/main/misc/scancode-hello/',
         packages=find_packages('src'),
         package_dir={'': 'src'},
         py_modules=[splitext(basename(path))[0] for path in glob('src/*.py')],
@@ -201,11 +257,11 @@ Create a ``PostScanPlugin`` class
 """""""""""""""""""""""""""""""""
 
 The ``PostScanPlugin`` class
-`PostScanPlugin code <https://github.com/nexB/scancode-toolkit/blob/develop/src/plugincode/post_scan.py>`_)
+`PostScanPlugin code <https://github.com/nexB/plugincode/blob/main/src/plugincode/post_scan.py>`_)
 inherits from the ``CodebasePlugin`` class (see
-`CodebasePlugin code <https://github.com/nexB/scancode-toolkit/blob/794d7acf78480823084def703b5d61ade12efdf2/src/plugincode/__init__.py#L139-L150>`_),
+`CodebasePlugin code <https://github.com/nexB/plugincode/blob/main/src/plugincode/__init__.py>`_),
 which inherits from the ``BasePlugin`` class (see
-`BasePlugin code <https://github.com/nexB/scancode-toolkit/blob/794d7acf78480823084def703b5d61ade12efdf2/src/plugincode/__init__.py#L38-L136>`_).
+`BasePlugin code <https://github.com/nexB/plugincode/blob/main/src/plugincode/__init__.py>`_).
 
 ::