Merge pull request #1028 from KnowledgeCaptureAndDiscovery/dev

Bringing issues for the next release

Author: dgarijo

.github/workflows/docker-publish.yml

@@ -0,0 +1,33 @@
+name: Build and Publish Docker Image after Release (view instruccions for Docker in readme.md file)
+
+on:
+  release:
+    types: [published]
+  # workflow_dispatch:
+
+jobs:
+  build-and-push:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v3
+
+      - name: Log in to Docker Hub
+        uses: docker/login-action@v3
+        with:
+          username: ${{ secrets.DOCKERHUB_USERNAME }}
+          password: ${{ secrets.DOCKERHUB_TOKEN }}
+
+      - name: Build and push Docker image
+        uses: docker/build-push-action@v5
+        with:
+          context: .
+          file: Dockerfile
+          push: true
+          tags: |
+            kcapd/somef:latest
+            kcapd/somef:${{ github.event.release.tag_name }}

.gitignore

@@ -24,3 +24,5 @@ repos.txt
 !package_neors.json
 !package_npm.json
 !test_data/api_responses/*.json
+uv.lock
+.python-version

README.md

@@ -95,6 +95,19 @@ We recognize the following properties:
 
 We use different supervised classifiers, header analysis, regular expressions, the GitHub/Gitlab API to retrieve all these fields (more than one technique may be used for each field) and language specific metadata parsers (e.g., for package files). Each extraction records its provenance, with the confidence and technique used on each step. For more information check the [output format description](https://somef.readthedocs.io/en/latest/output/)
 
+### Confidence values in header analysis
+
+When extracting metadata using header analysis, SOMEF assigns a confidence value based on the length 
+of the header. Shorter headers are more likely to be a good fit for a category, while longer headers 
+may contain additional context that makes the classification less reliable:
+
+| Header length | Confidence |
+|---------------|------------|
+| 1–3 words     | 1.0        |
+| 4–6 words     | 0.8        |
+| 7–10 words    | 0.5        |
+| 11+ words     | 0.1        |
+
 ## Documentation
 
 See full documentation at [https://somef.readthedocs.io/en/latest/](https://somef.readthedocs.io/en/latest/)
@@ -362,6 +375,10 @@ The following command extracts all metadata available from [https://github.com/d
 somef describe -r https://github.com/dgarijo/Widoco/ -o test.json -t 0.8
 ```
 
+We recommend having a high value for the `threshold` parameter, 0.8 (default) or above.
+Additional configuration parameters (such as the `similarity_threshold` for header analysis) 
+can be set in `~/.somef/config.json`. See the [usage documentation](https://somef.readthedocs.io/en/latest/usage/) for details.
+
 Try SOMEF in Binder with our sample notebook: [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/KnowledgeCaptureAndDiscovery/somef/HEAD?filepath=notebook%2FSOMEF%20Usage%20Example.ipynb)
 
 ## Contribute:

config.json

@@ -2,5 +2,6 @@
 	"description" : "./models/description.p",
 	"citation" : "./models/citation.p",
 	"installation" : "./models/installation.p",
-	"invocation" : "./models/invocation.p"
+	"invocation" : "./models/invocation.p",
+	"similarity_threshold": 0.8
 }

docs/output.md

@@ -133,6 +133,19 @@ The following table summarized the properties used to describe a `category`:
 | **source** | No | Url | URL of the source file used for the extraction. |
 | **technique** | Yes | String | Technique used for the extraction. One of the following list: Supervised classification, header analysis, regular expression, GitHub API, File exploration, Code parsing |
 
+### Confidence values in header analysis
+
+When extracting metadata using header analysis, SOMEF assigns a confidence value based on the length 
+of the header. Shorter headers are more likely to be a good fit for a category, while longer headers 
+may contain additional context that makes the classification less reliable:
+
+| Header length | Confidence |
+|---------------|------------|
+| 1–3 words     | 1.0        |
+| 4–6 words     | 0.8        |
+| 7–10 words    | 0.5        |
+| 11+ words     | 0.1        |
+
 ### Result
 Field returning the extracted output from the code repository. An example can be seen below for a citation found in BibteX format in a README file of a code repository:
 
@@ -423,6 +436,7 @@ The table below summarizes the mapping between the SOMEF internal JSON structure
 
 | Codemeta / Schema.org Field  | SOMEF Category | Description |
 | :---                  | :---                  | :--- |
+| `applicationCategory` | `application_domain`  | Categories |
 | `author`              | `author`              | Principal authors |
 | `buildInstructions`   | `installation` / `documentation` | Installation or build instructions |
 | `creditText`          | `citation` (Software)   | Human-readable citation for the software  *1*|
@@ -445,6 +459,7 @@ The table below summarizes the mapping between the SOMEF internal JSON structure
 | `logo`                | `logo`                  | Project logo URL |
 | `maintainer`          | `maintainer`            | Project maintainers |
 | `name`                | `name`                  | Software name |
+| `schema:owner`        | `owner`                  | Software owner |
 | `programmingLanguage` | `programming_languages` | Languages used |
 | `readme`              | `readme_url`            | README file URL |
 | `referencePublication`| `citation` (Papers)     || References to the main publication associated with this software component (as per author preference) *1*|

docs/setupcfg.md

@@ -0,0 +1,52 @@
+The following metadata fields can be extracted from a setup.cfg file.   
+These fields are defined in the [setuptools declarative configuration specification](https://setuptools.pypa.io/en/latest/userguide/declarative_config.html), and are mapped according to the [CodeMeta crosswalk for Python Distutils](https://github.com/codemeta/codemeta/blob/master/crosswalks/Python%20Distutils%20(PyPI).csv).
+
+| Software metadata category  |    SOMEF metadata JSON path    | SETUP.CFG metadata file field     |
+|--------------------------------|-----------------------------|----------------------------------------|
+| author - value                   |  author[i].result.value       |     metadata.author                  |
+| author - email                   |  author[i].result.email       |     metadata.author_email            |
+| author - name                   |  author[i].result.name        |     metadata.author                |
+| code_repository           |  code_repository[i].result.value  |  project_urls (source, repository, code) |
+| description               |  description[i].result.value |  metadata.description                |
+| documentation             |  documentation[i].result.value  |  project_urls (Documentation, docs) |       
+| license - value                   |  license[i].result.value  |   metadata.license or metadata.license_files                   |
+| license - name                   |  license[i].result.name    |  metadata.license           *(1)*   |
+| license - spdx id                   |  license[i].result.spdx_id |    metadata.license if "spdx.org/licenses/"   *(1)* |  
+| has_package_file          |  has_package_file[i].result.value    |   URL of the setup.cfg file      |
+| homepage                  |  homepage[i].result.value          |   metadata.url or project_urls (Homepage)     |
+| keywords                  |  keywords[i].result.value          |     metadata.keywords                 |
+| package_id                |  package_id[i].result.value |   metadata.name  |
+| requirements - value              | requirements[i].result.value  |   options.install_requires or options.setup_requires *(2)*   |
+| requirements - name              |  requirements[i].result.name   |   options.install_requires or options.setup_requires -> name  *(2)*     |
+| requirements - version              |  requirements[i].result.version   | options.install_requires or options.setup_requires -> version *(2)*   |
+| runtime_platform - value         |  runtime_platform[i].result.value |  options.python_requires -> "Python" + version  *(3)*  |
+| runtime_platform - name          |  runtime_platform[i].result.name |   options.python_requires -> "Python"  *(3)* |
+| runtime_platform - version          |  runtime_platform[i].result.version |   options.python_requires  *(3)* |
+| version - value                   |  version[i].result.value |  metadata.version |
+| version - tag                  |  version[i].result.tag                            |     metadata.version                            |
+
+---
+
+*(1)*  
+- Look for the name and spdx_id in a local dictionary with all licenses.
+
+*(2)*  
+- Examples of requirements
+```
+[options]
+install_requires =
+    astropy
+    ctapipe >= 0.12
+    h5py ~= 3.1.0
+    
+setup_requires =
+    setuptools >= 40.6.0
+    wheel
+
+```
+
+*(3)*  
+- Example:
+```
+python_requires = >= 3.10.0
+```

docs/supported_languages.md

@@ -12,7 +12,7 @@ To know more about the extraction details for each type of file, click on it.
 | JavaScript | [`package.json`](./packagejson.md), [`bower.json`](./bower.md) | 
 | Julia | [`Project.toml`](./julia.md) | 
 | PHP | [`composer.json`](./composer.md) |  
-| Python | [`setup.py`](./setuppy.md), [`pyproject.toml`](./pyprojecttoml.md), [`requirements.txt`](./requirementstxt.md) | 
+| Python | [`setup.py`](./setuppy.md), [`setup.cfg`](./setupcfg.md), [`pyproject.toml`](./pyprojecttoml.md), [`requirements.txt`](./requirementstxt.md) | 
 | R | [`DESCRIPTION`](./description.md) | 
 | Ruby | [`*.gemspec`](./gemspec.md) |  
 | Rust | [`Cargo.toml`](./cargo.md) |  

docs/supported_metadata_files.md

@@ -21,6 +21,7 @@ SOMEF can extract metadata from a wide range of files commonly found in software
 | `pyproject.toml`   | Python                     | Modern Python project configuration file used by tools like Poetry and Flit |   [🔍](./pyprojecttoml.md)| [📄](https://packaging.python.org/en/latest/guides/writing-pyproject-toml/)| [PEP 621](https://peps.python.org/pep-0621/)| [Example](https://github.com/KnowledgeCaptureAndDiscovery/somef/blob/master/pyproject.toml) | 
 | `requirements.txt` | Python                     | Lists Python package dependencies | [🔍](./requirementstxt.md)| [📄](https://pip.pypa.io/en/stable/reference/requirements-file-format/)| [Latest](https://pip.pypa.io/en/stable/reference/requirements-file-format/)| [Example](https://github.com/oeg-upm/FAIR-Research-Object/blob/main/requirements.txt) |
 | `setup.py`         | Python                     | Package file format used in python projects | [🔍](./setuppy.md)| [📄](https://setuptools.pypa.io/en/latest/references/keywords.html)| [v75.0.0](https://github.com/pypa/setuptools)| [Example](https://github.com/oeg-upm/soca/blob/main/setup.py) | 
+| `setup.cfg` | Python | Configuration file for setuptools used to define package metadata and options in a declarative way |  [🔍](./setupcfg.md)| [📄](https://setuptools.pypa.io/en/latest/userguide/declarative_config.html) | [v75.0.0](https://github.com/pypa/setuptools)|[Example](https://github.com/oeg-upm/soca/blob/main/setup.cfg)|
 | `DESCRIPTION`      | R                          | Metadata file for R packages including title, author, and version |   [🔍](./description.md) | [📄](https://cran.r-project.org/doc/manuals/R-exts.html#The-DESCRIPTION-file)| [v4.4.1](https://cran.r-project.org/doc/manuals/r-release/R-exts.html) |  [Example](https://github.com/cran/ggplot2/blob/master/DESCRIPTION) |
 | `*.gemspec`        | Ruby                       | Manifest file serves as the package descriptor used in Ruby gem projects. |  [🔍](./gemspec.md)| [📄](https://guides.rubygems.org/specification-reference/)| [v3.5.22](https://github.com/rubygems/rubygems)|[Example](https://github.com/rubygems/rubygems/blob/master/bundler/bundler.gemspec) |
 | `cargo.toml`       | Rust                       | Manifest file serves as the package descriptor used in Rust projects |  [🔍](./cargo.md) | [📄](https://doc.rust-lang.org/cargo/reference/manifest.html)| [v0.85.0](https://github.com/rust-lang/cargo) | [Example](https://github.com/rust-lang/cargo/blob/master/Cargo.toml) |

docs/usage.md

@@ -82,8 +82,33 @@ If you prefer to export as a [Codemeta](https://codemeta.github.io/) JSON-LD, ju
 somef describe -r https://github.com/dgarijo/Widoco/ -c test.json
 ```
 
-For more information about the output types supported by SOMEF, please see [the output format help page](https://somef.readthedocs.io/en/latest/output/).
+For more information about the output types supported by SOMEF, please see [the output format help page](https://somef.readthedocs.io/en/latest/output/). 
 
 We recommend having a high value for the `threshold` parameter, 0.8 (default) or above.
 
+## Configuration parameters
+
+SOMEF uses a configuration file located at `~/.somef/config.json` that can be edited to customize its behavior. 
+To generate it, run `somef configure`. The following parameters are available:
+
+### Similarity threshold
+
+Controls the minimum similarity score required for a README header to be matched to a 
+category (e.g., installation, usage, license). SOMEF uses WordNet path similarity to 
+compare header words against known category terms.
+
+- **Default value**: `0.8`
+- **Range**: `0.0` to `1.0` (higher values = stricter matching, lower values = more permissive)
+
+To change it, edit your `~/.somef/config.json`:
+
+```json
+{
+    "similarity_threshold": 0.75
+}
+```
+
+Note: This parameter is different from the `-t` threshold used in `somef describe`, 
+which controls the confidence of the supervised classifiers.
+
 To see a live usage example, try our Binder Notebook: [![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/KnowledgeCaptureAndDiscovery/somef/HEAD?filepath=notebook%2FSOMEF%20Usage%20Example.ipynb)