False Positive Detection of `LGPL-2.0-plus` and Other Licenses

[alok1304]

When I scan this text GNU Library General Public Licence we got "detected_license_expression": null but here we can clearly see this is a LGPL license we got only license_clues for this text.
Almost 2.8k files have this license sentence see: https://github.com/search?q=%22GNU+Library+General+Public+Licence%22&type=code

Here we can see the result that is getting from scancode-toolkit.

{
      "path": "check.txt",
      "type": "file",
      "detected_license_expression": null,
      "detected_license_expression_spdx": null,
      "license_detections": [],
      "license_clues": [
        {
          "license_expression": "lgpl-2.0 OR mulle-kybernetik",
          "license_expression_spdx": "LGPL-2.0-only OR LicenseRef-scancode-mulle-kybernetik",
          "from_file": "check.txt",
          "start_line": 1,
          "end_line": 1,
          "matcher": "3-seq",
          "score": 10.64,
          "matched_length": 5,
          "match_coverage": 10.64,
          "rule_relevance": 100,
          "rule_identifier": "lgpl-2.0_or_mulle-kybernetik4.RULE",
          "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/lgpl-2.0_or_mulle-kybernetik4.RULE",
          "matched_text": "GNU Library General Public Licence",
          "matched_text_diagnostics": "GNU Library General Public Licence"
        }
      ],
      "percentage_of_license_text": 100.0,
      "scan_errors": []
    }

Problem

problem in licence word Look the text, in the text licence word is present according SPDX license list matching guidelines both word license or license have same meaning.

Lets take new text GNU Library General Public License here licence is replaced with license now scan this text.

And see the result.

{
      "path": "check.txt",
      "type": "file",
      "detected_license_expression": "lgpl-2.0-plus",
      "detected_license_expression_spdx": "LGPL-2.0-or-later",
      "license_detections": [
        {
          "license_expression": "lgpl-2.0-plus",
          "license_expression_spdx": "LGPL-2.0-or-later",
          "matches": [
            {
              "license_expression": "lgpl-2.0-plus",
              "license_expression_spdx": "LGPL-2.0-or-later",
              "from_file": "check.txt",
              "start_line": 1,
              "end_line": 1,
              "matcher": "1-hash",
              "score": 100.0,
              "matched_length": 5,
              "match_coverage": 100.0,
              "rule_relevance": 100,
              "rule_identifier": "lgpl-2.0-plus_186.RULE",
              "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/lgpl-2.0-plus_186.RULE",
              "matched_text": "GNU Library General Public License",
              "matched_text_diagnostics": "GNU Library General Public License"
            }
          ],
          "identifier": "lgpl_2_0_plus-07b4098a-cf1f-a108-69d6-3021dbfb3959"
        }
      ],
      "license_clues": [],
      "percentage_of_license_text": 100.0,
      "scan_errors": []
    }

Here we detect the correct license.

[alok1304]

Let's see some examples:

1. For EUPL license we have only rule for EUPL licence i.e https://github.com/aboutcode-org/scancode-toolkit/blob/develop/src/licensedcode/data/rules/eupl-1.2_17.RULE
   for this we have only 408 files see this : https://github.com/search?q=%22EUPL+licence%22&type=code

we have no rule for this EUPL license but there are many file which have this sentences about 3.9k files
see this: https://github.com/search?q=%22EUPL+license%22&type=code
when we scan this text we got nothing in license detection.

2. And one more example is: NCGL-UK License
   we have rule for this text only i.e Non-Commercial Government Licence https://github.com/aboutcode-org/scancode-toolkit/blob/develop/src/licensedcode/data/rules/ncgl-uk-2.0_1.RULE this works fine when we scan this text.

"license_detections": [
        {
          "license_expression": "ncgl-uk-2.0",
          "license_expression_spdx": "NCGL-UK-2.0",
          "matches": [
            {
              "license_expression": "ncgl-uk-2.0",
              "license_expression_spdx": "NCGL-UK-2.0",
              "from_file": "check.txt",
              "start_line": 1,
              "end_line": 1,
              "matcher": "1-hash",
              "score": 100.0,
              "matched_length": 4,
              "match_coverage": 100.0,
              "rule_relevance": 100,
              "rule_identifier": "ncgl-uk-2.0_1.RULE",
              "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/ncgl-uk-2.0_1.RULE",
              "matched_text": "Non-Commercial Government Licence",
              "matched_text_diagnostics": "Non-Commercial Government Licence"
            }
          ],
          "identifier": "ncgl_uk_2_0-d5129bb3-e2ce-b870-f190-5b0518b0adf5"
        }
      ]

we have no rule for this text Non-Commercial Government License when we scan this text we got this result

"license_detections": [
        {
          "license_expression": "proprietary-license",
          "license_expression_spdx": "LicenseRef-scancode-proprietary-license",
          "matches": [
            {
              "license_expression": "proprietary-license",
              "license_expression_spdx": "LicenseRef-scancode-proprietary-license",
              "from_file": "check.txt",
              "start_line": 1,
              "end_line": 1,
              "matcher": "2-aho",
              "score": 100.0,
              "matched_length": 2,
              "match_coverage": 100.0,
              "rule_relevance": 100,
              "rule_identifier": "proprietary-license_544.RULE",
              "rule_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/rules/proprietary-license_544.RULE",
              "matched_text": "Non-Commercial Government License",
              "matched_text_diagnostics": "Non-Commercial"
            }
          ],
          "identifier": "proprietary_license-c6a32c86-c1b9-39ae-0881-23f79f294711"
        }
      ]

here we see false positive license detection due to license word.

possible solution is to add new rule for this.

[alok1304]

Solution of this is simply add a new rule but we have to add many rules for this, there some other similar word like licenced or licensed.

what I do for this is I simply change the query_text if any word that have equivalent word meaning like licence or license
I changed licence to license they work fine but I faced a problem supposed we have licence word in rule there is no rule for license word when I change the query_text the licence word is changed to license but there is no rule for this. we got nothing in license detection. For solve this problem we have add new rule for license word.

[pombredanne]

@alok1304 Hey :) Following yesterday weekly call discussion, how are thing working out?

[alok1304]

@pombredanne Not working ;]
the code which we implement yesterday i.e

        tokens_by_tid={tid: ts for ts, tid in dictionary.items()}
        self.tokens_by_tid =  [
            ts for _tid , ts in sorted(tokens_by_tid.items(), key=itemgetter(1))]

in this we sort by token id(tid) but we have same id for token license and licence, this not work we have always unique key.
we got problem in reindexing
traceback result

File "C:\Users\lenovo\Documents\GitHub\web dev\scancode-toolkit\src\licensedcode\index.py", line 244, in __init__
    self._add_rules(
    ~~~~~~~~~~~~~~~^
        rules,
        ^^^^^^
    ...<2 lines>...
        _license_tokens=_license_tokens,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "C:\Users\lenovo\Documents\GitHub\web dev\scancode-toolkit\src\licensedcode\index.py", line 571, in _add_rules        
    assert len_tokens == highest_tid + 1 == len(dictionary), msg

like that this is happened due to duplicate id for license word. This not work.

My questions is why we are doing this things??
in previous code we simply sort by tid, in new/modified code we simply sort by ts(token) but we have duplicate key/tid that create problem.

I tried on both side that means on query side and index side but the problem is in reindexing, got a lot of rule duplication
this is also not working.
can we do something else? @pombredanne what's your thought on this problem.

if we solve this problem we reduces a lot of false-positive detection and reduction of a lots of rules also. see example 1 for EUPL license : #4190 (comment)

[pombredanne]

@alok1304 this happens to be much more involved that I though initially: See:

• Support equivalent words in license detection #4190 #4215

It required quite a few data change together with the code changes and the deprecation of close to 1000 rules, and a big update to the legalese.

[alok1304]

@pombredanne thanks for the update.

this happens to be much more involved that I though initially: See:

• Support equivalent words in license detection #4190 #4215

It required quite a few data change together with the code changes and the deprecation of close to 1000 rules, and a big update to the legalese.

Yes I see, this is a really big update, initially I thought it was simple but I see lots of changes in the data with code changes.
hope this issue is helpful, From now onwards we have reduced a lot of rules and also false positive detection due to equivalent words.

[pombredanne]

Yes I see, this is a really big update, initially I thought it was simple but I see lots of changes in the data with code changes.
hope this issue is helpful, From now onwards we have reduced a lot of rules and also false positive detection due to equivalent words.

Yes this issue and your efforts have been super helpful there! let's see how this is handled by the PR