Matched text diagnostics should be in a new attribute

[AyanSinhaMahapatra]

Currently when we have --matched-text-diagnostics enabled we add this diagnostics directly in the matched_text attribute for the match. This is problematic as this matched_text is important in many downstream cases like generating attribution, showing diffs etc, and so this should be in a new attribute matched_text_diagnostics.
There would be a downside of increased file size for the result file btw.

[tehami02]

How can I ger started with this matched_text issue( as there are many files that have "matched_text" )...I am beginner to scancode toolkit but I have used it using docker and I want to work on this issue.

[AyanSinhaMahapatra]

In the LicenseMatch.to_dict function, if license_text_diagnostics is True we get the diagnostics at https://github.com/nexB/scancode-toolkit/blob/develop/src/licensedcode/match.py#L773, but this should be stored in a seperate variable and also returned seperately as mentioned above https://github.com/nexB/scancode-toolkit/blob/develop/src/licensedcode/match.py#L797

[tehami02]

Could this updated code solve our issue, also I am unable to figure out how would I test these changes like what data should I scan and what are the commands to run such scans. UPDATED CODE -

    matched_text = None
    matched_text_diagnostics = None
    if include_text:
        if license_text_diagnostics:
            matched_text_diagnostics = self.matched_text(whole_lines=False, highlight=True)
        else:
            if whole_lines:
                matched_text = self.matched_text(whole_lines=True, highlight=False)
            else:
                matched_text = self.matched_text(whole_lines=False, highlight=False)

    result = {}

    # Detection Level Information
    result['score'] = self.score()
    result['start_line'] = self.start_line
    result['end_line'] = self.end_line
    result['matched_length'] = self.len()
    result['match_coverage'] = self.coverage()
    result['matcher'] = self.matcher

    # LicenseDB Level Information (Rule that was matched)
    result['license_expression'] = self.rule.license_expression
    result['rule_identifier'] = self.rule.identifier
    result['rule_relevance'] = self.rule.relevance
    result['rule_url'] = self.rule.rule_url

    if include_text:
        result['matched_text'] = matched_text
    if license_text_diagnostics:
        result['matched_text_diagnostics'] = matched_text_diagnostics

    return result

[AyanSinhaMahapatra]

@tehami02

also I am unable to figure out how would I test these changes

See https://scancode-toolkit.readthedocs.io/en/stable/contribute/contrib_dev.html#running-tests, you can run the unit tests already present, and not new full scans.

what data should I scan and what are the commands to run such scans

The basic example should be sufficient: https://scancode-toolkit.readthedocs.io/en/stable/tutorials/how_to_run_a_scan.html

It's not helpful to have code snippets, instead fork the repo, push your code in a branch and open a PR, even if it's not ready :P
Additionally the tests you run and the outputs should give you more info if you need to change anything from the failing tests.

[tehami02]

@AyanSinhaMahapatra Sorry I had closed my last PR due to commit issues and it has been a month. Should I re-open PR for this issue or try some other issue.

[AyanSinhaMahapatra]

@tehami02 feel free to reopen the PR/open another PR.

[krishna9358]

@AyanSinhaMahapatra Hii, I am new to open source but I would love to resolve this issue. Can you guide me to get to know more about the issue and how to resolve it ?

[AyanSinhaMahapatra]

@krishna9358, here @tehami02 has already opened a PR for this #3510 which is being used in #3620, so you'd have to take up another issue instead, see https://github.com/nexB/scancode-toolkit/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22 for some good first issues. Thanks!