back2source: Run pipeline for JavaScript

[pombredanne]

We have an existing map_deploy_to_devel pipeline to analyze deployment of JavaScript apps using symbols found map files. There is a plan to further improve this with this issue:

• back2source: Refine ScanCode.io d2d pipeline for JavaScript #1405

The plan in #1405 is to refine the analysis using symbols and string matching between minified bundles and sources including JS and TS.

The purpose of this issue #1477 is to evaluate at some scale the results of #1405. And:

1. Collect a bunch of npm projects that use minification and TS
2. Run the pipeline on this.
3. Collect issues and either report as bugs in our code to fix (and fix them)
4. ... or as real issues upstream, then push mini report on this issues for reference here

[Illucious]

Could you provide some context for this

[pombredanne]

@Illucious I updated the description of this issue
If you want to help here, finding a few popular npm packages that contain minified or webpacked bundles in their released npms would be useful towards this, but we will need #1405 completed before we can do actual d2d runs

[pombredanne]

FYI, d2d is short for "Deploy To Development"

[tdruez]

@pombredanne We need the list of inputs to make progress on this one.

[chinyeungli]

Updated the CSV
back2source_javascripts_projects_list-2025-01-23.csv

[tdruez]

Run on the 100 projects list provided at https://github.com/user-attachments/files/18530393/back2source_javascripts_projects_list-2025-01-23.csv

1. Using back2source_javascript_projects_list-2025-01-23.csv

2. Batch create the projects

docker compose -f /opt/scancodeio/docker-compose.yml run --rm \
    --volume $PWD:/input-data:ro \
    web scanpipe batch-create \
    --input-list /input-data/back2source_javascript_projects_list-2025-01-23.csv \
    --pipeline map_deploy_to_develop:JavaScript \
    --label back2source-JavaScript \
    --execute --async

3. Use the "Report" action on the filtered list of projects by "back2source-JavaScript" label.

Results: back2source-report-JavaScript.xlsx

[AyanSinhaMahapatra]

I have triggered a rescan of all the 99 javascript projects from the above csvs now that we have the following PRs merged and we have javascript symobls matching in place thorugh the Javascript optional step:

• Refine ScanCode.io d2d pipeline for JavaScript using symbol mapping #1622
• Refine ScanCode.io d2d pipeline for JavaScript using string literals mapping #1652