Feature/batch info provider import

[barisgit]

• Add ability to batch update parts from info providers

Works in the following way:

1. User selects parts from any table and chooses action 'Bulk import from info provider'

2. Gets redirected to step 1, where the user needs to choose how to search for parts (eg. user mpn to search on LCSC and MOUSER and at the same time use lcsc spn to search on LCSC)

4. Upon clicking "Search all parts", the server searches for all parts at once, presenting user with similar interface to single part import but for multiple parts, while at the same time creating a row in a new table 'bulk_info_provider_import_jobs' so that the progress is not lost upon refreshing. Optionally user can also check to prefetch details for every part in this stage (if the search returns too many parts, for example if we were to search by a vague keyword, this could mean it takes a long time, but can be prevented by choosing supplier part numbers instead). Prefetched data gets automatically stored in cache

6. The job is then available to be opened from tools pane as well. On the job pane the user is presented with a list of all parts, where each part has search results. When user clicks update part they get redirected to info provider edit, with additional query parameter that tells page it is associated with a job. This way once saving part there is a button to go back, but at the same time mark part as completed in a job.

---

Note: I have by a mistake continued to work right on top of my other addition (excel support), so it would be best to first finish that or maybe even discard it and do everything here.

[codecov]

Codecov Report

❌ Patch coverage is 73.55649% with 316 lines in your changes missing coverage. Please review.
✅ Project coverage is 61.35%. Comparing base (8b417d6) to head (66f7629).
⚠️ Report is 7 commits behind head on master.

Files with missing lines	Patch %	Lines
...rc/Controller/BulkInfoProviderImportController.php	56.15%	171 Missing ⚠️
src/Controller/PartController.php	43.75%	27 Missing ⚠️
...Form/InfoProviderSystem/BulkProviderSearchType.php	0.00%	19 Missing ⚠️
...foProviderSystem/PartProviderConfigurationType.php	0.00%	19 Missing ⚠️
src/Services/EntityMergers/Mergers/PartMerger.php	5.00%	19 Missing ⚠️
src/Services/ImportExportSystem/EntityImporter.php	83.52%	14 Missing ⚠️
src/Entity/Parts/Part.php	7.69%	12 Missing ⚠️
...ices/InfoProviderSystem/Providers/LCSCProvider.php	83.33%	11 Missing ⚠️
src/Entity/BulkInfoProviderImportJob.php	94.90%	8 Missing ⚠️
src/Services/Parts/PartsTableActionHandler.php	11.11%	8 Missing ⚠️
... and 3 more

Additional details and impacted files

@@             Coverage Diff              @@
##             master     #982      +/-   ##
============================================
+ Coverage     59.83%   61.35%   +1.51%     
- Complexity     6050     6326     +276     
============================================
  Files           529      542      +13     
  Lines         20625    21730    +1105     
============================================
+ Hits          12341    13332     +991     
- Misses         8284     8398     +114     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[barisgit]

Here is description of excel functionality pull report

• Allow using xlsx and xls to import parts
• Update example csv to better show what is possible
• Add required tests (PASS)
• Fix parts merger when using update from info supplier to consider parts that already have a spn populated a duplicate

[barisgit]

Added a tab for filtering by job statuses and such

[barisgit]

Do not merge yet, there is a bug when user doesn't check prefetch and it still fetches info, causing timeout on larger sets.

[barisgit]

Add priority support in mapping table

[barisgit]

Hey @jbtronics any clue when this could be merged/rejected? Otherwise we will need to continue on our own fork, not providing any contributions here anymore.

[jbtronics]

I will try to release the V2 version in the next few days, it's basically only missing some documentation. Then I can review and merge the PRs.

As some base stuff changed, it will probably be easier to merge the PRs into the V2 changes than the other way around....

[barisgit]

All done, it should be better now, one thing i did was just refresh page (with js) after clicking a button in a bulk import job, maybe this code be done better. Also, why are github actions not running?

[barisgit]

@jbtronics Can you take a quick look at structure to see if it is better so you don't have to wait for me if you find out there are additional things to do

[jbtronics]

Its much better now. The controller is good now I think, extracting most of the logic into a service sounds good too.

Still, I do not really like the large complex arrays that get passed around (especially between different logical units). The performBulkSearch in the BulkInofProviderService returns a large complex array, which is basically impossible to understand without looking into the source code. I think returning a DTO would be helpful for making that easier to understand what it contains and more maintainable in the long run. For the internal passing around of data inside the service the array is okay, but then the arrays should have some type and or structure hints for parameter and return value, otherwise its very hard to understand what is done there... I have already started to add some typehints, but they are still missing in some places.

The field mapping could maybe also be extracted into a DTO, instead of an array, as it is used in many places. And the form could directly interact on these objects then.

Im not too sure about how useful the BulkSearchRequestDTO is, because it is only used in the one method signature, and is directly created before the method call and split up immediatley afterwards again. Here replacing the dto with named arguments in the performBulkSearch is probably doing the same.

I do not think its too useful to have BulkSearchResultDTO as child class of SearchResultDTO, as this needs to be updated everytime a new field is updated. I think it is more useful to just encapsulate the SearchResultDTO there. But i am not sure how it is used later.
Also its name is at least a bit misleading, as it is not really the counterpart to BulkSearchRequestDTO (in the sense that is directly returned by the performBulkSearch), nor returned by the BatchInfoProviderInterface

[barisgit]

@jbtronics Please tell me if you see any more bugs or areas for improvement, also i fixed a bug where "Add Mapping" wasn't working.

[jbtronics]

I did some refactoring to remove the remains of the arrays everywhere. Also did some other improvements.

I have now merged the PR, thank you.

Maybe some notes: The structure how the search results are serialized to the DB have changed slightly to your version, so if you had db entries already, you should probably delete all import jobs.

In the BulkInfoProviderImportController::researchAllParts() method, you used the entitymanager clear method to save a bit of memory. This caused problems in my tests (the flush complained about a missing user entity), so i commented the clear out.

For the future, it would probably be cool to make the whole data retrieval thing which can take some time truely asynchronous from the web tasks. This should be not so difficult with the symfony/messenger component, but so far Part-DB does not use it yet.