npe2api shouldn't list plugins that have been taken down from PyPI/conda-forge

[jni]

Based on this Zulip thread, I looked around for mmv_hitl4trk in the repo and found it listed in index.json:

npe2api/public/index.json

Lines 53 to 54 in d477c45

	"mmv_h4tracks": "1.1.1",
	"mmv_hitl4trk": "1.0.0",

It seems to me that plugins that have been removed from PyPI and conda should not appear in npe2api, but I'm not 100% sure whether there are some npe2api use cases that require historical data.

CC @lennartkowitz @DragaDoncila @jaimergp

[Czaki]

I just downloaded wheel from pypi, and it contains all metadata required for being visible:

[jni]

@Czaki that's the new package? The old package is not available on PyPI anymore?

[jni]

The old package name is mmv_hitl4trk

[Czaki]

old name is listed in deleted section of classifiers.
https://github.com/napari/npe2api/blob/main/public/classifiers.json#L4241

However, it is listed in an endpoint that is used by the plugin manager

But without any available version (also for correct name).

So it suggests that error is in https://github.com/napari/npe2api/blob/main/scripts/reindex.py or in a plugin manager code.

Maybe @tlambert03 remember what is expected behavior.

[tlambert03]

didn't this come up before @DragaDoncila?

yes, as @Czaki said, the source of truth for plugin status is classifiers.json, and the presence of mmv-hitl4trk in the deleted section of that file should be considered the important info here. There's been more hands on reindex.py, and I'm not up to date anymore on exactly how napari is using the various endpoints offered by npe2api. So the question of how napari (the application) ends up integrating that fact is not something I can answer. I would start by looking at the code in napari proper, and see how it's gathering items... figure out what end-points it's hitting, and then determine whether you want to actually change the index here, by modifying reindex.py... or whether you want to change the displaying in napari based on, for example, an empty list of pypi_versions...

Originally, I intended for there to be a two-step fetch process from npe2api (one to classifiers to get the active/withdrawn/deleted status, and a second one to fetch details) ... but I think there was a push to combine all of the info that napari wanted into a single endpoint (can't remember now if that was "summary" or "extended_summary" and I can't remember what the difference between those is). So i'm no longer sure about where the data is getting aggregated

[Czaki]

Currently, napari iterates over entries of extended_summary and puts it in plugin manager dialog. Nothing more.
extended_summary contains information about available versions of package

[tlambert03]

ok, then one possible solution here is for the plugin manager dialog to skip packages for which both pypi_versions and conda_versions are empty. If you want, you can also update the logic of reindex...

[Czaki]

Ok. So it looks like we have a clear view of the situation and could make a decision now. Thanks @tlambert03 for the information.

[DragaDoncila]

@jni do you have a preference for whether we update reindex to just not include in extended_summary any packages that have no conda and no pypi versions, or whether we update the plugin manager to read the info from the extended_summary?

We should at a minimum update reindex to read the visibility from the manifest and include it in extended_summary and make sure the plugin manager reads that...

[jni]

I don't have a strong preference but I don't see a reason why packages that aren't in either package repository should appear in the extended_summary... (If we ever e.g. allow direct installs from GitHub, then we can treat that as another package repository and apply the same logic.)

[DragaDoncila]

OK so sounds like action item is to update the reindex script to exclude packages with no listed versions in any repository and include manifest visibility in the extended_summary.