transfer packages from conda-forge to bioconda

[beckermr]

This issue tracks a new procedure for transferring packages from conda-forge to bioconda. The plan is to test it out once and if it works, then formalize it in the docs.

Steps:

1. copy the artifacts from conda-forge to bioconda
2. mark them broken on conda-forge (a special procedure for conda-forge that removes them from the main label repodata, but keeps them available for lockfiles, explicit env exports, etc.)
3. add the package to bioconda
4. change the conda-forge recipe by having it be a simple metapackage that installs from bioconda via bioconda::

I think this would:

• fix solver priority issues so that everything is pulled from bioconda
• ensure that folks using only conda-forge can get at least get some sort of solver problem or issue or even an install of the package for updated versions
• ensure that folks using both bioconda and conda-forge can get updated versions
• avoid a circular dependence in the conda-forge feedstock on using bioconda to build a package (instead it is only used to point to bioconda via the metapackage, which is less fragile)

If this works, we'll formalize it in the docs.

To Do Items:

• try this with r-nmf
  • copy the artifacts from conda-forge to bioconda
  • mark them broken on conda-forge (a special procedure for conda-forge that removes them from the main label repodata, but keeps them available for lockfiles, explicit env exports, etc.) (fix: mark all of r-nmf broken admin-requests#2014)
  • add the package to bioconda (Add r-nmf (migration from conda-forge for versions with bioconductor dependencies) bioconda/bioconda-recipes#64030)
  • change the conda-forge recipe by having it be a simple metapackage that installs from bioconda via bioconda:: (fix: point to bioconda r-nmf-feedstock#27)
  • possibly backfill old package versions with metapackages as well
• formalize in the docs if it works

xref: conda-forge/cfep#52

cc @conda-forge/core for comments and advice

[dlaehnemann]

Nice, thanks for getting this rolling right away. Let me know if I can do anything on 1. and 2., or once they are done. I can then re-trigger the CI on the existing r-nmf pull request over on bioconda, to see if the channel_priority: strict works out.

[beckermr]

We only need to mark the conda-forge artifacts as broken in order to test the channel priority issue.

[beckermr]

We hit the first issue. conda-build doesn't understand dependencies of packages of the same name on different channels. https://github.com/conda-forge/conda-forge-webservices/actions/runs/24342131036/job/71073427054

[beckermr]

See here for the build error: https://github.com/conda-forge/r-nmf-feedstock/actions/runs/24342144213/job/71073502908?pr=27

[beckermr]

OK so it turns out conda is not smart enough to solve for packages that point to themselves on another channel. Quick question @dlaehnemann. Can we add a package on bioconda that is named bioconda-r-nmf which is a metapackage that points to bioconda::r-nmf (with the channel specified)? Then conda-forge could depend on that to avoid the circular dependence.

[dlaehnemann]

Argh, that's annoying.

But wouldn't this workaround mean, that we would have to keep updating not only the r-nmf package on bioconda with every new version, but also the bioconda-r-nmf package alongside it? That sounds like something that is easily forgotten during updates. Or would this only apply for those versions where a conda-forge artefact has been moved over to bioconda?

Also, I was just thinking that if we then put back a recipe that builds a package of the same name for conda-forge (for any version, even just the old ones), this will re-trigger the channel_priority: strict issue, right? Because then we again have a package of that name, not marked as broken, on conda-forge. So I think we have a catch-22...?!

This came up while I was trying something completely different: I also looked if the build error could be prevented by making conda_build behaviour smarter. I had a quick look at the respective code and at least the [MatchSpec instance behind ms](https://github.com/conda/conda-build/blob/26ac2051ddd2d9f0cb232ec3d19cb8c0f72fa56c/conda_build/metadata.py#L1666) that is used in the ms.name, should also have an ms.channelfield associated with it. But theself.namethis is compared against is aMetaDatainstance and I couldn't see whether or where this might have anychannel` info.

[beckermr]

But wouldn't this workaround mean, that we would have to keep updating not only the r-nmf package on bioconda with every new version, but also the bioconda-r-nmf package alongside it? That sounds like something that is easily forgotten during updates. Or would this only apply for those versions where a conda-forge artefact has been moved over to bioconda?

You can use a single recipe with multiple outputs.

Also, I was just thinking that if we then put back a recipe that builds a package of the same name for conda-forge (for any version, even just the old ones), this will re-trigger the channel_priority: strict issue, right? Because then we again have a package of that name, not marked as broken, on conda-forge. So I think we have a catch-22...?!

My hope was that the pointer to the bioconda package would resolve this.

[dlaehnemann]

You can use a single recipe with multiple outputs.
Ah, OK. Yeah, I'm currently still sorting out the proper handling of multiple outputs in recipes for bioconda. But such a metapackage should be possible, once that is sorted.

Also, I was just thinking that if we then put back a recipe that builds a package of the same name for conda-forge (for any version, even just the old ones), this will re-trigger the channel_priority: strict issue, right? Because then we again have a package of that name, not marked as broken, on conda-forge. So I think we have a catch-22...?!

My hope was that the pointer to the bioconda package would resolve this.
But from the current pull request, this would build an artefact and would upload it, right? And then we will have it in the repodata JSON file and the channel_priority will complain again. Or am I misunderstanding the exact mechanisms, here?

Maybe we simply cannot solve the case of people only using conda-forge and wanting to use a conda-forge-to-bioconda migrated package. And we would break existing environment definitions with r-nmf and only conda-forge, unless they also have a lock file in place...

[beckermr]

Yes you may be correct, but currently we've hit some other bugs that make it hard to test things.

For some reason rattler-build is not finding packages on bioconda's channels.