deps: Add official storage plugins

[jonasfreimuth]

Proposed solution to #87.

Summary by CodeRabbit

• Chores / Maintenance

  • Refreshed remote storage integration dependency version constraints for multiple transfer backends (including object/blob, filesystem, and various scientific/academic endpoints) to improve consistency and compatibility across environments.

• Documentation

  • Updated the release notes to reflect dependency constraint changes only (no new protocol features were introduced).

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 5409f056-b2f8-4abf-a55a-14085c7aac03

📥 Commits

Reviewing files that changed from the base of the PR and between 90e87b3 and 0f63419.

⛔ Files ignored due to path filters (1)

• pixi.lock is excluded by !**/*.lock

📒 Files selected for processing (1)

• pixi.toml

---

📝 Walkthrough

Walkthrough

pixi.toml adds multiple Snakemake storage plugin dependencies under [dependencies].

Changes

Storage plugin dependency updates

Layer / File(s)	Summary
Dependency entries pixi.toml	Adds storage backend package constraints to [dependencies].

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~2 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: adding official storage plugin dependencies.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[jonasfreimuth]

Drafted, as during testing I realized adding the SFTP plugin appears to break snakemake 😬

[jonasfreimuth]

Removed the SFPT plugin now. I have already merged it on our fork, see the building action here. I'll undraft once that completed successfully.

[m-jahn]

OK, thanks for testing @jonasfreimuth .
Two question before we merge:

1. on your fork of the catalog, did your workflow now build and display as expected?
2. regarding all the storage plugins, do you think we could only include the top N used ones? I'm not sure some of these are used at all in automatic tests. I'd like to keep the number of software dependencies as low as possible.

[m-jahn]

I just saw that in your build the linting is still failing because of a storage provider error:
https://tron-bioinformatics.github.io/snakemake-workflow-catalog/docs/workflows/TRON-Bioinformatics/oblx.html#linting-tron-bioinformatics-oblx

[jonasfreimuth]

1. on your fork of the catalog, did your workflow now build and display as expected?

No, but that's expected I think because it doesn't rebuild the oblx page due to no changes having happened there that would warrant a rebuild. I have manually generated the rulegraph before using the Pixi env from this PR. I'll try and fudge with the data.json to force a rebuild on our fork's main branch.

2. regarding all the storage plugins, do you think we could only include the top N used ones? I'm not sure some of these are used at all in automatic tests. I'd like to keep the number of software dependencies as low as possible.

Sure tbh I'm fine with whatever, so long as it includes the http storage plugin.

[jonasfreimuth]

I've reset the update time of oblx in the data.json to 1.0 about four weeks ago, I believe that should ensure it's not skipped during page generation. Relevant run

[jonasfreimuth]

Ok, very sorry but apparently that was not enough. I'll draft this again and let you know once things work properly on our fork.

[jonasfreimuth]

Ok, I also removed the snakemake-storage-plugin-webdav plugin, the alternative pypi installation of a plugin caused issues as the interface seems to have been overwritten by the pypi version as well causing errors as below.

Error log

``` INFO:root:Checking repos 0-1 (total: 1), of which 1 have a Snakefile. INFO:root:0 of 1 repos done. INFO:root:Processing TRON-Bioinformatics/oblx. ERROR:root:Traceback (most recent call last): File "/project/.pixi/envs/default/bin/snakemake", line 6, in from snakemake.cli import main File "/project/.pixi/envs/default/lib/python3.13/site-packages/snakemake/__init__.py", line 15, in from snakemake.shell import shell File "/project/.pixi/envs/default/lib/python3.13/site-packages/snakemake/shell.py", line 17, in from snakemake.utils import format, argvquote, cmd_exe_quote File "/project/.pixi/envs/default/lib/python3.13/site-packages/snakemake/utils.py", line 19, in from snakemake.common.configfile import _load_configfile File "/project/.pixi/envs/default/lib/python3.13/site-packages/snakemake/common/__init__.py", line 25, in from snakemake.exceptions import NestedCoroutineError File "/project/.pixi/envs/default/lib/python3.13/site-packages/snakemake/exceptions.py", line 10, in from snakemake_interface_common.exceptions import WorkflowError, ApiError ModuleNotFoundError: No module named 'snakemake_interface_common' ```

As for further reducing the list of plugins, I wouldn't know by what metric. The plugin catalog doesn't give usage rankings I don't think. We could check the github stars, but to me that isn't really a metric that translates well enough to the useage numbers we would be interested in.

[jonasfreimuth]

Based on that, the latest build seems to work: https://tron-bioinformatics.github.io/snakemake-workflow-catalog/docs/workflows/TRON-Bioinformatics/oblx.html

[m-jahn]

As for further reducing the list of plugins, I wouldn't know by what metric. The plugin catalog doesn't give usage rankings I don't think.

That's a valid point, and I thought about this as well. I'm not using storage plugins, can you quickly point out to me how those are used? With a profile, like one would use executor plugins (we do use the snakemake slurm plugin for example)?

[jonasfreimuth]

That's a valid point, and I thought about this as well. I'm not using storage plugins, can you quickly point out to me how those are used? With a profile, like one would use executor plugins (we do use the snakemake slurm plugin for example)?

You generally wrap a URI in a storage call, then you can configure different storage interfaces in the Snakefile (e.g., credentials, limits, timeouts, etc). The plugin is then responsible for retrieving and potentially keeping the retrieved file in the .snakemake/storage dir. We don't actually configure anything, we just do the wrapping. An example of how we use it can be seen here.

That's also why we need to include the storage plugins in the env. Executor plugins for example are not integrated that close into a workflow, they just help with executing it, but a workflow can generally run without a plugin. When a workflow depends on external resources though, the stroage plugin is required to retrieve those resources in all cases. THat is at least my understanding.

[m-jahn]

Thanks for testing. The performance of the generate-catalog action is also about the same as before, which is good.

[jonasfreimuth]

Great! Thanks for reviewing and helping this get done!