Skip to content

Sync up unique cache paths for Singularity images with cwltool#433

Open
adamnovak wants to merge 2 commits into
common-workflow-language:mainfrom
adamnovak:robust-sif-cache
Open

Sync up unique cache paths for Singularity images with cwltool#433
adamnovak wants to merge 2 commits into
common-workflow-language:mainfrom
adamnovak:robust-sif-cache

Conversation

@adamnovak

Copy link
Copy Markdown

This and common-workflow-language/cwltool#2284 should make cwltool and cwl-docker-extract agree on what filenames to use for SIF images, even when the image names are unusual.

They should agree for everything they both support, which is Docker-format specifiers with and without tags. cwl-docker-extract doesn't support Singularity-isms like direct SIF file references or docker:// URIs even when run in Singularity mode.

@codecov

codecov Bot commented Jun 4, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 80.95238% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 37.79%. Comparing base (a466f42) to head (e980f78).

Files with missing lines Patch % Lines
src/cwl_utils/tests/test_image_puller.py 76.47% 2 Missing and 2 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #433      +/-   ##
==========================================
+ Coverage   37.77%   37.79%   +0.01%     
==========================================
  Files          50       51       +1     
  Lines       36760    36777      +17     
  Branches     9531     9533       +2     
==========================================
+ Hits        13886    13899      +13     
- Misses      19941    19943       +2     
- Partials     2933     2935       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@adamnovak

Copy link
Copy Markdown
Author

I can't figure out how to run just the tests I wrote here and lint just for one Python, so I can run the tests locally in a reasonable amount of time.

@adamnovak

Copy link
Copy Markdown
Author

Looks like it's tox -e py314-unit -- src/cwl_utils/tests/test_image_puller.py and tox -e py314-lint.

Comment on lines -89 to +94
CHARS_TO_REPLACE = ["/", ":"]
NEW_CHAR = "_"
CHARS_TO_REPLACE = ["_", "/"]
NEW_STRINGS = ["___", "_s_"]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @adamnovak ! Thank you for this and the linked PR. I worry that this will cause existing stored images to be ignored. Any idea on how we can diverge less from the previously naming scheme?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One problem that's being solved here is name collisions: it's possible with the previous naming scheme to have two distinct images (mostly pairs like a/b_c and a_b/c) share a cache key.

The obvious way to re-use existing cached images would be to look and see if the files already exist under the old scheme. Maybe that's OK even if there were possible collisions, since we won't create any new collisions and if people have images cached they must have been working.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The obvious way to re-use existing cached images would be to look and see if the files already exist under the old scheme.

yes, please and thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants