Fix result bucket for GHA-triggered prod runs

[sjspielman]

I'm opening this issue following the adventure in #145. Two things to note:

1. We might update instructions in our release template to state that the manually issued staging run should be run without resume. I had some caching problems the first time around there
2. Most importantly, there seems to be a bug in how the result bucket is named on GHA-triggered prod runs, but not on manually-triggered prod runs. It's not immediately clear how this bug came to be since the associated files have not changed recently - the GHA yaml was last modified 2024-11 and run_nextflow.sh was last modified 2025-01 and we've had successful release runs since those dates.
   • Editing to note that I did not carefully go through the logic rules in these scripts on my end when opening this issue

[jashapiro]

Looking at the runs that were done, it seems that the most recent manual "prod" runs were not run with the v0.1.5 tag for the revision, which is probably fine, but a potential source of the differences.

[jashapiro]

I believe the result bucket is actually being set correctly. I looked at the runs that were tagged with the v0.1.5 version, and those seem to be properly set: e.g., https://cloud.seqera.io/orgs/CCDL/workspaces/OpenScPCA/watch/3inC7jE2tpIz61/v2/configuration

I believe all of the unexpected behavior was related to the use of -resume, which may not be behaving as expected. While in theory changes to the contents of any called scripts should invalidate the cache, I am not sure that is happening. A short term fix would be to remove the -resume flag on prod runs, which seems reasonable.

Since this is not expected behavior, I will do a bit of testing to see if it is a bug in Nextflow itself.

[jashapiro]

Confirming that this is an apparent bug in Nextflow: scripts within bin are checked for modification, but scripts within modules are not

[jashapiro]

Turns out this bug should not actually affect us in our current usage, since we are using wave containers: nextflow-io/nextflow#6128

I tested locally with wave containers, and indeed this does avoid the bug (as long as the contents of the script change, touch is not sufficient).

Nonetheless, I think the appropriate response is to remove -resume from the prod runs.

[jashapiro]

Adding a note that part of this will be to add run names to production runs, for easier tracking

[allyhawkins]

I think we want to go ahead and remove the -resume flag from the prod run triggered by GHA.
This is the run that was just triggered from the latest release and it has the correct results bucket, but it cached pretty much everything and all the dates of the files in the bucket are the old dates and not from today. I'm kind of surprised since we changed some of the parameters, but it seems that -resume is not working as we would expect.

https://cloud.seqera.io/orgs/CCDL/workspaces/OpenScPCA/watch/46tieLedr2DLPw/v2/tasks

[allyhawkins]

Interestingly, the run that creates the test results did re-run everything
https://cloud.seqera.io/orgs/CCDL/workspaces/OpenScPCA/watch/2rM4r3Q9PinVEX/v2/tasks

[jashapiro]

I think we want to go ahead and remove the -resume flag from the prod run triggered by GHA. This is the run that was just triggered from the latest release and it has the correct results bucket, but it cached pretty much everything and all the dates of the files in the bucket are the old dates and not from today. I'm kind of surprised since we changed some of the parameters, but it seems that -resume is not working as we would expect.

https://cloud.seqera.io/orgs/CCDL/workspaces/OpenScPCA/watch/46tieLedr2DLPw/v2/tasks

The caching is kind of expected, because the prod run should in general be identical to the staging run from the day before, just with different publishDir settings. But why it isn't publishing the changed files seems to be getting back to the original complaint when this issue was filed.