Dynamic resources with 1 or more input files

[jchorl]

First, thanks for the awesome tool and all the hard work.

Bug report

Specifying dynamic resources over multiple inputs fails when a single input is passed. Note the issue occurs when inputs are passed in the same method (i.e. collected before being passed).

The goal is to set dynamic ram on a process that takes multiple inputs together (collected). Sometimes, only a single input is passed.

Expected behavior and actual behavior

It should succeed.

Steps to reproduce the problem

Nextflow config:

process {
    container = 'cimg/aws:2023.08'
    executor = 'awsbatch'
    queue = '<my-queue>'
} 
    
aws {
    region = 'ca-central-1'
}

Script:

#!/usr/bin/env nextflow
nextflow.enable.dsl=2

process genFile {
  cpus 1
  memory '1 GB' 
  input:
    val x 
  output:
    path 'out.txt'
  script:
    """
    echo '$x' > out.txt
    """
}

process catFiles {
  cpus 1
  memory { inputs*.size().sum() < 1.GB ? 1.GB : 2.GB }
  input:
    path inputs, stageAs: 'in*.txt'
  output:
    path 'catted.txt'
  script:
    """
    cat $inputs > catted.txt
    """
}

workflow {
  Channel.of(0) | genFile | collect | catFiles | view
}

Error:

bash-4.2# nextflow run main.nf -bucket-dir s3://<bucket>/nextflow
N E X T F L O W  ~  version 23.04.1
Launching `main.nf` [golden_bohr] DSL2 - revision: 36677f67a9
executor >  awsbatch (1)
[c0/673e46] process > genFile (1) [100%] 1 of 1 ✔
[-        ] process > catFiles    [  0%] 0 of -1
ERROR ~ No such file or directory: in.txt

 -- Check script 'main.nf' at line: 19 or see '.nextflow.log' file for more details

Note:

If we change the workflow to:

workflow {
  Channel.of(0..1) | genFile | collect | catFiles | view
}

Then it actually works.

Commenting out memory { inputs*.size().sum() < 1.GB ? 1.GB : 2.GB } also causes it to work.

How can we dynamically compute ram when the input could be 1 or 2 files?

Program output

nextflow.log

Environment

nextflow/nextflow:23.04.3 docker image

In an attempt to cover all bases, I also tried with 23.08.1-edge, same issue.

Additional context

Open to workarounds!

[bentsherman]

This will be fixed by #3706 , then you will be able to specify arity: true for the path input and it will always be a list.

[jchorl]

Thanks Ben! Should I close the issue, or leave it open until the PR is merged? I'm fine either way.

[bentsherman]

meh, I linked it to the PR so it will be closed automatically

[bentsherman]

Fixed by 42504d3