How to use `rcOutDir` option to process multiple samples and avoid out of memory errors?

[bernardo-heberle]

Hello,

I am running 9 cDNA samples sequenced on the PromethION with Bambu. Each ".bam" file is ~30GB long after filtering the reads, so the total amount of data being processed by Bambu is ~270GB. I noticed that the memory requirements are getting pretty high, as the job will fail with 500GB of RAM memory, but will complete if I increase the RAM to 1000GB. Here is the command I am running:

se_novel <- bambu(reads = bam, annotations = bambuAnnotations, rcOutDir = "./bambu_processed_files/", genome = fa_file, lowMemory=TRUE, ncore=8, opt.discovery = list(min.sampleNumber = 5, min.readCount = 5))

The bam variable is a vector with the paths for the 9 ".bam" files.

I was wondering if I am using the rcOutDir option correctly?

It is my understanding that this option is supposed to help with runs utilizing multiple samples, but I am not sure if there is an intermediary step that I am missing. I ask this because in the long run I intend to use bambu to process several dozens, if not hundreds, of cDNA samples generated on the PromethION. However, with the increase in RAM requirements, memory will probably become a limiting factor for processing a larger number of samples.

Any help with this and/or tips on how to avoid out of memory errors with a large number of large samples will be much appreciated!

[cying111]

Hi @bernardo-heberle, usually the processing memory hikes when preprocessing the bam files, so what we would recommend for very large samples, is to 1) process them individually with discovery mode only discovery = TRUE, quantification = FALSE and save the read class file to rcOutDir as what you have done, and then after all files are processed, 2) supply all read class files' paths as a character vector to the rcFile argument rcFile = path_to_all_read_class_files and then re-run bambu with both discovery and quantification mode on. One tip is that when you process the large sample bam file with only discovery mode on, you can set a specific value to yieldSize, which controls the read in of bam files, say 1,000,000 yieldSize = 1000000, so that the memory usage is a bit controlled. Another tip, is that for both steps, you might want to use a limited number of cpus, when you fail with a certain number of cpus, you can always lower the cpu number and try again.

[bernardo-heberle]

Thank you for the quick and effective response @cying111! The solutions you suggested worked well and I am able to run all of the samples with no issues now.