Excessive Memory Usage and Continuous Growth When Processing hog_big in FastOMA with 2570 Species

[alvinliu89757]

Hi

I am using FastOMA to infer orthologous groups from protein sequences of 2570 species. The overall process seemed to be progressing smoothly initially. However, the computation has been stuck on the hog_big processing step for over a week. During this time, I've observed a severe and continuous surge in memory (RAM) usage.

Initially, I allocated 4 TB of memory, which was completely filled. I then increased the available memory to 12 TB to accommodate the growth, but the memory footprint continues to increase and is consuming this additional capacity as well. The memory statue: size 12T used 4.8T avail 7.3T use% 40%.

Now statue:
executor> local (178) [f4/408a97] check_input (1) | 1 of 1, cached: 1 ✔ [8a/4d0f3d] oma…peruviana_NRRL66754.fasta) | 2570 of 2570, cached: 2570 ✔ [f7/5dac82] infer_roothogs (1) | 1 of 1, cached: 1 ✔ [25/b25ad3] batch_roothogs (1) | 1 of 1, cached: 1 ✔ [77/1e0f3d] hog_big (1054) | 2497 of 7378, cached: 2445, retries: 117 [a7/42e896] hog_rest (1231) | 2106 of 2106, cached: 2106 ✔ [- ] collect_subhogs - [- ] ext…airwise_ortholog_relations - [- ] fastoma_report - [2c/6fe723] NOTE: Process hog_big (1004) terminated with an error exit status (137) -- Execution is retried (1)

My run parameter:

`#!/bin/bash
#SBATCH --job-name=NP_batchScan_P1
#SBATCH -N 1
#SBATCH --partition=smp
#SBATCH --cpus-per-task=36
#SBATCH --output=%x_%j.out
#SBATCH --error=%x_%j.err

source /public/home/user/miniconda3/bin/activate
conda activate FastOMA-v04

CONFIG_PATH="/public/home/user/FOMA_ALF/custom_memory.config"

cd /public/home/user/FastOMA_0.4.0/FastOMA

nextflow run FastOMA.nf
-resume
-c $CONFIG_PATH
--input_folder /public/home/user/FOMA_ALF/allSpe
--output_folder /public/home/user/FOMA_ALF/results
--omamer_db /public/home/user/FastOMA/database/LUCA.h5`

my config set
`ss {
// Force the specific process to run (if it fits)
withName: hog_big {
memory = '128.GB'
}
}

// Tell the executor (local machine) it is allowed to use up to 480 GB total
executor {
name = 'local'
memory = '480.GB'
}
`

[alpae]

Hi, if you submit the pipeline like this, you essentially run all processes on a single node. the nextflow pipeline will submit many jobs in parallel, overloading this node. A better strategy is usually to run each of the nextflow processes as a single slurm job. the orchestrating nextflow job can be run on the login node (maybe in screen session), or be submitted itself to slurm (make sure that you can submit other slurm jobs from within a compute node in that case).

also, I suggest you update to the latest version (0.5.1), as it simplifies the profiles and configs quite a bit.

nextlfow run -profile slurm,conda,large \
   --input /public/home/user/FOMA_ALF/allSpe \
   --output_folder /public/home/user/FOMA_ALF/results \
   --omamer_db /public/home/user/FastOMA/database/LUCA.h5

(profile slurm submits each process as a slurm job, conda creates the appropriate conda environment, and large makes bigger estimates for time and memory than usual, which is encouraged for large datasets).

if you want to provide a hard cap on the amount of resources each process can have a most, you can create an additional config file with the proper resource limits:

process {
   resourceLimits = [cpus: 32, memory: 512.GB, time: 72.h ]
}

I hope these are a few pointers that are helpful. If you prefer to run everything on a single node as a single slurm job, I think you need to ensure that not too many processes get spawned at the same time by specifying a queueSize value. however, I haven't been looking into this at all for local exectution.

[alvinliu89757]

Thank you for your quick reply.

I have already installed version 0.5.1. However, my original task has been running for over a week. Should I run the entire pipeline from the beginning, or is there a resume option available?

[alpae]

nextflow's -resume option is in principle able to restart from where it left. however, with upgrading from 0.4.0 to 0.5.1, it's quite unlikly that this will work and it probably will restart from scratch.
But you can probably reuse the omamer mappings. for that, you would need to provide the --hogmap_in parameter to the hogmap folder from the initial output folder. See https://github.com/DessimozLab/FastOMA?tab=readme-ov-file#using-omamers-output

[alvinliu89757]

Hi there,

I just wanted to say a big thanks to the developers. Since upgrading to version 0.5.1, the dramatic memory acceleration issues we were facing have been addressed.

For anyone else encountering memory caps on HPC clusters, we were able to successfully stabilize the run by forcing Conda and applying a global memory limit with a specific override for the hog_big process.

Here is the configuration that worked for us:

custom_memory.config

// 1. Force Conda to be enabled
conda.enabled = true

// 2. Process Configuration
process {
    // General settings for all processes
    conda = '/path/to/miniconda3/envs/FastOMA-v05'
    executor = 'local'

    // GLOBAL SAFETY CAP:
    // This sets the default memory for ALL actions to 460 GB.
    // This ensures no task requests more than the machine has available.
    memory = '460 GB'

    // 3. Specific override for the crashing process
    // The error in previous logs was from 'hog_big' (requiring 512 GB).
    // We explicitly force it down here to ensure it launches.
    withName: 'hog_big' {
        memory = '460 GB'
    }
}

Then Run script

#!/bin/bash
#SBATCH --job-name=NP_batchScan_P1
#SBATCH -N 1
#SBATCH --partition=supersmp
#SBATCH --cpus-per-task=36
#SBATCH --mem=460G
#SBATCH --output=%x_%j.out
#SBATCH --error=%x_%j.err

source /path/to/miniconda3/bin/activate
conda activate FastOMA-v04

# Note: We save the path to your config file before changing directories
CONFIG_PATH="/path/to/FOMA_ALF/custom_memory.config"

cd /path/to/FastOMA_0.5.2/FastOMA

nextflow run FastOMA.nf \
  -resume \
  -profile conda,large \
  -c $CONFIG_PATH \
  --input /path/to/FOMA_ALF/allSpe \
  --output_folder /path/to/FOMA_ALF/results \
  --omamer_db /path/to/FastOMA/database/LUCA.h5

Thank you for the thoughtful updates and clear documentation in v0.5.1—they made troubleshooting straightforward and allowed us to continue our work without disruption. We truly appreciate your continued support and development of FastOMA!

Best regards