Advanced Slurm Job Scheduling

In the High Performance Computing (HPC) environment, all users execute their calculation or simulation via submitting a job. Ever since then, more and more complex and heavy jobs have been designed. It is quite common that in today’s scientific researches involve multiple different inputs with same calculations or a sequence of jobs which depend on each previous job’s result. Imagine you are to run the same calculation with 100 different input files. Are you going to submit the jobs one by one? Luckily, SLURM has some built-in features to simplify and automate the submission.

Slurm Job Array

SLURM job arrays allow users to submit and manage a collection of similar jobs quickly and easily. However, this feature is only supported for batch mode jobs. The Slurm job parameter for job array tasks is –array or -a. The option argument can be specified are array index values, a range of index values, and an optional step size. Jobs which are part of a job array will have the environment variable SLURM_ARRAY_TASK_ID set to its array index value. Let’s take a look at the example below:

# Submit a job array with index values between 0 and 31
$ sbatch --array=0-31    -N1 tmp

# Submit a job array with index values of 1, 3, 5 and 7
$ sbatch --array=1,3,5,7 -N1 tmp

# Submit a job array with index values between 1 and 7
# with a step size of 2 (i.e. 1, 3, 5 and 7)
$ sbatch --array=1-7:2   -N1 tmp

# Submit a job array with with index values between 0 and 15
# with 4 allowed simultaneous task
$ sbatch -array=0-15%4   -N1 tmp

With job arrays, a few additional environment variables are introduced,

• SLURM_ARRAY_JOB_ID
• SLURM_ARRAY_TASK_ID
• SLURM_ARRAY_TASK_COUNT
• SLURM_ARRAY_TASK_MAX
• SLURM_ARRAY_TASK_MIN

SLURM_ARRAY_JOB_ID will be set to the first job ID of the array. SLURM_ARRAY_TASK_ID will be set to the job array index value. SLURM_ARRAY_TASK_COUNT will be set to the number of tasks in the job array. SLURM_ARRAY_TASK_MAX will be set to the highest job array index value. SLURM_ARRAY_TASK_MIN will be set to the lowest job array index value. Let’s take a look at the example below:

# A job is submitted as follow
$ sbatch --array=1-3 -N1 tmp
# This job will generate an array containing 3 jobs.
Submitted batch job 15
# then the environment variables will be set as follows:

SLURM_JOB_ID=15
SLURM_ARRAY_JOB_ID=15
SLURM_ARRAY_TASK_ID=1
SLURM_ARRAY_TASK_COUNT=3
SLURM_ARRAY_TASK_MAX=3
SLURM_ARRAY_TASK_MIN=1

SLURM_JOB_ID=16
SLURM_ARRAY_JOB_ID=15
SLURM_ARRAY_TASK_ID=2
SLURM_ARRAY_TASK_COUNT=3
SLURM_ARRAY_TASK_MAX=3
SLURM_ARRAY_TASK_MIN=1

SLURM_JOB_ID=17
SLURM_ARRAY_JOB_ID=15
SLURM_ARRAY_TASK_ID=3
SLURM_ARRAY_TASK_COUNT=3
SLURM_ARRAY_TASK_MAX=3
SLURM_ARRAY_TASK_MIN=1

Now, we know the environment variables and Slurm parameters for the job array. How is it going to work?

Let’s start with a simple submission script, array.sh as follow:

#!/bin/bash -l
#SBATCH -p cpu-epyc
#SBATCH -N 1
#SBATCH -n 4
#SBATCH --mem=8G
#SBATCH --array=1-3

echo "Hello HPC!"

The script above will submit a job array consisting of 3 jobs which print “Hello HPC!” for each task. So, how can we utilize this feature to submit a calculation with multiple different inputs?

Arrange Input Files Name

From the script above, we modify the “echo” command to:

echo "Hello HPC from job ${SLURM_ARRAY_JOB_ID}_${SLURM_ARRAY_TASK_ID}"

Now, each job in the array will then print the line with the job ID and task ID. For example:

[george@umhpc] $ sbatch array.sh
Submitted batch job 15
[george@umhpc] $ cat slurm-15_1.out slurm-15_2.out slurm-15_3.out
"Hello HPC from job 15_1"
"Hello HPC from job 15_2"
"Hello HPC from job 15_3"

With the knowledge above, if we arrange our input file name correctly, we will be able to execute the same calculation with different input files easily. For example:

[george@umhpc] $ ls
array.sh  input_1.in  input_2.in  input_3.in 
[george@umhpc] $ cat array.sh
#!/bin/bash -l
#SBATCH -p cpu-epyc
#SBATCH -N 1
#SBATCH -n 4
#SBATCH --mem=8G
#SBATCH --array=1-3

srun my_app < input_${SLURM_ARRAY_TASK_ID}.in > output_${SLURM_ARRAY_TASK_ID}.out
[george@umhpc] $ sbatch array.sh
Submitted batch job 18
[george@umhpc] $ ls
array.sh  input_1.in  input_2.in  input_3.in  output_1.out  output_2.out  output_3.out

Read Input Files Name From A File

In some circumstances, the input file name might not be modifiable. You can prepare a list of filenames in a file and read the names into a list and navigate the list by index. For example:

[george@umhpc] $ cat list.txt
h2o.fasta
nh3.fasta
naoh.fasta
[george@umhpc] $ mapfile -t < list.txt filenames
[george@umhpc] $ echo ${filenames[1]}
[george@umhpc] $ nh3.fasta
# Applying the above approach to scripts
[george@umhpc] $ cat array.sh
#!/bin/bash -l
#SBATCH -p cpu-epyc
#SBATCH -N 1
#SBATCH -n 4
#SBATCH --mem=8G
#SBATCH --array=0-2

mapfile -t < list.txt filenames
srun my_app < ${filenames[${SLURM_ARRAY_TASK_ID}]} > ${filenames[${SLURM_ARRAY_TASK_ID}]}.out
[george@umhpc] $ sbatch array.shSubmitted batch job 21
[george@umhpc] $ ls
array.sh  h2o.fasta  h2o.out  list.txt  naoh.fasta  naoh.out  nh3.fasta nh3.out 

Slurm Job Dependency

Sometimes, you want to do the analysis with the result of a calculation after the job is finished. Queuing the jobs directly might result in an early start of the analysis job while the calculation is still in the process which can cause the job to fail and waiting for the calculation to finish then submit another analysis job seems like a not “smart” way. Luckily, Slurm integrated a dependency job feature to serve this purpose. The Slurm job parameter for dependency jobs is –dependency or -d. The option argument can be specified are logic operator, job finish status, job id and duration in minutes. Let’s take a look into some use case:

[george@umhpc] $ cat job1.sh
#!/bin/bash -l
#SBATCH -p cpu-epyc
#SBATCH -N 1
#SBATCH -n 4
#SBATCH --mem=8G

echo $((1+1)) > result1.out
sleep 30s
[george@umhpc] $ cat job2.sh
#!/bin/bash -l
#SBATCH -p cpu-epyc
#SBATCH -N 1
#SBATCH -n 4
#SBATCH --mem=8G

echo $((2+$(cat result1.sh))) > result2.out
[george@umhpc] $ sbatch job1.sh
Submitted batch job 24
# Start job2.sh after job1.sh is completed successfully.
[george@umhpc] $ sbatch --dependency=afterok:24 job2.sh
Submitted batch job 25
[george@umhpc] $ squeue --me
JOBID PARTITION    NAME    USER  ST    TIME  NODES  NODELIST(REASON)
  24  cpu-epyc job1.sh  george   R    0:05      1  cpu12
  25  cpu-epyc job2.sh  george  PD    0:00      1  (Dependency)
# job2.sh will start after job1.sh is done.
# After both jobs are done
[george@umhpc] $ ls
job1.sh  job2.sh  result1.out  result2.out
[george@umhpc] $ cat result1.out result2.out
2
4

With this feature, you can start to queue a series of jobs which will start automatically under a certain job status. So that, you can submit and manage your jobs more efficiently.

Complex Integration of Job Array and Job Dependency

Now, we have learnt how to use job arrays and job dependency. What can we do next? Imagine you need to do a series of calculations with different input files and you need to analyze all the results with another application. You can automate all the job submissions by simple scripting. For example:

[george@umhpc] $ cat calculation.sh
#!/bin/bash -l
#SBATCH -p cpu-epyc
#SBATCH -N 1
#SBATCH -n 4
#SBATCH --mem=8G
#SBATCH --array=1-6

srun my_app < input_${SLURM_ARRAY_TASK_ID}.in > output_${SLURM_ARRAY_TASK_ID}.out
[george@umhpc] $ cat analysis.sh
#!/bin/bash -l
#SBATCH -p cpu-epyc
#SBATCH -N 1
#SBATCH -n 4
#SBATCH --mem=8G

for i in {1..6}
do
my_analysis_app < output_${i}.out > analysis_${i}
done
[george@umhpc] $ cat automate.sh
#!/bin/bash

# Store the stdout as variable
jobstring=$(sbatch calculation.sh)# Get the last string before the last space into a variable
jobid=${jobstring##* }
sbatch --dependency=afterok:${jobid} analysis.sh[george@umhpc] $ chmod +x automate.sh[george@umhpc] $ ./automate.sh

Assuming your calculation and analysis must be done separately with different applications. One Slurm job submission script can be defined to execute a series of jobs with a job array and one Slurm job submission script can be defined to analyze the result from the series of jobs. An automated script will help you to submit the calculation job array and submit the analysis as dependency job. In this way, your calculation and analysis can be done automatically which can save your time.

Automation Help You to Be More Productive and Efficient

As the boundary of science keeps expanding, it is fairly usual that there are Job Y which depend on Job X, a series of large amounts of calculations or large amounts of data analysis. Utilizing the features of Slurm can help you to manage your HPC jobs more efficiently and less time consuming.