Skip to content

Slurm Job Scheduler

CCI uses slurm to allocate resources for compute jobs. Slurm is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system for large and small Linux clusters. A full description of the software can be found at https://slurm.schedmd.com/overview.html . We use cons_tres as our resource selection algorithm.

All compute jobs must be submitted to the slurm queue and not directly run on the frontend node. Users running jobs on the frontend node will be locked out of the system.

Quick Reference

Commands

  • squeue lists your jobs in the queue
  • sinfo lists the state of all machines in the cluster
  • sbatch submits batch jobs
  • sprio lists the relative priorities of pending jobs in the queue and how they are calculated
  • sacct display accounting and submission data for jobs
  • scancel is used to cancel jobs
  • salloc allocates a node for interactive use

Resources

Watching Slurm commands

When using watch to monitor the output of a Slurm command, please make sure to set an update interval of at least 60 seconds, ex: watch -n 120 squeue. The output is unlikely to change any faster than this and updating too frequently can cause undue load on the scheduler.

Resource specification

Options of interest (see the manual page for sbatch for a complete list):

 -n, --ntasks=ntasks         number of tasks to run (-n 1 is a default and better left unspecified) 
 -N, --nodes=N               number of nodes on which to run (N = min[-max])
 -c, --cpus-per-task=ncpus   number of cpus required per task
     --ntasks-per-node=n     number of tasks to invoke on each node
     --cpus-per-gpu          
 -i, --input=in              file for batch script's standard input
 -o, --output=out            file for batch script's standard output
 -e, --error=err             file for batch script's standard error
 -p, --partition=partition   partition requested
 -t, --time=minutes          time you expect your job to compelete under
 -D, --chdir=path            change remote current working directory
 -D, --workdir=directory     set working directory for batch script
     --mail-type=type        notify on state change: BEGIN, END, FAIL or ALL
     --mail-user=user        who to send email notification for job state changes

Note that any of the above can be specified in a batch file by preceeding the option with #SBATCH. All options defined this way must appear first in the batch file with nothing separating them. For example, the following will send the job's output to a file called joboutput.<the job's ID>:

#SBATCH -o joboutput.%J

Example job submission scripts

See also: Modules for any additional options/requirements of specific MPI implementations. Typically, it is necessary to load the same modules at runtime (before calling srun) that were used when building a binary.

Simple (non-MPI)

A simple (non-MPI) job can be started by just calling srun:

#!/bin/bash -x
srun ./a.out your_application_name

For example, the above jobs could be submitted to run 16 tasks on 1 nodes, in the partition "cluster", with the current working directory set to /foo/bar, email notification of the job's state turned on, a time limit of four hours (240 minutes), and STDOUT redirected to /foo/bar/baz.out as follows (where script.sh is the script):

sbatch -p cluster -N 1 -n 16 --mail-type=ALL --mail-user=example@rpi.edu -t 240 -D /foo/bar -o /foo/bar/baz.out ./script.sh

Note: In a simple, non-MPI case, running multiple tasks will create multiple instances of the same binary.

Interactive

Interactive jobs are supported. See the srun command manual page for details. Here is a usage example launching a shell on the compute node allocated to an interactive session:

salloc -t 100 --gres=gpu:8 srun --pty bash -i

Or by an alternative method to allocate and connect seperatly:

salloc -t 100 --gres=gpu:8 
ssh "$SLURM_JOB_NODELIST"

OpenMPI

Open MPI & Slurm
Example job batch script slurmOpenMpi.sh: =======

IBM Spectrum MPI or Mellanox HPC-X

These implementations do not have direct Slurm support and it is necessary to use mpirun. You must have passwordless SSH keys setup for mpirun to work. If mpirun outputs ORTE was unable to reliably start one or more daemons. then you need to setup SSH keys.

Example job batch script slurmSpectrum.sh:

1
2
3
4
5
6
7
#!/bin/bash -x

module load spectrum-mpi
mpirun --bind-to core --report-bindings -np $SLURM_NPROCS /path/to/your/executable

#kokkos users should add: --kokkos-num-devices=N
#where N is the number of gpus being used on each node

GPU-Direct

To enable GPU-Direct ('CUDA aware MPI') pass the -gpu and -mca pml_pami_enable_gdrcpy 1 flags to mpirun, ex:

mpirun -gpu -mca pml_pami_enable_gdrcpy 1 /path/to/your/executable

Job arrays

Job arrays provide a way to submit collections of similar jobs. https://slurm.schedmd.com/job_array.html Two additional environment variables will be passed, SLURM_ARRAY_JOB_ID and SLURM_ARRAY_TASK_ID

#!/bin/bash
#SBATCH --job-name=array-example
#SBATCH --output=slurm-%A.%a.out
#SBATCH --error=slurm-%A.%a.err
#SBATCH --nodes=1
#SBATCH --ntasks=1               # total number of tasks across all nodes
#SBATCH --time=06:00:00
#SBATCH --array=0-6              # job array with index values 0, 1, 2, 3, 4, 5, 6

echo "SLURM_ARRAY_JOB_ID: $SLURM_ARRAY_JOB_ID."
echo "SLURM_ARRAY_TASK_ID: $SLURM_ARRAY_TASK_ID"
echo "Executing on the node:" $(hostname)

module load openmpi
./my-executable <options>

Many small tasks

For many small tasks running simultaneously or in quick succession, it is often better to submit one large job rather than many small jobs. On some systems, doing otherwise leads to resource fragmentation and poor scheduler performance. This is particularitly true on DCS and NPL if your program cannot fully utilize a single GPU.

Example: This example requests 1 node with 1 GPU and run 4 programs on that GPU at the same time.

sbatch.sh

#!/bin/sh
#SBATCH --job-name=test
#SBATCH -t 03:00:00
#SBATCH -N1
#SBATCH --gres=gpu:1

srun overload.sh &
wait

overload.sh

./my-executable <options> > testing1.log &
./my-executable <options> > testing2.log &
./my-executable <options> > testing3.log &
./my-executable <options> > testing4.log &
wait

Note the ampersand (&) at then end of each executable invocation and the wait command at the end. This will run all 4 jobs in parallel within the single allocation and wait until all 4 are complete.


Last update: June 20, 2022