Slurm Job Scheduler
CCI uses slurm to allocate resources for compute jobs. Slurm is an open source, fault-tolerant, and highly scalable cluster management and job scheduling system for large and small Linux clusters. A full description of the software can be found at https://slurm.schedmd.com/overview.html . We use cons_tres as our resource selection algorithm.
All compute jobs must be submitted to the slurm queue and not directly run on the frontend node. Users running jobs on the frontend node will be locked out of the system.
- squeue lists your jobs in the queue
- sinfo lists the state of all machines in the cluster
- sbatch submits batch jobs
- sprio lists the relative priorities of pending jobs in the queue and how they are calculated
- sacct display accounting and submission data for jobs
- scancel is used to cancel jobs
- salloc allocates a node for interactive use
- Slurm Documentation: https://slurm.schedmd.com/archive/slurm-20.11.8/
- Slurm FAQ in the official Slurm documentation: http://www.schedmd.com/slurmdocs/faq.html
Watching Slurm commands¶
watch to monitor the output of a Slurm command, please make sure
to set an update interval of at least 60 seconds, ex:
watch -n 120 squeue.
The output is unlikely to change any faster than this and updating too
frequently can cause undue load on the scheduler.
Options of interest (see the manual page for sbatch for a complete list):
-n, --ntasks=ntasks number of tasks to run (-n 1 is a default and better left unspecified) -N, --nodes=N number of nodes on which to run (N = min[-max]) -c, --cpus-per-task=ncpus number of cpus required per task --ntasks-per-node=n number of tasks to invoke on each node --cpus-per-gpu -i, --input=in file for batch script's standard input -o, --output=out file for batch script's standard output -e, --error=err file for batch script's standard error -p, --partition=partition partition requested -t, --time=minutes time you expect your job to compelete under -D, --chdir=path change remote current working directory -D, --workdir=directory set working directory for batch script --mail-type=type notify on state change: BEGIN, END, FAIL or ALL --mail-user=user who to send email notification for job state changes
Note that any of the above can be specified in a batch file by preceeding the option with #SBATCH. All options defined this way must appear first in the batch file with nothing separating them. For example, the following will send the job's output to a file called joboutput.<the job's ID>:
#SBATCH -o joboutput.%J
Example job submission scripts¶
See also: Modules for any additional options/requirements of specific MPI implementations. Typically, it is necessary to load the same modules at runtime (before calling srun) that were used when building a binary.
A simple (non-MPI) job can be started by just calling srun:
#!/bin/bash -x srun ./a.out your_application_name
For example, the above jobs could be submitted to run 16 tasks on 1 nodes, in the partition "cluster", with the current working directory set to /foo/bar, email notification of the job's state turned on, a time limit of four hours (240 minutes), and STDOUT redirected to /foo/bar/baz.out as follows (where script.sh is the script):
sbatch -p cluster -N 1 -n 16 --mail-type=ALL --firstname.lastname@example.org -t 240 -D /foo/bar -o /foo/bar/baz.out ./script.sh
Note: In a simple, non-MPI case, running multiple tasks will create multiple instances of the same binary.
Interactive jobs are supported. See the srun command manual page for details. Here is a usage example launching a shell on the compute node allocated to an interactive session:
salloc -t 100 --gres=gpu:8 srun --pty bash -i
Or by an alternative method to allocate and connect seperatly:
salloc -t 100 --gres=gpu:8 ssh "$SLURM_JOB_NODELIST"
Open MPI & Slurm
Example job batch script
IBM Spectrum MPI or Mellanox HPC-X¶
These implementations do not have direct Slurm support and it is
necessary to use mpirun. You must have passwordless SSH keys setup for
mpirun to work. If mpirun outputs
ORTE was unable to reliably start one or more daemons. then you need
to setup SSH keys.
Example job batch script
1 2 3 4 5 6 7
To enable GPU-Direct ('CUDA aware MPI') pass the
-mca pml_pami_enable_gdrcpy 1 flags to mpirun, ex:
mpirun -gpu -mca pml_pami_enable_gdrcpy 1 /path/to/your/executable
Job arrays provide a way to submit collections of similar jobs. https://slurm.schedmd.com/job_array.html Two additional environment variables will be passed, SLURM_ARRAY_JOB_ID and SLURM_ARRAY_TASK_ID
#!/bin/bash #SBATCH --job-name=array-example #SBATCH --output=slurm-%A.%a.out #SBATCH --error=slurm-%A.%a.err #SBATCH --nodes=1 #SBATCH --ntasks=1 # total number of tasks across all nodes #SBATCH --time=06:00:00 #SBATCH --array=0-6 # job array with index values 0, 1, 2, 3, 4, 5, 6 echo "SLURM_ARRAY_JOB_ID: $SLURM_ARRAY_JOB_ID." echo "SLURM_ARRAY_TASK_ID: $SLURM_ARRAY_TASK_ID" echo "Executing on the node:" $(hostname) module load openmpi ./my-executable <options>
Many small tasks¶
For many small tasks running simultaneously or in quick succession, it is often better to submit one large job rather than many small jobs. On some systems, doing otherwise leads to resource fragmentation and poor scheduler performance. This is particularitly true on DCS and NPL if your program cannot fully utilize a single GPU.
Example: This example requests 1 node with 1 GPU and run 4 programs on that GPU at the same time.
#!/bin/sh #SBATCH --job-name=test #SBATCH -t 03:00:00 #SBATCH -N1 #SBATCH --gres=gpu:1 srun overload.sh & wait
./my-executable <options> > testing1.log & ./my-executable <options> > testing2.log & ./my-executable <options> > testing3.log & ./my-executable <options> > testing4.log & wait
Note the ampersand (&) at then end of each executable invocation and the
wait command at the end. This will run all 4 jobs
in parallel within the single allocation and wait until all 4 are complete.