This article describes basic Slurm usage. Brief “how-to” topics include, in this order:

  A simple Slurm job script
  Slurm logoSubmit the job
  List jobs
  Get job details
  Suspend a job (root only)
  Resume a job (root only)
  Kill a job
  Hold a job
  Release a job
  List partitions
  Submit a job that's dependant on a prerequisite job being completed

Simple Slurm job script:

$ cat my-slurm-job.sh

#!/bin/bash

# set the number of nodes #SBATCH –nodes=1

# set max wallclock time #SBATCH –time=100:00:00

# set name of job #SBATCH –job-name=MyTestjob5

# mail alert at start, end and abortion of execution #SBATCH –mail-type=ALL

# send mail to this address #SBATCH –mail-user=user@zamren.com

# run the application

echo “In the directory: `pwd`” echo “As the user: `whoami`” echo “write this is a file“ > analysis.output sleep 60

Submit the job:

$ sbatch slurm-job.sh

Submitted batch job 106

List jobs:

$ squeue

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
  106 defq      slurm-jo  rstober   R   0:04      1 atom01

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 319 AllNodes sim100M. omangete R 1:09:34 8 zm-node[004-011] Get job details:

$ scontrol show job 106 JobId=106 Name=slurm-job.sh

 UserId=rstober(1001) GroupId=rstober(1001)
 Priority=4294901717 Account=(null) QOS=normal
 JobState=RUNNING Reason=None Dependency=(null)
 Requeue=1 Restarts=0 BatchFlag=1 ExitCode=0:0
 RunTime=00:00:07 TimeLimit=UNLIMITED TimeMin=N/A
 SubmitTime=2013-01-26T12:55:02 EligibleTime=2013-01-26T12:55:02
 StartTime=2013-01-26T12:55:02 EndTime=Unknown
 PreemptTime=None SuspendTime=None SecsPreSuspend=0
 Partition=defq AllocNode:Sid=atom-head1:3526
 ReqNodeList=(null) ExcNodeList=(null)
 NodeList=atom01
 BatchHost=atom01
 NumNodes=1 NumCPUs=2 CPUs/Task=1 ReqS:C:T=*:*:*
 MinCPUsNode=1 MinMemoryNode=0 MinTmpDiskNode=0
 Features=(null) Gres=(null) Reservation=(null)
 Shared=0 Contiguous=0 Licenses=(null) Network=(null)
 Command=/home/rstober/slurm/local/slurm-job.sh
 WorkDir=/home/rstober/slurm/local

Suspend a job (root only):

# scontrol suspend 135 # squeue

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
  135  defq simple.s  rstober  S   0:10   1    atom01

Resume a job (root only):

# scontrol resume 135 # squeue

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
  135  defq simple.s  rstober  R   0:13   1    atom01

Kill a job. Users can kill their own jobs, root can kill any job.

$ scancel 135 $ squeue

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)

Hold a job:

$ squeue

JOBID PARTITION     NAME     USER  ST       TIME  NODES NODELIST(REASON)
  139      defq   simple  rstober  PD       0:00      1 (Dependency)
  138      defq   simple  rstober   R       0:16      1 atom01

$ scontrol hold 139 $ squeue

JOBID PARTITION     NAME     USER  ST       TIME  NODES NODELIST(REASON)
  139      defq   simple  rstober  PD       0:00      1 (JobHeldUser)
  138      defq   simple  rstober   R       0:32      1 atom01

Release a job:

$ scontrol release 139 $ squeue

JOBID PARTITION     NAME     USER  ST       TIME  NODES NODELIST(REASON)
  139      defq   simple  rstober  PD       0:00      1 (Dependency)
  138      defq   simple  rstober   R       0:46      1 atom01

List partitions:

$ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST defq* up infinite 1 down* atom04 defq* up infinite 3 idle atom[01-03] cloud up infinite 2 down* cnode1,cnodegpu1 cloudtran up infinite 1 idle atom-head1

Submit a job that's dependant on a prerequisite job being completed:

Here's a simple job script. Note that the Slurm -J option is used to give the job a name.

#!/usr/bin/env bash

#SBATCH -p defq #SBATCH -J simple

sleep 60

Submit the job

$ sbatch simple.sh Submitted batch job 149

Now we'll submit another job that's dependent on the previous job. There are many ways to specify the dependency conditions, but the “singleton” is the simplest. The Slurm -d singleton argument tells Slurm not to dispatch this job until all previous jobs with the same name have completed.

$ sbatch -d singleton simple.sh Submitted batch job 150 $ squeue

JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
  150 defq   simple  rstober  PD  0:00  1 (Dependency)
  149 defq   simple  rstober   R  0:17  1 atom01

Once the prerequisite job finishes the dependent job is dispatched.

$ squeue

JOBID PARTITION NAME USER ST TIME  NODES NODELIST(REASON)
  150 defq   simple  rstober   R   0:31  1 atom01