Single core

Programs that do not use any parallelisation, which includes most programs, are known as serial or sequential programs. They only use one CPU core at a time, so many instances can run at the same time on one of Hamilton's multi-core compute nodes.

An example job script to run a program called my_serial_program would be:

#!/bin/bash
 
# Request resources:
#SBATCH -c 1           # 1 CPU core
#SBATCH --mem=1G       # memory required, up to 250G on standard nodes.
#SBATCH --time=1:0:0   # time limit for job (format:  days-hours:minutes:seconds)
#SBATCH --gres=tmp:1G  # temporary disk space required on the compute node ($TMPDIR),
#                        up to 400G
# Run in the 'shared' queue (job may share node with other jobs)
#SBATCH -p shared
 
# Commands to be run:
module load my_module
./my_serial_program

If saved in a file called my_serial_job.sh, this could be submitted to the queue with the command:

sbatch my_serial_job.sh

Shared memory

Some programs can use more than one CPU core at a time, but are limited to a single compute node. They typically use programming techniques such as OpenMP or threading to achieve this. We call them shared memory programs, because the parallelisation requires that all CPU cores have access to the same RAM/memory.

An example job script to run a program called my_sharedmemory_program would be:

#!/bin/bash
 
# Request resources:
#SBATCH -c 2          # number of CPU cores, one per thread, up to 128
#SBATCH --mem=1G      # memory required, up to 250G on standard nodes
#SBATCH --time=1:0:0  # time limit for job (format:  days-hours:minutes:seconds)
#SBATCH --gres=tmp:1G # temporary disk space required on the compute node ($TMPDIR), 
#                       up to 400G
# Run in the 'shared' queue (job may share node with other jobs)
#SBATCH -p shared
 
# Commands to be run:
module load my_module
./my_sharedmemory_program

If saved in a file called my_shared_job.sh, this can be submitted to the queue with the command:

sbatch my_shared_job.sh

Distributed memory

Programs can be written to take advantage of CPU cores and memory spread across multiple compute nodes. They typically use the low-level library called MPI (Message Passing Interface) to allow communication between many copies of the same program, each with access to its own CPU core and memory. We call this a distributed memory programming model.

An example job script to run an MPI program called my_mpi_program would be:

#!/bin/bash
 
# Request resources:
#SBATCH -n 1           # number of MPI ranks (1 per CPU core)
#SBATCH --mem=1G       # memory required per node, in units M, G or T
#SBATCH --time=1:0:0   # time limit for job (format:  days-hours:minutes:seconds)
#SBATCH --gres=tmp:1G  # temporary disk space required on each compute node ($TMPDIR)
#SBATCH -N 1           # number of compute nodes.  
 
# Smaller jobs can run in the shared queue.  
# Larger jobs that will occupy one or more whole nodes should use the multi queue.
#SBATCH -p shared 
 
# Commands to be run.  
# Note that mpirun will automatically launch the number of ranks specified above 
module load my_module
mpirun ./my_mpi_program

If saved in a file called my_dist_job.sh, this can be submitted to the queue with the command:

sbatch my_dist_job.sh

Note that it is not normally necessary to specify the number of tasks/ranks in the mpirun command; this will be matched to the number allocated in the line #SBATCH -n.

Hybrid

Using a mixed MPI / OpenMP model can have benefits, for example, to reduce the memory and computation dedicated to halo exchanges between different processes in grid-based codes.  Codes written in this hybrid way can run on multiple CPU cores across one or more nodes.

For these codes we recommend running one MPI rank per CPU socket (two MPI ranks per compute node on Hamilton). An example job script would be:

#!/bin/bash
 
# Request resources:
#SBATCH -n 1                    # number of MPI ranks
#SBATCH -c 1                    # number of threads per rank (one thread per CPU core)
#SBATCH --ntasks-per-socket=1   # number of MPI ranks per CPU socket
#SBATCH -N 1                    # number of compute nodes. 
#SBATCH --mem=1G                # memory required per node, in units M, G or T
#SBATCH --gres=tmp:1G           # temporary disk space on each compute node ($TMPDIR)
#SBATCH -t 1:0:0                # time limit for job (format: days-hours:minutes:seconds) 
 
# Smaller jobs can run in the shared queue. 
# Larger jobs that will occupy one or more whole nodes should use hte multi queue.
#SBATCH -p shared 
 
# Commands to be run. 
# Note that mpirun will automatically launch the number of ranks specified above 
module load my_module o provide the CPUs requested)  
mpirun ./my_hybrid_program

If saved in a file called my_hybrid_job.sh, this can be submitted to the queue with the command:

sbatch my_hybrid_job.sh

High memory

Jobs that require >250GB memory (per node) should run in the bigmem queue. The nodes in this queue each have 1.95TB memory. An example job script my_bigmem_job.sh might be:

#!/bin/bash
 
# Request resources:
#SBATCH -c 1            # number of CPU cores, up to 128 for shared-memory programs
#SBATCH --mem=260G      # memory required, up to 1.95T
#SBATCH --time=1:0:0    # time limit for job (format:  days-hours:minutes:seconds)
#SBATCH --gres=tmp:1G   # temporary disk space required on the compute node ($TMPDIR),
#                         up to 400G
# Run in the bigmem queue (job may share node with other jobs)
#SBATCH -p bigmem
 
# Commands to be run:
module load my_module
./my_bigmem_program

If saved in a file called my_bigmem_job.sh, this can be submitted to the queue with the command:

sbatch my_bigmem_job.sh

Array

An array job allows you to submit a set of similar jobs from a single job script. For example, an array might perform the same analysis on multple datasets.  Array job scripts have an additional line beginning #SBATCH --array= that defines the array.  

Once an array job has been submitted, Slurm will expand it to a set of separate jobs, or array 'tasks'.  Any Slurm resource requests, such as the number of CPU cores required, apply to each task separately.

An environment variable, $SLURM_ARRAY_TASK_ID, can be used to differentiate the work done in different tasks.  

Example script:

#!/bin/bash

# Example array job script
 
#SBATCH -c 2          # number of CPU cores, one per thread, up to 128
#SBATCH --mem=1G      # memory required, up to 250G on standard nodes
#SBATCH --time=1:0:0  # time limit for job (format:  dd-hh:mm:ss)
#SBATCH --gres=tmp:1G # temporary disk space required on the compute 
#                       node (TMPDIR), up to 400G
#SBATCH -p shared     # Queue/partition
 
# This example has array indices from 1 to 1000 in steps of 2.
# Arrays can also have irregular indices.  
# Up to 10,000 tasks are permitted in any one array.
 
#SBATCH --array=1-1000:2
 
# Commands to be run: 
module load my_module
 
# Run my_program on a set of files called my_data_1 to my_data_1000
# that are held in separate directories.
cd $NOBACKUP/my_sub_dir_${SLURM_ARRAY_TASK_ID}
my_program -i my_data_${SLURM_ARRAY_TASK_ID}

If saved in a file called my_array_job.sh, this can be submitted to the queue with the command:

sbatch my_array_job.sh

Job dependencies are helpful to link array jobs together and form analysis pipelines for multiple datasets.   For example, the following example submits three job arrays, defined in files array1.sh, array2.sh and array3.sh.  Each task in array2.sh and array3.sh will only start after the corresponding task in the previous array has completed.  After all of array3.sh has completed, the non-array job defined in nonarray.sh will run.  A summary of common dependency options is given in the table below the script and there is more information about dependencies on the sbatch man page.

#!/bin/bash
  
# Example of a work pipeline using Slurm job dependencies

# To run this script, type "source <scriptname>"
 
# Submit the first array
jobid_1=$(sbatch array1.sh)
 
# Submit dependent arrays
jobid_2=$(sbatch --dependency=aftercorr:$jobid_1 array2.sh)
jobid_3=$(sbatch --dependency=aftercorr:$jobid_2 array3.sh)
 
# Submit non-array job.  See below for dependency types.
sbatch --dependency=afterok:$jobid_3 nonarray.sh

This script can be run interactively rather than submitted to Slurm.  If saved in a file submit_pipeline.sh, it could be run with the command:

source submit_pipeline.sh

Common dependency types 

Note that squeue does not display dependencies by default but they can be included with a command such as:

squeue --me --Format=JobArrayId,Name,Dependency:.40