Slurm Partitions and Queues

Partition Details

Partition Attributes

• Partition: The name of the queue or logical grouping of nodes. Specify with –partition (or -p) in sbatch/srun.
• Nodes: A list or range of compute node hostnames in that partition (e.g. gpu[01-04] expands to gpu01 gpu02 gpu03 gpu04).
• #Nodes: The total count of individual nodes covered by the Nodes field.
• Max Time: The maximum wall‑clock time a job may run in this partition. Slurm will cap jobs beyond this limit.
• AllowGroups: Unix group(s) permitted to submit jobs here. ALL means no restriction; otherwise list specific group(s).
• Default?: YES if this is the default partition when none is specified; NO otherwise.
• QoS: The Slurm Quality‑of‑Service level applied to jobs in this partition (affects priority and fairshare).
• DefaultTime: The time Slurm applies if the user omits –time (prevents runaway jobs; typically 01:00:00).
• PreemptMode: What Slurm does when higher‑priority work needs these nodes. REQUEUE returns preempted jobs to the queue.

There are five queues, or partitions of nodes, that can be used for submitting jobs.

• The cmp partition is the default partition if none is specified and contains all the cpu and mem nodes (i.e. all nodes without a GPU processor). This partition should be used for all compute intensive tasks that do not require a GPU processor.

• The gpu partition contains all GPU programming capable nodes (384GB RAM, 2x NVIDIA V100 GPUs) and should be used if you require a GPU processor. Note you will also need to explicitly request a GPU processor as explained in Requesting GPU Resources. 

• The mem partition contains all high-memory nodes (2TB RAM, no GPU) and should be used if you require the extra memory.

• The cpu partition contains a set of compute related nodes (512GB RAM, no GPU).

• The amd partition contains a single (256GB RAM, 8x MI100 GPU) node with a GPU that has AMD's CDNA architecture, as opposed to NVIDIA's CUDA.

• The all partition is comprised of all the nodes in the cpu, gpu, and mem partitions. It is there to allow users to request parallelism across the breadth of the cluster. Note that the all partition has a lower time limit to restrict the ability to schedule all the nodes for an extend period of time.

• The condo partition is composed of a heterogeneous set of nodes purchased by individual faculty members. You are welcome to run jobs on this partition when the nodes are available, but be aware that your job may be terminated if someone from the faculty member's lab requests the resources.

It is important to specify which partitions you need and your necessary resources as it will provide you with the best scheduling availability for your task.

The Slurm sinfo command provides information about the nodes and partitions. For example, if you were able to execute sinfo on an idle cluster you might see the following:

PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST

all          up 1-00:00:00     10  idle cpu[01-04],gpu[01-04],mem[01-02]

amd          up 2-00:00:00      1  idle amd01

cmp*         up 4-04:00:00      6  idle cpu[01-04],mem[01-02]

cpu          up 4-04:00:00      4  idle cpu[01-04]

gpu          up 2-00:00:00      4  idle gpu[01-04]

mem          up 4-04:00:00      2  idle mem[01-02]

condo        up 2-00:00:00      9  idle bio[01-03],dmlab01,oignat01,rgrotjahn[01-04]

Note that each queue or partition has different limits on both the number of nodes and the maximum time they can be reserved for. See for more information.

A user can specify the partition using the –partition parameter in either the srun or sbatch commands. 

Use scontrol to see the details on individual nodes, which is especially useful for the heterogeneous condo queue. The CfgTRES line will tell you the number of CPUs and GPUs and amount of RAM available on the whole node. For example, gpu01 has 80 CPUs, 368 GB of RAM, and 2 GPUs.

Where as the following will schedule a job for 30 minutes of processing.

Where as the following will schedule a job for 30 minutes of processing.

There is a maximum time for which a given partition of nodes can be reserved. This varies based on the partition. The sinfo command will display the maximum possible time allowed per partition. If you try to schedule a job for more than the maximum time allowed, the scheduler will override your request.