Loading

Cgroups Guide

Cgroups Overview

For a comprehensive description of Linux Control Groups (cgroups) see the cgroups documentation at kernel.org. Detailed knowledge of cgroups is not required to use cgroups in SLURM, but a basic understanding of the following features of cgroups is helpful:

Use of Cgroups in SLURM

SLURM provides cgroup versions of a number of plugins.

SLURM Cgroups Configuration Overview

There are several sets of configuration options for SLURM cgroups:

Currently Available Cgroup Plugins

proctrack/cgroup plugin

The proctrack/cgroup plugin is an alternative to other proctrack plugins such as proctrack/linux for process tracking and suspend/resume capability. proctrack/cgroup uses the freezer subsystem which is more reliable for tracking and control than proctrack/linux.

To enable this plugin, configure the following option in slurm.conf:

ProctrackType=proctrack/cgroup

There are no specific options for this plugin in cgroup.conf, but the general options apply. See the cgroup.conf man page for details.

task/cgroup plugin

The task/cgroup plugin is an alternative other task plugins such as task/affinity plugin for task management. task/cgroup provides the following features: The task/cgroup plugin uses the cpuset, memory and devices subsystems.

To enable this plugin, configure the following option in slurm.conf:

TaskPlugin=task/cgroup

There are many specific options for this plugin in cgroup.conf. The general options also apply. See the cgroup.conf man page for details.

jobacct_gather/cgroup plugin

At present, jobacct_gather/cgroup should be considered experimental.

The jobacct_gather/cgroup plugin is an alternative to the jobacct_gather/linux plugin for the collection of accounting statistics for jobs, steps and tasks. The cgroup plugin may provide improved performance over jobacct_gather/linux. jobacct_gather/cgroup uses the cpuacct and memory subsystems. Note: the cpu and memory statistics collected by this plugin do not represent the same resources as the cpu and memory statistics collected by the jobacct_gather/linux plugin (sourced from /proc stat).

To enable this plugin, configure the following option in slurm.conf:

JobacctGatherType=jobacct_gather/cgroup

There are no specific options for this plugin in cgroup.conf, but the general options apply. See the cgroup.conf man page for details.

Organization of SLURM Cgroups

SLURM cgroups are organized as follows. A base directory (mount point) is created at /cgroup, or as configured by the CgroupMountpoint option in cgroup.conf. All cgroup hierarchies are created below this base directory. A separate hierarchy is created for each cgroup subsystem in use. The name of the root cgroup in each hierarchy is the subsystem name. A cgroup named slurm is created below the root cgroup in each hierarchy. Below each slurm cgroup, cgroups for SLURM users, jobs, steps and tasks are created dynamically as needed. The names of these cgroups consist of a prefix identifying the SLURM entity (user, job, step or task), followed by the relevant numeric id. The following example shows the path of the task cgroup in the cpuset hierarchy for taskid#2 of stepid#0 of jobid#123 for userid#100, using the default base directory (/cgroup):

/cgroup/cpuset/slurm/uid_100/job_123/step_0/task_2

Note that this structure applies to a specific compute node. Jobs that use more than one node will have a cgroup structure on each node.

Cleanup of SLURM Cgroups

Linux provides a mechanism for the automatic removal of a cgroup when its state changes from non-empty to empty. A cgroup is empty when no processes are attached to it and it has no child cgroups. The SLURM cgroups implementation allows this mechanism to be used to automatically remove the relevant SLURM cgroups when tasks, steps and jobs terminate. To enable this automatic removal feature, follow these steps:

Last modified 6 June 2012