Loading

Accounting and Resource Limits

SLURM can be configured to collect accounting information for every job and job step executed. Accounting records can be written to a simple text file or a database. Information is available about both currently executing jobs and jobs which have already terminated. The sacct command can report resource usage for running or terminated jobs including individual tasks, which can be useful to detect load imbalance between the tasks. The sstat command can be used to status only currently running jobs. It also can give you valuable information about imbalance between tasks. The sreport can be used to generate reports based upon all jobs executed in a particular time interval.

There are three distinct plugin types associated with resource accounting. The SLURM configuration parameters (in slurm.conf) associated with these plugins include:

The use of sacct to view information about jobs is dependent upon AccountingStorageType being configured to collect and store that information. The use of sreport is dependent upon some database being used to store that information.

The use of sacct or sstat to view information about resource usage within jobs is dependent upon both JobAcctGatherType and AccountingStorageType being configured to collect and store that information.

Storing the accounting information into text files is very simple. Just configure the appropriate plugin (e.g. AccountingStorageType=accounting_storage/filetxt and/or JobCompType=jobcomp/filetxt) and then specify the pathname of the file (e.g. AccountingStorageLoc=/var/log/slurm/accounting and/or JobCompLoc=/var/log/slurm/job_completions). Use the logrotate or similar tool to prevent the log files from getting too large. Send a SIGHUP signal to the slurmctld daemon after moving the files, but before compressing them so that new log files will be created.

Storing the data directly into a database from SLURM may seem attractive, but requires the availability of user name and password data not only for the SLURM control daemon (slurmctld), but also user commands which need to access the data (sacct, sreport, and sacctmgr). Making possibly sensitive information available to all users makes database security more difficult to provide, sending the data through an intermediate daemon can provide better security and performance (through caching data) and SlurmDBD provides such services. SlurmDBD (SLURM Database Daemon) is written in C, multi-threaded, secure and fast. The configuration required to use SlurmDBD will be described below. Storing information directly into database would be similar.

Note that SlurmDBD relies upon existing SLURM plugins for authentication and SLURM sql for database use, but the other SLURM commands and daemons are not required on the host where SlurmDBD is installed. Install the slurmdbd, slurm-plugins, and slurm-sql RPMs on the computer when SlurmDBD is to execute. If you want munge authentication, which is highly recommended, you will also need to install the slurm-munge RPM.

If SlurmDBD is configured for use but not responding then slurmctld will utilize an interal cache until SlurmDBD is returned to service. The cached data is written by slurmctld to local storage upon shutdown and recovered at startup. If SlurmDBD is not available when slurmctld starts, a cache of valid bank accounts, user limits, etc. based upon their state when the daemons were last communicating will be used. Note that SlurmDBD must be responding when slurmctld is first started since no cache of this critical data will be available. Job and step accounting records generated by slurmctld will be written to a cache as needed and transfered to SlurmDBD when returned to service.

Infrastructure

With the SlurmDBD, we are able to collect data from multiple clusters in a single location. This does impose some constraints on the user naming and IDs. Accounting is maintained by user name (not user ID), but a given user name should refer to the same person across all of the computers. Authentication relies upon user ID numbers, so those must be uniform across all computers communicating with each SlurmDBD, at least for users requiring authentication. In particular, the configured SlurmUser must have the same name and ID across all clusters. If you plan to have administrators of user accounts, limits, etc. they must also have consistent names and IDs across all clusters. If you plan to restrict access to accounting records (e.g. only permit a user to view records of his jobs), then all users should have consistent names and IDs.

The best way to insure security of the data is by authenticating communications to the SlurmDBD and we recommend MUNGE for that purpose. If you have one cluster managed by SLURM and execute the SlurmDBD on that one cluster, the normal MUNGE configuration will suffice. Otherwise MUNGE should then be installed on all nodes of all SLURM managed clusters, plus the machine where SlurmDBD executes. You then have a choice of either having a single MUNGE key for all of these computers or maintaining a unique key for each of the clusters plus a second key for communications between the clusters for better security. MUNGE enhancements are planned to support two keys within a single configuration file, but presently two different daemons must be started with different configurations to support two different keys (create two key files and start the daemons with the --key-file option to locate the proper key plus the --socket option to specify distinct local domain sockets for each). The pathname of local domain socket will be needed in the SLURM and SlurmDBD configuration files (slurm.conf and slurmdbd.conf respectively, more details are provided below).

Whether you use any authentication module or not you will need to have a way for the SlurmDBD to get uid's for users and/or admin. If using MUNGE, it is ideal for your users to have the same id on all your clusters. If this is the case you should have a combination of every clusters /etc/passwd file on the database server to allow the DBD to resolve names for authentication. If using MUNGE and a users name is not in the passwd file the action will fail. If not using MUNGE, you should add anyone you want to be an administrator or operator to the passwd file. If they plan on running sacctmgr or any of the accounting tools they should have the same uid, or they will not authentic correctly. An LDAP server could also server as a way to gather this information.

Slurm JobComp Configuration

Presently job completion is not supported with the SlurmDBD, but can be written directly to a database, script or flat file. If you are running with the accounting storage, you may not need to run this since it contains much of the same information. If you would like to configure this, some of the more important parameters include:

SLURM Accounting Configuration Before Build

While the SlurmDBD will work with a flat text file for recording job completions and such this configuration will not allow "associations" between a user and account. A database allows such a configuration.

MySQL is the preferred database, PostgreSQL is supported for job and step accounting only. The infrastructure for PostgresSQL for use with associations is not yet supported, meaning sacctmgr will not work correctly. If interested in adding this capability for PostgresSQL, please contact us at slurm-dev@schedmd.com.

To enable this database support one only needs to have the development package for the database they wish to use on the system. The slurm configure script uses mysql_config and pg-config to find out the information it needs about installed libraries and headers. You can specify where your mysql_config script is with the --with-mysql_conf=/path/to/mysql_config option when configuring your slurm build. A similar option is also available for PostgreSQL. On a successful configure, output is something like this: 

checking for mysql_config... /usr/bin/mysql_config
MySQL test program built properly.

SLURM Accounting Configuration After Build

For simplicity sake we are going to reference everything as if you are running with the SlurmDBD. You can communicate with a storage plugin directly, but that offers minimal security.

Several SLURM configuration parameters must be set to support archiving information in SlurmDBD. SlurmDBD has a separate configuration file which is documented in a separate section. Note that you can write accounting information to SlurmDBD while job completion records are written to a text file or not maintained at all. If you don't set the configuration parameters that begin with "AccountingStorage" then accounting information will not be referenced or recorded.

SlurmDBD Configuration

SlurmDBD requires its own configuration file called "slurmdbd.conf". This file should be only on the computer where SlurmDBD executes and should only be readable by the user which executes SlurmDBD (e.g. "slurm"). This file should be protected from unauthorized access since it contains a database login name and password. See "man slurmdbd.conf" for a more complete description of the configuration parameters. Some of the more important parameters include:

MySQL Configuration

While SLURM will create the database automatically you will need to make sure the StorageUser is given permissions in MySQL to do so. As the mysql user grant privileges to that user using a command such as:

GRANT ALL ON StorageLoc.* TO 'StorageUser'@'StorageHost';
(The ticks are needed)

(You need to be root to do this. Also in the info for password usage there is a line that starts with '->'. This a continuation prompt since the previous mysql statement did not end with a ';'. It assumes that you wish to input more info.)

Live example:

mysql@snowflake:~$ mysql
Welcome to the MySQL monitor.Commands end with ; or \g.
Your MySQL connection id is 538
Server version: 5.0.51a-3ubuntu5.1 (Ubuntu)

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

mysql> grant all on slurm_acct_db.* TO 'slurm'@'localhost';
Query OK, 0 rows affected (0.00 sec)

You may also need to do the same with the system name in order for mysql to work correctly:

mysql> grant all on slurm_acct_db.* TO 'slurm'@'system0';
Query OK, 0 rows affected (0.00 sec)
where 'system0' is the localhost or database storage host.

or with a password...

mysql> grant all on slurm_acct_db.* TO 'slurm'@'localhost'
    -> identified by 'some_pass' with grant option;
Query OK, 0 rows affected (0.00 sec)

The same is true in the case, you made to do the same with the system name:

mysql> grant all on slurm_acct_db.* TO 'slurm'@'system0'
    -> identified by 'some_pass' with grant option;
where 'system0' is the localhost or database storage host.

This will grant user 'slurm' access to do what it needs to do on the local host or the storage host system. This must be done before the SlurmDBD will work properly. After you grant permission to the user 'slurm' in mysql then you can start SlurmDBD and the other SLURM daemons. You start SlurmDBD by typing its pathname '/usr/sbin/slurmdbd' or '/etc/init.d/slurmdbd start'. You can verify that SlurmDBD is running by typing 'ps aux | grep slurmdbd'. After the SlurmDBD and slurmctld daemons start, you can verify that the database was created by using the mysql command 'show databases;'. You can display the tables that slurm created in the database by using the mysql command 'use slurm_acct_db;' and then 'show tables;'.

Use the mysql 'show databases;' command

mysql> show databases;

+--------------------+
| Database           |
+--------------------+
| information_schema |
| slurm_acct_db      |
| test               |
+--------------------+

3 rows in set (0.00 sec)

Select the database that you created.

mysql> use slurm_acct_db;

Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed

Now do a mysql 'show tables;' command.

mysql> show tables;

+---------------------------+
| Tables_in_slurm_acct_db   |
+---------------------------+
| acct_coord_table          |
| acct_table                |
| assoc_day_usage_table     |
| assoc_hour_usage_table    |
| assoc_month_usage_table   |
| assoc_table               |
| cluster_day_usage_table   |
| cluster_event_table       |
| cluster_hour_usage_table  |
| cluster_month_usage_table |
| cluster_table             |
| job_table                 |
| last_ran_table            |
| qos_table                 |
| resv_table                |
| step_table                |
| suspend_table             |
| table_defs_table          |
| txn_table                 |
| user_table                |
| wckey_day_usage_table     |
| wckey_hour_usage_table    |
| wckey_month_usage_table   |
| wckey_table               |
+---------------------------+

24 rows in set (0.02 sec)

mysql> quit

If the database is not created or SlurmDBD is not running you can use the -v option when you start SlurmDBD to get more detailed information.

Tools

There are a few tools available to work with accounting data, sacct, sacctmgr, and sreport. These tools all get or set data through the SlurmDBD daemon.

See the man pages for each command for more information.

Web interfaces with graphical output is currently under development and should be available in the Fall of 2009. A tool to report node state information is also under development.

Database Configuration

Accounting records are maintained based upon what we refer to as an Association, which consists of four elements: cluster, account, user names and an optional partition name. Use the sacctmgr command to create and manage these records. There is an order to set up accounting associations. You must define clusters before you add accounts and you must add accounts before you can add users.

For example, to add a cluster named "snowflake" to the database execute this line:

sacctmgr add cluster snowflake

Add accounts "none" and "test" to cluster "snowflake" with an execute line of this sort:

sacctmgr add account none,test Cluster=snowflake \
  Description="none" Organization="none"

If you have more clusters you want to add these accounts, to you can either not specify a cluster, which will add the accounts to all clusters in the system, or comma separate the cluster names you want to add to in the cluster option. Note that multiple accounts can be added at the same time by comma separating the names. Some description of the account and the organization which it belongs must be specified. These terms can be used later to generated accounting reports. Accounts may be arranged in a hierarchical fashion, for example accounts chemistry and physics may be children of the account science. The hierarchy may have an arbitrary depth. Just specify the parent='' option in the add account line to construct the hierarchy. For the example above execute

sacctmgr add account science \
 Description="science accounts" Organization=science
sacctmgr add account chemistry,physics parent=science \
 Description="physical sciences" Organization=science

Add users to accounts using similar syntax. For example, to permit user da to execute jobs on all clusters with a default account of test execute:

sacctmgr add user da DefaultAccount=test

If AccountingStorageEnforce=associations is configured in the slurm.conf of the cluster snowflake then user da would be allowed to run in account test and any other accounts added in the future. Any attempt to use other accounts will result in the job being aborted. Account test will be the default if he doesn't specify one in the job submission command.

Partition names can also be added to an "add user" command with the Partition='partitionname' option to specify an association specific to a slurm partition.

Cluster Options

When either adding or modifying a cluster, these are the options available with sacctmgr:

Account Options

When either adding or modifying an account, the following sacctmgr options are available:

User Options

When either adding or modifying a user, the following sacctmgr options are available:

Limit enforcement

When limits are developed they will work in this order. If a user has a limit set SLURM will read in those, if not we will refer to the account associated with the job. If the account doesn't have the limit set we will refer to the cluster's limits. If the cluster doesn't have the limit set no limit will be enforced.

All of the above entities can include limits as described below and in the Resource Limits document.

Modifying Entities

When modifying entities, you can specify many different options in SQL-like fashion, using key words like where and set. A typical execute line has the following form: 

sacctmgr modify <entity> set <options> where <options>

For example:

sacctmgr modify user set default=none where default=test

will change all users with a default account of "test" to account "none". Once an entity has been added, modified or removed, the change is sent to the appropriate SLURM daemons and will be available for use instantly.

Removing Entities

Removing entities using an execute line similar to the modify example above, but without the set options. For example, remove all users with a default account "test" using the following execute line:

sacctmgr remove user where default=test

Note: In most cases, removed entities are preserved, but flagged as deleted. If an entity has existed for less than 1 day, the entity will be removed completely. This is meant to clean up after typographic errors.

Last modified 3 February 2012