Slurm Generic Resource (GRES) Plugin API
Overview
This document describes Slurm generic resource (GRES) plugins and the API that defines them. It is intended as a resource to programmers wishing to write their own Slurm GRES plugins.
Slurm GRES plugins must conform to the Slurm Plugin API with the following specifications:
const char *plugin_type="major/minor"major must be gres. minor can be any suitable name representing the GRES type of the plugin.
const char *plugin_nameSome descriptive name for the plugin. There is no requirement with respect to its format.
const uint32_t plugin_versionIf specified, identifies the version of Slurm used to build this plugin and any attempt to load the plugin from a different version of Slurm will result in an error. If not specified, then the plugin may be loaded by Slurm commands and daemons from any version; however, this may result in difficult to diagnose failures due to changes in the arguments to plugin functions or changes in other Slurm functions used by the plugin.
We include samples in the Slurm distribution for:
- gpu — Manage GPUs (Graphics Processing Units).
- mps — Manage MPS (CUDA Multi-Process Service).
API Functions
All of the following functions are required. Functions which are not implemented must be stubbed.
int init(void)
Description:
Called when the plugin is loaded, before any other functions are
called. Put global initialization here.
Arguments: None.
Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.
void fini(void)
Description:
Called when the plugin is removed. Clear any allocated storage here.
Arguments: None.
Returns: None.
Note: init() and fini() are not the same as those described in the dlopen(3) system library. The C run-time system co-opts those symbols for its own initialization. The system _init() is called before the Slurm init(), and the Slurm fini() is called before the system's _fini().
int node_config_load(List gres_conf_list, node_config_load_t *config)
Description:
This function is called by the slurmd daemon after the slurm.conf
and gres.conf files have been read.
It can be used to validate or infer the system configuration by testing the
actual hardware resources available or just confirm that an entry for the
resource was included in the gres.conf file.
Arguments:
gres_conf_list
(input/output) a list of configuration records generated by reading the
slurm.conf and gres.conf files
config
(input) Additional data. Contains fields
cpu_cnt and
xcpuinfo_mac_to_abs.
Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.
void job_set_env(char ***job_env_ptr, void *gres_ptr, int node_inx)
Description:
This function is called by the slurmd daemon after the getting a job
credential and can be used to set environment variables for the job based
upon GRES state information in that credential.
Arguments:
job_env_ptr
(input/output) pointer to the job's environment variable structure.
gres_ptr
(input) pointer to the job's GRES allocation information.
node_inx
(input) zero origin node index, used to interpret node-specific GRES data.
Returns: None.
void step_set_env(char ***job_env_ptr, void *gres_ptr, uint32_t flags)
Description:
This function is called by the slurmd daemon after the getting a job
step credential and can be used to set environment variables for the job step
based upon GRES state information in that credential.
Arguments:
job_env_ptr
(input/output) pointer to the job step's environment variable structure.
gres_ptr
(input) pointer to the step's GRES allocation information.
flags
(input) Various flags to alter behavior. Currently, only a verbose flag is used
to print verbose GRES binding information to stderr for each task.
Returns: None.
void send_stepd(int fd)
Description:
This function is called by the slurmd daemon to send any needed
information to the slurmstepd step shepherd.
Arguments:
fd
(input) file descriptor to write information to.
Returns: None.
void recv_stepd(int fd)
Description:
This function is called by the slurmstepd step shepherd to read any
needed information from the slurmd daemon.
Arguments:
fd
(input) file descriptor to read information from.
Returns: None.
int job_info(gres_job_state_t *job_gres_data, uint32_t node_inx, enum gres_job_data_type data_type, void *data)
Description:
This function is used to extract plugin-specific data from the job's GRES
data structure. Note that
enum gres_job_data_type values
GRES_JOB_DATA_COUNT and
GRES_JOB_DATA_BITMAP are processed in common
code rather than within the plugin and return data types of
uint32_t* and
bitstr_t**, respectively.
Arguments:
job_gres_data
(input) Information about the job's GRES resources.
node_inx
(input) Zero origin index within the job's resource allocation for which
data is desired.
data_type
(input) Type of information to be gathered from the data structure.
data
(output) Pointer to data within job_gres_data.
No data is copied or needs to be freed.
Data type depends upon the value of
data_type.
Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.
int step_info(gres_step_state_t *step_gres_data, uint32_t node_inx, enum gres_step_data_type data_type, void *data)
Description:
This function is used to extract plugin-specific data from the step's GRES
data structure. Note that
enum gres_job_data_type values
GRES_JOB_DATA_COUNT and
GRES_JOB_DATA_BITMAP are processed in common
code rather than within the plugin and return data types of
uint32_t* and
bitstr_t**, respectively.
Arguments:
step_gres_data
(input) Information about the step's GRES resources.
node_inx
(input) Zero origin index within the job's resource allocation for which
data is desired.
data_type
(input) Type of information to be gathered from the data structure.
data
(output) Pointer to data within step_gres_data.
No data is copied or needs to be freed.
Data type depends upon the value of data_type.
Returns:
SLURM_SUCCESS on success, or
SLURM_ERROR on failure.
List get_devices(void)
Description:
This function returns the list of GRES devices.
Arguments: None.
Returns:
Returns a List of GRES device records of type
gres_slurmd_conf_t.
void step_hardware_init(bitstr_t *usable_gres, char *settings)
Description:
Configure device hardware corresponding to all the GRES devices of the plugin
type. The slurmstepd calls this function while privileged and before
tasks are forked and executed. The
gres/gpu plugin sets GPU frequencies
here.
Arguments:
usable_gres
(input) A bit string specifying all GRES devices of the plugin type allocated
to the step.
settings
(input) A string containing device hardware settings to be set for all specified
hardware devices.
Returns: None.
void step_hardware_fini(void)
Description:
Do hardware configuration after the step is finished while privileged. This is
meant to allow Slurm to undo hardware configuration changes performed by
step_hardware_init().
The slurmstepd calls this function while privileged and after tasks
complete. The gres/gpu plugin resets GPU frequencies to high here.
Arguments: None.
Returns: None.
gres_epilog_info_t *epilog_build_env(gres_job_state_t *gres_job_ptr)
Description:
Given a job's GRES allocation data, translated that to the data required by
epilog_set_env() to set environment variables
for the Prolog and Epilog programs.
Arguments:
gres_job_ptr
(input) job's GRES allocation data.
Returns: Data structure containing the information required by epilog_set_env() to set environment variables for the Prolog and Epilog programs.
void epilog_set_env(char ***epilog_env_ptr, gres_epilog_info_t *epilog_info, int node_inx)
Description:
Set GRES specific environment variables for the Prolog and
Epilog programs.
Arguments:
epilog_env_ptr
(input) environment variables set for the Prolog and Epilog
programs. This array may be reallocated as needed to contain additional
environment variables.
epilog_info
(input) GRES specific job allocation information. Built by
epilog_build_env().
node_inx
(input) zero-origin index of this node in the job's allocation. Needed to
identify the resources on a specific node allocated to this job.
Returns: None.
Last modified 30 October 2020