Slurm REST API

Slurm provides a REST API daemon named slurmrestd. This daemon is designed to allow clients to communicate with Slurm via a REST API (in addition to the command line interface (CLI) or C API).

Prerequisites

slurmrestd requires additional libraries for compilation:

Stateless

Slurmrestd is stateless as it does not cache or save any state between requests. Each request is handled in a thread and then all of that state is discarded. Any request to slurmrestd is completely synchronous with the Slurm controller (slurmctld or slurmdbd) and is only considered complete once the HTTP response code has been sent to the client. Slurmrestd will hold a client connection open while processing a request. Slurm database commands are commited at the end of every request, on the success of all API calls in the request.

Sites are strongly encouraged to setup a caching proxy between slurmrestd and clients to avoid having clients repeatedly call queries, causing usage to be higher than needed (and causing lock contention) on the controller.

Run modes

Slurmrestd currently supports two run modes: inet service mode and listening mode.

Inet Service Mode

The Slurmrestd daemon acts as an Inet Service treating STDIN and STDOUT as the client. This mode allows clients to use inetd, xinetd, or systemd socket activated services and avoid the need to run a daemon on the host at all times. This mode creates an instance for each client and does not support reusing the same instance for different clients.

Listening Mode

The Slurmrestd daemon acts as a full UNIX service and continuously listens for new TCP connections. Each connection is independently authenticated.

Configuration

slurmrestd can be configured either by environment variables or command line arguments. Please see the doc/man/man1/slurmrestd.1 man page for details.

IPv6 Support

Slurmrestd supports IPv6 clients. On dual stack systems, it will likely provide the source IP as an IPv6 IP for IPv4 clients . Slurmrestd must still be able to talk to slurmctld and slurmdbd via IPv4.

OpenAPI Specification (OAS)

Slurmrestd is compliant with OpenAPI 3.0.2 . The OAS can be viewed at the following URLs:

  • /openapi.json
  • /openapi.yaml
  • /openapi/v3

The OAS is designed to be the primary documentation for the request methods. There are several third party tools that automatically generate documentation against the OAS output listed by openapi.tools.

To view the currently generated API reference: Slurm REST API Reference

Plugins

As of Slurm 20.11, the REST API uses plugins for authentication and generating content. These plugins can be optionality listed or selected via command line arguments as described in the slurmrestd documentation.

Plugin life cycle

Plugins provided with Slurm in any release are considered supported by that release. Due to the nature of the plugins, one plugin can be supported accross multiple Slurm releases to ensure (limited) forward client compatibility. As the slurmrestd plugins are relatively new, we are currently only explicitly deprecating plugins and expect any plugin not marked for deprecation to continue to exist in the next Slurm release. When a plugin is marked for removal in the next major Slurm release, the OpenAPI specification dictates that the given plugin will have all of the paths tagged with 

"deprecated": "true"

Sites are always encouraged to use the latest stable plugin version available for code development. There is no guarantee of compatibility between different versions of the same plugin with clients. Clients should always make sure to validate their code when migrating to newer versions of plugins. Plugin versions will always be included in the path for every method provided by a given plugin to ensure no two plugins will provide the same path.

As the plugins utilize the Slurm API internally, plugins that existed in previous versions of Slurm should continue to be able to communicate with the two previous versions of Slurm, similar to other components of Slurm. Newer plugins may have required RPC changes which will exclude them from working with previous Slurm versions. For instance, the openapi/dbv0.0.36 plugin will not be able to communicate with any slurmdbd servers prior to the slurm-20.11 release.

As with all other plugins in Slurm, sites are more than welcome to write their own plugins and are suggested to submit them as code contributions via bugzilla, tagged as a contribution. The plugin API provided may change between releases which could cause site specific plugins to break.

Security

The Slurm REST API is written to provide the necessary functionality for clients to control Slurm using REST commands. It is not designed to be directly internet facing. Only unencrypted and uncompressed HTTP communications are supported. Slurmrestd also has no protection against man in the middle or replay attacks. Slurmrestd should only be placed in a trusted network that will communicate with a trusted client.

JSON Web Token (JWT) Authentication

slurmrestd supports using JWT to authenticate users. JWT can be used to authenticate user over REST protocol.

  • User Name Header: X-SLURM-USER-NAME
  • JWT Header: X-SLURM-USER-TOKEN
SlurmUser or root can provide alternative user names to act as a proxy for the given user. While using JWT authentication, slurmrestd should be run as a unique, unprivileged user and group. Slurmrestd should be provided an empty SLURM_JWT environment variable at startup to activate JWT authentication. This will allow users to provide their own JWT tokens while authenticating to the proxy.

When using JWT, it is important that AuthAltTypes=auth/jwt be configured in your slurm.conf for slurmrestd.

Local Authentication

slurmrestd supports using UNIX domain sockets to have the kernel authenticate local users. slurmrestd must be run as SlurmUser or root to allow multiple different users at the same time but it is not required if users wish to run as only themselves. Slurmrestd must be located in the Munge security domain in order to function and communicate with Slurm in local authentication mode.

Last modified 17 December 2020