Multi Mode

If you want to run more than one instance of one daemon type on a single machine, you have to use multi mode.

General Information about Multi Mode

In BeeGFS, multi mode is a way to run multiple instances of one type of service on one server. This can be useful if for example, more than one BeeGFS file system or more than one metadata target is used on a single server. Please take into account that when using multi mode within a single file system, BeeGFS will treat them as individual machines. So, when creating for example mirror buddy groups with the --automatic option, one group might have both services on a single server. In that case, a manual configuration for Buddy Groups is necessary.

For multi mode, you will need to create a separate configuration file for the other daemon instance, using different network ports, a different storage directory, a different log file, and so on. If the second daemon instance on a machine should become part of the same file system instance (i.e. it registers at the same management daemon as the first daemon instance on this machine), then you would also need to set a different NodeID manually for the second daemon instance. (See “Configuration & Tuning” section on this page for information on how to manually set the NodeID.)

Setting up Multi Mode

You can use the BeeGFS systemd service template to start multiple instances of one service:

# systemctl start beegfs-<service>@<name>.service

Where <service> is the name of the service to be started (e.g. meta, storage, …) and <name> is a user defined string, representing the instance about to be started.

The single mode configuration file in /etc/beegfs/beegfs-<service>.conf will not be used when starting in multi mode. Instead, each instance has its own configuration file, stored in /etc/beegfs/<name>.d/beegfs-<service>.conf.

In order to start multiple beegfs-client services on a single server, the procedure is different. In that case, it is only necessary to add more lines to the file /etc/beegfs/beegfs-mounts.conf.

To have different BeeGFS instances mounted on the same machine, you have to supply a client configuration file for each instance. You can name the different client configuration files /etc/beegfs/beegfs-client-<name>.conf with <name> being a descriptor for the particular instance. The names of these files will be referenced in the client mount file /etc/beegfs/beegfs-mounts.conf.

Each line in the client mount file consists of two space-separated values. The first value is the directory where you want to mount the file system. The second value is the client configuration file for this mount point. You will typically have a line like this in the /etc/beegfs/beegfs-mounts.conf file:

/mnt/beegfs /etc/beegfs/beegfs-client.conf

It is possible to specify multiple mount/config entries in this file (one mount/config pair per line) if you need to mount different BeeGFS instances on the same client. Optionally, you can also specify mount options for each mount/config entry. In the example below, the first line mounts a writable BeeGFS FS on /mnt/scratch and a read-only BeeGFS FS on /mnt/software.

/mnt/scratch /etc/beegfs/beegfs-client-scratch.conf beegfs rw
/mnt/software /etc/beegfs/beegfs-client-software.conf beegfs ro

Start the service with:

# systemctl start beegfs-client.service

Example Configuration

In this example, we are going to create a setup with the metadata and the storage daemons running in multi-mode, in a single BeeGFS namespace. This is an adapted version of the Quick Start Guide that only describes the adaptions for the beegfs-meta and beegfs-storage services.

$ ssh root@node01
# /opt/beegfs/sbin/beegfs-setup-mgmtd -p /data/beegfs/beegfs_mgmtd

$ ssh root@node02
# mkdir /etc/beegfs/inst1.d
# mkdir /etc/beegfs/inst2.d
# cp /etc/beegfs/beegfs-meta.conf /etc/beegfs/inst1.d/
# cp /etc/beegfs/beegfs-meta.conf /etc/beegfs/inst2.d/
# vi /etc/beegfs/inst1.d/beegfs-meta.conf # Adapt "connMetaPort*", etc.
# vi /etc/beegfs/inst2.d/beegfs-meta.conf # Adapt "connMetaPort*", etc.
# /opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/inst1.d/beegfs-meta.conf \
                                     -p /data/beegfs/beegfs_meta1 -s 21 -S meta1-inst1 -m node01
# /opt/beegfs/sbin/beegfs-setup-meta -c /etc/beegfs/inst2.d/beegfs-meta.conf \
                                     -p /data/beegfs/beegfs_meta2 -s 22 -S meta1-inst2 -m node01

$ ssh root@node03
# mkdir /etc/beegfs/inst1.d
# mkdir /etc/beegfs/inst2.d
# cp /etc/beegfs/beegfs-storage.conf /etc/beegfs/inst1.d/
# cp /etc/beegfs/beegfs-storage.conf /etc/beegfs/inst2.d/
# vi /etc/beegfs/inst1.d/beegfs-storage.conf # Adapt "connStoragePort*", etc.
# vi /etc/beegfs/inst2.d/beegfs-storage.conf # Adapt "connStoragePort*", etc.
# /opt/beegfs/sbin/beegfs-setup-storage -c /etc/beegfs/inst1.d/beegfs-storage.conf \
                                        -p /mnt/myraid11/beegfs_storage \
                                        -s 31 -S stor1-inst1 -i 311 -m node01
# To add a second storage target on this same machine:
# /opt/beegfs/sbin/beegfs-setup-storage -c /etc/beegfs/inst1.d/beegfs-storage.conf \
                                        -p /mnt/myraid12/beegfs_storage \
                                        -s 31 -S stor1-inst1 -i 312
# /opt/beegfs/sbin/beegfs-setup-storage -c /etc/beegfs/inst2.d/beegfs-storage.conf \
                                        -p /mnt/myraid21/beegfs_storage \
                                        -s 32 -S stor1-inst2 -i 321 -m node01
# /opt/beegfs/sbin/beegfs-setup-storage -c /etc/beegfs/inst2.d/beegfs-storage.conf \
                                        -p /mnt/myraid22/beegfs_storage \
                                        -s 32 -S stor1-inst2 -i 322

$ ssh root@node1 systemctl start beegfs-mgmtd
$ ssh root@node2 systemctl start beegfs-meta@inst1
$ ssh root@node2 systemctl start beegfs-meta@inst2
$ ssh root@node3 systemctl start beegfs-storage@inst1
$ ssh root@node3 systemctl start beegfs-storage@inst2