Node Management¶
See also
Adding Nodes¶
Adding a new node to BeeGFS is very simple and does not require a downtime. You just have to take the following steps. Please, read the whole procedure before taking any action.
Make sure that the following properties are set to “true” in file
/etc/beegfs/beegfs-mgmtd.conf
, in order for the management service to allow new nodes and targets to be added to the system. You will find detailed documentation of those properties on the bottom of the file.sysAllowNewServers = true sysAllowNewTargets = true
If you change the properties above, please restart the management service afterward.
# systemctl restart beegfs-mgmtd
Install BeeGFS packages on the new node and configure the services, as described in the installation procedure.
Start the service, as seen in the example below.
# systemctl start beegfs-storage
Use the command
beegfs-check-servers
to list the server nodes in your system and check if your new node is listed. If not, check the log files of the added service and the management service to see if any error message informs you why the node could not be added.$ less /var/log/beegfs-mgmtd.log $ less /var/log/beegfs-meta.log $ less /var/log/beegfs-storage.log
Set the management properties back to false, to prevent accidental registrations of new nodes and targets in your system. Please, restart the management service afterward.
sysAllowNewServers = false sysAllowNewTargets = false
Removing Nodes¶
Removing Metadata Nodes¶
Note
It is not possible to safely remove metadata nodes from the system unless the node is completely empty. For example, this would be the case if a new metadata node was registered accidentally. There is no data migration tool for metadata.
To check if a metadata node is empty, locate its data directory and check the dentries
and
inodes
as well as buddymir/{dentries,inodes}
directory for files. The node can be considered
empty if the following conditions are met:
no files (only directories) are present in the directories
dentries
andbuddymir/dentries
and their subdirectoriesno files except
38/51/root
and35/5B/disposal
are present within theinodes
directory and its subdirectoriesno files except
38/51/root
and23/40/mdisposal
are present within thebuddymir/inodes
directory and its subdirectories
For example, you can use
# find /data/beegfs/meta/{dentries,inodes,buddymir/dentries,buddymir/inodes} -type f
If the result is
/data/beegfs/meta/inodes/38/51/root
/data/beegfs/meta/inodes/35/5B/disposal
/data/beegfs/meta/buddymir/inodes/23/40/disposal
the node can be considered empty. If any more files are listed, removing the node is not safe.
If the metadata node is empty, you can execute
# beegfs-ctl --removenode --nodetype=meta <nodeID>
to remove it from the system.
Removing Storage Nodes¶
Warning
Removing a node will not automatically migrate the files that were stored on this node, thus use this option carefully.
To safely remove a storage node, all of the attached storage targets have to be empty.
Data can be moved off specific targets and distributed over the remaining ones using the
beegfs-ctl --migrate
command:
# beegfs-ctl --migrate --nodeid=<nodeID> /mnt/beegfs
Warning
Do not forcefully remove a non-empty node. This will lead to errors and possible data corruption.
Once all targets of the node are empty, it can be removed from the system:
# beegfs-ctl --removenode --nodetype=storage <nodeID>
See also
Moving a node to another host¶
Moving the Management Node¶
Take the following steps to move the management service to a different machine. Please, read the whole procedure before taking any action. We suggest downtime to move a management target to a new server!
Stop the management service on the current machine. If the service is configured to run in Multi Mode, be careful to stop the right instance.
# service beegfs-mgmtd stop # umount /data/ssd
If you don’t need the management service running on the current machine, please uninstall it.
Check if the storage device where the management directory resides can be moved or connected to the new machine. If so, unmount it, move it, mount it on a directory of the new machine, and make sure it is configured to be mounted at boot time. The management directory is defined by option
storeMgmtdDirectory
in file/etc/beegfs/beegfs-mgmtd.conf
.Otherwise, if the device cannot be moved or connected to the new machine, copy the management directory from a backup (or from the device, remounted somewhere else) to a partition of the new machine.
Check if the management service is already installed on the new machine. If not, please install it. Do not start the service at this point. If the management service is already installed on the new server machine, please configure the service to run in multi-mode.
Then, edit the file
/etc/beegfs/beegfs-mgmtd.conf
on the new machine and set optionstoreMgmtdDirectory
to the path to the management service directory. If the service is configured to run in multi-mode, be careful to use the configuration file and directory of the right instance.Make sure that the file
nodeID
exists on the service directory and that it contains the ID that the management service daemon should use to identify itself. If the file does not exist, just create it with the same content of theoriginalNodeID
file.If the new machine has a different hostname, edit the configuration file of each other BeeGFS service running in your system, which are associated to that management service, and write the new hostname as the value of option
sysMgmtdHost
.Make sure that the new machine is reachable from the other BeeGFS servers, specially if you are using the client configuration options
connInterfacesFile
orconnNetFilterFile
to restrict which network interfaces can be used by BeeGFS services. Its IP address may be changed freely, because BeeGFS keeps no record of such addresses.Start or restart the management service.
# service beegfs-mgmtd restart
Test if the target is working properly.
Check if log files contain any error message.
$ less /var/log/beegfs-mgmtd.log $ less /var/log/beegfs-meta.log $ less /var/log/beegfs-storage.log $ less /var/log/beegfs-client.log
List all management nodes of your system and check if the moved one appears online.
$ beegfs-ctl --listnodes --nodetype=mgmt --reachable
Moving a Metadata Node¶
Take the following steps to move a metadata target to a different machine. Please, read the whole procedure before taking any action. This description focuses on the transfer of the storage device from one server to another, but it is possible to use a backup. If you are going to move the metadata target by means of a backup, please read the backup metadata instructions (see Metadata Daemon Backup). We suggest downtime to move a metadata target to a new server!
Stop the metadata service on the current machine and unmount the metadata target device. If the service is configured to run in multi-mode, be careful to stop the right instance.
# service beegfs-meta stop # umount /data/ssd
If you don’t need the metadata service running on the current machine, please uninstall it.
Check if the metadata target device can be moved or connected to the new machine. If so, mount it on a directory of the new machine and make sure it is configured to be mounted at boot time.
Otherwise, if the metadata target device cannot be moved or connected to the new machine, copy the data from a backup (or from the device, remounted somewhere else) to a storage device of the new machine (see Metadata Daemon Backup).
Check if the metadata service is already installed on the new machine. If not, please install it. If the metadata service is already installed on the new server machine, please configure the service to run in Multi Mode. Do not start the service at this point.
Then, edit the file
/etc/beegfs/beegfs-meta.conf
on the new machine and add the path of the moved metadata target to optionstoreStorageDirectory
. If the service is configured to run in multi-mode, be careful to use the configuration file and directory of the right instance.Make sure that the file
nodeID
exists on the service directory and that it contains the ID that the metadata service daemon should use to identify itself with the management service. If the file does not exist, just create it with the same content of theoriginalNodeID
file.Start or restart the metadata service.
# service beegfs-meta restart
Test if the target is working properly.
Check if log files contain any error message.
$ less /var/log/beegfs-mgmtd.log $ less /var/log/beegfs-meta.log $ less /var/log/beegfs-storage.log $ less /var/log/beegfs-client.log
List all metadata nodes of your system and check if the moved one appears online.
$ beegfs-ctl --listnodes --nodetype=meta --reachable