Client Node Tuning¶
Parallel Network Requests¶
Each BeeGFS client establishes multiple network connections to the same server, which allows the
client to have multiple network requests in flight to this server.
The number of connections from a particular client to the same server can be configured by setting
the value of connMaxInternodeNum in /etc/beegfs/beegfs-client.conf.
Increasing the number of connections may improve performance and responsiveness for certain
workloads. When increasing the value, it is extremely important to keep the resulting RAM usage for
network buffers on the servers in mind, especially for InfiniBand and larger cluster setups. Make
sure to read the comments for connMaxInternodeNum and connRDMABufSize in
beegfs-client.conf to learn more about the server-side RAM usage. Note that connRDMABufSize
needs to be an integer multiple of 4 KiB. On a compute node, it usually doesn’t make sense to set
this number higher than the number of CPU cores. On a cluster login node, setting this value higher
than the number of CPU cores may help to improve responsiveness when multiple users are active.
BeeGFS clients establish connections only when they are needed (and drop them after some idle time).
Use the command beegfs-net on a client to see the number of currently established connections to
each of the servers. beegfs-net is contained in the beegfs-utils package. The total space
used by the buffers (connRDMABufSize x connRDMABufNum) should be larger or equal to the data
chunk size, so that the messages exchanged between client and storage servers do not need to be
split to fit into the buffers available. The default RDMA settings (connRDMABufSize = 64 KB,
connRDMABufNum = 12) are OK for the default chunk size of 512 KB. If you set a chunk size
of 1 MB and a buffer size of 64 KB, the number of buffers should be at least 1 MB / 64 KB + 4
additional buffers for protocol, so 20 in this example.
Remote fsync¶
BeeGFS clients have a configuration option to control behavior when a user application calls
fsync(). The option is called tuneRemoteFSync in /etc/beegfs/beegfs-client.conf. The
client can either enforce that data is committed to the server disks on fsync()
(tuneRemoteFSync=true) or only make sure that data is transferred to the server-side cache
(tuneRemoteFSync=false). Disabling remote fsync can significantly reduce disk seeks and thus
improves performance for applications that use a lot of fsync() calls.
Disable locate/mlocate/updatedb¶
Some Linux distributions install a locate tool which scans all file systems once per day
to build a database of existing files. These tools can create unnecessary load on the filessytem if
they are active on the client nodes.
Either deactivate this service if you don’t need it or edit the file /etc/updatedb.conf to make
sure that the beegfs file system type is contained in the PRUNEFS list and your BeeGFS
mount point is contained in the PRUNEPATHS list.
Communication retries¶
BeeGFS clients support a time-limited retry mode for communications,
which will make them keep on trying to complete their I/O operation in case a server is
temporarily unreachable. The connCommRetrySecs option in /etc/beegfs/beegfs-client.conf
sets the time limit for communication retries. Retries can be stopped by:
Interrupting the application waiting for the I/O operation, e.g. CTRL + c.
Disabling retries for the entire mountpoint:
echo false > /proc/fs/beegfs/<...>/conn_retries_enabled
Preferred servers¶
Clients can prefer certain storage servers for new files and directories.
For example, due to a certain network topology. See options tunePreferredMetaFile and
tunePreferredStorageFile in /etc/beegfs/beegfs-client.conf. Note: These options only affect the
placement of new files. Clients having those options can set still read all files, regardless on
which server they are located.
