To perform a manual installation, you have to download the BeeGFS packages and set up a few basic configuration options.
It is recommended that you start by reading the first three chapters below in order to get a basic understanding of the general installation procedure. See Quick Start Guide as a reference while performing the actual installation on your system.
Download and Installation of Packages¶
To start an installation of BeeGFS:
Download the BeeGFS repository file for your Linux distribution from the table below to the specified directory.
Make the repository available on all nodes, either by storing it in your node boot image or by copying it to all nodes that you want to use with BeeGFS.
Install the BeeGFS services with your distribution’s package manager (e.g. “
yum install beegfs-storageon RedHat). An overview of available packages is provided in the next section.
You can download the packages directly instead of using repositories.
BeeGFS can be used in heterogeneous environments with different architectures and/or distributions.
Linux Base Distribution
Repository File (Save to…)
Red Hat & CentOS
(and derivatives, e.g. Fedora, …)
(and derivatives, e.g. OpenSuse, …)
Debian GNU Linux
apt, please do not forget to run
apt-get update after adding the repository.
Repository Signature Keys¶
BeeGFS repositories and packages are digitally signed. If you would like to verify the package signatures, you can add the public BeeGFS GPG key to your package manager:
For RPM Packages: https://www.beegfs.io/release/beegfs_7.2/gpg/RPM-GPG-KEY-beegfs
For Debian Packages: https://www.beegfs.io/release/beegfs_7.2/gpg/DEB-GPG-KEY-beegfs
To add the key on RedHat/SuSE, use the following command:
# rpm --import https://www.beegfs.io/release/beegfs_7.2/gpg/RPM-GPG-KEY-beegfs
To add the key on Debian, use the following command:
# wget -q https://www.beegfs.io/release/beegfs_7.2/gpg/DEB-GPG-KEY-beegfs -O- | apt-key add -
The following table shows the roles that need to be defined to run BeeGFS and the corresponding package for each role. BeeGFS has been designed to allow running any combination of services (e.g., metadata and storage server) on the same node. Additional information about the roles of BeeGFS nodes can be found in Architecture.
Management Server (one node)
Metadata Server (at least one node)
Storage Server (at least one node)
Mon - InfluxDB based Monitoring Server (optional)
BeeGFS utilities for administrators
The management server daemon, and the optional monitoring daemon has only moderate RAM and CPU requirements. As these services are not critical for file system performance, you do not need to run them on a dedicated machine. They are typically running on the cluster master node.
- Each BeeGFS service (including the client) comes with:
a config file:
an init script:
Installation does not start any of the services directly because there are a few configuration options that need to be set before the first startup. These will be described in the following sections.
Init scripts are replaced by systemd on most distributions and are only required by older operating systems installations. The only exception is the client script, which is used to trigger automatic client building and running (see below).
After installation, all services will be set to automatic startup at system boot. If you do not
want to run BeeGFS services automatically during system boot, use e.g. the
chkconfig tool to
disable automatic startup.
Another way to disable BeeGFS services startup would be to set
START_SERVICENAME="NO" in the
corresponding service configuration file (
/etc/default/beegfs-X). Note that this would have the side
effect of completely disabling init script startup of the service, not only during system boot.
Building the Client Kernel Module¶
Most parts of BeeGFS come pre-built and ready to use when you install the packages. However, the client kernel module of BeeGFS still needs to be compiled for your specific Linux kernel version.
In addition to that, it might be necessary to install the library package for RDMA support
libbegfs-ib), which contains the RDMA abstraction layer of BeeGFS for userspace applications
BeeGFS Client Kernel Module¶
BeeGFS client kernel modules are automatically built on startup of the
/etc/init.d/beegfs-client) for the currently running Linux kernel, so no manual build step is
required. The init script will also detect client package updates and kernel updates and
automatically runs a client module rebuild on next startup.
However, InfiniBand support is disabled by default. To enable IB support, you need to modify the
buildArgs parameter in the build configuration file
Default value (without InfiniBand support):
With InfiniBand support:
buildArgs=-j8 BEEGFS_OPENTK_IBVERBS=1 OFED_INCLUDE_PATH=/usr/src/openib/include
buildArgs example above, OFED header files are installed to
/usr/src/openib. If you are using the default kernel IB modules (i.e., not the separate OFED
modules), then the option
OFED_INCLUDE_PATH does not need to be set at all.
To force a rebuild of the client kernel modules, you need to be root on a client node and then call:
# /etc/init.d/beegfs-client rebuild
After a successful module build, the beegfs-client needs to be restarted:
# /etc/init.d/beegfs-client restart
If the client module is unable to load,
/var/log/messages) might provide
BeeGFS RDMA Library for Server Services¶
This section is only relevant if you are using InfiniBand or RoCE hardware. If not, just skip to the next section.
If you have your hardware and the correspondent drivers installed, then it is only necessary to
libbeegfs-ib package to use RDMA with the server services (
There are three configuration settings that are mandatory to be set for a specific environment. The following table shows the three settings and the configuration files, in which they will be applied:
Configuration File (in
Client Mount Directory
These are the three options that need to be set before you run the services.
Set the value of the
storeStorageDirectoryoption to configure where BeeGFS should store its internal data. BeeGFS stores all data in a subdirectory of an existing partition that has been formatted with any of the standard local Linux file systems (typically XFS for storage servers and ext4 for metadata servers).
Note that different services on the same machine cannot share the same storage directory, so different directories have to be used, e.g.
/data/beegfs-metafor metadata servers and
/data/beegfs-storagefor storage servers.
Set the value of the
sysMgmtdHostoption in the configuration files of the metadata servers, storage servers, clients, and the optional mon to the IP address or hostname of the management daemon. This will enable automatic server discovery for the clients.
Client Mount Directory
The client mount file consists of two space-separated values. The first value is the directory where you want to mount the file system, and the second value is the client configuration file for this mount point. You will typically have a line like this in the
It is also possible to specify multiple mount/config entries in this file (one mount/config pair per line) if you need to mount different BeeGFS instances on the same client. Optionally, you can also inform mount options for each mount/config entry. In the example below, the first line mounts a writable BeeGFS FS on
/mnt/scratchand a read-only BeeGFS FS on
/mnt/scratch /etc/beegfs/beegfs-client-scratch.conf beegfs rw /mnt/software /etc/beegfs/beegfs-client-software.conf beegfs ro
When you have made the changes to the configuration files, it is time to start the BeeGFS services.
The services will register themselves at the management server, so clients will automatically know about all available servers. The instructions below are for single instances. If you want to run multiple instances of one service on the same machine, please refer to Multi Mode.
BeeGFS clients have a mount sanity check and cancel a mount operation if servers are
unreachable. If you want to mount even with unreachable servers, set
systemctl start beegfs-<service> command to start the services.
Advanced Configuration and Tuning¶
Runtime configuration options can be queried and changed with the BeeGFS Control Tool (
command on the client nodes). This tool is part of the beegfs-utils package.
beegfs-ctl tool reads a
beegfs-client.conf file from the default location, if it exists.
While the tool can also be used without a client configuration file, it is usually much more convenient to
use when such a basic client configuration exists on the corresponding machine.
On a client with an active BeeGFS mount, several configuration options can also be queried through
procfs entries (see
There are also several convenience helper scripts contained in the beegfs-utils package (e.g.
/usr/bin/beegfs-net to show the currently established client connections), which either call
beegfs-ctl internally or read output from
Static configuration options can be set by editing the corresponding configuration files
/etc/beegfs/beegfs-X.conf). If you haven’t done so yet, you might want to take a look at these
files to find out which options you have in general. There is a description for each of the settings
in the corresponding configuration file.
Prevent accidental registration of new nodes/targets¶
BeeGFS contains some safety options that you might want to set when your
initial file system setup is complete:
storeAllowFirstRunInit=false forbids daemon startup if the
storage directory is empty, e.g. because the corresponding RAID volume is not mounted properly.
beegfs-mgmtd.conf forbids registration of new servers (e.g. to make
sure a new server is not registered accidentally in a production environment).
sysMountSanityCheckMS can be used to let the client refuse mounting the file system if not all
servers are reachable. (You will typically want to disable this option for clients that are
running together with server daemons on the same machine.)