Manual Installation

To perform a manual installation, you have to download the BeeGFS packages and set up a few basic configuration options.

It is recommended that you start by reading the first three chapters below in order to get a basic understanding of the general installation procedure. See Quick Start Guide as a reference while performing the actual installation on your system.

Download and Installation of Packages

Package Repositories

To start an installation of BeeGFS:

1. Visit the BeeGFS download page and follow the step-by-step directions to add the BeeGFS package repositories to your Linux distribution’s package manager.

   • You will need to make the repository available on all servers, either by adding it to a common boot image, or by copying it to all servers that will run BeeGFS services or mount BeeGFS.

2. Using your distribution’s package manager (e.g., apt or dnf) install the appropriate BeeGFS packages to each of the servers in your environment, depending which BeeGFS service(s) they will run or if they will need to mount the file system.

   • See the package descriptions below for an overview of the available packages.

3. (Optional) If you are using the BeeGFS Hive Edition, add your license to /etc/beegfs/license.pem on the machine that will run your management service.

Note

BeeGFS can be used in heterogeneous environments with different architectures and/or distributions. For example your BeeGFS server services could run on x86 machines using Rocky Linux and your clients could be arm machines running Ubuntu.

Package Descriptions

The following table shows the roles that need to be defined to run BeeGFS and the corresponding package for each role. BeeGFS has been designed to allow running any combination of services (e.g., metadata and storage server) on the same node. Additional information about the roles of BeeGFS nodes can be found in Architecture.

BeeGFS Roles and Corresponding Packages

Management Server (one node) Manages configuration and group membership Hostname or IP address must be known by other nodes at service start time	beegfs-mgmtd
Enterprise Feature Support (optional for the management) library for enabling enterprise features on the management	libbeegfs-license
Metadata Server (at least one node) Stores directory information and allocates file space on storage servers	beegfs-meta
Storage Server (at least one node) Stores raw file contents	beegfs-storage
Client Kernel module to mount the file system	beegfs-client,
RDMA Support libraries for RDMA support for Metadata and Storage Services	libbeegfs-ib
Mon - InfluxDB based Monitoring Server (optional) Continuous monitoring of servers Live statistics	beegfs-mon
BeeGFS tools for administrators beegfs tool for command-line administration	beegfs-tools
BeeGFS utilities for administrators beegfs-fsck tool to check/correct file system consistency	beegfs-utils

Note

The management server daemon, and the optional monitoring daemon has only moderate RAM and CPU requirements. As these services are not critical for file system performance, you do not need to run them on a dedicated machine. They are typically running on the cluster master node.

Each BeeGFS service (including the client) comes with:

• a config file: /etc/beegfs/beegfs-X

• systemd units

Installation does not start any of the services directly because there are a few configuration options that need to be set before the first startup. These will be described in the following sections.

Init scripts

Because init scripts have been replaced by systemd on all distributions officially supported by BeeGFS, starting in BeeGFS v8 only the client includes an init script that is used in building the client module and mounting the file system (see below).

Building the Client Kernel Module

Most parts of BeeGFS come pre-built and ready to use when you install the packages. However, the client kernel module of BeeGFS still needs to be compiled for your specific Linux kernel version.

In addition to that, it might be necessary to install the library package for RDMA support (libbegfs-ib), which contains the RDMA abstraction layer of BeeGFS for userspace applications and services.

BeeGFS Client Kernel Module

BeeGFS client kernel modules are automatically built on startup of the beegfs-client service (/etc/init.d/beegfs-client) for the currently running Linux kernel, so no manual build step is required. The init script will also detect client package updates and kernel updates and automatically runs a client module rebuild on next startup.

Unlike in earlier versions, RDMA support is available by default.

If you want to use a thirdparty driver though, you have to specify it in the file /etc/beegfs/beegfs-client-autobuild.conf like this: buildArgs=-j8 OFED_INCLUDE_PATH=/usr/src/openib/include.

To build the client without RDMA support, add BEEGFS_NO_RDMA=1 to buildArgs.

To force a rebuild of the client kernel modules, you need to be root on a client node and then call:

# /etc/init.d/beegfs-client rebuild

After a successful module build, the beegfs-client needs to be restarted:

# /etc/init.d/beegfs-client restart

If the client module is unable to load, dmesg / journalctl -k or journalctl -u beegfs-client might provide useful information.

BeeGFS RDMA Library for Server Services

Note

This section is only relevant if you are using InfiniBand or RoCE hardware. If not, just skip to the next section.

If you have your hardware and the correspondent drivers installed, then it is only necessary to install the libbeegfs-ib package to use RDMA with the server services (beegfs-meta and beegfs-storage).

Basic Configuration

There are three configuration settings that are mandatory to be set for a specific environment. The following table shows the three settings and the configuration files, in which they will be applied:

Basic Settings and Corresponding Configuration Files

Setting	Configuration File (in /etc/beegfs)
Storage Paths	beegfs-mgmtd.toml beegfs-meta.conf beegfs-storage.conf
Management Host	beegfs-meta.conf beegfs-storage.conf beegfs-client.conf
Client Mount Directory	beegfs-mounts.conf

These are the three options that need to be set before you run the services.

• Storage Paths

  For the management node optionally customize the path to the database using db-file then run /opt/beegfs/sbin/beegfs-mgmtd --init to initialize the database and follow the steps to configure TLS.

  For metadata and storage nodes, use the storeMetaDirectory and storeStorageDirectory options to configure where BeeGFS should store internal data. BeeGFS stores all data in a subdirectory of an existing partition that has been formatted with any of the standard local Linux file systems (typically XFS for storage servers and ext4 for metadata servers). By default these services will automatically initialize their directories when they first start, but it is recommend you manually initialize them using /opt/beegfs/sbin/beegfs-setup-[meta|storage]. This allows setting specific numeric IDs if desired and applies recommended settings.

  Note that different services on the same machine cannot share the same storage directory, so different directories have to be used, e.g. /data/beegfs-meta for metadata servers and /data/beegfs-storage for storage servers.

• Management Host

  Set the value of the sysMgmtdHost option in the configuration files of the metadata servers, storage servers, clients, and the optional monitoring service, to the IP address or hostname of the management daemon. This will enable automatic server discovery for the clients.

• Client Mount Directory

  The client mount file consists of two space-separated values. The first value is the directory where you want to mount the file system, and the second value is the client configuration file for this mount point. You will typically have a line like this in the beegfs-mounts.conf file:

  /mnt/beegfs /etc/beegfs/beegfs-client.conf
  

  It is also possible to specify multiple mount/config entries in this file (one mount/config pair per line) if you need to mount different BeeGFS instances on the same client. Optionally, you can also inform mount options for each mount/config entry. In the example below, the first line mounts a writable BeeGFS FS on /mnt/scratch and a read-only BeeGFS FS on /mnt/software.

  /mnt/scratch /etc/beegfs/beegfs-client-scratch.conf beegfs rw
  /mnt/software /etc/beegfs/beegfs-client-software.conf beegfs ro
  

When you have made the changes to the configuration files, it is time to start the BeeGFS services.

Connection authentication

We strongly recommend to configure connection authentication with a connAuthFile at this point. Please see Authentication for more information. If connection authentication is not needed, connDisableAuthentication has to be set to true on all nodes.

Service Startup

The services will register themselves at the management server, so clients will automatically know about all available servers. The instructions below are for single instances. If you want to run multiple instances of one service on the same machine, please refer to Multi Mode.

Note

BeeGFS clients have a mount sanity check and cancel a mount operation if servers are unreachable. If you want to mount even with unreachable servers, set sysMountSanityCheckMS=0 in the file beegfs-client.conf.

Use systemctl start beegfs-<service> command to start the services.

Advanced Configuration and Tuning

Runtime configuration options can be queried and changed with the BeeGFS Control Tool beegfs provided in the beegfs-tools package. This tool is typically installed on a BeeGFS client as some modes require a BeeGFS mount point.

The beegfs tool does not rely on any external configuration file. Instead required configuration is specified as using flags, or optionally exported as environment variables. For example the management address of the file system you wish to interact with can be specified like:

beegfs node list --mgmtd-addr=hostname:8010

Alternatively you could export an environment variable to avoid having to specify the flag every time you run the command. This could be exported once for the current session, or made persistent using your .bashrc file or equivalent for your shell:

export BEEGFS_MGMTD_ADDR=hostname:8010
beegfs node list

See beegfs help for a list of all global flags. All global flags can be set using environment variables by converting them to uppercase, replacing hyphens (-) with underscores (_) and adding the prefix BEEGFS_. For example auth-file is specified as BEEGFS_AUTH_FILE=/some/path.

On a client with an active BeeGFS mount, several configuration options can also be queried through procfs entries (see /proc/fs/beegfs).

Static configuration options can be set by editing the corresponding configuration files (/etc/beegfs/beegfs-X.conf). If you haven’t done so yet, you might want to take a look at these files to find out which options you have in general. There is a description for each of the settings in the corresponding configuration file.

See also

Management Service Metadata Node Tuning Storage Node Tuning

Network Tuning Client Side Caching Modes Client Node Tuning

Striping RDMA Support

Prevent accidental registration of new nodes/targets

BeeGFS contains some safety options that you might want to set when your initial file system setup is complete:

• registration-disable=false in beegfs-mgmtd.toml forbids registration of new servers (e.g. to make sure a new server is not registered accidentally in a production environment).

  • This does not affect client registration.

• storeAllowFirstRunInit=false in beegfs-meta.conf and beegfs-storage.conf forbids service startup if the storage directory is empty, e.g. because the corresponding RAID volume is not mounted properly.

  • There is no corresponding setting on the management because it does not automatically initialize the database if it does not already exist.

• sysMountSanityCheckMS in beegfs-client.conf can be used to let the client refuse mounting the file system if not all servers are reachable.

  • You will typically want to disable this option for clients that are running together with server services (e.g., management, metadata, or storage) on the same machine.

Backups

We strongly recommend to setup a backup solution like rsync, in order to backup the management target (often /beegfs/mgmtd/) to a safe external location. Please keep in mind that RAID or buddy mirroring is no replacement for an external backup.

See also

Backup