Manual Installation

To perform a manual installation, you have to download the BeeGFS packages and set up a few basic configuration options.

It is recommended that you start by reading the first three chapters below in order to get a basic understanding of the general installation procedure. See Quick Start Guide as a reference while performing the actual installation on your system.

Download and Installation of Packages

Package Repositories

To start an installation of BeeGFS:

  1. Download the BeeGFS repository file for your Linux distribution from the table below to the specified directory.

  2. Make the repository available on all nodes, either by storing it in your node boot image or by copying it to all nodes that you want to use with BeeGFS.

  3. Install the BeeGFS services with your distribution’s package manager (e.g. “yum install beegfs-storage on RedHat). An overview of available packages is provided in the next section.

You can download the packages directly instead of using repositories.

Note

BeeGFS can be used in heterogeneous environments with different architectures and/or distributions.

BeeGFS Repository Files

Linux Base Distribution

Version

Package Manager

Repository File (Save to…)

Red Hat & Rocky/Alma

8

yum

RHEL8

(Save to: /etc/yum/repos.d/)

SuSE Linux

(and derivatives, e.g. OpenSuse, …)

15

zypper

SLES15

(Save to: /etc/zypp/repos.d/)

Debian GNU Linux

10

(buster)

apt

Debian10

(Save to: /etc/apt/sources.list.d/)

11

(bullseye)

apt

Debian11

(Save to: /etc/apt/sources.list.d/)

Ubuntu

18.04

(bionic)

apt

Ubuntu18

(Save to: /etc/apt/sources.list.d/)

20.04

(focal)

apt

Ubuntu20

(Save to: /etc/apt/sources.list.d/)

Note

When using apt, please do not forget to run apt-get update after adding the repository.

Repository Signature Keys

BeeGFS repositories and packages are digitally signed. If you would like to verify the package signatures, you can add the public BeeGFS GPG key to your package manager:

https://www.beegfs.io/release/beegfs_7.3.2/gpg/GPG-KEY-beegfs

To add the key on RedHat/SuSE, use the following command:

# rpm --import https://www.beegfs.io/release/beegfs_7.3.2/gpg/GPG-KEY-beegfs

To add the key on Debian, use the following command:

# wget -q https://www.beegfs.io/release/beegfs_7.3.2/gpg/GPG-KEY-beegfs -O- | apt-key add -

Package Descriptions

The following table shows the roles that need to be defined to run BeeGFS and the corresponding package for each role. BeeGFS has been designed to allow running any combination of services (e.g., metadata and storage server) on the same node. Additional information about the roles of BeeGFS nodes can be found in Architecture.

BeeGFS Roles and Corresponding Packages

Management Server (one node)

  • Manages configuration and group membership

  • Hostname or IP address must be known by other nodes at service start time

beegfs-mgmtd

Metadata Server (at least one node)

  • Stores directory information and allocates file space on storage servers

beegfs-meta

Storage Server (at least one node)

  • Stores raw file contents

beegfs-storage

Client

  • Kernel module to mount the file system

  • Requires userspace helper daemon for logging and hostname resolution

beegfs-client, beegfs-helperd

RDMA Support

  • libraries for RDMA support for Metadata and Storage Services

libbeegfs-ib

Mon - InfluxDB based Monitoring Server (optional)

  • Continuous monitoring of servers

  • Live statistics

beegfs-mon

BeeGFS utilities for administrators

  • beegfs-ctl tool for command-line administration

  • beegfs-fsck tool for file system checking

  • Several small helper scripts

beegfs-utils

BeeGFS Common

  • Common files for all packages

beegfs-common

Note

The management server daemon, and the optional monitoring daemon has only moderate RAM and CPU requirements. As these services are not critical for file system performance, you do not need to run them on a dedicated machine. They are typically running on the cluster master node.

Each BeeGFS service (including the client) comes with:

  • a config file: /etc/beegfs/beegfs-X

  • systemd units

  • an init script: /etc/init.d/beegfs-X

Installation does not start any of the services directly because there are a few configuration options that need to be set before the first startup. These will be described in the following sections.

Init scripts

Warning

Init scripts are replaced by systemd on most distributions and are only required by older operating systems installations. The only exception is the client script, which is used to trigger automatic client building and running (see below).

After installation, all services will be set to automatic startup at system boot. If you do not want to run BeeGFS services automatically during system boot, use e.g. the chkconfig tool to disable automatic startup.

Another way to disable BeeGFS services startup would be to set START_SERVICENAME="NO" in the corresponding service configuration file (/etc/default/beegfs-X). Note that this would have the side effect of completely disabling init script startup of the service, not only during system boot.

Building the Client Kernel Module

Most parts of BeeGFS come pre-built and ready to use when you install the packages. However, the client kernel module of BeeGFS still needs to be compiled for your specific Linux kernel version.

In addition to that, it might be necessary to install the library package for RDMA support (libbegfs-ib), which contains the RDMA abstraction layer of BeeGFS for userspace applications and services.

BeeGFS Client Kernel Module

BeeGFS client kernel modules are automatically built on startup of the beegfs-client service (/etc/init.d/beegfs-client) for the currently running Linux kernel, so no manual build step is required. The init script will also detect client package updates and kernel updates and automatically runs a client module rebuild on next startup.

Unlike in earlier versions, RDMA support is available by default.

If you want to use a thirdparty driver though, you have to specify it in the file /etc/beegfs/beegfs-client-autobuild.conf like this: buildArgs=-j8 OFED_INCLUDE_PATH=/usr/src/openib/include.

To force a rebuild of the client kernel modules, you need to be root on a client node and then call:

# /etc/init.d/beegfs-client rebuild

After a successful module build, the beegfs-client needs to be restarted:

# /etc/init.d/beegfs-client restart

If the client module is unable to load, dmesg (or /var/log/messages) might provide useful information.

BeeGFS RDMA Library for Server Services

Note

This section is only relevant if you are using InfiniBand or RoCE hardware. If not, just skip to the next section.

If you have your hardware and the correspondent drivers installed, then it is only necessary to install the libbeegfs-ib package to use RDMA with the server services (beegfs-meta and beegfs-storage).

Basic Configuration

There are three configuration settings that are mandatory to be set for a specific environment. The following table shows the three settings and the configuration files, in which they will be applied:

Basic Settings and Corresponding Configuration Files

Setting

Configuration File (in /etc/beegfs)

Storage Paths

  • beegfs-mgmtd.conf

  • beegfs-meta.conf

  • beegfs-storage.conf

Management Host

  • beegfs-meta.conf

  • beegfs-storage.conf

  • beegfs-client.conf

Client Mount Directory

  • beegfs-mounts.conf

These are the three options that need to be set before you run the services.

  • Storage Paths

    Set the value of the storeMgmtDirectory/storeMetaDirectory/storeStorageDirectory option to configure where BeeGFS should store its internal data. BeeGFS stores all data in a subdirectory of an existing partition that has been formatted with any of the standard local Linux file systems (typically XFS for storage servers and ext4 for metadata servers).

    Note that different services on the same machine cannot share the same storage directory, so different directories have to be used, e.g. /data/beegfs-meta for metadata servers and /data/beegfs-storage for storage servers.

  • Management Host

    Set the value of the sysMgmtdHost option in the configuration files of the metadata servers, storage servers, clients, and the optional mon to the IP address or hostname of the management daemon. This will enable automatic server discovery for the clients.

  • Client Mount Directory

    The client mount file consists of two space-separated values. The first value is the directory where you want to mount the file system, and the second value is the client configuration file for this mount point. You will typically have a line like this in the beegfs-mounts.conf file:

    /mnt/beegfs /etc/beegfs/beegfs-client.conf

    It is also possible to specify multiple mount/config entries in this file (one mount/config pair per line) if you need to mount different BeeGFS instances on the same client. Optionally, you can also inform mount options for each mount/config entry. In the example below, the first line mounts a writable BeeGFS FS on /mnt/scratch and a read-only BeeGFS FS on /mnt/software.

    /mnt/scratch /etc/beegfs/beegfs-client-scratch.conf beegfs rw
/mnt/software /etc/beegfs/beegfs-client-software.conf beegfs ro

When you have made the changes to the configuration files, it is time to start the BeeGFS services.

Connection authentication

We strongly recommend to configure connection authentication with a connAuthFile at this point. Please see Authentication for more information. If connection authentication is not needed, connDisableAuthentication has to be set to true on all nodes.

Service Startup

The services will register themselves at the management server, so clients will automatically know about all available servers. The instructions below are for single instances. If you want to run multiple instances of one service on the same machine, please refer to Multi Mode.

Note

BeeGFS clients have a mount sanity check and cancel a mount operation if servers are unreachable. If you want to mount even with unreachable servers, set sysMountSanityCheckMS=0 in the file beegfs-client.conf.

Use systemctl start beegfs-<service> command to start the services.

Advanced Configuration and Tuning

Runtime configuration options can be queried and changed with the BeeGFS Control Tool (beegfs-ctl command on the client nodes). This tool is part of the beegfs-utils package.

The beegfs-ctl tool reads a beegfs-client.conf file from the default location, if it exists. While the tool can also be used without a client configuration file, it is usually much more convenient to use when such a basic client configuration exists on the corresponding machine.

On a client with an active BeeGFS mount, several configuration options can also be queried through procfs entries (see /proc/fs/beegfs).

There are also several convenience helper scripts contained in the beegfs-utils package (e.g. /usr/bin/beegfs-net to show the currently established client connections), which either call beegfs-ctl internally or read output from /proc/fs/beegfs.

Static configuration options can be set by editing the corresponding configuration files (/etc/beegfs/beegfs-X.conf). If you haven’t done so yet, you might want to take a look at these files to find out which options you have in general. There is a description for each of the settings in the corresponding configuration file.

See also

Prevent accidental registration of new nodes/targets

BeeGFS contains some safety options that you might want to set when your initial file system setup is complete: storeAllowFirstRunInit=false forbids daemon startup if the storage directory is empty, e.g. because the corresponding RAID volume is not mounted properly. sysAllowNewServers=false in beegfs-mgmtd.conf forbids registration of new servers (e.g. to make sure a new server is not registered accidentally in a production environment). sysMountSanityCheckMS can be used to let the client refuse mounting the file system if not all servers are reachable. (You will typically want to disable this option for clients that are running together with server daemons on the same machine.)

Backups

We strongly recommend to setup a backup solution like rsync, in order to backup the management target (often /beegfs/mgmtd/) to a safe external location. Please keep in mind that RAID or buddy mirroring is no replacement for an external backup.

See also

Backup