Quick Start Guide

This section provides example commands for the manual installation procedure. For general information on manual setup, please have a look at Advanced Topics.

Example Setup

In this example, we use the following hardware and software configuration:

  • Software: RHEL 8 or similar on all nodes

  • Host Services: (see note on dedicated hosts below) - node01: Management Server - node02: Metadata Server - node03: Storage Server - node04: Client

  • Storage:

    • Storage servers with RAID-6 data partition formatted with XFS mounted to /data. (See Storage Node Tuning)

    • Metadata servers with RAID-1 data partition formatted with ext4 mounted to /data. (See Metadata Node Tuning)

Note

In this example, we are using dedicated hosts for all BeeGFS services. This is just to show the different installation steps for each service. BeeGFS allows running any combination of services (including client and storage/metadata service) on the same machine.

Especially the management is not performance-critical and thus are typically not running on dedicated machines.

Note

Starting over with a fresh installation

In case you already had BeeGFS installed on one or more of your nodes (for testing, for example) and want to start over with a completely fresh system, you need to make sure that you use fresh data directories for all BeeGFS services.

Step 1: Package Download and Installation

First, the BeeGFS package repository file for your distribution needs to be downloaded. Repository files for several distributions are available at the BeeGFS package repository.

For our RHEL 8 example, we download this repository file and store it in the directory /etc/yum.repos.d on all nodes:

$ ssh root@node0X wget -O /etc/yum.repos.d/beegfs_rhel8.repo https://www.beegfs.io/release/beegfs_7.2.3/dists/beegfs-rhel8.repo

Now we can install the packages from the repository:

$ ssh root@node01 yum install beegfs-mgmtd                               # management service
$ ssh root@node02 yum install beegfs-meta libbeegfs-ib                   # metadata service; libbeegfs-ib is only required for RDMA
$ ssh root@node03 yum install beegfs-storage libbeegfs-ib                # storage service; libbeegfs-ib is only required for RDMA
$ ssh root@node04 yum install beegfs-client beegfs-helperd beegfs-utils  # client and command-line utils

Note

The additional beegfs-helperd contains the userspace helper daemon for the client, which provides logging and DNS lookup functionality to the client module. The beegfs-utils package contains e.g., the beegfs-ctl command line tool which can show statistics and perfom administrative tasks.

To enable support for remote direct memory access (RDMA) based on the OFED ibverbs API, please install the additional libbeegfs-ib package. BeeGFS will automatically enable RDMA on startup if the corresponding hardware and drivers are present.

Now that all services are installed, the next step is to configure the automatic client module build.

Step 2: Client Kernel Module Autobuild

Note

This step is only relevant if you have RDMA-capable network hardware (InfiniBand, Omni-Path, RoCE). Otherwise, skip this step.

To enable support for remote direct memory access (RDMA) based on the OFED ibverbs API, the corresponding BeeGFS client kernel module build parameters need to be added. (Use your favorite text file editor, we will use vim below.)

$ ssh node04
$ vim /etc/beegfs/beegfs-client-autobuild.conf

Find the option “buildArgs” and set it to enabled ibverbs support:

buildArgs=-j8 BEEGFS_OPENTK_IBVERBS=1

If you installed separate OFED kernel modules, add the OFED_INCLUDE_PATH:

buildArgs=-j8 BEEGFS_OPENTK_IBVERBS=1 OFED_INCLUDE_PATH=/usr/src/openib/include

Note

Defining OFED_INCLUDE_PATH is only required if you installed separate kernel driver modules. If you are not sure, check the returned path information of this command: $ modinfo ib_core

Separate OFED modules are usually located in the /lib/modules/<kernel_version> directory.

Step 3: Basic Configuration

Before we can run the services, we need to make a few basic settings.

Management Service

The management service needs to know where it can store its data. It will only store node information like connectivity data, so it will not require much storage space, and its data access is not performance critical. This service does not require a dedicated machine.

$ ssh root@node01
$ /opt/beegfs/sbin/beegfs-setup-mgmtd -p /data/beegfs/beegfs_mgmtd

Metadata Service

The metadata service needs to know where it can store its data and where the management service is running. Typically, you will have multiple metadata services running on different machines.

Optionally, you can also define a custom numeric metadata service ID (range 1..65535). As this service is running on a server with name node02 in our example, we will also pick number 2 as metadata service ID here.

$ ssh root@node02
$ /opt/beegfs/sbin/beegfs-setup-meta -p /data/beegfs/beegfs_meta -s 2 -m node01

Note

The metadata service will store metadata as extended attributes (xattr) on the underlying file system for performance reasons. Xattrs have to be supported and enabled on the underlying file system. Please see the chapter Metadata Node Tuning on how to enable extended attributes for ext4 file systems.

Storage Service

The storage service needs to know where it can store its data and how to reach the management server. Typically, you will have multiple storage services running on different machines and/or multiple storage targets (e.g., multiple RAID volumes) per storage service.

Optionally, you can also define a custom numeric storage service ID and numeric storage target ID (both in range 1..65535). As this service is running on a server with name node03 in our example, we will pick number 3 as ID for this storage service and we will use 301 as storage target ID to show that this is the first target (01) of storage service 3.

$ ssh root@node03
$ /opt/beegfs/sbin/beegfs-setup-storage -p /mnt/myraid1/beegfs_storage -s 3 -i 301 -m node01

To add a second storage target on this same machine:

$ /opt/beegfs/sbin/beegfs-setup-storage -p /mnt/myraid2/beegfs_storage -s 3 -i 302

Client

The client needs to know where the management service is running.

$ ssh root@node04
$ /opt/beegfs/sbin/beegfs-setup-client -m node01

The client mount directory is defined in a separate configuration file. This file will be used by the beegfs-client service startup script. By default, BeeGFS will be mounted to /mnt/beegfs. Thus, you need to perform this step only if you want to mount the file system to a different location.

$ ssh root@node04
$ vim /etc/beegfs/beegfs-mounts.conf

The first entry defines the mount directory. The second entry refers to the corresponding configuration file for this mount point.

Step 4: Service Startup

BeeGFS services can be started in arbitrary order by using the corresponding init.d or systemctl service scripts. All services create log files (/var/log/beegfs-...).

$ ssh root@node01 systemctl start beegfs-mgmtd
$ ssh root@node02 systemctl start beegfs-meta
$ ssh root@node03 systemctl start beegfs-storage
$ ssh root@node04 systemctl start beegfs-helperd
$ ssh root@node04 systemctl start beegfs-client

Note

BeeGFS clients have a mount sanity check and cancel a mount operation if servers are unreachable. If you want to mount even with unreachable servers, set sysMountSanityCheckMS=0 in the file /etc/beegfs/beegfs-client.conf.

Note

Some Linux distributions enable SELinux by default. If you are seeing an “Access denied” error when you access the BeeGFS mount, read ‘Access denied’ error on the client, even with correct permissions.

Congratulations, your parallel file system is now up and running!

Step 5: Check Connectivity

Note

This step is especially relevant if you have RDMA-capable network hardware (InfiniBand, Omni-Path, RoCE) to ensure that RDMA is used as the transport protocol. However, you can also check that BeeGFS is using the intended routes to the servers by using the commands below.

Check the detected network interfaces and transport protocols from a client node with the following commands:

$ ssh node04

$ beegfs-ctl --listnodes --nodetype=meta --nicdetails
$ beegfs-ctl --listnodes --nodetype=storage --nicdetails
$ beegfs-ctl --listnodes --nodetype=client --nicdetails
$ beegfs-net                # Displays connections the client is actually using
$ beegfs-check-servers      # Displays possible connectivity of the services
$ beegfs-df                 # Displays free space and inodes of storage and metadata targets

Please check your log files (/var/log/beegfs-X.log), if you do not see expected RDMA connections and verify that the libbeegfs-ib is installed.

Make sure that the interfaces are listed in your order of preference (i.e. the primary interface should be listed first in the output of beegfs-ctl --listnodes) and that you only see interfaces that you want BeeGFS to use. (see Configure allowed network interfaces)

Note that BeeGFS clients establish connections only when needed and also drop idle connections, so you might e.g. not see metadata server connections in beegfs-net until you performed a metadata operation on the client mount like e.g., ls.