Quick Start Guide¶
This section provides example commands for the manual installation procedure. For general information on manual setup, please have a look at Advanced Topics.
In this example, we use the following hardware and software configuration:
Software: RHEL 8 or similar on all nodes
Host Services: (see note on dedicated hosts below) -
node01: Management Server -
node02: Metadata Server -
node03: Storage Server -
In this example, we are using dedicated hosts for all BeeGFS services. This is just to show the different installation steps for each service. BeeGFS allows running any combination of services (including client and storage/metadata service) on the same machine.
Especially the management is not performance-critical and thus are typically not running on dedicated machines.
Starting over with a fresh installation
In case you already had BeeGFS installed on one or more of your nodes (for testing, for example) and want to start over with a completely fresh system, you need to make sure that you use fresh data directories for all BeeGFS services.
Step 1: Package Download and Installation¶
First, the BeeGFS package repository file for your distribution needs to be downloaded. Repository files for several distributions are available at the BeeGFS package repository.
For our RHEL 8 example, we download this repository file and store it in the directory /etc/yum.repos.d on all nodes:
$ ssh root@node0X wget -O /etc/yum.repos.d/beegfs_rhel8.repo https://www.beegfs.io/release/beegfs_7.2.4/dists/beegfs-rhel8.repo
Now we can install the packages from the repository:
$ ssh root@node01 yum install beegfs-mgmtd # management service $ ssh root@node02 yum install beegfs-meta libbeegfs-ib # metadata service; libbeegfs-ib is only required for RDMA $ ssh root@node03 yum install beegfs-storage libbeegfs-ib # storage service; libbeegfs-ib is only required for RDMA $ ssh root@node04 yum install beegfs-client beegfs-helperd beegfs-utils # client and command-line utils
beegfs-helperd contains the userspace helper daemon for the client, which
provides logging and DNS lookup functionality to the client module. The
package contains e.g., the
beegfs-ctl command line tool which can show statistics and
perfom administrative tasks.
To enable support for remote direct memory access (RDMA) based on the OFED ibverbs API, please install the additional libbeegfs-ib package. BeeGFS will automatically enable RDMA on startup if the corresponding hardware and drivers are present.
Now that all services are installed, the next step is to configure the automatic client module build.
Step 2: Client Kernel Module Autobuild¶
This step is only relevant if you have RDMA-capable network hardware (InfiniBand, Omni-Path, RoCE). Otherwise, skip this step.
To enable support for remote direct memory access (RDMA) based on the OFED ibverbs API, the corresponding BeeGFS client kernel module build parameters need to be added. (Use your favorite text file editor, we will use vim below.)
$ ssh node04 $ vim /etc/beegfs/beegfs-client-autobuild.conf
Find the option “buildArgs” and set it to enabled ibverbs support:
If you installed separate OFED kernel modules, add the
buildArgs=-j8 BEEGFS_OPENTK_IBVERBS=1 OFED_INCLUDE_PATH=/usr/src/openib/include
OFED_INCLUDE_PATH is only required if you installed separate kernel driver
modules. If you are not sure, check the returned path information of this command:
$ modinfo ib_core
Separate OFED modules are usually located in the
Step 3: Basic Configuration¶
Before we can run the services, we need to make a few basic settings.
The management service needs to know where it can store its data. It will only store node information like connectivity data, so it will not require much storage space, and its data access is not performance critical. This service does not require a dedicated machine.
$ ssh root@node01 $ /opt/beegfs/sbin/beegfs-setup-mgmtd -p /data/beegfs/beegfs_mgmtd
The metadata service needs to know where it can store its data and where the management service is running. Typically, you will have multiple metadata services running on different machines.
Optionally, you can also define a custom numeric metadata service ID (range 1..65535). As this
service is running on a server with name
node02 in our example, we will also pick number 2 as
metadata service ID here.
$ ssh root@node02 $ /opt/beegfs/sbin/beegfs-setup-meta -p /data/beegfs/beegfs_meta -s 2 -m node01
The metadata service will store metadata as extended attributes (xattr) on the underlying file system for performance reasons. Xattrs have to be supported and enabled on the underlying file system. Please see the chapter Metadata Node Tuning on how to enable extended attributes for ext4 file systems.
The storage service needs to know where it can store its data and how to reach the management server. Typically, you will have multiple storage services running on different machines and/or multiple storage targets (e.g., multiple RAID volumes) per storage service.
Optionally, you can also define a custom numeric storage service ID and numeric storage target ID (both in range 1..65535). As this service is running on a server with name node03 in our example, we will pick number 3 as ID for this storage service and we will use 301 as storage target ID to show that this is the first target (01) of storage service 3.
$ ssh root@node03 $ /opt/beegfs/sbin/beegfs-setup-storage -p /mnt/myraid1/beegfs_storage -s 3 -i 301 -m node01
To add a second storage target on this same machine:
$ /opt/beegfs/sbin/beegfs-setup-storage -p /mnt/myraid2/beegfs_storage -s 3 -i 302
The client needs to know where the management service is running.
$ ssh root@node04 $ /opt/beegfs/sbin/beegfs-setup-client -m node01
The client mount directory is defined in a separate configuration file.
This file will be used by the
beegfs-client service startup script.
By default, BeeGFS will be mounted to
Thus, you need to perform this step only if you want to mount the file system to a different location.
$ ssh root@node04 $ vim /etc/beegfs/beegfs-mounts.conf
The first entry defines the mount directory. The second entry refers to the corresponding configuration file for this mount point.
Step 4: Service Startup¶
BeeGFS services can be started in arbitrary order by using the corresponding
systemctl service scripts. All services create log files (
$ ssh root@node01 systemctl start beegfs-mgmtd $ ssh root@node02 systemctl start beegfs-meta $ ssh root@node03 systemctl start beegfs-storage $ ssh root@node04 systemctl start beegfs-helperd $ ssh root@node04 systemctl start beegfs-client
BeeGFS clients have a mount sanity check and cancel a mount operation if servers are unreachable.
If you want to mount even with unreachable servers, set
sysMountSanityCheckMS=0 in the file
Some Linux distributions enable SELinux by default. If you are seeing an “Access denied” error when you access the BeeGFS mount, read ‘Access denied’ error on the client, even with correct permissions.
Congratulations, your parallel file system is now up and running!
Step 5: Check Connectivity¶
This step is especially relevant if you have RDMA-capable network hardware (InfiniBand, Omni-Path, RoCE) to ensure that RDMA is used as the transport protocol. However, you can also check that BeeGFS is using the intended routes to the servers by using the commands below.
Check the detected network interfaces and transport protocols from a client node with the following commands:
$ ssh node04 $ beegfs-ctl --listnodes --nodetype=meta --nicdetails $ beegfs-ctl --listnodes --nodetype=storage --nicdetails $ beegfs-ctl --listnodes --nodetype=client --nicdetails $ beegfs-net # Displays connections the client is actually using $ beegfs-check-servers # Displays possible connectivity of the services $ beegfs-df # Displays free space and inodes of storage and metadata targets
Please check your log files (
/var/log/beegfs-X.log), if you do not see expected RDMA
connections and verify that the
libbeegfs-ib is installed.
Make sure that the interfaces are listed in your order of preference (i.e. the primary interface
should be listed first in the output of
beegfs-ctl --listnodes) and that you only see interfaces
that you want BeeGFS to use. (see Configure allowed network interfaces)
Note that BeeGFS clients establish connections only when needed and also drop idle connections, so
you might e.g. not see metadata server connections in
beegfs-net until you performed a metadata
operation on the client mount like e.g.,