Quick Start Guide¶
This section provides example commands for the manual installation procedure. For general information on manual setup, please have a look at Advanced Topics.
Example Setup¶
In this example, we use the following hardware and software configuration:
Software: RHEL 8 or similar on all nodes
Host Services: (see note on dedicated hosts below) -
node01: Management Server -node02: Metadata Server -node03: Storage Server -node04: ClientStorage:
Storage servers with RAID-6 data partition formatted with XFS mounted to
/data. (See Storage Node Tuning)Metadata servers with RAID-1 data partition formatted with ext4 mounted to
/data. (See Metadata Node Tuning)
Note
In this example, we are using dedicated hosts for all BeeGFS services. This is just to show the different installation steps for each service. BeeGFS allows running any combination of services (including client and storage/metadata service) on the same machine.
Especially the management is not performance-critical and thus are typically not running on dedicated machines.
Note
Starting over with a fresh installation
In case you already had BeeGFS installed on one or more of your nodes (for testing, for example) and want to start over with a completely fresh system, you need to make sure that you use fresh data directories for all BeeGFS services.
Step 1: Package Download and Installation¶
First, the BeeGFS package repository file for your distribution needs to be downloaded. Repository files for several distributions are available at the BeeGFS package repository.
For our RHEL 8 example, we download this repository file and store it in the directory /etc/yum.repos.d on all nodes:
$ ssh root@node0X wget -O /etc/yum.repos.d/beegfs_rhel8.repo https://www.beegfs.io/release/beegfs_7.3.2/dists/beegfs-rhel8.repo
Now we can install the packages from the repository:
$ ssh root@node01 yum install beegfs-mgmtd # management service
$ ssh root@node02 yum install beegfs-meta libbeegfs-ib # metadata service; libbeegfs-ib is only required for RDMA
$ ssh root@node03 yum install beegfs-storage libbeegfs-ib # storage service; libbeegfs-ib is only required for RDMA
$ ssh root@node04 yum install beegfs-client beegfs-helperd beegfs-utils # client and command-line utils
Note
The additional beegfs-helperd contains the userspace helper daemon for the client, which
provides logging and DNS lookup functionality to the client module. The beegfs-utils
package contains e.g., the beegfs-ctl command line tool which can show statistics and
perfom administrative tasks.
To enable support for remote direct memory access (RDMA) based on the OFED ibverbs API, please install the additional libbeegfs-ib package. BeeGFS will automatically enable RDMA on startup if the corresponding hardware and drivers are present.
Now that all services are installed, the next step is to configure the automatic client module build.
Step 2: Client Kernel Module Autobuild¶
Note
This step is only relevant if you have RDMA-capable network hardware (InfiniBand, Omni-Path, RoCE). Otherwise, skip this step.
Unlike in earlier versions, RDMA support based on the OFED ibverbs API is available by default.
If you want to use a thirdparty driver though, you have to specify it in the file /etc/beegfs/beegfs-client-autobuild.conf like this: buildArgs=-j8 OFED_INCLUDE_PATH=/usr/src/openib/include and rebuild the client:
$ ssh node04
$ vim /etc/beegfs/beegfs-client-autobuild.conf
Add the OFED_INCLUDE_PATH to the buildArgs parameter:
buildArgs=-j8 OFED_INCLUDE_PATH=/usr/src/openib/include
Note
Defining OFED_INCLUDE_PATH is only required if you installed separate kernel driver
modules. If you are not sure, check the returned path information of this command:
$ modinfo ib_core
Separate OFED modules are usually located in the /lib/modules/<kernel_version> directory.
Step 3: Basic Configuration¶
Before we can run the services, we need to make a few basic settings.
Management Service¶
The management service needs to know where it can store its data. It will only store node information like connectivity data, so it will not require much storage space, and its data access is not performance critical. This service does not require a dedicated machine.
$ ssh root@node01
$ /opt/beegfs/sbin/beegfs-setup-mgmtd -p /data/beegfs/beegfs_mgmtd
Metadata Service¶
The metadata service needs to know where it can store its data and where the management service is running. Typically, you will have multiple metadata services running on different machines.
Optionally, you can also define a custom numeric metadata service ID (range 1..65535). As this
service is running on a server with name node02 in our example, we will also pick number 2 as
metadata service ID here.
$ ssh root@node02
$ /opt/beegfs/sbin/beegfs-setup-meta -p /data/beegfs/beegfs_meta -s 2 -m node01
Note
The metadata service will store metadata as extended attributes (xattr) on the underlying file system for performance reasons. Xattrs have to be supported and enabled on the underlying file system. Please see the chapter Metadata Node Tuning on how to enable extended attributes for ext4 file systems.
Storage Service¶
The storage service needs to know where it can store its data and how to reach the management server. Typically, you will have multiple storage services running on different machines and/or multiple storage targets (e.g., multiple RAID volumes) per storage service.
Optionally, you can also define a custom numeric storage service ID and numeric storage target ID (both in range 1..65535). As this service is running on a server with name node03 in our example, we will pick number 3 as ID for this storage service and we will use 301 as storage target ID to show that this is the first target (01) of storage service 3.
$ ssh root@node03
$ /opt/beegfs/sbin/beegfs-setup-storage -p /mnt/myraid1/beegfs_storage -s 3 -i 301 -m node01
To add a second storage target on this same machine:
$ /opt/beegfs/sbin/beegfs-setup-storage -p /mnt/myraid2/beegfs_storage -s 3 -i 302
Client¶
The client needs to know where the management service is running.
$ ssh root@node04
$ /opt/beegfs/sbin/beegfs-setup-client -m node01
The client mount directory is defined in a separate configuration file.
This file will be used by the beegfs-client service startup script.
By default, BeeGFS will be mounted to /mnt/beegfs.
Thus, you need to perform this step only if you want to mount the file system to a different location.
$ ssh root@node04
$ vim /etc/beegfs/beegfs-mounts.conf
The first entry defines the mount directory. The second entry refers to the corresponding configuration file for this mount point.
Connection authentication¶
We strongly recommend to configure connection authentication with a connAuthFile at this point.
Please see Authentication for more information. If connection authentication is
not needed, connDisableAuthentication has to be set to true on all nodes.
Step 4: Service Startup¶
BeeGFS services can be started in arbitrary order by using the corresponding init.d or
systemctl service scripts. All services create log files (/var/log/beegfs-...).
$ ssh root@node01 systemctl start beegfs-mgmtd
$ ssh root@node02 systemctl start beegfs-meta
$ ssh root@node03 systemctl start beegfs-storage
$ ssh root@node04 systemctl start beegfs-helperd
$ ssh root@node04 systemctl start beegfs-client
Note
BeeGFS clients have a mount sanity check and cancel a mount operation if servers are unreachable.
If you want to mount even with unreachable servers, set sysMountSanityCheckMS=0 in the file
/etc/beegfs/beegfs-client.conf.
Note
Some Linux distributions enable SELinux by default. If you are seeing an “Access denied” error when you access the BeeGFS mount, read ‘Access denied’ error on the client, even with correct permissions.
Congratulations, your parallel file system is now up and running!
Step 5: Check Connectivity¶
Note
This step is especially relevant if you have RDMA-capable network hardware (InfiniBand, Omni-Path, RoCE) to ensure that RDMA is used as the transport protocol. However, you can also check that BeeGFS is using the intended routes to the servers by using the commands below.
Check the detected network interfaces and transport protocols from a client node with the following commands:
$ ssh node04
$ beegfs-ctl --listnodes --nodetype=meta --nicdetails
$ beegfs-ctl --listnodes --nodetype=storage --nicdetails
$ beegfs-ctl --listnodes --nodetype=client --nicdetails
$ beegfs-net # Displays connections the client is actually using
$ beegfs-check-servers # Displays possible connectivity of the services
$ beegfs-df # Displays free space and inodes of storage and metadata targets
Please check your log files (/var/log/beegfs-X.log), if you do not see expected RDMA
connections and verify that the libbeegfs-ib is installed.
Make sure that the interfaces are listed in your order of preference (i.e. the primary interface
should be listed first in the output of beegfs-ctl --listnodes) and that you only see interfaces
that you want BeeGFS to use. (see Configure allowed network interfaces)
Note that BeeGFS clients establish connections only when needed and also drop idle connections, so
you might e.g. not see metadata server connections in beegfs-net until you performed a metadata
operation on the client mount like e.g., ls.
