Running BeeGFS in Containers

Overview

Traditionally, BeeGFS server services are installed using a particular Linux distribution’s package manager (yum, apt, zypper) and managed (start/stop) by systemd. An alternative approach is to package applications into a container image that contains all of the dependencies needed by that application. This can drastically simplify installing and managing applications, providing a consistent runtime environment for the application that can be isolated from other processes running on the system.

Note

Starting with BeeGFS 7.4.1, the hosting service for BeeGFS images has moved to GitHub Container Registry which provides free usage for public packages ensuring users aren’t restricted by Docker Hub rate limits. Simply use the image names “ghcr.io/thinkparq/beegfs-<SERVICE>” going forward.

Definitions

Base OS: The operating system running on a physical or virtual machine that is used to run containers.

Container Image: Lightweight, standalone, executable package of software that includes everything needed to run an application.

Container: Running instance of a container image.

Prerequisites

BeeGFS container images are available through GitHub Container registry and are distributed using the Docker Image Manifest Version 2, Schema 2 format, which means they should generally work with any container engine or container runtime that supports either the Docker or Open Container image (OCI) formats. Depending on your use case you will typically either use a high-level container engine like Docker or Podman, a high-level runtime like Containerd, or a low level runtime like runc to actually run BeeGFS images.

For simplicity this guide will focus on how to get up and running using Docker. To follow along you will need to install the correct server version of Docker Engine based on your Linux distribution and architecture. Note there is a distinction between the open source Docker Engine available for various Linux platforms and Docker Desktop which requires a subscription to use on Linux, MacOS, and Windows. This guide only requires use of the open source Docker Engine and the following steps were specifically validated using Ubuntu 20.04 and Docker version 20.10.19.

This guide expects network interface(s) including IP addresses are already configured on the host used to run BeeGFS containers. These can be externally facing interfaces (optionally RDMA enabled), or the loopback device if you just want to test on an isolated system. If you want to test RDMA the appropriate packages/modules must already be installed/configured.

It may also be helpful to be familiar with the standard BeeGFS quick start guide as the documentation will occasionally refer back to standard installation steps to help the reader understand how those translate to running BeeGFS in containers.

Downloading/Pulling BeeGFS Images

Instead of configuring your package manager (apt, dnf, yum, etc) to download/install BeeGFS packages, provided the servers you are using have internet access simply run:

docker pull ghcr.io/thinkparq/beegfs-mgmtd:latest
docker pull ghcr.io/thinkparq/beegfs-meta:latest
docker pull ghcr.io/thinkparq/beegfs-storage:latest

Optionally to pull images for a specific version, specify it like:

docker pull ghcr.io/thinkparq/beegfs-mgmtd:7.4.1
docker pull ghcr.io/thinkparq/beegfs-meta:7.4.1
docker pull ghcr.io/thinkparq/beegfs-storage:7.4.1

Warning

When running in production it is strongly recommended to specify a version tag to avoid the file system upgrading unexpectedly.

If the servers you are using do not have internet access first pull the image on a machine with internet access, then save them as tar files using docker save so they can be copied to air-gapped servers and loaded with docker load.

Note

The earliest available BeeGFS images are for BeeGFS 7.3.1.

Image Verification

All BeeGFS server Docker images with tags latest and version numbers are signed with Cosign.

Note

With the move to GitHub Container Registry BeeGFS images will no longer be signed using Docker Content Trust.

Cosign

Cosign signatures can be verified with the Cosign application. The Cosign verify command is used to verify the Docker images’ signatures $ cosign verify --key <pub key path> <image uri>. All BeeGFS server Docker images are signed with the following public key:

-----BEGIN PUBLIC KEY-----
MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEAR5OydewbwaPKjd/5SqQ4CbGOH68
E5RSXAH6n+WRe3NL0UN7aeI2kAoqyI1wOG0TdCx0RSb4q0KrKSSjVWsVXg==
-----END PUBLIC KEY-----

For example, to verify the ghcr.io/thinkparq/beegfs-mgmtd image with tag 7.4.1:

cosign verify --key cosign_bee_containers.pub ghcr.io/thinkparq/beegfs-mgmtd:latest

   Verification for ghcr.io/thinkparq/beegfs-mgmtd:latest --
   The following checks were performed on each of these signatures:
     - The cosign claims were validated
     - Existence of the claims in the transparency log was verified offline
     - The signatures were verified against the specified public key

   [{"critical":{"identity":{"docker-reference":"ghcr.io/thinkparq/beegfs-mgmtd"},"image":{"docker-manifest-digest":"sha256:085ac76077cdddf1e79773cd7c431fe326c27eccc6cac519f29d4608808e470e"},"type":"cosign container image signature"},"optional":{"Bundle":{"SignedEntryTimestamp":"MEYCIQDRBdyKec9RWCqB/oUkPzc8IhnthTOswlBIPYDTHrqhzwIhALQWtt9vdF4DlSS3HPGb45HO/s+khmpT1rv5xQkzrjoW","Payload":{"body":"eyJhcGlWZXJzaW9uIjoiMC4wLjEiLCJraW5kIjoiaGFzaGVkcmVrb3JkIiwic3BlYyI6eyJkYXRhIjp7Imhhc2giOnsiYWxnb3JpdGhtIjoic2hhMjU2IiwidmFsdWUiOiJmODNlNDRjODkwZmE0ZDNkZDE2ZTM1MTBjYzk4ZjcyNDRhYjIyZTFlZTliNzBlZWNkZDhjNjBhZDNmM2E5YTNlIn19LCJzaWduYXR1cmUiOnsiY29udGVudCI6Ik1FVUNJUURINkoyYnNBTFVwd0xBY29DU3VMeEFPTFErWnY1RFp0d1dHREYyUmJiUTRRSWdFbEc0SjYwZkwwVVg3N1JUM0RkUVhKb01ZME4zdTZ4NGNDTytaWUlzZkU4PSIsInB1YmxpY0tleSI6eyJjb250ZW50IjoiTFMwdExTMUNSVWRKVGlCUVZVSk1TVU1nUzBWWkxTMHRMUzBLVFVacmQwVjNXVWhMYjFwSmVtb3dRMEZSV1VsTGIxcEplbW93UkVGUlkwUlJaMEZGUVZJMVQzbGtaWGRpZDJGUVMycGtMelZUY1ZFMFEySkhUMGcyT0FwRk5WSlRXRUZJTm00clYxSmxNMDVNTUZWT04yRmxTVEpyUVc5eGVVa3hkMDlITUZSa1EzZ3dVbE5pTkhFd1MzSkxVMU5xVmxkelZsaG5QVDBLTFMwdExTMUZUa1FnVUZWQ1RFbERJRXRGV1MwdExTMHRDZz09In19fX0=","integratedTime":1695912793,"logIndex":39180748,"logID":"c0d23d6ad406973f9559f3ba2d1ca01f84147d8ffc5b8445c224f98b9591801d"}},"ref":"6a205c621f37eb2e4d16a068f35981963fa8f1dc","repo":"ThinkParQ/beegfs-containers","run":"6340586990"}}]

Quick Start: BeeGFS Containers

This section provides an example to allow you to quickly get started with BeeGFS containers. More detail on what these commands do and alternative/more secure options will be provided in the following sections.

Note

The quick start example stores data outside the container under /tmp. Note this may be wiped after a reboot on some Linux distributions.

Steps

Step 1: Define shell variables with the interface(s) BeeGFS should use for communication, and the IP that should be used for the BeeGFS management service (typically the IP of the first interface).

export beegfs_connInterfacesList=<INTERFACES> # Example: ib0,ib1
export beegfs_sysMgmtdHost=<IPv4 ADDRESS> # Example: 100.127.1.39
set +o history
export beegfs_connAuthSecret=mysecret
set -o history

Step 2: Start the BeeGFS management service. The --volume argument accepts the absolute path and name of a directory where the file system can store its management data on the host machine.

docker run --name="beegfs-mgmtd" --privileged --network="host" \
  --volume=/tmp/mgmt_tgt_mgmt01:/mnt/mgmt_tgt_mgmt01 \
  --env beegfs_setup_1="beegfs-setup-mgmtd -p /mnt/mgmt_tgt_mgmt01 -C -S mgmt_tgt_mgmt01" \
  --env CONN_AUTH_FILE_DATA=$beegfs_connAuthSecret \
  -d ghcr.io/thinkparq/beegfs-mgmtd:latest \
  storeAllowFirstRunInit=false \
  connInterfacesList=$beegfs_connInterfacesList storeMgmtdDirectory=/mnt/mgmt_tgt_mgmt01

Step 3: Start the BeeGFS metadata service. The --volume argument accepts the absolute path and name of a directory where the file system can store its metadata on the host machine.

docker run --name="beegfs-meta" --privileged --network="host" \
  --volume=/tmp/meta_01_tgt_0101:/mnt/meta_01_tgt_0101 \
  --env beegfs_setup_1="beegfs-setup-meta -C -p /mnt/meta_01_tgt_0101 -s 1 -S meta_01" \
  --env CONN_AUTH_FILE_DATA=$beegfs_connAuthSecret \
  -d ghcr.io/thinkparq/beegfs-meta:latest \
  sysMgmtdHost=$beegfs_sysMgmtdHost storeAllowFirstRunInit=false \
  connInterfacesList=$beegfs_connInterfacesList storeMetaDirectory=/mnt/meta_01_tgt_0101

Step 4: Start the BeeGFS storage service. The --volume argument accepts the absoute path to a storage target. A storage target is a directory where the file system can store raw user file contents on the host machine.

docker run --name="beegfs-storage" --privileged --network="host" \
  --volume=/tmp/stor_01_tgt_101:/mnt/stor_01_tgt_101 --volume=/tmp/stor_01_tgt_102:/mnt/stor_01_tgt_102 \
  --env beegfs_setup_1="beegfs-setup-storage -C -p /mnt/stor_01_tgt_101 -s 1 -S stor_01_tgt_101 -i 101" \
  --env beegfs_setup_2="beegfs-setup-storage -C -p /mnt/stor_01_tgt_102 -s 1 -S stor_01_tgt_101 -i 102" \
  --env CONN_AUTH_FILE_DATA=$beegfs_connAuthSecret \
  -d ghcr.io/thinkparq/beegfs-storage:latest \
  sysMgmtdHost=$beegfs_sysMgmtdHost storeAllowFirstRunInit=false \
  connInterfacesList=$beegfs_connInterfacesList storeStorageDirectory=/mnt/stor_01_tgt_101,/mnt/stor_01_tgt_102

Step 5: Verify the BeeGFS metadata and storage service were able to register with the management service.

# docker logs beegfs-mgmtd
[...]
(1) 21:35:05 DGramLis [HBeatMgr] >> New root directory metadata node: beegfs-meta meta_01 [ID: 1]
(2) 21:35:05 DGramLis [Node registration] >> New node: beegfs-meta meta_01 [ID: 1]; RDMA; Source: 100.128.1.39
(2) 21:35:08 DirectWorker1 [Change consistency states] >> Metadata node is coming online. ID: 1
(2) 21:35:08 XNodeSync [Assign node to capacity pool] >> Metadata node capacity pool assignment updated. NodeID: 1; Pool: Normal; Reason: Free capacity threshold
(3) 21:35:23 Worker1 [RegisterTargetMsgEx.cpp:54] >> New target registered. targetID: 0-634DCA9B-1; newTargetNumID: 101
(3) 21:35:23 Worker2 [RegisterTargetMsgEx.cpp:54] >> New target registered. targetID: 1-634DCA9B-1; newTargetNumID: 102
(2) 21:35:23 DGramLis [Node registration] >> New node: beegfs-storage stor_01_tgt_101 [ID: 1]; RDMA; Source: 100.128.1.39
(2) 21:35:26 DirectWorker1 [Change consistency states] >> Storage target is coming online. ID: 101
(2) 21:35:26 DirectWorker1 [Change consistency states] >> Storage target is coming online. ID: 102

Step 6 (optional): Mount the BeeGFS file system on one or more clients.

Step 7 (optional): Stop the BeeGFS containers and clean up.

docker stop beegfs-mgmtd beegfs-meta beegfs-storage; \
docker rm beegfs-mgmtd beegfs-meta beegfs-storage; \
rm -rf /tmp/mgmt_tgt_mgmt01/ && rm -rf /tmp/meta_01_tgt_0101/ && rm -rf
/tmp/stor_01_tgt_101 && rm -rf /tmp/stor_01_tgt_102/; \
unset beegfs_connInterfacesList beegfs_sysMgmtdHost beegfs_connAuthSecret

Advanced Topics

This section goes in-depth on how to execute BeeGFS Docker images using Docker and Docker Compose.

Running BeeGFS in Containers using Docker Run

The docker run command allows you to create and execute containers based on existing Docker images. Refer to the full Docker run reference for additional details on the flags described in this section.

To setup BeeGFS you first need to initialize the directories used to store data (management, metadata, and storage targets). This is typically done by executing the beegfs-setup-<SERVICE> command, and for the storage service multiple commands may be needed if there are multiple targets. When running BeeGFS in containers the setup commands are instead specified as environmental variable(s), then startup and initialization scripts inside the container handle setup if the provided targets have not been initialized. This is also how we’ll specify a shared secret so we can enable connection authentication. The first part of the command to run the BeeGFS management service looks like:

docker run --name="beegfs-mgmtd" \
  --env beegfs_setup_1="beegfs-setup-mgmtd -p /data/mgmt_tgt_mgmt01 -C -S mgmt_tgt_mgmt01" \
  --env CONN_AUTH_FILE_DATA="shared_secret_string" \

Note

For containers it is important to always use -S and specify the string ID. By default BeeGFS uses the hostname, which for containers can change when they are recreated, or if the network mode is set to “host” will match the base OS hostname which would break things if multiple containers of the same type are running on the same server.

By default anything written to the /mnt/mgmt_tgt_mgmt01 directory won’t persist if the container is removed (for example if the server reboots). To avoid this we can provide one or more locations outside the container (already mounted to the base OS) that will persist beyond the life of the container. When running BeeGFS in production this will likely represent some faster storage device, for example an LUN from an internal RAID adapter or presented from an external storage system over a storage area network (SAN). Regardless of the backing device, after it is formatted/mounted in the base OS, you can bind mount it into the container by specifying:

--volume=/mnt/mgmt_tgt_mgmt01:/data/mgmt_tgt_mgmt01 \

Typically we want to expose each BeeGFS service to the outside world so it can be accessed by clients and services running on multiple servers. While there are a number of ways to configure networking in Docker, the simplest and most performant is to simply use the host’s network stack inside the container:

--network="host" \

Note

The use of the host network mode gives the container full access to local system services such as D-bus, and is consider insecure if hard separation between a BeeGFS container and other containers or processes running in the base OS is required. However for many common BeeGFS use cases this approach may be acceptable. For example if a particular server is only serving one BeeGFS file system and not hosting other applications (or BeeGFS file systems) in a multi-tenant environment. If separation is required it is possible to use other network modes to allow external access in a more secure manner, for example using the default Docker bridge network mode. If RDMA is in use the performance impact of alternate approaches should be minimal, but have not been extensively tested or evaluated with BeeGFS.

Next we’ll specify the name of the container image we want to execute using the -d flag to start in detached mode:

-d ghcr.io/thinkparq/beegfs-mgmtd:latest \

Typically any “runtime” arguments would be set at /etc/beegfs/beegfs-<SERVICE>.conf, but when we run BeeGFS in containers we just specify them as arguments to the container:

storeAllowFirstRunInit=false connDisableAuthentication=true \
connInterfacesList=<COMMA_SEPARATED_LIST> storeMgmtdDirectory=/data/mgmt_tgt_mgmt01

Note

The value for storeMgmtdDirectory must match what was specified in the -p section of the setup command (/mnt/mgmt_tgt_mgmt01).

If we put it all together the docker run command for the management service looks like:

docker run --name="beegfs-mgmtd" \
  --env beegfs_setup_1="beegfs-setup-mgmtd -p /data/mgmt_tgt_mgmt01 -C -S mgmt_tgt_mgmt01" \
  --env CONN_AUTH_FILE_DATA="shared_secret_string" \
  --volume=/mnt/mgmt_tgt_mgmt01:/data/mgmt_tgt_mgmt01 \
  --network="host" \
  -d ghcr.io/thinkparq/beegfs-mgmtd:latest \
  storeAllowFirstRunInit=false connDisableAuthentication=true \
  connInterfacesList=<COMMA_SEPARATED_LIST> storeMgmtdDirectory=/data/mgmt_tgt_mgmt01

The command for the metadata service will look similar to the management service:

docker run --name="beegfs-meta" \
  --env beegfs_setup_1="beegfs-setup-meta -C -p /data/meta_01_tgt_0101 -s 1 -S meta_01" \
  --env CONN_AUTH_FILE_DATA="shared_secret_string" \
  --volume=/mnt/meta_01_tgt_0101:/data/meta_01_tgt_0101 \
  --network="host" \
  -d ghcr.io/thinkparq/beegfs-meta:latest \
  sysMgmtdHost=<MGMT_IP> storeAllowFirstRunInit=false connDisableAuthentication=true \
  connInterfacesList=<COMMA_SEPARATED_LIST> storeMetaDirectory=/data/meta_01_tgt_0101

In the command for the storage service we can specify multiple targets including additional setup commands, volumes, and extending the storeStorageDirectory list:

docker run --name="beegfs-storage" \
  --env beegfs_setup_1="beegfs-setup-storage -C -p /data/stor_01_tgt_101 -s 1 -S stor_01_tgt_101 -i 101" \
  --env beegfs_setup_2="beegfs-setup-storage -C -p /data/stor_01_tgt_102 -s 1 -S stor_01_tgt_101 -i 102" \
  --env CONN_AUTH_FILE_DATA="shared_secret_string" \
  --volume=/mnt/stor_01_tgt_101:/data/stor_01_tgt_101 --volume=/mnt/stor_01_tgt_102:/data/stor_01_tgt_102 \
  --network="host" \
  -d ghcr.io/thinkparq/beegfs-storage:latest \
  sysMgmtdHost=<MGMT_IP> storeAllowFirstRunInit=false connDisableAuthentication=true \
  connInterfacesList=<COMMA_SEPARATED_LIST> storeStorageDirectory=/data/stor_01_tgt_101,/data/stor_01_tgt_102

Note

Before running the above commands ensure to adjust the sysMgmtdHost and connInterfacesList for each service to match your system.

Using RDMA with BeeGFS Containers

By default Docker containers are “unprivileged” and cannot perform functionality necessary to take advantage of RDMA-capable hardware like NVIDIA InfiniBand adapters. For simplicity the quick start section used Docker’s --privileged flag, which gives all Linux capabilities to the container and allows RDMA to work. A more secure approach is to add select capabilities to the container and provide access to specific devices which will be demonstrated in this section. These additional docker run flags will need to be specified before the Docker image name.

First we need to add the IPC_LOCK Linux capability to the container, this is required so BeeGFS is able to lock memory used for RDMA connections:

--cap-add=IPC_LOCK \

Next we need to give the container access to the InfiniBand devices we want BeeGFS to use. By default the container will be able to read, write, and mknod these devices. BeeGFS will need access to the RDMA communication manager (rdma_cm) and each of the InfiniBand interfaces (devices nodes indicated by uverbsX) you want the container to use:

--device=/dev/infiniband/rdma_cm --device=/dev/infiniband/uverbs0 --device=/dev/infiniband/uverbs1 \

If we put it all together for the management service, the command looks like:

docker run --name="beegfs-mgmtd" \
  --env beegfs_setup_1="beegfs-setup-mgmtd -p /data/mgmt_tgt_mgmt01 -C -S mgmt_tgt_mgmt01" \
  --env CONN_AUTH_FILE_DATA="shared_secret_string" \
  --volume=/mnt/mgmt_tgt_mgmt01:/data/mgmt_tgt_mgmt01 \
  --network="host" \
  --cap-add=IPC_LOCK \
  --device=/dev/infiniband/rdma_cm --device=/dev/infiniband/uverbs0 --device=/dev/infiniband/uverbs1 \
  -d ghcr.io/thinkparq/beegfs-mgmtd:latest \
  storeAllowFirstRunInit=false \
  connInterfacesList=<COMMA_SEPARATED_LIST> storeMgmtdDirectory=/data/mgmt_tgt_mgmt01

For the metadata service the command looks like:

docker run --name="beegfs-meta" \
  --env beegfs_setup_1="beegfs-setup-meta -C -p /data/meta_01_tgt_0101 -s 1 -S meta_01" \
  --env CONN_AUTH_FILE_DATA="shared_secret_string" \
  --volume=/mnt/meta_01_tgt_0101:/data/meta_01_tgt_0101 \
  --network="host" \
  --cap-add=IPC_LOCK \
  --device=/dev/infiniband/rdma_cm --device=/dev/infiniband/uverbs0 --device=/dev/infiniband/uverbs1 \
  -d ghcr.io/thinkparq/beegfs-meta:latest \
  sysMgmtdHost=<MGMT_IP> storeAllowFirstRunInit=false \
  connInterfacesList=<COMMA_SEPARATED_LIST> storeMetaDirectory=/data/meta_01_tgt_0101

For the storage service the command looks like:

docker run --name="beegfs-storage" \
  --env beegfs_setup_1="beegfs-setup-storage -C -p /data/stor_01_tgt_101 -s 1 -S stor_01_tgt_101 -i 101" \
  --env beegfs_setup_2="beegfs-setup-storage -C -p /data/stor_01_tgt_102 -s 1 -S stor_01_tgt_101 -i 102" \
  --env CONN_AUTH_FILE_DATA="shared_secret_string" \
  --volume=/mnt/stor_01_tgt_101:/data/stor_01_tgt_101 --volume=/mnt/stor_01_tgt_102:/data/stor_01_tgt_102 \
  --network="host" \
  --cap-add=IPC_LOCK \
  --device=/dev/infiniband/rdma_cm --device=/dev/infiniband/uverbs0 --device=/dev/infiniband/uverbs1 \
  -d ghcr.io/thinkparq/beegfs-storage:latest \
  sysMgmtdHost=<MGMT_IP> storeAllowFirstRunInit=false \
  connInterfacesList=<COMMA_SEPARATED_LIST> storeStorageDirectory=/data/stor_01_tgt_101,/data/stor_01_tgt_102

Note

Before running the above commands adjust the specified –devices, sysMgmtdHost, and connInterfacesList for each service to match your system.

Automatically Starting Containers after a Reboot

In a traditional BeeGFS deployment services are automatically configured to start using systemd. By default containers will not automatically restart if they exit (or the system reboots) unless you specify a restart policy as part of the Docker run command.

Additional Docker Commands

Viewing a Container’s Logs

BeeGFS containers will not log to /var/log either inside the container or in the base OS. Instead use docker logs <CONTAINER_NAME> to view the logs for a particular service.

Executing Commands in a Container

If you need to execute a command in the container do not use “docker attach” since we didn’t allocate a pseudo-tty (-t) or tell the container to listen on stdin (-i) when we started it. Instead use “docker exec” to run additional commands inside the container.

For example if you want a bash shell inside the container run docker exec -it <container-name> /bin/bash`. From here you can run any available commands inside the container and use ``exit when you want to return to the base OS leaving the container running in the background.

Note

The shell prompt (root@<HOSTNAME>:/#) won’t change if you’re using the -network="host" option. You can verify you’re attached to the container by running “cat /proc/1/cgroup” and verifying the paths container /docker/.

Stopping a Container

To stop a container run docker stop <CONTAINER_NAME>

Removing a Container

When you stop a container it will not automatically be removed. This can cause issues if say you need to update configuration. To remove a container run docker rm <CONTAINER_NAME>.

Running BeeGFS containers using Docker Compose

Docker Compose is a tool designed to simplify defining how multiple containers should run. With Compose you store the configuration representing how you want your containers to run in a YAML file that can manage multiple containers with a single command. This is drastically simpler than having to write/remember long Docker run commands, and also allows you to version control your container configuration. Each of the flags to Docker run has a corresponding configuration option in Docker Compose, see the Compose file reference for full details.

Defining a Docker Compose File

The above RDMA enabled BeeGFS containers example can be converted to a Docker Compose file as follows:

version: '3'
services:

  beegfs-mgmtd:
    container_name: beegfs-mgmtd
    image: ghcr.io/thinkparq/beegfs-mgmtd:latest
    environment:
      - beegfs_setup_1="beegfs-setup-mgmtd -p /data/mgmt_tgt_mgmt01 -C -S mgmt_tgt_mgmt01"
      - CONN_AUTH_FILE_DATA=shared_secret_string
    volumes:
      - /mnt/mgmt_tgt_mgmt01:/data/mgmt_tgt_mgmt01
    network_mode: "host"
    cap_add:
      - IPC_LOCK
    devices:
      - /dev/infiniband/rdma_cm
      - /dev/infiniband/uverbs0
      - /dev/infiniband/uverbs1
    command: "storeAllowFirstRunInit=false connInterfacesList=<COMMA_SEPARATED_LIST> storeMgmtdDirectory=/data/mgmt_tgt_mgmt01"
    restart: unless-stopped

  beegfs-meta:
    container_name: beegfs-meta
    image: ghcr.io/thinkparq/beegfs-meta:latest
    environment:
      - beegfs_setup_1=beegfs-setup-meta -C -p /data/meta_01_tgt_0101 -s 1 -S meta_01
      - CONN_AUTH_FILE_DATA=shared_secret_string
    volumes:
      - /mnt/meta_01_tgt_0101:/data/meta_01_tgt_0101
    network_mode: "host"
    cap_add:
      - IPC_LOCK
    devices:
      - /dev/infiniband/rdma_cm
      - /dev/infiniband/uverbs0
      - /dev/infiniband/uverbs1
    command: "sysMgmtdHost=<MGMT_IP> storeAllowFirstRunInit=false connInterfacesList=<COMMA_SEPARATED_LIST> storeMetaDirectory=/data/meta_01_tgt_0101"
    restart: unless-stopped
    depends_on:
      - beegfs-mgmtd

  beegfs-storage:
    container_name: beegfs-storage
    image: ghcr.io/thinkparq/beegfs-storage:latest
    environment:
      - beegfs_setup_1=beegfs-setup-storage -C -p /data/stor_01_tgt_101 -s 1 -S stor_01_tgt_101 -i 101
      - beegfs_setup_2=beegfs-setup-storage -C -p /data/stor_01_tgt_102 -s 1 -S stor_01_tgt_101 -i 102
      - CONN_AUTH_FILE_DATA=shared_secret_string
    volumes:
      - /mnt/stor_01_tgt_101:/data/stor_01_tgt_101
      - /mnt/stor_01_tgt_102:/data/stor_01_tgt_102
    network_mode: "host"
    cap_add:
      - IPC_LOCK
    devices:
      - /dev/infiniband/rdma_cm
      - /dev/infiniband/uverbs0
      - /dev/infiniband/uverbs1
    command: "sysMgmtdHost=<MGMT_IP> storeAllowFirstRunInit=false connInterfacesList=<COMMA_SEPARATED_LIST> storeStorageDirectory=/data/stor_01_tgt_101,/data/stor_01_tgt_102"
    restart: unless-stopped
    depends_on:
      - beegfs-mgmtd
      - beegfs-meta

Note

Note the use of the restart unless-stopped option, and depends_on option to control how containers start and stop to ensure the management service starts first, followed by the metadata then storage service.

Interacting with the Docker Compose File

Save the above file as docker-compose.yml, ensure the current working directory is set to the same directory, then use the following commands to interact with it.

Starting All Containers

Start all containers with docker-compose up. By default this command aggregates the output of each container, and when you exit (ctrl+c) the containers will stop. Alternatively use docker-compose up --detach to start containers in the background and leave them running.

Viewing All Container Logs

If you started Docker Compose in detached mode you can use docker-compose logs to view the aggregated output of each container.

Stopping All Containers

Stop all containers with docker-compose stop. As with docker stop you will need to remove the containers if you want to reconfigure anything.

Removing All Containers

Remove all containers with docker-compose rm

Stopping and Removing All Containers

To stop and remove containers with one command use docker-compose down.