Release Notes v8.0.0

The BeeGFS 8.0.0 release introduces new data management capabilities, simplifies file system management and third-party software integration, and sets the stage for future enhancements.

General Changes/Improvements

• Replaces network communication for managing BeeGFS with protocol buffers and gRPC, simplifying integration of third-party applications written in multiple programming languages.

  • This specifically applies to new services such as the management and allows for TLS encryption.

• Replaces the custom hashing algorithm used for connection based authentication with SHA256.

• Removes the beegfs-common package that contained deprecated init scripts for BeeGFS daemons.

• Simplifies client installation by removing the beegfs-helperd package and service. This was formerly used for client hostname resolution and logging.

  • Hostname resolution has been replaced with a mount script, and the client now exclusively logs to the system journal. This ensures all client log messages can be found in the journal, consistent with the behavior of most modern kernel module implementations.

• Removes the need to configure the TCP and UDP port used by clients and services separately. Going forward BeeGFS will always use the same port number for both TCP and UDP. No change to existing configuration files is required as long as the TCP/UDP port for each component type is the same.

• Cleans up leftover artifacts in the codebase from previously deprecated or removed components.

Note

With version 8, BeeGFS will more strictly adhere to semantic versioning guarantees. The initial release of version 8 preemptively makes a number of backwards incompatible changes required for planned functionality that can be rolled out as minor releases in the version 8 release series.

Management Service

• The management service has been redesigned from the ground up, is now written in Rust, and incorporates an SQL database, simplifying expansion of its role going forward.

  • Handles automatically importing BeeGFS 7 configuration into the new BeeGFS 8 management database with built-in schema migrations for seamless version upgrades as new features are added.

  • Simplifies configuration using a new TOML based format.

  • Improves supportability with improved logging that defaults to the system journal.

• Improves various Buddy Mirroring functionality, notably reducing shutdown and restart times in most scenarios when Buddy Mirroring is in use.

• Introduces a licensing system to streamline support for users with a support contract and help prevent accidental use of enterprise features in the BeeGFS Hive Edition. See licensing for more details including how to obtain a license key.

• Replaces node and target string IDs with user reconfigurable aliases that can assist in identification of underlying hardware resources based on user-defined naming conventions.

Metadata Service

• Persists File System Modification Events using a new on-disk event queue, preventing events from being dropped in most scenarios, such as service restarts or server reboots. The size of the queue is configurable, allowing events to be held for some time if downstream listeners/subscribers are temporarily unavailable due to network or other issues.

• Adds new types of File System Modification Events:

  • Different open events depending on the combination of read/write flags.

  • A special event indicating when a file is closed for writing on all clients.

• Expands the details with each File System Modification Event to include:

  • The user ID (actor ID) that triggered the event. This is useful for audit logging.

  • The timestamp when the event was triggered.

  • The number of links to the entry.

• Adds support for determining the number of read/write sessions on file entries, which is useful when designing workflows that need to know if a file is closed on all clients.

• Introduces a new internal locking mechanism for file entries to temporarily prevent client access until specific operations are completed, for example re-striping or synchronizing file contents.

• Incorporates file data states (e.g., local, locked, offloaded) that will be used in the future for richer integration with data management (HSM) software that needs to manage file contents.

Note

Collectively this functionality forms the foundation of the new BeeGFS Data Management API and provides richer integration with external data management solutions that monitor and react to file system activity and control the state of file system entries. Continuing to expand on this API will be a central focus of the BeeGFS 8 release series.

BeeGFS Command Line Interface

• Replaces the beegfs-ctl command-line interface with a new beegfs tool written in Go that unifies file system and data management.

  • Shorter, more concise commands with modern syntax featuring a noun-verb pattern and hierarchical command subcommand structure.

  • New scheme for referencing BeeGFS nodes, targets, pools, mirrors, etc. either by their type:id or alias (see entity IDs for more details).

  • Most commands now return structured output, which is printed as a table by default with filterable columns. Commands that support table output can alternatively output JSON or NDJSON.

  • No longer uses a configuration file and is instead fully configurable using environment variables and/or flags. Automatically determines the management service when BeeGFS is mounted often eliminating the need to specify any configuration when running the tool.

• Introduces new modes to enhance supportability:

  • Integrates previously standalone scripts including beegfs-check-servers, beegfs-df and beegfs-net and removes those scripts from the beegfs-utils package. Expanded check mode includes new health checks and the ability to watch the health of a file system when running benchmarks or other tests.

  • New node ping mode more precisely measures the round-trip latency to all/specific server nodes from the BeeGFS client kernel module.

  • Collect support bundles containing the output of all commands typically requested by support in a convenient .tar.gz bundle.

• Simplifies use of existing modes:

  • Allows updating or getting info about multiple entries by providing multiple paths using standard Linux glob patterns and wildcards, or by recursively walking a directory tree.

  • Modes like benchmark and stats now default to executing the most commonly used functionality when run without any additional options.

• Integrated support for new and existing enterprise data management functionality:

  • The copy mode allows parallel data movement between BeeGFS and other POSIX file systems (requires the beegfs-copy package to be installed).

  • The index modes allow creating, updating, and querying the file system Index (requires the beegfs-hive-index package to be installed).

  • The remote modes allow data in BeeGFS to be synced with Remote Storage Targets (requires Remote Storage Target services to be deployed).

• Uses a beegfs system group for ownership of configuration files required to execute the CLI tool. This improves security by no longer requiring beegfs to be a setuid binary.

• Separates the frontend CLI utility from the backend library functionality that handles communication with BeeGFS servers, enabling other Go applications to easily integrate with BeeGFS.

Remote Storage Targets

• Adds support for configuring one or more Remote Storage Targets (RSTs) on files and directories that represent external storage providers. Currently, object storage providers with an S3-like interface are supported, with plans to add additional protocols in the future.

  • RST configuration behaves similarly to storage pools with files automatically inheriting configuration from their parent directory unless explicitly set. Unlike storage pools, RST configuration on file entries can be changed at any time without needing to recreate the file.

• Introduced two new BeeGFS services, BeeGFS Remote and BeeGFS Sync, that provide the system to track and drive data movement to Remote Storage Targets.

  • The Remote node coordinates transferring data in parallel across a scalable pool of Sync nodes. These nodes can run alongside other BeeGFS services or on dedicated nodes depending on the requirements.

BeeGFS Watch (new beegfs-event-listener service)

• Introduces a new beegfs-event-listener service written in Go that runs on metadata nodes to forward File System Modification Events to external subscribers.

  • The old beegfs-event-listener is still available to support legacy use cases.

• Uses protocol buffers and gRPC for communication to subscribers, with libraries available in multiple languages to make implementation of custom subscribers simple.

  • Uses bidirectional gRPC streams to reliably deliver events to each subscriber, even if a subscriber temporarily disconnects due to network or other issues.

  • Additional protocols and subscriber types may be supported based on user interest.

• Watch will be used as the mechanism going forward to update Hive Index and other BeeGFS utilities.

BeeGFS Hive Index

• Adds a new rescan mode that allows the index to be updated for an entire directory tree or a specific subdirectory, with options to include or exclude recursion.

  • Recursive Mode: Scans the selected directory along with all existing and new child directories.

  • Non-Recursive Mode: Updates only the specified subdirectory while also detecting and indexing any newly created child directories. This avoids redundant rescans of existing directories, reducing unnecessary processing when there are no filesystem modifications. This flexibility ensures efficient index updates without re-scanning unchanged directories.

• Add commands for managing the Index to the new beegfs tool (CTL).

  • Results from index find, ls, and stats will show paths directly from the BeeGFS directory tree, ensuring better clarity and usability. Previously, paths were derived from the index directory, making it difficult for users to map back to filesystem paths.

• Removes the automatic index update functionality (replaced by the rescan mode).

  • The original approach to immediate index updates required verification with the filesystem right after each system call, adding significant load to an already busy filesystem. A heavily loaded filesystem also generates a large number of events that must be processed in the correct order to maintain consistency between the index and the filesystem. These scalability and correctness issues were addressed by adding a new approach to incrementally rescan the file system.

BeeOND

• Adapts BeeOND to work with the new mgmtd and the new beegfs command line utility.

  • Adds new command line argument (-G) to configure the mgmtd gRPC port. This is only needed if a gRPC port other than the default is configured in a configuration file.

  • Adds new command line arguments to enable connection authentication, TLS encryption and load license certificates.

Warning

Because BeeOND is considered a short-lived scratch file system with main focus on a straightforward setup with minimal configuration, as of BeeGFS 8, BeeOND defaults to starting without connection authentication or TLS encryption for the communication between the mgmtd and the command line configuration utility. By default, no configuration other than the BeeOND command line arguments is needed. New command line arguments have been added to enable connection authentication (-C) and TLS encryption (-E) if needed. Enabling them will add a requirement for a configuration directory that is available on all nodes (see option -f) and contains the required files needed.

Fixes

No fixes were made in 8.0.0 that were not also back-ported to the 7.4.x release series.

Known Issues and Limitations

General:

• The Metadata daemon does not work reliably on RHEL/CentOS 8 and SLES 15.1 and 15.2 due to a problem in the versions of glibc. The problem was fixed in RHEL/CentOS 8.1 and SLES 15.3.

• The client module does not compile on SLES 15.2 with Mellanox OFED 5.2

• We observed some issues with RoCE in the client module on newer kernels that currently affect RHEL 9.4 and Ubuntu 24.04 where the kernel locks up on RDMA connection attempts to meta and storage servers. Infiniband based RDMA connections work as expected.

Index:

• Incremental updates via the legacy bee-index systemd service and event listener are no longer supported.

Remote Storage Targets:

• Downloading empty (0 byte long) files from a remote target is not currently supported.

Supported Linux Distributions and Kernels

Supported distributions

Packages are provided for the x86_64 and aarch64 architecture and the following distributions:

• RHEL 8 and 9 (packages can also be used on RockyLinux and Alma Linux)

• SLES 15

• Debian 10, 11 and 12

• Ubuntu 18.04, 20.04 and 22.04, 24.04

The distributions we provide packages for are fully supported by the BeeGFS server services (mgmtd, meta, storage, mon). To keep the test matrix manageable and because kernel APIs tend to change frequently, even in distribution minor releases, the BeeGFS client generally only supports the default slow moving distribution kernels (not hwe, backports and similar). Custom kernel builds might or might not work and are generally not supported. There can also be incompatibilities between the client and very recent distribution kernel relases. We advise to have a look at the Client Build Testing section below for more information on what has been tested.

The same is true for RDMA driver versions that are generally not problematic on the servers, but can cause issues on the client. The following Mellanox OFED driver versions are supported: 25.01, 24.10, 24.07, 23.10, 5.8, 5.7, 5.6, 5.5, 5.4, 5.3, 5.2, 5.1, 5.0, 4.9

With the introduction of the DOCA repositories, we now support Mellanox OFED versions included in the following DOCA releases: 1.5-LTS, 2.5-LTS, and 2.9-LTS. Note that Mellanox OFED versions and DOCA versions follow different versioning schemes, so the supported Mellanox versions in each DOCA release may differ. Always refer to the specific DOCA release notes for detailed compatibility information.

The full integration test suite was run on Debian 10, Debian 11, Debian 12, OpenSUSE 15.4, RHEL 8.5, Rocky 8.7, Rocky 9.4, Ubuntu 20.04, Ubuntu 22.04 and Ubuntu 24.04.

Client build testing

Client build testing was done on the following OS and RDMA driver combinations. Other kernel versions might or might not work. If a version of MOFED was successfully tested on one OS, it is very likely that it will also work in combination with other kernels.

Tested distribution and driver version combinations (click to expand)

• RHEL 8.3: Kernel RDMA drivers, MOFED 4.9, 5.0, 5.1, 5.2, 5.3, 5.4

• AlmaLinux 8.4: Kernel RDMA drivers, MOFED 5.3, 5.4

• AlmaLinux 8.5: Kernel RDMA drivers, MOFED 5.3, 5.4, 5.5

• AlmaLinux 8.7: Kernel RDMA drivers, MOFED 5.8

• AlmaLinux 8.10: MOFED 24.10

• AlmaLinux 9.0: Kernel RDMA drivers, MOFED 5.6, 5.7

• Rocky Linux 8.4: Kernel RDMA drivers, MOFED 5.3, 5.4

• Rocky Linux 8.5: Kernel RDMA drivers, MOFED 5.5

• Rocky Linux 8.6: Kernel RDMA drivers, MOFED 5.6

• Rocky Linux 8.8: Kernel RDMA drivers, MOFED 5.8

• Rocky Linux 8.9: Kernel RDMA drivers, MOFED 23.10

• Rocky Linux 9.1: Kernel RDMA drivers, MOFED 5.8

• Rocky Linux 9.2: Kernel RDMA drivers

• Rocky Linux 9.3: Kernel RDMA drivers, MOFED 23.10, 24.1

• Rocky Linux 9.4: Kernel RDMA drivers, MOFED 24.07

• Rocky Linux 9.5: MOFED 24.10

• SLES 15.1: Kernel RDMA drivers, MOFED 5.0

• SLES 15.2: Kernel RDMA drivers, MOFED 5.1, 5.4

• SLES 15.3: Kernel RDMA drivers, MOFED 5.4, 5.5, 5.6, 5.8

• SLES 15.4: Kernel RDMA drivers, MOFED 5.8

• SLES 15.5: Kernel RDMA drivers, MOFED 5.8

• Debian 10: Kernel RDMA drivers, MOFED 5.2, 5.3, 5.4

• Debian 11: Kernel RDMA drivers, MOFED 5.6

• Debian 12: Kernel RDMA drivers, MOFED 24.10

• Ubuntu 18.04: Kernel RDMA drivers

• Ubuntu 20.04: Kernel RDMA drivers, MOFED 5.4, 24.10

• Ubuntu 22.04.1, 22.04.2, 22.04.3: Kernel RDMA drivers, MOFED 5.6, 5.7, 5.8, 24.10, 25.01

• Ubuntu 24.04.0: Kernel RDMA drivers, MOFED 24.04, 24.10, 25.01

• Ubuntu 24.10.0: Kernel RDMA drivers, MOFED 24.10

Version Interoperability

BeeGFS 8.0.0 clients, servers, and other utilities are not compatible with older versions of BeeGFS due to changes in network and on-disk formats.

Upgrading from Older Versions

To upgrade from an older version, please refer to the Upgrade Guide.