Release Notes v8.1

The BeeGFS 8.1 release primarily focuses on expanding data management capabilities.

Note

Please refer to GitHub for patch release notes.

General Changes/Improvements

  • Support for RHEL / Rocky Linux / AlmaLinux 10.

Management Service

  • For advanced use cases the file system UUID can be specified when initializing the database with a hidden --fs-uuid=<v4-uuid>. This is primarily intended for automated deployment tooling that needs to pregenerate the FS UUID before deploying any BeeGFS services.

Metadata Service

  • New file access flags can be set to block file operations by regular clients.

    • Read/write locks block open syscalls with conflicting file access modes and return EWOULDBLOCK.

    • Specific clients can be allowed to bypass access checks and manage the contents of locked files by setting a new sysBypassFileAccessCheckOnMeta parameter in beegfs-client.conf.

    • A new OPEN_BLOCKED file system modification event is triggered that external data management software can listen for and use to trigger some action (e.g., restore a file’s contents).

  • New file data states can be optionally set by data management software to communicate additional details using BeeGFS tooling.

Note

This functionality is part of the new BeeGFS Data Management API and is intended to provided richer integration with external data management solutions that needs to monitor and react to file system activity and control the state of file system entries. Continuing to expand on this API will be a central focus of the BeeGFS 8 release series.

BeeGFS Command Line Interface

  • Cosmetic improvements to help text and error outputs to simplify use and improve debuggability.

  • Positional arguments provided to commands that do not support them are no longer silently ignored.

  • The Remote job list command now defaults to returning jobs for a single path, and adds a new --recurse flag for listing jobs from multiple paths. The command also no longer requires specifying --mount=none to list jobs for paths that no longer exist in BeeGFS.

Remote Storage Targets

  • Files being synced are now locked to prevent inadvertent user access that would disrupt the sync.

    • This also greatly improves the overall performance of the remote targets feature by collecting most information once upfront and reusing it in subsequent steps. This also reduces the work done by the Remote node, instead pushing expensive tasks out to run in parallel on Sync nodes.

  • Pushing files can optionally offload file contents from BeeGFS and leave behind a “stub file”.

    • Stub files contain only a special URL pointing to the remote target and remote file where the file’s contents have been offloaded that can later be used to restore the contents with a pull.

    • Regular clients are not able to open stub files for reading or writing, and will instead get an EWOULDBLOCK (resource temporarily unavailable) error.

  • Pulling files can now download multiple remote files at once based on glob patterns and wildcards.

    • Files can be pulled using a different local path than the remote path, and then kept in sync if the remote path is later updated (requires specifying the --overwrite flag).

    • By default if the name of a remote file contains a slash implying a directory hierarchy, that structure is maintained when pulled into BeeGFS. Optionally the --flatten flag can be used to replace slashes with underscores and pull all remote files into the current directory.

    • Files can be pulled as stubs to recreate a remote directory structure inside BeeGFS that can be gradually rehydrated depending on what file contents are actually needed.

  • Missing Remote database entries are automatically recreated based on the actual state of the local and remote files whenever a push/pull is executed. This prevents unnecessary syncing if the database is lost due to disk failure, or if historical jobs for a path are deleted.

Fixes

  • Client: Uninstalling the package on SUSE 15.6 no longer fails with: Failed to execute /usr/lib/systemd/systemd-sysv-install: No such file or directory.

  • Copy: The --list-diff mode now consistently returns “1” if there are differences or “0” if the source/destination are in sync. Previously the return code depended on the number of differences.

  • CTL: The client stats did not respect the --raw flag and would always use IEC prefixed units.

  • Index: Command output no longer inserts an extra newline before printing output and does not add extra whitespace at line ends. These could break using the output as input to other applications.

  • Meta: Creation of cross-directory hardlinks between mirrored/unmirrored source/destination directories now correctly returns “Not Supported”.

  • Remote Storage Targets: Pulling empty objects from S3 buckets is now supported.

Known Issues and Limitations

General:

  • The Metadata daemon does not work reliably on RHEL/CentOS 8 and SLES 15.1 and 15.2 due to a problem in the versions of glibc. The problem was fixed in RHEL/CentOS 8.1 and SLES 15.3.

  • The client module does not compile on SLES 15.2 with Mellanox OFED 5.2

  • We observed some issues with RoCE in the client module on newer kernels that currently affect RHEL 9.4 and Ubuntu 24.04 where the kernel locks up on RDMA connection attempts to meta and storage servers. Infiniband based RDMA connections work as expected.

Remote Storage Targets:

  • The Remote and Sync services must be restarted if metadata nodes are added/removed.

Supported Linux Distributions and Kernels

Supported distributions

Packages are provided for the x86_64 and aarch64 architecture and the following distributions:

  • RHEL 8, 9, and 10 (packages can also be used on Rocky Linux and AlmaLinux).

  • SLES 15

  • Debian 11 and 12

  • Ubuntu 20.04 and 22.04, 24.04

The distributions we provide packages for are fully supported by the BeeGFS server services (mgmtd, meta, storage, mon). To keep the test matrix manageable and because kernel APIs tend to change frequently, even in distribution minor releases, the BeeGFS client generally only supports the default slow moving distribution kernels (not hwe, backports and similar). Custom kernel builds might or might not work and are generally not supported. There can also be incompatibilities between the client and very recent distribution kernel relases. We advise to have a look at the Client Build Testing section below for more information on what has been tested.

The same is true for RDMA driver versions that are generally not problematic on the servers, but can cause issues on the client. The following Mellanox OFED driver versions are supported: 25.04, 25.01, 24.10, 24.07, 23.10, 5.8, 5.7, 5.6, 5.5, 5.4, 5.3, 5.2, 5.1, 5.0, 4.9

With the introduction of the DOCA repositories, we now support Mellanox OFED versions included in the following DOCA releases: 1.5-LTS, 2.5-LTS, and 2.9-LTS. Note that Mellanox OFED versions and DOCA versions follow different versioning schemes, so the supported Mellanox versions in each DOCA release may differ. Always refer to the specific DOCA release notes for detailed compatibility information.

The full integration test suite was run on Debian 11, Debian 12, OpenSUSE 15.6, Rocky Linux 8.10, Rocky Linux 9.4, CentOS Stream 10, Ubuntu 20.04, Ubuntu 22.04 and Ubuntu 24.04.

Client build testing

Client build testing was done on the following OS and RDMA driver combinations. Other kernel versions might or might not work. If a version of MOFED was successfully tested on one OS, it is very likely that it will also work in combination with other kernels.

Tested distribution and driver version combinations (click to expand)

  • RHEL 8.4: Kernel RDMA drivers, Mellanox OFED 4.9, 5.4

  • Rocky Linux 8.8: Kernel RDMA drivers, Mellanox OFED 5.8

  • Rocky Linux 8.9: Kernel RDMA drivers

  • AlmaLinux 8.10: Kernel RDMA drivers, Mellanox OFED 23.10, 24.10

  • Rocky Linux 9.1: Kernel RDMA drivers, Mellanox OFED 5.8

  • Rocky Linux 9.3: Kernel RDMA drivers

  • AlmaLinux 10.0: Kernel RDMA drivers

  • Debian 11: Kernel RDMA drivers

  • Debian 12: Kernel RDMA drivers, Mellanox OFED 23.10, 24.10

  • Ubuntu 20.04: Kernel RDMA drivers, Mellanox OFED 24.10

  • Ubuntu 22.04.1, 22.04.2, 22.04.3: Kernel RDMA drivers, Mellanox OFED 5.8, 23.10, 24.10, 25.01, 25.04

  • Ubuntu 24.04.0: Kernel RDMA drivers, Mellanox OFED 24.10, 25.01

  • Ubuntu 24.10.0: Kernel RDMA drivers, Mellanox OFED 24.10

  • SLES 15.1: Kernel RDMA drivers

  • SLES 15.3: Kernel RDMA drivers, Mellanox OFED 5.4

  • OpenSUSE 15.5: Kernel RDMA drivers, Mellanox OFED 5.8

  • OpenSUSE 15.6: Kernel RDMA drivers

Version Interoperability

Starting with version 8, BeeGFS more strictly adheres to semantic versioning guarantees. This means BeeGFS 8.1.0 components are compatible with components running any BeeGFS 8.y.z release, however the use of new functionality introduced in 8.1.0 requires the involved components to be running 8.1.0.

Note that BeeGFS 8 is not compatible with older major versions of BeeGFS due to changes in network and on-disk formats.

Upgrading from Older Versions

To upgrade from an older version, please refer to the Upgrade Guide.