Release Notes v7.4.7

The BeeGFS 7.4.7 release adds support for kernels up to Linux 6.12, improves ACL and directory-listing performance, and fixes various bugs.

General Changes/Improvements

  • ACL Performance: Adds client-side ACL caching, controlled with a new sysACLsRevalidate option. The default cache mode only revalidates ACLs after sysDirSubentryCacheValidityMS (default 1s) / sysFileSubentryCacheValidityMS (default 0s). This slightly delays ACL change propagation, but can significantly improve performance when ACLs are enabled. To preserve legacy behavior and propagate ACL changes immediately, set sysACLsRevalidate=always, which fetches ACLs on every access.

  • File System Checker (fsck): Re-enables orphaned chunk deletion as a repair action, with a new interactive confirmation prompt.

    • Previously disabled to prevent accidental data loss, since deleting orphaned chunks is not always the correct recovery procedure. The new prompt explains when deletion is appropriate and requires typing a confirmation phrase to prevent accidental acknowledgements.

    • When running fsck with the automatic repair option (discouraged), orphaned chunk deletion is always skipped.

  • Directory Listing Performance: Implements buffer-size aware directory listing that significantly improves performance for large directories. The metadata server now fills responses based on serialized entry size rather than a fixed 100-entry limit, maximizing network buffer utilization while preventing buffer overflows.

    • Configuration change: The client tuneMsgBufSize default has been increased from 64 KB to 1 MB to match the metadata server’s tuneWorkerBufSize (which was already 1 MB). The effective buffer size for directory listings is min(tuneMsgBufSize, tuneWorkerBufSize).

    • Performance tuning: For extremely large directories with very long filenames, these buffer sizes can be increased further, though 1 MB should be sufficient for most use cases.

    • Backward compatibility: Adds a handshake mechanism between client and server using network message flags to enable seamless upgrades in any order. When both client and metadata server are running 7.4.7 or later, the new buffer-size aware mode is automatically used. If either component is running an older version, the system automatically falls back to legacy behaviour which uses fixed-size directory listing responses with a maximum of 100 entries per response.

Fixes

  • Client: Adds missing client configuration parameters to /proc/fs/beegfs/*/config.

  • Client: Fixes a build failure on newer kernels that expect to use iov_iter_get_pages2 / iov_iter_alloc_pages2 when compiling the kernel module with GPU Direct Storage support (NVFS).

  • Client: Fixes a crash in the fsync path when GPU Direct Storage Support (NVFS) is enabled.

  • Client: Improved rebuild times by performing feature detection in parallel and caching results.

  • CTL: Fixes an issue with beegfs-ctl --migrate unnecessarily failing to migrate files that had inherited ACLs.

  • Fsck: Chunk fetcher now ignores ENOENT on vanished chunk files and directories, allowing fsck to complete on live systems even while files are being deleted. Note that running repair actions on a database created by running beegfs-fsck on a live system is still discouraged.

  • Meta: Fixes a performance regression in unlink compared to 7.2.15 that can be observed in deletion heavy workloads.

  • Meta: Fixes an issue with disposal file cleanup that could leave orphaned inodes. Incorrect link count handling could remove disposal dentries while the inode was still in use, leaving it without any associated dentry. These orphaned inodes could only be removed with beegfs-fsck. The cleanup logic now preserves disposal dentries until the inode is fully released, ensuring proper removal of inodes and storage chunks by the normal disposal cleanup process.

  • Meta: Multiple improvements to prevent race conditions affecting various combinations of metadata operations, in particular:

    • Fixes a race between hard link, rename, and open/close operations that could cause an inode to be prematurely freed while still in use, resulting in a crash during close.

    • Fixes a race between rename, create, and link operations that could cause a hard link to target the wrong inode, resulting in a dangling dentry pointing to non-existent inode data.

      • Addresses situations where create encounters dangling dentries leftover from this/similar conditions, and the dentries have NULL stripe patterns which would previously cause a crash on buddy mirrored systems. Now the system detects and correctly handles dentries in this state, propagating results like EXISTS where appropriate.

    • Improves file-locking logic when dentries and inodes reside on different meta nodes. Previous behavior could lead to inconsistencies, particularly with cross-directory hard links.

    • Improves reliability of directory deletions with concurrent directory access. This eliminates cases where deletions could intermittently fail with “Device or resource busy” (EBUSY) errors.

      Note this release makes a number of proactive changes intended to mitigate potential races when executing various combinations of metadata operations in parallel on the same entries. Not all of these issues were actually observed in production workloads, and some were only triggered by synthetic stress tests designed to simulate extreme concurrency.

  • Meta: Creation of cross-directory hardlinks between mirrored/unmirrored source/destination directories now correctly returns “Not Supported”.

Known Issues and Limitations

  • The Metadata daemon does not work reliably on RHEL/CentOS 8 and SLES 15.1 and 15.2 due to a problem in the versions of glibc. The problem was fixed in RHEL/CentOS 8.1 and SLES 15.3.

  • The client module does not compile on SLES 15.2 with Mellanox OFED 5.2

  • We observed some issues with RoCE in the client module on newer kernels that currently affect RHEL 9.4 and Ubuntu 24.04 where the kernel locks up on RDMA connection attempts to meta and storage servers. Infiniband based RDMA connections work as expected.

Supported Linux Distributions and Kernels

Supported distributions

Packages are provided for the x86_64 and aarch64 architecture and the following distributions:

  • RHEL 8 and 9 (packages can also be used on RockyLinux and Alma Linux)

  • SLES 15

  • Debian 10, 11 and 12

  • Ubuntu 18.04, 20.04 and 22.04, 24.04

The distributions we provide packages for are fully supported by the BeeGFS server services (mgmtd, meta, storage, mon). To keep the test matrix manageable and because kernel APIs tend to change frequently, even in distribution minor releases, the BeeGFS client generally only supports the default slow moving distribution kernels (not hwe, backports and similar). Custom kernel builds might or might not work and are generally not supported. There can also be incompatibilities between the client and very recent distribution kernel relases. We advise to have a look at the Client Build Testing section below for more information on what has been tested.

The same is true for RDMA driver versions that are generally not problematic on the servers, but can cause issues on the client. The following Mellanox OFED driver versions are supported: 25.01, 24.10, 24.07, 23.10, 5.8, 5.7, 5.6, 5.5, 5.4, 5.3, 5.2, 5.1, 5.0, 4.9

With the introduction of the DOCA repositories, we now support Mellanox OFED versions included in the following DOCA releases: 1.5-LTS, 2.5-LTS, and 2.9-LTS. Note that Mellanox OFED versions and DOCA versions follow different versioning schemes, so the supported Mellanox versions in each DOCA release may differ. Always refer to the specific DOCA release notes for detailed compatibility information.

The full integration test suite was run on Debian 10, Debian 11, Debian 12, OpenSUSE 15.4, RHEL 8.5, Rocky 8.7, Rocky 9.4, Ubuntu 20.04, Ubuntu 22.04 and Ubuntu 24.04.

Client build testing

Client build testing was done on the following OS and RDMA driver combinations. Other kernel versions might or might not work. If a version of MOFED was successfully tested on one OS, it is very likely that it will also work in combination with other kernels.

Tested distribution and driver version combinations (click to expand)

  • RHEL 8.3: Kernel RDMA drivers, MOFED 4.9, 5.0, 5.1, 5.2, 5.3, 5.4

  • AlmaLinux 8.4: Kernel RDMA drivers, MOFED 5.3, 5.4

  • AlmaLinux 8.5: Kernel RDMA drivers, MOFED 5.3, 5.4, 5.5

  • AlmaLinux 8.7: Kernel RDMA drivers, MOFED 5.8

  • AlmaLinux 8.10: MOFED 24.10

  • AlmaLinux 9.0: Kernel RDMA drivers, MOFED 5.6, 5.7

  • Rocky Linux 8.4: Kernel RDMA drivers, MOFED 5.3, 5.4

  • Rocky Linux 8.5: Kernel RDMA drivers, MOFED 5.5

  • Rocky Linux 8.6: Kernel RDMA drivers, MOFED 5.6

  • Rocky Linux 8.8: Kernel RDMA drivers, MOFED 5.8

  • Rocky Linux 8.9: Kernel RDMA drivers, MOFED 23.10

  • Rocky Linux 9.1: Kernel RDMA drivers, MOFED 5.8

  • Rocky Linux 9.2: Kernel RDMA drivers

  • Rocky Linux 9.3: Kernel RDMA drivers, MOFED 23.10, 24.1

  • Rocky Linux 9.4: Kernel RDMA drivers, MOFED 24.07

  • Rocky Linux 9.5: MOFED 24.10

  • SLES 15.1: Kernel RDMA drivers, MOFED 5.0

  • SLES 15.2: Kernel RDMA drivers, MOFED 5.1, 5.4

  • SLES 15.3: Kernel RDMA drivers, MOFED 5.4, 5.5, 5.6, 5.8

  • SLES 15.4: Kernel RDMA drivers, MOFED 5.8

  • SLES 15.5: Kernel RDMA drivers, MOFED 5.8

  • Debian 10: Kernel RDMA drivers, MOFED 5.2, 5.3, 5.4

  • Debian 11: Kernel RDMA drivers, MOFED 5.6

  • Debian 12: Kernel RDMA drivers, MOFED 24.10

  • Ubuntu 18.04: Kernel RDMA drivers

  • Ubuntu 20.04: Kernel RDMA drivers, MOFED 5.4, 24.10

  • Ubuntu 22.04.1, 22.04.2, 22.04.3: Kernel RDMA drivers, MOFED 5.6, 5.7, 5.8, 24.10, 25.01

  • Ubuntu 24.04.0: Kernel RDMA drivers, MOFED 24.04, 24.10, 25.01

  • Ubuntu 24.10.0: Kernel RDMA drivers, MOFED 24.10

Version Interoperability

BeeGFS 7.4.7 is fully compatible with 7.4.6, 7.4.5, 7.4.4 and 7.4.3, mostly (see warning below) compatible with 7.4.x < 7.4.3 but not fully compatible with older versions of BeeGFS.

Warning

Necessary changes to a message used in metadata hard link creation cause an incompatibility between metadata services >= 7.4.3 and metadata services running versions < 7.4.3. It is therefore necessary to upgrade all metadata nodes in a BeeGFS system to versions >= 7.4.3 at once. Rolling upgrades of metadata services from older versions are not supported. All services other than metadata are not affected by this and can be upgraded in a rolling fashion or used together with other 7.4.x versions.

The incompatibilities between 7.4.7 and 7.3.x are limited to hard link functionality and a change in the monitoring message format. Please refer to the Upgrade Guide for more information.

Upgrading from Older Versions

To upgrade from an older version, please refer to the Upgrade Guide.