Release Notes v7.4.0

The BeeGFS 7.4.0 release adds support for cross-directory hard links, introduces an improved monitoring framework and updates the client module to work with new kernels. It also includes a lot of smaller bug fixes.

Warning

Before upgrading to this version from any version <7.3.1, please read the section about mandatory Authentication carefully. If you have not configured a connAuthFile, services will no longer start without configuration changes. For more background on why we make this change, please refer to our blog post on the topic.

Cross-directory hard links

The term hard link stands for a directory entry on a file system that associates a name with an inode, the structure that defines a file or directory. While every file in a file system has only one inode (in fact, the inode is the file), it can usually have multiple names in different places on the file system. The number of names it has is also referred to as its link count. When a file is created, it has a single name and a link count of one. When additional hard links to that file are created, the link count gets incremented and whenever the last of these links is deleted, the inode is removed as well and the file ceases to exist.

Because of the way BeeGFS optimizes inode access by embedding file inodes into their directory entries, the two used to be somewhat inseperable up until now. BeeGFS has always been able to represent a hard link on BeeGFS by a hard link in the underlying file system, but because different directories in BeeGFS can live on different nodes and hard links are limited to a single underlying file system, BeeGFS couldn’t support links across directory boundaries up until now.

Version 7.4.0 introduces a mechanism to split the inode from its directory entry in a process we call de-inlining. When an inode is de-inlined it gets physically moved to a different place in the underlying data structure but stays on the same node as its directory entry. The directory entry becomes a pointer to that inode. Because directory entry and inode are now separate objects, the directory entry is free to move around and new directory entries can be created and point to the same inode, increasing its link count.

This, however, comes at the cost of adding an additional lookup whenever the inode is accessed via one of its names, because the link needs to be resolved before the inode can be accessed. To keep that performance impact limited to the files that actually have multiple hard links, de-inlining only happens when a second link to a file is created. That way, metadata perfomance remains unaffected for the vast majority of files in the file system and there is no impact at all if hard links are not used.

7.4.0 also adds a mode to beegfs-fsck to find and migrate existing same-directory hardlinks to the new format, so they can benefit from the same flexibilty that de-inlined inodes provide. For a guide on how to check your file system and migrate existing hardlinks, please read through our BeeGFS v7.4.0 Upgrade Guide.

General Changes/Improvements

• New monitoring framework that uses Telegraf to provide system level metrics in addition to BeeGFS specific metrics. Completely reworked Grafana panels to visualize and correlate information from different sources to make performance monitoring and analysis a lot easier. For more information about how to set up the new monitoring system, see our article on the Monitoring service.

• Support for Linux versions up to 6.1, which is used in Debian 12. Also fixes build issues with RHEL 9.2 kernels.

• A new mount option grpid is now supported to enable global setgid like behavior. See Project directory quota tracking for more information on where this is useful.

Fixes

• Update 7.4.0p1: Fixed an issue with beegfs-meta crashing because of a race condition while accessing the internal inode store. This is related to the hard links updates, but can occur for operations unrelated to hard links as well. This patch update is distributed through the same repository as 7.4.0 and we encourage everybody who is running 7.4.0 to upgrade to 7.4.0p1, especially on the metadata servers.

• Quota will now be enforced correctly when a storage target and buddy group have the same ID.

• Fixed an issue that was introduced in 7.3.3 and 7.2.9 where configuring a connNetFilterFile on the mgmtd could lead to interfaces not being correctly advertised to other nodes in the system.

Known Issues and Limitations

• The Metadata daemon does not work reliably on RHEL/CentOS 8 and SLES 15.1 and 15.2 due to a problem in the versions of glibc. The problem was fixed in RHEL/CentOS 8.1 and SLES 15.3.

• The client module might hang if the IBV device(s) used are taken offline during operation.

• The client module does not compile on SLES 15.2 with Mellanox OFED 5.2

Supported Linux Distributions and Kernels

Packages are provided for the x86_64 and aarch64 architecture and the following distributions:

• RHEL 8 and 9 (packages can also be used on RockyLinux and Alma Linux)

• SLES 15

• Debian 10, 11 and 12

• Ubuntu 18.04, 20.04 and 22.04

The following Mellanox OFED driver versions are supported: 5.8, 5.7, 5.6, 5.5, 5.4, 5.3, 5.2, 5.1, 5.0, 4.9

The full integration test suite was run on Alma 8.5, Alma 9.0, Debian 10, Debian 11, Debian 12, OpenSUSE 15.4, Rocky 8.5, Rocky 8.6, Rocky 8.7, Rocky 9.1, Rocky 9.2, Ubuntu 20.04 and Ubuntu 22.04.

Client build testing:

• RHEL 8.3: no OFED, OFED 4.9, 5.0, 5.1, 5.2, 5.3, 5.4

• AlmaLinux 8.4: no OFED, OFED 5.3, 5.4

• AlmaLinux 8.5: no OFED, OFED 5.3, 5.4, 5.5

• AlmaLinux 8.7: no OFED, OFED 5.8

• AlmaLinux 9.0: no OFED, OFED 5.6, 5.7

• Rocky Linux 8.4: no OFED, OFED 5.3, 5.4

• Rocky Linux 8.5: no OFED, OFED 5.5

• Rocky Linux 8.6: no OFED, OFED 5.6

• Rocky Linux 9.1: no OFED, OFED 5.8

• Rocky Linux 9.2: no OFED

• SLES 15.1: no OFED, OFED 5.0

• SLES 15.2: no OFED, OFED 5.1, 5.4

• SLES 15.3: no OFED, OFED 5.4, 5.5, 5.6, 5.8

• SLES 15.4: OFED 5.5

• Debian 10: no OFED, OFED 5.2, 5.3, 5.4

• Debian 11: no OFED, OFED 5.6

• Debian 12: no OFED

• Ubuntu 18.04: no OFED

• Ubuntu 20.04: no OFED, OFED 5.4

• Ubuntu 22.04: no OFED, OFED 5.6, 5.7

Version Interoperability

BeeGFS v7.4.0 is not fully compatible with older versions of BeeGFS.

The incompatibilities between 7.4.0 and 7.3.x are limited to hard link functionality and a change in the monitoring message format. Please refer to the Upgrade Guide for more information.

Upgrading from Older Versions

To upgrade from an older version, please refer to the Upgrade Guide.