Release Notes v7.4.2

The BeeGFS 7.4.2 release adds more fine-grained configuration for RDMA connections, Grafana alert functionality and fixes issues with RDMA device removal and the buffered cache mode in the client and metadata service issues introduced with the hard links code.

General Changes/Improvements

• The client module has been adapted to build on Linux versions up 6.2.

• Added a new configuration option connRDMAFragmentSize that can be used to reduce fragmentation of the RDMA buffers used for data transfers between the client and the servers. The default value is 4096, which preserves legacy behavior. We recommend to increase the fragment and buffer sizes for improved RDMA performance and larger connRDMABufSize. Please see the client configuration file for more detailed documentation.

• Added new configuration options connRDMAMetaBufNum, connRDMAMetaBufSize and connRDMAMetaFragmentSize to enable more fine-grained control over RDMA paramters for connections to meta servers. connRDMAMetaBufNum and connRDMAMetaBufSize default to 0 to preservce legacy behavior, which is to use the same values as storage connections which are defined by connRDMABufNum and connRDMABufSize. connRDMAMetaFragmentSize defaults to 4096, also to preserve legacy behavior. Meta messages are usually much smaller than messages to storage and therefore don’t require as much buffer space. Please refer to the client configuration file for more detailed documentation.

• Added new example alert configuration to the monitoring framework. Please note that these alerts need to be imported and manually enabled first. Please see the alert setup documentation for a detailed setup and configuration guide.

Fixes

• Fixed an issue with the buffered cache mode that could lead to an EFAULT (Bad address) on read().

• Fixed an issue that could lead to client crashes if an IPoIB device went down and caused an RDMA_CM_EVENT_DEVICE_REMOVAL event, which was improperly handled. This fix also adds a periodic rescan of available NICs, which will update the connection pool component and enables it to react to changes in local NIC availability.

• Some documentation updates and corrections.

Fixes related to changes around hard links

1. Fixed an issue in the meta service where a race between the file creation and deletion code paths could trigger an existing bug when hard links are involved, which lead to metadata service crashes.

2. Fixed an issue where a code change needed for hard link support could lead to failures of an ongoing resync in the buddy mirroring modification resyncer.

Known Issues and Limitations

• The Metadata daemon does not work reliably on RHEL/CentOS 8 and SLES 15.1 and 15.2 due to a problem in the versions of glibc. The problem was fixed in RHEL/CentOS 8.1 and SLES 15.3.

• The client module does not compile on SLES 15.2 with Mellanox OFED 5.2

Issues related to changes around hard links

In BeeGFS version 7.4.0 we released cross-directory hard links support. Most design decisions proved to work well on customer systems systems but a few decisions eventually proved to be problematic and to cause rare race conditions with specific workloads. Buddy mirrored systems are generally more likely to experience these issues, because additional locking is needed to coordinate synchronous updates between the buddies.

Most of the issues have already been fixed in version 7.4.1 and 7.4.2, but rigorous internal testing of corner cases involving stress testing with high numbers of conflicting operations inspired by the issues we saw on customer systems have surfaced a few more rare issues that we are still working on fixing:

• There is still a theoretical race between some metadata operations, similar to issue number 1 in the “Fixes” section above but much less likely to occur. We are working on fully fixing this class of race conditions by refactoring how metadata operations access and update different in-memory structures.

  Impact: This class of errors leads to meta service crashes and require a restart of the meta service to get the system running again.

• In rare cases, races between file deletions and other operations can lead to dangling dentries, orphaned inodes and orphaned chunks.

  Impact: No direct impact on file system availability. Issues can be fixed with beegfs-fsck.

• In buddy mirrored systems, there can be rare occurences of “Different return codes” in normal operation, which will lead to unnecessary resyncs. The same race conditions can lead to errors in the modification resyncer as described in fix number 2 above. These will lead to aborted resyncs which leave secondaries in state “Bad” and require manual intervention to get the mirror in sync again (beegfs-ctl --startresync). The fix in the 7.4.2 release makes these issues very unlikely to occur, but doesn’t completely rule them out. We are working on a some code restructuring to completely eliminate this class of errors in the future.

  Impact: No direct impact on file system availability. Unnecessary resyncs or failing resyncs that may need to be manually restarted.

Supported Linux Distributions and Kernels

Packages are provided for the x86_64 and aarch64 architecture and the following distributions:

• RHEL 8 and 9 (packages can also be used on RockyLinux and Alma Linux)

• SLES 15

• Debian 10, 11 and 12

• Ubuntu 18.04, 20.04 and 22.04

The following Mellanox OFED driver versions are supported: 5.8, 5.7, 5.6, 5.5, 5.4, 5.3, 5.2, 5.1, 5.0, 4.9

The full integration test suite was run on Debian 10, Debian 11, Debian 12, OpenSUSE 15.4, RHEL 8.5, Rocky 8.7, Rocky 9.2, Ubuntu 20.04, Ubuntu 22.04 and Ubuntu 23.04.

Client build testing:

• RHEL 8.3: no OFED, OFED 4.9, 5.0, 5.1, 5.2, 5.3, 5.4

• AlmaLinux 8.4: no OFED, OFED 5.3, 5.4

• AlmaLinux 8.5: no OFED, OFED 5.3, 5.4, 5.5

• AlmaLinux 8.7: no OFED, OFED 5.8

• AlmaLinux 9.0: no OFED, OFED 5.6, 5.7

• Rocky Linux 8.4: no OFED, OFED 5.3, 5.4

• Rocky Linux 8.5: no OFED, OFED 5.5

• Rocky Linux 8.6: no OFED, OFED 5.6

• Rocky Linux 8.8: no OFED, OFED 5.8

• Rocky Linux 9.1: no OFED, OFED 5.8

• Rocky Linux 9.2: no OFED

• SLES 15.1: no OFED, OFED 5.0

• SLES 15.2: no OFED, OFED 5.1, 5.4

• SLES 15.3: no OFED, OFED 5.4, 5.5, 5.6, 5.8

• SLES 15.4: OFED 5.5

• Debian 10: no OFED, OFED 5.2, 5.3, 5.4

• Debian 11: no OFED, OFED 5.6

• Debian 12: no OFED

• Ubuntu 18.04: no OFED

• Ubuntu 20.04: no OFED, OFED 5.4

• Ubuntu 22.04: no OFED, OFED 5.6, 5.7, 5.8

Version Interoperability

BeeGFS 7.4.2 is fully compatible with 7.4.x but not fully compatible with older versions of BeeGFS.

The incompatibilities between 7.4.2 and 7.3.x are limited to hard link functionality and a change in the monitoring message format. Please refer to the Upgrade Guide for more information.

Upgrading from Older Versions

To upgrade from an older version, please refer to the Upgrade Guide.