BeeGFS v8.0.0 Upgrade Guide

Please read and understand the information below completely before attempting the upgrade.

Possible upgrade paths

The following upgrade paths from older versions are supported:

  • v6.x ⟶ v7.4.6 ⟶ v8.0.0

  • v7.x ⟶ v7.4.6 ⟶ v8.0.0

When upgrading from BeeGFS 7.3 or earlier, if hard links exist in the file system they should first migrated to the new format before upgrading from v7.4.6 to v8.0.0. Refer to the 7.4.6 upgrade guide for more details.

Before Upgrading

  1. BeeGFS 8 introduces license enforcement. Review the licensing documentation and obtain a license if needed and install it on your management node(s). The default path expected by all BeeGFS 8 components is /etc/beegfs/license.pem.

  2. The BeeGFS management and beegfs (CTL) tool now support TLS encryption. Familiarize yourself with the options to configure TLS in BeeGFS and decide how you want to configure TLS and obtain TLS certificates and keys if needed.

    1. The recommended/easiest approach is to place your cert file and key on your management node at /etc/beegfs/cert.pem and /etc/beegfs/key.pem and install the same cert file at /etc/beegfs/cert.pem on all nodes that will use the new beegfs tool. These are the default paths expected by all BeeGFS 8 components.

  3. If you are using File System Modification Events note this feature has been extensively reworked. Ensure your listeners are ready for these changes:

    • If you are using the beegfs-event-listener the event packet format has changed significantly and applications that consume events will need to be updated accordingly.

    • If you are using a custom third-party listener please consult with support for assistance migrating to the new file event protocol and event format.

  4. Check all servers and clients are on supported Linux distributions and kernel versions.

Upgrade to v8.0.0

Warning

BeeGFS 8 introduces license enforcement. Ensure you know if this system will need a license before you proceed and already have a license file if needed.

  1. Run the following commands and save the output:

    • If for some reason the automatic import step below fails, the metadata node or buddy group that owns the root inode will be required by support to manually import your BeeGFS 7 management data into the new database format:

      beegfs-ctl --getentryinfo --verbose /mnt/beegfs

    • If for some reason both members of a mirror come up in “needs-resync”, it is important to know the original primary target:

      beegfs-ctl --listtargets --longnodes --state --spaceinfo --mirrorgroups --nodetype=meta
beegfs-ctl --listtargets --longnodes --state --spaceinfo --mirrorgroups --nodetype=storage

  2. If you use buddy mirroring, make sure that all buddies are in GOOD state in the previous output before proceeding.

  3. Stop the BeeGFS system:

    • stop/unmount all clients

    • stop all storage services

    • stop all meta services including any event listeners (if in use)

    • stop the management service

  4. Refer to the backup guide and take a full system backup if possible. If a full backup is not possible ensure you at least take a minimal backup of important system configuration.

    Note

    A minimal backup of important system configuration after stopping all services and before upgrading packages is strongly recommended. Proceed without any backups at your own risk!

  5. Download the BeeGFS 8 package repository file for your distribution to all nodes. Visit the BeeGFS download page and follow the step-by-step directions to add the BeeGFS package repositories to your Linux distribution’s package manager.

  6. On the management node upgrade the beegfs-mgmtd package and complete the following manual steps required to migrate to the new management service. The management service is upgraded first to ensure your v7 configuration can be migrated to the new v8 database format. If there are any issues with the migration this simplifies rolling back the management upgrade so you can remain on BeeGFS 7 while you work with support to correct any configuration issues.

    1. Import your BeeGFS 7 management data into the new database format by running:

      /opt/beegfs/sbin/beegfs-mgmtd --import-from-v7=/path/to/old/management/dir

      Note

      The automatic import may not work with every installation. This is because the new management enforces stricter requirements to ensure the system state is valid. For example if you have targets assigned to storage pools that don’t exist the import will fail. If the migration fails manual intervention is necessary, DO NOT continue with the upgrade and contact ThinkParQ support for assistance. Optionally you can downgrade all BeeGFS packages and continue using BeeGFS 7 in the meantime.

    2. The old beegfs-mgmtd.conf configuration file does not work with the new management service. Administrators are required to manually transfer any customized settings from their old configuration file to the new beegfs-mgmtd.toml format before initially starting the v8.0.0 management service. Notably:

      1. The path to your auth-file must be updated with the actual path or authentication disabled if not in use for your environment. If your auth file already exists at /etc/beegfs/conn.auth no changes are required to the configuration file.

      2. If quotas are in use ensure to reenable quota tracking and optionally enforcement. You will also need to specify which user and group IDs should be queried by the quota system. Refer to the management configuration file and quota documentation for details.

      Note

      The new management service only supports logging to the system journal or stderr.

    3. Follow the steps to Configure TLS for a server service. If you are using TLS and have already placed your cert and key files at /etc/beegfs/cert.pem and /etc/beegfs/key.pem on your management node then nothing else needs to be done.

    4. If you wish to use Hive Enterprise features setup licensing by installing the libbeegfs-license package and downloading your license to /etc/beegfs/license.pem.

    5. The management’s systemd unit file has changed so run systemctl daemon-reload then start the management service. Verify the service starts and check journalctl -u beegfs-mgmtd for any errors.

  7. Upgrade the beegfs-meta and beegfs-storage packages on all metadata and storage nodes. Review the release notes for any configuration options that are no longer valid, or new options you may wish to configure on your metadata and storage services. Notably:

    1. If your metadata servers are configured to log file system modification events (i.e., you have enabled sysFileEventLogTarget in beegfs-meta.conf), configure the new metadata settings to customize how much disk space is used to persist events, and where events are persisted if you do not want them to be stored in the metadata target. Update and restart whatever application is listening for events to ensure it understand the new file event protocol. If you are using the beegfs-event-listener this can be done by updating the beegfs-utils package on all metadata servers.

  8. Start all metadata and storage services in the following order:

    1. If you are using buddy mirroring start all primary metadata and storage services in each buddy group then check their logs to ensure they have reconnected to the management service before starting secondary services. Failure to do so may cause unnecessary switchovers.

    2. Metadata and/or storage services that are not using buddy mirroring can be started in any order.

    Always verify all services have started and connected to the management before moving on.

  9. Upgrade the beegfs-client package on all client nodes.

    1. It is recommended (but not required) to remove deprecated client configuration options from (beegfs-client.conf). Specifically connHelperdPortTCP, logHelperdIP, and logType are no longer needed as the client now always logs to the system journal.

  10. Start the beegfs-client service on all client nodes to mount BeeGFS.

    1. If mounting BeeGFS fails check journalctl -k for any errors related to BeeGFS. If you see errors like Unable to proceed without a working root metadata node contact support as it is likely the v7 import was unable to identify/set the root metadata node in the database.

    2. Once BeeGFS is mounted verify you can interact with files and directories inside BeeGFS. If you see a Remote I/O error and all services are reachable by this node, contact support to verify your v7 system was imported correctly.

    If you see these or similar errors don’t panic! BeeGFS is designed to fail gracefully if something about the system state after an upgrade is not quite right. Most issues can be quickly corrected without needing to rollback to BeeGFS 7, simply contact support for assistance.

  11. Upgrade the beegfs-utils package on any clients/other nodes where it is installed. Install the new beegfs tool by installing the beegfs-tools package and configure it as follows:

    1. If BeeGFS is mounted and your conn auth file is present at the default /etc/beegfs/conn.auth path and are using a self-signed TLS certificate that is present at the default /etc/beegfs/cert.pem path then no further configuration is needed and the management node will be auto determined.

    2. If you have multiple different BeeGFS instances mounted or if BeeGFS is not mounted you will need to specify the management address. The new tool does not use a configuration file, but rather uses flags and/or environment variables for configuration. This means if you want persistent configuration for the tool it can be set in your ~/.bashrc file (or equivalent for your shell), for example:

      echo "export BEEGFS_MGMTD_ADDR='<BEEGFS-MGMTD-IP-OR-HOSTNAME>:8010'" >> ~/.bashrc
source ~/.bashrc

    Note

    The beegfs-ctl tool formerly provided by the beegfs-utils package has been replaced by the new beegfs tool. The beegfs-utils package is now only needed on nodes that need to run beegfs-fsck or on metadata nodes using the modification events feature.

  12. After finishing the upgrade to BeeGFS 8 run the following commands to verify the upgrade:

    1. Verify the root inode is owned by the same metadata node as before. This should happen automatically if you use the import-from-v7 functionality in the new management service.

      beegfs entry info /mnt/beegfs

    2. Verify all nodes and targets are online and in a good state: beegfs health check

      beegfs health check

      Tip

      If the “Available Capacity” check warns that targets are low on free space, you can adjust the “capacity pool” thresholds defined in the beegfs-mgmtd.toml so they are better suited to your environment.

  13. Lastly, remove the now unused packages beegfs-common and beegfs-helperd.

FAQs

The following section addresses common questions when upgrading BeeGFS.

Why does the management log the database schema is outdated?

After upgrading a system already running BeeGFS 8 to a newer version, the management service refuses to start and logs an error like:

Database schema version 1 is outdated. Please upgrade to latest version 2

Starting in BeeGFS 8, management data is stored in a SQL database. New BeeGFS versions may introduce changes to the database schema, for example to add new fields. The management includes a mechanism for upgrading (i.e., “migrating”) the database schema after an upgrade by manually running:

/opt/beegfs/sbin/beegfs-mgmtd --upgrade

Note

If your database is located at a non-standard path, ensure to also specify --db-file.

The upgrade process will first backup the current database in the same directory as the existing one, with the old version as the suffix. For example by default backups will exist at a path like:

/var/lib/beegfs/mgmtd.sqlite.v1

The backup is mainly needed if you need to downgrade BeeGFS. Because the database size is typically small, it is recommended to keep around at least one previous version indefinitely.

Provided the backup succeeds the database upgrade will be attempted. If for some reason the upgrade fails an error will be logged, for example if you tried to upgrade between unsupported versions. If the resolution is not clear please open a support ticket. If you choose to downgrade in the meantime, please perform the following steps:

  • Record the error(s) logged from the --upgrade.

  • Downgrade all BeeGFS packages.

Note if the upgrade fails the original database file will be left untouched, so there should be no reason to overwrite the current database file with the backup after a failed database upgrade. The backup is intended if you need to downgrade BeeGFS for some other reason.