Node Management

See also

Target Management

Adding Nodes

Adding a new node to BeeGFS is very simple and does not require a downtime. You just have to take the following steps. Please, read the whole procedure before taking any action.

1. Ensure registration of new targets and nodes is not disabled in /etc/beegfs/beegfs-mgmtd.toml.

   registration-disable = false
   

2. If the above setting changes, the management service must be restarted before continuing.

   # systemctl restart beegfs-mgmtd
   

3. Install BeeGFS packages on the new node and configure the services, as described in the installation procedure.

4. Start the service, as seen in the example below.

   # systemctl start beegfs-storage
   

5. Use the command beegfs health check to list the server nodes in your system and check if your new node is listed. If not, check the logs of the management and new service to see if there are any relevant messages.

   $ journalctl -u beegfs-mgmtd
   $ journalctl -u beegfs-meta
   $ journalctl -u beegfs-storage
   

6. Disable node and target registrations on the management to prevent accidental registrations. Remember to restart the management service after making this change.

   registration-disable = true
   

Removing Nodes

See also

Removing Storage Targets, Data Migration, Entity IDs

Removing Metadata Nodes

Note

It is not possible to safely remove metadata nodes from the system unless the node is completely empty. For example, this would be the case if a new metadata node was registered accidentally. There is no data migration tool for metadata.

To check if a metadata node is empty, locate its data directory and check the dentries and inodes as well as buddymir/{dentries,inodes} directory for files. The node can be considered empty if the following conditions are met:

• no files (only directories) are present in the directories dentries and buddymir/dentries and their subdirectories

• no files except 38/51/root and 35/5B/disposal are present within the inodes directory and its subdirectories

• no files except 38/51/root and 23/40/mdisposal are present within the buddymir/inodes directory and its subdirectories

For example, you can use

$ find /data/beegfs/meta/{dentries,inodes,buddymir/dentries,buddymir/inodes} -type f

If the result is

/data/beegfs/meta/inodes/38/51/root
/data/beegfs/meta/inodes/35/5B/disposal
/data/beegfs/meta/buddymir/inodes/23/40/disposal

the node can be considered empty. If any more files are listed, removing the node is not safe.

If the metadata node is empty, you can execute the following command to remove it from the system:

$ beegfs node delete <entityID>

Removing Storage Nodes

Warning

Removing a node will not automatically migrate the files that were stored on this node, thus use this option carefully.

To safely remove a storage node, all of the attached storage targets have to be empty. Data can be moved off specific targets and distributed over the remaining ones using the beegfs entry migrate command:

$ beegfs entry migrate --from-nodes=<nodeID>,[nodeID] /mnt/beegfs/

Warning

Do not forcefully remove a non-empty node. This will lead to errors and possible data corruption.

Once all targets of the node are empty, it can be removed from the system:

$ beegfs node delete <entityID>

Moving a node to another host

Moving the Management Node

Take the following steps to move the management service to a different machine. Please, read the whole procedure before taking any action. We suggest downtime to move a management target to a new server!

1. Stop the management service on the current machine. If the service is configured to run in Multi Mode, be careful to stop the right instance.

   # systemctl stop beegfs-mgmtd
   # umount /data/ssd
   

2. If you don’t need the management service running on the current machine, please uninstall it.

3. If the storage device where the management database resides should be used on the new machine, unmount it and perform the necessary steps to reconnect and mount the device on the new machine ensuring it is configured to be mounted at boot time. The management database is located based on the db-file setting in /etc/beegfs/beegfs-mgmtd.toml, which may need to be adjusted depending where the device is mounted.

   1. Otherwise, if the device cannot or should not be moved or connected to the new machine, copy the management database from a backup (or from the device, remounted somewhere else) to a partition of the new machine.

4. Check if the management service is already installed on the new machine. If not, please install it. Do not start the service at this point. If the management service is already installed on the new server machine, please configure the service to run in multi-mode.

5. Then, edit the file /etc/beegfs/beegfs-mgmtd.toml on the new machine and set option db-file to the path to the management service directory. If the service is configured to run in multi-mode, be careful to use the configuration file and directory of the right instance.

6. If the new machine has a different hostname, edit the configuration file of each other BeeGFS service running in your system, which are associated to that management service, and write the new hostname as the value of option sysMgmtdHost.

7. Make sure that the new machine is reachable from the other BeeGFS servers, specially if you are using the client configuration options connInterfacesFile or connNetFilterFile to restrict which network interfaces can be used by BeeGFS services. Its IP address may be changed freely, because BeeGFS keeps no record of such addresses.

8. Start or restart the management service.

   # systemctl restart beegfs-mgmtd
   

9. Test if the target is working properly.

   • Check if the logs contain any error messages:

     $ journalctl -u beegfs-mgmtd
     $ journalctl -u beegfs-meta
     $ journalctl -u beegfs-storage
     $ journalctl -u beegfs-client
     $ journalctl -k // For client kernel module logs.
     

   • Verify the management node is online with the expected NICs listed:

     $ beegfs node list --node-type=management --with-nics --reachability-check
     

Moving a Metadata Node

Take the following steps to move a metadata target to a different machine. Please, read the whole procedure before taking any action. This description focuses on the transfer of the storage device from one server to another, but it is possible to use a backup. If you are going to move the metadata target by means of a backup, please read the backup metadata instructions (see Management Daemon Backup). We suggest downtime to move a metadata target to a new server!

1. Stop the metadata service on the current machine and unmount the metadata target device. If the service is configured to run in multi-mode, be careful to stop the right instance.

   # systemctl stop beegfs-meta
   # umount /data/ssd
   

2. If you don’t need the metadata service running on the current machine, please uninstall it.

3. Check if the metadata target device can be moved or connected to the new machine. If so, mount it on a directory of the new machine and make sure it is configured to be mounted at boot time.

4. Otherwise, if the metadata target device cannot be moved or connected to the new machine, copy the data from a backup (or from the device, remounted somewhere else) to a storage device of the new machine (see Management Daemon Backup).

5. Check if the metadata service is already installed on the new machine. If not, please install it. If the metadata service is already installed on the new server machine, please configure the service to run in Multi Mode. Do not start the service at this point.

6. Then, edit the file /etc/beegfs/beegfs-meta.conf on the new machine and add the path of the moved metadata target to option storeStorageDirectory. If the service is configured to run in multi-mode, be careful to use the configuration file and directory of the right instance.

7. Make sure that the file nodeNumID exists in the storage directory and that it contains the ID that the metadata service daemon should use to identify itself with the management service.

8. Start or restart the metadata service.

   # systemctl restart beegfs-meta
   

9. Test if the target is working properly.

   • Check if the logs contain any error messages:

     $ journalctl -u beegfs-mgmtd
     $ journalctl -u beegfs-meta
     $ journalctl -u beegfs-storage
     $ journalctl -u beegfs-client
     $ journalctl -k // For client kernel module logs.
     

   • List all metadata nodes of your system and verify the one that was moved is online with the expected NICs listed:

     $ beegfs node list --node-type=meta --with-nics --reachability-check
     

Moving a Storage Node

We recommend Moving a Storage Target to another node.