Node Management

See also

Target Management

Adding Nodes

Adding a new node to BeeGFS is very simple and does not require a downtime. You just have to take the following steps. Please, read the whole procedure before taking any action.

1. Make sure that the following properties are set to “true” in file /etc/beegfs/beegfs-mgmtd.conf, in order for the management service to allow new nodes and targets to be added to the system. You will find detailed documentation of those properties on the bottom of the file.

   sysAllowNewServers = true
   sysAllowNewTargets = true
   

2. If you change the properties above, please restart the management service afterward.

   # systemctl restart beegfs-mgmtd
   

3. Install BeeGFS packages on the new node and configure the services, as described in the installation procedure.

4. Start the service, as seen in the example below.

   # systemctl start beegfs-storage
   

5. Use the command beegfs-check-servers to list the server nodes in your system and check if your new node is listed. If not, check the log files of the added service and the management service to see if any error message informs you why the node could not be added.

   $ less /var/log/beegfs-mgmtd.log
   $ less /var/log/beegfs-meta.log
   $ less /var/log/beegfs-storage.log
   

6. Set the management properties back to false, to prevent accidental registrations of new nodes and targets in your system. Please, restart the management service afterward.

   sysAllowNewServers = false
   sysAllowNewTargets = false
   

Removing Nodes

Removing Metadata Nodes

Note

It is not possible to safely remove metadata nodes from the system unless the node is completely empty. For example, this would be the case if a new metadata node was registered accidentally. There is no data migration tool for metadata.

To check if a metadata node is empty, locate its data directory and check the dentries and inodes as well as buddymir/{dentries,inodes} directory for files. The node can be considered empty if the following conditions are met:

• no files (only directories) are present in the directories dentries and buddymir/dentries and their subdirectories

• no files except 38/51/root and 35/5B/disposal are present within the inodes directory and its subdirectories

• no files except 38/51/root and 23/40/mdisposal are present within the buddymir/inodes directory and its subdirectories

For example, you can use

# find /data/beegfs/meta/{dentries,inodes,buddymir/dentries,buddymir/inodes} -type f

If the result is

/data/beegfs/meta/inodes/38/51/root
/data/beegfs/meta/inodes/35/5B/disposal
/data/beegfs/meta/buddymir/inodes/23/40/disposal

the node can be considered empty. If any more files are listed, removing the node is not safe.

If the metadata node is empty, you can execute

# beegfs-ctl --removenode --nodetype=meta <nodeID>

to remove it from the system.

Removing Storage Nodes

Warning

Removing a node will not automatically migrate the files that were stored on this node, thus use this option carefully.

To safely remove a storage node, all of the attached storage targets have to be empty. Data can be moved off specific targets and distributed over the remaining ones using the beegfs-ctl --migrate command:

# beegfs-ctl --migrate --nodeid=<nodeID> /mnt/beegfs

Warning

Do not forcefully remove a non-empty node. This will lead to errors and possible data corruption.

Once all targets of the node are empty, it can be removed from the system:

# beegfs-ctl --removenode --nodetype=storage <nodeID>

See also

Removing Storage Targets Data Migration

Moving a node to another host

Moving the Management Node

Take the following steps to move the management service to a different machine. Please, read the whole procedure before taking any action. We suggest downtime to move a management target to a new server!

1. Stop the management service on the current machine. If the service is configured to run in Multi Mode, be careful to stop the right instance.

   # service beegfs-mgmtd stop
   # umount /data/ssd
   

2. If you don’t need the management service running on the current machine, please uninstall it.

3. Check if the storage device where the management directory resides can be moved or connected to the new machine. If so, unmount it, move it, mount it on a directory of the new machine, and make sure it is configured to be mounted at boot time. The management directory is defined by option storeMgmtdDirectory in file /etc/beegfs/beegfs-mgmtd.conf.

4. Otherwise, if the device cannot be moved or connected to the new machine, copy the management directory from a backup (or from the device, remounted somewhere else) to a partition of the new machine.

5. Check if the management service is already installed on the new machine. If not, please install it. Do not start the service at this point. If the management service is already installed on the new server machine, please configure the service to run in multi-mode.

6. Then, edit the file /etc/beegfs/beegfs-mgmtd.conf on the new machine and set option storeMgmtdDirectory to the path to the management service directory. If the service is configured to run in multi-mode, be careful to use the configuration file and directory of the right instance.

7. Make sure that the file nodeID exists on the service directory and that it contains the ID that the management service daemon should use to identify itself. If the file does not exist, just create it with the same content of the originalNodeID file.

8. If the new machine has a different hostname, edit the configuration file of each other BeeGFS service running in your system, which are associated to that management service, and write the new hostname as the value of option sysMgmtdHost.

9. Make sure that the new machine is reachable from the other BeeGFS servers, specially if you are using the client configuration options connInterfacesFile or connNetFilterFile to restrict which network interfaces can be used by BeeGFS services. Its IP address may be changed freely, because BeeGFS keeps no record of such addresses.

10. Start or restart the management service.

    # service beegfs-mgmtd restart
    

11. Test if the target is working properly.

    • Check if log files contain any error message.

      $ less /var/log/beegfs-mgmtd.log
      $ less /var/log/beegfs-meta.log
      $ less /var/log/beegfs-storage.log
      $ less /var/log/beegfs-client.log
      

    • List all management nodes of your system and check if the moved one appears online.

      $ beegfs-ctl --listnodes --nodetype=mgmt --reachable
      

Moving a Metadata Node

Take the following steps to move a metadata target to a different machine. Please, read the whole procedure before taking any action. This description focuses on the transfer of the storage device from one server to another, but it is possible to use a backup. If you are going to move the metadata target by means of a backup, please read the backup metadata instructions (see Metadata Daemon Backup). We suggest downtime to move a metadata target to a new server!

1. Stop the metadata service on the current machine and unmount the metadata target device. If the service is configured to run in multi-mode, be careful to stop the right instance.

   # service beegfs-meta stop
   # umount /data/ssd
   

2. If you don’t need the metadata service running on the current machine, please uninstall it.

3. Check if the metadata target device can be moved or connected to the new machine. If so, mount it on a directory of the new machine and make sure it is configured to be mounted at boot time.

4. Otherwise, if the metadata target device cannot be moved or connected to the new machine, copy the data from a backup (or from the device, remounted somewhere else) to a storage device of the new machine (see Metadata Daemon Backup).

5. Check if the metadata service is already installed on the new machine. If not, please install it. If the metadata service is already installed on the new server machine, please configure the service to run in Multi Mode. Do not start the service at this point.

6. Then, edit the file /etc/beegfs/beegfs-meta.conf on the new machine and add the path of the moved metadata target to option storeStorageDirectory. If the service is configured to run in multi-mode, be careful to use the configuration file and directory of the right instance.

7. Make sure that the file nodeID exists on the service directory and that it contains the ID that the metadata service daemon should use to identify itself with the management service. If the file does not exist, just create it with the same content of the originalNodeID file.

8. Start or restart the metadata service.

   # service beegfs-meta restart
   

9. Test if the target is working properly.

   • Check if log files contain any error message.

     $ less /var/log/beegfs-mgmtd.log
     $ less /var/log/beegfs-meta.log
     $ less /var/log/beegfs-storage.log
     $ less /var/log/beegfs-client.log
     

   • List all metadata nodes of your system and check if the moved one appears online.

     $ beegfs-ctl --listnodes --nodetype=meta --reachable
     

Moving a Storage Node

We recommend Moving a Storage Target to another node.