Management Service

The following section includes advanced details about the internal workings of the new management service introduced with BeeGFS 8. If you are looking for details how to upgrade from the BeeGFS 7 to BeeGFS 8 management service, refer to the 8.0.0 upgrade guide. If you are looking for how to upgrade the database schema refer to the upgrade troubleshooting section.

Entity IDs

Entities in BeeGFS including nodes, targets, and mirrors have an associated “entity ID” that consists of three parts:

  • An alias: This is an arbitrary user defined string that can be changed at any time. Aliases are meant to be a user friendly way for identifying entities, for example by a hostname or volume name. Aliases must be unique across all entities in BeeGFS.

    • Note, formerly aliases were known as string IDs and could not be changed.

  • A numerical ID: This is a numerical ID that only needs to be unique amongst a particular entity type (e.g. nodes, targets, mirrors).

    • When interacting with entities using their numerical ID in the beegfs tool you will often specify the numerical ID with a type prefix since these IDs are not unique across all entity types. For example if you are running a command to interact with a nodes, you would use meta:1 or storage:1 to distinguish between the meta node or storage node with ID 1.

  • A unique ID (UID): This ID is automatically generated for each entity when it is first added, and cannot be changed. They are mainly meant for internal use and typically users will not need to interact with entities using their UIDs.

Database

The BeeGFS management service uses a SQLite database to store information about the systems structure, the registered nodes, targets, groups, pools; quota information and more. Generally users do not need to be concerned about this database, other than potentially customizing the path where it exists using db-file. This section gives advanced details about the database internals which may be helpful to understand how BeeGFS works and analyze/debug BeeGFS installations. Basic knowledge of SQL is required.

Warning

Before interacting with the database you should create a Backup and run your queries against the database backup instead of the active database in use by the management service. Please contact ThinkParQ support for assistance if you need to modify the active database.

Schema

The relationship between the different kinds of “entities” (nodes, targets, storage pools and buddy groups) of a BeeGFS cluster is encoded in and enforced by the database schema.

Some of the relations are further restricted by additional foreign keys, enforcing correct links between entries in “generic” tables like targets and type specific tables like root_inode. These are the reason for most of the double relationship lines.

The exact schema definition can extracted from your database using

sqlite3 mgmtd.sqlite .schema > schema.sql

Note that management creates the database schema incrementally to allow easy upgrades of the database schema. Thus you can’t find the complete schema definition directly in the source code.

A rough description of the tables and relations follows below.

See also

Why does the management log the database schema is outdated?

Entities

  • On top is the entities table. It contains exactly one entry for each entity as defined above and defines its global UID and global alias.

  • The nodes table contains exactly one entry for each node and has a 1:1 relation with entities. It holds node related data, e.g. node type, numeric id and the nodes TCP and UDP port.

  • The targets table contains exactly one entry for each target and has a 1:1 relation with entities. It holds target related data, e.g. target type, numeric id, free space and inode information. Note that BeeGFS does not really support meta targets at the moment, but internally, the management daemon (and also some other stuff) has exactly one meta target with the same numerical ID as the meta node.

  • The buddy_groups table contains exactly one entry for each buddy group and has a 1:1 relation with entities. It holds buddy group data, e.g. buddy group type, numeric id and its primary and secondary target.

  • The pools table contains exactly one entry for each storage pool and has a 1:1 relation with entities. It holds storage pool related data.

Other

  • The node_nics table stored the network interfaces (e.g. IP address, name and type) reported by and assigned to each node. It has a n:1 relationship with nodes.

  • The root_inode table contains the meta node or buddy group that has the root inode.

  • The quota_default_limits table contains the default quota limits for storage pools.

  • The quota_limits table contains quota limits for specific users / groups per storage pool.

  • The quota_usage table contains the used quota for users / groups per storage target.

  • The config table contains the management / system configuration.

  • The *_types tables contain information which type of various things exist and their name/description.

Views

The schema contains several views that make it easier to access relevant information together. A view is a predefined select statement which can be select from as it was a table.

The following types of views currently exist:

  • (nodes|targets|*)_ext: Extends a entity specific table with some often used data (e.g. the alias joined from the entities table)

  • (meta|storage|*)_(nodes|targets|*): Limits a generic table to one type (meta / storage / *)

Note that these exist mainly to shorten the SQL statements in the source code when accessing some often used data. They are not meant for convienience when investigating the db. They are intentionally very minimal to avoid fetching unneeded data in operations.

That said, the _ext views provide slightly more data, so they can be useful (the buddy_groups_ext in particular).

Extracting Information

To obtain information about a BeeGFS system, the database can be queried manually. Since SQLite databases are contained within one file (not counting a potential write ahead log), they can be easily shared (e.g. with ThinkParQ support).

Querying an in-use database is also possible and useful to get information about the running system.

Warning

Do only use SELECT statements on the in-use database, do not modify it, unless you know what you are doing. Read the section below.

There are several graphical database exploration tools available, but the most straight forward way is to use the sqlite3 command line tool:

  1. Open the file: sqlite3 mgmtd.sqlite.

  2. Set .headers on and .mode columns to make SELECT output human readable.

  3. Do your queries.

Example queries

Show targets including their aliases:

SELECT * FROM targets_ext;

The information in the targets table might not be enough, so we join the nodes table:

SELECT * FROM targets_ext
INNER JOIN nodes_ext USING (node_id, node_type);

Show only meta targets:

SELECT * FROM targets_ext
INNER JOIN nodes_ext USING (node_id, node_type)
WHERE node_type = 1;

Note that the 1 for meta nodes is taken from the node_types table - 1 refers to “meta”. These values are fixed and should never be changed.

Modifying the database

If necessary, the management database can be modified by hand. Since most changes (known exceptions: the config table) to the system state are made persistent (= written to the database) immediately, this can usually even be done while management is still running.

Warning

Manually modifying the database can affect its integrity. While most of it is protected by foreign keys (which are not necessarily enforced, see below), some aspects might not be. So, only do that if you know what you are doing.

Warning

SQLite does not enforce foreign key relations unless configured to do so. This is disabled by default and should be enabled by executing pragma foreign_keys = true; after opening the file with sqlite3 or enabled in the settings of your database exploration tool. When using the C interface, make sure, the connection settings SQLITE_DBCONFIG_ENABLE_TRIGGER and SQLITE_DBCONFIG_ENABLE_FKEY are set. See https://sqlite.org/c3ref/ c_dbconfig_defensive.html for more information.