Access Control Lists

BeeGFS supports Access Control Lists (ACLs) in two independent forms, both stored as Extended Attributes of the metadata files on the metadata server:

• POSIX ACLs – the traditional getfacl/setfacl model, described in the remainder of this section

• NFSv4 ACLs – a richer, NFSv4-style model (experimental), described under NFSv4 ACLs

The two are separate: they use different extended attributes, tools, and enforcement rules, and a deployment normally uses one or the other. Everything below, up to NFSv4 ACLs, refers to POSIX ACLs.

POSIX ACLs require the client nodes to run Linux kernel version 3.1 or above because the ACL enforcement facilities BeeGFS relies on are not available on older kernels.

To enable POSIX ACLs, edit the metadata configuration file (/etc/beegfs/beegfs-meta.conf) and check if the following options are set to true.

storeClientXAttrs       = true
storeClientACLs         = true

Then, edit the client configuration file (/etc/beegfs/beegfs-client.conf) and check if the following options are set to true.

sysXAttrsEnabled = true
sysACLsEnabled   = true

If you change the configuration files, remember to restart their respective services afterward.

Note that enabling POSIX ACLs comes at some cost of reduced metadata performance, because additional data needs to be transferred and checked.

POSIX ACL caching and revalidation

Until BeeGFS 8.2, the client module took a very conservative approach to caching and revalidating ACLs. Locally cached ACLs were invalidated on every access, leading to increased latency and metadata load because of large numbers of RPCs to the meta servers needed to revalidate and fetch them every time they were accessed on any client in the system.

BeeGFS 8.2 addresses this by changing the default behavior and introducing a new configuration option sysACLsRevalidate that can be set to either cache or always. The new default cache option configures the client to cache ACLs for the same time other file or directory attributes including access permissions are cached. That time can be adjusted by changing the values of tuneDirSubentryCacheValidityMS (for directories) and tuneFileSubentryCacheValidityMS (for files). Even with the default time settings of 1s for directories and no cache (0s) for files, the cache option for sysACLsRevalidate has a significant positive effect on performance because ACLs for path components will be cached during path resolution and repeated access to the same directory. The effects can be improved by setting longer cache times, which comes at the expense of using a slightly increased propagation time for ACLs between clients. In the vast majority of cases, the performance benefits should outweigh that cost, but in case strict guarantees for ACL propagations are required, sysACLsRevalidate can be set to always to revert back to legacy behavior and revalidate ACLs on every access.

Additional performance considerations with Extended Attributes

The Linux kernel uses a security mechanism that automatically removes setuid/setgid bits and capabilities from files when they are changed. This is done to prevent users from executing binaries with elevated privileges that were changed after the privileges were originally set. That mechanism requires that, by default, the kernel has to check each file for existing capabilities on every write which leads to a large overhead in metadata RPCs to fetch the “security.capability” extended attribute. To optimize this, Linux allows file systems to set a flag (S_NOSEC) on the file, which short-circuits these checks.

The sysXAttrsCheckCapabilities client configuration option configures the file system mount to allow the client to set that flag on new inodes either immediately and without any checks at all (sysXAttrsCheckCapabilities = never), after a first lookup of the extended attribute returns an empty result (sysXAttrsCheckCapabilities = cache) or not at all to always check (sysXAttrsCheckCapabilities = always). The flag will automatically be cleared when capabilities are modified on this client. It will, however, currently not be cleared if a different client modifies capabilities or sets setuid/setgid bits, which can lead to capabilities not being cleared, even after the file is written to. If this is a concern, the option should be set to always.

As long as BeeGFS is mounted using the “nosuid” mount option (which is recommended and the default setting), elevating privileges via setuid/setgid bits and capabilities is disabled and it is safe to set sysXAttrsCheckCapabilities to never.

NFSv4 ACLs

Warning

NFSv4 ACL support is experimental: the enforcement semantics are still being refined and may change in future BeeGFS releases. Do not rely on them for security-critical access control yet, and review Known issues and limitations before deploying.

In addition to POSIX ACLs, BeeGFS supports NFSv4-style ACLs, stored as the system.nfs4_acl extended attribute on the metadata server. To enable them, set the following in the metadata configuration file (/etc/beegfs/beegfs-meta.conf):

storeClientXAttrs = true
storeNFSv4ACLs    = true

and in the client configuration file (/etc/beegfs/beegfs-client.conf):

sysNFSv4ACLsEnabled = true

Enabling sysNFSv4ACLsEnabled implicitly requires extended-attribute support on the client. Restart the respective services after changing the configuration.

For more general information about NFSv4 ACLs that is not covered in this BeeGFS-specific guide consider reading the nfs4_acl(5) manpage or their specification in section 6 of RFC 7530.

Setting and reading NFSv4 ACLs

NFSv4 ACLs are managed with the nfs4-acl-tools package (nfs4_getfacl, nfs4_setfacl, nfs4_editfacl), which read and write the system.nfs4_acl extended attribute that BeeGFS stores on the metadata server.

Read the ACL of a file or directory:

nfs4_getfacl /mnt/beegfs/project

An NFSv4 ACL consists of zero or more Access Control Entries (ACEs). A file that has no ACL set returns an empty ACL (zero ACEs) rather than an error.

Each ACE is written as type:flags:principal:permissions:

• type – A (allow) or D (deny). U (audit) and L (alarm) ACEs are accepted and stored but ignored for access decisions.

• flags – e.g. g marks the principal as a group, fd/di/oi/ni are inheritance flags (see below).

• principal – one of the special principals OWNER@, GROUP@, EVERYONE@, a numeric UID/GID (e.g. 1001), or an Active Directory SID string (S-1-5-21-...). See SID resolution and Active Directory configuration.

• permissions – the NFSv4 permission letters (r w a x t T n N c C d D o y), or the nfs4_setfacl shorthand aliases R / W / X for generic read / write / execute. The aliases expand to R = rntcy, W = watTNcCy (plus D on directory ACEs), X = xtcy. See the nfs4_setfacl(1) man page for the full set.

Grant a user read and execute, and append a catch-all deny (the recommended whitelist form):

nfs4_setfacl -a "A::1001:RX" /mnt/beegfs/project
nfs4_setfacl -a "D::EVERYONE@:RWX" /mnt/beegfs/project

Replace the whole ACL, or remove it entirely:

nfs4_setfacl -s "A::OWNER@:RWX,A:g:GROUP@:RX,D::EVERYONE@:RWX" /mnt/beegfs/project
setfattr -x system.nfs4_acl /mnt/beegfs/project   # remove the ACL

Note

On Linux, because the VFS checks coarse access modes, BeeGFS expands each one into a bundle of ACE bits (see Permission granularity for detailed definitions). An ALLOW ACE only decides a request if it grants the whole bundle:

• read – r, t, n, c

• write – w, a, T, N (plus D on directories)

• execute / directory lookup – x

An ALLOW that grants only r leaves the remaining read bits undecided, so the request falls through to the POSIX mode bits instead of being allowed by the ACL. The simplest way to avoid this is to use the R / W / X aliases above: each is a superset of the corresponding bundle (the extra bits, such as SYNCHRONIZE, are among those BeeGFS ignores), so RX, RWX and the like always grant a complete bundle. Note that nfs4_getfacl displays the expanded letters, not the R/W/X shorthand.

Setting or removing the ACL requires ownership of the file (or CAP_FOWNER). See Reading and modifying the ACL itself. The blob is validated on the client before it is stored, so a malformed ACL is rejected with an error and never written.

Inheritance

ACEs carrying inheritance flags on a directory are propagated to newly created entries: files inherit FILE_INHERIT ACEs, subdirectories inherit DIRECTORY_INHERIT (and carry FILE_INHERIT forward as INHERIT_ONLY for their own children), and NO_PROPAGATE_INHERIT stops propagation one level down, following RFC 3530/5661. Inheritance is applied once, at creation time – changing a directory’s ACL does not retroactively rewrite the ACLs of existing children.

How NFSv4 ACLs are resolved and enforced

Unlike a classic NFSv4 server, BeeGFS does not resolve or enforce ACLs centrally. The metadata server stores the ACL as an opaque blob. Each client resolves the named principals against its own local identity information and evaluates the ACL at access time. A principal is matched directly, with no prefix or name translation, so the supported forms are:

• the special principals OWNER@, GROUP@ and EVERYONE@ (matched structurally, never resolved)

• a numeric ID – interpreted as a UID unless the ACE carries the group flag (g in nfs4_setfacl, ACE4_IDENTIFIER_GROUP on the wire), in which case it is a GID

• an Active Directory Security Identifier (SID, begins with S-1), resolved as described in SID resolution and Active Directory configuration

The same group flag distinguishes a user SID from a group SID. A user ACE applies to the accessing process’s UID. A group ACE applies when its group is the process’s primary group or one of its supplementary groups. Any other form, including NFS-style name@domain, cannot currently be resolved and matches no subject (the ACE is skipped). Because resolution is per client, principal mapping must be configured consistently across the cluster, or a principal may resolve on one client but not another. When the ACL does not decide a request, BeeGFS falls back to the POSIX mode-bit check.

Note

Preferably write NFSv4 ACLs as a whitelist: grant access with explicit ALLOW entries for the trusted principals, and terminate the ACL with a catch-all DENY EVERYONE@ entry.

This pattern is robust regardless of which identities a given client can resolve:

• An ALLOW whose principal a client cannot resolve simply grants nothing and falls through to the catch-all, so the whitelist fails safe. The catch-all itself (EVERYONE@ or the mode bits) needs no resolution and is enforced consistently on every client

• The trailing DENY EVERYONE@ also makes the ACL self-contained: every access right not covered by an ALLOW entry is decided by the ACL itself, independent of the file’s mode bits

Relying on the POSIX mode bits as the catch-all is an equally valid alternative. Instead of (or in addition to) a trailing DENY EVERYONE@, you can let unmatched requests fall back to the mode-bit check and keep those bits restrictive (e.g. 0600 or 0700), so that any subject not covered by an ALLOW entry is denied. This is just as robust against unresolved principals – an unresolved ALLOW grants nothing and falls through to the mode bits, which need no name resolution. The trade-off is where authority lives: with the mode-bit backstop, safety depends on those bits staying restrictive (a later chmod that widens them widens fallback access), whereas a trailing DENY EVERYONE@ keeps the ACL authoritative regardless of the mode bits. Use the explicit DENY when you want the ACL to stand on its own. Rely on the mode bits when you prefer to keep the familiar Unix permissions as the backstop.

On a correctly configured cluster – where every client resolves the same identities – the inverse “blacklist” pattern (a broad ALLOW combined with DENY entries for specific named principals) is enforced as written. It is, however, less robust: such a denial depends on the accessing client being able to resolve that named principal, so on a client with incomplete identity information it may not take effect. The whitelist form avoids this dependency and is therefore the safer choice.

SID resolution and Active Directory configuration

Special and numeric principals need no external lookup. Active Directory SIDs are resolved per client through the kernel request_key() upcall: the client requests a key of type logon whose description is the SID prefixed with os: (user / owner SID) or gs: (group SID), and expects the resolved id back as a 4-byte little-endian uid/gid. This matches the in-kernel SMB/CIFS client’s key format, so the lookup is served by cifs.idmap (from cifs-utils) backed by either winbind or SSSD.

Prerequisites

SID resolution relies on standard SMB/CIFS identity infrastructure, configured per the Samba, SSSD, and cifs-utils documentation. On every client:

• The host is joined to the Active Directory domain with a working SID-to-UID/GID resolver – winbind (Samba) or SSSD. Use a deterministic mapping (e.g. winbind idmap_rid / idmap_ad, or SSSD POSIX attributes) so that every client, and the POSIX ownership stored on the metadata servers, resolves a given SID to the same id.

• cifs-utils is installed and its id-mapping plugin is pointed at that backend (idmapwb.so for winbind, cifs_idmap_sss.so for SSSD, selected via /etc/cifs-utils/idmap-plugin).

BeeGFS-specific configuration

The only BeeGFS-specific step is to route the kernel logon key upcall to cifs.idmap. BeeGFS issues SID lookups as logon keys (with os:/gs: descriptions), rather than the key types the CIFS client uses for its own mounts, so add a rule for them to /etc/request-key.conf:

create  logon   *               *               /usr/sbin/cifs.idmap %k

To confirm resolution works, trigger an access to a file with a SID-based ACL and check keyctl show and the kernel log (beegfs-client debug builds log the SID and the resolved id).

Note

Resolution is per client, so every client must map a given SID to the same UID/GID, and those ids must match the POSIX ownership stored on the metadata servers. Use the same backend and mapping configuration across the whole cluster; divergent mappings cause ACL enforcement to differ between clients.

Permission granularity

The Linux VFS checks permissions using coarse access modes – read, write, execute – rather than the fine-grained operation set that NFSv4 ACLs describe. When evaluating an ACL, BeeGFS therefore maps each VFS access mode onto a fixed group of NFSv4 ACE permission bits:

• read → READ_DATA (r), READ_ATTRIBUTES (t), READ_NAMED_ATTRS (n), READ_ACL (c)

• write → WRITE_DATA (w), APPEND_DATA (a), WRITE_ATTRIBUTES (T), WRITE_NAMED_ATTRS (N) (plus DELETE_CHILD (D) on directories)

• execute / directory lookup → EXECUTE (x)

Two consequences follow from this coarse mapping:

• The evaluator cannot distinguish operations that the VFS bundles into one access mode. Reading file data, reading attributes, and reading the ACL all arrive as a single “read” check, so an ACL cannot, for example, permit reading data while denying reading attributes – a single ALLOW or DENY decides the whole group.

• Several ACE permission bits are never requested through this path and therefore have no effect on access decisions: WRITE_ACL, WRITE_OWNER, DELETE, and SYNCHRONIZE. Modifying the ACL and changing ownership are governed by file ownership instead (see above), and deleting an entry is governed by the parent directory’s write / DELETE_CHILD permission.

In practice, enforcement granularity is limited to read / write / execute (plus the attribute and named-attribute bits bundled with them). ACEs that grant or deny only the unmapped bits are accepted and stored, but do not change whether an operation is permitted.

Reading and modifying the ACL itself

Access to the NFSv4 ACL as an object – reading it or replacing it – is governed by POSIX-style rules rather than by the ACL’s own READ_ACL/WRITE_ACL bits:

• Reading the ACL is not restricted. Any user who can resolve the path can read the ACL (e.g. via nfs4_getfacl), the same way POSIX ACLs and mode bits are world-readable – even for a directory whose mode is 0700. A DENY entry covering READ_ACL (the c permission) is stored but is not enforced for ACL reads.

• Modifying the ACL requires ownership. Only the file’s owner (or a privileged process with CAP_FOWNER) may set or remove the ACL. Granting WRITE_ACL (C) to another principal does not let that principal change the ACL, and a DENY covering WRITE_ACL does not stop the owner from changing it.

A useful consequence of the second rule: an overly restrictive ACL – for example a catch-all D::EVERYONE@ – cannot lock the owner out of control. It denies normal data access until it is changed, but the owner can always rewrite or remove it with nfs4_setfacl (and root always can), because that operation is gated by ownership, not by the ACL’s contents.

Note

This differs from a classic NFSv4 server, which treats READ_ACL and WRITE_ACL as enforceable permissions. BeeGFS accepts and stores those bits for compatibility but does not use them to control access to the ACL itself. That access follows the POSIX model (the owner may change it, anyone may read it).

Caching

Like POSIX ACLs, NFSv4 ACLs are cached on the client per inode for the same validity period as other file and directory attributes, controlled by tuneDirSubentryCacheValidityMS (directories, default 1s) and tuneFileSubentryCacheValidityMS (files, default 0s, i.e. revalidated on every access). This mirrors the cache behavior described under POSIX ACL caching and revalidation. There is no NFSv4-specific equivalent of sysACLsRevalidate, so NFSv4 ACL caching is always governed by these timers. Changing an ACL invalidates the cache immediately on the client that makes the change, but the update becomes visible to other clients only after their cached entry expires. Longer cache times improve performance at the cost of slightly increased propagation time between clients.

Known issues and limitations

• Experimental. Enforcement semantics may change in future releases (see the warning above). Validate behavior for your workload before depending on it

• Principal forms. Only special, numeric, and AD SID principals resolve. name@domain currently does not

• Per-client enforcement. Identity mapping must be configured consistently across clients. Prefer the whitelist pattern so unresolved principals fail safe (see above)

• Idmapped mounts are not yet supported. BeeGFS otherwise allows idmapped mounts, but the NFSv4 ACL evaluator does not currently apply the mount’s ID mapping: owner matching and principal resolution use the underlying (host) UIDs/GIDs rather than the mapped values, so on an idmapped mount ACL decisions are evaluated against the unmapped identities and may be inconsistent with the POSIX mode-bit fallback (which is idmap-aware). Support for idmapped mounts together with NFSv4 ACLs is planned for a future release. Until then, prefer non-idmapped mounts where NFSv4 ACL enforcement matters

• Coarse permission granularity. Access maps to read/write/execute bundles, the WRITE_ACL, WRITE_OWNER, DELETE and SYNCHRONIZE bits have no effect on access decisions, and the ACL object itself follows the POSIX model (owner may always rewrite or remove it and it is world-readable) regardless of READ_ACL/WRITE_ACL

• SID payload size. The id-mapping upcall must return exactly a 4-byte id. Other payload sizes are treated as unresolved

• Caching and propagation delay. ACLs are cached on the timers described under Caching, so a change on one client becomes visible to others only after their cached entry expires

• Size limits. An ACL may contain at most 4096 ACEs, each principal at most 256 bytes, with a total serialized size of 32 KiB. Principals longer than 256 bytes cannot be resolved

• No partial update. XATTR_REPLACE semantics are not supported. A set replaces the whole ACL

• Mode bits are independent. chmod changes the POSIX mode but does not regenerate or synchronize the NFSv4 ACL, and inheritance is applied only at creation time