Client Metadata Caching

The default BeeGFS client caches metadata for only a short interval (milliseconds), controlled by tuneDirSubentryCacheValidityMS, tuneFileSubentryCacheValidityMS, and tuneENOENTCacheValidityMS. This offers a configurable tradeoff between speed and correctness for different workloads but limits the amount of work the client can satisfy locally.

In addition, BeeGFS version 8.4 and newer provide an alternate, long-term client-side metadata caching mechanism that allows clients to cache metadata (attributes, lookup results) for extended periods without relying on short, time-based expiry. Cache entries are kept valid until the owning metadata server explicitly notifies the client to invalidate them. This drastically reduces the number of metadata RPCs from client to server for read-mostly metadata workloads — those dominated by repeatedly reading the same metadata rather than modifying it (e.g., repeated stat(), directory traversal, ls -l), where it can significantly improve performance.

The mechanism does not target general metadata-heavy workloads. Workloads with a high rate of metadata changes (for example, heavy file creation and deletion) trigger frequent invalidations and benefit little, so they should not be expected to speed up.

Warning

This feature is currently considered experimental. It must be enabled on a coordinated set of clients and metadata servers. Please review the Compatibility and Limitations sections carefully before enabling it in production — most notably the possibility of temporarily stale cached metadata under concurrent modification.

Overview

When remote invalidation is enabled, the client and metadata server cooperate as follows:

  • When the client does a read request for some metadata object that is eligible for caching (currently: only inode metadata), the metadata server records the client as a watcher of that object in an in-memory tracking structure (the InvalWatch component).

  • The client may then cache the inode’s metadata indefinitely until invalidated.

  • When something happens on the metadata server that would change the cached state for that inode (e.g., another client modifies the file, the inode is removed, etc.), the server queues an invalidation for every registered watcher.

  • A dedicated component on the client (the InvalReader) maintains a persistent connection to each metadata server and pulls these invalidation lists. When the client receives an invalidation for an inode, it marks the inode cache state as invalid. The inode is not refreshed at that moment. The next time that inode is accessed (e.g., by a stat(), lookup, or any operation that needs its attributes), the client sees the invalid cache state and sends an RPC to the metadata server to refetch the current attributes. Inodes that are never touched again pay no cost.

The result is a cache whose entries can remain valid indefinitely as long as the underlying inode is not modified, while still respecting cross-client consistency.

Note

Invalidation is not delivered instantaneously. After another client modifies metadata, there is a bounded window of invalidation-delivery latency during which this client can still return the old stat() data. The metadata server batches invalidations behind a small, fixed delay (on the order of a millisecond), so the latency is always at least roughly that, and longer under load. Applications that need instant cross-client metadata coherence (e.g., using mtime or fil size as a synchronization signal) should be evaluated for compatibility or should not rely on this caching mechanism.

Configuration

Client side metadata caching with remote invalidation is disabled by default. It must be turned on independently on every metadata server that should participate, and on every client that should benefit.

Metadata Servers

Edit /etc/beegfs/beegfs-meta.conf on every metadata node where the feature needs to be enabled and set:

sysRemoteInvalEnabled    = true

This activates the InvalWatch component, which tracks per-client watcher state and pushes invalidations.

The amount of memory used by InvalWatch is bounded by the number of inodes it is allowed to track at once. This is controlled by:

tuneInvalWatchMaxObjects = 50000000

The default is 50 million tracked inodes which, at the current size of 128 bytes per entry amounts to ~6 GiB of memory usage. When the limit is reached, older entries are evicted (forcing the clients that cached them to re-fetch on next access). Increase this value for very large working sets if metadata-server memory permits, decrease it if memory is constrained.

The metadata server also maintains a separate, bounded queue of pending invalidations for each watching client. The maximum size of this per-client queue is controlled by:

tuneInvalWatchQueueSize = 4m

The value accepts a human-readable size suffix (k (kiB), m (MiB), g (GiB)) and must stay below 4 GiB (the queue cursors are 32-bit). The default is 4 MiB per client. When a client cannot keep up and its queue fills, the queue overflows and that client is forced to resync, dropping its whole cached metadata for this server (see Error Handling). Raising this value lets a client absorb larger bursts of cross-client metadata changes before overflowing, at the cost of up to that much additional metadata-server memory per active client.

Restart the metadata service after changing any of these options:

# systemctl restart beegfs-meta

Clients

Edit /etc/beegfs/beegfs-client.conf on every client that should use remote-invalidation-based caching and set:

sysRemoteInvalEnabled = true

This activates the InvalReader component on the client, which spawns a dedicated thread per metadata server to receive invalidation notifications.

Restart the client service for the change to take effect:

# systemctl restart beegfs-client

It is safe to enable the feature on a subset of clients first while leaving others on the default short-lived cache, as long as all metadata servers have already been upgraded to a version that supports the new protocol.

Warning

sysRemoteInvalEnabled = true on the client requires that every metadata node the client communicates with be upgraded to a version that supports the new protocol. Enabling the option against an unsupported metadata node will cause communication failures. See Compatibility.

Requirements and Compatibility

Read this section before enabling the feature. The protocol changes introduced by remote invalidation have important implications for rolling upgrades and mixed-version deployments.

  • Required BeeGFS version: Both metadata nodes and clients must be running BeeGFS 8.4 or later. All metadata nodes must be upgraded before any client enables the feature, and any client that sets sysRemoteInvalEnabled = true must itself be on BeeGFS 8.4 or later (older clients do not ship the InvalReader component and cannot use the feature at all).

  • Per-server opt-in: The feature can be left disabled on individual upgraded metadata servers. An upgraded server with sysRemoteInvalEnabled = false will still speak the new protocol but will not track watchers and will not push invalidations. Clients with the feature enabled will continue to work against such servers, but they will fall back to the short-lived, time-based metadata cache for entries owned by those servers. Enabling the feature on a metadata server while clients still have it disabled carries no performance drawback: such a server tracks no watchers and pushes no invalidations until a client actually opts in, so it behaves exactly as before. This makes it safe to enable on all metadata servers first and then try the feature on just one client.

  • Per-client opt-in: The feature can be left disabled on individual upgraded clients. An upgraded client with sysRemoteInvalEnabled = false continues to operate normally and uses the default short-lived, time-based metadata cache against all metadata servers, regardless of whether those servers have the feature enabled. A client with the feature disabled sees no change in behavior or performance compared to before — it neither registers as a watcher nor receives invalidations. This allows the feature to be rolled out to a subset of clients first (for example, for evaluation on selected workloads) while leaving the rest on the default behavior, making it safe to try on just one or two clients.

  • Both sides must be enabled: For the caching benefit to apply to a given metadata server, both the client and that server must have sysRemoteInvalEnabled = true.

  • Metadata mirroring: If you use metadata mirroring, enable the feature on both the primary and the secondary of each buddy group so caching will continue to work after a failover.

Note

Once the feature is enabled on a client, any downgrade of a metadata server below the supported minimum version will break that client’s ability to talk to that server. Plan downgrades accordingly: disable the feature on clients first, then downgrade servers.

Observability

Client

The client exposes a per-mount procfs entry summarizing the effectiveness of the cache:

$ cat /proc/fs/beegfs/<clientID>/inode_cache_stats
inodeCacheHits: <number of attribute lookups served from the client cache>
inodeCacheMisses: <number of attribute lookups that had to be fetched from a metadata server>
inodeCacheInvalidations: <number of inodes invalidated by remote push since mount>

A high hit-to-miss ratio together with a low invalidation rate indicates the cache is providing the expected benefit for the running workload.

The effective value of the configuration option is also visible in the standard config procfs entry:

$ grep sysRemoteInvalEnabled /proc/fs/beegfs/<clientID>/config

Metadata Server

The metadata server exposes InvalWatch internals (number of tracked inodes, number of registered watchers, queued invalidations, evictions) through the metadata server statistics interface. A ready-made Grafana dashboard for monitoring client metadata caching is included with the BeeGFS Grafana assets; see Monitoring service for how to wire up monitoring and dashboards.

Error Handling

The feature is designed to fail safely. The cache is only an optimization, never a source of truth, so whenever a client cannot reliably keep its cache coherent with a metadata server it discards the affected cache and re-fetches metadata from the server. These events impact performance (a temporary burst of metadata RPCs) but never correctness.

Global fallback

If a client hits repeated, unrecoverable errors while reading invalidations from a metadata server, it permanently disables remote invalidation for the entire mount and reverts to the default, time-based metadata caching. This fallback is:

  • One-way — it is never re-enabled automatically at runtime. Unmounting and remounting the file system is the only way to turn the feature back on.

  • Per client mount — it is local to that mount and does not affect other clients.

Because a fallen-back client is silently slower rather than broken, the main external signal is a log message. Grep the client log for falling back to time-based to detect it; the client logs a one-time warning of the form:

Client meta cache invalidation DISABLED: falling back to time-based metadata caching for the
lifetime of this mount.

The fallback is triggered only by a streak of repeated errors of the same kind (for example, invalidation-queue overflows, invalid read-position errors, or communication timeouts), or by a failure to allocate the invalidation buffers. The transient, recoverable conditions described below do not, on their own, trigger it.

Queue overflow

The metadata server maintains a bounded per-client queue of pending invalidations, sized by tuneInvalWatchQueueSize (see Configuration). The client drains this queue by pulling invalidation lists from the server. Each pull transfers at most 512 KB of invalidation data per response. A larger backlog is therefore drained over several consecutive reads.

If a client falls too far behind and its queue overflows before it can be drained, the server flags the condition on the next read. The client then drops its entire cached metadata for that metadata server and resynchronizes: all cached inodes for that server are marked invalid at once, so the next access to each of them issues a refresh RPC, producing a temporary burst of metadata traffic until the working set is re-cached. A single overflow is recoverable; only a repeated streak of overflows escalates to the global fallback above.

Metadata server restart or failover

After a metadata server restarts, or a buddy mirror fails over, the server no longer holds the client’s watcher state. The client detects this loss of synchronization and invalidates its whole cache for that metadata server. As with a queue overflow, subsequent accesses re-fetch metadata, causing a burst of refresh RPCs until the cache is repopulated. This is expected behavior and requires no operator intervention.

Limitations

  • Metadata cache coherence under concurrent modification. Cache validity for a cached inode is tracked by a single per-inode valid flag: an incoming invalidation (delivered by the InvalReader) clears it, and a completed metadata refresh sets it. These two operations are not serialized against an in-flight refresh. As a result there is a narrow race: if an inode is modified while this client is fetching that inode’s metadata, the invalidation for that change can be processed before the refresh commits its result. The refresh then re-marks the entry valid with data that predates the change, and the invalidation is effectively lost, so the client may serve stale cached metadata for that inode. The staleness is bounded and self-healing — it is cleared by the next invalidation for that inode, by a metadata-server restart or failover, or by a subsequent refresh — but its duration is not guaranteed. This affects only concurrent modification of the same inode (typically from another client) during this client’s refresh window; a client’s own metadata operations update its cache directly and are not affected.

  • The feature is experimental in the current release; the configuration option names, default values, and observability surface may change in future versions.

  • Self-invalidations limit the benefit for modify-heavy workloads. A client that reads an inode becomes a watcher of it, and the metadata server fans out every invalidation to all watchers — including the client that made the change. A client that modifies an inode therefore invalidates its own cached copy and must re-fetch it on the next access, even though it already knows the new state. Workloads dominated by a single client creating, deleting, and re-statting many files see little or no benefit and can even run slower than with the default time-based cache, because much of the invalidation traffic is the client invalidating its own cache. Concretely, with the feature enabled on both the client and the metadata server, there is an observable reduction in file-create performance, while metadata read operations (for example, stat and lookups) are either improved or unchanged. This is the expected trade-off: the feature optimizes read-mostly metadata workloads at the cost of some additional work on the modify path, so take it into account when enabling it for create-heavy workloads.

  • Enabling remote invalidation does not change the consistency guarantees of BeeGFS for opened files. POSIX semantics for data are still governed by the existing file cache and sysSyncOnClose / open/close semantics; this feature only governs how long metadata (attributes, dentries) can be cached client-side.

  • Memory usage on metadata servers grows linearly with the total number of inodes cached across all watching clients, bounded by tuneInvalWatchMaxObjects. Size this value appropriately before enabling the feature on large clusters.

  • Negative lookups (ENOENT, i.e. non-existent entries) are not cached long-term by this feature. The remote-invalidation mechanism only tracks and invalidates entries that exist on the metadata server; a watcher is registered on a successful lookup or stat, not for a name that does not resolve. Negative entries therefore remain governed by the short, time-based tuneENOENTCacheValidityMS timeout.

  • This feature is independent of the existing Client Side Caching Modes mechanism (tuneFileCacheType), which governs file data caching. Both can be configured independently.