Context FS: a filesystem you compose.

Key takeaways

• Mount it like any other directory on Linux, macOS and Windows through FUSE. Applications run unchanged.
• Files spread across the tiers you compose (RAM, NVMe, erasure-coded arrays, Amazon S3 or Google Cloud buckets) by score, with distributed metadata and no metadata server.
• Every file is also tags and blobs in the Context Transfer Engine, so search, assimilation and the other technologies use it without a copy.

• 10 µsstat from the shared-memory mirror
• 0metadata servers: inodes are homed by path
• 3 OSesone FUSE daemon for Linux, macOS and Windows
• N + Mdrives in an erasure-coded array that grows in place

What it is

Context FS is a product of CLIO Core, not the whole of it. It is the filesystem adapter (clio_cte_fuse) plus the filesystem module, composed over the Context Transfer Engine. Directories, inodes and file data are all stored as CTE tags and blobs, so the filesystem inherits tiering, replication, durability and indexing from the engine, and has no central metadata server: each inode is homed on a node by its path.

FUSE on every desktop and cluster

$ clio_run start &
$ mkdir -p ~/clio-mnt
$ CLIO_WITH_RUNTIME=0 clio_cte_fuse ~/clio-mnt -f &
$ cp -r ./results ~/clio-mnt/
$ ls ~/clio-mnt/results

One daemon mounts on all three operating systems with each platform's native driver:

• Linux: libfuse3, included in the pip wheel.
• Windows: WinFsp (FUSE 3 compatibility layer), mounted as a drive letter, included in the pip wheel.
• macOS: macFUSE, or FSKit on macOS 15.4 and newer; build from source.

Writes go straight through, so a file's size is exact as soon as write() returns. Attribute and negative-lookup caches last 1 s, and a shared-memory mirror answers stat without a round trip to the runtime. See the Linux, macOS and Windows tutorials.

Tiering

Each tier in the YAML gets a capacity and a score between 0 and 1. New data lands on the fastest tier with room; data you stop touching is demoted to slower, larger tiers, and a replica on a persistent tier keeps files across restarts.

1. ram::hotDRAM, for the working setscore 1.0
2. /mnt/nvmeLocal NVMe, for staged datascore 0.9
3. safe_bdevErasure-coded array of disksscore 0.4
4. s3:// · gs://A bucket as the capacity tierscore 0.1

Compose local tiers walks through a full configuration, and Placement and tiering explains the policy.

Expandable RAID with safe_bdev

safe_bdev is a block device made of other block devices. It protects data with Reed-Solomon erasure coding: for N drives that must survive f simultaneous failures it uses k = N − f data drives and m = f parity drives, so 8 drives surviving 2 failures is RS(6, 2) with 25% overhead. Unlike a classic RAID array, it grows in place:

• Add drives at runtime as data or parity members (safe_bdev_add_bdev). New blocks are allocated round-robin over the members that have room, so total data capacity is the sum of the members' capacities and an added drive's space is usable immediately, even when the array was already mostly full.
• Mixed sizes. Each member fills to its own capacity; stripes narrower than k are protected as mirrors until enough members have written into them.
• Replace and rebuild. Remove a failed member and name a replacement; the runtime rebuilds its shards while the filesystem stays mounted, and the dashboard shows recovery progress.

The array is an ordinary tier, so Context FS on top of it needs no changes. The cr_safe_bdev_integration_docker test composes three data drives and one added parity drive under a mounted Context FS, removes a member, rebuilds it, and checks that the data still reads back.

Amazon S3 and Google Cloud as tiers

The block device module has S3 and Google Cloud Storage transports, so a bucket can be a tier like any disk. Give it the lowest score and only data the organizer demotes reaches it. Separately, the assimilation engine imports existing s3:// and gs:// objects, including anonymous reads of public buckets. The cloud storage tutorial covers both.

Performance

Metadata latency for one operation each, through the FUSE mount on Linux (from Filesystem and FUSE):

• stat, shared-memory mirror~10 µs
• stat, filesystem module~31 µs
• readdir, 10 entries255 µs
• rename~570 µs
• readdir, 100 entries592 µs

About 91% of each operation's wall time is kernel transit; the daemon itself spends about 9.6 µs of CPU time per operation.

Against OrangeFS on Ares: IO500

IO500 on 4 nodes of the Ares cluster, 32 MPI ranks (8 per node). Each node: two Xeon Silver 4114 (40 cores), 46 GB of memory, a Toshiba RD400 NVMe drive, a Seagate ST1000LM049 1 TB HDD, and 40 GbE. Every node is both a server and a client of the filesystem under test.

• Context FS: release build of site-technologies (c89f7f10f), 20% DRAM / 80% NVMe per node: a 3 GB DRAM tier, pre-faulted at start (CLIO_PREFAULT=0), over a 12 GB NVMe tier. No replication.
• OrangeFS 2.10 (f4854f7): every node a metadata and data server, mounted through pvfs2fuse, with all storage on the NVMe drive or all on the HDD. TroveMethod directio, so OrangeFS reaches its device with O_DIRECT, as Context FS's file tier does.
• IO500 (5c2237d) with a 60 s stonewall and data capped at about 27 GB, 2.3x the aggregate DRAM tier, so most data lands on NVMe. These are not valid IO500 list submissions, which need a 300 s stonewall. One run per system.

Every phase (bandwidth in GiB/s, metadata in thousands of operations per second; higher is better):

Context FS scores 1.6x OrangeFS on NVMe and 5.6x OrangeFS on HDD. It leads on writes, file-per-process reads and every create, stat, read and find phase, by 7-17x on stat and 8x on find. It loses two phases. OrangeFS on NVMe reads the single shared ior-hard file faster (0.21 vs 0.12 GiB/s). Deletes are Context FS's weak point: it removes files at 35-143 per second against about 680 for OrangeFS, so each delete phase took about 22 minutes. At OrangeFS's delete rate, Context FS's metadata score would be 2.35 kIOPS instead of 1.33.

Two things in this setup cost OrangeFS bandwidth. directio keeps the server page cache from standing in for the device, and pvfs2fuse uses the FUSE 2 API, which splits writes into pieces of at most 128 KiB, so a 2 MiB IOR transfer reaches the device as 16 synchronous writes. OrangeFS through its kernel module, or with its default buffered alt-aio method, would post higher bandwidth. pvfs2fuse also needed a four-line fix: when several nodes create the shared ior-hard file at once, the losers got EEXIST from an open(O_CREAT) without O_EXCL, which aborted IO500 on NVMe every time. The patched client opens the existing file instead, as POSIX requires.

Checkpointing a simulation through the tiers

From the Eternia paper's evaluation: Gray-Scott on 2 Aurora nodes, 8 steps, a checkpoint every 2 steps. Each checkpoint is a copy-on-write snapshot that CLIO Core drains down the tier stack in the background, in a layout the HDF5 VOL can read. It is compared with a conventional synchronous checkpoint, in which the simulation waits until the snapshot reaches persistent storage; in the Lustre-heavy composition that is a simulation writing its checkpoint to the parallel file system. Total runtime in seconds, lower is better.

The asynchronous snapshot is 15-25% faster in every composition. Persisting the grid four times still moves 25-62 s of data against 10-48 s of computation; the snapshot lets that traffic proceed in the background instead of blocking the run on the slowest tier. For a science application at larger scale, see the Gray-Scott comparison against DDN, Hermes and cuFile.

Against ext4: cloning the Linux kernel

• ext439 s
• Context FS171 s

Linux kernel git clone, one node, lower is better. A clone is tens of thousands of small-file creates and lookups; a local kernel filesystem has no FUSE crossing, and Context FS pays one per metadata operation.

Against NTFS on a Windows workstation

The same Python script (1 GiB written in 1 MiB writes and fsynced, read back, then 2,000 files of 4 KiB created, stat'ed and listed) on an NTFS directory and on a Context FS mount at Z:. Release build of dev (f88db54bf), default runtime configuration bound to 127.0.0.1, WinFsp 2.0. Ryzen 7 3800X, 32 GB RAM, WD Black SN850X NVMe. Bars are the mean of two runs.

Bars are the mean of two runs; the two runs were within 1,081-1,258 vs 2,347-3,838 MB/s for writes and 2,122-2,278 vs 965-1,354 MB/s for reads (NTFS vs Context FS).

Read the write result carefully: in this run the runtime registered a single storage target, the DRAM tier, so Context FS writes landed in memory and were not on disk when fsync returned, which NTFS guarantees. With a persistent disk tier active, writes also pay for the replica. Reads, file creation and metadata cross the WinFsp kernel boundary on every call, which is where NTFS leads. Defender real-time scanning was on for both. Single machine, two runs each.

Where to go next

• Mount your first filesystem
• Filesystem and FUSE design and durability and recovery