Performance pitfalls and tuning knobs.

Where each knob is read

Context FS has two processes, and a variable only works in the one that reads it.

• The runtime (clio_run start) owns the tiers and the block devices. CLIO_PREFAULT and the block-device YAML keys apply here.
• The mount (clio_cte_fuse) owns the kernel interface. CLIO_FUSE_* and CLIO_CFS_* apply here.

Set each variable in the environment of the right process before it starts. Changing one later has no effect until that process restarts.

Pitfall 1: cold RAM-tier page faults

This is the most common reason a first run is slow. A RAM tier is a large shared-memory mapping that the operating system backs lazily: a page of physical memory is only committed the first time it is touched. Without help, the first write into each 4 KiB page takes a page fault inside the copy, and a write-heavy job pays that fault for every page of fresh tier it fills.

The runtime softens this by itself. When an allocation crosses the populated watermark, it bulk-faults the next populate_unit bytes (64 MiB by default) in one step, so copies stop faulting page by page. That still leaves a stall every time the watermark moves, and the first touch of the tier is still cold.

The fix is CLIO_PREFAULT, which faults the RAM tier in when it is composed rather than while your job runs:

# Fault in the whole RAM tier at startup
CLIO_PREFAULT=0 clio_run start

# Or only the first 4 GB of it
CLIO_PREFAULT=4GB clio_run start

• 0 means the whole mapping. Any other value is a size string such as 512MB or 4GB, clamped to the tier's size. Unset (the default) keeps incremental population.
• The runtime logs how much it pre-faulted and how long it took: look for CLIO_PREFAULT=… pre-faulted … bytes in its output.
• Population resumes past the pre-faulted prefix, so a partial prefault still works for a tier larger than it.

Related tier keys

These go on a block device or storage entry in the compose YAML.

• populate_unit (size string, default 64 MiB): how much the RAM tier bulk-faults each time allocation crosses the watermark. Raising it means fewer, larger stalls. 0 turns population off and gives pure demand faulting, the slowest setting for writes.
• alloc (auto, eager or lazy): when a memory-backed device commits its pages. eager commits the whole size up front, and falls back to lazy if the device has no size.
• growth_unit (size string, default 1 GiB): file-backed tiers grow their backing file in steps of this size instead of reserving the whole capacity at startup. RAM and GPU tiers ignore it.

Pitfall 2: GPU copies from unpinned memory

On a node with a GPU, the RAM tier pins the memory it commits, so copies between the tier and the GPU run asynchronously at full speed. Unpinned shared memory turns each of those copies into a slower, synchronous, driver-staged copy. CLIO_BDEV_NO_PIN=1 turns pinning off. Only set it if pinned memory is scarce, and expect GPU copies to slow down.

Pitfall 3: the kernel's 4 KiB writes

Without the writeback cache, FUSE hands the filesystem one 4 KiB write for every dirty page, so a large write becomes many small operations. Context FS avoids this for write-only file handles: files opened O_WRONLY use direct I/O, so each write() arrives at its full size. Compilers, git and archive tools create files this way. This is on by default; CLIO_FUSE_DIRECT_WRONLY=0 turns it off.

The kernel writeback cache (CLIO_FUSE_WRITEBACK=1) batches writes even further but is off by default for correctness. With it on, the kernel's view of a file's size can run ahead of what has been published, and a git clone was observed to corrupt its pack file. Only turn it on for single-writer workloads that call fsync.

Pitfall 4: coherence costs on multi-node mounts

On one node, the mount caches file attributes and directory entries for 1 second, which removes most metadata round trips. On a multi-node mount (CLIO_FUSE_MULTINODE), attributes are not cached by default, so that a reader on another node never sees a stale file size. Directory entries are still cached for 1 second: the namespace is spread across nodes, and without that cache every path component of every call becomes its own network round trip.

• CLIO_FUSE_ATTR_CACHE_S (seconds): attribute cache lifetime. Default 1 on one node and 0 on multi-node. Raise it for read-mostly data that no other node changes; set it to 0 when several processes or mounts share files and need exact sizes.
• CLIO_FUSE_ENTRY_CACHE_S (seconds): directory entry cache lifetime. Default 1 on multi-node, otherwise the attribute value. Deep directory trees such as .git/objects benefit most.
• CLIO_FUSE_STATFS_TTL_MS (default 1000): how long statfs results (free space) are cached.

Pitfall 5: write windows and memory

Writes return before they reach the tiers. They are kept in a shared-memory window and published in the background. That makes writes fast, but the window holds memory, and once it is full a writer waits for the oldest writes to finish.

• CLIO_FUSE_WRITE_INFLIGHT_MB (default 256): the mount's in-flight write window in MiB. 0 removes the bound. Raise it for fast tiers and large sequential writes, and lower it when many writers fill memory.
• CLIO_CFS_ASYNC_WRITES (default on): asynchronous writes. 0 makes every write wait for completion. Use that only to rule out write-behind when debugging.
• CLIO_CFS_WRITE_WINDOW_COUNT (default 256) and CLIO_CFS_WRITE_WINDOW_BYTES (default 64 MiB): the client's write-behind window by count and by bytes. The byte budget is global, not per file.
• CLIO_FUSE_ASYNC_FLUSH and CLIO_FUSE_ASYNC_CLOSE (both default on): close() does not wait for the server's bookkeeping round trip. Set them to 0 to make close wait.

Fast paths that are on by default

These are on by default and are worth checking only when a run is unexpectedly slow. Each has a switch to turn it off, meant for debugging.

• CLIO_FUSE_SIEVE: file data goes through the mount's buffer directly, and only namespace operations become runtime tasks. 0 sends all I/O through tasks.
• CLIO_CFS_SHM_READ: reads are served from shared memory without a round trip to the runtime. 0 forces every read through the runtime.
• CLIO_DIRECT_READ and CLIO_DIRECT_WRITE, read by the runtime: a node-local RAM tier is read and written in place, without dispatching a separate task.
• CLIO_TASK_BATCHING, read by the runtime: workers dequeue tasks in batches.

Runtime threads

The runtime's worker count comes from num_threads in the runtime section of its configuration (4 in the default configuration). On a many-core node serving several busy writers, more workers raise throughput; on a laptop, the default leaves cores for the application.

A starting configuration

# Runtime: fault in the RAM tier up front (commits its full size)
CLIO_PREFAULT=0 clio_run start &

# Mount: larger write window for big sequential writes
CLIO_WITH_RUNTIME=0 CLIO_FUSE_WRITE_INFLIGHT_MB=1024 clio_cte_fuse ~/clio-mnt -f

Change one knob at a time and measure. The API reference lists every environment variable the mount reads, and the Context Runtime API covers the runtime configuration file.