Durability and recovery

Four levels of "safe"

Data on a volatile tier (RAM, HBM, pinned memory) reaches only the first level until a flush moves it to a persistent tier.

What is persisted, and where

Persistent state lives under a single storage root (CLIO_STORAGE_ROOT or clio_run start --disk, default ~/.clio):

Without a metadata_log_path, the core persists nothing. The bytes on disk would survive, but nothing would record which blob they belong to. The shipped configuration sets the path.

Ordering invariants

Most of the reliability design comes down to doing things in the right order. Each rule below exists because the opposite order lost data at scale or under a file-system stress test.

The metadata log

The metadata log is sharded by worker, so concurrent workers do not contend on one file. Every record carries a sequence number from a single per-container counter. Replay merges all shards and sorts by that sequence before applying anything. Replaying each shard in file order once let an older full-layout record overwrite a newer one and shorten a blob after a crash. Records that carry a blob's layout are full-state rather than deltas, so replaying one is idempotent.

Snapshots are expensive (about a second per 18k blobs before the fix that made them occasional). A full snapshot is therefore rebuilt only when one of these holds:

• the log has grown past its capacity (32 MB by default);
• unlogged changes such as scores are pending and the last snapshot is over a minute old;
• a snapshot is explicitly requested.

Snapshotting yields regularly so it does not stall a worker.

What a restart does

clio_run start always recovers existing state. --fresh is the only way to discard it.

1. Pools come back. The runtime replays its per-node pool log, in creation order, and recreates every durable pool through its module's restart path. Pools from the server's compose configuration are treated as durable.
2. Block devices rebuild their allocators from the allocation log. Gaps below the highest live byte become free space, and the heap resumes after the last live byte.
3. The core restores metadata from the snapshot, then replays the merged log.
4. Volatile data is accounted as lost, not as zeros. Blocks that lived on RAM, HBM or pinned memory are dropped. A blob keeps the prefix before its first lost block, and the rest is marked lost. Reads into the lost range fail with an I/O error. The replication layer can serve those bytes from another copy if one exists.
5. Sizes and caches are made consistent. Tag sizes are recomputed in one linear pass (an earlier quadratic pass kept a 6-node cluster from ever recovering). Node-local cache copies are discarded, because invalidations may have been missed while the node was down.
6. Restored space is reserved on persistent tiers that do not log their own allocations, so new writes cannot land on restored data. The cost is that holes below the reserved high-water mark stay unusable.
7. Higher layers reconcile.
   • The stream module re-queues appends staged before the crash and finishes interrupted merges. It holds size operations until the filesystem confirms each file's size, or two minutes pass.
   • The filesystem resumes inode numbering past its durable reservation and destroys orphaned inodes.
   • The replication layer pulls back any changes another node made while standing in for this one.

What survives what

What fsync means end to end

Through the FUSE filesystem, fsync on a file does the following, in order:

1. Drains the client's write-behind pages.
2. Publishes the file's size.
3. Waits for any pending asynchronous replication of the file.
4. Syncs the file's tag: data is moved to a persistent tier and its devices are fsynced, then the metadata log is fsynced.
5. Fsyncs the stream size log.
6. Syncs the parent directory's tag, so the name is durable too.

If a node that may have held this file's unsynced writes died since the last sync, fsync returns an I/O error rather than claiming durability it cannot vouch for. Measured cost: syncing to the device adds roughly 23–35% to fsync-heavy workloads such as sqlite, tar and appends.

Node failure

• Default: a single copy. With the default replication factor of 1, a node's blobs are unavailable while it is down. Requests for them fail, after a network timeout or immediately in fail-fast mode.
• Remote copies. With factor N, the owner writes N−1 copies to the next nodes in ring order before acknowledging. A dead successor is skipped: the write succeeds with one copy fewer, and nothing re-replicates it later.
• Failover. With failover_to_successor, requests for a dead owner's blobs go to its first live successor, which serves its shadow copy and records every change. When the owner restarts, it pulls those changes back before serving. The successor also pushes them periodically.
• Peer restarts are detected by incarnation. Every message carries the sender's incarnation number. When a peer comes back as a new process, requests that were in flight to the old one fail promptly and are retried, rather than waiting forever for a reply that will never come.
• Membership is conservative. Liveness comes from probing peers that have outstanding requests or have gone quiet. The gossip-based failure detector is off by default, because at 256 nodes wide collective operations starved its probes and healthy nodes were declared dead. As a consequence, automatic container recovery seldom triggers. Durability across node loss is the replication layer's job, not the runtime's.

Tradeoffs at a glance