Context Runtime: the process under everything.

What it is

The runtime, started with clio_run start, loads shared libraries called ChiMods, creates pools of them, and serves tasks that clients submit to those pools. A client is any process that calls CLIO_INIT: a Python script, the FUSE daemon, an MPI rank, or another module inside the runtime. The runtime also serves a web dashboard and, in a cluster, talks to the runtimes on the other nodes listed in its hostfile.

Key concepts

• Module (ChiMod). A directory with a clio_mod.yaml, a set of task structs, a client library that programs link, and a runtime library the daemon loads. It defines numbered methods; ids 0 to 9 are reserved for Create, Destroy and Monitor.
• Pool and container. A pool is a running instance of a module with a name and a major.minor id. A container is the pool's instance on one node; the container id equals the node id, and the number of containers is fixed by the hostfile. Two pools of one module can run side by side with different settings.
• Task. A method call on a pool: a struct with input and output fields. A task lives in the submitting process; its inputs are serialized into a buffer that crosses into the runtime, and only its outputs come back. Tasks are not shared memory.
• Coroutine handlers. A container's handler for a method is a coroutine. When it awaits a subtask or a device, its worker thread runs other tasks. The runtime supports C++20 stackless coroutines and, with CLIO_CORE_ENABLE_BOOST_COROUTINES, Boost.Context fibers behind the same macros.
• Pool query. Every task carries a PoolQuery that says which containers run it: this node, one container by id or hash, a range, every container, a specific node, or wherever the module decides. Collective queries fold tasks from many submitters into one run.
• Transports. A client reaches the runtime over shared memory on the same machine, a Unix socket when shared memory is unavailable, or TCP across machines and containers, chosen automatically unless CLIO_IPC_MODE says otherwise. Runtimes talk to each other over TCP. GPU kernels can submit tasks through a GPU queue.
• Interposition. A module can stand in front of another, handle some methods and forward the rest to its next_pool_id. The cache, indexer, replication and compressor modules are built this way.
• Compose file. The compose: list of the configuration names the pools to create at startup, in order. clio_run compose start and stop create and destroy the pools of another file while the runtime keeps serving.

Scheduling

The runtime starts a fixed number of worker threads (runtime.num_threads, four by default) with roles: a scheduler, network send and receive workers, a GPU-queue worker, and I/O workers in cost classes. Tasks are mapped by predicted cost, learned from observed run times per module and method. A monitor moves the queue of a worker stuck on one task to a fresh worker, grows a saturated class, and retires idle extra workers. Runtime and block devices describes the mechanism and its tradeoffs.

Elasticity and failure handling

• Durable pools. Pools created from the compose file, or with restart: true, are recorded in each node's pool log and recreated by the next clio_run start. --fresh discards the log with the rest of the node's state.
• Adding nodes. clio_run start --induct registers a node with an existing cluster. Every container receives an Expand callback and may re-partition its data.
• Migration and recovery. A container can be moved with its Migrate callback (clio_run migrate), and rebuilt on another node after a node loss with Recover; ScheduleRecover lets the module choose where. The defaults do nothing beyond re-initialising, so a module that holds state must implement them.
• Restarted peers are detected from an incarnation number on every message. Requests in flight to the old incarnation fail and are retried.
• Silent peers are probed and declared dead after CLIO_PROBE_SILENCE_S seconds (default 30). A send to an unreachable peer is retried for CLIO_NET_RETRY_TIMEOUT_S seconds (default 30) before its task fails; CLIO_NET_DEAD_FAIL_FAST=1 fails tasks to a dead peer at once.
• Gossip detection (the swim section) exists but is off by default, because wide collectives starved its probes at scale. With it off, a dead node produces failed tasks, not automatic container recovery. Data availability across node loss comes from the replication module.
• Clients survive a restarted runtime. A heartbeat detects the loss, the client waits for the runtime to return, reconnects and resends. The runtime reclaims the shared-memory segments of dead clients after runtime.client_reap_grace_s seconds.

How it relates to the other technologies

• Context Transfer Engine: the core, cache, indexer, replication, compressor and block-device modules. The runtime's block-device module, clio_bdev, is what every storage tier is built on.
• Context FS: the filesystem and stream modules, plus the FUSE daemon, which is a runtime client.
• Context Assimilation Engine and Context Exploration Engine: modules and clients for importing and querying data.
• Memory expansion: paged vectors whose pages are tasks on the CTE pools, including tasks submitted from GPU kernels.
• Machine learning: clio-ml algorithms that run over those vectors.

Where to go next

• Install: the runtime is in every install; building a module needs the headers and CMake package.
• Usage: write, build, load and compose a ChiMod.
• API reference: clio_run, client initialisation, PoolQuery, the task and coroutine macros, the container callbacks, the CMake functions, the configuration file and environment variables.
• Write a chain operator: a complete module in the storage chain, built and run end to end.