API reference · CLIO Core

clio_run

usage

clio_run &lt;command&gt; [options]

start: start the runtime in the foreground, recover every piece of persistent state it finds, and compose the pools in the configuration.
stop: stop the local runtime.
status: report whether the local runtime answers.
compose <start|stop|rm|list>: manage pools from a compose file.
config export --path <file>: write the node's effective configuration to a file.
monitor: print worker statistics.
migrate: move a container to a different node.
refresh: regenerate ChiMod method files for module development.

The nested forms clio_run runtime start|stop|status and clio_run repo refresh still work. clio_run restart was removed: start always recovers state, and start --fresh discards it.

start

usage

clio_run start [--fresh] [--induct] [--ephemeral] [--disk &lt;path&gt;]
               [--viz | --no-viz] [--viz-port &lt;port&gt;] [--viz-bind &lt;addr&gt;]

--fresh: discard this node's persistent state (pools, metadata logs, allocator logs, data) and start empty.
--induct: register this node with all existing cluster nodes.
--ephemeral: skip the configured compose and start with only the admin pool (sets CLIO_EPHEMERAL=1).
--disk <path>: storage root for this run, created if missing (sets CLIO_STORAGE_ROOT; default ~/.clio).
--viz, --no-viz: serve or skip the dashboard. A daemon serves it by default; an embedded runtime does not.
--viz-port <port> (default 8080; 0 picks a free port) and --viz-bind <addr> (default 127.0.0.1). Naming either turns the dashboard on.

stop

usage

clio_run stop [--force] [--grace-period &lt;ms&gt;]

--grace-period <ms>: drain budget for in-flight work (default 5000; 0 means the default). A graceful stop also moves RAM-tier data to disk within this budget.
--force: immediate, ungraceful stop.

If the runtime does not answer, stop on Linux and macOS finds its process, escalates from SIGTERM to SIGKILL, and sweeps leftover shared-memory and socket files. On Windows that escalation is not available and stop reports the failure.

status

Probes the local runtime with a heartbeat and exits with a code that scripts can test:

0: RUNNING, with the process id when it can be found.
1: STOPPED (clean): no runtime and no stale artifacts.
2: STOPPED (stale artifacts): no responsive runtime, but leftover segments or sockets exist. They are listed on stdout.

The probe waits up to 2 s unless CLIO_WAIT_SERVER is set.

compose

usage

clio_run compose start &lt;config.yaml&gt;
clio_run compose stop  &lt;config.yaml&gt;
clio_run compose rm    &lt;config.yaml&gt;
clio_run compose list  [--restartable]

start: create the pools in the file's compose: list, in order, through the admin pool. Pools with restart: true are recorded in each node's pool log and recreated by the next clio_run start.
stop: destroy the listed pools but keep them in the pool log, so they are still restartable.
rm: destroy the listed pools and drop them from the pool log. The compose file itself is kept.
list: list the active containers in the local runtime. --restartable lists the pool-log entries instead, read from <conf_dir>/wal without contacting the runtime.

The old forms compose <file> and compose --unregister <file> still work with a deprecation warning.

Compose file entries

A compose file is any YAML file with a compose: list; the runtime's own configuration is one. The runtime reads these keys of each entry and hands everything else to the module as a YAML string:

mod_name: the module's chimod_lib_name, such as clio_bdev or clio_cte_core.
pool_name: the pool's name. For block devices it is also the device path (ram::name or a file path).
pool_query: routing of the create task. One of local, broadcast, dynamic, direct_id:N, direct_hash:N, range:OFFSET:COUNT, physical:NODE, many_to_one:HASH[:KEY[:NS]].
pool_id: "major.minor" as a string.
restart: true records the pool for recreation on the next start.

yaml · tiers.yaml

compose:
  - mod_name: clio_bdev
    pool_name: "ram::scratch"
    pool_query: local
    pool_id: "320.0"
    bdev_type: ram
    capacity: "256MB"
    restart: true

Module keys are documented with each module: the module reference for storage and chain modules, and the Context FS reference for the filesystem and stream modules.

Client initialisation

c++ · clio_runtime/clio_runtime.h

namespace clio::run {
enum class RuntimeMode { kClient, kServer, kRuntime = kServer };

bool CLIO_INIT(RuntimeMode mode, bool default_with_runtime = false,
               bool is_restart = false);
void CLIO_RUNTIME_FINALIZE();
}

mode: kClient for a program; kServer (alias kRuntime) for a process that is the runtime, which fails if one is already running on the port.
default_with_runtime: used when CLIO_WITH_RUNTIME is unset. In client mode "with runtime" means try to be the runtime and attach if the port is already bound, so several processes launched together elect one host. false only attaches.
is_restart: true recovers the node's persistent state (a plain clio_run start); false discards it.
Returns false when no runtime could be started or reached within CLIO_WAIT_SERVER.

Call CLIO_RUNTIME_FINALIZE() before the process exits; it closes the sockets and joins the background threads the client started.

PoolQuery

Factories in clio_runtime/pool_query.h. net_timeout is the per-task network timeout in seconds; -1 uses the runtime default.

PoolQuery::Local(u32 parallelism = 32): the container on this node. The default for creates.
PoolQuery::Dynamic(float net_timeout = -1): the module's own routing decides per task. The CTE routes each blob to its owner this way.
PoolQuery::Broadcast(float net_timeout = -1): every container runs the task; each copy's outputs are merged into the original with AggregateOut.
PoolQuery::DirectId(ContainerId container_id, float net_timeout = -1): one container by id.
PoolQuery::DirectHash(u32 hash, float net_timeout = -1): one container chosen by hashing a key you supply.
PoolQuery::Range(u32 offset, u32 count, float net_timeout = -1): a contiguous range of containers.
PoolQuery::Physical(u32 node_id, float net_timeout = -1): the container on a specific node.
PoolQuery::ManyToOne(u32 container_hash, u64 batch_key = 0, u64 batch_for_ns = 10000): tasks with the same pool, method, hash and key that arrive within the window are folded into one task with AggregateIn, run once, and the result is sent back to every submitter. The neighbourhood width is networking.neighborhood_size.
PoolQuery::AllToOne(u32 container_hash, u64 batch_key = 0): like ManyToOne, but the folded task runs once a task from every container in the pool has arrived. A barrier or reduction.
PoolQuery::Null(): no routing.
PoolQuery::FromString(const std::string&): parses the pool_query strings of a compose entry.

Setters on any query: SetNetTimeout(float seconds); SetTtl(float seconds), a hard cap on how long the origin waits for a cross-node task (negative for none); SetParallelism(u32), threads used when a GPU submits the task.

Tasks and handlers

Task fields

A task derives from clio::run::Task and is constructed with (TaskId, PoolId, PoolQuery, method). The base class carries task_id_, pool_id_, pool_query_, method_, task_flags_ and return_code_. Field annotations IN, OUT and INOUT document direction; SerializeIn and SerializeOut decide what travels, and both must call the base version first. Strings use clio::run::priv::string with CLIO_PRIV_ALLOC.

Coroutine macros

From clio_runtime/task.h. They expand to C++20 coroutine keywords by default and to Boost fiber calls when CLIO_CORE_ENABLE_BOOST_COROUTINES is on, so handler code is the same for both.

CLIO_TASK_BODY_BEGIN / CLIO_TASK_BODY_END: bracket a handler body.
CLIO_CO_AWAIT(expr): suspend until a future or nested handler completes. Never hold a plain lock across it.
CLIO_CO_RETURN: finish the handler.
CLIO_TASK_CC(ContainerClass): at the end of the runtime source file, defines the entry points the module manager loads.

Coroutine-aware locks: clio::run::CoMutex (clio_runtime/comutex.h) and clio::run::CoRwLock (clio_runtime/corwlock.h) with the guards ScopedCoMutex, ScopedCoRwReadLock and ScopedCoRwWriteLock.

Container callbacks

Virtual methods of clio::run::Container (clio_runtime/container.h) a module may override:

void Init(const PoolId&, const std::string& pool_name, u32 container_id = 0): first-time initialisation.
TaskResume Run(u32 method, shared_ptr<Task>): dispatch by method id.
void Restart(const PoolId&, const std::string&, u32 container_id = 0): warm start on the same node. Default calls Init.
u32 ScheduleRecover(): which node should recover this container; default -1 lets the admin choose.
void Recover(const PoolId&, const std::string&, u32 container_id = 0): rebuild on another node after a node loss. Default calls Init.
void Expand(const Host& new_host): a node joined. Default no-op.
void Migrate(u32 dest_node_id): move state to another node. Default no-op.
u64 GetWorkRemaining() const: pending work, used when draining.

The admin module also provides AsyncHeartbeat, AsyncCompose(PoolConfig), AsyncDestroyPool(PoolQuery, PoolId, flags) and AsyncListContainers(PoolQuery) on clio::run::admin::Client, which the clio_run subcommands use.

CMake functions

Provided by ClioCoreCommon.cmake, loaded by find_package(clio-core CONFIG). Both read clio_mod.yaml in the current source directory and search upward for clio_repo.yaml.

cmake

add_clio_module_client(
  [LIB_NAME &lt;name&gt;]
  SOURCES &lt;files&gt;...
  [COMPILE_DEFINITIONS ...] [LINK_LIBRARIES ...] [LINK_DIRECTORIES ...]
  [INCLUDE_LIBRARIES ...] [INCLUDE_DIRECTORIES ...])

add_clio_module_runtime(&lt;same arguments&gt;)

Targets are <LIB_NAME>_client and <LIB_NAME>_runtime, or <package>_<module>_client and _runtime without LIB_NAME. The output file name follows the target, so LIB_NAME must equal the module's chimod_lib_name.
Aliases <namespace>::<module>_client and _runtime are created for the YAML namespace.
Both link clio::run::cxx; the runtime library also links the client and defines CLIO_RUNTIME=1. Modules other than admin and bdev automatically link the admin and bdev clients when those targets exist.
Headers under include/ are installed; WINDOWS_EXPORT_ALL_SYMBOLS is set so module DLLs link without annotations.

Exported targets after find_package: clio::run::cxx, clio::run::admin_client, clio::run::admin_runtime, clio::run::bdev_client, clio::run::bdev_runtime, and when built, clio::cte::core_client, clio::cte::core_runtime, clio::cae::core_client, clio::cae::core_runtime.

Configuration file

The runtime reads the first file that exists: $CLIO_SERVER_CONF, ~/.clio/clio.yaml, ~/.clio/clio_run.yaml. Sizes accept B, KB, MB, GB, TB. ${CLIO_STORAGE_ROOT} and other variables expand in paths. The installed clio_default.yaml documents every key in comments; the defaults below are its values.

networking

port (9413): base RPC port. The runtime binds port, port+1 and port+3 and refuses to start if any is taken.
neighborhood_size (32): width of a ManyToOne neighbourhood.
hostfile (unset): one host or host:port per line for a cluster.
wait_for_restart (30) and wait_for_restart_poll_period (1): seconds to wait for peers at startup and between retries.

runtime

num_threads (4): worker threads.
queue_depth (1024): task queue depth per worker.
local_sched ("default"): local scheduler.
first_busy_wait (10000): microseconds to busy-wait before sleeping.
client_reap_grace_s (30): seconds a dead, idle client's shared memory is kept before reclaim; 0 defers to the next start.
task_progress_interval_ms (5000): how long a cross-node task may sit quietly before its node is probed; 0 disables.
learning_rate (0.2): learning rate of the task cost model.
conf_dir (per-user runtime directory): where learned models and the pool log live.
ipc_namespace (""): extra suffix for shared-memory segment names, to run two runtimes with the same user and port.

viz

enabled: serve the dashboard. Unset, a daemon serves it and an embedded runtime does not.
port (8080; 0 picks a free port), bind ("127.0.0.1"), max_threads (16).

swim

enabled (false): gossip failure detection and container recovery.
direct_probe_timeout_sec (300.0), indirect_probe_timeout_sec (150.0), suspicion_timeout_sec (3600.0).

compose

The list of pools to create at startup, in order; see compose file entries. Only clio_bdev is strictly required.

Environment variables

Environment values win over the configuration file and are re-applied after every reload.

Locating and reaching the runtime

CLIO_SERVER_CONF: path of the configuration file.
CLIO_STORAGE_ROOT (~/.clio): directory for on-disk state; expanded as ${CLIO_STORAGE_ROOT} in the file.
CLIO_PORT (9413): overrides networking.port.
CLIO_SERVER_ADDR: address clients use to reach the runtime.
CLIO_BIND_ADDR: address the runtime binds its TCP sockets to. On Windows set both this and CLIO_SERVER_ADDR to 127.0.0.1.
CLIO_IPC_MODE: SHM, IPC or TCP. Unset, the client picks shared memory for a runtime on the same machine, otherwise a Unix socket, otherwise TCP.
CLIO_WITH_RUNTIME: 0 attaches to a running runtime only; 1 makes sure one is available. Unset, the program's default_with_runtime applies.
CLIO_WAIT_SERVER: seconds a client waits for the runtime. status uses 2, stop 5, the FUSE mount 600.
CLIO_IPC_NAMESPACE: overrides runtime.ipc_namespace.
CLIO_NUM_THREADS: overrides runtime.num_threads.
CLIO_EPHEMERAL=1: what --ephemeral sets.
CLIO_REPO_PATH: directories searched for module libraries, :-separated (; on Windows).

Failure detection

CLIO_TASK_PROGRESS_INTERVAL_MS (5000): overrides runtime.task_progress_interval_ms.
CLIO_PROBE_SILENCE_S (30): a peer silent this long is declared dead and the tasks waiting on it fail.
CLIO_NET_RETRY_TIMEOUT_S (30): how long a send to an unreachable peer is retried before its task fails.
CLIO_NET_DEAD_FAIL_FAST (0): 1 fails a task addressed to a peer already declared dead at once.

Dashboard

CLIO_VIZ_ENABLE (1/0), CLIO_VIZ_PORT, CLIO_VIZ_BIND, CLIO_VIZ_MAX_THREADS: override the viz section.
CLIO_VIZ_PATH: :-separated directories to serve dashboard assets from instead of those next to the loaded module libraries.

Logging and debugging

CTP_LOG_LEVEL (info): debug, info, success, warning, error, fatal.
CTP_LOG_OUT: log file path; unset logs to the console.
CLIO_ALLOW_PTRACE=1: allow a debugger to attach on hosts with restricted ptrace scope.