CLIO Core

Docs · Context Runtime

Build a ChiMod.

A ChiMod is a directory with a YAML descriptor, a public header set and two shared libraries: a client library that user programs link, and a runtime library the daemon loads. This page walks through writing one, building it, loading it and composing it. The signatures it relies on are in the API reference; a complete worked example is Write a chain operator.

Template
context-runtime/modules/MOD_NAME
External example
context-runtime/test/unit/external-chimod
Worked example
/develop/operator.html

1. Lay out the module

Start from the template in context-runtime/modules/MOD_NAME. Replace MOD_NAME with your module name and clio_runtime with your include namespace:

files
MOD_NAME/
  CMakeLists.txt
  clio_mod.yaml                           module name, namespace, method ids
  include/clio_runtime/MOD_NAME/
    autogen/MOD_NAME_methods.h            method id constants and names
    MOD_NAME_tasks.h                      CreateParams and one struct per task
    MOD_NAME_client.h                     client: Async* methods returning futures
    MOD_NAME_runtime.h                    the container class
  src/
    MOD_NAME_client.cc                    usually just includes the client header
    MOD_NAME_runtime.cc                   handler bodies, ends with CLIO_TASK_CC
    autogen/MOD_NAME_lib_exec.cc          dispatch: Run, Save/LoadTask, NewTask, ...

An external repository that holds several modules adds a clio_repo.yaml at its root. The CMake helpers search up the directory tree for it to learn the namespace:

yaml · clio_repo.yaml
name: external-chimod
namespace: external_test
version: 1.0.0
description: External ChiMod development test repository

modules:
  - simple_mod

2. Number the methods

clio_mod.yaml names the module and assigns an integer id to every method. Ids 0 to 9 are reserved for the inherited methods; your own start at 10. A method set to -1 is not implemented.

yaml · clio_mod.yaml
module_name: MOD_NAME
namespace: clio::run
version: 1.0.0

# Inherited methods
kCreate: 0        # create the container
kDestroy: 1       # destroy the container
kNodeFailure: -1  # not implemented
kRecover: -1
kMigrate: -1
kUpgrade: -1
kMonitor: 9       # monitor container state

# Custom methods (start from 10)
kCustom: 10

The methods header repeats the ids as constants, defines kMaxMethodId (one past the largest id) and returns the method names the dashboard and the scheduler's cost model use:

c++ · autogen/MOD_NAME_methods.h
namespace clio::run::MOD_NAME::Method {
GLOBAL_CROSS_CONST clio::run::u32 kCreate = 0;
GLOBAL_CROSS_CONST clio::run::u32 kDestroy = 1;
GLOBAL_CROSS_CONST clio::run::u32 kMonitor = 9;
GLOBAL_CROSS_CONST clio::run::u32 kCustom = 10;
GLOBAL_CROSS_CONST clio::run::u32 kMaxMethodId = 11;

inline const std::vector<std::string>& GetMethodNames() {
  static const std::vector<std::string> names = [] {
    std::vector<std::string> v(kMaxMethodId);
    v[kCreate] = "Create";
    v[kDestroy] = "Destroy";
    v[kMonitor] = "Monitor";
    v[kCustom] = "Custom";
    return v;
  }();
  return names;
}
}  // namespace clio::run::MOD_NAME::Method

3. Define the tasks

The tasks header defines the creation parameters, aliases the inherited tasks, and declares one struct per custom method.

CreateParams and the inherited tasks

CreateParams is the configuration the container receives at creation. It must carry a static chimod_lib_name naming the library the runtime loads, a cereal serialize, and only fields that cereal can serialize. Create, destroy and monitor come from the admin module:

c++ · MOD_NAME_tasks.h
#include <clio_runtime/clio_runtime.h>
#include <clio_runtime/admin/admin_tasks.h>
#include "autogen/MOD_NAME_methods.h"

namespace clio::run::MOD_NAME {

struct CreateParams {
  clio::run::u32 worker_count_;
  clio::run::u32 config_flags_;

  // Required: the library the module manager loads (lib<name>_runtime.so).
  static constexpr const char* chimod_lib_name = "clio_MOD_NAME";

  CreateParams(clio::run::u32 worker_count = 1, clio::run::u32 config_flags = 0)
      : worker_count_(worker_count), config_flags_(config_flags) {}

  template<class Archive>
  void serialize(Archive& ar) { ar(worker_count_, config_flags_); }
};

using CreateTask  = clio::run::admin::GetOrCreatePoolTask<CreateParams>;
using DestroyTask = clio::run::admin::DestroyTask;
using MonitorTask = clio::run::admin::MonitorTask;

A module that is created from the compose file reads its YAML entry in a LoadConfig(const clio::run::PoolConfig &) method on its params struct. The filesystem module's FilesystemConfig parses next_pool_id, metadata_log_path, stream_pool_id, dir_split_entries and case_insensitive this way from pool_config.config_, which holds the entry's remaining keys as a YAML string.

A custom task

A task struct marks each field IN, OUT or INOUT. It needs a default constructor, an emplace constructor that sets the method id, SerializeIn (fields sent to the runtime), SerializeOut (fields sent back), Copy (used when the runtime replicates a task) and AggregateOut (merges a replica's output fields back into the original). Both serializers must call the base class first. Strings that cross the boundary use clio::run::priv::string allocated with CLIO_PRIV_ALLOC.

c++ · MOD_NAME_tasks.h (continued)
struct CustomTask : public clio::run::Task {
  INOUT clio::run::priv::string data_;
  IN clio::run::u32 operation_id_;

  CustomTask() : clio::run::Task(), data_(CLIO_PRIV_ALLOC), operation_id_(0) {}

  explicit CustomTask(const clio::run::TaskId &task_node,
                      const clio::run::PoolId &pool_id,
                      const clio::run::PoolQuery &pool_query,
                      const std::string &data, clio::run::u32 operation_id)
      : clio::run::Task(task_node, pool_id, pool_query, Method::kCustom),
        data_(CLIO_PRIV_ALLOC, data), operation_id_(operation_id) {
    task_id_ = task_node;
    pool_id_ = pool_id;
    method_ = Method::kCustom;
    task_flags_.Clear();
    pool_query_ = pool_query;
  }

  CTP_CROSS_FUN ~CustomTask() {}

  template<typename Archive>
  CTP_CROSS_FUN void SerializeIn(Archive& ar) {
    Task::SerializeIn(ar);
    ar(data_, operation_id_);
  }
  template<typename Archive>
  CTP_CROSS_FUN void SerializeOut(Archive& ar) {
    Task::SerializeOut(ar);
    ar(data_);
  }
  void Copy(const ctp::ipc::FullPtr<CustomTask> &other) {
    Task::Copy(other.template Cast<Task>());
    data_ = other->data_;
    operation_id_ = other->operation_id_;
  }
  // Merge ONLY the OUT fields of a replica. Never delegate to Copy():
  // that would overwrite this task's identity.
  void AggregateOut(const ctp::ipc::FullPtr<clio::run::Task> &other_base) {
    Task::AggregateOut(other_base);
    data_ = other_base.template Cast<CustomTask>()->data_;
  }
};

}  // namespace clio::run::MOD_NAME

Tasks that take part in the collective routing modes also define AggregateIn, which folds another submitter's input into this task before the handler runs once for the whole batch. The template's ManyToOneSumTask sums value_ that way.

Tasks are copied, not shared

A task lives in the submitting process. Its input fields are serialized into a buffer that crosses into the runtime, and only the output fields come back. Do not design a task around the idea that both sides see the same memory.

4. Write the container and its handlers

The runtime class derives from clio::run::Container, exposes using CreateParams = ...; for the CLIO_TASK_CC macro, and declares one handler per method. A handler takes a clio::run::shared_ptr<TaskType> & and returns clio::run::TaskResume.

c++ · MOD_NAME_runtime.h
#include &lt;clio_runtime/clio_runtime.h&gt;
#include &lt;clio_runtime/container.h&gt;
#include &lt;clio_runtime/comutex.h&gt;
#include &lt;clio_runtime/corwlock.h&gt;
#include "MOD_NAME_tasks.h"
#include "MOD_NAME_client.h"

namespace clio::run::MOD_NAME {

class Runtime : public clio::run::Container {
 public:
  using CreateParams = clio::run::MOD_NAME::CreateParams;

  void Init(const clio::run::PoolId&amp; pool_id, const std::string&amp; pool_name,
            clio::run::u32 container_id = 0) override;
  clio::run::TaskResume Run(clio::run::u32 method,
                            clio::run::shared_ptr&lt;clio::run::Task&gt; task_ptr) override;

  clio::run::TaskResume Create(clio::run::shared_ptr&lt;CreateTask&gt; &amp;task);
  clio::run::TaskResume Custom(clio::run::shared_ptr&lt;CustomTask&gt; &amp;task);
  clio::run::TaskResume Monitor(clio::run::shared_ptr&lt;MonitorTask&gt; &amp;task);
  clio::run::TaskResume Destroy(clio::run::shared_ptr&lt;DestroyTask&gt; &amp;task);

  clio::run::u64 GetWorkRemaining() const override;
  // Serialization and copy plumbing (SaveTask, LoadTask, AllocLoadTask,
  // LocalSaveTask, LocalLoadTask, LocalAllocLoadTask, NewCopyTask, NewTask,
  // AggregateOut, AggregateIn): see the dispatch file.

 private:
  Client client_;   // the module's own client, for subtasks
};

}  // namespace clio::run::MOD_NAME

Handler bodies sit between CLIO_TASK_BODY_BEGIN and CLIO_TASK_BODY_END, wait on a future with CLIO_CO_AWAIT, and finish with CLIO_CO_RETURN. The file ends with CLIO_TASK_CC(Namespace::Runtime), which defines the entry points the module manager looks up.

c++ · MOD_NAME_runtime.cc
#include "clio_runtime/MOD_NAME/MOD_NAME_runtime.h"

namespace clio::run::MOD_NAME {

clio::run::TaskResume Runtime::Create(clio::run::shared_ptr&lt;CreateTask&gt; &amp;task) {
  CLIO_TASK_BODY_BEGIN
  // Init() already ran. Read task-&gt;GetParams() here if you need the config.
  CLIO_CO_RETURN;
  CLIO_TASK_BODY_END
}

clio::run::TaskResume Runtime::Custom(clio::run::shared_ptr&lt;CustomTask&gt; &amp;task) {
  CLIO_TASK_BODY_BEGIN
  // A subtask through the module's own client. The worker runs other tasks
  // while this one waits; the result is read from the future afterwards.
  auto sub = client_.AsyncCustom(task-&gt;pool_query_, "child", 0);
  CLIO_CO_AWAIT(sub);
  task-&gt;data_ = sub-&gt;data_;
  task-&gt;return_code_ = 0;
  CLIO_CO_RETURN;
  CLIO_TASK_BODY_END
}

clio::run::TaskResume Runtime::Destroy(clio::run::shared_ptr&lt;DestroyTask&gt; &amp;task) {
  CLIO_TASK_BODY_BEGIN
  task-&gt;return_code_ = 0;
  CLIO_CO_RETURN;
  CLIO_TASK_BODY_END
}

clio::run::u64 Runtime::GetWorkRemaining() const { return 0; }

}  // namespace clio::run::MOD_NAME

CLIO_TASK_CC(clio::run::MOD_NAME::Runtime)
  • Never hold a lock across CLIO_CO_AWAIT. The worker thread runs other tasks while the handler is suspended. For locks that may be held across a wait, use the coroutine-aware clio::run::CoMutex and CoRwLock with the ScopedCoMutex, ScopedCoRwReadLock and ScopedCoRwWriteLock guards.
  • Several workers run handlers concurrently. Container state touched by more than one task needs an atomic or a lock.
  • Monitor answers with msgpack. Pack the reply into task->results_[container_id_] and the dashboard turns it into JSON (see the operator guide).
  • State that must survive restarts or node loss needs Restart, Recover, Expand or Migrate overrides; the defaults re-initialise or do nothing.

The dispatch file

The dispatch file implements the virtual methods of Container by switching on the method id. Init calls the base Init, builds the module's own client, and registers the cost model and method names. Run casts the task to its concrete type and awaits the handler:

c++ · autogen/MOD_NAME_lib_exec.cc (excerpt)
void Runtime::Init(const clio::run::PoolId &amp;pool_id, const std::string &amp;pool_name,
                   clio::run::u32 container_id) {
  clio::run::Container::Init(pool_id, pool_name, container_id);
  client_ = Client(pool_id);
  DefineModel(Method::kMaxMethodId);
  SetMethodNames(Method::GetMethodNames());
}

clio::run::TaskResume Runtime::Run(clio::run::u32 method,
                                   clio::run::shared_ptr&lt;clio::run::Task&gt; task_ptr) {
  CLIO_TASK_BODY_BEGIN
  switch (method) {
    case Method::kCreate: {
      auto&amp; typed = task_ptr.template Cast&lt;CreateTask&gt;();
      CLIO_CO_AWAIT(Create(typed));
      break;
    }
    case Method::kCustom: {
      auto&amp; typed = task_ptr.template Cast&lt;CustomTask&gt;();
      CLIO_CO_AWAIT(Custom(typed));
      break;
    }
    // ... kDestroy, kMonitor ...
    default: break;
  }
  CLIO_CO_RETURN;
  CLIO_TASK_BODY_END
}

The same switch appears in SaveTask and LoadTask (archive << *typed and archive >> *typed), LocalSaveTask and LocalLoadTask, NewTask (CLIO_IPC->NewTask<T>()), NewCopyTask (allocate, then Copy), AggregateOut and AggregateIn. Copy the template's file and change the case list; the operator guide shows the whole file as an X-macro.

5. Write the client

A client derives from clio::run::ContainerClient and is bound to a pool id. Each method allocates a task through the IPC manager, sends it, and returns a clio::run::Future<TaskType>. Wait() blocks until the task completes; the fields are then read through ->. Create goes to the admin pool as a GetOrCreatePoolTask, which loads the module if needed and creates the pool if it does not exist.

c++ · MOD_NAME_client.h
namespace clio::run::MOD_NAME {

class Client : public clio::run::ContainerClient {
 public:
  Client() = default;
  explicit Client(const clio::run::PoolId&amp; pool_id) { Init(pool_id); }

  clio::run::Future&lt;CreateTask&gt; AsyncCreate(const clio::run::PoolQuery&amp; pool_query,
                                            const std::string&amp; pool_name,
                                            const clio::run::PoolId&amp; custom_pool_id) {
    auto* ipc_manager = CLIO_CPU_IPC;
    auto task = ipc_manager-&gt;NewTask&lt;CreateTask&gt;(
        clio::run::CreateTaskId(), clio::run::kAdminPoolId, pool_query,
        CreateParams::chimod_lib_name, pool_name, custom_pool_id, this);
    return ipc_manager-&gt;Send(task);
  }

  clio::run::Future&lt;CustomTask&gt; AsyncCustom(const clio::run::PoolQuery&amp; pool_query,
                                            const std::string&amp; input_data,
                                            clio::run::u32 operation_id) {
    auto* ipc_manager = CLIO_CPU_IPC;
    auto task = ipc_manager-&gt;NewTask&lt;CustomTask&gt;(
        clio::run::CreateTaskId(), pool_id_, pool_query, input_data, operation_id);
    return ipc_manager-&gt;Send(task);
  }

  clio::run::Future&lt;MonitorTask&gt; AsyncMonitor(const clio::run::PoolQuery&amp; pool_query,
                                              const std::string&amp; query) {
    auto* ipc_manager = CLIO_CPU_IPC;
    auto task = ipc_manager-&gt;NewTask&lt;MonitorTask&gt;(
        clio::run::CreateTaskId(), pool_id_, pool_query, query);
    return ipc_manager-&gt;Send(task);
  }
};

}  // namespace clio::run::MOD_NAME

6. Build against an installed prefix

Install CLIO Core as described in Install, then build the module with CMAKE_PREFIX_PATH pointing at the prefix. add_clio_module_client and add_clio_module_runtime read the module name from clio_mod.yaml and the namespace from clio_repo.yaml, and create the targets <package>_<module>_client and <package>_<module>_runtime. LIB_NAME overrides that prefix; it must then match chimod_lib_name.

cmake · CMakeLists.txt
cmake_minimum_required(VERSION 3.20)
project(MOD_NAME CXX)
set(CMAKE_CXX_STANDARD 20)

find_package(clio-core CONFIG REQUIRED)

add_clio_module_client(
  LIB_NAME clio_MOD_NAME
  SOURCES src/MOD_NAME_client.cc)
add_clio_module_runtime(
  LIB_NAME clio_MOD_NAME
  SOURCES src/MOD_NAME_runtime.cc src/autogen/MOD_NAME_lib_exec.cc)

add_executable(my_program main.cc)
target_link_libraries(my_program clio_MOD_NAME_client clio::run::admin_client)
shell
$ cmake -S . -B build -DCMAKE_PREFIX_PATH=$HOME/clio-inst
$ cmake --build build

The external module test in context-runtime/test/unit/external-chimod uses find_package(clio_run REQUIRED) and find_package(clio_run_admin REQUIRED) instead of the umbrella package, with the namespace coming from its clio_repo.yaml; its targets are then external_test::simple_mod_client and clio::run::admin_client. Either form works against an installed tree.

7. Let the runtime find it

When a create task names a chimod_lib_name, the module manager scans directories for a shared library whose name ends in _runtime.so (.dylib on macOS, .dll on Windows). The scan order is:

  1. the directory that holds the runtime's own library;
  2. each entry of CLIO_REPO_PATH, separated by : (; on Windows);
  3. each entry of LD_LIBRARY_PATH;
  4. ./lib, ../lib and /usr/local/lib.

So the usual way to run a module from its build directory is to point CLIO_REPO_PATH at it before starting the daemon. The runtime log reports Loaded ChiMod: <name> for each library it loads.

shell
$ export CLIO_REPO_PATH=$HOME/MOD_NAME/build
$ export CLIO_SERVER_CONF=$HOME/clio.yaml
$ clio_run start &amp;

8. Create the pool

From a client program

A program calls clio::run::CLIO_INIT once before touching any client and clio::run::CLIO_RUNTIME_FINALIZE() before it exits. With CLIO_WITH_RUNTIME=0 in the environment it attaches to the running daemon.

c++ · main.cc
#include &lt;clio_runtime/clio_runtime.h&gt;
#include &lt;clio_runtime/admin/admin_client.h&gt;
#include &lt;clio_runtime/MOD_NAME/MOD_NAME_client.h&gt;

int main() {
  // Attach to a running runtime (CLIO_WITH_RUNTIME=0), else start one here.
  if (!clio::run::CLIO_INIT(clio::run::RuntimeMode::kClient, true)) return 1;

  constexpr clio::run::PoolId kMyPool = clio::run::PoolId(7001, 0);
  clio::run::MOD_NAME::Client client(kMyPool);

  auto pq = clio::run::PoolQuery::Local();
  auto created = client.AsyncCreate(pq, "my_pool", kMyPool);
  created.Wait();

  auto fut = client.AsyncCustom(pq, "hello", 1);
  fut.Wait();
  // fut-&gt;data_ and fut-&gt;return_code_ are now valid.

  clio::run::CLIO_RUNTIME_FINALIZE();
  return 0;
}

From the compose file

Add an entry to the compose: list of the runtime's configuration, or to a separate file given to clio_run compose start. The runtime reads four keys and restart; everything else in the entry is handed to the module as a YAML string for its LoadConfig.

yaml · clio.yaml
compose:
  - mod_name: clio_MOD_NAME      # chimod_lib_name of the module
    pool_name: my_pool
    pool_query: local            # routing used for the create task
    pool_id: "7001.0"            # "major.minor"
    restart: true                # recreate on the next clio_run start
    # module-specific keys follow; see the operator guide for next_pool_id
shell
$ clio_run compose start my_pool.yaml
$ clio_run compose list
$ clio_run compose stop my_pool.yaml

Each entry must come after the entry its next_pool_id names. The compose commands, the pool_query strings and the PoolQuery factories are in the API reference.

Where to go next