The modules you compose.

How modules fit together

A module (a ChiMod) is a unit of code that the runtime hosts as a pool. A pool has a name, a numeric pool_id, and containers on the nodes it spans, and it executes its operations as tasks in the runtime's worker threads. Clients and other modules address a pool by its id.

The data-path modules form an interposition chain. Each one is configured with a next_pool_id: it handles the operations it cares about (the cache serves reads it holds, the indexer records what is written, replication fans writes out to copies) and forwards everything else to the pool below. The CTE core sits at the bottom of the chain and places data on block devices. Because each interposer forwards what it does not handle, you can remove one from the chain, or insert another, without changing the modules around it. The interposition chain covers the mechanism.

The default composition

The runtime's default configuration (context-runtime/config/clio_default.yaml) composes this stack, top to bottom. Requests enter at the top and travel down:

1. clio_cae_coreAssimilation: imports external data into the store400.0
2. clio_cte_cacheKeeps hot copies on fast tiers; the chain top for writes and reads563.0
3. clio_cte_indexerIndexes what is written for keyword and name search564.0
4. clio_cte_replicationWrites durable copies to persistent tiers561.0
5. clio_cte_coreTags, blobs, scores and placement across targets512.0
6. clio_bdevBlock devices: RAM, files, cloud buckets301.0

The filesystem (clio_cte_filesystem) and the stream module (clio_cte_stream) also enter the data path at the cache. The compressor, when composed, sits between the indexer and replication. A trimmed version of the default YAML:

compose:
  - mod_name: clio_bdev
    pool_name: "ram::chi_default_bdev"
    pool_query: local
    pool_id: "301.0"
    bdev_type: ram
  - mod_name: clio_cte_core
    pool_name: cte_main
    pool_query: local
    pool_id: "512.0"
    storage:
      - path: "ram::cte_ram_tier1"
        bdev_type: "ram"
        score: 1.0
      - path: "${CLIO_STORAGE_ROOT}/cte_disk_tier.dat"
        bdev_type: "file"
        capacity_limit: "10GB"
        score: 0.2
  - mod_name: clio_cte_replication
    pool_name: clio_cte_replication
    pool_query: local
    pool_id: "561.0"
    next_pool_id: "512.0"
  - mod_name: clio_cte_indexer
    pool_name: clio_cte_indexer
    pool_query: local
    pool_id: "564.0"
    next_pool_id: "561.0"
  - mod_name: clio_cte_cache
    pool_name: clio_cte_cache
    pool_query: local
    pool_id: "563.0"
    next_pool_id: "564.0"

Module pages