CTE core module · CLIO Core

What it does

The core stores blobs (named byte ranges) inside tags (named groups, roughly a bucket or a file) on storage targets (block-device pools: RAM, pinned host memory, HBM, a backing file, S3, GCS or a no-op device). Every blob has a score between 0.0 and 1.0. The score selects the tier at placement time and drives later reorganization: 1.0 means the fastest tier, 0.0 the slowest. A new blob without an explicit score lands at 1.0; a put that omits the score keeps the blob's current one.

Use the core directly when you want a tiered key-value style store with explicit control over placement. Use it through the cache, replication, compressor and indexer modules, or the filesystem, when you want those policies added on top. None of them bypass the core; see CTE core design for the data path, distribution and concurrency model.

Where it sits

Pool. The core is composed at pool 512.0, the constant kCtePoolId that every CTE client and adapter resolves to by default. One container runs per node.
Below it. Each entry in its storage list becomes a bdev pool that the core creates and registers as a target. A target can instead bind to an already composed pool such as safe-bdev with existing_pool_id.
Above it. The interposer modules speak the core's own task interface and stack with next_pool_id. The canonical chain in the default configuration is cache (563.0) to indexer (564.0) to optional compressor (562.0) to replication (561.0) to core (512.0). Each entry must be composed after the pool it forwards to.
CAE. The Context Assimilation Engine core sits at its own pool 400.0 and forwards tag creation, put, get and semantic search to the CTE core. The default configuration warns against placing CAE in front of the CTE at 512.0, because the two modules' method IDs collide.

Composing it

This entry is the CTE section of context-runtime/config/clio_default.yaml with the comments removed. A RAM tier with capacity 0g takes 80% of system DRAM; a file tier grows lazily, so its capacity is a cap, not an upfront allocation.

yaml

compose:
  - mod_name: clio_cte_core
    pool_name: cte_main
    pool_query: local
    pool_id: "512.0"
    storage:
      - path: "ram::cte_ram_tier1"
        bdev_type: "ram"
        capacity_limit: "0g"
        score: 1.0
      - path: "${CLIO_STORAGE_ROOT}/cte_disk_tier.dat"
        bdev_type: "file"
        capacity_limit: "10GB"
        score: 0.2
        persistence_level: "temporary"
    performance:
      metadata_log_path: "${CLIO_STORAGE_ROOT}/cte_metadata_log"
      transaction_log_capacity: "32MB"
    dpe:
      dpe_type: "max_bw"
    targets:
      neighborhood: 1
      default_target_timeout_ms: 30000
      poll_period_ms: 5000

To use a pool that is already composed as a tier, replace the path, bdev_type and capacity_limit keys with the pool's identifier. The pool must implement the bdev task interface, and the core reads its free space from that pool's statistics. This form is exercised by test/unit/test_cte_existing_pool.cc.

yaml

    storage:
      - existing_pool_id: "350.0"
        existing_pool_module: "clio_safe_bdev"
        score: 0.5

Operations

Method IDs come from context-transfer-engine/core/clio_mod.yaml; the client surface is include/clio_cte/core/core_client.h. Every client method has an asynchronous form that returns a future, and the Tag wrapper adds blocking convenience calls.

Targets

RegisterTarget and UnregisterTarget add or remove a storage target on a node. Registration takes the target name, bdev type, total size and optionally an existing pool to attach and a file growth unit. Compose registers the storage entries for you; clients call this only when adding tiers at run time.
ListTargets, GetTargetInfo (score, capacity, performance figures) and StatTargets, which refreshes free space and performance for every target and runs periodically.
GetCapacity reports remaining capacity.

Blobs

PutBlob writes a byte range at an offset, with an optional score and a Context. Variants: a shared-memory pointer, a private buffer (the runtime copies it once, or not at all when the caller is inside the runtime), a vectored put of several regions in one task, and a deferred put that returns immediately and is tracked in a process-wide registry with flow control. Conditional forms put only if the blob is absent or still at a given version.
GetBlob reads a byte range, with the same shared-memory, private-buffer, vectored and deferred variants. A generational get waits for a writer to publish a given generation. When the bytes are in the local RAM tier and the metadata mirror is attached, a private-buffer get copies them without a round trip.
DelBlob, TruncateBlob, GetBlobSize, GetBlobScore, GetBlobInfo (score, size, blocks and targets) and GetResidency.
ReorganizeBlob sets a new score and moves the blob to the tier that score selects. ReorganizeHint passes a phase hint to the data organizer. Evict frees the lowest-scored blobs off a tier.
Pod variants of put, get and reorganize, and batched multi-put, multi-get and multi-score forms, exist for GPU callers with fixed-length names.

Queries

TagQuery matches tag names against a regular expression. BlobQuery matches blobs by tag and blob name patterns. ListLocalBlobs lists blobs matching a name pattern on the queried containers.
SemanticSearch runs BM25 keyword search over blob contents with a tag and blob filter and a top-k limit. TemporalSearch returns blobs whose last-modified time falls in an inclusive range, filtered by tag and blob patterns and sorted by timestamp.
PollTelemetryLog returns telemetry records newer than a given logical time.

Client API

In C++ the Tag wrapper covers the common case. Its blocking put from a private buffer defaults the score to 1.0; the asynchronous and shared-memory forms default to -1.0, which means keep the existing score.

cpp

#include &lt;clio_cte/core/core_client.h&gt;

clio::cte::core::Tag tag("experiment/run1");      // GetOrCreateTag
std::string payload = "hello";
tag.PutBlob("step0", payload.data(), payload.size());   // offset 0, score 1.0
std::string out(payload.size(), '\0');
tag.GetBlob("step0", out.data(), out.size());
float score = tag.GetBlobScore("step0");
tag.ReorganizeBlob("step0", 0.2f);                  // demote to a slower tier

The Python module is clio_cte_core_ext, built with nanobind from context-transfer-engine/wrapper/python/core_bindings.cc. Blob payloads are bytes. The Tag class binds PutBlob, GetBlob, GetBlobScore, GetBlobSize, GetContainedBlobs, ReorganizeBlob and GetTagId; the Client class adds DelBlob, RegisterTarget, TagQuery, BlobQuery, SemanticSearch, TemporalSearch, PollTelemetryLog, the conditional and deferred puts, and the asynchronous forms, which return a future with wait, done and result. All blocking calls release the GIL.

python

import clio_cte_core_ext as cte

cte.clio_init(cte.RuntimeMode.kClient)
tag = cte.Tag("experiment/run1")
tag.PutBlob("step0", b"hello", 0)
size = tag.GetBlobSize("step0")
data = tag.GetBlob("step0", size, 0)
tag.ReorganizeBlob("step0", 0.0)

client = cte.get_cte_client()
client.DelBlob(tag.GetTagId(), "step0")

The Python API page covers the module in more depth.

Configuration

Keys are parsed in context-transfer-engine/core/src/core_config.cc; defaults are the constructors in core_config.h. Sizes accept suffixes from k to p (powers of 1024).

storage (list)

path: ram::<name> for memory tiers, a file path for file tiers, a URL with a prefix for s3 and gcs. Required unless binding to an existing pool.
bdev_type: file, ram, hbm, pinned, noop, s3 or gcs.
capacity_limit: size string. 0 is valid only for ram and means 80% of system DRAM.
score: 0.0 to 1.0. Absent means automatic scoring from measured bandwidth.
persistence_level: volatile (default), temporary or long_term. Placement filters and persistent replicas key off this, so a tier left at volatile can never satisfy a persistence request.
growth_unit (file tiers): lazy-growth step, default 1 GB. The first unit is reserved at creation.
existing_pool_id, existing_pool_module: bind to an already composed pool; the module name is for logging only.

performance

metadata_log_path: directory for the metadata write-ahead log and snapshots. Empty (the built-in default) disables metadata durability; the shipped configuration sets it.
transaction_log_capacity: WAL capacity, default 32 MB. A full snapshot is rebuilt when the WAL exceeds it.
flush_metadata_period_ms (5000) and flush_data_period_ms (10000): periods of the metadata and data flush tasks. flush_data_min_persistence (1) is the lowest persistence level the data flush writes to.
fsync_mode: durable (default) returns from fsync once bytes sit on a tier at the minimum persistence and are synced along with the WAL; deferred returns at once and relies on the periodic flushes.
stat_targets_period_ms (1000): how often each target's capacity and free space are re-read. Validated to 10 to 60000.
max_concurrent_operations (64), score_threshold (0.7), score_difference_threshold (0.05), target_stat_interval_ms (5000, marked legacy in the header).

targets

neighborhood: how many nodes, in ring order, a container registers persistent targets from. The built-in default is 4; the shipped configuration sets 1, so placement is owner-local. Memory tiers are always node-local regardless.
default_target_timeout_ms (30000), poll_period_ms (5000).
failover_to_successor (false): route a dead owner's blobs to the first live successor, which serves the replication module's remote copy.

dpe and organizer

dpe.dpe_type: max_bw (default), round_robin or random.
organizer (top level of the entry, not under performance): none (default), frecency, grayscott, cyclic, scatter or hotset. organizer_tasks (1) periodic replicas partition the blob space; organizer_period_ms (5000) is the round period.

The config class can also load a standalone CTE configuration file from the path in CLIO_SERVER_CONF; an unset variable means built-in defaults, not an error.

Placement and organizer hooks

Two policy interfaces are pluggable without touching the data path.

Data placement engine. DataPlacementEngine::SelectTargets in include/clio_cte/core/dpe/dpe.h receives the available targets, the blob's score and the size to place, and returns an ordered list of targets. Policies live one per file under dpe/ and are registered in DpeFactory::CreateDpe. The placement page describes how max_bw uses the score and capacity.
Data organizer. DataOrganizer::Reorganize in include/clio_cte/core/data_organizer/data_organizer.h is a coroutine called by each periodic DynamicReorganize task. It receives a read-only snapshot per blob (score, last read and write, access count, size) and applies new scores through the runtime's reorganize logic. Policies are registered in DataOrganizerFactory::Get. With organizer: none no periodic task is spawned and placement changes only through explicit ReorganizeBlob calls.

How other modules attach

Every interposer derives from CoreInterposer in include/clio_cte/core/core_interposer.h. It forwards any core method it does not override to the next pool verbatim, routes tasks exactly as the core does (blob verbs to the blob's owner by hash, tag creation by tag-name hash), and numbers its own verbs at 100 and above so they never shadow a core method. Compose order must put each interposer after the pool its next_pool_id names.

Replication (561.0) writes durable copies on the next nodes by hash before the put is acknowledged and keeps the shadow copies that failover_to_successor serves.
Compressor (562.0, optional) transforms bytes on the way down and marks the blob with a transform flag the core persists.
Indexer (564.0) sits above the compressor so its read-backs see logical bytes, and keeps its own index log.
Cache (563.0) is the top of the chain. It keeps a raw node-local copy, routes reads to the submitter's node, and keeps the zero-IPC shared-memory path live even for compressed blobs. A client bound to the chain attaches the core's metadata mirror with AttachShmCacheOf.
Filesystem maps files to tags and blobs, and points at the top of whatever chain is composed. Its fsync becomes SyncTag.

Behaviors worth knowing

Durability: Blob layouts are logged to the WAL after the bytes are on the device. Without metadata_log_path the disk tier's bytes survive a restart but nothing remembers which blob they belong to. Bytes that lived only in RAM are gone after a restart; reads into that range fail with an I/O error rather than returning zeros.
Consistency: Each blob has one owner container that serialises its mutations. Reads of one blob run concurrently with each other and never block behind writers except in-place overwrites, which drain readers first. Conditional puts fail with distinct return codes.
Capacity: A put with no room evicts droppable (cache) data on the local node and retries once. Authoritative data is never dropped by the put path; demotion happens through the organizer or explicit reorganization.
Startup: A put that arrives before targets are registered waits up to two minutes instead of failing.

Current limits

The method manifest in clio_mod.yaml has drifted from the generated method header: methods 36 to 42, MultiPutBlob (48) and RegisterReplicaContainer (49) exist in the header but not the manifest, so the manifest's own comment says to treat the header as authoritative until reconciled.
The organizer validation error message lists only none and frecency even though all six names are accepted.
The shared-memory metadata mirror is sized once at startup and holds at most 16 blocks per blob; larger or fragmented blobs take the normal request path.
Client-mode puts have no zero-IPC fast path. A direct write into the RAM segment could race the organizer relocating the blob, so writes always go through a task.
Node failure, recovery, migrate and upgrade lifecycle methods are not implemented by this module.

How it is tested

context-transfer-engine/test/unit/: Catch2 tests for the core, including test_core_functionality.cc, test_tag_operations.cc, test_reorganize_blob.cc, test_cte_config_errors.cc, test_cte_config_dpe.cc, test_cte_existing_pool.cc, test_data_organizer.cc, test_cte_transaction_log.cc, test_query.cc, test_evict.cc, test_cte_shm_metadata_cache.cc, test_torn_read.cc and the interposer stack tests.
context-transfer-engine/test/integration/: multi-node Docker Compose scenarios (distributed, restart, coherence, client_crash, replication_persist and others) with their own clio_config.yaml files.
context-transfer-engine/wrapper/python/: Python tests such as test_bindings.py, test_cte_async.py, test_cte_conditional.py and test_cte_semantic_search.py.