CLIO Core

Docs · Context Transfer Engine

The Context Transfer Engine.

The Context Transfer Engine (CTE) is the storage engine under everything else in CLIO Core. It stores named byte ranges in named groups across the memory and storage tiers you compose, places each one by a score, moves it between tiers on request, and answers queries by name, by keyword and by time.

Module
clio_cte_core
Core pool
512.0
C++
clio_cte/core/core_client.h
Python
clio_cte_core_ext

What it is

The CTE is a tiered key-value store that runs as a module inside the Context Runtime. One container of it runs on every node. A program talks to it through a client library in C++ or Python, or indirectly through a filesystem mount, an I/O adapter, a paged vector or an import job.

The problem it solves is placement. A node has several kinds of memory and storage with very different speeds and capacities: HBM, DRAM, pinned host memory, local NVMe, a parallel filesystem, object storage. Code that wants to use all of them has to decide where each piece of data goes, move it when the decision changes, and remember where everything is. The CTE takes that work over. You write a blob with a score; the engine chooses the tier, records the location, and moves the blob later when you or a policy change the score.

Data model

  • Tag. A named group of blobs, roughly a bucket. Each tag has a TagId with a major_ and minor_ part. The name maps to the id, and one container owns the tag. A tag can carry extended attributes and aliases. Context FS uses one tag per file.
  • Blob. A named byte range inside a tag. Puts and gets address a blob by tag id and blob name plus an offset and a size, so a blob can be written and read in pieces. Blob operations are routed to the blob's owner container by hashing the tag and name.
  • Score. Every blob has a placement score from 0.0 to 1.0. 1.0 means the fastest tier, 0.0 the slowest. A new blob written without an explicit score lands at 1.0. A put that passes the sentinel -1.0 keeps the blob's current score.
  • Targets and tiers. Each entry in the storage list of the core's compose entry becomes a block-device pool registered as a target with its own score and capacity. Target types are ram, file, hbm, pinned, noop, s3 and gcs, or an already composed pool such as an erasure-coded array through existing_pool_id. Memory tiers are node-local. Persistent tiers can be registered from a neighborhood of nodes.
  • Placement. At put time the data placement engine (dpe.dpe_type, max_bw by default) picks targets for the blob from its score and the targets' free space. A blob's placement changes later only through reorganization: an explicit ReorganizeBlob with a new score, an Evict that frees the lowest-scored blobs off a tier, or a periodic data organizer policy when one is configured.

Queries

Three kinds of query are answered without reading the blobs back into the caller. TagQuery and BlobQuery match names against regular expressions from metadata. SemanticSearch is BM25 keyword search over blob contents, served by the indexer module rather than the core. TemporalSearch returns blobs whose last-modified timestamp lies in a range. The usage guide shows each one.

The interposition chain

The core is the bottom of a chain of modules that all speak its task interface: cache (563.0), indexer (564.0), optional compressor (562.0), replication (561.0), core (512.0). A client bound to the top of the chain gets all of their behavior. A client bound to 512.0 bypasses them. The chain design page explains the mechanism and the CTE core module page lists every configuration key.

How the other technologies use it

  • Context FS maps every file to a tag and every file page to a blob, and turns fsync into SyncTag.
  • The Context Assimilation Engine imports outside data into tags and blobs, and can register itself as a tag's fault handler so a blob is ingested the first time it is read.
  • The Context Exploration Engine is a bundle, query and retrieve API over tags, blobs and the three queries above.
  • Memory expansion stores every page of a paged CPU or GPU vector as a blob named by its page number.
  • Machine learning keeps its datasets in those paged vectors.
  • The Context Runtime hosts the CTE as a composed pool and runs its tasks on its workers.

Where to go next

  • Install: what the pip wheel and the other install methods include, and how to link the C++ client.
  • Usage: put and get, tags, queries, reorganization and asynchronous calls in C++ and Python.
  • API reference: the C++ Tag and Client classes, the Python module and the compose YAML keys.