CLIO Core

Docs · Context Assimilation Engine

The Context Assimilation Engine.

The Context Assimilation Engine (CAE) brings data from outside the stack into the Context Transfer Engine, and can describe it on the way in. It is a runtime module that runs one assimilator per source scheme, a command-line tool that submits YAML lists of transfers, and an optional summarizer that labels data as it is written.

Module
clio_cae_core, pool 400.0
Tool
clio_cae
Interposer
clio_cae_summarizer, pool 401.0
Source
context-assimilation-engine/

What it is

Everything in CLIO Core stores data as tags holding blobs in the Context Transfer Engine (CTE). The CAE is the part that turns something else into tags and blobs: a file on disk, a dataset in an HDF5 file, an object in a bucket, a URL, or a string inside a program. Once imported, that data is tiered, replicated, searched and mounted like anything written through the normal client.

The CAE has three parts:

  • The core module, clio_cae_core. A pool in the runtime that receives a list of transfers, picks an assimilator for each source and writes the result into a destination tag. The default configuration composes it at pool 400.0.
  • The clio_cae tool. Reads an OMNI file, a YAML list of transfers, and submits it to the core module as one task.
  • The summarizer, clio_cae_summarizer. An optional interposer that forwards every CTE operation and, after a matching put, asks a language model to label the blob and stores the reply beside it.

The problem it solves

Scientific data arrives in many containers and from many places. Applications that read it through the CTE want one addressing scheme, tag and blob, and want the data where the CTE can place and index it. The CAE is the single entry point for that: one YAML format for every source, one task that does the copying inside the runtime, and one set of rules for naming the resulting blobs. It also solves two second-order problems:

  • Ingest cost. A transfer can be registered without copying anything. The CAE then ingests a blob the first time something reads it. See On-demand ingest and pagify.
  • Meaning. Data that passes through the summarizer gets a model-written label stored as a sibling blob, and data that passes through the indexer becomes keyword-searchable. See Describing content in the I/O path.

Key concepts

  • Transfer. One source, one destination and a format. In YAML it is one entry under transfers. In C++ and Python it is an AssimilationCtx. Transfers in a list run in order, each finishing before the next starts.
  • Source URL. The part of src before :// or :: is the scheme and selects the assimilator: file::, string::, hdf5::, http://, https://, s3://, gs:// or gcs://, and globus://.
  • Destination. dst is iowarp::<tag name>. The binary and string assimilators write into that tag. The HDF5 assimilator writes one tag per dataset, named <tag name>/<dataset path>.
  • Blob naming. A file becomes blobs chunk_0, chunk_1, and so on, 1 MiB each by default. A string::name source becomes one blob called name.
  • Assimilator. A C++ class implementing BaseAssimilator. Reads one kind of source and puts blobs through the CTE client. Shipped assimilators are listed on the usage page; new ones are described on the extending page.
  • Interposer. A module that speaks the CTE core's task interface and forwards to the pool named by its next_pool_id. The summarizer is one. The CAE core also forwards GetOrCreateTag, PutBlob, GetBlob and SemanticSearch, so a CTE client can be pointed at it.

How it relates to the other technologies

  • Context Runtime hosts the CAE core and the summarizer as pools, runs their task coroutines, and loads their configuration from the compose list.
  • Context Transfer Engine is where everything lands. The CAE core writes through its next_pool_id, by default the top of the interposition chain at 563.0, so imports pass the cache, the indexer and replication before reaching the core.
  • Context Exploration Engine calls the CAE for its context_bundle operation and the CTE for query and retrieve. If you want to import from Python, that is the API to use.
  • Context FS does not use the CAE. Files written through a mount are already tags and blobs. The default configuration composes the CAE beside the filesystem chain, not inside it.
  • Machine learning uses the CAE to map a staged float32 file onto a dataset's tag, one blob per page, so a model can train on a file without loading it.
  • Memory expansion uses the same CTE fault mechanism the CAE uses for on-demand ingest, with its own fault handler rather than an assimilator.

Where to go next

  • Install: which packages include the CAE and the CMake options for HDF5, S3, Google Cloud Storage and Globus.
  • Usage: writing OMNI files, running clio_cae, importing from the cloud, and labeling data in the I/O path.
  • API reference: the OMNI schema, the tool, AssimilationCtx, the C++ client, compose keys and environment variables.
  • Writing an assimilator: the BaseAssimilator interface and the factory.
  • On-demand ingest and pagify: ingest on fault and numbered pages over named blobs.