CLIO Core

Technologies · Context

Context Assimilation Engine: bring any data in.

Pipeline data rarely starts in one place. The Context Assimilation Engine turns files, datasets and objects from wherever they live into tags and blobs in the stack, so they can be tiered, replicated, searched, mounted and trained on like anything written through CLIO Core.

Module
clio_cae_core (pool 400.0)
Sources
file, HDF5, HTTP(S), S3, Google Cloud, Globus
Submit
clio_cae transfers.yaml
Labels
clio_cae_summarizer

Key takeaways

  • Brings files, HDF5 datasets, web downloads, S3, Google Cloud and Globus data into the stack with one YAML file.
  • Ingests on fault: register a source and only the parts that are read are pulled in, when they are read.
  • Describes data in the I/O path: the summarizer labels it and the indexer makes it searchable as it lands.
  • Serves imported data as numbered pages, so memory expansion and clio-ml can work on it directly.

One file describes the import

An OMNI file lists transfers: a source URL, a destination tag and a format, with optional byte ranges and dependencies. clio_cae transfers.yaml submits it to the running engine, and the source scheme selects the assimilator: file::, hdf5::, http(s)://, s3://, gs:// or globus://. Each HDF5 dataset becomes its own tag, so later work can address one variable instead of a whole file.

yaml · transfers.yaml
name: ocean_run_42
transfers:
  - src: hdf5::/scratch/run42/output.h5
    dst: iowarp::run42
    format: hdf5
  - src: s3://climate-obs/sst/2024.bin
    dst: iowarp::sst_obs
    format: binary

Only what is read

A transfer marked on_fault copies nothing up front. The engine registers itself as the destination tag's fault handler on every node, and the first read of a missing blob pulls exactly that blob from the source and stores it on the composed tiers. Large archives become usable at once, and data that is never touched is never moved. A reserve keeps re-fetchable data from filling the last of a tier.

Described as it lands

The engine and its summarizer speak the Context Transfer Engine's own interface, so they can sit in the write path. The summarizer sends matching blobs to a language model and stores the reply beside the data as an ordinary blob; the indexer makes both keyword-searchable. Imported data is findable by what it says without a separate pass.

Pages for paged consumers

Memory expansion and clio-ml address data as page 0, page 1, and so on. The engine can serve numbered pages assembled from named blobs, and with an empty blob prefix an imported file's blobs are the pager's pages. That is how clio-ml trains on an assimilated CSV or Parquet table without first loading it into memory.

Where to go next