Technologies
The technologies, at a glance.
CLIO Core is a composable I/O stack: one runtime, one store and a set of modules you compose in YAML. Each technology below is a way of using that stack, and they all share the same data, so a file written through the mount can be searched, paged into GPU memory or trained on without a copy.
- Storage
- Context FS, Context Transfer Engine
- Context
- Assimilation and Exploration Engines
- Runtime
- Context Runtime
- Compute
- Memory expansion, machine learning
Context FS
Storage
- Mount it like any other directory on Linux, macOS and Windows through FUSE. Applications run unchanged.
- Files spread across the tiers you compose (RAM, NVMe, erasure-coded arrays, Amazon S3 or Google Cloud buckets) by score, with distributed metadata and no metadata server.
- Every file is also tags and blobs in the Context Transfer Engine, so search, assimilation and the other technologies use it without a copy.
Context Assimilation Engine
Context
- Brings files, HDF5 datasets, web downloads, S3, Google Cloud and Globus data into the stack with one YAML file.
- Ingests on fault: register a source and only the parts that are read are pulled in, when they are read.
- Describes data in the I/O path: the summarizer labels it and the indexer makes it searchable as it lands.
- Serves imported data as numbered pages, so memory expansion and clio-ml can work on it directly.
Context Assimilation Engine overview · Docs
Context Exploration Engine
Context
- One small API in C++ and Python: bundle, query, retrieve and destroy.
- Finds data by name pattern, by keyword relevance (BM25) or by time window, across every node at once.
- Sees everything written through Context FS, the Context Transfer Engine or the assimilation engine, with no import step.
- Returns matches packed into one size-bounded buffer, ready for an analysis or a model.
Context Exploration Engine overview · Docs
Context Transfer Engine
Storage
- A key-value store for C++ and Python: tags hold blobs, with partial reads and writes by offset.
- Each blob is placed on a tier by its score; changing the score moves it.
- Regex, keyword and time-window queries, and a shared-memory path for clients on the same node.
- The substrate every other technology writes to.
Context Transfer Engine overview · Docs
Context Runtime
Runtime
- Hosts every module as a long-lived pool and runs work as tasks, rather than as a fixed set of MPI ranks.
- One abstraction for moving data over shared memory, sockets, TCP and storage tiers.
- Recompose the stack, add nodes and recover from failures while it runs.
Context Runtime overview · Docs
Memory expansion
Memory
- Paged vectors for CPU and GPU code whose data is larger than device memory.
- Pages spill to DRAM and storage tiers; placement, coherence and persistence are handled for you.
- Measured within 0.88-1.19x of MPI, oneCCL and Intel SHMEM on up to 64 Aurora nodes.
Memory expansion overview · Docs
Machine learning
Compute
- clio-ml: the Spark MLlib algorithm set, over 30 modules, on CPU, CUDA and SYCL.
- Trains out of core: data larger than GPU memory streams from the tiers you compose.
- Measured 8-59x faster per iteration than Spark MLlib.
Machine learning overview · Docs
How the numbers on these pages were measured
Every figure on the technology pages comes from a benchmark in the repository, with its hardware and its caveats. Where a comparison favors another system, the page says so. Most runs are single runs on one machine or a small allocation; treat them as measured examples, not guarantees for your hardware. Each page links the source document so you can rerun it.