Docs
What CLIO Core does for you.
CLIO Core is a runtime of storage, memory and compute modules that you compose into the I/O path a pipeline needs. It places data across RAM, NVMe, disk, GPUs and cloud buckets, and exposes it as a filesystem, a key-value store, paged memory or training data.
- Written for
- Scientists, HPC users and developers
- Works on
- Linux, macOS, Windows
- License
- BSD 3-Clause
The short version
You start one background process per node, the runtime, with a YAML file that lists the modules and storage tiers you want. What you put in front of them decides the product you get: Context FS mounts a directory that any program can use, the key-value store gives C++ and Python direct access to tags and blobs, memory expansion pages GPU and CPU arrays larger than memory, and clio-ml trains models on that data. The quickest way to see it work is the filesystem, so these docs start there.
Underneath every product, CLIO Core decides where each piece of data lives. Recently used data stays in RAM. Colder data moves to NVMe or disk. A persistent copy protects it across restarts. An index records the words in each file, so you can later ask "which files mention sea ice extent?" and get a ranked answer.
What you can do with it
Mount a filesystem
Start the runtime, mount a directory, copy data in, and unmount. Takes about five minutes.
02Compose storage tiers
Combine RAM, NVMe, SSD and disk in one store with a short YAML file.
03Search from Python
Find files by keyword relevance or name pattern, and read them back.
04Watch the runtime
See nodes, workers and storage tiers live in the built-in web dashboard.
The documentation, by technology
Each technology has its own documentation: what it is, how to install it, how to use it and its API.
Context FS
Start the runtime, mount the filesystem, the default composition, and where to go for advanced ones.
02Context Assimilation Engine
Bring files, HDF5, web and cloud data into the stack, and describe content as it is written.
03Context Exploration Engine
Bundle, query and retrieve data by name, keywords or time, and compose storage for search.
04Context Transfer Engine
Tags, blobs and scores: the data model and the C++ and Python APIs.
05Context Runtime
Build a ChiMod: modules, tasks, pools and pool queries.
06Memory expansion
CPU and GPU vectors larger than memory: API, configuration and examples.
07Machine learning
Compose storage and memory for clio-ml, and the API of each algorithm.
The core concepts page explains tags, blobs, scores and pools in more depth. The design section covers the internals.
Where to start
Install
pip install iowarp-coreworks on most workstations. See Installation for conda, Docker, Spack and source builds.Mount and copy a file
Follow Basic usage end to end.
Shape your storage
Use Compose local tiers when the defaults no longer fit.