Docs
What CLIO Core does for you.
CLIO Core stores your data across RAM, NVMe and disk, mounts that storage as a normal directory, and indexes what you write so you can search it from Python.
- Written for
- Scientists, HPC users and developers
- Works on
- Linux, macOS, Windows
- License
- BSD 3-Clause
The short version
Most people use CLIO Core as a filesystem. You start one background process, the runtime, and mount a directory. Anything that can open a file can then use it: shell tools, Python, Jupyter, simulation codes and visualization tools.
Under that directory, CLIO Core decides where each piece of data lives. Recently used data stays in RAM. Colder data moves to NVMe or disk. A persistent copy protects it across restarts. An index records the words in each file, so you can later ask "which files mention sea ice extent?" and get a ranked answer.
What you can do with it
Mount a filesystem
Start the runtime, mount a directory, copy data in, and unmount. Takes about five minutes.
02Compose storage tiers
Combine RAM, NVMe, SSD and disk in one store with a short YAML file.
03Search from Python
Find files by keyword relevance or name pattern, and read them back.
04Watch the runtime
See nodes, workers and storage tiers live in the built-in web dashboard.
How the pieces fit
You will see these names in commands and logs:
| Name | What it is | You meet it as |
|---|---|---|
| Runtime | The background process on each node. It hosts every other component. | clio_run start |
| CTE | The Context Transfer Engine: tiered storage of tags (named containers) and blobs (named byte ranges). | Storage tiers in clio.yaml |
| Filesystem | A POSIX namespace built from CTE tags. Every file is a tag, and its data is stored as 1 MiB page blobs. | clio_cte_fuse |
| Indexer | Keeps a keyword (BM25) index of what is written, for search. | SemanticSearch in Python |
| CAE | The Context Assimilation Engine imports external data: files, HDF5, S3 and Google Cloud Storage. | AssimilationCtx |
| CEE | The Context Exploration Engine: a high-level API to bundle, query and retrieve. | import clio_cee |
The core concepts page explains tags, blobs, scores and pools in more depth. The design section covers the internals.
Where to start
Install
pip install iowarp-coreworks on most workstations. See Installation for conda, Docker, Spack and source builds.Mount and copy a file
Follow Basic usage end to end.
Shape your storage
Use Compose local tiers when the defaults no longer fit.