Install · CLIO Core

What each package includes

The general installation page covers every method. This page says what each one gives you for the CAE.

pip wheel (pip install iowarp-core). Built with CLIO_CORE_ENABLE_CAE=ON and CLIO_CORE_ENABLE_HDF5=OFF; the S3, GCS and Globus options are left at their OFF defaults. The wheel contains the CAE core client and runtime libraries, so clio_run composes the CAE pool from the default configuration and the Python clio_cee module can submit transfers. The wheel does not install a clio_cae command: its console scripts are clio_run, clio_cte_fuse, clio_cte_bench and clio_run_thrpt_bench. Use clio_cee to import from a pip install.
conda (iowarp-core on the iowarp channel). Includes the CAE and installs clio_cae under $CONDA_PREFIX/bin. The recipe's cloud SDK lines are commented out; to add S3 or GCS, build the recipe yourself with the matching CMake flag.
Docker (iowarp/deploy-cpu). Built with the build-cpu-release preset, which leaves CLIO_CORE_ENABLE_HDF5 and the cloud options at their OFF defaults. The image has clio_cae with the file, string and HTTP assimilators. The HDF5 VOL connector it ships is a CTE adapter, not the CAE HDF5 assimilator.
Spack. The iowarp package has variants +cae (default on), +s3_cae for s3:// import and +gcs for gs:// import. Both cloud variants require +cae.
.deb, .rpm, AppImage, Windows installer. Ship the native binaries without Python bindings. The Linux packages include clio_cae. The Windows presets build the CAE with HDF5 off.
Source. Everything below.

Build options

All options are set at configure time on the top-level project. The CAE is on by default; the sources that need an SDK are off by default.

Option	Default	Effect
`CLIO_CORE_ENABLE_CAE`	ON	Builds `context-assimilation-engine`: the core module, the summarizer, `clio_cae` and the tests.
`CLIO_CORE_ENABLE_HDF5`	OFF	HDF5 support in the CAE: the `hdf5::` assimilator, the `hdf5_vol` format and `.h5` export. Needs the HDF5 library.
`CAE_ENABLE_S3`	OFF	`s3://` import. Needs the AWS SDK for C++ (component `s3`) for the `cae_s3_tool` helper and Poco (Net, NetSSL, Crypto) for the in-process read path. If either is missing, configure warns and turns the option off.
`CAE_ENABLE_GCS`	OFF	`gs://` and `gcs://` import. Needs google-cloud-cpp with the storage component. Missing SDK: warn and turn off.
`CAE_ENABLE_GLOBUS`	OFF	`globus://` import and the `globus` format. Needs Poco (Net, NetSSL, Crypto, JSON) and nlohmann_json. Missing: warn and turn off.
`CLIO_CAE_ENABLE_SUMMARY`	OFF	Builds the summarizer's live-model tests and the semantic-search tests. They need a running Ollama server. The summarizer module itself builds regardless.

The summarizer compiles its HTTP client when libcurl and nlohmann_json are found. Without libcurl the module still builds and forwards operations, but labeling returns false. http:// and https:// sources are POSIX only: the assimilator spawns curl, and the cae_http_tool helper is not built on Windows.

Build from source

Start from the general source instructions, then add the CAE options you need. A Linux example with HDF5 and S3:

shell

$ sudo apt install -y libhdf5-dev libpoco-dev
$ git clone --recurse-submodules https://github.com/iowarp/clio-core.git
$ cd clio-core
$ cmake --preset release \
    -DCLIO_CORE_ENABLE_HDF5=ON \
    -DCAE_ENABLE_S3=ON
$ cmake --build build -j"$(nproc)"
$ sudo cmake --install build

The AWS SDK for C++ is not an apt package on most distributions; install it from source or through CI/ci-deps.sh, Spack (+s3_cae) or conda before configuring. Check the configure output for these lines:

text

found AWS SDK + Poco (CAE S3 support: ENABLED)
found google-cloud-cpp storage (CAE GCS support: ENABLED)
CAE summarization support: ENABLED (libcurl + nlohmann_json found)
    Context Assimilation:       ON

The install puts clio_cae in bin, with clio_cae_omni as a symlink (a copy on Windows) for older scripts, and cae_s3_tool and cae_http_tool beside it when built.

Runtime configuration

The default configuration already composes the CAE core at pool 400.0 with next_pool_id: "563.0". If you wrote your own clio.yaml, the entry you need is:

yaml

compose:
  # ... CTE core and chain entries first ...
  - mod_name: clio_cae_core
    pool_name: cae_main
    pool_query: local
    pool_id: "400.0"
    next_pool_id: "563.0"

The entry must come after the pool its next_pool_id names. Point next_pool_id at "512.0" to write to the bare core, skipping the cache and the indexer. The summarizer entry is in the default file, commented out; see the usage page to enable it.

Check the install

Start the runtime with clio_run start. From a pip install, import an inline string through the CEE and list it back:

python

import clio_cee as cee
ctx = cee.ContextInterface()
rc = ctx.context_bundle([cee.AssimilationCtx(
    src='string::hello', dst='iowarp::cae_check', format='binary',
    src_data="hello from the CAE")])
print("bundle rc", rc)
print(ctx.context_query("cae_check", ".*"))   # ['hello']
ctx.context_destroy(["cae_check"])

With a native install, check the tool against a real file:

shell

$ clio_run start &amp;
$ head -c 3M /dev/urandom &gt; /tmp/sample.bin
$ printf 'transfers:\n  - src: file::/tmp/sample.bin\n    dst: iowarp::sample\n    format: binary\n' &gt; /tmp/sample.yaml
$ clio_cae /tmp/sample.yaml
$ cte_search 'sample.*'

A successful run prints ParseOmni completed successfully! and the number of transfers scheduled. The tool exits 1 if the runtime is not running, if the OMNI file is invalid, if a transfer's scheme was not compiled in, or if nothing was scheduled.