CLIO Core

Docs · Machine learning

Install clio-ml.

clio-ml is built from inside a CLIO Core configure. The iowarp-core wheel does not include it. A CPU-only clio-ml wheel exists for Python users; GPU backends and the assimilate path need a source build.

Core option
CLIO_CORE_ENABLE_CLIO_ML
Python
CLIO_ML_PYTHON
Backends
CPU, CUDA, SYCL
Assimilate
CLIO_CORE_ENABLE_CAE=ON

What you get from each method

  • pip install iowarp-core installs CLIO Core only. It does not contain clio-ml. See the general install page for that wheel.
  • pip install clio-ml is the package the clio-ml repository's release workflow builds. It is the CPU backend over cpu_vector, built with the Context Assimilation Engine off, so clio_ml.assimilate is unavailable in it. It depends on numpy and needs Python 3.10 or newer.
  • Source build gives every backend and the assimilate path. This is the method the rest of this page describes.
  • conda, Docker, Spack: the CLIO Core recipes and images under installers/ and docker/ do not build clio-ml.

Prerequisites

  • A CLIO Core checkout with clio-ml at external/clio-ml. The core CMake adds it as a subdirectory; clio-ml does not vendor the runtime.
  • CMake 3.20 or newer and a C++20 compiler.
  • For the CUDA backend, the CUDA toolchain the core build uses with CLIO_CORE_ENABLE_CUDA=ON. For SYCL, the oneAPI DPC++ compiler the core build uses with CLIO_CORE_ENABLE_SYCL=ON.
  • For the Python package, a Python interpreter with development headers. nanobind 2.9.2 is fetched at configure time.
  • For assimilating tables from Python: pyarrow for Parquet. CSV uses pyarrow when installed, then pandas, then the standard csv module.

Build as part of CLIO Core

Configure the core checkout with clio-ml enabled. The backend follows the core build: CUDA when CLIO_CORE_ENABLE_CUDA is on, SYCL when CLIO_CORE_ENABLE_SYCL is on without CUDA, and otherwise the CPU backend over cpu_vector.

CPU

shell
$ cd clio-core
$ cmake -S . -B build \
    -DCLIO_CORE_ENABLE_CLIO_ML=ON \
    -DCLIO_ML_PYTHON=ON
$ cmake --build build

CUDA

shell
$ cd clio-core
$ cmake -S . -B build \
    -DCLIO_CORE_ENABLE_CUDA=ON \
    -DCLIO_CORE_ENABLE_CLIO_ML=ON \
    -DCLIO_ML_PYTHON=ON
$ cmake --build build

SYCL

shell
$ cd clio-core
$ cmake -S . -B build \
    -DCLIO_CORE_ENABLE_SYCL=ON \
    -DCLIO_CORE_ENABLE_CLIO_ML=ON \
    -DCLIO_ML_PYTHON=ON \
    -DCLIO_ML_SYCL_AOT_DEVICE=pvc      # optional: ahead-of-time for one device
$ cmake --build build

CMake options

  • CLIO_CORE_ENABLE_CLIO_ML (core, default OFF): adds external/clio-ml to the build. It forces CLIO_ML_ENABLE_CUDA to the value of CLIO_CORE_ENABLE_CUDA.
  • CLIO_ML_PYTHON (default OFF): builds the clio_ml Python package with nanobind and stages it at <build>/python/clio_ml.
  • CLIO_ML_ENABLE_TESTS (default ON): builds the per-algorithm unit tests and registers the clio_ml_python and clio_ml_python_tables ctests.
  • CLIO_ML_SYCL_AOT_DEVICE (SYCL only, default empty): ahead-of-time compiles the kernels for one Intel device, for example pvc. Empty means JIT.
  • CLIO_CORE_ENABLE_CAE (core, default ON): the Context Assimilation Engine. When its client target exists, the Python module is compiled with CLIO_ML_CAE=1 and clio_ml.assimilate works. The configure log prints clio-ml: Python assimilate() enabled (CAE client found) or clio-ml: CAE client not found; clio_ml.assimilate disabled.
Standalone configure

The clio-ml README shows a standalone cmake -S . -B build -DCLIO_CORE_ROOT=/path/to/core invocation with CLIO_ML_ENABLE_CUDA and CLIO_ML_PYTHON. The engine CMake file states that the algorithm libraries are in-tree only, because they use the core build's targets and helpers. Use the core option above for a working build.

Build the Python wheel yourself

external/clio-ml/pyproject.toml builds with scikit-build-core and points the CMake source directory two levels up, at the clio-core root. The wheel is therefore built from inside a clio-core checkout and installs only the clio_ml_python component. Its CMake defines turn on CLIO_CORE_ENABLE_CLIO_ML and CLIO_ML_PYTHON and turn off CUDA, ROCm, SYCL and the CAE, so a wheel built this way is CPU-only without assimilate. There is no source distribution.

Use the staged package

Put the staged directory on PYTHONPATH. On Windows, Python does not search PATH for an extension's DLL dependencies, so list the directories holding the runtime and backend DLLs in CLIO_ML_DLL_PATH, separated by ;, before importing.

Linux and macOS

shell
$ export PYTHONPATH=$PWD/build/python
$ python -c "import clio_ml; print(clio_ml.backend, clio_ml.has_cae)"

Windows

powershell
&gt; $env:PYTHONPATH = "$PWD\build\python"
&gt; $env:CLIO_ML_DLL_PATH = "$PWD\build\bin\Release;$PWD\build\bin;$PWD\vcpkg_installed\x64-windows\bin"
&gt; $env:CLIO_BIND_ADDR = "127.0.0.1"
&gt; python -c "import clio_ml; print(clio_ml.backend, clio_ml.has_cae)"

Check that it works

A fit on an in-memory array needs no runtime, so it is the quickest check of the extension:

shell
$ python -c "import numpy as np, clio_ml as cm; X = np.random.default_rng(0).normal(size=(512, 8)).astype(np.float32); print(cm.fit_kmeans(X, k=4, iters=5).counts)"

To exercise the paged path and the runtime, run the registered ctests from the build directory. They set PYTHONPATH, CLIO_BIND_ADDR=127.0.0.1 and CLIO_ML_DLL_PATH themselves. The tables test exits with status 1 and prints SKIP assimilate when the build has no CAE.

shell
$ ctest --test-dir build -L clio_ml --output-on-failure

Where to go next

  • Usage: assimilate a table and train on it.
  • API reference: the clio_ml package and the C++ headers.
  • Overview: concepts and the algorithm list.