CLIO Core

Docs · Machine learning

clio-ml: machine learning on paged data.

clio-ml is a library of Spark-MLlib-style algorithms written over CLIO Core's paged vectors. A training set is a Dataset whose pages live in Context Transfer Engine tiers and stream through a page cache, so a fit is not bounded by GPU or host memory. The same code runs on CPU, CUDA and SYCL, and every algorithm is also a ChiMod.

Source
external/clio-ml
Backends
CPU, CUDA, SYCL
Python
clio_ml (nanobind)
Algorithms
26

What it is

clio-ml is a separate repository that builds inside a CLIO Core checkout at external/clio-ml. It is laid out as a ChiMod repository: one top-level directory per algorithm, each with a C++ API header under include/clio_ml/, a ChiMod client and runtime under src/, and a clio_mod.yaml. Shared code lives in common/: the Dataset wrapper, backend selection, the cluster exchange, and the three paged engines (k-means, generalized linear models, and a generic pass over the rows) for CPU, CUDA and SYCL.

The Python package clio_ml wraps all of it. Every algorithm is a fit_<name> function that accepts either a NumPy array (in memory, no runtime) or a paged Dataset (through the runtime, out of core when the cache is smaller than the data).

The problem it solves

MLlib-style algorithms are usually bounded by the memory of the device that runs them. clio-ml stores the rows in a paged vector instead. Each algorithm reads its data one page at a time and accumulates partial results per block in a fixed order. For a fixed data layout, block count and page size, the result is bit-identical whether the page cache holds the whole dataset or a fraction of it, and whether one node or several take part. Fits run for a fixed iteration count rather than to a tolerance, which keeps runs comparable.

Key concepts

  • Dataset. A row-major float32 matrix of n_samples rows by n_features columns, stored in one paged vector under one Context Transfer Engine tag. Page i is the blob named "i" in that tag.
  • Row layout. Features come first, then the label columns the algorithm expects. clio_ml.COLUMNS[name] gives the number of trailing non-feature columns (0 for k-means, 1 for a label, 2 for AFT's time and event).
  • Page geometry. A row never straddles a page, so page_bytes / 4 must be a multiple of the column count. The constructor throws otherwise.
  • Blocks and cache frames. nblocks device blocks each own a contiguous page range. set_size cache frames per block bound the page cache. Fewer frames than pages makes the fit out of core; the result's stats.evicts counts the pages written back.
  • Backend. Chosen at build time: CUDA when the core build has CUDA, SYCL when it has SYCL without CUDA, otherwise CPU over cpu_vector. clio_ml.backend reports which one.
  • Options and results. Each algorithm has a C++ Options struct and a Result struct. In Python, options are keyword arguments named exactly as the struct fields, and results are the native objects.

How it relates to the other technologies

  • Memory expansion provides the paged vectors (gpu_vector and cpu_vector) that a Dataset wraps. Cache, tiering and coherence rules are the same.
  • Context Transfer Engine holds every page as a blob in a tag. The pool's storage tiers decide where pages live when they are not in the cache.
  • Context Assimilation Engine maps a staged float32 file onto a dataset's tag, one blob per page. This is how clio_ml.assimilate trains on a CSV or Parquet file without loading it into memory.
  • Context Runtime runs the ChiMods. Each algorithm is a module named ml_<key> that fits its node's shard and merges across nodes.

Algorithms

Each page gives the Python and C++ API with option defaults, the row layout and a short example.

Clustering

  • kmeans: k-means (Lloyd)
  • bkm: bisecting k-means
  • gmm: Gaussian mixture model
  • pic: power iteration clustering

Regression

  • linreg: linear regression
  • enet: elastic net and Lasso
  • pglm: Poisson generalized linear model
  • iso: isotonic regression
  • aft: accelerated failure time survival regression
  • fm: factorization machine regression
  • gbt: gradient-boosted regression trees

Classification

  • logreg: binary logistic regression
  • mlr: multinomial logistic regression
  • svm: linear support vector machine
  • nb: Gaussian naive Bayes
  • knn: k-nearest neighbours
  • tree: decision tree
  • rf: random forest
  • gbtc: gradient-boosted tree classifier
  • mlp: multilayer perceptron

Decomposition, recommendation, text and patterns

  • pca: principal component analysis
  • als: alternating least squares
  • lda: latent Dirichlet allocation
  • w2v: word2vec
  • fim: frequent itemsets and association rules
  • seq: sequential pattern mining

Where to go next

  • Install: build clio-ml inside CLIO Core, pick a backend, enable the assimilate path.
  • Usage: train on a CSV or Parquet table, on a NumPy array, or from C++; compose storage; run as ChiMods.
  • API reference: the clio_ml Python package, the C++ Dataset class and AssimilateFloat32File.
  • The algorithm pages listed above.