Usage · CLIO Core

Train on a CSV or Parquet table

Large tables usually arrive as CSV or Parquet files, often bigger than memory. clio_ml.assimilate streams the columns you name into a page-aligned float32 staging file, and the Context Assimilation Engine maps that file onto the dataset's tag, one blob per page. With on_fault=True (the default) nothing is copied up front: the CAE reads a page from the staging file the first time training touches it, and from then on the page lives on the tiers you composed.

python · assimilated Parquet

import clio_ml as cm

cm.init()
ds, info = cm.assimilate("/data/trips.parquet",          # or a .csv
                         columns=["distance", "duration", "passengers", "fare"],
                         cache_pages=256,                # fewer than info["n_pages"]: out of core
                         stage_dir="/scratch/clio")      # default: $CLIO_ML_STAGE_DIR or next to the file
print(info["n_rows"], "rows in", info["n_pages"], "pages")

model = cm.fit_linreg(ds, iters=50)                      # label column last
print(model.weights, "evicts", model.stats.evicts)

ds2 = cm.attach(info["tag"], info["n_rows"], info["n_cols"])   # reopen the same tag
centers = cm.fit_kmeans(ds2, k=8, iters=20, init_centroids=X0)  # X0: k x features
cm.finalize()

Columns keep the order given. Put label columns last; clio_ml.COLUMNS gives the count of trailing non-feature columns per algorithm. An unknown column raises KeyError naming it; a non-numeric column raises TypeError.
Missing values raise ValueError unless you pass fill_value=.
Readers: Parquet needs pyarrow. CSV uses pyarrow when installed, then pandas, then the standard csv module. The file is read in batches of 65536 rows, never whole.
Ingest: on_fault=True pulls pages on first use, so keep the staging file in place while the dataset is used. on_fault=False ingests every page immediately.
Reuse: cm.attach(tag, n_rows, n_cols) opens a dataset over a tag assimilated earlier, with the same page geometry. cm.stage_table(...) writes a staging file without a runtime.
Requires a clio-core build with CLIO_CORE_ENABLE_CAE=ON; cm.has_cae reports it, and assimilate raises RuntimeError without it. The package's tables test checks that a k-means and a linear regression fit on an assimilated CSV or Parquet file, with ingest on fault and eagerly, are bit-equal to the same fits on the rows passed as an array.

Train on a NumPy array

Every fit_* accepts a 2-D float32-convertible array. Two ways to run it: in memory with no runtime, or paged through the runtime with cm.dataset(X). Both return the same native result objects. This example fits linear regression both ways and checks that the paged fit, run out of core, is bit-equal to the in-memory fit. It follows the package's own test.

python

import os
import numpy as np

os.environ.setdefault("CLIO_SERVER_CONF", "/path/to/clio_ml.yaml")
os.environ.setdefault("CLIO_BIND_ADDR", "127.0.0.1")
import clio_ml as cm

rng = np.random.default_rng(1)
X = rng.normal(size=(4096, 7)).astype(np.float32)
y = X @ np.array([1.5, -2, 0.5, 3, -1, 0.25, -0.75], np.float32) + 0.5
data = np.column_stack([X, y]).astype(np.float32)   # 7 features, then the label

# 1. In memory: no runtime needed.
plain = cm.fit_linreg(data, iters=20)
print(plain.weights[:7], "intercept", plain.weights[7])
print("prediction", cm.predict("linreg", plain, data[0, :7]))

# 2. Paged: a cache smaller than the data (8 pages here) runs out of core.
cm.init()
ds = cm.dataset(data, cache_pages=4)
paged = cm.fit_linreg(ds, iters=20)
print("evicts", paged.stats.evicts,
      "bit-equal", np.array_equal(plain.weights, paged.weights))
cm.finalize()

Options are keyword arguments named exactly as the C++ options fields, so fit_kmeans(X, k=8, iters=20) sets KMeansOptions::k and KMeansOptions::iters. An unknown name raises TypeError. Nested options take a dict, for example fit_rf(X, num_trees=4, tree={"max_depth": 3}).
k-means and GMM take initial centroids or means. For an array they default to the first k rows; for a Dataset you must pass init_centroids= or init_means=. k-nearest neighbours needs queries=.
predict(name, result, x) evaluates one feature row for the algorithms that have a predictor.
The in-memory path calls the algorithm's *Plain function with a 16 KiB page geometry and at most four blocks. clio_ml.native exposes every Fit*, Fit*Plain, Fit*Reference, Mine* and *Predict function directly.

Train from C++

Start the runtime in client mode and the Context Transfer Engine client, build a Dataset, preload or assimilate it, then call the algorithm's Fit* function with its options struct. The header for each algorithm is named on its page.

cpp

#include "clio_ml/glm.h"

// Runtime up: clio::run::CLIO_INIT(clio::run::RuntimeMode::kClient, true)
// and clio::cte::core::CLIO_CTE_CLIENT_INIT().
constexpr clio_ml::u32 kNf = 7, kCols = kNf + 1, kBlocks = 4;
constexpr clio_ml::u64 kPageBytes = 16 * 1024;           // 4096 floats = 512 rows
std::vector&lt;float&gt; data = /* rows of 7 features then the label */;
const clio_ml::u64 rows = data.size() / kCols;

clio_ml::Dataset ds("linreg_demo", {0}, kPageBytes, kBlocks, /*set_size=*/16, rows, kCols);
ds.Vec().Preload(data.data(), data.size());

clio_ml::GlmOptions opt;
opt.iters = 100;
opt.learning_rate = 0.1f;
clio_ml::GlmResult r = clio_ml::FitLinearRegression(ds, opt);
// r.weights: 7 coefficients then the intercept; r.stats.evicts > 0 when out of core
float yhat = clio_ml::GlmPredict(clio_ml::GlmKind::kLinear, r.weights, data.data());

To train on a staged file instead of preloading, construct the Dataset first (it creates the tag), do not preload it, and call clio_ml::AssimilateFloat32File(tag, path, ds.NumPages(), ds.PageBytes(), true) from clio_ml/assimilate.h. The file must hold NumPages() * PageBytes() bytes laid out as the dataset lays them out: rows of NumFeatures() float32 values, never straddling a page, with the tail of the last page zero-filled. clio_ml.stage_table writes such a file from CSV or Parquet.

Compose storage and memory for training

A paged fit needs a runtime with a Context Transfer Engine pool. Every page of a Dataset is a blob in a tag named after the dataset, so the pool's storage tiers decide where pages live when they are not in the cache. The clio-ml tests use a RAM tier on a private port; add file tiers as in Compose local tiers when the data exceeds DRAM.

yaml · clio_ml.yaml

networking:
  port: 9463
runtime:
  num_threads: 4
  queue_depth: 1024
compose:
  - mod_name: clio_cte_core
    pool_name: clio_cte
    pool_query: local
    pool_id: 512.0
    storage:
      - path: "ram::clio_ml"
        bdev_type: "ram"
        capacity_limit: "512MB"
        score: 1.0
      - path: "/mnt/nvme/clio_ml/tier"
        bdev_type: "file"
        capacity_limit: "200GB"
        score: 0.9
    dpe:
      dpe_type: "max_bw"

Point the client at the file and bind to loopback, which the Windows build requires:

shell

$ export CLIO_SERVER_CONF=$PWD/clio_ml.yaml
$ export CLIO_BIND_ADDR=127.0.0.1
$ export CLIO_SERVER_ADDR=127.0.0.1

clio_ml.init() starts the runtime in-process and connects the CTE client; clio_ml.finalize() stops it. In C++ the same two calls are clio::run::CLIO_INIT(RuntimeMode::kClient, true) and clio::cte::core::CLIO_CTE_CLIENT_INIT().

The Dataset and its cache

A Dataset is a row-major float32 matrix stored in a paged vector: gpu_vector on CUDA and SYCL builds, cpu_vector on the CPU build. These are the paged vectors of memory expansion, so the same cache, tiering and coherence rules apply. The constructor takes the cache geometry:

page_bytes: the page size. A row must never straddle a page, so page_bytes / 4 must be a multiple of the column count; the constructor throws otherwise.
nblocks: how many device blocks the fit launches. Each block owns a contiguous range of pages.
set_size: cache frames per block. nblocks * set_size frames smaller than the page count makes the fit out of core, and the result's stats.evicts counts the pages written back.
gpu_ids: the devices to cache on (ignored on the CPU backend).

The Python helpers clio_ml.dataset, attach and assimilate compute this geometry from the row count and column count: whole rows per page, at most four blocks, and cache_pages frames in total (all pages when omitted).

Running as ChiMods

Each algorithm is also a module named ml_<key> (listed in clio_repo.yaml) with two methods beyond the container lifecycle:

Compute (method 10), sent with PoolQuery::Broadcast(): every container opens its node's shard, the tag <tag>_n<container id>, and runs the library Fit* on it. Replicas' models are merged; any disagreement sets the task's mismatch_ flag.
Gather (method 11), sent with PoolQuery::AllToOne: the per-pass all-gather. Each member's partial is written into its container-id slot, so the merge order is fixed and the cluster fit stays bit-identical to a single-device fit with nodes * nblocks blocks.

The C++ client is clio_ml::chimod::Client<Algo>, for example KMeansClient. AsyncCreate(query, pool_name, pool_id) creates the pool, and AsyncCompute(tag, global_rows, cols, page_bytes, nblocks, set_size, compute_id, params) runs the fit; params is the algorithm's options encoded as floats by a helper such as KMeansParams(k, iters, threads, init). The task returns model_ (the flattened model), faults_, evicts_, replicas_, mismatch_ and error_. The ChiMod path has no Python binding.

Where to go next

API reference: every function and property named above, with signatures and defaults.
Overview: the algorithm pages.
Context Assimilation Engine: what happens to the staging file.