API reference · CLIO Core

Python package: clio_ml

Module attributes

clio_ml.backend: "cpu", "cuda" or "sycl", whichever the extension was built for.
clio_ml.has_cae: True when the build links the Context Assimilation Engine client, so assimilate works.
clio_ml.COLUMNS: dict from algorithm name to the number of trailing non-feature columns a row has. 0 when every column is data (kmeans, pca, lda, fim, seq), 1 for one label column, 2 for aft (time, event). None for als, pic, w2v and iso, whose rows are fixed records and take no column argument.
clio_ml.native: the extension module. It exposes every Fit*, Fit*Plain, Fit*Reference, Mine* and *Predict function, every Options, Model and Result class, VecStats, GlmKind, Dataset and cae_assimilate.
clio_ml.GlmKind: enum with linear, logistic and svm.

Runtime

init(): starts the in-process runtime in client mode (CLIO_INIT(RuntimeMode::kClient, true)) and the Context Transfer Engine client. Raises RuntimeError if either fails. Required by Dataset, dataset, attach, assimilate and every paged fit. The in-memory fits do not need it.
finalize(): stops the runtime (CLIO_RUNTIME_FINALIZE()).

Datasets

dataset(X, tag=None, page_bytes=16384, nblocks=4, cache_pages=None, gpu_ids=(0,)): a paged Dataset holding X. X is converted to a C-contiguous float32 2-D array; anything else raises ValueError. The page size is rounded down to whole rows, the block count to at most one block per page, and set_size is cache_pages (all pages when None) divided across the blocks. The tag defaults to clio_ml_py_<n>. Ends with Dataset.preload(X).
attach(tag, n_rows, n_cols, page_bytes=16384, nblocks=4, cache_pages=None, gpu_ids=(0,)): a Dataset over pages that already exist under tag, or that the CAE will fault in, with no preload. n_rows, n_cols and page_bytes must match the geometry the pages were written with.
assimilate(src, columns, tag=None, cache_pages=None, stage_dir=None, on_fault=True, fill_value=None, page_bytes=16384, nblocks=4, gpu_ids=(0,)): stages columns of the CSV or Parquet file src with stage_table, builds a Dataset with attach, and maps the staging file onto its tag through the CAE. Returns (dataset, info) where info is a dict with tag, n_rows, n_cols, n_pages, staged_path and on_fault. The tag defaults to clio_ml_<file stem>_<pid>_<n>. The staging file is <tag>.f32 in stage_dir, else $CLIO_ML_STAGE_DIR, else the source file's directory. Raises RuntimeError when has_cae is false.
stage_table(src, columns, out_path, page_bytes, fill_value=None, batch_rows=65536): writes columns of src as a page-aligned, row-major float32 file with whole rows per page and a zero-padded last page. Returns (n_rows, n_cols, page_bytes, n_pages), where page_bytes is the row-aligned size actually used. Needs no runtime. A Parquet source needs pyarrow. Empty columns or a file with no rows raise ValueError, an unknown column KeyError, a non-numeric column TypeError, and missing values ValueError unless fill_value is given.

Dataset class

Dataset(tag, gpu_ids, page_bytes, nblocks, set_size, n_samples, n_features) binds the C++ constructor below. Read-only properties: num_rows, num_features, num_pages, elems_per_page, num_blocks, page_bytes. Method preload(x) writes a C-contiguous float32 array of exactly num_rows * num_features elements into the paged store; a different size raises ValueError.

Fitting

fit_<name>(X, **options) exists for every name in COLUMNS. X is a 2-D array (in memory, no runtime) or a Dataset (paged). Options are keyword arguments; each is set on a fresh instance of the algorithm's native options class, so the defaults are the C++ defaults. An unknown name raises TypeError. A dict value fills a nested options object field by field, for example fit_rf(X, num_trees=4, tree={"max_depth": 3}). The mapping from name to native stem and options class:

name → native stem, options class, trailing columns

kmeans  KMeans               KMeansOptions               0
linreg  Glm (GlmKind.linear) GlmOptions                  1
logreg  Glm (GlmKind.logistic) GlmOptions                1
svm     Glm (GlmKind.svm)    GlmOptions                  1
nb      NaiveBayes           NaiveBayesOptions           1
pca     Pca                  PcaOptions                  0
knn     KnnClassify          KnnOptions                  1
tree    DecisionTree         DecisionTreeOptions         1
rf      RandomForest         RandomForestOptions         1
gbt     Gbt                  GbtOptions                  1
gbtc    Gbt (logistic=True)  GbtOptions                  1
als     Als                  AlsOptions                  fixed records
gmm     Gmm                  GmmOptions                  1
mlp     Mlp                  MlpOptions                  1
bkm     BisectingKMeans      BisectingKMeansOptions      1
fm      Fm                   FmOptions                   1
pglm    PoissonGlm           PoissonGlmOptions           1
lda     Lda                  LdaOptions                  0
fim     FrequentItemsets     FrequentItemsetOptions      0   (Mine*)
aft     Aft                  AftOptions                  2
pic     Pic                  PicOptions                  fixed records
w2v     Word2Vec             Word2VecOptions             fixed records
enet    ElasticNet           ElasticNetOptions           1
iso     Isotonic             IsotonicOptions             fixed records
seq     SequentialPatterns   SequentialPatternOptions    0   (Mine*)
mlr     MultinomialLogistic  MultinomialLogisticOptions  1

For a Dataset, fit_<name> calls the native Fit<Stem>(ds, ..., options) (Mine<Stem> for fim and seq, KnnClassify for knn). For an array it calls Fit<Stem>Plain(X.ravel(), n_rows, n_features, elems_per_page, nblocks, ..., options) with a 16 KiB page holding whole rows and at most four blocks; the n_features argument is omitted for the fixed-record algorithms.
Extra arguments: fit_kmeans(..., init_centroids=) and fit_gmm(..., init_means=) default to the first k rows of an array (k defaults to 8 for kmeans and 2 for gmm when not given) and are required for a Dataset. fit_knn(..., queries=) is always required.
Results are the native *Result objects. Paged results carry stats, a VecStats with faults, puts, evicts, get_errors and put_errors.

Prediction

predict(name, result, x) evaluates one feature row with a fitted result. x is converted to a contiguous float32 array (a float for iso). The dispatch:

linreg, logreg, svm: GlmPredict(kind, result.weights, x).
pglm: PoissonGlmPredict(result.coef, x).
iso: IsotonicPredict(result.model, float(x)).
nb, tree, rf, gbt, gbtc, gmm, mlp, bkm, fm: the algorithm's *Predict(result.model, x).
mlr: MultinomialLogisticPredict(result, x); aft: AftPredictMedian(result, x).
Any other name raises ValueError.

Environment variables

CLIO_ML_DLL_PATH (Windows only): directories, separated by ;, added with os.add_dll_directory before the extension is imported.
CLIO_ML_STAGE_DIR: default directory for assimilate staging files when stage_dir is not given.
CLIO_SERVER_CONF, CLIO_BIND_ADDR, CLIO_SERVER_ADDR: read by the runtime that init() starts. See the Context Runtime docs. On Windows set both addresses to 127.0.0.1.

C++

clio_ml/dataset.h

cpp

namespace clio_ml {
using u32 = clio::run::u32;
using u64 = clio::run::u64;

class Dataset {
 public:
  Dataset(const std::string &amp;tag_name, const std::vector&lt;int&gt; &amp;gpu_ids,
          u64 page_bytes, u32 nblocks, u32 set_size, u64 n_samples,
          u32 n_features,
          clio::run::PoolId storage_pool_id = clio::run::PoolId::GetNull(),
          u32 nsets = 0);   // GPU cache sets; 0 = one per block

  u64 NumRows() const;
  u32 NumFeatures() const;
  u64 RowsPerPage() const;
  u64 PageBytes() const;
  u64 ElemsPerPage() const;
  u64 NumElems() const;      // n_samples * n_features
  u64 NumPages() const;
  u32 NumBlocks() const;     // drivers must launch exactly this many blocks
  u64 RowOffset(u64 r) const;

  PagedVec &amp;Vec();           // cpu_vector::Vector&lt;float&gt; or gpu_vector::Vector&lt;float&gt;
  clio::cte::gpu_vector::DeviceVector&lt;float&gt; GetDevice(int gpu_id);  // GPU builds only
};
}

The constructor throws std::invalid_argument unless page_bytes / sizeof(float) is a positive multiple of n_features. Rows are never padded to a page boundary.
On the CPU backend (CLIO_ML_CPU=1) gpu_ids and nsets are ignored and the vector is a cpu_vector with the same geometry.
PagedVec and VecStats (PagedVec::Stats) come from clio_ml/backend.h, which selects the vector namespace from CLIO_ML_CPU.
Load data with ds.Vec().Preload(ptr, count), or with the assimilate path below.

clio_ml/assimilate.h

cpp

namespace clio_ml {
inline bool HasCae();   // true when built with CLIO_ML_CAE

inline void AssimilateFloat32File(const std::string &amp;tag,
                                  const std::string &amp;path,
                                  std::uint64_t num_pages,
                                  std::uint64_t page_bytes,
                                  bool on_fault);
}

tag is the dataset's tag name without the iowarp:: prefix; path is the local staged float32 file; num_pages and page_bytes are Dataset::NumPages() and Dataset::PageBytes().
It submits one CAE AssimilationCtx with src = "file::" + path, dst = "iowarp::" + tag, format = "binary", range_size = num_pages * page_bytes, chunk_size = page_bytes and an empty blob_prefix, so blob i holds file bytes [i*page, (i+1)*page), named like the pager's pages.
on_fault = true copies nothing up front and makes the CAE the tag's fault handler; the file is read for the life of the tag. false ingests every page now.
Throws std::runtime_error when the build has no CAE or the ingest task fails.
Construct the Dataset first (it creates the tag) and do not preload it.

Build-time macros

CLIO_ML_CPU: 1 on the CPU backend, 0 on CUDA and SYCL.
CLIO_ML_CAE: 1 when the Python extension links the CAE client; defaults to 0.
CTP_ENABLE_SYCL: set on the SYCL backend; the extension reports backend = "sycl" when it is defined and CLIO_ML_CPU is 0.

Where to go next

Per-algorithm options, result fields and C++ signatures: the algorithm pages listed on the overview.
Usage: worked examples for each path.