CLIO Core

Docs · Machine learning

gbt: gradient-boosted regression trees.

Gradient-boosted regression trees with squared loss, the equivalent of Spark MLlib's GBTRegressor. Each level of each tree is one paged pass: the kernel evaluates the prior ensemble per row from resident metadata, accumulates gradient sums per node, feature and bin, and the host picks the variance-reduction split.

Module
ml_gbt
Header
clio_ml/gbt.h
Rows
features, then target
Limits
num_trees at most 64, max_depth at most 6

What it does

Leaf value is -sum g / count and the prediction is sum_t learning_rate * leaf_t(x). The same header and options drive the classifier when logistic is true. Bit-reproducible across paging and block counts.

Rows

n_features feature columns followed by the regression target. clio_ml.COLUMNS["gbt"] is 1.

Python

python
import numpy as np, clio_ml as cm
rng = np.random.default_rng(1)
X = rng.normal(size=(4096, 7)).astype(np.float32)
y = X[:, 0] * X[:, 1] + 0.1 * rng.normal(size=4096)
data = np.column_stack([X, y]).astype(np.float32)

r = cm.fit_gbt(data, num_trees=3, max_depth=3, learning_rate=0.3)
print(r.model.M, "trees of depth", r.model.depth)
print(cm.predict("gbt", r, data[0, :7]))      # regression value
  • fit_gbt(X, **options).
  • Options (GbtOptions): num_trees (20), max_depth (3), num_bins (32), learning_rate (0.3), logistic (False), l2 (0.0), min_hessian (1.0; equals the minimum rows per child for regression), threads (256).
  • Result GbtResult: model and stats. model (GbtModel): nf, B, depth, maxN (slots per tree), M (trees), lr, mn and inv (binning), and feat, split, leaf, value (each M*maxN).
  • predict("gbt", r, x) returns the raw regression value.

C++

cpp
struct GbtOptions {
  u32 num_trees = 20;     // <= 64
  u32 max_depth = 3;      // <= 6
  u32 num_bins = 32;
  float learning_rate = 0.3f;
  bool logistic = false;  // true: binary log-loss classifier
  float l2 = 0.0f;
  float min_hessian = 1.0f;
  u32 threads = 256;
  ClusterExchange cluster;
};
struct GbtModel {
  u32 nf = 0, B = 0, depth = 0, maxN = 0, M = 0;
  float lr = 0.0f;
  std::vector<float> mn, inv;
  std::vector<float> feat, split, leaf, value;  // M*maxN each
  std::vector<float> Flatten() const;
};
struct GbtResult { GbtModel model; VecStats stats; };
GbtResult FitGbt(Dataset &ds, const GbtOptions &opt);
GbtResult FitGbtPlain(const float *data, u64 n_rows, u32 nf, u64 elems_per_page,
                      u32 nblocks, const GbtOptions &opt);
GbtResult FitGbtReference(/* same arguments */);
float GbtPredict(const GbtModel &m, const float *x);        // raw margin / value
float GbtPredictProba(const GbtModel &m, const float *x);   // sigmoid of GbtPredict

The ChiMod is module ml_gbt.

Related: gradient-boosted classifier, decision tree, random forest.