CLIO Core

Docs · Machine learning

w2v: word2vec.

Skip-gram embeddings with negative sampling, the equivalent of Spark MLlib's Word2Vec.

Module
ml_w2v
Header
clio_ml/word2vec.h

What it does

Skip-gram embeddings with negative sampling, the equivalent of Spark MLlib's Word2Vec. Input and output embeddings are learned by full-batch Adam on the logistic loss, one paged pass per step, with gradients summed in a fixed row order so results are bit-reproducible across paging and node counts. You generate the pairs and negative samples.

Rows

Fixed records [center id, context id, label, pad]: label 1 for a positive pair, 0 for a negative sample.

Python

python
import numpy as np, clio_ml as cm
r = cm.fit_w2v(P, vocab_size=5000, dim=64)     # P: (center, context, label, 0)
vectors = np.asarray(r.model.params)[:5000 * 64].reshape(5000, 64)
  • fit_w2v(X, **options) accepts an in-memory array or a paged Dataset from cm.dataset(X) or cm.assimilate(...).
  • Options: vocab_size (required), dim (8), iters (300), learning_rate (0.05, Adam), seed (1), threads (256).
  • Result Word2VecResult: model (V, d, params = input then output embeddings), loss_history, stats.

C++

cpp
Word2VecResult FitWord2Vec(Dataset &ds, const Word2VecOptions &opt);
Word2VecResult FitWord2VecPlain(const float *data, u64 n_pairs, u64 elems_per_page,
                                u32 nblocks, const Word2VecOptions &opt);
u32 Word2VecNearest(const Word2VecModel &m, u32 w);  // nearest word by cosine

The paged call needs the runtime; the Plain call runs the same kernel over a plain copy of the data without one.

Related: LDA. Back to the machine learning overview.