CLIO Core

Docs · Memory expansion

Memory expansion.

Memory expansion gives a program a vector larger than the memory it runs in. cpu_vector pages into host memory from any thread; gpu_vector pages into device memory from inside a kernel. Every page is a blob in the Context Transfer Engine, so a vector is bounded by the tiers you compose, not by DRAM or HBM.

Headers
clio_cte/cpu_vector/cpu_vector.h, clio_cte/gpu_vector/gpu_vector.h
Namespaces
clio::cte::cpu_vector, clio::cte::gpu_vector
Page p is
blob "p" in the vector's tag

What it is

A paged vector is a CTE tag plus a page cache. The vector has a logical length in elements and a page size in bytes. Page p is the blob named p in decimal inside the tag, page_bytes long, written at blob score 0.5. In front of the tag sits a set-associative cache of page frames in host memory (cpu_vector) or device memory (gpu_vector). The program drives the cache explicitly: it fetches the element ranges it is about to touch, holds a page to read or write it, flushes the ranges it wrote, and unpins what it fetched.

The problem it solves is the dataset that does not fit. Out-of-core code written by hand has to split data into chunks, move them in and out of memory, and keep track of which chunk is where. The vectors take that over, and because the pages are ordinary CTE blobs, a page that does not fit in the cache spills to DRAM, then to local storage, then to whatever slower tier the runtime composes. Placement, movement between tiers and persistence are the CTE's job.

Key concepts

  • Pages and blobs. A cpu_vector and a gpu_vector opened on the same tag with the same page size and element type see the same data. Any CTE client, including the Python module, can read a page as a blob.
  • The cache. nsets sets of set_size frames. A page's home set is computed from its number. The cache holds at most nsets * set_size pages, and the page-sized memory behind the frames can be capped lower with capacity_pages.
  • Four verbs. Fetch makes element ranges resident and pins their pages. HoldPage returns a view onto the part of a range that lies on one page; it never fetches. Flush writes ranges back to the CTE; there is no dirty bit, so an unflushed write is lost on eviction. UnpinRange releases the pins a fetch took. A page that is never unpinned can never be evicted.
  • Generations. The first argument of Fetch and Flush is a generation number. 0 means an ordinary transfer. Otherwise a flush stamps the generation on the extents it wrote, and a fetch waits until the writer has published at least that generation. This is how several nodes or kernels share one vector without invalidation traffic.
  • Tiers. Pages are written at score 0.5. With the default max_bw placement engine a tier scored above 0.5 is not chosen on its own; a prefetch hint or a rescore moves a page into it. The placement page explains the rule.
  • Lazy copies. Copy(new_name) makes a copy-on-write snapshot. It creates the copy's tag with the checkpoint module (pool 566.0, clio_cte_checkpoint) registered as the tag's fault handler and the source tag as the fault parameter. Nothing is duplicated until a page of the copy is touched; Copy(new_name, true) or MaterializeAll() forces every page to real bytes.

How it relates to the other technologies

  • The Context Transfer Engine stores the pages, places them by score and serves fetches and flushes. The vector constructors take the CTE pool to use; gpu_vector also reads CLIO_GV_STORAGE_POOL, so pages can go through the cache module at 563.0 for node-local caching of remote pages.
  • The Context Assimilation Engine uses the same fault-handler mechanism as Copy to ingest a file on first access. With an empty blob_prefix it names the ingested blobs 0, 1, and so on, which are exactly the page names a vector faults on, so a vector can be opened over a file the CAE has not read yet.
  • Machine learning stores every Dataset in a gpu_vector on CUDA and SYCL builds and a cpu_vector on the CPU build, and uses the CAE's ingest-on-fault to bring training tables in page by page.
  • The Context Runtime completes the transfers. A kernel submits its fetches and flushes as tasks that the runtime's CPU workers execute; the gpu section of the runtime configuration sizes that queue.

Where to go next

  • Install: build options for CUDA and SYCL, the headers and CMake targets, and what the pip wheel contains.
  • Usage: a host-thread example with cpu_vector and a CUDA kernel with gpu_vector.
  • API reference: both class templates, their constructors and methods, and the configuration keys and environment variables.
  • Memory expansion technology page: the measurements behind it.