Context Transfer Engine: a key-value store over every tier.

Key takeaways

• A key-value store for C++ and Python: tags hold blobs, with partial reads and writes by offset.
• Each blob is placed on a tier by its score; changing the score moves it.
• Regex, keyword and time-window queries, and a shared-memory path for clients on the same node.
• The substrate every other technology writes to.

• 42.6kYCSB-B ops/s, against Redis 16.8k
• ~2xRedis on read-heavy YCSB workloads
• 15-16 GB/sasync GPU checkpoint into a RAM tier
• 2.0xthe HDF5 async VOL on that checkpoint

The data model

A tag is a named collection, a blob is a named value inside it. Puts and gets take an offset, so large values can be written and read in pieces. Each put carries a score from 0 to 1 that the placement policy uses to choose a tier, and the organizer can move a blob later with ReorganizeBlob. Queries match tag and blob names with regular expressions, and the default setup indexes what you write for keyword (BM25) search.

import clio_cte_core_ext as cte

cte.clio_init(cte.RuntimeMode.kClient, False)
cte.initialize_cte("", cte.PoolQuery.Dynamic())

tag = cte.Tag("experiments/run-42")          # get or create the namespace
tag.PutBlob("params", b'{"dt": 0.01}')       # put bytes (zero-copy in runtime mode)
size = tag.GetBlobSize("params")
print(tag.GetBlob("params", size, 0))        # get bytes back
print(tag.GetContainedBlobs())               # list keys in the tag

The same operations exist in C++ (clio::cte::core::Tag::PutBlob, GetBlob), with asynchronous variants such as AsyncPutBlob, and on the client for whole-store queries (TagQuery, BlobQuery, SemanticSearch). The Python API reference lists them all.

Performance against Redis, RocksDB and DynamoDB

The repository's YCSB binding (context-transfer-engine/benchmark/ycsb) runs the standard workloads against CLIO Core and three other stores. Throughput in operations per second, higher is better:

Higher is better. 100k records of about 1.1 KB, 100k operations, 4 threads, one WSL2 machine. CLIO Core measured 34,566-55,866 on workload C across runs; the bar shows the low end.

How to read it:

• CLIO Core's shared-memory read path is about twice as fast as Redis on the read-heavy workloads, B and C.
• The update-heavy workloads, A and F, are slower than Redis because the binding implements an update as a full-record read followed by a full-record rewrite, where Redis updates one hash field. A partial put at an offset is the planned fix.
• Embedded RocksDB has no inter-process hop at all and leads every workload. CLIO Core is a separate runtime process that many clients and nodes share.
• DynamoDB Local is the local emulator, not the AWS service. All records sit in one tag, so scale-out across tags is not exercised. These are single-node numbers on a development machine.

GPU checkpoints through the store

The kvhdf5 adapter writes HDF5-style datasets from GPU kernels as CTE blobs. Its scaling campaign checkpoints a Gray-Scott simulation on an RTX 4090 with local NVMe, comparing asynchronous blob puts against HDF5:

• CLIO async puts~15.6 GB/s
• HDF5 async VOL~7.8 GB/s
• Threaded raw writer7.2 GB/s
• Inline raw writer5.8 GB/s

I/O-bound GPU checkpoint (8 steps) into a RAM tier, higher is better. The HDF5 async VOL figure is derived from the campaign's measured 2.0x ratio. On an NVMe file tier every arm is disk-bound at about 1.1 GB/s.

Absolute numbers are this machine's; the campaign notes that the ratios carry over but the bandwidths must be re-measured on your hardware.

Put and get bandwidth on a workstation

clio_cte_bench, put then get, 4 client threads with 8 requests in flight each, against a separate runtime with a 16 GB DRAM tier (cte_config_ram.yaml). Release build of dev (f88db54bf) on Windows: Ryzen 7 3800X, 32 GB RAM. The client reaches the runtime over loopback TCP, the transport Windows auto-selects.

Aggregate bandwidth: 1,474 MB/s at 1 MiB, 945 MB/s at 64 KiB, 106 MB/s at 4 KiB.

One run each. Longer 64 KiB and 4 KiB runs (80,000 operations) stopped partway with put errors on this build, so the small-value rows use shorter runs.

Where to go next

• Core concepts: tags, blobs, scores
• CTE data model and data path
• Query stored data from Python