On-demand ingest and pagify.

Ingest on fault

A transfer context can be marked on_fault. It is not an OMNI key; programs that build an AssimilationCtx in C++ set it, and the Python binding does not expose it. ParseOmni then ingests nothing. Instead it:

1. Asks the source's assimilator for BlobBytes(ctx). A 0 answer means the assimilator cannot ingest per blob, and the transfer fails with result code -20. Of the shipped assimilators only the binary file assimilator implements this.
2. Encodes the context and stores it under a key fault:<16 hex digits>, an FNV-1a hash of the encoding, both in memory and as a blob of that name in the registry tag __cae_on_fault, written with the kNoFault flag. The registry lets the CAE on any node serve the fault.
3. Registers the CAE pool as the destination tag's fault handler on every container of the pool it forwards to, using the CTE's GetOrCreateTag variant that takes a fault pool and fault parameters, with the key as the parameter.

From then on, a GetBlob for a blob the tag does not hold is dispatched by the CTE to the CAE's GetBlob handler with the key in the task context. The handler:

1. Looks the key up in memory, or, when the registration was made on another node, loads and decodes the context from the registry tag.
2. Calls the assimilator's ReadBlob for exactly that blob on a separate I/O thread pool, because it is blocking source I/O. A short final blob is zero-filled to the blob size.
3. Stores the blob in the tag with a put flagged kNoFault, so the next read does not fault, then serves the read from the staged bytes. A size-0 request, which the CTE sends before a partial write, only stores.

Blob size and naming come from the context: chunk_size bytes per blob, 0 meaning the assimilator's default of 1 MiB for files, and blob_prefix followed by the blob index. The default prefix is chunk_; an empty prefix names blobs 0, 1, and so on, which are the page names a gpu_vector pager faults on. The fields carried to other nodes are src, dst, format, src_token, dst_token, s3_region, s3_profile, src_data, blob_prefix, range_off, range_size and chunk_size.

clio::cae::core::AssimilationCtx ctx("file::/data/run42/field.bin", "iowarp::run42_field", "binary");
ctx.on_fault = true;
ctx.chunk_size = 4 * 1024 * 1024;   // 4 MiB blobs
ctx.blob_prefix = "";               // blobs "0", "1", ...
auto f = cae.AsyncParseOmni({ctx});
f.Wait();
// Nothing copied yet. The first GetBlob of "7" reads bytes [28 MiB, 32 MiB) of the file.

Storage policy

A fault-ingested blob can always be re-ingested, so it is not allowed to take the last of a tier. Three environment variables, read by the runtime, control this:

• CLIO_CAE_FAULT_RESERVE_MB, default 512. The handler keeps this many megabytes free across the node's targets and serves from the source without storing when a store would cross that line.
• CLIO_CAE_FAULT_STORE, default on. Set to 0 to never store, giving pure read-through: every read of the tag goes to the source.
• CLIO_CAE_FAULT_STATS. Set to 1 to log the handler's stage counters every 5 seconds.

A store that fails because the tag cannot hold more drops the partial blob, serves from the source for the next 256 faults, then tries to store again. CLIO_CAE_REF_IO_THREADS, default 32, sizes the pool that performs the blocking reads.

The same mechanism elsewhere

The CTE fault hook is not specific to the CAE. The cpu_vector and gpu_vector adapters implement Copy() as a lazy copy-on-write checkpoint by registering their own module as the copy's fault handler with the source tag's name as the parameter. The HDF5 VOL connector binds its tag lazily on first access. The memory expansion documentation covers the vector side.

Pagify: numbered pages over named blobs

A paged consumer such as the gpu_vector addresses its data as page 0, page 1, page 2, and knows nothing about blob names. Pagify turns a page number into bytes. It is split into two headers:

• PageMap in page_map.h is pure arithmetic. It holds a page size and a sorted list of PageExtent entries, each saying which bytes of which blob land at which offset of the global page space. ToPage(page_id) returns the PageSlice list composing one page; FromPage(blob, off, size) returns the PageRef list of pages holding a byte range of one blob. Build(page_size, extents) sorts the extents and drops overlaps, first writer wins. Gaps are legal and read as zero.
• Pagify in pagify.h does the I/O over a map with injected callables: ReadPage(page_id, dst, BlobReadFn) and WritePage(page_id, src, BlobWriteFn). The assembly is therefore testable without a runtime, and the same code serves the CAE pool or a direct client.

The map travels with the data. PageMap::Serialize() produces a text block, and PageMap::Parse() reads it back, returning an empty map on malformed input:

pagemap
page_size=1048576
total=2097152
chunk_0 0 0 1048576
chunk_1 0 1048576 1048576

Each extent line is blob_name blob_off page_off size. The text is stored as a plain blob named pagemap in the same tag as the data. The BuildPageMap hook on BaseAssimilator is where an assimilator would produce that map, but no shipped assimilator overrides it; today the program that ingests the data writes the pagemap blob itself, as the pagify test does.

Reading a page

The CAE core's GetBlob handler treats a blob name of the form page:<id> as a page request. It loads the tag's pagemap blob once per tag, zeroes the caller's buffer, and reads every slice of that page directly into its place. A page composed of several blobs, a blob covering only part of a page, and gaps covered by no extent all read correctly, with gaps as zero. A tag with no pagemap blob fails the page read with a warning. Any blob name without the prefix forwards unchanged, so pagify costs an unpaged deployment one string comparison and is opt-in per request.

// A CTE client pointed at the CAE pool (CLIO_CTE_POOL=400.0, or the CAE client's passthrough).
auto buf = CLIO_IPC->AllocateBuffer(page_size);
auto g = cae.AsyncGetBlob(tag_id, "page:1", 0, page_size, 0, buf.shm_);
g.Wait();   // buf holds page 1 assembled from its slices

The end-to-end test in test/unit/test_cae_pagify.cc writes four blobs and a map into a tag, then reads page:0 and page:1 through a CTE client pointed at the CAE pool: page 0 is assembled from three blobs, and page 1 holds 1000 bytes at offset 500 with zeros around them. test_page_map.cc and test_pagify.cc cover the arithmetic and the I/O without a runtime.