Context Assimilation Engine core.

What assimilation does

Assimilation copies a source (a file, a dataset, an object) into the Context Transfer Engine as one tag holding one or more blobs. After that, the data is served by the CTE tiers like anything written through the normal client: it can be read by name, searched, exported again, or served to a filesystem view. Use the CAE when data already exists somewhere else and you want it inside the stack without writing a loader against the CTE client yourself.

The unit of work is a transfer: a source URL, a destination tag and a format. A list of transfers is an OMNI file. The client tool parses the file, performs any local preprocessing, and sends the whole list to the CAE pool in one ParseOmni task. The pool runs the transfers in order, each one finishing before the next starts.

Where it sits

The CAE is a pool inside the CLIO runtime, composed alongside a CTE core pool. Its handlers call the CTE client to create tags and put blobs, so a CTE pool must exist before the CAE pool is created. Two placements are in use:

• Side by side. The CAE pool is created at its own ID (400.0 by convention) and only ever acts when an OMNI file is submitted. Blobs are written to the global CTE core pool.
• As an interceptor. The CAE is created at the CTE's own entry pool ID and given a next_pool_id. The operations GetOrCreateTag, PutBlob, GetBlob and SemanticSearch that arrive at the CAE are forwarded unchanged to that next pool, which may be the CTE core directly or another interposer such as the summarizer or the indexer. A CTE client pointed at the CAE pool works unchanged. When next_pool_id is not set, forwarding falls back to the global CTE core pool ID.

The method IDs of the forwarded operations are asserted at compile time to match the CTE core's, which is what lets a task built by the CTE client dispatch to a CAE handler.

Composing it

The minimal deployment from the module's example configuration creates a CTE core and a CAE next to it. The CTE entry must come first.

compose:
  - mod_name: clio_cte_core
    pool_name: cte_main
    pool_query: local
    pool_id: "512.0"
  - mod_name: clio_cae_core
    pool_name: cae_main
    pool_query: local
    pool_id: "400.0"

The interceptor placement, adapted from the summarizer test configuration, puts the real CTE core at 513.0 and the CAE at the entry ID 512.0 with a summarizer between them:

compose:
  - mod_name: clio_cte_core
    pool_name: cte_core
    pool_query: local
    pool_id: "513.0"
  - mod_name: clio_cae_summarizer
    pool_name: clio_cae_summarizer
    pool_query: local
    pool_id: "401.0"
    next_pool_id: "513.0"
  - mod_name: clio_cae_core
    pool_name: cae_labeler
    pool_query: local
    pool_id: "512.0"
    next_pool_id: "401.0"

The only module-specific compose key is next_pool_id, written as "major.minor". Any other key on the entry is ignored by this module. Dynamic routing queries on the forwarded operations are resolved to the local container, so an intercepted task runs on the node that received it.

Supported sources and formats

The source scheme in src selects the assimilator. The scheme is everything before ://, or before :: for the custom forms.

• file::/path: a local file, read in chunks of 1 MB by default and stored as blobs named chunk_0, chunk_1 and so on in the destination tag. The runtime process must be able to read the path. Byte ranges apply.
• string::name: an inline payload carried in the request itself, stored verbatim as one blob named by the part after the scheme. Used by programs that build transfers in code rather than from a file.
• hdf5::/path.h5: an HDF5 file. Each dataset becomes a tag named <dst>/<dataset path>, large datasets are split into several blobs, and dataset_filter selects datasets by glob. Requires a build with CLIO_CORE_ENABLE_HDF5=ON.
• http:// and https://: a web download through a curl child process. Compiled in on POSIX builds only; a Globus web URL under https://app.globus.org is handed to the Globus assimilator instead. Credentials come from ~/.clio/cae_http_auth.ini.
• s3://bucket/key: Amazon S3 or an S3-compatible store such as MinIO. Requires CAE_ENABLE_S3=ON and the AWS SDK for C++. Credentials, region and endpoint come from the standard AWS environment of the runtime process: the access key variables or an AWS_PROFILE, AWS_DEFAULT_REGION, and S3_ENDPOINT or AWS_ENDPOINT_URL. Public buckets are read unsigned when AWS_NO_SIGN_REQUEST=1 is set or no credentials exist. The container keeps a pool of keep-alive connections across imports.
• gs:// or gcs://bucket/object: Google Cloud Storage through Application Default Credentials, with an optional GCS_ENDPOINT override. Requires CAE_ENABLE_GCS=ON and google-cloud-cpp.
• globus://: a Globus endpoint, with the access token taken from src_token or GLOBUS_ACCESS_TOKEN. Requires CAE_ENABLE_GLOBUS=ON.

The cloud and Globus options default to off and disable themselves with a warning when their SDK is not found. A source whose scheme was not compiled in fails the OMNI file with "No assimilator found".

The format field takes binary, hdf5, hdf5_vol or globus. hdf5_vol stages an HDF5 file in the cache layout of the CTE's HDF5 VOL connector, under the tag the connector itself would derive, so an application opening the file through the connector is served from the CTE tier from its first read. Its chunk size is read from CLIO_VOL_CHUNK_SIZE in the runtime's environment and must match the application's; a mismatch is a cache miss, not a wrong read.

The OMNI file

The test file for S3 ingestion is a complete example:

name: test_s3_assim

transfers:
  - src: s3://clio-cae-test/example_key
    dst: iowarp::test_s3_assim_tag
    format: binary
    depends_on: ""
    range_off: 0
    range_size: 0

Per-transfer keys the loader reads:

src, dst, format

Required. dst is iowarp::<tag>, or iowarp:: with no name for hdf5_vol.

depends_on

The exact src or dst of an earlier transfer. Transfers already run in order; this key documents and checks the dependency. A dependency on a later transfer, on itself, or on nothing in the list rejects the whole file before anything runs.

range_off, range_size

Byte range within the source. Zero size means the whole file.

src_token, dst_token

Authentication tokens. Environment variables are expanded.

dataset_filter

include_patterns and exclude_patterns lists of glob patterns for HDF5 datasets. * crosses /.

run, run_out

A script that transforms the source before ingest. It runs in the client, never in the runtime, with the source path or URL and run_out as its two arguments. The transfer then ingests run_out. If any run step fails, nothing is submitted. Relative paths resolve against the OMNI file's directory.

Any other key on a transfer is logged as a warning and ignored. The HDF5 example file in the tests carries name and description keys on its transfers; these are ignored by the loader, and its depends_on by transfer name does not match what the dependency check accepts.

Running it

1. Start the runtime with a configuration whose compose list includes the CTE and CAE pools:
   shell

   $ export CLIO_X=/path/to/clio_config.yaml
   $ clio_run start

2. Submit the OMNI file:
   shell

   $ clio_cae /path/to/transfers.yaml

The tool connects to the CAE pool at 400.0, loads the file, executes any run steps, and waits for ParseOmni. It exits non-zero when the assimilation result code or the task framework code is non-zero, and also when transfers were requested but zero assimilations were scheduled. Cloud credentials are read by the runtime process, not by the tool, so set them where the runtime runs.

Operations

• ParseOmni. Takes a serialized list of transfer contexts, checks dependencies, and runs each one through its assimilator in order. Returns the number of transfers scheduled, a result code and an error message. When replicas are aggregated, counts are summed and the first failure wins.
• ProcessHdf5Dataset. Processes one dataset of one HDF5 file into a tag prefix. It accepts a routing query so individual datasets can be sent to specific nodes.
• ExportData. Writes every blob in a tag to a file, in hdf5 or binary format, and reports bytes exported. A tag that does not exist is an error; an empty tag writes nothing.
• ImportData. The inverse of export for HDF5: reads the dataset whose in-file path equals the tag name and stores it in the key-value HDF5 form the key-value store uses.
• GetOrCreateTag, PutBlob, GetBlob, SemanticSearch. Interceptor passthroughs forwarded to the next pool.
• Monitor. The standard container status hook.

The C++ client exposes each of these as an asynchronous call on clio::cae::core::Client, plus CLIO_CAE_CLIENT_INIT for a process-wide singleton. The library links as clio::cae::core_client from the clio-core CMake package.

Behaviors worth knowing

• Transfers are sequential and fail fast. A failing transfer stops the file; later transfers do not run, and the count of scheduled transfers tells you how far it got.
• Ingest on fault. A transfer context can be marked on_fault. The CAE then ingests nothing, publishes the context in a registry tag so every node can see it, and registers itself as the destination tag's fault handler on every CTE container. A read of a blob the tag does not hold makes the source's assimilator fetch exactly that blob, which then serves the read and stays in the tag. When the tier is full, faults are served from the source without storing. This flag is not an OMNI key; it is set by programs that build contexts directly.
• Blob names and sizes. Binary sources are split into blobs of chunk_size bytes (1 MB when unset) named with blob_prefix and an index. Both fields are on the context, not the OMNI file.
• No secrets in OMNI files. Cloud credentials are resolved from the runtime's environment at assimilation time. Proxy settings follow the usual https_proxy variables, with loopback always direct.
• Scheduler hints. The module reports per-task cost estimates so the runtime can tell a 4 KiB put from a 1 MiB put when the CAE is the interceptor.

Current limits

• Node failure, recovery, migration and upgrade methods are not implemented for this module.
• HTTP sources need a POSIX build; S3, GCS, Globus and HDF5 are compile-time options that are off by default.
• Configuration parsing is best-effort: a malformed next_pool_id leaves the default in place without an error.
• ExportData and ImportData support HDF5 only when the module was built with HDF5; export also supports binary.
• The Destroy handler is not run on ordinary runtime shutdown; the container destructor does the cleanup instead.

How it is tested

Tests live under context-assimilation-engine/test/unit and are enabled with BUILD_TESTING. They cover the OMNI loader and dependency check, the transfer context and its serialization, binary and ranged assimilation, HDF5 import and export, the CTE interceptor round trip, string sources, the page-map and pagify helpers, S3 credentials and striped S3 reads, and HTTP authentication and fetching. Sample OMNI files sit next to them in binary_assim, hdf5_assim, s3_assim and gcs_assim. A Globus integration test with its own runtime configuration is in test/integration/globus_matsci. Sample datasets for the tests, including an HDF5 conversion of a CSV file, are in context-assimilation-engine/data.