CLIO Core

Docs · Context Assimilation Engine

Using the Context Assimilation Engine.

Most imports are an OMNI file and one command. This page walks through the file, the tool, each source scheme, and the two ways to describe data as it is written. Every key and default is listed on the API reference page.

Submit
clio_cae transfers.yaml
From code
clio_cee.context_bundle or AsyncParseOmni
Needs
A running runtime

Import a file

An OMNI file is a YAML document with a transfers list. Each transfer names a source, a destination tag and a format. Write one, start the runtime, and submit it:

yaml · transfers.yaml
name: import_results

transfers:
  - src: file::/data/run42/field.bin
    dst: iowarp::run42_field
    format: binary
shell
$ clio_run start &
$ clio_cae transfers.yaml

The file becomes the tag run42_field holding blobs chunk_0, chunk_1, and so on, 1 MiB each. Top-level keys other than transfers, such as name, are ignored. A per-transfer key the loader does not know is logged as a warning and ignored.

The tool initializes a client of the running runtime, loads the file, executes any run steps locally, then submits the whole list as one ParseOmni task and waits. Transfers run in order inside the runtime, each finishing before the next starts. The tool exits 1 when:

  • the runtime is not running, or the OMNI file cannot be read or is missing transfers, src, dst or format;
  • a run step fails, in which case nothing is submitted;
  • the assimilation result code or the task framework code is non-zero, for example a depends_on that names a later transfer, or a source scheme that was not compiled in;
  • transfers were requested but zero were scheduled.

Success means the transfers were processed by the CAE pool. Dataset-level problems inside an assimilator, such as a full tier (PutBlob failed), are logged by the runtime, not the tool. Confirm the blobs landed with cte_search 'run42_field.*', or by listing the tag from the CTE Python client.

Sources and formats

The scheme of src, everything before :// or ::, selects the assimilator. The format key is required on every transfer; the values the code acts on are binary, hdf5, hdf5_vol and globus.

  • file::/path. Reads a local file in chunks and stores blobs chunk_0, chunk_1, and so on. range_off and range_size select a byte range. The path must be visible to the runtime process, not only to the machine running clio_cae.
  • string::name. Stores the bytes of src_data verbatim as one blob named name. OMNI files have no key for src_data, so this scheme is for programs that build contexts in code, such as clio_cee.
  • hdf5::/path.h5. Discovers every dataset in the file. With format: hdf5, each dataset becomes its own tag named <dst tag>/<dataset path>, and large datasets are split into several blobs. dataset_filter selects datasets with include and exclude glob patterns. With format: hdf5_vol the file is staged in the layout the CTE's HDF5 VOL connector expects, so an application opening the file through the connector is served from the CTE from its first read. Needs CLIO_CORE_ENABLE_HDF5=ON.
  • http:// and https://. Downloads through a curl child process, with src_token as the bearer token when set. POSIX builds only. A URL under https://app.globus.org is handed to the Globus assimilator instead.
  • s3://bucket/key. Amazon S3 or an S3-compatible store. Reads run in the runtime over Poco with SigV4 signing; the standalone cae_s3_tool helper links the AWS SDK and is used for seeding and benchmarks. Needs CAE_ENABLE_S3=ON.
  • gs://bucket/object or gcs://bucket/object. Google Cloud Storage through Application Default Credentials. GCS_ENDPOINT overrides the endpoint. Needs CAE_ENABLE_GCS=ON.
  • globus://. A Globus endpoint, with the access token from src_token or the GLOBUS_ACCESS_TOKEN environment variable. Needs CAE_ENABLE_GLOBUS=ON.

A scheme whose support was not compiled in logs an error naming the CMake option to turn on, the factory returns no assimilator, and ParseOmni fails with result code -2.

Several transfers and dependencies

Transfers run in the order listed. depends_on names an earlier transfer by its exact src or dst URL. Because of the ordering it changes nothing at run time; it is a check. A dependency that names a later transfer, the transfer itself, or nothing in the list fails the whole file before anything is scheduled, with result code -3.

yaml
transfers:
  - src: file::/data/run42/mesh.bin
    dst: iowarp::run42_mesh
    format: binary
  - src: file::/data/run42/field.bin
    dst: iowarp::run42_field
    format: binary
    depends_on: iowarp::run42_mesh
  - src: hdf5::/data/run42/diagnostics.h5
    dst: iowarp::run42_diag
    format: hdf5
    dataset_filter:
      include_patterns: ["/temperature/*"]
      exclude_patterns: ["/temperature/raw*"]

Glob patterns support *, ? and [], and * crosses /. If the include patterns match no dataset the transfer fails with result code -9 and the tool prints the likely cause.

Preprocess a source with a run step

A transfer may carry run and run_out. The script runs in the clio_cae process, as the user running the tool, before anything is submitted. It never runs inside the runtime. It is called with two arguments: the source, which is the local path for file:: and hdf5:: sources and the URL unchanged for other schemes, and the output path. If it exits 0 and run_out exists as a regular file, the transfer ingests run_out instead of the original source: as hdf5:: when the format starts with hdf5, as file:: otherwise. Later depends_on entries that named the old source are rewritten. Any failure stops the whole ingest.

yaml
transfers:
  - src: file::/data/A46_xx.csv
    dst: iowarp::a46_parquet
    format: binary
    run: ./to_parquet.sh           # called as: to_parquet.sh &lt;src&gt; &lt;run_out&gt;
    run_out: /scratch/a46.parquet  # the file the script must write

Both paths are environment-expanded, and relative paths resolve against the OMNI file's directory. run_out without run, or the reverse, is an error.

Import from the cloud

Credentials are read by the process that performs the read. For s3:// and gs:// that is the runtime, so set the AWS or Google environment where clio_run starts. The exceptions are the two S3 hints the client can forward: a program that builds contexts in code can set s3_region and s3_profile on the context, and they become AWS_DEFAULT_REGION and AWS_PROFILE for the S3 helper. OMNI files have no key for them.

yaml
transfers:
  - src: s3://my-bucket/reference/baseline.bin
    dst: iowarp::baseline
    format: binary
  - src: gs://my-bucket/reference/baseline-2.bin
    dst: iowarp::baseline_2
    format: binary
  - src: https://data.example.org/run42.tar
    dst: iowarp::run42_tar
    format: binary
    src_token: ${DATA_TOKEN}

src_token and dst_token are environment-expanded when the file is loaded, so a token can stay out of the file. The cloud storage tutorial walks through S3 and Google Cloud imports end to end, and the environment variable list covers the S3 and HTTP tuning knobs.

Import from a program

The Context Exploration Engine wraps the CAE for Python and C++: context_bundle takes a list of AssimilationCtx and submits one ParseOmni task on pool 400.0.

python
import clio_cee as cee
ctx = cee.ContextInterface()
ctx.context_bundle([
    cee.AssimilationCtx(src="file::/data/run42/field.bin", dst="iowarp::run42_field", format="binary"),
    cee.AssimilationCtx(src="string::notes", dst="iowarp::run42_field", format="binary",
                        src_data="field.bin is the final pressure field"),
])

In C++ the CAE client is clio::cae::core::Client; the API page lists its methods. The C++ context has fields the Python binding does not expose, including include_patterns, exclude_patterns, on_fault, chunk_size and blob_prefix.

Export back to a file

The CAE core also has ExportData and ImportData tasks, reachable from the C++ client. AsyncExportData(tag_name, output_path, format) writes every blob of a tag to one file, with format "hdf5" or "binary". AsyncImportData(tag_name, input_path, "hdf5") reads a dataset from an HDF5 file into a tag. Both HDF5 directions need CLIO_CORE_ENABLE_HDF5=ON; without it the task returns -1 with the message HDF5 support not compiled in. There is no command-line front end for them.

Describing content in the I/O path

Because the CAE core and the summarizer speak the CTE's task interface, they can be placed where ordinary puts and gets pass through them, and describe data as it is written rather than in a later pass.

  • Indexer. The default CAE entry forwards to the chain top at 563.0, which runs through the indexer at 564.0. Every imported blob is therefore tokenized and answers SemanticSearch. Point next_pool_id at 512.0 to skip that.
  • Summarizer labels. clio_cae_summarizer forwards every verb and, after a successful PutBlob whose tag and blob names match a rule in label_matches, sends the bytes to the model named by the rule with the prompt named by the rule, and stores the reply as {blob_name}_label in the same tag with the same placement score. The label is an ordinary blob. Matching uses regex search, not full match, so .*\.txt$ matches names ending in .txt. An invalid regex disables that rule with a warning. Inference is synchronous on the handling worker, so scope the rules tightly.
  • CAE as interceptor. Compose clio_cae_core at 512.0 with next_pool_id pointing at the real core, and a CTE client or the filesystem runs through it unchanged. Only GetOrCreateTag, PutBlob, GetBlob and SemanticSearch are forwarded, so this layout is for dedicated configurations, not the default file.

To enable the summarizer in the default configuration, uncomment its block, put it after the cache entry, and point the CAE core's next_pool_id at it:

yaml
  - mod_name: clio_cae_summarizer
    pool_name: clio_cae_summarizer
    pool_query: local
    pool_id: "401.0"
    next_pool_id: "563.0"
    label_endpoint: "http://127.0.0.1:11434"
    label_api: ollama
    label_prompts:
      summarize: "Summarize the following text in one short sentence."
    label_matches:
      - tag_re: ".*\\.txt$"
        blob_re: ".*"
        model: "gemma3:1b"
        prompt: "summarize"
        context_length: 4096
        num_predict: 0

  - mod_name: clio_cae_core
    pool_name: cae_main
    pool_query: local
    pool_id: "400.0"
    next_pool_id: "401.0"

With label_api: openai the summarizer posts to {label_endpoint}/chat/completions and sends a bearer token read from label_api_key_file or the variable named by label_api_key_env; both are re-read on every request. The summarizer module page covers chunking and the request bodies, and the operators tutorial shows the compose change step by step.

Where to go next