Using the Context Exploration Engine.

Connecting to the runtime

Constructing a ContextInterface does nothing. The first method call initializes the runtime client, then the CAE client, which initializes the CTE client. If the runtime is not running the call logs an error and returns a failure value: 1 for bundle and destroy, an empty list for query and retrieve. A failed initialization is remembered for the life of the process, so start the runtime before the first call.

Two environment variables matter, and both are read when the client initializes, so set them before the first call. In Python, set them before the import to be safe.

• CLIO_CTE_POOL selects the CTE pool the client talks to, as major.minor. Keyword search is answered by the indexer module, so set it to 564.0 (the indexer) or 563.0 (the chain top, which forwards to the indexer). Without it the client binds to the bare core at 512.0 and semantic queries return nothing. Regex and temporal queries work either way.
• CLIO_WITH_RUNTIME is not needed. CEE initializes the client with the attach-only default. Setting it to 1 makes the process try to start a runtime itself and attach to the existing one if the port is taken.

import os
os.environ["CLIO_CTE_POOL"] = "563.0"

import clio_cee as cee
ctx = cee.ContextInterface()

On Windows, run the runtime and the client with CLIO_BIND_ADDR=127.0.0.1 and CLIO_SERVER_ADDR=127.0.0.1. If the runtime reads a non-default configuration, point CLIO_SERVER_CONF at the same file in the client's environment.

Import data with a bundle

context_bundle takes a list of AssimilationCtx and submits them as one ParseOmni task to the CAE pool at 400.0. It returns 0 on success or the task's result code; an empty list returns 0 with a warning. Transfers run in order inside the runtime.

rc = ctx.context_bundle([
    cee.AssimilationCtx(src="string::ice", dst="iowarp::climate_docs", format="binary",
                        src_data="Arctic sea ice extent fell to a record low."),
    cee.AssimilationCtx(src="string::co2", dst="iowarp::climate_docs", format="binary",
                        src_data="Atmospheric CO2 passed 420 ppm."),
    cee.AssimilationCtx(src="file::/data/run42/field.bin", dst="iowarp::run42",
                        format="binary"),
])
assert rc == 0

• A string::name source stores src_data verbatim as one blob called name. Pass it through the constructor's src_data argument or set the attribute.
• A file::/path source becomes blobs chunk_0, chunk_1, and so on, 1 MiB each. The path must be readable by the runtime process.
• hdf5::, http(s)://, s3://, gs:// and globus:// sources work when the runtime was built with their support. The binding does not expose the HDF5 dataset filters.
• range_off and range_size select a byte range; src_token is the HTTP bearer token or Globus token.

The CAE usage page describes every source scheme and the result codes a bundle can return.

Find blobs with a query

context_query(tag_re, blob_re, max_results=0, prompt="", time_begin=0, time_end=0) returns a list of blob names. Both patterns are full-string regular expressions, so write .*part.* for a substring. The arguments pick one of three modes, checked in this order:

1. Temporal. If time_begin or time_end is non-zero, the call is a CTE TemporalSearch: blobs whose last-modified time lies in the inclusive range, sorted ascending by time. 0 for either bound means no bound. max_results caps the list; 0 means unlimited.
2. Semantic. Otherwise, if prompt is non-empty, the call is a CTE SemanticSearch: BM25 keyword scoring of the prompt against the contents of every indexed blob that matches both patterns, best first. max_results is the top-k; 0 falls back to 10.
3. Regex. Otherwise the call is a CTE BlobQuery and returns every blob whose tag and blob names match. 0 means unlimited.

ctx.context_query("climate_docs", ".*")                                   # ['ice', 'co2'] in some order
ctx.context_query("climate_docs", ".*", max_results=1, prompt="sea ice")  # ['ice']
ctx.context_query("run42", "chunk_.*", time_begin=t0)                     # chunks modified at or after t0

Keyword search is exact-token BM25 over the text the indexer saw. There is no embedding model. Binary blobs match nothing useful; scope them out with the indexer's tag_re and blob_re keys, described under composing storage.

Read data with retrieve

context_retrieve resolves blobs with the same three modes, then reads their bytes. The result is a list with one str holding every retrieved blob, or an empty list when nothing matched or nothing could be read.

packed = ctx.context_retrieve("climate_docs", "ice")                  # ['Arctic sea ice extent fell to a record low.']
packed = ctx.context_retrieve("run42", "chunk_.*", max_context_size=64 * 1024 * 1024)
best   = ctx.context_retrieve("climate_docs", ".*", max_results=3, prompt="carbon dioxide")

• The buffer is the blobs' bytes concatenated in the order the query returned them. There are no headers, lengths or separators. To split it, query the names first and read sizes with the CTE client, or give each blob a fixed size.
• max_results defaults to 1024 here, not 0. The same 0 rules apply as for query.
• max_context_size, default 256 MiB, is allocated up front as one shared-memory buffer. Retrieval stops at the first blob that would not fit; blobs already copied are returned. Size it to what you will read.
• Blobs are fetched batch_size at a time (default 32) with AsyncGetBlob. Within a batch the tag lookup and size lookup are sequential.
• A blob of size 0 is skipped. A GetBlob that fails logs a warning but its reserved bytes stay in the buffer.
• Each blob's tag is resolved with GetOrCreateTag, so a tag name that no longer exists is created empty as a side effect.
• nanobind returns the buffer as str. Bytes that are not valid UTF-8 may not survive the conversion; for binary data read through the CTE client instead.

Delete contexts

context_destroy(["climate_docs", "run42"]) calls CTE DelTag for each name and waits. It returns 0 when every delete succeeded and 1 if any failed; an empty list returns 0. Deleting a tag deletes its blobs and, when the indexer is in the chain, its index entries. context_splice is declared but not implemented: it logs a warning and returns 1 without touching the runtime.

The same calls in C++

#include <clio_cee/api/context_interface.h>

iowarp::ContextInterface ctx;

clio::cae::core::AssimilationCtx item;
item.src = "file::/data/run42/output.bin";
item.dst = "iowarp::run42";
item.format = "binary";
if (ctx.ContextBundle({item}) != 0) { /* runtime not reachable or import failed */ }

auto names  = ctx.ContextQuery("run42", ".*");                            // regex listing
auto ranked = ctx.ContextQuery("run42", ".*", 5, "temperature gradient");  // BM25, top 5
auto packed = ctx.ContextRetrieve("run42", ".*", 1024, 64 * 1024 * 1024);  // one std::string
ctx.ContextDestroy({"run42"});

Link with find_package(clio_cee_api REQUIRED) and target_link_libraries(app PRIVATE clio::cee::api). In C++ the context has fields the Python binding lacks, including include_patterns, exclude_patterns, on_fault, chunk_size and blob_prefix.

How CEE sees Context FS

Context FS stores files as CTE tags and blobs, so anything written through a mount is visible to CEE without an import step:

• Each file is a tag named by its path inside the mount.
• File data is split into 1 MiB pages, one blob per page, named by page index (0, 1, ...). Each file also has a small attribute blob named ~i.
• The filesystem pool (560.0) writes through the chain top at 563.0, so pages pass the indexer and are keyword-searchable when they are text.

So ctx.context_query(".*notes/.*", "[0-9]+", prompt="sea ice") ranks the pages of files under notes/. The blob pattern [0-9]+ keeps the ~i blob out. The limitation is that context_query returns blob names only, which for files are page numbers with no file name attached. When the file name matters use clio_cte_core_ext directly: its BlobQuery returns (tag name, blob name) pairs and its search results carry the tag id. The query files from Python tutorial shows that path. context_retrieve does work on files and returns the matched pages concatenated.

Composing storage for search

CEE has no configuration of its own. What it can find and how fast it can read depends on the chain composed in the runtime's YAML. The default context-runtime/config/clio_default.yaml already composes a working layout; the points below are what to keep when writing your own.

• Keep the indexer in the chain, and route every writer through it. The default order is cache (563.0) to indexer (564.0) to replication (561.0) to core (512.0), with the optional compressor (562.0) between indexer and replication. The CAE core (400.0), the filesystem (560.0) and the stream pool (565.0) all set next_pool_id: "563.0", so imports and files are indexed. A writer pointed at 512.0 bypasses the index. Scope the index with the indexer's tag_re and blob_re keys so binary data is not tokenized, and set index_log_path so the index survives a restart.
• Add the summarizer for content labels. clio_cae_summarizer at 401.0 runs a prompt over each put that matches a rule and stores the reply as a sibling blob <blob>_label in the same tag. Give it next_pool_id: "563.0" so the label passes through the indexer, then point the CAE core's next_pool_id at "401.0". A BM25 prompt then matches the model's description as well as the raw text, and blob_re=".*_label" restricts a query to labels. The CAE usage page has the full block.
• Give retrieval a fast tier. context_retrieve is a GetBlob per blob. Blobs land on the highest-scoring tier with room, and the cache at the chain top keeps a node-local copy of blobs read from other nodes. Promote a working set with the CTE client's ReorganizeBlob before a large retrieve.
• Bound the result. max_context_size is allocated in shared memory in full, so size it to what you will read rather than leaving the 256 MiB default for small queries.

Excerpt of a chain that satisfies all of the above; the core, replication and cache entries are as in the default file:

compose:
  - mod_name: clio_cte_core
    pool_name: cte_main
    pool_query: local
    pool_id: "512.0"
    # storage tiers as in the default file

  - mod_name: clio_cte_replication
    pool_name: clio_cte_replication
    pool_query: local
    pool_id: "561.0"
    next_pool_id: "512.0"

  - mod_name: clio_cte_indexer
    pool_name: clio_cte_indexer
    pool_query: local
    pool_id: "564.0"
    next_pool_id: "561.0"
    index_log_path: "${CLIO_STORAGE_ROOT}/cte_indexer_index"
    tag_re: ".*"
    blob_re: ".*"

  - mod_name: clio_cte_cache
    pool_name: clio_cte_cache
    pool_query: local
    pool_id: "563.0"
    next_pool_id: "564.0"

  - mod_name: clio_cae_summarizer
    pool_name: clio_cae_summarizer
    pool_query: local
    pool_id: "401.0"
    next_pool_id: "563.0"
    label_endpoint: "http://127.0.0.1:11434"
    label_prompts:
      summarize: "Summarize the following text in one short sentence."
    label_matches:
      - tag_re: ".*"
        blob_re: ".*"
        model: "gemma3:1b"
        prompt: "summarize"

  - mod_name: clio_cae_core
    pool_name: cae_main
    pool_query: local
    pool_id: "400.0"
    next_pool_id: "401.0"

Each entry must come after the pool its next_pool_id names. Clients then set CLIO_CTE_POOL=563.0 or 564.0. Module pages document each key: indexer, cache, summarizer, CTE core.

Run the MCP server

context-exploration-engine/iowarp-cei-mcp/server.py is a Model Context Protocol server built on FastMCP. Despite the directory name it wraps the CTE Python module clio_cte_core_ext, not clio_cee, and the runtime client runs inside the server process. It searches for the module in the build's bin directory, CTE_PYTHON_PATH and PYTHONPATH. Install its requirements as on the install page, then register it with a client:

{
  "mcpServers": {
    "iowarp-cte": {
      "command": "python3",
      "args": ["/full/path/to/context-exploration-engine/iowarp-cei-mcp/server.py"],
      "env": {
        "PYTHONPATH": "/path/to/build/bin",
        "CLIO_SERVER_CONF": "/path/to/clio.yaml"
      }
    }
  }
}

Call initialize_cte_runtime first. With CLIO_SERVER_CONF set, it uses that configuration; without it, it writes a one-tier file-backed configuration under the temp directory on a free port between 9129 and 9199 and sets CLIO_SERVER_CONF to it. In both cases it calls clio_init(RuntimeMode.kClient, True), which tries to start a runtime in the server process and attaches to an existing one if the port is already bound, then initialize_cte, then registers a 512 MB file target at pool 700.0. The README warns that the query tools crash the server if called before initialization. Every tool returns a JSON string. The tool list is on the API page. test_mcp_tools.py exercises every tool over stdio.

Limits

• context_splice is unimplemented.
• context_query returns blob names without tag names; use clio_cte_core_ext.Client.BlobQuery when the tag matters.
• The retrieve buffer has no framing and is returned as a Python str.
• Keyword search is exact-token BM25 over ASCII text; there is no embedding model. See the indexer page.
• The Python AssimilationCtx exposes nine fields; dataset filters and on-demand ingest are C++ only.
• The MCP server exposes CTE operations, not the clio_cee methods, and has no semantic or temporal search tool.

Tests

context-exploration-engine/api/test holds Catch2 tests for bundle, query (including the three modes in test_context_query_modes.cc) and destroy, plus the Python tests test_context_interface.py, test_context_interface_enhanced.py and test_context_retrieve_roundtrip.py. All need a running runtime; the directory's clio_config.yaml is the configuration they use. api/demo/demo.py is a self-contained walk through bundle, query, retrieve and destroy on a 1 MB temporary file.