CLIO Core

Docs · Context Exploration Engine

Context Exploration Engine API reference.

Signatures and defaults below are taken from context-exploration-engine/api/include/clio_cee/api/context_interface.h and api/src/python_bindings.cc. The Python names are snake_case versions of the C++ methods with the same defaults.

Header
clio_cee/api/context_interface.h
Module
clio_cee (nanobind)
CMake
find_package(clio_cee_api), clio::cee::api
MCP
iowarp-cei-mcp/server.py

ContextInterface (C++)

cpp
#include <clio_cee/api/context_interface.h>

namespace iowarp {
class ContextInterface {
 public:
  ContextInterface();      // does not touch the runtime
  ~ContextInterface();

  int ContextBundle(const std::vector<clio::cae::core::AssimilationCtx> &bundle);

  std::vector<std::string> ContextQuery(const std::string &tag_re,
                                        const std::string &blob_re,
                                        unsigned int max_results = 0,
                                        const std::string &prompt = "",
                                        uint64_t time_begin = 0,
                                        uint64_t time_end = 0);

  std::vector<std::string> ContextRetrieve(const std::string &tag_re,
                                           const std::string &blob_re,
                                           unsigned int max_results = 1024,
                                           size_t max_context_size = 256 * 1024 * 1024,
                                           unsigned int batch_size = 32,
                                           const std::string &prompt = "",
                                           uint64_t time_begin = 0,
                                           uint64_t time_end = 0);

  int ContextSplice(const std::string &new_ctx, const std::string &tag_re,
                    const std::string &blob_re);

  int ContextDestroy(const std::vector<std::string> &context_names);
};
}

Initialization

Every public method first calls a private EnsureInitialized(). On the first call it runs clio::run::CLIO_INIT(RuntimeMode::kClient, false), then CLIO_CAE_CLIENT_INIT(), which initializes the CTE client, then checks that the IPC manager exists. If CLIO_INIT fails, the failure is cached in a static flag for the rest of the process and every later call fails immediately. On failure the methods return 1 (bundle, destroy) or an empty vector (query, retrieve) and log an error.

ContextBundle(bundle)

  • bundle: vector of AssimilationCtx. Empty: logs a warning, returns 0.
  • Creates a clio::cae::core::Client on kCaePoolId (400.0), calls AsyncParseOmni(bundle) and waits.
  • Returns: 0 on success; the task's result_code_ when non-zero; 1 on an exception. The CAE result codes are on the CAE API page.

ContextQuery(tag_re, blob_re, max_results, prompt, time_begin, time_end)

  • tag_re, blob_re: full-string regular expressions over tag and blob names.
  • max_results (0): result cap. 0 means unlimited for regex and temporal, and 10 for semantic.
  • prompt (""): BM25 query text. Empty disables semantic mode.
  • time_begin, time_end (0): inclusive bounds on last-modified time; 0 means no bound. Any non-zero value selects temporal mode.
  • Mode selection, in order: temporal if either time bound is non-zero; else semantic if prompt is non-empty; else regex.
  • Calls: AsyncTemporalSearch(tag_re, blob_re, time_begin, time_end, max_results, PoolQuery::Broadcast()), AsyncSemanticSearch(tag_re, blob_re, prompt, k, PoolQuery::Broadcast()) with k = max_results > 0 ? max_results : 10, or AsyncBlobQuery(tag_re, blob_re, max_results, PoolQuery::Broadcast()).
  • Returns: blob names only. Temporal results are sorted ascending by timestamp; semantic results are ordered by descending score; regex results are in the order the runtime returned them. Empty vector on failure.

ContextRetrieve(tag_re, blob_re, max_results, max_context_size, batch_size, prompt, time_begin, time_end)

  • Same mode selection and CTE calls as query, but keeps (tag name, blob name) pairs.
  • max_results (1024): same 0 rules as query.
  • max_context_size (268435456, 256 MiB): size of one buffer allocated with CLIO_IPC->AllocateBuffer before any read. The loop stops at the first blob that would exceed it.
  • batch_size (32): number of AsyncGetBlob futures in flight per batch. For each blob in a batch the code waits on AsyncGetOrCreateTag(tag_name) and AsyncGetBlobSize(tag_id, blob_name) sequentially, skips the blob if the tag id is null or the size is 0, then issues AsyncGetBlob(tag_id, blob_name, 0, size, 0, ptr) into the next free offset of the buffer. All futures of the batch are then waited; a non-zero return code logs a warning and leaves the reserved bytes in place.
  • Returns: a vector with one std::string holding the bytes written to the buffer, back to back and in query order, or an empty vector when no blob matched or no bytes were copied. The buffer is freed before return.

ContextSplice(new_ctx, tag_re, blob_re)

Not implemented. Logs a warning and returns 1 without initializing anything.

ContextDestroy(context_names)

  • context_names: tag names. Empty: logs a warning, returns 0.
  • For each name, waits on AsyncDelTag(name).
  • Returns: 0 if every delete returned 0, else 1.

clio_cee (Python)

Built with nanobind from api/src/python_bindings.cc. The module docstring is IOWarp Context Exploration Engine API - Python Bindings. Lists map to Python lists and strings to str.

python
import clio_cee as cee

class cee.ContextInterface:
    def __init__(self)
    def context_bundle(self, bundle: list[AssimilationCtx]) -> int
    def context_query(self, tag_re: str, blob_re: str, max_results: int = 0,
                      prompt: str = "", time_begin: int = 0, time_end: int = 0) -> list[str]
    def context_retrieve(self, tag_re: str, blob_re: str, max_results: int = 1024,
                         max_context_size: int = 268435456, batch_size: int = 32,
                         prompt: str = "", time_begin: int = 0, time_end: int = 0) -> list[str]
    def context_splice(self, new_ctx: str, tag_re: str, blob_re: str) -> int
    def context_destroy(self, context_names: list[str]) -> int

Arguments can be passed by keyword. Behavior and return values are those of the C++ methods above. context_retrieve returns a list of zero or one str; nanobind converts the packed bytes to str, so binary payloads that are not valid UTF-8 are not safely recoverable through this call.

AssimilationCtx

The CAE's clio::cae::core::AssimilationCtx, bound as cee.AssimilationCtx. Three constructors:

python
cee.AssimilationCtx()
cee.AssimilationCtx(src, dst, format, depends_on="", range_off=0, range_size=0,
                    src_token="", dst_token="")
cee.AssimilationCtx(src, dst, format, src_data, depends_on="", range_off=0, range_size=0,
                    src_token="", dst_token="")

The third form is selected by passing a fourth positional argument or the src_data keyword. Read-write attributes:

AttributeTypeDefaultMeaning
srcstrrequiredSource URL, for example file::/path/to/file or string::<blob_name>.
dststrrequiredDestination URL, iowarp::<tag_name>.
formatstrrequiredData format: binary, hdf5, hdf5_vol or globus.
depends_onstr""The src or dst of an earlier item in the same bundle.
range_offint0Byte offset in the source.
range_sizeint0Bytes to read; 0 means to the end.
src_tokenstr""Source credential (HTTP bearer token, Globus access token).
dst_tokenstr""Destination credential.
src_datastr""Inline payload for a string:: source, stored verbatim as one blob named by the part after string::.

repr() gives <AssimilationCtx src='...' dst='...' format='...'>. The C++ fields s3_region, s3_profile, include_patterns, exclude_patterns, on_fault, chunk_size and blob_prefix are not bound; see the CAE API page for them.

Environment variables

  • CLIO_CTE_POOL (major.minor): binds the CTE client to an interposing pool. Set 563.0 or 564.0 for semantic queries to reach the indexer. A malformed value is ignored with an error in the log.
  • CLIO_WITH_RUNTIME: 1 or true makes the client try to start a runtime in-process; otherwise it only attaches. CEE passes attach-only as the default.
  • CLIO_SERVER_CONF: the runtime configuration file the client reads. Must describe the same runtime the server started.
  • CLIO_BIND_ADDR, CLIO_SERVER_ADDR: bind and connect addresses; use 127.0.0.1 for both on Windows.
  • Library target clio_cee_api (shared, SOVERSION 1, C++20). Alias clio::cee::api. Links clio::run::cxx, clio_cte_core_client and clio_cae_core_client publicly.
  • Installed CMake package clio_cee_api under lib/cmake/clio_cee_api, version 1.0.0, compatibility AnyNewerVersion. Its config finds clio_run, clio_cte_core and clio_cae_core.
  • Python module target clio_cee, built only with CLIO_CORE_ENABLE_PYTHON=ON, installed to PYTHON_SITE_PACKAGES (and to iowarp_core/ext in the pip wheel).
  • Tests are added under api/test when BUILD_TESTING is on.

MCP tools

Defined in context-exploration-engine/iowarp-cei-mcp/server.py with FastMCP under the server name IOWarp Context Transfer Engine (CTE) MCP Server. Every tool returns a JSON string. The tools wrap clio_cte_core_ext; there is no tool for clio_cee methods and no semantic or temporal search.

Safe before initialization

  • get_client_status(): available, initialized, pool_query_available, message, plus note or warning when something is missing.
  • get_cte_types(): which of TagId, BlobId, CteTelemetry and PoolQuery are bound, and the CteOp values.
  • initialize_cte_runtime(): sets CLIO_REPO_PATH and LD_LIBRARY_PATH to the module's directory, uses CLIO_SERVER_CONF or generates a one-tier file configuration (port 9129 to 9199, num_threads: 2, a 512MB file tier under the temp directory), calls clio_init(RuntimeMode.kClient, True), waits 0.5 s, calls initialize_cte(config_path, PoolQuery.Dynamic()), and registers a 512 MB file target at pool id 700.0. Returns success, runtime_init, client_init, cte_init, messages, and target_registered and target_path when it got that far. Progress is also written to cte_init_log_<pid>.json in the temp directory.

Data operations

  • put_blob(tag_name, blob_name, data, offset=0): Tag(tag_name).PutBlob(blob_name, data.encode(), offset). Returns data_size and success.
  • get_blob(tag_name, blob_name, size=0, offset=0): size 0 means the blob's size; a blob of size 0 is reported as not found. Returns data when the bytes are UTF-8, otherwise data_base64.
  • get_blob_size(tag_name, blob_name): size and size_bytes.
  • list_blobs_in_tag(tag_name): blobs and count.
  • delete_blob(tag_name, blob_name): Client.DelBlob(tag_id, blob_name); success.

Query and placement

  • tag_query(tag_regex, max_tags=0): Client.TagQuery with PoolQuery.Dynamic(); tags and count.
  • blob_query(tag_regex, blob_regex, max_blobs=0): Client.BlobQuery; blobs as (tag name, blob name) pairs and count.
  • poll_telemetry_log(minimum_logical_time=0): entries with op, off, size, tag_id (major, minor, is_null), logical_time, and mod_time and read_time when serializable.
  • reorganize_blob(tag_id_major, tag_id_minor, blob_name, new_score): Client.ReorganizeBlob; result_code and success.

The query and placement tools return an error object when the client is not initialized, but the README documents that calling them before initialize_cte_runtime can still crash the server process.