The block-device module (bdev)

What it does

The module exposes a small task interface. Each operation is a task routed to one container of the pool; on a single node the container is the local one.

• AllocateBlocks takes a byte count and returns a list of blocks, each an offset, a size and a size class. One request may come back as several extents.
• Write and Read take a block list and a buffer. Writes report bytes written and an I/O error category; reads report bytes read.
• FreeBlocks returns blocks to the allocator.
• GetStats returns the configured performance figures, the configured capacity, the remaining allocatable bytes and a predicted device lifetime in days (999999 means healthy).
• Sync makes written data and allocator state durable. It runs fdatasync on a file device and is a no-op on memory devices.
• SetLifespan overrides the lifetime figure, which the CTE's placement filter reads. FlushAllocLog flushes and compacts the allocator log. Update publishes GPU memory pointers for the hbm and pinned types. Monitor answers dashboard queries.

The pool keeps no per-object metadata. Recording which blocks belong to which blob is the CTE's job; see the CTE core. When the caller is in the same process as the runtime, the client runs allocate, free, read and write inline instead of sending a task.

Where it sits

A bdev pool is a leaf. It calls nothing below it, and it is called by the CTE core (one bdev per storage tier), by safe-bdev (one bdev per array member), and by tests and benchmarks directly. In a compose file it must appear before any pool that refers to it. The runtime design page covers the allocator and file-tier internals in more depth.

Composing a bdev directly

The pool name doubles as the device location: a file path for the file type, a bucket (with optional prefix) for s3 and gcs, and an arbitrary unique name for memory types. The default configuration composes one RAM device this way:

compose:
  - mod_name: clio_bdev
    pool_name: "ram::chi_default_bdev"
    pool_query: local
    pool_id: "301.0"
    bdev_type: ram          # file, ram, hbm, pinned, noop, s3, gcs
    capacity: "0g"          # 0 on a ram device = 80% of system DRAM

  - mod_name: clio_bdev
    pool_name: "/mnt/nvme/chi_bdev"
    pool_query: local
    pool_id: "302.0"
    bdev_type: file
    capacity: "100GB"
    alloc_log: "/mnt/nvme/chi_bdev.alloc_log"

Configuration keys

bdev_type

One of file (default), ram, hbm, pinned, noop, s3, gcs. Setting the type also sets default performance figures ordered so that hbm outranks ram, which outranks pinned, which outranks everything else.

capacity

A size string such as 2GB. Default 0. For a RAM device 0 means 80% of physical DRAM; for s3 and gcs 0 means 1 TiB of address space; for a file device it is the maximum the backing file may grow to.

alloc

Page commit policy for ram and pinned devices: auto (default, commit everything at creation when a capacity was given), eager, or lazy. An unknown value falls back to auto with a warning.

io_depth

Asynchronous I/O queue depth per worker for file devices. Default 32.

alignment

Allocation granularity in bytes. Default 4096. Memory devices lower a default or zero value to 512 because sub-4 KiB blobs otherwise pin a full 4 KiB block.

growth_unit

File devices only. How far the backing file grows, and reserves disk, at a time. Default 1GB.

populate_unit

RAM devices only. How many bytes of the shared mapping are pre-faulted when an allocation crosses the populated watermark. Default 64MB; 0 disables population.

alloc_log

Path of the allocator write-ahead log. A file device logs to <pool_name>.alloc_log when this is unset; setting it moves the log and makes even clio_run start --fresh recover from it.

persistence_level

volatile, temporary or long_term. Only an explicit volatile disables the allocator log on a file device.

perf_metrics

A map of read_bandwidth_mbps, write_bandwidth_mbps, read_latency_us, write_latency_us and iops. These override the type defaults and are what the placement engine ranks tiers by; they are not measured.

How a CTE tier becomes a bdev

You rarely compose a bdev by hand. Each entry under the CTE's storage list creates one bdev pool per node in the target neighborhood. The keys are the same ideas under slightly different names:

  - mod_name: clio_cte_core
    pool_name: clio_cte_core
    pool_id: "512.0"
    storage:
      - path: "ram::cte_ram_tier1"      # memory tiers use ram::<name>
        bdev_type: "ram"
        capacity_limit: "0g"
        score: 1.0
      - path: "${CLIO_STORAGE_ROOT}/cte_disk_tier.dat"
        bdev_type: "file"
        capacity_limit: "10GB"
        growth_unit: "1GB"
        score: 0.2
        persistence_level: "temporary"

• path becomes the bdev pool name, with _node<N> appended for the node the target is registered on. For an S3 or GCS tier that suffix lands on the key prefix, so use a bucket/prefix path rather than a bare bucket.
• capacity_limit maps to the bdev's capacity. A RAM tier with 0 gets the same 80% of DRAM default.
• growth_unit and persistence_level pass through unchanged. score belongs to the CTE's placement engine, not to the bdev.
• Memory tiers register on their own node only; file and object-store tiers span the configured neighborhood.
• A storage entry with existing_pool_id and existing_pool_module attaches to a pool composed earlier (for example a safe-bdev) instead of creating a bdev.

Transports

The runtime picks a transport from the type at pool creation. All of them share the same allocator and the same task interface.

ram

Host memory mapped as one sparse shared-memory segment, named from the runtime's process ID and the pool ID so client processes can attach it and read blob payloads without a round trip. Pages are faulted in one populate_unit at a time as allocations advance, so committed memory tracks the allocation high-water mark rather than the declared capacity. The environment variable CLIO_PREFAULT pre-faults a prefix at creation instead (0 for the whole mapping, or a size string). Contents are lost on restart.

pinned and hbm

pinned allocates page-locked host memory through the GPU API so device transfers can run asynchronously; it is not shared-memory backed. hbm allocates one device buffer covering the whole capacity. GPU copies are enqueued on a stream and the worker yields until they finish. In a GPU-enabled build a failed device allocation for hbm is fatal; it does not fall back to host memory. In a build without GPU support both types are ordinary host memory and hbm logs a warning saying so. Both are volatile.

file

A backing file driven by per-worker asynchronous I/O contexts. The backend is chosen at build time in the order NIXL, io_uring, libaio, then the platform default (POSIX AIO, or I/O completion ports on Windows). If the preferred backend cannot open the file at run time, for example io_uring under a container seccomp profile, the transport falls back to POSIX AIO. Behaviors worth knowing:

• The file starts at one growth unit and is extended on demand. Where the filesystem supports it, each step reserves real blocks so a full disk fails at allocation time. A failed growth is remembered for two seconds so requests past that point fail fast.
• Remaining capacity is the allocator's free space capped by what the host filesystem can still give the file, sampled at most once a second.
• Writes of at least 64 KiB whose offset and length are aligned are bounced through an aligned buffer so they can reach the direct-I/O file descriptor.
• The transport records the identity of the file it opened. If the path later names a different file, reopen attempts fail instead of silently creating an empty file.
• Allocations and frees are appended to the allocator log before blocks are handed out or reused. The log is flushed on every call and fsynced every 50 ms by a thread owned by the transport. On restart the live extents are replayed so no live byte is handed out twice.
• A test-only fault marker, <file>.fail, makes every read, write and sync fail with a device fault while it exists.

s3

Compiled in when CLIO_ENABLE_AMAZON_DRIVE is defined. Each block is one object named [prefix/]block_<offset>; a never-written block reads back as zeros when the GET returns 404. The pool name is parsed as bucket, bucket/prefix or s3://bucket/prefix. Requests are signed with AWS Signature Version 4 by a header-only REST client over Poco; no AWS SDK is linked. One keep-alive connection is kept per worker. The bucket is created if it does not exist and S3_ALLOW_BUCKET_CREATE is 1. Configuration comes from the environment of the process that starts the runtime:

• AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are required; pool creation fails without them. AWS_SESSION_TOKEN is optional.
• AWS_DEFAULT_REGION, default us-east-1.
• S3_ENDPOINT overrides the endpoint and switches to path-style addressing, which is how MinIO and the test stub are reached. Unset means real AWS with the bucket in the hostname.

gcs

Compiled in when CLIO_ENABLE_GOOGLE_CLOUD is defined. Same object layout and zero-fill convention as s3, with the pool name parsed as bucket, bucket/prefix or gcs://bucket/prefix. Authentication is an OAuth2 bearer token; a fresh HTTPS session is opened per call.

• GCS_ACCESS_TOKEN is required, typically the output of gcloud auth print-access-token. The service-account JWT flow is not implemented.
• GCS_ENDPOINT, default https://storage.googleapis.com.
• GCS_PROJECT_ID, used only when creating the bucket; default clio-prototype.

noop

No transport is created. Allocate, free, write and read return success without touching any storage. It exists to measure the runtime's own overhead.

The allocator

Every transport uses the same allocator: a bump pointer over the capacity plus segregated free lists from 512 B to 1 MB, one set per worker, with stealing between workers. Blocks are returned with a size class so frees go back to the right list. When the heap is exhausted a large request is assembled from several free blocks, so a single allocation can return more than one extent and callers must handle block lists. Nothing is compacted or coalesced, so long churn fragments free space into blocks of 1 MB or less. File devices run the blocking part of growth off the runtime worker so a multi-second reservation does not stall other tasks.

Capacity and health

GetStats reports the configured capacity, not the physical device under it. Remaining space is the allocator's view, further capped on file devices by host free space. Several file devices on one disk each see the whole free space, so the figure is an upper bound. The lifetime figure defaults to healthy and can be set with SetLifespan; a Monitor query can also fetch a prediction from the optional failure-prediction service under hw_failure_monitor/, which is outside the normal data path.

Limits

• Allocator state for s3 and gcs devices is held in memory only; their objects persist but the allocator restarts from zero after a restart.
• Within one task, blocks are written one at a time. Concurrency comes from many tasks in flight.
• An hbm or pinned device in a build without GPU support is host memory.
• The GCS transport opens a new connection per request; the S3 transport reuses one per worker.

How it is tested

Tests live in context-runtime/modules/bdev/test.

• cr_all_bdev_tests runs test_bdev_chimod.cc: creation, 4 KiB allocation, lazy file growth, the volatile log skip, a deleted backing file, host-disk capping, near-full-disk growth, RAM and pinned I/O, bounds checks, parallel I/O, allocator-log recovery and compaction, and the coalescing allocator cases.
• cr_bdev_explicit_backend_shm, _tcp and _ipc run the file device over each client transport, and cr_bdev_force_net routes every task through the network path on loopback.
• cr_bdev_sigv4 checks the signer against known-answer vectors generated by gen_sigv4_vectors.py. cr_bdev_s3_rest and cr_bdev_s3_rest_resume drive the REST client against s3_stub_server.py, an in-memory S3 stand-in that verifies signatures, so no credentials or network are needed.
• cr_bdev_prediction_integration (Linux only) starts the Python model server and runs one prediction round trip; it skips when the Python dependencies or models are absent.