Safe block device (safe-bdev)

What it does

The module reuses the bdev task types for its data plane, so a CTE or any other bdev client can use it without change: AllocateBlocks, FreeBlocks, Write, Read, GetStats and Sync carry the same method IDs as in bdev. On top of those it adds management operations:

AddBdev appends a member as a data column or, with the parity flag, as a parity column.
RemoveBdev marks a member faulty or cleanly removed. Nothing is copied; reads reconstruct around it.
RecoverBdev seats a replacement pool in a failed member's position and rebuilds that member's chunks onto it.
BuildParity re-encodes stripes whose parity is stale. It is registered as a periodic task every 50 ms.
FlushAllocLog flushes and compacts the allocator log, also periodic at 50 ms.

Use it when a CTE tier must outlive a disk. For a tier that only needs to survive a node, see replication and the durability design.

Where it sits

The array is a pool composed after its member bdev pools and before the CTE that uses it. The CTE attaches to it as an existing pool rather than creating a bdev. The array calls each member through an ordinary bdev client. With distributed: true members live on the nodes their node_id names, the array runs on the first member's node, and management tasks sent to any other node's container return an error code telling the caller to route to the home node.

Composing an array

This is the integration-test configuration: three file members as data, with a parity member added at startup through AddBdev. The order matters, because each member must exist before the array that names it.

yaml

compose:
  - mod_name: clio_bdev
    pool_name: "/mnt/bdev0.dat"
    pool_query: local
    pool_id: "301.0"
    bdev_type: file
    capacity: "256MB"
    alloc_log: "/mnt/bdev0.alog"
  # ... bdev1 at 302.0 and bdev2 at 303.0 the same way ...

  - mod_name: clio_safe_bdev
    pool_name: "safe0"
    pool_query: local
    pool_id: "350.0"
    max_failures: 1
    alloc_log: "/mnt/safe0.alog"
    members:
      - pool_name: "/mnt/bdev0.dat"
        node_id: 0
        pool_id_major: 301
        pool_id_minor: 0
      - pool_name: "/mnt/bdev1.dat"
        node_id: 0
        pool_id_major: 302
        pool_id_minor: 0
      - pool_name: "/mnt/bdev2.dat"
        node_id: 0
        pool_id_major: 303
        pool_id_minor: 0

  - mod_name: clio_cte_core
    pool_name: cte_main
    pool_query: local
    pool_id: "512.0"
    storage:
      - path: "safe0"
        existing_pool_id: "350.0"
        existing_pool_module: "clio_safe_bdev"
        score: 1.0

A member entry may carry parity: true to seat it as a parity column at creation; the disk-failure tests compose four data and two parity members this way. Creation refuses more parity members than max_failures.

Configuration keys

max_failures: How many simultaneous member failures the array must tolerate, and the most parity members it will accept. Default 1.
members: A sequence of pool_name, node_id, pool_id_major, pool_id_minor and optional parity. Members are referenced, not created.
alloc_log: Path of the array's write-ahead log. Empty disables all persistence. When set, the member manifest lives at <alloc_log>.members and the degraded-write journal at <alloc_log>.journal.
distributed: Default false. True routes member I/O to each member's node and faults members whose node dies.
intent_sync: Default true: stripe intents are fsynced before any member write, so parity stays consistent across a power loss. False hands intents to the kernel only, which is consistent across a process crash but not a power loss, and skips an fsync per small write.

Layout

Every member is split into fixed 64 KiB chunks behind a 64 KiB superblock at offset 0 that records the array's identity, the member's role and its index. A data member's chunk for slot s is at byte 65536 + s × 65536. The logical offset handed to callers encodes the location directly:

cpp

chunk  = offset / 65536
member = chunk / 2^32       // data column
slot   = chunk % 2^32

There is no chunk map to persist. Each data member owns a band of 2^32 slots, so held offsets stay valid when members are added, removed or recovered, because a member's index never changes.

A stripe is slot s on every data member, whether or not the chunk is allocated, and its parity lives at slot s on every parity member. Parity row j is a Cauchy Reed-Solomon combination over GF(2^8) of all data columns, with coefficients that do not depend on the number of columns. Consequences:

A write updates parity from the change alone (old XOR new over the written range) when the stripe is healthy and current; otherwise the stripe is re-encoded whole.
Adding a data member adds a column of zeros, so existing parity stays valid and nothing moves.
A freed chunk still counts in its stripe, so frees never touch parity and recovery rebuilds every slot below the array's high-water mark, not just live ones.
The number of data members plus max_failures must not exceed 256, and each role holds at most 256 members.

Parity is written before a write is acknowledged, under a per-stripe lock. Before any member write, the stripes it touches are logged as intents in the allocator log and, by default, fsynced. Once the parity is written the intents are cleared. After a crash, every stripe still holding an intent is marked dirty and re-encoded before it can be used for reconstruction. The log also records which data members were down when each write began, so a restart can tell a write that never landed from one that did.

Allocation and mixed sizes

Blocks are placed one chunk at a time, round-robin over the data members that still have room, each member filling to its own capacity. When a request has a chunk for every active member, a whole stripe is placed at once. Small concurrent writers are spread over distinct recent slots so they do not serialize on one stripe lock. Adjacent chunks on the same member are coalesced into one block, so a multi-chunk allocation usually comes back as several blocks.

Members may differ in size. Each member's slot count comes from its own GetStats at creation or when added. Because every data slot needs a matching parity slot, each data member's usable slots are clamped to the smallest parity member's count, never below what it has already used. A parity member added later must hold at least as many slots as any data member has used, and a recovery replacement must hold at least the slots the failed member had used; otherwise the operation is refused. The array's capacity reported to the CTE is the sum of the active data members' clamped slots; a down member contributes only its live chunks.

Adding members at runtime

AddBdev takes a member pool name, node, pool ID and a parity flag. The command-line tool clio_safe_bdev_recover --add data|parity creates a file bdev and calls it, and the dashboard page under viz/ offers the same action.

As data. The member's superblock is read; a blank member gets one written, and a member owned by another array is refused. It joins the round-robin with a fresh column index. No data moves. If some existing member reports a predicted lifetime under seven days, AddBdev instead treats that member as failing and migrates it onto the new one.
As parity. Refused once the parity count has reached max_failures. Otherwise every written slot is marked dirty and the background builder computes the new parity column off the critical path.

Each membership change appends a snapshot to the member manifest, which is compacted periodically.

Removal, failure and rebuild

Automatic faulting. A member I/O that fails with a fatal device error (media error or vanished device) marks that member faulty and persists the fact. A transient error, a full device or a rejected range fails only that request. In a distributed array, a dead node faults every member on it.
RemoveBdev sets the member to faulty or removed. From then on reads of its chunks decode from any N of the N + m columns, reading only the byte range requested, and writes keep its bytes in the parity. Writes to a stripe with a down member reconstruct the stripe first and save the down column's chunk in the journal so a crash mid-write cannot lose it.
RecoverBdev replaces the failed member's client with the new pool, marks it recovering in the manifest, and rebuilds every slot below the array high-water mark: data members from survivors and parity, parity members by re-encoding. Writes that arrive during the rebuild are redone onto the replacement. The flag in the manifest means an interrupted rebuild resumes at the next start. Progress counters are visible through Monitor.
Restart. Creation reopens the allocator log, replays the member manifest and the per-member live slot sets, reads each member's superblock and refuses one owned by a foreign array, seats members found dead as down, and fails if more members are down than max_failures. Beyond that budget a read returns an error, never wrong bytes.

Sync encodes every dirty stripe, syncs every active member, then fsyncs the log, so a plain bdev client's Sync on the array means durable and parity-protected.

Limits

The chunk size is a compile-time constant of 64 KiB.
Without alloc_log there is no persistence: membership, allocator state, intents and the journal are all disabled.
Parity placement is dedicated: parity members hold only parity.
A removed member keeps its index; there is no way to shrink the array or reclaim a column.
The disk-failure tests are validated on Linux only.

The design note DESIGN-dynamic-view.md in the module directory describes variable-width stripes whose membership is derived from live chunks. The implementation moved to fixed-width stripes over every data column, as the runtime header documents, so the note's sections on stripe width and mirror-then-widen behavior no longer match the code.

How it is tested

Tests live in context-runtime/modules/safe-bdev/test.

cr_safe_bdev_ec_tests covers the header-only coding core with no runtime: field axioms, reconstruction from any k shards, incremental parity, and add, remove and recover at the array level.
cr_safe_bdev_config_tests parses compose YAML into the creation parameters and round-trips them through serialization.
cr_all_safe_bdev_tests runs the runtime suite: round-trip recovery, parity rebuild with a data member down, striping, allocation spread, reclaim, partial chunks, superblock reattach and foreign refusal, adding data without reshuffle, allocator-log restart, interrupted recovery resumed across a restart, and capacity after adding data then parity.
cr_safe_bdev_concurrency_tests runs the data plane with 16 worker threads so several tasks enter the container at once.
cr_safe_bdev_stress_tests grows an array from one to four data members, unplugs and recovers, with and without concurrent I/O.
cr_safe_bdev_disk_fail_tests (Linux only) kills file members with the bdev fault marker: degraded service, faults kept across a restart, rebuild under load, crashes between data and parity, crashes during degraded writes, delta parity on small rewrites, sync with dying disks and two members lost.
test_safe_bdev_dist_stress.cc builds a distributed driver that is not a ctest; it needs a live multi-node runtime and the harness under test/integration/safe-bdev.