CLIO Core

Technologies · Memory

Memory expansion: GPU memory without a ceiling.

Eternia is a software distributed shared memory built on CLIO Core. It extends GPU memory across nodes and down a tiered hierarchy of DRAM and storage, and takes placement, coherence and persistence off the programmer's hands.

Interface
Paged vector (gpu_vector, cpu_vector)
GPUs
Intel Max 1550 (SYCL), NVIDIA (CUDA)
Tiers
HBM, DRAM, DAOS, Lustre, any CTE tier
Scale
Measured to 64 Aurora nodes

Key takeaways

  • Paged vectors for CPU and GPU code whose data is larger than device memory.
  • Pages spill to DRAM and storage tiers; placement, coherence and persistence are handled for you.
  • Measured within 0.88-1.19x of MPI, oneCCL and Intel SHMEM on up to 64 Aurora nodes.
  • 0.88-1.19xMPI's time on four workloads, 4 to 64 Aurora nodes
  • 2xlarger problems than DDN, Hermes and cuFile can run
  • 1.22-1.24xfaster than the fastest of them where they still run
  • 52 mscommunication at 64 nodes, against about 7 s for the libraries

What it is

Scientific and AI applications increasingly produce datasets that exceed GPU memory, and buying more HBM is expensive in both money and energy. Today the alternatives are distributed GPU memory libraries, which are bounded by aggregate device memory, and hand-written out-of-core code, which is complex, rarely reusable and pays heavy data-movement penalties.

Eternia gives kernels a paged vector whose pages live in CLIO Core's tiers. The interface is explicit on purpose: kernels issue their own batched, asynchronous fetches and flushes, so data movement overlaps computation.

  • Generational coherence keeps pages cached in GPU memory consistent across nodes and tiers, without sharer tracking or invalidations.
  • Tiered by construction. Pages that do not fit in the GPU's frame cache spill to DRAM, then to DAOS, Lustre or any tier the runtime composes, so a dataset is bounded by the tier stack, not by device memory.
  • Persistent checkpoints. Checkpoints are copy-on-write snapshots in the tiered store, drained to persistent storage in the background and readable in portable formats such as HDF5.
  • Shipped with its expertise. An agent skill encodes how to use the interface well, and derives application-specific placement policies from an application's own kernels.

Against MPI, oneCCL and Intel SHMEM

Four workloads with very different access patterns (k-means, a Gray-Scott stencil, the particle-mesh Ewald spread and gather of a molecular-dynamics step, and model-parallel neural-network training) were written once on Eternia's paged vector and again as standalone MPI, oneCCL and Intel SHMEM codes. They weak-scale from 4 to 64 Aurora nodes, one GPU tile per node, with each node's share fitting in GPU memory. Every run ends in a correctness gate: identical checksums, bit-equal mesh totals, identical weight digests.

Weak scaling on Aurora from 4 to 64 nodes for kmeans, Gray-Scott, gmx and lbann. Each bar is time per iteration for Eternia, MPI, oneCCL and Intel SHMEM, split into communication and compute.
Weak scaling on Aurora, one GPU tile per node. Each bar is time per iteration, split into communication on the slowest rank (hatched) and compute. Flat is ideal, except for lbann, whose model grows with the node count. Intel SHMEM failed validation for lbann at 4 and 8 nodes and has no result at 64.

The libraries cache nothing and need no coherence, so they are the upper bound. Eternia stays within 20% of MPI at every scale measured and matches or beats oneCCL and Intel SHMEM in several configurations:

  • gmx is at parity (1.00-1.06x MPI) and faster than all three libraries at 24 nodes (0.88x). Its halo is a handful of whole mesh planes that Eternia publishes once per pass.
  • lbann trains faster on Eternia at 4 and 8 nodes (0.90x and 0.98x) and stays within 7% at 16 and 64.
  • kmeans (1.10-1.19x) and Gray-Scott (1.12-1.16x) hold steady as the machine grows. At 64 nodes Intel SHMEM falls to 1.53x MPI on Gray-Scott while Eternia holds 1.12x.
gmx communication on the slowest rank, 64 nodes, seconds (lower is better)
  • Eternia0.052
  • MPI~7
  • oneCCL~7
  • Intel SHMEM~7
Gray-Scott time relative to MPI, 64 nodes (lower is better)
  • MPI1.00x
  • Eternia1.12x
  • Intel SHMEM1.53x

Where the residual cost comes from: the kernels themselves run within 2-6% of MPI's. The gap is small control-path exchanges, such as k-means' 4 KB centroid reduction, which ride the runtime's TCP transport where the libraries use the fabric directly.

Larger problems than file systems and I/O buffers

A GPU simulation that outgrows memory would otherwise write its output through a parallel file system or an I/O buffering layer. Gray-Scott on 4 Delta nodes (four NVIDIA A40 GPUs, 256 GB of DRAM per node) grows from 32 GB to 256 GB of data, against the same code writing through the DDN Exascaler file system over POSIX, through the tiered buffering system Hermes, and through NVIDIA cuFile (GPUDirect Storage). Runtime in seconds, lower is better:

32 GB
  • Eternia40.2
  • cuFile49.2
  • Hermes53.6
  • DDN138.5
64 GB
  • Eternia93.8
  • cuFile116.2
  • Hermes129.6
  • DDN223.4
128 GB
  • Eternia165.3
  • cuFile201.1
  • Hermes227.9
  • DDN549.6
192 GB and 256 GB
  • Eternia, 192 GB281.5
  • Eternia, 256 GB451.3
  • DDN, Hermes, cuFileout of memory

Gray-Scott on 4 Delta nodes as the dataset grows. Beyond 128 GB the DDN, Hermes and cuFile editions exhaust memory and no longer run.

The alternatives keep the whole grid resident in memory while they compute and write it, so the largest problem they can run is bounded by the memory of the nodes they run on. Eternia pages the grid through its tiers, so it runs twice the data, and runs the sizes the others can handle 1.22-1.24x faster than the fastest of them.

Where these numbers come from

All results on this page are from the Eternia paper's evaluation (external/eternia), run on Argonne's Aurora (two Xeon CPU Max 9470, 1 TB DDR5 and six Intel Data Center GPU Max 1550 per node, Slingshot 11) and NCSA's Delta. Every timed region runs for at least two minutes, and every run is checked against its reference.

Where to go next