RDMA

RDMA stands for Remote Direct Memory Access. It enables a computer to directly access memory on another computer without involving its CPU or OS, enabling low-latency, high-throughput networking essential for cluster and data center workloads.

RDMA operates over specialized networks such as InfiniBand, RoCE (RDMA over Converged Ethernet), and iWARP. It uses one-sided operations where the initiator specifies a remote memory address and transfers occur directly between memory regions managed by NICs. Memory regions must be registered and protected via a protection domain. Communication is driven by work requests posted to queue pairs and completion events reported on completion queues. The network interface handles data movement, often achieving zero-copy transfers and bypassing kernel involvement on the remote side.

Software stacks include the RDMA verbs API, librdmacm, and higher-level libraries and runtimes such as MPI, UCX, and storage stacks. Programming typically involves creating protection domains, memory registration, queue pairs, and using RDMA read/write or atomic operations, sometimes with polling or interrupts for completions.

Applications and benefits: RDMA is used in high-performance computing, distributed databases, in-memory caches, and storage systems, and in NVMe over Fabrics deployments. Benefits include low latency (often microseconds or less), high sustained bandwidth, CPU offload, and reduced memory copies, which improve scalability.

Considerations and limitations: it requires RDMA-capable hardware and a compatible network fabric, careful configuration of permissions and memory registration, and software stacks. Security relies on fabric-level controls and memory protection domains. Not all workloads benefit equally; for small messages, latency may be dominated by network setup.