memoryjoin

Memoryjoin is a term sometimes used in data processing to describe a join operation where the participating datasets are kept resident in main memory during the computation. In such setups, the join is performed with minimal disk I/O, allowing faster access to records and better cache locality. The concept applies to both batch processing and streaming analytics, where at least one of the inputs remains in memory for the duration of the join.

Common implementations of memoryjoin rely on established join algorithms adapted for in-memory operation. A memory-based hash join builds a hash table from one input in memory and probes it with the other input. A memory-resident sort-merge join sorts inputs in memory and then merges them on matching keys. Nested-loop joins are typically avoided at scale but may be used when one relation is very small or memory is constrained. In memory-limited environments, memoryjoin may spill intermediate results to disk or employ hybrid approaches that partition data and operate on subsets that fit in RAM.

Usage scenarios include in-memory databases, real-time analytics, and streaming pipelines where a dimension table is cached in memory to join with streaming facts, or where both sides are kept in memory to reduce latency. Performance hinges on memory availability, data skew, key cardinality, and the efficiency of the chosen join method.

Although "memoryjoin" is not a universally standardized term, it is closely related to in-memory join techniques such as in-memory hash join and in-memory sort-merge join and is often discussed in the context of single-machine or near-memory-disaggregated systems.