mapreduce

MapReduce is a programming model and an associated software framework for processing large data sets across distributed clusters. It enables developers to write applications that automatically parallelize and execute across many machines by expressing computation as Map and Reduce operations.

In the Map phase, the map function processes input records to produce intermediate key-value pairs. The framework groups and shuffles these pairs by key, so all values with the same key are sent to the same reducer. In the Reduce phase, the reduce function aggregates or transforms the values for each key to produce final outputs. Often, a combiner can be used as a mini-reducer to optimize data transfer.

The framework handles partitioning, scheduling, data locality, fault tolerance, and persistence. It typically runs on clusters using a distributed file system like HDFS, which stores input and output data and supports replay in case of failures. If a map or reduce task fails, it is re-executed on another node, ensuring fault tolerance.

Originating at Google in 2004, MapReduce was popularized by the Apache Hadoop project, which provides an open-source implementation. Hadoop's MapReduce, together with the Hadoop Distributed File System and YARN resource manager, became a dominant platform for batch processing. Other systems and newer engines offer MapReduce-like capabilities or alternative processing models, such as in-memory processing in Spark.

Use cases include log analysis, large-scale indexing, data warehousing, and ETL tasks. Limitations include overhead for small jobs, less suitability for interactive or iterative workloads, and latency that makes it less ideal for real-time processing; streaming frameworks provide alternatives for continuous data processing.