spark

Spark is an open-source distributed computing system designed for fast, scalable data processing. Originating at the University of California, Berkeley's AMP Lab, Spark was released as open source in 2010 and later became a top-level project of the Apache Software Foundation in 2014.

Its core abstraction is the Resilient Distributed Dataset (RDD), which enables in-memory, fault-tolerant computations over a cluster. Spark employs a directed acyclic graph (DAG) execution engine that optimizes and schedules tasks across nodes, supporting iterative algorithms and interactive data analysis. It can run on different cluster managers such as standalone, Hadoop YARN, Apache Mesos, and Kubernetes.

Spark provides several high-level APIs and libraries: Spark SQL with DataFrames and Datasets for structured data; MLlib for machine learning; GraphX for graph processing; Structured Streaming for real-time data streams; and Spark Streaming as its earlier streaming API. It supports multiple programming languages, including Scala (native), Java, Python (PySpark), and R (SparkR).

Common use cases include batch ETL, data warehousing workloads, iterative machine learning, and real-time analytics. Spark is designed to handle large-scale data workloads in memory, though it can spill to disk for large datasets. It integrates with other big data technologies and is widely deployed in cloud and on-premises environments.