sparkssmall

Sparkssmall is an open-source, lightweight data processing framework designed for small-scale clusters, edge environments, and educational use. It provides a familiar API for batch and streaming workloads while emphasizing low memory usage and a small runtime footprint. The project aims to offer the core programming model of larger systems in a compact package suitable for devices with limited resources.

Origin and development: Sparkssmall originated as a community-driven project forked from larger distributed data frameworks to explore modular design and minimal dependencies. Development began in the early 2020s with a focus on portability across CPUs and single-board computers.

Features: Core engine supports batch processing and micro-batch streaming, a DataFrame-like API, and SQL querying. It includes a compact in-memory cache, pluggable storage connectors (local filesystem, S3-compatible, and HDFS-like), and a lightweight scheduler. The runtime can run on a single node or small clusters, with optional backends for GPU or CPU acceleration.

Architecture and API: Sparkssmall uses driver and worker components; tasks are scheduled to workers; built-in fault tolerance uses lineage or checkpointing; offers bindings for Python, Java, and Scala with a subset of common Spark APIs.

Reception and status: The project is maintained by a growing, volunteer community and released under an open-source license. It prioritizes ease of setup and predictable resource use, while acknowledging that feature parity with larger systems is limited. It is used in education, rapid prototyping, and small-scale data experiments.