kubikfod

Kubikfod is an open-source data processing and storage framework designed for modular, composable pipelines in distributed systems. It provides a plugin-based architecture that enables developers to assemble processing chains from independent components called kubiklets. The project emphasizes portability across cloud and edge environments and aims to simplify data orchestration and transformation at scale.

Architecture and data model: Kubikfod uses a microservices-like orchestration layer and pluggable runtimes. Data flows through pipelines defined declaratively in YAML or JSON, with support for both batch and streaming processing. It relies on a message bus for event delivery and provides facilities such as a schema registry, data lineage, and versioned artifacts to support reproducibility and governance.

Core components: The runtime executes pipelines, the kubiklet plugin system provides processing units, and the registry stores schemas, configurations, and artifacts. Storage adapters enable interaction with object stores, distributed filesystems, or databases. Access control is handled through role-based permissions and integration with external identity providers via OIDC or SAML.

History and development: Kubikfod originated from an open-source collaboration known as the Kubik Collective, with initial design work between 2018 and 2019. The first stable release appeared in 2020, and development has continued through an active community contributing plugins and integrations. The project is released under the Apache 2.0 license.

Usage and reception: Organizations use Kubikfod for edge-to-cloud data ingestion, data transformation, and orchestration of analytics workloads. It competes with other data orchestration and ETL frameworks and is noted for its emphasis on portability, extensibility, and reproducibility in pipelines.