tolesnpro

tolesnpro is an open‑source data processing framework designed for scalable analytics on large, distributed datasets. It combines a high‑performance execution engine with a simple, declarative query language that allows users to write expressions in a syntax similar to SQL but with additional support for streaming data and machine‑learning pipelines. The framework is written largely in Go and leverages a message‑passing interface for inter‑node communication, making it particularly well suited for deployment in containerised environments such as Kubernetes.

The project began in 2019 as an internal research tool at the Centre for Analytics Innovation, where a team of scientists and engineers sought to unify batch and real‑time processing into a single, unified engine. A public release under the Apache 2.0 licence followed in 2020, and the core team subsequently partnered with several cloud‑service providers to offer managed deployments. Its current maintainers include representatives from both academia and industry, and the project is governed by a community‑driven code‑review process.

Key technical features of tolesnpro include in‑memory columnar storage, optional column pruning, adaptive query optimisation, and built‑in support for common statistical functions and data‑science libraries. The system supports integration with popular data sources such as Hadoop Distributed File System, Amazon S3, and streaming platforms like Apache Kafka. Users can also embed custom UDFs (user‑defined functions) written in Go or Python, thanks to a native foreign‑function interface.

In practice, tolesnpro is employed by organisations that require both high‑throughput analytics and low‑latency insights, such as telemetry providers, e‑commerce platforms, and scientific research consortia. Community contributions focus on extending connector support, enhancing optimisation strategies, and improving observability tooling. The project roadmap includes a planned shift toward a serverless compute model and tighter integration with the Spark ecosystem to broaden its appeal to existing big‑data users.