Systemds

SystemDS is an open-source system for scalable machine learning and data science. It originated from IBM’s SystemML and has been developed with collaboration among IBM Research, TU Wien, and the Apache Software Foundation. The project provides a declarative machine learning language and a compiler/runtime that optimize and execute ML workloads on large datasets, enabling data scientists to express algorithms at a high level while the system handles efficient execution.

At its core, SystemDS offers a declarative language for ML, commonly referred to as DML, along with a Python API and Java API. Users write programs that describe data transformations and learning algorithms, and SystemDS’ optimizer converts these programs into efficient execution plans. The runtime supports distributed and out-of-core computation, aiming to minimize manual tuning by selecting suitable data representations and operator implementations. It is designed to work with big data platforms and can run in local mode or on clusters.

Key features include matrix- and tensor-oriented operations, optimized linear algebra routines, and a library of common ML tasks such as regression, classification, clustering, and matrix factorization. SystemDS emphasizes end-to-end workflows that combine data preparation with modeling, enabling compact scripts that can scale to large datasets. It supports sparse and dense data, automatic memory management, and plan optimization to improve performance on distributed environments.

SystemDS is distributed under an Apache-2.0 license and is available as an open-source project with ongoing community development. It is used in academic and industry contexts to explore scalable ML workflows without extensive low-level optimization.