codebookmodeller

CodebookModeller is an open-source software framework for designing, training, and evaluating codebooks used in vector quantization and dictionary-based representations. A codebook is a finite set of codewords that map input vectors to discrete indices, enabling data compression, efficient search, and feature quantization. The project emphasizes modularity and reproducibility, with components that can be swapped or extended.

Its architecture separates core concepts such as codebooks, codewords, distance metrics, and optimization strategies. Codebook backends implement various quantization schemes, including k-means-based Lloyd quantization, hierarchical codebooks, product quantization, and residual/vector quantization. The framework provides multiple distance measures, training pipelines, and evaluation tools for distortion, rate-distortion, and codeword utilization.

CodebookModeller offers a Python API with a fit/transform style, designed to integrate with common machine learning workflows. It supports data serialization, interoperability with standard formats, and experiment tracking. The design favors configurability and testability, with clear interfaces and extensible components.

Typical workflow involves selecting a quantization scheme and parameters, training on a dataset, and evaluating using prescribed metrics. Trained codebooks can be used to encode data into compact indices or to accelerate approximate nearest neighbor search, with decoding options that yield controllable reconstruction error.

Applications include data compression, image and audio quantization, feature quantization in machine learning pipelines, and model or inference acceleration. The project targets both research and production environments and welcomes community contributions and ongoing development.