kazanlm

KazanLM is a family of transformer-based language models developed by researchers affiliated with Kazan University and partners. The project seeks to advance multilingual natural language processing by providing scalable tools that can be used for research, education, and industry. The models are released in several sizes and are available under an open-source license.

KazanLM uses a decoder-only transformer architecture. Available versions span from roughly 2 billion to 20 billion parameters. Training data comprises multilingual text from publicly available sources and licensed corpora, with a focus on broad linguistic coverage and regional languages. The training pipeline emphasizes efficiency, employing techniques such as mixed-precision training, gradient checkpointing, and quantization-aware training.

The models support a range of tasks, including instruction following, text generation, summarization, translation, and code generation. They are accessible via an API and open-source repositories, and users can fine-tune or adapt them with task-specific data using standard frameworks.

Evaluations show competitive performance for models in the 2–10 billion parameter range on several standard benchmarks, though larger commercial models may outperform them on certain tasks. Researchers note strengths in multilingual generation and accessibility, while pointing to limitations in factual reliability, potential biases, and the need for safety mitigations.

See also: transformer (machine learning), large language model, multilingual NLP.