Korpused

Korpused, the Estonian term for linguistic corpora, are large, structured collections of language data used for linguistic research and language technology development. They compile authentic text and speech samples from diverse sources to approximate the usage patterns of a language or its varieties. By providing a quantitative basis for analysis, korpused support tasks such as frequency studies, grammar discovery, lexicon development, and training of NLP systems.

Common types include text corpora (written texts such as novels, newspapers, or web content), speech corpora (recorded spoken language with transcripts), and multimodal corpora (combining text with audio, video, or other data). Many korpused are annotated at various levels, including part-of-speech tagging, lemmatization, syntactic parsing, named entity recognition, and discourse structure. Parallel or translation corpora pair texts in multiple languages to support translation studies and machine translation.

Building a korpus involves data collection, cleaning, annotation, and quality control. Legal and ethical considerations, licensing, and privacy constraints affect what data can be included. Annotations may be produced manually by linguists or semi-automatically using tools, with inter-annotator agreement used to assess reliability. Metadata describes sources, genres, and temporal or regional variation.

Access to korpused ranges from open, freely available resources to restricted databases requiring licenses or institutional access. They support a wide range of applications, including linguistic research, lexicography, language documentation, language education, and the development of natural language processing tools such as spell checkers, search engines, and voice interfaces.

Representativeness and bias remain concerns in corpus design. Ongoing efforts aim to expand regional varieties, register coverage, and cross-linguistic comparability, while improving annotation standards and interoperability through common schemas and formats.