tekstikogude

Tekstikogude (text corpora) are large, structured collections of natural language text that are used for linguistic research and natural language processing. They typically consist of raw texts plus metadata (language, source, date, genre) and may include linguistic annotations such as tokenization, lemmatization, part-of-speech tags, syntactic parses, named entities, or semantic roles.

They can be monolingual, bilingual or multilingual, and vary by size, genre, domain, and time period. General-purpose corpora aim for broad coverage; specialized corpora focus on a field (law, medicine); balanced corpora try to evenly represent genres; diachronic corpora track language change over time.

Creation involves collecting texts from publishers, websites, digitization of print sources, and preparing data through cleaning, deduplication, and copyright clearance. Annotations are added via manual annotation or automated NLP tools, often in standardized formats (TEI-XML, JSON, CoNLL-like tabular formats). Common licensing ranges from open (Creative Commons) to restricted to ensure rights holders' permissions.

Uses include linguistic analysis (morphology, syntax, semantics), lexicography and language resource development, training and evaluating NLP models (tokenizers, POS taggers, parsers, language models), and benchmarking. Corpus design considerations include size, balance across genres and registers, representativeness, annotation quality, and reproducibility.

Examples of well-known corpora include general-language corpora and domain-specific collections, multilingual corpora, and web-scale corpora. In Estonian contexts, text corpora support language research, education, and technology development, often with emphasis on licensing and accessibility to researchers and developers.