Textkorpora

Textkorpora are large, structured collections of natural language texts used for linguistic research and natural language processing. They are typically stored with metadata and, in many cases, annotations such as lemmas, part-of-speech tags, or syntactic parses to enable statistical analysis, model training, and cross-linguistic comparison.

There are several main types: monolingual corpora for a single language; multilingual corpora containing texts in multiple languages; parallel corpora with sentence-aligned translations used for machine translation; and domain-specific or balanced corpora designed to reflect particular genres or registers.

Textkorpora are created from diverse sources – books, newspapers, web texts, transcripts – and undergo preprocessing (tokenization, normalization, deduplication) and, when available, annotation (POS, lemmas, syntax, named entities). Metadata such as language, domain, size, date, and license support reproducibility and discovery.

Prominent examples in the German and international context include DeReKo (Deutsches Referenzkorpus), DWDS Korpus, and the Leipzig Corpora Collection, alongside well-known English corpora such as COCA and the Brown Corpus, and multilingual resources like Europarl and OpenSubtitles.

Applications include language modeling, parsing, part-of-speech tagging, machine translation, information retrieval, and corpus-based sociolinguistic analysis.

Key challenges include copyright and licensing, representativeness and bias, annotation quality, versioning and provenance, privacy considerations for user-generated data, and the storage and compute requirements of large-scale corpora.