corpuswide

Corpuswide is a term used in corpus linguistics and related fields to describe analyses, statistics, or models that are computed for an entire text corpus rather than for individual documents or smaller subsets. A corpuswide approach seeks to describe the corpus as a whole, capturing aggregate patterns that hold across its included texts, genres, or time periods.

Common corpuswide analyses include computing overall word frequency distributions, type-token ratios across the complete corpus, dispersion of items, and corpus-wide collocation or co-occurrence networks. It may also refer to training or evaluating models on the full corpus to reflect corpus-level characteristics rather than those of single documents. Methodologically, corpuswide work typically involves preprocessing and annotation at scale (tokenization, lemmatization, part-of-speech tagging, parsing), followed by aggregating metrics across all tokens or texts and reporting aggregated statistics with appropriate measures of uncertainty.

Applications of corpuswide analysis span dictionary development and lexicography (estimating word frequencies and lexical coverage), vocabulary testing for language teaching, and linguistic research on distributional patterns such as Zipf’s law and Heaps’ law. It is also used in benchmarking natural language processing tools and in comparing different corpora to understand genre, register, or diachronic variation at a macro level.

Limitations include dependence on the representativeness and quality of the underlying corpus. Corpuswide results can obscure important subcorpus differences, and aggregation may mask variation across texts. Computational demands can be substantial for very large corpora, and biases in collection, annotation, or processing steps can affect interpretation.