typetoken

Typetoken, or type-token distinction, is a concept used in linguistics and text analysis to describe the vocabulary present in a text. Tokens are the individual word occurrences as they appear in running text, counting repetitions. Types are the distinct word forms, typically obtained after normalization such as lowercasing and lemmatization; for example, in the sentence "The cat sat on the mat," there are six tokens and five types when case is ignored and the two occurrences of "the" are counted as one type.

Type-token ratio (TTR) is the most common measure of lexical diversity, defined as the number of types divided by the number of tokens. It increases with more varied vocabulary but decreases as text length grows, making comparisons across texts of different lengths problematic. TTR is also sensitive to how tokens are defined (for instance, punctuation and numbers can be included or excluded) and to whether inflected forms are treated as separate types.

Use of typetoken statistics appears in studies of first and second language development, authorship attribution, readability assessment, and corpus linguistics. To address length effects, researchers employ alternative measures such as the Measure of Textual Lexical Diversity (MTLD), VocD (vocabulary diversity), and HD-D (Hypergeometric Distribution D), as well as moving-average type-token ratio (MATTR). The choice of measure depends on the language, the text genre, and the research question, but all rely on the basic distinction between types and tokens to gauge lexical richness.