typtokenratio

Type-token ratio is a measure used in linguistics and natural language processing to quantify lexical diversity. It is calculated by dividing the number of unique words (types) in a text by the total number of words (tokens) in that text. A higher type-token ratio indicates greater lexical richness, meaning a wider variety of words are used relative to the text's overall length. Conversely, a lower ratio suggests more repetition of words.

The type-token ratio is a simple metric, but its interpretation can be influenced by text length. Shorter texts tend to have naturally higher type-token ratios because there is less opportunity for word repetition. For this reason, comparisons are often made between texts of similar lengths or the ratio is analyzed in conjunction with other diversity measures. It is a useful tool for analyzing writing style, identifying potential issues in text generation, and comparing different corpora. For example, a children's book might have a lower type-token ratio than a complex academic paper. The calculation is straightforward: count all the words, then count how many distinct words there are, and divide the latter by the former.