vocabsize

Vocabsize refers to the size of the vocabulary used by a language model or the number of distinct tokens in a text corpus after preprocessing. It depends on tokenization and normalization choices and whether punctuation, digits, or special tokens are counted. In corpus linguistics, vocabsize serves as a measure of lexical diversity and typically grows with corpus size, often following sublinear growth patterns described by Heaps' law.

In machine learning, vocabsize is often a model parameter that determines the number of embeddings and the size of the output layer. A larger vocabsize increases memory usage and computation. To manage this, many modern models use subword tokenization (such as BPE or WordPiece), which expands the repertoire of units while keeping a manageable vocabsize; new words can be formed from subword units, reducing out-of-vocabulary occurrences. In such setups vocabsize usually refers to the set of subword units plus any special tokens.

Counting rules vary: some pipelines include punctuation; others remove it. Special tokens like <PAD>, <UNK>, and <EOS> are typically excluded from the count when measuring embedding or output dimensions.

Vocabsize is context-dependent and shaped by preprocessing choices. It is a practical parameter rather than an intrinsic linguistic property, balancing coverage, memory usage, and processing speed.