Stemmed

Stemmed refers to text or tokens that have undergone stemming, a preprocessing step used in linguistics and information retrieval. Stemming reduces words to their stem or root form by removing affixes, with the aim of grouping morphologically related terms. The resulting stem may not correspond to a complete word in the language, but it captures the core lexical content for processing.

Stemming is typically performed by rule-based algorithms that apply language-specific suffix or infix rules. Common English stemmers include the Porter stemmer and the Snowball stemmer, while Lovins and other rule sets exist for other languages. Stemmed forms are used to build search indexes, enabling matches between related terms such as run, running, and runs.

Differences from lemmatization: stemming prioritizes simplicity and speed over linguistic precision. Lemmatization maps a word to its dictionary lemma using morphological analysis, often preserving valid words. Example outcomes can differ: running may become run by both approaches, but better may become bet or better depending on the stemmer, illustrating possible over-stemming.

Limitations include over-stemming and under-stemming, language dependence, and reduced interpretability of stems. Stemming is widely used in information retrieval and text mining, particularly for large-scale corpora, where approximate matching is advantageous. In multilingual contexts, languages with rich morphology may require specialized stemmers or the combination of stemming with other normalization techniques.