lemmatization

Lemmatization is the process of reducing inflected or variant forms of a word to its lemma, the canonical base form found in dictionaries. It aims to map a word to a single, meaningful representative form, typically the headword, rather than producing arbitrary stems. This distinguishes lemmatization from stemming, which often results in non-dictionary forms or incomplete roots.

Lemmatization relies on morphological analysis and often part-of-speech tagging to determine the correct lemma. Because the lemma can depend on the word’s role in a sentence, many lemmatizers require POS information and language-specific resources such as lexicons or morphological rules. Approaches range from rule-based systems using handcrafted dictionaries to statistical or neural methods trained on large corpora; contemporary tools frequently combine these resources to improve accuracy.

Examples illustrate the process: "running" maps to "run" when treated as a verb, "better" maps to "good" when used as an adjective, and "cars" maps to "car". Irregular forms, polysemy, and multiword expressions pose challenges, as the appropriate lemma can depend on context. Proper nouns and low-resource languages add further complexity, and lemmatization quality depends on the coverage of the underlying linguistic resources.

Applications of lemmatization span information retrieval, text normalization for NLP pipelines, machine translation, question answering, and various text analytics tasks. By reducing inflected forms to a common base, lemmatization helps unify tokens for matching, counting, and semantic interpretation, improving the consistency and interpretability of language processing systems.