Stopwords

Stopwords are words that occur very frequently in a language but carry little lexical meaning on their own. In natural language processing and information retrieval, they are often filtered out during preprocessing to reduce noise and computational overhead. Typical stopwords include articles, pronouns, prepositions, conjunctions, and auxiliary verbs, such as the, of, and, is, it.

Stopword removal is usually performed after tokenization and before feature extraction. It can be accomplished with predefined lists, frequency-based filtering, or user-defined domain lists. The lists are language-specific and can vary by corpus and application. Some systems use stopword lists tailored to a domain, such as legal or biomedical texts, where common function words may be informative or not.

The practice has benefits and drawbacks. Removing stopwords can reduce dimensionality, speed up search, and improve signal-to-noise ratio for certain tasks. However, stopwords can be important for syntax, negation, and meaning in some contexts, especially in sentiment analysis, question answering, or when full text integrity is required. In modern NLP with deep learning, many pipelines either retain stopwords or rely on models that learn contextual representations, diminishing the need for explicit removal.

Stopword lists exist for many languages and are often created by analyzing large corpora or by hand. Some systems provide configurable options to enable or disable stopword removal depending on the task.