tfidf

TF-IDF, short for term frequency–inverse document frequency, is a numeric statistic used to evaluate how important a word is to a document within a collection or corpus. The idea is to weigh terms that appear frequently in a document but are relatively rare across the corpus higher than common words.

Term frequency TF(t, d) measures how often term t occurs in document d. Inverse document frequency IDF(t, D) measures how rare the term is across the whole corpus D. A common formulation is IDF(t, D) = log(N / DF(t)), where N is the total number of documents and DF(t) is the number of documents containing t. The TF-IDF weight for term t in document d is w(t, d) = TF(t, d) × IDF(t, D). In practice, TF can be raw counts or normalized (for example, divided by the document length), and IDF can be smoothed or base-10 logarithmic.

TF-IDF vectors are often normalized (such as to unit length) to enable meaningful comparisons between documents using cosine similarity. This weighting emphasizes terms that are distinctive for a document relative to the corpus, assisting in tasks that rely on text representation.

Applications include information retrieval and search engine ranking, document classification, clustering, and other text mining tasks where feature weighting improves discrimination between documents. Limitations include ignoring word order and semantics, treating words independently, and dependence on the chosen corpus. It may require frequent updates in dynamic corpora. Variants and improvements exist, such as sublinear TF scaling, smooth IDF, and BM25, which adapt the basic idea to different modeling goals.