tfidfpohjaiset

TF-IDF (Term Frequency-Inverse Document Frequency) is a numerical statistic used to reflect how important a word is to a document in a collection or corpus. It is widely used in information retrieval and text mining. The TF-IDF score increases proportionally to the number of times a word appears in the document, but is offset by the frequency of the word in the corpus, which helps to adjust for the fact that some words appear more frequently in general.

The term frequency (TF) is the number of times a word appears in a document. It is calculated as the raw count of a term in a document divided by the total number of terms in the document. This gives a measure of how frequently a term appears in a document.

The inverse document frequency (IDF) is a measure of how much information the word provides, that is, whether the term is common or rare across all documents. It is calculated as the logarithm of the total number of documents divided by the number of documents containing the term. This downscales the weight of common terms and upscales the weight of rare terms.

The TF-IDF score for a term in a document is then calculated as the product of the TF and IDF scores. This gives a measure of the importance of a term to a document in a corpus.

TF-IDF is used in various applications such as search engines, text summarization, and document clustering. It helps to identify the most relevant documents for a given search query by giving higher weights to terms that are important to a document but not common across all documents.