informationretrieval

Information retrieval is the science of obtaining information items that satisfy an information need from large collections. It focuses on identifying relevant documents or records and ranking them in order of usefulness to the user, rather than simply returning all matches. IR systems are widely used in web search, digital libraries, and enterprise information discovery.

A typical IR pipeline includes collecting and preprocessing documents, building an index, processing a user query, scoring documents with a retrieval model, and presenting ranked results. A common data structure is the inverted index, which maps terms to the documents that contain them. Retrieval models range from classic approaches such as Boolean retrieval and vector space models to probabilistic methods. TF-IDF and BM25 are widely used weighting schemes, while language-model based rankers estimate the likelihood of a query given a document or vice versa.

Query processing often includes tokenization, normalization, stop-word removal, stemming or lemmatization, and sometimes query expansion. Modern systems may apply personalization, diversity considerations, and fast re-ranking using compact representations.

Evaluation in information retrieval uses metrics such as precision, recall, F1, and rank-based measures like mean reciprocal rank and normalized discounted cumulative gain. Standard test collections and tasks, such as TREC or CLEF, support offline evaluation, while live user studies inform interactive systems.

Recent trends in information retrieval include neural ranking models and transformer-based re-ranking, learning to rank, and dense vector representations paired with approximate nearest-neighbor search for scalable retrieval. Challenges include scalability, multilingual and multi-modal data, user intent understanding, fairness, and privacy concerns.