Ngrammit

ngrammit is a software toolkit designed to extract and analyze n-grams from text data. It provides utilities to generate unigrams through higher-order n-grams, count frequencies, and compute statistics to support natural language processing, corpus linguistics, and text mining. The library emphasizes a modular design that can be integrated into preprocessing pipelines for research or production tasks. Key features include tokenization options, case normalization, punctuation handling, optional stopword filtering, and support for skip-gram configurations. It can operate on large corpora via streaming processing or on in-memory datasets and can read plain text as well as structured inputs such as JSON or CSV.

Output from ngrammit includes frequency tables, relative frequencies, and ranking of n-grams by measures such as simple counts or more advanced collocation scores. The tool is suitable for language modeling, discovery of common word sequences, feature extraction for machine learning classifiers, and comparative corpus studies. While intended to be general-purpose, users should be mindful of limitations around memory usage for high-order n-grams and sparse data in small corpora; parameter choices such as the maximum n, minimum frequency, and smoothing settings can significantly affect results.

ngrammit is released as open-source software with a permissive license. It is maintained by a community of contributors and is designed to be language- and framework-agnostic, with adapters to common data science stacks. See also: N-gram, language model, Markov chain, text mining.