chunker

Chunker is a term used in natural language processing to describe a component or model that performs shallow parsing, grouping adjacent words into chunks such as noun phrases (NPs) and verb phrases (VPs). Unlike full syntactic parsers, a chunker does not attempt to build a complete hierarchical parse tree; instead it identifies spans of text with local syntactic categories.

Typically, chunkers operate on text that has been part-of-speech tagged. They can be rule-based, using regular expressions over POS tags to recognize common phrase patterns, or statistical, trained as sequence labeling models. Common modeling approaches include maximum entropy, conditional random fields, and support vector machines. The output is often encoded using the Inside–Outside–Beginning (IOB) scheme, for example B-NP, I-NP, O, etc., to mark the boundaries and types of chunks.

Common chunking tasks emphasize noun phrase chunking, though verb phrases and other categories may also be identified. Prominent implementations include the OpenNLP chunker, the Stanford chunker, and rule-based chunkers in the NLTK toolkit. Some modern NLP frameworks provide chunking as part of a broader pipeline, with noun phrasing accessible through pre-trained models or lightweight regex-based modules; spaCy, for instance, yields noun phrase chunks as part of its parse results.

Data sets such as the CoNLL-2000 NP chunking corpus have driven development and evaluation, using precision, recall, and F1 to assess boundary and label accuracy. Chunking is widely used as a preprocessing step for information extraction, question answering, and as a feature source for downstream tasks like named entity recognition and relation extraction. Its primary limitation is that it provides only shallow structure and depends on the quality of POS tagging.