tokener

A tokener is a software component that reads a stream of characters and converts it into a sequence of tokens that a parser can process. It performs the lexical analysis step of parsing, identifying meaningful symbols such as keywords, punctuation, numbers, and strings, and often tracks the current position in the input for error reporting.

Tokeners are typically designed to be lightweight and may operate incrementally, offering features such as lookahead, skip over whitespace, and specialized readers for strings, numbers, and literals. They often expose a small interface that includes methods for obtaining the next token, peeking at upcoming tokens, and handling end-of-input or errors. In some implementations, a tokener also accumulates enough state to provide helpful error messages, such as the line and column where parsing failed.

In practice, the term appears in various programming ecosystems. A well-known example is the JSONTokener class in the Java org.json library, which consumes JSON text and yields tokens representing objects, arrays, values, and punctuation. Other languages and libraries adopt similar concepts under different names, with tokener-like components serving as the first stage of a JSON, XML, or domain-specific language parser.

Tokeners are distinct from full parsers: a tokener focuses on breaking input into tokens, while a parser builds a hierarchical structure from those tokens. In many projects, the tokener is a specialized, reusable utility within the broader parsing pipeline, and its design can influence the simplicity or complexity of the subsequent parser. See also: lexer, tokenizer, JSONTokener.