getAllTokens

getAllTokens is a function or method used in tokenization components of programming languages, compilers, and natural language processing libraries. It returns the complete sequence of tokens produced from a given input, preserving the original order. The function is intended to provide a full token stream for downstream processing such as parsing, syntax highlighting, or feature extraction.

Inputs and outputs commonly observed with getAllTokens include: the input material, which may be a string, a file content, or a data stream; and an options object that can control which tokens are included, such as token types, whether to include or exclude whitespace, comments, or punctuation, and possibly normalization or state options. The typical output is an array or list of token objects or strings. Each token often carries metadata such as type, value or text, start and end positions, and sometimes line and column information to aid error reporting and analysis.

In usage, getAllTokens is employed when a complete token sequence is required for downstream stages like parsing, lexical analysis, or syntax-aware processing. It can be implemented as a standalone function or as a method on a tokenizer or lexer object, and may be available as an iterator or generator in addition to returning a collection.

Variants and related terminology vary by library. Some systems expose a similar function named tokenize or lexAll, while others provide a streaming token generator. getAllTokens is typically contrasted with incremental or on-demand tokenization, where tokens are produced as needed rather than in a single pass.