ujjsontok

ujjsontok is a lightweight tokenizer component used to break JSON input into a sequence of tokens in the ujjson family of tooling. It is designed to work alongside a JSON parser, providing the lexical analysis stage required to interpret JSON documents. The tokenizer exposes a stream of tokens corresponding to the lexical elements of JSON: structural tokens for { } [ ] :, and ,; literal tokens for true, false, null; as well as value tokens for strings and numbers. It can report the location (line and column) of each token and supports streaming input, allowing large JSON documents to be tokenized incrementally.

Design and features: The module emphasizes predictable, deterministic token emission, with Unicode support inside string values and proper handling of escape sequences. It typically supports strict mode, which enforces standard JSON semantics, and may offer lenient mode for common non-conforming inputs. Error reporting includes token position and a description of the parse error.

Usage: In a typical workflow, ujjsontok is fed chunks of a JSON stream and emits tokens to a parser or a higher-level event handler. Developers integrate it by attaching callbacks or by consuming an iterator or generator interface. The API generally includes functions to initialize a tokenizer, supply input, and fetch the next token, along with metadata about the token's type and value.

Copyright and licensing: As with other ujjson components, ujjsontok is released under an open-source license and maintained by contributors in the JSON tooling ecosystem. It is designed to be cross-language in intent, with bindings or ports to several languages via wrappers or interop interfaces.