utf16

UTF-16 is a Unicode Transformation Format that encodes Unicode code points using 16-bit code units. It is a variable-length encoding: most characters from the Basic Multilingual Plane (U+0000 to U+FFFF) are represented by a single 16-bit unit, while characters from supplementary planes (U+10000 to U+10FFFF) are encoded using pairs of 16-bit units known as surrogate pairs.

In UTF-16, the 16-bit code units can be stored in either big-endian or little-endian order. The encoding forms are UTF-16BE (big-endian) and UTF-16LE (little-endian), or a UTF-16 with a leading Byte Order Mark (BOM) U+FEFF that signals endianness. The surrogate pair mechanism uses a high surrogate in the range 0xD800–0xDBFF followed by a low surrogate in 0xDC00–0xDFFF; together they encode code points from U+10000 to U+10FFFF. The single-unit range U+0000–U+D7FF and U+E000–U+FFFF encodes directly, while the surrogate values themselves are not valid scalar values.

UTF-16 is used by several major systems and languages. Windows APIs commonly operate on UTF-16 strings; Java represents strings as sequences of UTF-16 code units, with surrogate pairs handling non-BMP characters. UTF-16 provides a compromise between fixed-width encodings and UTF-8 in terms of space efficiency for BMP scripts, but requires careful handling of endianness and surrogate pairs to ensure correct processing and data interchange, especially when interfacing with UTF-8 or ASCII-based systems.