UnicodeEncoding

Unicode encoding is the method by which Unicode code points are represented as sequences of bytes in memory or on disk. An encoding defines how a code point such as U+1F600 is serialized and subsequently decoded by software. The most common Unicode encodings are UTF-8, UTF-16, and UTF-32, each with different trade-offs for space and processing.

UTF-8 is a variable-length encoding using one to four bytes per code point. It is ASCII-compatible for the first 128 code points, which helps interoperability with existing software. The high bits of each initial byte indicate the total length, and continuation bytes use the pattern 10xxxxxx. UTF-8 is widely used on the Web and in many programming environments.

UTF-16 encodes most characters in 2-byte units but uses surrogate pairs to represent code points above U+FFFF. Code points in the Basic Multilingual Plane map to a single 16-bit unit; supplementary code points use two 16-bit units. Endianness matters; UTF-16 data can be little- or big-endian, and a Byte Order Mark may be used at the start of a stream.

UTF-32 uses fixed 4-byte units, mapping each code point directly to a 32-bit value. This makes processing straightforward but is space-inefficient for typical text, especially for multilingual content.

Adoption and handling: UTF-8 is the de facto standard for the Web; UTF-16 remains common in some platforms and APIs, while UTF-32 appears mainly in internal processing. Encodings should be validated, as invalid sequences may trigger errors or be replaced by a placeholder character. Misinterpretation of encoding can alter content and introduce security risks.