RSAESOAEP

RSAES-OAEP, or RSA Encryption Scheme with Optimal Asymmetric Encryption Padding, is a padding scheme for RSA encryption that adds randomness and structure to plaintext before RSA is applied. It is defined in the PKCS #1 standard (and in RFC 8017) and is widely used to provide semantic security against adaptive chosen-ciphertext attacks when RSA is used for encryption. The scheme relies on a hash function and a mask generation function (MGF1) to produce an encoded message of the same length as the RSA modulus.

To encode, let k be the modulus length in octets and hLen the hash output length. OAEP uses a label L (often empty) and computes lHash = Hash(L). The data block DB is lHash || PS || 0x01 || M, where PS is a string of zero octets chosen to fit length constraints and M is the plaintext. A random seed of length hLen is generated. A seedMask is obtained by applying MGF1 to DB, and a dataMask by applying MGF1 to the seed. The maskedSeed = seed XOR seedMask and maskedDB = DB XOR dataMask. The encoded message EM is 0x00 || maskedSeed || maskedDB and is then RSA-encrypted with the public key to produce the ciphertext c. The message length must satisfy mLen <= k - 2*hLen - 2; otherwise encoding fails.

Decryption reverses this process: RSA decrypts to EM, which is split into maskedSeed and maskedDB; the seeds and data blocks are recovered by XOR with the corresponding masks, and M is extracted after verifying that lHash matches and that the 0x01 separator is present. OAEP security rests on the unpredictability of the seed and the properties of MGF1 with a strong hash, giving resistance to chosen-plaintext and chosen-ciphertext attacks in the appropriate model.

OAEP is commonly used with RSA for encryption in protocols and libraries, and is generally preferred over PKCS #1 v1.5 padding for encryption. For signatures, RSASSA-PSS is used instead.