BLEU

BLEU, short for Bilingual Evaluation Understudy, is an automatic metric for evaluating machine translation quality by comparing a candidate translation to one or more reference translations. It was introduced by Kishore Papineni and colleagues at IBM in 2002 and has since become a standard tool in MT research and development.

The core idea of BLEU is to measure n-gram precision between the candidate and references, using a modified precision that clips each n-gram count in the candidate to the maximum number of times that n-gram appears in any reference. BLEU typically combines n-gram precisions for n from 1 to 4 using a geometric mean and then applies a brevity penalty to penalize translations that are too short compared with the references. The brevity penalty equals one when the candidate is longer than the references; otherwise it decreases exponentially as the candidate length falls short of the references. The final BLEU score is the product of the brevity penalty and the n-gram precision term, and scores are often reported as percentages.

BLEU can be computed at the corpus level, which tends to be more stable, or at the sentence level, which is more volatile. Results depend on preprocessing choices such as tokenization and handling of punctuation, and are influenced by the number and quality of reference translations used.

Strengths of BLEU include its speed, language-agnostic design, and strong historical use for benchmarking. Limitations include reliance on reference translations, insensitivity to semantic adequacy or fluency beyond n-gram overlap, vulnerability to gaming through length, and the need for smoothing when zero overlaps occur. Variants and extensions, especially BLEU-4, remain common in reporting MT quality.