meteor

METEOR is an automatic evaluation metric used to assess the quality of machine translation outputs. Introduced in 2005 by Banerjee and Lavie, the name stands for Metric for Evaluation of Translation with Explicit Ordering. METEOR differentiates itself from some n-gram–based metrics by using a flexible word matching and alignment process between a candidate translation and one or more reference translations.

The core method aligns unigrams (words) from the candidate to the references via exact matches, stemming, synonymy, and paraphrase matches, often leveraging lexical resources such as WordNet. From these alignments, METEOR computes precision and recall of the matched unigrams and combines them into a harmonic mean (a form of F-score) with a tunable parameter that can weight recall versus precision. A fragmentation penalty is applied to reflect how contiguous or dispersed the matched segments are; greater fragmentation lowers the final score. The typical final METEOR score is a function of the F-score and the penalty, yielding a value between 0 and 1.

METEOR can operate with a single reference or multiple references, in which case the maximum score across references is usually reported. Over time, variants and language-specific modules have been developed to improve alignment quality and applicability to different languages and domains.

In practice, METEOR is widely used in machine translation research and development as a supplementary metric to BLEU and others. It tends to correlate better with human judgments on certain language pairs, particularly for morphologically rich languages, due to its emphasis on synonymy, paraphrase, and flexible word matching. However, it is more computationally intensive and relies on lexical resources beyond simple surface forms.