BLEURT

BLEURT is a learned evaluation metric for natural language generation that assigns a continuous, real-valued quality score to a candidate text with respect to a reference. It is designed to better reflect human judgments of translation, summarization, and other NLG outputs than traditional word-overlap metrics.

BLEURT builds on a pre-trained transformer encoder to obtain representations of the candidate and reference text. A regression head processes these representations and outputs a quality score. The model is trained on datasets containing human ratings of text quality across diverse tasks, and training data are augmented with synthetic perturbations to widen coverage of linguistic phenomena, improving robustness to minor edits and lexical variation.

Empirical evaluations report that BLEURT achieves higher correlations with human judgments than BLEU, ROUGE, and METEOR on several benchmarks, particularly for semantic adequacy and fluency. It is used in research to evaluate system outputs and to guide development, including hyperparameter tuning and model comparisons.

Limitations include dependence on the domain and language of the training data, computational cost relative to traditional metrics, and potential biases inherited from the training corpus. The metric and its pretrained models are typically released openly, enabling researchers to fine-tune or apply it to their own tasks.