chrF

ChrF, short for character F-score, is an automatic evaluation metric used to assess the quality of machine translation and other text generation outputs. It measures the overlap of character n-grams between a system’s hypothesis and one or more reference translations by computing a precision, a recall, and a resulting F-score. By focusing on character sequences rather than words, ChrF is language-agnostic and more robust to morphological variation, typos, and minor spelling differences, making it especially useful for morphologically rich languages.

Calculation is based on character n-grams. A typical configuration uses n-grams from 2 to 6 characters, though other ranges (such as 1 to 6) are also used. For a given hypothesis H and reference R, the number of matching n-grams is counted. Precision is the ratio of matched n-grams to the total n-grams in H, and recall is the ratio to the total n-grams in R. The F-score is computed as (1 + beta^2) * P * R / (beta^2 * P + R), where beta balances the importance of recall versus precision. In practice, ChrF scores for different n-gram orders are often combined, commonly by averaging, yielding a single ChrF value. Variants such as CHRF and CHRF++ exist, with CHRF++ extending the approach by also incorporating word-level n-grams to capture lexical aspects.

ChrF is widely used in machine translation evaluation, including in major conferences and shared tasks, and is implemented in many evaluation toolkits. Its language-agnostic nature and robustness to spacing and morphology make it a popular complement to traditional word-based metrics like BLEU. However, ChrF depends on reference quality and preprocessing choices, and, like all automatic metrics, may not fully align with human judgments in every language pair or domain.