crossencoder

A cross-encoder is a neural network architecture used to assess the relationship between two text inputs by encoding them jointly in a single forward pass. Unlike bi-encoders, which encode each input separately, a cross-encoder processes the pair together in one transformer, producing a relevance or similarity score.

In a typical setup, the two texts are concatenated with a separator and fed to a transformer such as BERT, RoBERTa, or DeBERTa. The final hidden state corresponding to the [CLS] token is fed through a classifier to yield a relevance probability or a continuous score.

Training uses labeled text pairs with objectives such as binary cross-entropy for relevance or regression for similarity. Both pointwise and pairwise loss formulations are common, and large improvements are often achieved with task-specific fine-tuning on curated data.

Common applications include reranking candidate results in information retrieval, answer selection in question answering, and other text-pair tasks requiring precise judgments of relevance or meaning.

A key trade-off is speed versus accuracy. Because each candidate must be scored by a full transformer on its pair, cross-encoders are computationally expensive at inference. They are typically deployed as a second-stage re-ranker after a fast bi-encoder or traditional retriever.

Variants include multilingual cross-encoders and models optimized for longer inputs. Practical use often requires truncating long texts and tuning batch sizes to fit hardware constraints.