BERT

BERT, in the field of natural language processing, refers to Bidirectional Encoder Representations from Transformers, a language representation model developed by Google AI Language and released in 2018. Built on the Transformer architecture, BERT introduces bidirectional context, enabling models to consider both left and right context during pre-training. Its approach combines learning from large unlabeled text with subsequent fine-tuning on specific tasks, reducing the need for extensive task-specific data.

BERT comes in two main sizes: Base and Large. Base has 12 Transformer encoder layers, a hidden size of 768, 12 attention heads and about 110 million parameters; Large has 24 layers, a hidden size of 1024, 16 heads and about 340 million parameters. It uses a WordPiece tokenizer with a vocabulary of roughly 30,000 tokens. Pre-training employs masked language modeling, where tokens are masked and predicted, and next sentence prediction to model sentence relationships. Data sources include English Wikipedia and books corpora, with later variants trained on larger, multilingual or domain-specific corpora.

Fine-tuning tailors the model to downstream tasks by adding task-specific outputs and training on labeled data for applications such as sentiment analysis, question answering and natural language inference. BERT achieved state-of-the-art results on benchmarks including GLUE and SQuAD and inspired a family of transformer-based models, such as RoBERTa and ALBERT, that sought efficiency or performance improvements. Limitations include substantial computational cost for pre-training and inference, sensitivity to domain shifts, and biases present in training data, which have prompted ongoing research into efficiency, robustness and fairness.

BERT is also a masculine given name or nickname in English- and Dutch-speaking regions. It is commonly a short form of names such as Bertram and Robert, and is used independently as well.