BERTBase

BERTbase is the base configuration of BERT (Bidirectional Encoder Representations from Transformers), a language representation model introduced by Google AI Language in 2018. It uses a bidirectional Transformer encoder to learn contextualized word representations from unlabeled text, enabling transfer learning for a wide range of natural language processing tasks. BERTbase is one of the two primary model sizes released with BERT, the other being BERT-large.

Model architecture and parameters: BERTbase consists of 12 Transformer encoder layers, a hidden size of 768, and 12 self-attention heads, totaling about 110 million parameters. It uses a WordPiece vocabulary of 30,000 tokens and is available in uncased and cased variants.

Pretraining: The model is pretrained on large unlabeled text using two objectives: Masked Language Modeling (MLM), where a fraction of input tokens are masked and the model must predict them, and Next Sentence Prediction (NSP), where the model predicts whether two sentences follow each other. The pretraining corpus primarily includes English Wikipedia and the BooksCorpus dataset, enabling broad linguistic knowledge.

Fine-tuning and use: After pretraining, BERTbase can be fine-tuned with task-specific layers on downstream NLP tasks such as classification, question answering, and named entity recognition. It has become widely adopted in research and industry and is available in common deep learning frameworks and libraries, often exposed as "bert-base-uncased" or "bert-base-cased" variants in platforms like TensorFlow, PyTorch, and HuggingFace Transformers.

Comparison to BERT-large: The base model is smaller and faster to run than BERT-large, which uses 24 layers, 1024 hidden units, 16 attention heads, and about 340 million parameters. The base model trades some raw performance for lower computational requirements, making it suitable for resource-constrained settings.