Large-Scale Pre-Trained Language Models

BERT

How and Why Does It Use The Transformer Architecture?

BERT stands for Bidirectional Encoder Representations from Transformers. As the name suggests, it generates representations using an encoder from Vaswani et al.’s Transformer architecture. However, there are notable differences between BERT and the original Transformer, especially in how they train those models.

This article discusses the following:

• Why unsupervised pre-training?
• Masked language model (MLM)
• Next Sentence Prediction (NSP)
• Supervised fine-tuning

Why Unsupervised Pre-Training?

Vaswani et al. employed supervised learning to train the original Transformer models for language translation tasks, which requires pairs of source and target language sentences. For example, a German-to-English translation model needs a training dataset with many German sentences and corresponding English translations. Collecting such text data may involve much work, but we require them to ensure machine translation quality. There is not much else we can do about it, or can we?