So, what is BERT? In simple words, BERT is an architecture that can be used for a lot of downstream tasks such as question answering, Classification, NER etc. One can assume a pre-trained BERT as a black box that provides us with H = 768 shaped vectors for each input token(word) in a sequence.

BERT stands for Bidirectional Encoder Representations from Transformers. We've already discussed how bidirectional pre-training with MLMs enables BERT to function, so let's cover the remaining letters in the acronym to get a better understanding of its architecture.

The artificial intelligence models behind popular chatbots developing faster than Moore's law, a measure of how quickly computer hardware performance increases. That suggests the developers of AI systems, known as large language models (LLMs), are becoming smarter at doing more with less.

LEGAL-BERT: The Muppets straight out of Law School. LEGAL-BERT is a family of BERT models for the legal domain, intended to assist legal NLP research, computational law, and legal technology applications.