Artificial Intelligence Models for Language Processing

BERT (Bidirectional Encoder Representations from Transformers) is a method where you can train data and use this for other NLP (Natural Language Processing) tasks such as machine translation, question answering, information retrieval, and etc. The goal of NLP is to identify human language for the task they need to do. According to GLUE (General Language Understanding Evaluation Benchmark), BERT outperforms other models when it comes to tasks that involve consuming data and using it for NLP tasks. 

    BERT has a process that consists of reading text and using it for other tasks. The first step of the process is called, pre-training where the model consumes text. When Bert was tested, it was fed with BooksCorpus which consists of 800M words and Wikipedia which consists of 2500M words (lists, tables, and headers were ignored). Before the text is consumed, 10-20% of words were masked out. It basically looks like fill in the blanks. This is why BERT is considered a masked language model as a few words have been masked out for training. When it comes to BERT figuring out what word they have to fill, unlike other models that look at its left of the sentence or just the right of the sentence, BERT looks out in both directions. For instance, if the sentence, I went to the library to study, was inserted in the BERT model as I went to the ____ to study, the BERT model would look at the blank from its left side of words and its right side of words. Although pre-training can be expensive, its next step is inexpensive. Fine-tuning is where someone can take a model and use it for similar tasks. In this case, BERT can be used for NLP tasks. Results can be produced within one hour and no pre-training is needed which is beneficial for NLP researchers. 

    NMT (Neural Machine Translation) is a machine translation approach that uses an artificial neural network to predict a sequence of words. A neural network is basically a bunch of algorithms inspired by the human brain. It has been used for many tasks such as Google translate, speech recognition programs, and handwritten translation. In NMT, a large single neural network is trained and sentences are read and a translation is outputted. According to the BLEU score (a score that determines the accuracy of the language conversion of the sentence/sentences), it has been getting better and better with longer sentences as time goes. It was used in tests where words had to be converted from English to French.

    NMT can be applied to many tasks but in this case, we will talk about translating words from one language to another. It starts with the input, a sentence/sentences, which gets processed to a neural network. In this case, the network is called a BiDirectional Recurrent Neural Network which deals with sequences and checks if the output of each word is suitable by checking what comes before that word and what comes after that word. It gets converted to a vector of numbers and inputted into the attention mechanism. Attention mechanism can be represented as a checkered type graph where one axis is the language of the input words and the other axis is the language of the output words. It can see which output word is the best match with the input word, and that is represented by a square where white means super aligned (meaning good to use).  This is very useful when it comes to deciding which word should heavily be focused on in terms of grammar. According to BLEU, NMT can work with more words in the input than it did without the attention mechanism. Once the attention mechanism process is over, the vector of numbers gets decoded into actual words. 

References

Bahdanau, Dzmitry, et al. “Neural Machine Translation by Jointly Learning To Align and Translate.”

Devlin, et al. “BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding.” ArXiv.org, 24 May 2019, https://arxiv.org/abs/1810.04805.

Google-Research. “Google-Research/Bert.” GitHub, 17 Oct. 2019, https://github.com/google-research/bert.

YouTube, 13 Feb. 2019, https://youtu.be/AIpXjFwVdIE.

YouTube, 16 May 2019, https://youtu.be/NwViH5eGt9w.