How transformers work: watch, read, then listen

Q@qa-pagemcp · 1 min read · 8 things · Updated Nov 17

Three videos, three papers and one album to read them by. About six hours, in this order.

1 album · 3 videos

You can understand how ChatGPT-style models work in a weekend, if you take things in the right order. Most people start with the famous paper and bounce off it. Start with pictures, then code, then the paper. By then it reads like a summary.

Watch (about 3 hours)

Read (about 3 hours)

Put the album on first.

Attention Is All You NeedAshish Vaswani et al. · PaperThe dominant sequence transduction models are based on complex recurrent or convolutional neural networks in an encoder-decoder configuration. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. Experiments on two machine translation tasks show these models to be superior in quality while being more parallelizable and requiring significantly less time to train. Our model achieves 28.4 BLEU on the…

The paper. After the videos, sections 3.1 to 3.3 read like a recap. Skip the training details.

The Illustrated Transformerjalammar.github.io

Keep this open in another tab while you read the first paper. Every diagram the paper should have had.

BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingJacob Devlin et al. · PaperWe introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial task-specific architecture modific…

Same parts, used to read instead of write. Figure 1 is the whole idea.

Language Models are Few-Shot LearnersTom B. Brown et al. · PaperRecent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performa…

What happened when it got big. Read the introduction and section 3.9, where it writes news.

What to take away

  • The model only ever does one thing: guess the next word.
  • Attention is how each word looks at the others to work out what it means here.
  • Nothing in the 2017 paper promises a machine you can talk to. Size did that, three years later.

If one of these loses you, it's probably the third video. Go back to the second and watch it again.

Made by @qa-pagemcp

Make yours