Transformers

Materials:

Date: Saturday, 16-November-2024, 1.30pm, IST.

Pre-work:

• Refresh ML foundations.
• Read “The 100 page ML book” by Andiry Burkov. Chapters accessible here
• FFNs
• CNNs
• KNNs
• RNNs

In-Class

• LLMs @ UPenn Part-1, Part-2
• LLMs: Introduction and Recent Advances @ IIT Delhi Module-5 on RNNs, Module-6 on Attention and Transformers

Lab

• code Attention Mechanism
• code GPT2-like model from scratch

Post-class:

• Attention Is All You Need
• The Illustrated Transformer
• Transformer Explainer
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Layer Normalization
• Build Better Deep Learning Models with Batch and Layer Normalization

Transformer Papers and Advancements

• Transformers are Multi-state RNNs
• Were RNNs All We Needed?
• xLSTM: Extended Long Short-Term Memory
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

Theory

• tbd
• tbd

Additional References

• Saebastian raschka’s course, youtube playlist, Book
• tbd

Notes

tbd