RNNs

Materials:

Date: Saturday, 12-October-2024, 1.30pm, IST.

Pre-work:

• Refresh ML foundations.
• Read “The 100 page ML book” by Andiry Burkov. Chapters accessible here
• FFNs
• CNNs
• KNNs

In-Class

• Chapter 9 from on RNNs from d2l.
• Sebastian Raschka’s lecture on RNNs pdf
• Understanding LSTMs Chris Cola’s Blog
• Lecture 14 on RNNs from from Mitesh Khapra’s CS7015
• Lecture 15 on GRUs, LSTMs, from Mitesh Khapra’s CS7015

Lab

• Sebastican Raschka’s tuotrial blog, RNN Classifier on IMDB notebook
• Sentiment Analysis, with RNNs
• micrograd a no dependency, tiny backprop, with a PyTorch like API, from the legendary Andreaj Karpathy
• BoolGrad backprop on Boolean Computational Graph, based on micrograd.

Post-class:

• GRU Analysis
• Blog - The unreasonable effectiveness of RNNs by Anreaj Karpathy

RNN Papers and Advancements

• Transformers are Multi-state RNNs
• Were RNNs All We Needed?
• xLSTM: Extended Long Short-Term Memory
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention

Theory

• tbd
• tbd

Additional References

• Saebastian Raschk’s course, youtube playlist, Book
• tbd

Notes

tbd