REQ: Kandenze Creative Applications of Deep Learning with TensorFlow III (incl. Google Brain's Magenta MIDI)
This course extends our existing background in Deep Learning to state of the art techniques in audio, image and text modeling. We'll see how dilated convolutions can be used to model long term temporal dependencies efficiently using a model called WaveNet. We'll also see how to inspect the representations in deep networks using a deep generator network, leading to some of the strongest insights into deep networks and the representations they learn. We'll then switch gears to one of the most exciting directions in Deep Learning thus far: Reinforcement Learning. We'll take a brief tour of this fascinating topic and explore toolkits released by OpenAI, DeepMind, and Microsoft. Finally, we're teaming up with Google Brain's Magenta Lab for our last session on Music and Art Generation. We'll explore Magenta's libraries using RNNs and Reinforcement Learning to create generative and improvised music.
Session 1: Modeling Music And Art: Google Brain’s Magenta Lab
We're teaming up with the Google Brain lab, Magenta to explore the generative creation of Music and Art! We'll explore their libraries which use RNNs and Reinforcement Learning to compose, generate, improvise, and even create duets of music.
1. Introduction to Magenta w/ Douglas Eck
2. Magenta Installation
3. MIDI Setup
4. Introduction to MIDI w/ Adam Roberts
5. Melody RNN: Pre-trained Model w/ Harry Potter
6. MIDI Processing with Magenta
7. Melody RNN: Preprocessing The Legend of Zelda
8. Melody RNN: Training the Legend of Zelda
9. Polyphony RNN: Introduction w/ Curtis Hawthorne
10. Drums and Improv RNN: Introduction w/ Ian Simon
11. Drums RNN: Jam w/ Adam Roberts and Ian Simoq
12. Drums RNN: Setup and Training
13. Magenta MIDI: Introduction w/ Adam Roberts
14. Magenta MIDI: Setup
15. AI Duet
16. Magenta NIPS Demo: Max + Ableton Live Set - Demo
17. Sageev Oore Introduction
18. Magenta Jam: Sageev Oore
19. Magenta Jam w/ Doug Eck, Adam Roberts, and Sageev Oore
20. Closing Thoughts and Homework
Session 2: Modeling Language: Natural Language Processing
This session develops an understanding in natural language processing covering word2vec, glove, seq2seq and attention mechanisms.
1. Introduction
2. Count-Based Methods
3. Modeling Sequences with N-Grams
4. Predict-Based Methods
5. Noise Contrastive Estimation
6. Word2Vec Implementation and Considerations
7. GloVe: Global Vectors - Overview
8. GloVe: Global Vectors - Pre-Trained Model Exploration
9. RNN Language Model: Seq2Seq
10. Seq2Seq: Overview
11. Seq2Seq: Special Tokens, Buckets, Dynamic Unrolling
12. Seq2Seq: Training Data
13. Seq2Seq: Preprocessing w/NLTK Part I
14. Seq2Seq: Preprocessing w/NLTK Part II
15. Seq2Seq: Making Training Pairs
16. Dynamic RNN Seq2Seq Model w/ Attention
17. Homework
Session 3: Autoregressive Image Modeling W/ PixelCNN
Session 4: Modeling Audio W/ Wavenet And NSynth
1. Introduction to WaveNet
2. Understanding Audio, Samples, and Sample Rates
3. Bit Depth and Mu Law Encoding
4. Dilated Convolution and Receptive Field Sizes
5. A note on Skip Connections, Residual Connections, and Gated Convolution
6. WaveNet Code
7. Fast WaveNet Generation
8. Introduction to Magenta w/Jesse Engel
9. Motivations for NSynth w/ Jesse Engel
10. Introduction to NSynth w/ Jesse Engel
11. NSynth Albeton Live Sampler w/ Jesse Engel
12. NSynth Training Code and AI Experiment Sampler
13. NSynth Pre-trained Model, Encoding and Decoding, and Fast Generation
14. Homework
Some knowledge of basic python programming is assumed, including how to start a python session, working with jupyter (ipython) notebook (for homework submissions), numpy basics including how to manipulate arrays and images, how to draw images with matplotlib, and how to work with files using the os package. You should also have completed the first course in the CADL program before taking this second course.
If a student signs up for the Creative Applications of Deep Learning program, it is recommended that these courses are taken sequentially.
Session 1: Modeling Music And Art: Google Brain’s Magenta Lab
We're teaming up with the Google Brain lab, Magenta to explore the generative creation of Music and Art! We'll explore their libraries which use RNNs and Reinforcement Learning to compose, generate, improvise, and even create duets of music.
1. Introduction to Magenta w/ Douglas Eck
2. Magenta Installation
3. MIDI Setup
4. Introduction to MIDI w/ Adam Roberts
5. Melody RNN: Pre-trained Model w/ Harry Potter
6. MIDI Processing with Magenta
7. Melody RNN: Preprocessing The Legend of Zelda
8. Melody RNN: Training the Legend of Zelda
9. Polyphony RNN: Introduction w/ Curtis Hawthorne
10. Drums and Improv RNN: Introduction w/ Ian Simon
11. Drums RNN: Jam w/ Adam Roberts and Ian Simoq
12. Drums RNN: Setup and Training
13. Magenta MIDI: Introduction w/ Adam Roberts
14. Magenta MIDI: Setup
15. AI Duet
16. Magenta NIPS Demo: Max + Ableton Live Set - Demo
17. Sageev Oore Introduction
18. Magenta Jam: Sageev Oore
19. Magenta Jam w/ Doug Eck, Adam Roberts, and Sageev Oore
20. Closing Thoughts and Homework
Session 2: Modeling Language: Natural Language Processing
This session develops an understanding in natural language processing covering word2vec, glove, seq2seq and attention mechanisms.
1. Introduction
2. Count-Based Methods
3. Modeling Sequences with N-Grams
4. Predict-Based Methods
5. Noise Contrastive Estimation
6. Word2Vec Implementation and Considerations
7. GloVe: Global Vectors - Overview
8. GloVe: Global Vectors - Pre-Trained Model Exploration
9. RNN Language Model: Seq2Seq
10. Seq2Seq: Overview
11. Seq2Seq: Special Tokens, Buckets, Dynamic Unrolling
12. Seq2Seq: Training Data
13. Seq2Seq: Preprocessing w/NLTK Part I
14. Seq2Seq: Preprocessing w/NLTK Part II
15. Seq2Seq: Making Training Pairs
16. Dynamic RNN Seq2Seq Model w/ Attention
17. Homework
Session 3: Autoregressive Image Modeling W/ PixelCNN
Session 4: Modeling Audio W/ Wavenet And NSynth
1. Introduction to WaveNet
2. Understanding Audio, Samples, and Sample Rates
3. Bit Depth and Mu Law Encoding
4. Dilated Convolution and Receptive Field Sizes
5. A note on Skip Connections, Residual Connections, and Gated Convolution
6. WaveNet Code
7. Fast WaveNet Generation
8. Introduction to Magenta w/Jesse Engel
9. Motivations for NSynth w/ Jesse Engel
10. Introduction to NSynth w/ Jesse Engel
11. NSynth Albeton Live Sampler w/ Jesse Engel
12. NSynth Training Code and AI Experiment Sampler
13. NSynth Pre-trained Model, Encoding and Decoding, and Fast Generation
14. Homework
Some knowledge of basic python programming is assumed, including how to start a python session, working with jupyter (ipython) notebook (for homework submissions), numpy basics including how to manipulate arrays and images, how to draw images with matplotlib, and how to work with files using the os package. You should also have completed the first course in the CADL program before taking this second course.
If a student signs up for the Creative Applications of Deep Learning program, it is recommended that these courses are taken sequentially.