1<!DOCTYPE html> 2<html> 3 <head> 4 <meta charset="utf-8"> 5 <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1"> 6 <title>Learning to generate lyrics and music with Recurrent Neural Networks</title> 7 <meta name="viewport" content="width=device-width, initial-scale=1"> 8 <meta name="description" content="Blog about Machine Learning and Computer Vision. Google Summer of Code blog posts. Scikit-image face detection algorithm implementation. 9"> 10 <link rel="canonical" 11 href="http://warmspringwinds.github.io/pytorch/rnns/2018/01/27/learning-to-generate-lyrics-and-music-with-recurrent-neural-networks/"> 12 13 <!-- Mathjax--> 14
14<script type="text/javascript" src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
14 15 16 <!-- Harmony styles --> 17 <link rel="stylesheet" type="text/css" href="/assets/css/main.css"> 18 19 <!-- Modernizr js --> 20
20<script async src="/assets/js/modernizr.js"></script>
20 21 22 <!-- IE Fixes --> 23 <!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries --> 24 <!-- WARNING: Respond.js doesn't work if you view the page via file:// --> 25 <!--[if lt IE 9]> 26
26<script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
26 27
27<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
27 28 29 <![endif]--> 30 </head> 31 <body class="theme-base-01"> 32 <header class="main-header"> 33 <div class="wc-container"> 34 <h1><a href="/">Daniil's blog</a></h1> 35 <h2>Machine Learning and Computer Vision artisan.</h2> 36 <ul> 37 <li> 38 <a href="http://warmspringwinds.github.io/about">About</a><span>/</span> 39 </li> 40 <li> 41 <a href="http://warmspringwinds.github.io/blog">Blog</a><span>/</span> 42 </li> 43</ul> 44 </div> 45 </header> 46 <div class="page-content wc-container"> 47 48 <div class="post"> 49 <h1>Learning to generate lyrics and music with Recurrent Neural Networks</h1> 50 <p class="post-meta"> 51 52 <span class="categories"> 53 Pytorch and rnns 54 </span> | 55 56 <span class="post-date"> 57 Jan 27, 2018 58 </span> 59 </p> 60 <div class="post"> 61 <p>A post showing an application of RNN-based generative models for lyrics and piano music generation.</p> 62 63<hr /> 64 65<h3 id="introduction">Introduction</h3> 66 67<p>In this post we will train RNN character-level language model on lyrics dataset of 68most popular/recent artists. Having a trained model, we will sample a couple of 69songs which will be a funny mixture of different styles of different artists. 70After that we will update our model to become a conditional character-level RNN, 71making it possible for us to sample songs conditioned on artist. 72And finally, we conclude by training our model on midi dataset of piano songs. 73While solving all these tasks, we will briefly explore some interesting concepts related to RNN 74training and inference like character-level RNN, conditional character-level RNN, 75sampling from RNN, truncated backpropagation through time and gradient checkpointing. All the code and 76trained models are <a href="https://github.com/warmspringwinds/pytorch-rnn-sequence-generation-classification">available on github</a> and were implemented in <a href="http://pytorch.org/">Pytorch</a>. The blog post can also <a href="https://github.com/warmspringwinds/pytorch-rnn-sequence-generation-classification/blob/master/blog_post.ipynb">be viewed in a jupyter notebook format</a>. If you are already familiar with the character-level language model and recurrent neural networks, feel free to skip respective sections or go directly to the results section.</p> 77 78<h3 id="character-level-language-model">Character-Level language model</h3> 79 80<p><img src="http://warmspringwinds.github.io/assets/img/character_level_model.jpg" alt="png" /></p> 81 82<p>Before choosing a model, letâs have a closer look at our task. Given current letter and all previous 83letters, we will try to predict the next character. During training we will just take a sequence, and use 84all its characters except the last one as an input and the same sequence starting from the second character as groundtruth (see the picture above; <a href="https://github.com/spro/practical-pytorch/blob/master/conditional-char-rnn/conditional-char-rnn.ipynb">Source</a>). We will start from the simplest model that ignores all the previous characters while making a prediction, improve this model to make it take only a certain number of previous characters into account, and conclude with a model that takes all the previous characters into consideration while making a prediction.</p> 85 86<p>Our language model is defined on a character level. We will create a dictionary which will contain 87all English characters plus some special symbols, like period, comma, and end-of-line symbol. Each charecter will be represented as one-hot-encoded tensor. For more information about character-level models and examples, I recommend <a href="https://github.com/spro/practical-pytorch">this resource</a>.</p> 88 89<p>Having characters, we can now form sequences of characters. We can generate sentences even now just by 90randomly sampling character after character with a fixed probability
90<script type="math/tex">p(any~letter)=\frac{1}{dictionary~size}</script>
90. 91Thatâs the most simple character level language model. Can we do better than this? Yes, we can compute the probabily of occurance of each letter from our training corpus (number of times a letter occures divided by the size of our dataset) and randomly sample letter using these probabilities. This model is better but it totally ignores the relative positional aspect of each letter. For example, pay attention on how you read any word: you start with the first letter, which is usually hard to predict, but as you reach the end of a word you can sometimes guess the next letter. When you read any word you are implicitly using some rules which you learned by reading other texts: for example, with each additional letter that you read from a word, the probability of a space character increases (really long words are rare) or the probability of any consonant after the letter ârâ is low as it usually followed by vowel. There are lot of similar rules and we hope that our model will be able to learn them from data. To give our model a chance to learn these rules we need to extend it.</p> 92 93<p>Letâs make a small gradual improvement of our model and let probability of each letter depend 94only on the previously occured letter (<a href="https://en.wikipedia.org/wiki/Markov_property">markov assumption</a>). So, basically we will have
94<script type="math/tex">p(current~letter|previous~letter)</script>
94. 95This is a <a href="https://en.wikipedia.org/wiki/Markov_chain">Markov chain model</a> (also try these <a href="http://setosa.io/ev/markov-chains/">interactive visualizations</a> if you are not familiar with it). We can also estimate the probability distribution
95<script type="math/tex">p(current~letter|previous~letter)</script>
95 from our training dataset. This model is limited because in most cases the probability of the current letter depends not only on the previous letter.</p> 96 97<p>What we would like to model is actually
97<script type="math/tex">p(current~letter|all~previous~letters)</script>
97. At first, the task seems intractable as the number of previous letters is variable and it might become really large in case of long 98sequences. Turns out Reccurent Neural Netoworks can tackle this problem to a certain extent by using shared weights and fixed size hidden state. This leads us to a next section dedicated to RNNs.</p> 99 100<h3 id="recurrent-neural-networks">Recurrent Neural Networks</h3> 101 102<p><img src="http://warmspringwinds.github.io/assets/img/rnn_unfold.jpg" alt="png" /></p> 103 104<p>Recurrent neural networks are a family of neural networks for processing sequential data. 105Unlike feedforward neural networks, RNNs can use their internal memory to process arbitrary sequences of inputs. 106Because of arbitrary size input sequences, they are concisely depicted as a graph with a cycle (see the picture; <a href="http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/">Source</a>). 107But they can be âunfoldedâ if the size of input sequence is known. They define a non-linear mapping from a current input
107<script type="math/tex">x_t</script>
107 and previous hidden state
107<script type="math/tex">s_{t-1}</script>
107 to the output
107<script type="math/tex">o_t</script>
107 and current hidden state
107<script type="math/tex">s_t</script>
107. Hidden state size has a predefined size and stores features which are updated on each step and affect the result of mapping.</p> 108 109<p>Now align the previous picture of the character-level language model and the ufolded RNN picture to see how 110we are using the RNN model to learn a character level language model.</p> 111 112<p>While the picture depicts the Vanilla RNN, we will use LSTM in our work as it is easier to train usually achieves better results.</p> 113 114<p>For a more elaborate introduction to RNNs, we refer reader to the <a href="http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/">following resource</a>.</p> 115 116<h3 id="lyrics-dataset">Lyrics dataset</h3> 117 118<p>For our experiments we have chosen <a href="https://www.kaggle.com/mousehead/songlyrics">55000+ Song Lyrics Kaggle dataset</a> which contains good variety of recent artists and more older ones. It is stored as a pandas file and we wrote a python wrapper around it to be able to use it for training purposes. You will have to download it yourself in order to be able to use our code.</p> 119 120<p>In order to be able to interpret the results better, I have chosen a subset of artists which I am 121more or less familiar with:</p> 122 123<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">artists</span> <span class="o">=</span> <span class="p">[</span> 124<span class="s">'ABBA'</span><span class="p">,</span> 125<span class="s">'Ace Of Base'</span><span class="p">,</span> 126<span class="s">'Aerosmith'</span><span class="p">,</span> 127<span class="s">'Avril Lavigne'</span><span class="p">,</span> 128<span class="s">'Backstreet Boys'</span><span class="p">,</span> 129<span class="s">'Bob Marley'</span><span class="p">,</span> 130<span class="s">'Bon Jovi'</span><span class="p">,</span> 131<span class="s">'Britney Spears'</span><span class="p">,</span> 132<span class="s">'Bruno Mars'</span><span class="p">,</span> 133<span class="s">'Coldplay'</span><span class="p">,</span> 134<span class="s">'Def Leppard'</span><span class="p">,</span> 135<span class="s">'Depeche Mode'</span><span class="p">,</span> 136<span class="s">'Ed Sheeran'</span><span class="p">,</span> 137<span class="s">'Elton John'</span><span class="p">,</span> 138<span class="s">'Elvis Presley'</span><span class="p">,</span> 139<span class="s">'Eminem'</span><span class="p">,</span> 140<span class="s">'Enrique Iglesias'</span><span class="p">,</span> 141<span class="s">'Evanescence'</span><span class="p">,</span> 142<span class="s">'Fall Out Boy'</span><span class="p">,</span> 143<span class="s">'Foo Fighters'</span><span class="p">,</span> 144<span class="s">'Green Day'</span><span class="p">,</span> 145 <span class="s">'HIM'</span><span class="p">,</span> 146 <span class="s">'Imagine Dragons'</span><span class="p">,</span> 147 <span class="s">'Incubus'</span><span class="p">,</span> 148 <span class="s">'Jimi Hendrix'</span><span class="p">,</span> 149 <span class="s">'Justin Bieber'</span><span class="p">,</span> 150 <span class="s">'Justin Timberlake'</span><span class="p">,</span> 151<span class="s">'Kanye West'</span><span class="p">,</span> 152 <span class="s">'Katy Perry'</span><span class="p">,</span> 153 <span class="s">'The Killers'</span><span class="p">,</span> 154 <span class="s">'Kiss'</span><span class="p">,</span> 155 <span class="s">'Lady Gaga'</span><span class="p">,</span> 156 <span class="s">'Lana Del Rey'</span><span class="p">,</span> 157 <span class="s">'Linkin Park'</span><span class="p">,</span> 158 <span class="s">'Madonna'</span><span class="p">,</span> 159 <span class="s">'Marilyn Manson'</span><span class="p">,</span> 160 <span class="s">'Maroon 5'</span><span class="p">,</span> 161 <span class="s">'Metallica'</span><span class="p">,</span> 162 <span class="s">'Michael Bolton'</span><span class="p">,</span> 163 <span class="s">'Michael Jackson'</span><span class="p">,</span> 164 <span class="s">'Miley Cyrus'</span><span class="p">,</span> 165 <span class="s">'Nickelback'</span><span class="p">,</span> 166 <span class="s">'Nightwish'</span><span class="p">,</span> 167 <span class="s">'Nirvana'</span><span class="p">,</span> 168 <span class="s">'Oasis'</span><span class="p">,</span> 169 <span class="s">
169'Offspring'</span><span class="p">,</span> 170 <span class="s">'One Direction'</span><span class="p">,</span> 171 <span class="s">'Ozzy Osbourne'</span><span class="p">,</span> 172 <span class="s">'P!nk'</span><span class="p">,</span> 173 <span class="s">'Queen'</span><span class="p">,</span> 174 <span class="s">'Radiohead'</span><span class="p">,</span> 175 <span class="s">'Red Hot Chili Peppers'</span><span class="p">,</span> 176 <span class="s">'Rihanna'</span><span class="p">,</span> 177 <span class="s">'Robbie Williams'</span><span class="p">,</span> 178 <span class="s">'Rolling Stones'</span><span class="p">,</span> 179 <span class="s">'Roxette'</span><span class="p">,</span> 180 <span class="s">'Scorpions'</span><span class="p">,</span> 181 <span class="s">'Snoop Dogg'</span><span class="p">,</span> 182 <span class="s">'Sting'</span><span class="p">,</span> 183 <span class="s">'The Script'</span><span class="p">,</span> 184 <span class="s">'U2'</span><span class="p">,</span> 185 <span class="s">'Weezer'</span><span class="p">,</span> 186 <span class="s">'Yellowcard'</span><span class="p">,</span> 187 <span class="s">'ZZ Top'</span><span class="p">]</span> 188</code></pre></div></div> 189 190<h3 id="training-unconditional-character-level-language-model">Training unconditional character-level language model</h3> 191 192<p>Our first experiment consisted of training of our character-level language model RNN 193on the whole corpus. We didnât take into consideration the artist information while training.</p> 194 195<h3 id="sampling-from-rnn">Sampling from RNN</h3> 196 197<p>Letâs try to sample a couple of songs after training our model. Basically, on each 198step our RNN will output logits and we can softmax them and sample from that distribution. 199Or we can use Gumble-Max trick and <a href="https://hips.seas.harvard.edu/blog/2013/04/06/the-gumbel-max-trick-for-discrete-distributions/">sample using logits directly</a> which is equivalent.</p> 200 201<p>One intersting thing about sampling is that we can partially define the input sequence ourselves and start sampling 202with that initial condition. For example, we can sample a song that starts with âWhyâ:</p> 203 204<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Why do you have to leave me? 205I think I know I'm not the only one 206I don't know if I'm gonna stay awake 207I don't know why I go along 208 209I don't know why I can't go on 210I don't know why I don't know 211I don't know why I don't know 212I don't know why I keep on dreaming of you 213</code></pre></div></div> 214 215<p>Well, that sounds like a possible song :D</p> 216 217<p>Letâs sample with a song that starts with âWellâ:</p> 218 219<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Well, I was a real good time 220I was a rolling stone 221I was a rock and roller 222Well, I never had a rock and roll 223There were times I had to do it 224I had a feeling that I was found 225I was the one who had to go 226</code></pre></div></div> 227 228<p>There is âtemperatureâ parameter that is used during sampling which controls the randomness of sampling 229process. When this parameter approaches zero, 230the sampling is equivalent to argmax and when it is close to infinity the sampling is equivalent to sampling 231from a uniform distribution. Have a look at the figure from a <a href="https://arxiv.org/pdf/1611.01144.pdf">relevant paper by Jang et al.</a>:</p> 232 233<p><img src="http://warmspringwinds.github.io/assets/img/sampling_temperature.png" alt="png" /></p> 234 235<p>When
235<script type="math/tex">\tau=1</script>
235, the distribution is not affected. If we decrease
235<script type="math/tex">\tau</script>
235, the distribution 236becomes more pronounced, meaning that value with bigger probability mass will have it increased. When
236<script type="math/tex">\tau</script>
236 will approach zero, sampling will be equivalent to armax, because the probability of that value will be close to one. When we start to icrease
236<script type="math/tex">\tau</script>
236 the distribution becomes more and more uniform.</p> 237 238<p>The previous sample was generated with a temperature paramter equal to
238<script type="math/tex">0.5</script>
238. 239Letâs see what happens when we increase it to
239<script type="math/tex">1.0</script>
239 and sample:</p> 240 241<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Why can't we drop out of time? 242We were born for words to see. 243Won't you love this. You're still so amazing. 244This could be that down on Sunday Time. 245Oh, Caroline, a lady floor. 246I thought of love, oh baby. 247</code></pre></div></div> 248 249<p>Letâs try increasing it even more:</p> 250 251<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Why - won't we grow up naked? 252We went quietly what we would've still give 253That girl you walked before our bedroom room 254I see your mind is so small to a freak 255Stretching for a cold white-heart of crashing 256Truth in the universal daughter 257 258I lose more and more hard 259I love you anytime at all 260Ah come let your help remind me 261Now I've wanted waste and never noticed 262 263I swear I saw you today 264You needed to get by 265But you sold a hurricane 266Well out whispered in store 267</code></pre></div></div> 268 269<p>Why donât we grow up naked, indeed? :D 270Well, you can see that trend that when we increase the temperature, sampled 271sentences become more and more random.</p> 272 273<h3 id="training-conditional-character-level-language-model">Training conditional character-level language model</h3> 274 275<p>Imagine if we could generate lyrics in a style of some particular artist. 276Letâs change our model, so that it can use this information during training.</p> 277 278<p>We will do this by adding an additional input to our RNN. So far, our RNN model 279was only accepting tensors containing one-hot encoded character on each step.</p> 280 281<p>The extention to our model will be very simple: we will have and additional one-hot encoded 282tensor which will represent the artist. So on each step the RNN will accept one tensor which will consist of concatenated tensors representing character and artist. Look <a href="https://github.com/spro/practical-pytorch/blob/master/conditional-char-rnn/conditional-char-rnn.ipynb">here for more</a>.</p> 283 284<h3 id="sampling-from-conditional-language-model-rnn">Sampling from conditional language model RNN</h3> 285 286<p>After training, we sampled a couple of songs conditined on artist. 287Below you can find some results.</p> 288 289<p>Him:</p> 290 291<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>My fears 292And the moment don't make me sing 293So free from you 294The pain you love me yeah 295 296Whatever caused the warmth 297You smile you're happy 298You sit away 299You say it's all in vain 300</code></pre></div></div> 301 302<p>Seems really possible, especially the fact the the word pain was used, which is 303very common in the lyrics of the artist.</p> 304 305<p>ABBA:</p> 306 307<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Oh, my love it makes me close a thing 308You've been heard, I must have waited 309I hear you 310So I say 311Thank you for the music, that makes me cry 312 313And you moving my bad as me, ah-hang wind in the hell 314I was meant to be with you, I'll never be playing up 315</code></pre></div></div> 316 317<p>Bob Marley:</p> 318 319<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Mercy on judgment, we got so much 320 321Alcohol, cry, cry, cry 322Why don't try to find our own 323I want to know, Lord, I wanna give you 324Just saving it, learned 325Is there any more? 326 327All that damage done 328That's all reason, don't worry 329Need a hammer 330I need you more and more 331</code></pre></div></div> 332 333<p>Coldplay:</p> 334 335<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Look at the stars 336Into life matter where you lay 337Saying no doubt 338I don't want to fly 339In my dreams and fight today 340 341I will fall for you 342 343All I know 344And I want you to stay 345Into the night 346 347I want to live waiting 348With my love and always 349Have I wouldn't wasted 350Would it hurt you 351</code></pre></div></div> 352 353<p>Kanye West:</p> 354 355<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>I'm everywhere for you 356The way that it couldn't stop 357I mean it too late and love I made in the world 358I told you so I took the studs full cold-stop 359The hardest stressed growin' 360The hustler raisin' on my tears 361I know I'm true, one of your love 362</code></pre></div></div> 363 364<p>Looks pretty cool but keep in mind that we didnât track the validation accuracy so some sampled lines could have been just memorized by our rnn. A better way to do it is to pick a model that gives best validation score during training (see the code for the next section where we performed training this way). We also noticed one interesting thing: the unconditional 365model usually performes better when you want to sample with a specified starting string.
366Our intuition is that when sampling from a conditional model with a specified starting string, 367we actually put two conditions on our model â starting string and an artist compared to the one condition 368in the case of previous model that we explored. And we didnât have enough data to model that conditional 369distribution well (every artist has relatively limited number of songs).</p> 370 371<p>We are making the code and models available and you can sample songs from our trained models 372even without gpu as it is not really computationally demanding.</p> 373 374<h3 id="midi-dataset">Midi dataset</h3> 375 376<p>Next, we will work with a <a href="http://www-etud.iro.umontreal.ca/~boulanni/icml2012">small midi dataset</a> consisting 377of approximately
377<script type="math/tex">700</script>
377 piano songs. We have used the <code class="highlighter-rouge">Nottingam</code> piano dataset (training split only).</p> 378 379<p>Turns out that any midi file can be <a href="http://nbviewer.jupyter.org/github/craffel/pretty-midi/blob/master/Tutorial.ipynb">converted to piano roll</a> which is just is a time-frequency matrix where each row is a different MIDI pitch and each column is a different slice in time. So each piano song from our dataset will be represented as a matrix of size
379<script type="math/tex">88\times song\_length</script>
379, where
379<script type="math/tex">88</script>
379 is a number of pitches of the piano. Here is an example of 380piano roll matrix:</p> 381 382<p><img src="http://warmspringwinds.github.io/assets/img/piano_roll_2.png" alt="png" /></p> 383 384<p>This representation is very intuitive and easy to interpret even for a person that is not familiar 385with music theory. Each row represents a pitch: top rows represent low frequency pitches and bottom 386rows represent high pitches. Plus, we have a horizontal axis which represents time. So if we play a sound 387with a certain pitch for a certian period of time, we will see a horizontal line. Overall, this is very 388similar to <a href=". 392For music character multiple keys can be pressed at one timestep (since we are working with polyphonic dataset). 393In this case, each timestep will be represented by a vector which can contain more than one
393<script type="math/tex">1</script>
393.</p> 394 395<h3 id="training-pitch-level-piano-music-model">Training pitch-level piano music model</h3> 396 397<p>Before starting the training, we will have to adjust our loss that we have used for language model 398to account for different input that we discussed in the previous section. In the language model, 399we had one-hot encoded tensor (character) as an input on each timestep and one-hot encoded tensor as output (predicted next character). As we had to make a single exlusive choice for predicted next character, we used 400<a href="https://rdipietro.github.io/friendly-intro-to-cross-entropy-loss/">cross-entropy loss</a>.</p> 401 402<p>But now our model outputs a vector which is no longer one-hot encoded (multiple keys can be pressed). Of course, we can treat all possible combinations of pressed keys as a separate class, but this is intractable. Instead, we will treat each element of the output vector as a binary variable (
402<script type="math/tex">1</script>
402 â pressing,
402<script type="math/tex">0</script>
402 â not pressing a key). We will define a separate loss 403for each element of the output vector to be binary cross-entropy. And our final loss will be an averaged sum of these binary cross-entropies. You can also read the code to get a better understanding.</p> 404 405<p>After making the aforementioned changes, we trained our model. In the next section, we will perform sampling 406and inspect the results.</p> 407 408<h3 id="sampling-from-pitch-level-rnn">Sampling from pitch-level RNN</h3> 409 410<p>We have sampled piano rolls during the early stages of optimization:</p> 411 412<p><img src="http://warmspringwinds.github.io/assets/img/piano_roll_early_sample.png" alt="png" /></p> 413 414<p>You can see that our model is starting to learn one common pattern that is common among the songs from 415our dataset: each song consists of two different parts. First part contains a sequence of pitches that are played separately and are very <a href="https://www.didjshop.com/BasicMusicalHarmony.html">distinguishable and are often singable</a> (also know as melody). If you look at the sampled piano roll, this part can be clearly seen in the bottom. If you also have a look at the top of our piano roll, we can see a group of pitches that are usually played together â this is harmony or a progression of chords (pitches that are played together throughout the song) which accompanies the melody.</p> 416 417<p>By the end of the training samples drawn from our model started to look like this:</p> 418 419<p><img src="http://warmspringwinds.github.io/assets/img/piano_roll_late_sample.png" alt="png" /></p> 420 421<p>As you can see they started to look more similar to the picture of the ground-truth piano roll that we showed 422in the previous sections.</p> 423 424<p>After training, we have sampled songs and analyzed them. We got one sample with <a href="https://www.youtube.com/watch?v=Iz8xQou2OqA">an interesting introduction</a>. While another sample features <a href="https://www.youtube.com/watch?v=fUdsWVIOeeU&feature=youtu.be&t=15s">a nice style transition</a>. At the same time we generated a couple of examples with low temperature parameter which resulted in songs with a slow tempo: <a href="https://www.youtube.com/watch?v=UoLyeauBsNk">first one</a> and a <a href="https://www.youtube.com/watch?v=Iz8xQou2OqA">second one here</a>. 425You can find the whole playlist <a href="https://www.youtube.com/watch?v=EOQQOQYvGnw&list=PLJkMX36nfYD000TG-T59hmEgJ3ojkOlBp">here</a>.</p> 426 427<h3 id="sequence-length-and-related-problems">Sequence length and related problems</h3> 428 429<p>Now letâs look at our problem from the gpu memory consumption and speed point of view.</p> 430 431<p>We greatly speed up computation by processing our sequences in batches. At the same time, as 432our sequences become longer (depending on the dataset), our max batch size starts to decrease. 433Why is it a case? As we use backpropagation to compute gradients, we need to store all the intermediate acitvations, which contribute the most to the memory consumption. As our sequence becomes longer, we need to store more activations, therefore, we can fit less examples in our batch.</p> 434 435<p>Sometimes, we either have to work with really long sequences or we want to increase our batch size or maybe you just have a gpu with small amount of memory available. There are multiple possible solutions to reduce memory
436consumption in this case, but we will mention two, which will have different trade-offs.</p> 437 438<p>First one is a <a href="https://www.quora.com/Whats-the-key-difference-between-backprop-and-truncated-backprop-through-time#sLRGO">truncated back propagation</a>. The idea is to split the whole sequence into subsequences and treat 439them as separate batches with an exception that we process these batches in the order of split and every next batch uses hidden state of previous batch as an initial hidden state. We also provide an implementation of this approach, so that you can get the better understanding. This approach is obviously not an exact equivalent of processing the whole sequence but it makes more frequent updates and consumes less memory. On the other hand, there is a chance that we might not be able to capture long-term dependencies that span beyond the length of one subsequence.</p> 440 441<p>Second one is <a href="https://medium.com/@yaroslavvb/fitting-larger-networks-into-memory-583e3c758ff9">gradient checkpointing</a>. This method gives us a possibilty to use less memory while training our model on the whole sequence on the expence of performing more computation. If you recall, previously we mentioned that the most memory during 442training is occupied by activations. The idea of gradient checkpointing consists of storing only every
442<script type="math/tex">n</script>
442-th activation and recomputing the unsaved activations later. This method is already <a href="https://github.com/openai/gradient-checkpointing">implemented in Tensorflow</a> and <a href="https://github.com/pytorch/pytorch/pull/4594">being implemented in Pytorch</a>.</p> 443 444<h3 id="conclusion-and-future-work">Conclusion and future work</h3> 445 446<p>In our work we trained simple generative model for text, extended our model to work with 447polyphonic music, briefly looked at how sampling works and how the temperature parameter affects our 448text and music samples â low temperature gives more stable results while high temperature adds more 449randomness which sometimes gives rise to very interesting samples.</p> 450 451<p>Future work can include two directions â more applications or deeper analysis of the already trained models. 452Same models can be applied to your spotify listening history, for example. After training on your 453listening history data, you can give it a sequence of songs that you have listened to in the previous hour or so, and it will sample a playlist for you for the rest of the day. Well, you can also do the same for your browsing history, which will be just a cool tool to analyze your browsing behaviour patterns. <a href="https://www.kaggle.com/uciml/human-activity-recognition-with-smartphones">Capture the accelerometer and gyroscope data</a> from your phone while doing different activities (exercising in the gym, working in the office, sleeping) and learn to classify these 454activity stages. After that you can change your music playlist automatically, based on your activity (sleeping â calm music of rain, exercising in the gym â high intensity music). In terms of medical applications, model can 455be applied to detect heart problems based on pulse and other data, similar to <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5391725/">this work</a>.</p> 456 457<p>It would be very interesting to analyze the neuron firings in our RNN trained 458for music generation like <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/">here</a>. To see if the model learned some simple music concepts implicitly (like our discussion 459of harmony and melody). The hidden representation of RNN can be used to cluster our music dataset to 460find similar songs.</p> 461 462<p>Letâs sample one last lyrics from our unconditional model to conclude this post :D :</p> 463 464<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>The story ends 465The sound of the blue 466The tears were shining 467The story of my life 468I still believe 469The story of my life 470</code></pre></div></div> 471 472 </div> 473 474 <!-- Discus comments feature --> 475 476 477 478 <div id="disqus_thread"></div> 479
479<script type="text/javascript"> 480 /* * * CONFIGURATION VARIABLES * * */ 481 var disqus_shortname = 'daniil-ml-cv-blog'; 482 483 /* * * DON'T EDIT BELOW THIS LINE * * */ 484 (function() { 485 var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true; 486 dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js'; 487 (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq); 488 })(); 489 </script>
489 490 <noscript>Please enable JavaScript to view the <a href="https://disqus.com/?ref_noscript" rel="nofollow">comments powered by Disqus.</a></noscript> 491 492 493 </div> 494 495 496 497 <div class="related"> 498 <h4>Related Posts</h2> 499 <ul class="posts"> 500 501 <li> 502 <span>23 Jan 2017 »</span> 503 <a href="http://warmspringwinds.github.io/tensorflow/tf-slim/2017/01/23/fully-convolutional-networks-(fcns)-for-image-segmentation/">Fully Convolutional Networks (FCNs) for Image Segmentation</a> 504 </li> 505 506 <li> 507 <span>21 Dec 2016 »</span> 508 <a href="http://warmspringwinds.github.io/tensorflow/tf-slim/2016/12/21/tfrecords-guide/">Tfrecords Guide</a> 509 </li> 510 511 <li> 512 <span>18 Dec 2016 »</span> 513 <a href="http://warmspringwinds.github.io/tensorflow/tf-slim/2016/12/18/image-segmentation-with-tensorflow-using-cnns-and-conditional-random-fields/">Image Segmentation with Tensorflow using CNNs and Conditional Random Fields</a> 514 </li> 515 516 </ul> 517 </div> 518 519 520 <div class="post-footer"> 521 <div class="column-1"> 522 523 <a href="http://warmspringwinds.github.io/tensorflow/tf-slim/2017/01/23/fully-convolutional-networks-(fcns)-for-image-segmentation/"><< Older</a> 524 525 </div> 526 <div class="column-2"><a href="http://warmspringwinds.github.io/ ">Home</a></div> 527 <div class="column-3"> 528 529 <span>Newer >></span> 530 531 </div> 532 </div> 533</div> 534 535 536 <footer class="main-footer"> 537 <div class="wc-container"> 538 <div class="column one"> 539 <h6>Few more links</h6> 540<ul class="menu"> 541 <li><a href="http://warmspringwinds.github.io/about">About</a></li> 542 <li><a href="http://warmspringwinds.github.io/blog">Blog</a></li> 543 <li><a href="http://warmspringwinds.github.io/faq">Help / FAQ</a></li> 544</ul> 545 </div> 546 <div class="column two"> 547 <h6>Follow me</h6> 548 549<ul class="social-media"> 550 551 552 553 <li> 554 <a title="warmspringwinds on Twitter" 555 href="https://twitter.com/warmspringwinds" 556 class="twitter wc-img-replace" target="_blank">Twitter</a> 557 </li> 558 559 560 561 <li> 562 <a title="warmspringwinds on Github" 563 href="https://github.com/warmspringwinds" 564 class="github wc-img-replace" target="_blank">Github</a> 565 </li> 566 567 568 569 570 571 572 573 574 575 576</ul> 577 </div> 578 </div> 579 <p class="wc-container disclaimer"> 580 581Powered by <a href="http://jekyllrb.com" target="_blank">Jekyll</a> 582 </p> 583 </footer> 584
584<script type="text/javascript"> 585 /* To avoid render blocking css */ 586 var cb = function() { 587 var l = document.createElement('link'); l.rel = 'stylesheet'; 588 l.href = 'http://fonts.googleapis.com/css?family=Ubuntu+Mono&subset=latin'; 589 var h = document.getElementsByTagName('head')[0]; h.parentNode.insertBefore(l, h); 590 }; 591 var raf = requestAnimationFrame || mozRequestAnimationFrame || 592 webkitRequestAnimationFrame || msRequestAnimationFrame; 593 if (raf) raf(cb); 594 else window.addEventListener('load', cb); 595 </script>
595 596 <!-- jQuery --> 597
597<script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
597 598 <!-- When no internet load JQuery from local --> 599
599<script>window.jQuery || document.write('
599<script src="/assets/js/jquery.min.js"><\/script>')</script>
599 600 <!-- Site js --> 601
601<script src="/assets/js/all.js"></script>
601 602 <!-- Google analytics --> 603
604<script type="text/javascript"> 605 606 var _gaq = _gaq || []; 607 _gaq.push(['_setAccount', 'UA-63320666-1']); 608 _gaq.push(['_trackPageview']); 609 610 (function() { 611 var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true; 612 ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js'; 613 var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s); 614 })(); 615 616</script>
616 617 618 </body> 619</html>
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.