PageSourceSearch

http://warmspringwinds.github.io/pytorch/rnns/2018/01/27/learning-…s-and-music-with-recurrent-neural-networks/

html warmspringwinds.github.io collected 2026-10-03 10:09:28 UTC 37,878 bytes, 619 lines download raw bytes

1<!DOCTYPE html>
2<html>
3    <head>
4        <meta charset="utf-8">
5        <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1">
6        <title>Learning to generate lyrics and music with Recurrent Neural Networks</title>
7        <meta name="viewport" content="width=device-width, initial-scale=1">        
8        <meta name="description" content="Blog about Machine Learning and Computer Vision. Google Summer of Code blog posts. Scikit-image face detection algorithm implementation.
9">
10        <link rel="canonical" 
11        href="http://warmspringwinds.github.io/pytorch/rnns/2018/01/27/learning-to-generate-lyrics-and-music-with-recurrent-neural-networks/">
12
13        <!-- Mathjax-->
14        
14<script type="text/javascript" src="http://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
14
15        
16        <!-- Harmony styles -->
17        <link rel="stylesheet" type="text/css" href="/assets/css/main.css">
18
19        <!-- Modernizr js -->
20        
20<script async src="/assets/js/modernizr.js"></script>
20    
21
22        <!-- IE Fixes -->
23        <!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries -->
24        <!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
25        <!--[if lt IE 9]>
26          
26<script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
26
27          
27<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
27
28
29        <![endif]-->        
30    </head>
31    <body class="theme-base-01">
32        <header class="main-header">
33            <div class="wc-container">
34                <h1><a href="/">Daniil's blog</a></h1>
35                <h2>Machine Learning and Computer Vision artisan.</h2>
36                <ul>
37	<li>
38		<a href="http://warmspringwinds.github.io/about">About</a><span>/</span>
39	</li>
40	<li>
41		<a href="http://warmspringwinds.github.io/blog">Blog</a><span>/</span>
42	</li>
43</ul>                
44            </div>
45        </header>
46        <div class="page-content wc-container">
47	
48	<div class="post">
49		<h1>Learning to generate lyrics and music with Recurrent Neural Networks</h1>
50		<p class="post-meta">
51			
52      <span class="categories">
53      Pytorch and rnns
54      </span> |
55	    
56	    <span class="post-date">
57    	Jan 27, 2018 
58	    </span>
59		</p>		
60		<div class="post">
61			<p>A post showing an application of RNN-based generative models for lyrics and piano music generation.</p>
62
63<hr />
64
65<h3 id="introduction">Introduction</h3>
66
67<p>In this post we will train RNN character-level language model on lyrics dataset of
68most popular/recent artists. Having a trained model, we will sample a couple of
69songs which will be a funny mixture of different styles of different artists.
70After that we will update our model to become a conditional character-level RNN,
71making it possible for us to sample songs conditioned on artist.
72And finally, we conclude by training our model on midi dataset of piano songs.
73While solving all these tasks, we will briefly explore some interesting concepts related to RNN
74training and inference like character-level RNN, conditional character-level RNN,
75sampling from RNN, truncated backpropagation through time and gradient checkpointing. All the code and
76trained models are <a href="https://github.com/warmspringwinds/pytorch-rnn-sequence-generation-classification">available on github</a> and were implemented in <a href="http://pytorch.org/">Pytorch</a>. The blog post can also <a href="https://github.com/warmspringwinds/pytorch-rnn-sequence-generation-classification/blob/master/blog_post.ipynb">be viewed in a jupyter notebook format</a>. If you are already familiar with the character-level language model and recurrent neural networks, feel free to skip respective sections or go directly to the results section.</p>
77
78<h3 id="character-level-language-model">Character-Level language model</h3>
79
80<p><img src="http://warmspringwinds.github.io/assets/img/character_level_model.jpg" alt="png" /></p>
81
82<p>Before choosing a model, let’s have a closer look at our task. Given current letter and all previous
83letters, we will try to predict the next character. During training we will just take a sequence, and use
84all its characters except the last one as an input and the same sequence starting from the second character as groundtruth (see the picture above; <a href="https://github.com/spro/practical-pytorch/blob/master/conditional-char-rnn/conditional-char-rnn.ipynb">Source</a>). We will start from the simplest model that ignores all the previous characters while making a prediction, improve this model to make it take only a certain number of previous characters into account, and conclude with a model that takes all the previous characters into consideration while making a prediction.</p>
85
86<p>Our language model is defined on a character level. We will create a dictionary which will contain
87all English characters plus some special symbols, like period, comma, and end-of-line symbol. Each charecter will be represented as one-hot-encoded tensor. For more information about character-level models and examples, I recommend <a href="https://github.com/spro/practical-pytorch">this resource</a>.</p>
88
89<p>Having characters, we can now form sequences of characters. We can generate sentences even now just by
90randomly sampling character after character with a fixed probability 
90<script type="math/tex">p(any~letter)=\frac{1}{dictionary~size}</script>
90.
91That’s the most simple character level language model. Can we do better than this? Yes, we can compute the probabily of occurance of each letter from our training corpus (number of times a letter occures divided by the size of our dataset) and randomly sample letter using these probabilities. This model is better but it totally ignores the relative positional aspect of each letter. For example, pay attention on how you read any word: you start with the first letter, which is usually hard to predict, but as you reach the end of a word you can sometimes guess the next letter. When you read any word you are implicitly using some rules which you learned by reading other texts: for example, with each additional letter that you read from a word, the probability of a space character increases (really long words are rare) or the probability of any consonant after the letter “r” is low as it usually followed by vowel. There are lot of similar rules and we hope that our model will be able to learn them from data. To give our model a chance to learn these rules we need to extend it.</p>
92
93<p>Let’s make a small gradual improvement of our model and let probability of each letter depend
94only on the previously occured letter (<a href="https://en.wikipedia.org/wiki/Markov_property">markov assumption</a>). So, basically we will have 
94<script type="math/tex">p(current~letter|previous~letter)</script>
94.
95This is a <a href="https://en.wikipedia.org/wiki/Markov_chain">Markov chain model</a> (also try these <a href="http://setosa.io/ev/markov-chains/">interactive visualizations</a> if you are not familiar with it). We can also estimate the probability distribution 
95<script type="math/tex">p(current~letter|previous~letter)</script>
95 from our training dataset. This model is limited because in most cases the probability of the current letter depends not only on the previous letter.</p>
96
97<p>What we would like to model is actually 
97<script type="math/tex">p(current~letter|all~previous~letters)</script>
97. At first, the task seems intractable as the number of previous letters is variable and it might become really large in case of long
98sequences. Turns out Reccurent Neural Netoworks can tackle this problem to a certain extent by using shared weights and fixed size hidden state. This leads us to a next section dedicated to RNNs.</p>
99
100<h3 id="recurrent-neural-networks">Recurrent Neural Networks</h3>
101
102<p><img src="http://warmspringwinds.github.io/assets/img/rnn_unfold.jpg" alt="png" /></p>
103
104<p>Recurrent neural networks are a family of neural networks for processing sequential data.
105Unlike feedforward neural networks, RNNs can use their internal memory to process arbitrary sequences of inputs.
106Because of arbitrary size input sequences, they are concisely depicted as a graph with a cycle (see the picture; <a href="http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/">Source</a>).
107But they can be “unfolded” if the size of input sequence is known. They define a non-linear mapping from a current input 
107<script type="math/tex">x_t</script>
107 and previous hidden state 
107<script type="math/tex">s_{t-1}</script>
107 to the output 
107<script type="math/tex">o_t</script>
107 and current hidden state 
107<script type="math/tex">s_t</script>
107. Hidden state size has a predefined size and stores features which are updated on each step and affect the result of mapping.</p>
108
109<p>Now align the previous picture of the character-level language model and the ufolded RNN picture to see how
110we are using the RNN model to learn a character level language model.</p>
111
112<p>While the picture depicts the Vanilla RNN, we will use LSTM in our work as it is easier to train usually achieves better results.</p>
113
114<p>For a more elaborate introduction to RNNs, we refer reader to the <a href="http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/">following resource</a>.</p>
115
116<h3 id="lyrics-dataset">Lyrics dataset</h3>
117
118<p>For our experiments we have chosen <a href="https://www.kaggle.com/mousehead/songlyrics">55000+ Song Lyrics Kaggle dataset</a> which contains good variety of recent artists and more older ones. It is stored as a pandas file and we wrote a python wrapper around it to be able to use it for training purposes. You will have to download it yourself in order to be able to use our code.</p>
119
120<p>In order to be able to interpret the results better, I have chosen a subset of artists which I am
121more or less familiar with:</p>
122
123<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">artists</span> <span class="o">=</span> <span class="p">[</span>
124<span class="s">'ABBA'</span><span class="p">,</span>
125<span class="s">'Ace Of Base'</span><span class="p">,</span>
126<span class="s">'Aerosmith'</span><span class="p">,</span>
127<span class="s">'Avril Lavigne'</span><span class="p">,</span>
128<span class="s">'Backstreet Boys'</span><span class="p">,</span>
129<span class="s">'Bob Marley'</span><span class="p">,</span>
130<span class="s">'Bon Jovi'</span><span class="p">,</span>
131<span class="s">'Britney Spears'</span><span class="p">,</span>
132<span class="s">'Bruno Mars'</span><span class="p">,</span>
133<span class="s">'Coldplay'</span><span class="p">,</span>
134<span class="s">'Def Leppard'</span><span class="p">,</span>
135<span class="s">'Depeche Mode'</span><span class="p">,</span>
136<span class="s">'Ed Sheeran'</span><span class="p">,</span>
137<span class="s">'Elton John'</span><span class="p">,</span>
138<span class="s">'Elvis Presley'</span><span class="p">,</span>
139<span class="s">'Eminem'</span><span class="p">,</span>
140<span class="s">'Enrique Iglesias'</span><span class="p">,</span>
141<span class="s">'Evanescence'</span><span class="p">,</span>
142<span class="s">'Fall Out Boy'</span><span class="p">,</span>
143<span class="s">'Foo Fighters'</span><span class="p">,</span>
144<span class="s">'Green Day'</span><span class="p">,</span>
145 <span class="s">'HIM'</span><span class="p">,</span>
146 <span class="s">'Imagine Dragons'</span><span class="p">,</span>
147 <span class="s">'Incubus'</span><span class="p">,</span>
148 <span class="s">'Jimi Hendrix'</span><span class="p">,</span>
149 <span class="s">'Justin Bieber'</span><span class="p">,</span>
150 <span class="s">'Justin Timberlake'</span><span class="p">,</span>
151<span class="s">'Kanye West'</span><span class="p">,</span>
152 <span class="s">'Katy Perry'</span><span class="p">,</span>
153 <span class="s">'The Killers'</span><span class="p">,</span>
154 <span class="s">'Kiss'</span><span class="p">,</span>
155 <span class="s">'Lady Gaga'</span><span class="p">,</span>
156 <span class="s">'Lana Del Rey'</span><span class="p">,</span>
157 <span class="s">'Linkin Park'</span><span class="p">,</span>
158 <span class="s">'Madonna'</span><span class="p">,</span>
159 <span class="s">'Marilyn Manson'</span><span class="p">,</span>
160 <span class="s">'Maroon 5'</span><span class="p">,</span>
161 <span class="s">'Metallica'</span><span class="p">,</span>
162 <span class="s">'Michael Bolton'</span><span class="p">,</span>
163 <span class="s">'Michael Jackson'</span><span class="p">,</span>
164 <span class="s">'Miley Cyrus'</span><span class="p">,</span>
165 <span class="s">'Nickelback'</span><span class="p">,</span>
166 <span class="s">'Nightwish'</span><span class="p">,</span>
167 <span class="s">'Nirvana'</span><span class="p">,</span>
168 <span class="s">'Oasis'</span><span class="p">,</span>
169 <span class="s">
169'Offspring'</span><span class="p">,</span>
170 <span class="s">'One Direction'</span><span class="p">,</span>
171 <span class="s">'Ozzy Osbourne'</span><span class="p">,</span>
172 <span class="s">'P!nk'</span><span class="p">,</span>
173 <span class="s">'Queen'</span><span class="p">,</span>
174 <span class="s">'Radiohead'</span><span class="p">,</span>
175 <span class="s">'Red Hot Chili Peppers'</span><span class="p">,</span>
176 <span class="s">'Rihanna'</span><span class="p">,</span>
177 <span class="s">'Robbie Williams'</span><span class="p">,</span>
178 <span class="s">'Rolling Stones'</span><span class="p">,</span>
179 <span class="s">'Roxette'</span><span class="p">,</span>
180 <span class="s">'Scorpions'</span><span class="p">,</span>
181 <span class="s">'Snoop Dogg'</span><span class="p">,</span>
182 <span class="s">'Sting'</span><span class="p">,</span>
183 <span class="s">'The Script'</span><span class="p">,</span>
184 <span class="s">'U2'</span><span class="p">,</span>
185 <span class="s">'Weezer'</span><span class="p">,</span>
186 <span class="s">'Yellowcard'</span><span class="p">,</span>
187 <span class="s">'ZZ Top'</span><span class="p">]</span>
188</code></pre></div></div>
189
190<h3 id="training-unconditional-character-level-language-model">Training unconditional character-level language model</h3>
191
192<p>Our first experiment consisted of training of our character-level language model RNN
193on the whole corpus. We didn’t take into consideration the artist information while training.</p>
194
195<h3 id="sampling-from-rnn">Sampling from RNN</h3>
196
197<p>Let’s try to sample a couple of songs after training our model. Basically, on each
198step our RNN will output logits and we can softmax them and sample from that distribution.
199Or we can use Gumble-Max trick and <a href="https://hips.seas.harvard.edu/blog/2013/04/06/the-gumbel-max-trick-for-discrete-distributions/">sample using logits directly</a> which is equivalent.</p>
200
201<p>One intersting thing about sampling is that we can partially define the input sequence ourselves and start sampling
202with that initial condition. For example, we can sample a song that starts with “Why”:</p>
203
204<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Why do you have to leave me?  
205I think I know I'm not the only one  
206I don't know if I'm gonna stay awake  
207I don't know why I go along  
208  
209I don't know why I can't go on  
210I don't know why I don't know  
211I don't know why I don't know  
212I don't know why I keep on dreaming of you   
213</code></pre></div></div>
214
215<p>Well, that sounds like a possible song :D</p>
216
217<p>Let’s sample with a song that starts with “Well”:</p>
218
219<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Well, I was a real good time  
220I was a rolling stone  
221I was a rock and roller  
222Well, I never had a rock and roll  
223There were times I had to do it  
224I had a feeling that I was found  
225I was the one who had to go  
226</code></pre></div></div>
227
228<p>There is “temperature” parameter that is used during sampling which controls the randomness of sampling
229process. When this parameter approaches zero,
230the sampling is equivalent to argmax and when it is close to infinity the sampling is equivalent to sampling
231from a uniform distribution. Have a look at the figure from a <a href="https://arxiv.org/pdf/1611.01144.pdf">relevant paper by Jang et al.</a>:</p>
232
233<p><img src="http://warmspringwinds.github.io/assets/img/sampling_temperature.png" alt="png" /></p>
234
235<p>When 
235<script type="math/tex">\tau=1</script>
235, the distribution is not affected. If we decrease 
235<script type="math/tex">\tau</script>
235, the distribution
236becomes more pronounced, meaning that value with bigger probability mass will have it increased. When 
236<script type="math/tex">\tau</script>
236 will approach zero, sampling will be equivalent to armax, because the probability of that value will be close to one. When we start to icrease 
236<script type="math/tex">\tau</script>
236 the distribution becomes more and more uniform.</p>
237
238<p>The previous sample was generated with a temperature paramter equal to 
238<script type="math/tex">0.5</script>
238.
239Let’s see what happens when we increase it to 
239<script type="math/tex">1.0</script>
239 and sample:</p>
240
241<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Why can't we drop out of time?  
242We were born for words to see.  
243Won't you love this. You're still so amazing.  
244This could be that down on Sunday Time.  
245Oh, Caroline, a lady floor.  
246I thought of love, oh baby.  
247</code></pre></div></div>
248
249<p>Let’s try increasing it even more:</p>
250
251<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Why - won't we grow up naked?  
252We went quietly what we would've still give  
253That girl you walked before our bedroom room  
254I see your mind is so small to a freak  
255Stretching for a cold white-heart of crashing  
256Truth in the universal daughter  
257  
258I lose more and more hard  
259I love you anytime at all  
260Ah come let your help remind me  
261Now I've wanted waste and never noticed  
262  
263I swear I saw you today  
264You needed to get by  
265But you sold a hurricane  
266Well out whispered in store
267</code></pre></div></div>
268
269<p>Why don’t we grow up naked, indeed? :D
270Well, you can see that trend that when we increase the temperature, sampled
271sentences become more and more random.</p>
272
273<h3 id="training-conditional-character-level-language-model">Training conditional character-level language model</h3>
274
275<p>Imagine if we could generate lyrics in a style of some particular artist.
276Let’s change our model, so that it can use this information during training.</p>
277
278<p>We will do this by adding an additional input to our RNN. So far, our RNN model
279was only accepting tensors containing one-hot encoded character on each step.</p>
280
281<p>The extention to our model will be very simple: we will have and additional one-hot encoded
282tensor which will represent the artist. So on each step the RNN will accept one tensor which will consist of concatenated tensors representing character and artist. Look <a href="https://github.com/spro/practical-pytorch/blob/master/conditional-char-rnn/conditional-char-rnn.ipynb">here for more</a>.</p>
283
284<h3 id="sampling-from-conditional-language-model-rnn">Sampling from conditional language model RNN</h3>
285
286<p>After training, we sampled a couple of songs conditined on artist.
287Below you can find some results.</p>
288
289<p>Him:</p>
290
291<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>My fears  
292And the moment don't make me sing  
293So free from you  
294The pain you love me yeah  
295  
296Whatever caused the warmth  
297You smile you're happy  
298You sit away  
299You say it's all in vain  
300</code></pre></div></div>
301
302<p>Seems really possible, especially the fact the the word pain was used, which is
303very common in the lyrics of the artist.</p>
304
305<p>ABBA:</p>
306
307<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Oh, my love it makes me close a thing  
308You've been heard, I must have waited  
309I hear you  
310So I say  
311Thank you for the music, that makes me cry  
312  
313And you moving my bad as me, ah-hang wind in the hell  
314I was meant to be with you, I'll never be playing up
315</code></pre></div></div>
316
317<p>Bob Marley:</p>
318
319<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Mercy on judgment, we got so much  
320  
321Alcohol, cry, cry, cry  
322Why don't try to find our own  
323I want to know, Lord, I wanna give you  
324Just saving it, learned  
325Is there any more?  
326  
327All that damage done  
328That's all reason, don't worry  
329Need a hammer  
330I need you more and more  
331</code></pre></div></div>
332
333<p>Coldplay:</p>
334
335<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Look at the stars  
336Into life matter where you lay  
337Saying no doubt  
338I don't want to fly  
339In my dreams and fight today
340
341I will fall for you  
342  
343All I know  
344And I want you to stay  
345Into the night  
346  
347I want to live waiting  
348With my love and always  
349Have I wouldn't wasted  
350Would it hurt you
351</code></pre></div></div>
352
353<p>Kanye West:</p>
354
355<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>I'm everywhere for you  
356The way that it couldn't stop  
357I mean it too late and love I made in the world  
358I told you so I took the studs full cold-stop  
359The hardest stressed growin'  
360The hustler raisin' on my tears  
361I know I'm true, one of your love
362</code></pre></div></div>
363
364<p>Looks pretty cool but keep in mind that we didn’t track the validation accuracy so some sampled lines could have been just memorized by our rnn. A better way to do it is to pick a model that gives best validation score during training (see the code for the next section where we performed training this way). We also noticed one interesting thing: the unconditional
365model usually performes better when you want to sample with a specified starting string.
366Our intuition is that when sampling from a conditional model with a specified starting string, 
367we actually put two conditions on our model – starting string and an artist compared to the one condition
368in the case of previous model that we explored. And we didn’t have enough data to model that conditional
369distribution well (every artist has relatively limited number of songs).</p>
370
371<p>We are making the code and models available and you can sample songs from our trained models
372even without gpu as it is not really computationally demanding.</p>
373
374<h3 id="midi-dataset">Midi dataset</h3>
375
376<p>Next, we will work with a <a href="http://www-etud.iro.umontreal.ca/~boulanni/icml2012">small midi dataset</a> consisting
377of approximately 
377<script type="math/tex">700</script>
377 piano songs. We have used the <code class="highlighter-rouge">Nottingam</code> piano dataset (training split only).</p>
378
379<p>Turns out that any midi file can be <a href="http://nbviewer.jupyter.org/github/craffel/pretty-midi/blob/master/Tutorial.ipynb">converted to piano roll</a> which is just is a time-frequency matrix where each row is a different MIDI pitch and each column is a different slice in time. So each piano song from our dataset will be represented as a matrix of size 
379<script type="math/tex">88\times song\_length</script>
379, where 
379<script type="math/tex">88</script>
379 is a number of pitches of the piano. Here is an example of
380piano roll matrix:</p>
381
382<p><img src="http://warmspringwinds.github.io/assets/img/piano_roll_2.png" alt="png" /></p>
383
384<p>This representation is very intuitive and easy to interpret even for a person that is not familiar
385with music theory. Each row represents a pitch: top rows represent low frequency pitches and bottom
386rows represent high pitches. Plus, we have a horizontal axis which represents time. So if we play a sound
387with a certain pitch for a certian period of time, we will see a horizontal line. Overall, this is very
388similar to <a href="![alt text](sampling_temperature.png" title="Logo Title Text 1">piano tutorials on youtube</a>.</p>
389
390<p>Now, let’s try to see the similarities between the character-level model and our new task. In the current case, we will have to predict the pitches that will be played on the next timestep, given all the previously played
391pitches. So, if you look at the picture of the piano roll, each column represents some kind of a musical character and given all the previous musical characters, we want to predict the next one. Let’s pay attention to the difference between a text character and a musical character. If you recall, each character in our language model was represented by one-hot vector (meaning that only one value in our vector is 
391<script type="math/tex">1</script>
391 and others are 
391<script type="math/tex">0</script>
391).
392For music character multiple keys can be pressed at one timestep (since we are working with polyphonic dataset).
393In this case, each timestep will be represented by a vector which can contain more than one 
393<script type="math/tex">1</script>
393.</p>
394
395<h3 id="training-pitch-level-piano-music-model">Training pitch-level piano music model</h3>
396
397<p>Before starting the training, we will have to adjust our loss that we have used for language model
398to account for different input that we discussed in the previous section. In the language model,
399we had one-hot encoded tensor (character) as an input on each timestep and one-hot encoded tensor as output (predicted next character). As we had to make a single exlusive choice for predicted next character, we used
400<a href="https://rdipietro.github.io/friendly-intro-to-cross-entropy-loss/">cross-entropy loss</a>.</p>
401
402<p>But now our model outputs a vector which is no longer one-hot encoded (multiple keys can be pressed). Of course, we can treat all possible combinations of pressed keys as a separate class, but this is intractable. Instead, we will treat each element of the output vector as a binary variable (
402<script type="math/tex">1</script>
402 – pressing, 
402<script type="math/tex">0</script>
402 – not pressing a key). We will define a separate loss
403for each element of the output vector to be binary cross-entropy. And our final loss will be an averaged sum of these binary cross-entropies. You can also read the code to get a better understanding.</p>
404
405<p>After making the aforementioned changes, we trained our model. In the next section, we will perform sampling
406and inspect the results.</p>
407
408<h3 id="sampling-from-pitch-level-rnn">Sampling from pitch-level RNN</h3>
409
410<p>We have sampled piano rolls during the early stages of optimization:</p>
411
412<p><img src="http://warmspringwinds.github.io/assets/img/piano_roll_early_sample.png" alt="png" /></p>
413
414<p>You can see that our model is starting to learn one common pattern that is common among the songs from
415our dataset: each song consists of two different parts. First part contains a sequence of pitches that are played separately and are very <a href="https://www.didjshop.com/BasicMusicalHarmony.html">distinguishable and are often singable</a> (also know as melody). If you look at the sampled piano roll, this part can be clearly seen in the bottom. If you also have a look at the top of our piano roll, we can see a group of pitches that are usually played together – this is harmony or a progression of chords (pitches that are played together throughout the song) which accompanies the melody.</p>
416
417<p>By the end of the training samples drawn from our model started to look like this:</p>
418
419<p><img src="http://warmspringwinds.github.io/assets/img/piano_roll_late_sample.png" alt="png" /></p>
420
421<p>As you can see they started to look more similar to the picture of the ground-truth piano roll that we showed
422in the previous sections.</p>
423
424<p>After training, we have sampled songs and analyzed them. We got one sample with <a href="https://www.youtube.com/watch?v=Iz8xQou2OqA">an interesting introduction</a>. While another sample features <a href="https://www.youtube.com/watch?v=fUdsWVIOeeU&amp;feature=youtu.be&amp;t=15s">a nice style transition</a>. At the same time we generated a couple of examples with low temperature parameter which resulted in songs with a slow tempo: <a href="https://www.youtube.com/watch?v=UoLyeauBsNk">first one</a> and a <a href="https://www.youtube.com/watch?v=Iz8xQou2OqA">second one here</a>.
425You can find the whole playlist <a href="https://www.youtube.com/watch?v=EOQQOQYvGnw&amp;list=PLJkMX36nfYD000TG-T59hmEgJ3ojkOlBp">here</a>.</p>
426
427<h3 id="sequence-length-and-related-problems">Sequence length and related problems</h3>
428
429<p>Now let’s look at our problem from the gpu memory consumption and speed point of view.</p>
430
431<p>We greatly speed up computation by processing our sequences in batches. At the same time, as
432our sequences become longer (depending on the dataset), our max batch size starts to decrease.
433Why is it a case? As we use backpropagation to compute gradients, we need to store all the intermediate acitvations, which contribute the most to the memory consumption. As our sequence becomes longer, we need to store more activations, therefore, we can fit less examples in our batch.</p>
434
435<p>Sometimes, we either have to work with really long sequences or we want to increase our batch size or maybe you just have a gpu with small amount of memory available. There are multiple possible solutions to reduce memory
436consumption in this case, but we will mention two, which will have different trade-offs.</p>
437
438<p>First one is a <a href="https://www.quora.com/Whats-the-key-difference-between-backprop-and-truncated-backprop-through-time#sLRGO">truncated back propagation</a>. The idea is to split the whole sequence into subsequences and treat
439them as separate batches with an exception that we process these batches in the order of split and every next batch uses hidden state of previous batch as an initial hidden state. We also provide an implementation of this approach, so that you can get the better understanding. This approach is obviously not an exact equivalent of processing the whole sequence but it makes more frequent updates and consumes less memory. On the other hand, there is a chance that we might not be able to capture long-term dependencies that span beyond the length of one subsequence.</p>
440
441<p>Second one is <a href="https://medium.com/@yaroslavvb/fitting-larger-networks-into-memory-583e3c758ff9">gradient checkpointing</a>. This method gives us a possibilty to use less memory while training our model on the whole sequence on the expence of performing more computation. If you recall, previously we mentioned that the most memory during
442training is occupied by activations. The idea of gradient checkpointing consists of storing only every 
442<script type="math/tex">n</script>
442-th activation and recomputing the unsaved activations later. This method is already <a href="https://github.com/openai/gradient-checkpointing">implemented in Tensorflow</a> and <a href="https://github.com/pytorch/pytorch/pull/4594">being implemented in Pytorch</a>.</p>
443
444<h3 id="conclusion-and-future-work">Conclusion and future work</h3>
445
446<p>In our work we trained simple generative model for text, extended our model to work with
447polyphonic music, briefly looked at how sampling works and how the temperature parameter affects our
448text and music samples – low temperature gives more stable results while high temperature adds more
449randomness which sometimes gives rise to very interesting samples.</p>
450
451<p>Future work can include two directions – more applications or deeper analysis of the already trained models.
452Same models can be applied to your spotify listening history, for example. After training on your
453listening history data, you can give it a sequence of songs that you have listened to in the previous hour or so, and it will sample a playlist for you for the rest of the day. Well, you can also do the same for your browsing history, which will be just a cool tool to analyze your browsing behaviour patterns. <a href="https://www.kaggle.com/uciml/human-activity-recognition-with-smartphones">Capture the accelerometer and gyroscope data</a> from your phone while doing different activities (exercising in the gym, working in the office, sleeping) and learn to classify these
454activity stages. After that you can change your music playlist automatically, based on your activity (sleeping – calm music of rain, exercising in the gym – high intensity music). In terms of medical applications, model can
455be applied to detect heart problems based on pulse and other data, similar to <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5391725/">this work</a>.</p>
456
457<p>It would be very interesting to analyze the neuron firings in our RNN trained
458for music generation like <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/">here</a>. To see if the model learned some simple music concepts implicitly (like our discussion
459of harmony and melody). The hidden representation of RNN can be used to cluster our music dataset to 
460find similar songs.</p>
461
462<p>Let’s sample one last lyrics from our unconditional model to conclude this post :D :</p>
463
464<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>The story ends  
465The sound of the blue  
466The tears were shining  
467The story of my life  
468I still believe  
469The story of my life 
470</code></pre></div></div>
471
472		</div>
473
474		<!-- Discus comments feature -->
475
476		
477
478		<div id="disqus_thread"></div>
479		
479<script type="text/javascript">
480			/* * * CONFIGURATION VARIABLES * * */
481			var disqus_shortname = 'daniil-ml-cv-blog';
482
483			/* * * DON'T EDIT BELOW THIS LINE * * */
484			(function() {
485				var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
486				dsq.src = '//' + disqus_shortname + '.disqus.com/embed.js';
487				(document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
488			})();
489		</script>
489
490		<noscript>Please enable JavaScript to view the <a href="https://disqus.com/?ref_noscript" rel="nofollow">comments powered by Disqus.</a></noscript>
491
492		
493	</div>
494
495
496	
497	<div class="related">
498		<h4>Related Posts</h2>
499		<ul class="posts">
500		    
501		    <li>
502			  <span>23 Jan 2017 &raquo;</span>
503			  <a href="http://warmspringwinds.github.io/tensorflow/tf-slim/2017/01/23/fully-convolutional-networks-(fcns)-for-image-segmentation/">Fully Convolutional Networks (FCNs) for Image Segmentation</a>
504		    </li>
505		    
506		    <li>
507			  <span>21 Dec 2016 &raquo;</span>
508			  <a href="http://warmspringwinds.github.io/tensorflow/tf-slim/2016/12/21/tfrecords-guide/">Tfrecords Guide</a>
509		    </li>
510		    
511		    <li>
512			  <span>18 Dec 2016 &raquo;</span>
513			  <a href="http://warmspringwinds.github.io/tensorflow/tf-slim/2016/12/18/image-segmentation-with-tensorflow-using-cnns-and-conditional-random-fields/">Image Segmentation with Tensorflow using CNNs and Conditional Random Fields</a>
514		    </li>
515		    
516		</ul>
517	</div>
518	
519
520	<div class="post-footer">
521		<div class="column-1">
522			
523				<a href="http://warmspringwinds.github.io/tensorflow/tf-slim/2017/01/23/fully-convolutional-networks-(fcns)-for-image-segmentation/"><< Older</a>
524			
525		</div>
526		<div class="column-2"><a href="http://warmspringwinds.github.io/ ">Home</a></div>
527		<div class="column-3">
528			
529				<span>Newer >></span>
530			
531		</div>
532	</div>
533</div>
534 
535
536        <footer class="main-footer">
537            <div class="wc-container">
538                <div class="column one">
539                    <h6>Few more links</h6>
540<ul class="menu">
541    <li><a href="http://warmspringwinds.github.io/about">About</a></li>
542    <li><a href="http://warmspringwinds.github.io/blog">Blog</a></li>
543    <li><a href="http://warmspringwinds.github.io/faq">Help / FAQ</a></li>    
544</ul>		                    
545                </div>
546                <div class="column two">
547                    <h6>Follow me</h6>
548
549<ul class="social-media">
550
551
552    
553    <li>
554        <a title="warmspringwinds on Twitter" 
555            href="https://twitter.com/warmspringwinds" 
556            class="twitter wc-img-replace" target="_blank">Twitter</a>
557    </li>   
558    
559
560    
561    <li>
562        <a title="warmspringwinds on Github" 
563            href="https://github.com/warmspringwinds" 
564            class="github wc-img-replace" target="_blank">Github</a>
565    </li>
566     
567
568    
569
570    
571
572    
573
574    
575
576</ul>
577                </div>
578            </div>
579            <p class="wc-container disclaimer">
580                
581Powered by <a href="http://jekyllrb.com" target="_blank">Jekyll</a>
582            </p>
583        </footer>
584        
584<script type="text/javascript">
585          /* To avoid render blocking css */
586          var cb = function() {
587            var l = document.createElement('link'); l.rel = 'stylesheet';
588            l.href = 'http://fonts.googleapis.com/css?family=Ubuntu+Mono&subset=latin';
589            var h = document.getElementsByTagName('head')[0]; h.parentNode.insertBefore(l, h);
590          };
591          var raf = requestAnimationFrame || mozRequestAnimationFrame ||
592              webkitRequestAnimationFrame || msRequestAnimationFrame;
593          if (raf) raf(cb);
594          else window.addEventListener('load', cb);
595        </script>
595
596        <!-- jQuery -->
597        
597<script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
597
598        <!-- When no internet load JQuery from local -->
599        
599<script>window.jQuery || document.write('
599<script src="/assets/js/jquery.min.js"><\/script>')</script>
599
600        <!-- Site js -->
601        
601<script src="/assets/js/all.js"></script>
601
602        <!-- Google analytics  -->
603        
604<script type="text/javascript">
605
606  var _gaq = _gaq || [];
607  _gaq.push(['_setAccount', 'UA-63320666-1']);
608  _gaq.push(['_trackPageview']);
609
610  (function() {
611    var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
612    ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
613    var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
614  })();
615
616</script>
616
617
618    </body>        
619</html>

Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.