1const e=`--- 2title: "Building Recurrent Neural Networks in Tensorflow" 3date: "2018-07-05" 4author: ataspinar 5categories: 6 - "classification" 7 - "machine-learning" 8 - "recurrent-neural-networks" 9 - "tensorflow" 10math: true 11comments: true 12img_path: /assets/img/posts/2018-07-05-building-recurrent-neural-networks-in-tensorflow/ 13--- 14 15In the previous blog posts we have seen how we can [build Convolutional Neural Networks in Tensorflow](http://ataspinar.com/2017/08/15/building-convolutional-neural-networks-with-tensorflow/) and also how we can use [Stochastic Signal Analysis techniques](http://ataspinar.com/2018/04/04/machine-learning-with-signal-processing-techniques/) to classify signals and time-series. In this blog post, lets have a look and see how we can build Recurrent Neural Networks in Tensorflow and use them to classify Signals. 16 17 18# 1. Introduction to Recurrent Neural Networks 19 20Recurrent Neural Nets (RNN) detect features in sequential data (e.g. time-series data). Examples of applications which can be made using RNN's are [anomaly detection in time-series data](https://www.elen.ucl.ac.be/Proceedings/esann/esannpdf/es2015-56.pdf), classification of [ECG](http://ieeexplore.ieee.org/abstract/document/668803/) and [EEG](http://www.clinph-journal.com/article/S1388-2457(01)00579-X/abstract) data, [stock market prediction](https://www.sciencedirect.com/science/article/pii/S1568494610002565), [speech recogniton](https://arxiv.org/abs/1303.5778), sentiment analysis, etc. 21 22This is done by [unrolling the data](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) into N different copies of itself (if the data consists of N time-steps) . In this way, the input data at the previous time steps $$ t_n - 1, t_n - 2, t_n - 3, ... , t_0 $$ can be used when the data at timestep $$t_n$$ is evaluated. If the data at the previous time steps is somehow correlated to the data at the current time step, these correlations are remembered and otherwise they are forgotten. 23 24By unrolling the data, the weights of the Neural Network are shared across all of the time steps, and the RNN can generalize beyond the example seen at the current timestep, and beyond sequences seen in the training set. 25 26This is a very short description of how an RNN works. For people who want to know more, At the end is some more reading material to get you up to speed. For now, what I would like you to remember is that Recurrent Neural Networks can learn whether there are temporal dependencies in the sequential data, and if there are, which dependencies / features can be used to classify the data. A RNN therefore is ideal for the classification of time-series, signals and text documents. 27 28So, Lets start with implementing RNN's in Tensorflow and using them to classify signals. 29 30# 2. Loading the Data 31 32This blog we will work with the CPU-friendly [Human Activity Recognition Using Smartphones](https://archive.ics.uci.edu/ml/datasets/human+activity+recognition+using+smartphones) dataset. This dataset contains measurements done by 30 people between the ages of 19 to 48. These people have a smartphone placed on the waist while doing one of the following six activities: 33 34- walking, 35- walking upstairs, 36- walking downstairs, 37- sitting, 38- standing or 39- laying. 40 41During these activities, sensor data is recorded at a constant rate of 50Hz. The signals are cut in fixed-width windows of 2.56 sec with 50% overlap. Since, these signals of 2.56 sec long have a sampling rate of 50 Hz, they will have 128 samples in total. For an illustration of this, see Figure 1a. 42 43The smartphone measures three-axial linear body acceleration, three-axial linear total acceleration and three-axial angular velocity. So per measurement, the signal has nine components in total (see Figure 1b). 44 45 46_Figure 1a. A plot of the first components (body acceleration measured in the x-direction) of the signal._ 47 48 49_Figure 1b. A 3D plot of the nine different components of a signal._ 50 51The dataset is already splitted into a training and a test part, so we can immediately load the signal into two different numpy ndarrays containing the training part and test part. 52 53\`\`\`python 54def read_signals(filename): 55 with open(filename, 'r') as fp: 56 data = fp.read().splitlines() 57 data = map(lambda x: x.rstrip().lstrip().split(), data) 58 data = [list(map(float, line)) for line in data] 59 data = np.array(data, dtype=np.float32) 60 return data 61 62def read_labels(filename): 63 with open(filename, 'r') as fp: 64 activities = fp.read().splitlines() 65 activities = list(map(int, activities)) 66 return np.array(activities) 67 68def randomize(dataset, labels): 69 permutation = np.random.permutation(labels.shape[0]) 70 shuffled_dataset = dataset[permutation, :, :] 71 shuffled_labels = labels[permutation] 72 return shuffled_dataset, shuffled_labels 73 74def one_hot_encode(np_array, num_labels): 75 return (np.arange(num_labels) == np_array[:,None]).astype(np.float32) 76 77def reformat_data(dataset, labels): 78 no_labels = len(np.unique(labels)) 79 labels = one_hot_encode(labels, no_labels) 80 dataset, labels = randomize(dataset, labels) 81 return dataset, labels 82 83INPUT_FOLDER_TRAIN = './UCI_HAR/train/InertialSignals/' 84INPUT_FOLDER_TEST = './UCI_HAR/test/InertialSignals/' 85 86INPUT_FILES_TRAIN = ['body_acc_x_train.txt', 'body_acc_y_train.txt', 'body_acc_z_train.txt', 87 'body_gyro_x_train.txt', 'body_gyro_y_train.txt', 'body_gyro_z_train.txt', 88 'total_acc_x_train.txt', 'total_acc_y_train.txt', 'total_acc_z_train.txt'] 89 90INPUT_FILES_TEST = ['body_acc_x_test.txt', 'body_acc_y_test.txt', 'body_acc_z_test.txt', 91 'body_gyro_x_test.txt', 'body_gyro_y_test.txt', 'body_gyro_z_test.txt', 92 'total_acc_x_test.txt', 'total_acc_y_test.txt', 'total_acc_z_test.txt'] 93 94##### 95 96train_signals, test_signals = [], [] 97for input_file in INPUT_FILES_TRAIN: 98 signal = read_signals(INPUT_FOLDER_TRAIN + input_file) 99 train_signals.append(signal)
100train_signals = np.transpose(np.array(train_signals), (1, 2, 0)) 101 102for input_file in INPUT_FILES_TEST: 103 signal = read_signals(INPUT_FOLDER_TEST + input_file) 104 test_signals.append(signal) 105test_signals = np.transpose(np.array(test_signals), (1, 2, 0)) 106 107##### 108 109LABELFILE_TRAIN = './UCI_HAR/train/y_train.txt' 110LABELFILE_TEST = './UCI_HAR/test/y_test.txt' 111train_labels = read_labels(LABELFILE_TRAIN) 112test_labels = read_labels(LABELFILE_TEST) 113 114##### 115 116train_dataset, train_labels = reformat_data(train_signals, train_labels) 117test_dataset, test_labels = reformat_data(test_signals, test_labels) 118\`\`\` 119 120The number of signals in the training set is 7352, and the number of signals in the test set is 2947. As we can see in Figure 2, each signal has a length of of 128 samples and 9 different components, so numerically it can be considered as an array of size 128 x 9. 121 122 123_Figure 2. A single signal can be represented by an array of size 128 x 9._ 124 125# 3. Recurrent Neural Networks in Tensorflow 126 127As we have also seen in the previous blog posts, our Neural Network consists of a \`tf.Graph()\` and a \`tf.Session()\`. The \`tf.Graph()\` contains all of the computational steps required for the Neural Network, and the \`tf.Session\` is used to execute these steps. 128 129The computational steps defined in the \`tf.Graph\` can be divided into four main parts; 130 1311. We initialize placeholders which are filled with batches of training data during the run. 1322. We define the RNN model and to calculate the output values (logits) 1333. The logits are used to calculate a loss value, which then 1344. is used in an Optimizer to optimize the weights of the RNN. 135 136 137\`\`\`python 138num_units = 50 139signal_length = 128 140num_components = 9 141num_labels = 6 142 143num_hidden = 32 144learning_rate = 0.001 145lambda_loss = 0.001 146total_steps = 5000 147display_step = 500 148batch_size = 100 149 150def accuracy(y_predicted, y): 151 return (100.0 * np.sum(np.argmax(y_predicted, 1) == np.argmax(y, 1)) / y_predicted.shape[0]) 152 153#### 154 155graph = tf.Graph() 156with graph.as_default(): 157 #1) First we put the input data in a tensorflow friendly form. 158 tf_dataset = tf.placeholder(tf.float32, shape=(None, signal_length, num_components)) 159 tf_labels = tf.placeholder(tf.float32, shape = (None, num_labels)) 160 161 #2) Then we choose the model to calculate the logits (predicted labels) 162 # We can choose from several models: 163 logits = rnn_model(tf_dataset, num_hidden, num_labels) 164 #logits = lstm_rnn_model(tf_dataset, num_hidden, num_labels) 165 #logits = bidirectional_lstm_rnn_model(tf_dataset, num_hidden, num_labels) 166 #logits = twolayer_lstm_rnn_model(tf_dataset, num_hidden, num_labels) 167 #logits = gru_rnn_model(tf_dataset, num_hidden, num_labels) 168 169 #3) Then we compute the softmax cross entropy between the logits and the (actual) labels 170 l2 = lambda_loss * sum(tf.nn.l2_loss(tf_var) for tf_var in tf.trainable_variables()) 171 loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=logits, labels=tf_labels)) + l2 172 173 #4. 174 # The optimizer is used to calculate the gradients of the loss function 175 optimizer = tf.train.AdamOptimizer(learning_rate).minimize(loss) 176 #optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss) 177 #optimizer = tf.train.AdagradOptimizer(learning_rate).minimize(loss) 178 179 # Predictions for the training, validation, and test data. 180 prediction = tf.nn.softmax(logits) 181 182with tf.Session(graph=graph) as session: 183 tf.global_variables_initializer().run() 184 print("\\nInitialized") 185 for step in range(total_steps): 186 #Since we are using stochastic gradient descent, we are selecting small batches from the training dataset, 187 #and training the convolutional neural network each time with a batch. 188 offset = (step * batch_size) % (train_labels.shape[0] - batch_size) 189 batch_data = train_dataset[offset:(offset + batch_size), :, :] 190 batch_labels = train_labels[offset:(offset + batch_size), :] 191 192 feed_dict = {tf_dataset : batch_data, tf_labels : batch_labels} 193 _, l, train_predictions = session.run([optimizer, loss, prediction], feed_dict=feed_dict) 194 train_accuracy = accuracy(train_predictions, batch_labels) 195 196 if step % display_step == 0: 197 feed_dict = {tf_dataset : test_dataset, tf_labels : test_labels} 198 _, test_predictions = session.run([loss, prediction], feed_dict=feed_dict) 199 test_accuracy = accuracy(test_predictions, test_labels) 200 message = "step {:04d} : loss is {:06.2f}, accuracy on training set {} %, accuracy on test set {:02.2f} %".format(step, l, train_accuracy, test_accuracy) 201 print(message) 202\`\`\` 203 204\`\`\` 205Initialized 206step 0000 : loss is 001.96, accuracy on training set 16.0 %, accuracy on test set 27.99 % 207step 0500 : loss is 000.75, accuracy on training set 60.0 %, accuracy on test set 60.67 % 208step 1000 : loss is 000.52, accuracy on training set 74.0 %, accuracy on test set 68.68 % 209step 1500 : loss is 000.66, accuracy on training set 70.0 %, accuracy on test set 69.22 % 210step 2000 : loss is 001.14, accuracy on training set 60.0 %, accuracy on test set 50.93 % 211step 2500 : loss is 001.00, accuracy on training set 74.0 %, accuracy on test set 65.97 % 212step 3000 : loss is 001.25, accuracy on training set 69.0 %, accuracy on test set 65.93 % 213step 3500 : loss is 001.40, accuracy on training set 79.0 %, accuracy on test set 69.66 % 214step 4000 : loss is 001.69, accuracy on training set 74.0 %, accuracy on test set 70.44 % 215step 4500 : loss is 002.00, accuracy on training set 77.0 %, accuracy on test set 70.68 % 216\`\`\` 217 218As you can see, there are different RNN Models and optimizers to choose from. 219 220GradientDescentOptimizer is a vanilla (simple) implementation of Stochastic Gradient Descent while other implementations like the AdaOptimizer, MomentumOptimizer and [AdamOptimizer](http://www.fast.ai/2018/07/02/adam-weight-decay/) dynamically adapt the learning rate to the parameters resulting in a more computational intensive process with better results. For a good explanation of the differences between all the different optimizers, have a look at [Sebastian Ruders' blog](http://ruder.io/optimizing-gradient-descent/). 221 222 223Besides the different types of optimizers, Tensorflow also contains different flavours of RNN's. We can choose from different types of [cells and wrappers](https://www.tensorflow.org/api_guides/python/contrib.rnn#Base_interface_for_all_RNN_Cells) use them to reconstruct different types of Recurrent Neural Networks. 224 225The basic types of cells are a \`BasicRNNCell\`, \`GruCell\`, \`LSTMCell\`, \`MultiRNNCell\`, These
225can be placed inside a static_rnn, dynamic_rnn or a static_bidirectional_rnn container. 226 227In Figure 3 we can see (on the left side) a schematic overview of the process-steps of constructing a RNN Model together with (on the right side) the lines of code accompanying these steps. 228 229 230_Figure 3. A schematic overview of a Recurrent Neural Network implemented in Tensorflow._ 231 232As you can see, we first split the data into a list of N different arrays with \`tf.unstack()\`. Then the type of cell is chosen and passed into the recurrent neural network together with the splitted data. 233 234Now that we have schematically seen how we can create a RNN model, lets have a look at how we can create the different types of models in more detail. 235 236## 3.1 Building the model for a RNN 237 238Above, we have seen what the computational steps of the Neural Network consists of. But we have not yet seen the contents of our \`rnn_model\`, \`lstm_rnn_model\`, \`bidirectional_lstm_rnn_model\`, \`twolayer_lstm_rnn_model\` or \`gru_rnn_model\`. Lets have a look at how these models are constructed in more detail in the few sections below. 239 240\`\`\`python 241def rnn_model(data, num_hidden, num_labels): 242 splitted_data = tf.unstack(data, axis=1) 243 244 cell = tf.nn.rnn_cell.BasicRNNCell(num_hidden) 245 246 outputs, current_state = tf.nn.static_rnn(cell, splitted_data, dtype=tf.float32) 247 output = outputs[-1] 248 249 w_softmax = tf.Variable(tf.truncated_normal([num_hidden, num_labels])) 250 b_softmax = tf.Variable(tf.random_normal([num_labels])) 251 logit = tf.matmul(output, w_softmax) + b_softmax 252 return logit 253\`\`\` 254 255As you can see, we first split the Tensor containing the data (size batch_size, 128, 9) into a list of 128 Tensors of size (batch_size, 9) each. This is used, together with \`BasicRNNCell\` as an input for the \`static_rnn\`, which gives us a list of outputs (also of length 128). The last output in this list (the last time step) contains information from all previous timesteps, so this is the output we will use to classify this signal. 256 257**BasicRNNCell** is the most basic and vanille cell present in Tensorflow. It is an basic implementation of a RNN cell and does not have an LSTM implementation like **BasicLSTMCell** has. The accuracy you can achieve with \`BasicLSTMCell\` therefore is higher than \`BasicRNNCelll\`. 258 259## 3.2 From BasicRNNCell to BasicLSTMCell (and beyond) 260 261Since it does not have LSTM implemented, BasicRNNCell has its limitations. Instead of a BasicRNNCell we can use a BasicLSTMCell or an LSTMCell. Both are comparable, but a **LSTMCell** has some additional options like peephole structures, clipping of values, etc. 262 263\`\`\`python 264def rnn_lstm_model(data, num_hidden, num_labels): 265 splitted_data = tf.unstack(data, axis=1) 266 267 cell = tf.nn.rnn_cell.BasicLSTMCell(num_hidden) 268 269 outputs, current_state = tf.nn.static_rnn(cell, splitted_data, dtype=tf.float32) 270 output = outputs[-1] 271 272 w_softmax = tf.Variable(tf.truncated_normal([num_hidden, num_labels])) 273 b_softmax = tf.Variable(tf.random_normal([num_labels])) 274 logit = tf.matmul(output, w_softmax) + b_softmax 275 return logit 276\`\`\` 277 278## 3.3 GruCell: A Gated Recurrent Unit Cell 279 280Besides BasicRNNCell and BasicLSTMCell, Tensorflow also contains **GruCell**, which is an abstract implementation of the Gated Recurrent Unit, proposed in 2014 by [Kyunghyun Cho et al](https://arxiv.org/abs/1406.1078). 281 282\`\`\`python 283def gru_rnn_model(data, num_hidden, num_labels): 284 splitted_data = tf.unstack(data, axis=1) 285 286 cell = tf.contrib.rnn.GRUCell(num_hidden) 287 288 outputs, current_state = tf.nn.static_rnn(cell, splitted_data, dtype=tf.float32) 289 output = outputs[-1] 290 291 w_softmax = tf.Variable(tf.truncated_normal([num_hidden, num_labels])) 292 b_softmax = tf.Variable(tf.random_normal([num_labels])) 293 logit = tf.matmul(output, w_softmax) + b_softmax 294 return logit 295\`\`\` 296 297## 3.4 bi-directional LSTM RNN 298 299The vanille RNN and LSTM RNN models we have seen so far, assume that the data at a step $$t$$ only depend on 'past' events. A bidirectional LSTM RNN, assumes that the output at step $$t$$ can also depen
299d on the data at future steps. This is not so strange if you think about applications in text analytics or speech recognition: subjects often precede verbs, adjectives precede nouns and in speech recognition the meaning of current sound may depend on the meaning of the next few sounds. 300 301To implement a bidirectional RNN, two BasicLSTMCell's are used; the first one looks for temporal dependencies in the backward direction and the second one for dependencies in the forward direction. 302 303\`\`\`python 304def bidirectional_rnn_model(data, num_hidden, num_labels): 305 splitted_data = tf.unstack(data, axis=1) 306 307 lstm_cell1 = tf.nn.rnn_cell.BasicLSTMCell(num_hidden, forget_bias=1.0, state_is_tuple=True) 308 lstm_cell2 = tf.nn.rnn_cell.BasicLSTMCell(num_hidden, forget_bias=1.0, state_is_tuple=True) 309 outputs, _, _ = tf.nn.static_bidirectional_rnn(lstm_cell1, lstm_cell2, splitted_data, dtype=tf.float32) 310 311 output = outputs[-1] 312 313 w_softmax = tf.Variable(tf.truncated_normal([num_hidden*2, num_labels])) 314 b_softmax = tf.Variable(tf.random_normal([num_labels])) 315 logit = tf.matmul(output, w_softmax) + b_softmax 316 return logit 317\`\`\` 318Â 319## 3.5 Two-layered RNN 320
321We have seen how we can implement a bi-directional LSTM by stacking two LSTM Cells on top of each other, where the first on looks for sequential dependencies in the forward direction, and the second one in the backward direction. You could also place two LSTM cells on top of each other, simply to increase the neural network strength. 322 323\`\`\`python 324def twolayer_rnn_model(data, num_hidden, num_labels): 325 splitted_data = tf.unstack(data, axis=1) 326 327 cell1 = tf.nn.rnn_cell.BasicLSTMCell(num_hidden, forget_bias=1.0, state_is_tuple=True) 328 cell2 = tf.nn.rnn_cell.BasicLSTMCell(num_hidden, forget_bias=1.0, state_is_tuple=True) 329 cell = tf.nn.rnn_cell.MultiRNNCell([cell1, cell2], state_is_tuple=True) 330 331 outputs, state = tf.nn.static_rnn(cell, splitted_data, dtype=tf.float32) 332 output = outputs[-1] 333 334 w_softmax = tf.Variable(tf.truncated_normal([num_hidden, num_labels])) 335 b_softmax = tf.Variable(tf.random_normal([num_labels])) 336 logit = tf.matmul(output, w_softmax) + b_softmax 337 return logit 338\`\`\` 339Â 340## 3.6 Multi-layered RNN 341 342In this RNN network, n layers of RNN are stacked on top of each other. The output of each layer is mapped into the input of the next layer, and this allows the RNN to hierarchically looks for temporal dependencies. With each layer the representational power of the Neural Network increases (in theory). 343 344\`\`\`python 345def multi_rnn_model(data, num_hidden, num_labels, num_cells = 4): 346 splitted_data = tf.unstack(data, axis=1) 347 348 lstm_cells = [] 349 for ii in range(0,num_cells): 350 lstm_cell = tf.nn.rnn_cell.BasicLSTMCell(num_hidden, state_is_tuple=True) 351 lstm_cells.append(lstm_cell) 352 353 cell = tf.nn.rnn_cell.MultiRNNCell(lstm_cells, state_is_tuple=True) 354 outputs, state = tf.nn.static_rnn(cell, splitted_data, dtype=tf.float32) 355 output = outputs[-1] 356 357 w_softmax = tf.Variable(tf.truncated_normal([num_hidden, num_labels])) 358 b_softmax = tf.Variable(tf.random_normal([num_labels])) 359 logit = tf.matmul(output, w_softmax) + b_softmax 360 return logit 361\`\`\` 362Â 363The num_layer parameter determines how many layers are used to determine the temporal dependencies in the data. The more layers you have, the higher the representation power of the RNN is. 364 365# 4. Classification results 366
367We have seen how we can build several different types of Recurrent Neural Networks. The question then is, how do these RNN's perform in practice? Does the accuracy really increase a lot with the number of layers or the number of hidden units? What is the effect of the chosen optimizer and the learning rate? 368 369In each image you can see the final accuracy in the test set for different learning rates, models, optimizers and hidden units. 370 371 372|  |  |  | 373|  |  |  | 374 375# 5. Conclusion and Final Words 376 377In this blog-post we have seen how we can build an Recurrent Neural Network in Tensorflow, from a vanille RNN model, to an LSTM RNN, GRU RNN, bi-directional or multi-layered RNN's. Such Recurrent Neural Networks are (powerful) tools which can be used for the analysis of time-series data or other data which is sequential in nature (like text or speech). 378 379What I have noticed so far is: 380 381- The most important factor to achieve high accuracy values is the chosen learning rate. It should be carefully tuned, first with large steps than with finer steps. 382- [AdamOptimizer usually performs best](http://www.fast.ai/2018/07/02/adam-weight-decay/). 383- More hidden units is not necessarily better. In any case, if you change the number of hidden units, you probably need to find the optimum value for learning rate again. 384- For the type of RNN also; more layers is not necessarily better. BasicRNNCell has the worst performance, but except for BasicRNNCell there is no single implementation which outperforms all others in all regards. If you implement a RNN containing a **BasicLSTMCell** and carefully tune the learning rate and implement some l2-regularization it should be good enough for most applications. 385- I am not that impressed with RNN's in general. Same accuracy values can be / are achieved with simple [stochastic analysis techniques](http://ataspinar.com/2018/04/04/machine-learning-with-signal-processing-techniques/) with much less effort. With Stochastic analysis techniques you also have the benefit of knowing what the characteristic feature of each type of signal is. 386 387[1] If you feel like you need to refresh your understanding of CNNâs, here are some good starting points to get you up to speed: 388- [Machine Learning is fun!](https://medium.com/@ageitgey/machine-learning-is-fun-part-5-language-translation-with-deep-learning-and-the-magic-of-sequences-2ace0acca0aa) 389- [Colah's blog](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) 390- [WildML on RNN](http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/) 391- [DeepLearning4J on RNN](https://deeplearning4j.org/lstm.html) 392- [The deeplearning book](http://www.deeplearningbook.org/contents/rnn.html) 393- [Some more resources](https://github.com/kjw0612/awesome-rnn) 394`;export{e as default};
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.