Keras: RNNs For Text Classification Tasks¶

When we work with datasets like time-series, text, speech, etc, any example of data has a dependency on previous examples that came before it. With such datasets, in order to predict the output of any text example, using the examples that came before it can give the best results. These kinds of datasets have sequences in data examples that can not be captured by a normal neural network of only dense layers as they can't capture sequences properly (It does not have any memory of previous examples to make predictions). To work with such datasets, Recurrent neural networks (RNNs) were developed. The RNNs and its variant (LSTM and GRU) are best suited for datasets like time series, text, speech, etc.

As a part of this tutorial, we have explained how we can create RNNs using Python deep learning library keras that can be used for text classification tasks. We have tried different versions of RNNs for solving the task. For encoding text data, we have used word embeddings approach. After training RNNs, we have also evaluated their performance by calculating various ML metrics and also explained predictions made by them using LIME algorithm.

In this tutorial, we have primarily created RNNs with vanilla Recurrent layer. If you are looking for a guide on a variant of RNN that consists of LSTM layers then please feel free to check the below link.

• Keras: LSTM Networks For Text Classification Tasks

Below, we have listed essential sections of tutorial to give an overview of the material covered.

Important Sections Of Tutorial¶

1. Prepare Data
   • 1.1 Load Dataset
   • 1.2 Tokenize And Vectorize Text Data
2. Approach 1: Single Recurrent Layer Network (Max Tokens=25, Embedding Length=30, RNN Output=50)
   • Truncate Data At Max Tokens
   • Define Network
   • Compile Network
   • Train Network
   • Evaluate Network Performance
   • Explain Network Predictions using LIME Algorithm
3. Approach 2: Single Recurrent Layer Network (Max Tokens=50, Embedding Length=30, RNN Output=50)
4. Approach 3: Single Bidirectional Recurrent Layer Network (Max Tokens=50, Embedding Length=30, RNN Output=50)
5. Approach 4: Stacking Multiple Recurrent Layers (Max Tokens=50, Embedding Length=30, RNN Output=50,60,75)
6. Approach 5: Stacking Multiple Bidirectional Recurrent Layers (Max Tokens=50, Embedding Length=30, RNN Output=50,60,75)
7. Summarized Results And Further Recommendations

Below, we have imported the necessary libraries and printed the versions we used in our tutorial.

1. Prepare Data ¶

In this section, we are preparing data to be given to the neural network. As we said earlier, we are going to use word embeddings approach for encoding text data. In order to encode text data using this approach, we need to follow 3 steps.

1. Create vocabulary of all unique tokens in the corpus. The vocabulary is a simple mapping from token to integer index. Each token is assigned a unique integer index starting from 0.
2. Tokenize each text example into tokens and assign indexes to each token using vocabulary.
3. Map indexes to real-valued vectors.

The first two steps will be completed in this section. The third step will be implemented through a neural network where we'll have an embedding layer that will map token indexes to their respective real-valued vectors (embeddings).

Below, we have included one image showing how word embeddings look.

1.1 Load Dataset¶

In this section, we have simply loaded AG NEWS dataset available from torchtext which we'll use for our tutorial. The dataset has text documents for four different categories (["World", "Sports", "Business", "Sci/Tech"]) of news.

1.2 Tokenize And Vectorize Text Data¶

In this section, we have completed two steps of encoding mentioned earlier.

First, we have created an instance of Tokenizer() and called fit_on_texts() method on it with train and test text documents. This call will tokenize each text example and populate vocabulary inside of Tokenizer object.

Next, we have called texts_to_sequences() method on Tokenizer object with train and test datasets. It'll tokenize each text example from datasets and retrieve indexes of those tokens. The output of this function call will be a list of indexes of tokens of text examples.

Below, we have explained with a simple example how one text example will be mapped to indexes using vocabulary.

text = "Hello, How are you? Where are you planning to go?"

tokens = ['hello', ',', 'how', 'are', 'you', '?', 'where',
            'are', 'you', 'planning', 'to', 'go', '?']

vocab = {
    'hello': 0,
    'bye': 1,
    'how': 2,
    'the': 3,
    'welcome': 4,
    'are': 5,
    'you': 6,
    'to': 7,
    '<unk>': 8,
}

vector = [0,8,2,4,6,8,8,5,6,8,7,8,8]

Approach 1: Single Recurrent Layer Network (Max Tokens=25, Embedding Length=30, RNN Output=50) ¶

Our first approach uses a single Recurrent layer for the text classification tasks. The network consists of an embedding layer, recurrent layer, and dense layer. The embedding layer is responsible for generating embeddings of input token indexes which will be given to the recurrent layer for processing. The output of the recurrent layer will be given to the dense layer for generating probabilities of target classes.

Truncate Data At Max Tokens¶

Earlier, when we converted our text examples to token indexes the length of each example is different because each text document has a different number of tokens (words). For our approach in this section, we have decided to use a maximum of 25 tokens per text example. To achieve this, we have called pad_sequences() function on our vectorized train and test datasets. This function will make sure that all examples have a maximum of 25 token indexes. For examples which has more than 25 tokens, it'll truncate tokens beyond 25 and for examples that have less than 25 tokens, it'll pad them with 0s to bring length to 25.

Define Network¶

In this section, we have defined the RNN that we'll use for our text classification task. Our network consists of 3 layers.

1. Embedding Layer
2. Recurrent Layer
3. Dense Layer

The embedding layer is the first layer of the network. It takes a list of token indexes as input and returns their respective embeddings. We create embedding layer using Embedding() constructor available from layers sub-module of keras. The layer takes vocabulary and embedding length as input and creates a weight matrix of shape (vocab_length, embed_len). This weight matrix has embeddings for each token. The embedding layer is simply responsible for mapping token indexes to their respective embeddings from this matrix. It takes input of shape (batch_size, max_tokens) = (batch_size, 25) and returns output of shape (batch_size, max_tokens, embed_len) = (batch_size, 25, 30).

The output of the embedding layer is given to the Recurrent layer for processing. We have created a recurrent layer using SimpleRNN() constructor of keras. We have provided its output units as 50. The recurrent layer works by looping through embeddings of tokens of a single text example. It takes input of shape (batch_size, max_tokens, embed_len) and returns output of shape (batch_size, max_tokens, rnn_out).

The output of the Recurrent layer is given to Dense layer which has 4 output units that are the same as the number of target classes. The dense layer applies softmax activation to output to create probabilities.

Compile Network¶

In this section, we have compiled our network to use RMSProp optimizer, cross entropy loss, and accuracy metric.

Train Network¶

Here, we have trained our network by calling fit() method. We have asked it to use a batch size of 512. The method trains for 5 epochs. We have provided test data as validation data for verifying its accuracy during training. We can notice from the loss and accuracy values getting printed after each epoch that our model is doing a good job at the text classification task.

Evaluate Network Performance¶

In this section, we have evaluated the performance of our trained network by calculating accuracy score, classification report and confusion matrix metrics on test predictions. We can notice from the accuracy score that our model is doing a good job. We have calculated these metrics using functions available from scikit-learn.

If you want to learn about various ML metrics available from sklearn then please check the below link which covers the majority of them in-depth.

• Scikit-Learn - Model Evaluation & Scoring Metrics

Apart from calculations, we have also plotted a confusion matrix for a better understanding of performance. We can notice from the visualization that the network is doing good at classifying text documents in Business category compared to all other categories.

We have created a confusion matrix plot using Python library scikit-plot. It provides visualization for many ML metrics. Please feel free to check the below link if you want to know about it in detail.

• Scikit-Plot: Visualizing Machine Learning Algorithm Results & Performance Metrics

Explain Network Predictions using LIME Algorithm¶

In this section, we have tried to explain the predictions made by our trained network using LIME algorithm. We have used a python library named lime which provides an implementation of the algorithm. It let us create a visualization that highlights words of text documents that contributed to predicting a particular target label.

If you are new to the concept of LIME and want to learn about it in depth then we recommend that you go through the below links as it'll help you better understand it.

• How to Use LIME to Understand sklearn Models Predictions?
• LIME: Interpret Predictions Of Keras Text Classification Networks

In order to create an explanation using LIME, we first need to create an instance of LimeTextExplainer which we have created below.

Below, we have first created a simple prediction function that takes a list of text examples as input and returns their prediction probabilities. Later on, we'll use this function to generate explanations.

After defining a function, we randomly selected a text example from the test dataset. Then, we have made predictions on it using our trained network. Our network correctly predicts the target label as Business for the selected text example. Next, we'll create an explanation for this selected example.

Below, we have first called explain_instance() method on LimeTextExplainer to create Explanation object. We have provided a text example, prediction function, and target label to the method for creating an explanation. Then, we have called show_in_notebook() method on the explanation object to create the visualization. From visualization, we can notice that words like 'concessions', 'financing', 'airlines', 'pensions', 'bankruptcy', etc will contribute to predicting target label as Business.

Approach 2: Single Recurrent Layer Network (Max Tokens=50, Embedding Length=30, RNN Output=50) ¶

Our approach in this section is precisely the same as our previous approach with only a difference in the maximum tokens we have used per text example. We have increased the token count to 50 per text example. The network architecture and rest of the code are exactly the same as the previous approach. We are doing this experiment to see whether increasing the tokens count per text example helps us improve accuracy further or not.

Truncate Data At Max Tokens¶

In this section, we have simply truncated our vectorized datasets to keep 50 tokens per text example.

Define Network¶

In this section, we have redefined the network that we'll use for the classification task in this section. It is an exact copy of the network we used in the previous section.

Compile Network¶

Here, we have compiled a network to use RMSProp optimizer, cross entropy loss, and accuracy metric.

Train Network¶

Below, we have trained our network using the same settings that we had used in the previous section. The loss and accuracy values getting printed after each epoch hint that our network seems to be doing a good job at the text classification task.

Evaluate Network Performance¶

In this section, we have evaluated the performance of the trained network by calculating accuracy score, classification report and confusion matrix metrics on test predictions. We can notice from the accuracy score that it has improved a little bit compared to our previous approach. We have also plotted the confusion matrix for reference purposes. From the confusion matrix plot, we can notice that our network is doing a good job at classifying text documents of Sports and World categories compared to Sci/Tech and Business categories.

Explain Network Predictions using LIME¶

In this section, we have explained predictions made by our trained network using LIME algorithm. The network correctly predicts the target category as Business. The visualization shows that words like 'airlines', 'united', 'seeks', 'cuts', 'pensions', 'million', 'labor', etc are contributing to predicting target label as Business.

Approach 3: Single Bidirectional Recurrent Layer Network (Max Tokens=50, Embedding Length=30, RNN Output=50) ¶

Our approach in this section again consists of a single recurrent layer but the recurrent layer is bidirectional this time. By default, recurrent layers in Keras are unidirectional which means that it process sequence in the forward direction from beginning to end. In the case of the bidirectional recurrent layer, the sequence is processed in both forward and backward directions to find out patterns in both directions. The majority of the code in this section is the same as the previous with the only difference being the usage of the bidirectional recurrent layer.

Define Network¶

Below, we have defined the network that we'll use in this section. It has the same architecture as our previous approach with the only difference that we have wrapped the recurrent layer inside of Bidirectional() constructor to make it bidirectional.

Compile Network¶

Here, we have compiled a network to use RMSProp as an optimizer, cross entropy as a loss function, and accuracy as an evaluation metric.

Train Network¶

Below, we have trained our network using the same settings we have been using for all our sections. We can notice from the accuracy and loss values getting printed after each epoch that our model is doing a good job at the classification task.

Evaluate Network Performance¶

In this section, we have evaluated the performance of our trained network by calculating accuracy score, classification report and confusion matrix metrics. We can notice from the accuracy score that it is quite better compared to our previous approaches. We have also plotted the confusion matrix for reference purposes.

Explain Network Predictions using LIME Algorithm¶

In this section, we have explained predictions made by our network using LIME. The network correctly predicts the target label as Business for selected text example from the test set. The visualization shows that words like 'airlines', 'pensions', 'bankruptcy', 'concessions', 'labor', 'financing', etc are contributing to predicting target label as Business.

Approach 4: Stacking Multiple Recurrent Layers (Max Tokens=50, Embedding Length=30, RNN Output=50,60,75) ¶

Our approach in this section uses RNN with multiple recurrent layers. We have stacked multiple recurrent layers to check whether they improve accuracy over single recurrent layers or not. The majority of the code in this section is the same as our earlier sections with the only difference being network architecture.

Define Network¶

Below, we have defined the network that we'll use for our task in this section. The network consists of a single embedding layer followed by 3 recurrent layers and one dense layer. The recurrent layers have 50,60 and 75 output units respectively.

Compile Network¶

Here, we have compiled our network to use RMSProp as an optimizer, cross entropy loss, and accuracy as an evaluation metric.

Train Network¶

Below, we have trained our network by using the same settings that we have used for all our previous approaches. We can notice from the loss and accuracy values that the network is doing a good job.

Evaluate Network Performance¶

In this section, we have evaluated the performance of our network by calculating accuracy, classification report and confusion matrix metrics on test predictions. We can notice from the accuracy score that it is the worst of all the approaches we tried till now. This is surprising as we had expected that stacking multiple recurrent layers will give more power to the network to better capture patterns but it turned out to be otherwise. We have also plotted the confusion matrix for reference purposes.

Explain Network Predictions using LIME Algorithm¶

In this section, we have explained network prediction using LIME. The network correctly predicts the target label as Business for the selected text example. According to visualization, words like 'united', 'further', etc are contributing to predicting the target label as Business.

Approach 5: Stacking Multiple Bidirectional Recurrent Layers (Max Tokens=50, Embedding Length=30, RNN Output=50,60,75) ¶

Our approach in this section is exactly the same as our previous approach with one minor change. We are again using multiple RNN layers in the network but this time all layers are bidirectional. The majority of the code is the same as in our previous section with only a change in network architecture.

Define Network¶

Below, we have defined a network that we'll be using in this section. The network architecture is the same as the previous approach with the only change that we have wrapped all RNN layers in Bidirectional() constructor to make them bidirectional. This will force all of them to look for sequences in both forward and backward directions.

Compile Network¶

Here, we have compiled our network to use RMSProp optimizer, cross entropy loss, and accuracy metric.

Train Network¶

Below, we have trained our network using the same settings that we have been using for all our previous approaches. We can notice from the loss and accuracy values getting printed after each epoch that our model is doing a good job at the text classification task.

Evaluate Network Performance¶

In this section, we have evaluated the performance of the trained network as usual by calculating the accuracy score, classification report and confusion matrix metrics. We can notice from the accuracy score that it is the highest accuracy of all our approaches. Apart from metrics calculations, we have also plotted the confusion matrix for reference purposes.

Explain Network Predictions using LIME Algorithm¶

Here, we have tried to explain predictions of our trained network using LIME algorithm. Our network correctly predicts the target category as Business for the selected text example. The visualization created using LIME shows that words like 'concessions', 'financing', 'employees', 'cuts', 'bankruptcy', 'pensions', etc are used to predict the target label as Business which makes sense as they are commonly used words in the business world.

7. Summarized Results And Further Recommendations ¶

Below, we have summarized all approaches we tried and their performance on the test dataset.

Further Suggestions¶

• Try different max tokens.
• Try different embedding lengths.
• Try different RNN layer output units.
• Train network for more epochs.
• Try different optimizers (Adam).
• Try different weight initialization for networks.
• Try to stack the different number of RNN layers.
• Try learning rate schedulers

References¶

• Keras: LSTM Networks For Text Classification Tasks
• Text Classification using Keras Networks
• Keras: Word Embeddings for Text Classification