PyTorch: Text Generation using LSTM Networks (Character-based RNN)¶

Text Generation also referred to as Natural Language Generation is a kind of Language Modeling problem where we build a model that tries to understand the structure of a text and produce another text. Tasks like machine translation, conversational systems (chatbots), speech-to-text, text summarization, etc at their core try to build language models. Now a day's deep learning models are developed for language modeling tasks. The language model in the case of text generation tries to predict the next token (character/word/n-gram) in text-based on previously seen tokens. In order to predict the next token in sequence, the language model needs to understand the sequence in which tokens are laid out. Deep Learning Recurrent Neural Networks (RNNs) and their variants (LSTM, GRU, etc) are quite good at understanding the sequence of input data hence can be used for language modeling tasks.

As a part of this tutorial, we have explained how we can create Recurrent Neural Networks consisting of LSTM layers using Python deep learning library PyTorch for text generation task. In this tutorial, we have used Character-based approach for text generation tasks where the model takes a specified number of characters as input and predicts the next character in the sequence. In the same way, we can also create networks that take a sequence of words as input and predicts the next word. We have used bag of words approach for encoding text data. We have used the Wikipedia text corpus available from torchtext library (PyTorch NLP tasks helper library) for our purpose. We have another tutorial on text generation using Pytorch which uses character embeddings for encoding text data. Please feel free to check it from the below link.

• Text Generation using PyTorch LSTM Networks (Character Embeddings)

Please make a NOTE that language models are generally big and take time to train until they can produce some meaningful text. It will be hard to train them on CPU and GPU can help with faster training hence we recommend training language models on GPU.

Below, we have listed important sections of Tutorial to give an overview of the material covered.

Important Sections Of Tutorial¶

1. Prepare Data
   • 1.1 Load Data
   • 1.2 Populate Vocabulary
   • 1.3 Reshape Examples to Create Sequence Of Data
   • 1.4 Create Data Loaders
2. Define LSTM Network
3. Train Network
4. Generate Text
5. Train Network More
6. Generate Text
7. Train Even More
8. Generate Text
9. Further Suggestions

Below, we have imported the necessary Python libraries and printed the versions that we have used in our tutorial.

1. Prepare Data ¶

In this section, we are preparing our data for training our network. As we said earlier, we are going to use character-based approach for text generation hence we'll feed a few characters to the network and make it predict the next character in the sequence. We have decided to use 100 characters sequence to network and make it predict the next character after them.

We'll be encoding data using bag of words approach. We'll follow the below steps to encode and prepare data.

1. Create a vocabulary of all unique characters of the data. A vocabulary is a simple mapping from characters to their integer index. Each unique character is assigned a unique index starting from 0.
2. Loop through data sequentially one character at a time. Take the first 100 characters as data features and the next character after them as the target value. E.g characters 1-100 data features and character 101 target value, characters 2-101 data features and character 102 target value, characters 3-102 data features and 103 target value, and so on.
3. Replace each character with their unique index as per vocabulary.

The data generated after following the above steps will be given to the LSTM network for processing. The network will process a sequence of 100 characters at a time and try to predict the next character. We have explained the steps in more detail below to make them easier to grasp.

1.1 Load Data¶

In this section, we have simply loaded our Wikipedia dataset. The dataset is already divided into the train, test, and validation sets. We'll use only the train set for our task. The train set has ~36k text examples. Each example represents a Wikipedia article.

1.2 Populate Vocabulary¶

In this section, we are building a vocabulary of all unique characters present in our dataset. In order to create a vocabulary, we have used build_vocab_from_iterator() function available from 'vocab' sub-module of torchtext library. The function accepts an iterator that returns a list of characters on each call. We have created a small function named build_vocabulary() that works as an iterator. The function takes datasets as input and loops through all datasets and their examples one at a time yielding list of characters. Our text examples have a special token named <unk> which represents the unknown character and we have done special handling of it to count it as one token instead of breaking it into characters.

After building vocabulary, we have printed vocabulary and the number of characters present in it.

1.3 Reshape Examples to Create Sequence Of Data¶

In this section, we are reorganizing our dataset examples so that they can be used to train our LSTM network. We are simply looping through each text example of our train dataset. For each text example, we are sliding a window of 100 characters. We are taking 100 characters as data features and the next character in the sequence as the target value, then we move the window by 1 character and continue the process until we reach the end of the text. We have also replaced each character with its integer index using our vocabulary. Please make a NOTE that we have not used all examples available from the dataset for the training model as it'll take quite long.

After organizing the dataset, we have converted them to torch tensors. We have also added one extra dimension at the end in order to feed data to the LSTM layer.

Below, we have tried to explain the process with a simple example.

vocab = {
'h':1,
'e':2,
'l':3,
'o':4,
' ':5,
',':6,
'w',7,
'a':8,
'r':9,
'y':10,
'u':11,
'?':12,
'c':13,
'm':14,
't':15,
'd':16,
'z':17,
'n':18
}

text_example = "Hello, How are you? Welcome to coderzcolumn?"
seq_length = 10

X_train = [
            ['h','e','l','l','o',',',' ', 'h','o','w'],
            [,'e','l','l','o',',',' ', 'h','o','w',' '],
            ['l','l','o',',',' ', 'h','o','w', ' ', 'a'],
            ['l','o',',',' ', 'h','o','w',' ', 'a', 'r'],
            ...
            ['d','e','r','z','c','o','l', 'u','m','n']
            ]
Y_train = ['e','l','l','o',',',' ', 'h','o','w',' ',..., '?']

X_train_vectorized = [
                        [1,2,3,4,5,6,1,4,7],
                        [2,3,4,5,6,1,4,7,5],
                        [3,4,5,6,1,4,7,5,1],
                        ...
                        [16,2,9,17,13,4,3,11,14,18]
                     ]
Y_train_vectorized = [1,2,3,4,5,6,1,4,7,5,1,...., 12]

1.4 Create Data Loaders¶

In this section, we have simply wrapped our torch tensors in the dataset and created a data loader from it. The data loader will let us process data in batches during the training process. We have set the batch size of 1024.

2. Define LSTM Network ¶

In this section, we have defined a neural network that we'll use for our task. Our task will be considered a classification task as our network predicts one of the characters from the vocabulary.

The network that we have defined consists of 2 LSTM layers and one linear layer. The output size of each LSTM layer is set at 256. The usage of two consecutive LSTM layers will help us better capture the sequence of characters found in the data. We have defined LSTM layers using LSTM() constructor where we have provided the value of num_layers parameter as 2 instructing it to stack to LSTM layers. The output of the second LSTM layer is given to Linear layer which has output units the same as the size of the vocabulary.

After defining the network, we initialized it, printed the shape of weights/biases of layers, and performed a forward pass for verification purposes.

If you are someone who is new to PyTorch or don't have a background on LSTM Networks then we recommend that you go through the below links as they will help you with the background. We have not covered the inner workings of LSTM in-depth here as it is already covered there.

• Create Simple PyTorch Neural Networks using 'torch.nn' Module
• PyTorch: LSTM Networks For Text Classification Tasks

3. Train Network ¶

Here, we are training our network. To simplify the training process, we have created a helper training function. The function takes the model, loss function, optimizer, train data loader, and a number of epochs as input. It then executes a training loop number of epochs times looping through whole training data in batches each time. For each batch of data, it performs a forward pass to make predictions, calculates loss, calculates gradients, and updates network parameters using gradients. It records the loss value for each batch and prints the average loss value of all batches at the end of each epoch.

Below, we are actually training our network using the training routine from the previous cell. We have initialized a number of epochs to 25 and the learning rate to 0.001. Then, we have initialized cross entropy loss, our LSTM model, and Adam optimizer. At last, we have called our training routine with the necessary parameters to perform training. We have trained the network for 25 epochs to see what kind of results it produces. We can notice from the loss value getting printed after each epoch that the network seems to be doing a good job at learning the sequence of characters.

4. Generate Text ¶

In this section, we are trying to generate data using our trained network. We have first retrieved a random text example from our organized train dataset. We have then printed the characters of that example. Then, we have a loop that generates 100 new characters. The logic starts with the initial randomly selected sequence and makes the next character prediction. It then removes the first character from the sequence and adds a newly predicted character at the end. Then, it makes another prediction and the process repeats for 100 characters.

We can notice from the results that our model is not making any spelling errors even though it is predicting one character at a time. The sequence of characters generated does not make much sense but seems like an English language sentence. It is also predicting punctuation marks. The model is a little deterministic and repeats the sequence of characters after some time. This can be avoided by introducing some kind of randomness to the output of the network.

The results look overall good as we have trained the network for just 25 epochs. Next, we'll train the network for more epochs and hopefully, it should improve results further.

5. Train Network More ¶

In this section, we have trained our network for another 50 epochs. We have also reduced the learning rate from 0.001 to 0.0003. We can notice from the loss values getting printed that it is decreasing at every epoch which means that our network is getting good at the text generation task.

6. Generate Text ¶

Here, we have again generated new characters using our more trained network. We have used the same example that we had used earlier. We can notice that results seem to have improved a little bit. The model is not making spelling mistakes and new words are generated for the same example. The network still seems deterministic and produces the same characters again and again. We can train the network further to see whether it helps or not.

7. Train Even More ¶

In this section, we have trained our network for another 50 epochs. We have reduced the learning rate from 0.0003 to 0.0001. We can notice from the loss values at the end of the epoch that the network is improving further.

8. Generate Text ¶

Here, we are again generating text on the same text example using our trained network. We can notice from the results this time that they are a little better compared to earlier. Though they are still deterministic.

9. Further Suggestions ¶

Below we have suggested a few more things that can be tried to improve network performance further.

1. Train the network for more epochs.
2. Try different combinations of LSTM layers. Maybe add more LSTM layers.
3. Try different hidden sizes for LSTM layers.
4. Try different sequence lengths. In our case, we tried a sequence length of 100 characters.
5. Try using an n-gram/word-based model instead of a character-based.
6. Try adding linear layers in the network after dense layers.
7. Try learning rate schedulers.
8. Try different character encodings like character embeddings, etc.
9. Try to add randomness to the prediction of the next character to make the predicted text look more natural. REFERENCE

This ends our small tutorial explaining how to design LSTM Networks using PyTorch for Text generation tasks. Please feel free to contact us if you questions

References¶

• Text Generation using PyTorch LSTM Networks (Character Embeddings)
• Create Simple PyTorch Neural Networks using 'torch.nn' Module
• PyTorch: LSTM Networks For Text Classification Tasks