In the first tutorial we went through how to write fiction using a premade artificial intelligence. In this tutorial we go through how to train your own custom AI. You will need to collect a corpus of literature and then train your neural network to imitate it. The training process can take hours or days to complete, so make sure you do this in a place where you can leave your computer uninterrupted for a while.
Your AI will write in the same style as the literature you give it to train on. If you give it Shakespeare, then it will write like Shakespeare. This gives you incredible creative power to craft different AIs with different voices.
As in the last tutorial, you can choose between a Mac or Windows tutorial from the tabs below.
This is a short summary of all of the commands and steps in this tutorial. This is mostly useful if you are going through it a second time and don't want to sift through the full details.
1: Install Docker 2: Create a Data File 1-3MB will train in hours, 3-10MB will train in days, 10+MB will train in weeks on a standard laptop.
3: Start Docker Container Terminal Window 1:
docker run -ti --rm --name literai crisbal/torch-rnn:base
4: Copy Data to Docker Container Terminal Window 2:
docker cp ~/Desktop/project/ literai:root/torch-rnn/data/
5: Preprocess Data Terminal Window 1:
python scripts/preprocess.py --input_txt data/project/project.txt --output_h5 data/project.h5 --output_json data/project.json
6: Train Your Model Terminal Window 1:
th train.lua -input_h5 data/project.h5 -input_json data/project.json -checkpoint_name output/checkpoint -rnn_size 512 -num_layers 2 -dropout 0.5 -gpu -1
7: Save Model to Computer Terminal Window 2:
docker cp literai:root/torch-rnn/output/ ~/Desktop/
8: Write Text Terminal Window 1:
th sample.lua -checkpoint output/checkpoint_XX000.t7 -length 2000 -temperature 0.7 -gpu -1
Terminal Window 1:
exit
9: Upload Story to Literai 1: Install Docker
If you completed this as a part of the tutorial 1, you can skip this step.
If you went straight to this tutorial the first step is to install Docker on your computer. You can find a link to download it here: https://docs.docker.com/docker-for-mac/.
Docker is a tool that runs a simple virtual computer filled with the code we want already installed and ready to go. This makes the whole process easier since you will only need to install one program, instead of many.
Once you have installed Docker open it from your Applications folder - you should see an icon for it on the menubar at the top of your mac. When you click on the icon it should say that Docker is running. Next you need to go to preferences and set the number of CPUs and Memory. Setting these values higher will make everything you run through Docker go faster. We recommend setting them to about 75-90% of what you have available.
2: Create a Data File
Creating an artificial intelligence that can write in English would seem to be an incredibly difficult challenge - but neural networks have made it incredibly easy. Neural networks learn how to do things by being given a large number of examples and slowly teaching themselves to imitate these styles.
The neural network you create in this tutorial won't know a single thing about spelling, punctuation, characters, plot, or even words. It will have to learn all of this from scratch from the literature you give it to learn on. Think of it like a child who learns to write by reading from a library with no tutor.
Choosing which books to give your AI is an exciting creative endeavor. The literature it learns to write on will teach it not just the basics of the language, but will also give it a specific "voice". You could give it only literature from one author, or one genre, or get it to imitate an interesting combination of works. The possibilities are endless. It doesn't even need to be in English. Within hours a neural network could be writing in a foreign language more fluently than you'll ever be able to.
For your first AI choose a simple topic, perhaps an author or book series you want to imitate. You will need to open up a simple text editor (eg. TextEdit - a program installed on every mac computer). Copy and paste the full text from the writing you want into the text file. Don't worry about leaving spacing between each book, just paste them in one after another.
Obviously you will need to find the text for the literature you want online. Google can get you most of the way there, but be sure to also check out Project Gutenberg, IMSDB, and MLDB. Also check out the data files attached to stories on Literai - you can use them to build off of the work others have already done bringing writings together. You can find stories that have data files attached on Literai by looking for the data icon:
At this point you will want to come up with a name for your project. Once you have your name chosen you should save your text file as project.txt and place it in a new folder with the same name on your Desktop. Throughout this tutorial we are going to keep using the word project and you will need to replace it in all command line arguments with whatever word you chose to name your project.
The larger your dataset the better your AI will be at writing, but the longer it will take to learn. As a general rule of thumb about 1MB (200,000 words) is the minimum size you would want to train on. Anything in the 3-10MB range is pretty ideal and will take a reasonable amount of time to run. Large datasets of 10s or 100s of MBs can create incredibly good writers, but they might take weeks or even months to train on a personal computer.
3: Start Docker Container
If you've already completed the first tutorial, this step is the same as last time - you can skip to copying the command below into the Terminal. If you haven't completed the first tutorial then read on.
Now that we have Docker running and our project set up we are ready to start! To get started we are going to need to open the Mac Application "Terminal". Terminal is a command line interface for your computer. It can be a little daunting if you've never used it before - but don't worry, you'll just be copying and pasting commands into it.
Our first step is to start up a Docker Container with all of the packages and code we want. We will be using a program called torch-rnn written in the programming language Lua by Justin Johnson (based on char-rnn by Andrej Karpathy). Cristian Baldi has created a Docker Image we can download automatically that has the program and all of its dependencies set up ready to go. (Note: the initial download is 500MB)
All you have to do is paste the following code into the terminal and hit return:
docker run -ti --rm --name literai crisbal/torch-rnn:base
It will take a few moments to download everything you need. When it is done the command prompt will come up again indicating that the process finished correctly. Now the terminal is controlling the inside of your Docker Container. If something went wrong make sure your internet connection is working and that Docker is running. In the future, when you do this step again it will happen very quickly, as it won't need to download anything.
4: Copy Data into Docker Container
The next step is to copy the data file from your computer into the Docker Container. To do this you need to open a new Terminal window (Cmd + N or right click on the terminal icon in the dock and choose "New Window"). We need a second window as our first window is now controlling the Docker Container.
Paste the following command into the second terminal window and hit return to copy the folder you made across. If your folder with your data file (project.txt) isn't on the Desktop you will need to modify the path in the code to link to the right place. Remember to replace project with your own project name.
docker cp ~/Desktop/project/ literai:root/torch-rnn/data/
Optional: To check that the folder was correctly copied, go back to your first terminal window, type cd torch-rnn/data/ and hit enter, then type ls and hit enter. You should see a list of files/folders that includes the one you just copied over. Now type cd .. and hit enter two times to go back to where you were.
5: Preprocess Data
Before we can train our neural network we need to run a couple of processes to get the data ready for it. There are some advanced parameters you might want to tweak at this step (see the tips and tricks section), but for the most part you'll probably want to use the default command below.
Go to the first Terminal window (the one controlling your Docker Container), and enter the following command before pressing enter:
python scripts/preprocess.py --input_txt data/project/project.txt --output_h5 data/project.h5 --output_json data/project.json
Optional: To check that the data files were processed correctly type cd torch-rnn/data/ into the first terminal window and hit enter, then type ls and hit enter. You should see a list of files/folders that includes project.h5 and project.json. Now type cd .. and hit enter to go back to where you were.
6: Train Your Model
This is the exciting (and long) part, where we train your neural network to write. Before we do that though we should make sure your computer won't shutdown or go to sleep while the process is running. If you are running this on a laptop make sure it is plugged in and that you are in a place where the laptop can stay open for a long time. Secondly, go to "System Preferences" and enter the "Energy Saver" mode. For both the "Battery" and "Power Adapter" tabs, move the slider for "turn display off after" to "never". This will prevent your computer from going to sleep. Make sure to put it back to normal after you are done with this tutorial.
Now that you are ready to train your neural network, go to the first Terminal Window (the one controlling the Docker Container), paste the following command and hit enter:
th train.lua -input_h5 data/project.h5 -input_json data/project.json -checkpoint_name output/checkpoint -rnn_size 512 -num_layers 2 -dropout 0.5 -gpu -1
You'll notice a few different parameters in the command above. See the tips and tricks section for more information on how to tweak these to get better results. In short though, the larger the rnn_size is the larger your neural network's brain is. In general you should increase rnn_size only when working with larger data files. Doing so will give you much better results but can drastically increase the time it takes to learn.
Now that the command is running you will see lines output to the terminal one after another. This is showing you information about each iteration of the model's learning - the value on the right (loss =) is an evaluation metric which indicates how good your model is. The smaller this number is the better. You'll notice it very slowly going down over time. Every 1000 iterations a copy of your model will be saved (a checkpoint) and a special evaluation will be done to see how good it is. Any checkpoint can be used to write, so you can actually see how your AI is getting better over time.
This process will take a very long time (hours to days depending on your data and parameters), so you can just leave it running. If you decide to end the run prematurely you can do so by hitting Ctrl+C. You can then just take the last checkpoint created for the next step.
7: Save Model to Your Computer
With the training now completed, you now have a neural network (model) capable of writing for you! The first thing you'll want to do is copy this model from the Docker Container on to your computer. Go to the second terminal window (that we used in step 4), paste the following command and hit enter:
docker cp literai:root/torch-rnn/output/ ~/Desktop/
When this is done running you will see a new folder on your desktop called "output". Rename it to your project name. Inside there will be a number of files with the name "checkpoint_XX000.t7" and "checkpoint_XX000.json" where XX is a specific number. Take the .t7 file with the highest number and rename it project.t7 . This file is your completely trained model. It can now be shared and used as the input (step 2) of the first tutorial.
But since we've already got a Docker Container running we don't need to start that all over again, instead we can jump straight to writing text.
8: Write Text
Now that everything is set up we are finally ready to write some text! For this step we need to go back to the first terminal window where we will copy and paste the command below. You will need to replace the XX in the checkpoint name with the number from the actual checkpoint file you are using (note that you only renamed it on your computer, not within the docker container).
th sample.lua -checkpoint output/checkpoint_XX000.t7 -length 2000 -temperature 0.7 -gpu -1
After a couple of minutes the computer should print out 2000 characters of text! Your AI has just written entirely novel literature! You can copy this text out of the terminal into a text editor (like Word or Google Docs).
To create more text, simply paste the command again (or hit the up arrow in the terminal to auto-populate it). You'll notice in the command is the phrase -length 2000 You can modify this number to be bigger or smaller to see more or less text. In practice you can sometimes get better results by generating lots of shorter snippets rather than a really long one. There are other parameters you can tweak here to get better results. The tips and tricks tutorial goes over these in more detail.
When you are done generating text it is important to enter one last command into the first terminal window:
exit
This will close down the Docker Container and will delete everything on it. Importantly, this will also free up the name "literai" which we gave to the container in step 3. If you forget to exit and then come back to write more fiction in the future the code we have will cause an error, since the name "literai" will still be taken. You can solve this by using a different name (eg literai2) in all of the commands, but for simplicity remember to type exit!
9: Upload Your Story to Literai
Now that you've generated text with your AI you can complete a story and upload it to Literai to share with the world.
This is your chance to be creative in selecting and editing the writing your AI produces. Find the funny interesting things it says and discard the others. On literai we strongly encourage quality over quantity. Computers can write infinite amounts of fiction, but humans only really want to read a finite amount, usually no more than a page of writing. Keep it short and grab the best stuff you can. We discourage directly editing the words the AI wrote, but fixing minor spelling mistakes, spacing issues, or renaming characters to create continuity is fine.
When your story is complete go to the Upload Story page on Literai and enter the story, a short synopsis about the story, and a description of how you created it under Methods. Remember to upload the model (.t7) file with your story, so other people can download it to write more like it. You should also upload your data (.txt) file so other people can build off of the literature you collected.
You've now successfully completed the second tutorial! Next you can read the How it Works or Tips and Tricks sections. Experiment with different literature and come up with some exciting new stories!
This is a short summary of all of the commands and steps in this tutorial. This is mostly useful if you are going through it a second time and don't want to sift through the full details.
1: Install Docker 2: Create a Data File 1-3MB will train in hours, 3-10MB will train in days, 10+MB will train in weeks on a standard laptop.
3: Start Docker Container Command Prompt 1:
docker run -ti --rm --name literai crisbal/torch-rnn:base
4: Copy Data to Docker Container Command Prompt 2:
docker cp C:\Users\Admin\Documents\project\ literai:root/torch-rnn/data/
5: Preprocess Data Command Prompt 1:
python scripts/preprocess.py --input_txt data/project/project.txt --output_h5 data/project.h5 --output_json data/project.json
6: Train Your Model Command Prompt 1:
th train.lua -input_h5 data/project.h5 -input_json data/project.json -checkpoint_name output/checkpoint -rnn_size 512 -num_layers 2 -dropout 0.5 -gpu -1
7: Save Model to Computer Command Prompt 2:
docker cp literai:root/torch-rnn/output/ C:\Users\Admin\Documents\
8: Write Text Command Prompt 1:
th sample.lua -checkpoint output/checkpoint_XX000.t7 -length 2000 -temperature 0.7 -gpu -1
Command Prompt 1:
exit
9: Upload Story to Literai 1: Install Docker
The first step for this tutorial is to install Docker on your computer. You can find a link to download it here: https://docs.docker.com/docker-for-windows/.
Docker is a tool that runs a simple virtual computer filled with the code we want already installed and ready to go. This makes the whole process easier since you will only need to install one program, instead of many.
Please note that Docker requires Hyper-V which only available on Windows Professional and up (not on Windows Home Edition). I ran into this problem on my own PC when making this tutorial. The only solution I know is to upgrade to Windows Pro for $100 - not fun, but it works. Otherwise, use the Mac tutorial on a Mac or Linux machine. [Update: Apparently it is possible to get docker to run on Windows Home edition using Virtual Box]
Once you have installed Docker you should see an icon for it in the bottom right of your taskbar. When you click on the icon it should say that Docker is running. Next you need to go to settings and set the number of CPUs and Memory. Setting these values higher will make everything you run through Docker go faster. We recommend setting them to about 75-90% of what you have available.
2: Create a Data File
Creating an artificial intelligence that can write in English would seem to be an incredibly difficult challenge - but neural networks have made it incredibly easy. Neural networks learn how to do things by being given a large number of examples and slowly teaching themselves to imitate these styles.
The neural network you create in this tutorial won't know a single thing about spelling, punctuation, characters, plot, or even words. It will have to learn all of this from scratch from the literature you give it to learn on. Think of it like a child who learns to write by reading from a library with no tutor.
Choosing which books to give your AI is an exciting creative endeavor. The literature it learns to write on will teach it not just the basics of the language, but will also give it a specific "voice". You could give it only literature from one author, or one genre, or get it to imitate an interesting combination of works. The possibilities are endless. It doesn't even need to be in English. Within hours a neural network could be writing in a foreign language more fluently than you'll ever be able to.
For your first AI choose a simple topic, perhaps an author or book series you want to imitate. You will need to open up a simple text editor (eg. notepad). Copy and paste the full text from the writing you want into the text file. Don't worry about leaving spacing between each book, just paste them in one after another.
Obviously you will need to find the text for the literature you want online. Google can get you most of the way there, but be sure to also check out Project Gutenberg, IMSDB, and MLDB. Also check out the data files attached to stories on Literai - you can use them to build off of the work others have already done bringing writings together. You can find stories that have data files attached on Literai by looking for the data icon:
At this point you will want to come up with a name for your project. Once you have your name chosen you should save your text file as project.txt and place it in a new folder with the same name in your Documents folder. Throughout this tutorial we are going to keep using the word project and you will need to replace it in all command line arguments with whatever word you chose to name your project.
The larger your dataset the better your AI will be at writing, but the longer it will take to learn. As a general rule of thumb about 1MB (200,000 words) is the minimum size you would want to train on. Anything in the 3-10MB range is pretty ideal and will take a reasonable amount of time to run. Large datasets of 10s or 100s of MBs can create incredibly good writers, but they might take weeks or even months to train on a personal computer.
3: Start Docker Container
If you've already completed the first tutorial, this step is the same as last time - you can skip to copying the command below into the Command Prompt. If you haven't completed the first tutorial then read on.
Now that we have Docker running and our project set up we are ready to start! To get started we are going to need to open the Windows program "Command Prompt". To open this go to the start menu and search for cmd. Command Prompt is a command line interface for your computer. It can be a little daunting if you've never used it before - but don't worry, you'll just be copying and pasting commands into it.
Our first step is to start up a Docker Container with all of the packages and code we want. We will be using a program called torch-rnn written in the programming language Lua by Justin Johnson (based on char-rnn by Andrej Karpathy). Cristian Baldi has created a Docker Image we can download automatically that has the program and all of its dependencies set up ready to go. (Note: the initial download is 500MB)
All you have to do is paste the following code into the command prompt and hit return:
docker run -ti --rm --name literai crisbal/torch-rnn:base
It will take a few moments to download everything you need. When it is done the command prompt will come up again indicating that the process finished correctly. Now the command prompt is controlling the inside of your Docker Container. If something went wrong make sure your internet connection is working and that Docker is running. In the future, when you do this step again it will happen very quickly, as it won't need to download anything.
4: Copy Data into Docker Container
The next step is to copy the data file from your computer into the Docker Container. To do this you need to open a new Command Prompt window (go to the start menu, search for cmd, and open it like before). We need a second window as our first window is now controlling the Docker Container.
Paste the following command into the second command prompt window and hit return to copy the folder you made of your model across. If the folder with your data file (project.txt) isn't oin your Documents you will need to modify the path in the code to link to the right place. Remember to replace project with your own project name.
docker cp C:\Users\Admin\Documents\project\ literai:root/torch-rnn/data/
5: Preprocess Data
Before we can train our neural network we need to run a couple of processes to get the data ready for it. There are some advanced parameters you might want to tweak at this step (see the tips and tricks section), but for the most part you'll probably want to use the default command below.
Go to the first Command Prompt (the one controlling your Docker Container), and enter the following command before pressing enter. (Notice that you can't paste using Ctrl+V in the window controlling your Docker container, you can instead right click on the line to paste instead).
python scripts/preprocess.py --input_txt data/project/project.txt --output_h5 data/project.h5 --output_json data/project.json
6: Train Your Model
This is the exciting (and long) part, where we train your neural network to write. Before we do that though we should make sure your computer won't shutdown or go to sleep while the process is running. If you are running this on a laptop make sure it is plugged in and that you are in a place where the laptop can stay open for a long time. Secondly, go to the Power Options section of your computer's Control Panel (you can search for this in the start menu). Choose the display and sleep settings to never put your computer to sleep. Make sure to put it back to normal after you are done with this tutorial.
Now that you are ready to train your neural network, go to the first command prompt (the one controlling the Docker Container), paste the following command and hit enter:
th train.lua -input_h5 data/project.h5 -input_json data/project.json -checkpoint_name output/checkpoint -rnn_size 512 -num_layers 2 -dropout 0.5 -gpu -1
You'll notice a few different parameters in the command above. See the tips and tricks section for more information on how to tweak these to get better results. In short though, the larger the rnn_size is the larger your neural network's brain is. In general you should increase rnn_size only when working with larger data files. Doing so will give you much better results but can drastically increase the time it takes to learn.
Now that the command is running you will see lines output to the command prompt one after another. This is showing you information about each iteration of the model's learning - the value on the right (loss =) is an evaluation metric which indicates how good your model is. The smaller this number is the better. You'll notice it very slowly going down over time. Every 1000 iterations a copy of your model will be saved (a checkpoint) and a special evaluation will be done to see how good it is. Any checkpoint can be used to write, so you can actually see how your AI is getting better over time.
This process will take a very long time (hours to days depending on your data and parameters), so you can just leave it running. If you decide to end the run prematurely you can do so by hitting Ctrl+C. You can then just take the last checkpoint created for the next step.
7: Save Model to Your Computer
With the training now completed, you now have a neural network (model) capable of writing for you! The first thing you'll want to do is copy this model from the Docker Container on to your computer. Go to the second command prompt (that we used in step 4), paste the following command and hit enter:
docker cp literai:root/torch-rnn/output/ C:\Users\Admin\Documents\
When this is done running you will see a new folder in your Documents folder called "output". Rename it to your project name. Inside there will be a number of files with the name "checkpoint_XX000.t7" and "checkpoint_XX000.json" where XX is a specific number. Take the .t7 file with the highest number and rename it project.t7 . This file is your completely trained model. It can now be shared and used as the input (step 2) of the first tutorial.
But since we've already got a docker container running we don't need to start that all over again, instead we can jump straight to writing text.
8: Write Text
Now that everything is set up we are finally ready to write some text! For this step we need to go back to the first command prompt where we will copy and paste the command below. You will need to replace the XX in the checkpoint name with the number from the actual checkpoint file you are using (note that you only renamed it on your computer, not within the docker container).
th sample.lua -checkpoint output/checkpoint_XX000.t7 -length 2000 -temperature 0.7 -gpu -1
After a couple of minutes the computer should print out 2000 characters of text! Your AI has just written entirely novel literature! You can copy this text out of the command prompt into a text editor (like Word or Google Docs).
To create more text, simply paste the command again (or hit the up arrow in the command prompt to auto-populate it). You'll notice in the command is the phrase -length 2000 You can modify this number to be bigger or smaller to see more or less text. In practice you can sometimes get better results by generating lots of shorter snippets rather than a really long one. There are other parameters you can tweak here to get better results. The tips and tricks tutorial goes over these in more detail.
When you are done generating text it is important to enter one last command into the first command prompt:
exit
This will close down the Docker Container and will delete everything on it. Importantly, this will also free up the name "literai" which we gave to the container in step 3. If you forget to exit and then come back to write more fiction in the future the code we have will cause an error, since the name "literai" will still be taken. You can solve this by using a different name (eg literai2) in all of the commands, but for simplicity remember to type exit!
9: Upload Your Story to Literai
Now that you've generated text with your AI you can complete a story and upload it to Literai to share with the world.
This is your chance to be creative in selecting and editing the writing your AI produces. Find the funny interesting things it says and discard the others. On literai we strongly encourage quality over quantity. Computers can write infinite amounts of fiction, but humans only really want to read a finite amount, usually no more than a page of writing. Keep it short and grab the best stuff you can. We discourage directly editing the words the AI wrote, but fixing minor spelling mistakes, spacing issues, or renaming characters to create continuity is fine.
When your story is complete go to the Upload Story page on Literai and enter the story, a short synopsis about the story, and a description of how you created it under Methods. Remember to upload the model (.t7) file with your story, so other people can download it to write more like it. You should also upload your data (.txt) file so other people can build off of the literature you collected.
You've now successfully completed the second tutorial! Next you can read the How it Works or Tips and Tricks sections. Experiment with different literature and come up with some exciting new stories!
