<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Fan Yeng-Loon on Medium]]></title>
        <description><![CDATA[Stories by Fan Yeng-Loon on Medium]]></description>
        <link>https://medium.com/@ringoshin?source=rss-189f93992c4a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*7gS4O7z8ii1tGXsCNn3UzA.jpeg</url>
            <title>Stories by Fan Yeng-Loon on Medium</title>
            <link>https://medium.com/@ringoshin?source=rss-189f93992c4a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 09:22:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ringoshin/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to install Tensorflow GPU on Ubuntu 18.04 LTS]]></title>
            <link>https://medium.com/analytics-vidhya/how-to-set-up-tensorflow-gpu-on-ubuntu-18-04-lts-7a09ffd5f30f?source=rss-189f93992c4a------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a09ffd5f30f</guid>
            <category><![CDATA[tensorflow-gpu]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[ubuntu]]></category>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[cuda]]></category>
            <dc:creator><![CDATA[Fan Yeng-Loon]]></dc:creator>
            <pubDate>Wed, 29 Jan 2020 08:44:25 GMT</pubDate>
            <atom:updated>2020-02-08T10:10:47.398Z</atom:updated>
            <content:encoded><![CDATA[<h3>Install TensorFlow GPU on Ubuntu 18.04 LTS</h3><h4>GPU accelerated deep learning with NVIDIA CUDA/cuDNN</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*aBcfRK-jUEB1y3EVM1YgQg.png" /></figure><h3><strong>TL;DR</strong></h3><p>How I install compatible NVIDIA CUDA Tookit and cuDNN packages on Ubuntu 18.04 LTS, so that I can speed up Deep Learning with TensorFlow/Keras using GPU on my laptop.</p><h3>Table of Contents</h3><ol><li><a href="#bd39">Disclaimer</a></li><li><a href="#757b">Introduction</a></li><li><a href="#cb8f">Installation Process</a><br>Step 1. <a href="#dcce">Identify Compatibility</a><br>Step 2. <a href="#7b17">Install Video Driver</a><br>Step 3. <a href="#8614">Install CUDA Toolkit</a><br>Step 4. <a href="#d7f8">Install CUDNN Library</a><br>Step 5. <a href="#1927">Update Environment Variables</a><br>Step 6. <a href="#0ec4">Install TensorFlow-GPU and Keras</a><br>Step 7. <a href="#059f">Run Verification Tests</a><br>Step 8. <a href="#ea17">Trial run on a CNN Model</a></li><li><a href="#9fdb">Miscellaneous Tips</a><br>a. <a href="#80bd">What to do if Tensorflow-GPU still does not load?</a><br>b. <a href="#9c3c">How to Manage CUDA Out-of-memory Warnings?</a><br>c. <a href="#6a59">How to Monitor GPU Usage?</a><br>d. <a href="#c7df">How to Kill (Release) Runaway Processes?</a><br>e. <a href="#2786">How I got Keras CuDNNLSTM to work with the latest spaCy?</a></li><li><a href="#32be">Wrap Up</a></li><li><a href="#3532">Handy References</a></li></ol><h3>Disclaimer</h3><p>Before we begin, some caveats are in order.</p><ul><li>This article is not written for Anaconda users, because when you install Tensorflow-gpu under Anaconda, it should take care of said CUDA drivers for you. You can check it out <a href="https://medium.com/@shaolinkhoa/install-tensorflow-gpu-2-0-alpha-on-anaconda-for-windows-10-ubuntu-ced099010b21"><strong>here</strong> </a>and <a href="https://docs.anaconda.com/anaconda/user-guide/tasks/tensorflow/"><strong>here</strong></a><strong>.</strong></li><li>This is primarily for Python users who code with an IDE and virtual environments outside of Anaconda.</li><li>Also, this is based on my experience with Ubuntu 18.04 LTS only. However, I suppose it is possible that the overall installation process can be easily adapted for Ubuntu 16.04 LTS too.</li><li>Sadly, there is no guarantee that your setup will definitely work, even if you have successfully installed and tested with compatible CUDA/cuDNN packages. Additional tweaks and experiments might be required.</li></ul><p><a href="#564e">^</a></p><h3>Introduction</h3><p>Before I joined the <a href="https://metis.kaplan.com.sg/"><strong>Metis</strong></a> Data Science boot camp journey, one of the suggested prerequisites was to prepare a Ubuntu laptop. Naturally, I went for it without hesitation. Installing Ubuntu on my laptop to dual-boot with Windows was a fabulous learning experience worthy of its own article.</p><p>On top of that, I wanted my DL projects to make good use of my laptop GPU (GTX 1050). As it turned out, I spent many hours googling and reinstalling of library packages. Hence, I have decided to blog this for my own reference and hopefully it will help you too if you are also interested.</p><h4>Prerequisites</h4><ul><li>You must have an NVIDIA graphics card installed on your machine. You can find out from the command line:</li></ul><pre><strong>$</strong> lspci | grep -i nvidia</pre><ul><li>You have installed Ubuntu 18.04 LTS (my references: <a href="https://itsfoss.com/install-ubuntu-1404-dual-boot-mode-windows-8-81-uefi/">here</a>, <a href="https://hackernoon.com/installing-ubuntu-18-04-along-with-windows-10-dual-boot-installation-for-deep-learning-f4cd91b58557">here</a> &amp; <a href="https://medium.com/@mishawagon/ubuntu-and-windows-10-dual-boot-setup-2d5e6b5842fe">here</a>)</li><li>You have set up a <a href="https://www.digitalocean.com/community/tutorials/how-to-install-python-3-and-set-up-a-programming-environment-on-ubuntu-18-04-quickstart">Python development environment</a> for machine learning</li></ul><p><a href="#564e">^</a></p><h3>Installation Process</h3><p>Before we begin, it is always good to know how to fall back to previous state should something go wrong. Fortunately, it is quite simple to uninstall as detailed <a href="https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#removing-cuda-tk-and-driver">here</a>:</p><p>To remove CUDA Toolkit:<br><strong>$</strong> sudo apt-get --purge remove &quot;*cublas*&quot; &quot;cuda*&quot;</p><p>To remove libcudnn drivers:<br><strong>$</strong> sudo apt-get --purge remove &quot;libcudnn7*&quot;</p><p>To remove NVIDIA drivers (optional):<br><strong>$</strong> sudo apt-get --purge remove &quot;*nvidia*&quot;</p><p><a href="#564e">^</a></p><h4>Step 1 — Identify Compatibility</h4><p>We need to ensure that your system has no compatibility issues with these:</p><ul><li>Video driver</li><li>CUDA Toolkit</li><li>cuDNN</li><li>Tensorflow GPU</li></ul><p>First, find out whether your NVIDIA video card is compatible <a href="https://developer.nvidia.com/cuda-gpus"><strong>here</strong></a>.</p><p>Next, take note on what is the graphics card driver version using one of these:</p><pre>$ modinfo nvidia | grep version<br>$ nvidia-settings</pre><p>Visit <a href="https://docs.nvidia.com/deploy/cuda-compatibility/index.html#binary-compatibility__table-toolkit-driver"><strong>here</strong></a> to identify which version of CUDA is compatible with your card.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*skyWRkhIesOXr2yenHhZyA.png" /><figcaption>Table 1: Cuda Toolkit and Compatible Video Driver</figcaption></figure><p>Finally, visit <a href="https://www.tensorflow.org/install/source#gpu"><strong>here</strong></a> to take note of which cuDNN and TensorFlow you can install based on your most compatible CUDA version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/1*mrZmlb7Cyf35Y4rIgHeZPw.png" /><figcaption>Table 2: Tensorflow GPU and respective compatible libraries</figcaption></figure><p>From my personal experience, although we should install only the compatible CUDA and respective cuDNN versions, we have a little room to vary the cuDNN versions depending on the default installation outcome. More on <a href="#a192">this</a> later.</p><p><a href="#564e">^</a></p><h4>Step 2 — Install Video Driver</h4><p>Now that you know you have a compatible NVIDIA video card, let’s install the latest driver.</p><p>Firstly, let’s compare what you have with what is available:</p><pre>$ modinfo nvidia | grep version<br>$ ubuntu-drivers devices</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*omLzNw4YORjgI_8xGA3OhA.png" /></figure><p>Let’s install the latest recommended driver from the driver list, if you have not done so. For my case, I updated my GTX 1050 to version 440 using this:</p><pre>$ sudo apt install <strong>nvidia-driver-440</strong></pre><blockquote><strong>Always</strong> ensure you have selected NVIDIA driver before rebooting.</blockquote><pre>$ prime-select query<br>$ prime-select nvidia</pre><p><a href="#564e">^</a></p><h4>Step 3 — Install CUDA Toolkit</h4><p>With the latest video driver in place, let’s install CUDA Tookit that is most compatible with your video driver.</p><p>Let’s see whether you have it installed, or worse, the wrong version. To check, use either of these commands:</p><pre>$ nvcc --version<br>$ apt list --installed | grep -i nvcc</pre><p>If you have an incompatible version installed and/or it has not been working out for you, it’s advisable to perform a clean uninstall:</p><pre>$ sudo apt-get — purge remove “*cublas*”<br>$ sudo apt-get — purge remove “cuda*”</pre><p>If you have the correct version installed, you can skip to the next step then.</p><p>To download your compatible CUDA Toolkit, visit <a href="https://developer.nvidia.com/cuda-toolkit-archive"><strong>here</strong></a><strong> </strong>and<strong> </strong>select the compatible version that you can use<strong>. </strong>We do not need to go<strong> </strong><a href="https://developer.nvidia.com/cuda-downloads"><strong>here</strong></a><strong> </strong>to install the latest CUDA because Tensorflow GPU libraries are usually not yet compatible with it (refer to <a href="#d8c3">Table 2 </a>above for a recap).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*4_jtc-08WRNXvwiZq5IJHw.png" /></figure><p>Once you have clicked on the respective version, you will be sent to a new page. From here, select all the relevant options for Ubuntu 18.04. Below is the screenshot of my selection for downloading CUDA Tookit 10.0, which is the compatible version for me to install TensorFlow GPU 1.13 and 1.14.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*dIHmd1WBDl5peSw7ItlStg.png" /></figure><p>Click on the ‘Download’ button to download the ‘Base Installer’ deb file and follow the instructions within the box, except the last instruction.</p><blockquote><strong>IMPORTANT</strong>: Do not upgrade your installed CUDA version after that.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iMe_UeKYpdu_57i63e5fKw.png" /></figure><p><a href="#564e">^</a></p><h4>Step 4 — Install cuDNN Library</h4><p>Now that we have CUDA installed, let’s install the compatible cuDNN. As always, let’s check if you already have one installed and if it does not work for you, then you will need to remove it.<br>$ apt list --installed | grep -i libcudnn</p><p>The versioning has two parts to it. For my case, I needed cuDNN version 7.6 and CUDA Toolkit version 10.0. Hence, I downloaded and installed exactly that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/1*qrhOP3e0SjZYdf1G-y5H0g.png" /></figure><p>If you need to remove the existing libcudnn:<br><strong>$</strong> sudo apt-get --purge remove “libcudnn7*”</p><p>Visit <a href="https://developer.nvidia.com/cudnn"><strong>here</strong></a> and click on “Download cuDNN” to proceed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*hF1b_8LKgdMP-eHnZvs6UQ.png" /></figure><p>You will then be prompted to either signup or login. You will have to signup if this is the first time you are here.</p><p>Once login, tick to agree to the terms and you should see this page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/1*3PHUOHPu6r6hiBOTnEXjeQ.png" /></figure><p>If your cuDNN version is not on the list, then select “<em>Archived cuDNN Releases</em>” for more options.</p><p>Remember to match both cuDNN and CUDA versions that are compatible for the TensorFlow you can/want to install. For my case, I chose cuDNN 7.6.3 with CUDA 10.0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/549/1*Wud1kmNE-DwRpnnvXwH_Lg.png" /></figure><p>Download all three Deb files for Ubuntu 18.04:</p><ul><li>cuDNN Runtime Library</li><li>cuDNN Developer Library</li><li>cuDNN Code Samples and User Guide</li></ul><p>To install, run these commands (but change the filenames to the ones you have downloaded):</p><pre>$ sudo dpkg -i <em>libcudnn7_7.6.5.32-1+cuda10.0_amd64.deb</em><br>$ sudo dpkg -i <em>libcudnn7-dev_7.6.5.32-1+cuda10.0_amd64.deb</em><br>$ sudo dpkg -i <em>libcudnn7-doc_7.6.5.32-1+cuda10.0_amd64.deb</em></pre><p>Let’s also hold the version from being accidentally/automatically updated:</p><pre>$ sudo apt-mark hold libcudnn7 libcudnn7-dev<br>$ apt-mark showhold</pre><p><a href="#564e">^</a></p><h4>Step 5 — Update Environment Variables</h4><p>Almost there! We’ll have to update a couple of environment variables. First, check if CUDA is already in your system path:</p><pre>$ echo $PATH<br>$ echo $LD_LIBRARY_PATH</pre><p>To update, use your favorite text editor to open the .bashrc file: <br>$ nano ~/.bashrc</p><p>Jump to the end of the file and append the following three lines, but make sure your change the CUDA version to match yours.</p><pre><br># NVIDIA CUDA Toolkit<br>export PATH=/usr/local/cuda-<strong>10.0</strong>/bin:$PATH<br>export LD_LIBRARY_PATH=/usr/local/cuda-<strong>10.0</strong>/lib64:$LD_LIBRARY_PATH</pre><p>Reboot to put these into effect.</p><blockquote><strong>NB</strong>: It is possible to execute the modifications without a reboot by running one of the commands below, but somehow it did not work for me when I subsequently ran the verification tests. If your tests fail the first time too, before you attempt anything else: Try rebooting once and running the tests again just to be sure.</blockquote><pre>$ source ~/.bashrc<br>$ . ~/.bashrc</pre><p><a href="#564e">^</a></p><h4>Step 6 — Install Tensorflow GPU and Keras</h4><p>Let’s install TensorFlow GPU and Keras if you have not done so yet. The TF version you need must be compatible with your CUDA Toolkit and cuDNN libraries (refer to Table 2).</p><p>For my case, my video card was compatible for CUDA 10.1 and below. This basically means that I can pretty much install any version of TensorFlow GPU. However, as my DL projects still run on older TF (TensorFlow) versions, I’ll only be installing the latest TF 1.14.</p><p>To install a specific TF version, run this:</p><pre>$ pip install “tensorflow-gpu==1.14.*”<br>$ pip install keras</pre><p>To check:</p><pre>$ pip list | grep -i tensor<br>$ pip list | grep -i keras</pre><p><a href="#564e">^</a></p><h4>Step 7 — Run Verification Tests</h4><p>We are getting so close now! Let’s run the following tests in this order:</p><ul><li>Step 7a. Verifying cuDNN is working</li><li>Step 7b. Verifying TensorFlow GPU is working</li></ul><h4>Step 7a — Verify cuDNN</h4><p>Run below commands as instructed <a href="https://docs.nvidia.com/deeplearning/sdk/cudnn-install/index.html#verify"><strong>here</strong></a>.</p><pre>$ cp -r /usr/src/cudnn_samples_v7/ $HOME<br>$ cd  $HOME/cudnn_samples_v7/mnistCUDNN<br>$ make clean &amp;&amp; make<br>$ ./mnistCUDNN</pre><p>If cuDNN is working correctly, you should be seeing this message:<br>Test passed!</p><h4>Step 7b — Verify Tensorflow GPU</h4><p>Launch ‘python’ from the command line and execute these codes:</p><pre>$ python<br>&gt;&gt;&gt; import tensorflow as tf<br>&gt;&gt;&gt; print(tf.test.gpu_device_name())<br>&gt;&gt;&gt; quit()</pre><p>If TF-GPU is correctly installed, it should be reporting a GPU device:<br>/device:GPU:0</p><p><a href="#564e">^</a></p><h4>Step 8. Trial run on a CNN model</h4><p>Visit <a href="https://www.tensorflow.org/tutorials/images/cnn"><strong>here</strong></a>, download the notebook and give it a spin.</p><p>If you run into CUDA OOM (out-of-memory) warnings, click <a href="#9c3c"><strong>here</strong></a> on how to resolve it.</p><p><a href="#564e">^</a></p><h3>Miscellaneous Tips</h3><p>Now that you have successfully installed a working TensorFlow GPU, let me share with you a couple of tips based on my own learning experience.</p><p><a href="#564e">^</a></p><h4>What to do if Tensorflow-GPU still does not load?</h4><ol><li>Ensure that the most <a href="#dcce">compatible versions</a> have been installed for the following: <a href="#7b17">NVIDIA video driver</a>, <a href="#8614">CUDA Toolkit</a>, <a href="#d7f8">cuDNN</a> and <a href="#0ec4">TensorFlow-GPU</a>.</li><li>Ensure that the <a href="#1927">PATH and LD_LIBRARY_PATH</a> have been correctly updated.</li><li>Reboot the machine and run the <a href="#059f">verification tests</a> again.</li><li>Based on various forum discussions, if you are not able to use Tensorflow-GPU with the current set up, you can try downgrading to different versions of Tensorflow-GPU, cuDNN and CUDA Toolkit (in that order). Run the verification tests after each permutation change.</li><li>Google the error messages or post in forums like StackOverflow for assistance.</li></ol><blockquote><strong>IMPORTANT</strong>: Always remember to uninstall the previous CUDA/cuDNN versions before installing a different one. Remember to update the environment variables too.</blockquote><p><a href="#564e">^</a></p><h4>How to Manage CUDA Out-of-memory Warnings</h4><p>Please beware that one of the common issues you might run into would be <strong>CUDA OOM (out-of-memory)</strong> warnings. Based on <a href="https://stackoverflow.com/questions/34199233/how-to-prevent-tensorflow-from-allocating-the-totality-of-a-gpu-memory">this</a> and <a href="https://riptutorial.com/tensorflow/example/31879/control-the-gpu-memory-allocation">this</a>, you can insert below codes into your script/notebook to help manage the issue.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2691858ccff0f28d02747fed21ff447b/href">https://medium.com/media/2691858ccff0f28d02747fed21ff447b/href</a></iframe><p><a href="#564e">^</a></p><h4>How to Monitor GPU Usage</h4><p>Run either of this in your command line:</p><pre>$ nvidia-smi<br>$ watch -n 5 nvidia-smi</pre><p>You would want to monitor the temperature, current memory usage and memory usage per process (and respective PIDs).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*n6l3BNG8j9tQwn0p5BPawA.png" /></figure><blockquote><strong>NB</strong>: Notice that the CUDA Version is listed as 10.2, which is different from my installed 10.0. According to <a href="https://stackoverflow.com/questions/53422407/different-cuda-versions-shown-by-nvcc-and-nvidia-smi">this</a> and <a href="https://docs.nvidia.com/cuda/cuda-runtime-api/driver-vs-runtime-api.html">this</a>, there are two different APIs, the runtime and the driver API. Our installed CUDA Toolkit refers to the runtime API, not the video driver API.</blockquote><p><a href="#564e">^</a></p><h4>How to Kill (Release) Runaway Processes</h4><p>If the GPU memory is not released, even after you have exited your IDE or notebook, then you can kill it manually instead. You do so by using its respective PID. For example, if I wanted to kill the last process 2471 shown above:</p><pre>$ sudo kill -9 2471</pre><p>If you are unsure which process to kill, you can start from the bottom up, especially with the process that is suspiciously hogging up a lot of memory.</p><p><a href="#564e">^</a></p><h4>How I got Keras CuDNNLSTM to work with the latest spaCy</h4><p>By installing only the following versions, after figuring it out how Google Colab did it:</p><ul><li>CUDA Toolkit 10.0</li><li>cuDNN 7.6</li><li>TensorFlow 1.15.x</li></ul><p><strong>Note 1:</strong> While retaining the compatible version of CUDA Toolkit 10.0, I was able to upgrade cuDNN from 7.4 to 7.6, even though this was not listed as compatible on the <a href="https://www.tensorflow.org/install/source#gpu">website</a>.</p><p><strong>Note 2: </strong>After having installed TF 1.15, I was having problems using Keras Convolutional Layers for my image classification projects. Eventually, it was resolved by installing TensorFlow 1.14.x to a new environment. <a href="https://docs.python-guide.org/dev/virtualenvs/">Virtual Environment </a>for the win!</p><p><a href="#564e">^</a></p><h3>Wrap Up</h3><p>Finally, I hope you are able to successfully install a working Tensorflow-GPU environment on your Ubuntu 18.04 LTS, that can fully accelerate all of your Deep Learning projects with your onboard GPU.</p><p>For those who cannot do so for whatever reasons, do not despair. You can always use cloud solutions, for eg., <a href="https://colab.research.google.com/">Google Colab</a>, which offers both GPU and TPU for free! Alternatively, you can also try out <a href="https://www.tensorflow.org/install/docker">Docker</a> instead.</p><p>Thank you for reading my article and I hope this has been useful for you!</p><p><a href="#564e">^</a></p><h3>Handy References</h3><ol><li><a href="https://docs.nvidia.com/cuda/index.html">CUDA Toolkit Documentation</a></li><li><a href="https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html">NVIDIA CUDA Installation Guide for Linux</a></li><li><a href="https://docs.nvidia.com/deeplearning/sdk/cudnn-install/index.html#install-linux">cuDNN Installation Guide on Linux</a></li><li><a href="https://www.tensorflow.org/install/gpu">TensorFlow GPU Support</a></li><li><a href="https://www.tensorflow.org/install/source#gpu">TensorFlow GPU Compatibility List</a></li><li><a href="https://www.tensorflow.org/install/pip">TensorFlow Pip Installation Guide</a></li></ol><p><a href="#564e">^</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a09ffd5f30f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/how-to-set-up-tensorflow-gpu-on-ubuntu-18-04-lts-7a09ffd5f30f">How to install Tensorflow GPU on Ubuntu 18.04 LTS</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Guinea Pig Breed Classification]]></title>
            <link>https://medium.com/data-science/guinea-pig-breed-classification-517fbb036ee6?source=rss-189f93992c4a------2</link>
            <guid isPermaLink="false">https://medium.com/p/517fbb036ee6</guid>
            <category><![CDATA[time]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[guinea-pigs]]></category>
            <category><![CDATA[image-classification]]></category>
            <dc:creator><![CDATA[Fan Yeng-Loon]]></dc:creator>
            <pubDate>Wed, 01 Jan 2020 19:41:00 GMT</pubDate>
            <atom:updated>2020-02-07T03:02:29.535Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to identify cavy breed using images through machine learning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dEq9woX-m4IAv2zRcpp_qQ.jpeg" /></figure><p><strong>TL:DR</strong> I used a deep learning model to identify the guinea pig’s breed from a picture. The model is an image CNN that applied transfer learning from an Inception V3 model with weights pre-trained on ImageNet. The source images were scraped from multiple image search engines. The final model is capable of classifying four different cavy breeds.</p><p>The codes and notebooks can be found <a href="https://github.com/ringoshin/cavy_breed_clf"><strong>here</strong></a>.</p><h3>Table of Contents</h3><ol><li><a href="#5908">Preface</a></li><li><a href="#d02c">Introduction</a></li><li><a href="#6cb5">Model Building and Findings</a><br>Step 0. <a href="#447c">Project Folder Formatting</a><br>Step 1. <a href="#9c5d">Raw Image Scraping</a><br>Step 2. <a href="#5011">Dataset Creation</a><br>Step 3. <a href="#14d4">Image Data Preprocessing</a><br>Step 4. <a href="#fb6b">Baseline Model Training</a><br>Step 5. <a href="#8b24">Deep Learning Model Training</a><br>Step 6. <a href="#1aab">Model Comparison and Selection</a></li><li><a href="#aa43">Conclusions</a></li></ol><h3>Preface</h3><p>This was my third project completed at <a href="https://metis.kaplan.com.sg/"><strong>Metis Data Science Bootcamp</strong></a>. We were learning about classifications, the respective classifiers and metrics at the time. I wanted to give image classification a shot, but not with <a href="https://www.kaggle.com/c/dogs-vs-cats"><strong>Dogs vs Cats</strong></a>. Since my wife and myself are avid cavy (guinea pig) lovers, so why not apply my domain knowledge and have some fun while we are at it? And that was how the journey began…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9yHJSI8CImGAIBwR" /><figcaption>Photo by <a href="https://unsplash.com/@bonniekdesign?utm_source=medium&amp;utm_medium=referral">Bonnie Kittle</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><a href="#d2b8">^</a></p><h3>Introduction</h3><p>Guinea pigs (or cavy) are commonly kept as household pets due to their manageable size (0.7–1.2kg), friendly nature and extreme cuteness. These cuddlies are rodents the size between a hamster and a chinchilla. They are herbivores and eat pretty much the same food as rabbits: pellets, hay, vegetables and fruits.</p><p>According to <a href="https://en.wikipedia.org/wiki/List_of_guinea_pig_breeds"><strong>Wikipedia</strong></a>, there are many breeds of cavy (13–18). Unlike dogs, which come in different size and shape, for cavies, it is all about the hair:</p><p>For this project, I chose to work with four most distinctively identifiable breeds:</p><ul><li><a href="https://www.goodhousekeeping.com/life/pets/g25751355/guinea-pig-breeds/?slide=2">Abyssinian</a> (short-haired with rosettes)</li><li><a href="https://www.goodhousekeeping.com/life/pets/g25751355/guinea-pig-breeds/?slide=4">American</a> (smooth, short-haired)</li><li><a href="https://www.goodhousekeeping.com/life/pets/g25751355/guinea-pig-breeds/?slide=13">Silkie/Sheltie</a> (smooth, long-haired)</li><li><a href="https://www.goodhousekeeping.com/life/pets/g25751355/guinea-pig-breeds/?slide=14">Skinny</a> (hairless).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*dhPUJGMgUMKX-31vRdv4CA.png" /><figcaption>Cavy Breeds used for Image Classification</figcaption></figure><p><a href="#d2b8">^</a></p><h3>Model Building and Findings</h3><p>For the rest of this article, you can follow me along with this Google Colab <a href="https://github.com/ringoshin/cavy_breed_clf/blob/master/cavy_breed_clf_train_and_predict_abridged.ipynb"><strong>notebook</strong></a>. It is an abridged version of the steps I took to complete the project.</p><p>All you have to do is to make a new subfolder called ‘<em>cavy_breed_clf</em>’ under your Google Drive’s ‘<em>Colab Notebooks</em>’ folder. Copy the following notebook and folders into ‘<em>cavy_breed_clf</em>’ and you should be good to go.</p><ul><li>/data (~163kB)</li><li>/images (~130MB)</li><li>/lib (~67kB)</li><li><a href="https://github.com/ringoshin/cavy_breed_clf/blob/master/cavy_breed_clf_train_and_predict_abridged.ipynb">cavy_breed_clf_train_and_predict_abridged.ipynb</a></li></ul><p>However, if you would like to see the full picture of how I got there, I have also sorted my scripts and notebooks in numerical order. I hope this will provide a clearer sense of the following processes: data preparation, training (classical ML, then Neural Network) and metrics/predictions.</p><p><a href="#d2b8">^</a></p><h4>Step 0. Project Folder Formatting</h4><p>This was the first project where I made the effort to organize my codes and documentation based on suggestions from <a href="https://blog.godatadriven.com/write-less-terrible-notebook-code"><strong>here</strong></a> and <a href="https://docs.python-guide.org/writing/structure/"><strong>here</strong></a>.</p><ul><li>/data — project datasets: raw, training, validation and testing</li><li>/images — raw images are stored in sub-folders ordered by breed, plus a sample sub-folder for adhoc images beyond the datasets</li><li>/lib/data_common.py — image data preprocessing routines</li><li>/lib/ml_common.py — classical ML handling processes, including classification metrics</li><li>/lib/nn_common.py — neural network DL handling processes</li></ul><p><a href="#d2b8">^</a></p><h4>Step 1. Raw Image Scraping</h4><p>Having a lot of good, clean and the right type of data is very important. It can be the most painful part, but also the most crucial part of your ML project. It will ensure your model will perform what it is set out to do, or at least a presentable MVP. During the bootcamp, we truly learnt this lesson the hard way.</p><p>AFAIK, there were no public cavy image dataset available for machine learning. Fortunately, there are many useful search engines out there based on this <a href="https://www.searchenginejournal.com/best-image-search-engines/299963/#close"><strong>article</strong></a>. For me, I relied mostly on <a href="https://images.google.com/"><strong>Google</strong></a>, <a href="https://images.search.yahoo.com/"><strong>Yahoo</strong></a> and <a href="https://www.bing.com/images/discover"><strong>Bing</strong></a>.</p><p>I was able to scrape 500 to 1000 raw images for each breed to start off with. I depended heavily on this wonderful Chrome extension <a href="https://chrome.google.com/webstore/detail/fatkun-batch-download-ima/nnjjahlikiabnchcpehcpkdeckfgnohf"><strong>Fatkun Batch Download Image</strong></a> to batch download the images.</p><p>Mind you, 1000 images per breed might sound a lot, but it was not. These were just raw and unlabeled. I had to pore through all 4000+ images during and post downloading to filter out unqualified candidates:</p><ul><li>Mislabeled animal species or cavy breed</li><li>Missing cavy in the image (yes, even search engines are not perfect)</li><li>Multiple cavies (better for the model to learn from just one per image)</li><li>Caricatures, instead of a real photo</li><li>Important breed features not in full display (eg. rosettes on Abyssinian)</li><li>Resolution was too low</li><li>Too out of focus</li></ul><p>Eventually, I trimmed it down to 1600+ images:</p><ul><li>Abyssinian: 553</li><li>American: 519</li><li>Silkie: 267</li><li>Skinny: 292</li></ul><p>It was a moderately imbalanced dataset, but I believed it should not impact the training of the models significantly.</p><p><a href="#d2b8">^</a></p><h4>Step 2. Dataset Creation</h4><p>After I have collected the required images, I sorted these into the respective sub-folders. Then, I ran a short <a href="https://github.com/ringoshin/cavy_breed_clf/blob/master/data/0a%20create_image_dataset.py">script</a> (also as a <a href="https://github.com/ringoshin/cavy_breed_clf/blob/master/data/0a%20create_image_dataset.ipynb">notebook</a>) to generate these datasets; raw, training, validation and testing. The latter three datasets would become the input data for all future ML training, validation and testing.</p><p>A quick sanity check to ensure the breeds were distributed evenly across the datasets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/597/1*_aXwRHURQuwaWtz1FUdDVw.png" /><figcaption>Breed Distribution across Training, Validation &amp; Test Datasets</figcaption></figure><p>Another check to be sure the features and targets were in good order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*hvV8mar5vKXzZ0O-Mz2fPw.png" /><figcaption>Sanity Check on Features and Targets</figcaption></figure><p><a href="#d2b8">^</a></p><h4>Step 3. Image Data Preprocessing</h4><p>Each image file will be transformed into ML ready format by resizing it to a specific image shape (150, 150) and then flatten it to an 1-D array. These will be saved as the features (X_train, X_val, X_test).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8b8b119d29cfea14750b6fa1665c59e3/href">https://medium.com/media/8b8b119d29cfea14750b6fa1665c59e3/href</a></iframe><p>The breed of each image will be one-hot-encoded to become the labels (y_train, y_val, y_test).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1f0b3c04ef12a36da7b5a524397d15cc/href">https://medium.com/media/1f0b3c04ef12a36da7b5a524397d15cc/href</a></iframe><p>A quick visual sanity check to make sure it was all good:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LhYDAInv8h_kXQA6cX4h6g.png" /></figure><p><a href="#d2b8">^</a></p><h4>Step 4. Baseline Model Training</h4><p>I decided right from the start that ‘Random Forest’ classifier would be my baseline model. Nevertheless, I took this as an learning opportunity to see how ‘well’ other classical classifiers would fare against my baseline model.</p><p>I created a routine (Vanilla_ML_Run) to train these four classifiers together:</p><ul><li><a href="https://scikit-learn.org/stable/modules/generated/sklearn.dummy.DummyClassifier.html">Dummy classifier</a></li><li><a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html">Logistic Regression</a></li><li><a href="https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.GaussianNB.html">Gaussian Naive Bayes</a></li><li><a href="https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html">Random Forest</a> (our baseline)</li></ul><p>Once the trained classifiers had been saved/reloaded, I compared the performances with the following:</p><ul><li><a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html">Classification report</a></li><li><a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html">Confusion matrix</a></li><li><a href="https://scikit-learn.org/stable/auto_examples/model_selection/plot_precision_recall.html#in-multi-label-settings">Precision-Recall curve</a></li><li><a href="https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html#plot-roc-curves-for-the-multilabel-problem">ROC curve</a></li></ul><p>NB: I highly recommend reading <a href="https://towardsdatascience.com/what-metrics-should-we-use-on-imbalanced-data-set-precision-recall-roc-e2e79252aeba">this</a> and <a href="https://machinelearningmastery.com/roc-curves-and-precision-recall-curves-for-classification-in-python/">this</a> for a good grasp of PR and ROC curves. And also <a href="https://datascience.stackexchange.com/questions/15989/micro-average-vs-macro-average-performance-in-a-multiclass-classification-settin">this</a> on Micro-average vs Macro-average.</p><h4>Step 4a. Gaussian NB Classifier</h4><p>Needless to say, Dummy classifier did not do very well and apparently, so was GaussianNB classifier. Precision and recall scores for ‘Silkie’ and ‘Skinny’ were pretty low. Let’s take a look at ‘Skinny’ as an example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/1*50ZE9MZ0yg4YW7rzFMWD1w.png" /></figure><ul><li>Low precision here meant that many of its predictions were not ‘Skinny’ (false positives)</li><li>Low recall meant that many actual ‘Skinny’ were wrongly identified as other breeds (false negatives)</li></ul><p>And the confusion matrix confirmed that understanding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*OrOMfScMGr1IuG-R2MsiGQ.png" /></figure><h4>Step 4b. Logistic Regression Classifier</h4><p>This classifier performed much better than GaussianNB, where most precision and recall scores hovered above or near 50.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*RTCO8nGwOFoqCxW5BtDeVA.png" /></figure><p>However, the PR curve seemed to tell a clearer story that this model did not perform very well for ‘Skinny’ especially. Its area was way less ideal than others and its curve deviated furthest from the micro-average curve too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*5a4wgC5dFaiSVZNHIFdpZw.png" /><figcaption>Precision-Recall Curve by Breed (area = Avg. Precision Score)</figcaption></figure><h4>Step 4c. Random Forest Classifier</h4><p>Random Forest appeared to be a better model than Logistic Regression, albeit just slightly. It looked like the overall higher scores were achieved at the expense of low recall scores for ‘Silkie’ and ‘Skinny’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/477/1*agQ0PAjXUbEmaHPIJZm_cw.png" /></figure><p>The F1 scores and the PR curve indicated strongly that this model did not perform very well for the smaller classes (‘Silkie’ and ‘Skinny).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/809/1*JpaTpNvu6YE7I6FOOR0Iuw.png" /><figcaption>Precision-Recall Curve by Breed (area = Avg. Precision Score)</figcaption></figure><p><a href="#d2b8">^</a></p><h4>Step 5. Deep Learning Model Training</h4><p>Now that we had our baseline model trained and reviewed, we could proceed next with two different DL models, namely:</p><ul><li>Multi-layer Image CNN (Convolution Neural Network)</li><li>Transfer Learning from Inception V3 using pre-trained weights on ImageNet</li></ul><p>Once trained, I would compare the model performances with the same metrics as before.</p><h4>Step 5a. Multi-layer Image CNN Building</h4><p>The architecture was fairly straightforward for an Image CNN model, of which you could find out more about it from <a href="https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html"><strong>here</strong></a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6eda0f15cd67ac78f23c9bc41cdef4b3/href">https://medium.com/media/6eda0f15cd67ac78f23c9bc41cdef4b3/href</a></iframe><p>After training for 100 epochs, the model achieved a training accuracy of 78% and validation accuracy of 67%. It definitely was more accurate than the baseline model, but I believed it was already overfitting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/1*RA27I27SoRQ_F97SPFO48A.png" /><figcaption>Training/Validation Accuracy vs No. of Epochs</figcaption></figure><p>Let’s look at the classification report. The overall scores were convincingly better as compared to the baseline model. Surprisingly, ‘Skinny’ was the best performer against the rest. I didn’t expect that!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*Gno1I-oaWTOHUys-lSQNog.png" /><figcaption>Classification Report</figcaption></figure><p>The precision and recall scores for the other breeds were interestingly uneven. The same with the PR and ROC curves too. ‘Skinny’ stood out very clearly against the other classes. It sure was cool to have trained a model capable of telling apart a hairless cavy from those with hair. :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/1*9BqOleEkjylPWt3luUx5mw.png" /><figcaption>Precision-Recall Curve by Breed (area = Avg. Precision Score)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/824/1*MPVC8fMdp4G2vSgeKEzYHw.png" /><figcaption>Receiver Operating Characteristic Curve (AUC = Area Under the Curve)</figcaption></figure><h4>Step 5b. Transfer Learning from InceptionV3 (ImageNet)</h4><p>This model is built using transfer learning technique as illustrated <a href="https://keras.io/applications/#fine-tune-inceptionv3-on-a-new-set-of-classes"><strong>here</strong></a>. It uses weights pretrained on <a href="http://www.image-net.org/">ImageNet</a>. For this specific model, I chose to freeze the first 230 layers.</p><p>After training for 100 epochs, the model achieved a 99 % training accuracy and 82% validation accuracy. It was definitely overfitting and the choppy, fluctuating validation curve seemed to support that as well. It could also mean that my data was still not enough.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*AbxRJSce2pN_J33vTYsK1g.png" /><figcaption>Training/Validation Accuracy vs No. of Epochs</figcaption></figure><p>When I looked at the classification report, precision and recall scores were extremely good overall, except for three very specific ones. It seemed like the model tended to mislabel ‘Abyssinian’ and ‘Silkie’ as ‘American’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*2EhnmzL5gaRpeB1YqeD79Q.png" /><figcaption>Classification Report</figcaption></figure><p>The confusion matrix also supported this observation. The model was mislabeling enthusiastically ‘Abyssinian’ (31) and ‘Silkie’ (9) as ‘American’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*WYwMLFMvm_8yGYzdfvKo-A.png" /></figure><p>Intuitively, this made some sense. Unlike ‘Skinny’, ‘Abyssinian’ being short-haired and ‘Silkie’ being smooth-coated, both breeds could be mistaken easily for an ‘American’ when viewed from certain angles.</p><p>NB: Of course, it could also be a simple case of badly labeled raw images. Something I would definitely want to look into to be sure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vW6i5SvR0xqWdGACDYY8jw.png" /></figure><p><a href="#d2b8">^</a></p><h4>Step 6. Model Comparison and Selection</h4><p>The InceptionV3 transfer learning model had the best scores overall.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a121250dee2c4a5e3e249e8e2d91bc5c/href">https://medium.com/media/a121250dee2c4a5e3e249e8e2d91bc5c/href</a></iframe><p>On top of that, it was able to brilliantly classify ‘Skinny’, where classical classifiers had generally failed (high recall). Notably, none of the models were able to confidently identify ‘Abyssinian’ as per se (low recall). Seemed like the <a href="https://www.wideopenpets.com/everything-you-need-to-know-about-abyssinian-guinea-pigs/">rosette patterns</a> unique to this breed were not that immediately obvious after all. Might need to get more quality images to improve.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eae02e482d0ab74c59d6c19f004c4bcc/href">https://medium.com/media/eae02e482d0ab74c59d6c19f004c4bcc/href</a></iframe><p>The deep learning models were able to differentiate ably between most breeds in the predictions (high precision).</p><p>Only ‘American’ suffered similarly across all models: During predictions, these models often saw other breeds as ‘American’ (low precision). The DL models were slightly better off, but not by much.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5926d1d0809738abc503b2996783fb76/href">https://medium.com/media/5926d1d0809738abc503b2996783fb76/href</a></iframe><p>When we compared the PR curve of all the models together, it became clear that we had a true winner.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/1*oxVF88LY2yN7Ku65WgtaOw.png" /></figure><p>Similarly, the transfer learning model led with the best ROC curve too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*dYURfynUFUMSvpBqqiBGDA.png" /></figure><p>Hence, the final model trained through transfer learning was clearly the best candidate. Just for fun, I even used it to predict out of scope images kept in the subfolder ‘<em>/images/new_samples’</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kw0uIj4PQFnxrJkdgWnu4w.png" /></figure><p><a href="#d2b8">^</a></p><h3>Conclusions</h3><p>It had been a fantastic and fun-filled project for sure. Even though I knew that transfer learning would work wonders, seeing it happened right before my eyes truly was exhilarating. This was especially encouraging when I had to work with a small dataset of mostly sub-par quality images. Can you imagine how much better the model will become if we have more data, better images?</p><p>Also, Deep Learning models were especially ‘clever’ at being able to tell apart ‘Skinny’ from the other breeds. Something that classical classifiers could not achieve. Then again, I had not tested with other classifiers to be sure, especially XGBoost.</p><p>And this is not the end for me. I would very much like to extend this project with the following goals:</p><ul><li>Add more quality images</li><li>Identify more breeds</li><li>Try other classifiers and pre-trained models</li><li>Deploy an API</li><li>Launch a web site</li><li>Build a mobile app</li></ul><p>When the site or app is ready, I will definitely be sharing this with other cavy lovers on social media.</p><p>Thank you for reading this and I hope you have enjoyed as much as I do!</p><p><a href="#d2b8">^</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=517fbb036ee6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/guinea-pig-breed-classification-517fbb036ee6">Guinea Pig Breed Classification</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I, Storytelling Bot]]></title>
            <link>https://medium.com/analytics-vidhya/i-storytelling-bot-3652812f3aff?source=rss-189f93992c4a------2</link>
            <guid isPermaLink="false">https://medium.com/p/3652812f3aff</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[word2vec]]></category>
            <category><![CDATA[lstm]]></category>
            <category><![CDATA[text-generation]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Fan Yeng-Loon]]></dc:creator>
            <pubDate>Sat, 28 Dec 2019 15:29:46 GMT</pubDate>
            <atom:updated>2024-01-16T07:46:11.622Z</atom:updated>
            <content:encoded><![CDATA[<h4>How a little bot spin narratives through machine learning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7agie6XiiWFKEtjH" /><figcaption>Photo by <a href="https://unsplash.com/@dustinlee?utm_source=medium&amp;utm_medium=referral">Dustin Lee</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>TLDR</strong>: I have created a <a href="https://www.analyticsvidhya.com/blog/2023/12/building-an-ai-storyteller-application-using-langchain-openai-and-hugging-face/">simple bot</a> to generate new text based on seeding text randomly chosen or entered. The final prediction candidate was a Deep Learning model using Keras/Tensorflow libraries: An LSTM model using word-tokenisation and pre-trained Word2vec (Gensim). The model was trained on a dataset built with free short stories available from <a href="https://paizo.com/pathfinder"><strong>Pathfinder</strong></a>.</p><p>The complete codes and notebooks can be found <a href="https://github.com/ringoshin/snaug"><strong>here</strong></a>.</p><h3>Table of Contents</h3><ol><li><a href="#6b49">Introduction</a></li><li><a href="#d415">Model Building and Selection</a><br>Step 0. <a href="#cfdf">Project Folder Formatting</a><br>Step 1. <a href="#f928">Raw Data Sourcing</a><br>Step 2. <a href="#6bdf">Data Preparation</a><br>Step 3. <a href="#65f3">Data Preprocessing</a><br>Step 4. <a href="#4d3f">Model Training</a><br>Step 5. <a href="#2d86">Sample Text Prediction</a><br>Step 6. <a href="#1a18">Model Selection</a><br>Step 7. <a href="#28b0">Text Visualization</a><br>Step 8. <a href="#f6c7">Final Model Text Generation</a></li><li><a href="#ad4e">Conclusions</a></li></ol><h3>Introduction</h3><p>This was my final project in <a href="https://metis.kaplan.com.sg/"><strong>Metis</strong></a><strong> Data Science Bootcamp</strong>. I wanted to try something <a href="https://www.analyticsvidhya.com/blog/2023/12/building-an-ai-storyteller-application-using-langchain-openai-and-hugging-face/">different</a> from my previous works, which dealt with predictions and image classifications. Hence, under my instructor’s guidance, I turned my attention to creativity through machine learning — text generation bot.</p><p>Text generation is one of many NLP (Natural Language Processing) applications. It can be both challenging and fun. And off we go…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cNNxPdc-x9rF2-vs" /><figcaption>Photo by <a href="https://unsplash.com/@kellysikkema?utm_source=medium&amp;utm_medium=referral">Kelly Sikkema</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><a href="#bb80">^</a></p><h3>Model Building and Selection</h3><p>You will find below the steps I took to build towards the final working model. I might sometimes illustrate only details from one of the three models to avoid being repetitive.</p><p><a href="#bb80">^</a></p><h4>Step 0. Project Folder Formatting</h4><p>I would prefer my codes (and documentation) to be as organized as possible and thus I leveraged some of the suggestions from <a href="https://blog.godatadriven.com/write-less-terrible-notebook-code"><strong>here</strong></a> and <a href="https://docs.python-guide.org/writing/structure/"><strong>here</strong></a>.</p><p>I have packaged commonly used modules into libraries stored <a href="https://github.com/ringoshin/snaug/tree/master/lib"><strong>here</strong></a><strong>:</strong></p><ul><li>nlplstm_class.py — Library of classes to encapsulate GPU usage, as well as for NLP using Keras/Tensorflow LSTM models</li><li>data-common.py — Data loading, data saving, data pre-processing and text generation related functions</li><li>text_viz_common.py — Functions that visualize generated text using spaCy library</li></ul><p>Without going into too much details, the customized LSTM classes kept things sane for me. I could at anytime easily train all of my models using GPU on Google Colab or any cloud platforms (for speed), while separately reload and predict with the trained models on my laptop without a powerful GPU (for conveniences or as a cheaper option).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5aeb07a5b3bc678729596a62a42a6679/href">https://medium.com/media/5aeb07a5b3bc678729596a62a42a6679/href</a></iframe><p><a href="#bb80">^</a></p><h4><strong>Step 1. Raw Data Sourcing</strong></h4><p>In the past, I have enjoyed a couple of <a href="https://paizo.com/pathfinder"><strong>Pathfinder</strong></a> games with my son. He was always one of the players, while I ran these games as the DM (dungeon master). A DM wears a lot of hats, one of which is to provide narratives to keep the game going.</p><p>Needless to say, I saw this as a ML learning opportunity: Let’s teach a bot how to generate random, but useful and relevant short narratives based on Pathfinder stories it has learnt.</p><p>Fortunately, I was able to download free short adventures from Pathfinder website, which became the dataset I used to train my model — all 15,000 usable words.</p><p>You can find the downloaded pdfs under “Pathfinder Beginner Box” subfolders <a href="https://github.com/ringoshin/snaug/tree/master/data"><strong>here</strong></a>.</p><p><a href="#bb80">^</a></p><h4><strong>Step 2. Data Preparation</strong></h4><p>I have applied the following mini-steps (mix of manual/scripted) to produce a single dataset suitable for our machine learning processes down the pipeline.</p><ul><li>Convert PDFs into text files (you can do this with any free online converters)</li><li>Strip off chunks of text that do not contain any valuable story elements</li><li>Concatenate text files into a single raw text file <a href="https://github.com/ringoshin/snaug/blob/master/data/textgen_pathfinder_raw.txt"><strong>here</strong></a></li><li>Retain only proper sentences using <a href="https://github.com/ringoshin/snaug/blob/master/0%20clean_raw_text_data.py"><strong>this</strong></a> and save it <a href="https://github.com/ringoshin/snaug/blob/master/data/textgen_pathfinder.txt"><strong>here</strong></a>.</li></ul><p>The resulting <em>“textgen_pathfinder.txt”</em> would be used as the source file for creating the dataset required to train the models below.</p><p><a href="#bb80">^</a></p><h4>Step 3. Data Preprocessing</h4><p>After a couple of initial exploratory experiments, I have decided to try out three different LSTM models, before settling for the best candidate:</p><ul><li><a href="https://github.com/ringoshin/snaug/blob/master/1a%20snaug_train_char_token_model_save_weights.ipynb"><strong>Character tokensation</strong></a><strong> </strong>(<a href="https://machinelearningmastery.com/text-generation-lstm-recurrent-neural-networks-python-keras/">reference</a>)</li><li><a href="https://github.com/ringoshin/snaug/blob/master/1b%20snaug_train_word_token_model_save_weights.ipynb"><strong>Word tokenisation and word embedding</strong></a> (<a href="https://machinelearningmastery.com/how-to-develop-a-word-level-neural-language-model-in-keras/">reference</a>)</li><li><a href="https://github.com/ringoshin/snaug/blob/master/1c%20snaug_train_pretrained_model_save_weights.ipynb"><strong>Word tokenisation and pre-trained Word2vec</strong></a> (<a href="https://stackoverflow.com/questions/42064690/using-pre-trained-word2vec-with-lstm-for-word-generation">reference</a>)</li></ul><p>I will take a closer look at how the third model was built, since it employed more or less the same techniques (tokenisation, embedding, LSTM/RNN, classification) as the first two models. First, we need to import these libraries:</p><pre>from keras.preprocessing.text import Tokenizer<br>from keras.utils import to_categorical</pre><pre>import string<br>import textwrap<br>import pickle</pre><pre>from lib.nlplstm_class import (TFModelLSTMCharToken, TFModelLSTMWordToken, TFModelLSTMWord2vec) <br>from lib.data_common import (load_doc, save_doc, clean_doc, prepare_char_tokens)<br>from lib.data_common import (build_token_lines, prepare_text_tokens, load_word2vec)<br>from lib.data_common import pathfinder_textfile, fixed_length_token_textfile</pre><p>Next, we load the document and tokenize it with this function:</p><pre>def clean_doc(doc):<br> # replace &#39;--&#39; with a space &#39; &#39;<br> doc = doc.replace(&#39;--&#39;, &#39; &#39;)</pre><pre> # replace &#39;-&#39; with a space &#39; &#39;<br> doc = doc.replace(&#39;-&#39;, &#39; &#39;)</pre><pre> # split into tokens by white space<br> tokens = doc.split()</pre><pre> # remove punctuation from each token<br> table = str.maketrans(&#39;&#39;, &#39;&#39;, string.punctuation)<br> tokens = [w.translate(table) for w in tokens]</pre><pre> # remove remaining tokens that are not alphabetic<br> tokens = [word for word in tokens if word.isalpha()]</pre><pre> # make lower case<br> tokens = [word.lower() for word in tokens]<br> return tokens</pre><p>Organize these tokens into fixed-length lines of tokens and save it:</p><pre>def build_token_lines(tokens, length=50):<br> length += 1<br> lines = list()<br> for i in range(length, len(tokens)):<br>  # select sequence of tokens<br>  seq = tokens[i-length:i]<br>  # convert into a line<br>  line = &#39; &#39;.join(seq)<br>  # store<br>  lines.append(line)<br> return lines</pre><p>Effectively, these fixed-length lines will be the dataset used to train the model.</p><p>Let’s use above paragraph as a toy example dataset of 14 words and let us set the fixed-length at 5 words. This toy example model will then be trained to predict the next word by learning from the previous 5 words. The resulting toy example training dataset would look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f03e2bdf586113044c3114cf9b8fd212/href">https://medium.com/media/f03e2bdf586113044c3114cf9b8fd212/href</a></iframe><p>Thus, that is how our actual dataset will look like too. We will then transform the fixed-length text tokens into a format ready for LSTM training:</p><pre>def prepare_text_tokens(lines):<br> # integer encode sequences of words<br> tokenizer = Tokenizer()<br> tokenizer.fit_on_texts(lines)<br> sequences = tokenizer.texts_to_sequences(lines)</pre><pre> # vocabulary size<br> vocab_size = len(tokenizer.word_index)<br> #print(tokenizer.word_index)</pre><pre> # split into X and y<br> npsequences = np.array(sequences)<br> X, y = npsequences[:,:-1], npsequences[:,-1]<br> y = to_categorical(y, num_classes=vocab_size+1)<br> seq_length = X.shape[1]<br> <br> return X, y, seq_length, vocab_size, tokenizer</pre><p>The third model is special in the sense that it uses pre-trained weights to sorta jumpstart its training. So, we will need to build this set of weights by training a Gensim Word2vec model with our fixed-length lines of tokens:</p><pre>def load_word2vec(lines):<br>  # split tokens up per line for Gensim Word2vec consumption<br>  sentences = [line.split() for line in lines]</pre><pre>  print(&#39;\nTraining word2vec...&#39;)<br>  # workers=1 will ensure a fully deterministrically-reproducible run, per Gensim docs<br>  word_model = Word2Vec(sentences, size=300, min_count=1, window=5, iter=100, workers=1)<br>  pretrained_weights = word_model.wv.syn0<br>  vocab_size, emdedding_size = pretrained_weights.shape<br>  print(&#39;Result embedding shape:&#39;, pretrained_weights.shape)</pre><pre>return vocab_size, emdedding_size, pretrained_weights</pre><p><a href="#bb80">^</a></p><h4>Step 4. Model Training</h4><p>Once we have gathered all the necessary components from the previous step, we can define the LSTM model we will use for the third model:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a7fff825a54f0ca2db6efbe72894aeaa/href">https://medium.com/media/a7fff825a54f0ca2db6efbe72894aeaa/href</a></iframe><p>The first layer we add to the model would be an embedding layer. The ‘vocab_size’ is the number of unique words in our dataset. Generally, most articles I have come across suggested arbitrarily setting the embedding dimensions (‘embedding_size’) within the range of 50–300. And this is also where we load the previously ‘pretrained_weights’ to the model.</p><pre>self.model.add(Embedding(input_dim=vocab_size,<br>                         output_dim=embedding_size,<br>                         weights=[pretrained_weights]))</pre><p>Now we add two LSTM layers and regularize with couple of dropouts.</p><pre>self.model.add(self.select_LSTM(embedding_size,return_sequences=True))<br>self.model.add(Dropout(0.2))<br>self.model.add(self.select_LSTM(embedding_size))<br>self.model.add(Dropout(0.2))</pre><p>We complete the model with two Dense layers, the latter being the output layer.</p><pre>self.model.add(Dense(embedding_size, activation=&#39;relu&#39;))<br>self.model.add(Dense((vocab_size+1), activation=&#39;softmax&#39;))</pre><p>With that in mind, let’s create an object of the above class by defining the parameters required:</p><pre># create new object that is an LSTM model using word tokenization<br># and pre-trained Word2vec model from Gensim to generate text<br>textgen_model_3 = TFModelLSTMWord2vec(use_gpu=True)</pre><pre>textgen_model_3.define(vocab_size=vocab_size, <br>                       embedding_size=emdedding_size, <br>                       pretrained_weights=pretrained_weights)</pre><p>Fundamentally, this is a classification exercise, hence ‘<em>categorical crossentropy</em>’ will be used:</p><pre># compile model<br>textgen_model_3.compile(loss=&#39;categorical_crossentropy&#39;, optimizer=&#39;adam&#39;, metrics=[&#39;accuracy&#39;])</pre><p>We can now train the model and save the weights, so that we can reload these weights from a different cloud platform or a local machine:</p><pre># fit model<br>history = textgen_model_3.fit(X, y, batch_size=128, epochs=50)</pre><pre># serialize model weights to HDF5 and save model training history<br>textgen_model_3.save_weights_and_history(fname_prefix=&quot;./model/pathfinder_wordtoken_w2v_model_50_epoch&quot;)</pre><p>This is the complete script of the preprocessing and model training processes for all 3 models:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1332f08fe50e173e929b73f6e712458c/href">https://medium.com/media/1332f08fe50e173e929b73f6e712458c/href</a></iframe><p>Colab notebooks for each of the model are also available <a href="https://github.com/ringoshin/snaug"><strong>here</strong></a>.</p><p>Last but not least, we will also need to save the models once after each successful training done:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f361465eac51cf5d229b6928ed6fa9c8/href">https://medium.com/media/f361465eac51cf5d229b6928ed6fa9c8/href</a></iframe><p>Respectively, the Colab notebook is available <a href="https://github.com/ringoshin/snaug/blob/master/2%20snaug_load_weights_save_models.ipynb"><strong>here</strong></a>.</p><p><a href="#bb80">^</a></p><h4>Step 5. Sample Text Prediction</h4><p>Now that we have trained our models and saved copies of each, let’s run some sample text predictions with all three models and see how each of them perform.</p><p>For model 1, it generates text by predicting one letter at a time, only to return the complete generated text when it reaches the ‘<em>n_chars</em>’ required:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/08c38f3b6fc9af3d78ac9861ef42b9ff/href">https://medium.com/media/08c38f3b6fc9af3d78ac9861ef42b9ff/href</a></iframe><p>For model 2 and 3, these will generate text by predicting the next word each time. The routine will return the generated text when it has reached the ‘<em>n_words</em>’ required’:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d37ebaa6687bb9d2b95f6e9b9ca0f7f0/href">https://medium.com/media/d37ebaa6687bb9d2b95f6e9b9ca0f7f0/href</a></iframe><p>Both text generation routines employed a rather ingenious helper function to add a degree of randomness to each prediction. This is achieved by adding a hyperparameter known as ‘temperature’ or ‘diversity’.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b1965d1d2d26aaf96ae1e733b2c75a4f/href">https://medium.com/media/b1965d1d2d26aaf96ae1e733b2c75a4f/href</a></iframe><p>Low temperature (near zero) will return predicted probabilities as it should be. The higher the temperature, the more random the return probabilities would be. This is best illustrated by testing the helper function on a toy probability array:</p><pre>preds=[0.05, 0.1, 0.35, 0.5]<br>print([preds[sample_predict(preds,0.05)] for _ in range(10)])<br>print([preds[sample_predict(preds,1)] for _ in range(10)])<br>print([preds[sample_predict(preds,5)] for _ in range(10)])</pre><p>Without the ‘temperature’, the prediction will always return ‘0.5’ as the largest probability. Adding temperatures now introduces an exciting level of uncertainty you can control.</p><pre>[0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5] <br>[0.5, 0.35, 0.35, 0.5, 0.5, 0.5, 0.1, 0.35, 0.5, 0.35] <br>[0.05, 0.05, 0.35, 0.5, 0.05, 0.1, 0.5, 0.5, 0.35, 0.1]</pre><p>For model 1, this is what I ran it with:</p><pre>temperature_table = [0, 0.7]</pre><pre>for temperature in temperature_table:<br>  generated = generate_seq_of_chars(textgen_model_1,<br>              num_unique_char, char2indices, indices2char,<br>              char_seed_text, maxlen, 300, temperature)</pre><pre>  print(&quot;&gt;&gt; generated text (temperature: {})&quot;.format(temperature))<br>  print(textwrap.fill(&#39;%s&#39; % (generated), 80))<br>  print()</pre><p>And the results were as follows:</p><pre>&gt;&gt; generated text (temperature: 0)  comes a dissong creck abread to eaper the ring to hear the sandpoint in the wind a small captain now a points and styyengess-demer, and a scccentions from a for the rap, the beliening of shobthess gropp and pcs who elemental in surprised to hel make a for gite and stealsh with a worken of golds wit  </pre><pre>&gt;&gt; generated text (temperature: 0.7)  combat—yagg, and is she robb as magnimar’s hork samp, and as not a points and following the beat of gold, in simpating the mapical mumber she wastreaks he enter the pcs may of sandpoint’s strypeled betore and to searing the maps nom a can grack fagilies she remares staight acamem, and for sceeters</pre><p>For model 2 and 3, I ran with this:</p><pre>temperature_table = [0, 1.0]<br>for temperature in temperature_table:<br>  generated = generate_seq_of_words(textgen_model_2, tokenizer,<br>                       seq_length, word_seed_text, 100, temperature)</pre><pre>  print(&quot;&gt;&gt; generated text (temperature: {})&quot;.format(temperature))<br>  print(textwrap.fill(&#39;%s&#39; % (generated), 80))<br>  print()</pre><p>And the results for model 2 was:</p><pre>&gt;&gt; generated text (temperature: 0)<br>or knowledge religion check defeating the undead is easier if the pcs extinguish the candle of the development section with the skeletons defeated the pcs can deal with the candle of night with a successful dc knowledge arcana or knowledge religion check the pcs learn this minor magic item cannot be extinguished save by snuffing the flame with live flowing blood hazelindra adds that the pcs can<br>keep the candle as long as they do not tell the academy of her connection to this situation the cemetery is half a mile west of the town and is accessible via a</pre><pre>&gt;&gt; generated text (temperature: 1.0)<br>or knowledge religion check defeating the undead is easier if the pcs extinguish the candle of the development section with the skeletons defeated the pcs can deal with the candle of night with a successful dc knowledge arcana or knowledge religion check the pcs learn this minor magic item cannot be extinguished save by snuffing the flame with live flowing blood hazelindra adds that the pcs can<br>keep the candle as long as they do not tell that about a long plum sized ruby calling it the fire of versade savasha versade has decided to display it publicly for the</pre><p>And for model 3:</p><pre>&gt;&gt; generated text (temperature: 0)<br>or knowledge religion check defeating the undead is easier if the pcs extinguish the candle of the development section with the skeletons defeated the pcs can deal with the candle of night with a successful dc knowledge arcana or knowledge religion check the pcs learn this minor magic item cannot be extinguished save by snuffing the flame with live flowing blood in order for the pcs to put out<br>its flame and prevent more undead from rising from graves along their path back to sandpoint they must douse the candle in blood from an open wound dealing at least points</pre><pre>&gt;&gt; generated text (temperature: 1.0)<br>or knowledge religion check defeating the undead is easier if the pcs extinguish the candle of the development section with the skeletons defeated the pcs can deal with the candle of night with a successful dc knowledge local or knowledge religion check defeating the undead is easier if the pcs extinguish the candle of the development section with the skeletons defeated the pcs can deal with the candle of night with a successful dc knowledge arcana or knowledge religion check the pcs learn this minor magic item cannot be extinguished save by snuffing the flame with live flowing blood in</pre><p>Here’s the complete code for generating text with various temperature for all three models:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f08a62f5c160337c27b1b05f914300ab/href">https://medium.com/media/f08a62f5c160337c27b1b05f914300ab/href</a></iframe><p>The colab version is available <a href="https://github.com/ringoshin/snaug/blob/master/3%20snaug_load_models_generate_text.ipynb"><strong>here</strong></a>.</p><p><a href="#bb80">^</a></p><h4>Step 6. Model Selection</h4><p>When I was deciding on the final candidate model, I reviewed over several factors like time spent to train, model complexity, model accuracy, generated text coherence, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*bFIU5TSRJ03xpVRVWmuiVQ.png" /></figure><p>In the end, I have settled for model 3, since it attained similar accuracy/loss against other two models faster, thus utilizing lesser resources. Also, when gauging from the sample text generations, the generated text was decently comprehensible in overall.</p><p><a href="#bb80">^</a></p><h4>Step 7. Text Visualization</h4><p>Now that we have a final candidate model, let’s add visualization values by using <a href="https://spacy.io/usage/linguistic-features#named-entities"><strong>spaCy NER</strong></a> (named entity recognition) feature. In particular, I had to generate new named entities to recognize domain specific names from the Pathfinder fantasy world.</p><p>The first step was to collect lists and dictionaries of new named entity types:</p><pre>god_name_list = [&#39;Erastil&#39;, &#39;Aroden&#39;, &#39;Desna&#39;, &#39;Sarenrae&#39;]<br>race_name_list = [&#39;Azlanti&#39;, &#39;Varisian&#39;, &#39;Thassilonian&#39;, &#39;Korvosan&#39;, &#39;Magnimarian&#39;]<br>...<br>sp_name_list = [&#39;Burning&#39;, &#39;Hands&#39;]</pre><pre>entity_names = {&#39;GOD&#39;: god_name_list, &#39;RACE&#39;: race_name_list, &#39;ORG&#39;: org_name_list, &#39;MOB&#39;: mob_name_list, &#39;PER&#39;: per_name_list, &#39;LOC&#39;: loc_name_list, &#39;SP&#39;: sp_name_list}</pre><pre>god_labels = [&#39;Erastil&#39;, &#39;Aroden&#39;, &#39;Desna&#39;, &#39;Sarenrae&#39;]<br>race_labels = [&#39;Azlanti&#39;, &#39;Varisian&#39;, &#39;Thassilonian&#39;, &#39;Korvosan&#39;, &#39;Magnimarian&#39;]<br>...<br>sp_labels = [&#39;Burning Hands&#39;]</pre><pre>entity_labels = {&#39;GOD&#39;: god_labels, &#39;RACE&#39;: race_labels, &#39;ORG&#39;: org_labels, &#39;MOB&#39;: mob_labels, &#39;PER&#39;: per_labels, &#39;LOC&#39;: loc_labels, &#39;SP&#39;: sp_labels}</pre><p>Next, we update the new named entities to spaCy <a href="https://spacy.io/usage/rule-based-matching"><strong>matcher</strong></a> with the help of <a href="https://spacy.io/usage/rule-based-matching#phrasematcher"><strong>PhraseMatcher</strong></a>:</p><pre>def get_matcher(nlp, entity_labels):<br>  matcher = PhraseMatcher(nlp.vocab)</pre><pre>  for entity, label_list in entity_labels.items():<br>    entity_patterns = [nlp(text) for text in label_list]<br>    matcher.add(entity, None, *entity_patterns)<br>  <br>  return matcher</pre><p>With that done, we can now use the updated matcher with <a href="https://spacy.io/api/top-level#displacy.render"><strong>displacy.render</strong> </a>to highlight important names unique to Pathfinder.</p><pre>doc = nlp(revised_text)<br>matches = matcher(doc)<br>spans = []</pre><pre>for match_id, start, end in matches:<br>  # get the unicode ID, i.e. &#39;COLOR&#39;<br>  rule_id = nlp.vocab.strings[match_id]  </pre><pre>  # get the matched slice of the doc<br>  span = doc[start : end]                </pre><pre>  # print(rule_id, span.text)<br>  spans.append(Span(doc, start, end, label=rule_id))<br>  doc.ents = spans</pre><pre>print()<br>print(&#39;-&#39;*95)</pre><pre>options = {&quot;ents&quot;: [&#39;GOD&#39;,&#39;MOB&#39;,&#39;PER&#39;,&#39;LOC&#39;,&#39;RACE&#39;,&#39;ORG&#39;,&#39;SP&#39;],<br>    &quot;colors&quot;: {&#39;GOD&#39;:&#39;#f2865e&#39;,&#39;MOB&#39;:&#39;#58f549&#39;,&#39;PER&#39;:&#39;#aef5ef&#39;,<br>    &#39;LOC&#39;:&#39;pink&#39;,&#39;RACE&#39;:&#39;#edcb45&#39;,&#39;ORG&#39;:&#39;#d88fff&#39;, &#39;SP&#39;:&#39;pink&#39;}}</pre><pre>print(&#39;Snaug_bot:&#39;)</pre><pre>if using_notebook:<br>  displacy.render(doc, style=&#39;ent&#39;, jupyter=True, options=options)<br>else:<br>  displacy.render(doc, style=&#39;ent&#39;, options=options)</pre><p>The complete functions to achieve these can be found <a href="https://github.com/ringoshin/snaug/blob/master/lib/text_viz_common.py"><strong>here</strong></a>.</p><p><a href="#bb80">^</a></p><h4>Step 8. Final Model Text Generation</h4><p>Hurrah! We are now ready to spin up the final working model. I added a loop to check for user input. If you only hit &lt;enter&gt;, it will randomly pick one seeding text from the original dataset and use that to generate hopefully something cool. To quit, just type ‘quit’ and hit &lt;enter&gt;.</p><pre>text_input=&#39;random&#39;<br>while True:<br>  text_input = input(&quot;Enter seeding text or hit &lt;ENTER&gt; to automate or &#39;quit&#39; to exit: &quot;)<br>  <br>  if text_input==&#39;quit&#39;:<br>    break<br>  else:<br>    if text_input==&#39;&#39;:<br>      text_input=&#39;random&#39;<br>      generate_and_visualize(lines, textgen_model, tokenizer,<br>             seq_length, nlp, matcher, entity_names, entity_labels,<br>             text_input=text_input)</pre><p>The complete colab notebook can be found <a href="https://github.com/ringoshin/snaug/blob/master/4%20snaug_final_model_generate_text_and_visualize.ipynb"><strong>here</strong></a><strong> </strong>and here’s an example output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2YrH15IhuboP7g3uIjaMcQ.png" /></figure><p><a href="#bb80">^</a></p><h3>Conclusions</h3><p>This has certainly been a fun project to work with, as I got to explore many different and sometimes diverse areas of ML. It’s pretty amazing to watch this simple app seemingly creating narratives all on its own. The only ingredients we have added were a few short stories with a common background, a fairly routine NLP LSTM model for text generation and some funky spaCy features to brighten things up a little.</p><p>All in all, we can definitely improve this model with below extended goals:</p><ul><li>Automate some of the manual and semi-auto processes</li><li>Attempt sentence and document embedding</li><li>Look into generating proper sentences with punctuation and all</li><li>Add a helper function to match up user input text closer to seeding text</li><li>Explore other text generation models out there</li></ul><p>I hope you have enjoyed reading this article and perhaps even find something useful for your own projects! :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3652812f3aff" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/i-storytelling-bot-3652812f3aff">I, Storytelling Bot</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>