<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Louis Teo on Medium]]></title>
        <description><![CDATA[Stories by Louis Teo on Medium]]></description>
        <link>https://medium.com/@louisteo9?source=rss-d1e21a5b6a23------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Stories by Louis Teo on Medium</title>
            <link>https://medium.com/@louisteo9?source=rss-d1e21a5b6a23------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 13:52:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@louisteo9/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Secret Guide To Human-Like Text Summarization]]></title>
            <link>https://medium.com/data-science/the-secret-guide-to-human-like-text-summarization-fcea0bfbe801?source=rss-d1e21a5b6a23------2</link>
            <guid isPermaLink="false">https://medium.com/p/fcea0bfbe801</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Louis Teo]]></dc:creator>
            <pubDate>Tue, 27 Apr 2021 12:04:00 GMT</pubDate>
            <atom:updated>2022-07-28T00:27:45.880Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JktSJvXWmkPyrVwj" /><figcaption>Photo by <a href="https://unsplash.com/@neonbrand?utm_source=medium&amp;utm_medium=referral">NeONBRAND</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Use Google’s state-of-the-art T5 model to summarize your content</h4><p>Summarization has become a very helpful way of tackling the issue of data overburden. In my earlier <a href="https://towardsdatascience.com/report-is-too-long-to-read-use-nlp-to-create-a-summary-6f5f7801d355">story</a>, I shared how you can create your personal text summarizer using extractive method — if you have tried that, you may have noticed that, because no new sentences were generated from the original content, at times you may have difficulties understanding the generated extractive summary.</p><p>In this story, I will share how I use Google’s T5 (Text-to-Text Transfer Transformer) model to create a human-like summarizer with just a few lines of codes!</p><p>As a bonus, I will also share my text summarizer pipelines where I combine both extractive and abstractive methods to generate meaningful summaries for PDF documents of any length…</p><h3><strong>Text Summarization Techniques</strong></h3><p>There are two techniques to summarize a long content:</p><p>i. <strong>Extractive summary</strong> — extracts important sentences from a long content.</p><p>ii. <strong>Abstractive summary</strong> — creates a summary by generating new sentences from an original content.</p><p>Abstractive summary is a comparatively more difficult technique as it involves deep learning, but thanks to Google’s pre-trained models that are made available to public, creating a meaningful abstractive summary is no longer a daunting machine learning task!</p><h3>T5 Text Summarizer</h3><p>You can build simple yet incredibly powerful abstractive text summarizer using Google’ T5 pre-trained model. I will use <a href="https://huggingface.co/transformers/">HuggingFace</a>’s state-of-the-art Transformers framework and <a href="https://pytorch.org/">PyTorch</a> to build a summarizer.</p><p><strong>Install packages</strong></p><p>Please ensure you have both Python packages installed.</p><pre>pip install torch<br>pip install transformers</pre><p><strong>Load model and tokenizer</strong></p><p>Load T5’s pre-trained model and its tokenizer.</p><pre>import torch<br>from transformers import AutoTokenizer, AutoModelForSeq2SeqLM</pre><pre>model = AutoModelForSeq2SeqLM.from_pretrained(&#39;t5-base&#39;)<br>tokenizer = AutoTokenizer.from_pretrained(&#39;t5-base&#39;)</pre><p>There are a total of five T5 models to choose from: t5-small, t5-base, t5-large, t-3B &amp; t5–11B. They each have different parameters. I will choose the ‘t5-base’ model, which has a total of 220 millions parameters. Feel free to try different T5 models.</p><p><strong>Input text</strong></p><p>Let’s load a CNN <a href="https://edition.cnn.com/2021/04/21/media/netflix-earnings-analysis/index.html">news</a> article about <strong><em>‘Netflix needs a Next Big Thing’ </em></strong><em>—</em> simply because this is rather interesting business news — and see how well our summarizer performs.</p><pre>text = &quot;&quot;&quot;New York (CNN Business)Netflix is synonymous with streaming, but its competitors have a distinct advantage that threatens the streaming leader&#39;s position at the top.</pre><pre>Disney has Disney+, but it also has theme parks, plush Baby Yoda dolls, blockbuster Marvel movies and ESPN. Comcast (CMCSA), Amazon (AMZN), ViacomCBS (VIACA), CNN&#39;s parent company WarnerMedia and Apple (AAPL) all have their own streaming services, too, but they also have other forms of revenue.</pre><pre>As for Netflix (NFLX), its revenue driver is based entirely on building its subscriber base. It&#39;s worked out well for the company — so far. But it&#39;s starting to look like the king of streaming will soon need something other than new subscribers to keep growing.</pre><pre>The streaming service reported Tuesday it now has 208 million subscribers globally, after adding 4 million subscribers in the first quarter of 2021. But that number missed expectations and the forecasts for its next quarter were also pretty weak.</pre><pre>That was a big whiff for Netflix — a company coming off a massive year of growth thanks in large part to the pandemic driving people indoors — and Wall Street&#39;s reaction has not been great.</pre><pre>The company&#39;s stock dropped as much as 8% on Wednesday, leading some to wonder what the future of the streamer looks like if competition continues to gain strength, people start heading outdoors and if, most importantly, its growth slows.</pre><pre>&quot;If you hit a wall with [subscriptions] then you pretty much don&#39;t have a super growth strategy anymore in your most developed markets,&quot; Michael Nathanson, a media analyst and founding partner at MoffettNathanson, told CNN Business. &quot;What can they do to take even more revenue out of the market, above and beyond streaming revenues?&quot;</pre><pre>Or put another way, the company&#39;s lackluster user growth last quarter is a signal that it wouldn&#39;t hurt if Netflix — a company that&#39;s lived and died with its subscriber numbers — started thinking about other ways to make money.</pre><pre><strong>An ad-supported Netflix? Not so fast</strong></pre><pre>There are ways for Netflix to make money other than raising prices or adding subscribers. The most obvious: selling advertising.</pre><pre>Netflix could have 30-second commercials on their programming or get sponsors for their biggest series and films. TV has worked that way forever, why not Netflix?</pre><pre>That&#39;s probably not going to happen, given that CEO Reed Hastings has been vocal about the unlikelihood of an ad-supported Netflix service. His reasoning: It doesn&#39;t make business sense.</pre><pre>&quot;It&#39;s a judgment call... It&#39;s a belief we can build a better business, a more valuable business [without advertising],&quot; Hastings told Variety in September. &quot;You know, advertising looks easy until you get in it. Then you realize you have to rip that revenue away from other places because the total ad market isn&#39;t growing, and in fact right now it&#39;s shrinking. It&#39;s hand-to-hand combat to get people to spend less on, you know, ABC and to spend more on Netflix.&quot;</pre><pre>Hastings added that &quot;there&#39;s much more growth in the consumer market than there is in advertising, which is pretty flat.&quot;</pre><pre>He&#39;s also expressed doubts about Netflix getting into live sports or news, which could boost the service&#39;s allure to subscribers, so that&#39;s likely out, too, at least for now.</pre><pre>So if Netflix is looking for other forms of near-term revenue to help support its hefty content budget ($17 billion in 2021 alone) then what can it do? There is one place that could be a revenue driver for Netflix, but if you&#39;re borrowing your mother&#39;s account you won&#39;t like it.</pre><pre>Netflix could crack down on password sharing — a move that the company has been considering lately.</pre><pre>&quot;Basically you&#39;re going to clean up some subscribers that are free riders,&quot; Nathanson said. &quot;That&#39;s going to help them get to a higher level of penetration, definitely, but not in long-term.&quot;</pre><pre><strong>Lackluster growth is still growth</strong></pre><pre>Missing projections is never good, but it&#39;s hardly the end of the world for Netflix. The company remains the market leader and most competitors are still far from taking the company on. And while Netflix&#39;s first-quarter subscriber growth wasn&#39;t great, and its forecasts for the next quarter alarmed investors, it was just one quarter.</pre><pre>Netflix has had subscriber misses before and it&#39;s still the most dominant name in all of streaming, and even lackluster growth is still growth. It&#39;s not as if people are canceling Netflix in droves.</pre><pre>Asked about Netflix&#39;s &quot;second act&quot; during the company&#39;s post-earnings call on Tuesday, Hastings again placed the company&#39;s focus on pleasing subscribers.</pre><pre>&quot;We do want to expand. We used to do that thing shipping DVDs, and luckily we didn&#39;t get stuck with that. We didn&#39;t define that as the main thing. We define entertainment as the main thing,&quot; Hastings said.</pre><pre>He added that he doesn&#39;t think Netflix will have a second act in the way Amazon has had with Amazon shopping and Amazon Web Services. Rather, Netflix will continue to improve and grow on what it already does best.</pre><pre>&quot;I&#39;ll bet we end with one hopefully gigantic, hopefully defensible profit pool, and continue to improve the service for our members,&quot; he said. &quot;I wouldn&#39;t look for any large secondary pool of profits. There will be a bunch of supporting pools, like consumer products, that can be both profitable and can support the title brands.&quot;&quot;&quot;&quot;</pre><p><strong>Tokenize Text</strong></p><p>T5 can be used to perform other tasks, such as text generation, translation, etc.; adding T5 specific prefix “summarize: ” will tell the model to perform the summarizing task.</p><pre>tokens_input = tokenizer.encode(&quot;summarize: &quot; + text,<br>                              return_tensors=&#39;pt&#39;,<br>                              max_length=tokenizer.model_max_length,<br>                              truncation=True)</pre><p>Here we will tokenize our text to the model’s maximum acceptable token input length. If the tokenized input exceeds the model’s maximum token length, it will be truncated.</p><p><strong>Generate Summary</strong></p><p>Let’s generate a summary by passing in the encoded tokens and then decode the generated summary back to text.</p><pre>summary_ids = model.generate(tokens_input, min_length=80, <br>                           max_length=150, length_penalty=15, <br>                           num_beams=2)</pre><pre>summary = tokenizer.decode(summary_ids[0], skip_special_tokens=True)</pre><p>The model takes encoded tokens and the following input arguments:</p><ul><li><strong>min_length</strong>: minimum length of tokenized text.</li><li><strong>max_length</strong>: maximum length of tokenized text.</li><li><strong>length_penalty</strong>: value &gt; 1 forces the model to generate a longer summary, value &lt; 1 forces the model to generate a shorter summary.</li><li><strong>num_beams</strong>: value 2 allows the model to explore tokens that generate <a href="https://en.wikipedia.org/wiki/Beam_search">more promising predictions</a>.</li></ul><p><strong><em>Note:</em></strong><em> Keeping the minimum and maximum tokenized text lengths between 80 and 150 and a length penalty of 15 will allow the model to generate a reasonable summary of 60 to 90 words.</em> <em>We will use the default values for the rest of input arguments (not shown above).</em></p><p><strong>Output Summary</strong></p><pre>Netflix (NFLX) reported Tuesday it now has 208 million subscribers globally. that number missed expectations and the forecasts for its next quarter were also pretty weak. the streaming service&#39;s stock dropped as much as 8% on Wednesday, leading some to wonder what the future of the streamer looks like. if competition continues to gain strength, people start heading outdoors and if, most importantly, its growth slows, it wouldn&#39;t hurt if Netflix started thinking about other ways to make money - like selling ads.</pre><p>Wow! It looks like a pretty decent summary.</p><p>…but if you read the full text and read the summary again, you will notice that the latter part of the full text did not get summarized — this is because the tokenized input get truncated after it exceeds the maximum model token input length of 512.</p><p>If you are worried about missing out some important details in the latter text, you can use a simple trick to solve the issue: perform extractive summarization to the original text first, followed by abstractive summarization.</p><h3>BERT Extractive Summary</h3><p>Before we proceed, make sure you have pip installed BERT extractive summarizer on your terminal.</p><pre>pip install bert-extractive-summarizer</pre><p><a href="https://iq.opengenus.org/bert-for-text-summarization/#:~:text=Machine%20Learning%20(ML)%20BERT&amp;text=Extractive%20Text%20summarization%20refers%20to,approach%20to%20perform%20NLP%20tasks.">BERT</a> stands for Bidirectional Encoder Representations from Transformers. It’s a Transformer-based machine learning technique for Natural Language Processing (NLP) developed by Google. It uses a powerful flat architecture with inter sentence transform layers to get the best result in summarization.</p><pre>from summarizer import Summarizer</pre><pre>bert_model = Summarizer()<br>ext_summary = bert_model(text, ratio=0.5)</pre><p>Below is the extractive summary generated by BERT. I purposely set it to produce a summary that is 50% in length of the original text by setting the summary ratio to 0.5. Feel free to use a different ratio to adjust your long document to the appropriate length.</p><pre>New York (CNN Business)Netflix is synonymous with streaming, but its competitors have a distinct advantage that threatens the streaming leader&#39;s position at the top. Disney has Disney+, but it also has theme parks, plush Baby Yoda dolls, blockbuster Marvel movies and ESPN. It&#39;s worked out well for the company - so far. But that number missed expectations and the forecasts for its next quarter were also pretty weak. Or put another way, the company&#39;s lackluster user growth last quarter is a signal that it wouldn&#39;t hurt if Netflix - a company that&#39;s lived and died with its subscriber numbers - started thinking about other ways to make money. Not so fast<br>There are ways for Netflix to make money other than raising prices or adding subscribers. His reasoning: It doesn&#39;t make business sense. &quot;It&#39;s a judgment call... It&#39;s a belief we can build a better business, a more valuable business [without advertising],&quot; Hastings told Variety in September. &quot; You know, advertising looks easy until you get in it. Then you realize you have to rip that revenue away from other places because the total ad market isn&#39;t growing, and in fact right now it&#39;s shrinking. It&#39;s hand-to-hand combat to get people to spend less on, you know, ABC and to spend more on Netflix.&quot; So if Netflix is looking for other forms of near-term revenue to help support its hefty content budget ($17 billion in 2021 alone) then what can it do? Netflix could crack down on password sharing - a move that the company has been considering lately. &quot;Basically you&#39;re going to clean up some subscribers that are free riders,&quot; Nathanson said. &quot; That&#39;s going to help them get to a higher level of penetration, definitely, but not in long-term.&quot; The company remains the market leader and most competitors are still far from taking the company on. We used to do that thing shipping DVDs, and luckily we didn&#39;t get stuck with that. We define entertainment as the main thing,&quot; Hastings said. He added that he doesn&#39;t think Netflix will have a second act in the way Amazon has had with Amazon shopping and Amazon Web Services. Rather, Netflix will continue to improve and grow on what it already does best. I wouldn&#39;t look for any large secondary pool of profits.</pre><p>Let’s now feed the extractive summary through our T5 model.</p><h3><strong>The Extractive-Abstractive Summary</strong></h3><pre>Netflix&#39;s lackluster user growth is a signal that it wouldn&#39;t hurt if it started thinking about other ways to make money. the company remains the market leader and most competitors are still far from taking the company on. the company could crack down on password sharing - a move that the company has been considering lately. &quot;it&#39;s a judgment call... it&#39;s a belief we can build a better business, a more valuable business,&quot; hastings said.</pre><p>Wow… the generated summary now covers the entire context of the original text.</p><p>For your convenience, I have summarized the codes below.</p><p><strong>You can also click </strong><a href="https://github.com/louisteo9/t5-text-summarizer"><strong>here</strong></a><strong> to go to my GitHub to get the Jupyter Notebooks for T5 text summarizer and text summarizer pipelines preparation, and the pipeline scripts that you can run on your terminal to summarize multiple PDF documents.</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ec92a4c7f21156ed5058bba6edf2ea2b/href">https://medium.com/media/ec92a4c7f21156ed5058bba6edf2ea2b/href</a></iframe><h3>BONUS….T5 Text Summarizer Pipelines</h3><p>I have built a text summarizer pipelines that can extract text from PDF documents, summarize the text and store both the original text and the summary into a SQLite database and output the summary to a text file.</p><p>To summarize a long PDF document, you can first apply extractive summarization to shorten the text before you feed it through the T5 model to generate a human-like summary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*__Bdg4DXeEaYL76J8Ff7vQ.png" /><figcaption>Image by author: run data pipeline to extract text data from pdf files and save to a SQLite database</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Aybqhw9MCtCNoA1Kzk22ng.png" /><figcaption>Image by author: run summarization pipeline (only T5) to summarize text data, save the summary to text file and store the summary to database</figcaption></figure><p><strong><em>Note:</em></strong><em> key in ‘1.0&#39; if you only want to summarize the text with T5 model.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L3nJ5pz16i7IxV_THU_cSg.png" /><figcaption>Image by author: run summarization pipeline (BERT &amp; T5) to summarize text data, save the summary to text file and store the summary to database</figcaption></figure><p><strong><em>Note:</em></strong><em> key in a ratio below ‘1.0’ (e.g. ‘0.5’) if you wish to shorten the text with BERT extractive summarization before running it through T5 summarization. It takes longer to generate a summary this way because each text is run through two different summarizers.</em></p><h3>Conclusion… and future work</h3><p>There you go — you only need 7 lines of codes (including importing libraries and modules) to get the Google’s T5 pre-trained model to summarize a content for you.</p><p>To make the summary of a long content more meaningful, you can apply extractive summarization to shorten the text first, followed by abstractive summarization.</p><p>T5 pre-trained models support Transfer-Learning: that means we can train the models further with our custom datasets.</p><p><strong>For future work</strong>, it will be interesting to see how the models perform after they have been custom-trained to summarize specific contents, e.g. medical journals and engineering journals.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fcea0bfbe801" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-secret-guide-to-human-like-text-summarization-fcea0bfbe801">The Secret Guide To Human-Like Text Summarization</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[6 Tips to Complete Udacity’s Data Scientist Nanodegree in 1 month]]></title>
            <link>https://medium.com/data-science/6-tips-to-complete-udacitys-data-scientist-nanodegree-in-1-month-b5f7351f22b3?source=rss-d1e21a5b6a23------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5f7351f22b3</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[udacity]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Louis Teo]]></dc:creator>
            <pubDate>Thu, 04 Feb 2021 13:32:59 GMT</pubDate>
            <atom:updated>2021-02-04T13:32:59.772Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zq6z2BKfxJi5r-Hk" /><figcaption>Photo by <a href="https://unsplash.com/@sickhews?utm_source=medium&amp;utm_medium=referral">Wes Hicks</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Maximize your learning with minimal time and cost</h4><p>Udacity’s data scientist nanodegree is one of the best online data science courses I have taken. This course has received a lot of positive feedback from past students, and it’s easy to see why — their learning material is easy to follow and their industry-based projects allow students to get practical and early exposure to solving real-world problems. What I like the most about Udacity is its quick turnaround time and personalized project review. The personalized project review allows me to know my strengths and weaknesses, so I can work to improve my skills immediately.</p><p><a href="https://www.udacity.com/course/data-scientist-nanodegree--nd025">Udacity’s home page</a> says on average it takes 4 months (at 10 hours per week) to complete the course, but I will share my tips on how I managed to complete the course in a month.</p><p><strong><em>Note:</em></strong><em> I had accumulated around 6 months of experience in using Python and some Machine Learning knowledge before I took the program. I decided to sign up to this program because I was stuck in a city far away from home due to a COVID-19 lockdown, and I had occasional time-off from my job.</em></p><p><em>If you are new to data science and machine learning, I will suggest you take the other free Udacity courses and some pre-requisite courses before you sign up for this course.</em></p><p><strong>Tip #1: To jumpstart, ask for available discount</strong></p><p>Since the COVID-19 outbreak in year 2020, Udacity has been offering various discounts because lockdown and the extra time freed by work-from-home arrangement present a great timing to many to upskill their tech skills.</p><p>Paying USD 399 per month was a lot to pay for me so I decided to ask for a discount from Udacity — and to my surprise Udacity offered it to me in early January 2021, which allowed me to sign up to the program almost right away.</p><p><em>Udacity offers special discounts from time to time, but it’s on a case-by-case basis, so be sure to ask for it before you sign up.</em></p><p><strong>Tip #2: Read through all the Project Rubrics.</strong></p><p>Udacity specifies all the requirements to pass their projects in their rubrics. Read them first before you start the project.</p><p>I know many of us like to jump straight into the project and do it right away, but if we realize much later that the rubrics requirements have not been fulfilled, we will need to re-work and re-submit our project for grading.</p><p>Also, sometimes Udacity put an EXTRA challenges section in the project, but it will be mentioned it’s not necessary to complete that section to pass the project. You can skip this EXTRA section and only come back to learn it after you graduate from your program.</p><p><em>Read the project rubrics before you start any project so that your project has a higher chance to be graded as ‘pass’ in your first submission.</em></p><p><strong>Tip #3: Start your project early and apply what you’ve learned right away</strong></p><p>The Nanodegree is organized in such a way that after relevant topics are covered, the student needs to complete a project. If you have watched all the videos before starting the project, you most likely will need to go through the videos again because you likely won’t remember all the concepts or knowledge that was taught, before you implement it in your project.</p><p><em>Start the project early, and understand what you need to know before you watch the lecture videos. Apply what you’ve learned straightaway to your project and finish it bit-by-bit…</em></p><p><strong>Tip #4: Focus on lecture videos related to the project</strong></p><p>There are tons of lecture videos to watch, and all of them are first-rate. But given what little time that you may have to spare, I recommend you only watch those in the core curriculum and, for the extra videos in the extra curriculum, only watch them if you need them to complete your project.</p><p>Once you have completed the projects and graduated from the program, you will have one year of free access to the course materials and videos still. You are allowed to download them for your future reference.</p><p><em>All the lecture videos are extremely good and easy to follow; however, you will find it too much to absorb all in a month’s time; so only watch the videos that you really need to know for now, and watch the rest later.</em></p><p><strong>Tip #5: Google your problems for immediate answers</strong></p><p>Many have said Google is your best teacher in learning anything, so make full use of Google search to find your answers immediately. Even the most seasoned programmers will tell you that they use Google search every day to find answers related their programming queries.</p><p>There is a specific way to search the answers on Google for Python - in fact it is taught in the course: always start your search with the keyword ‘Python’, then followed by your question (e.g. python UnboundLocalError: local variable). This tells the search engine to prioritize results that are explicitly related to the tools you’re using.</p><p><strong>Tip #6: Refer to students’ answers for guide to complete the projects</strong></p><p>The projects in Udacity are very industry-based, and this means that they relate to real-world problems — also in other words, they could be very tough.</p><p>I wouldn’t have been able to complete the projects without any reference to other students’ work. There are many different approaches to solve a problem. You can look at the approach taken by a student and try and implement it in your workbook. You can in fact improve the approach as well with your codes.</p><p>If you find one that’s too difficult to understand, refer to the next workbook you can find.</p><p><em>You can easily find on Google other students’ work published on GitHub by searching the project title. If you refer to and want to use their answers, then be sure to quote the reference because it is part of Udacity’s requirements that you acknowledge the work of others that you refer to.</em></p><h3><strong>Conclusion</strong></h3><p>Let’s recap the main tips and suggestions:</p><p>a) If you know how to use Python for data analysis and have some machine-learning background knowledge, then you are ready to start the nanodegree anytime. <strong>Don’t be discouraged </strong>— they have plenty of resources for pre-requisites if you get stuck. Learn to <strong>Google your problems effectively to find immediate solutions.</strong></p><p>b) <strong>Start your project early</strong>, and<strong> </strong>always <strong>keep the project rubrics in mind</strong> when you work on your project. <strong>Apply what you learn</strong> <strong>right away</strong> and finish the project bit-by-bit.</p><p>c) Manage your time well. <strong>Dedicate more hours in the start of your course </strong>to solidify your foundations, and you will progress fast towards the end of the program before you know it. If you only learn 2 hours every day, it is easy to forget a lot of things down the road, and you will then spend a lot of time re-watching the videos.</p><p>c) <strong>Refer to students’ answers in GitHub.</strong> Try to understand why they approached a problem a certain way before you implement it in your workbook. Remember to quote your reference as part of Udacity’s requirements on reference to external work.</p><p>Have fun in learning to build cool industrial projects with Udacity! Feel free to ask me any questions on this or follow me on <a href="https://louisteo9.medium.com/">medium</a> and <a href="https://github.com/louisteo9">Github</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b5f7351f22b3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/6-tips-to-complete-udacitys-data-scientist-nanodegree-in-1-month-b5f7351f22b3">6 Tips to Complete Udacity’s Data Scientist Nanodegree in 1 month</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Used Machine Learning To Create A Stock Price Predictor]]></title>
            <link>https://ai.plainenglish.io/how-i-create-my-stock-price-predictor-f38c229b758e?source=rss-d1e21a5b6a23------2</link>
            <guid isPermaLink="false">https://medium.com/p/f38c229b758e</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Louis Teo]]></dc:creator>
            <pubDate>Wed, 27 Jan 2021 04:30:23 GMT</pubDate>
            <atom:updated>2021-02-04T08:26:27.394Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dW0QbfpyTz82Koh2" /><figcaption>iPhoto by<a href="https://unsplash.com/@nick604?utm_source=medium&amp;utm_medium=referral"> Nick Chon</a>g on<a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral"> Unsplas</a>h</figcaption></figure><h4>Improve your technical analysis skills using Machine Learning</h4><p><em>Disclaimer: Please use the information in the story with caution. You should conduct your own due diligence on the stock price predictor tool before using it.</em></p><p>There are two analyses in a stock price analysis — fundamental analysis and technical analysis.</p><p><strong>Fundamental analysis</strong> is focused on analysis of the company’s future growth based on current business environment and financial performance such as revenues, earnings and profit margin.</p><p><strong>Technical analysis </strong>is focused on predicting price movements using statistical method. This analysis guides traders on what is most likely to happen given past information.</p><p>Most investors use <strong>both</strong> fundamental and technical analyses to make sound decisions.</p><h3>Problem Statement</h3><p>Predicting stock price is an intensive and time-consuming task. A “sell” or “buy” decision made at the right timing can drastically affect your profits.</p><p>With the wealth of information that is available in the form of historical stock prices and company performance data and the advent of computing power, Machine Learning can be used to predict stock price trend using technical analysis and allow investors to make sound decisions on when to buy or sell their stocks, thus maximizing the profits.</p><p>In this story, we will work with historical stock price from <a href="https://www.quandl.com/data/EOD/INTC-Intel-Corporation-INTC-Stock-Prices-Dividends-and-Splits">Intel Inc</a>. We will implement two machine learning algorithms — Moving Average and LSTM — in predicting the future daily closing prices of the company. Then, we will choose the best out of the two algorithms to develop our own stock price predictor.</p><h3>Evaluation Metrics</h3><p>To evaluate the effectiveness of the models we developed, we use <em>Root Mean Square Error (RMSE) </em>and <em>Mean Absolute Percentage Error (MAPE)</em>.</p><p>For both evaluation metrics, the lower the values, the better our model will be in predicting stock price. The winning algorithm should also have MAPE below 5%. In other words, on average you want your prediction be within +/- 5% of the actual price.</p><h3>Dataset</h3><p>The dataset <a href="https://www.quandl.com/data/EOD/INTC-Intel-Corporation-INTC-Stock-Prices-Dividends-and-Splits">Intel Inc</a> looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*gZA1RL0TUuBrczYeTlOelg.png" /></figure><p>There are many variables in the dataset, but we are only interested in the closing price in the ‘Close’ column and the date in the ‘Date’ column.</p><p>You will find some dates are missing — and that’s because market is closed on weekends and public holidays: 2017–12–23 (Saturday), 2017–12–24 (Sunday), 2017–12–25 (Christmas Day).</p><p>Profit and loss is calculated using the closing price. Thus, the closing price is our target variable.</p><p>Let’s plot the historical closing price.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*R09DahTc55_bzDdw4D3YIQ.png" /></figure><p>From the plot, we can see the stock price data has been collected in a complete form — no anomaly is found in the data. We can also safely ignore the missing dates on weekends and public holidays — when market is not traded.</p><p>We will split our dataset to 80% training data and 20% testing data. The model will be trained using the training data and evaluated using the 20% testing data.</p><h3>Moving Average</h3><p><a href="https://www.investopedia.com/articles/active-trading/052014/how-use-moving-average-buy-stocks.asp#:~:text=The%20moving%20average%20(MA)%20is,time%20period%20the%20trader%20chooses.">Moving Average</a> is a simple technical analysis method that smoothens out price values by creating a continuously updating average price. The average price is usually taken over a period of time - it can be 30 minutes, 10 days, 2 weeks, 1 month, or any period an investor chooses.</p><p>Moving Average is a popular strategy because it can be adjusted to any timeframe, and so it is suitable for both long-term and short-term investors. It helps to cut down the ‘noise’ on the price chart, thereby allowing investors to get a basic idea of which way the stock price is moving.</p><p>The timeframe or length you choose for a moving average (also known as look back period) can have a big impact on how effective it is. A shorter timeframe will react much quicker to price change than a longer time frame, e.g. 20-day moving average tracks more closely on the actual price than 100-day moving average.</p><p>To identify significant, long-term support and resistance levels and overall stock price trends, 50-day, 100-day and 200-day timeframes are commonly used. The 200-day timeframe is in fact considered as the most significant in <a href="https://www.investopedia.com/ask/answers/122414/what-are-most-common-periods-used-creating-moving-average-ma-lines.asp#:~:text=Traders%20and%20market%20analysts%20commonly,averages%20are%20the%20most%20common.">stock trading</a>.</p><p>Below is the plot showing the prediction of stock price using testing data size (218) as our Moving Average timeframe. The 218-day timeframe is chosen purely because I want to simplify my codes in Python. For the purpose of this algorithm comparison, you can treat 218-day and 200-day as similar timeframes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*TdT7QGccwoNe6zUkkzafJg.png" /><figcaption>Predicting stock price using Moving Average</figcaption></figure><p>From the plot, you can see the prediction did not go so well. The predicted values are mostly of the same range as historical pricing in the training data for the timeframe we chose.</p><h3>Long Short Term Memory (LSTM)</h3><p><a href="https://machinelearningmastery.com/gentle-introduction-long-short-term-memory-networks-experts/">LSTM</a> is a deep learning technique that was developed to overcome the vanishing and exploding gradients when training in traditional Recurrent Neural Network (RNN). The ability of LSTM to take in important information from past and ignore unimportant information, makes it very powerful in predicting future trend, especially in stock price prediction. For a more detailed understanding of LSTM, you can read this <a href="https://www.analyticsvidhya.com/blog/2017/12/fundamentals-of-deep-learning-introduction-to-lstm/">article</a>.</p><p>For now, let’s treat LSTM as a ‘black box’ and implement it. We will use past 60 days’ stock prices as our input variable and the 61th day’s stock price as our output variable, because I believe 60 days is a sufficient timeframe for the algorithm to learn the price trend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*BkaPbuy0zmQbWZO0QLXHRA.png" /><figcaption>Predicting stock price using LSTM</figcaption></figure><p>Wow! By just looking at the plot above, we can see LSTM performs quite well out-of-the-box.</p><h3><strong>The Winning Algorithm</strong></h3><p>Below is the evaluation result for the two algorithms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/273/1*Mrh6DOuEpFXqOp4-thv5uw.png" /></figure><p><strong>LSTM</strong> is the winner as it gives lower RMSE and MAPE, and the MAPE is below 5%.</p><h3><strong>Model Improvement</strong></h3><p>We can improve our model performance further through hyperparameters tuning.</p><h4>Tune Hyperparameters</h4><p>In machine learning, <a href="https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)">hyperparameter</a> is a parameter whose value is used to control the learning process; it cannot be inferred while fitting the machine to the training set.</p><p>Some of the LSTM hyperparameters that we can tune are:</p><ul><li>number of epochs</li><li>batch size</li><li>different optimizers</li></ul><p>I won’t be going through the full hyperparameter tuning, as it would take a long time; instead, I will show you the effect of different hyperparameters on model performance VS the initial hyperparameters we set, then pick the wining hyperparameter from each of the trainings and re-train our LSTM model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*Je_Bql7jAlrIlTlRFCMZOA.png" /></figure><p>Performance comparison before and after the LSTM hyperparameter tuning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/252/1*K4d00Kwb0pLrdv9z1o_BGg.png" /></figure><p><strong>You can find my Jupyter Notebook </strong><a href="https://github.com/louisteo9/stock-price-prediction"><strong>here</strong></a><strong> for more details on my codes.</strong></p><h3>Reflection</h3><p>LSTM algorithm works very well ‘out-of-the-box’ (without needing much tuning on its hyperparameters) in predicting our stock prices.</p><p>Although LSTM can be tuned for various hyperparameters to improve our LSTM model performance, we will only use the initial parameters to create our stock price predictor in view of shorter model training time.</p><h3>Create My Stock Price Predictor</h3><p>Let’s package the LSTM model training and predicting stock price to Python scripts.</p><p>Click <a href="https://github.com/louisteo9/stock-price-prediction">here</a> to download the Python scripts from my Github. You can start using it as an exercise to predict any stock prices of your choice.</p><p>Below are the complete scripts for <strong>train.py</strong> and <strong>predictor.py.</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/144e2a9bfd5ecbda6ade02d3e4d3c7b6/href">https://medium.com/media/144e2a9bfd5ecbda6ade02d3e4d3c7b6/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7b62eb0c8f4c926557295a0287969ea/href">https://medium.com/media/d7b62eb0c8f4c926557295a0287969ea/href</a></iframe><h3>Conclusion</h3><p>You have learnt how machine learning can be used to perform technical analysis in stock price prediction, and that LSTM algorithm works pretty good in predicting prices.</p><p>You have created your personal stock price predictor in Python. The price prediction, I should hope, looks pretty good in supplement to your fundamental analysis for a stock.</p><p>Note that LSTM alone cannot be expected to be sufficient to identify at 100% accuracy whether a stock price will increase or decrease, as stock prices are affected by many fundamental factors such as company’s earnings, future growth and profitability from producing and selling goods and services. Price is also driven by supply and demand for a company’s stock.</p><p><strong><em>Future work</em></strong></p><p>For future work, it will be interesting to explore how LSTM works on different time series problems.</p><p>It will also be interesting to explore how other time series forecasting techniques such as ARIMA and Facebook Prophet perform in stock price prediction and see how their performances compare with the algorithms above.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f38c229b758e" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/how-i-create-my-stock-price-predictor-f38c229b758e">I Used Machine Learning To Create A Stock Price Predictor</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beginner’s guide to creating a powerful chatbot]]></title>
            <link>https://medium.com/data-science/beginners-guide-to-creating-a-powerful-chatbot-48fc6b073e55?source=rss-d1e21a5b6a23------2</link>
            <guid isPermaLink="false">https://medium.com/p/48fc6b073e55</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[business]]></category>
            <dc:creator><![CDATA[Louis Teo]]></dc:creator>
            <pubDate>Sat, 07 Nov 2020 16:29:42 GMT</pubDate>
            <atom:updated>2020-11-10T00:34:24.289Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FqgeT26baoexFMoX" /><figcaption>Photo by <a href="https://unsplash.com/@askkell?utm_source=medium&amp;utm_medium=referral">Andy Kelly</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4><a href="https://towardsdatascience.com/tagged/hands-on-tutorials">Hands-on Tutorials</a></h4><h4>Ever thought of using a chatbot to help answer your customers’ standard inquiries?</h4><p>Are you facing too many standard requests and questions from customers and struggling to cope with them? Are you looking for a way to expand your customer service without incurring much costs?</p><p>In my previous <a href="https://towardsdatascience.com/report-is-too-long-to-read-use-nlp-to-create-a-summary-6f5f7801d355">story</a>, I shared how I tackled data overburden issues by creating a personal text summarizer to summarize a document.</p><p>In this story, I will show you how you can easily create a powerful chatbot to handle your growing customer requests and inquiries. I will also show you how to deploy your chatbot to a web application using Flask.</p><h3>First, why chatbot is particularly relevant in times like these, and what it is</h3><p>The Covid-19 pandemic has hit the world hard. Many businesses have suffered major losses due to lockdown / movement controls. To ride through this tough time, many were forced to move their businesses online.</p><p>One of the main, common problems faced by online business owners is having to respond to an overwhelming number of questions and requests from customers. For those with limited manpower resources, it’s impossible to deal with all requests in time.</p><p>To solve this problem, many business owners have turned to using chatbot for their customer service.</p><p><strong><em>What is a chatbot:</em></strong> a chatbot is AI-powered intelligent software that is capable of having conversations with human and performing human-like tasks. Chatbots are present in many smart devices (e.g. Siri (iOS), Google Assistant (Android), Cortana (Microsoft), Alexa (Amazon)), websites and applications.</p><p><strong><em>Why use a chatbot: </em></strong>According to a research carried out by <a href="https://www.hubspot.com/stories/chatbot-marketing-future">HubSpot</a>, 71% of users were willing to use quick-to-respond chat apps to receive customer assistance, and many do so because they want their problems to be solved <strong>FAST (and efficiently, of course)</strong>. A chatbot, if configured intelligently, indeed can unlock great values for businesses by allowing manpower resources to be focused on critical operations while maintaining the same level of customer satisfaction.</p><h3>How to create a chatbot?</h3><p>I will divide this into three major sections — but know that you can go directly to the section that is of your interest:</p><p><a href="#c03d">(a) Building a chatbot using Jupyter notebook</a></p><p><a href="#38b8">(b) Running chatbot in terminal</a></p><p><a href="#9bba">(c) Deploying chatbot as web app using Flask</a></p><h3>(a) Building a chatbot using Jupyter notebook</h3><p>Thanks to the ChatterBot library in Python, creating a chatbot is no longer a daunting machine learning task that it used to be.</p><p>Now, let’s get our hands dirty…</p><p><strong>(1) Install the ChatterBot library</strong></p><p>We will begin by installing the ChatterBot library. Installation commands for terminal are as follows:</p><pre>pip install chatterbot</pre><p>The ChatterBot text corpus (language resource consisting of a large and structured set of texts) is distributed in its own Python package, so you need to install it separately:</p><pre>pip install chatterbot_corpus</pre><p>If you have not installed <strong>spaCy </strong>(an open source library for advanced Natural Language Processing) before, then please install it now because the ChatterBot library needs spaCy library to work:</p><pre>pip install spacy</pre><p>Install spaCy <strong>English </strong>(‘en’)<strong> </strong>model after installing the spaCy library:</p><pre>python -m spacy download en</pre><p><strong>(2) Create chatbot instance</strong></p><p>We will develop our chatbot using <strong>Jupyter Notebook</strong> before we package the entire chatbot to Python scripts. Let’s begin by importing the module we need:</p><pre>from chatterbot import ChatBot</pre><p>We will create a chatbot instance, name our bot as <strong>Buddy </strong>and specify <strong>read_only</strong> parameter to <strong>True</strong> because we only want our chatbot to learn from our training data.</p><p>By creating a chatbot instance, a chatbot database named <strong>db.sqlite3</strong> will be created for you.</p><pre>bot = ChatBot(&#39;Buddy&#39;, read_only = True)</pre><p><strong>(3) Train the chatbot</strong></p><p>It’s time to train our chatbot… (<em>What, just like that? </em>Yes — it’s no joke!)</p><p>The training simply means feeding conversations into the chatbot database. See the example below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/1*SZrW22_-V3aWziCB_JeJ0Q.png" /><figcaption>Conversation feeds (Image by author)</figcaption></figure><p>As soon as the chatbot is fed with ‘conversation 1’ and ‘conversation 2’, the chatbot will store the conversations in its ‘knowledge graph’ database in the correct order.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*dJdv-1lT7XxQeLVhd9W_3w.png" /><figcaption>Knowledge graph stored in database (Image by author)</figcaption></figure><p>After that, you can initiate a conversation using any of the statements above.</p><p>For example, you can start a conversation with statement 6 (“ What payment methods do you accept?”), and the chatbot will respond with statement 7 (“We accept debit cards and major credit cards).</p><p>With this concept in mind, let’s train our chatbot with ‘conversation 1’ and ‘conversation 2’.</p><p>We do so by importing the <strong>ListTrainer</strong> module, instantiating it by passing the chatbot object (<strong>Buddy</strong>) and calling the <strong>train()</strong> method to pass a list of sentences.</p><pre>from chatterbot.trainers import ListTrainer<br>trainer = ListTrainer(bot)</pre><pre>trainer.train([<br>    &quot;Hi, can I help you&quot;,<br>    &quot;Who are you?&quot;,<br>    &quot;I am your virtual assistant. Ask me any questions...&quot;,<br>    &quot;Where do you operate?&quot;,<br>    &quot;We operate from Singapore&quot;,<br>    &quot;What payment methods do you accept?&quot;,<br>    &quot;We accept debit cards and major credit cards&quot;,<br>    &quot;I would like to speak to your customer service agent&quot;,<br>    &quot;please call +65 3333 3333. Our operating hours are from 9am to 5pm, Monday to Friday&quot;<br>    <br>])</pre><pre>trainer.train([<br>    &quot;What payment methods do you offer?&quot;,<br>    &quot;We accept debit cards and major credit cards&quot;,<br>    &quot;How to contact customer service agent&quot;,<br>    &quot;please call +65 3333 3333. Our operating hours are from 9am to 5pm, Monday to Friday&quot;<br>    <br>])</pre><p><strong><em>Chatbot Testing</em></strong></p><p>To check if we have fed our conversations correctly to our chatbot, let’s test it out by feeding the chatbot with just a simple input statement - ‘payment method’.</p><pre>response = bot.get_response (&#39;payment method&#39;)<br>print(response)</pre><p>The chatbot responded to us with:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wevURd9k5MdAhmYshXVurA.png" /></figure><p>This response is exactly what we want.</p><p>We will create a <strong>while loop </strong>to<strong> </strong>enable our chatbot to respond to each of our queries continuously<strong>. </strong>We will put an end to the loop and stop the program when we get ‘Bye’ or ‘bye’ statement from the user.</p><pre>name = input(&#39;Enter Your Name: &#39;)</pre><pre>print (&#39;Welcome to Chatbot Service! Let me know how can I help you&#39;)</pre><pre>while True:<br>    <br>    request = input(name+&#39;:&#39;)<br>    <br>    if request==&quot;Bye&quot; or request==&#39;bye&#39;:<br>        print(&#39;Bot: Bye&#39;)<br>        break<br>    else:<br>        response=bot.get_response(request)<br>        print(&#39;Bot: &#39;, response)</pre><p>Let’s try it out…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*4cloF8NlSCQ1fruguEKidw.gif" /><figcaption>Chatbot demo in Jupyter Notebook (GIF by author)</figcaption></figure><p>Congratulations! You have just created your very first, working chatbot!</p><p><strong><em>Corpus data training</em></strong></p><p>Of course you would want your chatbot to be able to have more conversations on top of those that we just fed in (!) — in that case, we need to train our chatbot further.</p><p>Fortunately, this task has been simplified for us. We can quickly train our chatbot to communicate in a more ‘human-like’ and smarter way with us, by using the available English corpus data in the package.</p><p><strong><em>Note: </em></strong><em>If you face problems in running the corpus training, please copy this </em><a href="https://github.com/gunthercox/chatterbot-corpus/tree/master/chatterbot_corpus"><em>chatterbot_corpus</em></a><em> folder to the file directory specified in the error message.</em></p><p>Just run the codes below, and your chatbot will be trained to have conversations in the following scope: <em>AI, botprofile, computers, conversations, emotion, food, gossip, greetings, health, history, humor, literature, money, movies, politics, psychology, science, sports &amp; trivia.</em></p><pre>from chatterbot.trainers import ChatterBotCorpusTrainer</pre><pre>trainer = ChatterBotCorpusTrainer(bot)</pre><pre>trainer.train(&#39;chatterbot.corpus.english&#39;)</pre><p>Let’s test if our chatbot has become smarter now….</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*QtOtvp7LtpyHKa7jmGX1mg.gif" /><figcaption>Chatbot testing after corpus data training (GIF by author)</figcaption></figure><p>Yes! It’s become smarter — it can tell you a joke now…</p><p><strong><em>Preprocess input</em></strong></p><p>ChatterBot comes with several built-in preprocessors that allow us to clean our input statement, before we get the statement processed by the bot’s logic adapter.</p><p>Cleaning makes our input statement more readable and analyzable by our chatbot. It removes ‘noise’ that can interfere with the text analysis from the input statement - e.g. additional space, Unicode characters and escaped html characters.</p><p>We will now include the preprocessors in our chatbot instance and rerun the chatbot instance with the codes below.</p><pre>bot = ChatBot(&#39;Buddy&#39;,<br>             read_only = True,<br>             preprocessors=[&#39;chatterbot.preprocessors.clean_whitespace&#39;,                            &#39;chatterbot.preprocessors.unescape_html&#39;,                          &#39;chatterbot.preprocessors.convert_to_ascii&#39;])</pre><p>Then, rerun the chatbot, and you can see we get the same response for <strong>who are you</strong> and <strong>whó are yóu</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jW8J4ksQDC8AcE-5xvdv1g.png" /></figure><p><strong><em>Low confidence response</em></strong></p><p>On top of that, we can also configure our chatbot to inform users if the input is not understood, with a default response:</p><p>‘<em>I am sorry, I do not understand. I am still learning. Please contact </em><a href="/service/mailto:abc@xxx.com"><em>abc@xxx.com</em></a><em> for further assistance.’</em></p><p>Let’s include the low confidence response to our chatbot instance and rerun the chatbot instance.</p><pre>bot = ChatBot(&#39;Buddy&#39;,<br>             logic_adapters = [<br>                 {<br><strong>                     &#39;import_path&#39;: &#39;chatterbot.logic.BestMatch&#39;,<br>                     &#39;default_response&#39;: &#39;I am sorry, I do not understand. I am still learning. Please contact </strong><a href="/service/mailto:abc@xxx.com"><strong>abc@xxx.com</strong></a><strong> for further assistance.&#39;,<br>                     &#39;maximum_similarity_threshold&#39;: 0.90</strong><br>                 }<br>             ],<br>             read_only = True,<br>             preprocessors=[&#39;chatterbot.preprocessors.clean_whitespace&#39;,<br>&#39;chatterbot.preprocessors.unescape_html&#39;,<br>&#39;chatterbot.preprocessors.convert_to_ascii&#39;])</pre><p>Then, rerun the chatbot, and let’s try to get a response from an unexpected input — our chatbot will reply with the <strong>default_response</strong> when it doesn’t understand a statement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a6KPSbHAW5obFn-as1ky5Q.png" /><figcaption>chatbot response with unexpected input (Image by author)</figcaption></figure><p>Congratulations! You have successfully built a chatbot using Jupyter notebook.</p><p><strong>Here is the link to </strong><a href="https://github.com/louisteo9/Chatbot"><strong>my Github</strong></a><strong> for the Jupyter Notebook on this.</strong></p><h3>(b) Running chatbot in terminal</h3><p><strong><em>Note:</em></strong><em> If you have come straight to this section (and skipped section </em><a href="#c03d"><em>(a) Building a chatbot using Jupyter notebook</em></a><em>), then please ensure you have installed all the required library and packages mentioned in that section before proceeding to run the chatbot in terminal.</em></p><p>We will use the Jupyter notebook above to write scripts that run the steps above and perform the chatbot training based on the conversation files specified by the user, and then see our chatbot in action...!</p><h4><strong>The complete scripts and instructions to run…</strong></h4><p>You can download my complete scripts <a href="https://github.com/louisteo9/chatbot">here</a> which you can use right away to train and run your chatbot!</p><ol><li>Create a <strong>training_data</strong> folder and store all the conversations you want to train in text file(s). The chatbot_training.py script will read all the text files in the <strong>training_data</strong> folder.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GY41UJEaYarb4-deaAGB1g.png" /></figure><p>The conversation in the text file should be inputted as one sentence per line:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*553l4O9Eh-h8s4PASJoMog.png" /></figure><p>2. Run <strong>chatbot_training.py</strong>. You will be asked to choose if you want to train the chatbot with English corpus data — select <strong>Y</strong> or <strong>N</strong>.</p><p>Select <strong>Y</strong> — your chatbot will be trained to have conversations in the following scope: <em>AI, botprofile, computers, conversations, emotion, food, gossip, greetings, health, history, humor, literature, money, movies, politics, psychology, science, sports &amp; trivia.</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5e166d3d3867fb0f03ccf08f9ff69e97/href">https://medium.com/media/5e166d3d3867fb0f03ccf08f9ff69e97/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*ZIfM_984nSYpPPAuOgab3A.gif" /></figure><p>3. Then, run the <strong>chatbot.py </strong>to<strong> </strong>launch the chatbot. Input some conversations and test if it responds properly.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/948619c06e83787c1cf996392859870a/href">https://medium.com/media/948619c06e83787c1cf996392859870a/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*p_jYSwWOtPhiqsHtAcTiWw.gif" /></figure><p>Tadaa… there you go! You have completed your chatbot training and run it in the terminal.</p><h3>(c) Deploying chatbot as web app using Flask</h3><p><strong><em>Note:</em></strong><em> If you have come straight this section (and skipped section </em><a href="#c03d"><em>(a) Building a chatbot using Jupyter notebook</em></a><em>), then please ensure you have installed all the required library and packages mentioned in that section before proceeding to deploy chatbot to web app using Flask.</em></p><p>We deploy our chatbot to a web application so that it can be used by customers.</p><p>To run our chatbot on a web application, we need to find a way for our application to receive incoming data and to return data. We can do this in any way we want — HTTP requests, web sockets etc.</p><p>There are a few existing examples available on the <a href="https://chatterbot.readthedocs.io/en/stable/faq.html">ChatterBot FAQ page</a> that show us how to do this. I will show you how to deploy a web application using Flask.</p><p>Download the example codes from <a href="https://github.com/louisteo9/chatbot">my Github</a>, edit the <em>index.html</em> file in <strong>template </strong>folder<strong> </strong>and <em>style.css</em> file in <strong>static</strong> folder to your liking. Leave the files as they are if you just want to test the chatbot on a web application.</p><p>After that, let’s run <strong>web_app.py.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*zKNCGy9UyabXdgki2r0F1g.gif" /></figure><p>This deploys our chatbot to a local server at <a href="http://127.0.0.1:5000/">http://127.0.0.1:5000/</a></p><p>Launch your web browser and go to the URL above to launch your chatbot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*9VJTzudXcMHizFNuunHdtg.gif" /></figure><h3>Conclusion, improvement tips… and what’s next?</h3><p>Congratulations (again)! You’ve successfully built your first business chatbot and deployed it to a web application using Flask. The chatbot, I should hope, did a pretty good job in answering some standard business questions you’ve had it trained on.</p><p>One of the things you can do to further improve the performance of your chatbot, is compile a list of FAQs that have been posted by your customers to date, provide answers to the FAQs and then train them on your chatbot.</p><p><strong><em>Why some chatbots failed to live up to expectations?</em></strong> Some chatbots failed simply because the standard questions and requests made to a business had not been analysed sufficiently, and as a result the chatbot did not receive the training it required. Training and improving your chatbot is a continuous process in the beginning and is similar to how humans learn new skills and knowledge. And once the skills are learned, they’re built in the chatbot and the chatbot does not need to be retrained, unless and until your business grows.</p><p>Next, you can look into deploying your chatbot to a Platform-as-a-Service (PaaS) of your choice, which can host and run your web application entirely from the cloud. One of the popular free PaaS’ that you can consider is <a href="https://www.heroku.com/">Heroku</a>.</p><p>Thank you for reading this story! Follow me on <a href="https://louisteo9.medium.com/">medium</a> for more tips on how to grow your business using DIY machine learning.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48fc6b073e55" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/beginners-guide-to-creating-a-powerful-chatbot-48fc6b073e55">Beginner’s guide to creating a powerful chatbot</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Report is too long to read? Use NLP to create a summary]]></title>
            <link>https://medium.com/data-science/report-is-too-long-to-read-use-nlp-to-create-a-summary-6f5f7801d355?source=rss-d1e21a5b6a23------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f5f7801d355</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[productivity]]></category>
            <dc:creator><![CDATA[Louis Teo]]></dc:creator>
            <pubDate>Thu, 29 Oct 2020 11:22:33 GMT</pubDate>
            <atom:updated>2021-02-08T09:44:23.472Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-LO-i2tda4IonjZW" /><figcaption>Photo by <a href="https://unsplash.com/@officestock?utm_source=medium&amp;utm_medium=referral">Sebastian Herrmann</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>A guide to creating your personal text summarizer</h4><p>Have you ever had one too many reports to read and you just want a quick summary of each report? Were you ever in a situation where everybody just wanted to read a summary instead of a full-blown report?</p><p>Summarization has become a very helpful way of tackling the issue of data overburden in the 21st century. In this story, I will show you how you can create your personal text summarizer using Natural Language Processing (NLP) in Python.</p><p><em>Foreword: Personal text summarizer is not hard to create — a beginner can easily do it!</em></p><h3><strong>What is text summarization?</strong></h3><p>It’s basically a task to generate an accurate summary while maintaining key information and not losing overall meaning.</p><p>There are two general types of summarization:</p><ul><li><strong>Abstractive summary</strong> &gt;&gt; generate new sentences from original text.</li><li><strong>Extractive summary</strong> &gt;&gt; recognize important sentences and create a summary using those sentences.</li></ul><h3><strong>Which summarization method should I use, and why?</strong></h3><p>I use extractive summary because I can apply this method to many documents without having to do a lot of (daunting) machine learning model training tasks.</p><p>Besides that, extractive summarization gives better summary outcome than abstractive summary, because abstractive summarization has to generate new sentences from the original text, which is a more difficult method than a data-driven approach to extract important sentences.</p><h3><strong>How to create your own Text Summarizer?</strong></h3><p>We will use word histogram to rank the importance of sentences and, subsequently, create a summary. The benefit of doing this is that you don’t need to train your model to use it for your document.</p><h4>Text Summarization Workflow</h4><p>Below is the workflow that we will be following…</p><p><em>import text&gt;&gt; &gt;&gt; clean text and split into sentences &gt;&gt; remove stop words &gt;&gt; build word histogram&gt;&gt; rank sentences&gt;&gt; select top N sentences for summary</em></p><p><strong>(1) Sample Text</strong></p><p>I used the text from a news article entitled <strong>Apple Acquires AI Startup For $50 Million To Advance Its Apps. </strong>You can find the original news article <a href="https://analyticsindiamag.com/apple-acquires-ai-startup-for-50-million-to-advance-its-apps/">here</a>.</p><p>You can also download the text document from <a href="https://github.com/louisteo9/personal-text-summarizer">my Github</a>.</p><p><strong>(2) Import libraries</strong></p><pre># Natural Language Tool Kit (NLTK)<br>import nltk<br>nltk.download(&#39;stopwords&#39;)<br>nltk.download(&#39;punkt&#39;)</pre><pre># Regular Expression for text preprocessing<br>import re</pre><pre># Heap (priority) queue algorithm to get the top sentences<br>import heapq</pre><pre># NumPy for numerical computing<br>import numpy as np</pre><pre># pandas for creating DataFrames<br>import pandas as pd</pre><pre># matplotlib for plot<br>from matplotlib import pyplot as plt<br>%matplotlib inline</pre><p><strong>(3) Import text and perform preprocessing</strong></p><p>There are many ways to do this. The goal here is to have a clean text that we can feed into our model.</p><pre># load text file<br>with open(&#39;Apple_Acquires_AI_Startup.txt&#39;, &#39;r&#39;) as f:<br>    file_data = f.read()</pre><p>Here, we use regular expression to do text preprocessing. We will (A) replace reference number, i.e. [1], [10], [20], with empty space (if any…), (B) replace one or more spaces with single space.</p><pre>text = file_data<br># replace reference number with empty space, if any..<br>text = re.sub(r&#39;\[[0-9]*\]&#39;,&#39; &#39;,text) </pre><pre># replace one or more spaces with single space<br>text = re.sub(r&#39;\s+&#39;,&#39; &#39;,text)</pre><p>Next, we form a <strong>clean text</strong> with lower case (without special characters, digits and extra spaces) and split it into individual word, for word score computation and formation of the word histogram.</p><p>The reason to form a clean text is so that the algorithm won’t treat, e.g. <strong>“understanding”</strong> and <strong>understanding</strong>, as two different words.</p><pre># convert all uppercase characters into lowercase characters<br>clean_text = text.lower()</pre><pre># replace characters other than [a-zA-Z0-9], digits &amp; one or more spaces with single space<br>regex_patterns = [r&#39;\W&#39;,r&#39;\d&#39;,r&#39;\s+&#39;]<br>for regex in regex_patterns:<br>    clean_text = re.sub(regex,&#39; &#39;,clean_text)</pre><p><strong>(4) Split (tokenize) text into sentences</strong></p><p>We split the text into sentences using NLTK <strong>sent_tokenize()</strong> method. We will evaluate the importance of each of the sentences, then decide if we should include each in our summary.</p><pre>sentences = nltk.sent_tokenize(text)</pre><p><strong>(5) Remove stop words</strong></p><p>Stop words are English words which do not add much meaning to a sentence. They can be safely ignored without sacrificing the meaning of the sentence. We already downloaded a file with English stop words in ‘(2) Import libraries’ section.</p><p>Here, we will get the list of stop words and store them in <strong>stop_word</strong> variable.</p><pre># get stop words list<br>stop_words = nltk.corpus.stopwords.words(&#39;english&#39;)</pre><p><strong>(6) Build word histogram</strong></p><p>Let’s evaluate the importance of each word based on how many times it appears in the entire text.</p><p>We will do so by (1) splitting the words in <strong>clean_text</strong>, (2) removing the stop words, and then (3) checking the frequency of each word as it appears in the text.</p><pre># create an empty dictionary to house the word count<br>word_count = {}</pre><pre># loop through tokenized words, remove stop words and save word count to dictionary<br>for word in nltk.word_tokenize(clean_text):<br>    # remove stop words<br>    if word not in stop_words:<br>        # save word count to dictionary<br>        if word not in word_count.keys():<br>            word_count[word] = 1<br>        else:<br>            word_count[word] += 1</pre><p>Let’s plot the word histogram and see the results.</p><pre>plt.figure(figsize=(16,10))<br>plt.xticks(rotation = 90)<br>plt.bar(word_count.keys(), word_count.values())<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*UdsTZJByLiZtWTSALDfT5w.png" /></figure><p>Ahhh… it’s a bit difficult to read the plot. Let’s convert it to horizontal bar plot and display only the top 20 words with a helper function below.</p><pre># helper function for plotting the top words.<br>def plot_top_words(word_count_dict, show_top_n=20):<br>    word_count_table = pd.DataFrame.from_dict(word_count_dict, orient = &#39;index&#39;).rename(columns={0: &#39;score&#39;})<br>    word_count_table.sort_values(by=&#39;score&#39;).tail(show_top_n).plot(kind=&#39;barh&#39;, figsize=(10,10))<br>    plt.show()</pre><p>Let’s display the top 20 words.</p><pre>plot_top_words(word_count, 20)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*hbG_AYOy98reTdZYnp2KjQ.png" /></figure><p>From the plot above, we can see the words <strong>‘ai’</strong> and <strong>‘apple’</strong> appear on the top. This makes sense because the article is about Apple acquiring an AI startup.</p><p><strong>(6) Rank sentences based on scores</strong></p><p>Now, we are going to rank the importance of each sentence based on sentence score. We will:</p><ul><li>remove sentences that have more than <strong>30</strong> words, recognizing that long sentences may not always be meaningful**;</li><li>then, add score (count) from each word that forms the sentence to form the sentence score.</li></ul><p>Sentences that have high scores will form our top sentences. The top sentences will form our summary later.</p><p>**<strong>Note:</strong> In my experience, any word count between <strong>25</strong> and <strong>30</strong> should give you a good summary.</p><pre># create empty dictionary to house sentence score    <br>sentence_score = {}</pre><pre># loop through tokenized sentence, only take sentences that have less than 30 words, then add word score to form sentence score<br>for sentence in sentences:<br>    # check if word in sentence is in word_count dictionary<br>    for word in nltk.word_tokenize(sentence.lower()):<br>        if word in word_count.keys():<br>            # only take sentence that has less than 30 words<br>            if len(sentence.split(&#39; &#39;)) &lt; <strong>30</strong>:<br>                # add word score to sentence score<br>                if sentence not in sentence_score.keys():<br>                    sentence_score[sentence] = word_count[word]<br>                else:<br>                    sentence_score[sentence] += word_count[word]</pre><p>We convert the <strong>sentence_score</strong> dictionary to a DataFrame and display the sentences and scores.</p><p><strong>Note</strong>: dictionary doesn’t allow you to sort the sentences based on scores, so you need to convert the data stored in dictionary to DataFrame.</p><pre>df_sentence_score = pd.DataFrame.from_dict(sentence_score, orient = &#39;index&#39;).rename(columns={0: &#39;score&#39;})<br>df_sentence_score.sort_values(by=&#39;score&#39;, ascending = False)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*354O9moWFr-MbtZH-MYVpw.png" /></figure><p><strong>(7) Select top sentences for summary</strong></p><p>We use heap queue algorithm to select the top 3 sentences and store them in <strong>best_sentences</strong> variable.</p><p>Usually 3–5 sentences will be enough. Depending on the length of your document, feel free to change the number of top sentences to be displayed.</p><p>In this case, I chose <strong>3</strong> because our text is a relatively short article.</p><pre># display the best 3 sentences for summary  <strong> </strong>          <br>best_sentences = heapq.nlargest(<strong>3</strong>, sentence_score, key=sentence_score.get)</pre><p>Let’s display our summarized text using <strong>print()</strong> and <strong>for loop</strong> functions.</p><pre>print(&#39;SUMMARY&#39;)<br>print(&#39;------------------------&#39;)</pre><pre># display top sentences based on their sentence sequence in the original text<br>for sentence in sentences:<br>    if sentence in best_sentences:<br>        print (sentence)</pre><p><strong>Here is the link to </strong><a href="https://github.com/louisteo9/personal-text-summarizer"><strong>my Github</strong></a><strong> to get the Jupyter notebook for this.</strong></p><p>Below is the complete Python script that you can use right away to summarize your text.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/10e7e06484da533a2422005cdee9b883/href">https://medium.com/media/10e7e06484da533a2422005cdee9b883/href</a></iframe><h3>Let’s see the algorithm in action!</h3><p>Below is the original text from a news article entitled <strong>Apple Acquires AI Startup For $50 Million To Advance Its Apps </strong><em>(the original news article can be found </em><a href="https://analyticsindiamag.com/apple-acquires-ai-startup-for-50-million-to-advance-its-apps/"><em>here</em></a><em>)</em>:</p><pre>In an attempt to scale up its AI portfolio, Apple has acquired Spain-based AI video startup — Vilynx for approximately $50 million.</pre><pre>Reported by Bloomberg, the AI startup — Vilynx is headquartered in Barcelona, which is known to build software using computer vision to analyse a video’s visual, text, and audio content with the goal of “understanding” what’s in the video. This helps it categorising and tagging metadata to the videos, as well as generate automated video previews, and recommend related content to users, according to the company website.</pre><pre>Apple told the media that the company typically acquires smaller technology companies from time to time, and with the recent buy, the company could potentially use Vilynx’s technology to help improve a variety of apps. According to the media, Siri, search, Photos, and other apps that rely on Apple are possible candidates as are Apple TV, Music, News, to name a few that are going to be revolutionised with Vilynx’s technology.</pre><pre>With CEO Tim Cook’s vision of the potential of augmented reality, the company could also make use of AI-based tools like Vilynx.</pre><pre>The purchase will also advance Apple’s AI expertise, adding up to 50 engineers and data scientists joining from Vilynx, and the startup is going to become one of Apple’s key AI research hubs in Europe, according to the news.</pre><pre>Apple has made significant progress in the space of artificial intelligence over the past few months, with this purchase of UK-based Spectral Edge last December, Seattle-based Xnor.ai for $200 million and Voysis and Inductiv to help it improve Siri. With its habit of quietly purchasing smaller companies, Apple is making a mark in the AI space. In 2018, CEO Tim Cook said in an interview that the company had bought 20 companies over six months, while only six were public knowledge.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*f9PYpAxKWMz7cGH7G-Fmug.gif" /></figure><p>… and the summarized text is as follows:</p><pre>In an attempt to scale up its AI portfolio, Apple has acquired Spain-based AI video startup — Vilynx for approximately $50 million.<br>With CEO Tim Cook’s vision of the potential of augmented reality, the company could also make use of AI-based tools like Vilynx.<br>With its habit of quietly purchasing smaller companies, Apple is making a mark in the AI space.</pre><h3>Conclusion… and one last tip</h3><p>Congratulations!</p><p>You have created your personal text summarizer in Python. The summary, I should hope, looks pretty decent.</p><p>It is important to note that we used <strong>word frequency</strong> in a document to rank the sentences. The advantage of using this method is that it does not require any prior training and can work on any piece of text. As another tip, you can further tweak the summarizer to your liking based on:</p><p>(1) <strong>The number of top sentences: </strong>The simple rule-of-thumb here is that the length of a summary should not be more than ¼ of the original text — it can be one sentence, one paragraph or multiple paragraphs depending on the length of the original text and your purpose of getting the summary. If the text you want to summarize is long, then you can increase the number of top sentences; or</p><p>(2) <strong>Sentence length:</strong> On average, a sentence length today ranges between <a href="https://techcomm.nz/Story?Action=View&amp;Story_id=106#:~:text=How%20many%20words%20should%20we,2009%3B%20Vincent%2C%202014).">15 and 20 words</a>. Therefore, limiting your summarizer to take only the sentences of length than 25–30 words is enough; however, feel free to increase and decrease the word count.</p><p>Thank you for reading this story. Follow me on <a href="https://louisteo9.medium.com/">medium</a> for more of my sharing on data science and machine learning.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f5f7801d355" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/report-is-too-long-to-read-use-nlp-to-create-a-summary-6f5f7801d355">Report is too long to read? Use NLP to create a summary</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Started With Machine Learning for Newbies]]></title>
            <link>https://medium.com/swlh/getting-started-with-machine-learning-for-newbies-b75f55f39ef6?source=rss-d1e21a5b6a23------2</link>
            <guid isPermaLink="false">https://medium.com/p/b75f55f39ef6</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[newbie]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Louis Teo]]></dc:creator>
            <pubDate>Sun, 25 Oct 2020 10:02:17 GMT</pubDate>
            <atom:updated>2021-01-17T13:32:06.822Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*xMB8eqalQlffUbC6.jpg" /><figcaption>Image source: <a href="https://cdn.techinasia.com/wp-content/uploads/2013/12/baby-entrepreneur.jpg">https://cdn.techinasia.com/wp-content/uploads/2013/12/baby-entrepreneur.jpg</a></figcaption></figure><h3>Getting Started With Machine Learning — for Newbies</h3><h3><strong>Introduction</strong></h3><p>You may often hear the term “Machine Learning” and wonder how you can apply it to the datasets you have. In this post, I will show you how to get started with Machine Learning by using a sample dataset obtained from <a href="https://archive.ics.uci.edu/ml/index.php">UCI Machine Learning Repository</a>, and by answering the following top 3 questions that one curious about Machine Learning often has:</p><ol><li>What is the overall Machine Learning workflow?</li><li>How to apply Machine Learning model to a dataset?</li><li>What is the best overall Machine Learning model that one can apply to almost all datasets, and what are the tips to improve overall model performance?</li></ol><h3><strong>(1) Machine Learning Workflow</strong></h3><p>If you are a fresh data scientist or have just started learning Machine Learning, you can use the following workflow to create a functional Machine Learning project. Machine Learning typically follows the following workflow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*RUcSRVeyAxuGfgpw-E5pEQ.png" /><figcaption>Image by Author</figcaption></figure><ol><li><strong>Exploratory Analysis &gt;&gt; </strong>first, look at your dataset.</li><li><strong>Data Cleaning &gt;&gt; </strong>clean up duplicate data, drop unwanted observations, fix data structural problem, remove outliers, etc. You need to perform this step because proper data can make or break your project. Having good data is more important than having fancy algorithms.</li><li><strong>Feature Engineering &gt;&gt; </strong>apply the domain knowledge that you know, to engineer some features that can improve model performance.</li><li><strong>Algorithm Selection &gt;&gt; </strong>select<strong> </strong>a few<strong> </strong>relevant algorithms to be applied.</li><li><strong>Model Training &gt;&gt; </strong>split the data into training and testing sets and feed the data into model.</li></ol><h3>(2) Applying Machine Learning model to a dataset</h3><p>I will briefly demonstrate how the entire Machine Learning workflow is applied from start to finish, including applying Machine Learning model to a dataset. We won’t be focusing on the fine details of each of the steps, but rather how they work together.</p><p>The goal here is to develop high-level intuition of the entire Machine Learning process, which is extremely important but often overlooked.</p><p>We will be using <strong>Jupyter Notebook</strong> for this exercise. You can find my detailed notebook <a href="https://github.com/louisteo9/student-performance">here</a>.</p><p>The notebook develops a model to predict student performance in the maths subject in two Portuguese secondary schools. We will apply regression model to the dataset.</p><p>The dataset can be downloaded <a href="https://archive.ics.uci.edu/ml/datasets/Student+Performance">here</a>.</p><p><strong><em>Exploratory Analysis</em></strong></p><p>First, we will start by getting to know the data. Our goal here is to understand the dataset at a high level. We will take note of anything that should be cleaned up and spot any chances to implement feature engineering.</p><p>In our case, we find that our data is comprised of two data types: ‘object’ and ‘int64’. ‘object’ is categorical feature, and ‘int64’ is numerical feature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZiRFNyc67RChymyhA6enhw.png" /><figcaption>Display first few rows of data</figcaption></figure><p>We can always dive deeper into the dataset by looking at the distribution of numerical features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*aXxec8e_KZBbjfiNAxtIPw.png" /><figcaption>Distribution of the numerical data</figcaption></figure><p>It’s also a good idea to look at the correlations between the numerical features. One of the best ways is to use <strong>Seaborn</strong>’s heatmap.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/409/1*PI0l6oPH3gniTKFiB4RQBQ.png" /><figcaption>Seatborn’s heatmap</figcaption></figure><p><strong><em>Data Cleaning</em></strong></p><p>Remember, good data is always more important than fancy algorithms. In the second step, we will drop duplicated data, deal with missing data, remove outliers and fix any data structural error.</p><p>If your data is collected in a complete form, such as in this case, then you are unlikely to find any missing data or any structural error.</p><p><strong><em>Feature Engineering</em></strong></p><p>Once the data cleaning step is done, we will perform some feature engineering based on the domain knowledge we know about the dataset.</p><p>In our case, we notice that the data in <strong>famsize</strong> (family size) column is in ordinal form; so, we will do some feature engineering to change the <strong>LE3</strong> (less than 3) to <strong>0</strong> and <strong>GT3</strong> (greater than 3) to <strong>1</strong>.</p><p><em>Dummy Variables</em></p><p>Machine learning algorithms cannot handle categorical features directly; they can only take in numerical value, so next we need to create <strong>dummy variables</strong> for categorical features in order to allow us to run machine learning algorithm on our dataset.</p><p><strong><em>Algorithm Selection</em></strong></p><p>For practical Machine Learning application, it’s better to allow more complexity in your models, and then apply other measures to protect it against overfitting. The reason is that if our models are too simple, they can’t learn complex patterns no matter how much data we have; if our models are complex, we can always apply techniques to avoid overfitting.</p><p>In this algorithm selection, we will select the right algorithms that can help us prevent overfitting.</p><p>Our picks are: 3x regularized regression algorithms and 2x tree ensemble algorithms.</p><p><em>Why 3x linear regression algorithms and 2x tree ensemble algorithms?</em></p><p>Linear regression algorithm suffers from two main problems: it’s prone to overfitting if we have too many input features, and it cannot easily express non-linear relationship.</p><p>Therefore, to assist us in finding the balance for model complexity without overfitting, we will also deploy 2x Tree Ensembles algorithms.</p><p>You can refer to the cheat sheet <a href="https://download.microsoft.com/download/3/5/b/35bb997f-a8c7-485d-8c56-19444dafd757/azure-machine-learning-algorithm-cheat-sheet-nov2019.pdf?WT.mc_id=docs-article-lazzeri">here</a> to help you choose the best algorithm for your dataset.</p><p><strong><em>Model Training</em></strong></p><p>Once you have done your proper data preparation work, model training is straightforward.</p><p>We split the our data into input variable (X) and output variable (y).</p><p>Before we start with the model training, we need to divide our data into training and testing sets. This is because we can only spend some to train our model and some to test our model, and we cannot use the same data for both. Testing data will be used to evaluate the model performance.</p><p>Usually 80% of the data will be used for training and 20% of the data for testing.</p><p>We will use Scikit-Learn <strong>train_test_split()</strong> function to split our data into training and testing sets.</p><p>Then, we instantiate all the 5 models. To ensure the repeatability of the results, we put an arbitrary seed number <strong>random_state = 1234</strong> to the model.<strong> </strong>You can choose any arbitrary seed number.</p><p>Then model training process is simply feeding the training data using <strong>.fit()</strong> method.</p><p><em>Example:</em></p><pre>rg_rf = RandomForestRegressor(randome_state=1234)</pre><pre>rg_rf.fit(X_train, y_train)</pre><p>Once model is trained, we need to evaluate the performance of our model.</p><p>For our dataset, there are two ways to evaluate our models.</p><ol><li><strong>R² score</strong>. In layman’s terms, R² is the variance in the target variable that can be explained by the model. The <strong>higher</strong> the R² score, the <strong>better</strong> the model performance.</li><li><strong>Mean Absolute Error (MAE)</strong>. In layman’s terms, MAE is the average absolute difference between the predicted and the actual values for our target variable. The lower the MAE, the better the model performance.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/1*-J_4-uMk62UI6BcGw0bcQw.png" /><figcaption>R² and MAE scores for each model in student performance dataset</figcaption></figure><p>From the results above, you can see <strong>Random Forest</strong> model scored the highest R² and the lowest MAE.</p><p>We can also visualize the results by actual result vs predicted result. If a model performs well, the predicted result will ‘track’ the actual result well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/319/1*i0tILNhJHYES54wZDpI9qg.png" /><figcaption>Performance for each model (predicted result vs actual result)</figcaption></figure><p>The predicted result ‘tracks’ actual result well. Let’s keep this in mind for now.</p><p>It is also vital to know the feature variables that have the most impact on our output (y) variables. We can easily visualize this by plotting the top 10 Gini importance score for our feature variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*R29DCJCvE9oDa2q3m_j8fw.png" /></figure><p>From the plot above, G2 results, in combination with the ‘absence’ rate, are the two most important features in determining the final grade results (G3).</p><p>We have completed applying machine learning model to our dataset.</p><h3>(3a) My favorite Machine Learning Model</h3><p>From the model training <a href="https://github.com/louisteo9/student-performance/blob/master/Student%20Performance.ipynb">here</a>, you can see that the <strong>Random Forest</strong> model performs best in the five chosen models.</p><p><strong>Random Forest</strong> model is my favorite model because it works well out-of-the-box, can model non-linear relationships and is generally very robust to outliers.</p><h3><strong>(3b) Three tips to improve overall model performance</strong></h3><p><strong><em>Feature engineering</em></strong> is one of the best ways to improve model performance and add value to applied machine learning process.</p><p>How?</p><ul><li>Based on your domain knowledge, engineer features that you think will potentially improve the performance of your model.</li><li>Create interaction and indicator features, and group sparse classes in categorical features, and encode ordinal categorical features as numeric.</li><li>Drop unwanted or redundant features (e.g. ID columns, text descriptions, metadata, etc).</li></ul><p><strong><em>Tune hyperparameters</em></strong> is crucial, as they control the overall behavior of a machine learning model. These parameters cannot be learnt through the dataset and are set before the model training. The ultimate goal is to find an optimal combination of hyperparameters that minimizes the loss function and gives better results.</p><p>We only need to choose a range of hyperparameters to be tuned via Cross-Validation (CV). CV will find the best combination of given hyperparameters for us.</p><p><strong><em>Try other algorithms. </em></strong>Refer to the cheat sheet <a href="https://download.microsoft.com/download/3/5/b/35bb997f-a8c7-485d-8c56-19444dafd757/azure-machine-learning-algorithm-cheat-sheet-nov2019.pdf?WT.mc_id=docs-article-lazzeri">here</a> to help you choose the best machine learning algorithm for your dataset.</p><p>Make your decision based on the nature of your data and the goal you want to achieve with you data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-pZWNt6J-8PPkYVM4V18cQ.png" /></figure><h3><strong>Summary</strong></h3><p>Congratulations…!</p><p>You have learnt the machine learning workflow and built a successful machine learning model using a dataset! You have also learnt that the best overall machine learning model is Random Forest - it works well out-of-the-box - and you have learnt a few tips to improve your overall model performance.</p><p>Let me know how you get on with machine learning.</p><p>Please share your comments below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b75f55f39ef6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/getting-started-with-machine-learning-for-newbies-b75f55f39ef6">Getting Started With Machine Learning for Newbies</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>