<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gilang R Ilhami on Medium]]></title>
        <description><![CDATA[Stories by Gilang R Ilhami on Medium]]></description>
        <link>https://medium.com/@gilangrilhami?source=rss-27416751a23e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*tavyfcCsS034RKhisVWmuA.jpeg</url>
            <title>Stories by Gilang R Ilhami on Medium</title>
            <link>https://medium.com/@gilangrilhami?source=rss-27416751a23e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 16:50:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gilangrilhami/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to implement Redis caching in a NodeJS project]]></title>
            <link>https://medium.com/@gilangrilhami/how-to-implement-redis-caching-in-a-nodejs-project-6a2d03b74ab3?source=rss-27416751a23e------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a2d03b74ab3</guid>
            <category><![CDATA[scalability]]></category>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[backend]]></category>
            <dc:creator><![CDATA[Gilang R Ilhami]]></dc:creator>
            <pubDate>Fri, 21 Aug 2020 15:40:24 GMT</pubDate>
            <atom:updated>2020-08-21T15:40:24.417Z</atom:updated>
            <content:encoded><![CDATA[<p><em>“Caching, caching, caching.” — Some guy who wrote a blog post on designing scalable system that I read</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*2-n1HLpSd0sEftpaXRaQnA.jpeg" /></figure><p>Back in the days, I remember a young backend engineer who had this passion of building a scalable system that can handle the load of 10,000 users with low-latency.</p><p>That developer had little knowledge of building data-intensive applications. All he had was a server, which contained contained an SQL database and a NodeJS project, and a dream that his system would scale.</p><p>After spending some time putting all the necessary technical infrastructure in place, as a prudent backend engineer, he decided to do some stress testing. He wanted to see if his masterpiece could deliver on his expectations.</p><p>After spending sometime creating the script, off he goes to stress test. The young engineer was quite pleased at first to see that the system he created was able to deliver fast response for hundreds of users.</p><p>However, as the number of users reach thousands, the latency started to drop, and so did his believe that his system is scalable. Eventually, the response time reach a rate of 40 seconds; and that’s just the median.</p><p>Despite the disappointment, he didn’t stop. After doing some analysis on the server performance, he discovered that the database was handling a huge load of read and write operations due to relational complexity!</p><p>And the the young backend engineer thought, “what if I could take some of the loads from the read operations?” A question that would lead him to dig deeper to the world of scalable systems, and discover the power of caching.</p><p>Okay, time to address the elephant in the room. Yes, that young backend engineer was ME.</p><p>In this blog post, I want to show a exactly the basic implementation that I did in order to increase response time of my NodeJS project using Redis caching.</p><h3>What is caching? What is Redis?</h3><p>The first time I learned about caching, I put a small importance on it. Today, as a backend engineer, I considered it an important element in order for me to develop low-latency systems.</p><p>Caching is basically the process of storing a subset of data into memory cache. A cache is a high speed data-storage layer that is transient in nature and allows faster data deliverables than the primary storage itself.</p><p>One of the most common technology used for caching is Redis.</p><p>I’m going too much into the details here, but in a nutshell Redis is an open-source in-memory data storage that is commonly used as a cache, database, and message queue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*Ci_QJsHk8Da_VZ94AO5_uw.jpeg" /><figcaption>Redis</figcaption></figure><p>It can be considered a NoSQL database that stores subset of data in the form of key-values. It supports various data such as <em>strings, hashes, lists, sets </em>and<em> sorted sets.</em></p><p>Alternative to Redis as an open-source tools for caching is Memcached, which we’re not going to talk about now, but I encourage any curious backend engineers to look it up.</p><h3>Source Code</h3><p>Now that we got the introduction and concept of Redis out of the way, I’m pretty sure that whoever reading can’t wait any longer but to actually start implementing the code.</p><p>That is actually what I’m going to show all of you here. I’m going to implement and structure the code in a way that how I would usually structure my NodeJS related projects.</p><p>As a use case, I will use the <a href="https://www.kaggle.com/jessemostipak/volcano-eruptions">Volcano Eruptions</a> data from Kaggle. I’ve already converted the csv file into a db.sqlite file, which we will be using as the database.</p><p>First of all, you need to get the code to start of your development. Here is the <a href="https://github.com/grilhami/How-to-implement-Redis-caching-in-a-NodeJS/tree/master/code">source code</a>. The structure if the directories and files should look something like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/258/1*83Yfw_IMG71TXnE4BK-mFA.jpeg" /><figcaption>Source Code Project Structure</figcaption></figure><p>The source code already included the controller, route, and model for creating a GET request to retrieve all the information of eruptions from the database.</p><p>There is one more folder named “middlewares”. This is where we will be including the Redis cache.</p><p>Make sure to install all the necessary packages for running the source code. The package.json file should be helpful.</p><p>Also, in order to use Redis, you need to to install Redis on Terminal or Command Prompt to run the Redis server. I used the Windows Linux Subsystem, so I installed it using the Linux terminal.</p><h3>The implementation</h3><p>Now that we got the source code ready, we can start integrating caching mechanism. For me, it’s much more convenient to include it as a middleware, which will be included in the route file.</p><p>To do that, we need to add a new file in our middleware folder. In this case, named the new file as “cacheMiddleware.js”.</p><p>Next we need to include the Redis package (already included in the Json file for installation), and connect it with the Redis server by creating a new client. The default port for the server is 6379.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5f4c166afbf6166999511aaa3c2b937c/href">https://medium.com/media/5f4c166afbf6166999511aaa3c2b937c/href</a></iframe><p>Alright, now that we got the Redis client connect, we need to create a function that will get the data from the cache. The function acts as a middleware and look something like…</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a12fd729d79875e0202fe22579a8dbb3/href">https://medium.com/media/a12fd729d79875e0202fe22579a8dbb3/href</a></iframe><p>The purpose of this function is to retrieve the all the eruptions data if it exist in the cache. Since the Redis store data in the form of key-values, we want to the get the values with the key “eruptions”.</p><p>If no data exist, then we just continue on. This might be a little hard to understand, but it will make more sense after we actually include this in the route as the middleware.</p><p>In the last part of the code, we also want to export the Redis client. This is important since we are the client in the controller to store data if no values were found with the corresponding key.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f74c2f803211bc34cdf973aa5b9f140f/href">https://medium.com/media/f74c2f803211bc34cdf973aa5b9f140f/href</a></iframe><p>Let’s talk a little bit about the code above. It is the controller for getting all eruption data.</p><p>First, we need to include the Redis client from the middleware file.</p><p>After that, we want to get the eruption data from the model, and store it in the cache using the key “eruptions” in the form of a JSON string. Note that this is why in the cache middleware we parse the value.</p><p>Here we store the data for 5 seconds only. After caching the data, return the value with a code of 200, which is a success. If anything went wrong, then we just return the error message with a code of 500, which means server error.</p><p>That’s all we need to do with the controller. Now the route.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2b0178e463c40e0f323e62bfb70168d2/href">https://medium.com/media/2b0178e463c40e0f323e62bfb70168d2/href</a></iframe><p>For the route, we need to include the controller from eruptionController file, and the cache function from cacheMiddleware file. Both of them will be included in the route.</p><p>This is where the next() function start to make sense. If no data in the cache is not found, then it will jump to the controller and retrieve the data from the database, and then store them in the cache.</p><p>And that is essentially how we integrate Redis into the project to perform caching. Now all we need to do is test it out. I will use Postman in order to make a request to the API running in a localhost.</p><p>Don’t forget that you need to have the Redis server running also. You can do by running redis-server in separate Terminal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xQXsswgbqK7b2Hoj0t5syg.jpeg" /></figure><p>As you can see, since we have a lot of data to retrieve, and it takes 1526 ms for a data size of 4.41 MB. Note that we currently do not have the data cache, so the API will fetch the data from the database first and then cache them</p><p>If we try to make a request again …</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FkTqnKKxnsMhMdMsFh1hBg.jpeg" /></figure><p>Now it takes only 229 ms to get retrieve the same data. Around 0.15 the time it takes to retrieve the data from the database.</p><p>After 5 second, however, it will attempt to retrieve the data from the database again since we set the time for caching for only 5 seconds. Feel free to set the time yourself.</p><h3>Conclusion</h3><p>In this article, we learned on how to implement caching in our NodeJS project using Redis. The implementation involve the following steps:</p><ul><li>Create a NodeJS application and organize it in the form of model, route, and controller structure.</li><li>Create a dedicated directory for the caching mechanism function, which would act as a middleware</li><li>Export the Redis client from the middleware file, and use it in the controller to store data in the cache if it previously did not exist.</li><li>Include the cahing mechanism function in the route. This way, it will first try to retrieve data from the cache. If no data exist, then query the primary database.</li></ul><p>And there you go! That is how I would implement Redis in a NodeJS application … well, only in the beginning of course.</p><p>As the application grows with various features being added, more complex mechanism are required in order to cache other kinds of data for different request.</p><p>Certain strategies are also to be considered to provide low-latency response. For example, time limiting certain data being stored to make room for other more frequently request data.</p><p>I would encourage anybody who wants to implement caching with Redis to check out <a href="https://redislabs.com/redis-best-practices/">this article on Redis best practices.</a></p><p>I wish this article is helpful for anybody out there who wish to be a better backend engineer.</p><p>If you have any feedback, or just want to talk more about backend stuff, please feel free to reach out to me on my <a href="https://www.linkedin.com/in/gilang-ilhami-832b96147/">Linkedin</a>. Would appreciate any kind of discussion.</p><p>Until the next article, cheers!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a2d03b74ab3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to turn your FAQ page into a Chatbot with IBM Watson?]]></title>
            <link>https://medium.com/@gilangrilhami/how-to-turn-your-faq-page-into-a-chatbot-with-ibm-watson-da71fcad6e82?source=rss-27416751a23e------2</link>
            <guid isPermaLink="false">https://medium.com/p/da71fcad6e82</guid>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ibm-watson]]></category>
            <category><![CDATA[business]]></category>
            <dc:creator><![CDATA[Gilang R Ilhami]]></dc:creator>
            <pubDate>Mon, 20 Apr 2020 17:06:36 GMT</pubDate>
            <atom:updated>2020-04-20T17:06:36.164Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Taking advantage of your FAQ page and turn it into a chatbot to develop strong engagement with your audience or customer?</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*JL0iarW81Qm5XrbGDAbDEA.jpeg" /><figcaption>Image Source: engagebay.com</figcaption></figure><p>What is the most important aspect of a business? Customers. Therefore, it only makes sense that we treat them well. According to McKinsey, <a href="https://industryanalysts.com/12417_greatamerica/">70 percent of the customer’s experience depends on how the customer feels they are being handled.</a></p><p>Therefore, it is essential that businesses find ways to enhance customer experiences. One of the ways is the use of chatbots. A chatbot is a piece of software that conducts an auditory or textual conversation.</p><p>There has been a great demand for a chatbot in recent years. In fact, if a company could get it right, 49 percent of consumers would rather use text, talk, or messaging to conduct all their customer service interactions. That’s according to an Aspect-led survey of 1,000 U.S. consumers.</p><p>In this post, I’m going to show you exactly how to turn your FAQ page into a chatbot using services provided by IBM Cloud, so that you can focus on providing better customer experience while saving time and resources.</p><h3><strong>Motivation</strong></h3><p>I’ve just finished my internship as a Data Modeler intern at a <a href="https://www.ibm-jti.com/">company</a> that was own 100% by IBM a few days ago. It was actually my second internship with a company related to IBM (previously I intern at IBM Indonesia).</p><p>During my internship, I was fortunate enough to be given a massive amount of responsibility from my manager to work on interesting projects. One of those projects is how we can create a chatbot for financial institutions.</p><p>Although I had some experience with IBM Cloud and concepts of AI, I had minimal experience in developing a chatbot system.</p><p>Luckily, IBM has provided a service called <a href="https://www.ibm.com/cloud/watson-assistant/">IBM Watson Assistant</a> that would allow any individual to simply create a chatbot. And so, I ended up doing some research and took the IBM Watson Assistant course on cognitiveclass.ai.</p><p>From then on, I was hooked on developing a chatbot system. I realized the potential of chatbot can do for not just enterprises, but also small businesses.</p><p>After doing some research, I found that there is a way that turns an already existing FAQ page into a chatbot by using two more IBM Cloud services: <a href="https://www.ibm.com/cloud/functions">IBM Cloud Function</a> and <a href="https://www.ibm.com/cloud/watson-discovery">IBM Discovery</a>.</p><p>I was familiar with both services. The first one is similar to the Lambda service provided by AWS, while the latter I used in my previous project at IBM Indonesia.</p><p>And now, I want to explain the exact steps on how I build a chatbot using IBM Watson; in this case, I’ll be using IBM Cloud’s FAQ page as an example.</p><h3>Step 1: Create an IBM Cloud account</h3><p>This is as straightforward as it seems. For those who do not have an account, all you need to do is go to <a href="https://www.ibm.com/cloud">https://www.ibm.com/cloud</a> and create an account.</p><p>When you sign up for an IBM Cloud account, you will be automatically assigned to a Lite plan account, which provides a free plan for all the services we are going to use in this blog post.</p><p>If you’d like to know more about IBM Cloud’s pricing plans, please take a look <a href="https://www.ibm.com/cloud/pricing">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2bdKBeEMh7-LpQzg9YCjYA.jpeg" /></figure><p>If you already have an IBM Cloud account, you can move one to the next step.</p><h3>Step 2: Create instances for Watson Assistant and Discovery</h3><p>If you’re already logged in to your IBM Cloud account, you should be looking at the dashboard section of your account</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C28Fi4nY6U6i6I7rqrf45Q.png" /><figcaption>IBM Cloud Dashboard</figcaption></figure><p>From here, what you want to do is click on ‘Create resource’. Then you’ll be redirected to the Catalog page where you enter ‘Watson Assistant’ on the search bar and click the corresponding service on the ‘Services’ section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*jNItID4zNmsSpv3FBY4lhA.gif" /></figure><p>Next, you’ll be asked to create an instance for the Watson Assistant service. There are a couple of settings stuff you could set here such as region, instance name, resource group, and tags.</p><p>In my case, I set the region to Seoul and let the name be set to default (we might rename each instance later on for convenience). After setting up, all you need to do is click the ‘Create’ button, and there’s your instance!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*r-T6fqQBuHwnrJ70zJZVJw.gif" /></figure><p>Note that for Lite plan accounts, you could only create on a free instance for every service. If you want to create a different instance for the same service, you might want to delete the existing one first.</p><p>Moving one, after creating the Watson Assistant service, you can do the same thing with the Discovery service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*ww2gxG7fcjSDx6JXgnGUPA.gif" /></figure><p>If after creating the Discovery service, you get something that looks like “This service might take time to provision. You are being redirected to the resource list now.”, then there’s no need to worry.</p><p>It just means that your service was created but needs some time to be activated.</p><p>Great! We got Watson Assistant and Discovery set up. Before we move on to the next step, we might want to change the name of the instances to that it’ll be easier to navigate each instance.</p><p>What we need to do is navigate to the Resource List, and on the left side of each instance row, that’ll be three dots; click the three dots and click the label ‘Edit name’.</p><p>For the Watson Assistant instance, we want to change the name to “Watson Assistant FAQ Chatbot”, and for Discovery, change the name to “Discovery FAQ Chatbot”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*zIe9mej0DucyLCrd9vJIUA.gif" /></figure><p>Great, we’re all set! Now you might ask, what bout the Cloud Function service? For that one, the setup is a little bit different, and that is what we are going to do in the next step.</p><h3>Step 3: Create action for Cloud Function</h3><p>Unlike the two previous services, we don’t need to create any instances for Cloud Function. Instead, we need to create what is called an ‘action’.</p><p>What is an ‘action’? An action is a small piece of code that can be invoked in response to an event or set to run automatically.</p><p>The benefit of using action is that you limit the amount of time that your code is running, which lowers your costs.</p><p>To create an action, we first need to navigate to the IBM Cloud Functions page. To do that, on the Catalog page, search for “Functions”. Click the “Functions” service on the “Services” section.</p><p>From there, you will be taken to the IBM Cloud Functions page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*60XlRFnfTqL_J68lPXVJag.gif" /></figure><p>After you’ve arrived at the page, you can now create an action. To do this, you need to click on the ‘Action’ label on the left side of the page. Next, click the ‘Create’ button.</p><p>There are options on what you can create on IBM Cloud Functions, but in this case, pick the one that says “Create Action”.</p><p>Pick a name for the action (in my case it’s ‘ibm-watson-faq-chatbot’) and what time of Runtime do you prefer (for me it’s Python 3.7).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*tdjaPylOZE7SSIhwAHLY7Q.gif" /></figure><p>Alright, that’s it for the IBM Cloud Functions for now. We will get back to Cloud Functions again and see how it bridges Watson Assistant and Discovery.</p><h3>Step 3: Setting up Watson Discovery for processing FAQ page</h3><p>Now that we got everything set up, there’s one more thing we need to do before processing the FAQ page; setting up the Discovery service.</p><p>First of all, we want to return to our Watson Discovery console. To do that, we just need to go to the Resource List page and click on the Discovery instance.</p><p>After that, you need to click on ‘Launch Watson Discovery’ to launch the console page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*eeM9LVyFCM40hcVO4iW9jg.gif" /></figure><p>Next, in order to process a document, we need to create what is called a ‘collection in’ Discovery. A collection is basically a set of documents.</p><p>Using a collection allows us to separate the documents we want to analyze into certain contexts; for example, when we want to separate documents for different audiences.</p><p>Going back to our console, we can create a collection by clicking the ‘Upload your own data’ button on the console. After that, you need to give a name (I gave the name ‘FAQ Page’ ) and language support for that collection.</p><p>The last thing to do is just click the ‘Create’ button, and you’ll be taken to the collection page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*ascwTwq00iK5P2A6HnlWIw.gif" /></figure><p>Alright, let’s get to that FAQ Page!</p><h3>Step 4: Processing FAQ page with Watson Discovery</h3><p>This probably a good time to get your FAQ page ready. Note that in order for this to work, you need to have your FAQ page in the form of a PDF file.</p><p>If you have any HTML to PDF converter, that would be convenient. Otherwise, just copy-paste your FAQ page onto a word document and save it as a PDF file, which is what I did with the<a href="https://www.ibm.com/cloud/support"> IBM Cloud Support page</a>.</p><p>Moving on, in this step, there are some sub-steps we need to do for this part. The steps are:</p><ul><li>Upload document</li><li>Annotate questions and answers</li><li>Re-upload document</li><li>Set field for labels and a separate document</li><li>Re-upload document (again)</li></ul><p>If there are some sub-steps that might be confusing, don’t worry. It’ll all come together as we go along. For now, try to keep them all in your head.</p><p>Alright, let’s upload that FAQ page to the collection. Just click on the ‘Upload data to get started’ writing and choose the file you want to upload. Alternatively, you can just drop and drag the file on to the page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*XdnhiWNdEcyeUoVPpkmU9w.gif" /></figure><p>If everything goes right, you should see something like the below image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8_719Mcgw7V1ZdFIXl7sUg.png" /></figure><p>Now let’s move on to the next sub-step, which is annotating the document for questions and answers. Looking back at the page, click on the ‘Configure data’ label (somewhere on the top right corner fo the page).</p><p>You will be taken to the annotation page (the ‘Identity field’ section). On the right side of the page, you can see a section called ‘Field labels’, where you can pick the type of label you need to annotate parts of the document.</p><p>The Discovery service has an OCR that detects text within a document and labels them. All we need to do is pick a color based on the label, and start marking the labeled text.</p><p>When you’re done with a certain page, just click the ‘Submit page’ button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*rlbOAvcIx8t4fJNFDHdY0Q.gif" /></figure><p>Note that as you are annotating the document, the Discovery service’s OCR will try to predict that some sentences fall under a certain label.</p><p>For example, most likely that everything that you annotate as questions has a question mark. After a few pages, Discovery will probably predict that everything that has a question mark is labeled as questions.</p><p>As you are going through more pages to annotate, the less number of text you have to label since the Discovery is helping you along the way.</p><p>When you finished, just clicked the ‘Apply changes to collection button’, and you’ll be asked to re-upload your document.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*j-83wDb7U95aTzEYkuAiRA.gif" /></figure><p>Now we are going to make sure that the Discovery service is focusing only on the labels that used on the document. We need to go to ‘Configure data’ again, and then go to the ‘Manage fields’ section.</p><p>From there, we want to turn off all of the field except the ‘question’ and ‘answer’ field (I also included ‘title’ since I have a title on my document).</p><p>Then, on the right side of the section, there’s an option to ‘improve query results by splitting your documents.’ Choose the ‘question’ field for the splitting.</p><p>Once you’re done, you will be asked to reupload the document again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*HnvZpGbKgThxDFwlhn99yA.gif" /></figure><p>If you check your collection page, there’s a counter that shows you how many documents are in that collection. When we first created the collection, the counter shows ‘1 document(s)’.</p><p>After doing the splitting of the document based on the ‘question’ field, the counter might show something different (in my case it was ’11 document(s)’).</p><p>That’s all regarding the Discovery service. It alone has other powerful functionality for various kinds of applications. I’m not going to dive too deep into this, but if you’d like to know more you can check out the <a href="https://cloud.ibm.com/docs/discovery?topic=discovery-getting-started">documentation</a>.</p><h3>Step 5: Taking note of Discovery credentials</h3><p>The next step should involve setting up the Cloud Functions service. However, before doing that, there is some information we need to take note of. They will be required if we want the chatbot to run smoothly.</p><p>There are 4 things that we need to gather:</p><ul><li>Collection ID</li><li>Environment ID</li><li>API Key</li><li>URL</li></ul><p>For the first two, you go to the collection page and search for a little square on the upper right side of the page that says ‘api’ and click on it. You will be shown the Collection ID and Environment ID.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nQswA_7x7141VBHfM0F1AA.png" /></figure><p>For the API key and the URL, we need to back to the Discovery page where we launch the console from. From there go to the ‘Service credentials’ section.</p><p>There should be auto-generated credentials where you can take the API Key and the URL. If there’s no credential that exists, you can create it by clicking the ‘New credential’ button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*edkZG4plkz96_jw8QMu8-g.png" /></figure><p>Great! Now that we got the credentials, we can start setting up the Cloud Functions.</p><h3>Step 6: Setting up Cloud Functions</h3><p>I want to mention that this part is quite technical and requires some knowledge about using the terminal or command prompt since we’ll be uploading some code in Python by the IBM Cloud CLI.</p><p>So for those who are non-programmers, it might be a good time to let you IT Specialist for this step. Otherwise, you can either higher one or take the time to get to know Python and terminal or command prompt.</p><p>Note that here I’m using IBM Cloud CLI on a Windows Linux Subsystem(WSL). For other installation, please take a look <a href="https://cloud.ibm.com/docs/cli?topic=cloud-cli-getting-started">here</a>.</p><p>Once you’ve installed the command line, there some configurations you need to do on the terminal. To see the steps, you need to go back to the Cloud Functions page, which is shown in Step 3.</p><p>From there, got to the ‘Getting Started’ section and click on ‘CLI’. From there you’ll see exactly the steps you need to do.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*6kTEtu043VPwq2FLdsLAvg.gif" /></figure><p>With that done, we know need to set up some software with the code that will be uploaded with the code to Cloud Functions.</p><p>Create a folder where it is easy for you to navigate to in terminal or command prompt. In that folder, create a new file named ‘__main__.py’, then open that file using a text editor.</p><p>Once the file is open, copy the code below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7b47456b5ad0a2dbd6b9b1289eb5cf1f/href">https://medium.com/media/7b47456b5ad0a2dbd6b9b1289eb5cf1f/href</a></iframe><p>Now go to the folder or directory which contains ‘__main__.py’ via terminal or command prompt, and run the following command</p><p>python3.7 -m pip install ibm-watson -t .</p><p>or if you’re current computer already runs on Python 3.7 by default, you can just use pip or pip3 instead of python3.7 -m pip.</p><p>If everything goes right you should see some folders or directories created alongside ‘__main__.py’. You should see something like below on your text editor</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/330/1*iwJmAeLxicL3xeU5mvp-rQ.png" /></figure><p>Okay, let’s pause a bit and understand what’s going on here.</p><p>Without getting too technical, in order to access the Discovery service on Cloud Functions, we need to use the IBM Watson Software Development Kit (SDK), IBM Cloud provides for the Python programming language.</p><p>The thing about the Cloud Functions does not come with the SDK. So we need to find an away to include it with our code. This is what installing through pip does.</p><p>IBM Watson SDK is also built on top of other libraries, so pip automatically install all other required dependencies.</p><p>Since we already have the script and the required libraries in one folder, all we need to do now is upload them. To do that we need to compress all files and folders into a zip file.</p><p>It’s quite easy doing that via the terminal on WSL. All you need to do is run the following</p><p>zip -r &lt;zip_file_name&gt;.zip .</p><p>The command above will basically compress all the files and folders within a certain folder. Replace &lt;zip_file_name&gt; with the name you want (mine is code.zip).</p><p>To upload the zip file, we’ll use the IBM Cloud CLI which we installed earlier. Still in the same directory, where your zip file was created, run the following command</p><p>ibmcloud fn action update &lt;action_name&gt; &lt;zip_file_name&gt;.zip --kind python:3.7</p><p>The command above updates the already existing action that we created in Step 3. If nothing goes wrong, you should get a message on the terminal that looks like this</p><p>ok: updated action &lt;action_name&gt;</p><p>and the action page should look something like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fKUaQu66pVRALFJK-ktt0Q.png" /></figure><p>As it says on the page, we can’t really see our code anymore. However, we can still test it if it works. Before doing that, we need to add some information to the action.</p><p>Remember those credentials I ask you to take note of? This is where we’re going to use them. On the right side of the page, click on ‘Parameters’. From there, the credentials will be the parameters.</p><p>You will need to add the following names as the parameter names:</p><ul><li>iam_apikey</li><li>url</li><li>environment_id</li><li>collection_id</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*SXplk8C2CCWKW9JK_bqY6Q.gif" /></figure><p>After you provide the parameter names, you need to provide the value for each parameter. Note that you need to include a<strong> double quotation mark </strong>for the parameter values. Once that done, make sure you click ‘Save’.</p><p>Now that we got all the parameters setup, we can try to test if the action actually works before integrating it with Watson Assistant.</p><p>Since the parameters are in place, we need to include something else. Going back to the ‘Code’ section, click on the ‘Invoke with parameters’. You’ll be prompted with a pop-up box.</p><p>In the pop-up, you want to enter the following:</p><p>{&quot;input&quot;: &quot;How do I reset my password?&quot;}</p><p>If you decide to use your own FAQ page, then you can change the ‘How do I reset my password?’ to question that is relevant to your FAQ page.</p><p>Then, you want to close the pop-up by clicking ‘Apply’. When you’re ready to test the function, just click on ‘Invoke’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*Av6IjeEQrPPBV1uxEIyuzA.gif" /></figure><p>If everything went well, you should see something like below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JOko97BQNiGPBBSbCXlL0Q.png" /></figure><p>Note that the result will vary according to your FAQ page or what you decide to ask other than the question provided above.</p><p>Aright, our action appears to be working just fine. One more thing we need to do and that is to make sure we can send values to the action, and to do that we need to wrap the action in the form of a Web Action.</p><p>Basically, a Web Action is a URL that can be used to trigger the action from any web app. This is really convenient for integrating the action with the Watson Assistant later on.</p><p>To do that, we need to go to the ‘Endpoints’ section. Then under the ‘Web Action’ section, select the checkbox with the label ‘Enable as Web Action’. Don’t forget to click ‘Save’ when you’re done.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*lx7wn2hM69CiiFd9QDSzCw.gif" /></figure><p>Now after enabling the checkbox, you want to note of the URL provided below the checkboxes. It will be important in the next step, which is setting the IBM Watson Assistant.</p><h3>Step 7: Configuring Watson Assistant</h3><p>In this part, we are going to make some configuration to Watson Assitant instance that you’ve already created.</p><p>Let’s go back to the Resource page and look for the Watson Assistant instance that you created earlier. Once you’re in the instance’s page, you want to launch the console by clicking ‘Launch Watson Assistant’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*vg9vs72u4KoqHZ-U1BgbKA.gif" /></figure><p>After opening the console, you’ll arrive at the ‘Assistants’ section of the page, where you’ll see your first assistant with the label ‘My first assistant’.</p><p>An ‘assistant’ is the cognitive bot you can customize for your business purposes and deploy to multiple channels.</p><p>An ‘assistant’ needs to be associated with a certain ‘skill’, which is basically the part that will process the question being asked to the bot. Since we have our assistant, all we need now is the skill.</p><p>To create a skill, we need to go to the ‘Skills’ section, which is accessible on the left side of the console page. When you’re on the ‘Skill’ section, you will see an already created skill called ‘My first skill’.</p><p>You can create your own skill by clicking the ‘Create skill’ button if you want to. But for now, I prefer to work on what we already have. Let’s start by renaming the skill so it’s more consistent with what we’re working on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*FAYWTEVOfVkIBQLtXsKJnQ.gif" /></figure><p>When you’re done renaming, click on the skill and you’ll be redirected to a skill’s page. On that page, there is much stuff that you can do to maximize the potential of Watson Assistant.</p><p>However, I’m not planning to go deep on terms, so we’re are only going to work on what we need right now: intents and dialog.</p><p>An intent represents the purposer of a user’s input; in this case, it’s the FAQ. While a dialog is a branching conversation flow, which determines how your application reacts when it recognizes the specified intents and entities.</p><p>Let’s start off with intents. To create an intent, click on the ‘Create intent’ button. You’ll be asked to give a name for the intent. For now, let’s just name it ‘faq’.</p><p>Next, you’ll need to provide some examples of the question from your FAQ. Watson Assistant will learn from these examples so that it can detect if a user is classified for a certain intent.</p><p>So for example in my case, I have the question ‘How do I reset my password?’ from the IBM Cloud FAQ page. That means I might want to add some similar sentences like ‘reset password’ or ‘i forgot my password’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*KAIAHUD2i0N71GgrbrnMpQ.gif" /></figure><p>The service automatically saves the examples you input to the intent. If you’re done, then just the blue arrow beside the name of your intent on the left side of the page.</p><p>Now that our intent is ready, there’s something we need to do before going to the dialog, and that is to connect the Watson Assistant service to the Cloud Functions action.</p><p>We can do that by going to the ‘Options’ section of the page and look for the URL input box under ‘Webhook setup’. Within that box, paste the URL of the action that we took note in the previous step.</p><p>Then, you want to add ‘.json’ to the end of that URL. This is important if you want to the chatbot to work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*AXHsEeMw0AarHr5fVYgYuQ.gif" /></figure><p>Once that done, we’re ready to do something about our dialog. The dialog is where we want to take the user input and send it via the webhook so that Cloud Functions can process it and get the desired response from Discovery.</p><p>First, we want to navigate to the ‘Dialog’ section. Then, you’ll see two nodes name ‘Welcome’ and ‘Anything else’. These are the nodes that will greet your user and handle inputs which intent is not clear.</p><p>What we want to do here is add a node below the ‘Welcome’ node. You can do that by clicking the three dots on the ‘Welcome’ node and select ‘Add node below’.</p><p>A new node will be added, and you’ll be asked to give that node a name. In my case, I named the node ‘FAQ’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*439rQ45y1S-feTrVuto-LQ.gif" /></figure><p>Once you determine the name, we need to enabled Webhook for the node. Select ‘Customize’ besides where you give the name for the node. Then you wan to look for the ‘Webhooks’ toggle and switching it to ‘On’.</p><p>Don’t forget to click ‘Apply’ when you’re done to make sure the change is saved.</p><p>Next, we need to make sure that the node can take the input when an intent is detected. Under the ‘If assistant recognizes’ label, select the intent that you created previously.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*5J2463fSGOAlGRP2fRAdkg.gif" /></figure><p>From there, a new section will appear called ‘Then callout to webhook’. This is the part that will actually capture the user’s input.</p><p>Let’s start with the ‘Parameter’ part. For the key, we just set to ‘input’, then for the value, we set to &quot;&lt;?input.text?&gt;&quot;.</p><p>The second part, which is the ‘Return variable’ field, we set to ‘$webhook_result_1’. Don’t forget the $ sign. Next is the ‘Assistant response’</p><p>Basically, the ‘Assitant response’ section will help you to manage the responses for a certain situation. You can also do this node by node in the Dialog tree, but we’re not going to cover that in this blog post.</p><p>Moving on, you’ll see two already fields under the ‘If assistant recognizes’ label. What we want to fill is under the ‘Respond with’ label.</p><p>For ‘$webhook_result_1’, we want to respond with ‘$webhook_result_1.message’, which will get the message of the response from the URL that connects to the Cloud Functions.</p><p>For ‘anything else’, you can put whatever you wish to respond with if a certain input is not included within your intent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*eadfcg8tXYPhI14ZsXs0tg.gif" /></figure><p>If that’s all set and done, we can now have a try on our Watson Assistant-powered chatbot. On the top right of the page, you’ll see the ‘Try it’ button, which will pop-up the chatbot on the right side of the page.</p><p>Try inputting something relevant to your FAQ. For example, I input, “How do I reset my password”. The result should be the same as the one you have on Cloud Function when invoking the action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*1KyJbAZiO2J8hKI4ClEZOg.gif" /></figure><p>Wow, look at that! It works like a charm!</p><p>At last, all the pieces have come together to create this FAQ chatbot. But wait! It works on the Watson Assistant console, but how can we share it with other people to see if it works fine?</p><p>And I’m pretty sure that we don’t want other people to sign in to our IBM Cloud account to try out the chatbot of course.</p><p>For that reason, the next and final step involves creating a link that you can share with other people to try out your bot.</p><h3>Step 8: Creating your Preview Link</h3><p>Usually in developing any product, you want to test it first before deploying it to the end-user. In this step, we are going to create a Preview Link.</p><p>A Preview Link it’s just a URL that when you click on it, it creates a simple interface for your chatbot so that other people can use it.</p><p>To start off, you need to go to the ‘Assistant’ section of the Watson Assistant console. Remember that already created assistant called ‘My first assistant’? Click on it.</p><p>The assistant is already connected to the skill we just developed. All you need to do is click ‘Add integration’, select the ‘Preview Link’ integration, and click ‘Create’. A link will be generated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*GoGYpOFFSDUWOd_62lBomg.gif" /></figure><p>Click on the link, and you’ll be taken to a web page where you can interact with your bot. Try to ask the same question as to the one you ask when trying out on the Watson Assistant console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*QTso5cxhnU_kibltQ6aeGg.gif" /></figure><p>Fantastic!</p><p>There you go, a working chatbot for your FAQ. So next time and end-user have a question that is commonly asked, the chatbot can answers questions for him or her.</p><h3>Conclusion</h3><p>We just converted our already existing FAQ page into a Watson-Assistant powered chatbot. The aim is to provide a better customer or user experience for those who have questions regarding your business or organization.</p><p>However, there are some improvements that we could do to make the chatbot much more interactive. One of them is by making the intents more specific instead of just ‘faq’.</p><p>For example, in my case, it’s making sure that when an end-user asks ‘How do I reset my password?’, it should be identified as a ‘#reset_password’ intent. This allows for better management of each question later on.</p><p>Another improvement is probably using ‘Entity’ in Watson Assistant. This allows for the detection of certain entities in an input.</p><p>For example, for an eCommerce website operating in the U.S., if a customer asks something like ‘How much does it cost to ship to Indonesia?”, you’ll want to identify ‘Indonesia’ as a country’s name to give an appropriate response.</p><p>All of the improvements I mentioned above are only on the side of Watson Assistant. There are also various ways we can improve the chatbot from the side of Cloud Functions and Discovery also.</p><p>I encourage you to have a try at the <a href="https://cognitiveclass.ai/badges/build-your-own-chatbot">Build your own chatbot course provided at cognitiveclass.ai</a>. It provides a deeper explanation of how to maximize the potential of Watson Assistant.</p><p>Also, I suggest looking at the documentation of Cloud Functions and Discovery and take the time to learn other features that these services provide.</p><p>That’s all from me right now. I hope this blog post was helpful and brings value to whatever you are doing. If you have any feedback, please let me know in the comment section or reach me out on Linkedin.</p><p>Until next time, cheers!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da71fcad6e82" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting Micronutrients using Neural Networks and Random Forest (Part 3)]]></title>
            <link>https://medium.com/data-science/predicting-micronutrients-using-neural-networks-and-random-forest-part-3-337ed6c5946a?source=rss-27416751a23e------2</link>
            <guid isPermaLink="false">https://medium.com/p/337ed6c5946a</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Gilang R Ilhami]]></dc:creator>
            <pubDate>Tue, 24 Sep 2019 08:29:24 GMT</pubDate>
            <atom:updated>2019-09-24T13:32:25.407Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://towardsdatascience.com/predicting-micronutrients-using-neural-networks-and-random-forest-part-2-1bd85497fdf1?source=your_stories_page---------------------------">UNICEF wants you to help them to predict important nutritions within foods using the power of machine learning.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bhHCu9Z6BVDy-6gn" /><figcaption>Photo by <a href="https://unsplash.com/@zoltantasi?utm_source=medium&amp;utm_medium=referral">Zoltan Tasi</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Greetings! Welcome to part 3 of the “Predicting Micronutrients using Neural Networks and Random Forest” blog series.</p><p>In the previous blog post, we took a little bit of look at how our data look like by doing some visualization techniques to our data.</p><p>Then, we tried to use two of the most common supervised machine learning algorithms: Linear Regression and Random Forest. However, it seems that we did not get the result we wanted.</p><p>If you are new to deep learning, note that although the content contains an introduction to deep learning, this is not a blog post about how deep learning works beautifully.</p><p>On the contrary, this blog post is to show that deep learning is <strong>not</strong> a silver bullet when trying to build a machine learning model or solving a data problem.</p><p>If you wish to see an example case of when deep learning does work, I’ll include some articles by the end of this blog post.</p><p>By now, we have only a couple of days to prepare our proposed model to our data science team in the UNICEF. Of course, with the kind of reputation the organization has, we don’t want to disappoint them, right?</p><p>Great, now that we’re on the same side, let’s get started.</p><h3>Overview<strong>:</strong></h3><p>This blog post contains the following:</p><ul><li>Introduction to Deep Learning</li><li>How does Deep Neural Network “learns”?</li><li>Coding a Deep Neural Network using Keras</li><li>The Reflection Part</li></ul><h3>Introduction to Deep Learning</h3><p>Will all the hype about deep learning, I assume that most of you who are reading this has some idea about what it is in general.</p><p>In short, deep learning is one of the various machine learning methods that mainly utilize stacked perceptrons to learn from data.</p><p>These stacked <strong>perceptrons</strong> are called neural networks, which was inspired by the complex biological structure of the human brain. Generally, a neural network has <strong>three main layers</strong>: <strong>input</strong>, <strong>hidden</strong> and <strong>output</strong>.</p><p>Each unit in the hidden layer is called a <strong>hidden unit</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/1*BMdnuQUqGEb_FSEmYIxw5g.jpeg" /><figcaption>A general structure of a neural network has an input, hidden, and output layer.</figcaption></figure><p>Now you might be thinking, “okay, but I still don’t get what so DEEP about this.” Well, you are absolutely right. However, in most practices, more hidden layers and hidden units are stacked in the architecture.</p><p>Some complicated networks used from hundreds of thousands or even millions of hidden layers and units. As the number of hidden layers and units grow, most experts will refer to the network as a <strong>Deep Neural Network</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*29y0xq-ScSQ6l1KEA-q3VQ.jpeg" /><figcaption>A Deep Neural Network</figcaption></figure><p>Another question that might pop into your head is “so what’s so special about this deep neural network?” One of the special parts of deep learning is the “learning” part of it.</p><p>We’ve covered the “deep” part, now let’s talk about the second part.</p><h3>How does a Deep Neural Network “learns”?</h3><p>The goal of various supervised machine learning algorithms is fundamentally similar: minimizing the error between the predicted value and the ground truth value.</p><p>Before the popularity of deep learning, most of machine learning processes attempt to do this by way of<strong> </strong><a href="https://machinelearningmastery.com/discover-feature-engineering-how-to-engineer-features-and-how-to-get-good-at-it/"><strong>feature engineering</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*5nl600Dn7qPhSxpW8dXnow.png" /></figure><p>Not going too much into the details, feature engineering is just a way to create a feature to make machine learning works. Features, in this case, means the variables of the data.</p><p>However, this is not the case in deep learning algorithms. Instead, an <a href="https://en.wikipedia.org/wiki/Mathematical_optimization#Computational_optimization_techniques"><strong>optimizations algorithm</strong></a> is used to find the best performing model. One of the most popular ones is called <a href="https://en.wikipedia.org/wiki/Gradient_descent"><strong>gradient descent</strong></a>.</p><p>Basically, gradient descent’s main goal is to find the local minimum of a loss function.</p><p>It is able to do this by taking the gradient of the error with respect to all the weights in the hidden layers through a process called <a href="https://en.wikipedia.org/wiki/Backpropagation"><strong>backpropagation</strong></a>.</p><p>Then, you want to take the negative step towards the gradient. We’ll try to use an analogy to understand that better.</p><p>We’re going not going to cover the math here, but visually it looks something like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*yHOan2LU2_0kpPUgezNEPQ.gif" /><figcaption>What a gradient descent process looks like in three dimensions.</figcaption></figure><p>Alternatively, we can use an analogy of someone blindfolded going down a hill. Would it be better just to take one big leap down to the bottom, or take some small step on some safe part of the hill?</p><p>If someone still wants to live another day, then the latter would be much better of course. Basically what we want to do is to take a small incremental step that will take us to the bottom of the hill.</p><p>These small steps are what “taking a negative step toward the gradients” means, but this time, it’s from the “error function” hill.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/245/1*pNa6RZiDd69oj1e-ybo-Xw.gif" /><figcaption>optimization using gradient descent be like…</figcaption></figure><p>Note that gradient descent has its variants. Choosing which to use with which deep learning algorithm is part of the art of training a deep neural network.</p><p>Specifically, we are using one of the most common variants called <strong>Adam</strong> optimizer. Again, we are not going to discuss that here, but feel free to refer to the <a href="https://arxiv.org/abs/1412.6980">original paper</a>.</p><p>Okay, now’s for the moment we’ve been waiting for, the coding part with Keras!</p><h3>Coding a Deep Neural Network using Keras</h3><p>Keras is an open-source neural network library written in python. It runs on top of various existing deep learning libraries such as Tensorflow, MXNet, and Microsoft Cognitive Toolkit.</p><p>Also, we will be using another tool called <a href="https://www.wandb.com/">Weights and Biases</a> which is used for tracking visualizing a neural network’s training progress. For now, we will focus our attention on tracking the training accuracy.</p><p>Before we start coding, note that we already have the dataset prepared. It has already been cleansed, scaled, and split into training and testing set in the first part of this blog series.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0e030fd7936d9391e02b93956042154d/href">https://medium.com/media/0e030fd7936d9391e02b93956042154d/href</a></iframe><p>With the data being ready, we can start coding the neural network. I assumed that the readers are familiar with Keras. If not, then the <a href="https://keras.io/">documentation</a> can be really helpful.</p><p>Let’s import all the required stuff from the Keras library.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5275d4cfc1011e46d4118898f57ff68c/href">https://medium.com/media/5275d4cfc1011e46d4118898f57ff68c/href</a></iframe><p>Here, we are using Sequential to stack the layers. There are four types of layer object that we will be using: Activation, Dense, BatchNormalization, Dropout.</p><p>Just to point out, Dense means the fully connected layer that we talked about before. The Activation, Dense, and BatchNormalization layers are quite straightforward based on their name.</p><p>If you are not familiar with the last two, then I highly encourage you to do a little bit of reading on them since they have been commonly used in practice.</p><p>As mentioned before, Adam is chosen as the optimizer. Also, we have mean_squared_error as the loss function since it’s the one we use for Random Forest and Linear Regression.</p><p>Okay, we got what we need from Keras, but we also need something to track the performance of our model as it’s training. This way, we can see if the accuracy stops improving after some period of time.</p><p>There are many choices open source tools that we can use, but the one I prefer is called <a href="https://www.wandb.com/">Weight &amp; Biases</a> (wandb). It provides support for Keras, Tensorflow, and Pytorch.</p><p>To use it, we need to sign up at the official website, download the SDK for Python, and do a little bit of setup.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/18a7c09d0ab50f06a7be0f8095b5b2b1/href">https://medium.com/media/18a7c09d0ab50f06a7be0f8095b5b2b1/href</a></iframe><p>wandb.init() initialized the chart that will display the chart. config allows us to set the parameters (batch size, learning rate, etc.) that will be sent to the wandb server for the axis of the charts; for example like epochs.</p><p>Now we need to set the parameters to config. Feel free to do some modification to parameters.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/34c40e1ef91cf01d5e3e2199519dc95b/href">https://medium.com/media/34c40e1ef91cf01d5e3e2199519dc95b/href</a></iframe><p>Great! Let’s get started with the neural network architecture. For starter, let’s just build a 2 hidden layers neural network. The first layer will have 128 hidden units and the second layer will have 64.</p><p>We will use a ReLU activation after each layer. Then we will do batch normalization, which will be followed by a dropout. The final layer has 6 output units.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/03a25e6acf6e34a743b8988b813f9e1d/href">https://medium.com/media/03a25e6acf6e34a743b8988b813f9e1d/href</a></iframe><p>We got our architecture, now we just need to compile our optimizer and loss function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/16998ae166484c071a4c642a05fc471e/href">https://medium.com/media/16998ae166484c071a4c642a05fc471e/href</a></iframe><p>And now, it’s time for TRAINING!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4bca8af59e8880243dce193b6c8b42f5/href">https://medium.com/media/4bca8af59e8880243dce193b6c8b42f5/href</a></iframe><p>As we can see here, we have the callbacks argument which takes a list as its input. We can have more than one performance tracker, but in this case, we will only use WandbCallback in the list.</p><p>To see the performance of the model, we can directly check our dashboard on the wandb’s website.</p><p>Depending on your RAM power, it might take a while to train the model. However, with a tracker, we can stop anytime we want if our performance seems to not improve at all.</p><p>Let’s see how we doing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/824/1*uQUdTkC4bN2Z8a3F5f8nkw.png" /></figure><p>Through the whole training, it seems that our model could only reach ~30% accuracy. Specifically, around the 20th epoch, the accuracy stops improving. Looks like we have to change the architecture a little bit.</p><p>Let’s try using less hidden units in both layers. Note that we need to rerun wandb.init() and config = wandb.config to start a new chart.</p><p>If not, then wandb assumes that we are continuing the previous training session when we are actually starting a new one.</p><p>That being said, let’s set up the architecture.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2fa0f659e6211942be9fb7ed7eb341c0/href">https://medium.com/media/2fa0f659e6211942be9fb7ed7eb341c0/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*HZf0JUhVwihJ9BTLFlJaEA.png" /></figure><p>And the accuracy is still around 30%. Hmm, I wonder what’s going on here? But there seem to be drastic changes in the training in the first 40 epochs.</p><p>There’s a lot of assumption here, but maybe decreasing the number of hidden units were not a really good idea. So what happens if we add more hidden units than the first architecture?</p><p>Let’s find out.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0afc73313e384baf7f113d4fb2e3e05c/href">https://medium.com/media/0afc73313e384baf7f113d4fb2e3e05c/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/813/1*Y_JDKQ63SfUDXV-Fg2umWg.png" /></figure><p>The accuracy is still ~30%? (and the confusion starts).</p><p>Well, all we have done so far is changing the number of hidden units. Maybe it’s time to spice things up to do something with the layers. Let’s do exactly just that! Instead of 2 layers, we’re going to do 5.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0fb224ba49bbd55974b42ca1c5e6b068/href">https://medium.com/media/0fb224ba49bbd55974b42ca1c5e6b068/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/849/1*5tVDun3d0E-HgJgLv0Bwmw.png" /></figure><p>Okay, now I’m pretty sure that we are not heading to the right direction here.</p><p>In this kind of situation, it’s really tempting to conclude that maybe that deep learning is not a viable solution in this case and move one.</p><p>However, it is always good practice to take a step back and take the time to understand <em>why </em>it’s not a viable solution.</p><p>Understanding why a solution did not work for a certain problem is the learning part of every failure. That way, when asked why something did not work, we will be able to explain why</p><p>So let’s move on the next part, the Reflection part.</p><h3>The Reflection Part</h3><p>First of all, let’s think more a little bit more about our data. Our dataset has around 8,000 data points.</p><p>In the eye of deep learning practitioner, that’s quite a small number of data points. Usually, deep neural networks require millions, even tens of millions of data points.</p><p>Also, we use many variables in our dataset that might act as noise instead of the actual signal. Maybe we don’t need to use all of them.</p><p>Maybe if we do a little bit more visualization, we can understand our data better and use a better approach other than deep learning. And that is exactly what we will be doing in the fourth and last part of this blog series.</p><p>Despite the whole deep learning hype, we now understand that deep learning is not always the answer; at least not in this case. There are other ways to solve the data problem.</p><p>By the way, as promised, here are some articles where deep learning works really well (with Keras of course):</p><ul><li><a href="https://machinelearningmastery.com/tutorial-first-neural-network-python-keras/">Your First Deep Learning Project in Python with Keras Step-By-Step</a></li><li><a href="https://www.kaggle.com/ngyptr/lstm-sentiment-analysis-keras">LSTM Sentiment Analysis | Keras</a></li><li><a href="https://towardsdatascience.com/image-recognition-with-keras-convolutional-neural-networks-e2af10a10114">Image Recognition with Keras: Convolutional Neural Networks</a></li></ul><p>The above articles are great examples that will show you the real power of deep neural networks when it is fed with a sufficient amount of data.</p><p>Thank you so much if you’ve read this blog post until the end. If you have further questions or want to give some feedback, please let me know in the comment, or message me on <a href="https://www.linkedin.com/in/gilang-ilhami-832b96147/">my Linkedin</a>.</p><p>I hope to see you in the next part. Till then, cheers!</p><p>About Me:</p><p><em>An Indonesian student currently studying in Korea and has a deep interest in the world of AI. Back-end and machine learning developer most of the time. Believe in continuous learning. Want to give back to the AI that has contributed so much in my pursuit of knowledge.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=337ed6c5946a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/predicting-micronutrients-using-neural-networks-and-random-forest-part-3-337ed6c5946a">Predicting Micronutrients using Neural Networks and Random Forest (Part 3)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting Micronutrients using Neural Networks and Random Forest (Part 2)]]></title>
            <link>https://medium.com/data-science/predicting-micronutrients-using-neural-networks-and-random-forest-part-2-1bd85497fdf1?source=rss-27416751a23e------2</link>
            <guid isPermaLink="false">https://medium.com/p/1bd85497fdf1</guid>
            <category><![CDATA[scikit-learn]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[random-forest]]></category>
            <category><![CDATA[decision-tree]]></category>
            <dc:creator><![CDATA[Gilang R Ilhami]]></dc:creator>
            <pubDate>Mon, 24 Jun 2019 17:35:55 GMT</pubDate>
            <atom:updated>2019-06-24T20:10:03.316Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>UNICEF wants you to help them to predict important nutritions within foods using the power of machine learning.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bXbE12_1fm9reBrv" /><figcaption>Photo by <a href="https://unsplash.com/@djulien?utm_source=medium&amp;utm_medium=referral">Julien R</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Welcome back! Glad you can join me in part 2 of this series of “Predicting Micronutrients using Neural Networks and Random Forest.”</p><p>In the previous blog post, I mentioned that UNICEF has invited us to join their data science team to predict micronutrients composition in certain foods by using the power of machine learning.</p><p>Also, we went through the process data cleansing with a dataset from the US Department of Agriculture.</p><p>Now, we are going to one of the most exciting parts of almost every data science project, building the machine learning model.</p><p>As you can probably tell by the title itself, we are going to use Random Forest and Neural Networks to build the model. But for part 2, we are not advancing to neural networks just yet.</p><p>In fact, it is recommended when you are working on a data science or machine learning project to start using simple models and move your way to a more complex model.</p><p>Okay, our colleagues in the UNICEF data science team are waiting for us in the meeting room, so let’s get going and build that machine learning model.</p><h3>Overview</h3><p>This blog post contains the following:</p><ul><li>Introduction</li><li>A brief introduction to machine learning</li><li>Multivariate Linear Regression: Linear Regression and Multivariate Regression</li><li>Random Forest: Decision Trees and Bootstrapping</li></ul><h3>A Brief Introduction to Machine Learning</h3><p>In the first part of the series, I mentioned some sources that act as prerequisites in order for anybody to be able to have a better understanding of what I am talking about in this series.</p><p>One of them was about “Introduction to Machine Learning.” However, I decided that we should address a minor introduction just to keep the flow nice for this blog post.</p><p>But if you wish to gain a better understanding, I highly recommend that you read <a href="https://towardsdatascience.com/introduction-to-machine-learning-db7c668822c4">this blog post</a> by <a href="https://towardsdatascience.com/@vihar.kurama">Vihar Kurama</a>.</p><h4>What is Machine Learning?</h4><p>We will have to be a little bit more critical about this question since there have been many attempts to define what is and what is not machine learning. Here is a couple of examples.</p><blockquote>Machine Learning: Field of study that gives computers the ability to learn without being explicitly programmed.</blockquote><blockquote>Arthur Samuel (1959)</blockquote><p>In the 1950s, Arthur Samuel was able to create a program that lets the computer learns to play checkers by playing against itself. In the end, even Samul himself lost to his own program in checkers.</p><p>The second definition is probably the most popular and well-known among the machine learning community.</p><blockquote>Well posed Learning Problem: A computer program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P, improves with experience E.</blockquote><blockquote>Tom Mitchell (1998)</blockquote><p>Let’s use our current mission to predict micronutrients within certain foods as an example.</p><p>In this case, E is the records of the amount of nutrient composition within each food, T is the task of predicting the micronutrients based on other nutrients, and P is how far are the predicted values from the ground truth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*5h4Ihn8oP5DySv27i0pgcw.png" /><figcaption>E: the records of the amount of nutrient composition within each food. T: the task of predicting the micronutrients based on other nutrients. P: how far are the predicted values from the ground truth.</figcaption></figure><p>In common practice, there are three different kinds of machine learning algorithms when it comes to tasks:</p><ul><li><strong>Supervised Learning</strong>. This is what we are actually doing when we have input variables and output variables for training.</li><li><strong>Unsupervised Learning</strong>. It is basically the same as supervised learning, but there are no output variables.</li><li><strong>Reinforcement Learning</strong>. This type of machine learning handles a task where there are no input or output variables. Instead, a computer program will interact with a dynamic environment to achieve a certain goal through trials and errors.</li></ul><p>When it comes to supervised learning, we actually trying to learn a function that maps the input variables to the output variables.</p><p>The form of this function is unknown and our job is to evaluate various machine learning algorithms and observe which one is better in approximating the underlying function.</p><p>Another way that we can think of a function is like a magical box that can never be opened. But the box can take input and make a prediction based on that input.</p><p>For example, if you put in an email from a suspiciously unknown email into that box, it can correctly tell you 100% the of the time if it is a spam or just one of your friend who’ve just create a new email.</p><p>Since we can’t see what is inside the box, then maybe we can create something similar to that box. And the form of this something doesn’t have to be a box. It can be a ball, prism, or any other shape.</p><p><em>(Let’s say we ended up with a ball)</em></p><p>The most important thing is that our ball can make an approximation that is as good as the prediction of the magical box. This means that it can predict 99% correctly instead of 100%.</p><h3>Multivariate Linear Regression</h3><p>As I mentioned before, when dealing with a data science or machine learning project, it is recommended that we start with a simpler model.</p><p>Here, we are going to cover Multivariate Linear Regression and implement it using Scikit-learn.</p><h4>Linear Regression</h4><p>Linear Regression is one of the most widely used machine learning algorithms for predictive modeling. It also commonly implemented in a supervised learning scenario.</p><p>It is called a linear model because it infers a linear relationship between our input and output variable. Basically, it gives us a linear equation that looks something like the image below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*J24-pcoWneExmxMPYUNBHw.png" /></figure><p><em>Note — input and output variables are usually referred to as independent and dependent variable respectively when we ware talking about Linear Regression. I’m going to use the terms interchangeably. This also goes for coefficients, weights, and parameters.</em></p><p>In the equation above we have one <strong>dependent variable</strong>, y, and one <strong>independent variable</strong>, x. Also, we have two betas that are the <strong>coefficients </strong>of the linear equation.</p><p>The Epsilon is the <strong>error term</strong> due to random sampling.</p><p>The first coefficient (beta null) is the intercept term, while the second one (beta one) is the weight that is assigned to the variable. The weight signifies the importance of the variable.</p><p>This means that if beta one is 2.5, then the unit of our variable (x one) will also increase by 2.5.</p><p>Note that the equation above has only one variable with one coefficient. When we have more than one, let’s say <em>n </em>variables, then we need to include <em>n </em>coefficients in the equation.</p><p>When we are building a predictive model using linear regression with multiple variables, it is usually more known as <strong>multivariable linear regression</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/1*76VX03TxCDOk06P4MBOdIQ.png" /><figcaption>Based on the preprocessing step that we’ve done in the previous blog post, we have 43 input variables for training. That means we will have 44 coefficients (43 for each variable plus the intercept term).</figcaption></figure><p>The goal of the linear model is to find the parameters that best learn the data by minimizing the residual sum of squares (RSS).</p><p>RSS is the total of the squared differences between the true value, y, and the predicted value by the model, ŷ (pronounced y-hat), which indicates an estimation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*1Tq7wnhF88I1VRfQ3PErqQ.png" /></figure><p>We are not going to cover the details of RSS in this blog post. For those who are interested in the detailed explanation and derivation, you can check it out <a href="http://www.stat.cmu.edu/~cshalizi/mreg/15/lectures/13/lecture-13.pdf">here</a>.</p><p>Okay, let’s hold here for a bit. Can you spot something’s missing from our discussion here? Something related to our proposed task in particular?</p><p>If you were thinking something like “but we have more then one output labels” or “why is there only one <em>y</em> in the linear equation?” then you deserve a promotion my friend (just kidding, maybe a candy would be enough?)</p><p>When we have to predict values for one output variable, it’s called <strong>univariate regression.</strong> In fact, this pretty common in most linear regression tutorials that you can find on the web.</p><p>But what to do when we have more than one output variables?</p><h4>Multivariate Regression</h4><p>In the case when we have to make predictions for multiple output variables, it is called a <strong>multivariate regression</strong>, which we will briefly need to cover first.</p><p>Remember the linear equation we talked about just a moment ago? We can actually further extend that formula to look like something like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*jGFA7NZ3_c_TlRm6GIsadA.png" /></figure><p>Basically, we now have more dependent variables, parameters, and error terms. Our goal is similar to univariate regression. Only this time, we have to find the parameters that minimize the RSS across all the output variables.</p><p>Also, it means that instead of having 44 parameters to estimate, now we have 264 parameters (44x6).</p><p>Wow, that’s a lot, right? Not really. This pretty common in most machine learning practice. Especially when we are talking about deep learning, where we can have parameters up to the thousands.</p><p>We’re not going to cover RSS in the context of the multivariate regression. However, if you want to know more in details about multivariate regression, please refer to this <a href="https://maitra.public.iastate.edu/stat501/lectures/MultivariateRegression.pdf">slide</a>.</p><p>The slide can be too technical for those who are not familiar with statistics. But If you are interested, give it a go!</p><p>Next, we are going to use Scikit-learn to implement multivariate linear regression. Don’t forget to take a break if you need to.</p><h4>It’s time to code!</h4><p>Alright, we are finally going to start coding. Make sure that you have Scikit-learn installed. Also, we are going to do data visualization in this section, so make sure you also have Matlotlib and Seaborn.</p><p>Before we build the model, let’s use some visualization to get a better understanding of our data.</p><p>Let’s analyze the strength of the linear relationship between every two variables. This is also known as the <a href="https://en.wikipedia.org/wiki/Correlation_and_dependence"><strong>correlation</strong></a> between every two variables.</p><p>We can build a correlation matrix using the corr attribute provided by pandas.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/13ffd62bee0be06fd8c5916139e8f6a6/href">https://medium.com/media/13ffd62bee0be06fd8c5916139e8f6a6/href</a></iframe><p>You will see something like this as the output.</p><p>Ah, look at all those numbers! I think we should make this in a way so that our team member can understand the correlation better.</p><p>Let’s use Seaborn to make visualization of the correlation matrix. There’s a snippet of code that we can use from <a href="https://seaborn.pydata.org/examples/many_pairwise_correlations.html">here</a> to quickly make a heatmap of the correlation matrix.</p><p>With some minor adjustments, we can run the following lines of code…</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9663a16e91903bfa4018968be87e18dd/href">https://medium.com/media/9663a16e91903bfa4018968be87e18dd/href</a></iframe><p>…and something like this below should appear.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QUYgL2mR4rUc0sR3n3lSUQ.png" /></figure><p>Alright, that looks much better. Let’s take a look.</p><p>We can see that some of the nutrients are either positively correlated or show no correlation with other nutrients, while only a couple is negatively correlated.</p><p>If you want to know the difference between positive correlation, negative correlation, and no correlation, please take a look at <a href="https://www.emathzone.com/tutorials/basic-statistics/positive-and-negative-correlation.html">here</a>.</p><p>Although the correlation matrix displays visually about the linear relationship between every two variables, it does not explain if two variables <strong>cause</strong> each other to go up or down.</p><p>In fact, a lot of time you here will people who are involved heavily statistics (including machine learning) say something like <a href="https://towardsdatascience.com/why-correlation-does-not-imply-causation-5b99790df07e">“correlation does not cause causation” or “correlation does not imply causation”</a>.</p><p>For that reason, it is a good idea to plot our data in order to get a better understanding of the relationship between each variable.</p><p>Let’s do just that with Matplotlib!</p><p>Looks like Carbohydrate and Iron shows a positive correlation. We can plot the data using the following line of code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/07f236f14afec5ef7ff3f5f58d349329/href">https://medium.com/media/07f236f14afec5ef7ff3f5f58d349329/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/1*JNJwdUu0nSL39mAWdY1hYQ.png" /></figure><p>Hmm, looks like even though the correlation matrix suggests that carbohydrate and iron are positively correlated, they seem to lack <a href="https://en.wikipedia.org/wiki/Collinearity"><strong>colinearity</strong></a>.</p><p>This why it is important to plot the data before building the model.</p><p>Okay, getting back to our data, it seems all cluttered up on the bottom part of the plot. How about we try this again with Fiber and Iron.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/1*v_h3DxFyyWQMZqTLxevDuA.png" /></figure><p>Looks like it is also the same for fiber and iron. However, in this case, the data points are all cluttered up on the bottom left o the plot.</p><p>Another thing that we should pay attention to is the high variability of <a href="https://www.mathsisfun.com/data/outliers.html"><strong>outliers</strong></a> shown on the plot.</p><p>Clearly, if our initial goal to predict the amount of iron based on the amount of carbohydrate or fiber, a linear model will definitely show poor results.</p><p>But we are dealing with multiple variables here, let’s move from 2D into 3D. Again, Matplotlib offers a great way to plot 3D plots.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0bbf0ebe8473f5950a66f9d251ed27f4/href">https://medium.com/media/0bbf0ebe8473f5950a66f9d251ed27f4/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*aT2RCEcetIGGQ0tqNHvXjA.png" /></figure><p>Okay, now this plot shows definitely that linear model won’t work here. Our data is highly non-linear and using a linear regression won’t provide us with any good performance.</p><p>But for the sake of fun, let’s try anyway.</p><p>First of all, we need to split the input data and labels in a training and test set. The Scikit-learn’s train_test_split function makes this process convenient.</p><p>We will split 75% of the data to the training set and 15% to the test set.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5c55ae87c1f6b80991b5a99349da62ef/href">https://medium.com/media/5c55ae87c1f6b80991b5a99349da62ef/href</a></iframe><p>It is highly suggested in most practice that we should an additional set called the <strong>validation set</strong>. But since our dataset is quite small, we don’t have to do that here.</p><p>Now, we need to standardize our data before coding the model. This is important because some of the nutrients use different units. For example, Vitamin A uses the UI unit, while Carbohydrate uses the gram.</p><p>We will be using Scikit-learn’s RobustScalerto standardize our data so that all of them are robust against the differences in units.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fc913d9ad108c8cfd90ffba1c8bf01d1/href">https://medium.com/media/fc913d9ad108c8cfd90ffba1c8bf01d1/href</a></iframe><p>Alright, looks like we are ready to use the data to build the model.</p><p>I assume that most of the readers are already familiar with Scikit-learn’s <a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html">LinearRegression</a> class. It is commonly used in many online tutorials.</p><p>However, the class assumes that only one output variable is provided.</p><p>For multiple output variable, we need to add a class, which is not usually mentioned in most tutorials, called MultiOutputRegressor.</p><p>To put it simply, it is a way by Scikit-learn to extend regressor that does not natively support multi-output regression.</p><p>We’ll start by simply importing the classes.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2a12765ed1814c7a588262e841f20796/href">https://medium.com/media/2a12765ed1814c7a588262e841f20796/href</a></iframe><p>Next, we can start fitting the model.</p><p>Normally, in most tutorials about linear regression, you will most likely see something like this.</p><p>We need to modify that just a bit to include the MultiOutputRegressor class.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fcdb49b6a4babc08935ec363dbbfa4bc/href">https://medium.com/media/fcdb49b6a4babc08935ec363dbbfa4bc/href</a></iframe><p>Looking good so far. Next, we need to check our model has performed well during training. Scikit-learn’s score attribute should do the trick.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dab48ad7a0d98c3f6330c6d7d2374be0/href">https://medium.com/media/dab48ad7a0d98c3f6330c6d7d2374be0/href</a></iframe><p>Our model has shown an accuracy of ~72%. What do we make of this? Well, it seems with just linear regression, we can get performance better than flipping a coin.</p><p>But let’s continue a little bit with a very important step in every machine learning practice, validating our training performance on the test set.</p><p>We do this to avoid <strong>overfitting</strong> during training. To put it simply, overfitting occurs when a model tries to predict a trend in data that is too noisy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*NJnMs89pwUO9ZjORpXBohA.png" /><figcaption>The black line is when the model was able to generalize well on the data. On the other hand, the blue line displays the line when the model has overfitted. (source: wikipedia.org)</figcaption></figure><p>So let’s check how the model performs with the test set. We can do the same as we did with the training set with some minor changes.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6d306b4d5340083a764c56d46401dcb8/href">https://medium.com/media/6d306b4d5340083a764c56d46401dcb8/href</a></iframe><p>Looks like our model was able to escape overfitting to a certain extent (difference of around 7% between the training and test set accuracy).</p><p>You may now have a whisper in your head saying “so how do we avoid overfitting and underfitting?” Well, it has something to do with the <a href="http://scott.fortmann-roe.com/docs/BiasVariance.html"><strong>bias-variance tradeoff</strong></a>.</p><p>If we go back to the plot, our data looks all cluttered up. Visually, this supposed to be a sign of high bias, which should result in underfitting. However, looks like our model has more chance to overfit instead.</p><p>While we did a good job to avoid overfitting and underfitting, it’s not good enough to show to our team in UNICEF.</p><p>Since the number of people, we can influence is large (I mean it is UNICEF), then we should increase the performance of our model.</p><p>Like I said before, the dataset itself is highly non-linear and consist of many outliers. Doing a linear regression with such dataset will surely not result in an optimal performance</p><p>As a fun exercise, you can try to plot the data again with the predicted value of the model. This should give a better visual understanding of our data(we will do more visualization in a later part of this series).</p><p>So basically, what we need to do is use a different model that can handle non-linear data, and robust against outliers.</p><p>Enter Random Forest!</p><h3><strong>Random Forest</strong></h3><p>If we try to find some of the most used machine learning model to increase performance in Kaggle, it will either be Random Forest or Gradient Boosting Trees.</p><p>In this blog post, however, we will be talking about the first one.</p><p>But before diving into the algorithm right away, we need to talk about decision trees and bootstrapping.</p><h4><strong>Decision Trees</strong></h4><p>Imagine walking in the park while observing your surroundings. For no obvious reason, you decided to look at a particular tree. It starts from the root and makes it ways to the top until the last leaf.</p><p>This the same thing with decision trees, but upside down. Decision trees algorithm is also known as CART, Classification and Regression Trees.</p><p>For now, we will focus our attention on Regression Trees since, of course, we are dealing with a multivariate regression task.</p><p>Unlike linear regression, decision trees algorithm does not concern with the assumption of linearity within our data. Also, it does not incorporate the need for any weights.</p><p>Instead, when using decision trees, we are attempting to use the satisfaction of the feature space to make predictions.</p><p>According to Chapter 8 of the Introduction to Statistical Learning book, there are two steps that we need to follow to build decision trees:</p><blockquote>1. We divide the predictor space — that is, the set of possible values for X1, X2,…,Xp — into J distinct and non-overlapping regions, R1, R2,…,RJ .<br>2. For every observation that falls into the region Rj , we make the same prediction, which is simply the mean of the response values for the training observations in Rj .</blockquote><p>Note that the author is using, <em>Xj</em> ,as the notation for the independent variables instead of <em>xi. </em>We are going to do the same thing from here on.</p><p>Now for those who are not familiar with the decision trees algorithm might find the above statement rather unclear.</p><p>But to put it simply, the first statement state that we need to choose a variable, and established regions on the observations (data points) based on those observations.</p><p>While the second one is stating that after we have our regions, we can make predictions by calculating the mean of the observations in each region.</p><p>There are two ways that we can do to understand this concept intuitively through visualization.</p><p>First, we can plot our data points and draw a line that separates two areas of it. Let’s use the fiber and carbohydrate to estimate the value of iron.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/1*MCL2HxldzL277_M1OnjVlw.png" /></figure><p>Here we see a line that splits the plot into two regions. If we look closer, the line starts the split at about the 41 marks in the axis that represents carbohydrate.</p><p>In terms of regression, decision trees algorithms make a prediction by averaging the data points in each region.</p><p>Then the algorithm will continue to split on a variable until the estimated value is just short from the ground truth.</p><p>Another way we can visualize decision trees is by using graphs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/344/1*iMj0YzgPRdDnjeJ6w05dzQ.png" /></figure><p>Here we can see that the graph starts the split at the top. The boxes are called the <strong>nodes</strong> while the arrows that point/connect to another node are called the <strong>edges</strong>.</p><p>If you noticed, the first node makes a split on the 41.32 value, which is almost the same as the plot.</p><p>Unlike real-life trees, decision trees are usually visualized by having the branches going downwards.</p><p>The two nodes that branched out of the top node are called the <strong>internal nodes</strong>. In the picture above, there is the word <em>value</em> inside each internal node.</p><p>Those are the values are the predictions made by the algorithm by averaging the number of observations in each region.</p><p>However, both the plot and the graph above have only a depth of one. If we let the tree grow to a depth of two, then we will have something like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Vpafn5uEf1CZ0F2x1hqJVA.png" /><figcaption>The plot and the graph with a depth of two.</figcaption></figure><p>On the plot, we can see that now that another line has made a split on the previous line. While the graph has more internal nodes which made a split the fiber.</p><p>For the graph, if we let the tree grows until it made the best split, it will have more internal nodes, which in the end might not be visually comprehensible anymore.</p><p>But the question is what is the best split? Well, the best split is the split that minimizes the error between the predicted value and the true value.</p><p>If we look closer at our graph, we can see that there’s the word <em>mse</em> in it.</p><p>This refers to <a href="https://en.wikipedia.org/wiki/Mean_squared_error"><strong>mean squared error</strong></a><strong> </strong>which is given by the following formula</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/1*RKrL9XwPp-MYDa20l9MMKA.png" /></figure><p>This is similar to RSS that we’ve covered before with some slight difference.</p><p>As we see that the formula has the <em>1/N</em> notation. This means that after summing all of the squared errors, we average them by the number of <em>N</em> observations.</p><p>MSE might look like RSS plus the averaging notation, but <a href="https://stats.stackexchange.com/questions/73540/mean-squared-error-and-residual-sum-of-squares">this discussion</a> might worth checking out before making that conclusion.</p><p>Anyways, understanding the decision trees algorithm is a crucial step in order to work with the Random Forest algorithm.</p><p>However, this is just a brief explanation of what decision trees are capable of and recommend you to check out <em>chapter 8 of the </em><a href="https://www-bcf.usc.edu/~gareth/ISL/"><em>Introduction to Statistical Learning</em></a><em> book</em>.</p><p>The book gives a better high-level mathematical explanation for decision trees.</p><p>Before we continue, I might need to mention one more important aspects when using decision trees. It’s called the <strong>Gini Index.</strong></p><p>In short, it is used to measure the probability of a certain variable being mistakenly classified when it is arbitrarily chosen. The Gini Index is based on the concept of entropy, which is the degree of uncertainty.</p><p>This concept is usually common in various blog posts that use decision tree related algorithm for classification tasks.</p><p>In a classification task, we want to measure the <strong>information gain </strong>in order to understand which features give the most information about a certain class.</p><p>The information is then used to explain the importance of each variable and use it understand which variable has the most effect on the prediction or most commonly understood as <strong>variable importance</strong>.</p><p>In terms of regression, and as we discussed before, we use <em>mse </em>find the best split.</p><p>We are probably going to discuss more variable importance in a featured blog post in this series.</p><p>Next, we will cover another essential aspect of Random Forest, Bootstrapping.</p><h4>Bootstrapping</h4><p>One of the challenges of working with machine learning projects is that gathering data can be very expensive and difficult.</p><p>Although various open datasets — like the one we are using — are available on the internet, some models require more data to enhance performance during training.</p><p>But gathering data can sometimes be expensive and difficult in practice. We can use a resampling method called Bootstrapping</p><p>In short, bootstrapping involves resampling data from our already available dataset randomly with replacement. “With replacement” basically means we can use the same observations twice in the new dataset.</p><p>We repeat the process for some amount of time until we have the desired amount of bootstrapped datasets. Each bootstrapped dataset contains the same number of observations of the original dataset.</p><p>Let’s use an example with our dataset and use the first five observations of the carbohydrate variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Aol1pcF3HqEy9niOAqcKkQ.png" /></figure><p>The first five observations are 0.06, 2.87, 00.0, 2.34, and 2.79.</p><p>Let’s denote the bootstrapped dataset as <em>Zi</em>. If we have 4 datasets, then each dataset will be named <em>Z1, Z2, Z3, </em>and <em>Z4</em> respectively.</p><p>If you look at the image above, all of the bootstrapped datasets’ observations have repeating value in it. This is what “with replacement” means.</p><p>So what do we do with these bootstrapped datasets? Well, we use them to make machine learning models of course.</p><p>This might sound a little bit strange since we already have our original dataset to train on.</p><p>But it will all make sense once we move on the exciting part of this topic, coding the Random Forest!</p><h4>Coding the Random Forest</h4><p>Now that we have some intuition about decision trees and bootstrapping, we can move on to talking about Random Forest.</p><p>You see, there’s a reason why the algorithm is called Random Forest. Think about the two important aspects that we’ve just discussed.</p><p>The algorithm uses bootstrapping to create randomly subsets of data with subsets of features from the original dataset. Then decision trees algorithm are used to build a model on each subset.</p><p>Then each prediction of each model is averaged to create one final prediction.</p><p>This is the same for multivariates tasks. The only difference is that each decision trees output values for Vitamin A, iron, Zinc, Calcium, Vitamin D, and Folate.</p><p>Let’s get started! The same with Linear Regression, we need to import the RandomForestRegression class and the MultiOutputRegressor class.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/745c12874dc5597e929926c836ed2f70/href">https://medium.com/media/745c12874dc5597e929926c836ed2f70/href</a></iframe><p>Now we can fit the data into the model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b0b850f3c36195c475bbffa2c3cad053/href">https://medium.com/media/b0b850f3c36195c475bbffa2c3cad053/href</a></iframe><p>As we can see there are some parameters that were set here. n_estimators to 100 because the next version of Scikit-learn will have that values as the default value.</p><p>max_features is just make sure that we subset the square root of the number of features. This is one of the hyperparameters that we need to tune for our model. We will cover this in another blog post.</p><p>Next, we can try to check the accuracy of our model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4461e08054010daae7726d0f0ac973d4/href">https://medium.com/media/4461e08054010daae7726d0f0ac973d4/href</a></iframe><p>Wow looks like our model performs really well on the training data. I mean, ~95% accuracy? That sounds pretty good, right?</p><p>Well, not really. We still have to check out the test accuracy, because what we want is the model to perform well on data that we have not observed before.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b1e5fe0e59e0486ef2733c2373304c2f/href">https://medium.com/media/b1e5fe0e59e0486ef2733c2373304c2f/href</a></iframe><p>Looks like the performance seems pretty good. Almost 20% better than linear regression</p><p>Unlike Linear Regression, our model overfits since the training accuracy is way larger than the test accuracy (around ~15% difference).</p><p>There are various ways that we can do to improve the model. However, let’s see if we can make the model using an algorithm that has gained high popularity in recent years.</p><p>In the next part of this series, we will build a Neural Network algorithm to train on our data.</p><p>What do you think will happen when we use Neural Network on our dataset? Will it be the same? better? or even worse?</p><p>Let me know what you think in the comments. Looking forward to the feedback and further discussions.</p><p>See you in part 3. Cheers!</p><p><em>An Indonesian student currently studying in Korea and has a deep interest in the world of AI. Recently moved to IBM as an AI Research Intern. Believe in continuous learning. Want to give back to the AI that has contributed so much in my pursuit of knowledge. When I’m not coding, I like to read up the latest trends in the space industry and play for my college’s American football team.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1bd85497fdf1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/predicting-micronutrients-using-neural-networks-and-random-forest-part-2-1bd85497fdf1">Predicting Micronutrients using Neural Networks and Random Forest (Part 2)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting Micronutrients using Neural Networks and Random Forest (Part 1)]]></title>
            <link>https://medium.com/data-science/predicting-micronutrients-using-neural-networks-and-random-forest-part-1-83a1469766d7?source=rss-27416751a23e------2</link>
            <guid isPermaLink="false">https://medium.com/p/83a1469766d7</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-cleaning]]></category>
            <dc:creator><![CDATA[Gilang R Ilhami]]></dc:creator>
            <pubDate>Tue, 01 Jan 2019 23:13:35 GMT</pubDate>
            <atom:updated>2019-06-24T16:44:45.522Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>UNICEF wants you to help them to predict important nutritions within foods using the power of machine learning.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2c55qtDQAd8RrD09" /><figcaption>Photo by <a href="https://unsplash.com/@dosejuice?utm_source=medium&amp;utm_medium=referral">Dose Juice</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Welcome, all! Today, we are going to implement machine learning algorithms to tackle one of the world’s hardest problems — world hunger.</p><p>I decided to break this tutorial into multiple parts to make sure we don’t miss any important details.</p><p>In this series, we are going to work on a regression problem and predict important micronutrients, which are the building blocks of healthy brains, bones, and bodies, based on other micronutrients and macronutrients.</p><p>We will use the programming language Python and two of the commonly used dependencies in machine learning to build our models, Keras and Scikit-Learn.</p><p>In this first part, we will talk more about the dataset and data cleansing it instead of building the model itself.</p><p>Data cleansing is a critical part of the process since the data that is provided to us is not always ready to be fed into the model.</p><p>Without further ado, let’s begin.</p><h3><strong>Overview</strong></h3><p>This blog post includes the following:</p><ul><li>Motivation</li><li>Prerequisites</li><li>Introduction</li><li>Dataset</li><li>Data Cleansing</li></ul><h3>Motivation</h3><p>In this blog post, I decided to cover something a little different regarding the task. Most of the time, machine learning tutorials cover univariate regression, where there is only one outcome variable during prediction.</p><p>This time, however, we will be doing a multivariate regression problem, where the outcome variable is more than one, using some popular machine learning algorithms.</p><p>I think this is important because many of real-world problems need to consider the relationship between multiple output variables.</p><p>Note — the topic of deep learning will come when we are using neural networks for regression. Most blog posts will compare how deep neural networks show better accuracy than other algorithms.</p><p>However, in this series specifically, I will cover a scenario where deep neural networks actually <em>underperformed</em> than other machine learning architecture.</p><p>Also, I will be introducing some useful tools that you can leverage to build machine learning algorithms.</p><h3>Prerequisites</h3><p>This blog post assumes that the reader has some familiarity with popular machine learning algorithms (such as linear regression and decision trees) and technical terms (such as overfitting, hidden layers, etc.,).</p><p>If you have a no prior knowledge or need some to fresh up on some of the things that I just mentioned above, here are some resources that I recommend:</p><ul><li>“Introduction to Statistical Learning” by Gareth James, Daniela Witten, Trevor Hastie, Robert Tibshirani</li><li>Machine Learning, Linear Regression, Random Forest<br>“Elements to Statistical Learning” by Trevor Hastie, Robert Tibshirani, Jerome Friedman</li><li><a href="https://bit.ly/2RorsKx">Introduction to Machine Learning</a></li><li><a href="https://bit.ly/2QYbebw">Multiple/Multivariate Regression</a></li><li><a href="https://bit.ly/2iv6dmW">Deep Learning and Neural Network</a></li></ul><h3>Introduction</h3><p>The United Nations Children’s Fund (UNICEF) has a request for you to join their data science team to solve an important issue that plays a vital role in the fight against world hunger — micronutrients (also known as vitamins and minerals).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ADqPsipCqfD68s0R" /><figcaption>Photo by <a href="https://unsplash.com/@perrygrone?utm_source=medium&amp;utm_medium=referral">Perry Grone</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Your job is to create a machine learning model that can make predictions of these deficiencies based on other nutrients (Protein, Carbohydrates, etc.,).</p><p>However, there is just one thing that they forgot to mention about the job, they don’t have the data (which is highly unlikely for an organization as big as UNICEF; but for the sake of this blog post, let’s just say they don’t).</p><p>Therefore, you will need to find a way to get the data in order to build the model.</p><p>But how? and from where?</p><h3>Dataset</h3><p>Luckily, one of the greatest creation in history, the internet, is filled with many open datasets that we can download and use to train our model.</p><p>In this case, we will be using <a href="https://www.unicef.org/nutrition/index_iodine.html">The USDA National Nutrient Database for Standard Reference (SR) </a>from the US Department of Agriculture’s Agricultural Research Service.</p><p>In this tutorial, we will be using the abbreviated excel file that you can download on <a href="https://www.ars.usda.gov/ARSUserFiles/80400525/Data/SR/SR28/dnload/sr28abxl.zip">here</a>.</p><p>The dataset contains the main source of food composition data in the United States with more than 8,000 data points with over 50 variables of micronutrients in each food (such as carbohydrate, protein, etc.,).</p><p>Now that we got the data, the first thing to do is data cleansing.</p><h3>Data Cleansing</h3><p>Naturally, data does not always come in the form of a clean CSV or text file that is ready to be fed in a machine learning model.</p><p>Most of the food that we take from nature are first to be cleaned and processed before we can buy them from the supermarket. This is also the same with data.</p><p>But first, let’s look at our data. Since the data is in the form of an Excel file, we will use Pandas read_excel function to open the file.</p><pre>import pandas as pd</pre><pre>path_to_data = &quot;datasets/ABBREV.xlsx&quot; # Path to the Excel file<br>dataset = pd.read_excel(path_to_data)<br>dataset.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*eQcPYQR4VgbFaZ-gGZXAqQ.png" /><figcaption>If you run on the code on Jupyter Notebook, then the table should look like this (might look different if you run a python script on terminal)</figcaption></figure><p>dataset.head() displays the first 5 rows of the dataset. By doing this, we can get some idea about what our data looks like.</p><p>As we can see, the table contains data about the nutrient compositions of each food (such as Butter, Cheese, atc.,).</p><p>The columns contain the name of each micronutrient and macronutrients of the food (Protein, Carbohydrate, etc.,).</p><p>To see all the variable of the data, you can run the following code below.</p><pre>dataset.columns</pre><pre># Output<br>Index([&#39;NDB_No&#39;, &#39;Shrt_Desc&#39;, &#39;Water_(g)&#39;, &#39;Energ_Kcal&#39;, &#39;Protein_(g)&#39;,<br>       &#39;Lipid_Tot_(g)&#39;, &#39;Ash_(g)&#39;, &#39;Carbohydrt_(g)&#39;, &#39;Fiber_TD_(g)&#39;,<br>       &#39;Sugar_Tot_(g)&#39;, &#39;Calcium_(mg)&#39;, &#39;Iron_(mg)&#39;, &#39;Magnesium_(mg)&#39;,<br>       &#39;Phosphorus_(mg)&#39;, &#39;Potassium_(mg)&#39;, &#39;Sodium_(mg)&#39;, &#39;Zinc_(mg)&#39;,<br>       &#39;Copper_mg)&#39;, &#39;Manganese_(mg)&#39;, &#39;Selenium_(µg)&#39;, &#39;Vit_C_(mg)&#39;,<br>       &#39;Thiamin_(mg)&#39;, &#39;Riboflavin_(mg)&#39;, &#39;Niacin_(mg)&#39;, &#39;Panto_Acid_mg)&#39;,<br>       &#39;Vit_B6_(mg)&#39;, &#39;Folate_Tot_(µg)&#39;, &#39;Folic_Acid_(µg)&#39;, &#39;Food_Folate_(µg)&#39;,<br>       &#39;Folate_DFE_(µg)&#39;, &#39;Choline_Tot_ (mg)&#39;, &#39;Vit_B12_(µg)&#39;, &#39;Vit_A_IU&#39;,<br>       &#39;Vit_A_RAE&#39;, &#39;Retinol_(µg)&#39;, &#39;Alpha_Carot_(µg)&#39;, &#39;Beta_Carot_(µg)&#39;,<br>       &#39;Beta_Crypt_(µg)&#39;, &#39;Lycopene_(µg)&#39;, &#39;Lut+Zea_ (µg)&#39;, &#39;Vit_E_(mg)&#39;,<br>       &#39;Vit_D_µg&#39;, &#39;Vit_D_IU&#39;, &#39;Vit_K_(µg)&#39;, &#39;FA_Sat_(g)&#39;, &#39;FA_Mono_(g)&#39;,<br>       &#39;FA_Poly_(g)&#39;, &#39;Cholestrl_(mg)&#39;, &#39;GmWt_1&#39;, &#39;GmWt_Desc1&#39;, &#39;GmWt_2&#39;,<br>       &#39;GmWt_Desc2&#39;, &#39;Refuse_Pct&#39;],<br>      dtype=&#39;object&#39;)</pre><p>Before we move on, I always like to check the datatype of each column. This is important because only float type will be used during training.</p><p>I will not get into details about what all of those variables mean. But if you are interested to find out, you can check the documentation that is included in the zip file when you downloaded the dataset.</p><p>Luckily, Pandas’ DataFrame class has an attribute dtypes that returns a Series with the data type of each column.</p><pre>dataset.dtypes</pre><pre>#Output</pre><pre>NDB_No                 int64<br>Shrt_Desc             object<br>Water_(g)            float64<br>Energ_Kcal             int64<br>Protein_(g)          float64<br>Lipid_Tot_(g)        float64<br>Ash_(g)              float64<br>Carbohydrt_(g)       float64<br>Fiber_TD_(g)         float64<br>Sugar_Tot_(g)        float64<br>Calcium_(mg)         float64<br>Iron_(mg)            float64<br>Magnesium_(mg)       float64<br>Phosphorus_(mg)      float64<br>Potassium_(mg)       float64<br>Sodium_(mg)          float64<br>Zinc_(mg)            float64<br>Copper_mg)           float64<br>Manganese_(mg)       float64<br>Selenium_(µg)        float64<br>Vit_C_(mg)           float64<br>Thiamin_(mg)         float64<br>Riboflavin_(mg)      float64<br>Niacin_(mg)          float64<br>Panto_Acid_mg)       float64<br>Vit_B6_(mg)          float64<br>Folate_Tot_(µg)      float64<br>Folic_Acid_(µg)      float64<br>Food_Folate_(µg)     float64<br>Folate_DFE_(µg)      float64<br>Choline_Tot_ (mg)    float64<br>Vit_B12_(µg)         float64<br>Vit_A_IU             float64<br>Vit_A_RAE            float64<br>Retinol_(µg)         float64<br>Alpha_Carot_(µg)     float64<br>Beta_Carot_(µg)      float64<br>Beta_Crypt_(µg)      float64<br>Lycopene_(µg)        float64<br>Lut+Zea_ (µg)        float64<br>Vit_E_(mg)           float64<br>Vit_D_µg             float64<br>Vit_D_IU             float64<br>Vit_K_(µg)           float64<br>FA_Sat_(g)           float64<br>FA_Mono_(g)          float64<br>FA_Poly_(g)          float64<br>Cholestrl_(mg)       float64<br>GmWt_1               float64<br>GmWt_Desc1            object<br>GmWt_2               float64<br>GmWt_Desc2            object<br>Refuse_Pct           float64<br>dtype: object</pre><p>Ahah, look at that. It seems that all of the columns are floats except for NDB_No, Shrt_Desc, Energy_Kcal, GmWt_Desc1, and GmWt_Desc2.</p><p>Also, you can see that some of these columns have the datatype object. This is because they contain text for each row entry.</p><p>For that reason, Pandas will consider them as categorical data. I’d like to use the term <em>false categorical variable</em> because they are not actually categorical data and just contain text within them.</p><p>To make the data cleansing process more simple, let’s separate these <em>false categorical variables</em> with the other numerical variable.</p><p>Since Pandas do not provide such a feature, we have to do it manually.</p><pre>tot_cols = dataset.columns<br>num_cols = dataset._get_numeric_data().columns<br>cat_cols = list(set(tot_cols)-set(num_cols))</pre><pre># Output of cat_cols</pre><pre>[&#39;GmWt_Desc1&#39;, &#39;GmWt_Desc2&#39;, &#39;Shrt_Desc&#39;]</pre><p>Great! Now we can move one to our first step.</p><p>First of all, note that NBD_No contains the index of each food and Shrt_Desc contains the name of the food itself. In this case, we can drop these variables since they won’t be used to build the model.</p><pre>drop_cols = [&#39;NDB_No&#39;, &#39;Shrt_Desc&#39;]<br>dataset = dataset.drop(drop_cols, axis=1)</pre><p>Next thing we need to set the <strong>labels</strong> for our training. Since we are dealing with a multivariate regression problem, that means the output variable will be more than one.</p><p>In other words, this means we have multiple labels.</p><p>As I mentioned before, the UNICEF wants you to make a prediction about important micronutrients within certain foods. But as we see in the columns, there are many of them.</p><p>How do we know which of them are going to be our labels?</p><p>Well, there’s no forward answer to this and it depends on the problem you are trying to solve. Since we don’t know much about nutrition (I assumed) we can check out the UNICEF’s official website.</p><p>According to the <a href="https://www.unicef.org/nutrition/index_iodine.html">UNICEF’s official website</a>, there are seven main deficiencies when it comes to micronutrients: Iodine, Vitamin A, Iron, Zinc, Calcium, Vitamin D, and Folate.</p><p>All of them are provided in our dataset except for Iodine. But after discussing with our supervisor in the team, he says it’s okay to proceed for the first iteration of building the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*zJnwFircxqSDL2-jnEhGbg.png" /><figcaption>Just to give a visualization of what we are trying to do.</figcaption></figure><p>For now, we will just put the name of the outcome variable in a list. We can do that in Python by doing the following.</p><pre>labels = [&#39;Calcium_(mg)&#39;, &#39;Iron_(mg)&#39;,&#39;Zinc_(mg)&#39;, &#39;Vit_A_IU&#39;, &#39;Vit_D_IU&#39;, &#39;Folate_Tot_(µg)&#39;]</pre><p>Alright, now we need to go back to those <em>false categorical variables. </em>Since we’ve already droppedNDB_No, Shrt_Desc, and all we need to do is what to do with GmWt_Desc1, and GmWt_Desc2.</p><p>Now there are two things you can do here.</p><p>First, you can just drop them because they are just descriptions of the GmWt_1 and GmWt_2 variables. A second alternative is to clean the text and just includes the numbers of each row entry.</p><p>I would recommend the first choice since these variables would most like do not have any significant impact on the performance of the model. However, for the sake of this blog post, I will include them.</p><p>Why? I’ll explain.</p><p>Data on the internet is not always clean. They can have redundant or chattery words in them. So it is good practice to somehow clear these text and represent them with numbers.</p><p>Computers can only compute numbers and not text type data.</p><p>For that reason, we need to remove those text in GmWt_Desc1 and GmWt_Desc2. In this case, that includes anything within parentheses.</p><p>Luckily, I’ve already a written a function to do just that.</p><pre>def clean_text(data):<br>    <br>    # If value is NaN we return the same value for cleansing later on<br>    if isinstance(data, str):<br>        <br>        # Remove any punctuations including words within parentheses<br>        clean = re.sub(r&#39;\([^)]*\)&#39;, &#39;&#39;, data)<br>        <br>        # Get numbers only<br>        clean = re.sub(&quot;\D&quot;,&quot;&quot;,clean)<br>        <br>        return clean<br>    <br>    else:<br>        return data</pre><p>We can use this function to remove punctuations and words within parentheses. Also, we want to get the number within sentences.</p><p>For example, if the entry in the row is “1 pat, (1” sq, 1/3&quot; high)”, then we only left with the number 1.</p><p>Pandas allows us to that simply with the attribute apply. This where the list of <em>false categorical variables </em>comes in handy.</p><pre># Remove text in GmWt_Desc1<br>dataset[cat_cols[0]] = dataset[cat_cols[0]].apply(clean_text)</pre><pre># Remove text in GmWt_Desc2<br>dataset[cat_cols[1]] = dataset[cat_cols[1]].apply(clean_text)</pre><p>Fantastic! Looks like we’re done, right? Not really. There is a couple of things that we need to do before we are ready to feed the data.</p><p>If you go back to the clean_text function, there was a comment that state “If value is NaN we return the same value for cleansing later on.”</p><p>Another common step in cleaning your dataset is dealing with <strong>missing data</strong>. This very common since many datasets that can be found on the internet is not completely filled.</p><p>Those missing data are replaced with NaN values. Same like text, computers cannot compute NaN values.</p><p>Let’s see if our dataset has any missing values for each column.</p><pre>dataset.isnull().sum()</pre><pre># Output</pre><pre>Water_(g)               1<br>Energ_Kcal              0<br>Protein_(g)             0<br>Lipid_Tot_(g)           0<br>Ash_(g)               325<br>Carbohydrt_(g)          0<br>Fiber_TD_(g)          594<br>Sugar_Tot_(g)        1832<br>Calcium_(mg)          348<br>Iron_(mg)             144<br>Magnesium_(mg)        739<br>Phosphorus_(mg)       579<br>Potassium_(mg)        426<br>Sodium_(mg)            83<br>Zinc_(mg)             706<br>Copper_mg)           1257<br>Manganese_(mg)       2160<br>Selenium_(µg)        1700<br>Vit_C_(mg)            818<br>Thiamin_(mg)          634<br>Riboflavin_(mg)       616<br>Niacin_(mg)           637<br>Panto_Acid_mg)       2242<br>Vit_B6_(mg)           905<br>Folate_Tot_(µg)      1261<br>Folic_Acid_(µg)      2039<br>Food_Folate_(µg)     1768<br>Folate_DFE_(µg)      2057<br>Choline_Tot_ (mg)    4016<br>Vit_B12_(µg)         1193<br>Vit_A_IU              711<br>Vit_A_RAE            1535<br>Retinol_(µg)         1806<br>Alpha_Carot_(µg)     3258<br>Beta_Carot_(µg)      3162<br>Beta_Crypt_(µg)      3270<br>Lycopene_(µg)        3292<br>Lut+Zea_ (µg)        3315<br>Vit_E_(mg)           2889<br>Vit_D_µg             3262<br>Vit_D_IU             3211<br>Vit_K_(µg)           3563<br>FA_Sat_(g)            349<br>FA_Mono_(g)           666<br>FA_Poly_(g)           665<br>Cholestrl_(mg)        410<br>GmWt_1                300<br>GmWt_Desc1            299<br>GmWt_2               3965<br>GmWt_Desc2           3965<br>Refuse_Pct             50<br>dtype: int64</pre><p>Wow, it seems we have plenty of missing data in our dataset.</p><p>Again, there are two things we can do here. First is by dropping those the rows that contain missing data for any columns. This means if a column has a NaN value, we drop them right away.</p><p>The second option is to do imputing, which means replacing the missing values. This option is more preferable because we are not throwing away other data that might be important during the training of the model.</p><p>The question is, what do we replace it with?</p><p>There’s no one answer to this. It depends on the problem you are facing and the model you are using. Some models are robust toward missing data, while others cannot handle them.</p><p>For example, later on, we are going to use a popular machine learning algorithm called Random Forest. This model has its own way of dealing with missing data.</p><p>However, we will not cover that in this series. But if you are interested to know about it, <a href="https://www.youtube.com/watch?v=nyxTdL_4Q-Q">this video</a> can be very helpful.</p><p>For simplicity, we will be using one of the most popular technique to replace missing data. And that is by using the median.</p><p>Why? That is an explanation for another blog post. In the meantime, let’s move forward.</p><p>To get the median value for each column, we can use the attribute median() provided by the DataFrame class. Then we can replace the missing value by using the fillna attribute.</p><pre>get_median = dataset.median()<br>dataset = dataset.fillna(get_median)</pre><p>An alternative way is using Scikit-learn’s <a href="https://scikit-learn.org/stable/modules/generated/sklearn.impute.SimpleImputer.html">SimpleImputer class</a>.</p><p>Now we need to check if no missing data values are in our dataset.</p><pre>dataset.isna().values.any()</pre><pre># Output<br>False</pre><p>Looks like all the missing values are gone.</p><p>Now, for the final step. Since we are dealing with a supervised learning task, we need to separate the dataset into input data and labels.</p><p>But before that, I want to mention something very quick.</p><p>If you pay attention to the list of variables we have, for Vitamin A and Vitamin D, we have two variables representing each of them.</p><p>For Vitamin A we have Vit_A_IU and Vit_A_RAE, and for Vitamin D we have Vit_D_IU and Vit_D_µg.</p><p>The IU stands for International Standard. I’m not going to explain about what IU is, but if you want to know more then you can check it out <a href="https://livewell.jillianmichaels.com/iu-vitamins-5480.html">here</a>.</p><p>I decided to choose Vit_A_IU and Vit_D_IU because I try to avoid too much variation in the difference of units of each variable.</p><p>Now that we got that out of the way, let’s move on to separating the input data and the labels.</p><p>Remember the list of label names that initially prepared before? This is where it’s going to come handy.</p><pre># List of names of the columns that we will drop for the input data<br>drop_cols = [&#39;Vit_D_µg&#39;, &#39;Vit_A_RAE&#39;] + labels</pre><pre># Get the data for labels<br>y = dataset[labels].values</pre><pre># Get the data for inputs<br>X = dataset.drop(drop_cols, axis=1).values</pre><p>The value attribute converts the data from a DataFrame object into Numpy arrays.</p><p>Awesome! We’re done with cleaning the data</p><p>We are making progress as part of our data science team. Now that the data is rid of text and missing values, we are ready to start building the model.</p><p>Join me in Part 2 of this series where we will finally take another step on the fight against world hunger by creating a machine learning model using Scikit-learn.</p><p>I hope this blog post was helpful and can’t wait to see you in part 2. Until then, cheers!</p><p>About Me:</p><p><em>An Indonesian student currently studying in Korea and has a deep interest in the world of AI. Backend Developer at a startup called 88spares. Believe in continuous learning. Want to give back to the AI that has contributed so much in my pursuit of knowledge. When I’m not coding, I like to read up the latest trends in the space industry and play for my college’s American football team.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83a1469766d7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/predicting-micronutrients-using-neural-networks-and-random-forest-part-1-83a1469766d7">Predicting Micronutrients using Neural Networks and Random Forest (Part 1)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>