<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Code Heroku - Medium]]></title>
        <description><![CDATA[Live Programming Courses - Medium]]></description>
        <link>https://medium.com/code-heroku?source=rss----3b5048e3b0e5---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Code Heroku - Medium</title>
            <link>https://medium.com/code-heroku?source=rss----3b5048e3b0e5---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 21:59:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/code-heroku" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Part 2 — How to Turn Your Machine Learning Scripts into Projects You Can Demo?]]></title>
            <link>https://medium.com/code-heroku/part-2-how-to-turn-your-machine-learning-scripts-into-projects-you-can-demo-467b3acff041?source=rss----3b5048e3b0e5---4</link>
            <guid isPermaLink="false">https://medium.com/p/467b3acff041</guid>
            <category><![CDATA[movie-review]]></category>
            <category><![CDATA[vuejs]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <dc:creator><![CDATA[Code Heroku]]></dc:creator>
            <pubDate>Wed, 15 Jul 2020 12:27:27 GMT</pubDate>
            <atom:updated>2020-07-15T12:27:27.330Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part 2 — How to Turn Your Machine Learning Scripts into Projects You Can Demo?</h3><p>I often see folks who aspire to become machine learning engineers but who don’t have any prior experience in software development ask this question — “How do I take my python scripts and use it in a real software project?” The answer usually is to expose your machine learning models with a REST API which can be then consumed in either a mobile or web application.</p><p><a href="https://medium.com/code-heroku/how-to-turn-your-machine-learning-scripts-into-projects-you-can-demo-cbc5611ca442">In the previous part of this tutorial </a>we gave you an approach on how to build such an API using a python based framework called web2py.</p><p>In this part of the tutorial we will walk you through the steps of building a front-end app with VueJS that will send a request to the movie recommendation API that we created in <a href="https://medium.com/code-heroku/how-to-turn-your-machine-learning-scripts-into-projects-you-can-demo-cbc5611ca442">Part-1</a> and display the recommended movies to our users.</p><h3>The End Result</h3><p>Live Demo: <a href="http://www.codeheroku.com/static/movies/index.html">http://www.codeheroku.com/static/movies/index.html</a></p><p>The screenshot below and the demo link will give you a sense of the final output that we are looking for.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rTji2xGSL0mXXbM0jvVMgw.png" /></figure><p>We essentially need a:</p><ol><li>A search box where the user enters the name of movies which they have seen before</li><li>Ability to Rate those movies with “Stars” and create a list of rated movies</li><li>And finally based on the rated movies, display a list of recommended movies to the user.</li></ol><h3>Download The Starter Kit</h3><p>Here is everything you need:</p><p><a href="https://drive.google.com/file/d/1WoLN9Ra4uHw2g0RWsqkybFqHpsHi__jT/view?usp=sharing">movie recommendation part 2.zip</a></p><ol><li>Extract the above zip file in your web2py project’s static folder and call it movies</li><li>Extract the posters.zip and move it inside the movies folder</li><li>Make sure your web app is up and running. (See Part-1 of this tutorial)</li><li>Open <a href="http://127.0.0.1:8000/movies/static/movies/index.html">http://127.0.0.1:8000/movies/static/movies/index.html</a></li></ol><blockquote><em>If you like this post, please do share it with your network and give it some claps! This will encourage us to produce more such content.</em></blockquote><blockquote><em>If you get stuck somewhere while building this project you can reach out to us via email: hello@codeheroku.com or on WhatsApp: +91–9967578720</em></blockquote><h3>Why VueJS?</h3><p>VueJS is arguably the most beginner friendly reactive front-end framework to build single page apps. it has grown tremendously in popularity over the past few years because of its easy to follow documentation and a smooth learning curve. Even though you can build this web page without VueJs, but using it will make your life much easier.</p><h3>Step 0: Become Familiar with VueJS</h3><p>When you are learning something new, never skip going over some basic examples of a new library or framework before trying to build the actual project you are working on. Most beginners take way too long to complete a project because they never take some time initially to understand the fundamentals of a library/tool and dive directly into building the intended project.</p><p>VueJS has some excellent tutorials on their official page, I suggest you at least go over the introduction section before moving forward: <a href="https://vuejs.org/v2/guide/index.html">https://vuejs.org/v2/guide/index.html</a></p><h3>Step 1: Create a Vue.js object</h3><p>Let’s start by creating a Vue.js object, wherein we will edit the parameters to directly change the content of the HTML.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c44d3a941274d639b33ee6f38aa0c191/href">https://medium.com/media/c44d3a941274d639b33ee6f38aa0c191/href</a></iframe><h3>Step 2: Load List of Movies</h3><p>Let’s familiarize some files in your starter kit</p><ol><li>Posters.zip contains posters for each movie in the MovieLens dataset, filename corresponds to the ‘movieId’. It is open-sourced at: <a href="https://github.com/babu-thomas/movielens-posters">https://github.com/babu-thomas/movielens-posters</a></li><li>Movies.zip contains an index.html and the test.html which we built in the <a href="https://medium.com/code-heroku/how-to-turn-your-machine-learning-scripts-into-projects-you-can-demo-cbc5611ca442">Part-1</a> of the tutorial</li><li>movies.json &amp; movies_small.json contain list of dictionaries, containing ‘movieId’, ‘title’, ‘genres’.<br>Note that we’re not considering ‘genres’ in our application, which could make our application more robust! Afterwards, you can explore it on your own.</li></ol><p>Let’s load the JSON to our HTML using Vue.js and jQuery, a popular JavaScript library designed to simplify HTML DOM tree traversal and manipulation.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/44731afd873f3140847c18286b7b351e/href">https://medium.com/media/44731afd873f3140847c18286b7b351e/href</a></iframe><p>Whenever index.html is called, this function would be the first to execute. You can press F12, or right-click and go to Inspect Element, open up the console and you’ll find “here”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G1DxpnrsXVPAbY-Lsyzu3w.png" /></figure><h3>Step 3: Create methods to add/remove a rating, and get recommendations</h3><p>Here’s where we will use a modified version of post processing from the API that we created in <a href="https://medium.com/code-heroku/how-to-turn-your-machine-learning-scripts-into-projects-you-can-demo-cbc5611ca442">Part-1</a>, with a bit of JavaScript touch.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/71c06d681a26fa806b33c008ed79d3d8/href">https://medium.com/media/71c06d681a26fa806b33c008ed79d3d8/href</a></iframe><p>In <em>addrating</em> and <em>removemovie</em> methods, after the operation, we set the search string back to an empty string (“”), and call the <em>getrecommendations</em> method again. The end outcome will update the predictions as soon as we add a rating, or remove an existing movie which we rated.</p><h3>Step 4: Create computed property in Vue object</h3><p>We will create a computed value as a derived value that will be automatically updated whenever one of the underlying values used to calculate it is updated. You don’t <em>call</em> a computed and it doesn’t accept any parameters, which is why it is different from <em>method</em> property.<br>Here’s a more discrete answer: <a href="https://stackoverflow.com/questions/44350862/method-vs-computed-in-vue">https://stackoverflow.com/questions/44350862/method-vs-computed-in-vue</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e1910029a2d880cea0c1bc9f2af5b085/href">https://medium.com/media/e1910029a2d880cea0c1bc9f2af5b085/href</a></iframe><p>Notes: <br>1. If the search string has length less than 2, it is discarded.<br>2. If this.all_movies returns a False value, an empty array is taken instead.<br>3. In line 74–79, we’re filtering movies if their index in the array of movies is greater than -1, i.e., they’re present in the array.<br>4. We’re bounding the resulting array with a length of 100.</p><p>With this, we have completed writing code to render our HTML!</p><h3>Step 5: Create Nav-bar</h3><p>We will use one of the most pretty and responsive CSS frameworks, <a href="https://getbootstrap.com/">Bootstrap</a>. You don’t need to download anything, we have provided the core CSS in the Starter Kit.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fd936cf0196436111d41e6f4ce3c2b88/href">https://medium.com/media/fd936cf0196436111d41e6f4ce3c2b88/href</a></iframe><p>With this much code, you’ll get a nice top nav-bar with “Movie Recommendation Engine” in clickable format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nAhz91zUxHrETlA3UXz0vw.png" /></figure><h3>Step 6: Add Search Box</h3><p>Seach Box connects to the <em>addmovie </em>function, which we created in Step 4.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/841fe7425aaa7b3fd6e41aa247150d60/href">https://medium.com/media/841fe7425aaa7b3fd6e41aa247150d60/href</a></iframe><p>To further your understanding, try to add rating numbers (1–5) in a square box, which would look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1hWezjd-QD0HZ7dlSvYlnQ.png" /></figure><p>Note: Answer to this is in index.html provided in the Starter Kit, but we encourage you to try it out on your own first.</p><h3>Step 7: Display watched movies</h3><p>The displayed movies ideally, should also have a ‘Remove’ utility, connecting to the <em>removemovie </em>method we created in Step 4.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/420ac880a7fae22dd2ee2574bff1f054/href">https://medium.com/media/420ac880a7fae22dd2ee2574bff1f054/href</a></iframe><p>With that done, after searching for a movie, and rating them, we should get something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*TYpCMwx7hRjyr6-xKNnDkA.gif" /></figure><p>We’re close to the endgame!</p><h3>Step 8: Display predictions</h3><p>If the movie has a corresponding poster, display it along with star rating, otherwise display the movie name with star rating.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7b75beb9bca61a6a66e3cf8419bd17d/href">https://medium.com/media/d7b75beb9bca61a6a66e3cf8419bd17d/href</a></iframe><p>And, we’re done!</p><p>Open <a href="http://127.0.0.1:8000/movie/static/movies/index.html">http://127.0.0.1:8000/movies/static/movies/index.html</a> and try out your very own recommendation system.</p><blockquote><em>If you like this post, please do share it with your network and give it some claps! This will encourage us to produce more such content.</em></blockquote><blockquote><em>If you get stuck somewhere while building this project you can reach out to us via email: hello@codeheroku.com or on WhatsApp: +91–9967578720</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=467b3acff041" width="1" height="1" alt=""><hr><p><a href="https://medium.com/code-heroku/part-2-how-to-turn-your-machine-learning-scripts-into-projects-you-can-demo-467b3acff041">Part 2 — How to Turn Your Machine Learning Scripts into Projects You Can Demo?</a> was originally published in <a href="https://medium.com/code-heroku">Code Heroku</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Object Detection in 2020 —  From RCNNs to YOLOv4]]></title>
            <link>https://medium.com/code-heroku/object-detection-in-2020-from-rcnns-to-yolov4-fd549b182b78?source=rss----3b5048e3b0e5---4</link>
            <guid isPermaLink="false">https://medium.com/p/fd549b182b78</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[faster-r-cnn]]></category>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[yolo-object-detection]]></category>
            <dc:creator><![CDATA[Code Heroku]]></dc:creator>
            <pubDate>Sun, 28 Jun 2020 09:54:04 GMT</pubDate>
            <atom:updated>2020-06-28T09:50:15.503Z</atom:updated>
            <content:encoded><![CDATA[<h3>Object Detection in 2020 — From RCNNs to YOLOv4</h3><h3>What is Object Detection?</h3><p>Thinking in terms of a layman, what actually is object detection?</p><p>Let’s walk through a sample image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*OHlBv6CJfJen_18EFWWNmQ.gif" /></figure><p>As a human, we understand that the image contains moving cars, But how do we inject this understanding to a computer? Pretty fascinating if you think about it.</p><p>The problem can be broken down into two sub-problems.</p><h4><strong>Object localization and Image classification.</strong></h4><p>Image classification task, if performed successfully, will tell us if the image contains a ‘banana’ or ‘apple’, or both. It assigns a label to an image.</p><p>Object localization task, if performed successfully, will tell us where the objects are in the image by drawing a bounding box around them.</p><h4><strong>Object detection is nothing but a combination of localization and classification.</strong></h4><p>In other words, it is the problem of finding and classifying a variable number of objects on an image.</p><p>So how does one approach with the challenge of object detection?</p><p>Classical Machine Learning approaches which used to work in realtime are in-fact, used today as well. One of them is<a href="https://en.wikipedia.org/wiki/Viola%E2%80%93Jones_object_detection_framework"> Viola-Jones framework</a>, an algorithm which uses<a href="https://en.wikipedia.org/wiki/Haar-like_feature"> Haar Features</a> for real-time face detection.</p><p>However, with the development of deep learning approaches, algorithms now have become much more robust in understanding not just spatial, but contextual information as well.</p><p>For example, consider a human body. Spatial features would include figuring out what a head looks like, or what a leg looks like. Contextual features would include information such as head is connected to neck, or arm is connected to the torso, and likewise.</p><h3>Existing Network Architectures:</h3><p>One of the best performing deep learning approaches is the RCNN family. It was first proposed in 2013, and stands for <em>Region based CNNs</em>.</p><p><strong>How does R-CNN work?</strong></p><p>It’s a multi-stage pipeline which outputs a label as well as a bounding box to a variable number of objects.</p><p>The first stage can broadly be referred to as <em>Image Segmentation</em> task. The purpose of this stage is to find out <em>Region Proposals, </em>or in simple words, parts of interest in the image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*TecJ5eqScXintCzD" /><figcaption>Source:<a href="https://devblogs.nvidia.com/wp-content/uploads/2016/11/figure16.png"> Nvidia blog</a></figcaption></figure><p>Now the further pipeline will be focused only on the regions which the first stage outputs. The algorithm used in the paper is as follows:</p><p>Selective Search:</p><p>1. Generate initial sub-segmentation, we generate many candidate regions</p><p>2. Use greedy algorithm to recursively combine similar regions into larger ones</p><p>3. Use the generated regions to produce the final candidate region proposals</p><p>However proposed to be optimal, this algorithm is slower than most of the other deep learning approaches, two of the main reasons include the <em>greedy approach </em>as<em> </em>there’s no <strong>learning</strong> involved, and the segmented regions are classified one at a time, requiring a lot of computation.</p><p>To tackle the computation required for classification of objects one at a time, <strong>Fast R-CNN</strong> was introduced in 2015, which used convolution outputs instead of fully connected layers, sharing a lot of computation.</p><h3><strong>How does Fast R-CNN work?</strong></h3><p>Let’s take an example, a normal Image Classification network architecture would look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*7VSqsNJ4-8loy7j-" /></figure><p>Now the same fully connected layers can be modified into Convolution layers, which in turn gives us the benefit of the shared computation.</p><p><em>[Note that the softmax layer after the above fully connected layer could be reproduced as an activation for the below convolution layer as well, but this isn’t done. Instead of binary outputs, a feature map is generated.]</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*VusSXVZD4XIwdzUk" /></figure><p>Source:<a href="https://www.coursera.org/specializations/deep-learning"> Deep Learning Specialization by Coursera</a></p><p>This solution was widely accepted for other areas of computer vision research as well. Having solved the classification of objects one at a time, this algorithm is yet again slow.</p><p><strong>Faster R-CNN comes to rescue!</strong></p><p>The selective algorithm in R-CNN is a “fixed” algorithm. Thereby a computer wouldn’t <em>learn</em> at this stage, and this might lead to bad candidates for region proposals.</p><p>Also, when the runtimes were compared, it was realized that region proposals were the bottlenecks in the performance of the family of algorithms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*5h8zu4uHK4SSB0Ek" /><figcaption>Source:<a href="https://missinglink.ai/guides/convolutional-neural-networks/faster-r-cnn-detecting-objects-without-wait/"> missinglink.ai</a></figcaption></figure><p><strong>So how does Faster R-CNN promised to be a better candidate?</strong></p><p>Instead of using the selective search algorithm on the feature map to identify the region proposals, a separate network was used to predict the region proposals. That, in turn, eliminated the selective search algorithm and led the overall network to learn the region proposals too.</p><p>And since it was implemented by convolutions, this algorithm became <strong>the fastest and most accurate object detection algorithm among its family.</strong></p><p>But let’s not hatch the eggs before they’re born, let’s look at a speed vs<a href="https://medium.com/@jonathan_hui/map-mean-average-precision-for-object-detection-45c121a31173"> mAP</a>(mean Average Precision) graph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*zkR_ozM4cBkL4BF3" /></figure><blockquote>Yolo inferences much faster!</blockquote><p>Why is that? Let’s compare the difference between YOLO and RCNN:</p><p>YOLO and Faster R-CNN both share some similarities. They both use an anchor box based network structure, and both use bounding both regression.</p><h3>How are they different?</h3><p>Thing that makes YOLO differ from Faster R-CNN is that it makes classification and bounding box regression at the same time.</p><p>However, Yolo does have it’s drawback in object detection. It doesn’t generalize well when objects in the image show rare aspects of ratio.</p><p>Faster R-CNN on the other hand, does detect small objects well, however it fails to do real-time detection with its two step architecture.</p><p>With that said, let’s jump onto setting up Yolov4 on a local machine, because it can be trained on a local not-so-expensive gpu.</p><h3>Training on a custom data set with YOLO v4</h3><p>Let’s start by downloading cuda and cudnn libraries, for GPU acceleration.</p><p><em>Note: Currently only NVIDIA gpus are supported, </em><a href="https://developer.nvidia.com/cuda-gpus"><em>https://developer.nvidia.com/cuda-gpus</em></a><em> check this link to see if your GPU is supported.</em></p><ol><li>Go to <a href="https://developer.nvidia.com/cuda-toolkit-archive">https://developer.nvidia.com/cuda-toolkit-archive</a>, download the 10.0 toolkit. If you’re on Linux, download the <strong>runfile(local)</strong> version.</li><li>While downloading, open a terminal and type the following command:</li></ol><pre>nvidia-smi</pre><p>If you get a nice structured layout with your GPU mentioned, in the following step, type ’n’ when the installer asks to install graphics drivers.</p><ol><li>After the download completes, open up a terminal to the location you’ve downloaded and run the following command:</li></ol><pre>sudo sh &lt;the file you downloaded&gt;</pre><p><em>Make sure you type ‘y’ when the installer asks to create a symlink with /usr/local/cuda</em></p><ol><li>After completion, the installer would ask you to add /usr/local/cuda to your PATH, and LD_LIBRARY_PATH environment variables.</li><li>If you’re on Bash shell, just type:</li></ol><pre>export PATH=/usr/local/cuda/bin:$PATH</pre><pre>export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH</pre><p>Alternatively, you could add these commands to <strong>~/.bashrc </strong>for everytime you open a terminal it’ll be automatically exported for you.</p><p>We’re done installing the cuda and cudnn libraries!</p><p>Let’s install OpenCV, to boost the data augmention speeds.</p><ol><li>Download the source files from OpenCV releases, you can use the latest 4.3.0 here: <a href="https://opencv.org/releases/">https://opencv.org/releases/</a></li><li>Download the source files from OpenCV contrib repository, here’s the link:</li></ol><p><a href="https://github.com/opencv/opencv_contrib/releases">https://github.com/opencv/opencv_contrib/releases</a></p><ol><li>Go to the OpenCV downloaded folder, inside the directory, create a folder named ‘build’.</li></ol><pre>cd opencv-x.x.x</pre><pre>mkdir build &amp;&amp; cd build</pre><ol><li>Run the following command to open up the gui of cmake:</li></ol><pre>cmake-gui</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*01iu_qLT6UIkOLgZ" /></figure><p>Initially, it’ll all be empty, press Configure to generate these configurations.</p><ol><li>The only important flag here is to check:</li></ol><p>BUILD_opencv_world</p><ol><li>Set OPENCV_EXTRA_MODULES_PATH to:</li></ol><pre>&lt;path to opencv_contrib&gt;/modules</pre><p>Press Generate.</p><ol><li>Now go to the <em>build</em> folder, type the following command:</li></ol><pre>Make -j8 &lt;where 8 are the number of CPU cores, if you don’t have an idea about the number of cores of your CPU, 1 works fine. This is helpful for parallel computation.&gt;</pre><p>Note: This step will take some time.</p><ol><li>After the above command finishes, if you don’t come across any error, type the following command:</li></ol><pre>export OpenCV_DIR=&lt;path to the build folder&gt;</pre><p>That’s it! We have completed building OpenCV library.</p><p>Till now we have installed the cuda and cudnn libraries for gpu computations, and opencv for faster data augmentations and processing of images and videos.</p><p>Let’s now compile Darknet.</p><p>Heard of TensorFlow, PyTorch? Darknet is yet another amazing framework built over C for computation extensive tasks. We will use Darknet to run inference of yolo.</p><p><strong>Compiling Darknet:</strong></p><ol><li>Clone the darknet repository given by AlexeyAB: <a href="https://github.com/AlexeyAB/darknet">https://github.com/AlexeyAB/darknet</a></li></ol><pre>git clone <a href="https://github.com/AlexeyAB/darknet.git">https://github.com/AlexeyAB/darknet.git</a></pre><pre>cd darknet</pre><pre>To compile it on Linux using CMake:<br>./build.sh</pre><pre>To compile it on Windows using CMake:</pre><pre>./build.ps1</pre><p>Alternatively, we can use make command with modified parameters such as:</p><p>GPU=1 to build with CUDA to accelerate by using GPU (CUDA should be in /usr/local/cuda)</p><p>CUDNN=1 to build with cuDNN v5-v7 to accelerate training by using GPU (cuDNN should be in /usr/local/cudnn)</p><p>CUDNN_HALF=1 to build for Tensor Cores (on Titan V / Tesla V100 / DGX-2 and later) speedup Detection 3x, Training 2x</p><p>OPENCV=1 to build with OpenCV 4.x/3.x/2.4.x — allows to detect on video files and video streams from network cameras or web-cams</p><p>DEBUG=1 to build debug version of Yolo</p><p>OPENMP=1 to build with OpenMP support to accelerate Yolo by using multi-core CPU</p><p>Where to make these changes? Simply edit the ‘Makefile’ using a text editor.</p><ol><li>If that runs successfully, there will be a binary named ‘darknet’, which we will use to run inference.</li></ol><p>We’re done!</p><p><strong>Training Yolov4 on coco dataset:</strong></p><p>Now that we have Darknet compiled, we’ll use it to train Yolov4 architecture on custom dataset.</p><p>Training-prerequisites:</p><p>Let’s assume we have a folder containing some images, upon which we will train the architecture.</p><ol><li>If the data is unlabeled, i.e. the ground truths are not available, we can use the GUI utility by AlexeyAB: <a href="https://github.com/AlexeyAB/Yolo_mark">https://github.com/AlexeyAB/Yolo_mark</a></li><li>If the data is labeled, we convert it to the format the pipeline accepts.</li><li>The format is as follows:</li></ol><p>&lt;object-class&gt; &lt;x_center&gt; &lt;y_center&gt; &lt;width&gt; &lt;height&gt;</p><p>object-class is the id to which the class is labeled (more on that below)</p><p>The rest 4 parameters are relative to the image size.</p><p><strong>So they must be in range of 0–1.</strong></p><p>Note: x_center and y_center are the center point of the box.</p><ol><li>For example: Suppose we have an image named <strong>img1.jpg</strong>, w.r.t to the image, <strong>img1.txt</strong> will be created using the utility, or already labeled data to be converted to the wanted format. The content of img1.txt should be like:</li></ol><pre>1 0.111 0.112 0.113 0.114</pre><pre>0 0.121 0.122 0.123 0.124</pre><pre>1 0.131 0.132 0.133 0.134</pre><p>3 lines denote the image contains 3 objects, where 2 of them belong to class 1.</p><ol><li>Copy the contents of yolo4-custom.cfg located at <em>&lt;darknet repository&gt;/cfg/</em> to a new file named as yolo-obj.cfg</li><li>Edit the content as follows:</li></ol><ul><li>Change <strong>max_batches</strong> to (classes*2000), if we have 4 classes, max_batches=8000.</li></ul><p>Note: Make sure the number of images in training data is larger than max_batches</p><ul><li>Change <strong>steps</strong> to (0.8*max_batches, 0.9*max_batches), if we have 4 classes steps=6400,7200.</li><li>Network size <strong>width</strong>=416, <strong>height</strong>=416 (or any multiple of 32)</li><li>Modify <strong>classes</strong>=80 to classes=&lt;number of classes&gt;, if we have 4 classes, classes=4.</li></ul><p>Note: This needs to be done for all the 3 layers of YOLO. (Find and Replace works wonders here)</p><ul><li>Modify <strong>filters</strong>=255 to filters=(classes + 5)x3 in the convolutional layers immediately before each 3 yolo layers, if we have 4 classes, filters=27</li></ul><ol><li>Create file <strong>obj.names</strong> in the directory <em>&lt;darknet repository&gt;/build/darknet/x64/data/</em>, with objects names — each in a new line.</li><li>Create file <strong>obj.data</strong> in the directory <em>&lt;darknet repository&gt;/build/darknet/x64/data/</em> with following data:</li></ol><p><strong>classes = &lt;number of classes&gt; </strong>for eg, classes = 4</p><p><strong>train = data/train.txt</strong></p><p><strong>valid = data/test.txt</strong></p><p><strong>names = data/obj.names</strong></p><p><strong>backup = backup/</strong></p><ol><li>Place the content of the directory where the images and their labels in .txt format are stored, to <em>&lt;darknet repository&gt;/build/darknet/x64/data/obj/</em></li><li>Create file <strong>train.txt</strong> in directory <em>&lt;darknet repository&gt;/build/darknet/x64/data/</em>, with filenames of the images, each filename in new line, with path relative to <strong><em>darknet binary</em></strong>, for eg., with following content:</li></ol><p><strong>data/obj/img1.jpg</strong></p><p><strong>data/obj/img2.jpg</strong></p><p><strong>data/obj/img3.jpg</strong></p><p>We’re done setting up the training-prerequisites!</p><p>We can now start training, or restore training from the <strong>backup</strong> folder we defined in <strong>obj.data </strong>(see step 8).</p><p>Before training, download the convolution data from here: <a href="https://drive.google.com/file/d/1JKF-bdIklxOOVy-2Cr5qdvjgGpmGfcbp/view">https://drive.google.com/file/d/1JKF-bdIklxOOVy-2Cr5qdvjgGpmGfcbp/view</a></p><p>Place it in <em>&lt;darknet repository&gt;/build/darknet/x64/</em></p><p>Open up a terminal and type:</p><pre><strong>./darknet detector train data/obj.data yolo-obj.cfg yolov4.conv.137</strong></pre><ul><li>File yolo-obj_last.weights will be saved to the &lt;darknet repository&gt;/build/darknet/x64/backup\ for each 100 iterations</li><li>File yolo-obj_xxxx.weights will be saved to the &lt;darknet repository&gt;/build/darknet/x64/backup\ for each 1000 iterations</li><li>To disable Loss-Window use:</li></ul><pre>./darknet detector train data/obj.data yolo-obj.cfg yolov4.conv.137 <strong>-dont_show</strong></pre><p>To see the mAP &amp; Loss-chart during training on remote server without GUI, use:</p><pre>./darknet detector train data/obj.data yolo-obj.cfg yolov4.conv.137 -dont_show <strong>-mjpeg_port 8090</strong> <strong>-map</strong></pre><p>Now open URL <a href="http://ip-address:8090/">http://ip-address:8090</a> in Chrome/Firefox browser</p><p><strong>We’ve started training!</strong></p><p>Use this command to test the trained model:</p><pre>./darknet detector test data/obj.data yolo-obj.cfg yolo-obj_<strong>8000</strong>.weights</pre><p>Note: 8000 is the training step for which weights have been stored</p><p><strong>Other Implementations:</strong></p><p>Yolov4 has also been written in TensorFlow, which can be converted to TFLite and used in an android application!</p><p>Here is the link: <a href="https://github.com/hunglc007/tensorflow-yolov4-tflite">https://github.com/hunglc007/tensorflow-yolov4-tflite</a></p><p>The TensorFlow version has been issued in PyPi, which can be used to train, infer, and convert to tflite all at the same place!</p><p>Here is the link: <a href="https://pypi.org/project/yolov4/">https://pypi.org/project/yolov4/</a></p><blockquote>Credits: This post was originally created by Aitik Gupta as part of summer internship program @Code Heroku. You can reach him at aitikgupta@gmail.com</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd549b182b78" width="1" height="1" alt=""><hr><p><a href="https://medium.com/code-heroku/object-detection-in-2020-from-rcnns-to-yolov4-fd549b182b78">Object Detection in 2020 —  From RCNNs to YOLOv4</a> was originally published in <a href="https://medium.com/code-heroku">Code Heroku</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Off-The-Shelf VS Custom Software Solutions]]></title>
            <link>https://medium.com/code-heroku/off-the-shelf-vs-custom-software-solutions-ba87534fa3e1?source=rss----3b5048e3b0e5---4</link>
            <guid isPermaLink="false">https://medium.com/p/ba87534fa3e1</guid>
            <category><![CDATA[product-development]]></category>
            <category><![CDATA[power-bi]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[cio]]></category>
            <dc:creator><![CDATA[Code Heroku]]></dc:creator>
            <pubDate>Sun, 28 Jun 2020 09:53:35 GMT</pubDate>
            <atom:updated>2020-06-28T09:55:36.837Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kXQ22995UxVXXfP2TpFkzg.png" /></figure><p>When leaders of an organization try to address a business need using software solutions, they are often faced with two choices -</p><p>1. Use an <em>Off-the-Shelf</em> software; OR</p><p>2. Build a custom software solution which can address their unique business requirements.</p><p>Examples for Off-the-shelf software solutions include: Content Management Systems (CMS) such as SharePoint, Drupal, WordPress which can be configured for your business needs. And its customizable counterpart will be a static website built using simple HTML / JavaScript.</p><p>Similarly in the world of BI / Analytics — Tableau, PowerBI, Qlik are examples for off-the-shelf software whereas its counterpart would be a custom dashboard built using JavaScript charts.</p><h3>Often leaders think that Off-The-Shelf Solutions are less expensive to deploy and thus best suited for small businesses or a product in early stages of development.</h3><p><em>In this post I hope to bust some myths around these off-the-shelf solution so that you can make a more informed decision for your organization!</em></p><p>Consider a mid-size financial consulting company which wants to build a website. The only dynamic sections of their website include — a News Page and a Careers Page that needs to be updated about every month.</p><h3>Scenario 1: Choosing Off-The-Shelf solution</h3><p>Since they need the ability for their HR manager to update the Careers page they choose to use a CMS (User-configurable software) which will apparently make it very easy for a non technical user to update the page. The CEO of ABC Inc, does some initial research and finds that they could use opensource CMS such as WordPress to build their website quickly. They find an expert freelancer to setup their website.</p><p>The developer uses readily available templates, several free plugins this platform offers to speed-up the development. Since there are some unique requirements on the News and Jobs pages, the developer also build some custom plugins to meet these additional requirements. After six months of hard work, the website is finally deployed. The HR manager is happy as they can post jobs without the need to communicate with a developer.</p><p>In these past six months, the developer made sure that updates are regularly applied to the CMS, know vulnerabilities are patched, backups are created and all other nuisances that might occur with a database driven application are taken care of. Everything works great! …. until now!</p><p>One years latter, the original developer of this website no longer works for the company. For the past few months no one has applied patches to the CMS or updated plugins. The news section needs to be updated but due to some recent updates to the CMS, custom plugin no longer work as expected. What do they do now? Hire another Developer to fix the website?</p><p>Note that even though the actual News &amp; Careers page updates were made by a non-technical person, the company might not realize but they always needed a Wordpress / Drupal developer to maintain and update their website, so the initial premise that this solution will allow them to maintain a website without a developer at lower cost was never valid.</p><h3>Scenario 2: Building a Custom Website</h3><p>Now, what if the company opted to create a static HTML website (customizable software). They contract a developer and purchase a website template for few bucks. The website is built in half the time since there is no CMS development overhead involved. Initially the HR manager learns from the developer on how to create and edit jobs page by copy-pasting from the HTML template. After doing this couple of times, they pick up this skill and use tools like online HTML editors to make the process easier. And because this is a static website it needs no patches or significant updates to be applied. Even after a few years when the developer who build this website no longer works for the company, the company is not significantly affected. If any updates are required, It is much easier and cheaper for the company to find a college kid who knows HTML/Javascript compared to finding an expert WordPress / Drupal Developer.</p><p>Bottom Line: Leaders often underestimate the time and expertise it needs to setup and maintain an off-the-shelf solution. Any customization to these products are not straight forward. Often such solutions can’t match the growing needs of your business and you will end up hiring developers to address this.</p><h3>Myth: This choice depends on the problem at hand</h3><p>You might also think that this choice solely depends on the problem at hand and your specific requirements. I would argue that a bigger factor in this decision is what business are you in? Ask your self -</p><blockquote>Is the problem you are solving, directly related to your core business?</blockquote><p>If the answer to this is yes, then you will be better served with a custom software solution in the longer run.</p><p>In cases when organization try to use off-the-shelf solution as part of their core product, for example a Health Care company using embedded PowerBI dashboards in their product, they are convinced that because these solutions can be configured to meet their needs, this will help to reduce their development time.</p><p>But what they don’t consider is that often this comes at the cost of added complexity, reduced maintainability and thus eventually added cost.</p><p>In fact, as you see in the plot below, I believe the end-user value degrades after a point with added configuration. To deliver high end-user value you must find the right balance between software’s maintainability and configurability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NLkYqFh7rNCyGJlm" /></figure><p>I am not against using configurable software platforms whether enterprise or opensource. If a SaS product satisfies all your requirements you should try it out. But If this is part of your core product / offering, It is in the best interest to build tools using minimal technology stack and thus keep the software at really low maintenance. I would suggest that you build an evaluation model for your company that will help you determine the best course of action.</p><h4>If you would like to get a free copy of an evaluation model we recommend to choose between Off-the-Shelf and Custom Software Development, you can email me at mihirt@codeheroku.com</h4><p>Do you have any stories from your organization that you would like to share? I would love to read them in the comments section. If you enjoyed reading this post, please do consider sharing this with your network!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba87534fa3e1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/code-heroku/off-the-shelf-vs-custom-software-solutions-ba87534fa3e1">Off-The-Shelf VS Custom Software Solutions</a> was originally published in <a href="https://medium.com/code-heroku">Code Heroku</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Turn Your Machine Learning Scripts into Projects You Can Demo?]]></title>
            <link>https://medium.com/code-heroku/how-to-turn-your-machine-learning-scripts-into-projects-you-can-demo-cbc5611ca442?source=rss----3b5048e3b0e5---4</link>
            <guid isPermaLink="false">https://medium.com/p/cbc5611ca442</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[recommendation-engine]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[machine-learning-projects]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Code Heroku]]></dc:creator>
            <pubDate>Mon, 09 Sep 2019 09:01:49 GMT</pubDate>
            <atom:updated>2020-07-16T08:11:25.876Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/871/1*9roJ0xrFi3L5XPITBFmAqA.png" /></figure><p>Recently we published an article and video on Building a Movie Recommendation Engine and one of the most common questions that I received was — “How do I turn this script/notebook into a project that I can demo?”. So, I decided to build a front-end for our recommendation engine. You can try it out here:</p><blockquote><a href="http://www.codeheroku.com/static/movies/index.html">http://www.codeheroku.com/static/movies/index.html</a></blockquote><blockquote>Part 2 of this series is now published: <a href="https://medium.com/code-heroku/part-2-how-to-turn-your-machine-learning-scripts-into-projects-you-can-demo-467b3acff041">Click Here</a></blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F3ecNC-So0r4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D3ecNC-So0r4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F3ecNC-So0r4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/297d28a518183f96eebd97ccb4a854c9/href">https://medium.com/media/297d28a518183f96eebd97ccb4a854c9/href</a></iframe><p>In our previous videos we have covered building a recommendation system using Collaborative Filtering. In this post, I would like to start from a point assuming that you have built a recommendation model and now want to create a web based user interface for it.</p><pre><strong>##Ingredients For this Recipe:</strong></pre><pre>1. Web2py Framework : <a href="https://mdipierro.pythonanywhere.com/examples/static/nightly/web2py_src.zip">Download Here</a><br>2. Recommendations Model File:  <a href="http://www.codeheroku.com/static/movies/item_similarity_df.csv">item_similarity_df.csv</a><br>3. Test HTML Page: <a href="http://www.codeheroku.com/static/movies/test.html">Download Here</a> <br>4. Completed Application Code with all above: <a href="https://drive.google.com/file/d/13coIj5qQCGksnh6GpQbXVWtImgtLX2GY/view?usp=sharing">Download Here</a></pre><p>Let’s go through high-level steps for building a user interface for your ML application:</p><ul><li>Selection of Tools and Languages</li><li>Building APIs/Services to expose your ML model</li><li>Building Client-side Web / Mobile App</li></ul><h3>Step 1: Selection of Tools and Programming Languages</h3><p>Whether you are a veteran software developer or somebody who is just starting out, this is a crucial step that will determine how much time and efforts you will need to build the application you envisioned.</p><blockquote>You must use the right tools for the right purpose!</blockquote><p>Assuming you have built your ML model in Python, it makes sense to use this powerful language for your web applications/API engine as well. It also means that you don’t have to port any of your code to any other programming language. Here’s my recommendation on a stack that you can’t go wrong with:</p><blockquote>Build APIs using a Python based framework (Flask, web2py, Django) and</blockquote><blockquote>Build your Front-end using JavaScript with VueJS or ReactJS</blockquote><p>If you love Python, you would like JavaScript. Using reactive packages like VueJS/ReactJS is optional but will make your life simpler especially to build a single page app as you won’t have to worry about refreshing pages and passing parameters from one page to another!</p><p><strong><em>In this post we will be using a Python based web framework called web2py for building APIs and VueJS for building reactive HTML front-end.</em></strong></p><h3>Step 2: Building APIs/Services to expose your ML model</h3><p>Our Machine Learning models will be exposed to outside world using a web service. We can then consume this service in a mobile / web application to make our apps smarter. We will create a service called getRecommendations</p><pre>@service.json<br><strong>def getRecommendations</strong>(watched_movies):<br>     <br>     ##Magic with our ML model goes here</pre><pre><strong>return</strong> recommended_movies</pre><p>This is a JSON API that takes in movies that a user has watched/rated in the past and returns a list of recommended movies. web2py is great for building APIs with minimal lines of code. So let’s go ahead and <a href="https://mdipierro.pythonanywhere.com/examples/static/nightly/web2py_src.zip">Download web2py from here</a>.</p><p>Once you have extracted the archive, you should see the following directory structure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*flxxo0AFQGp8suTmmeuVHA.png" /><figcaption>web2py directory structure</figcaption></figure><p>A quick note on some important folders / files:</p><ol><li>Applications: All your web2py apps/projects are contained in this directory. You can backup your projects and install new ones by simply copy-pasting your project folder inside this directory.</li><li>web2py.py: We can start our web server by running this script:</li></ol><pre><strong>&gt;&gt; cd Downloads/web2py_src<br>&gt;&gt; python web2py.py</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/369/1*lLLxCh0rc0UJGDLpWRRjgA.png" /></figure><p>Choose a password of your choice and hit start server. Next, go to admin, use the password you just used and then create a simple application called “movies”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*blFK8oEGtfjHh5nsoqQiKw.gif" /></figure><p>Now, if we want we could write the rest of the code in the buit-in web based editor that web2py provides. But I am not a huge fan of this editor! So, let’s go back to the web2py directory and edit the app in our favorite code editor— Sublime/Atom/VS Code.</p><p>Head to the applications folder and you should see a new folder created for our movies app. Here’s the directory structure of our web2py movies app:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*_T39P7kkDAiAguxqudWx6w.png" /></figure><p>Web2py inherently supports MVC (Model View Controller) architecture.</p><ul><li><strong>M</strong>odels: define database schema and validations</li><li><strong>V</strong>iews: hosts your HTML code</li><li><strong>C</strong>ontrollers: Python code where you will write business logic and manages interactions between front-end and the database.</li></ul><p>Even though MVC is a good architectural pattern, the world has moved on to using micro-services architecture so that our front-end is independent of the controller logic (mostly). Right now, we will be focusing only on the controllers where we will build our APIs. Open the controller file located at applications/movies/controllers/default.py</p><p>First, let me show you how easy it is to use web2py and create a simple API. let’s create an API called <em>myapi </em>that takes in two parameters a &amp; b and returns a string concatenation of these two parameters:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8c123630651495ab90d584efdeea60a1/href">https://medium.com/media/8c123630651495ab90d584efdeea60a1/href</a></iframe><p>Ok, don’t worry yet — In lines 1 through 5 we wrote some web2py specific code which is used to initialize web services in our app. The @service.json decorator tells web2py that we want to expose myapi as a JSON web service. You can also try @service.xml or @service.csv to build services in other protocols.</p><p>Now, let’s call our API from the browser and see if it actually works. Go to the following URL in your browser: <a href="http://127.0.0.1:8000/movies/default/call/json/myapi?a=3&amp;b=2">http://127.0.0.1:8000/movies/default/call/json/myapi?a=3&amp;b=2</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/1*nYt5aEkOOUdiW9MLfnkMWw.png" /><figcaption>Testing our simple API by passing two parameters and getting a concat of two strings in the result.</figcaption></figure><p>Congratulations! you have just created your first API!</p><p>Now its time to build our Recommendations API. Recall our API signature look something like this:</p><pre>@service.json<br><strong>def getRecommendations</strong>(watched_movies):<br>     <br>     ##Magic with our ML model goes here</pre><pre><strong>return</strong> recommended_movies</pre><p>In the first step, we will load the recommendations model that we built using Collaborative Filtering in our app. We have created an “item_similarity_df” in our collaborative filtering tutorial. <a href="http://www.codeheroku.com/static/movies/item_similarity_df.csv">Click Here to download this CSV file</a> and place it in your applications/movies/static folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*FLP7M9k3lh8qJNE6nk0E1A.png" /></figure><p>Since we do not want to load the data frame every time someone uses our app we will cache it using web2py’s cache.ram method.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a1f719b316be00084acb2ddd563e69ef/href">https://medium.com/media/a1f719b316be00084acb2ddd563e69ef/href</a></iframe><p>Next up, comes the part we were waiting for: Building the recommendation API. This is pretty much the same method we created during <a href="https://youtu.be/3ecNC-So0r4">collaborative filtering class</a> with minute changes on the data type of watched_movies. We have also included a helper method that excludes movies that user has already watched/rated from the list of recommendations.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/917bdd2fa27ff6b0dbef0e396ad495ce/href">https://medium.com/media/917bdd2fa27ff6b0dbef0e396ad495ce/href</a></iframe><h3>Testing Your Recommendations API</h3><p>To test our API we need to send a POST request to the API located at /movies/default/call/json/get_recommendations with watched_movies parameter.</p><p>To make things easier for you, we have created a test HTML page that sends this request. Go ahead and <a href="http://www.codeheroku.com/static/movies/test.html">download it from here</a> and save this page in your applications/movies/static folder. Right click &gt; Save page as (uhumm..uhumm)</p><p>You can access this page on your browser at URL <a href="http://localhost:8000/movies/static/test.html">http://127.0.0.1:8000/movies/static/test.html</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A-DzzgWu_6uzmcvQOV8Fdg.png" /><figcaption>This test.html page sends a post request with a sample movie the user has rated and returns recommended movies for the user.</figcaption></figure><p>If you see a page like this that means your API is working as expected and you are all set for the next part of this tutorial.</p><p>In the next part we will build the HTML/JavaScript based front-end using VueJS to call this API and display recommended movies for our users in a nice looking interface. We will decide to publish the next part based on the feedback and number of responses we get for this post. So if you are keen on getting the next part, please do comment on this post or reach out to us on WhatsApp.</p><blockquote>Part 2 of this series is now published: <a href="https://medium.com/code-heroku/part-2-how-to-turn-your-machine-learning-scripts-into-projects-you-can-demo-467b3acff041">Click Here</a></blockquote><pre><strong>##Ingredients For this Recipe:</strong></pre><pre>1. Web2py Framework : <a href="https://mdipierro.pythonanywhere.com/examples/static/nightly/web2py_src.zip">Download Here</a><br>2. Recommendations Model File:  <a href="http://www.codeheroku.com/static/movies/item_similarity_df.csv">item_similarity_df.csv</a><br>3. Test HTML Page: <a href="http://www.codeheroku.com/static/movies/test.html">Download Here</a> <br>4. Completed Application Code with all above: <a href="https://drive.google.com/file/d/13coIj5qQCGksnh6GpQbXVWtImgtLX2GY/view?usp=sharing">Download Here</a></pre><blockquote>If you like this post, please do share it with your network and give it some claps! This will encourage us to produce more such content.</blockquote><blockquote>If you get stuck somewhere while building this project you can reach out to us via email: hello@codeheroku.com or on WhatsApp: +91–9967578720</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cbc5611ca442" width="1" height="1" alt=""><hr><p><a href="https://medium.com/code-heroku/how-to-turn-your-machine-learning-scripts-into-projects-you-can-demo-cbc5611ca442">How to Turn Your Machine Learning Scripts into Projects You Can Demo?</a> was originally published in <a href="https://medium.com/code-heroku">Code Heroku</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Indian Premier League Analysis]]></title>
            <link>https://medium.com/code-heroku/indian-premier-league-analysis-c939e755e45b?source=rss----3b5048e3b0e5---4</link>
            <guid isPermaLink="false">https://medium.com/p/c939e755e45b</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[cricket]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[ipl]]></category>
            <dc:creator><![CDATA[Sudeepto Mukhopadhyay]]></dc:creator>
            <pubDate>Fri, 14 Jun 2019 06:28:12 GMT</pubDate>
            <atom:updated>2019-06-15T08:01:41.827Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*MsB7NkZA8CiiPmqnhWaTTA.jpeg" /></figure><p>The Indian Premier League (IPL) is a professional Twenty20 cricket league in India contested during March or April and May of every year by eight teams representing eight different cities in India. The league was founded by the Board of Control for Cricket in India (BCCI) in 2008. IPL has an exclusive window in ICC Future Tours Programme.</p><p>The IPL is the most-attended cricket league in the world and in 2014 ranked sixth by average attendance among all sports leagues. In 2010, the IPL became the first sporting event in the world to be broadcast live on YouTube. The brand value of IPL in 2018 was US$6.3 billion, according to Duff &amp; Phelps. According to BCCI, the 2015 IPL season contributed ₹11.5 billion to the GDP of the Indian economy.</p><p>There have been twelve seasons of the IPL tournament. The current IPL title holders are the Mumbai Indians, who won the 2019 season.</p><p>Let us go straight on to the topic of discussion. We will analyse the IPL data set and get some good insights from it. For the Analysis purpose, I am using <strong><em>Jupyter Notebook</em> </strong>with <strong><em>Python</em></strong>.</p><p>Before we start, here is a link to the python notebook:<br><a href="https://github.com/sudeepto22/ipl-stats/blob/master/IPL%20Analysis.ipynb"><strong><em>IPL Analysis</em></strong></a></p><h3><strong><em>Import necessary Packages:</em></strong></h3><p>Let’s import all the packages we require for this Analysis. We need a package where we can do numerical operations; we need a way to read a data set and also we need a package to visualize the analysis.</p><pre>import numpy as np<br>import pandas as pd<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br>sns.set_style(&quot;darkgrid&quot;)</pre><h3>Read the data sets</h3><p>We have 2 data sets, matches.csv and deliveries.csv in ipl folder. Let’s read that csv and put it in pandas DataFrame.</p><pre>df_matches = pd.read_csv(‘ipl/matches.csv’)<br>df_deliveries = pd.read_csv(‘ipl/deliveries.csv’)</pre><p>Take a quick look on both the data sets.</p><pre>df_matches.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rBWlgDD9TMeFUQbfYREOEw.png" /></figure><pre>df_deliveries.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3fuht9pk3RlUpW0NaAjaDQ.png" /></figure><p>So, we now have the whole setup for working on the data. We will now get to some analysis with this data. We will be analyzing the following:</p><ul><li>Most dismissals by a Wicket Keeper</li><li>Toss Call for a given Venue</li><li>Number of wickets taken by a bowler</li><li>Highest and lowest runs in IPL History</li><li>Average runs in all IPL seasons</li></ul><h3>Most dismissals by a Wicket Keeper</h3><p>Before working on this analysis, let’s get our domain knowledge clear. The first question to ask is, who a wicket-keeper is? I know that sounds dumb, but we need to drill down to the minute information possible.</p><blockquote>The <strong>wicket-keeper</strong> in the sport of cricket is the player on the fielding side who stands behind the wicket or stumps being watchful of the batsman and be ready to take a catch, stump the batsman out and run out a batsman when occasion arises.<br>- Wikipedia</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/151/1*bhAkd_4c7dhfUJVJ4Chm3A.jpeg" /><figcaption>MS Dhoni behind the wickets</figcaption></figure><p>As we can see that, a wicket-keeper is responsible for stumping, catching and run out a batsman. Out of these, running out a batsman is a team wicket and not an individual one. So, we can say that performance of a wicket-keeper depends upon the number of stumping and catches. Of course, we can argue that the wicket-keeper is also responsible for saving boundaries, etc. But here, we are looking for the wicket-keepers with maximum stumping and catches.</p><p>So, from the data set, we get the unique kinds of dismissals</p><pre>df_deliveries[‘dismissal_kind’].unique()</pre><pre>array([nan, &#39;caught&#39;, &#39;bowled&#39;, &#39;run out&#39;, &#39;lbw&#39;, &#39;caught and bowled&#39;,&#39;stumped&#39;, &#39;retired hurt&#39;, &#39;hit wicket&#39;, &#39;obstructing the field&#39;],dtype=object)</pre><p>So, we need to extract those rows, having dismissal kind as stumped as of now. If we take the rows having dismissal kind as caught, then they can be any fielder and not just a wicket-keeper. So, we need to first get the wicket-keepers list by seeing at the stumping as only a wicket-keeper is responsible for stumping. After that we can extract all the wicket-keepers and check how many catches the wicket-keeper has caught.</p><p>Let’s take the dismissal kind as <em>stumped</em></p><pre><strong># Take the dismissal kind as stumped and count the number of stumping a fielder(wicket-keeper) has done<br></strong>wicket_keeper_stumping = df_deliveries[df_deliveries[‘dismissal_kind’]==’stumped’][‘fielder’].value_counts().reset_index()</pre><pre><strong># Renaming the column accordingly</strong><br>wicket_keeper_stumping.rename(columns={‘index’:’wicket_keeper’,’fielder’:’stumping’}, inplace=True)</pre><pre><strong># Get first 5 rows</strong><br>wicket_keeper_stumping.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/289/1*121vFS-j7yKOqXOJgec86w.png" /></figure><p>From the above DataFrame, we got to know how many wicket-keepers are there and hence we can list out all such wicket-keepers.</p><pre>wicket_keeper_list = wicket_keeper_stumping[‘wicket_keeper’].tolist()<br>wicket_keeper_list</pre><pre><br><strong># Output:</strong><br>[‘MS Dhoni’, ‘RV Uthappa’, ‘KD Karthik’, ‘WP Saha’, ‘PA Patel’, ‘AC ilchrist’, ‘NV Ojha’, ‘RR Pant’, ‘KC Sangakkara’, ‘Q de Kock’, ‘MS isla’, ‘SP Goswami’, ‘AB de Villiers’, ‘KM Jadhav’, ‘BB McCullum’, ‘AP Tare’, ‘DH Yagnik’, ‘KL Rahul’, ‘SV Samson’, ‘H Klaasen’, Kamran kmal’, ‘YV Takawale’, ‘CM Gautam’, ‘MV Boucher’, ‘PR Shah’, ‘KB Arun arthik’, ‘N Saini’, ‘M Rawat’, ‘J Bairstow’, ‘Vishnu Vinod’, ‘DJ acobs’, ‘AT Rayudu’, ‘Ishan Kishan’, ‘Gurkeerat Singh’, ‘KH evdhar’]</pre><p>Now, we can get the number of catches by wicket-keeper as we have the complete list of the same.</p><pre><strong># Take the dismissal kind as caught and fielder should be present in the wicket_keeper_list</strong><br>wicket_keeper_catches = df_deliveries[(df_deliveries[‘dismissal_kind’]==’caught’) &amp; (df_deliveries[‘fielder’].isin(wicket_keeper_list))][‘fielder’].value_counts().reset_index()</pre><pre><strong># Rename the columns accordingly</strong><br>wicket_keeper_catches.rename(columns={‘index’:’wicket_keeper’,’fielder’:’catches’}, inplace=True)</pre><pre><strong># Get first 5 rows</strong><br>wicket_keeper_catches.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/281/1*_7wwl3jUSpzxKInMYKeEIQ.png" /></figure><p>We now need to merge these two new data sets with wicket-keeper as key. We need to do a left outer join on wicket_keeper_stumping and wicket_keeper_catches and get the new DataFrame. There are some NaN values as well; we need to make it to 0 and convert its type to integer. Also, we can calculate the total number of stumping and catches by a wicket-keeper and sort it according to this total.</p><pre><strong># Left Outer Join on wicket_keeper_stumping and</strong> <strong>wicket_keeper_catches</strong><br>wicket_keeper_dismissals = pd.merge(wicket_keeper_stumping,wicket_keeper_catches,on=’wicket_keeper’,how=’outer’)</pre><pre><strong># Convert the NaN values to 0</strong><br>wicket_keeper_dismissals.catches = wicket_keeper_dismissals.catches.fillna(0)</pre><pre><strong># Convert the data-type of catches to integer (from float)</strong><br>wicket_keeper_dismissals.catches = wicket_keeper_dismissals.catches.astype(int)</pre><pre><strong># Get the total by adding number of stumping and catches</strong><br>wicket_keeper_dismissals[‘total’] = wicket_keeper_dismissals.stumping + wicket_keeper_dismissals.catches</pre><pre><strong># Sort the DataFrame by total</strong><br>wicket_keeper_dismissals = wicket_keeper_dismissals.sort_values([‘total’], ascending=False).reset_index()</pre><pre><strong># Drop the index column</strong><br>wicket_keeper_dismissals.drop(‘index’, axis=1, inplace=True)</pre><pre><strong># Get first 5 rows</strong><br>wicket_keeper_dismissals.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/1*bhAI-ivf0pX34ghwPSFH-A.png" /></figure><p>Also, let us visualize the information we have derived with some kind of charts. Let’s visualize the to 10 wicket-keepers with respect to the dismissals as stumping and catches.</p><pre>wicket_keeper_dismissals_top10 = wicket_keeper_dismissals.head(10)</pre><pre>plt.figure(figsize=(10,4))</pre><pre>plt.bar(wicket_keeper_dismissals_top10[&#39;wicket_keeper&#39;],wicket_keeper_dismissals_top10[&#39;total&#39;], label=&#39;Total&#39;)</pre><pre>plt.bar(wicket_keeper_dismissals_top10[&#39;wicket_keeper&#39;],wicket_keeper_dismissals_top10[&#39;stumping&#39;], label=&#39;Stumping&#39;)</pre><pre>plt.xticks(rotation=90)<br>plt.title(&#39;Wicket Keeper Stats&#39;)<br>plt.xlabel(&#39;Wicket Keepers&#39;)<br>plt.ylabel(&#39;Dissmissal&#39;)<br>plt.legend(loc=&#39;upper right&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*edmIr1VxBexrYhqJZdtdUg.png" /></figure><p>From the above chart, we can infer the following:<br>KD Karthik, MS Dhoni and RV Uthappa have been a great wicketkeeper in IPL. Also, we can say that, when it comes to stumping, MS Dhoni performs the best followed by RV Uthappa and KD Karthik.</p><h3>Toss Call for a given Venue</h3><p>One of the important decisions before playing cricket is toss. Of course, a mere toss decision would not affect a player’s caliber or performance. Still, let us find out how a toss decision in each city impacts the game. This is just an analysis and we cannot and should not consider it as prediction as we have not kept into consideration the predictive models.</p><p>Let’s create a new DataFrame which contains all the useful information related to toss, viz. city, toss_decision, toss_winner, winner and the unique city names</p><pre>df_toss = df_matches[[‘city’,’toss_decision’,’toss_winner’,’winner’]]</pre><pre><strong># Function to get the unique cities in a particular data frame</strong><br>def get_unique_citites(df):<br>    city_dict = df.groupby([‘city’]).groups<br>    city_list = []<br>    for city,stats in city_dict.items():<br>        city_list.append(city)<br>    print(city_list)<br>get_unique_citites(df_toss)</pre><pre><br><strong># Output</strong><br>[&#39;Abu Dhabi&#39;, &#39;Ahmedabad&#39;, &#39;Bangalore&#39;, &#39;Bengaluru&#39;, &#39;Bloemfontein&#39;, &#39;Cape Town&#39;, &#39;Centurion&#39;, &#39;Chandigarh&#39;, &#39;Chennai&#39;, &#39;Cuttack&#39;, &#39;Delhi&#39;, &#39;Dharamsala&#39;, &#39;Durban&#39;, &#39;East London&#39;, &#39;Hyderabad&#39;, &#39;Indore&#39;, &#39;Jaipur&#39;, &#39;Johannesburg&#39;, &#39;Kanpur&#39;, &#39;Kimberley&#39;, &#39;Kochi&#39;, &#39;Kolkata&#39;, &#39;Mohali&#39;, &#39;Mumbai&#39;, &#39;Nagpur&#39;, &#39;Port Elizabeth&#39;, &#39;Pune&#39;, &#39;Raipur&#39;, &#39;Rajkot&#39;, &#39;Ranchi&#39;, &#39;Sharjah&#39;, &#39;Visakhapatnam&#39;]</pre><p>From the above list, we can see that Bangalore and Bengaluru are the same city as Bangalore city name is changed to Bengaluru. So, we should update Bangalore to Bengaluru</p><pre><strong># Deep Copy the data frame</strong><br>df_toss = df_toss.copy(deep=True)</pre><pre><strong># Update Bangalore to Bengaluru</strong><br>df_toss.city.loc[df_toss[‘city’]==’Bangalore’] = ‘Bengaluru’<br>get_unique_citites(df_toss)</pre><pre><strong># Output</strong><br>[&#39;Abu Dhabi&#39;, &#39;Ahmedabad&#39;, &#39;Bengaluru&#39;, &#39;Bloemfontein&#39;, &#39;Cape Town&#39;, &#39;Centurion&#39;, &#39;Chandigarh&#39;, &#39;Chennai&#39;, &#39;Cuttack&#39;, &#39;Delhi&#39;, &#39;Dharamsala&#39;, &#39;Durban&#39;, &#39;East London&#39;, &#39;Hyderabad&#39;, &#39;Indore&#39;, &#39;Jaipur&#39;, &#39;Johannesburg&#39;, &#39;Kanpur&#39;, &#39;Kimberley&#39;, &#39;Kochi&#39;, &#39;Kolkata&#39;, &#39;Mohali&#39;, &#39;Mumbai&#39;, &#39;Nagpur&#39;, &#39;Port Elizabeth&#39;, &#39;Pune&#39;, &#39;Raipur&#39;, &#39;Rajkot&#39;, &#39;Ranchi&#39;, &#39;Sharjah&#39;, &#39;Visakhapatnam&#39;]</pre><p>Let’s clean the data now. We need to look for any missing value and handle it. Let us look at the unique winners, we should do this to check if there is a draw and hence will get NaN value in <em>winner</em></p><pre>df_toss.winner.isna().sum()</pre><pre><strong># Output</strong><br>4</pre><p>There are in total 4 draw matches. So, we should remove those rows for analysis</p><pre>df_toss = df_toss[pd.notnull(df_toss[‘winner’])]<br>df_toss.winner.isna().sum()</pre><pre><strong># Output</strong><br>0</pre><p>Let us look at the count of the data frame</p><pre>df_toss.count()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/242/1*U1AGunULyva1PJPhZ7zGXA.png" /></figure><p>We need to check for city column as well and delete those rows too</p><pre>df_toss[df_toss[‘city’].isna()]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*BedXDY8roTHmBesHFxUgtg.png" /></figure><p>Remove those NaN city rows</p><pre>df_toss = df_toss[pd.notnull(df_toss[‘city’])]<br>df_toss.city.isna().sum()</pre><pre># Output<br>0</pre><p>Lets check the count of the data frame now</p><pre>df_toss.count()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/1*oB2syzji3UkTCJt2pMPJnw.png" /></figure><p>Looks perfect…!</p><p><strong><em>Approach </em></strong>: If <em>toss_winner</em> is a <em>winner</em> then <em>toss_decision</em> is <em>true</em> i.e. decision should be same as <em>toss_decision</em> else it should be the opposite</p><pre>def decision(row):<br>    if row[‘toss_winner’] == row[‘winner’]:<br>        val = row[‘toss_decision’]<br>    else:<br>        if row[‘toss_decision’] == ‘bat’:<br>            val = ‘field’<br>        else:<br>            val = ‘bat’<br>    return val</pre><pre><strong># Apply the above function to the toss DataFrame</strong><br>df_toss[&#39;decision&#39;] = df_toss.apply(decision, axis=1)</pre><pre><strong># Get first 5 rows</strong><br>df_toss.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*T6NOXEK90JavfP0SV9fDcw.png" /></figure><p>Now, remove <em>toss_decision</em>, <em>toss_winner</em> and <em>winner</em> from our data set or just take <em>city</em> and <em>decision</em> columns</p><pre>df_toss = df_toss[[‘city’,’decision’]]<br>df_toss.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/245/1*kohs2Pqh60D1mM3CP5ua_A.png" /></figure><p>We now have to use one hot encoder to convert categorical variables</p><pre>df_decision = pd.get_dummies(df_toss.decision)<br>df_decision.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/142/1*b_9-y4DUxl77ubMBy5R7WQ.png" /></figure><p>Concatenate these two data frames and remove the decision field</p><pre>df_toss = pd.concat([df_toss,df_decision], axis=1)<br>df_toss.drop([‘decision’], axis=1, inplace=True)<br>df_toss.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/1*7vwCVnodXWwM5TcONynQlw.png" /></figure><p>We have to now group by the field <em>city </em>and get all the stats like total and percentages.</p><pre><strong># Group by city and then get sum of all the other columns</strong><br>df_toss = df_toss.groupby(‘city’).sum().reset_index()</pre><pre><strong># Get the total matches played</strong><br>df_toss[‘total’] = df_toss.bat+df_toss.field</pre><pre><strong># Calculate batting decision percentage</strong><br>df_toss[‘bat_decision’] = round(((df_toss.bat)/df_toss.total)*100,2)</pre><pre><strong># Calculate feilding decision percentage</strong><br>df_toss[‘field_decision’] = round(((df_toss.field)/df_toss.total)*100,2)</pre><pre><strong># Get first 5 rows</strong><br>df_toss.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*iKG_Em0oQtNY3rJxKx4IXw.png" /></figure><p>Now drop <em>bat</em>, <em>field</em> and <em>total</em> columns from the data frame</p><pre>df_toss.drop([‘bat’,’field’,’total’], axis=1, inplace=True)<br>df_toss</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*aWM0EYs9hDYvoAzfjkmgkg.png" /></figure><p>For a instance, if we choose Kolkata as the city and you field first, there is a 61% chance that you will win the game. Again, we are not sure about it, but from the history data, we can give this statement.</p><h3>Number of wickets taken by a bowler</h3><p>Again, like how we did the analysis for wicket-keeper, we need some domain knowledge. We need to know in how many ways a bowler can take wickets, his own individual wicket.</p><blockquote>Bowling, in cricket, is the action of propelling the ball toward the wicket defended by a batsman. A player skilled at bowling is called a bowler; a bowler who is also a competent batsman is known as an all-rounder.<br>- Wikipedia</blockquote><p>A bowler is said to have taken wickets for himself when the dismissal kinds are as follows:</p><ul><li>caught</li><li>bowled</li><li>lbw</li><li>caught and bowled</li><li>stumped</li></ul><p>We need to create a new DataFrame say bowlers, and get the count of the above dismissal grouped by a player(bowler). Our target is to get all the bowlers who has taken 100 or more wickets.</p><pre>bowlers = df_deliveries[[‘bowler’,’dismissal_kind’]]</pre><pre>bowlers = bowlers[(bowlers.dismissal_kind.isin([‘caught’, ‘bowled’, ‘lbw’, ‘caught and bowled’, ‘stumped’]))]</pre><pre>bowlers = bowlers.groupby(‘bowler’).count().reset_index()</pre><pre>bowlers.rename(columns={‘dismissal_kind’:’no_wickets’}, inplace=True)</pre><pre>bowlers = bowlers.sort_values(by=’no_wickets’, ascending=False)</pre><pre>bowlers = bowlers[(bowlers.no_wickets&gt;=100)]</pre><pre>bowlers</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*aa0rvbaFunqLBplRdvC1VQ.png" /></figure><p>Let’s plot the above findings</p><pre>plt.figure(figsize=(15,5))<br>plt.bar(bowlers.bowler, bowlers.no_wickets)<br>plt.xticks(rotation=90)<br>plt.ylim(99,171)<br>plt.title(‘Bowlers having more than 100 wickets in IPL’)<br>plt.xlabel(‘Bowlers’)<br>plt.ylabel(‘No of wickets’)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tJQyRJSLnc3j7q25oK7lOQ.png" /></figure><p>We can see SL Malinga has taken the most number of wickets in IPL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/279/1*PAkzN2FPYO2cQht8gWItCw.jpeg" /><figcaption>SL Malinga bowling style</figcaption></figure><h3>Highest and lowest runs in IPL History</h3><p>We will now analyse the top 10 extreme performances i.e top 10 highest runs and top 10 lowest runs scored.</p><p>For this case, let us not consider the match having no result or the match which was stopped due to some emergency like rain because of which D/L method was used. We have to exclude super over innings as well from the DataFrame which may contribute to low scoring runs which is not what we want.</p><pre><strong># Get id and season from df_matches(with no &#39;no result&#39; and no D/L method) DataFrame </strong><br>df_id = pd.DataFrame(df_matches[(df_matches.result!=’no result’) &amp; (df_matches.dl_applied==0)][[‘id’,’season’]])</pre><pre><strong># Rename the id column to match with id column of df_deliveries DataFrame</strong><br>df_id.rename(columns={‘id’:’match_id’}, inplace=True)</pre><pre><strong># Join the two DataFrames</strong><br>df_proper_matches = pd.merge(df_id, df_deliveries, on=’match_id’, how=’inner’)</pre><pre><strong># Make sure super_over column is not taken and group by matches and inning and get the sum and sort by the total_runs and take necessary columns</strong><br>df_proper_matches = df_proper_matches[(df_proper_matches.is_super_over!=1)].groupby([‘match_id’,’inning’]).sum().sort_values(by=’total_runs’, ascending=True).reset_index()[[‘match_id’,’inning’,’total_runs’]]</pre><pre><strong># Take innings 1 and 2 only</strong><br>df_proper_matches = df_proper_matches[df_proper_matches.inning&lt;=2]</pre><pre><strong># Again rename the id column</strong><br>df_proper_matches.rename(columns={‘match_id’:’id’}, inplace=True)</pre><pre><strong># Put first 10 rows to lowest and last 10 to highest</strong><br>lowest, highest = df_proper_matches.head(10), df_proper_matches.tail(10)</pre><pre><strong># Merge for getting the lowest and highest DataFrames</strong></pre><pre>lowest = pd.merge(lowest,df_matches,on=’id’,how=’inner’)[[‘season’,’inning’,’team1&#39;,’team2&#39;,’toss_winner’,’toss_decision’,’total_runs’]]</pre><pre>highest = pd.merge(highest,df_matches,on=’id’,how=’inner’)[[‘season’,’inning’,’team1&#39;,’team2&#39;,’toss_winner’,’toss_decision’,’total_runs’]].sort_values(by=’total_runs’,ascending=False)</pre><pre><strong># Print the DataFrame, lowest</strong><br>lowest</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MHoacjQfo5FyIqMmauPWRw.png" /></figure><p>We need to format this data in such a way that we know who the run scorer is and against whom they have played in which season and what was the score. For this purpose, I have created a function to get the scoring_team and the against_team.</p><pre>def score_format(df):<br>    scoring_team = &quot;&quot;<br>    against = &quot;&quot;<br>    if(df.toss_decision == &#39;field&#39;):<br>        if(df.inning == 1):<br>            if(df.toss_winner!=df.team1):<br>                scoring_team = df.team1<br>                against=df.team2<br>            else:<br>                scoring_team = df.team2<br>                against=df.team1<br>        else:<br>            if(df.toss_winner==df.team1):<br>                scoring_team = df.team1<br>                against=df.team2<br>            else:<br>                scoring_team = df.team2<br>                against=df.team1<br>    else:<br>        if(df.inning == 1):<br>            if(df.toss_winner==df.team1):<br>                scoring_team = df.team1<br>                against=df.team2<br>            else:<br>                scoring_team = df.team2<br>                against=df.team1<br>        else:<br>            if(df.toss_winner!=df.team1):<br>                scoring_team = df.team1<br>                against=df.team2<br>            else:<br>                scoring_team = df.team2<br>                against=df.team1<br>    return scoring_team,against</pre><pre><strong># Apply the above function to lowest DataFrame</strong><br>lowest[&#39;low_scoring_team&#39;] = lowest.apply(lambda row : score_format(row)[0], axis = 1)<br>lowest[&#39;against&#39;] = lowest.apply(lambda row : score_format(row)[1], axis = 1)</pre><pre><strong># Apply the above function to highest DataFrame</strong><br>highest[&#39;high_scoring_team&#39;] = highest.apply(lambda row : score_format(row)[0], axis = 1)<br>highest[&#39;against&#39;] = highest.apply(lambda row : score_format(row)[1], axis = 1)</pre><p>Get the list of low scoring team</p><pre>lowest = lowest[[‘low_scoring_team’,’against’,’total_runs’,’season’]]</pre><pre>lowest</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/735/1*p9wm6BaTJ0C72qJc5EBkfQ.png" /></figure><p>Get the list of high scoring team</p><pre>highest = highest[[&#39;high_scoring_team&#39;,&#39;against&#39;,&#39;total_runs&#39;,&#39;season&#39;]]</pre><pre>highest</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*3BaHjIMJOAqPhm92hqDFyA.png" /></figure><p>Royal Challengers Bangalore has both highest and lowest score in IPL history.</p><h3>Average runs in all IPL seasons</h3><p>For getting season average for all IPL matches, we need to merge both the data sets and group buy id and inning and again group by id to get the total runs. After getting the total runs in a match, we need to divide it by 2 (2 innings) for getting the average of score of the match. Then we can take the mean of season wise score (group by season)</p><pre><strong># Group by match_id and inning and get the sum of total_runs and again group it by match_id and get the sum to get the total score in both the innings</strong><br>df_matchwise_runs = df_deliveries.groupby([&#39;match_id&#39;,&#39;inning&#39;]).sum()[&#39;total_runs&#39;].reset_index().groupby(&#39;match_id&#39;).sum()[&#39;total_runs&#39;].reset_index()</pre><pre><strong># Create a new column, average_runs by dividing the total runs by 2</strong><br>df_matchwise_runs[&#39;average_runs&#39;] = df_matchwise_runs[&#39;total_runs&#39;]/2</pre><pre><strong># Drop the column, total_runs</strong><br>df_matchwise_runs.drop(&#39;total_runs&#39;, axis=1, inplace=True)</pre><pre><strong># Take id and season from df_matches DataFrame</strong><br>df_id = df_matches[[&#39;id&#39;,&#39;season&#39;]]</pre><pre><strong># Rename the id column to match it with the id column of df_matchwise_runs</strong><br>df_id = df_id.rename(columns={&#39;id&#39;:&#39;match_id&#39;})</pre><pre><strong># Join the id and the matchwise runs DataFrames</strong><br>df_matchwise_runs = pd.merge(df_id, df_matchwise_runs, on=&#39;match_id&#39;, how=&#39;inner&#39;)</pre><pre><strong># Drop the match_id column</strong><br>df_matchwise_runs.drop(&#39;match_id&#39;, axis=1, inplace=True)</pre><pre><strong># Get the season average by grouping the data by season and taking mean of the average_runs colum</strong><br>season_average = df_matchwise_runs.groupby(&#39;season&#39;).mean().reset_index()</pre><pre><strong># Round the average_runs and convert it to integer</strong><br>season_average.average_runs = round(season_average.average_runs).astype(int)</pre><pre><strong># Display the DataFrame</strong><br>season_average</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/267/1*66qONNQW52aT8YXTsVa58g.png" /></figure><p>Let us plot this on a time-series line chart,</p><pre>plt.figure(figsize=(15,4))<br>plt.plot(season_average.season,season_average.average_runs, ‘bo’)<br>plt.plot(season_average.season,season_average.average_runs)<br>plt.title(‘Average runs per season’)<br>plt.xlabel(‘Season’)<br>plt.ylabel(‘Average Runs’)<br>plt.xticks(np.arange(np.min(season_average.season), np.max(season_average.season)+1, step=1)) <br>plt.yticks(np.arange(np.min(season_average.average_runs), np.max(season_average.average_runs)+1, step=4))<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G8S3IuAip6WGuRbmrLlL1g.png" /></figure><p>From the above time-series graph, we can say that in season 2009, the average score was very less and in season 2018, the average score was highest.</p><p>That is it for now. I would like to thank <strong>Code Heroku</strong> for giving me this opportunity and encouraging me to write blogs and share my knowledge to the world. Also, I would like to thank <a href="https://www.kaggle.com/nowke9">Navaneesh Kumar</a> from Kaggle who has uploaded those data sets.</p><p>Link to the data sets:<br> <a href="https://www.kaggle.com/nowke9/ipldata">IPL Data</a></p><p>Link to my Jupyter Notebook for this IPL Analysis:<br><a href="https://github.com/sudeepto22/ipl-stats/blob/master/IPL%20Analysis.ipynb">IPL Analysis</a></p><p><em>Please share this with all your friends and hit that❤ button below to spread it around even more. Also, add any other tips or tricks that you use for Data Analysis below in the comments.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c939e755e45b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/code-heroku/indian-premier-league-analysis-c939e755e45b">Indian Premier League Analysis</a> was originally published in <a href="https://medium.com/code-heroku">Code Heroku</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Build A ML model and Get Predictions using TensorFlow : (1/3)]]></title>
            <link>https://medium.com/code-heroku/how-to-build-a-ml-model-and-get-predictions-using-tensorflow-1-3-454c94f9abef?source=rss----3b5048e3b0e5---4</link>
            <guid isPermaLink="false">https://medium.com/p/454c94f9abef</guid>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Code Heroku]]></dc:creator>
            <pubDate>Tue, 23 Apr 2019 13:40:17 GMT</pubDate>
            <atom:updated>2019-04-22T07:08:44.462Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to Build A ML model and Get Predictions using TensorFlow : (1/3)</h3><p><em>Exploring the Dataset : To Predict Baby’s Weight.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ugo7kFBW_6Cowe0-4VC9EA.png" /><figcaption>TensorFlow</figcaption></figure><p>Welcome. This is the 1st Tutorial from the series : <strong><em>How to Build A ML model and Get Predictions using TensorFlow</em></strong>.</p><p>In this series, we are going to use <strong>tf.estimator</strong> API.<br>Its a high-level <strong>TensorFlow API</strong> that greatly simplifies machine learning programming. It can be used for training, evaluation, prediction purposes in machine learning environment.</p><p>On <strong>Google Cloud Platform</strong> (GCP) , we can use <strong>Cloud ML Engine</strong> to train machine learning models in <strong>TensorFlow</strong> and other <strong>Python ML</strong> libraries (eg. scikit-learn).</p><p><strong>Cloud ML Engine</strong> gives us a service for training and deploying TensorFlow models. If you have data that fits in memory, pretty much any machine learning framework will work. But once your data sets get larger than the memory, you need a more sophisticated ML framework.</p><blockquote><em>This is where TensorFlow comes in.</em></blockquote><p>The technology we are going to use is : <strong>Cloud Datalab</strong>.</p><p>With the help of Cloud Datalab, we can explore, transform, visualize and process data. The data may be present in <strong>Google BigQuery</strong>, <strong>Compute Engine</strong> and <strong>Cloud Storage</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*RBQpyMm-lWoHCn9VvEzTEg.png" /><figcaption>BigQuery</figcaption></figure><p>Besides Cloud Datalab, the other piece of technology we will be using is BigQuery. It connects to GCP, hence accessing BigQuery is a breeze.</p><p>Based on this, we can then use the data to build data pipelines for deployment to BigQuery or to create the machine ML models.</p><p>The <strong>Dataset of births</strong> that we need, is stored in <strong>BigQuery</strong>.</p><blockquote><em>BigQuery is a serverless data warehouse that operates at massive scale.</em></blockquote><p>Also, What’s noteworthy about Datalab is that the Datalab is open source, and developers can simply fork it and/or submit pull requests on GitHub.</p><h3>Overview of the Series:</h3><ol><li>En route, in this tutorial you <strong>explore</strong> the data, <strong>visualize</strong> the dataset, pick features for your machine learning model.</li><li>Then, in the second tutorial, you create a sample dataset so that you can do local development of a TensorFlow model. Which, then, we will use to develop the <strong>TensorFlow Estimator API model</strong>.</li><li>Once we have the prototype in place, we will change over to P<strong>roductionization</strong>. The training of the model will be distributed and scaled out on Cloud ML Engine. Then , trained model will be deployed as a service</li></ol><h3>Prerequisites :</h3><blockquote>To get started you are going to need a Google cloud account. The free account is sufficient for this purpose.</blockquote><h4>Google Cloud Datalab :</h4><p>You may ask, why do I need a Google account when I can use Jupyter, Python and TensorFlow on my own resources?</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*1Ub_Ci08hteK6lid3ERniw.png" /><figcaption>Google Cloud Datalab</figcaption></figure><p>The answer is — you can easily access <strong>BigQuery</strong> sized data collections directly from the notebook running on your laptop.</p><blockquote>To get started go to the Datalab <a href="https://cloud.google.com/datalab/">home page</a>.</blockquote><p>To say a few words about <strong>Cloud Datalab</strong> — it is a very powerful tool created to explore, analyze, transform and visualize the data and build machine learning models on Google Cloud Platform.</p><h4>Setting up :</h4><p><em>(Note : we will be using this setup throughout the series)</em></p><ul><li><em>Sign up for the Cloud </em><a href="https://cloud.google.com/datalab/"><em>Datalab</em></a><em> :</em><br>It is basically free as you’ll be getting an initial $300 credit.</li><li><em>Create a bucket using your GCP Console :</em><br>In your GCP Console, click on the <strong>Navigation menu</strong> (Three bars /Top-Left) and select <strong>Storage.</strong></li><li>Click on <strong>Create Bucket. </strong>Choose — Regional bucket and set an unique name. Then, click <strong>Create</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*fqPG3qXKpEuEFEPFuY9gaA.png" /></figure><ul><li>Open Cloud Shell. The Cloud Shell icon is at the top right of the GCP <a href="https://console.cloud.google.com/">web console</a>.</li></ul><blockquote><em>Google Cloud Shell, gives developers command-line access to their compute resources on Google Cloud Platform.</em></blockquote><pre>gcloud compute zones list</pre><ul><li>Pick a zone which is closer to you. The listed regions are the regions that currently support Cloud ML Engine jobs.<br>For example, if you are in the US, you can choose <strong>us-east1-c</strong> as your zone.</li><li>Replace &lt;ZONE&gt; with us-east1-c</li></ul><pre>datalab create mydatalabvm --zone us-east1-c</pre><blockquote>Be patient. Datalab will take about 5 minutes to start.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G8J5dm5bcxib7Nhb6t8ebA.png" /><figcaption>Cloud Console Prompt for the Web Preview</figcaption></figure><ul><li>Click on <strong>Web Preview</strong> icon on the top-right corner of the Cloud Shell toolbar. Click <strong>Change Port</strong> and enter the port <strong>8081</strong> and click <strong>Change and Preview</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*maCYhz66Qxl-OXgJrxkMmg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/437/1*cQLVJrsXL4NKCC39cQmj5w.png" /></figure><h4>Cloning the Repo :</h4><ul><li>In <strong>Cloud Datalab</strong> home page , navigate into <strong>notebooks</strong> and add a new notebook using the icon</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/110/1*wdZMWZJ-gHdgCWTFqo6wuw.png" /></figure><ul><li>Rename this notebook as <strong>repocheckout</strong>.</li><li>In the new notebook, enter the following commands in the cell, and click on <strong>Run</strong> (on the top navigation bar) to run the commands:</li></ul><pre>%bash git clone <a href="https://github.com/GoogleCloudPlatform/training-data-analyst">https://github.com/GoogleCloudPlatform/training-data-analyst</a> rm -rf training-data-analyst/.git</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5aagcTkPGaRKIaCT" /></figure><ul><li>Confirm that you have <strong>cloned the repo</strong> by going back to <strong>Datalab Browser</strong> and ensuring that there exists a directory called <strong>training-data-analyst</strong></li><li>Lets head to the exploration!</li></ul><h3>Explore and Visualize the Dataset :</h3><p>Now, you may have been thinking that — why are we exploring the data? Why shouldn’t we just feed the data into the machine learning model?</p><p>Isn’t it a machine learning model?. Isn’t it the ML model’s job to figure out what datasets are needed and what aren’t?!</p><blockquote><em>Well, real life doesn’t work that way.</em></blockquote><p>Many times the data, as it is recorded, isn’t what you expect.</p><p>The real world data is surprisingly confusing, and if we use that data without understanding it first, we will end up using the data in a way that will make productionization very hard.</p><blockquote><em>Because in the productionization, we have to deal with the data as it comes in, in real time.</em></blockquote><p>The goal of this tutorial is to investigate which features have influence on the baby’s weight. Which we are going to predict.</p><p>For example, if you want to know whether the field pluraity or the field is_malehas some influence on the weight, you can generate a bar chart for - the average weight of babies sharing the same pluralityvalue, or the same is_male value.</p><p>However, it’s not enough to just check whether the average weight of the baby in these different conditions is different, you have to look at the whole distribution of values too.</p><p>Checking the number of babies for each value is necessary because if you don’t have enough sample data for any range of input values, predictions using those values may turn out to be inaccurate.</p><h3>Get started :</h3><p><em>Start with running the following code in the Datalab Notebook.</em></p><pre>BUCKET = &#39;cloud-training-demos-ml&#39;<br>PROJECT = &#39;cloud-training-demos&#39;<br>REGION = &#39;us-central1&#39;</pre><pre><strong>import</strong> <strong>os</strong><br>os.environ[&#39;BUCKET&#39;] = BUCKET<br>os.environ[&#39;PROJECT&#39;] = PROJECT<br>os.environ[&#39;REGION&#39;] = REGION</pre><pre>%%bash<br><strong>if</strong> ! gsutil ls | grep -q gs://<strong>${</strong>BUCKET<strong>}</strong>/; <strong>then</strong><br> gsutil mb -l <strong>${</strong>REGION<strong>}</strong> gs://<strong>${</strong>BUCKET<strong>}</strong><br><strong>fi</strong></pre><h3>Explore Data :</h3><p>The data is <strong>Natality</strong> data : record of births in the US. Natality stands for <em>the ratio of the number of births to the size of the population (birth rate).</em></p><blockquote>We are Exploring BigQuery Dataset using Cloud Datalab.</blockquote><p>Here, the goal is to predict the baby’s weight given a number of factors about the pregnancy and the baby’s mother.</p><p>The hash of the year-month will be used for that — this way, twins born on the same day won’t end up in different cuts of the data.</p><h4>1. Create SQL query :</h4><p><em>Using natality data after the year 2000.</em></p><pre>query = &quot;&quot;&quot;<br>SELECT<br> weight_pounds,<br> is_male,<br> mother_age,<br> plurality,<br> gestation_weeks,<br> ABS(FARM_FINGERPRINT(CONCAT(CAST(YEAR AS STRING), CAST(month AS STRING)))) AS hashmonth<br>FROM<br> publicdata.samples.natality<br>WHERE year &gt; 2000<br>&quot;&quot;&quot;</pre><h4>2. Call BigQuery :</h4><p><em>And examine in dataframe.</em></p><pre><strong>import</strong> <strong>google.datalab.bigquery</strong> <strong>as</strong> <strong>bq</strong><br>df = bq.Query(query + &quot; LIMIT 100&quot;).execute().result().to_dataframe()<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/1*-WY4X06_d3i2KBAcpqGqRg.png" /><figcaption>BigQuery DataFrame</figcaption></figure><p>Let’s write a query to find the unique values for each of the columns and the count of those values. This is important to ensure that we have enough examples of each data value, and to verify our hunch that the parameter has predictive value.</p><h4>3. Create a function :</h4><p><em>That finds the number of records and the average weight for each value of the chosen column.</em></p><pre><strong>def</strong> get_distinct_values(column_name):<br> sql = &quot;&quot;&quot;<br>SELECT<br> <strong>{0}</strong>,<br> COUNT(1) AS num_babies,<br> AVG(weight_pounds) AS avg_wt<br>FROM<br> publicdata.samples.natality<br>WHERE<br> year &gt; 2000<br>GROUP BY<br> <strong>{0}</strong><br> &quot;&quot;&quot;.format(column_name)<br> <strong>return</strong> bq.Query(sql).execute().result().to_dataframe()</pre><h4>4. Bar plot :</h4><p><em>To see </em><em>is_malewith </em><em>avg_wtlinear and </em><em>num_babieslogarithmically.</em></p><pre>df = get_distinct_values(&#39;is_male&#39;)<br>df.plot(x=&#39;is_male&#39;, y=&#39;num_babies&#39;, kind=&#39;bar&#39;);<br>df.plot(x=&#39;is_male&#39;, y=&#39;avg_wt&#39;, kind=&#39;bar&#39;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*uF6xK0a26EM2Mh0II3EC4g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*snMzLHx_cLlo9KkVsCClmg.png" /></figure><h4>5. Line plot :</h4><p><em>To see </em><em>mother_age with </em><em>avg_wt linear and </em><em>num_babies logarithmically.</em></p><pre>df = get_distinct_values(&#39;mother_age&#39;)<br>df = df.sort_values(&#39;mother_age&#39;)<br>df.plot(x=&#39;mother_age&#39;, y=&#39;num_babies&#39;);<br>df.plot(x=&#39;mother_age&#39;, y=&#39;avg_wt&#39;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/1*XRj2wcUF8Oe--Mkg-tQaZQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*sQA1eTmQ89nMxpImGqipvQ.png" /></figure><h4>6. Bar plot :</h4><p><em>To see plurality(singleton, twins, etc.) with </em><em>avg_wt linear and </em><em>num_babies logarithmic.</em></p><pre>df = get_distinct_values(&#39;plurality&#39;)<br>df = df.sort_values(&#39;plurality&#39;)<br>df.plot(x=&#39;plurality&#39;, y=&#39;num_babies&#39;, logy=<strong>True</strong>, kind=&#39;bar&#39;);<br>df.plot(x=&#39;plurality&#39;, y=&#39;avg_wt&#39;, kind=&#39;bar&#39;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*bNsokn1dDMPPMEmdwm4kgA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*NoS0g9kkOF2c0c4t-398Ng.png" /></figure><h4>7. Bar plot :</h4><p><em>To see </em><em>gestation_weeks with </em><em>avg_wt linear and </em><em>num_babies logarithmic.</em></p><pre>df = get_distinct_values(&#39;gestation_weeks&#39;)<br>df = df.sort_values(&#39;gestation_weeks&#39;)<br>df.plot(x=&#39;gestation_weeks&#39;, y=&#39;num_babies&#39;, logy=<strong>True</strong>, kind=&#39;bar&#39;);<br>df.plot(x=&#39;gestation_weeks&#39;, y=&#39;avg_wt&#39;, kind=&#39;bar&#39;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*2nzLFsPgFsgWCveSbgQCNg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*rE9OKDca92F7Lt6cpOi8Ow.png" /></figure><h4>All these factors seem to play a part in the baby’s weight :</h4><ul><li>Male babies are heavier on average than female babies.</li><li>Teenage and older moms tend to have lower-weight babies.</li><li>Twins, triplets, etc. have lower weight than single births.</li><li>Premature babies weigh in lower as do babies born to single moms.</li></ul><p>In addition to this, it is important to check whether you have enough data(here — no. of babies). Otherwise, the model prediction against values that doesn’t have enough data may not be reliable.</p><blockquote>In the next article, We are going to create a smaller dataset for <strong>model development </strong>and for <strong>local training</strong>. We will then create a <strong>TensorFlow</strong> model using the high-level <strong>Estimator API</strong>. The <strong>TensorFlow</strong> model will be trained on the sampled dataset.</blockquote><p><strong><em>Thanks for the read!</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=454c94f9abef" width="1" height="1" alt=""><hr><p><a href="https://medium.com/code-heroku/how-to-build-a-ml-model-and-get-predictions-using-tensorflow-1-3-454c94f9abef">How to Build A ML model and Get Predictions using TensorFlow : (1/3)</a> was originally published in <a href="https://medium.com/code-heroku">Code Heroku</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Crawling Robot With Q Learning]]></title>
            <link>https://medium.com/code-heroku/building-a-crawling-robot-with-q-learning-251e2550d824?source=rss----3b5048e3b0e5---4</link>
            <guid isPermaLink="false">https://medium.com/p/251e2550d824</guid>
            <category><![CDATA[q-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Code Heroku]]></dc:creator>
            <pubDate>Sat, 30 Mar 2019 09:07:51 GMT</pubDate>
            <atom:updated>2019-03-30T09:07:51.689Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><em>This post is a part of our </em><a href="http://www.codeheroku.com/course?course_id=1"><em>Introduction to Machine Learning</em></a><em> course at </em><a href="http://www.codeheroku.com/"><em>Code Heroku</em></a><em>.</em></blockquote><p>In our <a href="https://medium.com/code-heroku/introduction-to-reinforcement-learning-67826ec177ea">Introduction to Reinforcement Learning</a> post, we have seen how to model a Reinforcement Learning problem using Markov Decision Process (MDP).</p><p>In case of model based learning, we try to create a model for reinforcement learning problems using Markov Decision Process (MDP). We try to calculate all the possible probabilities associated with each state (and probabilities of getting rewards) and then extract a policy from it.</p><p>But, will calculating all the probabilities be computationally feasible for most real world problems? No, right? That’s why the concept of model free learning got introduced in reinforcement learning.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FVDsugzVbnms%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DVDsugzVbnms&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FVDsugzVbnms%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/504dad0120109cc09561444f82822504/href">https://medium.com/media/504dad0120109cc09561444f82822504/href</a></iframe><p>In case of model free learning, the only thing that we are concerned about is - being in the current state, what is the next best action to take?</p><p>There are two types of model free learning - Monte Carlo Learning and TD Learning. Monte Carlo Learning takes a sequence of actions for solving a Reinforcement Learning problem and then in the end, it tries to evaluate - how good the “sequence of actions” were. Whereas, TD - Learning takes a single action at a time and tries to evaluate how good the individual action was. Q Learning is a type of TD - Learning that we are going to learn today.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RU84QhcDFgBxiaPbGocbEQ.png" /></figure><p>The problem with Monte Carlo Learning is - unless we reach an end state, we won’t be able to evaluate our actions. So, it can’t be implemented for problems which don’t have any end state. For example, in our crawling robot problem, there is no end state; The robot’s only goal is to walk. That’s why we are applying Q Learning for solving this problem.</p><p>We know that in case of reinforcement learning problems, we can take multiple actions from a state. For example, consider the following image of PacMan. The agent can either move in upward, downward, left or right direction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/453/1*JOzsb5j8zt9NMQ5sS72JmQ.jpeg" /></figure><p>But, to achieve the final goal, the agent needs to move in the direction which gives maximum reward (value).</p><p>Q(s,a) is a function which takes a state-action pair as input and then tells us the value (reward) associated with taking this action ‘a’ on state ‘s’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UjpewBT207JW1bE6" /></figure><p>In order to maximize our rewards, we should select the maximum Q(s,a) value for choosing the value of the state ( V(s) ).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*Mz6HbN0Wd5ITh6NR78jAqg.png" /></figure><p>Let’s see if you actually understood this idea or not. Given the following table of Q(s,a), how would you calculate the V(s) for each state?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/0*hF-pkmnUdeI0jz2f" /></figure><p>We can see that the top left block (state) has two possible values of Q(s,a). If it goes to the state at its right, it will get a Q(s,a) of 90. If it goes to the state below it, it will get a Q(s,a) of 72. So, in order to get maximum rewards, we should select V(s) as maximum of all possible Q values from the current state = max(90,72) = 90. This way we can calculate the V(s) for the rest of the states as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*k8uO4KfP46Zxq3Q0gIT6vA.png" /></figure><p>Over here, we have seen how we can calculate V(s) from Q(s,a) values. But, we still don’t know how to calculate the Q(s,a) values itself. Let’s see how to do this.</p><p>The Q value of a state-action pair can is nothing but the summation of the instant reward and all of future rewards.</p><p>Q(s,a) = Instant Reward + Future Reward</p><p>And for calculating the value of “future reward”, we need to select the maximum reward of the next state and multiply it with a discount factor — gamma (Ɣ).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*ixWd5BSuTk2RiD2R-6Ddiw.png" /></figure><p>Based on this equation, we are going to update our Q values.</p><p>Now that we have a clear idea of Q Learning, let’s try to implement a crawling robot with this.</p><p>For implementing this, we will use a crawling environment which was originally created for University of Berkeley’s CS188 course. This environment file can be downloaded from our <a href="https://github.com/codeheroku/Introduction-to-Machine-Learning/blob/master/Reinforcement%20Learning/RL3%20QLearning/crawler_env.py">course Github repo</a>. Along with this, we will create two more files - Agents.py and play.py for for simulating the whole environment.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6b8807813491219e3ec17367ce5c083c/href">https://medium.com/media/6b8807813491219e3ec17367ce5c083c/href</a></iframe><blockquote>You can download the Python scripts and associated files of this article from<a href="https://github.com/codeheroku/Introduction-to-Machine-Learning/tree/master/Reinforcement%20Learning/RL3%20QLearning"> Course Github Repo</a>.</blockquote><p>Inside the play.py file, let us first initialize the environment and agent.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/183489dfbeed89a48361cd9b200b2498/href">https://medium.com/media/183489dfbeed89a48361cd9b200b2498/href</a></iframe><p>For this crawling robot problem, can you figure out what are the states? What are the actions? And what are the rewards?</p><p>We can think of the position of the arms as the states, the movement of the arms as actions and the velocity of the robot as a good reward signal.</p><p>We will keep a track of all the rewards using total_rewards variable. Then we will iterate the learning process for 30,000 times and will print average rewards after each 5,000 steps.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1f6ccf964a2e03b331d308c9be560101/href">https://medium.com/media/1f6ccf964a2e03b331d308c9be560101/href</a></iframe><p>In order to make this thing to work, we need to define the learn() function in agents.py file. We will implement the formula of Q(s,a) inside this function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e1daa222bad5294df995d57fdf43dc08/href">https://medium.com/media/e1daa222bad5294df995d57fdf43dc08/href</a></iframe><p>But, instead of directly assigning the new value to Q, we will use a parameter, alpha, to gradually increase the new value over the old value. It is similar to learning rate in supervised learning.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7cbebf0bbdc0d78693b1bcb8433a24c7/href">https://medium.com/media/7cbebf0bbdc0d78693b1bcb8433a24c7/href</a></iframe><p>Now we need to define the choose_action() function. In this function, we would define our action to choose the maximum Q value for a state. But, sometimes we will intentionally choose random actions to add some randomness to our learning process. This will make sure that our learning process won’t be biased towards taking some specific action.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9ffa6b31e00078d3860030b28138951a/href">https://medium.com/media/9ffa6b31e00078d3860030b28138951a/href</a></iframe><p>Now if we try to run the above file, we will see an output screen where the robot is trying to crawl and getting better at it as the number of steps are increasing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*Os8aUt-yDN8c5K-7zfEGLw.gif" /></figure><blockquote>You can download the completed Python scripts and associated files of this article from<a href="https://github.com/codeheroku/Introduction-to-Machine-Learning/tree/master/Reinforcement%20Learning/RL3%20QLearning"> Course Github Repo</a>.</blockquote><blockquote>If this article was helpful to you, check out our <a href="http://www.codeheroku.com/course?course_id=1">Introduction to Machine Learning</a> Course at <a href="http://www.codeheroku.com/">Code Heroku</a> for a complete guide to Machine Learning.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=251e2550d824" width="1" height="1" alt=""><hr><p><a href="https://medium.com/code-heroku/building-a-crawling-robot-with-q-learning-251e2550d824">Building a Crawling Robot With Q Learning</a> was originally published in <a href="https://medium.com/code-heroku">Code Heroku</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Artificial Intelligence, Machine Learning and Deep Learning - Are they same?]]></title>
            <link>https://medium.com/code-heroku/artificial-intelligence-machine-learning-and-deep-learning-are-they-same-375bdc384953?source=rss----3b5048e3b0e5---4</link>
            <guid isPermaLink="false">https://medium.com/p/375bdc384953</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Code Heroku]]></dc:creator>
            <pubDate>Sun, 24 Mar 2019 07:34:44 GMT</pubDate>
            <atom:updated>2019-03-25T14:03:58.983Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mlsGMqOaDLnzYl3QGmeqfg.jpeg" /></figure><p>One of the most common questions that beginners have when they start to learn Machine Learning is - “Are Artificial Intelligence(AI), Machine Learning(ML) and Deep Learning(DL) the same thing?”</p><p>And that’s a question which I personally hear a lot from so many people. So, I thought why not make a blog post to explain it?</p><h4>Artificial Intelligence (AI)</h4><p>Artificial intelligence refers to the capability of a machine to imitate intelligent human behaviour. In other words, any machine that is able to perform tasks that would otherwise require human intelligence viz., making decisions or giving (meaningful) responses in a conversation can be said to have some kind of artificial intelligence.</p><p>It is a broad area of computer science, which deals with the simulation of intelligent behavior in computers.</p><p>Virtual assistants are very good examples of implementation of AI. With the power of AI, virtual assistants such as Google Assistant or Siri (by Apple) can nowadays imitate human behaviour with high fidelity. Have a look at the following video which demonstrates how Google Assistant uses AI to book an appointment for the user.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FpKVppdt_-B4%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DpKVppdt_-B4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FpKVppdt_-B4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5e60af3e071a660a00c3623f237b4d71/href">https://medium.com/media/5e60af3e071a660a00c3623f237b4d71/href</a></iframe><p>AI can be implemented through huge collection of explicitly programmed if-else statements, or a complex statistical model. But, the end goal of AI remains same - “Simulation of human intelligence in machines”.</p><p>The practice of explicitly programming if-else statements for implementing AI models is known as “Good Old-Fashioned Artificial Intelligence” (GOFAI), which was the dominant paradigm of AI research from the mid-1950s until the late 1980s.</p><p>Some AI models might be termed as “weak AI” models since they only focus on getting the work done without figuring out how human reasoning works. One such example could be - bots created using Artificial Intelligence Markup Language (AIML). These bots try to give responses by simply matching some basic patterns against the input using an XML-like structure of conditions without trying to “understand” the conversation.</p><p>On the other hand, AI systems which pretend to think exactly like humans do, are termed as “strong AI” models. We have already seen an example of this while discussing about Google Assistant.</p><h4>Machine Learning (ML)</h4><p>While AI is a broad area of cognitive computing, Machine Learning is a particular subset of AI which mainly deals with the study of how machines learn.</p><p>Doesn’t make sense? Let me clear that for you.</p><p>Early AI researchers found some problems way much harder to solve with the early available techniques of AI (GOFAI). Hard coded algorithms were simply not useful for solving those problems.</p><p>For example, how would you make a computer to distinguish between pictures of cats and dogs? Will you write hard-coded algorithms for identifying each picture individually? What happens if it sees a picture that it has never seen before? What happens if it sees a partial picture? Will it still be able to recognize it?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*bRdPPhlQXP4GjKH8yxzNFg.png" /></figure><p>It turns out that for solving these kind problems, instead of just imitating human behaviour, researchers started to find ways to imitate how humans learn. And that’s what gave rise to the field of Machine Learning. It is the field of study that gives computers the ability to learn without being explicitly programmed.</p><p>More specifically, in case of machine learning problems, you need to give the computer some data (lots of cat and dog images in this case), tell it to “learn” from that data, and then predict the correct output for new, unseen data.</p><p>This is nothing but a mathematical function (model) which maps inputs to its correct outputs. The task of Machine Learning engineers is to figure out and implement this “mathematical function” in such a way that programmers won’t need to write “hard-coded” algorithms for solving each individual problem.</p><h4><strong>Deep Learning (DL)</strong></h4><p>Deep Learning is a subfield of machine learning which is concerned with algorithms inspired by the structure and function of the brain.</p><p>It consists of a particular type of mathematical model that can be thought of as a collection of simple blocks, distributed across multiple layers. Each of these blocks are specialized to perform a particular task and can be adjusted to better predict the final outcome.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*L5l85Gf_fl9km3qY" /></figure><p>Human brain consists of billions of neurons interconnected to each other. Each neuron receives a signal, processes the signal, and passes it on to the other neurons. This is how the information is processed in our brain.</p><p>These blocks try to simulate the behaviour of neuron cells present in human brain for performing a particular task. Hence, the collection of these blocks can be referred to as Artificial Neural Network (ANN).</p><p>Deep Learning is called “deep” because the composition of these blocks are stacked on top of each other forming multiple layers. The deeper you go, the more complex features are extracted.</p><p><strong>Conclusion</strong></p><p>Although terms like artificial intelligence, machine learning and deep learning are used interchangeably but, they are not the same thing. Machine learning is the subset of artificial intelligence and deep learning is a subset of machine learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*5QPCUW9K5tHS-dsh" /><figcaption>Image source: Nvidia</figcaption></figure><p>Here are the key takeaways:</p><ul><li>Artificial Intelligence deals with imitating human intelligence.</li><li>Machine Learning deals with imitating how humans learn.</li><li>Deep learning deals with imitating how human brain processes data with the help of neurons.</li></ul><blockquote>If you enjoyed reading this article, please have a look at our <a href="http://www.codeheroku.com/course?course_id=1">Introduction to Machine Learning</a> course at <a href="http://www.codeheroku.com/">Code Heroku</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=375bdc384953" width="1" height="1" alt=""><hr><p><a href="https://medium.com/code-heroku/artificial-intelligence-machine-learning-and-deep-learning-are-they-same-375bdc384953">Artificial Intelligence, Machine Learning and Deep Learning - Are they same?</a> was originally published in <a href="https://medium.com/code-heroku">Code Heroku</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Exploratory Data Analysis (EDA)]]></title>
            <link>https://medium.com/code-heroku/introduction-to-exploratory-data-analysis-eda-c0257f888676?source=rss----3b5048e3b0e5---4</link>
            <guid isPermaLink="false">https://medium.com/p/c0257f888676</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Code Heroku]]></dc:creator>
            <pubDate>Sun, 17 Mar 2019 06:35:22 GMT</pubDate>
            <atom:updated>2019-03-25T13:46:50.335Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RXdMb7Uk6mGqWqPguHULaQ.png" /></figure><blockquote>“Torture the data, and it will confess to anything.”</blockquote><blockquote>— Ronald Coase</blockquote><p>If you are someone who is familiar with data science, I can confidently say that you must have realized the power of the above statement.</p><p>With proper use of data one could rule the entire world as well!</p><p>But, raw, unprocessed data isn’t of much use unless you derive insights from it.</p><p>Exploratory Data Analysis (EDA) is the process of visualizing and analyzing data to extract insights from it. In other words, EDA is the process of summarizing important characteristics of data in order to gain better understanding of the dataset.</p><p>In this article, we are going to introduce you to the process of EDA through the analysis of the automobile dataset available <a href="https://drive.google.com/file/d/1g7ewVnbl3zHW33eOD9DoDwJPc4f-OPU2/view?usp=sharing">here</a>. We will talk about some common methods used for EDA and will let you know how to apply them for extracting meaningful insights from raw data.</p><p>Here is a quick overview of the things that you are going to learn in this article:</p><ul><li>Descriptive Statistics</li><li>Grouping of Data</li><li>Handling missing values in dataset</li><li>ANOVA: Analysis of variance</li><li>Correlation</li></ul><p>At first, you need to download the <a href="https://drive.google.com/file/d/1g7ewVnbl3zHW33eOD9DoDwJPc4f-OPU2/view?usp=sharing"><em>automobile dataset from this link</em></a>.</p><h3>Descriptive statistics</h3><p>Descriptive statistics analysis helps to describe the basic features of dataset and obtain a brief summary of the data.</p><p>The describe() method in Pandas library helps us to have a brief summary of the dataset. It automatically calculates basic statistics for all numerical variables excluding NaN (we will come to this part later) values.</p><p>Let’s import all the libraries and read the data.</p><pre>import pandas as pd<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br>import numpy as np<br>from scipy import stats</pre><pre>df = pd.read_csv(&#39;automobile.csv&#39;) #read data from CSV file</pre><p>We can check if the data is successfully imported by displaying the first 5 rows of dataframe using head() method.</p><pre>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*sZQZKxyUq870f6WI2pQLmw.png" /></figure><p>Now, let’s apply the describe() method over this dataset and see the results. It displays a description of mean, standard deviation, quartiles and maximum &amp; minimum values.</p><pre>df.describe()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*3Ebggt7p-p4cbgfLoNLvwg.png" /></figure><p>It helps us to have a brief overview of the data. For example, by seeing the mean of “diesel” and “gas” column, we can say that 90% of the engines run on gas while almost 10% engines run on diesel.</p><p>Now we know how how to get brief summary of numerical data using describe() method. But, what if we have categorical data? How can we get a summary of categorical data? The value_counts() method will be useful in this case.</p><p>Let’s see an example of this:</p><pre>&gt;&gt;&gt; df[‘num-of-doors’].value_counts()</pre><pre>four 115</pre><pre>two 86</pre><p>The above output tells us that our dataset has 115 cars with four doors and 86 cars with two doors.</p><p><strong>Box plots</strong></p><p>Another way of analyzing numerical data would be to use box plots. Box plot shows us the median of the data, which represents where the middle data point is. The upper and lower quartiles represent the 75 and 25 percentile of the data respectively. The upper and lower extremes shows us the extreme ends of the distribution of our data. Finally, it also represents outliers, which occur outside the upper and lower extremes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*UEU8TgRHjvHIArYDpFwjjg.png" /></figure><p>For example, the following code shows us the distribution of price for cars with different number of cylinders.</p><pre>sns.boxplot(x=&#39;num-of-cylinders&#39;,y=&#39;price&#39;,data=df)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g8L2JOXIl7GDaAVTryYl5A.png" /></figure><p>We can see that the price of eight-cylinder cars lies between 35,000 to 45,000, while the price of four-cylinder cars lies between 5,000 to 19,000. It also tells us that the average price of five-cylinder cars and six-cylinder cars is almost same.</p><p><strong>Scatter plots</strong></p><p>Often, we see some continuous variables in our data within a specific range. For example, in our dataset, engine size and price are continuous variables. What if we want to understand the relationship between these continuous variables? Could engine size possibly predict the price of the car?</p><p>A great way to visualize this relationship would be to use a scatter plot. Scatter plots represent each relationship between two continuous variables as individual data point in a 2D graph.</p><p>We will use the scatter() method of matplotlib library to visualize the scatter plot.</p><pre>plt.scatter(df[&#39;engine-size&#39;],df[&#39;price&#39;])</pre><pre>plt.xlabel(&#39;Engine Size&#39;)</pre><pre>plt.ylabel(&#39;Price&#39;)</pre><pre>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a5Jl0zca0TnGMi5f72PO3A.png" /></figure><p>From the above output, we can interpret that there is a linear relationship between engine size and price. Cars with bigger engines might be costlier than the cars with small-sized engines. This thing totally makes sense, right?</p><p><strong>Histograms</strong></p><p>Histogram shows us the frequency distribution of a variable. It partitions the spread of numeric data into parts called as “bins” and then counts the number of data points that fall into each bin. So, the vertical axis actually represents the number of data points in each bin.</p><p>Let’s see an example of this. We will see the distribution of “peak-rpm” using histogram.</p><pre>count,bin_edges = np.histogram(df[&#39;peak-rpm&#39;])</pre><pre>df[&#39;peak-rpm&#39;].plot(kind=&#39;hist&#39;,xticks=bin_edges)</pre><pre>plt.xlabel(&#39;Value of peak rpm&#39;)</pre><pre>plt.ylabel(&#39;Number of cars&#39;)</pre><pre>plt.grid()</pre><pre>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zr0XAhEqxmtOdWqkOd8VyQ.png" /></figure><p>The above output tells us that there are 10 cars which have peak rpm between 4395 and 4640, around 42 cars have peak rpm between 4640 and 4885 and so on.</p><h3>Grouping of data</h3><p>Assume that you want to know the average price of different types of vehicles and observe how they differ according to body styles and number of doors. A nice way to do this would be to group the data according to “body-style” and “num-of-doors” and then see the average price across each category. The groupby() method from Pandas library helps us to accomplish this task.</p><pre>df_temp = df[[‘num-of-doors’,’body-style’,’price’]]</pre><pre>df_group = df_temp.groupby([‘num-of-doors’,’body-style’],as_index=False).mean()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/824/1*w3rbv8vIgmAkO3e-ku7xpw.png" /></figure><p>The above output tells us that two door hardtop and two door convertibles are the most expensive cars, whereas, four door hatchbacks are the cheapest.</p><p>A table of this form is not very easy to read. So, we can convert this table to a pivot table using the pivot() method, which would allow us to read this table in a better fashion.</p><pre>df_pivot = df_group.pivot(index=&#39;body-style&#39;,columns=&#39;num-of-doors&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/1*nUv_EmsscWYOdnNWbnvZhg.png" /></figure><p>The price data now becomes a rectangular grid, which is easier to visualize.</p><h3>Handling missing values</h3><p>When no data value is stored for a feature in a particular observation, we say this feature has missing values. Examining this is important because when some of your data is missing, it can lead to weak or biased analysis.</p><p>We can detect missing values by applying isnull() method over the dataframe. The isnull() method returns a rectangular grid of boolean values which tells us if a particular cell in the dataframe has missing value or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*BUjtNl4HGOxu3mHIEl75iw.png" /></figure><p>As you can see, analyzing a grid of this style to detect missing value is not very convenient, so we will use heatmaps to visually detect these missing values.</p><p><strong>Heatmap</strong></p><p>Heatmap takes a rectangular data grid as input and then assigns a color intensity to each data cell based on the data value of the cell. This is a great way to get visual clues about the data.</p><p>We will generate a heatmap of the output of isnull() in order to detect missing values.</p><pre>sns.heatmap(df.isnull())</pre><pre>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yXMKNPL3pTRU6OlawijJDA.png" /></figure><p>This indicates that “stroke” and “horsepower-binned” columns have few missing values.</p><p>We can handle missing values in many ways:</p><ul><li><strong>Delete</strong>: You can delete the rows with the missing values or delete the whole column which has missing values. The dropna() method from Pandas library can be used to accomplish this task.</li><li><strong>Impute</strong>: Deleting data might cause huge amount of information loss. So, replacing data might be a better option than deleting. One standard replacement technique is to replace missing values with the average value of the entire column. For example, we can replace the missing values in “stroke” column with the mean value of stroke column. The fillna() method from Pandas library can be used to accomplish this task.</li><li><strong>Predictive filling</strong>: Alternatively, you can choose to fill missing values through predictive filling. The interpolate() method will perform a linear interpolation in order to “guess” the missing values and fill the results in the dataset.</li></ul><h3>ANOVA (Analysis of Variance)</h3><p>ANOVA is a statistical method which is used for figuring out the relation between different groups of categorical data. The ANOVA test, gives us two measures as result:</p><p>F-test score: It calculates the variation between sample group means divided by variation within sample group.</p><p>P value: It shows us the confidence degree. In other words, it tells us whether the obtained result is statistically significant or not.</p><p>Let’s take an example to understand this better.</p><p>The following bar chart shows the average price of different car makes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*svo-8oB4YqXOBOBck80L3A.png" /><figcaption>Average price for different makes</figcaption></figure><p>We can see that the average price of “audi” and “volvo” is almost same. But, the average price of “jaguar” and “honda” differ significantly.</p><p>So, we can say that there is very small variance between “audi” and “volvo” because their average price is almost same. While the variance between “jaguar” and “honda” is significantly high. Let’s verify this using the ANOVA method.</p><p>The ANOVA test can be performed using the f_oneway() method from Scipy library .</p><pre>temp_df = df[[&#39;make&#39;,&#39;price&#39;]].groupby([&#39;make&#39;])</pre><pre>stats.f_oneway(temp_df.get_group(&#39;audi&#39;)[&#39;price&#39;],temp_df.get_group(&#39;volvo&#39;)[&#39;price&#39;])</pre><p>This gives us the following result:</p><pre>F_onewayResult(statistic=0.014303241552631388, pvalue=0.9063901597143602)</pre><p>The result confirms what we guessed at first. Since the variance between the price of “audi” and “volvo” is very small, we got a F-test score which is very small (around 0.01) and a p value around 0.9.</p><p>Let’s do this test once more between “jaguar” and “honda” and see the results.</p><pre>stats.f_oneway(temp_df.get_group(&#39;jaguar&#39;)[&#39;price&#39;],temp_df.get_group(&#39;honda&#39;)[&#39;price&#39;])</pre><p>This gives us the following result:</p><pre>F_onewayResult(statistic=400.925870564337, pvalue=1.0586193512077862e-11)</pre><p>Notice that in this case, we got a very high F-Test score(around 401) with a p value around 1.05 * 10^-11 because, the variance between the average price of “jaguar” and “honda” is huge.</p><h3>Correlation</h3><p>Correlation is a statistical metric for measuring to what extent different variables are interdependent.</p><p>In other words, when we look at two variables over time, if one variable changes, how does</p><p>this effect change in the other variable?</p><p>For example, smoking is known to be correlated with lung cancer. Since, smoking increases the chances of lung cancer.</p><p>Another example would be the relationship between the number of hours a student studies and the score obtained by that student. Because, we expect the student who studies more to obtain higher marks in the exam.</p><p>We can see the correlation between different variables using the corr() function. Then we can plot a heatmap over this output to visualize the results.</p><pre>correlation_matrix = df.corr()</pre><pre>sns.heatmap(correlation_matrix, annot=True)</pre><pre>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i387yhkZFdy1WZj-lVY9nQ.png" /><figcaption>Heatmap of correlation matrix</figcaption></figure><p>From the above heatmap, we can see that engine size and price are positively correlated(score of 0.87) with each other while, highway-mpg and price are negatively correlated(score of -0.7) with each other. In other words, it tells us that cars with larger engine sizes will be costlier than cars with small engine sizes. It also tells us that expensive cars generally have less MPG as compared to cheaper cars.</p><p>Let’s verify this relationship by plotting regression plots between these variables.</p><pre>sns.regplot(x=&#39;engine-size&#39;,y=&#39;price&#39;,data=df)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S4R0pWiaXE4C9grAI4nVRw.png" /></figure><p>The above plot shows the positive correlation between engine size and price.</p><pre>sns.regplot(x=&#39;highway-mpg&#39;,y=&#39;price&#39;,data=df)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ExAYhChxR2Bl6MZPST2RoQ.png" /></figure><p>The above plot shows us the negative correlation between “highway-mpg” and “price”.</p><p>This was a brief introduction to Exploratory Data Analysis. Follow our publication to get regular updates on these kind of tutorials.</p><blockquote>If you enjoyed reading this article, please have a look at our <a href="http://www.codeheroku.com/course?course_id=1">Introduction to Machine Learning</a> course at <a href="http://www.codeheroku.com/">Code Heroku</a>.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0257f888676" width="1" height="1" alt=""><hr><p><a href="https://medium.com/code-heroku/introduction-to-exploratory-data-analysis-eda-c0257f888676">Introduction to Exploratory Data Analysis (EDA)</a> was originally published in <a href="https://medium.com/code-heroku">Code Heroku</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Reinforcement Learning]]></title>
            <link>https://medium.com/code-heroku/introduction-to-reinforcement-learning-67826ec177ea?source=rss----3b5048e3b0e5---4</link>
            <guid isPermaLink="false">https://medium.com/p/67826ec177ea</guid>
            <category><![CDATA[introduction]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[markov-decision-process]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Code Heroku]]></dc:creator>
            <pubDate>Thu, 14 Mar 2019 08:11:00 GMT</pubDate>
            <atom:updated>2019-03-14T14:01:57.274Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction to Reinforcement Learning — Part 1</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ngDqK7YZ0tZ2oB_4yhelWw.png" /></figure><blockquote>This post is a part of our <a href="http://www.codeheroku.com/course?course_id=1">Introduction to Machine Learning</a> course at <a href="http://www.codeheroku.com/">Code Heroku</a>.</blockquote><p>Till now in our <a href="http://www.codeheroku.com/course?course_id=1">Introduction to Machine Learning</a> course, we have seen that a supervised machine learning algorithm takes in <strong><em>Features</em></strong> of a dataset (e.g., number of hours a student studies) and <strong><em>Labels</em></strong> (e.g., their score on an exam) as inputs and estimates a <strong><em>Function</em></strong> (model) which minimizes the difference between the expected and predicted results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*QE4Wk04rlPZ2Ty3bvplkqw.png" /></figure><p>Now think about how a kid learns to ride a bicycle. No one gives him/her datasets to analyze. They learn by trying different things and then observing the effects of their actions. If the kind of actions they perform lead to balancing the bike they get to enjoy the ride or else they end-up falling and hurting themself. Similarly, can we also teach computers to learn from their mistakes? And the answer lies with Reinforcement Learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6ffe83S_FtUuuN9wPeBowQ.png" /></figure><blockquote>Reinforcement learning is the closest one could get to the natural form of learning. It is a paradigm of Machine Learning and Artificial Intelligence which deals with making a set of sequential decisions under uncertainty.</blockquote><h4>What is Reinforcement Learning?</h4><p>Let’s say we are given a robot inside a room and our goal is to reach to the green block which gives us +1 point. There is also a red block and if we end up there we get -1 point. After reaching to both of these blocks(states) our simulation comes to an end. There is one challenge, because the way mechanics of this world and robot are setup whenever the robot tries to move forward, in only 80% of the cases it actually moves forward. In the other 10% of cases, it goes either goes to the right and in remaining 10% it tries to go the left but since there is no place to go it will stay at the same spot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*IHCgRcSaSf305XmCz0s_aQ.png" /></figure><p>In real world where all our actions do not always end up with intended consequences (i.e. the world is stochastic) conventional planning algorithms are useless. Given this challenge, what is the best sequence of actions that we can take so as to ultimately reach to our goal?</p><p>Reinforcement Learning tries to find answer to this question.</p><p>The goal of any reinforcement learning algorithm is to find a set of actions that will result in maximum rewards (now or in near future).</p><p>And what’s the solution? Well, the only hope in this case to take an action and then observe its effects i.e. We not only need data from our sensors but also access to the physical/simulated environment. Also note that in most blocks (states) in this room except for the terminating states we do not get any feedback(reward) from the environment.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FzeomVun0OoY%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DzeomVun0OoY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FzeomVun0OoY%2Fhqdefault.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/4848ff7680e811a87dcca162dc2bbbbd/href">https://medium.com/media/4848ff7680e811a87dcca162dc2bbbbd/href</a></iframe><h3><strong>Reinforcement Learning Problem Formulation</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*sm7er02CXaN-XELHEBA5fA.png" /></figure><h4>Let’s Make sure you understand some RL Lingo</h4><p><strong>States </strong>is your current observation of the world. In the previous example, the positions/blocks in the room were the states of our environment.</p><p><strong>Actions </strong>the set of actions your RL agent can perform in the environment. In the previous example, the actions that our robot could perform were moving Up, Down, Left and Right.</p><p><strong>Rewards </strong>are the feedback that you might get from the environment. In the previous example our robot gets a reward of +1 for being in the green block and -1 for being the red block.</p><p>Let’s see if you get it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*rZpPl59l7t8JRM2lso-qWw.png" /></figure><p>What do you think? What are the states, actions and rewards for the RL problem of playing the game of Pac-Man.</p><p><strong>States</strong>: What could be an efficient way to represent a state in this game? We can ask ourselves what does the agent (Pac-Man) need to know in order to make a good move? It needs to know its own position, positions of all the ghosts and position of all the food. Right? So, a good representation of the state might be a tuple containing values of</p><p>{agent_loc:33; ghost_locations:[12,22,32], food_locations: [10,20,30]}</p><p><strong>Actions</strong>: Our agent can move only in four directions in this 2D grid so our actions are {Up,Down,Left,Right}</p><p><strong>Rewards</strong>: Our end goal is to obtain maximum possible score in the game. So, after performing an action, the increase in score can be a good reward signal for our agent. Or, if your intention is to survive in the game as long as possible, time spent in the game can be a good reward signal for the agent.</p><h4>Markov Decision Process (MDP):</h4><p>To formalize the Reinforcement Learning problem at hand we use Markov Decision Process (MDP). MDPs give us a good way to create a mathematical model of the RL problem and adhere to the Markovian property that the results of our actions only depends on the present state and everything that has happened before this is irrelevant. This might seem a bit limiting at first since intuitively we know that our set of actions/states are correlated and not independent of each other. But as always, we will work our way around this assumption.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*Dv_Wc8k2gTMK5hE-d0SvLg.png" /></figure><p>S denotes all the states.</p><p>A denotes all the actions.</p><p>P(s’|s,a) denotes the transition probabilities. It indicates the probability of reaching state s’ from state s given that you take the action <em>a</em>.</p><p>P(r|s,a) denotes the reward probabilities. It indicates the chance of getting reward R given that you take an action <em>a </em>on state <em>s.</em></p><p>s_0 denotes the initial state of the world.</p><blockquote>The goal of any reinforcement learning algorithm is to come up with a policy which maps all the states to actions.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*FULdw2OPOxVqBErmhhfHww.png" /></figure><p>There could be many possible policies. But, we need to select an optimal policy which will give us the maximum value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MImNTEWc0eKZvKf97WI4gQ.png" /></figure><p>Now the question is, “How do we get this maximum value?”</p><p>Well, we know that we want to get maximum rewards using a policy. Right?</p><p>So, if we sum all our rewards from t = 0 to infinity for a given policy, we should get the value of that policy. And because these rewards are stochastic, we always don’t get these rewards. So, the value of policy is actually the expectation over the sum of rewards.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QxqMciEuQ2gkzfIUYt9Kbg.png" /></figure><p>But wait, the sum of all the rewards from t = 0 to infinity will not be bounded. We will run out of memory if we try to compute this value. So let’s introduce a new factor, gamma (Ɣ) into our equation. This gamma (Ɣ) is called as the discount factor, which will between 0 to 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2-al3GmAmCNw3xoTn95D4Q.png" /></figure><p>Gamma (Ɣ) indicates the importance of a reward in our equation. Initially, the value of gamma (Ɣ) is set to 1. And then, the value of gamma (Ɣ) gradually decreases with each iteration. This is a way of saying that our future rewards are less valuable than our current reward. So if someone gives you a $100 check right now versus $100 one year latter, the one that you get right now is more valuable, right?</p><p>As the value of gamma (Ɣ)decreases in each step, the sum of discounted rewards will eventually converge (remember geometric series?).</p><p>Thus we have formalized the problem at hand and we will see latter posts how we go about solving this. Please let us know in the comments if this was helpful.</p><blockquote><em>If this article was helpful to you, check out our </em><a href="http://www.codeheroku.com/course?course_id=1"><em>Introduction to Machine Learning</em></a><em> Course at </em><a href="http://www.codeheroku.com/"><em>Code Heroku</em></a><em> for a complete guide to Machine Learning.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=67826ec177ea" width="1" height="1" alt=""><hr><p><a href="https://medium.com/code-heroku/introduction-to-reinforcement-learning-67826ec177ea">Introduction to Reinforcement Learning</a> was originally published in <a href="https://medium.com/code-heroku">Code Heroku</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>