<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Maajid Khan on Medium]]></title>
        <description><![CDATA[Stories by Maajid Khan on Medium]]></description>
        <link>https://medium.com/@nmaajidkhan?source=rss-27ae3f584d17------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*NLsrDTzX6X2Zl_yqCbDoBA.png</url>
            <title>Stories by Maajid Khan on Medium</title>
            <link>https://medium.com/@nmaajidkhan?source=rss-27ae3f584d17------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 05:25:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nmaajidkhan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[5 places where Initializer List is a compulsion in C++]]></title>
            <link>https://medium.com/analytics-vidhya/5-places-where-initializer-list-is-a-compulsion-in-c-7b0417d927d1?source=rss-27ae3f584d17------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b0417d927d1</guid>
            <category><![CDATA[hacks]]></category>
            <category><![CDATA[syntax]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[cplusplus]]></category>
            <category><![CDATA[cpp]]></category>
            <dc:creator><![CDATA[Maajid Khan]]></dc:creator>
            <pubDate>Sat, 31 Oct 2020 10:54:35 GMT</pubDate>
            <atom:updated>2020-10-31T17:06:07.501Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/437/1*SH_DSvio2AXnWRj2Qzuf-g.png" /><figcaption><strong>INITIALIZER LISTS</strong></figcaption></figure><p>Initializer List is used in initializing the data members of a class. The list of members to be initialized is indicated with constructor as a comma-separated list followed by a colon.</p><p>The syntax begins with a colon(:) and then each variable along with its value separated by a comma. The initializer list does not end in a semicolon.</p><p><strong>Syntax:</strong><br>Constructorname(datatype value1, datatype value2):datamember{value1},datamember{value2}<br>{<br> ...<br>}</p><p>Here are 5 places where Initializer List is a compulsion in c++ and you cannot directly value assignment to the member variables</p><ol><li><strong>You cannot initialize non static constant data member of your class without an initializer list</strong></li></ol><p>When your class has a const data member <strong>_x, </strong>it has be to initialized using a initializer list, you can’t assign the value to it inside the constructor</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*dRhN2NYlVWe_mUkQLnVhYA.png" /><figcaption><strong>[Correct] works fine</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/1*33dq30ZiOSPXpp2R7T-Liw.png" /><figcaption><strong>error: </strong>uninitialized const member in ‘const int’ [-fpermissive]|<br><strong>[ wrong] error: </strong>assignment of read-only member ‘Base::_x’</figcaption></figure><p><strong>2. You cannot initialize a reference variable of your class without an initializer list</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*Jauu2hILE2T7I6tTF6-xJQ.png" /><figcaption><strong>[Correct] works fine</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*zRgxiLK-MtfoG1H_YmtmjA.png" /><figcaption><strong>error:</strong> uninitialized reference member in ‘int&amp;’ [-fpermissive]|</figcaption></figure><p><strong>3. you cannot initialize one class data member inside another class (class Two) if the first class (class One) is not having default constructor for itself.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/1*wg0qTK1D1VqdrF0AseBNRQ.png" /><figcaption><strong>[Correct] works fine</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/371/1*spRBvelIEzcR6cksUDuezQ.png" /><figcaption><strong>error:</strong> no matching function for call to ‘One::One()’|</figcaption></figure><p>so here, when <strong>Two two(one) </strong>is called in the main() function, In Line 27: when one is passed as an argument x (One x) for the class Two constructor, we expect that the default constructor of class One is called.</p><p>As we haven’t defined the default constructor for class One, it throws an error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/1*9avaMMTemF_OSjxbAwdHrw.png" /><figcaption><strong>[Correct] works fine. </strong>works as we have added the default constructor for class One()</figcaption></figure><p><strong>4. You cannot initialize your Base class data member from Child class without using the initializer list</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*xzoeArhpKyQV-oyPHToiCQ.png" /><figcaption><strong>[Correct] works fine</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*mnk-XO_gS8G9GEkctR1ncA.png" /><figcaption><strong>error: </strong>‘int Base::_x’ is private within this context</figcaption></figure><p><strong>5. when you have temporary variable exactly similar to data member name, you have to use an initializer list</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*25OuI7esQz_bIimJ5DmvUQ.png" /><figcaption><strong>[Correct] works fine</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*PAHeYNx6vvJOgPhYgj6-cg.png" /><figcaption><strong>[Wrong] </strong>Prints random value for the variable _x</figcaption></figure><p>Thanks for Reading, will upload the relevant code in my Github page<br><a href="https://github.com/MaajidKhan">https://github.com/MaajidKhan</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b0417d927d1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/5-places-where-initializer-list-is-a-compulsion-in-c-7b0417d927d1">5 places where Initializer List is a compulsion in C++</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transfer Learning — Life Just Got Easier!!]]></title>
            <link>https://medium.com/analytics-vidhya/transfer-learning-life-just-got-easier-11805a375039?source=rss-27ae3f584d17------2</link>
            <guid isPermaLink="false">https://medium.com/p/11805a375039</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[transfer-learning]]></category>
            <category><![CDATA[flask]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[keras]]></category>
            <dc:creator><![CDATA[Maajid Khan]]></dc:creator>
            <pubDate>Tue, 24 Dec 2019 12:29:09 GMT</pubDate>
            <atom:updated>2019-12-31T13:49:15.545Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Transfer Learning (Keras)— Life Just Got Easier!!</strong></h3><h4><strong>Pick your model, deploy it and play around (Resnet50 + VGG16 + Xception) -&gt; Ensemble Them..</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*UNOh6VHcHt9eqWW1KDCXEw.jpeg" /></figure><p><strong>A web App for Image Classification using various Deep Learning Models.</strong></p><p>In the recent time deep learning has achieved a lot of popularity because of its performance in various machine learning algorithms. Deep learning tries to model high level abstractions in information by utilizing multiple processing layers. Multi-Class Image Classification is a major research point with wide application prospects in Artificial Intelligence field these days. This course project describes the use of this learning methods like VGG16, ResNet50, Xception, etc.; to perform Multi-Class image classification and implement these deep learning algorithms on a large-scale Multi-Class Image Classification dataset from ImageNet yearly challenge task. We will be using a large trained deep convolutional neural network to classify the 1.2 million high-resolution images into the 1000 different classes. The implementation performs the categorization of various images into its most likely class and shows illustrative examples of the prediction probability. A web app is developed using python’s flask API to classify these images into different categories using the above-mentioned models. Our goal is to find a computational efficient algorithm which classifies the image correctly with a higher accuracy.</p><p><strong>Transfer Learning:</strong></p><p>If not for <strong>Transfer Learning</strong>, Machine Learning is a pretty tough thing to do for an absolute beginner. At the lowest level, machine learning involves computing a function that maps some inputs to their corresponding outputs. Though the function itself is just a bunch of addition and multiplication operations, when passed through a nonlinear activation function and stacking a bunch of these layers together, <strong>functions can be made, to learn literally anything</strong>, provided there’s enough data to learn from, and an enormous amount of computational power.</p><p><strong>That’s where transfer learning comes into play.</strong> In transfer learning, we take the pre-trained weights of an already trained model (one that has been trained on millions of images belonging to 1000’s of classes, on several high power GPU’s for several days) and use these already learned features to predict new classes. <em>The advantages of transfer learning are that, there is no need of an extremely large training dataset and not much computational power is required as we are using pre-trained weights and only have to learn the weights of the last few layers.</em></p><p>The reason why it works so well is that, we use a network which is pre-trained on the imagenet dataset and this network has already learnt to recognize the trivial shapes and small parts of different objects in its initial layers. By using a pre-trained network to do transfer learning, we are simply adding a few dense layers at the end of the pre-trained network and learning what combination of these already learnt features help in recognizing the objects in our new dataset.</p><p>Hence we are training only a few dense layers. Furthermore, we are using a combination of these already learnt trivial features to recognize new objects. All this helps in making the training process very fast and require very less training data compared to training a conv net from scratch.</p><h4><strong>Hardware Requirements:</strong></h4><p>· Quad core Intel Core i7 Skylake or higher processor</p><p>· 8GB RAM or higher (8GB is okay but not for the performance you may want and or expect)</p><p>· <strong>GPU</strong>: RTX 2070/ RTX 2080 Ti/ GTX 1070/ GTX 1080/ GTX 1070 Ti</p><p>· <strong>CPU</strong>: 1–2 cores per GPU depending how you preprocess data</p><p>· Hard drive for data (&gt;= 1TB)</p><p>· An additional monitor</p><p>· PSU has enough PCIe connectors (6+8pins)</p><p>· Blower-style fans for handling multiple GPUs</p><p>· <strong>Motherboard</strong> with multiple PCIe slots</p><h4><strong>Software Requirements:</strong></h4><p>· Windows OS</p><p>· Python 3.6+</p><p>· Anaconda Navigator (Tool)</p><p>· Spyder IDE</p><p>· Flask API (Python)</p><p>· Numpy package</p><p>· Os package</p><p>· Keras package</p><p>· WSGIServer package</p><p>· Pandas package</p><p>· Pycharm IDE (For flask projects)</p><p><strong>Detailed Design:</strong></p><p>This project is used to develop a web app to classify images into different categories using a pre-trained model on the Imagenet dataset. ImageNet is a common academic data set in machine learning for training an image recognition system. The basic idea is to allow the user to upload an image from his/her system and run the different deep learning image classification models over it to classify in different classes with a confidence (probability) score.</p><p><strong>System Architecture:</strong></p><p>The overall architecture of this system is divided into 3 stages:</p><p><strong>Stage-I: (Transfer Learning stage)</strong></p><p>This is the initial stage of the project. A huge data (Imagenet) with class labels is taken and are trained using various deep learning models. These models learn by extracting patterns from the features and they undergo training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*a2NZRegz1dLxcYdYz5EhSg.png" /><figcaption><strong>Figure 1: Transfer Learning Stage (Stage-I)</strong></figcaption></figure><p>The next step in working with neural networks is training. During training, the network is fed input data (images, in this case), and the net’s output, or guesses, are compared to the expected results (the image’s labels). With each run through the data, the network’s weights are modified to decrease the error rate of the guesses; that is, they are adjusted so that the guesses better match the proper label of an image. (Training a large network on millions of images can take a lot of computing resources, thus the interest of distributing pre-trained nets like this.)</p><p>Once trained, the network can be used for inference, or making predictions about the data it sees. Inference happens to be a much less compute-intensive process.</p><p><strong>Stage-II: (Building Hybrid classification model)</strong></p><p>In this stage, we load the pre-trained model weights we generated from stage-I. Then we use this weights to train our deep learning models ResNet50, VGG16 and Xception. Then we infer top 3 accuracy results from each classifier. Then, we ensemble the top 1 class output of each model and display the final result. This way the accuracy of predicting the correct class of the image significantly improves.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*Mc1Jo_JhPlJWzJ2mYj6ZaQ.png" /><figcaption><strong>Figure 2: Building Hybrid classification model (Stage-II)</strong></figcaption></figure><p><strong>Stage-III: (Deploying the model as a web Application)</strong></p><p>In this stage, we will be using Flask API of python to develop a web application to deploy our newly built deep learning image classification model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*7hTvltHgM51ybNdNYYD9pg.png" /><figcaption><strong>Figure 3: Deployment of model as Web App (Stage-III)</strong></figcaption></figure><p>We upload the image to be tested locally from the machine into the web app. The web app makes an API call, computes the model and returns back the output on to the user screen.</p><p><strong>Implementation:</strong></p><p><strong>Resnet50 Model with Flask web App code snippet:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*AsdBYeL89NqdJEA_Fbcppw.png" /><figcaption>ResNet50 Model code snippet</figcaption></figure><p><strong>Flask Web Application UI:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/319/1*KxpdEsgWUsv_7GuVltHQMw.png" /><figcaption><strong>Flask web App for single Model (Resnet50)</strong></figcaption></figure><p><strong>Flask web App for our 3 Models:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HnxUaN9ip2vIL90CpJ9DRA.jpeg" /><figcaption><strong>Image Predictor web page for our 3 Models</strong></figcaption></figure><p>Now that we have our 3 top deep learning models predicting the results on the same image (Each giving Top 3 predictions), we can choose the best prediction to be one with the highest probability.</p><p>In the above case, we can take the result to be “Chihuahua” predicted by ResNet50 model with 99.97% Accuracy.</p><p><strong>The code for this project can be found at GitHub:</strong></p><p><a href="https://github.com/MaajidKhan/Deploying3DeepLearningModels-Flask-">https://github.com/MaajidKhan/Deploying3DeepLearningModels-Flask-</a></p><p><strong>References:</strong></p><p>[1] <a href="https://in.mathworks.com/matlabcentral/fileexchange/59133-neural-network-toolbox-tm--model-for-alexnet-network">https://in.mathworks.com/matlabcentral/fileexchange/59133-neural-network-toolbox-tm--model-for-alexnet-network</a></p><p>[2] Lee, H., Grosse, R., Ranganath, R. and Ng, A.Y., 2009, June. Convolutional deep belief</p><p>networks for scalable unsupervised learning of hierarchical representations. In <em>Proceedings of the 26th annual international conference on machine learning</em> (pp. 609–616). ACM.</p><p>[3] Deep Learning with MATLAB — matlab expo2018</p><p>[4] Introducing Deep Learning with the MATLAB — Deep Learning E-Book provided by the</p><p>mathworks.</p><p>[5] Singh, N. and Singh, S., 2017, March. Object classification to analyze medical imaging data using deep learning. In <em>2017 International Conference on Innovations in Information, Embedded and Communication Systems (ICIIECS)</em> (pp. 1–4). IEEE.</p><p>[6] Azizah, L.M.R., Umayah, S.F., Riyadi, S., Damarjati, C. and Utama, N.A., 2017, November. Deep learning implementation using convolutional neural network in mangosteen surface defect detection. In <em>2017 7th IEEE International Conference on Control System, Computing and Engineering (ICCSCE)</em>(pp. 242–246). IEEE.</p><p>[7] Shiddieqy, H.A., Hariadi, F.I. and Adiono, T., 2017, October. Implementation of deep-learning based image classification on single board computer. In <em>2017 International Symposium on Electronics and Smart Devices (ISESD)</em> (pp. 133–137). IEEE.</p><p>[8] Touvron, H., Vedaldi, A., Douze, M. and Jégou, H., 2019. Fixing the train-test resolution discrepancy. <em>arXiv preprint arXiv:1906.06423</em>.</p><p>[9] Chollet, F., 2017. Xception: Deep learning with depthwise separable convolutions. In <em>Proceedings of the IEEE conference on computer vision and pattern recognition</em> (pp. 1251–1258).</p><p>[10] Krizhevsky, A., Sutskever, I. and Hinton, G.E., 2012. Imagenet classification with deep convolutional neural networks. In <em>Advances in neural information processing systems</em> (pp. 1097–1105).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11805a375039" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/transfer-learning-life-just-got-easier-11805a375039">Transfer Learning — Life Just Got Easier!!</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Get your hands dirty with Hadoop Map Reduce Job using Cloud Era platform -Big Data]]></title>
            <link>https://medium.com/analytics-vidhya/get-your-hands-dirty-with-hadoop-map-reduce-job-using-cloud-era-platform-big-data-f929fe0f12b4?source=rss-27ae3f584d17------2</link>
            <guid isPermaLink="false">https://medium.com/p/f929fe0f12b4</guid>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[hdfs]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[cloudera]]></category>
            <dc:creator><![CDATA[Maajid Khan]]></dc:creator>
            <pubDate>Thu, 12 Sep 2019 15:06:22 GMT</pubDate>
            <atom:updated>2019-09-16T10:44:05.321Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Get your hands dirty with Hadoop Map Reduce Job using Cloudera platform</strong></h3><h3><strong>Analysis on Sales Data for Chain of Stores using Cloud Era Platform</strong></h3><h3>What we will be Achieving through this blog!!</h3><blockquote><strong>•Analyse different use-cases where MapReduce is used.</strong></blockquote><blockquote><strong>•Implement basic MapReduce Concepts.</strong></blockquote><blockquote><strong>•Run a MapReduce program.</strong></blockquote><blockquote><strong>•Understand Input Splits concept in MapReduce.</strong></blockquote><blockquote><strong>•Understand Map Reduce Job Submission flow.</strong></blockquote><blockquote><strong>•Implement Job Tracker and Job Tracer in MapReduce.</strong></blockquote><h3><strong>What’s MapReduce?</strong></h3><p>As a result of technologies such as cloud computing, virtualization and big data, MapReduce is the new programming paradigm used to process big data, called big data. MapReduce calculations take place in thousands of single computers associated with the cloud. Thus, it can leverage the graphics processing units (GPU’s) associated with the cloud with its parallel processing capabilities. Real-world companies are moving from traditional computing to cloud computing and traditional data mining to big data analytics. The reason behind this is the exponential growth of data. Storing and processing such data requires a large data ecosystem associated with cloud computing. In this context, the MapReduce programming model is supported by distributed programming frameworks such as Hadoop. However, it is very difficult to secure MapReduce calculations against malicious attacks.</p><p>There are many cloud storage mechanisms in the literature. However, securing the MapReduce programming paradigm in Hadoop and the Big Data ecosystem remains to be explored. In this article, we proposed a differential privacy algorithm to protect big data from malware Mapper and Reducer. We built an application prototype to demonstrate proof of concept. The result showed the utility of the proposed approach.</p><p>Recently, companies have begun to place greater emphasis on data and data analysis for strategic decision-making. Due to the exponential growth of data in real-world applications, it has become essential to have a good ecosystem that can be used to store and manage big data characterized by volume, speed, and variety. The emergence of cloud computing has enabled the processing of large data by providing a large pool of fashionable shared computing resources. At the same time, a Hadoop distributed programming framework appeared. This infrastructure supports the storage and processing of a huge amount of data and leverages parallel processing with thousands of standard computers. Because large data processing requires computing power and storage, cloud computing has become the way to do it. With cloud computing, virtualization, big data, Hadoop and other big data platforms, an eco-system is created in a distributed environment. MapReduce is the programming model used for large data processing. This is the new programming approach that contains the map and reduces the tasks. The work is done in an environment distributed by many standard computers. As noted, Amazon provides its own MapReduce programming model, called Elastic MapReduce, based on Hadoop. Therefore, Hadoop is the framework used to handle big data by supporting MapReduce programming. However, security issues arise when the mapper or gearbox is compromised. Mapping and job reduction are vulnerable to various types of attacks. In this article, we used the mapper and reducer to calculate the total sales of each store in the United States.</p><h3><strong><em>The Idea</em></strong></h3><p>To analyse and determine the revenue earned across different stores in a country/region.</p><p>Use the core concepts, HDFS and MapReduce of Big Data Hadoop Platform to solve the problem.</p><p>Processing large volumes of data in parallel by dividing the work into a set of independent tasks.</p><p>Use MapReduce process file to break it into chunks and process it in parallel.</p><h3><strong>Preliminary concept of data</strong></h3><h3><strong>A. Data Description</strong></h3><p>The data set is taken from Udacity, an online learning platform providing wide range of courses. The data set contains the data regarding the sales data of each store of Walmart in a particular year. This data set contains categorical variables and numerical variables. There are 6 attributes and more than 10,00,000 instances.</p><p><strong>The following are the necessary fields:</strong></p><p>· Date</p><p>· Time</p><p>· Location</p><p>· Item</p><p>· Price</p><p>· Card</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*VWx1lXo6G7l87Hb-7Ylojw.png" /><figcaption><strong>Fig. 1 Store Sales Dataset</strong></figcaption></figure><p>The data set can be downloaded from this link:</p><p><a href="https://drive.google.com/file/d/1Ar12GZISlf-JMQSj1FQsy0_XhSqMH4F9/view?usp=sharing"><strong>https://drive.google.com/file/d/1Ar12GZISlf-JMQSj1FQsy0_XhSqMH4F9/view?usp=sharing</strong></a></p><h3><strong>B. Methodology</strong></h3><p><strong>Hadoop Framework</strong></p><p>Hadoop is an open source framework that can be used for the processing of huge volume of data, i.e. the size of the data may be in peta bytes. It is a distributed computing framework. The architecture of the Hadoop is Master Slave architecture where the bottom layer of Hadoop includes other machines which works in parallel. The large volume of data is split into different data set which runs in different machines. Each machine will execute the task. Each machine will have a task tracker and job tracker. The main components of Hadoop framework are HDFS (Hadoop Distributed File System) and MapReduce concept.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*9adeUEgPMpLrwraXSl3_tA.png" /><figcaption><strong>Fig. 2 High Level Design of Hadoop Framework</strong></figcaption></figure><p><strong>MapReduce Concept</strong></p><p>MapReduce is a programming model. It simplifies the processing by splitting in parallel the large volume of data and send in into different machines. It supports both structured as well as unstructured data formats. The main programming language used is Java but also supports other programming languages such as python, c, c++. Other data analytics programming tool such as R can also be integrated with the Hadoop.</p><p><strong>MapReduce programming has two functions:</strong></p><p><strong>a. Mapper function</strong></p><p>In this function, the large input is divided into sub problem inputs or tasks. These tasks are distributed to the nodes where these nodes also divide the sub tasks into sub -sub tasks. Once the processing is done, it maps into key value pair format. Before the mapping is sent to reducer function, they are shuffled and sorted based on the keys.</p><p><strong>b. Reducer function</strong></p><p>This function is used for processing such as finding the total count for each keys or other analysis that needs to be found out. It produces a single result or set of results.</p><p><strong>HDFS — Hadoop Distributed File System</strong></p><p>Before processing, the large files from the local disk or real time data are stored in the distributed file system of Hadoop known as HDFS. It has nodes for the storage and for computation. It is possible to create, delete or manage the files within the HDFS. It also allows the creation of directories. It has data node and name node for storing and accessing the files.</p><h3><strong>C.</strong> <strong>Process Design</strong></h3><p>Suppose there is a big file of sales data that contains the sales data of each store of Walmart in a particular year. The rows of the file look something like this. We need to find out the total sales of each individual store in 2012.Now as the file will contain millions of items it is not feasible to process the file serially. Map Reduce helps us to divide the file into smaller chunks, process those chunks on different machines in a cluster and then combine the results.</p><p>Instead of one machine doing the job we will have a set of machines called mappers and reducers that will help to run this process in parallel. What will be the job of the mappers and the reducers? The file will be divided into smaller chunks and we will give one chunk to each mapper. The job of the mapper will be to take the chunk and separate the sales data of each store. For example if a mapper gets the above records he will make three piles namely NYC , MIAMI and LA and keep the sales data of each in the particular pile. Each reducer will be assigned a group of stores. The reducer will collect the data from the mappers of its assigned store and sum up the values of sales of that store. For example, if the first reducer is assigned NYC he will collect the NYC sales data from each mapper and sum up the values to get the total sales for NYC. Each reducer goes through his piles in alphabetical order. So, the second reducer will process the sales of LA before Miami.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*AZImlpDphmxbw8-p3U39nw.png" /><figcaption><strong>Mappers-Reducers</strong></figcaption></figure><p>So, the mappers are just programs each of which acts on a small chunk of the file. The mappers produce an intermediate key value pair. In our case it is the store name and the Item Price. Once the mappers have done their job a phase known as Shuffle and Sort takes place. Shuffle is the movement of records from mappers to the reducers that have been assigned those records. Sort is the sorting of data by the particular reducer. The reducers get a key and a list of values. In our case the store Name and the sales data of each store. It iterates through all the values and produces the total sales result of each store in the end.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*r-cjBluhGTnTDb01fUygOg.png" /><figcaption><strong>Key-Value pairs</strong></figcaption></figure><h3><strong>D. Implementation</strong></h3><p>Now it’s time to get hands on with Hadoop Map Reduce. Before running a Map Reduce job you need to install certain things that will help to setup the Hadoop Environment on your machine. You can download the <a href="http://www.cloudera.com/content/support/en/downloads/download-components/download-products.html">CDH </a>package from cloudera site and try to set up the environment. Or if you are lazy like me just download this <a href="http://content.udacity-data.com/courses/ud617/Cloudera-Udacity-Training-VM-4.1.1.c.zip.">Virtual Machine</a> which already has the environment setup and run it with Virtual Box. I have already installed CloudEraDistribution using Oracle VM VirtualBox Manager. You can do the same, just google it out or watch an youtube video.</p><p>Usually Map Reduce code is written in java. But with a feature called Hadoop Streaming you can write your mappers and reducers in any language. We will use python to write the Map reduce job.</p><p>I am assuming that you have already installed the virtual machine. Once you open the machine go to the directory where your project files are located. It will have an mapper and reducer program. It contains a file ``` <strong>purchases.txt ```</strong>that contains the sales data for a dummy store. First, we need to add this input into the Hadoop cluster. To run any command on the Hadoop cluster we need to append it with <strong>Hadoop fs</strong>.</p><p>1. Go to the folder where your project files are located in your machine.</p><p>2. Put the <strong>purchases.txt</strong> file in the hadoop cluster using<em> </em><strong><em>hadoop fs -put purchases.txt</em></strong></p><p>3. Check if the file is in the hadoop cluster using<em> </em><strong><em>hadoop fs -ls</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*mLl-1QQgRzyA7iM6iC_-Rw.png" /><figcaption><strong>HDFS (CloudEra)</strong></figcaption></figure><p>Now let’s take a look at the mapper and reducer code. Our purchases.txt data looks like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*cyL9hdSPaoG71wl4g17UWw.png" /><figcaption><strong>purchases.txt sample</strong></figcaption></figure><p>The project folder contains the<strong> mapper.py</strong> and<strong> reducer.py</strong> files.</p><p><strong>The mapper code is shown below.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*oSHoqqckekOkEfKIUB8xaw.png" /><figcaption><strong>mapper code (mapper.py)</strong></figcaption></figure><p>The mapper reads the chunk of the file and splits the file by using tab delimiter. Then it prints out only the store location and item cost as the output.</p><p><strong>The reducer code is as shown.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*eHBTRi7HoHbHKD2Itg6EXw.png" /><figcaption><strong>reducer code (reducer.py)</strong></figcaption></figure><p>We are assuming that we are having only one reducer that will get the sorted input from the mappers. The reducer.py takes in the sorted input and keeps on checking whether the new key is equal to the previous key. When the change occurs, it prints the store name and the total sales for the store.</p><p>Once you have written the mappers and reducers you can start the map reduce job using the command:</p><blockquote><strong>hadoop jar /usr/lib/hadoop-0.20-mapreduce/contrib/streaming/hadoop-streaming-2.6.0-mr1-cdh5.12.0.jar -mapper mapper.py -reducer reducer.py -file mapper.py -file reducer.py -input myinput -output joboutput</strong></blockquote><p>After you run this, it takes some time to compute. You can watch how mappers and reducers are running in the output terminal. The terminal will shows variety of information like the percentage of the completion of job.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*JQwYtiXYVulbCvrL4DMxjw.png" /><figcaption><strong>Output Terminal-1</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*dYieRCFIaVItfMMrZRAetQ.png" /><figcaption><strong>Output Terminal-2</strong></figcaption></figure><p>The output of our program gets saved in the file <strong>joboutput/part-00000</strong></p><p>You can view the output in the same terminal by running the command <strong>hadoop fs -ls joboutput</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/471/1*XLjCtY6j_SF0ghuqh4Db1Q.jpeg" /><figcaption><strong>Output (Total Revenue generated across each store)</strong></figcaption></figure><p>Instead of viewing the whole output, you can view a smaller part of our output by using the command <strong>hadoop fs -cat joboutput/part-00000 | less</strong></p><p>Once your task is done, now you can delete that joboutput file from out HDFS system. Reason being, we can’t have different output workloads being generated with the same output file name. This can be achieved by running the following command <strong>hadoop fs -rm -r -f joboutput</strong></p><p>Meanwhile, you can export the data into a text file and save it in your directory if you want for future use.</p><h3><strong>E. Conclusion</strong></h3><p>To analyse and determine the revenue earned across different stores in a country/region, we used the core concepts — HDFS and MapReduce from Big Data Hadoop Platform to solve this problem.<strong> We </strong>Processed large volumes of data in parallel by dividing the work into a set of independent tasks.<strong> We u</strong>sed MapReduce process file to break the data into chunks and process it in parallel and were able to find out the revenue earned by individual store in US using reducers.</p><p>That is all for this post. I hope you really understand the myth behind HDFS and MapReduce Framework and how to implement it.</p><p>The code for this project can be found at my GitHub page:</p><p>``<strong> https://github.com/MaajidKhan/HadoopMapReduce_BigData ``</strong></p><p>If you like this post, hit the clap icon below and share this post with your friends and network. Thanks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f929fe0f12b4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/get-your-hands-dirty-with-hadoop-map-reduce-job-using-cloud-era-platform-big-data-f929fe0f12b4">Get your hands dirty with Hadoop Map Reduce Job using Cloud Era platform -Big Data</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploy your Machine Learning Model using Flask — Made Easy Now]]></title>
            <link>https://medium.datadriveninvestor.com/deploy-your-machine-learning-model-using-flask-made-easy-now-635d2f12c50c?source=rss-27ae3f584d17------2</link>
            <guid isPermaLink="false">https://medium.com/p/635d2f12c50c</guid>
            <category><![CDATA[flask]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Maajid Khan]]></dc:creator>
            <pubDate>Wed, 11 Sep 2019 19:21:47 GMT</pubDate>
            <atom:updated>2019-09-12T14:33:58.551Z</atom:updated>
            <content:encoded><![CDATA[<h3>Deploy your Machine Learning Model using Flask — Made Easy Now</h3><p>A quick cheat sheet to deploy your ML model in 10 min.</p><h4><strong>You’ve</strong> built a machine learning model using python. It turns out to be super cool with good prediction results. It would be nice if there’s a way you can share this thing with your friends and family. So, how can we do this? Is there a simpler way of doing this?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*5OGb3HSY8V4rcAgrdw1FvA.png" /><figcaption><strong>Flask + ML model</strong></figcaption></figure><p>The easiest way of doing it is by deploying the model using flask. When we start learning machine learning, initially we do it by running a simple supervised learning model. The most important and the easiest one to understand and use is a <strong>Regression model</strong>. This blog is under the assumption that the person has a fair knowledge of running python code and is familiar with simple ML libraries like <strong>Sci-kit Learn</strong>, P<strong>andas</strong>, N<strong>umpy</strong>.</p><p>Different users have different choices when it comes to the type of python IDE they want to use. Some of the popular IDE’s are <strong>Spyder</strong>, <strong>Jupyter</strong> Notebook, <strong>PyCharm</strong> editor or you can just run it on python compiler.</p><p><a href="https://www.datadriveninvestor.com/2019/03/03/editors-pick-5-machine-learning-books/">DDI Editor&#39;s Pick: 5 Machine Learning Books That Turn You from Novice to Expert | Data Driven Investor</a></p><p>When you want to run line by line and check the output, it’s idle to go with Jupyter notebooks. When you want to run a chunk of code altogether, Spyder is the better choice and if you want your whole project to look organized, has a lot of files and you want to make it look in a structured way, it’s good to go with PyCharm<strong> </strong>IDE.</p><p>In our blog, we will be using Spyder framework as an IDE provided by <strong>Anaconda Navigator</strong>. You can use anything of your choice and with which you are comfortable with.</p><p>In this post, we will be building a simple regression model using <strong>scikit-learn’s</strong> built-in <strong>LinearRegression()</strong> function. First, let us have a look at how our data looks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/378/1*o_yd6119thHZ2dH1oQUBuA.png" /><figcaption><strong>hiring.csv (Dataset)</strong></figcaption></figure><p>It’s a very simple data set with 4 attributes and 8 rows. All the columns are numerical columns, so we need to do any data pruning or cleaning up the data to convert categorical data into numerical before you build your classifier (regression model). This data used here is clean and is ready to be applied for a classifier.</p><p>Normally, you have to first clean up your data like filling up empty values in your data set, converting your categorical data into numerical data, normalization, etc; as part of our data cleaning and pruning. But this blog purpose is mainly serving the purpose of deploying the model using flask, so we have considered simpler data which is already clean and requires no further pre-processing.</p><p>Here, the class label will be the last salary column, which we will be predicting using our model. After our model is built, we will be saving our trained model using a library called pickle.</p><p><strong>Pickle</strong></p><p>Python pickle module is used for serializing and de-serializing a Python object structure. Any object in Python can be pickled so that it can be saved on disk. What pickle does is that it “serializes” the object first before writing it to file. Pickling is a way to convert a python object (list, dict, etc.) into a character stream. The idea is that this character stream contains all the information necessary to reconstruct the object in another python script.</p><p>Here’s the code of building a simple regression model and then saving the model by serializing it using pickle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*y_ONiFkvU_1eVt0B-fkpMA.png" /><figcaption><strong>model.py</strong></figcaption></figure><p>When you run this, your model is now trained and your model gets saved in the directory where your project files are stored in your local machine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*3x-4cGE7ZTYGN5MiJ6MAIA.png" /><figcaption><strong>Project directory</strong></figcaption></figure><p>Now that your model is ready and saved. It’s time we start building a flask model.</p><h3><strong>Setting up flask-app over the localhost:</strong></h3><p>Make sure flask is installed.</p><p>``` <strong>pip install flask</strong> ```</p><p><strong>The flask code can be explained in three sections:</strong></p><h4><strong>1.Loading the saved model</strong></h4><p>We load the model.pkl file and initialize the flask app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/335/1*3WrJyB8eQAoxD8m_T7IKaQ.png" /><figcaption><strong>Loading the saved model ( model.pkl )</strong></figcaption></figure><p><strong>2. Redirecting the API to the home page index.html</strong></p><p>After initializing the app, we have to tell Flask what we want to do when the web page loads. The line @app.route(&quot;/&quot;, methods = [&quot;GET&quot;,&quot;POST&quot;]) tells Flask what to do when we load the home page of our website.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/303/1*hXQtiwodxH9rwxKopTIlIQ.png" /><figcaption><strong>Redirecting the API (Routing)</strong></figcaption></figure><p>We use <strong>@app.route(‘/’)</strong> to define functions which are used to redirect them into any number of URI with respect to the API. So, when you start the flask server, it redirects to index.html file by default in our case.</p><p><strong>3. Redirecting the API to predict the result (salary)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*LZKRaJweVHQSeV4O5gRqhA.png" /><figcaption><strong>Predict API</strong></figcaption></figure><p>Since it is a <strong>‘POST’</strong> request, it will be reading the input values from request.form.values(). Now that we have the input values in the variable <strong>int_features</strong>, we will convert it into an array and then use the model to predict it and round the final prediction to two decimal places.</p><p>When we click on the predict button in index.html, it predicts the salary for the values entered by the user (3 inputs), then passes on the variable ``` <strong>output</strong> ``` outputted from the model and sends it back to index.html template as <strong>prediction_text</strong>.</p><p><strong>Let us have a look over index.html file:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*w0R7YsuNl8MYU-1IMFqwxg.png" /><figcaption><strong>index.html</strong></figcaption></figure><p>The <strong>{{ prediction_text }}</strong> placeholder you see here is where your output prediction(salary predicted) from the model will be placed in our index.html file.</p><p><strong>4. Starting the flask server</strong></p><p>This will call app.run() and run our web page locally, hosted on your computer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/187/1*ivHuUZ_DeGP1U9YNpdBqPQ.png" /><figcaption><strong>Starting the Flask Server</strong></figcaption></figure><p>The following script starts the flask server on localhost and default port (5000) making <a href="http://127.0.0.1:5000/">http://127.0.0.1:5000/</a></p><p>Just paste <a href="http://127.0.0.1:5000/"><strong>http://127.0.0.1:5000/</strong></a> (or) <a href="http://localhost:5000/"><strong>http://localhost:5000</strong></a> on browser and press enter to see the server working.</p><h3><strong>The overall structure of the project:</strong></h3><p>This project has four major parts:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/134/1*uMTkqHMGFxFUFtNj48pGpw.png" /><figcaption><strong>Project Directory Structure</strong></figcaption></figure><p><strong>1. model.py </strong>— This contains code for our Machine Learning model to predict employee salaries based on training data in <strong>‘hiring.csv’</strong> file.</p><p><strong>2. app.py </strong>— This contains Flask APIs that receives employee details through GUI or API calls, computes the predicted value based on our model and returns it.</p><p><strong>3. template </strong>— This folder contains the HTML template (<strong>index.html</strong>) to allow user to enter employee detail and displays the predicted employee salary.</p><p><strong>4. static </strong>— This folder contains the <strong>css</strong> folder with <strong>style.css</strong> file which has the styling required for out index.html file.</p><h3><strong>Running the project:</strong></h3><p><strong>1. Ensure that you are in the project home directory. Create the machine learning model by running below command from command prompt -</strong></p><p>```</p><p><strong>python model.py</strong></p><p>```</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/1*QFUTTexrwmdNLhpp_QsMSw.png" /><figcaption><strong>model.py</strong></figcaption></figure><p>This would compile our linear regression classifier, build the trained model and creates a serialized version of our model and save it as a file named <strong>model.pkl</strong></p><p><strong>2. Run app.py using below command to start Flask API</strong></p><p>```</p><p><strong>python app.py</strong></p><p>```</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*X63lXT4ImOIh_R-K3H_mFQ.png" /><figcaption><strong>app,py</strong></figcaption></figure><p>By default, flask will run on port 5000.</p><p><strong>3. Navigate to URL </strong><a href="http://127.0.0.1:5000/"><strong>http://127.0.0.1:5000/</strong></a><strong> (or) </strong><a href="http://localhost:5000/"><strong>http://localhost:5000</strong></a></p><p>You should be able to view the homepage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*-9tejiu1kCLY4isH7w8Cdg.png" /><figcaption><strong>Home page (index.html)</strong></figcaption></figure><p>Enter valid numerical values in all 3 input boxes and hit the button <strong>Predict</strong>.</p><p>If everything goes well, you should be able to see the predicted salary value on the HTML page!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/1*PJEPpqblPb-2LI0q_t0ChA.png" /><figcaption><strong>Predicted value</strong></figcaption></figure><p>That is all folks. It is that easy to deploy an application using flask. I hope you liked this post. Now, you can start working on this and building your own application. The code for this project can be found at my GitHub page:</p><blockquote><strong>https://github.com/MaajidKhan/DeployMLModel-Flask</strong></blockquote><p>Thanks for reading, and keep an eye out for my next article on deploying a deep learning model using Flask!</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fb2a0d6%3Fas_embed%3Dtrue&amp;dntp=1&amp;display_name=Upscribe&amp;url=https%3A%2F%2Fupscri.be%2Fb2a0d6%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href">https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=635d2f12c50c" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/deploy-your-machine-learning-model-using-flask-made-easy-now-635d2f12c50c">Deploy your Machine Learning Model using Flask — Made Easy Now</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>