<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shatakshi Singh on Medium]]></title>
        <description><![CDATA[Stories by Shatakshi Singh on Medium]]></description>
        <link>https://medium.com/@shatakshisingh2k?source=rss-94ea5048038c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ivWq1S8mhXk5xQN6_YuIpA.png</url>
            <title>Stories by Shatakshi Singh on Medium</title>
            <link>https://medium.com/@shatakshisingh2k?source=rss-94ea5048038c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 00:28:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shatakshisingh2k/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[What Working in Tech Quietly Taught Me]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/swlh/what-working-in-tech-quietly-taught-me-35c49c45283a?source=rss-94ea5048038c------2"><img src="https://cdn-images-1.medium.com/max/2600/1*kbceecnmy64oRloHHYdUNA.jpeg" width="2998"></a></p><p class="medium-feed-snippet">Now, a few years into working as a data engineer, I realize how incomplete my picture of &#x201C;being good in tech&#x201D; used to be. A few years ago&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/swlh/what-working-in-tech-quietly-taught-me-35c49c45283a?source=rss-94ea5048038c------2">Continue reading on The Startup »</a></p></div>]]></description>
            <link>https://medium.com/swlh/what-working-in-tech-quietly-taught-me-35c49c45283a?source=rss-94ea5048038c------2</link>
            <guid isPermaLink="false">https://medium.com/p/35c49c45283a</guid>
            <category><![CDATA[engineering]]></category>
            <category><![CDATA[experience]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[tech]]></category>
            <dc:creator><![CDATA[Shatakshi Singh]]></dc:creator>
            <pubDate>Thu, 11 Jun 2026 13:37:21 GMT</pubDate>
            <atom:updated>2026-06-12T10:25:34.400Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Mastering Workflow Automation: A Quick Introduction to Apache Airflow]]></title>
            <link>https://shatakshisingh2k.medium.com/getting-started-with-apache-airflow-a-beginners-guide-f25e67bbc829?source=rss-94ea5048038c------2</link>
            <guid isPermaLink="false">https://medium.com/p/f25e67bbc829</guid>
            <category><![CDATA[why-use-airflow]]></category>
            <category><![CDATA[when-to-use-airflow]]></category>
            <category><![CDATA[what-is-airflow]]></category>
            <category><![CDATA[advantages-of-airflow]]></category>
            <category><![CDATA[substitutes-of-airflow]]></category>
            <dc:creator><![CDATA[Shatakshi Singh]]></dc:creator>
            <pubDate>Tue, 04 Apr 2023 18:19:03 GMT</pubDate>
            <atom:updated>2023-04-04T18:23:59.284Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iPF_70zBWvdhV_G5Ft41Wg.png" /></figure><p>Apache Airflow is an <strong>open-source</strong> platform that allows users to programmatically create, schedule, and monitor workflows, which are composed of a series of tasks that are executed on a regular basis. Airflow provides a flexible and extensible framework for defining and executing workflows. This makes it a popular choice for data engineering and ETL (Extract, Transform, Load) tasks in many organizations.</p><p>There are several other similar platforms like Apache Airflow that are used for workflow management and automation. Some popular alternatives to Airflow include:</p><ol><li><strong>Luigi</strong>: Developed by Spotify, Luigi is another Python-based workflow management system that supports complex workflows and dependencies.</li><li><strong>Oozie</strong>: Developed by Apache, Oozie is a Java-based workflow scheduler and coordinator that is designed to integrate with the Hadoop ecosystem.</li><li><strong>Azkaban</strong>: Azkaban is a Java-based workflow management system developed at LinkedIn that is designed to simplify the process of creating and running jobs on Hadoop clusters.</li><li><strong>Jenkins</strong>: While primarily known as a continuous integration and delivery tool, Jenkins can also be used for workflow management and automation.</li><li><strong>Kubernetes</strong>: While not strictly a workflow management system, Kubernetes provides powerful tools for container orchestration and can be used to automate and manage complex distributed workflows.</li></ol><p>Each of these platforms has its own pros and cons. The best choice for your use case will depend on your specific needs and requirements.</p><p>Airflow is particularly useful when there is a need to manage and automate complex workflows with dependencies between tasks. Some specific use cases where Airflow can be particularly helpful include:</p><ol><li><strong>Data processing pipelines</strong>: Airflow can help you manage and automate the various stages of your data processing pipeline, from data ingestion and cleaning to transformation and analysis.</li><li><strong>ETL workflows</strong>: Airflow is well-suited for managing extract, transform, and load (ETL) workflows, which can involve multiple steps and dependencies.</li><li><strong>Machine learning workflows</strong>: Airflow can help you manage and automate the various stages of your machine learning workflow, from data preparation and feature engineering to model training and evaluation.</li><li><strong>Reporting and analysis workflows</strong>: Airflow can help you automate the process of generating and delivering reports and dashboards, which can involve multiple data sources and complex dependencies.</li></ol><p>Overall, if you need to manage and automate complex workflows with dependencies between tasks, then Airflow can be a powerful and flexible tool to help you achieve your goals.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f25e67bbc829" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Visualisation]]></title>
            <link>https://shatakshisingh2k.medium.com/data-visualisation-764a59427c12?source=rss-94ea5048038c------2</link>
            <guid isPermaLink="false">https://medium.com/p/764a59427c12</guid>
            <dc:creator><![CDATA[Shatakshi Singh]]></dc:creator>
            <pubDate>Sun, 16 Jan 2022 07:08:39 GMT</pubDate>
            <atom:updated>2022-01-16T07:08:39.317Z</atom:updated>
            <content:encoded><![CDATA[<p>Part-3</p><p>Bar Graphs are commonly used to represent data. Lets see how to plot attractive and insightful bar graphs using matplotlib.</p><h4>1. Plot a simple bar graph with two arrays.</h4><pre>plt.bar([0,1,2,3],[12,23,21,8])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/1*f7jly6PUA8j3ozPrZh5mDA.png" /></figure><p>You can see the bars are centered around 0,1,2 and 3 x-coordinate.</p><p>Now let&#39;s see how can we plot graphs to compare data of two years. Suppose you are given production of grain in 2018 and 2019 for 3 months.</p><pre>plt.bar([0,1,2],[34,46,23]) #Production in 2018</pre><pre>plt.bar([0,1,2],[23,45,43]) #Production in 2019</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*MxG-jnjXM3ih8z3uvdot3A.png" /></figure><p>OOPS! Data of 2018 and 2019 overlapped. This happened because both are centered around 0,1, and 2.</p><p>So we will change the position on the x-axis for one of them. Let’s use and array <em>pos_x=np.array([0,1,2]) </em>to define position on x-axis for one of them. And for other we will add 0.5.</p><pre>pos_x=np.array([0,1,2])</pre><pre>plt.bar(pos_x,[34,46,23]) #Production in 2018</pre><pre>plt.bar(pos_x+0.25,[23,45,43]) #Production in 2019</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*Ii4ApNJKfGHXnIxfKtHkQA.png" /></figure><p>It still overlaps. Now we can either reduce the thickness or increase the gap in pos_x values. Let’s try to reduce the width.</p><pre>pos_x=np.array([0,1,2])</pre><pre>plt.bar(pos_x,[34,46,23], width=0.5) #Production in 2018</pre><pre>plt.bar(pos_x+0.5,[23,45,43], width=0.5) #Production in 2019</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*PxTYiZRm7wtguV-HU-d3Qg.png" /></figure><p>Great, No Overlapping now! Still, it would look good if there is some space in between data of different months.</p><pre>pos_x=np.array([0,1,2])*2</pre><pre>plt.bar(pos_x,[34,46,23], width=0.5) #Production in 2018</pre><pre>plt.bar(pos_x+0.5,[23,45,43], width=0.5) #Production in 2019</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/1*1puGOtG8wadOcqbiVOWRxg.png" /></figure><p>Looks good now.</p><h4>2. Give Labels</h4><p>We will now give labels to each color and will plot a legend.</p><pre>pos_x=np.array([0,1,2])*2</pre><pre>plt.bar(pos_x,[34,46,23], width=0.5, label=”2018&quot;) #Production in 2018</pre><pre>plt.bar(pos_x+0.5,[23,45,43], width=0.5, label=”2019&quot;) #Production in 2019</pre><pre>plt.legend()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*_gtHDBY5vVmnwn75xCOkKQ.png" /></figure><h4>3. Put labels on x-axis</h4><p>To make it easy to interpret we will give the names of months for which production is compared. We do it using the <em>tick_labels </em>parameter.</p><pre>pos_x=np.array([0,1,2])*2</pre><pre>plt.bar(pos_x,[34,46,23], width=0.5, label=”2018&quot;, tick_label=[“Jan”, “Feb”, “Mar”]) #Production in 2018</pre><pre>plt.bar(pos_x+0.5,[23,45,43], width=0.5, label=”2019&quot;) #Production in 2019</pre><pre>plt.legend()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/1*pJGbO_H10tx7j0dIsBt0IQ.png" /></figure><h4>4. Give a title to the plot and both the axes</h4><pre>pos_x=np.array([0,1,2])*2</pre><pre>plt.bar(pos_x,[34,46,23], width=0.5, label=”2018&quot;, tick_label=[“Jan”, “Feb”, “Mar”]) #Production in 2018</pre><pre>plt.bar(pos_x+0.5,[23,45,43], width=0.5, label=”2019&quot;) #Production in 2019</pre><pre>plt.title(“Wheat Production”)</pre><pre>plt.xlabel(“Months”)</pre><pre>plt.ylabel(“Production in Kgs”)</pre><pre>plt.legend()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/427/1*odplXrOAE3vVfVUJQJE8iA.png" /></figure><p>Now let&#39;s try to change the theme to make it look more attractive.</p><p>Above your existing code just write this one line.</p><pre>plt.style.use(“dark_background”)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/1*mXEiYoUAA6QJhsyEGxB2hg.png" /></figure><h4>5. Change the color of bars</h4><pre>plt.style.use(“dark_background”)</pre><pre>pos_x=np.array([0,1,2])*2</pre><pre>plt.bar(pos_x,[34,46,23], width=0.5, label=”2018&quot;, tick_label=[“Jan”, “Feb”, “Mar”], color=”blue”) #Production in 2018</pre><pre>plt.bar(pos_x+0.5,[23,45,43], width=0.5, label=”2019&quot;, color=”orange”) #Production in 2019</pre><pre>plt.title(“Wheat Production”)</pre><pre>plt.xlabel(“Months”)</pre><pre>plt.ylabel(“Production in Kgs”)</pre><pre>plt.legend()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/1*6jhNjOQtmQvY8uNnhMtFfA.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=764a59427c12" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Visualisation using Matplotlib for beginners (Part 2)]]></title>
            <link>https://shatakshisingh2k.medium.com/data-visualisation-in-python-d0744298be18?source=rss-94ea5048038c------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0744298be18</guid>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[matplotlib]]></category>
            <category><![CDATA[graphs-are-fun]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Shatakshi Singh]]></dc:creator>
            <pubDate>Sat, 12 Jun 2021 20:47:29 GMT</pubDate>
            <atom:updated>2021-06-12T20:50:58.924Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*imGHKkDzkszm_qM-" /><figcaption>Photo by <a href="https://unsplash.com/@chrisliverani?utm_source=medium&amp;utm_medium=referral">Chris Liverani</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In the <a href="https://shatakshisingh2k.medium.com/data-visualisation-using-matplotlib-for-beginners-a382a537f77d"><em>previous blog</em></a>, we saw how to plot simple line plots and do some formatting to make them look attractive. In case you haven&#39;t checked it out…check it now!</p><p>After the line plots let’s work with Scatter Plots.</p><h3>Scatter Plots</h3><p>We will use the same data as we used for the line plot in the <a href="https://shatakshisingh2k.medium.com/data-visualisation-using-matplotlib-for-beginners-a382a537f77d"><em>previous blog</em></a>.</p><h4>1. Make a simple scatter plot</h4><p>We have three arrays x,y1, and y2.</p><pre>plt.scatter(x,y1)</pre><pre>plt.scatter(x,y2)</pre><pre>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/1*uxTrhMZ9y0xCm_uVxuF_Eg.png" /></figure><h4>2. Put Titles</h4><p>The process remains the same as we did for the line plot</p><pre>plt.xlabel(“Time”)</pre><pre>plt.ylabel(“Speed”)</pre><pre>plt.title(“Speed v/s Time Curve”)</pre><h4>3. Change the symbols</h4><p>Till now you were getting dots on your plot. You can change this symbol using the marker parameter.</p><p>We can use symbols like stars and triangles.</p><pre>plt.scatter(x,y1, color=”red”, label=”A”, marker=”*”)</pre><pre>plt.scatter(x,y2, color=”green”, label=”B”)</pre><pre>plt.legend()</pre><pre>plt.xlabel(“Time”)</pre><pre>plt.ylabel(“Speed”)</pre><pre>plt.title(“Speed v/s Time Scatter Plot”)</pre><pre>plt.show</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/1*TlUYcUTxeF1aSH9K9WPrIg.png" /></figure><pre>plt.scatter(x,y1, color=”red”, label=”A”, marker=”^”)</pre><pre>plt.scatter(x,y2, color=”green”, label=”B”)</pre><pre>plt.legend()</pre><pre>plt.xlabel(“Time”)</pre><pre>plt.ylabel(“Speed”)</pre><pre>plt.title(“Speed v/s Time Scatter Plot”)</pre><pre>plt.show</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/1*qRLO6f2LqjZRNaz63Ttxog.png" /></figure><h4>4. Adjust the size of any plot</h4><p>We can adjust the size of any plot with help of <em>plt. figure(figsize=(2,2)). </em>You can give any dimensions in place of (2,2), and see the output.</p><pre>plt.figure(figsize=(2,2))</pre><pre>plt.scatter(x,y1, color=”red”, label=”A”, marker=”^”)</pre><pre>plt.scatter(x,y2, color=”green”, label=”B”)</pre><pre>plt.legend()</pre><pre>plt.xlabel(“Time”)</pre><pre>plt.ylabel(“Speed”)</pre><pre>plt.title(“Speed v/s Time Scatter Plot”)</pre><pre>plt.show</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/173/1*7P7c4YuXUDy8l90CHEgBbQ.png" /></figure><pre>plt.figure(figsize=(10,5))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*zN7KUIc7Qp3sfKG-4BAOxA.png" /></figure><pre>plt.figure(figsize=(8,2))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*_hA4AeDpF6sDltpVFFczuA.png" /></figure><p>You can choose the best fit as per your need. Keep experimenting with more parameters.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0744298be18" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Visualisation using Matplotlib for beginners.]]></title>
            <link>https://shatakshisingh2k.medium.com/data-visualisation-using-matplotlib-for-beginners-a382a537f77d?source=rss-94ea5048038c------2</link>
            <guid isPermaLink="false">https://medium.com/p/a382a537f77d</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python-matplotlib]]></category>
            <category><![CDATA[data-visualisation]]></category>
            <category><![CDATA[matplotlib]]></category>
            <dc:creator><![CDATA[Shatakshi Singh]]></dc:creator>
            <pubDate>Sat, 21 Nov 2020 14:40:37 GMT</pubDate>
            <atom:updated>2020-11-21T14:40:37.384Z</atom:updated>
            <content:encoded><![CDATA[<p>Making it easy to understand data using insightful and attractive plots!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KGnOXhPBkzaIsJrE" /><figcaption>Photo by <a href="https://unsplash.com/@lukechesser?utm_source=medium&amp;utm_medium=referral">Luke Chesser</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>While solving any data science related problem the first step is to understand the problem statement. Then we acquire data and clean it. Then we do the exploratory analysis for which we use data visualization to get insights. Then comes the modelling and finally we can share the results.</p><p>Let’s see how can we visualize data using the following types of plots:</p><ul><li>Line plots</li><li>Scatter Plots</li><li>Bar Graphs</li><li>Pie Charts</li><li>Histograms</li></ul><h3>Line Plots</h3><p>To begin let us see how to plot the function y=f(x) on the graph. To do this we need two arrays x and y. where y=f(x). Let us take two functions for y1 and y2.</p><h4><strong>1.Get the data</strong></h4><pre>x=np.arange(10)<br>y1= x**2<br>y2=2*x+34</pre><pre>print(x)<br>print(y1)<br>print(y2)</pre><pre>Output---------</pre><pre>[0 1 2 3 4 5 6 7 8 9]<br>[ 0  1  4  9 16 25 36 49 64 81]<br>[34 36 38 40 42 44 46 48 50 52]</pre><h4>2. Plot Simple curves</h4><p>To simply plot these functions we use plot function as given below.</p><pre>plt.plot(x,y1)<br>plt.show</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/383/1*uC5EsBIew8YsMlMDs4NXAw.png" /><figcaption>You will get this graph as output. Similarly, you can plot for other functions as well.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*rlxdacCxRXOh4vgH.jpeg" /><figcaption>Credits: imgflip.com</figcaption></figure><h4>3. Use themes to make the plot look more attractive</h4><p>Matplotlib provides some themes to make the plot look more attractive. To get a list of these themes, write the following code.</p><pre>themes=plt.style.available<br>print(themes)</pre><p>Let us try to use one of the theme from this list, say <em>Seaborn-paper</em>. You can try out other themes as well.</p><pre>plt.style.use(“seaborn-paper”)<br>plt.plot(x,y1)<br>plt.show</pre><p>Output:-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/1*uOya9SIV0rAAUGD9AoN4Tg.png" /><figcaption>Using the seaborn paper theme we got the gray colour border and curve, and grd on the graph.</figcaption></figure><h4>4. Change the color of curve</h4><p>Let’s see how to change the color of curve by plotting both y1 and y2 in different colors.</p><pre>plt.style.use(“seaborn-paper”)<br>plt.plot(x,y1, color=”red”)<br>plt.plot(x,y2, color=”green”)<br>plt.show</pre><p>Output:-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*5K3dfdEP-eUYRwPoCsHHoA.png" /><figcaption>We can plot different curves on same plot using different colors</figcaption></figure><h4>5. Give titles</h4><p>Now see how to put titles on your plot.</p><ul><li>plt.xlabel(“ ”): to put a label on x axis.</li><li>plt.ylabel(“ ”): to put a label on y axis.</li><li>plt.title(“ ”): to gve a title to the plot.</li></ul><p>See the code and output given below.</p><pre>plt.xlabel(“Time”)<br>plt.ylabel(“Speed”)<br>plt.title(“Speed v/s Time Curve”)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*uq97EI3bxhkKF0Jxcrk6Pw.png" /><figcaption>We got the label for x and y axis along with the title for curve.</figcaption></figure><h4>6. Give labels to different curves an plot legend</h4><p>We use label parameter to gve labels to the different color curves.</p><pre>plt.style.use(“seaborn-paper”)<br>plt.plot(x,y1, color=”red”, label=”A”)<br>plt.plot(x,y2, color=”green”, label=”B”)<br>plt.legend()</pre><pre>plt.xlabel(“Tme”)<br>plt.ylabel(“Speed”)<br>plt.title(“Speed v/s Time Curve”)</pre><pre>plt.show</pre><p>Output:-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*A9qCrwnxbmQrFK7x1sJ08A.png" /></figure><h4>7. Change style of curve</h4><p>We use linestyle parameter to change the style. Modify this line n the above code as given here.</p><pre>plt.plot(x,y1, color=”red”, label=”A”, linestyle=”dashed”)</pre><p>Output:-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/1*r66_pVudwl_oL5fXqYjyDA.png" /></figure><h4>8. Putting a marker</h4><p>To see the points using which we plotted our curve we use marker.</p><pre>plt.plot(x,y1, color=”red”, label=”A”, linestyle=”dashed”, marker=”o”)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/1*7iEAG9aKRmFNpMm86QS2Xg.png" /></figure><p>Now you know about some basic parameters used to plot attractive and insightful line plots. Experiment more with different parameters.</p><blockquote>“You can achieve simplicity in the design of effective charts, graphs and tables by remembering three fundamental principles: restrain, reduce, emphasize.”</blockquote><blockquote>— <a href="http://www.garrreynolds.com/">Garr Reynolds</a></blockquote><p>Here’s the <a href="https://github.com/shatakshisingh24/medium-blog-codes/blob/master/Data%20Visualizaton.ipynb">github</a> link.</p><p>In case the code doesn’t work for you, let me know. I will update the code. Another blog for more Scatter plots and pie charts coming soon!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a382a537f77d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implement your first Neural Network]]></title>
            <link>https://medium.com/analytics-vidhya/implement-your-first-neural-network-8aa43c7c54a6?source=rss-94ea5048038c------2</link>
            <guid isPermaLink="false">https://medium.com/p/8aa43c7c54a6</guid>
            <category><![CDATA[image-classsification]]></category>
            <category><![CDATA[mnist-dataset]]></category>
            <category><![CDATA[mnist]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[beginner]]></category>
            <dc:creator><![CDATA[Shatakshi Singh]]></dc:creator>
            <pubDate>Tue, 11 Aug 2020 11:45:28 GMT</pubDate>
            <atom:updated>2020-08-16T14:38:36.247Z</atom:updated>
            <content:encoded><![CDATA[<h3>Build a multi-class image classification model with the MNIST dataset.</h3><h4>For beginners|Along with source code</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*PHJidwI0wgNBJ0K8EZJOxA.png" /><figcaption>Source: <a href="https://www.google.com/url?sa=i&amp;url=https%3A%2F%2Fwww.kdnuggets.com%2F2018%2F02%2F8-neural-network-architectures-machine-learning-researchers-need-learn.html&amp;psig=AOvVaw3W-ckJDBF8yR037N1yX0b3&amp;ust=1597231201178000&amp;source=images&amp;cd=vfe&amp;ved=0CAMQjB1qFwoTCLC494qEk-sCFQAAAAAdAAAAABAD">KDnuggts</a></figcaption></figure><p>Below are the steps to build a model that can classify handwritten digits with an accuracy of more than 95%. While reading this article I suggest you simultaneously try the code in colab notebook. Follow the steps and observe the outputs.</p><p>You can find the complete code on <a href="https://github.com/shatakshisingh24/Master-Data-Science-Course/blob/master/Session%20-22%20Deep%20Learning/Neural_Networks.ipynb">GitHub </a>as well.</p><h3>1. Prepare the input data</h3><p><strong><em>Step-1 Import the required libraries</em></strong></p><pre>import numpy as np<br>import keras<br>from keras.datasets import mnist<br>import matplotlib.pyplot as plt</pre><ul><li><a href="https://numpy.org/doc/1.18/reference/index.html">Numpy</a> has many functions that provide support for arrays. An image is nothing but a NumPy array containing pixels of the data points.</li><li><a href="https://keras.io/getting_started/">Keras library</a> has many functions that make it really easy to build models.</li><li><a href="https://matplotlib.org/">Matplotlib</a> helps to plot images and visualize results.</li></ul><p><strong><em>Step-2 Load the MNIST dataset</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/1*6hESC6qB-lxRpXzw_rf_sQ.png" /><figcaption>Sample of MNIST dataset containing hand-written digits.</figcaption></figure><p><strong><em>Dataset- </em></strong>This dataset consists of 60,000 28x28 grayscale images of the 10 digits (0–9), along with a test set of 10,000 images[1]. More info can be found at the <a href="http://yann.lecun.com/exdb/mnist/">MNIST homepage</a>.</p><pre>(x_train, y_train), (x_test, y_test)=mnist.load_data()</pre><pre>Downloading data from <a href="https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz">https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz</a> 11493376/11490434 [==============================] - 0s 0us/step</pre><p><strong><em>Step-3 Reshape the input data</em></strong></p><pre>#View the shape of loaded dataset<br>x_train.shape<br>x_test.shape<br>y_train.shape<br>x_test.shape</pre><p>The size of input images is <em>78x78</em>. But we can pass only flattened arrays in a Deep Neural Network. So let&#39;s reshape the images.</p><pre>#Reshape the dataset</pre><pre>x_test=x_test.reshape(-1,784)<br>x_train=x_train.reshape(-1, 784)</pre><h3>2. Build your model</h3><p>We will build a <strong><em>Sequential mode</em></strong>l using Keras layers. In a sequential model output of the previous layer is treated as an input for the next layer.</p><p><strong><em>Step-4 Import the libraries required to build a model</em></strong></p><pre>from keras.models import Sequential<br>from keras.layers import Dense</pre><ul><li>For building a sequential model we import Sequential class from <em>keras.models</em>.</li><li>We import Dense class to add dense layers in our network. In a dense layer, each neuron of a layer is connected to all the neurons of the next layer. In simple words, a dense layer is a fully connected layer.</li></ul><p><strong><em>Step-5 Add layers to your network</em></strong></p><pre>model=Sequential()</pre><p>We created an object of Sequential class called a model. Now we will add the required layers to the model using model.add().</p><pre>model.add(Dense(units=64, activation=’relu’,input_shape= (784, )))<br>model.add(Dense(units= 64, activation=’relu’))<br>model.add(Dense(units= 128, activation=’relu’))<br>model.add(Dense(units= 64, activation=’relu’))<br>model.add(Dense(units= 10, activation=’softmax’))</pre><p>In this model, we will add 5 dense layers. For each layer, we have to give values for a few parameters. The two important parameters are <em>units</em> and <em>activation</em>. However, there are many other parameters like <em>kernal_initializer</em>, <em>bias_initializer</em>, etc, which will be initialized with their default values.</p><ul><li><em>Units:</em> refers to the number of neurons in a layer. We generally increase the number of neurons preferably in the form of 2^n. In multiclass classification number of units in the last layer is equal to a number of different classes.</li><li><em>Activation: </em>The activation function to be used in each layer. In our model, we have used Relu(Rectified Linear Unit function) in hidden layers and Softmax function in the output layer.</li><li><em>input_shape</em>: This argument is passed only for the first layer because the first layer does not know what kind of input will be given by the user. Hence we have to give it explicitly. The other layers will get the input of the same shape as the shape of the output of the previous layer.</li></ul><blockquote>The output is given as <strong>activation(dot(input, kernal)+bias)</strong> where activation is the element-wise activation function passed as the activation argument, kernal is a weights matrix created by the layer, and bias is a bias vector created by the layer.</blockquote><p>View the model summary if you want</p><pre>model.summary()<br></pre><pre>Model: &quot;sequential&quot; _________________________________________________________________ Layer (type)                 Output Shape              Param #    ================================================================= dense (Dense)                (None, 64)                50240      _________________________________________________________________ dense_1 (Dense)              (None, 64)                4160       _________________________________________________________________ dense_2 (Dense)              (None, 128)               8320       _________________________________________________________________ dense_3 (Dense)              (None, 64)                8256       _________________________________________________________________ dense_4 (Dense)              (None, 10)                650        ================================================================= Total params: 71,626 Trainable params: 71,626 Non-trainable params: 0 _________________________________________________________________</pre><p><strong><em>Step-6 Compile the model</em></strong></p><p>Now we will compile the model and give the following parameters :</p><ul><li><em>Optimizer</em>: It optimizes the loss function. Here we will use adam optimizer.</li><li><em>loss</em>: This model is performing multiclass classification so we will use categorical_crossentropy loss.</li><li><em>metrics</em>: It defines on what basis we have to evaluate our model. Here, we will use accuracy as the metrics.</li></ul><pre>model.compile(optimizer=”adam”, loss=’categorical_crossentropy’,metrics=[‘accuracy’] )</pre><p><strong><em>Step-7 Convert the output vector to one hot vector</em></strong></p><p>To know what is one-hot encoding and why it is necessary read <a href="https://machinelearningmastery.com/why-one-hot-encode-data-in-machine-learning/">this</a>.</p><pre>from keras.utils import to_categorical<br>y_train=to_categorical(y_train)</pre><h3>3. Train your model</h3><p><strong><em>Step-8 Fit the model on your training dataset</em></strong></p><p>Now we will fit the model on training data and save the model as ‘hist’. We have to give the values of the following parameters:</p><ul><li><em>x</em>: the input vector of training data.</li><li><em>y</em>: output vector of training data, which was one-hot encoded.</li><li><em>batch_size</em>: it is significant for large datasets. We chose 32 as the batch size which means in every iteration 32 examples will be processed.</li><li><em>epochs</em>: the number of epochs</li><li><em>validation_split</em>: the ratio of total data to be used for validation.</li></ul><p>There are many other parameters but they will be initialized with their default values.</p><pre>hist=model.fit(x=x_train, y=y_train, batch_size=32, epochs=10, validation_split=0.2, shuffle=True)<br></pre><pre>Epoch 1/10 1500/1500 [==============================] - 3s 2ms/step - loss: 0.8803 - accuracy: 0.8268 - val_loss: 0.3275 - val_accuracy: 0.9077<br> Epoch 2/10 1500/1500 [==============================] - 3s 2ms/step - loss: 0.2615 - accuracy: 0.9238 - val_loss: 0.2284 - val_accuracy: 0.9383 <br>Epoch 3/10 1500/1500 [==============================] - 3s 2ms/step - loss: 0.2054 - accuracy: 0.9406 - val_loss: 0.1965 - val_accuracy: 0.9447 <br>Epoch 4/10 1500/1500 [==============================] - 3s 2ms/step - loss: 0.1703 - accuracy: 0.9504 - val_loss: 0.1894 - val_accuracy: 0.9503 <br>Epoch 5/10 1500/1500 [==============================] - 3s 2ms/step - loss: 0.1489 - accuracy: 0.9569 - val_loss: 0.2132 - val_accuracy: 0.9452 <br>Epoch 6/10 1500/1500 [==============================] - 3s 2ms/step - loss: 0.1319 - accuracy: 0.9620 - val_loss: 0.1746 - val_accuracy: 0.9558<br> Epoch 7/10 1500/1500 [==============================] - 3s 2ms/step - loss: 0.1182 - accuracy: 0.9655 - val_loss: 0.1546 - val_accuracy: 0.9583 <br>Epoch 8/10 1500/1500 [==============================] - 3s 2ms/step - loss: 0.1066 - accuracy: 0.9695 - val_loss: 0.1533 - val_accuracy: 0.9605 <br>Epoch 9/10 1500/1500 [==============================] - 4s 2ms/step - loss: 0.0952 - accuracy: 0.9722 - val_loss: 0.1680 - val_accuracy: 0.9617 <br>Epoch 10/10 1500/1500 [==============================] - 4s 3ms/step - loss: 0.0868 - accuracy: 0.9747 - val_loss: 0.1775 - val_accuracy: 0.9574</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/1*fdjm2M0F2mArp7xtRMhEhg.jpeg" /><figcaption>3 layer Neural Network to classify Hand-written digit Source:S<a href="https://www.google.com/url?sa=i&amp;url=https%3A%2F%2Fwww.slideshare.net%2Fmirjalil%2Fintroduction-to-neural-networks-and-deep-learning&amp;psig=AOvVaw16FRYS_iTqUKjwV3EsAw01&amp;ust=1597231375926000&amp;source=images&amp;cd=vfe&amp;ved=0CAMQjB1qFwoTCKjmwdiEk-sCFQAAAAAdAAAAABAD">lideshare</a></figcaption></figure><h3>4.Test your model</h3><p><strong><em>Step-9 Predict outcome for any random datapoint</em></strong></p><p>Now lets test one random example from out dataset, say example no 999. To make predictions we use model.predict() function.</p><pre>model.predict_classes(x_test[999].reshape(-1,784))</pre><p>The output given is 9 in my code. To check if the prediction made by your model is correct plot the data using plt.imshow().</p><pre>plt.imshow(x_test[999].reshape(28,28), cmap=’gray’)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/1*jvNkRljHFFBuchxh9KES6g.png" /></figure><p>Our model predicted the input digit correctly.</p><h3>References</h3><p>[1] Cohen, G., Afshar, S., Tapson, J., &amp; Van Schaik, A. (2017, May). EMNIST: Extending MNIST to handwritten letters. In <em>2017 International Joint Conference on Neural Networks (IJCNN)</em> (pp. 2921–2926). IEEE.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8aa43c7c54a6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/implement-your-first-neural-network-8aa43c7c54a6">Implement your first Neural Network</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>