<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Praneel Nihar on Medium]]></title>
        <description><![CDATA[Stories by Praneel Nihar on Medium]]></description>
        <link>https://medium.com/@u.praneel.nihar?source=rss-ffa230f1042d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*FUPlAo3gesjlx2G2v1_AsQ.jpeg</url>
            <title>Stories by Praneel Nihar on Medium</title>
            <link>https://medium.com/@u.praneel.nihar?source=rss-ffa230f1042d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 21:52:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@u.praneel.nihar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Facebook Prophet For Time Series Forecasting in Python (Part1)]]></title>
            <link>https://medium.com/data-science/facebook-prophet-for-time-series-forecasting-in-python-part1-d9739cc79b1d?source=rss-ffa230f1042d------2</link>
            <guid isPermaLink="false">https://medium.com/p/d9739cc79b1d</guid>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <category><![CDATA[fbprophet]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Praneel Nihar]]></dc:creator>
            <pubDate>Tue, 08 Sep 2020 18:15:57 GMT</pubDate>
            <atom:updated>2020-09-14T12:50:36.417Z</atom:updated>
            <content:encoded><![CDATA[<h3>Facebook Prophet For Time Series Forecasting in Python</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*iyWl0kveGRhTvPyXDDSlgA.png" /><figcaption>Prophet Forecasting</figcaption></figure><p>Prophet is an open source time series forecasting algorithm designed by Facebook for ease of use without any expert knowledge in statistics or time series forecasting. Prophet builds a model by finding a best smooth line which can be represented as a <strong>sum of the following components:</strong></p><p><strong>y(t) = g(t) + s(t) + h(t) + ϵₜ</strong></p><ul><li>Overall growth trend. g(t)</li><li>Yearly seasonality. s(t)</li><li>Weekly seasonality. s(t)</li><li>Holidays effects h(t)</li></ul><p>In this blog post, we will see some of the useful functions present in the library <strong>fbprophet </strong>listed below with an example.</p><ol><li><strong>Prophet.fit</strong></li><li><strong>Prophet.predict</strong></li><li><strong>Prophet.plot</strong></li><li><strong>Prophet.plot_components</strong></li><li><strong>Prophet.add_seasonality</strong></li><li><strong>Prophet.add_regressors</strong></li><li><strong>Prophet.seasonalities</strong></li><li><strong>Prophet.predictive_samples</strong></li></ol><p>Let’s start by describing the sample data set that we will be using for our demonstration.</p><h3>Data Description</h3><p>We will be using a synthetic daily time series data(shown below) with columns (<strong>date</strong>, <strong>target</strong>, <strong>regr1</strong>, <strong>regr2</strong>) for 180 days, where <strong>target</strong> is a value which we want to be predicted for each day and <strong>regr1</strong>, <strong>regr2</strong> are external factor which effect the target value.</p><p>Let&#39;s see the data visually by plotting the target and regressor columns.</p><pre># Importing Libraries<br>import pandas as pd<br><br># loading the time series data into a dataframe<br>df = pd.read_csv(&#39;ts_with_2regressors.csv&#39;)<br>df[&#39;date&#39;] = pd.to_datetime(df[&#39;date&#39;], format=&#39;%Y-%m-%d&#39;)<br></pre><pre># plotting the time series data<br>df.plot(x=&#39;date&#39;, y=&#39;target&#39;, figsize=(20, 5), title=&#39;Time series Data&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*dLsTb7MMTjMiX9iHrIK2DQ.png" /><figcaption>Sample Time Series Data</figcaption></figure><pre># plotting the regressors<br>ax = df.plot(x=&#39;date&#39;, y=&#39;target&#39;, figsize=(20, 5), title=&#39;Regressors Effect&#39;)</pre><pre>ticks, _ = plt.xticks()</pre><pre>y_min = df.target.min()<br>y_max = df.target.max()</pre><pre>plt.vlines(x=list(df[df[&#39;regr1&#39;] == 1][&#39;date&#39;].values), ymin=y_min, ymax=y_max, colors=&#39;purple&#39;, ls=&#39;--&#39;, lw=2, label=&#39;regr1&#39;)</pre><pre>plt.vlines(x=list(df[df[&#39;regr2&#39;] == 1][&#39;date&#39;].values), ymin=y_min, ymax=y_max, colors=&#39;green&#39;, ls=&#39;:&#39;, lw=2, label=&#39;regr2&#39;)</pre><pre>plt.legend(bbox_to_anchor=(1.04, 0.5), loc=&quot;center left&quot;)</pre><pre>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YJvnSNSpxthgx3NLofFNmQ.png" /><figcaption>Regressors Effect</figcaption></figure><p>We can see that the spikes in the time series data are due to the regressors (<strong>regr1</strong>, <strong>regr2</strong>). We will see how to capture and model these regressors in the coming sections.</p><h3><strong>Installation of Prophet:</strong></h3><p>As with every python library you can install <strong>fbprophet</strong> using pip. The major dependency that Prophet has is <strong>pystan</strong>.</p><pre># Install pystan with pip before using pip to install fbprophet</pre><pre>pip install pystan<br>pip install fbprophet</pre><p>Let us now see how to use the above functions:</p><h3>Generating a Forecast:</h3><ul><li>Prophet follows the sklearn model API. We create an instance of the Prophet class and then call its fit(<strong>Prophet.fit</strong>) and predict(<strong>Prophet.predict</strong>) methods.</li><li>The input to Prophet is always a data frame with two columns: <strong><em>ds</em></strong> and <strong><em>y</em></strong>. The <strong>ds</strong> (date stamp) column should be of a format expected by Pandas, ideally <strong>YYYY-MM-DD</strong> for a date or <strong>YYYY-MM-DD HH:MM:SS</strong> for a time stamp. The <strong>y</strong> column must be numeric, and represents the measurement we wish to forecast.</li><li><em>For demonstration we will use the </em><strong><em>target</em></strong><em> values for first 150 days as training data and predict target for all 180 days.</em></li></ul><p><strong>Note</strong> : For this step we will be considering only the <strong>date</strong> and <strong>target</strong> columns</p><pre># Creating train and predict dataframe<br>df = df.rename(columns={&#39;date&#39;:&#39;ds&#39;, &#39;target&#39;:&#39;y&#39;})<br>df_train = df[[&#39;ds&#39;, &#39;y&#39;]].iloc[:150]<br>df_predict = df[[&#39;ds&#39;]]</pre><pre><br># Fitting a Prophet model<br>model = Prophet()<br>model.fit(df_train)<br>forecast = model.predict(df_predict)</pre><pre>forecast.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1021/1*Lwu32V09NIFETpYrIYwrRw.gif" /><figcaption>Forecast Head GIF</figcaption></figure><pre># plotting the actual and forecast values<br><br>ax = (df.plot(x=&#39;ds&#39;,y=&#39;y&#39;,figsize=(20,5),title=&#39;Actual Vs Forecast&#39;))<br>forecast.plot(x=&#39;ds&#39;,y=&#39;yhat&#39;,figsize=(20,5),title=&#39;Actual vs Forecast&#39;, ax=ax)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*P9PSfQ0lYvfA7V1_0cAJug.png" /><figcaption>Actual vs forecast</figcaption></figure><p>From the above output we can see that prophet pretty much gave a good fit on the data, but still it is unable to capture the sudden jumps in the data. These jumps are basically caused by the external regressors which prophet is unable to detect by default. We will see how to model Prophet to capture these external factors in the coming sections.</p><h3>Plotting the Forecast :</h3><ul><li>We can plot the forecast and the components by calling the <strong>Prophet.plot</strong> &amp; <strong>Prophet.plot_components</strong> method and passing in the forecast dataframe as shown below</li><li>The forecast plot is a single graph containing a scatter plot of historical data points indicated by black dots and the forecast/fitted curve indicated by a blue line. The graph also contains a light blue shaded region which corresponds to the uncertainty bands.</li><li>The components plot is a group of plots corresponding to various time series components(<strong>trend</strong>, <strong>seasoanilities</strong>) and external effects.</li></ul><pre># Plotting the generated forecast<br>fig1 = model.plot(forecast, uncertainty=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*S5gmHhIzLteIukoWXbhc3Q.png" /><figcaption>Forecast Output Plot</figcaption></figure><pre># Plotting the forecast components.<br>fig2 = model.plot_components(forecast)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*3Op5Dwc1jEvoailAzRIdDw.png" /><figcaption>Forecast Component Plot</figcaption></figure><p>As mentioned in the starting <strong>Prophet</strong> estimates the <strong>trend</strong> and <strong>weekly_seasonality</strong> based on the training data.</p><p>Let us now understand the above 2 Plots:</p><h3><strong>Forecast Output Plot:</strong></h3><ul><li><strong>X-axis</strong> represents the date values (<strong>ds ) </strong>for both history and future dates.</li><li><strong>Y-axis </strong>represents the target values(<strong>y</strong>, <strong>yhat</strong>)for both history and future dates.</li><li>In the graph the <strong>black dotted points</strong> represent the historical training data points.</li><li>The <strong>blue line </strong>represents the forecasts generated for both history and future.</li><li>Along with there is <strong>light blue region </strong>which represents the uncertainty bands(We will see more about this in the coming sections.)</li></ul><h3>Forecast Component Plot:</h3><ul><li><strong>X-axis</strong> represents the date values (<strong>ds ) </strong>for both history and future dates.</li><li><strong>Y-axis </strong>represents the prophet estimate for respective forecast compoent (<strong>trend</strong>, <strong>seasonality</strong>)</li><li>Graph1: <strong>trend </strong>value for all dates(history and future).</li><li>Graph2: <strong>weekly_seasonality </strong>a weekly profile for each day in a week based on the training data.</li></ul><p>As we can see it is very easy to start and get a reasonable forecast model on your time series data using prophet.</p><h3>Adding Custom Seasonalities</h3><ul><li>In Prophet we can model custom seasonalities using the functions <strong>Prophet.add_seasonality</strong>.</li><li>By default Prophet automatically models a additive <strong>daily</strong>, <strong>weekly</strong>, <strong>yearly</strong> seasonalities based on available training data.</li><li>We can get the details of inferred seasonalities using the function <strong>Prophet.seasonalities</strong></li><li>Let us now use the above method to model a <strong>monthly</strong> seasonality.</li></ul><pre># Modelling a custom monthly seasonality<br>model2 = Prophet()<br>model2.add_seasonality(name=&#39;custom_monthly&#39;, period=30.5, fourier_order=10)</pre><pre>model2.fit(df_train)<br>forecast2 = model2.predict(df_predict)</pre><pre>print(model2.seasonalities)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/817/1*6DMzh2A3fwMctvohc0S0jg.png" /></figure><pre>fig1 = model2.plot(forecast2, uncertainty=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*kJoiG04GwScgfaxGZ0uiOg.png" /></figure><pre>fig2 = model2.plot_components(forecast2, uncertainty=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*kGUdd410GYS0U2CV-P3qgQ.png" /></figure><p>From the above graph We can see that the Prophet has modeled a custom_monthly seasonality and the forecast is also a bit modified compared to the default forecast.</p><h3>Adding External Regressors</h3><p>Till now the prophet model is not able to model some of points in the training data. We know that these values deviate from the regular value because of the external regressors (<strong>regr1</strong>, <strong>regr2</strong>).</p><p>let us now see how to capture these values and model them.</p><ul><li>Similar to seasonalities prophet also has a way to capture/model external factors which have an effect on the target value using the function <strong>Prophet.add_regressors</strong>.</li><li>In the sample data we are using we have mentioned that there are two external regressors which are effecting the target value.</li><li>In order to model and predict the regressor effects, both the training and prediction dataframes should contain the regressor data.</li><li>Let us now see how to model these regressors using the above function.</li></ul><p><strong>Note</strong> : The regressor’s should be numeric values, you will have to perform one hot encoding if the regressor contains string data.</p><pre># adding regressor data in historical and future dates</pre><pre>df_train3 = (df[[&#39;ds&#39;, &#39;y&#39;, &#39;regr1&#39;, &#39;regr2&#39;]]<br>             .iloc[:150]<br>             .copy())<br>df_predict3 = df[[&#39;ds&#39;, &#39;regr1&#39;, &#39;regr2&#39;]].copy()</pre><pre># modelling external regressors prior to model fitting</pre><pre>model3 = Prophet()<br>model3.add_regressor(&#39;regr1&#39;)<br>model3.add_regressor(&#39;regr2&#39;)<br><br><br># fit and predcit<br>model3.fit(df_train3)<br>forecast3 = model3.predict(df_predict3)</pre><pre># Plot the forecast<br>fig1 = model3.plot(forecast3, uncertainty=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*iyWl0kveGRhTvPyXDDSlgA.png" /><figcaption>Prophet model with external regressors</figcaption></figure><pre># plot model components<br>fig2 = model3.plot_components(forecast3, uncertainty=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*9mdVyCuGrcqq6IxddPzScA.png" /><figcaption>Prophet Model Components with external regressors</figcaption></figure><p>From the above output we can observe that the model has pretty much captured the two external effects, from the ouptut graph we can that there is approx 5% and 20% lift on the target value w.r.t the regressors.</p><h3>Prophet Posterior Samples</h3><ul><li>By default prophet generates 1000 posterior samples for each day inorder to estimating the upper and lower bound’s of the uncertainity bands. On a given day the mean of the posterior samples is almost equal to the forecast value <strong>yhat</strong></li><li>Prophet has a way to access the posterior samples for a particular day both in history and future using the function <strong>Prophet.predictive_samples</strong></li><li>We can modify the no of samples at the time of prophet instantiation using the parameter <strong>uncertainty_samples</strong></li><li>let us now generate a data frame with posterior samples for 1 week in the prediction data frame.</li></ul><pre># Select one week from prediction df<br>df_1wk = df_predict3.iloc[:7]<br>df_1wk</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*1e8TFNY_9JFKE6ySfsl-aA.png" /></figure><pre># fetching the posterior samples.<br>samples = model3.predictive_samples(df_1wk)</pre><pre>df_samples = pd.DataFrame(data=samples[&#39;yhat&#39;], index=df_1wk[&#39;ds&#39;]).reset_index()</pre><pre>df_samples</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cCYfDZ1xVGqREEe5rIcewQ.png" /></figure><p>Let us now compute the mean of posterior samples for one day and compare it with that day’s forecast.</p><pre># Forecast <br>forecast3[forecast3[&#39;ds&#39;] == &#39;2018-09-02&#39;][[&#39;ds&#39;, &#39;yhat&#39;]]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/705/1*D3sVRyEtLwDCyYLohqv6gQ.png" /></figure><pre># Mean of the posterior samples.<br>df_samples[df_samples[&#39;ds&#39;] == &#39;2018-09-02&#39;].set_index(&#39;ds&#39;).mean(axis=1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*su_3moW4NlPclzEUffyEXg.png" /></figure><p>We can see that the mean of the posterior samples is almost equal to the forecast value for a particular day.</p><h3>Conclusion:</h3><p>This brings us to the end of our demonstration. Hope you find the post informative. You can find the data and Jupyter notebook used in the below repository:</p><p><a href="https://github.com/upraneelnihar/ML-Projects/tree/master/Time_series">upraneelnihar/ML-Projects</a></p><p>Do try modelling your time series data using prophet and share your thoughts in the comments section.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9739cc79b1d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/facebook-prophet-for-time-series-forecasting-in-python-part1-d9739cc79b1d">Facebook Prophet For Time Series Forecasting in Python (Part1)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building Multi Page Web App Using Streamlit]]></title>
            <link>https://medium.com/@u.praneel.nihar/building-multi-page-web-app-using-streamlit-7a40d55fa5b4?source=rss-ffa230f1042d------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a40d55fa5b4</guid>
            <category><![CDATA[streamlit]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Praneel Nihar]]></dc:creator>
            <pubDate>Sun, 09 Aug 2020 19:58:15 GMT</pubDate>
            <atom:updated>2020-08-13T05:31:53.846Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*sz5xk7rcuaIIhTYF" /></figure><p>In the <a href="https://medium.com/@u.praneel.nihar/streamlit-for-building-web-apps-9b8ab6b829fb"><strong><em>previous blog</em></strong></a> we saw how to install streamlit and build a simple hello world app. In this blog we will see how to build a multi page web app using streamlit and then create a general framework to build multi page apps.</p><h3>Creating a Multi Page App:</h3><p>As mentioned already streamlit does not have a implicit feature to add a tab for a new page in a single page, but using some existing features we can create a work around for building multi page apps.</p><p>We can create multiple apps and navigate across each of them in a main app using a radio button. Let’s see how to do that.</p><ol><li>Create app1.py and app2.py .</li></ol><pre># app1.py<br>import streamlit as st<br>def app():<br>    st.title(&#39;APP1&#39;)<br>    st.write(&#39;Welcome to app1&#39;)<br></pre><pre># app2.py<br>import streamlit as st<br>def app():<br>    st.title(&#39;APP2&#39;)<br>    st.write(&#39;Welcome to app2&#39;)</pre><p>2. Create a main app app.py and add a navigator using radio buttons.</p><pre>#app.py</pre><pre>import app1<br>import app2<br>import streamlit as st</pre><pre>PAGES = {<br>    &quot;App1&quot;: app1,<br>    &quot;App2&quot;: app2<br>}</pre><pre>st.sidebar.title(&#39;Navigation&#39;)<br>selection = st.sidebar.radio(&quot;Go to&quot;, list(PAGES.keys()))<br>page = PAGES[selection]<br>page.app()</pre><p>Now you run the app.py to access and navigate through both the apps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*deUq9egdVs7Eyd-41-eUCA.png" /><figcaption>Multipage Streamlit App</figcaption></figure><p>Though the code looks simple it is hard to maintain when the no of pages increase. Hence using the above idea i have created a framework <strong>multiapp.py</strong>(<a href="https://github.com/upraneelnihar/streamlit-multiapps/blob/master/multiapp.py"><strong>click here</strong></a><strong> </strong>to view the code) through which we can create a multi page app which is simple and maintainable as shown below:</p><p><strong>Steps to create the app:</strong></p><ol><li>Import class MultiApp from <strong>multiapp.py </strong>in your app and create a new instance for it.</li><li>Add a new app using the class function add_app(title, func) which takes title and function as the argument.</li><li>Finally run the multi app using class function run() .</li></ol><p>Below are examples to create a multi page app using the above framework.</p><pre># new_app1.py<br>from multiapp import MultiApp<br>import streamlit as st</pre><pre>def foo():<br>    st.title(&quot;Hello Foo&quot;)<br>def bar():<br>    st.title(&quot;Hello Bar&quot;)</pre><pre>app = MultiApp()</pre><pre>app.add_app(&quot;Foo&quot;, foo)<br>app.add_app(&quot;Bar&quot;, bar)</pre><pre>app.run()</pre><p>It is also possible keep each application in a separate file.</p><pre># new_app2.py<br>import foo<br>import bar<br>from multiapp import MultiApp</pre><pre>app = MultiApp()</pre><pre>app.add_app(&quot;Foo&quot;, foo.app)<br>app.add_app(&quot;Bar&quot;, bar.app)</pre><pre>app.run()</pre><p>Clone the below GitHub repository and run the app to see a sample multi page web app as shown below.</p><p><a href="https://github.com/upraneelnihar/streamlit-multiapps">https://github.com/upraneelnihar/streamlit-multiapps</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*Z0o0yLjAZoCEagxaQvvRyw.gif" /><figcaption>Sample Multipage Web App</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a40d55fa5b4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Streamlit for Building Web Apps]]></title>
            <link>https://medium.com/@u.praneel.nihar/streamlit-for-building-web-apps-9b8ab6b829fb?source=rss-ffa230f1042d------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b8ab6b829fb</guid>
            <category><![CDATA[web-app-development]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[streamlit]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Praneel Nihar]]></dc:creator>
            <pubDate>Sun, 09 Aug 2020 19:57:30 GMT</pubDate>
            <atom:updated>2020-08-13T06:14:52.155Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*GRlB6MfwySroLvUd" /></figure><p><a href="https://www.streamlit.io/"><strong>Streamlit </strong></a>is a open source python library for building fast, performant and beautiful data apps. It is a great tool to develop interactive apps quickly for demonstrating a solution.</p><p>In this blog we will see how to build a simple hello world web app using streamlit. In the end we will discuss about streamlit’s capability to build a multi page web app.</p><h3>Building a Streamlit App:</h3><p>We can build a simple streamlit app by following the below 4 steps:</p><ol><li>Make sure you have python 3.6+ installed in your system.</li><li>Install streamlit using PIP:</li></ol><pre>$ pip install streamlit</pre><p>3. Create a python script by naming it hello.py , enter the following code and save it.</p><pre>import streamlit as st</pre><pre>st.title(&#39;Hello World&#39;)<br>st.write(&#39;Welcome to my first app.&#39;)</pre><p>4. Run the app by running the following command in a terminal:</p><pre>streamlit run hello</pre><p>That’s it in a few seconds you will see the app running in your default browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jkw4Eto2vOId62cyvVyIgQ.png" /><figcaption>Hello World Streamlit App</figcaption></figure><p>The biggest advantage of streamlit is that you can build an app just by knowing how to write a python script as shown above. To look at the various features of library<strong><em> </em></strong><a href="https://docs.streamlit.io/en/stable/index.html"><strong><em>click here</em></strong></a></p><p><strong><em>As the library is still in early stages there is no default feature to add multiple pages in a single app but we can create this by using some other features.</em></strong></p><p>In the next blog we will see how to build multi page web apps using streamlit and create a generic framework which is simple to maintain. Click on the below link to go the next blog.</p><p><a href="https://medium.com/@u.praneel.nihar/building-multi-page-web-app-using-streamlit-7a40d55fa5b4"><strong>Building Multi Page Web App Using Streamlit</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b8ab6b829fb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tips and Tricks to Process Large Data in Pandas]]></title>
            <link>https://medium.com/@u.praneel.nihar/tips-and-tricks-to-process-large-data-in-pandas-c1e412496499?source=rss-ffa230f1042d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1e412496499</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[data-wrangling]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[pandas-dataframe]]></category>
            <dc:creator><![CDATA[Praneel Nihar]]></dc:creator>
            <pubDate>Mon, 27 Jan 2020 12:30:28 GMT</pubDate>
            <atom:updated>2020-01-29T12:20:36.201Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*heQem5vdGxEd3l_y.jpg" /></figure><p>As data scientists the first and foremost skill to have is the ability to be process and analyze data. Python pandas has been the most popular tool for data wrangling and data analysis. Usually pandas is really good and efficient in processing small data (usually from 100MB up to 1GB) where performance is rarely a concern. Once the data size increases we experience memory and performance issues. By understanding how pandas interprets data and by using some tricks we can efficiently process large data files .</p><p>In this blog post we will see some of the common issues faced while processing large data in pandas and tricks to overcome them along with a example.</p><h3>TL, DR:</h3><p>The default settings that Pandas uses upon reading a file into a data frame uses a lot of memory. A short analysis of the data, followed by a few type casting operations can achieve a significant reduction in memory usage, without any loss of data. I have demonstrated this on a publicly available Instacart data set. The default file read results in a 11GB data frame in memory, which can be reduced to 1GB data frame with a few type casting operations — a 10X reduction !</p><p>As mentioned above we will be using the kaggle data set for <a href="https://www.kaggle.com/c/instacart-market-basket-analysis"><strong>Instacart market basket analysis</strong></a><strong>.</strong> The data set contains data for around 3lakh historical orders with multiple products per each order. Overall the data set has size of 700MB with around 30 million rows and 9 columns.</p><p>Let us now load this data into a pandas data frame and analyze the data.</p><pre>import pandas as pd</pre><pre>df_aisles = pd.read_csv(&#39;Datasets/aisles.csv&#39;)<br>df_departments = pd.read_csv(&#39;Datasets/departments.csv&#39;)<br>df_order_prior = pd.read_csv(&#39;Datasets/order_products__prior.csv&#39;)<br>df_order_train = pd.read_csv(&#39;Datasets/order_products__train.csv&#39;)<br>df_orders = pd.read_csv(&#39;Datasets/orders.csv&#39;)<br>df_products = pd.read_csv(&#39;Datasets/products.csv&#39;)</pre><pre>print(&#39;aisles: \t&#39;,df_aisles.shape) <br>print(&#39;departments: \t&#39;,df_departments.shape) <br>print(&#39;order_prior: \t&#39;,df_order_prior.shape)<br>print(&#39;order_train: \t&#39;,df_order_train.shape)<br>print(&#39;orders: \t&#39;,df_orders.shape)<br>print(&#39;products: \t&#39;,df_products.shape)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*082jRwfvomNmntJ2KpiQ-g.png" /></figure><p>The data set has information corresponding to various entities(click <a href="https://www.kaggle.com/c/instacart-market-basket-analysis/data">here</a> to look at file descriptions). After looking at the file descriptions we can observe that each entity has multiple data types(Int, float, string). In order to analyze how pandas interprets these data types let us merge all of these entities into a single data frame using pandas <strong><em>merge </em></strong>as shown below.</p><pre># Creating the historical orders data by merging all the data frame.</pre><pre>df_his_orders = (df_order_prior<br>                 .merge(df_orders, on=[&#39;order_id&#39;], how=&#39;left&#39;)<br>                 .merge(df_products, on=[&#39;product_id&#39;], how=&#39;left&#39;)<br>                 .merge(df_aisles, on=[&#39;aisle_id&#39;], how=&#39;left&#39;)<br>                 .merge(df_departments, on=[&#39;department_id&#39;], how=&#39;left&#39;))</pre><pre>print(df_his_orders.shape)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/1*rNZJy0wKP-2Eiqnx0OKzkw.png" /></figure><pre># Analyzing the memory taken by the pandas dataframe<br>df_his_orders.info(memory_usage=&#39;deep&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*qnjPU0xaI8Tr3Er0tu-HZw.png" /></figure><p>We can see that by default pandas interprets all number columns as 64 bit integer or float values ( <strong><em>int64</em></strong>,<strong><em>float64)</em></strong> and all string columns as <strong><em>object</em></strong> values. Each number value in the above data frame occupies a size of 8bytes, and each string value has the size equal to sum of size taken for each character.</p><p>let us now see how to optimize the default pandas interpretation of various dtypes and reduce the data frames size.</p><h3>Int64 &amp; Float64 Columns</h3><p>Let us fetch and analyze the data for numeric columns.</p><pre># Fetching Integer and Float columns<br>int_columns = [col for col in df_his_orders.columns if df_his_orders[col].dtype == &#39;int64&#39;]</pre><pre>float_column = [col for col in df_his_orders.columns if df_his_orders[col].dtype == &#39;float64&#39;]</pre><pre># Printing Min and Max Values of `int_columns`<br>for column in int_columns:<br>    print(column, <br>          df_his_orders[column].min(),<br>          df_his_orders[column].max())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/1*3vUlVLGGDDpPGA8fB9Q4tA.png" /></figure><p>From the above output we can see that none of the integer column really need a 64 bit integer to store the data . W.r.t the min and max values of each column below are the actual byte sizes required by each column:</p><ul><li><em>order_id, product_id, user_id</em> : <strong><em>uint32</em></strong> range(0 to 4294967295)</li><li><em>add_to_cart_order, order_number,order_number, order_dow, order_hour_of_day, aisle_id, department_id</em> : <strong><em>uint16</em></strong> range(0 to 65,535)</li></ul><p><strong>Note</strong>: We are choosing unsigned integer as there are no negative values in the data</p><p>Pandas has an in built method <strong><em>pd.to_numeric </em></strong>to downcast a number to it’s respective lower byte size. Sometimes we may face data loss issues while down casting a numeric column to their respective <strong><em>signed </em></strong>or <strong><em>unsigned</em></strong> value, hence it is recommended that you analyze the data values as above before down casting the values.</p><p>Let us now downcast the integer columns and analyze the data frame size.</p><pre># Downcasting the numeric columns to respective unsigned integers<br>for column in int_columns:<br>    df_his_orders[column] = pd.to_numeric(df_his_orders[column], downcast=&#39;unsigned&#39;)</pre><pre># Printing max and min values of the float columns<br>for column in float_column:<br>    print(column, df_his_orders[column].min(),df_his_orders[column].max())</pre><pre>#Printing the unique values in the float column<br>print(list(df_his_orders[&#39;days_since_prior_order&#39;].unique()))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*I6sqhX0U8UrW04LnvRJh4A.png" /></figure><p>We can see that the float values are not actually floats but due to the missing value <strong><em>nan</em></strong> pandas is interpreting it as a float column. Let us fill the missing value with 0 for now and downcast the column.</p><pre># Filling the ,missing value with 0 nand downcasting the data.</pre><pre>df_his_orders[&#39;days_since_prior_order&#39;].fillna(value=0, inplace=True)</pre><pre>df_his_orders[&#39;days_since_prior_order&#39;] = (pd<br>                                       .to_numeric(df_his_orders[&#39;days_since_prior_order&#39;], <br>                                                     downcast=&#39;unsigned&#39;))<br></pre><pre># Anlaysing the memory usage of the dataframe after downcasting Int64 and Float64 values</pre><pre>df_his_orders.info(memory_usage=&#39;deep&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*NGuN88WugHkA91Y3xPcLpg.png" /></figure><p>We can see that by analyzing the data values and down casting them to their respective lower sizes we have achieved a 2GB decrease in the data frame memory size.</p><h3>Object Columns</h3><p><strong><em>Object</em></strong> columns are the columns which contribute to most of the size of a pandas data frame. The size of each object column is equal to sum of size taken by all characters present in the column.</p><p>Let us now analyze the data present in the <strong><em>object</em></strong> columns and see how we can optimize the size taken.</p><pre># Fetching object column names<br>object_columns = [col for col in df_his_orders.columns if df_his_orders[col].dtype == &#39;object&#39;]</pre><pre># Printing out no of unique values present in each object columns<br>for column in object_columns:<br>    print(column, df_his_orders[column].nunique())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*sm4ZDrzgVC5TvN0UlGvyKA.png" /></figure><p>From the above output we can see that each <strong><em>object</em></strong> column has only few unique values which are repeated across the data frame and constituting for unnecessary memory storage.</p><p>Let us perform a small trick here and see what happens, we will be converting the <strong><em>object</em></strong> column to <strong><em>categorical </em></strong>column using the following command and then analyze the data frame.</p><p><strong>df[column_name].astype(&#39;category&#39;)</strong>.</p><pre># converting the object columns to categorial columns<br>for column in object_columns:<br>    df_his_orders[column] = df_his_orders[column].astype(&#39;category&#39;)</pre><pre># Anlaysing the memory usage of the dataframe after converting object to category <br>df_his_orders.info(memory_usage=&#39;deep&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*yWPMicw-fZ6VUFknw_ITHg.png" /></figure><p>We can see that by converting <strong><em>object</em></strong> columns to <strong><em>categorical</em></strong> columns we have achieved around 10 times reduction in the size of the data frame.</p><p><strong>What is categorical datatype?</strong></p><ul><li>Often in real-time, data includes the text columns, which are repetitive. Pandas <strong><em>categorical</em></strong> is datatype which can take on only a limited, and usually fixed number of possible values.</li><li>By default pandas stores categorical columns as <strong><em>object</em></strong> columns. The reason why this default interpretation is inefficient is because pandas creates a pointer for each value of this columns in the memory.</li><li>By converting the <strong><em>object</em></strong> columns to <strong><em>categorical </em></strong>we are forcing pandas to use a virtual mapping table where all unique values are mapped via an integer instead of a pointer. for eg: if we have 100 unique values in a column it will use a <strong><em>uint8 </em></strong>integer to create the mapping. This reduces the memory required drastically in case of large data.</li></ul><p><strong>Note:</strong> <strong>df[column_name].astype</strong> can also be used for converting integers and float columns from <strong><em>int64</em></strong> to their respective downcasted versions but the only problem is that you have to manually enter the size(eg: <strong><em>int16</em></strong>, <strong><em>int8</em></strong>)to which it needs to be downcasted in <strong>astype</strong> whereas <strong>pd.to_numeric</strong> does that by itself.</p><h3>Conclusion</h3><p>By using some simple tricks listed below we can process large data in pandas efficiently.</p><ul><li>Downcast numeric column values to their respective signed or unsigned values using <strong>pd.to_numeric</strong> or <strong>df.astype</strong>.</li><li>Convert object column values to categorical values using <strong>df.astype(&#39;category&#39;)</strong></li></ul><p>please go to the below GitHub link to download the full notebook.</p><p><a href="https://github.com/upraneelnihar/Blogs/tree/master/pandas_blogs">https://github.com/upraneelnihar/Blogs/tree/master/pandas_blogs</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1e412496499" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Improving Read, Write, Store Performance by Changing File Formats (Serialization Protocols)]]></title>
            <link>https://medium.com/@u.praneel.nihar/improving-read-write-store-performance-by-changing-file-formats-serialization-protocols-bfdb13114004?source=rss-ffa230f1042d------2</link>
            <guid isPermaLink="false">https://medium.com/p/bfdb13114004</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Praneel Nihar]]></dc:creator>
            <pubDate>Wed, 22 Jan 2020 09:50:23 GMT</pubDate>
            <atom:updated>2020-01-22T09:54:54.615Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hrB8_SjgMxH5XOSl.jpg" /></figure><p>As data scientists, we use CSV files and Pandas a lot. When data files grow in size, we experience slow performance, memory issues, etc. Can we do some simple things to improve performance in a big way here?</p><p>To answer this question, I compared <strong>Read-Write Speeds</strong> and <strong>Storage Space</strong> required using different serialization protocols (i.e. file formats) - <strong><em>CSV</em></strong>, <strong><em>HDF</em></strong>, <strong><em>JSON</em></strong>, <strong><em>MSGPACK</em></strong>, <strong><em>PARQUET</em></strong>, <strong><em>PICKLE</em></strong>, using data sets of various sizes. I used a 4GB RAM Dual core, Linux machine.</p><p>Here are results from my experiments….</p><h3>TLDR:</h3><ul><li>On read speeds, PICKLE was 10x faster than CSV, MSGPACK was 4X faster, PARQUET was 2–3X faster, JSON/HDF about the same as CSV</li><li>On write speeds, PICKLE was 30x faster than CSV, MSGPACK and PARQUET were 10X faster, JSON/HDF about the same as CSV</li><li>On storage space, GZIPPED PARQUET gave 40X reduction, GZIPPED CSV gave 10X reduction (didn’t compare the rest)</li></ul><h3>Data from Experiments</h3><p><strong>Comparison of Read speeds</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*C1Z8A1KCs3tY0n8sinBxIQ.jpeg" /><figcaption>Read speeds in seconds</figcaption></figure><p><strong>Comparison of Write speeds</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*M82Su36kIHLx37oHN0qorw.jpeg" /><figcaption>Write speeds in seconds</figcaption></figure><p><strong>Comparison of Storage space required</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*rqBgIduTrSxsbbYh3rcxFg.jpeg" /><figcaption>Storage space comparison</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bfdb13114004" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part2:  Melt, Pivot and Pivot_table]]></title>
            <link>https://medium.com/@u.praneel.nihar/part2-melt-pivot-and-pivot-table-18764f92a57?source=rss-ffa230f1042d------2</link>
            <guid isPermaLink="false">https://medium.com/p/18764f92a57</guid>
            <category><![CDATA[pivot]]></category>
            <category><![CDATA[stack]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[data-cleaning]]></category>
            <dc:creator><![CDATA[Praneel Nihar]]></dc:creator>
            <pubDate>Tue, 05 Nov 2019 03:40:44 GMT</pubDate>
            <atom:updated>2020-08-13T17:59:39.177Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part2: Melt, Pivot and Pivot_table</h3><p>This page is the secondpart of the blog <strong>Reshaping data in pandas. </strong>click <a href="https://medium.com/p/e2400be1ab2">here</a> to go back to main page</p><h3>Melt</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/913/0*O3SGf68-vlON_H8L.png" /><figcaption>Pandas Melt</figcaption></figure><p>Pandas has different ways to accomplish the same task, the difference being readability and performance. Pandas data frame method name <strong><em>melt</em></strong> works similarly to <strong><em>stack</em></strong> but gives more flexibility.The method takes in the below 5 parameters out of which two parameters namely <strong><em>id_vars</em></strong> and <strong><em>value_vars</em></strong> are crucial to understand how to reshape your data.</p><ul><li><strong><em>id_vars</em></strong>: list of column names that you want to preserve as columns and not reshape (list,tuple or ndarray)(optional).</li><li><strong><em>value_vars</em></strong>: list of column names that you want to reshape as columns (list,tuple or ndarray) (optional).</li><li><strong><em>var_name</em></strong>: Name to use for the variable column, defaults to <strong><em>variable.</em></strong>(scalar)(optional)</li><li><strong><em>value_name</em></strong>:Name to use for the value column, defaults to `value`(scalar)(optional).</li><li><strong><em>col_level</em></strong>:If column are multi index then use this level to melt (int or string)(optional).</li></ul><p>All of the above parameters mentioned are optional. Let us try to understand the usage of each of the parameters by applying it to our sample dataset.</p><p><strong>Working of Melt:</strong></p><ul><li>The <strong><em>id_vars</em></strong> or the identification variables remain in the same column but repeat for each of the <strong><em>value_vars</em></strong></li><li>One crucial aspect of <strong><em>melt</em></strong> is that it ignores the values in index, in fact it drops the existing index and replaces it with the <strong><em>RangeIndex</em></strong>. so if you have values in index that you want to keep, you need to do a <strong><em>reset_index</em></strong> before applying <strong><em>melt</em></strong>.</li></ul><p>Let’s apply the melt function on our sample sensors data frame.</p><pre>#<em> Sample data</em><br>df_sensors</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*R5wUGl8bXectKa2Y6wsVfQ.png" /></figure><pre># Applying Melt on sample data<br>df_sensors.melt(id_vars=[&#39;Sensor&#39;, &#39;Metric&#39;],<br>                value_vars=[&#39;2017&#39;, &#39;2018&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*4VJFuJXSgXe_WYIzc3vAog.png" /></figure><p><strong><em>melt </em></strong>works similar to <strong><em>stack</em></strong> but gives more flexibility<strong><em>. </em></strong>By default <strong><em>melt </em></strong>assigns <em>variable </em>and <em>value </em>as the names for <em>value_vars</em>, this can be overrided by passing the desired column names as an input to the parameters <strong><em>var_name</em></strong> and <strong><em>value_name </em></strong>as shown below<strong><em>.</em></strong></p><pre># <em>Setting the names of variable and value columns.</em><br>df_sensors.melt(id_vars=[&#39;Sensor&#39;, &#39;Metric&#39;],<br>                value_vars=[&#39;2017&#39;, &#39;2018&#39;],<br>                var_name=&#39;Year&#39;,<br>                value_name=&#39;value&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*bcMdwrKmC5E32Y4bmOw-Sw.png" /></figure><blockquote><strong>Point to Remember: </strong>The advantage of <strong><em>melt</em></strong> over <strong><em>stack</em></strong> is that you can mention the column name which you want to preserve in the index without explicitly setting them as index .</blockquote><h3>Pivot:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/915/0*DBIsbGKzrfZSXvPq.png" /></figure><p><strong><em>pivot</em></strong> method is similar to <strong><em>unstack</em></strong> but is more easy to use. It takes in 3 parameters (mentioned below) as input which are <strong><em>index</em></strong>, <strong><em>columns</em></strong> and <strong><em>values</em></strong></p><ul><li><strong><em>index</em></strong>: Column to use for the new frames index, if none use current index(string or object).</li><li><strong><em>columns</em></strong>: Column to use for the new frames columns (string or object).</li><li><strong><em>values</em></strong> : Column(s) to use for the new frames values (string or object).</li></ul><p><strong>Working of pivot:</strong></p><ul><li>Each parameter takes in a single column as a string.</li><li>The <strong><em>index</em></strong> remains in the vertical and becomes the new index.</li><li>The values of the columns referenced by <strong><em>columns</em></strong> becomes new column names.</li><li>The values referenced by the <strong><em>values</em></strong> are tiled to correspond the intersection of their former index and columns label.</li></ul><p>Let us consider the previous melted output and see how to apply <strong><em>pivot </em></strong>in order to reshape it to original form.</p><pre># <em>loading melted data as a dataframe<br></em>df_melted = df_sensors.melt(id_vars=[&#39;Sensor&#39;, &#39;Metric&#39;],<br>                value_vars=[&#39;2017&#39;, &#39;2018&#39;],<br>                var_name=&#39;Year&#39;,<br>                value_name=&#39;value&#39;)<br>df_melted</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*MuaiLcE9Dys3ji-uux6Qjw.png" /></figure><p>We cannot use <strong><em>pivot </em></strong>directly to pivot the <strong><em>Year</em></strong> column , as mentioned above pivot raises an <strong><em>ValueError</em></strong> when there are duplicate values in the index columns.</p><pre># <em>Pivot raises a ValueError</em><br>df_melted.pivot(index=[&#39;Sensor&#39;, &#39;Metric&#39;], columns=&#39;Year&#39;, values=&#39;value&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*NRydSGCvVylk86mkxf9TRg.png" /></figure><p>In order to do that we’ll have to set the index columns explicitly before applying <strong><em>pivot </em></strong>as shown below.</p><pre># <em>Setting the index and applying pivot</em><br>(df_melted<br> .set_index([&#39;Sensor&#39;, &#39;Metric&#39;])<br> .pivot(columns=&#39;Year&#39;)[&#39;value&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*pX38K3PNxKmpICyANTRH1Q.png" /></figure><pre># <em>Processing the pivot output</em><br>(df_melted<br> .set_index([&#39;Sensor&#39;, &#39;Metric&#39;])<br> .pivot(columns=&#39;Year&#39;)[&#39;value&#39;]<br> .reset_index()<br> .rename_axis([None], axis=1))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*sqdQdoSJVYh6zS_kLXCalQ.png" /></figure><blockquote><strong>Points to Remember</strong>:</blockquote><blockquote><strong><em>pivot</em></strong> raises a <strong><em>ValueError</em></strong> when any index or column combinations has duplicate values.</blockquote><blockquote><strong><em>pivot</em></strong> cannot take more than one index at a time.In order to achieve that you will have to explicitly set the index and apply pivot on columns.</blockquote><h3>Pivot_table:</h3><p><strong><em>pivot_table</em></strong> is a versatile and flexible function. Below mentioned are the list of the input parameters to the fucntion.. Below mentioned are the input parameters to the function.</p><ul><li><strong><em>index</em></strong>: column(s) which is intended to stay as index. (column, list, array, Grouper)</li><li><strong><em>columns</em></strong>: column(s) which are pivoted (column, list, array, Grouper).</li><li><strong><em>values</em></strong> : column to aggregate.</li><li><strong><em>aggfunc</em></strong>: function, list of aggregation functions.(defaults to <strong><em>mean</em></strong>)</li><li><strong><em>fill_value</em></strong>: Scalar to fill for missing values in the result (scalar, default None).</li><li><strong><em>margins</em></strong>: whether to add all rows/columns (Bool, default False )(eg: subtotal or grand total)</li><li><strong><em>dropna</em></strong>: Do not include columns whose values for all rows are NaN (Bool, default True).</li><li><strong><em>margins_name</em></strong>: Name of the row / column that will contain the totals when margins is True (string, default All).</li></ul><p><strong>Working of pivot_table:</strong></p><ul><li><strong><em>pivot_table</em></strong> is a generalization of <strong><em>pivot</em></strong> that can handle duplicate values for one <strong>pivoted</strong> index/column pair. Specifically, you can give <strong><em>pivot_table</em></strong> a list of aggregation functions using keyword argument <strong><em>aggfunc</em></strong>. The default <strong><em>aggfunc</em></strong> of <strong><em>pivot_table</em></strong> is <strong><em>numpy.mean</em></strong>.</li><li><strong><em>pivot_table</em></strong> also supports using multiple columns for the index and column of the <strong>pivoted</strong> table. A hierarchical index will be automatically generated for you.</li><li><strong><em>pivot_table </em></strong>also has some additional default parameters named <strong><em>margins</em></strong> , <strong><em>margins_name </em></strong>which indicates whether to add rows and columns total in the output or not.</li></ul><p>Let us see how can we apply <strong><em>pivot_table </em></strong>to the previous melted output in order to convert it to original form.</p><pre># <em>loading melted data as a dataframe</em><br>df_melted = df_sensors.melt(id_vars=[&#39;Sensor&#39;, &#39;Metric&#39;],<br>                value_vars=[&#39;2017&#39;, &#39;2018&#39;],<br>                var_name=&#39;Year&#39;,<br>                value_name=&#39;value&#39;)<br>df_melted</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*n9v5AK8OGGBNeJfRb-m82w.png" /></figure><pre># <em>Applying pivot_table on melted output</em><br>(df_melted<br> .pivot_table(index=[&#39;Sensor&#39;, &#39;Metric&#39;],<br>              columns=&#39;Year&#39;,<br>              values=&#39;value&#39;))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*kk4xy9ALC97dzwfUYtjdCQ.png" /></figure><pre># <em>Processing the Output</em><br>(df_melted<br> .pivot_table(index=[&#39;Sensor&#39;, &#39;Metric&#39;],<br>              columns=&#39;Year&#39;,<br>              values=&#39;value&#39;)<br> .rename_axis([None],axis=1)<br> .reset_index())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*b-l1voOmiyugN7M9JNy7VA.png" /></figure><p><strong><em>pivot_table</em></strong> is widely used to summarize the numerical data. You can perform multiple aggregations like count, sort, average, total etc using <strong><em>pivot_table . </em></strong>We can use the parameters <strong><em>aggfunc, </em></strong><strong><em>margins </em></strong>to perform any no of aggregations at a time on the chosen column values.</p><pre># <em>Performing aggregations using aggfunc.</em><br>df_melted.pivot_table(index=[&#39;Sensor&#39;, &#39;Year&#39;],<br>                     columns=&#39;Metric&#39;,<br>                     values=&#39;value&#39;,<br>                     aggfunc=[np.sum, np.mean],<br>                     fill_value=0,<br>                     margins=True,<br>                     margins_name=&#39;Total&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*G6nQ0GxwcORA1kwp8qm-IQ.png" /></figure><h4>Key Differences between <strong><em>pivot</em></strong> and <strong><em>pivot_table:</em></strong></h4><ul><li><strong><em>pivot</em></strong> is used for pivoting without aggregation. Therefore, it can’t deal with duplicate values for one index/column pair whereas <strong><em>pivot_table </em></strong>can be used for both the purposes.</li><li><strong><em>pivot</em></strong> doesn’t accept a list for index <strong><em>pivot_table</em></strong> accepts.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18764f92a57" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part1: Stack, Unstack]]></title>
            <link>https://medium.com/@u.praneel.nihar/part1-stack-unstack-2060af57b4a0?source=rss-ffa230f1042d------2</link>
            <guid isPermaLink="false">https://medium.com/p/2060af57b4a0</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[pivot]]></category>
            <category><![CDATA[stack]]></category>
            <category><![CDATA[data-cleaning]]></category>
            <dc:creator><![CDATA[Praneel Nihar]]></dc:creator>
            <pubDate>Tue, 05 Nov 2019 03:39:26 GMT</pubDate>
            <atom:updated>2020-08-13T17:57:52.663Z</atom:updated>
            <content:encoded><![CDATA[<p>This page is the first part of the blog <strong>Reshaping data in pandas. </strong>click <a href="https://medium.com/p/e2400be1ab2">here</a> to go back to main page.</p><h3>Stack</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/0*2VhoZlZC9njz1txT.png" /><figcaption>Pandas Stack Image.</figcaption></figure><p>In pandas terminology <strong>stacking</strong> is referred as transforming horizontal column names to vertical column values . <strong><em>stack </em></strong>function converts the data from a wide to long format.Below mentioned are the input parameters to the function.</p><ul><li><strong><em>level</em></strong>: Prescribed level(s) to stack from column axis onto index axis (int,str, list, default -1).</li><li><strong><em>dropna</em></strong>: Whether to drop rows with missing values in the resulting frame, defaults (bool, default True).</li></ul><p>Let us start by anaylsing the sample dataset shown below.</p><pre>#<em> Sample data</em><br>df_sensors</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*R5wUGl8bXectKa2Y6wsVfQ.png" /></figure><p>We can see that the two year values <strong><em>2017 </em></strong>and<strong><em> </em></strong><strong><em>2018 </em></strong>are present as column names, but ideally both of them depict a single variable which can named as <strong><em>Year</em></strong>. We can resturcture the two Year values as a single column using <strong><em>stack </em></strong>as shown below.</p><pre># Stacking the Year columns into a single column.<br>(df_sensors.set_index([&#39;Sensor&#39;, &#39;Metric&#39;])<br> .stack()<br> .reset_index()<br> .rename(columns={&#39;level_2&#39;: &#39;Year&#39;, 0: &#39;Value&#39;}))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/1*86XVsODyUnC-XrfssmPp4Q.png" /><figcaption>Desired Output.</figcaption></figure><p>Let us now understand the above code snippet in a step wise manner</p><p><strong>Step1:</strong></p><pre># <em>Applying stack on sample data</em><br>df_sensors.stack()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*VKYWCeNBwO-J90A1WsByAg.png" /></figure><p>We can see that by default <strong><em>stack</em></strong>takes in all the columns and converts them into a single vertical column at the inner most level. The above output is a series with hierarchical index, all the column names are added as an index at the innermost level and each row is repeated as values.</p><p>If there are any columns which we want to stay in the index we have to explicitly set them using <strong><em>set_index</em></strong> method before performing the <strong><em>stack</em></strong>.</p><p><strong>Step 2:</strong></p><p>Let us now apply <strong><em>stack </em></strong>by setting the <strong><em>Metric </em></strong>column as index and analyze the output.</p><pre>df_sensors.set_index([&#39;Metric&#39;]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/1*I4N0tXemke6hZNwlpP5B9g.png" /></figure><pre>df_sensors.set_index([&#39;Metric&#39;]).stack()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/1*8nON2UqIlzjOLrDxjgWgHA.png" /></figure><p>The output of above step is a <strong><em>series</em></strong> with hierarchical index. In order to convert it to a data frame we have to use pandas helper method <strong><em>reset_index</em></strong></p><pre><em># Processing the stacked output</em><br>(df_sensors<br> .set_index([&#39;Metric&#39;])<br> .stack()<br> .reset_index())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*IGO8_iGOkqJDbzaXQzATzw.png" /></figure><p>We can see that the output is still messy i.e in <strong><em>level_1 </em></strong>column there are multiple data types hence the data is still not in a desired form.</p><p><strong>Step 3:</strong></p><p>In order to convert the multiple year columns to a single year column we have to set the columns <strong><em>Sensor</em></strong>, <strong><em>Metric</em></strong> as index and apply <strong><em>stack </em></strong>.</p><pre># <em>Stacking the Year Columns</em><br>df_sensors.set_index([&#39;Sensor&#39;, &#39;Metric&#39;]).stack()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*q1dfQaoCWeczMLW1Yw_qQg.png" /></figure><p>The above output is a hierarchical indexed series which is in the desired form. In order to convert it to a dataframe we have to use the helper method’s <strong><em>reset_index </em></strong>and <strong><em>rename </em></strong>as shown below<strong><em>:</em></strong></p><pre># <em>Processing the stacked output</em><br>(df_sensors.set_index([&#39;Sensor&#39;, &#39;Metric&#39;])<br> .stack()<br> .reset_index()<br> .rename(columns={&#39;level_2&#39;: &#39;Year&#39;, 0: &#39;Value&#39;}))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/1*86XVsODyUnC-XrfssmPp4Q.png" /></figure><p>When more than one column header is present in the dataset we can stack the desired column header using the <strong><em>level</em></strong> parameter of the stack function as hown below:</p><p>Let us add a column header named <strong><em>Value </em></strong>on top of the Year columns .</p><pre># <em>Add a column level `Value` on top of Year columns</em><br>df_sensors2 = df_sensors.set_index([&#39;Sensor&#39;, &#39;Metric&#39;])<br>df_sensors2.columns = pd.MultiIndex.from_product([[&#39;Value&#39;], df_sensors2.columns])<br>df_sensors2</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/1*Z9dsixUaHO3F2yYDf2mtBA.png" /></figure><p><strong><em>Level 0</em></strong>: [Value]</p><p><strong><em>Level 1 : </em></strong>[ 2017, 2018]</p><p>Now we can stack the two year columns by specifying the level parameter while applying stack</p><pre># <em>Using Level parameter to stack the year columns</em><br>df_sensors2.stack(level=1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*e_e-7TFN8qCnujJPI_oBQw.png" /></figure><pre><em># Processing the output</em><br>(df_sensors2<br> .stack(level=1)<br> .reset_index()<br> .rename(columns={&#39;level_2&#39;:&#39;Year&#39;}))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*TTg3eYMpYbW8PcwGXBYRsA.png" /></figure><blockquote><strong>Points to Remember</strong>:</blockquote><blockquote>By default stack takes in all columns present at the inner most level in the column axis and stacks them row wise . If you want any column to stay in index you need to set your index column explicitly using <strong><em>pd.DataFrame.set_index</em></strong> method before performing <strong><em>stack</em></strong> and then use <strong><em>pd.DataFrame.reset_index()</em></strong> to convert the output to a data frame.</blockquote><h3>Unstack:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/0*AsK-EkbNYMGb6WtN.png" /><figcaption>Pandas Unstack Image</figcaption></figure><p><strong><em>unstack </em></strong>is an inverse operation of <strong><em>stack</em></strong> i.e it converts a data frame from long to wide format. By default it takes the inner most index values and returns a data frame by reshaping them as the columns. Below are the parameters for the <strong><em>unstack</em></strong> method.</p><ul><li><strong><em>level</em></strong>: level(s) of index to <strong><em>unstack</em></strong>, defaults to -1 (int or string or list of these)</li><li><strong><em>fill_value</em></strong>: replace NAN with value specified if <strong><em>unstack</em></strong> produces missing values.</li></ul><p>In order to understand the <strong><em>unstack </em></strong>method let us take the stacked output and see how we can invert the <strong><em>stack</em></strong> operation.</p><pre># <em>Loadind stacked data into a dataframe</em><br>df_stacked = (df_sensors.set_index([&#39;Sensor&#39;, &#39;Metric&#39;])<br>              .stack()<br>              .reset_index()<br>              .rename(columns={&#39;level_2&#39;: &#39;Year&#39;, 0: &#39;Value&#39;}))</pre><pre>df_stacked</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/824/1*LjRUI0oGoThdjKgPSWsTYg.png" /></figure><p>We can now invert the stacked output by following the below three steps:</p><ul><li>Set the columns [<strong><em>Sensor</em></strong>, <strong><em>Metric</em></strong> and <strong><em>Year</em></strong>] as index.</li><li>Apply <strong><em>unstack</em></strong>.</li><li>Process the output to a consumable form.</li></ul><pre># <em>Setting the index columns</em><br>df_stacked.set_index([&#39;Sensor&#39;, &#39;Metric&#39;,&#39;Year&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*l8HPyNHef5FSZcgiWe7FuQ.png" /></figure><pre># <em>Applying unstack</em><br>(df_stacked<br> .set_index([&#39;Sensor&#39;, &#39;Metric&#39;,&#39;Year&#39;])<br> .unstack())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*oIeani150S2FbCJEaVrJeA.png" /></figure><pre># <em>Processing the Output</em><br>dfx = (df_stacked<br>       .set_index([&#39;Sensor&#39;, &#39;Metric&#39;, &#39;Year&#39;])<br>       .unstack())<br>dfx.columns = dfx.columns.droplevel([0])<br>dfx.rename_axis([None], axis=1).reset_index()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/1*Yb-NxfUNEbBfmebRxDO5vg.png" /></figure><p>Let us see some more examples of <strong><em>unstack </em></strong>for better understanding:</p><ol><li><strong>Unstacking</strong> <strong><em>sensors</em> metric values as columns </strong>: We can use <strong><em>unstack</em></strong> operation on the stacked data frame to obtain the desired result as shown below.</li></ol><pre># <em>Setting Metric as innermost index level</em><br>df_stacked.set_index([&#39;Sensor&#39;, &#39;Year&#39;, &#39;Metric&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*EloW9uFScnlDl2r4yZnfng.png" /></figure><pre># <em>Unstacking the Metric column values.</em><br>df_stacked.set_index([&#39;Sensor&#39;, &#39;Year&#39;, &#39;Metric&#39;]).unstack()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*-SaCMiYo3wi1aLq6CxxivQ.png" /></figure><pre>#<em> Processing the output</em><br>dfy = (df_stacked<br>       .set_index([&#39;Sensor&#39;, &#39;Year&#39;, &#39;Metric&#39;])<br>       .unstack())<br><br>dfy.columns = dfy.columns.droplevel([0])<br>dfy.rename_axis([None], axis=1).reset_index()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*rbeueAkTiYGLAtMfrNk-Fg.png" /></figure><p>2. <strong>Unstacking both </strong><strong><em>Year </em>and </strong><strong><em>Metrics</em> as columns : </strong>Using the <strong><em>level </em></strong>parameter we can <strong><em>unstack </em></strong>methods to reshape multi<strong><em>stack</em></strong> function converts a data frame from <strong>wide</strong> to <strong>long</strong> format,ple column values as individual hierarchical columns as shown below.</p><pre># <em>Unstacking at multiple index levels at a time.<br></em>(df_stacked<br> .set_index([&#39;Sensor&#39;, &#39;Year&#39;, &#39;Metric&#39;])<br> .unstack(level=[-3,-2]))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*CdeWlmmme1BnMlO3STmXhw.png" /></figure><pre># <em>Changing the order of the unstacked levels.</em><br>(df_stacked<br> .set_index([&#39;Sensor&#39;, &#39;Year&#39;, &#39;Metric&#39;])<br> .unstack(level=[-2,-1]))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*EsqTyh33ELcyqkySEh4Ejw.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2060af57b4a0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reshaping  data using pandas.]]></title>
            <link>https://medium.com/@u.praneel.nihar/reshaping-data-using-pandas-e2400be1ab2?source=rss-ffa230f1042d------2</link>
            <guid isPermaLink="false">https://medium.com/p/e2400be1ab2</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-cleaning]]></category>
            <category><![CDATA[pivot]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[stack]]></category>
            <dc:creator><![CDATA[Praneel Nihar]]></dc:creator>
            <pubDate>Tue, 05 Nov 2019 03:36:09 GMT</pubDate>
            <atom:updated>2020-08-12T13:42:35.032Z</atom:updated>
            <content:encoded><![CDATA[<h3>Reshaping data using pandas.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-5vO5AVNHDDCRVn4.png" /><figcaption>Reshaping data in pandas</figcaption></figure><p>Data analysis is the first and foremost step in machine learning life cycle. It includes inspecting, cleaning, transforming and modelling data with a goal to discover useful information, informing conclusions and support decision making. Python’s pandas library is one of the powerful and widely used tool for data analysis. Transforming or reshaping data into understandable form is one of the key aspect in data analysis. In this blog we will try to understand some of the widely used pandas methods listed below for reshaping data:</p><ul><li><strong><em>stack</em></strong></li><li><strong><em>unstack</em></strong></li><li><strong><em>melt</em></strong></li><li><strong><em>pivot</em></strong></li><li><strong><em>pivot_table</em></strong></li></ul><h3><strong>Why should we reshape or restructure data?</strong></h3><p>Real world data is not always in a consumable form. It contains lots of missing entries and errors. It is easy to extract data from the rows and columns of a data but there are situations when we need the data in a format that is different from format in which we received it. Therefore it is important to clean and restructure the data to a consumable form.<em> </em>Reshaping data includes converting columns to rows, rows to columns and performing aggregation to bring the data into a form which is easy to analyze.</p><h3>Sample data :</h3><p>Let us create a small sample data set which we will be using throughout the blog . We will be creating a flow sensor data for 3 metric values over the last 2 years. The below image consists the data for 2 sensors(<strong><em>sensor1</em></strong><em>,</em><em> </em><strong><em>sensor2</em></strong><em>) </em>and 3 metric values(<strong><em>Pressure</em></strong><em>, </em><strong><em>Temperature</em></strong><em> , </em><strong><em>Flow rate</em></strong>) measured over the last 2 years (<strong><em>2017</em></strong><em>,</em><strong><em> 2018</em></strong>).</p><pre>import pandas as pd<br>import numpy as np</pre><pre># Sensors data of 2 sensors for last 2 years.<br>iterables = [[&#39;sensor1&#39;, &#39;sensor2&#39;],<br>             [&#39;Pressure&#39;, &#39;Temperature&#39;, &#39;Flow&#39;]]</pre><pre># Setting the names of variable and value columns.index = pd.MultiIndex.from_product(iterables,names=[&#39;Sensor&#39;, &#39;Metric&#39;])</pre><pre>df_sensors = pd.DataFrame(np.random.randint(low=40, high=100,size=(6,2)),<br>                          index=index,<br>                          columns=[&#39;2017&#39;, &#39;2018&#39;]).reset_index()</pre><pre>df_sensor</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/261/1*EQ_16-F0gGYeg-vpgKo7GQ.png" /><figcaption>Sample Data</figcaption></figure><p>Let us now try to understand the usage of each of the above mentioned methods by applying them on the created sample data. We will be using some of the pandas helper methods <strong>set_index</strong>, <strong>reset_index</strong>, <strong>rename</strong> and <strong>rename_axis</strong> to add final touches to the data frame in order to bring the output in a consumable form.</p><p>For better understanding of each method the blog is divided into two logical parts where in each part explains a set of opposite operations with functionality description of each method along with some key points to note :</p><ul><li><a href="https://medium.com/@u.praneel.nihar/part1-stack-unstack-2060af57b4a0">Part1: Stack, Unstack</a></li><li><a href="https://medium.com/@u.praneel.nihar/part2-melt-pivot-and-pivot-table-18764f92a57">Part2: Melt, Pivot and Pivot_table</a></li></ul><p>I hope you get a basic understanding of each of the above method after reading the two parts. Try applying them on different data sets and share your experiences in comments section.</p><p>please go to the below GitHub link to download the full notebook.</p><p><a href="https://github.com/upraneelnihar/Blogs/tree/master/pandas_blogs">https://github.com/upraneelnihar/Blogs/tree/master/pandas_blogs</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e2400be1ab2" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>