<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Bisman Preet Singh on Medium]]></title>
        <description><![CDATA[Stories by Bisman Preet Singh on Medium]]></description>
        <link>https://medium.com/@bismanpreetsingh?source=rss-bb6928ce1fde------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*Msv8z8ALhD57-Pqrzds46Q.jpeg</url>
            <title>Stories by Bisman Preet Singh on Medium</title>
            <link>https://medium.com/@bismanpreetsingh?source=rss-bb6928ce1fde------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 00:17:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@bismanpreetsingh/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[5 Ways & Resources to get into Product Management]]></title>
            <link>https://medium.productcoalition.com/5-ways-resources-to-get-into-product-management-beff26ca80c7?source=rss-bb6928ce1fde------2</link>
            <guid isPermaLink="false">https://medium.com/p/beff26ca80c7</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[marketing]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[product]]></category>
            <category><![CDATA[management]]></category>
            <dc:creator><![CDATA[Bisman Preet Singh]]></dc:creator>
            <pubDate>Tue, 14 Jul 2020 08:54:32 GMT</pubDate>
            <atom:updated>2020-12-24T17:02:43.263Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JzaMXO_14XfYe6ymyw8qtw.jpeg" /><figcaption>Image Source: Unsplash</figcaption></figure><p>Product Management is one of the most fascinating careers in tech right now and is also one of the hottest careers all over the world owing to the growth opportunities in this field. Product Managers grow to become CEOs, CXOs, and founders. Wonder why that is the case?</p><h3><strong>The reason</strong></h3><p>As a Product Manager, you get to work with designers, engineers, analysts, data scientists, finance, marketing, third-party vendors, customer service, and whatnot. The list is endless! Every day is different and you need to have at least a basic level of understanding in all the domains to understand the problems.</p><p>You learn technical, business, design, and communication skills in a very short period by interacting with so many different stakeholders daily and getting the work done without having any formal authority over anyone. These skills help PMs become great leaders who help turnaround the company by onboarding and motivating everyone to work towards a common vision.</p><h3><strong>Is it the right fit for you?</strong></h3><p>It seems like a glamorous world to people who want to get into Product Management and to some extent it is! In reality, Product Management is a thankless job and you are constantly on your toes to work on the next big thing, solving customer problems, driving business growth, etc.</p><blockquote>“All that glitters is not gold”</blockquote><blockquote>- William Shakespeare</blockquote><p>Make no mistake, I love working as a Product Manager and I don’t see myself doing anything else. But if you are thinking to get into Product Management, you need to first ensure that it is the right fit for you. Ask yourself honestly the following five important questions:</p><blockquote>1. Can you empathize with customers?</blockquote><blockquote>2. Do you love solving problems?</blockquote><blockquote>3. Do you like working with different stakeholders?</blockquote><blockquote>4. Do you have the attitude for getting shit done?</blockquote><blockquote>5. Can you handle a lot of meetings (a lot might be an understatement) and emails?</blockquote><p>If the answer to all the above questions is <strong>YES</strong>, you are already one step closer to becoming a Product Manager and you have come to the right place. Now, let us see how we can get you there.</p><h3><strong>How to get into Product Management</strong></h3><p><strong>1.</strong> <strong>Starting your own company</strong>: Nothing can teach you Product Management better than starting your company. Reason? Because as a founder, you do everything on your own (at least in the starting). You’re your company’s Data Analyst, Marketer, Product Manager, Customer Support, and whatnot. You become a jack of all trades and master of none, which is exactly what is needed to be a great Product Manager because you interact with so many different stakeholders every day. It is a fun <strong>rollercoaster ride</strong> with great learning experience but it can be quite expensive if your ultimate goal is just to get into Product Management. However, if you get that opportunity and are up for the challenge, I’d recommend trying your hand at <strong>entrepreneurship</strong> at least once. I started my startup and <strong>learned so much</strong> in the process, especially things that don’t work, which brings you one step closer to things that will work.</p><blockquote><strong><em>“I have not failed. I’ve just found 10,000 ways that won’t work.”</em></strong></blockquote><blockquote><strong><em>- Thomas Edison</em></strong></blockquote><p><strong>2.</strong> <strong>Associate Product Manager (APM) Programs/Internship Programs</strong>: This route is most suited for fresh graduates. Big companies such as Facebook, Google, Amazon, Apple, Microsoft, etc. have APM programs that help people with <strong>no experience</strong> get into Product Management. These programs help you learn fast because of the <strong>well-designed curriculum</strong>, which includes working on live high user base products, access to mentorship and coaching, rotation within different product organizations, and the opportunity to build a network but are <strong>very competitive</strong>. You can find a list of all APM programs <a href="https://apmlist.com/">here</a> and signup for the programs to get a notification when they do open up. The other route is getting a summer internship at a big company. You can find internships <a href="https://www.linkedin.com/jobs/search/?geoId=103644278&amp;keywords=product%20management%20intern&amp;location=United%20States">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7iW9y3e-8OnPxe2qbljqPg.png" /><figcaption>Image Source: RocketBlocks</figcaption></figure><p><strong>3.</strong> <strong>Transition within the company</strong>: This is one of the quickest ways to get into Product Management without spending a penny. It doesn’t matter which role you’re working in. There will always be something in your role that you can <strong>mold into Product Manager responsibilities</strong> or at least try to get closer to the product team within your organization. Meet Product Managers and try to understand their role, <strong>extend your help</strong> in any which way you can, and <strong>express your desire</strong> to join their team. I have seen people moving from Marketing, Customer Experience, Operations, Analytics, Engineering to Product Management. So can you! You can find more details on how to do that transition <a href="https://blog.hubspot.com/service/become-product-manager">here</a>.</p><blockquote>“If you really want to do something, you’ll find a way. If you don’t, you’ll find an excuse.”</blockquote><blockquote>- Jim Rohn</blockquote><p>4. <strong>MBA:</strong> A lot of people also pursue an MBA to get into middle or senior level Product Manager roles at big companies, which pay handsomely to freshly minted MBAs. Fortune 500 companies <strong>recruit MBAs directly from campus</strong> for these roles and other companies are constantly looking for MBAs to fill up Product Manager roles. You might be thinking why? Some of the main reasons are that MBAs get to interact with a lot of people from different backgrounds during MBA, work with them on a couple of wide range of projects, give presentations and take mandatory courses such as Finance, Strategy, Operations, Data Analytics, etc. So, MBA is an <strong>overall package</strong> that is useful for the Product Management job. But it does come with a price, a hefty price! You can find more details on how an MBA can help <a href="https://blogs.haas.berkeley.edu/the-berkeley-mba/do-i-need-an-mba-to-be-a-product-manager-part-i">here</a> and the Product Management plan <a href="https://blog.usejournal.com/how-to-get-into-product-management-esp-for-the-mba-folks-4374bab31be8">here</a>.</p><blockquote>“Get an education because no one can take that from you.”</blockquote><blockquote>- Nomar Garciaparra</blockquote><p>5. <strong>PM Bootcamps/Programs</strong>: There are other Product Management specific programs/boot camps offered by companies such as <a href="https://medium.com/productschool.com">Product School</a>, <a href="https://generalassemb.ly/">General Assembly.</a> These are comparatively less expensive than an MBA and are seeing a rising trend for Product Manager aspirants. These might also be helpful for people who want to transition within the company but lack some skills and don’t want to spend a lot on higher education. These boot camps help you with the <strong>skills and knowledge</strong> required for Product Management by building digital products, getting mentorship, teaching case studies, etc. The best part is that these courses are taught by featured instructors from fortune 500 companies.</p><blockquote>“An investment in knowledge pays the best interest.”</blockquote><blockquote>- Benjamin Franklin</blockquote><h3><strong>Resources to get you started ASAP</strong></h3><p>All of the above steps might take some planning and time to execute but no one is stopping you to start learning the ins and outs of Product Management. Check out some of the following well-known resources that will help you get a better idea of Product Management responsibilities and how to crack Product Management interviews.</p><p>1. <a href="https://www.amazon.com/Cracking-PM-Interview-Product-Technology-ebook/dp/B00ISYMUR6/ref=sr_1_3?dchild=1&amp;keywords=Cracking+the+PM+Interview&amp;qid=1593817167&amp;sr=8-3">Cracking the PM Interview: How to Land a Product Manager Job in Technology</a></p><p>2. <a href="https://www.amazon.com/INSPIRED-Create-Tech-Products-Customers-ebook/dp/B077NRB36N/ref=sr_1_2?dchild=1&amp;keywords=Inspired%3A+How+to+Create+Tech+Products+Customers+Love&amp;qid=1593817012&amp;sr=8-2">Inspired: How to Create Tech Products Customers Love</a></p><p>3. <a href="https://www.amazon.com/Eric-Ries-ebook/dp/B004J4XGN6/ref=tmm_kin_swatch_0?_encoding=UTF8&amp;qid=1593817040&amp;sr=8-1">The Lean Startup: How Today’s Entrepreneurs Use Continuous Innovation to Create Radically Successful Businesses</a></p><p>4. <a href="https://www.amazon.com/Swipe-Unlock-Technology-Business-Strategy/dp/1976182190/ref=sr_1_2_sspa?dchild=1&amp;keywords=product+management+books&amp;qid=1593817085&amp;sr=8-2-spons&amp;psc=1&amp;spLa=ZW5jcnlwdGVkUXVhbGlmaWVyPUEzOURQWTI2RUk3WUo0JmVuY3J5cHRlZElkPUEwNTc3NTE2M0tKNUczOTNGMzFLUCZlbmNyeXB0ZWRBZElkPUEwNDEzOTczRFRCT1A5STRHSkZKJndpZGdldE5hbWU9c3BfYXRmJmFjdGlvbj1jbGlja1JlZGlyZWN0JmRvTm90TG9nQ2xpY2s9dHJ1ZQ==">Swipe to Unlock: The Primer on Technology and Business Strategy</a></p><p>5.<a href="https://www.amazon.com/Decode-Conquer-Answers-Management-Interviews/dp/0615930417/ref=sr_1_4?dchild=1&amp;keywords=Decode+and+Conquer%3A+Answers+to+Product+Management+Interviews&amp;qid=1593817134&amp;sr=8-4"> Decode and Conquer: Answers to Product Management Interviews</a></p><p>In case you’re wondering how I got into Product Management - I started my own company and worked on e-Commerce products, developed and launched some products during my MBA, and worked in a Product role at my University.</p><p>Happy learning &amp; Best of luck! Let me know if you find this blog helpful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=beff26ca80c7" width="1" height="1" alt=""><hr><p><a href="https://medium.productcoalition.com/5-ways-resources-to-get-into-product-management-beff26ca80c7">5 Ways &amp; Resources to get into Product Management</a> was originally published in <a href="https://medium.productcoalition.com/">Product Coalition</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Predicting Sales: Time Series Analysis & Forecasting with Python]]></title>
            <link>https://medium.com/analytics-vidhya/predicting-sales-time-series-analysis-forecasting-with-python-b81d3e8ff03f?source=rss-bb6928ce1fde------2</link>
            <guid isPermaLink="false">https://medium.com/p/b81d3e8ff03f</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Bisman Preet Singh]]></dc:creator>
            <pubDate>Tue, 28 Jan 2020 21:59:30 GMT</pubDate>
            <atom:updated>2020-06-23T04:05:57.688Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*qGhy8mb-KwikYc2dIGf3YQ.png" /></figure><p>One of the most important tasks for any retail store company is to analyze the performance of its stores. The main challenge faced by any retail store is predicting in advance the sales and inventory required at each store to avoid over-stocking and under-stocking. This helps the business to provide the best customer experience and avoid getting into losses, thus ensuring the store is sustainable for operation.</p><p>In this post, I’ll use Rossmann store <a href="https://www.kaggle.com/c/rossmann-store-sales">data</a> available on Kaggle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/1*kNVl-iiV6YshJBXTlMUvWg.png" /></figure><p>Rossmann operates over 3,000 drug stores in 7 European countries. The challenge is to predict their daily sales for up to six weeks in advance. Store sales are influenced by many factors, including promotions, competition, school and state holidays, seasonality, and locality.</p><p>This post is divided into two parts: EDA &amp; Forecasting</p><h3><strong>Part A) Exploratory Data Analysis (EDA)</strong></h3><p>Let’s start by first importing the required libraries followed by data exploration.</p><pre># Importing required libraries<br>import numpy as np<br>import pandas as pd, datetime<br>import seaborn as sns<br>from statsmodels.tsa.stattools import adfuller<br>import matplotlib.pyplot as plt<br>get_ipython().run_line_magic(&#39;matplotlib&#39;, &#39;inline&#39;)<br>from time import time<br>import os<br>from math import sqrt<br>from statsmodels.tsa.seasonal import seasonal_decompose<br>from statsmodels.graphics.tsaplots import plot_acf, plot_pacf<br>import itertools<br>import statsmodels.api as sm<br>from statsmodels.tsa.stattools import acf,pacf<br>from statsmodels.tsa.arima_model import  ARIMA<br>from sklearn import model_selection<br>from sklearn.metrics import mean_squared_error, r2_score<br>from pandas import DataFrame<br>import xgboost as xgb<br>from fbprophet import Prophet<br>import warnings<br>warnings.filterwarnings(&#39;ignore&#39;)</pre><pre># Importing store data<br>store = pd.read_csv(&#39;./data/store.csv&#39;)<br>store.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BSBvpx-kzy_6aYyFByLm7Q.jpeg" /></figure><p>The above table gives us information about 1115 stores owned by Rossman.</p><pre># Importing train data<br>train = pd.read_csv(&#39;./data/train.csv&#39;, index_col=&#39;Date&#39;, parse_dates = True)<br>train.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*6TSJPsDBssOKp9MsHs0XbQ.jpeg" /></figure><p>We can see from the above table that the date is one of the columns. What makes this analysis unique is that ‘Date’ is one of the important factors influencing sales and it acts as an independent variable. To put it simply, this is a time-series data i.e a series of data points ordered in time.</p><ol><li><strong>Trends &amp; Seasonality</strong></li></ol><p>Let’s see how the sales vary with month, promo, promo2 (second promotional offer) and year.</p><pre># Sales trend over the months and year<br>sns.factorplot(data = train_store_joined, x =&quot;Month&quot;, y = &quot;Sales&quot;, <br>               col = &#39;Promo&#39;, # per store type in cols<br>               hue = &#39;Promo2&#39;,<br>               row = &quot;Year&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*VE_f7uNgiyMjWSFge_JgCg.png" /></figure><p>The above graph tells us that sales tend to spike in December, which makes sense because of the Christmas and holiday season. So, this confirms that the sales vary with the ‘Date’ (time) and there is a seasonality factor present in our data.</p><pre># Sales trend over days<br>sns.factorplot(data = train_store_joined, x = &quot;DayOfWeek&quot;, y = &quot;Sales&quot;, hue = &quot;Promo&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/1*jK4iQ_Kipelko-ceVmRYtg.png" /></figure><p>We can see from the above trend that there are no promotions on the weekends i.e Saturday and Sunday, which makes sense as stores want to earn a maximum profit during the time when people do their house chores.</p><p>The sales tend to increase on Sunday because people shop during the weekend. We can also see that the maximum sale happens on Mondays when there are promotional offers.</p><p><strong>2. Stationarity of Time Series</strong></p><p>In order to use time series forecasting models, we need to ensure that our time series data is stationary i.e constant mean, constant variance and constant covariance with time.</p><p>There are 2 ways to test the stationarity of time series:</p><p><strong>a) Rolling Mean</strong>: A rolling analysis of a time series model is often used to assess the model’s stability over time. The window is rolled (slid across the data) on a weekly basis, in which the average is taken on a weekly basis. Rolling Statistics is a visualization test, where we can compare the original data with the rolled data and check if the data is stationary or not.</p><p><strong>b) Dicky -Fuller test</strong>: This test provides us the statistical data such as p-value to understand whether we can reject the null hypothesis. The null hypothesis is that data is not stationary and the alternative hypothesis says that data is stationary. If the p-value is less than the critical value (say 0.5), we will reject the null hypothesis and say that data is stationary.</p><p>Let’s check the stationarity of a store of type ‘a’.</p><pre># Data Preparation: input should be float type<br>train[&#39;Sales&#39;] = train[&#39;Sales&#39;] * 1.0</pre><pre># Assigning one store from each category<br>sales_a = train[train.Store == 2][&#39;Sales&#39;]</pre><pre># Trend<br>sales_a.resample(&#39;W&#39;).sum().plot(ax = ax1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1U2JvtMHnDHVcxBhaALNxg.jpeg" /></figure><p>The above graph tells us that sales tend to peak at the end of the year.</p><pre># Function to test the stationarity<br>def test_stationarity(timeseries):<br>    <br>    # Determing rolling statistics<br>    roll_mean = timeseries.rolling(window=7).mean()<br>    roll_std = timeseries.rolling(window=7).std()</pre><pre># Plotting rolling statistics:<br>    orig = plt.plot(timeseries.resample(&#39;W&#39;).mean(), color=&#39;blue&#39;,label=&#39;Original&#39;)<br>    mean = plt.plot(roll_mean.resample(&#39;W&#39;).mean(), color=&#39;red&#39;, label=&#39;Rolling Mean&#39;)<br>    std = plt.plot(roll_std.resample(&#39;W&#39;).mean(), color=&#39;green&#39;, label = &#39;Rolling Std&#39;)<br>    plt.legend(loc=&#39;best&#39;)<br>    plt.show(block=False)<br>    <br>  # Performing Dickey-Fuller test:<br>    print(&#39;Results of Dickey-Fuller Test:&#39;)<br>    result = adfuller(timeseries, autolag=&#39;AIC&#39;)<br>    print(&#39;ADF Statistic: %f&#39; % result[0])<br>    print(&#39;p-value: %f&#39; % result[1])<br>    print(&#39;Critical Values:&#39;)<br>    for key, value in result[4].items():<br>           print(key, value)</pre><pre># Testing stationarity of store type a<br>test_stationarity(sales_a)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*4EklT8IlBJmaPxNFze1t1g.jpeg" /></figure><p>We can see from the above plot and statistical test that mean and variation doesn’t change much with time, i.e they are constant. Thus, we don’t need to perform any transformation (needed when time series is not stationary).</p><p>Now, let’s see the seasonality and trend using decomposition plots.</p><pre># Plotting seasonality and trend<br>def plot_timeseries(sales,StoreType):</pre><pre>fig, axes = plt.subplots(2, 1, sharex=True, sharey=False)<br>    fig.set_figheight(10)<br>    fig.set_figwidth(15)</pre><pre>decomposition= seasonal_decompose(sales, model = &#39;additive&#39;,freq=365)</pre><pre>estimated_trend = decomposition.trend<br>    estimated_seasonal = decomposition.seasonal<br>    estimated_residual = decomposition.resid<br>    <br>    axes[1].plot(estimated_seasonal, &#39;g&#39;, label=&#39;Seasonality&#39;)<br>    axes[1].legend(loc=&#39;upper left&#39;);<br>    <br>    axes[0].plot(estimated_trend, label=&#39;Trend&#39;)<br>    axes[0].legend(loc=&#39;upper left&#39;);</pre><pre>plt.title(&#39;Decomposition Plots&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*__BMD7rd77TG3jJdR3vEBQ.png" /></figure><p>From the above plots, we can see that there are seasonality and trend present in our data. So, we’ll use the forecasting models that take both of these factors into consideration.</p><h3><strong>Part B) Forecasting: Predictive Modelling</strong></h3><ol><li><strong>Evaluation Metrics</strong></li></ol><p>There are two popular metrics used in measuring the performance of regression (continuous variable) models i.e MAE &amp; RMSE.</p><p><strong>Mean Absolute Error (MAE)</strong>: It is the average of the absolute difference between the predicted values and observed values.</p><p><strong>Root Mean Square Error (RMSE)</strong>: It is the square root of the average of squared differences between the predicted values and observed values.</p><p>MAE is easier to understand and interpret but RMSE works well in situations where large errors are undesirable. This is because the errors are squared before they are averaged, thus penalizing large errors. In our case, RMSE suits well because we want to predict the sales with minimum error (i.e penalize high errors) so that inventory can be managed properly.</p><p>So, we’ll choose RMSE as a metric to measure the performance of our models.</p><p><strong>2. Predictive Modelling</strong></p><p><strong>Model 1: Autoregressive Integrated Moving Average (ARIMA)</strong></p><p>We will use one of the most commonly used methods for time-series forecasting, known as ARIMA.</p><p>ARIMA models are denoted by ARIMA(p, d, q).</p><p>p, d, and q represent seasonality, trend, and noise in data respectively. We’ll first create all possible combinations of p, d, and q as follows:</p><pre># Define the p, d and q parameters to take any value between 0 and 3<br>p = d = q = range(0, 2)</pre><pre># Generate all different combinations of p, q and q triplets<br>pdq = list(itertools.product(p, d, q))</pre><pre># Generate all different combinations of seasonal p, q and q triplets<br>seasonal_pdq = [(x[0], x[1], x[2], 12) for x in list(itertools.product(p, d, q))]</pre><pre>print(&#39;Examples of parameter combinations for Seasonal ARIMA: &#39;)<br>print(&#39;SARIMAX: {} x {}&#39;.format(pdq[1], seasonal_pdq[1]))<br>print(&#39;SARIMAX: {} x {}&#39;.format(pdq[1], seasonal_pdq[2]))<br>print(&#39;SARIMAX: {} x {}&#39;.format(pdq[2], seasonal_pdq[3]))<br>print(&#39;SARIMAX: {} x {}&#39;.format(pdq[2], seasonal_pdq[4]))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*IszQIOCJ4ERsrS3tmO_W8g.jpeg" /></figure><p><strong>Hyperparameter tuning for ARIMA</strong></p><p>In order to choose the best combination of the above parameters, we’ll use a <strong>grid search</strong>. The best combination of parameters will give the lowest Akaike information criterion (AIC) score. AIC tells us the quality of statistical models for a given set of data.</p><pre># Determing p,d,q combinations with AIC scores.<br>for param in pdq:<br>    for param_seasonal in seasonal_pdq:<br>        try:<br>            mod = sm.tsa.statespace.SARIMAX(train_arima,<br>                                            order=param,<br>                                            seasonal_order=param_seasonal,<br>                                            enforce_stationarity=False,<br>                                            enforce_invertibility=False)</pre><pre>results = mod.fit()</pre><pre>print(&#39;ARIMA{}x{}12 - AIC:{}&#39;.format(param, param_seasonal, results.aic))<br>        except:<br>            continue</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/1*wULpCPJwCLs2ARfvrUmR4g.jpeg" /></figure><p>I’ve only included a snapshot of how the grid search looks. The above iteration suggested that <strong>SARIMAX(1, 1, 1)x(0, 1, 1, 12)12</strong> is the best parameter combination with the lowest <strong>AIC: 1806.29.</strong></p><p><strong>Fitting the ARIMA model</strong></p><pre># Fitting the data to ARIMA model <br>model_sarima = sm.tsa.statespace.SARIMAX(train_arima,<br>                                order=(1, 1, 1),<br>                                seasonal_order=(0, 1, 1, 12),<br>                                enforce_stationarity=False,<br>                                enforce_invertibility=False)</pre><pre>results_sarima = model_sarima.fit()</pre><pre>print(results_sarima.summary().tables[1])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/935/1*G47mhT26epRE39MciTe0kw.jpeg" /></figure><p>Let’s check diagnostic plots to visualize the performance of our model.</p><pre># Checking diagnostic plots<br>results_sarima.plot_diagnostics(figsize=(10, 10))<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*SSfLyezI74KzLGFOqTzrgA.png" /></figure><p>The Normal Q-Q plot shows that the ordered distribution of residuals follows the distribution similar to normal distribution. Thus, our model seems to be pretty good.</p><p><strong>Model Prediction</strong></p><pre># Model Prediction<br># Predictions are performed for the 11th Jan&#39; 2015 onwards of the train data.</pre><pre>pred = results_sarima.get_prediction(start=pd.to_datetime(&#39;2015-01-11&#39;), dynamic = False)</pre><pre># Get confidence intervals of forecasts<br>pred_ci = pred.conf_int()</pre><pre>ax = train_arima[&quot;2014&quot;:].plot(label = &quot;observed&quot;, figsize=(15, 7))<br>pred.predicted_mean.plot(ax = ax, label = &quot;One-step ahead Forecast&quot;, alpha = 1)<br>ax.fill_between(pred_ci.index, <br>                pred_ci.iloc[:, 0], <br>                pred_ci.iloc[:, 1], <br>                color = &quot;k&quot;, alpha = 0.05)</pre><pre>ax.set_xlabel(&quot;Date&quot;)<br>ax.set_ylabel(&quot;Sales&quot;)</pre><pre>plt.legend<br>plt.show()</pre><pre>train_arima_forecasted = pred.predicted_mean<br>train_arima_truth = train_arima[&quot;2015-01-11&quot;:]</pre><pre># Calculating the error<br>rms_arima = sqrt(mean_squared_error(train_arima_truth, train_arima_forecasted))<br>print(&quot;Root Mean Squared Error: &quot;, rms_arima)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*QMem6seYYznUHFmqdFZkFg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/1*mFX41emlQ9sg53A9cqMNtg.jpeg" /></figure><p>The above plot shows that our predicted values catch up to the observed values in the dataset. Our forecasts seem to align with the ground truth very well and show a spike in December as expected. RMSE is also reasonably low in our case.</p><p>So, final ARIMA model can be represented as <strong>SARIMAX(1, 1, 1)x(0, 1, 1, 12)12. </strong>This is the best we can do with ARIMA, so let’s try another model to see whether we can decrease the RMSE.</p><p><strong>Model 2: Prophet</strong></p><p>Prophet is an open-source tool by Facebook. This procedure is used for forecasting time series data based on an additive model where non-linear trends are fit with yearly, weekly, and daily seasonality, plus holiday effects.</p><p><strong>Baseline model</strong></p><p>Our baseline (initial) model will use the default parameters. Let’s see how it performs.</p><pre># Creating a train dataset<br>train_prophet = train.copy()<br>train_prophet.reset_index(level=0, inplace=True)</pre><pre># Converting col names to specific names as required by Prophet library<br>train_prophet = train_prophet.rename(columns = {&#39;Date&#39;: &#39;ds&#39;,<br>                                &#39;Sales&#39;: &#39;y&#39;})</pre><pre># Downsampling to week because modelling on daily basis takes a lot of time<br>ts_week_prophet = train_prophet.set_index(&quot;ds&quot;).resample(&quot;W&quot;).sum()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3i_fSCXdSJ354FRsqZd_2Q.jpeg" /></figure><p><strong>Fitting the Prophet model</strong></p><pre># Fitting data to Prophet model<br>prophet_1 = Prophet() <br>prophet_1.fit(ts_week_prophet_train)</pre><p><strong>Model Prediction</strong></p><pre>future_1 = prophet_1.make_future_dataframe(periods = 52, freq = &quot;W&quot;) <br>forecast_1 = prophet_1.predict(future)<br>forecast_1[[&quot;ds&quot;, &quot;yhat&quot;, &quot;yhat_lower&quot;, &quot;yhat_upper&quot;]].tail()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*1UjUuNjVCQr-WwK4EoheYg.jpeg" /></figure><pre># Visualizing predicions of forecast<br>prophet.plot(forecast_1);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*p7HAygrTocsQ_dDbXmLOLw.png" /></figure><p>We can see from the above plot that the predictions are decent enough but let’s look at the RMSE to get a better idea.</p><pre># Checking the RMSE of Prophet model<br>metric_prophet_1 = forecast_1.set_index(&#39;ds&#39;)[[&#39;yhat&#39;]].join(ts_week_prophet_train.set_index(&#39;ds&#39;).y).reset_index()<br>metric_prophet_1.dropna(inplace=True)<br>rms_prophet_1 = mean_squared_error(metric_prophet_1.y, metric_prophet_1.yhat)<br>rms_prophet_1</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/217/1*RWEvgHKqv_r0LBtM4ECiVw.jpeg" /></figure><p>Wow! RMSE, in this case, is too large and we need to do something about it. Let’s see if we can reduce it by manipulating some of the parameters.</p><h4><strong>Hyperparameter tuning for Prophet</strong></h4><p>There are a lot of parameters available in the Prophet model. One of the most important ones is <strong>‘holidays’</strong>. This lets us parse holidays explicitly while training the model. We’ll create a new ‘holidays’ data frame by taking observations when there was a school or state holiday.</p><p>We’ll also use three more parameters viz.</p><ul><li><strong>interval_width</strong>: It defines the uncertainty level to make the prediction. The default value is 0.8 but we’ll take 0.95 because we want to be more certain in our predictions.</li><li><strong>growth</strong>: We know that ‘Sales’ can take any value and there is no saturation point. So, we’ll take ‘linear’ growth instead of ‘logarithmic’.</li><li><strong>yearly_seasonality</strong>: We’ll explicitly pass it as ‘True’ because we know that there is a yearly seasonality (discussed above) present in our data.</li></ul><pre># Encoding state holiday categorical variable<br>train_prophet[&quot;StateHoliday_cat&quot;] = train_prophet[&quot;StateHoliday&quot;].map({0:0, &quot;0&quot;: 0, &quot;a&quot;: 1, &quot;b&quot;: 1, &quot;c&quot;: 1})</pre><pre># Choosing only required cols<br>train_prophet = train_prophet[[&#39;ds&#39;, &#39;y&#39;, &#39;SchoolHoliday&#39;, &#39;StateHoliday_cat&#39;]]<br>train_prophet.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*6as_Bi7saptg0vx99kP2eA.jpeg" /></figure><pre># Modelling holidays - creating holidays dataframe<br>state_dates = train_prophet[(train_prophet.StateHoliday_cat == 1)].loc[:, &quot;ds&quot;].values<br>school_dates = train_prophet[(train_prophet.SchoolHoliday == 1)].loc[:, &quot;ds&quot;].values</pre><pre>state = pd.DataFrame({&quot;holiday&quot;: &quot;state_holiday&quot;,  &quot;ds&quot;: pd.to_datetime(state_dates)})<br>school = pd.DataFrame({&quot;holiday&quot;: &quot;school_holiday&quot;, &quot;ds&quot;: pd.to_datetime(school_dates)})</pre><pre>holidays = pd.concat((state, school))<br>holidays.head()</pre><pre># Dropping holiday columns because not needed any more<br>train_prophet_clean = train_prophet.drop([&quot;SchoolHoliday&quot;, &quot;StateHoliday_cat&quot;], axis = 1)</pre><pre># Downsampling to week because modelling on daily basis takes a lot of time<br>ts_week_prophet = train_prophet_clean.set_index(&quot;ds&quot;).resample(&quot;W&quot;).sum()</pre><pre># Resetting the index<br>ts_week_prophet_train = ts_week_prophet.reset_index()<br></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/266/1*VPBLPXAcpDmZemWzzDgyRw.jpeg" /></figure><p><strong>Fitting the hyper tuned Prophet model</strong></p><pre># Fitting data to Prophet model<br>prophet_2 = Prophet(holidays = holidays, interval_width = 0.95, growth=&#39;linear&#39;, yearly_seasonality = True) <br>prophet_2.fit(ts_week_prophet_train)<br>print(&quot;done&quot;)</pre><p><strong>Model Prediction</strong></p><pre>future_2 = prophet_2.make_future_dataframe(periods = 52, freq = &quot;W&quot;) <br>forecast_2 = prophet_2.predict(future)<br>forecast_2[[&quot;ds&quot;, &quot;yhat&quot;, &quot;yhat_lower&quot;, &quot;yhat_upper&quot;]].tail() # We have a new dataframe, which includes, the forecast and the uncertainity invervals.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*Ahzd8-arcZdLKZcfJwqXRA.jpeg" /></figure><pre># Visualizing predicions of forecast<br>prophet.plot(forecast_2);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*SpeaxUAjsQzCl1wDHonioQ.png" /></figure><pre># Visualizing trend and seasonality components<br>prophet.plot_components(forecast_2);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*J4aSYNOAeCyQQksBFVPVqA.png" /></figure><p>The first plot shows that the total sales on a weekly basis are increasing. The second plot shows the holiday gaps in the dataset and the third plot shows that the store sees very high sales in the last week of December (because of the Christmas holidays).</p><pre># Checking the RMSE of Prophet model<br>metric_prophet_2 = forecast_2.set_index(&#39;ds&#39;)[[&#39;yhat&#39;]].join(ts_week_prophet_train.set_index(&#39;ds&#39;).y).reset_index()<br>metric_prophet_2.dropna(inplace=True)<br>rms_prophet_2 = mean_squared_error(metric_prophet_2.y, metric_prophet_2.yhat)<br>rms_prophet_2</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/227/1*k7tjC4iUwwXS3xmBLm6Ctg.jpeg" /></figure><p>Our baseline Prophet model used default parameters and we got RMSE of 53782649094881.14 and after hyper tuning, we got RMSE of 52478331938232.15. Although the final model is performing better, it is still performing poorly as compared to ARIMA. So, let’s try another model.</p><p><strong>Model 3: XGBoost</strong></p><p>XGBoost is an optimized distributed gradient boosting library designed to be highly <strong>efficient</strong>, <strong>flexible</strong> and <strong>portable</strong>. Although it is not specifically designed for time-series data, it is known to perform extremely well in all kinds of regression problems.</p><pre># Dropping Customers and Sale per customer<br>ts_xgboost = train_store_joined.copy()<br>ts_xgboost = ts_xgboost.drop([&#39;Customers&#39;, &#39;SalePerCustomer&#39;, &#39;PromoInterval&#39;], axis=1)</pre><pre># Combining similar columns into one column and dropping old columns<br>ts_xgboost[&#39;CompetitionOpen&#39;] = 12 * (ts_xgboost.Year - ts_xgboost.CompetitionOpenSinceYear) + (ts_xgboost.Month - ts_xgboost.CompetitionOpenSinceMonth)<br>ts_xgboost[&#39;PromoOpen&#39;] = 12 * (ts_xgboost.Year - ts_xgboost.Promo2SinceYear) + (ts_xgboost.WeekOfYear - ts_xgboost.Promo2SinceWeek) / 4.0<br>ts_xgboost = ts_xgboost.drop([&quot;CompetitionOpenSinceMonth&quot;, &quot;CompetitionOpenSinceYear&quot;], axis = 1)<br>ts_xgboost = ts_xgboost.drop([&quot;Promo2SinceWeek&quot;, &quot;Promo2SinceYear&quot;], axis = 1)</pre><pre># Converting categorical cols to numerical cols and removing old cols<br>mappings = {0:0, &quot;0&quot;: 0, &quot;a&quot;: 1, &quot;b&quot;: 1, &quot;c&quot;: 1}<br>ts_xgboost[&quot;StateHoliday_cat&quot;] = ts_xgboost[&quot;StateHoliday&quot;].map(mappings)<br>ts_xgboost[&quot;StoreType_cat&quot;] = ts_xgboost[&quot;StoreType&quot;].map(mappings)<br>ts_xgboost[&quot;Assortment_cat&quot;] = ts_xgboost[&quot;Assortment&quot;].map(mappings)<br>ts_xgboost = ts_xgboost.drop([&quot;StateHoliday&quot;, &quot;StoreType&quot;, &quot;Assortment&quot;], axis = 1)</pre><pre># Splitting the data<br>features = ts_xgboost.drop([&quot;Sales&quot;], axis = 1)<br>target = ts_xgboost[&quot;Sales&quot;]</pre><pre>X_train, X_test, y_train, y_test = model_selection.train_test_split(features, target, test_size = 0.20)</pre><p><strong>Baseline Model</strong></p><p>Our baseline (initial) model will use the default parameters. Let’s see how it performs.</p><pre># Tuning parameters - using default metrics<br>params = {&#39;max_depth&#39;:6, &quot;booster&quot;: &quot;gbtree&quot;, &#39;eta&#39;:0.3, &#39;objective&#39;:&#39;reg:linear&#39;}</pre><pre>dtrain = xgb.DMatrix(X_train, y_train)<br>dtest = xgb.DMatrix(X_test, y_test)<br>watchlist = [(dtrain, &#39;train&#39;), (dtest, &#39;eval&#39;)]</pre><pre># Training the model<br>xgboost = xgb.train(params, dtrain, 100, evals=watchlist,early_stopping_rounds= 100, verbose_eval=True)<br>         <br># Making predictions<br>preds = xgboost.predict(dtest)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3ziwzXbCER4vwcsYyCMQnQ.jpeg" /></figure><pre># RMSE of model<br>rms_xgboost = sqrt(mean_squared_error(y_test, preds))<br>print(&quot;Root Mean Squared Error for XGBoost:&quot;, rms_xgboost)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*dOMmJyZfHzshhtrGI9euSg.jpeg" /></figure><p>It performs pretty well at least in comparison to Prophet. Let’s see if we can further reduce the RMSE.</p><h4>Hypertuning for XGBoost</h4><p>Now let’s try to decrease the RMSE of XGBoost by passing different values for our hyperparameters in the XGBoost model.</p><ul><li><strong>eta</strong>: It defines the learning rate i.e step size to learn the data in the gradient descent modeling (the basis for XGBoost). The default value is 0.3 but we want to keep the learning rate low to avoid overfitting. So, we’ll choose 0.2 as eta.</li><li><strong>max_depth</strong>: Maximum depth of a tree. The default value is 6 but we want our model to be more complex and find good predictions. So, let’s choose 10 as max depth.</li><li><strong>gamma</strong>: Minimum loss reduction required to make a further partition on a leaf node of the tree. The larger gamma is, the more conservative the algorithm will be. The default value is 0, let’s choose a little higher value so as to get good predictions.</li></ul><pre># Tuning parameters<br>params_2 = {&#39;max_depth&#39;:10, &#39;eta&#39;:0.1,  &#39;gamma&#39;: 2}</pre><pre>dtrain = xgb.DMatrix(X_train, y_train)<br>dtest = xgb.DMatrix(X_test, y_test)<br>watchlist = [(dtrain, &#39;train&#39;), (dtest, &#39;eval&#39;)]</pre><pre># Training the model<br>xgboost_2 = xgb.train(params_2, dtrain, 100, evals=watchlist,early_stopping_rounds= 100, verbose_eval=True)<br>         <br># Making predictions<br>preds_2 = xgboost_2.predict(dtest)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/979/1*QUfzOJoUzEdBzO7aoHBSSw.jpeg" /></figure><pre># RMSE of model<br>rms_xgboost_2 = sqrt(mean_squared_error(y_test, preds_2))<br>print(&quot;Root Mean Squared Error for XGBoost:&quot;, rms_xgboost_2)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/1*XH-EuGFCIJ2ESh4fZ11QNA.jpeg" /></figure><p>After hyper tuning, we see that our model’s RMSE decreased. Let’s see what features impact the sales of a store.</p><pre># Let&#39;s see the feature importance<br>fig, ax = plt.subplots(figsize=(10,10))<br>xgb.plot_importance(xgboost_2, max_num_features=50, height=0.8, ax=ax)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*dGzPQ24ZWWhZbnlSoDYXnA.png" /></figure><p>As expected, there are five major reasons affecting the sales of a store viz. the <strong>number of stores, competition distance, day of the week, is the competition open, and promotions</strong>.</p><p>Our final XGBoost model after hyper tuning is the one with <strong>‘max_depth’:10, ‘eta’:0.1, ‘gamma’: 2 and RMSE score of 1191.90</strong>, which is great! Now, let&#39;s compare the performance of all models</p><p><strong>3. Results</strong></p><p>We used the Root Mean Squared Error (RMSE) to evaluate and validate the performance of various models. Let’s see which model performed better and why/why not.</p><pre># Comparing performance of above three models - through RMSE<br>rms_arima = format(float(rms_arima))<br>rms_prophet_2 = format(float(rms_prophet_2))<br>rms_xgboost_2 = format(float(rms_xgboost_2))</pre><pre>model_errors = pd.DataFrame({<br>    &quot;Model&quot;: [&quot;SARIMA&quot;, &quot;Prophet&quot;, &quot;XGBoost&quot;],<br>    &quot;RMSE&quot;: [rms_arima, rms_prophet_2, rms_xgboost_2]<br>})</pre><pre>model_errors.sort_values(by = &quot;RMSE&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/330/1*UqT4XnzXl6GIfe675V0ADA.jpeg" /></figure><p><strong>4. Model Comparison &amp; Selection</strong></p><p>a) We can see from the above table that SARIMA performs the best followed by XGBoost and Prophet.</p><p>b) It makes sense because SARIMA is designed specifically for seasonal time series data while XGBoost is a general (though powerful) machine learning approach with various applications.</p><p>c) Prophet is a good choice for producing quick forecasts as it doesn’t require strong technical skills. It is easy to implement at scale. The reason for its poor performance here is probably because of a lack of data. It works best with time series that have strong seasonal effects and several seasons of historical data. Prophet is robust to missing data and shifts in the trend, and typically handles outliers well.</p><p>Based on the above analysis, we’ll choose ARIMA as our final model to predict the sales because it gives us the least RMSE and is well suited to our needs of predicting time series seasonal data. We chose <strong>ARIMA(1, 1, 1)x(0, 1, 1, 12)12 </strong>as the final parameter combination with <strong>AIC of 1806.29 and RMSE of 739.06.</strong></p><p><strong>5. Conclusion</strong></p><p><strong>Reflection</strong></p><ul><li>The most interesting thing about the data was that the category of stores having the highest sales didn’t have the highest sale per customer. It might be because those stores sell small items, which are needed on a daily basis.</li><li>Another interesting thing was that running a promotion for the second time didn’t help in increasing sales. It is probably because customers already purchased whatever they wanted during the first promotional sale.</li></ul><p><strong>Acknowledgments</strong></p><p><a href="https://machinelearningmastery.com/arima-for-time-series-forecasting-with-python/">https://machinelearningmastery.com/arima-for-time-series-forecasting-with-python/</a></p><p><a href="https://www.digitalocean.com/community/tutorials/a-guide-to-time-series-forecasting-with-arima-in-python-3">https://www.digitalocean.com/community/tutorials/a-guide-to-time-series-forecasting-with-arima-in-python-3</a></p><p><a href="https://xgboost.readthedocs.io/en/latest/python/python_intro.html">https://xgboost.readthedocs.io/en/latest/python/python_intro.html</a></p><p><a href="https://facebook.github.io/prophet/docs/quick_start.html">https://facebook.github.io/prophet/docs/quick_start.html</a></p><p>For more details, please check out the source code on <a href="https://github.com/bisman16/Kaggle_Rossmann_Store_Sales_Forecasting"><strong>Github</strong></a>.</p><p>If you enjoyed reading this story, please click the 👏 button and share it to help others learn machine learning. I’d love to hear your thoughts, so feel free to leave comments below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b81d3e8ff03f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/predicting-sales-time-series-analysis-forecasting-with-python-b81d3e8ff03f">Predicting Sales: Time Series Analysis &amp; Forecasting with Python</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[So, you want to be a developer?]]></title>
            <link>https://medium.com/@bismanpreetsingh/so-you-want-to-be-a-developer-1c86e23a3c51?source=rss-bb6928ce1fde------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c86e23a3c51</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Bisman Preet Singh]]></dc:creator>
            <pubDate>Fri, 20 Dec 2019 17:35:48 GMT</pubDate>
            <atom:updated>2019-12-20T17:35:48.653Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rffnex5tzarzkzVOvux7xQ.jpeg" /><figcaption>Image credit: Photo by Fabian Grohs on Unsplash</figcaption></figure><p><strong>Introduction</strong></p><p>Are you an aspiring developer or engineer who is looking to break into the field? It’s always a hassle to prioritize the things to learn because of a plethora of information available out there. I believe the best way to get insights is to hear directly from people working in the industry.</p><p>I derived some insights by doing data analysis on Stack Overflow’s 2019 annual developer survey taken by 90,000 developers from 170 countries. Let’s find answers to some of the questions that might help you get your foot in the industry.</p><p><strong>1. What languages and databases should I learn?</strong></p><p>Everyone has his/her own preference for languages and databases based on the ease of use, functionality, tech stack at company and use case. However, there are some common languages that are used in almost every company.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*rBl9uK07GXqFuh__IPPY4g.png" /></figure><blockquote><strong>JavaScript, HTML/CSS, SQL, Python, and Java are the top five languages used in the industry.</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*pmRc68shXRkgEZCK0bsrYg.png" /></figure><blockquote><strong>MySQL, PostgreSQL, Microsoft SQL Server, SQLite, MongoDB are the top five databases used in the industry.</strong></blockquote><p><strong>2. Should I pursue higher education and which subjects should I take?</strong></p><p>It’s a well-known fact that you don’t need a degree to break into the field. You can start coding in your room and with plenty of free and paid online tutorials, you can be as good a coder as anyone who joined a degree program. In fact, big companies such as Google, Apple, Facebook, etc. don’t care anymore whether you have a degree.</p><p>But if you prefer a structured learning experience, then you might want to join a degree program at a University. Let’s see what kind of backgrounds do developers have.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*KVPi9plVljjuc5cgfbSCag.png" /></figure><p>We can see from the above graph that a majority of software developers have some sort of higher education. Around 67% of the respondents have at least a bachelor’s degree, 22.35% of the respondents have a Master’s degree, 12.21% dropped out of college and 4% have a professional or doctoral degree.</p><blockquote><strong>Overall, 71% of the respondents have some sort of higher education.</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*RZ5MtlGmN1Ffk3nt89LXgw.png" /></figure><blockquote><strong>We can see from the above graph that around 82.5% of the software engineers have a degree in Computer Science, Engineering (civil, electrical, mechanical), Information Technology, Web development or design.</strong></blockquote><p><strong>3. What does job and career satisfaction look like?</strong></p><p>You always want to work in a field that gives you satisfaction and makes you happy about the work you do on a daily basis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*4q6yPubT4cSFGYSPkNxZsQ.png" /></figure><p>It can be seen that 65% of the respondents are satisfied with their job while 22% of the respondents are dissatisfied with their job. It’s pretty decent as compared to some of the other professions and we have to acknowledge the fact that job satisfaction depends on a lot of factors such as age, compensation, organization size, etc. Let’s see how satisfying is this career choice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*4FHnfawmLMmc8VCi9puEVQ.png" /></figure><blockquote><strong>We see that 74% of the people are satisfied with their career and it’s much higher than a lot of other career options.</strong></blockquote><p><strong>4. How much will I get paid and what factors influence the pay?</strong></p><p>Well, we have identified by now what kind of technology you need to learn, what does the job satisfaction look like and educational background to get into this field, one last item you might be wondering about is how much developers are paid.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/1*qcYUkmwgltMdbxrvp9zQMg.jpeg" /></figure><p>As you can imagine, salaries depend on the country you’re working in. I’ve taken the salaries for the top 10 countries based on the highest number of respondents. The above table tells us the median salaries in each country. As expected, the United States pays the highest followed by Canada, Germany, and the United Kingdom.</p><blockquote><strong>After running several machine learning models, I found out that Salary depends on the country, years you have been coding, age, hours per week spent, organization size, ethnicity, and job satisfaction.</strong></blockquote><h4><strong>Interesting Insight — Pay Discrepancy</strong></h4><p>It seems from the above table that developers in countries such as India, Brazil, the Russian Federation, and Poland are paid very low as compared to those in other countries. What’s going on here?</p><p>The problem is that Stack Overflow asked developers for their salaries in local currency and then converted those salaries into US$ as per the <strong>exchange rate</strong> at that time. The problem with this approach is that it doesn’t take the cost of living into account. This explains the huge difference in salaries of developers in different countries.</p><p>To get a real picture, I used <strong>Purchasing Power Parity (PPP)</strong>, an approach used by economists to compare different economies. I used the World Bank’s PPP conversion factor for analysis. Let’s see the adjusted salary (in terms of international dollars), a common currency to compare economies after factoring PPP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/1*Ak-XxW_5LPLR3lOxDteY3w.jpeg" /></figure><p>It can be seen from the above plot that the highest difference in the salaries after adjusting for the cost of living is seen for Poland, the Russian Federation, India, and Brazil. It’s better to use Adjusted Salary in order to make comparisons.</p><p>After adjusting the salary, Poland and the Russian Federation jumped up a few spots and have comparable salaries with other countries while India and Brazil still witness lower salaries.</p><p><strong>Conclusion</strong></p><p>1. We have found that JavaScript, HTML/CSS, SQL, and Python<strong><em> </em></strong>are the preferred languages. MySQL, PostgreSQL, Microsoft SQL Server, SQLite, and MongoDB are the preferred databases for developers<strong><em>.</em></strong></p><p>2. 71% of the respondents have some sort of higher education with a background in Computer Science, Engineering (civil, electrical, mechanical), Information Technology, Web development or design</p><p>3. 65% of the respondents are satisfied with their job and 74% of the respondents are satisfied with their career.</p><p>4. Finally, the United States pays the highest salary followed by Canada, Germany, and the United Kingdom. Salary depends on the country, years spent in coding, age, workweek hrs, organization size, ethnicity, job satisfaction.</p><p>For more details, please feel free to check out the analysis on my Github repository <a href="https://github.com/bisman16/Stack_Overflow_Analysis/blob/master/Stack_Overflow_Analysis.ipynb">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c86e23a3c51" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>