<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sudip Roy on Medium]]></title>
        <description><![CDATA[Stories by Sudip Roy on Medium]]></description>
        <link>https://medium.com/@sudip.1987roy?source=rss-7e59c4ac97d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*B-ESpeiDQD_rppGX</url>
            <title>Stories by Sudip Roy on Medium</title>
            <link>https://medium.com/@sudip.1987roy?source=rss-7e59c4ac97d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 04:52:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sudip.1987roy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[IPL 2023 Analysis — Build 10 Beautiful Intuitive Charts using Seaborn]]></title>
            <link>https://medium.com/@sudip.1987roy/ipl-2023-analysis-build-10-beautiful-intuitive-charts-using-seaborn-8f46be24b5d2?source=rss-7e59c4ac97d------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f46be24b5d2</guid>
            <category><![CDATA[ipl-2023]]></category>
            <category><![CDATA[seaborn-tutorial]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[cricket]]></category>
            <category><![CDATA[visualization]]></category>
            <dc:creator><![CDATA[Sudip Roy]]></dc:creator>
            <pubDate>Sun, 13 Aug 2023 15:57:31 GMT</pubDate>
            <atom:updated>2023-08-13T16:06:08.927Z</atom:updated>
            <content:encoded><![CDATA[<h3>IPL 2023 Analysis — Build 10 Beautiful Intuitive Charts using Seaborn</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*KH8wbOanP1HAfxITXiQI4w.jpeg" /></figure><p><strong>About IPL:</strong></p><p>The Indian Premier League or the IPL is a men’s Twenty20 (T20) cricket league that is annually held in India and contested by ten city-based franchise teams. The BCCI founded the league in 2007. The competition is usually held in summer between March and May every year.</p><p>The IPL is the most-popular cricket league in the world. In 2014, it was ranked sixth by average attendance among all sports leagues. The brand value of the league in 2022 was INR 90,038 crore (USD 11 billion). According to BCCI, the 2015 IPL season contributed INR 1,150 crore (USD 140 million) to the GDP of the economy of India.</p><p>The 2023 edition of the IPL was contested between 10 teams in which Chennai Super Kings emerged as the winner defeating Gujarat Titans in the finals of the tournament played at Narendra Modi Stadium in Ahmedabad.</p><p><strong>Site URL:</strong> <a href="https://colab.research.google.com/corgiredirector?site=https%3A%2F%2Fwww.iplt20.com%2F">https://www.iplt20.com/</a></p><p><strong>Problem Statement:</strong></p><p>Visualize the performance of various teams in the 2023 edition of the IPL played between different teams in the group stages of the tournament using Seaborn Library in Python.</p><p><strong>Why Seaborn?</strong></p><p>Seaborn is one of the most powerful visualization library in Python. Seaborn is used to analyse and depict results with meaningful charts. It is built on top of matplotlib library and supports most of the functionalities of matplotlib.</p><p><strong>API Link:</strong> <a href="https://colab.research.google.com/corgiredirector?site=https%3A%2F%2Fseaborn.pydata.org%2Findex.html">https://seaborn.pydata.org/index.html</a></p><p><strong>Datasets:</strong> I have used 5 datasets to highlight various facets of the IPL tournament. Link for the datasets and the complete notebook will be shared at the end of the article.</p><p><strong>Motivation: </strong>The primary motivation or the end result of my article is to showcase different charts that we can use for performing exploratory data analysis on any dataset and state observations, facts or figures to justify the visual representations. So, I will skip all the intermediate coding exercises that I have performed on the data to get the final results. But be rest assured, just like I mentioned, I will attach a link at the end of the article for you to go through the complete notebook.</p><p>So let’s begin and build some beautiful charts using Seaborn. Shall we ?</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ftenor.com%2Fembed%2F4767352&amp;display_name=Tenor&amp;url=https%3A%2F%2Ftenor.com%2Fview%2Fdespicable-me-minions-lets-get-started-happy-excited-gif-4767352&amp;image=https%3A%2F%2Fmedia.tenor.com%2Fr3XdvPsAV3kAAAAC%2Fdespicable-me-minions.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=tenor" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/f9cd27fa232dc101672874efef4882d0/href">https://medium.com/media/f9cd27fa232dc101672874efef4882d0/href</a></iframe><h3><strong>Chart Type 1: Count Plot</strong></h3><p>CountPlot shows the counts of observations in each categorical bin using bars. CountPlot cannot be used for quantitative variables. (Example: Number of people who performed transaction in an ATM across 7 days of the week, Total footfall across different apparel stores in Delhi)</p><p><strong>Let’s try and answer few questions from IPL 2023 to understand Count Plot</strong></p><h4>A. Was Toss a big factor in IPL 2023 ?</h4><p><strong>Code Snippet:</strong></p><pre># Size of the Chart<br>fig = plt.figure(figsize=(6,6))<br><br># Plotting the Count Plot<br>ax = sns.countplot(x=match_details_df[&quot;Toss A Factor ?&quot;])<br><br># Displaying values corresponding to each bar<br>ax.bar_label(ax.containers[0])<br><br># Setting Title Name<br>ax.set_title(&#39;Count of Matches won by Team Winning Toss in IPL 2023&#39;, \<br>fontsize =10, fontweight =&#39;bold&#39;)<br><br># Displaying Chart<br>plt.plot()</pre><p><strong>Results:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/1*NVwMszjdW9ImMZ50Tp1hbQ.png" /></figure><p><strong>Observation:</strong></p><blockquote>- Toss really a big factor in the outcome of the match. Hmmmmmm…. not really. Seems like more than half of the matches played was won by the team losing the toss. So, not really a big plus in winning the toss. What say people?</blockquote><h3>Chart Type 2: Bar Chart or Manhattan Chart</h3><p>A Bar Chart or Manhattan Chart is a chart or a graph that presents categorical data with rectangular bars with heights or lengths proportional to the values that they represent. Manhattan Chart comes in handy when we want to display comparison of numbers, which could be both discrete and continuous across different categories in the dataset. (Example: Total Sales across different regions in the state, Average Price per Unit of item at a Walmart store, Temperature in different cities of India)</p><p><strong>Let’s try and answer few questions from IPL 2023 to understand Bar Chart</strong></p><h4><strong>B. Which batting team is making the best use of the powerplay ?</strong></h4><p><strong>Code Snippet:</strong></p><pre># Size of the Chart<br>fig = plt.figure(figsize=(10,6))<br><br># Choosing color palette for teams<br>palette = [&#39;yellow&#39;, &#39;crimson&#39;, &#39;grey&#39;, &#39;indigo&#39;, &#39;cyan&#39;, &#39;blue&#39;, \<br>&#39;red&#39;, &#39;lightgreen&#39;, &#39;magenta&#39;, &#39;orange&#39;]<br><br># Plotting the Bar Chart<br>ax = sns.barplot(data=analysis1, x=&quot;Team&quot;, y=&quot;Powerplay_Score&quot;, \<br>palette = palette)<br><br># Displaying values corresponding to each bar<br>ax.bar_label(ax.containers[0])<br><br># Setting Title Name<br>ax.set_title(&#39;Average Runs Scored by Teams Inside Powerplay in IPL 2023&#39;, \<br>fontsize =10, fontweight =&#39;bold&#39;)<br><br># Displaying Chart<br>plt.plot()</pre><p><strong>Results:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/1*70olLqYRJ99OEk3VMFn4eA.png" /></figure><p><strong>Facts:</strong></p><blockquote>No surprises there eh? Chennai Super Kings (CSK), Gujarat Titans (GT), Royal Challengers Bangalore (RCB) and Rajasthan Royals (RR) had scored a great deal of the runs in the powerplay and the large marjority of which could be attributed to the opening partnership of these 4 teams. Ruturaj Gaikwad and Devon Conway for CSK, Shubhman Gill and Wriddhiman Saha for GT, Virat Kohli and Faf Du Plessis for RCB and Yashasvi Jaiswal and Jos Buttler for RR were the key contributors in the powerplay.</blockquote><blockquote>On the other hand Kolkata Knight Riders (KKR), Lucknow Super Giants (LSG), Delhi Capitals (DC), Sunrisers Hyderabad (SRH) and Punjab Kings (PK) had a decent start in the powerplays as one of the opening pair stuggled to get going in the first 6 overs of the innings. David Warner was the only shinning knight for DC in the first six overs, KKR never really got partnership going due to the struggles with different combinations at the top of the order, KL Rahul and Shikhar Dhawan struggled to get going in the powerplay for LSG and PK respectively while SRH never really had a smooth start to the innings in the first 6 overs of the match.</blockquote><blockquote>Mumbai Indians (MI) although chased well in the second half of the tournament but they struggled to set the tone in the powerplay overs, particularly while setting target for the opposition teams.</blockquote><h3>Chart Type 3: Histogram</h3><p>Histogram is a graph that shows the frequency of numerical data using bars.The height of a bar represents the distribution frequency of a variable. Histogram gives a very good intuition about the spread/distribution of the data in a variable. (Example: Maximum, Mininum or Median job offers of students in XYZ College)</p><p><strong>Let’s try and answer few questions from IPL 2023 to understand Bar Chart</strong></p><h4>C. What is the distribution of runs scored in IPL 2023 ?</h4><p><strong>Code Snippet:</strong></p><pre># Size of the Chart<br>fig = plt.figure(figsize=(10,6))<br><br># Plotting Distribution of Runs using Histplot<br>ax = sns.histplot(data=analysis4, x=&quot;Runs&quot;, kde = True, bins = 30)<br><br># Setting Title Name<br>ax.set_title(&#39;Distribution of runs scored in IPL 2023&#39;, fontsize =10, \<br>fontweight =&#39;bold&#39;)<br><br># Annotation for Outlier<br>ax.text(62, 1.50, &quot;Outlier&quot;, fontsize = 10, fontweight =&#39;bold&#39;, \<br>color = &quot;red&quot;, ha = &quot;center&quot;, va = &quot;center&quot;)<br><br># Displaying Chart<br>plt.plot()</pre><p><strong>Results:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*qfPu5frn76b0-2cYv7QjVQ.png" /></figure><p><strong>Observation:</strong></p><blockquote>All the scores made by teams lie in the range of 105 to 230, with 185+ seeming like a competitive total for most of the teams.</blockquote><p><strong>Facts:</strong></p><blockquote>There is also a big score of 250+ posted by Lucknow Super Giants (LSG) against Punjab Kings (PK), while Rajasthan Royals got bundled out for mere 59 runs against Royal Challengers Bangalore (RCB), the same is highlighted as an ‘Outlier’ in the above graph.</blockquote><h3>Chart Type 4: BoxPlot</h3><p>A boxplot is a graph that gives a visual indication of how a dataset’s mean, median, mode, minimum, maximum and outlier values are spread out and compare to each other. Boxplots are drawn as a box with a vertical line down the middle, and has horizontal lines attached to each side known as “whiskers”. (Example: Maximum, Mininum or Median job offers of students in XYZ College)</p><p><strong>Let’s try and answer few questions from IPL 2023 to understand Bar Chart</strong></p><h4>D. Is there a significant difference in the Runs Scored by Teams in IPL 2023 ?</h4><p><strong>Code Snippet:</strong></p><pre># Size of the Chart<br>fig = plt.figure(figsize=(12, 7))<br><br># Choosing color palette for teams<br>palette = [&#39;yellow&#39;, &#39;crimson&#39;, &#39;grey&#39;, &#39;indigo&#39;,&#39;cyan&#39;, &#39;blue&#39;, \<br>&#39;red&#39;,&#39;lightgreen&#39;,&#39;magenta&#39;,&#39;orange&#39;]<br><br># Plotting BoxPlot<br>ax = sns.boxplot(data=analysis4, x=&quot;Team&quot;, y=&quot;Runs&quot;, palette = palette)<br><br># Setting Title Name<br>ax.set_title(&#39;Distribution of runs scored by Teams in IPL 2023&#39;, \<br>fontsize =10, fontweight =&#39;bold&#39;)<br><br># Displaying Chart<br>plt.plot()</pre><p><strong>Results:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AGKArDVvv4425r2OSe8Epg.png" /></figure><p><strong>Observation:</strong></p><blockquote>The above graph shows the distribution of scores for teams in the IPL 2023 season.</blockquote><h3>Chart Type 5: Violin Plot</h3><p>A violin plot plays a similar role as a box and whisker plot. It shows the distribution of quantitative data across several levels of one or more categorical variables. Just like Histogram and Box Plot, Violin Plot gives a very good intuition about the spread/distribution of the data in a variable.(Example: Maximum, Mininum or Median job offers of students in XYZ College)</p><p><strong>Let’s try and answer few questions from IPL 2023 to understand Bar Chart</strong></p><h4>E. Is there a significant difference in the Runs Scored by Teams in IPL 2023 ?</h4><p><strong>Code Snippet:</strong></p><pre># Size of the Chart<br>fig = plt.figure(figsize=(20, 7))<br><br># Choosing color palette for teams<br>palette = [&#39;yellow&#39;, &#39;crimson&#39;, &#39;grey&#39;, &#39;indigo&#39;, &#39;cyan&#39;, &#39;blue&#39;, \<br>&#39;red&#39;,&#39;lightgreen&#39;,&#39;magenta&#39;,&#39;orange&#39;]<br><br># Plotting ViolinPlot<br>ax = sns.violinplot(data=analysis4, x=&quot;Team&quot;, y=&quot;Runs&quot;, palette = palette)<br><br># Setting Title Name<br>ax.set_title(&#39;Distribution of runs scored by Teams in IPL 2023&#39;, \<br>fontsize =10, fontweight =&#39;bold&#39;)<br><br># Displaying Chart<br>plt.plot()</pre><p><strong>Results:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KzRlKdz42eLYChVPJ2ElLw.png" /></figure><p><strong>Observation:</strong></p><blockquote>This output is similar to the results that we performed in the boxplot section and just like I mentioned previously an entirely different way of representing the results of distribution of runs scored by Teams in IPL 2023.</blockquote><h3><strong>Chart Type 6: Contour Plot or Density Plot</strong></h3><p>A density plot is a representation of the distribution of a numeric variable that uses a kernel density estimate to show the probability density function of the variable. In short Contour Plot or Density Plot gives an idea about concentration/density of points in a chart. (Example: Map of a Region showing the area where the concentration of trees is dense or intense as compared to the neighbouring or surrounding areas)</p><p><strong>Let’s try and answer few questions from IPL 2023 to understand Bar Chart</strong></p><h4>F. Who were more effective in IPL 2023 — Seamers or Spinners ?</h4><p><strong>Code Snippet:</strong></p><pre># Size of the Chart<br>fig = plt.figure(figsize=(10,6))<br><br># Plotting contour plot using kdeplot<br>ax = sns.kdeplot(data=analysis5, x=&quot;Economy_Rate&quot;, y=&quot;Wickets&quot;, \<br>hue=&quot;Seamer_or_Spinner&quot;)<br><br># Setting Title Name<br>ax.set_title(&#39;Density of Wickets vs Economy Rate for Bowlers in \<br>IPL 2023&#39;, fontsize =10, fontweight =&#39;bold&#39;)<br><br># Displaying Chart<br>plt.plot()</pre><p><strong>Results:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*3jjsLqk8cFoHvDNnqLEiHA.png" /></figure><p><strong>Observation:</strong></p><blockquote>Seems like most of the spinners had an economy rate in the range of 7 to 9 runs and they have picked up on an average 9 to 15 wickets. The seamers on the other hand had a slightly higher economy rate between 8 to 11 runs and they have picked up on an average 5 to 15 wickets. Also the wickets in India seem to offer a little more to the spinners when compared to fast bowlers, which could very well explain the density of the rings.</blockquote><h3><strong>Chart Type 7: HeatMap</strong></h3><p>A heat map is a 2-dimensional data visualization technique that represents the magnitude of individual values within a dataset as a colour. The variation in colour may be by hue or intensity. (Example: Correlation between different factors that could cause breast cancer)</p><p><strong>Let’s try and answer few questions from IPL 2023 to understand Bar Chart</strong></p><h4>G. Who were more effective in IPL 2023 — Seamers or Spinners ?</h4><p><strong>Code Snippet:</strong></p><pre># Size of the Chart<br>plt.figure(figsize=(6,6))<br><br># Choosing color palette<br>colormap = sns.color_palette(&quot;OrRd&quot;)<br><br># Plotting the HeatMap<br>ax = sns.heatmap(analysis10, annot = True, fmt=&quot;.0f&quot;, linewidths=.5, \<br>linecolor=&#39;white&#39;, cmap=colormap)<br><br># Displaying Extras at the top<br>ax.xaxis.tick_top()<br><br># Setting the orientation of Labels in Y Axis<br>plt.yticks(rotation=0)<br><br># Setting Title Name<br>ax.set_title(&#39;Extras Conceded by Teams in IPL 2023&#39;, fontsize =10, \<br>fontweight =&#39;bold&#39;)<br><br># Displaying Chart<br>plt.plot()</pre><p><strong>Results:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*k5mMzSUDGwCjEtlWBtCJiw.png" /></figure><p><strong>Facts:</strong></p><blockquote>Chennai Super Kings (CSK) and Mumbai Indians (MI) have conceded a lot of extras in the tournament as they had to start the tournament with an inexperienced bowling line up owing to too many injuries and retirements of their marquee players.</blockquote><blockquote>Royal Challengers Bangalore (RCB) also had an farily weak bowling line up and most of their go to bowlers were off the mark in the tournament leading to too many extras. Delhi Capitals (DC) also had a poor outing in the tournament when it came to bowling and most of their bowlers gave away too many extras throughout the tournament.</blockquote><h3><strong>Chart Type 8: Line Chart</strong></h3><p>A line chart or line plot, is a type of chart which displays information as a series of data points called ‘markers’ connected by straight line segments. This is the most basic type of chart used in finance, and it typically only depicts a security’s closing prices over time. Line charts can be used for any timeframe, but they most often use day-to-day price changes. (Example: Stock Price of Nike over a period of 30 days, Change in temperature of Bengaluru over a 24 hour period)</p><p><strong>Let’s try and answer few questions from IPL 2023 to understand Bar Chart</strong></p><h4>H. How have scores have been chased successfully in IPL 2023 ?</h4><p><strong>Code Snippet:</strong></p><pre># Defining runs and axes for plotting Charts<br>fig, axes = plt.subplots(2,1,figsize=(20,10))<br><br># Plotting Line Chart on Sub Plot 1<br>sns.lineplot(data = analysis4, x = &quot;Match_No&quot;, y = &quot;Runs&quot;, \<br>hue = &quot;Innings&quot;, markers = True, style = &quot;Innings&quot;, ax=axes[0])<br><br># Setting X Axis parameters for Sub Plot 1<br>axes[0].set_xlim(0, 75)<br><br># Setting Title names for Sub Plot 1<br>axes[0].set_title(&#39;Runs Scored by Teams in each match of IPL 2023&#39;, \<br>fontsize =10, fontweight =&#39;bold&#39;)<br><br># Defining Color Band<br>colour = [&#39;red&#39; if val &lt; 0 else &#39;green&#39; for val in analysis11.diffs]<br><br># Plotting Bar Chart on Sub Plot 2<br>sns.barplot(data = analysis11, x = &quot;Match_No&quot;, y = &quot;diffs&quot;, \<br>width=0.7, palette = colour, ax=axes[1])<br><br># Setting X Axis parameters for Sub Plot 2<br>axes[1].set_xlim(-1, 73)<br><br># Setting Title names for Sub Plot 2<br>axes[1].set_title(&#39;Margin of Win/Lose by Teams in each match of \<br>IPL 2023&#39;, fontsize =10, fontweight =&#39;bold&#39;)<br><br># Displaying values corresponding to each bar<br>axes[1].bar_label(axes[1].containers[0])<br><br># Displaying Chart<br>plt.show()</pre><p><strong>Results:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2fz5lr37f9cpdFG71kMcZw.png" /></figure><p><strong>Observation:</strong></p><blockquote>The first graph gives us an idea about how teams have scored and chased throughout the league phase of the tournament.</blockquote><blockquote>The second graph gives us an indication of margin of win or lose throughout the tournament and fits well with our analysis that we performed in the count plot section of this notebook, where we saw that 37 matches were won by teams batting first.</blockquote><h3>Chart Type 9: Scatter Plot</h3><p>A scatter plot is used to represent values for two different numeric variables. The position of each dot on the horizontal and vertical axis indicates values for an individual data point. Scatter plots are used to observe relationships between variables and could serve a could indicator of dependency of one numeric variable on the other variable. (Example: Price of a second hand car vs Age of the car)</p><p><strong>Let’s try and answer few questions from IPL 2023 to understand Bar Chart</strong></p><h4>I. How have scores have been chased successfully in IPL 2023 ?</h4><p><strong>Code Snippet:</strong></p><pre># Size of the Chart<br>fig = plt.figure(figsize=(10,6))<br><br># Plotting Scatter Plot<br>ax = sns.scatterplot(data=analysis8, x=&quot;Runs&quot;, y=&quot;Strike_Rate&quot;, \<br>hue=&quot;Batter_Type&quot;, style = &quot;Batter_Type&quot;)<br><br># Setting Title Name<br>ax.set_title(&#39;Runs Scored vs Strike Rate of Batsman in IPL 2023&#39;, \<br>fontsize =10, fontweight =&#39;bold&#39;)<br><br># Annotation for Faf Du Plessis<br>ax.text(730, 159, &quot;F DuPlessis&quot;, fontsize = 7, fontweight =&#39;bold&#39;, \<br>color = &quot;lightgreen&quot;, ha = &quot;center&quot;, va = &quot;center&quot;)<br><br># Annotation for Rashin Khan<br>ax.text(68, 237.3, &quot;R Khan&quot;, fontsize = 7, fontweight =&#39;bold&#39;, \<br>color = &quot;grey&quot;, ha = &quot;center&quot;, va = &quot;center&quot;)<br><br># Annotation for MS Dhoni<br>ax.text(140, 191, &quot;MS Dhoni&quot;, fontsize = 7, fontweight =&#39;bold&#39;, \<br>color = &quot;#ccad60&quot;, ha = &quot;center&quot;, va = &quot;center&quot;)<br><br># Annotation for Shubham Gill<br>ax.text(680, 147, &quot;S Gill&quot;, fontsize = 7, fontweight =&#39;bold&#39;, \<br>color = &quot;grey&quot;, ha = &quot;center&quot;, va = &quot;center&quot;)<br><br># Annotation for Devon Conway<br>ax.text(585, 144, &quot;D Conway&quot;, fontsize = 7, fontweight =&#39;bold&#39;, \<br>color = &quot;#ccad60&quot;, ha = &quot;center&quot;, va = &quot;center&quot;)<br><br># Annotation for Yashasvi Jaiswal<br>ax.text(625, 159, &quot;Y Jaiswal&quot;, fontsize = 7, fontweight =&#39;bold&#39;, \<br>color = &quot;magenta&quot;, ha = &quot;center&quot;, va = &quot;center&quot;)<br><br># Annotation for Virat Kohli<br>ax.text(639, 135, &quot;V Kohli&quot;, fontsize = 7, fontweight =&#39;bold&#39;, \<br>color = &quot;lightgreen&quot;, ha = &quot;center&quot;, va = &quot;center&quot;)<br><br># Displaying Chart<br>plt.plot()</pre><p><strong>Results:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*ZOejoAf0_rwh-WzGuENtQg.png" /></figure><p><strong>Observation:</strong></p><blockquote>The above scatterplot seems to suggest the fact that as the number of runs scored by the batsman increased slowly and gradually, the strike rate also improved considerably.</blockquote><p><strong>Facts:</strong></p><blockquote>Players who scored less runs and had a higher strike rate like Rashid Khan and MS Dhoni, were playing the role of a finisher and more often than not ended up scoring at a strike rate of more than 200 as they were looking for boundaries as the fag end of the innings. Faf Du Plessis, Shubhman Gill, Virat Kohli, Yashasvi Jaiswal and Devon Conway were the leading run scores in the league phase of the tournament with an impressive strike rate above 140+</blockquote><h3><strong>Chart Type 10: Pie Chart</strong></h3><p>A pie chart is a circular statistical graphic which is divided into slices to illustrate numerical proportion. In a pie chart, the arc length of each slice is proportional to the quantity it represents. (Example: marks obtained by students in a class)</p><p><strong>Let’s try and answer few questions from IPL 2023 to understand Bar Chart</strong></p><h4>J. Which Team has made the best choice of batsman as Impact Sub ?</h4><p><strong>Code Snippet:</strong></p><pre># Size of the Chart<br>plt.figure(figsize=(10,8))<br><br># Choosing Color Palette<br>palette_color = [&#39;yellow&#39;, &#39;crimson&#39;, &#39;grey&#39;, &#39;indigo&#39;,&#39;cyan&#39;, \<br>&#39;blue&#39;,&#39;red&#39;,&#39;lightgreen&#39;,&#39;magenta&#39;,&#39;orange&#39;]<br><br># Plotting Pie Chart<br>plt.pie(analysis12[&#39;Runs&#39;], labels=analysis12[&#39;Team_x&#39;], \<br>colors=palette_color, autopct=&#39;%1.1f%%&#39;)<br><br># Setting Title Name<br>plt.title(f&#39;Percentage of Runs Scored for Each Team by Impact Subs \<br>in IPL 2023&#39;, fontsize =10, fontweight =&#39;bold&#39;)<br><br># Displaying Chart<br>plt.show()</pre><p><strong>Results:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/1*_4UMBDbhTPeXej9-urtu6A.png" /></figure><p><strong>Observation:</strong></p><blockquote>The above chart gives an idea about the percentage of runs scored by the Impact Subs for their respective teams out of the total runs scored by the Impact Subs during the entire tournament.</blockquote><p><strong>Tip:</strong></p><blockquote>Even though the chart gives a good indication of the share of pie for respective teams but pie chart should be avoided when the number of categories/entries are more than 4 as they fail to paint a clear picture of the share in absolute terms. To enforce the understanding, I have performed the same analysis with the help of bar chart in the below diagram and it seems to perform much better when compared to pie chart.</blockquote><p><strong>With that I leave you guys with a favourite memory of mine from the IPL 2023 season. Hope you guys liked my analysis.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*AAoE0BvkoeJL6D7O3pQ51w.jpeg" /></figure><h3><strong>GitHub Link to the Solution Notebook</strong></h3><p><a href="https://github.com/SC30GSWNBA/Visual_Analysis_IPL_2023">GitHub - SC30GSWNBA/Visual_Analysis_IPL_2023: Build 10 beautiful charts on IPL 2023 datasets using Seaborn</a></p><p><strong><em>Please do visit my other articles in case you enjoyed reading this one.</em></strong></p><p><strong>Link: </strong><a href="https://medium.com/@sudip.1987roy/mercari-price-suggestion-using-ann-10d9bfe3524e">Mercari Price Suggestion using ANN — Step-by-Step Approach</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f46be24b5d2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mercari Price Suggestion using ANN — Step-by-Step Approach]]></title>
            <link>https://medium.com/@sudip.1987roy/mercari-price-suggestion-using-ann-10d9bfe3524e?source=rss-7e59c4ac97d------2</link>
            <guid isPermaLink="false">https://medium.com/p/10d9bfe3524e</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[count-vectorizer]]></category>
            <category><![CDATA[price-prediction]]></category>
            <category><![CDATA[text-mining]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <dc:creator><![CDATA[Sudip Roy]]></dc:creator>
            <pubDate>Sat, 15 Jul 2023 12:19:50 GMT</pubDate>
            <atom:updated>2023-07-15T16:31:55.511Z</atom:updated>
            <content:encoded><![CDATA[<p>Mercari, is an online platform that provides community-powered shopping app price suggestion. Determining the right price of products on online platforms is a real deal for sellers considering just how many products are sold online. Clothing has strong seasonal pricing trends and is heavily influenced by brand names, while electronics have fluctuating prices based on product specs.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ftenor.com%2Fembed%2F12611385&amp;display_name=Tenor&amp;url=https%3A%2F%2Ftenor.com%2Fview%2Fso-i-am-confusion-confused-calculating-complicated-gif-12611385&amp;image=https%3A%2F%2Fmedia.tenor.com%2FVgabnA4VyCcAAAAC%2Fso-i-am-confusion.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=tenor" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/7a316654b92d44277a537f5840e6fb37/href">https://medium.com/media/7a316654b92d44277a537f5840e6fb37/href</a></iframe><p><strong>Problem Statement:</strong> Build an algorithm that automatically suggests the right product prices based on user-inputted text descriptions of their products, including details like product category name, brand name, and item condition.</p><p><strong>About data:</strong></p><p>We will be using user-inputted text descriptions of their products, including details like product category name, brand name, and item condition. To know more about the dataset in details, please go through the below to explore the public data repository.</p><p><a href="https://www.kaggle.com/competitions/mercari-price-suggestion-challenge/data">Mercari Price Suggestion Challenge</a></p><p><strong>Table of Contents:</strong></p><p>1. Reading the Input File</p><p>2. Exploratory Data Analysis</p><p>3. Data Pre-processing</p><p>4. Creation of New Features</p><p>5. Visual Analysis</p><p>6. Probability Modelling</p><p>7. Word Frequency</p><p>8. Price Prediction using ANN</p><p>9. GitHub Link to the Solution Notebook</p><p>10. References</p><p>Let’s Begin with our Analysis…</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ftenor.com%2Fembed%2F9488740&amp;display_name=Tenor&amp;url=https%3A%2F%2Ftenor.com%2Fen-GB%2Fview%2Felijah-mikaelson-shall-we-the-originals-daniel-gillies-lets-go-gif-9488740&amp;image=https%3A%2F%2Fmedia.tenor.com%2F-5Q5YG_kSQMAAAAC%2Felijah-mikaelson-shall-we.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=tenor" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/bf44edc5849f7d36182a0ea8c65c97d1/href">https://medium.com/media/bf44edc5849f7d36182a0ea8c65c97d1/href</a></iframe><p><strong>1. Reading the Input File</strong></p><p>The file format provided for this challenge is in tab separated format, which could easily be read using the read_csv function of the pandas library</p><p><strong>Python Code:</strong></p><pre>mercari_df = pd.read_csv(&#39;/content/train.tsv&#39;, sep = &#39;\t&#39;)</pre><p><strong>2. Exploratory Data Analysis</strong></p><p>Once the dataset has been loaded into Google Collaboratory, we need to perform basic EDA like check for the amount of rows or columns, null values(if any), amount of duplicate rows etc.</p><pre># Inspecting first few rows of the dataFrame<br><br>mercari_df.head(10)<br><br># Basic information about the dataset<br><br>mercari_df.info()<br><br># Checking for duplicate rows<br><br>mercari_df.duplicated().any()<br><br># Deleting duplicate rows from the dataset<br><br>mercari_df.drop_duplicates(keep=&quot;first&quot;, inplace = True)</pre><p><strong>3. Data Standardization</strong></p><p>After we are done with inspecting the dataframe and removed all the duplicate rows in the dataset, we need to impute the missing values in those columns (if any).</p><pre># Imputing Missing Values<br><br>mercari_df.category_name.fillna(&#39;NotAvailable&#39;, inplace=True)<br>mercari_df.brand_name.fillna(&#39;NotAvailable&#39;, inplace=True)<br>mercari_df.item_description.fillna(&#39;NotAvailable&#39;, inplace=True)</pre><p>After we are done with missing value imputation we need to address the duplicate or redundant entries in any column of the dataset. In case there are duplicate or redundant entries, we need to standardize those columns before we perform any sort of modeling on the dataset as it might impact the model performance during price prediction. Code snippet provided below is one of many such instances where I found duplicate or similar entries in text columns of the dataframe.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Ftenor.com%2Fembed%2F8314801&amp;display_name=Tenor&amp;url=https%3A%2F%2Ftenor.com%2Fen-GB%2Fview%2Fmowing-the-lawn-gif-8314801&amp;image=https%3A%2F%2Fmedia.tenor.com%2FTx0f4xWS9BwAAAAC%2Fmowing-the-lawn.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=tenor" width="600" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/bfcdd4ae1a5e98af5453f04fae1862dc/href">https://medium.com/media/bfcdd4ae1a5e98af5453f04fae1862dc/href</a></iframe><pre>mercari_df[&#39;brand_name&#39;].replace(to_replace=[&#39;georgio armani&#39;, <br>&#39;armani jeans&#39;, &#39;giorgio armani&#39;, &#39;a|x armani exchange&#39;, <br>&#39;a/x armani exchange&#39;, &#39;armani exchange&#39;, &#39;armani collezioni&#39;, &#39;armani&#39;, <br>&#39;emporio armani&#39;], value=&quot;armani&quot;, inplace = True)</pre><p><strong>4. Creation of New Features</strong></p><p>During this whole process of EDA, we also need to check if we can create new features out of the existing columns which might be helpful from the perspective of price prediction.</p><pre># Creating New Features<br><br>mercari_df = mercari_df.join(mercari_df[&#39;category_name&#39;].str.\<br>split(&#39;/&#39;, expand=True).rename(columns={0:&#39;sub_category1&#39;, \<br>1:&#39;sub_category2&#39;, 2:&#39;sub_category3&#39;}))<br><br>mercari_df[&#39;sub_category3&#39;] = mercari_df[mercari_df.columns[-3:]].\<br>apply(lambda x: &#39;,&#39;.join(x.dropna().astype(str)),axis=1)<br><br>mercari_df.drop(columns = [3,4], inplace = True)<br><br>mercari_df.head()</pre><p><strong>5. Visual Analysis</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FxUPGcEw56dJj3fJFJe%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2FBaywatchMovie-funny-comedy-xUPGcEw56dJj3fJFJe&amp;image=https%3A%2F%2Fmedia4.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExdm8xYnpvYmgwdzB1NmNpa3M0YWc2NnpqYzN4OTZ1ZzdxYzZydDFuZiZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw%2FxUPGcEw56dJj3fJFJe%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="435" frameborder="0" scrolling="no"><a href="https://medium.com/media/a224b7a9a6518fb0a57eb7a3c9605c60/href">https://medium.com/media/a224b7a9a6518fb0a57eb7a3c9605c60/href</a></iframe><p>Visual Analysis provides a clear picture of the dataset in terms of what lies within each and every attribute. Results of visual analysis helps us address any anomalies or hidden patterns in the dataset which might otherwise go unnoticed. Before we begin with any sort of modeling on the dataset at hand, we must address any issues (if any) and understand what lies within the data. In my analysis, I have performed visual analysis on various combinations of the attributes. For the sake of this blog, I will provide only a handful of this code snippets. Full code snippets will be provided at the end of the documentation.</p><pre># Univariate Analysis - Price Points<br><br>plt.figure(figsize=(12,6))<br><br>ax = sns.histplot(data=mercari_df[mercari_df[&#39;price&#39;] &lt;= 150], \<br>x=&quot;price&quot;, kde=True)<br><br>ax.bar_label(ax.containers[0])<br>plt.title(&#39;Distribution of Price&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l4Y8QZNlXRJBknyu8eeqtA.png" /><figcaption>Univariate Analysis</figcaption></figure><pre># Bivariate Analysis <br># Price Analysis of Top 10 Products with Price &gt; $100<br><br>fig,axes=plt.subplots(2,1,figsize=(24,10))<br><br>price_greater_than_t = mercari_df[mercari_df[&#39;price&#39;] &gt; 100]<br><br>sns.countplot(y=&#39;name&#39;, data=price_greater_than_t, \<br>order = price_greater_than_t[&#39;name&#39;].value_counts().index[:10], ax=axes[0])<br><br>axes[0].bar_label(axes[0].containers[0])<br>axes[0].set_title(&#39;Count of Top 10 Products by Name having Price &gt; $100&#39;)<br><br>products = price_greater_than_t[&quot;name&quot;].value_counts().index[:10]<br><br>df_products1 = price_greater_than_t[price_greater_than_t[&#39;name&#39;].\<br>isin(products)].groupby([&#39;name&#39;]).agg({&#39;price&#39;:&#39;mean&#39;}).reset_index()<br><br>df_products1[&#39;type&#39;] = &#39;mean&#39;<br><br>df_products2 = price_greater_than_t[price_greater_than_t[&#39;name&#39;].\<br>isin(products)].groupby([&#39;name&#39;]).agg({&#39;price&#39;:&#39;median&#39;}).reset_index()<br><br>df_products2[&#39;type&#39;] = &#39;median&#39;<br>df_products = pd.concat([df_products1, df_products2]).\<br>reset_index(drop = True)<br><br>sns.barplot(data=df_products, x=&quot;name&quot;, y=&quot;price&quot;, \<br>order = price_greater_than_t[&#39;name&#39;].value_counts().index[:10], hue=&quot;type&quot;)<br><br>axes[1].bar_label(axes[1].containers[0])<br>axes[1].bar_label(axes[1].containers[1])<br>axes[1].set_title(&#39;Mean-Median Price Analysis of Top 10 \<br>Products having Price &gt; $100&#39;)<br><br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3aArAWXV1T67DcOtnWp_fQ.png" /><figcaption>Bivariate Analysis</figcaption></figure><p><strong>6. Probability Modelling</strong></p><p>Once we are done with visual analysis, we could start with probability based modeling to determine normality, similarity/dissimilarity of products, evenness of distribution of products across various categories to check the dependency of attributes on each other. To perform probability based modeling we need to define our hypothesis to validate results accordingly. I performed once such analysis on condition of products to check if price is similar or different based on condition of products.</p><p><strong><em>Let us first set-up the hypothesis framework</em></strong></p><p><strong><em>Null Hypothesis:</em></strong> Price has no effect on the condition id of the product</p><p><strong><em>Alternate Hypothesis:</em></strong> Price has effect on the condition id of the product</p><p><strong><em>Significance Level(α):</em></strong> 0.05 or 5%</p><p><strong><em>Test Statistic:</em></strong> One Way ANOVA</p><p>ANOVA Tests are massively impacted by outliers, Let’s remove these outliers with the help of IQR Method and then run these tests</p><pre># Outlier Treatment for condition1<br><br>condition1 = mercari_df[mercari_df.item_condition_id == 1]<br><br>q1 = condition1[&#39;price&#39;].quantile(0.25)<br>q3 = condition1[&#39;price&#39;].quantile(0.75)<br>iqr = q3 - q1<br><br>condition1 = condition1[(condition1[&#39;price&#39;]&gt;(q1 - 1.5*iqr) ) &amp; \<br>(condition1[&#39;price&#39;]&lt;(q3 + 1.5*iqr))]<br><br>condition1 = condition1[&#39;price&#39;]<br><br># Outlier Treatment for condition2<br><br>condition2 = mercari_df[mercari_df.item_condition_id == 2]<br><br>q1 = condition2[&#39;price&#39;].quantile(0.25)<br>q3 = condition2[&#39;price&#39;].quantile(0.75)<br>iqr = q3 - q1<br><br>condition2 = condition2[(condition2[&#39;price&#39;]&gt;(q1 - 1.5*iqr) ) &amp; \<br>(condition2[&#39;price&#39;]&lt;(q3 + 1.5*iqr))]<br><br>condition2 = condition2[&#39;price&#39;]<br><br># Outlier Treatment for condition3<br><br>condition3 = mercari_df[mercari_df.item_condition_id == 3]<br><br>q1 = condition3[&#39;price&#39;].quantile(0.25)<br>q3 = condition3[&#39;price&#39;].quantile(0.75)<br>iqr = q3 - q1<br><br>condition3 = condition3[(condition3[&#39;price&#39;]&gt;(q1 - 1.5*iqr) ) &amp; \<br>(condition3[&#39;price&#39;]&lt;(q3 + 1.5*iqr))]<br><br>condition3 = condition3[&#39;price&#39;]<br><br># Outlier Treatment for condition4<br><br>condition4 = mercari_df[mercari_df.item_condition_id == 4]<br><br>q1 = condition4[&#39;price&#39;].quantile(0.25)<br>q3 = condition4[&#39;price&#39;].quantile(0.75)<br>iqr = q3 - q1<br><br>condition4 = condition4[(condition4[&#39;price&#39;]&gt;(q1 - 1.5*iqr) ) &amp; \<br>(condition4[&#39;price&#39;]&lt;(q3 + 1.5*iqr))]<br><br>condition4 = condition4[&#39;price&#39;]<br><br># Outlier Treatment for condition4<br><br>condition5 = mercari_df[mercari_df.item_condition_id == 5]<br><br>q1 = condition5[&#39;price&#39;].quantile(0.25)<br>q3 = condition5[&#39;price&#39;].quantile(0.75)<br>iqr = q3 - q1<br><br>condition5 = condition5[(condition5[&#39;price&#39;]&gt;(q1 - 1.5*iqr) ) &amp; \<br>(condition5[&#39;price&#39;]&lt;(q3 + 1.5*iqr))]<br><br>condition5 = condition5[&#39;price&#39;]</pre><p><strong>Calculation of Standard Deviation</strong></p><ul><li>Note that the standard deviation of the population is not known.</li><li>Hence, let us consider a sample size of 1000 across all categories.</li></ul><pre># Creating Samples<br><br>sample1 = condition1.sample(1000)<br>sample2 = condition2.sample(1000)<br>sample3 = condition3.sample(1000)<br>sample4 = condition4.sample(1000)<br>sample5 = condition5.sample(1000)</pre><p><strong>Let us now calculate,</strong></p><p>The degree of freedom(<strong>dfn</strong>) for different conditions, and</p><p>The degree of freedom(<strong>dfd</strong>) for all data points within different conditions.</p><pre>dfn = 5<br>dfd = len(sample1) - 1 + len(sample1) - 1 + len(sample1) - 1 + \<br>len(sample1) - 1 + len(sample1) - 1</pre><p><strong>Critical Region at 95% confidence interval</strong></p><pre>from scipy.stats import f<br><br>critical_region = f.ppf(0.95, dfn=dfn, dfd=dfd)<br>print(f&#39;The critical region for the hypothesis at 95% confidence \<br>interval: {critical_region}&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*GRSnVwaYYH46Ed34cRZq-w.png" /></figure><p><strong>Let us now calculate the f-statistic based on data points</strong></p><pre>from scipy.stats import f_oneway<br><br>f_oneway(sample1, sample2, sample3, sample4, sample5)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*aUWuR9dCTEhapBM9m_jM0g.png" /></figure><p>Since, pvalue is lesser than Significance Level(α) i.e. 0.05, we reject the null hypothesis that — Price has no effect on the condition id of the product.</p><p>This implies that at 95% confidence level, the mean price of products is statistically different for condition id of the product.</p><p>There is no limit to the amount of probability based modeling that we can do on the dataset but more the amount of analysis done, more better informed we are in terms of the dataset understanding.</p><p><strong>7. Word Frequency</strong></p><p>Since, the whole price prediction is based on text mining, it would be good to visualize the frequently occurring words in the text columns of the dataset.</p><p>To figure out the most frequently occurring ‘words’ or ‘bag of words’ in the text based attributes in the dataset, we need to perform text cleaning and then vectorize them with the help of Count Vectorizer.</p><pre># Importing Stopwords from nltk library<br><br>stop_words = set(stopwords.words(&#39;english&#39;))<br>mercari_df[&quot;item_description&quot;] = mercari_df[&quot;item_description&quot;].\<br>replace({&quot;No description yet&quot;: &quot;NoDescription&quot;})</pre><pre># Cleaning the textual attributes<br><br>for attribute in [&#39;item_description&#39;,&#39;name&#39;,&#39;sub_category1&#39;,&#39;\<br>sub_category2&#39;,&#39;sub_category3&#39;,&#39;brand_name&#39;]:<br>  cleaned_words = []<br>  for sentence in mercari_df[attribute]:<br>    sentence = sentence.lower()<br>    sentence = re.sub(&#39;[^A-Za-z0-9]+&#39;, &#39; &#39;, sentence)<br>    sentence = &#39; &#39;.join(word for word in sentence.split() if word not in \<br>    stop_words)<br><br>    cleaned_words.append(sentence.strip())<br><br>mercari_df[attribute] = cleaned_words</pre><pre># Frequently Occuring Words or Bag of Words in Item Description Column<br><br>vectorizer1 = CountVectorizer(min_df=10, max_features=20000)<br>vectorized_attr1 = vectorizer1.fit_transform\<br>(mercari_df[&quot;item_description&quot;])<br><br>df1 = pd.DataFrame()<br>df1[&quot;words&quot;] = vectorizer1.get_feature_names_out()<br>df1[&quot;frequency&quot;] = np.asarray(vectorized_attr1.sum(axis=0))[0]<br><br>x1 = df1.sort_values(&quot;frequency&quot;, axis=0, ascending=False).head(30)\<br>[&quot;words&quot;]<br><br>y1 = df1.sort_values(&quot;frequency&quot;, axis=0, ascending=False).\<br>head(30)[&quot;frequency&quot;]<br><br>vectorizer2 = CountVectorizer(min_df=30, ngram_range=(3, 3), \<br>max_features=10000)<br>vectorized_attr2 = vectorizer2.fit_transform\<br>(mercari_df[&quot;item_description&quot;])<br><br>df2 = pd.DataFrame()<br>df2[&quot;words&quot;] = vectorizer2.get_feature_names_out()<br>df2[&quot;frequency&quot;] = np.asarray(vectorized_attr2.sum(axis=0))[0]<br><br>x2 = df2.sort_values(&quot;frequency&quot;, axis=0, ascending=False).head(30)\<br>[&quot;words&quot;]<br><br>y2 = df2.sort_values(&quot;frequency&quot;, axis=0, ascending=False).head(30)\<br>[&quot;frequency&quot;]<br><br>fig, ax = plt.subplots(2,1,figsize=(30, 12))<br><br>ax[0].bar(x1, y1, color = &#39;royalblue&#39;)<br>ax[0].bar_label(ax[0].containers[0])<br>ax[0].set_title(&#39;Top 30 Frequently Occuring words in the Item \<br>Description column of the dataset&#39;)<br>ax[0].set_xlabel(&#39;Words&#39;)<br>ax[0].set_ylabel(&#39;Frequency&#39;)<br><br>ax[1].bar(x2, y2, color = &#39;royalblue&#39;)<br>ax[1].bar_label(ax[1].containers[0])<br>ax[1].set_title(&#39;Top 30 Frequently Occuring Set of Words in the \<br>Item Description column of the dataset&#39;)<br>ax[1].set_xlabel(&#39;Bag of Words&#39;)<br>ax[1].set_ylabel(&#39;Frequency&#39;)<br><br>plt.xticks(rotation = 45)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CPTOHf2yfJD2Jm5tj_HZsw.png" /><figcaption>Word Frequency Count</figcaption></figure><p><strong>8. Price Prediction using ANN</strong></p><p>For Price Prediction of products, we will consider a fraction(10%) of the dataset and then run our models to avoid RAM issues while executing on Google Collaboratory.</p><pre># Considering fraction of the dataset for price prediction<br><br>mercari_df = mercari_df.sample(frac = 0.1)</pre><p><strong>Creating Input and Output Columns</strong></p><pre># Creating Input and Output Columns<br><br>X = mercari_df.drop(columns = [&#39;price&#39;])<br>y = np.log1p(mercari_df[&#39;price&#39;])</pre><p><strong>Test Train Split for Modeling</strong></p><pre># Creating Test-Train Split<br><br>X_train, X_test, y_train, y_test = train_test_split(X, y, \<br>test_size=0.25, random_state=42)</pre><p><strong>Vectorizing Input Columns</strong></p><pre># Function to tranform and vectorize training data<br><br>def vectorize_train_test_data(df_train, df_test):<br><br>  binarized_attrs1 = []<br>  binarized_attrs2 = []<br>  vectorized_attrs1 = []<br>  vectorized_attrs2 = []<br><br>  for attr in [&#39;item_condition_id&#39;, &#39;category_name&#39;, &#39;brand_name&#39;,\<br>&#39;shipping&#39;, &#39;sub_category1&#39;, &#39;sub_category2&#39;, &#39;sub_category3&#39;]:<br><br>      binary_obj = LabelBinarizer(sparse_output=True)<br>      binarized_attr1 = binary_obj.fit_transform(df_train[attr])<br>      binarized_attrs1.append(binarized_attr1)<br>      binarized_attr2 = binary_obj.transform(df_test[attr])<br>      binarized_attrs2.append(binarized_attr2)<br><br><br><br>  for attr in [&#39;name&#39;, &#39;item_description&#39;]:<br><br>    vectorizer = CountVectorizer(min_df=10, max_features=30000)<br>    vectorized_attr1 = vectorizer.fit_transform(df_train[attr])<br>    vectorized_attrs1.append(vectorized_attr1)<br>    vectorized_attr2 = vectorizer.transform(df_test[attr])<br>    vectorized_attrs2.append(vectorized_attr2)<br><br><br>    if attr == &#39;name&#39;:<br>      vectorizer = CountVectorizer(min_df=20, ngram_range=(2, 2), \<br>max_features=20000)<br>      vectorized_attr1 = vectorizer.fit_transform(df_train[attr])<br>      vectorized_attrs1.append(vectorized_attr1)<br>      vectorized_attr2 = vectorizer.transform(df_test[attr])<br>      vectorized_attrs2.append(vectorized_attr2)<br><br>    else:<br><br>      vectorizer = CountVectorizer(min_df=30, ngram_range=(3, 3), \<br>max_features=10000)<br>      vectorized_attr1 = vectorizer.fit_transform(df_train[attr])<br>      vectorized_attrs1.append(vectorized_attr1)<br>      vectorized_attr2 = vectorizer.transform(df_test[attr])<br>      vectorized_attrs2.append(vectorized_attr2)<br><br><br>  X_train = hstack((binarized_attrs1[0], binarized_attrs1[1], \<br>binarized_attrs1[2], binarized_attrs1[3], binarized_attrs1[4], \<br>binarized_attrs1[5], binarized_attrs1[6], vectorized_attrs1[0], \<br>vectorized_attrs1[1], vectorized_attrs1[2], vectorized_attrs1[3])).tocsr()<br><br>  X_test = hstack((binarized_attrs2[0], binarized_attrs2[1], \<br>binarized_attrs2[2], binarized_attrs2[3], binarized_attrs2[4], \<br>binarized_attrs2[5], binarized_attrs2[6], vectorized_attrs2[0], \<br>vectorized_attrs2[1], vectorized_attrs2[2], vectorized_attrs2[3])).tocsr()<br><br><br>  return X_train, X_test<br><br>X_train, X_test = vectorize_train_test_data(X_train, X_test)</pre><p><strong>Designing the Neural Network Layer</strong></p><pre># Designing the Neural Network Layer<br><br>model = Sequential([<br>Dense(144, kernel_initializer=&#39;normal&#39;, activation=&#39;relu&#39;),<br>Dropout(0.2),<br>Dense(360, kernel_initializer=&#39;normal&#39;, activation=&#39;relu&#39;),<br>Dropout(0.2),<br>Dense(240, kernel_initializer=&#39;normal&#39;, activation=&#39;relu&#39;),<br>Dense(1, kernel_initializer=&#39;normal&#39;, activation=&#39;linear&#39;)])</pre><p><strong>Defining Error Metric</strong></p><pre># Defining Error Metric<br><br>msle = MeanSquaredLogarithmicError()<br>model.compile(loss=msle, optimizer=Adam(learning_rate=0.01), \<br>metrics=[msle])</pre><p><strong>Training the Model</strong></p><pre># Fitting the Training Model<br><br>training_model = model.fit(<br>X_train.toarray(),<br>y_train.values,<br>epochs=100,<br>batch_size=64,<br>validation_split=0.25)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KKy9hajPkoDoxWFuLlq53w.png" /><figcaption>Training Epochs</figcaption></figure><p><strong>Error Comparison — Training vs Validation</strong></p><pre># Function to plot training vs validation loss<br><br>def training_vs_validation_plot(training_model):<br>  plt.plot(training_model.history[&#39;mean_squared_logarithmic_error&#39;], \<br>color = &#39;orange&#39;)<br>  plt.plot(training_model.history[&#39;val_mean_squared_logarithmic_error&#39;], \<br>color = &#39;red&#39;)<br>  plt.xlabel(&quot;Number of Epochs&quot;)<br>  plt.ylabel(&#39;Mean Square Logarithmic Error&#39;)<br>  plt.legend([&#39;Training&#39;, &#39;Validation&#39;])<br>  plt.title(&#39;Mean Square Logarithmic Error - Training vs Validation&#39;)<br>  plt.show()<br><br>training_vs_validation_plot(training_model)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/1*xRMQqZiTCNVZBhAGVBNi8A.png" /><figcaption>Training vs Validation Error</figcaption></figure><p><strong>Prediction Results</strong></p><pre># Prediction Results<br><br>prediction_df = pd.DataFrame()<br>prediction_df[&#39;Actual_Price&#39;] = np.expm1(np.array(y_test))<br>prediction_df[&#39;Prediction_Price&#39;] = np.expm1(model.predict(X_test))<br>prediction_df.head(10)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*gJES45UjitlP1sEcyssuBw.png" /><figcaption>Prediction Results</figcaption></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FkEbGNrbKsXfytpU8An%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FkEbGNrbKsXfytpU8An%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FkEbGNrbKsXfytpU8An%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/82efe392ea65f2a59193f0462f2a7d5c/href">https://medium.com/media/82efe392ea65f2a59193f0462f2a7d5c/href</a></iframe><p><strong>9. GitHub Link to the Solution Notebook</strong></p><p><a href="https://github.com/SC30GSWNBA/Mercari_Price_Suggestion_Challenge/blob/main/Mercari_Price_Suggestion_Case_Study.ipynb">Mercari_Price_Suggestion_Challenge/Mercari_Price_Suggestion_Case_Study.ipynb at main · SC30GSWNBA/Mercari_Price_Suggestion_Challenge</a></p><p><strong>10. References</strong></p><ol><li>Basics of CountVectorizer : <a href="https://towardsdatascience.com/basics-of-countvectorizer-e26677900f9c">https://towardsdatascience.com/basics-of-countvectorizer-e26677900f9c</a></li><li>Label Binarizer: <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelBinarizer.html">https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelBinarizer.html</a></li><li>Regression Analysis using ANN in Tensorflow: <a href="https://www.analyticsvidhya.com/blog/2021/08/a-walk-through-of-regression-analysis-using-artificial-neural-networks-in-tensorflow/">https://www.analyticsvidhya.com/blog/2021/08/a-walk-through-of-regression-analysis-using-artificial-neural-networks-in-tensorflow/</a></li></ol><p>Thank you for reading folks. Please like and share if you like my solution notebook.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=10d9bfe3524e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>