<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Akshay Sharma on Medium]]></title>
        <description><![CDATA[Stories by Akshay Sharma on Medium]]></description>
        <link>https://medium.com/@akshay.sharma8426?source=rss-c1c8910ec894------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*oeq5XkoIsJJkreuj44-R9A.jpeg</url>
            <title>Stories by Akshay Sharma on Medium</title>
            <link>https://medium.com/@akshay.sharma8426?source=rss-c1c8910ec894------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 13:14:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@akshay.sharma8426/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Evaluation Metrics for Common ML Problems]]></title>
            <link>https://medium.com/@akshay.sharma8426/evaluation-metrics-for-common-ml-problems-8010365e3d7?source=rss-c1c8910ec894------2</link>
            <guid isPermaLink="false">https://medium.com/p/8010365e3d7</guid>
            <category><![CDATA[evaluation]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[learning]]></category>
            <dc:creator><![CDATA[Akshay Sharma]]></dc:creator>
            <pubDate>Tue, 08 Oct 2019 18:23:42 GMT</pubDate>
            <atom:updated>2019-10-08T18:23:42.892Z</atom:updated>
            <content:encoded><![CDATA[<h3>Accuracy</h3><p>This is the simplest measurement as it measures how often a classifier makes a correct prediction. <strong><em>It is the ratio between the number of correct predictions and the total number of predictions</em></strong>. One drawback of the accuracy metric is that it does not make a distinction between your various classes. In some cases, it may be more imperative that you have the absolute minimum number of false-negatives or false-positives and in those cases, accuracy will not be giving you any information of whether your classifier is doing what you hope.</p><h3>Confusion Matrix</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/356/0*bDhs-tNELMdbIV8x.png" /></figure><p>This neatly shows you the results of your classifier. It lists out the total number of true positives, true negatives, false positives, and false negatives. This gives you a clearer picture of how your classifier is actually performing and if you have a preference for true negatives or true positives then you are given enough information in regards to that.</p><p>Within a confusion matrix, you can take a look at sub-accuracy. You can see how accurate your positive class predictions are and how accurate your negative class predictions are.</p><h3>Log-Loss</h3><p>This metric can be used if your classifier outputs a numeric probability instead of an outright class label. The probability would be the likelihood of the sample being in class 1 or 2. Probabilities provide a greater context of how well your classifier is performing. For example, if a random sample has a class label of ‘2’ but your classifier thinks its ‘1’ and gives it a 51% chance of being class ‘1’, then in actuality, your classifier is really close to making the right decision. If you simply looked at the confusion matrix, you would not get this sort of insight of how close your classifier was in making the right decision. Log-loss is a measure to gauge the noise that is present in your classifier rather than providing you the true labels. By minimizing log-loss, you maximize the accuracy of the classifier.</p><h3>AUC</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*vn9iQPjITRuWdGkK.png" /><figcaption>ROC Curve</figcaption></figure><p>Also known as area under the curve and is commonly associated with the ROC curve (Receiver Operating Characteristic curve). The ROC curve shows the sensitivity of your classifier by plotting the rate of true positives to the rate of false positives. This graph shows you how many more true positives you can obtain as you allow your classifier to make more false positive predictions. Since the ROC curve essentially shows you your classifiers behavior as you go from the range of predicting everything as negative to everything as positive, the AUC summarizes the graph into a single number allowing for easy comparison of different models. The idea is to maximize the AUC value as it indicates the true positive rate is at its highest value.</p><h3>Precision-Recall</h3><p>Precision and Recall are two different metrics that are often used together. Precision looks to answer the question of “Out of all of the items the classifier believes to be in the positive class, how many are truly relevant?” Recall looks to answer the question of “Out of all the Positive items in your data set, how many does your classifier pick up on”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/0*5Ty7UYDXeeA32DsT.png" /></figure><p>By taking both these metrics into consideration, the F1 score is born. The F1 score is a measure of a classifier’s accuracy. It can be considered to be the harmonic mean of the different values between precision and recall. A harmonic mean tends to favor the lower value between the two elements. A F1 value of 1 is perfect whereas a value of 0 is the worst.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/1*VtU4Rl_Su-JRNja3zs7zaA.png" /></figure><h3>Root-Mean-Square Error</h3><p>RMSE is one of the most commonly used metric for regression models. This value measures the square root of the average squared distance between the actual value and the predicted value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/0*W3pnzs5Vsaoit3bq.jpg" /></figure><p>Some issues with the RMSE is that it is very sensitive to outliers in your data. This is because the difference between the predicted and actual value is squared, so if there is a large difference, it has a large impact. This may be a benefit depending on your situation. However, if you do not want outliers to have a large impact on your metric, then you can use the Mean Absolute Error.</p><h3>Mean Absolute Error</h3><p>The MAE measures the average magnitude of the errors in a set of predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/315/0*p4yPlTc5qSlD2N_M.gif" /></figure><p>Since it provides the absolute difference between predicted and actual values, it does not amplify large differences. This is why that as the sample size increases, MAE tends to be a lower value than RMSE.</p><h3>Conclusion</h3><p>As you can see, there are many metrics you can rely on when evaluating your classification or regression models. However, when deciding on which metrics you want to use, you need to have a thorough understanding of your data. This is because each metric has its own pros and cons and you may be misrepresenting the reliability of your model if you end up choosing a metric that does not accurately reflect the performance of your model.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8010365e3d7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Steps Every ML Project Goes Through Part 2/2]]></title>
            <link>https://medium.com/@akshay.sharma8426/the-steps-every-ml-project-goes-through-part-2-2-abec5e9e0789?source=rss-c1c8910ec894------2</link>
            <guid isPermaLink="false">https://medium.com/p/abec5e9e0789</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Akshay Sharma]]></dc:creator>
            <pubDate>Tue, 01 Oct 2019 16:46:31 GMT</pubDate>
            <atom:updated>2019-10-01T16:46:31.438Z</atom:updated>
            <content:encoded><![CDATA[<p>This blog is a continuation of the similarly titled one from last week. In this edition, I will be discussing the steps a Machine Learning project goes through after having completed the steps involving the data. If you did not get a chance to read the first post, please see the link below before continuing.</p><p><a href="https://medium.com/@akshay.sharma8426/the-steps-every-ml-project-goes-through-part-1-53ca9492652c">The Steps Every ML Project Goes Through Part 1</a></p><h3>Selecting and Training a Model</h3><p>After having done the hard work in the previous steps, this step is going to be relatively simple in comparison. With the data all ready to be fitted into various models, all you really have to do is import the corresponding model from SKLearn and fit your training data into the model.</p><p>If you wanted to use linear regression, then that can be done by the following:</p><blockquote>from sklearn.linear_model import LinearRegression</blockquote><blockquote>your_variable = LinearRegression()</blockquote><blockquote>your_variable.fit(input_features, labels)</blockquote><p>The process is generally the same regardless of which model you end up using.</p><p>However, the first iteration of a model is almost never the best version. You have to evaluate the performance of your model in order to improve it. If your errors are too large, then it is likely that your current iteration of the model is underfitting the training data. Underfitting means that the current set of features do not provide enough information for the model to make accurate predictions. In order to tackle this underfitting problem, you can change up your model, create new features, or reduce the constraints on your model.</p><p>If you run a new model, like a decision tree, then you can use the code similar to above but instead use: <strong><em>from sklearn.tree import DecisionTreeRegressor</em></strong></p><p>A method you should employ to evaluate your model is to use cross-validation. Cross-validation breaks up your training data into n-subsets. It then trains and evaluates your model n-times using the other subsets. To read more about employing cross-validation in your project, please check out the documentation on it below:</p><p><a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html">sklearn.model_selection.cross_val_score</a></p><p>The goal of this step is to use many ML models and see which ones perform well on your data at a base level. From here, you take it to the next step and begin fine-tuning your models to get the best performance.</p><h3>Tuning your Models</h3><p>One of the most common ways to fine-tune your models is to do a grid-search. You can use SKlearn’s GridSearchCV function to easily implement it into your project. What GridSearch does is that it helps you find the optimal combination of hyper parameter values for your model using cross validation. You merely put in your instantiated model, your parameters you want to test out in the form of a list of dictionaries, the number of folds for your cross validation, and the metric you want to optimize for. Something to note is that this can take a very long time. As such, you won’t want to test every value possible, but rather take into account the hardware you are using and adjust accordingly. I would suggest starting conservatively and observing how long it takes and then adding more parameters.</p><p>After you have run the grid search, you can determine your best combination of hyper parameters through your variable for instantiating <strong><em>GridSearchCV.best_params_.</em></strong></p><p>An alternative to grid search would be randomized search. Like I mentioned earlier, grid search can take a very long time to run as it goes through each possible combination of hyper parameters. By calling the GridSearchCV class, you can evaluate a n-number of random combinations for each hyperparameter. This method allows you to control your computing power as it is directly correlated with the number of random combinations you have selected to run.</p><p>At this stage, you should be able to understand the importance of all your attributes. As such, you should take a look at the ones with extremely low values and consider dropping them to simplify your model and get rid of any potential source of noise.</p><p>Finally, you should now test your finely tuned model on your test set. It is normal for the performance to drop when analyzing the test set. However, it is important not to further tweak your model to optimize for this test set. Because you will just run into more problems when trying to predict another set of data that the model has not been trained on. If you find your model is performing terribly on your test set, then you need to work with you training set and continue tuning your models to help it be more predictive. This usually means simplifying your model, but it depends on each individual situation.</p><h3>Presenting your Solution</h3><p>This step is merely to provide a high level overview of all of the previous steps. If you are completing a personal project, then on your Github Readme, you need to indicate what you learned during the course of the project, which models worked well with your data and which ones did not, and any assumptions you may have made during the course of your project. The point of this is to create a nice summary using your EDA and visualizations from earlier to help another Data Scientist arrive at your conclusion too.</p><h3>Launching and Maintaining your System</h3><p>Once you launch your model, you need to be wary of the degradation of your model. The longer your model is in deployment, the more likely it is the data you are predicting starts to change. This will weaken the performance of your model and it will need to be retrained. It is essential that once you deploy your model that you write a small function that monitors the performance of your model and alerts you if the performance dips below a certain threshold. This will let you know that it is time to retrain your model on new data.</p><h3>Conclusion</h3><p>I hope these past 2 blogs have given you insight on what it takes to conduct a proper project within Data Science. As you can see, the bulk of the work is in the beginning where you handle your data and ensure its quality. This is also why many say that more than 80% of a Data Scientist’s time is dedicated towards ETL procedures and the remainder is spent developing and deploying models as well as communicating the results of the project.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=abec5e9e0789" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Steps Every ML Project Goes Through Part 1]]></title>
            <link>https://medium.com/@akshay.sharma8426/the-steps-every-ml-project-goes-through-part-1-53ca9492652c?source=rss-c1c8910ec894------2</link>
            <guid isPermaLink="false">https://medium.com/p/53ca9492652c</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[students]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Akshay Sharma]]></dc:creator>
            <pubDate>Wed, 25 Sep 2019 16:55:24 GMT</pubDate>
            <atom:updated>2019-09-25T16:55:24.647Z</atom:updated>
            <content:encoded><![CDATA[<h3>Understand the problem at a high level</h3><p>When you are assigned a project or if you decide to take on a project, the first question you need to ask is: what is the objective? This is important because obtaining this understanding will frame this entire process. Typically, building a model is not the end goal. The end goal is what solution or benefit will the model bring. Once you understand this, you will be able to frame what algorithms you select, what performance measures you use to evaluate your model, and how much you tweak it.</p><h3>Obtain the data</h3><p>There are many ways of obtaining your data and it ultimately depends on what type of project you are working on. If you are completing a project based on an idea you came up with, you are likely going to be obtaining your data through webscraping or API calls. This will mean running various functions and combining different sources of data. No matter how you initially obtain your data, you will end up using the Pandas library to visualize and manipulate your data.</p><p>Once you have obtained your data and put it into a Pandas dataframe, one of the first steps to take is to quickly look at your data structure. This can be achieved by simply using <strong><em>Dataframe.head()</em></strong>.</p><p>This will return the top 5 (or any number you put in between the parenthesis) rows of your data frame.</p><p>Using <strong><em>Dataframe.info()</em></strong> is also a great way to quickly look at the various columns in your dataframe and see the number of non-null values and each attribute type. This will help you identify columns where you are missing data and if your columns with numbers are incorrectly saved as strings rather than floats.</p><p>This is also the stage where you identify your categorical variables and continuous variables because this is the stage where you get to understand your data.</p><p>To quickly take a look at the distributions of your data, it is convenient to just do <strong><em>Dataframe.hist(bins =). </em></strong>The bins value indicates how many groups you want in your data. Visualizing your data helps you understand how skewed your data may be and if you need to transform it to help your ML model better detect patterns.</p><p>To quickly look at the descriptive statistics of your data, it is convenient to use <strong><em>Dataframe.describe(). </em></strong>This method quickly provides descriptive statistics regarding the values in each of your columns for you to get a sense of the values. This can help you identify if one of the columns is in a different scale that might present an issue later on.</p><p>First thing to do is to separate a chunk of your data as your test set. Typically this will be 20 to 25 percent of your total data set. You want to separate out data because you want the model you end up making to be able to generalize to the patterns of data it has not been trained on. This tests its predictive power which is the whole purpose of a model. To be able to perform well on unseen data. You want to take out your test data by randomly selecting rows of data from your dataframe. You may also use the train_test_split module from SKLearn to do this automatically for you.</p><h3>Visualize and manipulate the data</h3><p>With your test data set aside, you have ensured you won’t be biased in your decision making in regarding to all of the future steps. So far, you have only developed a high level understanding of what type of data you are working with. In this stage, the goal is to develop a strong understanding of your data in order to enter the stage where you prepare your ML algorithms.</p><p>It is good practice to make a copy of your dataframe and work on this during this stage because you may end up making some changes that you may not want to keep when you start modeling. A simple <strong><em>.copy()</em></strong> will do the trick at the end of your variable name for your dataframe.</p><p>You can get really into the weeds during this step when you want to analyze your date. One way to do this is to create a scatter plot and fiddle with the parameters to get a sense of how all your variables relate to one another. You can manipulate the size of your circles with the values of one variable, you can manipulate the color to be another variable, and set a range on your color map to distinguish between high and low values. You can gain a true understanding of all the parameters you can tune by taking a look at by looking at the documentation of pandas and pyplot.</p><p>Another way to determine relationships with your variables and your final one is to take a look at the correlation coefficients. These values range from -1 to 1. The strength of relationship increases as the values get closer to 1. The strength of an inverse relationship increases as the values get closer to -1. The relationship weakens if the values approach 0.</p><p>You can use the scatter_matrix function from pandas which plots every numerical column of data against each other. This will result in scatter plots showing variable relationships among each other.</p><p>You may also decide to combine your features together to make new ones that you think may better define the relationship to your dependent variable. You can reaffirm whether your new features do a good job by finding their correlation coefficient values as I mentioned above.</p><p>The point of this step is to understand the quirks of your data. This entails finding the relationships in your data and making the decision to remove certain columns or rows of data that do not generalize to the rest of the data, such as outliers. At the end of the day you want your ML to be general enough to predict well rather than be trained on noise present in your data.</p><h3>Prepare the data for ML algorithms</h3><p>When you want to start the process of building out your Machine Learning models, you need to make sure your data is consistent. This means you need to address any missing data points.</p><p>Typically, you have three options when dealing with missing data. You can get rid of the rows where your data is missing, you can get rid of the entire attribute of a lot of data within that column is missing, or you can set the missing values manually using a variety of techniques.</p><p>If you decide to go with the first option of getting rid of only the values missing, you can easily do <strong><em>Dataframe.dropna(subset=[‘column_name’])</em></strong>.</p><p>If you decide to go with the second option of getting rid of the entire column, you can easily do <strong><em>Dataframe.drop(‘column_name’, axis = 1)</em></strong></p><p>Finally, if you opt in for the last option, you can use the <strong><em>Dataframe[‘column_name’].fillna(technique)</em></strong>. You can opt in to use the median value of your column or the mean, or whatever you think is best. You will need to save your value you compute for this technique in order to use your test set that you already separated out from your data.</p><p>At this stage you will also need to deal with any categorical data you may have by converting it to numbers. For this situation you can use the LabelEncoder function from sklearn. For more information on how you can use the sklearn library to deal with categorical data, please visit the following link:</p><p><a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html">LabelEncoder</a></p><p>When you are dealing with multiple categories within a column, you may want to one-hot encode your categories instead. This technique creates a separate column for each category and gives the value of 1 or 0 depending on whether that row of data fits that or not. This solves the problem of accidently creating a relationship between low and high values when you simply encode your categories. If you are dealing with nominal categorical data, then order does not matter within your categories. As such, you would not want to accidentally establish that relationship by using a label encoder. This can be done by using the <strong>LabelBinarizer</strong> class from the SKlearn library.</p><p>Another important task that needs to be tackled at this stage involves feature scaling. ML algorithms do not perform well if the various numerical features have very different scales. The two most common methods to scale include <strong><em>min-max scaling</em></strong> and <strong><em>standardization</em></strong>.</p><p>Min-Max scaling (AKA normalization) adjusts the values within the column to be between 0 and 1. This is done by subtracting out the minimum value from your value and then dividing by the maximum value minus the minimum. You can also simply use the <strong><em>MinMaxScaler</em></strong> from SKLearn to automatically scale your data.</p><p>Standardization works by subtracting out the mean value and then dividing by the variance within the column. This results in the rescaled data having a mean of 0 and a standard deviation of 1. One thing to note here is that the rescaled values are not confined to a specific range. While this may hurt the performance of some algorithms, it provides an advantage in that it is not as affected by outliers in your data. For example, if you had an extremely high outlier, the min-max scaler would make that equal to 1 while making the rest of your data points confined to really small values. This situation is not present if you use standardization instead. As such, the scaler you use depends on your data. This is why the early exploratory data analysis is key as I mentioned above. Similar to the <strong><em>MinMaxScaler</em></strong> from SKLearn, you can use the <strong><em>StandardScaler</em></strong> to automate the scaling process of your data.</p><h3>Conclusion</h3><p>I hope the information above is helpful as you begin tackling data science projects and working with data. In order to keep the length of this post short, I will be releasing part 2 of this blog next week where I will discuss the steps involving: training your model, tuning your model, presenting your solution, and launching your model.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=53ca9492652c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Different Types of Distributions You Will Encounter As A Data Scientist]]></title>
            <link>https://medium.com/mytake/understanding-different-types-of-distributions-you-will-encounter-as-a-data-scientist-27ea4c375eec?source=rss-c1c8910ec894------2</link>
            <guid isPermaLink="false">https://medium.com/p/27ea4c375eec</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[distribution]]></category>
            <dc:creator><![CDATA[Akshay Sharma]]></dc:creator>
            <pubDate>Wed, 18 Sep 2019 02:46:35 GMT</pubDate>
            <atom:updated>2019-09-20T19:01:16.879Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*DmPUIjvecL7KllOamoFSDw.png" /><figcaption>Common types of distributions</figcaption></figure><p>As a Data Scientist, you will be looking at a lot of data. While that may be common sense, you also need to understand that not all data is the same. Some of your data sets may be continuous distributions while others may be discrete. Some columns of your table may follow a Gaussian distribution while others may be exponential. Due to numerous types of data distributions possible, it is important to establish a solid understanding of the most common types and be familiar of the situations where you are likely to see one.</p><h3>Discrete vs Continuous Distributions</h3><p>A simple way to understand if your data is discrete or continuous is the answer the following question: Are the number outcomes finite?</p><p>If the answer to the above is yes, then you have a discrete dataset. Otherwise, you likely have a continuous dataset.</p><p>But what does finite mean, exactly? Well, if you imagine a roll of a fair dice, you know that you have exactly 1/6 chance of rolling a 1, 2, 3, 4, 5, or 6. These values are finite because there is no chance of getting a 1.2 or 2.6 or other smaller value that isn’t exactly 1 or 2. These outcomes are a set of values rather than a continuous length of values.</p><p>An example of a continuous distribution would be weather. On any given day, the weather channel may say it is 62 degrees Fahrenheit. The actual temperature may not be exactly 62. It could be 62.01, 62.001, or even 62.00000001. All these values are different from the other and as such as referred to as a continuous range of values.</p><h3>Examples of Discrete Distributions</h3><p>A few common examples of discrete distributions include the Bernoulli Distribution, the Poisson Distribution, and the Uniform Distribution.</p><p>The Bernoulli Distribution essentially represents the probability of success of an experiment. An example would be tossing a coin where the 2 outcomes are given a probability of occurrence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/0*fmxaCqSx0KxxqGXS.png" /></figure><p>The Poisson Distribution represents the probability of <strong><em>n</em></strong> events in a given time period when the overall rate of occurrence is constant. In simpler terms, it gives you a probability of how often something might happen. We can use the amount of mail you receive everyday. A Poisson Distribution would give you the probability of how many pieces of mail you receive each day.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/453/0*eI0LXSmAsFwWbsqf.gif" /></figure><p>The Uniform Distribution is the simplest distribution to explain. Is is when all the outcomes are equally likely.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*AYtr5eq0Kc8MR4Zu.gif" /></figure><h3>Example of Continuous Distribution</h3><p>The Gaussian or normal distribution is the most common distribution that you will come across. This follows a bell shape, which is the name you may recognize it by, and is found in many real world data, such as height and weight.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6yZT9pwR239KO4CT.png" /></figure><h3>Conclusion</h3><p>There are many different types of distributions and while each one describes a different situation, the normal distribution serves as the foundation within the Data Science community. Since this is the most common distribution that is naturally occurring, it is vital to begin your understanding of distributions with the Gaussian distribution. To begin your journey, please feel free to check out the links below that will help you learn.</p><ul><li><a href="https://stattrek.com/probability-distributions/normal.aspx">Normal Distribution</a></li><li><a href="https://www.statisticshowto.datasciencecentral.com/probability-and-statistics/normal-distributions/central-limit-theorem-definition-examples/">Central Limit Theorem: Definition and Examples in Easy Steps</a></li></ul><p><a href="https://www.analyticsvidhya.com/blog/2017/09/6-probability-distributions-data-science/">https://www.analyticsvidhya.com/blog/2017/09/6-probability-distributions-data-science/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27ea4c375eec" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mytake/understanding-different-types-of-distributions-you-will-encounter-as-a-data-scientist-27ea4c375eec">Understanding Different Types of Distributions You Will Encounter As A Data Scientist</a> was originally published in <a href="https://medium.com/mytake">MyTake</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Making a Text Generator Using Markov Chains]]></title>
            <link>https://medium.com/analytics-vidhya/making-a-text-generator-using-markov-chains-e17a67225d10?source=rss-c1c8910ec894------2</link>
            <guid isPermaLink="false">https://medium.com/p/e17a67225d10</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[text-generation]]></category>
            <category><![CDATA[markov-chains]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Akshay Sharma]]></dc:creator>
            <pubDate>Mon, 26 Aug 2019 15:42:23 GMT</pubDate>
            <atom:updated>2019-09-05T04:26:18.277Z</atom:updated>
            <content:encoded><![CDATA[<p>Having a computer sound like a person with text being auto-generated sounds like the future. There have been small proof of concepts, such as Gmail’s predictive text as you write out an email, and the point where this concept is truly realized may be arriving sooner rather than later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*i6VdHhgcYAhJYdf0" /></figure><p>A small project I completed, that doesn’t use neural networks to generate text, used Yelp Reviews to generate new reviews using Markov Chains. The text generated, although not perfect, does an incredible job of capturing the context, sentiment, and style of a review written by a typical user.</p><p><a href="https://github.com/hiowatah/Yelp-Review-Generator">hiowatah/Yelp-Review-Generator</a></p><h3>What is a Markov Chain?</h3><p>Markov Chains are mathematical systems that go from one state to another. A few rules associated with this broad statement are as follows: the next state is entirely dependent on the previous state. The next state is determined on a probabilistic basis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/0*HsJN6HRot2UMdDfo" /><figcaption>Example Image of Markov Chain from Brilliant.org</figcaption></figure><p>To put this into the context of a text generator, imagine an article you recently read. That article contains x number of words where there are probably many words used multiple times. For each word in the article, the words directly after are grouped together where words that occur more often are weighted more heavily. When generating text, a random word is chosen and from that a random word from the list of words is selected continuously until the desired word count is reached.</p><h3>An Example Using Python</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3cc2d372315720bfed07360f0b0169b1/href">https://medium.com/media/3cc2d372315720bfed07360f0b0169b1/href</a></iframe><p>In python terms, you create a dictionary of every unique word in your corpus. From there, you make the values of your dictionary a list of words that appear after each unique word. The more often a word appears after another, the higher the probability that it will be selected in the text generator.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/1*VtphDqdyGZbzKDt4f28rkg.png" /><figcaption>Sample text generated</figcaption></figure><p>As you can see from above, the text generated looks, at a high level, as if a human being wrote it. It is when you look deeper into the words that it becomes a little more evident that it doesn’t make sense holistically. Something to fix this could be adding weights to the probabilities of the words selected.</p><p>While the concept is simple, the hardest part of creating a generator using Markov Chains is to ensure you have enough text in your corpus so the text you generate doesn’t end up being the same words over and over.</p><h3>Conclusion</h3><p>Markov Chains can work wonderfully in generating text to mimic a human being’s style. However, in order to effectively generate text, your corpus needs to be filled with documents that are similar. In the example above, I captured 3 star reviews from Yelp. However, it contains phrases like manure, office buildings, nfl, and theater. These are generally unrelated and would not be posted in a typical review. In order to correct this, you will need to keep documents discussing similar topics (i.e. pizza parlors) in the same corpus and use that for Markov Chains. This way the text generated will all be pizza related. That being said, for such a simple execution, the result of the text generated is remarkable and much easier to obtain than heavily trained neural networks!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e17a67225d10" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/making-a-text-generator-using-markov-chains-e17a67225d10">Making a Text Generator Using Markov Chains</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How To Tell If Something Is OP!?]]></title>
            <link>https://medium.com/@akshay.sharma8426/how-to-tell-if-something-is-op-1e2fd3a5fbd?source=rss-c1c8910ec894------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e2fd3a5fbd</guid>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[gaming]]></category>
            <dc:creator><![CDATA[Akshay Sharma]]></dc:creator>
            <pubDate>Wed, 21 Aug 2019 18:26:59 GMT</pubDate>
            <atom:updated>2019-08-21T18:26:59.090Z</atom:updated>
            <content:encoded><![CDATA[<p>In video games, the term OP (over powered) gets tossed around very liberally. Sometimes it is true, sometimes it is said as a defense mechanism to explain why you lost, and other times it is just bewildering. The point of this blog is to allow gamers to use statistical analysis to determine if something is actually OP or not.</p><p>The basis of my blog is based on a Data Science Project I did relating to Qiyana, the latest champion to be released in League of Legends. The Github link to my readme and my notebooks is below.</p><p><a href="https://github.com/hiowatah/Analysis-of-Qiyana-LoL">hiowatah/Analysis-of-Qiyana-LoL</a></p><h3>Step 1: Obtain Data on The Items of Comparison</h3><p>This step is straight-forward, but oftentimes not easily doable. In order to compare 2 things and say one is statistically stronger and therefore, ‘OP’, you need to obtain a lot of data points for the 2 things.</p><p>In my project, I obtained match histories of over 300 players that were the top rank in the North American Server of League of Legends. This resulted in 30,000 data points, which after some data massaging, boiled down to roughly 12,000 total matches. Once you have your raw data, you need to pick out the data points that you want to compare. For example, you wouldn’t compare a support/healer character with an assassin to determine if the assassin was ‘OP’ because they serve different roles.</p><p>For my project, I picked out champions that served a similar niche as Qiyana. This included melee attack damage assassin type champions. As a result, my comparison included Zed, Qiyana, Yasuo, and Talon.</p><p>The matches included data on the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/0*MKqjHW4_9PIisI5r.png" /></figure><p>The reason you see 20,949 data points is because a single match can contain multiple champions and so in the 12,000 matches I had, many of them contained one or more champions in the cluster I analyzed.</p><h3>Step 2: Create an Index Using Your Data</h3><p>It is very difficult to look at many different columns and make the bold claim of something being ‘OP’ because, more often than not, the champions will have different strengths and weaknesses and so they may have higher values in some columns and lower values in others. This is why you need to create an index, which consolidates the values of each column into a single number.</p><p>This part is where bias may trickle in and effectively, ruin your whole analysis. As I mentioned before, champions will have higher values in some columns than others, and so if you add more weight to certain columns, you are essentially guaranteeing that that champion will be labeled ‘OP’.</p><p>For my project, I had the following weights based on my understanding of the role of assassin.</p><blockquote>Kills: 1.25</blockquote><blockquote>Deaths: 1.25</blockquote><blockquote>Assists: 0.50</blockquote><blockquote>Killing_Spree: 1.00</blockquote><blockquote>Total_Damage: 1.00</blockquote><blockquote>Damage_To_Objectives: 0.60</blockquote><blockquote>Damage_To_Turrets: 0.60</blockquote><blockquote>Damage_Taken: 1.00</blockquote><blockquote>Gold_Earned: 0.60</blockquote><blockquote>Vision_Score: 0.80</blockquote><blockquote>Crowd_Controls: 0.75</blockquote><p>Kills and Deaths deserve more weight because of the feast or famine role assassins play in team comps. CC, Vision, Gold, Damage to objectives/turrets deserve less weight because these scores can be more easily influenced by the length of the game.</p><h3>Step 3: Plot the Distribution of Your Data</h3><p>Now that you have a means to measure the different items you are comparing to determine if something is ‘OP’ or not, you need to start analyzing it with the mindset of “On average, is this item scoring higher than the others. If so, is it statistically significant?”</p><p>This essentially means that if the average index score is 5 and the next item has an index score of 4: is the difference statistically significant, or can is it highly probably that if you had different data points (samples) that the index scores of the 2 items could be reversed, or at least closer in value?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/0*D33gklSFNrF3k-Kn.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/0*yVO1nfZ0siGb1Lkk.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/0*tKzgw6Xwnh7WBUfo.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/0*hkLAH49YSPPu27bN.png" /></figure><p>These distribution plots are fairly normally distributed, but they clearly all are positively skewed.</p><h3>Step 4: Look At The Average Values</h3><p>To finally determine if something is ‘OP’ you need to now compare the average values between them and see if there is a significant difference. This involves running a t-test and obtaining the p-value. The p-value will give you the probability that the comparison was due to random chance. And so if the value is below your alpha, which is basically the inverse of your confidence level (confident at 95% means alpha of 5%), then you can confidently reject the null hypothesis that the champion is not ‘OP’ compared to other champions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/0*ME7U66fcYhjdLuqj.png" /></figure><h3>Step 5: Conclude</h3><p>When you compare multiple items, you run an ANOVA test. This is also called analysis of variance. If you want to compare 2 items, you run a Student T-Test.</p><p>For my project, I kept it simple and simply did a t-test between each champion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8TG88oDQxE4a6AaE.png" /></figure><p>Running a T-test on the array of scores for each champion, the only relationship that is statistically signficant and can reject the Null Hypothesis is between the other 3 champions and Talon. Their average scores are statistically higher which may indicate that Riot games will need to make Talon a little bit stronger so player performances on this character improve.</p><p>And so, while I cannot say anything is ‘OP’ for these 4 champions, I can claim that Talon is ‘UP’ or under-powered. And so can you, if you apply statistical scrutiny to the next thing you scream out is ‘OP’ after you die to it!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e2fd3a5fbd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Deeper Look Into Laning in League of Legends]]></title>
            <link>https://medium.com/@akshay.sharma8426/a-deeper-look-into-laning-in-league-of-legends-ef3ce9a4f13a?source=rss-c1c8910ec894------2</link>
            <guid isPermaLink="false">https://medium.com/p/ef3ce9a4f13a</guid>
            <category><![CDATA[league-of-legends]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Akshay Sharma]]></dc:creator>
            <pubDate>Wed, 07 Aug 2019 17:47:57 GMT</pubDate>
            <atom:updated>2019-08-07T17:47:57.381Z</atom:updated>
            <content:encoded><![CDATA[<p>Over the past few weeks, I have been working on a project designed to look at the types of play-styles in each lane for the online game League of Legends. If you would like to look at the code (Python) for this project, please see the github link below:</p><p>( <a href="https://github.com/hiowatah/Analysis-of-Qiyana-LoL">https://github.com/hiowatah/Analysis-of-Qiyana-LoL)</a></p><p>A quick overview of the game: There are 3 lanes (Top, Middle, Bottom) and 2 additional roles (Jungle, Support). The game is essentially a tower defense game with 2 teams of 5 battling it out to take down the other teams base.</p><p>The question I wanted to answer was: how many differing playstyles are in the game which help players keep coming back to keep this game one of the most popular PC games out there. To answer this question, I obtained data using Riot Games’, the developer of League of Legends, API. I looked at the best players in North America (Rank: Challenger) to truly understand the optimal playstyle of each champion and how they should be played.</p><h3>Obtaining the Data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/0*7IzUOlOKe_TfpDT6.png" /></figure><p>The graph above shows the regression plot of the wins and losses of the players ranked Challenger when I pulled my data. As the plot indicates, these players have a greater than 50% win rate which means they win more games than they lose. This is vital since winning increases a players Match Making Rating which results in a higher rank.</p><p>The second step, after obtaining the summoner names of the players ranked Challenger was to obtain their unique Account Ids. This was achieved through a second API call.</p><p>The third and final step was to obtain the match histories of the players. I obtained 100 match histories per player resulting in 30,000 total data points. However, as I was obtain match histories, which includes 9 other players per match, their were bound to be many duplicate entries. As such, I dropped all duplicates which resulted in a total data set of around 12,000 match histories to analyze.</p><h3>Segregating the different roles</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/0*-Ho7wfAJxXAkXidE.png" /></figure><p>Trying to group the champions played in each role proved to be a very difficult step during the EDA process. Because there are 300 players in Challenger and based on the graph above, a majority of them are Bottom or Support, a player will be “auto-filled” into a role other then their primary role. They will then “role swap” with another player in order to obtain their primary role. For example, if someone queued up for a game hoping to play Middle or Jungle, but the game determined their role to be Bottom, then they would try to trade their role to play Middle or Jungle but the game data will still determine them to be “Bottom”. As such, the graph above and the overall data can be very misleading if looked through the lens of lane assignment.</p><p>In order to obtain an accurate picture of the current state of the game, the analysis should be determined on the common champions played in each role and manually segregate champions into different roles and perform an analysis that way.</p><p>This is not the ideal situation as champions like Irelia are commonly played in the top lane and mid lane. However, for clustering purposes, the best approach is to manually assign a primary lane to each champion and group them through that. For this, I used <a href="http://na.op.gg/champion/statistics">http://na.op.gg/champion/statistics</a> to look at the different champions in each role.</p><h3>The Clusters</h3><p>After going through the champion roster and assigning each champion a main role, I was able to use K-means clustering to assess the different types of playstyles within each role. This means that I was able to see the different types of champions being played in the top lane, jungle, mid lane, bottom lane, and support.</p><p>In the section below, I will be providing the number of clusters (play-styles) in each lane, the types of champions in each cluster, and a high level assessment on the differentiating factor of the clusters.</p><p>I primarily relied on elbow-plots to determine how many natural clusters existed within each role. Elbow-plots show the point of diminishing returns between the number of clusters and some metric. In my case, I used the sum of squared differences between each data point and their closest cluster center.</p><h4>Top</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/435/0*HtTUZOryYA0oaecL.png" /><figcaption>Elbow Plot for Top Lane</figcaption></figure><p>Based on the elbow plot above, it is fairly clear that the point of diminishing returns begins at k = 3 but really becomes evident at k = 4. For this reason, I have settled on having 4 clusters for the top lane.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/0*1XI5hRI5sWqEcaa3.png" /><figcaption>Champions in each cluster for Top Lane</figcaption></figure><p>As you can see from the table above, the 4 clusters have pretty distinguishing features.</p><p>From a high level perspective, cluster 1 contains champions that are generally tanky and have a form of crowd-control.</p><p>Cluster 2 contains champions that have a high amount of sustain in lane or an ability that instantly gives them more HP through transformations.</p><p>Cluster 3 contains champions that either are lane bullies or late game scalers. The commonality between those 2 are that they are likely to get many kills in a game. This is why this cluster averages the most kills out of all the clusters.</p><p>The last cluster is interesting as it contains just three champions. The commonality with these champions is that they run at you and are considered to be “feast or famine”. These champions are one-dimensional and if they don’t get ahead in the early game, they are incredibly easy to shut down, especially at the highest level of play.</p><h4>Jungle</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/435/0*5i8dJm865pJSYXmJ.png" /><figcaption>Elbow Plot for Jungle</figcaption></figure><p>Based on the elbow plot above, it is fairly clear that the point of diminishing returns begins at k = 3. For this reason, I have settled on having 3 clusters for the jungle.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/0*05MgoAV2YyChmN1m.png" /><figcaption>Champions in each cluster for Jungle</figcaption></figure><p>Based on the average values from each cluster it is evident that cluster 1 contains the high damage, carry-style junglers.</p><p>On average, champions in cluster 1 get two more kills than the champions in the other clusters. They also average 3k more damage over the course of a game over other junglers as well.</p><p>Champions in cluster 2 fit the mold of being more supportive and provide crowd-control capabilities to the team. As such, these champions average the lowest gold of the clusters as well as a crowd control of ~46 while the other 2 clusters average a crowd control score of ~13 and ~20.</p><p>The last cluster contains the tanks. Some of these champions are not necessarily tanks by nature, but they are considered to be early game champions that fall off in the late game, such as Xin Zhao, Lee Sin, and Reksai. This is why these champions start to build tanky items to maintain value to their team. As such, the champions in this cluster average the lowest damage out of the clusters and the highest damage taken. This cluster averages over 20% more damage taken then the next highest cluster.</p><h4>Mid</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/0*gxx4xhryRgexgrKe.png" /><figcaption>Elbow Plot for Mid Lane</figcaption></figure><p>Based on the elbow plot above, it is fairly clear that the point of diminishing returns begins at k = 3. For this reason, I have settled on having 3 clusters for the mid lane.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/0*MWSpX1_ufVPcoxpm.png" /><figcaption>Champions in each cluster for Mid Lane</figcaption></figure><p>Based on the selection of champions in each cluster alone, it is fairly evident what the playstyles and distinguishing play-patterns are within each cluster.</p><p>Looking at cluster 1, these champions are predominantly mages with long range and some form of crowd-control. As such, champions in this cluster average a crowd-control score of 23.14 while the other 2 clusters hover around 14. Champions in this cluster are also a main source of damage for any team composition, however, they don’t need kills to be useful. As such, they average 1 whole kill less than the other main cluster, cluster 2, but they average a higher kill/death/assist ratio.</p><p>Champions in cluster 2 are mostly assassins. This means they are high risk — high reward and often need to get up close and personal with the enemy in order to deal their damage. Since they expose them to the enemy team, they average the most damage taken out of the other 2 groups in the mid lane. They average over 23% more damage taken then the next closest cluster. As I mentioned before, this group averages the most kills out of the clusters, averaging upwards of 6 kills a game. This is due to the fact that earning kills nets you more gold and you need the gold to maintain your advantage state in the game. If an assassin falls behind in the game, their team is pretty much guaranteed to lose. Earning more kills nets you more gold individually which is why these assassin champions average a lot more.</p><p>The last cluster is pretty interesting as it only contains one champion. This champion, Ziggs, has a few unique qualities in that he is long range and has abilities to take down enemy structures, like towers, very quickly. As such, Ziggs averages more than 6x the damage dealt to enemy structures per game than any other cluster in the mid lane. Because all of Ziggs abilities are area-of-effect, he also averages the most damage to objectives and as a whole out of the other clusters. This is why he is a cluster of 1 because Ziggs is essentially an all encompassing champion.</p><h4>Bot</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/0*U6fv10a_NnKmkrEh.png" /><figcaption>Elbow Plot for Bot Lane</figcaption></figure><p>Based on the elbow plot above, it is fairly clear that the point of diminishing returns begins at k = 4. For this reason, I have settled on having 4 clusters for the mid lane.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/0*j7A2wkUlVTbrtIjA.png" /><figcaption>Champions in each cluster for Bot Lane</figcaption></figure><p>The bot lane is interesting as the playstyle of ADCs tend to be very similar. Their duty is to dish out as much damage while taking the least damage possible due to their squishy nature. However, looking at the clusters in the bot lane, it makes sense that the role is segmented this way.</p><p>Looking at cluster 1, these champions tend to be more supportive in nature by providing crowd control capabilities to their team. The data corroborates this assessment as they average the highest assists out of the bot lane clusters, averaging nearly 10% more than then next closest cluster and they average the highest crowd controls scores as well, averaging nearly 41% more than the next closest cluster.</p><p>Looking at cluster 2, the champions in this cluster all control the space around them, either with abilities that displace enemies or through traps that prevent enemies from approaching or the ability to move large distances themself to create more space between them and the enemy. The champions in this cluster are mostly lane-bullies with the ability to shove lane and oppress the other side. As such, they average the most damage out of all the clusters to objectives and turrets. This is because they are able to obtain lane priority.</p><p>Looking at cluster 3, these champions have the shortest range and are auto attack reliant. Since they need to get up close to the enemy in order to deal damage, they average the highest damage taken out by 12% from the next closest cluster. On the other-side, they also average the most kills out of the other ADCs, but only by a marginal amount.</p><p>Looking at the final cluster, this cluster only contains one champion who has the most unique playstyle out of the marksmen. Ezreal is a poke based marksman who primarily relies on his spells to deal damage. As such, it makes sense that he averages the least amount of kills but the most damage dealt per game.</p><h4>Support</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/435/0*0AEO7F5MV3kw6vAv.png" /><figcaption>Elbow plot for Support</figcaption></figure><p>Based on the elbow plot above, it is fairly clear that the point of diminishing returns begins at k = 6. For this reason, I have settled on having 6 clusters for support.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ljB2QSUgfSw8AL2U.png" /><figcaption>Champions in each cluster for Support</figcaption></figure><p>Support has the highest number of clusters out of any other lane indicating the various types of play-styles this role can play. Based on the champions in the different clusters and my domain knowledge, being a bot lane main, I can say that each cluster is definitely distinctive in it’s play-style.</p><p>Looking at cluster 1, these champions primarily provide heals and sheilds to you and your team. Their main duty is to empower you to deal more damage. As such, they average the least kills and the highest vision score — which is their primary duty as a supportive style support.</p><p>Looking at cluster 2, these champions have a high amount of damage negating abilities. As such, the champions in this cluster average the highest amount of damage taken than any other cluster.</p><p>Looking at cluster 3, these long ranged high damage dealer mages purpose is to poke the enemy bot lane down so your bot laner can free farm. As such, these champions average the highest kills and damage dealt out of the other clusters for supports. This cluster averages just over 36% more damage dealt than the best cluster.</p><p>Looking at cluster 4, this cluster contains the hybrid champions of clusters 1 and 3. These champions help prevent their damage dealers from getting locked down in a fight through their supportive abilities as well as have high damaging skills to poke down their lane opponents. The data I have does not accurately represent the value these types of champions bring to a team and thus this cluster has stats that make this the average for supports.</p><p>Looking at cluster 5, these are the tanky high crowd control type of supports. As such, they average the second highest damage taken, second highest crowd control, as well as the least damage dealt.</p><p>Looking at the last cluster, these champions are the long range poke based champions which big shields/heals. As such, they average the second most damage done and the least damage taken out of all the clusters for support. They also average the highest crowd control score by the number of slows and roots in their kit. In my opinion, these 2 champions are definitely the most annoying to lane against as these champions, by nature, prevent the enemy from doing anything while dealing damage.</p><h3>Conclusion</h3><p>In my post above, I provided descriptive statistics between each cluster for the different roles in League of Legends. If you would like to quickly look at the raw data behind those stats, please check out my github repository for this project. I have a separate Juptyer Notebook for each role to make it easier to deep dive on whatever you are interested in.</p><p><a href="https://github.com/hiowatah/Analysis-of-Qiyana-LoL">hiowatah/Analysis-of-Qiyana-LoL</a></p><p>Based on my clustering analysis, it looks like there are a few defined play-styles within each role and multiple champions within most clusters to keep the game fresh for a wide variety of players who prefer a single play-style.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ef3ce9a4f13a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystifying ASAP: Evaluating your K-means Clustering Model]]></title>
            <link>https://medium.com/@akshay.sharma8426/demystifying-asap-evaluating-your-k-means-clustering-model-e1a438294d3a?source=rss-c1c8910ec894------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1a438294d3a</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[k-means]]></category>
            <dc:creator><![CDATA[Akshay Sharma]]></dc:creator>
            <pubDate>Tue, 30 Jul 2019 03:33:51 GMT</pubDate>
            <atom:updated>2019-07-30T03:33:51.141Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*M4kAma885x_0RCOe.png" /></figure><h3>Overview of Clustering</h3><p>Clustering techniques are a type of unsupervised machine learning algorithm. They are used, as the name suggests, to group similar data points together. The way grouping works is the distance between the different groups together is high and the distance between each data point inside the cluster is low. This is known as intra-class (the different groups) and inter-class (within each group).</p><p>Clustering techniques determine similarity through distance which can be measured through a number of means, such as Manhattan, Euclidean, or Minkowski.</p><p>There are many types of clustering techniques that you can employ. The distinction I want to make in this blog is between hierarchical and nonhierarchical algorithms.</p><p>Looking into hierarchical algorithms, there are 2 types that you can use. Agglomerative and divisive. The difference between these is that the former starts with <strong><em>k</em></strong> clusters where <strong><em>k</em></strong> is the number of observations and the algorithm begins to combine the 2 most similar clusters together and the latter starts with one giant cluster and begins to divide towards x clusters.</p><p>A nonhierarchical algorithm requires you to choose the initial clusters. The center point of each cluster is selected at random and the centers begin moving until the optimal points are reached for the <strong><em>k</em></strong> clusters.</p><h3>A closer look at K-means</h3><p>K-Means Clustering is a type of nonhierarchical algorithm. This means we need to decide on the number of clusters we want before we employ this technique. There are many scenarios where this is a hard decision as one wouldn’t know how many clusters there should be in your data. To help you evaluate how many clusters, or the value of k, to settle on, there are many metrics you can look at before finalizing your decision. Among them are the: <strong>Calinski Harabasz Score</strong>, the <strong>Elbow Plot</strong>, and the <strong>Silhouette Score</strong>.</p><h4><strong>Calinski Harabasz Score</strong></h4><p>The Calinski-Harabasz score evaluates a cluster based on the average intra and inter cluster sum of squares. The higher the score, the better the clustering is for your data. This rings true with what we talked about earlier where you want to maximize the distance between different clusters and minimize the distance between data points within each cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/0*OsknoDvMLucS6aeL.png" /></figure><h4><strong>Silhouette Score</strong></h4><p>The silhouette score is calculated based on the mean intra-cluster distance and the mean nearest-cluster distance for each sample. The values range from 1 to -1 where the higher the value the better the clustering fit is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/0*Hv9seLnl4orJRoAy.png" /></figure><h4>Elbow Plot</h4><p>An elbow plot is a general term which essentially plots a metric on the y-axis and a value of k on the x-axis. The purpose of an elbow plot is to determine the point of diminishing returns of naturally occurring clusters in a data set. The value of k is determined by the point where the plot bends like an arm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/0*aDuQQHquXGksbpqq.png" /></figure><h3>Conclusion</h3><p>Determining the optimal number of clusters in your data depends on the circumstances. There is no right answer to fit 100% of the problems. However, with these 3 metrics, you will have more information in determining what number of clusters makes the most sense for you, your data, and the story you are trying to tell.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1a438294d3a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My Experience With A Data Science Boot Camp]]></title>
            <link>https://medium.com/@akshay.sharma8426/my-experience-with-a-data-science-boot-camp-59457d31bcad?source=rss-c1c8910ec894------2</link>
            <guid isPermaLink="false">https://medium.com/p/59457d31bcad</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[online-learning]]></category>
            <category><![CDATA[bootcamp]]></category>
            <dc:creator><![CDATA[Akshay Sharma]]></dc:creator>
            <pubDate>Mon, 20 May 2019 18:37:05 GMT</pubDate>
            <atom:updated>2019-05-20T18:37:05.090Z</atom:updated>
            <content:encoded><![CDATA[<p>10 weeks ago, I decided to branch out of public accounting and pursue my newfound passion in Data Science. I had gotten exposure to conducting risk analysis in my day to day job and through my companies internal resources, I discovered this “new” field of Data Science and instantly became hooked. The following blog talks about my experience so far of going through an immersive (in-person) bootcamp.</p><p>Before making the life-altering decision of quitting my job and doing this bootcamp, I spent a few months trying to self-teach myself the various aspects of Data Science. However, my approach, in hindsight, was not the best as I broke down the various aspects until I had mastered them. This meant I spent most of my time initially trying to teach myself Python for a few months and then moving on to basic statistics.</p><p>The progress I was making was not that at a pace I was content with and I realized I had to commit all the way to learning these new skills if I was serious about wanting to do this full-time. As a result, I joined Flatiron School for Data Science and, as of today, have completed 10 out of the 15 weeks here.</p><h3>Advantages of a Bootcamp</h3><ol><li>The biggest aspect to a bootcamp, besides the curriculum, is the amount of diversity within my class. Everyone has a different background from recent college grads to people with their own businesses and years of experience. These are people I probably would not have come across had I not done this in person bootcamp and having the opportunity to work with them has been a great experience.</li><li>Having a structured method of being introduced to various topics has been an immense help for me as well. The purpose of a bootcamp is to give you a wide exposure to all aspects of Data Science and teach you the ability to teach yourself. In that sense you are exposed to various topics, such as ML algorithms and it is up to you to pick and choose the ones you want to get in depth experience on.</li><li>Learning with others is always better than self-teaching or learning by yourself. At my bootcamp, I have gotten the chance to work with a classmate on a coding challenge every morning and just walking through the problem and learning from each other has been a real beneficial experience and is a great way of solidifying the materials I have been learning.</li></ol><h3>Challenges of a Bootcamp</h3><ol><li>15 weeks is a short time to being exposed to so many new topics. It is a real challenge learning something new every single day if you don’t feel comfortable yet with the material you learned the day before. However, this is in line with the old adage, “get comfortable with being uncomfortable”. You certainly have the put in the time to learn the materials fully, but it is definitely a commitment with time dedicated to this going far beyond the in-class commitment of 40 hours a week.</li><li>Because 15 weeks is so short, some topics that you may really want to dive into have to be put to the side because you are consistently learning new things. These topics will have to be revisited post bootcamp when you have more time to dive into the topics you may really be interested in.</li><li>There will be some topics that just won’t click with you. The concept may be something you are completely unfamiliar with and you may get stuck on it. Since you learn something new every day this will pose a challenge to you because as a human being, you want to learn the previous material before moving on to the next topic. However, this way of thinking does not work in a bootcamp setting because of the sheer amount of material covered. This goes back to my point one of being comfortable with being uncomfortable and realizing that you will have to revisit the topic over the weekend or some later point.</li></ol><h3>Concluding Thoughts</h3><p>My experience so far has been overwhelmingly positive in a holistic sense. I don’t think I ever would have been able to learn all that I have in the time frame that I have if I had not been at this bootcamp. With 5 weeks remaining and the final 2 projects remaining to complete, I hope to utilize all that I have learned so far and set out to starting my career as a Data Scientist.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=59457d31bcad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Rise of Streaming and Twitch]]></title>
            <link>https://medium.com/@akshay.sharma8426/the-rise-of-streaming-and-twitch-e0d11d9b0bfb?source=rss-c1c8910ec894------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0d11d9b0bfb</guid>
            <category><![CDATA[gaming]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[streaming]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Akshay Sharma]]></dc:creator>
            <pubDate>Thu, 09 May 2019 16:57:26 GMT</pubDate>
            <atom:updated>2019-05-09T16:57:26.899Z</atom:updated>
            <content:encoded><![CDATA[<h3>What is live-streaming?</h3><p>Live-streaming has become more and more popular over the past few years. Live-streaming is when someone records either themselves or something they are doing (such as playing a video game) and broadcasting that recording live. This has become a very popular over the past couple of years, mainly due to the ability for the broadcaster and the audience to interact with one another in real time. Technology has significantly lowered the barrier of entry in this space as you can simply start streaming from your phone nowadays. However, in order to connect a streamer to an audience, you need a platform to stream on. One of the most popular ones is Twitch.tv.</p><h3>What is Twitch tv?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hu5Qw6gMp3BmcCVxkRzV4w.png" /><figcaption>Top games being streamed on 5/7/2019</figcaption></figure><p>Twitch is mostly used to stream video games as you can see in the screenshot above. Users can make an account and start streaming right away under any category they wish to, as long as the content abides by the terms and conditions (AKA common sense).</p><p>So why are video games the most popular on streaming platforms? This is because gamers used to go to Youtube tutorials/guides to learn how to beat a specific level/enemy or get better at a game they were playing. However, why do that when you can go directly to a livestream and ask the gamer or even the other viewers? The accessibility to other people that are interested in the same game as you provides streaming a huge advantage over the old medium of recording videos and posting them online. The speed of interaction between dozens, hundreds, or even thousands of people cannot be challenged.</p><h3>Big Channels on Twitch</h3><p>Not all is as glorious or great as what I have listed so far. Because streaming is becoming more mainstream by the day, the competition for viewers is much more cutthroat than one would expect for a platform meant to unite gamers (and others). While it is simple to set up a stream, the hard part is getting people to watch you. There are certain unwritten pre-requisites which every channel who hopes to make it “big” has to pass.</p><ol><li>Be <strong>amazing</strong> at the game you are streaming</li><li>Have a entertaining personality so people will want to watch you</li></ol><p>Users need a reason to watch a stream. They want to get better at a game/character and see that you play that game/character at a very high level and will watch you to learn. The problem with people just starting off with streaming now is that it is hard to stand out from the crowd. There is typically already someone who is amazing at the game you are streaming and so you would need to cannibalize their viewers. This is not even considering the fact that there is a sharp distinction between being good and being amazing. Most gamers fall in the bucket of being good at a game. People don’t want to watch good, they want to witness something amazing.</p><p>This second part has given rise to a bunch of gimmicks/personalities on this platform such as Tyler1 and Dr.Disrespect.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgfycat.com%2Fifr%2Fincrediblenippyflyingfish&amp;url=https%3A%2F%2Fgfycat.com%2Fincrediblenippyflyingfish&amp;image=https%3A%2F%2Fthumbs.gfycat.com%2FIncredibleNippyFlyingfish-size_restricted.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=gfycat" width="418" height="242" frameborder="0" scrolling="no"><a href="https://medium.com/media/74f4bf471ddf83983c7a08bdc4ed0f5c/href">https://medium.com/media/74f4bf471ddf83983c7a08bdc4ed0f5c/href</a></iframe><p>People supplement the gap between their skill and the “best player streamer” with their character gimmick. While gimmick has a negative connotation to it, I am simply using it to mean a technique streamers use to stand out from the thousands of other streamers.</p><h3>Why Twitch?</h3><p>Amazon. On August 25, 2014, Amazon acquired Twitch for $970 million. Since that acquisition, they have enhanced the twitch experience by giving amazon prime members perks such as a free subscription (more on that below) as well as a catalog of free games/items in games. The biggest reason Amazon and other large, traditional companies are interested in this platform is the demographic of its users. According to a recent Goldman Sachs insight article, 79% of viewers are under 35 years old and the audience on YouTube and Twitch is larger than HBO, Netflix &amp; ESPN combined.</p><p>Since twitch is a platform dominated by games, trend analysis of consumers in this demographic can easily be tracked by publishers and developers in the gaming industry. This allows them to quickly capitalize on emerging trends and consumer behavior which led to the great ‘Battle Royale’ craze, where developers of the ‘Call of Duty’ and ‘Battlefield’ franchises along with numerous independent developers in the past year started pumping out Battle Royale modes in their games due to games like ‘Player Unknown’s Battlegrounds’ and ‘Fortnite’ blowing up in the media and gaming circles.</p><h3>Why Do People Want to Stream?</h3><p>Streaming has been put on a pedestal with the idea that “I can get paid to just play video games all day?” Reports of big time streamers such as Ninja earning as estimated $1M per <strong>month </strong>also feeds into the notion of it being an easy and lucrative gig.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2Fl0HUoSlpH5dvjXeeI%2Ftwitter%2Fiframe&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fl0HUoSlpH5dvjXeeI%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2Fl0HUoSlpH5dvjXeeI%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="241" frameborder="0" scrolling="no"><a href="https://medium.com/media/28a95cf19915cbcff34ba53e26fbb4f6/href">https://medium.com/media/28a95cf19915cbcff34ba53e26fbb4f6/href</a></iframe><p>Streamers can earn money through some of the following ways:</p><ol><li>Running ads on their channel (prone to ad block)</li><li>Viewers subscribing to their channel (viewer pays $5 minimum and streamers gets 50% to 80% of that amount)</li><li>Donations from viewers</li><li>Partnering with companies/developers to run promoted materials (i.e. drinking red bull on stream or playing an upcoming game to market it to viewers)</li></ol><p>Some may wonder why would anyone donate money to a streamer. While I don’t have a psychology background, I do surmise that viewers who really enjoy getting <strong><em>recognized</em></strong> by the streamer feel inclined to donate. Streamers typically have sound alerts on their stream where if you donate above a certain threshold, your donation along with your username and message to the streamer get read out loud. This in a way is like getting a direct interaction with the streamer whom the viewer may look up which would fuel their decision to give money to the streamer.</p><h3>Potential Data Analysis of Streams</h3><p>As an up and coming data scientist, there are definitely a lot of interesting questions I can look into using the Twitch API. Some of the items worth looking into include:</p><ol><li>You can take a deeper look into the sources of revenue of streamers by obtaining their subscriber counts over time</li><li>How a game is doing with its players after it has launched (games typically have a large viewership the first few days of launch but may fail to maintain these large numbers for a consistent period of time)</li><li>Training a machine to ‘watch’ all the streams and ‘listen’ for their donation sound alert and pulling the clip for that time period. By doing this you can obtain information on how much money they make through donations in a given time frame. This along with their subscribers can give a healthy estimate of a streamers revenue earnings.</li></ol><h3>Conclusion</h3><p>There are a lot of interesting insights and topics to delve into when it comes to streams and the Twitch experience. You can look at it from a business perspective with how successful a game launch is. Or you can look at it from a potential streamer perspective to identify trends of viewers of the platform to determine which game you want to focus on to maximize your market share of the viewers. You can also look at it from a psychologist’s perspective to try and understand why some streamers get more donations than others and why some viewers donate much more often that others.</p><p>This platform continues to grow each year and as esports becomes more mainstream, established platforms like Twitch will continue to thrive and provide valuable insights on a seemingly endless number of topics.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0d11d9b0bfb" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>