<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gary Schwaeber on Medium]]></title>
        <description><![CDATA[Stories by Gary Schwaeber on Medium]]></description>
        <link>https://medium.com/@gsstats?source=rss-2ff2a37ea872------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Z5hhTnft4tfBPAU7.jpg</url>
            <title>Stories by Gary Schwaeber on Medium</title>
            <link>https://medium.com/@gsstats?source=rss-2ff2a37ea872------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 15:37:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gsstats/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building An NHL Game Prediction Model, Part 2]]></title>
            <link>https://gsstats.medium.com/building-an-nhl-game-prediction-model-part-2-9932bf914ed?source=rss-2ff2a37ea872------2</link>
            <guid isPermaLink="false">https://medium.com/p/9932bf914ed</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[nhl]]></category>
            <dc:creator><![CDATA[Gary Schwaeber]]></dc:creator>
            <pubDate>Sat, 26 Jun 2021 21:11:12 GMT</pubDate>
            <atom:updated>2021-06-26T21:11:12.503Z</atom:updated>
            <content:encoded><![CDATA[<p>Data Acquisition</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T1ZE1lKks6eNoD1zHVd5VA.jpeg" /><figcaption>Courtesy of Taylor Friehl via Unsplash</figcaption></figure><h3><strong>Overview</strong></h3><p>In <a href="https://gsstats.medium.com/building-an-nhl-game-prediction-model-part-1-1ee7596ff91b">part one</a> of the series on creating an NHL game prediction model, I discussed my personal motivation and the application of the model to a betting strategy. In this post, I will go into how I got the data, cleaned it, and got it organized to be usable for modeling.</p><p>I wanted to start by making a top down model. This means I would be using team based stats as my features as opposed to building up a model from the stats of the players on the teams. The exception to this is the goalie. Goaltenders in hockey are considered to be the most important players on the ice. The play and quality of a goaltender can almost single handedly win a game for a team. For many teams, their starter and backup differ in quality significantly and who is starting will greatly impact the outcome.</p><p>In order to train the model I needed to get both the historical results as well as statistics that measure various elements of team and goaltending strength at that point in time. To get these point in time measurements, I would need game logs of statistics. Then, I would need to calculate statistics on a rolling game basis at every point in time.</p><h3><strong>Data Acquisition</strong></h3><p>Natural Stat Trick is one of the most popular websites hosting hockey stats. I had prior experience scraping data from NST, that I wrote about in a prior<a href="https://medium.com/nerd-for-tech/scraping-data-tables-from-natural-stat-trick-using-beautiful-soup-3251191bc3e1"> blog post</a>. Essentially I was able to use the Pandas read_HTML method to scrape NST. I had to do numerous scrapes to get team game logs with 5v5 data, powerplay data, penalty kill data, as well as scrape game logs for each individual goalie.</p><h4><strong>Team</strong></h4><p>The function for the scraping team game logs is below. The key stats coming from this function are Corsi, Fenwick, Goals, Expected Goals, Shots, and High Danger Chances per game.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c88bf7b86ea339d60d69fcd957af3480/href">https://medium.com/media/c88bf7b86ea339d60d69fcd957af3480/href</a></iframe><p>The nst_to_sched dictionary allows me to convert from the team name which comes from NST, to the team abbreviation which is was used when I later scrape the NHL API to get the official schedule and results.</p><p>The URL is dynamic. I can use this same function to scrape to different seasons and different situations ( 5v5, 5v5 score and venue adjusted, powerplay, and penalty kill).</p><p>I created a key by concatenating team abbreviation with the date. The keys are used to join the point in time statistics from this data frame with the correct game on the modeling data frame.</p><h4><strong>Goalie</strong></h4><p>I scraped game logs for individual goalies. Since the beginning of the 2016 season, there were approximately 150 different goalies who appeared in a game. Each goalie required one page request and only two seasons worth of games could be scraped at a time. With making such a large number of requests, I had to set a timer in between page requests so that the server could handle it and my IP would not get banned. I was able to get the goalie ids from scraping the NHL API. The goalie_ids dictionary contains the goalie name as the key and the goalie id from the NHL API as the value.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f73dee85291117e7d79d86353b08edc2/href">https://medium.com/media/f73dee85291117e7d79d86353b08edc2/href</a></iframe><h4><strong>ELO Ratings</strong></h4><p>I wanted to include ELO rating as one of the features for the modeling. ELO is a general measure of team strength based on the game results, score, and quality of opponent. ELO ratings are more commonly used in board games such as chess but can be adapted for the NHL. Fortunately, after many google searches, I was able to find someone who was calculating and hosting historical ELO ratings that I could import. Thank you to <a href="https://github.com/NeilPaine538/NHL-Player-And-Team-Ratings">Neil Paine</a> for providing the data for that feature.</p><h4><strong>Official Game Results</strong></h4><p>In looking for official game results from the NHL, I initially found it difficult to engage with the NHL API as I found no official documentation. I was able to find the python library hockey_scraper that allowed me to scrape the schedule for the seasons I wanted to model on. In this code I am also creating the target variable, whether the home team won, and the keys I used to merge the features from the data frames onto this data frame for modeling and analysis later.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/019ac3287eb027a1c3ec34dd75d4e811/href">https://medium.com/media/019ac3287eb027a1c3ec34dd75d4e811/href</a></iframe><p><strong>Feature Engineering</strong></p><p>After acquiring the team, goalie, ELO, and schedule data, I now have three data frames holding my features, and one holding the schedule where I will combine all my features for modeling.,</p><p>In the team and goalie data frames, one row represents one game where the columns are different stats occurring in that game. I needed a way to efficiently calculate sums and averages of various statistics for the prior X number of games for every team on a rolling basis.</p><p>With some help from stack overflow, I found the basis of code that can accomplish what I wanted to do. The below code sample will sum the FF (Fenwick For, aka unblocked shot attempts) for a team from the prior 40 games.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f24335f1462942b281fce0708d2d061a/href">https://medium.com/media/f24335f1462942b281fce0708d2d061a/href</a></iframe><p>Going through the code from left to right</p><p>Group by team</p><p>Then the transform will apply a function but maintain the same number of rows as what is before it</p><p>The lambda function is summing on a 40 game rolling basis</p><p>The .shift at the end will exclude the current row from the calculation. This is important because I do not want the stats from a game influencing the models ability to predict that game. I only want to know what the stats of the team were prior to them playing that game.</p><p>Similar code is applied to other team and goalie stats in order to generate the rolling statistics needed for modeling.</p><h3><strong>Combining It All Together</strong></h3><p>Now that I have the data for my features I need to combine the various data frames into one to be used for modeling. I use pandas merge function and the keys I created by concatenating the team abbreviation with the date to join the tables and import the features for the home and away teams.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/11a4c9cffe6e057ebc2c356c9d31cd6c/href">https://medium.com/media/11a4c9cffe6e057ebc2c356c9d31cd6c/href</a></iframe><h3><strong>Conclusion</strong></h3><p>In this blog, I displayed the various ways I was able to scrape the data, format it, calculate the features, and then combine into one data frame for modeling. Getting this code down was one of the more time consuming parts of the entire project. In the next post, I will do a deeper dive into the features I chose to use as well as how many rolling games ended up being most predictive. See the project <a href="https://github.com/gschwaeb/NHL_Game_Prediction">github</a> for all the code and results of the modeling. Thank you for reading and see you next time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9932bf914ed" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building An NHL Game Prediction Model, Part 1]]></title>
            <link>https://medium.com/analytics-vidhya/building-an-nhl-game-prediction-model-part-1-1ee7596ff91b?source=rss-2ff2a37ea872------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ee7596ff91b</guid>
            <category><![CDATA[nhl]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Gary Schwaeber]]></dc:creator>
            <pubDate>Fri, 18 Jun 2021 19:05:49 GMT</pubDate>
            <atom:updated>2021-06-30T09:00:27.588Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JOP7I3xzsdugkbmiD77_0A.jpeg" /><figcaption>Credit: Chris Liverani, Unsplash</figcaption></figure><p>For my data science capstone project at Flatiron School, I built an NHL Game Prediction model. Over my next couple blog posts I am going through the various steps and lessons I took in order to build the model. Now without further ado….</p><h3><strong>Motivation</strong></h3><p>I have always been a huge hockey fan, and more recently also became a big data nerd. But even before jumping into the world of data and pursuing it as a career path, I was always interested in reading the journalists who delve deep into the analytics of the game. Over the past decade, I have seen a greater acceptance by people to use advanced statistics as more meaningful ways to evaluate the game. So when it came to choosing my capstone project, working with NHL data felt natural. Inspired by Dom Luszczyszyn, a writer for The Athletic who created a game prediction model which is one of if not the best publicly published models out there, I decided to embark on building my own game prediction model.</p><h3><strong>Business Problem</strong></h3><p>Trying to solve some problem with data is at the heart of any data science project. The problem an NHL game prediction model solves comes down to the simple idea: Can I bet on hockey games more intelligently than gut intuition.</p><p>Unlike in Football or Basketball where betting against the spread is the most popular type of betting, the moneyline is king in the NHL due to lower scoring games. So you’re just betting on who will win the game regardless of how much a team won by. There is a different cost to betting on the underdog vs the favorite, and that cost is the moneyline. When betting the moneyline, the way to gain an edge is if you know the truer probability of the game outcome then the implied probability from the moneyline. Over the course of the season, if your internally derived game probabilities are superior to the bookmakers’, you can be profitable.</p><h4><strong>Calculating Implied Probability</strong></h4><p>Let’s take a look at this example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yf4PKSab92vrXDHOtARpgA.png" /><figcaption>Game line courtesy of DraftKings</figcaption></figure><p>Tonight the Vegas Golden Knights are playing the Montreal Canadiens. We can see from the moneyline column that Vegas is favored because their moneyline figure is negative. The -175 means you would have to risk $175 to win $100 profit. For the underdog Canadiens, the +148 figure indicates you need to risk only $100 to profit $148.</p><p>To calculate the implied probability of Vegas (the favorite) winning, the formula is:</p><p>175 / (100+175) = 63.6%.</p><p>To calculate the implied probability of Montreal (the underdogs) winning, the formula is:</p><p>100 / (148+100) = 40.3%.</p><p>You may have noticed that those probabilities add up to more than 100%. That’s because the books take a commission (aka Vig) on both sides of the bet. They require you to risk more $ (or receive less payout depending on how you look at it) to make your ‘investment’.</p><h4><strong>Applying The Model Output To A Betting Strategy</strong></h4><p>With all the above in mind, I needed to create a model that would output the probability of each team in the matchup winning. If the model outputs a probability of a team winning that was higher than the implied probability from the moneyline, you would bet on that team. Of course you can always bring some qualitative analyses into your decision making, but that is the general gist of how you would use the model.</p><p>When I got to the modeling stage of my project, I needed to use machine algorithms that could output probabilities. For this reason I tried logistic regression, AdaBoost, Gradient Boosting and a Neural Network to see which one would score best. The scoring metric I used to judge the models was log-loss. Log-loss is indicative of how close the prediction probability is to the corresponding actual/true value (0 or 1 in case of binary classification). The more the predicted probability diverges from the actual value, the higher is the log-loss value, <a href="https://towardsdatascience.com/intuition-behind-log-loss-score-4e0c9979680a">Source</a>.</p><h3><strong>Conclusion</strong></h3><p>In this post I went over my motivation for building an NHL game prediction model and the use for it. In later posts I will discuss how I got the data, cleaned it, engineered features, selected features, went through the modeling process, and provide some insights from the data and lessons I learned along the way. You can check out the Github repository for this project <a href="https://github.com/gschwaeb/NHL_Game_Prediction">here</a>. As of this writing, I am continuously working on fine tuning the model, and making the code more dynamic so that it’s ready to be rolled out for the 2021–2022 season.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ee7596ff91b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/building-an-nhl-game-prediction-model-part-1-1ee7596ff91b">Building An NHL Game Prediction Model, Part 1</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Cloud Computing is Important For Business]]></title>
            <link>https://gsstats.medium.com/why-cloud-computing-is-important-for-business-2962222c9dff?source=rss-2ff2a37ea872------2</link>
            <guid isPermaLink="false">https://medium.com/p/2962222c9dff</guid>
            <category><![CDATA[cloud-services]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[cloud-storage]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Gary Schwaeber]]></dc:creator>
            <pubDate>Sun, 11 Apr 2021 17:32:41 GMT</pubDate>
            <atom:updated>2021-04-13T19:17:13.445Z</atom:updated>
            <content:encoded><![CDATA[<p>What job-seeking data professionals can do about it</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SXerXUgs8gYc_iKLf4FL4g.jpeg" /><figcaption>Image Courtesy of <a href="https://unsplash.com/@jlcruz_photography">Jordan M. Lomibao</a> via Unsplash</figcaption></figure><h3>What is cloud computing?</h3><blockquote>Cloud computing is the delivery of computing services — including servers, storage, databases, networking, software, analytics, and intelligence — over the Internet (“the cloud”) to offer faster innovation, flexible resources, and economies of scale. —<a href="https://azure.microsoft.com/en-us/overview/what-is-cloud-computing/"> Microsoft Azure</a></blockquote><p>Many companies today are dealing in petabytes of data (1 petabyte is one quadrillion bytes). Walmart goes through 2 petabytes of data per hour. Can you imagine how much data a company like Facebook or Netflix goes through?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/1*r8gqd42Y3jafTxuRfR-2ew.jpeg" /><figcaption>Petabytes — humongous</figcaption></figure><h3>The Business Case for Cloud Computing</h3><p>Companies are increasingly turning to cloud computing to handle this large data flows. Data security and data modernization are two of the other biggest factors driving executive decision makers to drop legacy in-house data storage solutions and move their data solutions to third party cloud vendors.</p><p>Some of the benefits of cloud computing are:</p><p><strong>Cost</strong> — Companies can drop their expensive and fixed costs for hardware, software, and maintenance costs to run data centers. Expenses tend to be lower with third part cloud providers as they can utilize economies of scale. Business’ expenses will change from fixed to variable as they now only need to pay for what they use.</p><p><strong>Security</strong> –Cloud computing providers have greater expertise and cyber security infrastructures than many companies would otherwise be able to develop in-house, protecting their data from increasingly common cyber attacks and data leaks.</p><p><strong>Reliability</strong> — Cloud providers supply data backups and disaster recovery as data is mirrored at multiple redundant sites</p><p><strong>Flexibility</strong> — Companies have access to technology services such as storage or computing power on an as needed basis from third party providers. This give companies of various sizes opportunities to access the same data storage, computing power, and infrastructure as the biggest fortune 500 companies.</p><p><strong>Global Access —</strong> Cloud computing providers have global infrastructure allowing companies to access computing resources anywhere in the world they would need</p><p>Ultimately, cloud computing infrastructures services allow businesses to focus on their core goals without worrying about something that is not their area of expertise.</p><h3>Opportunities for Data Professionals</h3><p>With all that being said, as a data science student and job-seeker who has never worked with a business cloud service provider, but sees this requirement on numerous job postings, how would I get experience? Fortunately many of the top providers have certifications that allows one to learn how to engage with their services and gain professional credibility. I have outlined some summary information for certifications from the top 3 cloud infrastructure providers below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*duz12vh8A0Wk2NzyC0dQuA.png" /><figcaption>AWS, Microsoft Azure, and Google Cloud are top 3 in market share for cloud vendors.</figcaption></figure><h4><strong>AWS Certifications</strong></h4><p>AWS Certified Cloud Practitioner Certification is recommended for people who are newer to cloud computing. Other options include more advanced certifications such as Solutions Architect, SysOps Administrator, Developer, and DevOps Engineer and specialty certs: Advanced Networking, Security, Machine Learning, Data Analytics, and Database.</p><p><strong>AWS Certified Cloud Practitioner Skills Emphasized:</strong></p><ul><li>Basic global infrastructure</li><li>Basic architectural principles</li><li>Key services and common use cases (ex: compute and analytics)</li><li>Basic security and compliance aspects and the shared security model</li><li>Billing, account management, and pricing models</li><li>Identify sources of documentation or technical assistance (for example, whitepapers or support tickets)</li><li>Describe basic/core characteristics of deploying and operating in the AWS Cloud</li></ul><p><strong>Exam Cost:</strong> 100 USD</p><p><strong>Valid For:</strong> 3 Years</p><p><strong>More Info:</strong> <a href="https://aws.amazon.com/certification/certified-cloud-practitioner/?ch=cta&amp;cta=header&amp;p=2">https://aws.amazon.com/certification/certified-cloud-practitioner/?ch=cta&amp;cta=header&amp;p=2</a></p><h4><strong>Google Cloud Certificates</strong></h4><p>At the Associate level (6+ months recommended experience) there is a Cloud Engineer Certification. Then for more advanced Professional Level Certifications (1+ years recommended experience) they offer certifications in Cloud Architect, Cloud Developer, Data Engineer, Cloud DevOps Engineer, Cloud Security Engineer, Cloud Network Engineer, Collaboration Engineer, and Machine Learning Engineer.</p><p><strong>Associate Level Cloud Engineer Skills Emphasized:</strong></p><p>· Set up a cloud solution environment</p><p>· Plan and configure a cloud solution</p><p>· Deploy and implement a cloud solution</p><p>· Ensure successful operation of a cloud solution</p><p>· Configure access and security</p><p><strong>Exam Cost:</strong> 125$</p><p><strong>Valid For:</strong> 2 Years</p><p><strong>More info:</strong> <a href="https://cloud.google.com/certification">https://cloud.google.com/certification</a></p><h4><strong>Microsoft Azure</strong></h4><p>The entry level certification is for Azure Fundamentals. There is also Certificates in Azure AI Fundamentals and Data Fundaments as well as numerous options for more <a href="https://query.prod.cms.rt.microsoft.com/cms/api/am/binary/RE4wyqh">advanced certifications</a></p><p><strong>Azure Fundamentals Skills Emphasized:</strong></p><p>· Describe cloud concepts</p><p>· Describe core Azure services</p><p>· Describe core solutions and management tools on Azure</p><p>· Describe general security and network security features</p><p>· Describe identity, governance, privacy, and compliance features</p><p>· Describe Azure cost management and Service Level Agreements</p><p><strong>Cost:</strong> 100$</p><p><strong>Valid For:</strong> 2 Years</p><p>More Info: <a href="https://docs.microsoft.com/en-us/learn/certifications/">https://docs.microsoft.com/en-us/learn/certifications/</a></p><h3>Conclusion</h3><p>Hopefully you now understand why the cloud services and third party cloud infrastructure providers are critical for modern businesses and have some resources for how to gain experience and earn some credibility in this area.</p><h3>Sources</h3><p><a href="https://info.cobaltiron.com/blog/petabyte-how-much-information-could-it-actually-hold">https://info.cobaltiron.com/blog/petabyte-how-much-information-could-it-actually-hold</a></p><p><a href="https://azure.microsoft.com/en-us/overview/what-is-cloud-computing/">https://azure.microsoft.com/en-us/overview/what-is-cloud-computing/</a></p><p><a href="https://aws.amazon.com/what-is-cloud-computing/">https://aws.amazon.com/what-is-cloud-computing/</a></p><p><a href="https://www.pcmag.com/news/what-is-cloud-computing">https://www.pcmag.com/news/what-is-cloud-computing</a></p><p><a href="https://365datascience.com/trending/cloud-computing/">https://365datascience.com/trending/cloud-computing/</a></p><p><a href="https://towardsdatascience.com/im-a-data-scientist-why-should-i-use-the-cloud-b896ded55bb9">https://towardsdatascience.com/im-a-data-scientist-why-should-i-use-the-cloud-b896ded55bb9</a></p><p><a href="https://www2.deloitte.com/us/en/insights/industry/technology/why-organizations-are-moving-to-the-cloud.html">https://www2.deloitte.com/us/en/insights/industry/technology/why-organizations-are-moving-to-the-cloud.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2962222c9dff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Best Practices For Your Machine Learning Workflow in Scikit-learn]]></title>
            <link>https://gsstats.medium.com/best-practices-for-your-machine-learning-workflow-in-scikit-learn-2fe857bcb962?source=rss-2ff2a37ea872------2</link>
            <guid isPermaLink="false">https://medium.com/p/2fe857bcb962</guid>
            <category><![CDATA[scikit-learn]]></category>
            <category><![CDATA[data-model]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-leak]]></category>
            <dc:creator><![CDATA[Gary Schwaeber]]></dc:creator>
            <pubDate>Tue, 30 Mar 2021 23:52:03 GMT</pubDate>
            <atom:updated>2021-03-31T21:27:48.126Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kSw-njRsFwVlqbTElIX8oQ.jpeg" /><figcaption>Photo Credit: Kevin Wu via Unsplash</figcaption></figure><p>Over the course of my data science learning, I have had a ton of exposure to Scikit-learn. Scikit learn is one of the most popular machine learning libraries for Python. In this post, I will discuss some important practices to keep in mind when using the Scikit-learn library for building effective models.</p><h3>Keep Your Data Preprocessing Consistent</h3><p>Scikit-learn provides numerous libraries and methods for doing data transformations. Any Dataset transformations used when training a model, must also be used on test data or when deployed in production systems. For example, if you use their StandardScaler to scale the X train to fit your model, you must also use StandardScaler on the X_test when getting predictions or model metrics. Not applying consistent processing steps will result in unexpected results and cause your model to not perform effectively.</p><p>The below code block, courtesy of Sci-kit learn’s documentation shows the <strong>wrong</strong> way to predict on the test data without processing it in the same manner as the training data which was used to fit the model</p><pre><strong>&gt;&gt;&gt; from</strong> <strong>sklearn.metrics</strong> <strong>import</strong> mean_squared_error</pre><pre><strong>&gt;&gt;&gt; from</strong> <strong>sklearn.linear_model</strong> <strong>import</strong> LinearRegression</pre><pre><strong>&gt;&gt;&gt; from</strong> <strong>sklearn.preprocessing</strong> <strong>import</strong> StandardScaler</pre><pre><strong>&gt;&gt;&gt; </strong>scaler = StandardScaler()</pre><pre><strong>&gt;&gt;&gt; </strong>X_train_transformed = scaler.fit_transform(X_train)</pre><pre><strong>&gt;&gt;&gt; </strong>model = LinearRegression().fit(X_train_transformed, y_train)</pre><pre><strong>&gt;&gt;&gt; </strong>mean_squared_error(y_test, model.predict(X_test))</pre><pre>62.80...</pre><p>The below code block shows the <strong>correct</strong> way</p><pre><strong>&gt;&gt;&gt; </strong>X_test_transformed = scaler.transform(X_test)</pre><pre><strong>&gt;&gt;&gt; </strong>mean_squared_error(y_test, model.predict(X_test_transformed))</pre><pre>0.90...</pre><h3>Avoid Data Leakage</h3><blockquote>Data leakage is the use of information in the model training process which would not be expected to be available at prediction time, causing the predictive scores (metrics) to overestimate the model’s utility when run in a production environment.</blockquote><p>Data leakage will cause your model to appear to perform better than it should.</p><p>Here are some things to consider to avoid data leakage when using Scikit-learn:</p><p>· Never fit on test data.</p><p>· When performing feature selection algorithms, only use training data. Train, test split your data first, then do feature selection on the training data.</p><p>· If you use some average or other statistic to feature engineer, impute, scale, or otherwise pre-process your data, you should only be using a statistic based on the training subset, otherwise information from your training subset will be influencing the model.</p><p>· Be aware of/avoid using leaky predictors as features. You should not use variables that were updated after the target was realized. For example if you are building a model to predict if people are sick, you shouldn’t fit the model using the fact that people have taken medicine. People take medicine because they are sick, not the other way around (usually). Deploying that model on unseen data would result in inaccurate results!</p><p><strong>Wrong</strong></p><pre><strong>&gt;&gt;&gt; </strong><em># Incorrect preprocessing: the entire data is transformed</em></pre><pre><strong>&gt;&gt;&gt; </strong>X_selected = SelectKBest(k=25).fit_transform(X, y)</pre><pre><strong>&gt;&gt;&gt; </strong>X_train, X_test, y_train, y_test = train_test_split(</pre><pre><strong>… </strong>X_selected, y, random_state=42)</pre><p><strong>Right</strong></p><pre><strong>&gt;&gt;&gt; </strong>X_train, X_test, y_train, y_test = train_test_split(</pre><pre><strong>... </strong>    X, y, random_state=42)</pre><pre><strong>&gt;&gt;&gt; </strong>select = SelectKBest(k=25)</pre><pre><strong>&gt;&gt;&gt; </strong>X_train_selected = select.fit_transform(X_train, y_train)</pre><p>The above guidelines should be followed for cross-validation test and train folds as well</p><h3><strong>Pipelines</strong></h3><p>Scikit-learn has an object called Pipelines that will help you to avoid many of the above pitfalls from bad code. One of Core Developers himself recommend everyone should be using Pipelines when using Scikit.</p><blockquote>Pipelines allow you do encapsulate all the preprocessing steps, feature selections, scaling, encoding of variables, and so on together with the final supervised model that you usually have in a single estimator. -<a href="https://datascience.columbia.edu/andreas-mueller">Andreas Muller</a>, Core Developer of Scikit-learn</blockquote><p>Pipelines are especially useful when doing cross validation as it will prevent data leakage from the training to test folds and allow the cross validation scoring to provide a more accurate picture of model performance.</p><h3>Conclusion</h3><p>There you have it some important practices you should be following when using Scikit-learn so that your models can perform effectively on unseen data!</p><h3>Sources</h3><p><a href="https://scikit-learn.org/stable/common_pitfalls.html">https://scikit-learn.org/stable/common_pitfalls.html</a></p><p><a href="https://scikit-learn.org/stable/modules/compose.html#pipeline">https://scikit-learn.org/stable/modules/compose.html#pipeline</a></p><p><a href="https://drgabrielharris.medium.com/python-how-scikit-learn-0-20-optimal-pipeline-and-best-practices-dc4dd94d2c09">https://drgabrielharris.medium.com/python-how-scikit-learn-0-20-optimal-pipeline-and-best-practices-dc4dd94d2c09</a></p><p><a href="https://towardsdatascience.com/want-to-truly-master-scikit-learn-2-essential-tips-from-the-official-developer-himself-dada6ff56b99">https://towardsdatascience.com/want-to-truly-master-scikit-learn-2-essential-tips-from-the-official-developer-himself-dada6ff56b99</a></p><p><a href="https://en.wikipedia.org/wiki/Leakage_(machine_learning)">https://en.wikipedia.org/wiki/Leakage_(machine_learning)</a></p><p><a href="https://www.kaggle.com/dansbecker/data-leakage">https://www.kaggle.com/dansbecker/data-leakage</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2fe857bcb962" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What NHL Teams Are Under or Over Performing?]]></title>
            <link>https://gsstats.medium.com/what-nhl-teams-are-under-or-over-performing-b090035b811d?source=rss-2ff2a37ea872------2</link>
            <guid isPermaLink="false">https://medium.com/p/b090035b811d</guid>
            <category><![CDATA[statsmodels]]></category>
            <category><![CDATA[regression-analysis]]></category>
            <category><![CDATA[nhl]]></category>
            <dc:creator><![CDATA[Gary Schwaeber]]></dc:creator>
            <pubDate>Fri, 19 Mar 2021 00:55:43 GMT</pubDate>
            <atom:updated>2021-03-19T00:55:43.961Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zeDbNgCKH10d_iEhCMJhwg.jpeg" /><figcaption>Credit: <a href="https://www.gettyimages.com/search/photographer?family=editorial&amp;photographer=Bruce+Bennett">Bruce Bennett</a> / Staff</figcaption></figure><p>Former NFL coach Bill Parcells once wisely said — “you are what your record says you are”. Ultimately this is true. By the end of the NHL season, I think there is truly a large enough sample of games where every team deserves their place in the standings. But in short samples, sometimes teams may overperform or underperform their record. I want to take a look at some team level statistics and see what correlates highly with a team’s record (given the weirdness of the NHL point system where the loser point exists for OT losses, a team’s record = their points %). Then I want to take a look at the current season as of 3/18/2021, where most teams are at around the 30 game mark, and see if I can spot any teams that are outperforming or underperforming their record.</p><p>I’ll start by grabbing data from Natural Stat Trick. If you are curious how I do this, I have another blog post about how to scrape Natural Stat Trick <a href="https://medium.com/nerd-for-tech/scraping-data-tables-from-natural-stat-trick-using-beautiful-soup-3251191bc3e1">here</a>. . I’m taking team stats from the 2013–2014 season up to last season. I will compare CF%, FF%, SF%, ‘GF%’, ‘xGF%’, ‘SCF%’, ‘SCSF%’, ‘HDCF%’,’HDSF%’, ‘SH%’, ‘SV%’, ‘PDO’. If you are unfamiliar with any of these stats, check out the glossary on <a href="https://www.naturalstattrick.com/glossary.php?teams">Natural Stat Trick</a>.</p><p>Onto the analysis! Looking at the correlation heatmap below, the top metrics that correlate with team record are GF%, PDO, SH%, and then xGF%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*NWckwgnQT12wk9aC-xx0QA.png" /></figure><p>It’s no surprise that GF% is so highly correlated with record given that a game result is a function of goal differential of 1 game and GF% is a function of goal differential across multiple games.</p><blockquote><strong>GF%</strong> - Percentage of total Goals in games that team played that are for that team. <em>GF*100/(GF+GA)</em></blockquote><p>xGF% is the best metric that does not include actual goals scored in its calculation. xGF% gauges the scoring opportunity generation and suppression of the skaters independent of goaltending ability for either team. The calculation for xGF% is derived from a model. Information for this metric can be found <a href="https://rpubs.com/evolvingwild/395136/">here</a>.</p><blockquote><strong>PDO</strong> — Shooting percentage plus save percentage. <em>(GF/SF)+(GA/SA)</em></blockquote><p>PDO is very strongly correlated with Point %. There is some debate as to whether PDO is luck based and thats team should regress to 100. In my opinion teams can sustain PDO above or below 100 depending on goaltending skill and some degree of shooting ability or shot selection. However, given the max and min season PDO for the period measured of 104 and 96 accordingly, it’s probable a team that has a PDO far outside that range will regress into that range over the course of the season. PDO is normally distributed with a mean of 1. There is only a .16% probability of a team finishing the season outside the range of .96–1.04 which is 3.15 standard deviations away from the mean in both directions.</p><p>I regressed GF% and against Points %. I also combined PDO and xGF% in a multiple linear regression model. It makes sense to combine these two given they measure difference components of team success and PDO has a very low correlation with xGF%, thus avoiding any multicollinearity issues.</p><p>Using stats model library in Python to fit and evaluate the data. I can see GF% has an R² .93 explaining 93% of the variability in Points %. The PDO and xGF% model has an R² of .89, so these features together explain 89% of the variability in Points %.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*ElDKam-pXF8FEcacWPvePg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*ECmsw7tMe2YFEG0vzoEXfw.png" /><figcaption>OLS Regression results for GF%(left) and PDO+xGF%(right) against Point %</figcaption></figure><p>Fitting the coefficients onto data from the current season, and by looking at the difference between actual points and points inferred from the model (AKA the residuals), I can see which teams records maybe don’t reflect their performance completely.</p><p>Based on GF%, Rangers, Kings, and Stars, standout as have a worse record than expected and the Avalanche and Panthers stand out as having a better record than expected. (Though of note, the Rangers thrashed the Flyers 9–0 last night, so it’s possible if I wrote this a few days ago they wouldn’t seem such a steep under-performer)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xSE76cTcES3yUKd1EzuC2A.png" /></figure><p>Looking at the residuals for the multiple regression on xGF% and PDO, the Rangers and Stars again show up as big under-performers while the Blue are the only overperformer that stands out though no team is overperforming by more than 10% points.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4ZZlCdycp0ne8PomvcsePg.png" /></figure><p>What conclusions can be drawn from this? I think the strongest evidence points to the Rangers and Stars (both at .500 Points %) are better than their records would indicate. Likely, if they performed to the same level rest of season, they would outperform their current Points %. Both are on the outside of the playoffs in their division, but I wouldn’t be surprised if either can make up enough ground to challenge for a spot by the end of season barring a significant roster change.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b090035b811d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scraping Data Tables from Natural Stat Trick Using Beautiful Soup]]></title>
            <link>https://gsstats.medium.com/scraping-data-tables-from-natural-stat-trick-using-beautiful-soup-3251191bc3e1?source=rss-2ff2a37ea872------2</link>
            <guid isPermaLink="false">https://medium.com/p/3251191bc3e1</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[beautifulsoup]]></category>
            <category><![CDATA[nhl]]></category>
            <category><![CDATA[advanced-analytics]]></category>
            <category><![CDATA[corsi]]></category>
            <dc:creator><![CDATA[Gary Schwaeber]]></dc:creator>
            <pubDate>Sat, 06 Mar 2021 20:55:30 GMT</pubDate>
            <atom:updated>2021-03-14T03:06:21.527Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PTXg3InjO1FBUpm2" /><figcaption>Credit: <a href="https://www.gettyimages.com/search/photographer?family=editorial&amp;photographer=Bruce+Bennett">Bruce Bennett</a> / Staff</figcaption></figure><p>In this post, I will demonstrate how to scrape data from tables on <a href="http://naturalstattrick.com/">Natural Stat Trick</a>, a website for getting standard and advanced NHL statistics. Natural Stat Trick is a reputable and frequently referenced data source for NHL analytics used by writers and fans alike.</p><p>First, using the Beautiful Soup library, I’ll go through the steps to scrape the html and convert the data to a Pandas DataFrame. Then, I’ll show a convenient one liner that works for tables on Natural Stat Trick and will work for tables on other webpages depending on their complexity. Finally, a tip on altering the URL to change the parameters of the data you can get.</p><p>Here is the <a href="https://github.com/gschwaeb/scraping_naturalstattrick">github repo</a> with a jupyter notebook with all the below code.</p><h3>Beautiful Soup Step By Step</h3><p>Import the below libraries</p><pre>import requests<br>from bs4 import BeautifulSoup<br>import pandas as pd</pre><p>The requests.get() method will download a webpage and return a response object</p><pre>url = “<a href="http://www.naturalstattrick.com/playerteams.php?fromseason=20202021&amp;thruseason=20202021&amp;stype=2&amp;sit=5v5&amp;score=all&amp;stdoi=oi&amp;rate=n&amp;team=ALL&amp;pos=S&amp;loc=B&amp;toi=0&amp;gpfilt=none&amp;fd=&amp;td=&amp;tgp=410&amp;lines=single&amp;draftteam=ALL">http://www.naturalstattrick.com/playerteams.php?fromseason=20202021&amp;thruseason=20202021&amp;stype=2&amp;sit=5v5&amp;score=all&amp;stdoi=oi&amp;rate=n&amp;team=ALL&amp;pos=S&amp;loc=B&amp;toi=0&amp;gpfilt=none&amp;fd=&amp;td=&amp;tgp=410&amp;lines=single&amp;draftteam=ALL</a>&quot;</pre><pre>req = requests.get(url)</pre><p>To confirm it worked with no errors, we can run the .status_code method. We are looking for this to return 200 which means we are all good. <a href="https://en.wikipedia.org/wiki/List_of_HTTP_status_codes">HTML Codes</a>.</p><pre>req.status_code</pre><p>Here we are passing the text of the html from the request into a Beautiful Soup object so we can use the Beautiful Soup library to parse it.</p><pre>soup = BeautifulSoup(req.content)</pre><p>We use the .children method to work through the levels of child nodes in the html. We’ll pass it to a list to access different levels. For this webpage there is only 1 child at the top level</p><pre>top = list(soup.children)</pre><p>At the second level, there are two children: header and body. The contents of the table will be in the body which is found at index 1.</p><pre>body = list(top[0].children)[1]</pre><p>Use the .find_all method to filter by all the data that has a specified html tag and put it into a list. We want all the ‘th’ tags to get the table headers.</p><pre>body.find_all(‘th’)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*ythVuyUdH02LnoQ0ShZ2Iw.png" /><figcaption>body.find_all(‘th’) return statement</figcaption></figure><p>The .text method will extract the text within the tags. We’ll pass the columns text into a list to use later.</p><pre>columns = [item.text for item in body.find_all(‘th’)]<br>columns</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*oA2njj_8y4zdHKAcpwcchg.png" /><figcaption>columns return statement</figcaption></figure><p>Now we’ll find all the ‘td’ tags where the data is contained</p><pre>body.find_all(‘td’)</pre><p>And we’ll pass the text of the data into a list</p><pre>data = [e.text for e in body.find_all(‘td’)]</pre><p>Here we will loop through the list of data to create a list of lists for each row of data in the table. Later we’ll use this to create a pandas DataFrame.</p><pre>start = 0<br>table= []<br>#loop through entire data<br>while start+len(columns) &lt;= len(data):<br>    player = []<br>    #use length of columns as iteration stop point to get list of info for 1 player <br>    for i in range(start,start+len(columns)):<br>        player.append(data[i])<br>    #add player row to list<br>    table.append(player)<br>    #start at next player<br>    start += len(columns)</pre><p>Read the list of lists to a Pandas DataFrame</p><pre>df = pd.DataFrame(table, columns = columns, dtype = ‘float’).set_index(‘’)</pre><p>And VOILA! We’ve got a Pandas Dataframe we can work with to do some analysis.</p><pre>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CMwIALw2zfwcPAYfbCbruA.png" /></figure><h3>The One Liner</h3><p>Now as mentioned, we can do all that scraping, searching and transforming to a DataFrame in one concise line of code! Pandas .read_html method reads HTML tables into a list of DataFrame objects and it’s built on BeautifulSoup4 and lxml. It can also cast the columns into the appropriate data types.</p><pre>df2 = pd.read_html(url, header=0, index_col = 0, na_values=[“-”])[0]<br>df2.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tj6mcgWski730wLyat7Bmg.png" /></figure><p>Much easier!</p><h3>A Note on URL Parameters</h3><p>By altering elements in the URL that you pass in the code, you can change what data comes back in the table. This allows for dynamically pulling different types of data. For example, you can switch to different seasons, dates, look at goalie stats, look at rates instead of counts, filter by different game situations etc…</p><p>I’ve pulled out all the potential parameters options from the URL as their own variables. Check out the Github notebook for a full guide on the parameter options. Using the python method .format, I can change the parameters in the URL string. Compared to the URL string I was previously working with above, I’ll change the stdoi parameter to ‘g’ to get goalie specific stats to demonstrate</p><pre>fromseason = 20202021</pre><pre>thruseason = 20202021</pre><pre>stype = 2 </pre><pre>sit = ‘5v5’</pre><pre>score = ‘all’ </pre><pre>stdoi = ‘g’ </pre><pre>rate = ‘n’</pre><pre>team = ‘ALL’ </pre><pre>pos = ‘S’ </pre><pre>loc = ‘B’ </pre><pre>toi = 0 </pre><pre>gpfilt = ‘none’ </pre><pre>fd = ‘’ </pre><pre>td = ‘’ </pre><pre>lines = ‘single’ </pre><pre>draftteam = ‘ALL’ </pre><pre>url = ‘<a href="http://www.naturalstattrick.com/playerteams.php?fromseason={}&amp;thruseason={}&amp;stype={}&amp;sit={}&amp;score={}&amp;stdoi={}&amp;rate={}&amp;team={}&amp;pos={}&amp;loc={}&amp;toi={}&amp;gpfilt={}&amp;fd={}&amp;td={}td&amp;tgp={}&amp;lines={}&amp;draftteam={}&#39;.format(">http://www.naturalstattrick.com/playerteams.php?fromseason={}&amp;thruseason={}&amp;stype={}&amp;sit={}&amp;score={}&amp;stdoi={}&amp;rate={}&amp;team={}&amp;pos={}&amp;loc={}&amp;toi={}&amp;gpfilt={}&amp;fd={}&amp;td={}td&amp;tgp={}&amp;lines={}&amp;draftteam={}&#39;.format(</a><br> fromseason,<br> thruseason,<br> stype, <br> sit,<br> score, <br> stdoi, <br> rate, <br> team, <br> pos, <br> loc, <br> toi, <br> gpfilt, <br> fd, <br> td, <br> tgp, <br> lines, <br> draftteam)</pre><pre>print(url)</pre><pre>df3 = pd.read_html(url, header=0, index_col = 0, na_values=[&quot;-&quot;])[0]<br>df3.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qQQyl_HQehoo-p3KssTFTA.png" /></figure><h3>Conclusion</h3><p>You now have the tools to grab data from the tables on Natural Stat Trick and other similar websites. Give pd.read_html a shot first and if that doesn’t work you can go step by step with the Beautiful Soup library. You can also dynamically update the URL parameters to get different types of data.</p><p>Although the purveyor of Natural Stat Trick allows access to the data to the public for free, if you do find yourself using the data frequently, consider subscribing to the <a href="https://www.patreon.com/naturalstattrick">Patreon</a> to support the continued operations of the site.</p><h3>Sources</h3><p>- <a href="http://www.naturalstattrick.com/">http://www.naturalstattrick.com/</a><br>- <a href="https://requests.readthedocs.io/en/master/user/quickstart/">https://requests.readthedocs.io/en/master/user/quickstart/</a><br>- <a href="https://theathletic.com/415611/2018/07/05/an-advanced-stats-primer-with-naturalstattricks-brad-timmins/">https://theathletic.com/415611/2018/07/05/an-advanced-stats-primer-with-naturalstattricks-brad-timmins/</a><br>- <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/">https://www.crummy.com/software/BeautifulSoup/bs4/doc/</a><br>- <a href="https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#reading-html-content">https://pandas.pydata.org/pandas-docs/stable/user_guide/io.html#reading-html-content</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3251191bc3e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>