<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Michael Dendinger on Medium]]></title>
        <description><![CDATA[Stories by Michael Dendinger on Medium]]></description>
        <link>https://medium.com/@m-dendinger19?source=rss-74b1064c6077------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*Vg-yKxznY_Lx-chN1YbpPw.jpeg</url>
            <title>Stories by Michael Dendinger on Medium</title>
            <link>https://medium.com/@m-dendinger19?source=rss-74b1064c6077------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 16:58:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@m-dendinger19/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Team That AI Built: The Finale]]></title>
            <link>https://m-dendinger19.medium.com/the-team-that-ai-built-the-finale-1744e58be0c7?source=rss-74b1064c6077------2</link>
            <guid isPermaLink="false">https://medium.com/p/1744e58be0c7</guid>
            <category><![CDATA[nfl]]></category>
            <category><![CDATA[sports-analytics]]></category>
            <category><![CDATA[fantasy-football]]></category>
            <category><![CDATA[football]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Michael Dendinger]]></dc:creator>
            <pubDate>Thu, 06 Jan 2022 22:01:31 GMT</pubDate>
            <atom:updated>2022-01-06T22:01:31.097Z</atom:updated>
            <content:encoded><![CDATA[<p>Now that the fantasy football season is over, I can write the resolution of my AI picked fantasy team. <a href="https://m-dendinger19.medium.com/the-team-that-ai-built-b8469b562cfc">Here</a> is a quick reminder of how my team was drafted and what it looked like in the first weeks of the season.</p><p>From the beginning of the season to the end, this is how my team looked at both points:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*vHWb1VOOoSIyYzbOLXibsg.png" /></figure><p>I was required to cut a few players like JuJu had a season ending injury and the Bills D had a good season, but I dropped them as I generally hot-swap my defenses every week. Overall, my team stayed as consistent as I could throughout the season.</p><p>At the beginning of the season, ESPN rated my team as a failure and projected that I would finish 10th out of 10 in the league. At the end of the regular season, I finished 4th!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*JTb04LS9ZlitbPYkW6IsmQ.png" /><figcaption>I am the team on the very bottom.</figcaption></figure><p>Overall, I was very proud of this outcome. I do want to see how my team as the independent functioned against the other teams in my league as well as how it performed in relation to my initial goal. I wanted a team that had high average scores with a low variance between their scores.</p><p>If we look at this, we can see the weekly scores of my team as it compares to the other teams:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4OzI-b6chmqix7-4Gs9L8A.png" /></figure><p>Just from this visual, I did not perform a low variance team. At my highest, had 141.62 points and my lows was 45.58 points. That is a 96.04 difference in scores, not low variance at all.</p><p>To see how my team’s running average score compared to my scores, I made this visualization as well:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*1Bq6cUl2pnVoo14ERyFBQA.png" /></figure><p>My teams moving average did stay fairly consistent throughout the season with some drops from weeks 9 and 10.</p><p>This is how my team compared to all teams with their running averages:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*e-j0aaeC0uOJdlTxnKovZw.png" /></figure><p>As we can see here, even though my team was built to have a high running average, I had the 5th highest running average. However, comparing my score just to my league’s scores could be misleading. Our league was pretty evenly split as far as records went this season with three teams that were 7–7 and two that were 6–8. I was one of the 7–7 teams. Our league performed, as a whole, fairly evenly.</p><h3>Conclusion</h3><p>I had a consistent team from the team I drafted, many players performed spectacularly, and I had pretty good luck with my defense throughout the season. So, why did my team not do better?</p><h4>Age</h4><p>I did not include age in my model for this task. Looking back, I probably should have. Many players will peak in their mid season years or earlier and then decline as they age. While some players can continue to perform, it is an outlier. Tom Brady is an outlier in many aspects, but having an injury prone O-Line can still cause him issues by him getting hit more often.</p><h4>Team Diversity</h4><p>My star performers on average were Tom Brady and Leonard Fournette. While it was great to have them most weeks, when there was a Bye week or one of them was having an off game, it impacted both players negatively. I probably should have included some way to keep start players on different teams, as to avoid this issue.</p><h4>Depth</h4><p>Many of my players played through most of the season, but I did not have good solutions for when start players were injured. Melvin Gordon and Jarvis Landry were injured for large parts of the season, but there were no viable replacements on the waiver wire for them at a given time. This would be a hard thing to fit into a model, but definitely worth looking into.</p><h4>Waiver Wire</h4><p>I tried to keep my team as true to form as I could throughout the season, as I wanted to have an accurate representation of this drafted team across the whole season. In the future, I should try to do a weekly analysis to find players who could have big weeks to replace injured players. This would be an analysis similar to what Watson does for each player with “Boom/Bust”.</p><p>Overall, I am satisfied with my outcome. I do want to hear what you think of this team and how it performed. Let me know what you think!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1744e58be0c7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Team that AI Built]]></title>
            <link>https://m-dendinger19.medium.com/the-team-that-ai-built-b8469b562cfc?source=rss-74b1064c6077------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8469b562cfc</guid>
            <category><![CDATA[football]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[fantasy-football]]></category>
            <category><![CDATA[sports-analytics]]></category>
            <category><![CDATA[espn]]></category>
            <dc:creator><![CDATA[Michael Dendinger]]></dc:creator>
            <pubDate>Wed, 15 Sep 2021 16:51:01 GMT</pubDate>
            <atom:updated>2021-09-15T16:51:01.177Z</atom:updated>
            <content:encoded><![CDATA[<h4>Draft and Week 1</h4><p>I am sure that everyone has seen me talking about my Sisyphus-esque task of creating an AI that could build me a fantasy team that would be able to win my league. Like Viktor Frankenstein, I believe I have finally created my monster to do so.</p><p>I will not share the exact code used to create that team, but I have shared the pre-processing and data engineering task that it took to create the data set that was the foundation for this work <a href="https://medium.com/analytics-vidhya/football-prediction-e98e7dff122d">here</a>.</p><h3>Data Strategy</h3><p>In my hands were 54 years of offensive performance data on players, I started my analysis. The first task I performed was creating a correlation chart on all of the performances of players overall and I got a correlation chart that is too big for Medium to post it on the page.</p><p>From here I was able to create some plans for my team creation. My final decision came down to focusing on non-volatile players (in a performance standard). This means that I created a model that would prioritize players who had high average scores with a low variance of score breakdown. So, for example, this is what Aaron Rodgers looks like in the analytic:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/358/1*zmppOcwZnaz4oI8pbQGdsw.png" /></figure><p>This would mean that Aaron Rodgers average fantasy score is 18.755 points with a variance of 8.568 points. If you are familiar with standard deviation, you will know that the variance is what the step of a standard deviation is for this player. This means that 50% of the time Aaron Rodgers will score between 10 and 26 points. While this isn’t a great insight, it does allow for us to know how confident we can be in a player. There was a little more work done as to relate to time series and trends up and down, but this was more or less the framework for the process.</p><h3>The Output</h3><p>Some of those reading this may be ahead of me a bit when they saw how I was going to build this and see some flaws already. From the process I used, my model created what I am going to call a “Reliability Score”. This represents how reliable a player is and how close they will be to their average score. This score gave me a document of all players who have played in the NFL for at least one season (so no rookies) and led me through my draft. The team that my model told me to get was the following.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/491/1*G9-yLNlXMWiliU40DT_ZOA.png" /></figure><p>I have gotten pretty much the same response from every single person I have shown this team to, “That is a very reliable team.” For those who may not follow football quite as closely, that basically means I don’t have any players who can be ‘explosive’ on the field. They show up, do their work, and go home. All are solid players, but none are outrageous. Russel Wilson as a first round pick is quite suspicious and Melvin Gordon as a second rounder is pretty much insane by most drafting strategies.</p><p>Without too much surprise, my team was projected to lose the league by an IBM predictor that someone else used to review our draft.</p><h3>Week 1</h3><p>My week 1 was very stressful as I was only projected to get around 81 points with this team and the team I was playing was projected to score above 100 points. That is usually a pretty significant death sentence in most fantasy scenarios.</p><p>However, if any of you watched last week’s games, I did pretty well with my team. The model I have for picking players to play chose this team and they did pretty well:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/1*Npm7jTjUH6ECa45XERzevg.png" /></figure><p>Now, most of these players scored pretty closely within their average and did get me the win! My opposition did choose to not start a defense though, so I very well could have lost this had a defense been played.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8469b562cfc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Football Prediction]]></title>
            <link>https://medium.com/analytics-vidhya/football-prediction-e98e7dff122d?source=rss-74b1064c6077------2</link>
            <guid isPermaLink="false">https://medium.com/p/e98e7dff122d</guid>
            <category><![CDATA[football]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[tensorflow]]></category>
            <dc:creator><![CDATA[Michael Dendinger]]></dc:creator>
            <pubDate>Tue, 02 Mar 2021 15:47:27 GMT</pubDate>
            <atom:updated>2021-03-02T16:22:58.796Z</atom:updated>
            <content:encoded><![CDATA[<h4>The What and the Where of Advanced Football Analytics Through Machine Learning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AtMlS80iXSBNssLjK6AQ4g.png" /><figcaption><a href="https://www.theringer.com/nfl/2018/12/19/18148153/nfl-analytics-revolution">https://www.theringer.com/nfl/2018/12/19/18148153/nfl-analytics-revolution</a></figcaption></figure><h3>The Beginning</h3><p>In July of this year I vowed to win my fantasy football league. I wanted to create an algorithm that would give me insights on how each player would be predicted to perform on a weekly basis. In the past, I have been burned by ESPN and Yahoo’s predictions, so I set out to create my own. For the first step, I just wanted to create an algorithm that could fairly accurately predict which NFL team would win their game that week.</p><h3>The Data</h3><p>For this topic, I was going to need a lot of data. After reading <a href="https://www.theringer.com/nfl/2018/12/19/18148153/nfl-analytics-revolution">some</a> <a href="https://www.advancedfootballanalytics.com/index.php/home/research/general/199-ranking-teams-with-optimization">articles</a> <a href="https://fivethirtyeight.com/methodology/how-our-nfl-predictions-work/">on</a> predictive football analytics and found that there were a lot of things that were easy to get: game scores, player performance, and game location. However, there is a lot of difficult metrics to find good historical data for such as: history of injuries, complete weather data, and game attendance. So, I needed to do research on what data was available for free and could be scraped in an acceptable amount of time.</p><p>On my journey to find the data, I first went to <a href="https://www.kaggle.com/">Kaggle</a>. Kaggle is known among the data science community as a hub for stored data sets on hundreds to thousands of topics. Luckily, I found a <a href="https://www.kaggle.com/tobycrabtree/nfl-scores-and-betting-data">data set</a> that had the score, location, and date of every NFL game from 1966 to the beginning of this season.</p><p>While scores are all and good, we need more than just that to create a good predictive model. As scores are impacted by the players on the field, I wanted to bring that into the analysis as well. To do this, I had to scrape <a href="https://www.pro-football-reference.com/">Pro Football Reference</a> to get the<a href="https://github.com/dendi19/FootballAnalytics/blob/main/BoxScores.ipynb"> offensive box scores</a> for every one of these games as well.</p><h3>The Metrics</h3><p>Now that I had my box score data for every offensive player from 1966, I needed to create a way for those analytics to be useful in predictive performance. To do this, <a href="https://github.com/dendi19/FootballAnalytics/blob/main/PerfEnrich.ipynb">I made it so</a> that every offensive metric would be aggregated over the season for a season average as well as a metric for the last week’s performance.</p><p>Now, I had 68 metrics to go over for my analysis, but there were still a few key factors I needed to include in this data. I needed the weather data, whether a team won or lost, and how that team as a whole performed and not just the player. To do this I aggregated all of those performance metrics and enriched the data with those team and location metrics.</p><p>So at the end of this, I had these metrics:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*wHR3fPpFqfxdLZB7N61UIQ.png" /></figure><p>Now to the actual crux of this process, the analysis and prediction. For the analysis of this work, I wanted to see how it looked in ArcGIS Insights. However, looking at all occurrences of offensive players (234,720) with all of the metrics (68) over 54 years, was a bit more than Insights online or on Desktop could effectively handle and it was going to be a bit more legwork for the Machine Learning process to function at the scale in which I wanted.</p><p>Once again, the goal of the future ML process was to predict a winner based on the team performance, so I don’t need a data set with all players anyway. To do this I ran an aggregation that found the average of every player’s performance and showed that as the teams performance. So if there were three running backs and they had 5, 6, and 12 carries respectively the team would have an average performance of 7.6 carries in the table.</p><p>For analysis purposes, I left this final table with team performance with win/loss data as one table. In a second table, I added latitude, longitude, wind, temperature, and elevation. I wanted to see how GIS plays in with learning how these teams perform.</p><h3>The Machine Learning</h3><p>For my purpose, I had to weigh how I was going to do my analysis: <a href="https://en.wikipedia.org/wiki/Logistic_regression#:~:text=Logistic%20regression%20is%20a%20statistical,a%20form%20of%20binary%20regression).">Regression</a>, <a href="https://en.wikipedia.org/wiki/Neural_network">Neural Networks</a>, <a href="https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm">K-Nearest</a>? For my purpose, due to time and personal mathematical limitations, I chose to go with the logistic regression process. This was chosen for this specific purpose due to the fact that I was predicting win/loss or 1/0. In a logistic regression, the model finds the likelihood that an event exists or does not exist. In Neural Networks and K-Nearest analysis, there is more nuance to the answers and they require things like <a href="https://en.wikipedia.org/wiki/Sigmoid_function">sigmoid functions</a> and heavy mathematics that I did not have the time or ability to ration out in an “academically appropriate” way.</p><p>When making the tables for the analysis, I had to create the training and evaluation tables for the analysis. In this specific case, I chose to split the data between 70% training and 30% evaluation. This means that the logistic regression function within <a href="https://www.infoworld.com/article/3278008/what-is-tensorflow-the-machine-learning-library-explained.html">Tensor Flow</a> will look at the 70% to try and predict what is happening in the 30%. Most cases suggest you have a minimum of 10,000 occurrences to 1,000 predicted events, but for the sake of this demo, I decided to do a 70/30 split to look at more general numbers and since there are only a few new records per year.</p><p>Once I had this split, I converted the string values of the team names into numeric values for the analysis to run effectively. This is because a <a href="https://en.wikipedia.org/wiki/Natural_language_processing">non-NLP</a> module in Tensor Flow cannot appropriately read string values.</p><p>Now that the data cleaning was done, I needed to turn the training and evaluation data sets into matrices. As most machine learning is a mix between high level statistics and heavy calculus, the data cannot be in a simple table form. Luckily, Tensor Flow has some fairly simple functions to convert the tables into matrices.</p><p>Once this was done I built the actual machine learning code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O409BQmDpWqPwyLz4-x5_g.png" /></figure><p>This looks like it may be confusing, but basically what this is doing is taking all of that data engineering work we did before and iterating through the data to find trends. A couple of important pieces to this that show how this functions are the num_epochs and batch_size. In this, Epochs refer to how many times you look at the evaluation data compared to the training data. So, in this case it is looking through the evaluation data 15 times to make sure we have looked at it enough times. The batch size refers to how many items are in an analysis every time the function runs. In this example, a batch contains 320 events that are looked at. With the epoch and batch is split into pieces of 320 batches and each batch is looked at 15 times.</p><h3>The ML Output</h3><p>Now for the point you all have been waiting for! When we run our analysis on our non-geoenriched data, we get an accuracy of 64% with this model. That means that 64% of the time this model can predict whether or not a football team is going to win the game. Now, for such a complicated sport with a lot of metrics, this is a big deal! However, we are a GIS company and we want to know the impact of the WHERE. Now, I ran the same ML model that I used without the GIS data and I got a pretty awesome output of 72% accuracy!</p><p>So, long story short, even a machine learning non-GIS minded model shows the importance of the where when it comes to predicting sports outcomes (or anything really).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/413/1*4U8tS6NIVdemKfIgbBdMKA.png" /></figure><h3>Potential Future Analysis</h3><p>Now, knowing the impact that GIS has in this type of work as well as player performance there are a lot of other ways to test this model and make it more robust.</p><ul><li>Find impact in the change of location by team</li><li>Develop a <a href="https://en.wikipedia.org/wiki/Monte_Carlo_method">Monte Carlo Simulation</a> to get a better idea of how each metric is important</li><li>Aggregate Career performance by each player</li><li>Player Age data</li><li>Weighted performance in play-offs</li><li>Weighted performance in Superbowls</li><li>Injury data</li><li>Defensive data</li><li>Collegiate performance</li><li>And a whole lot more…</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e98e7dff122d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/football-prediction-e98e7dff122d">Football Prediction</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scrape ESPN Fantasy Data with Selenium]]></title>
            <link>https://m-dendinger19.medium.com/scrape-espn-fantasy-data-with-selenium-4d3b1fdb39f3?source=rss-74b1064c6077------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d3b1fdb39f3</guid>
            <category><![CDATA[fantasy-football]]></category>
            <category><![CDATA[sports-analytics]]></category>
            <category><![CDATA[selenium]]></category>
            <category><![CDATA[espn]]></category>
            <category><![CDATA[web-scraping]]></category>
            <dc:creator><![CDATA[Michael Dendinger]]></dc:creator>
            <pubDate>Thu, 03 Dec 2020 23:00:15 GMT</pubDate>
            <atom:updated>2020-12-28T17:10:48.103Z</atom:updated>
            <content:encoded><![CDATA[<p>When it comes to scraping data from a fantasy league to an analytical framework, ESPN does not make it easy. If you have found this article, you probably have also found that there used to be an API for ESPN that made data collection easy, but today we need to do a lot more legwork. We will need to go about this in several ways. Normally scraping a website we could use something like the Requests library with the addition of the BeautifulSoup library. Now, in a normal case, these are great and are simple to use, but when we have to log into a website to get our data, we need some other steps. The reason behind this is that when Requests goes to a website, it opens it in an internal browser. This means that even if you have all your passwords saved in with your cookies, you will still need to log in.</p><p>Thus we introduce Selenium. For those who don’t know, Selenium will actually open a browser for python to click through. Selenium can be a bit difficult to set up, <a href="https://selenium-python.readthedocs.io/installation.html#drivers">but here is the link to that process</a>. From there, you can use the Selenium functions to navigate the site autonomously to collect the data.</p><p>As ESPN holds the fantasy data behind your password, we are going to need Selenium to log into your own account, so lets go through the steps to set up our script.</p><pre>import time<br>from selenium import webdriver<br>from selenium.common.exceptions import TimeoutException<br>from selenium.common.exceptions import NoSuchElementException<br>from selenium.webdriver.support.ui import WebDriverWait # available since 2.4.0<br>from selenium.webdriver.support import expected_conditions as EC # available since 2.26.0<br>from selenium.webdriver.common.keys import Keys<br>from selenium.webdriver.common.by import By<br>from selenium.webdriver.common.action_chains import ActionChains<br>from pywebcopy import save_website<br>from pywebcopy import WebPage<br>import pywebcopy<br>from bs4 import BeautifulSoup<br>import pandas as pd<br>pywebcopy.config[&#39;bypass_robots&#39;] = True<br>from datetime import datetime</pre><p>These are all different pieces of the Selenium library that are needed. I personally, like to use datetime to test how long certain pieces of the process take and ESPN will need to load, so that is why we bring in the time library.</p><p>Once we have our libraries imported, we can open our internet browser that we will use for our navigation.</p><pre>driver = webdriver.Chrome(ChromeDriverManager().install())</pre><p>This will open a Chrome browser, but you can set it to a different browser should you choose from the link above in the ‘Drivers’ section. Now that we have our browser open, we can start going through the log in and data collection process.</p><pre>driver.get(&#39;<a href="https://www.espn.com/&#39;">https://www.espn.com/&#39;</a>);<br>time.sleep(8)</pre><pre>##Open Initial Log In Location<br>search_box = driver.find_element_by_id(&#39;global-user-trigger&#39;)<br>search_box.click()<br>time.sleep(2)<br>print(&#39;Open Log In Tab&#39;)</pre><pre>##Click on the Log In location<br>nextbox = driver.find_element_by_xpath(&quot;//a[<a href="http://twitter.com/data">@data</a>-affiliatename=&#39;espn&#39;]&quot;)<br>nextbox.click()<br>print(&#39;Click Login&#39;)</pre><pre>##Switch to iFrame to enter log in credentials<br>time.sleep(2)<br>driver.switch_to.frame(&quot;disneyid-iframe&quot;)<br>username = driver.find_element_by_xpath(&quot;//input[<a href="http://twitter.com/placeholder">@placeholder</a>=&#39;Username or Email Address&#39;]&quot;)<br>print(&#39;Switching to iFrame&#39;)</pre><pre>##Submit Username and Password<br>time.sleep(2)<br>username.send_keys(&#39;USERNAME&#39;)<br>password = driver.find_element_by_xpath(&quot;//input[<a href="http://twitter.com/placeholder">@placeholder</a>=&#39;Password (case sensitive)&#39;]&quot;)<br>password.send_keys(&#39;PASSWORD&#39;)<br>time.sleep(2)<br>print(&#39;Logging In&#39;)</pre><pre>##Submit credentials<br>button = driver.find_element_by_xpath(&quot;//button[<a href="http://twitter.com/class">@class</a>=&#39;btn btn-primary btn-submit ng-isolate-scope&#39;]&quot;)<br>button.click()<br>driver.page_source</pre><pre>##Open Link Page<br>time.sleep(8)<br>search_box = driver.find_element_by_id(&#39;global-user-trigger&#39;)<br>search_box.click()<br>print(&#39;Going to Fantasy Link&#39;)</pre><pre>##Selecting Fantasy League<br>time.sleep(2)<br>leaguego = driver.find_element_by_partial_link_text(&#39;FANTASY TEAM NAME&#39;)<br>leaguego.click()<br>print(&#39;Entering League&#39;)</pre><pre>site = driver.page_source</pre><p>Once you are here, you will need to actually collect the data from these pages. To do this you will need to use the BeautifulSoup library. This is where we actually collect the tables from the html format of the pages and to read the tables.</p><pre>time.sleep(8)<br>page = driver.page_source<br>soup = BeautifulSoup(page, &#39;html.parser&#39;)<br>tables = soup.find_all(&quot;table&quot;)</pre><p>This will take the table from your team, or the team of the page you are currently looking at. Be aware that you are on the correct week for your analysis. You either need to time your automation of this to fit in with when all scores are on the most recent page, or you will need to use Selenium to select the correct week for your scrape. Once I have this done, I personally like to format the table and add a couple of my own columns.</p><pre>time.sleep(8)<br>table = tables[0]<br>tab_data = [[cell.text for cell in row.find_all([&quot;th&quot;,&quot;td&quot;])]<br>                        for row in table.find_all(&quot;tr&quot;)]<br>df = pd.DataFrame(tab_data)<br>gdf = df.drop([0,11,19])</pre><pre>gdf.columns = gdf.iloc[0]<br>gdf = gdf[1:]<br>#gdf.drop(columns=[&#39;action&#39;])</pre><pre>gdf[&#39;Team&#39;] = &#39;Fantasy Team Name&#39;<br>gdf[&#39;Week&#39;] = 01<br>gdf[&#39;Time&#39;] = datetime.now()</pre><p>Now that you have this, the analysis is up to you! Let me know what you are doing analysis on or other improvements that could be added into this format!</p><p>— —</p><p>Here is the repository for this code along with some extras:</p><p><a href="https://github.com/dendi19/FFL">https://github.com/dendi19/FFL</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d3b1fdb39f3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spellcheck in Python]]></title>
            <link>https://medium.com/analytics-vidhya/spellcheck-in-python-2cbd7474f73f?source=rss-74b1064c6077------2</link>
            <guid isPermaLink="false">https://medium.com/p/2cbd7474f73f</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[spelling]]></category>
            <category><![CDATA[spelling-bee]]></category>
            <category><![CDATA[python-libraries]]></category>
            <category><![CDATA[new-york-times]]></category>
            <dc:creator><![CDATA[Michael Dendinger]]></dc:creator>
            <pubDate>Fri, 31 Jul 2020 19:01:47 GMT</pubDate>
            <atom:updated>2020-08-06T05:18:30.097Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently, I was working on a project as a mental test. I was looking at the <a href="https://www.nytimes.com/puzzles/spelling-bee">New York Times Spelling Bee</a> game and I thought to myself, “I bet it would be really easy to make a Python script that could give me all words that would work within the Spelling Bee game.” Now, I know, developing a code to beat this game defeats the point of the mental exercise, but I thought I would try anyway.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*iSWLELGcHILsBEvL8KXR2A.png" /><figcaption>Spelling Bee Layout</figcaption></figure><p>For those who are unfamiliar with Spelling Bee, you are given seven letters to make words with. In the center of the list, you have one character that must be used in every word. Each word needs to be a minimum of four characters long.</p><p>It is a simple concept, but I thought I could speed up the process. I figured I could get every iteration of the characters and then compare them to some kind of spellchecker to verify they are actually words. From there I would be able to quickly input the words and call it a day.</p><p>Creating the iteration portion wasn’t very difficult. I did it in an admittedly inefficient way for my mental exercise and just did nested loops for the words.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EmpTZG_LsKPbGx53VyLAaw.png" /></figure><p>Now, like I said, this is a pretty over-coded way to do this, but I tried using a couple iteration tools like using the Iteration library and importing the Combinations function. This however would only give me a minimal number of iterations and wouldn’t function to the level I needed it to get to. For example, I was only getting around 300 combination with the Combinations function. However, when I used the loop for the four character combinations, I found 2401 combinations. The five word combination found 16807.</p><p>Now, once I had my lists of combinations as a part of my test, I need to find which combinations are actually words. Though for computational efficiency, I first removed all combinations of without the necessary letter. In my test case the letter ‘T’. This cleans our list down from roughly 19,000 to around 14,000.</p><p>Now, to the crux of this discussion. How to find which words are actual words! My first thought was, “Hey, I bet there is a spell check Python library.” I was right and wrong. First, I found <a href="https://pypi.org/project/pyspellchecker/">PySpellChecker</a>, I liked the idea. However, it doesn’t exactly give me what I was looking for. It tells me what the word could be, but it doesn’t give me a boolean True or False for being spelled correctly. As far as I can find, no Python library does this. So, my logical test was to see if the suggested word from Pyspellchecker was equal to the the word that existed and I got a mess.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/1*_eyvZF90LRzOOL67u44xqw.png" /></figure><p>Boy is this a mess for my work. If the combos are close to a word, spellchecker does a pretty good job, but if you just feed it a jumble you do not get a coherent answer. Sifting through this was a huge headache and I thought to look to the actual ambassadors of the written word. Merriam-Websters Dictionary. Merriam-Webster luckily has an <a href="https://dictionaryapi.com/">API</a>! However, the free version of the API only allows for 1,000 requests a day and that just won’t do for my test. <a href="https://developer.oxforddictionaries.com/#plans">Oxford Dictionary</a> has a similar pricing set for their API as well.</p><p>I would like to advertise <a href="https://www.postman.com/">Postman</a> real quick! I love using this to get API responses and if you haven’t used it but get frustrated with API requests, you have to start now! (I am not paid by them by any means, but they are great!</p><p>Now, when I used Postman and got my request scripted out for me, I got these results from Merriam-Webster:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*Eoajj8tMWEm0lreIIQExRA.png" /></figure><p>This is better, but then rare words like “Hight” (which means “being named” and doesn’t use the necessary letters) still occur and obfuscate the practice that I am going through.</p><p>So, in my experience, if you are trying to do simple spelling corrections, feel free to use PySpellChecker, but if you are trying to find coherent results in the thousands of combinations, you may have to pay for an API and that may also not bring in the result you want.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2cbd7474f73f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/spellcheck-in-python-2cbd7474f73f">Spellcheck in Python</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Film Analytics: Releases Through the Decades]]></title>
            <link>https://m-dendinger19.medium.com/film-analytics-releases-through-the-decades-42da380393f3?source=rss-74b1064c6077------2</link>
            <guid isPermaLink="false">https://medium.com/p/42da380393f3</guid>
            <category><![CDATA[film]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[movies]]></category>
            <category><![CDATA[pop-culture]]></category>
            <category><![CDATA[history]]></category>
            <dc:creator><![CDATA[Michael Dendinger]]></dc:creator>
            <pubDate>Thu, 09 Jul 2020 22:36:00 GMT</pubDate>
            <atom:updated>2020-07-09T22:36:00.908Z</atom:updated>
            <content:encoded><![CDATA[<p>In the summer of 1977 the world saw the release of one of the most culturally impactful films ever released, <em>Star Wars Epsiode IV: A New Hope</em>. Before that fateful day in May, people saw summer films as movies that were just supposed to be throw films that weren’t going to win awards or bring in the big bucks. Star Wars changed everything. From 1977 on the summer was the time for big, blockbusters to be released and for teens to go out in droves to theaters.</p><p>In my own curiosity, I wanted to see what release trends looked like for movies throughout history. The data scientist inside me was extremely excited for this project and I took to it quickly.</p><p>I decided to take the history of American movies and see how they were released throughout the years. I decided to focus on movies that came out after 1927 (the first year when movies were eligible for Oscars) and see what trends looked like.</p><h3>Hypothesis:</h3><p>Now before I show any of the analysis, I wanted to share what my hypothesis was for this endeavor. I thought I would see few movies in the early days (20s-40s) and then see a gradual rise in film releases until the late 90s and then they would drop off again. My theory was that films have gotten pretty expensive to make in the last couple of decades, so we would see fewer movies but the movies we did have would be big budget movies.</p><h3>Analysis:</h3><p>Let’s see what the first chart I got looked like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-wnGTZo40j8DRf0utjyx8w.png" /></figure><p>This is not exactly what I was expecting to see. My hypothesis has already been blown out of the water and at the first site of this chart, I questioned my Python skills and I thought I had messed up.</p><h3>What Happened?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HzPMbd9NsHItPctofJYgUA.jpeg" /><figcaption><a href="https://en.wikipedia.org/wiki/Television_in_the_United_States">https://en.wikipedia.org/wiki/Television_in_the_United_States</a></figcaption></figure><p>In the chart above, it shows that there were, on average, almost 400 movies released a year until the 1950s! This is pretty much the opposite of my hypothesis and I wanted to know why. This got me to thinking what was going on in the time between 1945 to 1960? After a quick Google search I found that there was one product that caused the rapid decline in movies, the television. According to <a href="https://www.nyu.edu/classes/stephens/History%20of%20Television%20page.htm">Mitchell Stevens in <em>The History of Television</em></a>, almost 50% of US households had a television by 1955. This had a huge impact on what happened in Hollywood from the 1950s to the culmination of what is considered “The New Hollywood” era of film.</p><p>Before Americans were able to see the quintessential Hollywood films of the 1960s that reinvigorated film attendance, Hollywood had to go through what <a href="https://books.google.com/books?id=Nz8EAAAAMBAJ&amp;lpg=PA146&amp;pg=PA146#v=onepage&amp;q&amp;f=true">Life Magazine called “the horrible decade” for Hollywood</a>. As theater attendance was plummeting, there was a need to find a new way to get movie-goers back into their seats. The initial plan was to make the theater experience a “spectacle”. Films got bigger, we saw things like The <em>Ten Commandments (1956)</em>, <em>Ben-Hur (1959)</em>, <em>North by Northwest (1959)</em>, <em>Paths of Glory (1957)</em>, and <em>Bridge over the River Kwai (1957)</em>. These movies came out as some of the most technically astounding films at the time. They took advantage of things like technicolor, surround sound, and film quality improvements to create cinema in a way that could not be touched by television. Though, as you can see, all of those films came out at the end of the decade. To start the decade off, we saw more of the dramatic films like <em>Rear Window (1954), Sunset Boulevard (1950), Dial M for Murder (1954), </em>and <em>Roman Holiday (1953). </em>While the films of the early 1950s were incredible, they weren’t getting people in the seats.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sUUs8BPW0Pn6p5q6mrAmgg.png" /></figure><p>We can see the impact of this trend by looking at this graph. This graph tracks the runtime of movies on average through the years. It is easy to see that Hollywood was making sure that movies were something to be experienced in a way that couldn’t be done at home. Hollywood tried to make that movie experience a full evening activity and something that could tell a more complex story.</p><p>One of the biggest impacts of how films were made came from the demographic of viewers, until the 1950s most <a href="https://books.google.com/books/about/American_Cinema_American_Culture.html?id=PBfuAAAAMAAJ">movie-goers were over 30 years old and had a lower quality of education.</a> However, there was a new demographic on the way that would change everything for Hollywood: Baby Boomers.</p><h3>The 1960s-70s and The New Hollywood</h3><p>Baby Boomers changed a lot of things in the United States and Hollywood was not an exception. As the youth of America were given bigger allowances and had more freedoms, they were able to use their time as they saw fit. One of the <a href="https://books.google.com/books/about/American_Cinema_American_Culture.html?id=PBfuAAAAMAAJ">biggest pastimes for individuals under 30 by the late 60s</a> was going to movies. Hollywood saw this trend and decided they needed to have more relatable films to the baby boomers and they decided to get some age appropriate directors into the mix and this opened the floodgates for some of the biggest names in movies we have today.</p><p>Now, when we talk about Hollywood of the 60s and 70s we can go on for many levels of textbooks and anecdotal evidence of how things changed, but I will just mention a few directors who came into their own during this time period. We saw: <a href="https://www.imdb.com/name/nm0000040/">Stanley Kubrick</a>, <a href="https://www.imdb.com/name/nm0000338/">Francis Ford Coppola</a>, <a href="https://www.imdb.com/name/nm0000186/?ref_=fn_al_nm_1">David Lynch</a>, <a href="https://www.imdb.com/name/nm0000517/">Terrence Malick</a>, <a href="https://www.imdb.com/name/nm0001681/">George A. Romero</a>, <a href="https://www.imdb.com/name/nm0000217/">Martin Scorsese</a>, <a href="https://www.imdb.com/name/nm0000631/">Ridley Scott</a>, and <a href="https://www.imdb.com/name/nm0000229/">Steven Spielberg</a>.</p><p>Whether you know this names or not, they completely changed film and how we see it today. There are many other influences as well but to list all of the men and women who helped make Hollywood the way we recognize it today will take a book.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NmFRdW6i5OjABIBGxNuUZw.jpeg" /><figcaption><a href="https://flashbak.com/waiting-in-line-to-see-star-wars-1977-2000-26505/">https://flashbak.com/waiting-in-line-to-see-star-wars-1977-2000-26505/</a></figcaption></figure><p>These directors made some of the most impactful movies in Hollywood. Whether it was Stanley Kubrick forcing us to look at our world differently in <em>2001: A Space Odessy</em>, George A. Romero giving us a new definition of horror with <em>The Night of the Living Dead</em>, Francis Ford Coppola showing us the horrors of war in <em>Apocalypse Now</em>, or Steven Spielberg making us fear the water again with <em>JAWS</em>. These directors helped revitalize Hollywood and bring Americans back to the movie theaters in the 1960s and 1970s.</p><p>As we discuss the films being released it would be remiss if we did not discuss the kinds of movies coming out at the time. If we look at this graph we can see how genre releases changed as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mzoOfgdotH5G_g8NVJnJNg.png" /></figure><p>We can see drama and comedy films have been consistently the most released films throughout American film history. I will go further into genre based analysis in the future, but for now we can use this to gauge how certain movies have been released through the years.</p><p>Along with the style of films and the popularity among teenagers to go out and see movies, there was another invention in route that would change the film industry again. In 1977 the VCR was released and at home movie viewing became popularized.</p><h3>The VCR and the 1980s-90s</h3><p>It may be safe for me to assume that most of the readers of this article are most familiar with the movies released from the 1980s to today. As they include many of the quintessential Coming-of-Age films like <em>Stand By Me</em>, <em>Dead Poets Society</em>, and <em>The Breakfast Club</em>. Many people, myself included, have a strong nostalgia for the films made in the 80s and 90s. We got to see the pinnacle of people like Spielberg, John Hughes, Rob Reiner, and Robert Zemeckis. These directors gave us movies that are still quoted and mirrored in pop culture in many ways. Whether it is Star Lord in Guardians of the Galaxy or stand up comedy from <a href="https://www.youtube.com/watch?v=ButlizwQXnU">John Mulaney</a>, we see the ripple from the 80s and 90s pop culture alive and well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*yMR7RnRicnBE6UmmBnCcpg.jpeg" /><figcaption><a href="https://www.newyorker.com/business/currency/last-blues-for-blockbuster">https://www.newyorker.com/business/currency/last-blues-for-blockbuster</a></figcaption></figure><p>The ability to take home, watch, and rewatch our favorite movies made it easy to quote and quickly recall parts of our favorite movies and turn it into our popular Zeitgeist. Many VHS tapes were worn out from watching <em>The Sandlot</em> or <em>Goonies</em> one too many times I am sure. The VCR made it so that people could buy their favorite movies and watch them at home, this made the culture of movies even more impactful than before. This is the era where you can see in the charts that movies started to come out in force again. Thanks to movies like <em>Star Wars</em> and <em>Indiana Jones,</em> there were more adventures in the theater that were making people come back to theaters in a way that hadn’t been seen for two decades!</p><p>This was the perfect time for big Hollywood blockbusters that tried to capitalize on the new found market of merchandising with other products like toys, cereal, and <a href="https://www.youtube.com/watch?v=fgRFQJCHcPw">flamethrowers</a>. However, Hollywood began to fly a little too close to the sun and the late 90s represents the era of some of the biggest film flops of all time. According to Wikipedia’s list of <a href="https://en.wikipedia.org/wiki/List_of_biggest_box-office_bombs">Biggest Box Office Bombs</a> roughly 20% of the biggest bombs came from 1995–2000. This era made Hollywood a lot more careful on which movies to release. In fairness, the late 90s is when we also saw the major films like <em>The Matrix</em>, <em>Saving Private Ryan</em>, and <em>Fight Club</em>. The success of these movies would directly reflect how the next two decades of films would be made.</p><h3>The 2000s-Today and The Internet</h3><p>This is the era when films and the film industry really has gone into a huge shake up. As you can see in the above release chart, you can see that the 2000s has been a pretty shaky time for how many movies are getting released every year. This can be caused by a few different pieces.</p><p>To start, we can look at the cost of the movies during this time period and if we look at another Wikipedia table showing the <a href="https://en.wikipedia.org/wiki/List_of_most_expensive_films">Most Expensive Films</a> (adjusted for inflation), of the over 70 movies on this list, only six of them were made before 2000. This means that studios are putting more money into the hits they expect to make, but it also has another risk with potential flops. In the Biggest Box Office Bombs list mentioned earlier, while 20% came from the late 90s, 30% have come from 2015-today and that includes a whopping 14 box office flops in 2016 alone. Then also on the flip side, the 2010s have seen some of the <a href="https://en.wikipedia.org/wiki/List_of_highest-grossing_films">Most Profitable Films</a> (adjusted for inflation) <em>in Avatar (2009), Star Wars: The Force Awakens (2015)</em>, and <em>Avengers: Endgame (2019).</em></p><p>However, the main volatility in the film industry in the last 20 years has been the internet. Whether it is online piracy, streaming, or general leaks, the film industry took a long time to finally make a form of cease-fire with the internet. As places like Netflix and Redbox grew, the Blockbusters of the world were slowly dying. As the brick and mortar stores tried to adapt to this <a href="https://money.cnn.com/2010/03/24/news/companies/blockbuster/?npt=NP1">new online form of rentals</a>, they were already too far behind and by 2010 Blockbuster had closed.</p><p>This made online film rentals the best way to access movies for the average consumer. This led to one of the biggest artist strikes in living memory with the <a href="https://en.wikipedia.org/wiki/2007%E2%80%9308_Writers_Guild_of_America_strike">2007 writers strike</a>. For those unfamiliar with this event, it was the culmination of film and TV writers demanding compensation for DVD material as well as online streaming. Many films were impacted by this but the strike deeply impacted TV. After this happened making movies became a little more expensive to make as contracts became more complicated and more features needed to have compensation to them. This can be seen in places like the Scrubs you watch on Netflix has different music from the original as they were not permitted to stream the songs in the original contracts.</p><p>In 2015 Netflix changed the game again and released the film <em>Beast of No Nation </em>and production studios around the world reared their angry heads. The goal of this movie was to get Netflix into the world of film production though many established production companies did not appreciate this and actively tried to keep <a href="https://www.cbsnews.com/news/cannes-film-festival-director-explains-decision-to-ban-netflix-from-palme-dor/">Netflix out contention for awards</a>. This has been the same for places like Amazon Studios and Hulu as well. Though I will dig deeper into awards for films in a later article.</p><p>From now on we will be watching a unique change in the film industry as Netflix and Amazon Studios put out big block busters right along the big producers as well as Best Picture winners like <em>Roma </em>from Netflix.</p><p>My name is Michael Dendinger.<br>I am a semi-self taught data scientist with a passion for all things data and I will be adding more analysis similar to this on a multitude of topics, but for now I will really be focusing on analytics for films. I hope you keep checking in!</p><p>This analysis was done with Python with the Pandas and Seaborn libraries. Film record data came from a mix of OMDB and IMDB.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42da380393f3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Problem with “Simple Questions” in Humanitarian Crises and International Development]]></title>
            <link>https://m-dendinger19.medium.com/the-problem-with-simple-questions-in-humanitarian-crises-and-international-development-8b1ef61a9cfd?source=rss-74b1064c6077------2</link>
            <guid isPermaLink="false">https://medium.com/p/8b1ef61a9cfd</guid>
            <category><![CDATA[data-for-development]]></category>
            <category><![CDATA[international-development]]></category>
            <category><![CDATA[data-for-good]]></category>
            <category><![CDATA[malnutrition]]></category>
            <category><![CDATA[humanitarian-aid]]></category>
            <dc:creator><![CDATA[Michael Dendinger]]></dc:creator>
            <pubDate>Thu, 05 Sep 2019 00:01:01 GMT</pubDate>
            <atom:updated>2019-09-05T15:28:59.338Z</atom:updated>
            <content:encoded><![CDATA[<p>How many women suffer from abuse around the world? How many children die from a lack of food? Where can I best put my donation money to help the most people?</p><p>In the last 10–15 years there has been a push to incorporate data into decision making in the humanitarian context. This has been a push as governments and organizations need to make tough decisions on what to fund with their annual budgets. Organizations and advocates need to be able to relate their growth and impact with hard facts, rarely do emotional pleas work unless there is a unique case for a specific humanitarian issue. Several organizations have taken it upon themselves to try and figure out better ways to use data for applied decision making. <a href="https://pardee.du.edu/pardee-center-and-ifs">The Pardee Center for International Development</a> is one such organization (I should disclose that I was employed by the Pardee Center while I was a graduate student at the University of Denver). The Pardee Center does a great job with what it has when it works with organizations like USAID, the Colombian government, and many other domestic and international bodies. However there is an issue when it comes to humanitarian issues and governance related development predictions: there are too many things to measure and too many ways to measure them. I found this issue most glaring when I conducted research on child malnutrition in Sub-Saharan Africa for a class project.</p><p>The question was: how best can we reduce child malnutrition. This question was the beginning of me using the phrase “I am too ignorant to ask the right questions, please help.” I went in to the resident epidemiology instructor in my program and asked her where I could find rates of child-malnutrition, she looked at me as if I had asked her how to describe quantum mechanics. She quickly told me that there are many ways to look at this issue but we focused on four main aspects of malnutrition: wasting, stunting, underweight, and deficiencies in vitamins and minerals. <a href="https://www.who.int/news-room/fact-sheets/detail/malnutrition">These are part of the WHOs description of ‘undernutrition’.</a> After we had established that, I said “okay, so where do I find numbers on that?” I received another look. All four of those measurements of ‘undernutrition’ had many different ways to measure depending on the organization and where they are in the world. That means a report from the UN on child malnutrition in Kenya may vary considerably from a similar report from the UN in Vietnam.</p><p>The WHO, UN, and other organizations are aware of these issues and try to make the issues as transparent as possible. If you look at <a href="https://data.unicef.org/topic/nutrition/malnutrition/">UNICEF’s data on child malnutrition</a>, you will find an extremely overwhelming amount of data that breaks down malnutrition rates by the different indicators, income of the family, and regions in which they live. Organizations like this include the way that data was collected extremely efficiently for what they can do.</p><figure><img alt="This is the cover page for the UNICEF data on child malnutrition" src="https://cdn-images-1.medium.com/max/1024/1*JZLsK3YASJfShsjpk5KdlA.png" /><figcaption>Cover Page for UNICEF’s report on Malnutrition</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*azSPHyG5Y0CnKCe98gnVHw.png" /><figcaption>Notes Section in the UNICEF Malnutrition Data</figcaption></figure><p>As you can see, this quickly becomes a lot more information one may be prepared for when asking “how many starving children are there?”</p><p>To show what this looks like when you try to comprehend this data, you get something similar to this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*RZTih1_DSS2Sc-g2MDgehA.png" /><figcaption>This is the number of times each footnote appears in the National number for Malnourished children.</figcaption></figure><p>As you can see, there are quite a few reports that have no footnotes and were collected without any issue or need for clarification. However, this is for a national number. This is not broken down by sex, age, income level, or region. Those are all broken into their own reports that have a similar number of footnotes in their reports. If you look, converted estimates is the most common form of data changes. This is due to issues of scale in how the data was collected and extrapolated to answer the larger question so that it wouldn’t cause a huge outlier in the data.</p><p>If you are overwhelmed, trust me so was I and this was supposed to be something that I could understand as a researcher and graduate student.</p><p>As we continue digging we find issues with reporting years, regions and countries that have shifted territorial ownership (Sudan/South Sudan for example), and reports of income by family. All of these things create an extremely complex question. The question is no longer “How many starving children are there?” it has changed into something more similar to “How many children under the age of five in the Tigray region of Ethiopia were at risk of stunting in 2016?” The second question can get a clear answer (69.5% of them), the first one not as much. Keep in mind, this question does not ask “How do we reduce this wasting?” or “How did they become victims of wasting?” or “How will this impact the economy in 25 years?” Those questions include many other reports that are just as complex if not more.</p><p>Where does this knowledge leave us? First of all, probably a moment of self-contemplation where we sit back and realize the world is much larger and more complex than we generally like to think. The overall message of this is that we should be sure to ask specific questions, so that when we get answers we know what is being told to us. Many people like to over simplify questions like this and many other things, but when we ask simple questions we will get simple answers that may come from a very complicated system with many nuances.</p><p>—</p><p>If you are interested in programs that are in effect to alleviate child malnutrition, I personally suggest looking at the <a href="https://thousanddays.org/why-1000-days/">1,000 days program</a> or at <a href="https://www.savethechildren.org/us/what-we-do/global-programs/health/nutrition">Save the Children</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8b1ef61a9cfd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Senator Sentiment Code Book]]></title>
            <link>https://m-dendinger19.medium.com/senator-sentiment-code-book-d11f2f7c5b2?source=rss-74b1064c6077------2</link>
            <guid isPermaLink="false">https://medium.com/p/d11f2f7c5b2</guid>
            <category><![CDATA[politics]]></category>
            <category><![CDATA[predictive-analytics]]></category>
            <category><![CDATA[senate]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[twitter]]></category>
            <dc:creator><![CDATA[Michael Dendinger]]></dc:creator>
            <pubDate>Tue, 14 May 2019 21:21:52 GMT</pubDate>
            <atom:updated>2019-05-14T21:33:10.642Z</atom:updated>
            <content:encoded><![CDATA[<p>This is an expansion to an analysis I will write on machine learning and senators tweets. I have linked the <a href="https://github.com/dendi19/SenatorData">Github</a> page where the code for this specific code is, in case you would like to do it yourself or walk through the code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*y_n13fOs3nC6DvSTjGjAJQ.png" /></figure><p>To scrape the tweets from the senators, I used <a href="https://www.socialseer.com/resources/us-senator-twitter-accounts/">this list</a> from the 115th congress. I then created a python script that would go through and verify that each twitter handle was correct by sending back a ‘200’ response. There were a couple of senators who had changed their handles from this list, explicitly Bernie Sanders of Vermont and Corey Gardner of Colorado. Next, I used <a href="https://stackoverflow.com/questions/28384588/twitter-api-get-tweets-with-specific-id">this</a> stack overflow thread as a guide to make sure I got as many tweets from each senator as possible with my free API. This format of tweet collection created a new CSV for every senator and took the tweet, time of tweet, and the senator’s name. I also used a 60 second pause between loops to make sure that I did not exceed the limit put on the API by Twitter. After all tweets had been collected there were 299,200 tweets in 100 CSVs.</p><p>Once the tweets were collected I started to work on the code book for how I would define what the sentiment of the tweets would be. I came up with these rules:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*cBXFLZAegFmtkSn0HUXozQ.png" /></figure><p>To make sure the different levels of the machine learning process would go by smoothly and quickly I collected ten random tweets from every senator and randomly sorted them into a training file. This made it so that I would have a training file with 1,000 examples that I could use for the training of the assisted machine learning process. There was a moment where I tried to use Yelp/Amazon reviews as a trainer, but it was not very effective for the analytical format. When completed there were 458 positive tweets, 325 neutral tweets, and 190 negative tweets.</p><p>The initial machine learning process used the Naive_Bayes: BernoulliNB process of analysis. As a test, I started off using the full 1,000 tweet sentiment analysis document that I had created to analyze the full 298,200 other tweets. This process took about 30 minutes on the Google Colab system that I was using. When all was said and done, the process was about 60% accurate. While this isn’t the best prediction, keep in mind, most tweets are roughly 70 characters and due to the nature of Twitter many tweets can be highly nuanced.</p><p>To try a different model, I used SKLearn and its LinearSVC. This process ran faster than the Naive_Bayes model and had the analysis done in 20 minutes on the same Google Colab system. However, this only provided a 50% accuracy in the sentiment assignment.</p><p>Overall, the sentiment of Tweets did not deliver a whole lot of actionable information. I did choose to do some more analysis on this data and some other interesting analysis.</p><p>When we look at the Democrat/Republican divide, the machine learning process is much more effective. This process was easy to create a training file for the machine learning code. I took the 299,200 and just added the demographic information for each senator through a couple quick Excel Vlookups. From here I had Senators birth year, state, and political party.</p><p>Using the LinearSVC model it reported an 80% accurate prediction of which political party the tweet belonged to. There was an issue with predicting Independent party tweets, since this set only had one official Independent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/455/1*Hvn1ORUM2ZLxNLpeqxJ2Fw.png" /></figure><p>I then decided to look at how this predicted when it came to the generation of tweet. For this, I made a category in the file to show which generation each senator was born into. This analysis with the LinearSVC model was not as impressive as the Democrat/Republican split (71% accuracy), but still interesting none-the-less.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/441/1*YEuVG0Wq2r3xZk_Zq7bhPQ.png" /></figure><p>Finally, I decided to look at how the state impacted the tweets. Since there are only two senators per state, so this analysis was not the best overall, but I thought it would be interesting to look at. Overall, it predicted correctly about 52% of the time, unless the senator was from Indiana, Iowa, or North Dakota (~70%).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/259/1*7_DmYwvVI0BOYEkroZ3IKw.png" /></figure><p>All in all, machine learning is not the end-all-be-all for this level of analysis. There could be some potential for different training material that is not directly related to Twitter, but there would need to be more research done on which text source may be more accurate for this specific context.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d11f2f7c5b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Rise of the German Far-Right]]></title>
            <link>https://m-dendinger19.medium.com/the-rise-of-the-german-far-right-8a82a7c576c5?source=rss-74b1064c6077------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a82a7c576c5</guid>
            <category><![CDATA[refugees]]></category>
            <category><![CDATA[germany]]></category>
            <category><![CDATA[politics]]></category>
            <category><![CDATA[nationalism]]></category>
            <category><![CDATA[tableau]]></category>
            <dc:creator><![CDATA[Michael Dendinger]]></dc:creator>
            <pubDate>Thu, 09 May 2019 20:33:00 GMT</pubDate>
            <atom:updated>2019-05-09T20:33:00.275Z</atom:updated>
            <content:encoded><![CDATA[<p>In 2017 Germany saw a sharp rise in representation from the far-right, nationalist party the Alternative for Germany (AfD). Many news sites and sources have done anecdotal research to find the cause of this rise and have declared white supremacy and racist sentiment is the cause for the rise. The AfD <a href="https://www.afd.de/wp-content/uploads/sites/111/2017/04/2017-04-12_afd-grundsatzprogramm-englisch_web.pdf">self-declares</a> to be in favor of smaller government and other stereotypical conservative opinions. Not once in their 2017 manifesto do they ever mention white supremacy or nationalism, but the leaders often use racist and xenophobic <a href="https://www.dw.com/en/afds-alexander-gauland-slammed-over-racist-remark-aimed-at-minister/a-40277497">rhetoric to stir up support</a> and have <a href="https://www.dw.com/en/afd-pegida-hold-side-by-side-events-in-dresden/a-38761338">even worked with</a> the German citizen action group <a href="https://en.wikipedia.org/wiki/Pegida">PEGIDA</a>. For a country that played host to one of the world’s most well documented and horrific genocide’s, how can something like this occur. Many people who support the AfD claim that their fears come from the problems with <a href="https://www.thelocal.de/20161130/heres-why-so-many-germans-vote-for-the-far-right-afd">globalization, crime, or economic issues caused by immigration</a>, but is that what is actually causing the rise?</p><p>For those who do not know, Germany has a <a href="https://www.cnbc.com/2017/09/15/german-elections-explained-chancellor-bundestag-voting-parties-and-merkel.html">two ballot voting system</a>. The point of this system is to allow Germans the ability to vote for parties in a way that, in theory, allows minorities to gain seats in the parliament that they may not get if there is a single ballot system. Germans also do not necessarily vote for an individual. While the parties of Germany must announce who will be their representatives and who they will make a coalition with before the election, they generally vote for the party. The graph below shows the number of votes all parties have received in the last 10 years.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/983/1*l-2xV8k2az-7HwmKSasYjg.png" /><figcaption>Data From: <a href="https://www.bundeswahlleiter.de/en/bundestagswahlen/2017/ergebnisse/bund-99.html">https://www.bundeswahlleiter.de</a></figcaption></figure><p>As is visible, the light blue line represents the AfD and its increase in votes received since it was formed in 2013. It is also visible that the core parties, the Christian Democratic Union (CDU) and the Social Democratic Party for Germany (SPD), have received fewer votes in the years since formation of the AfD. To figure out more on the movement toward nationalist voting, we need to see who is voting for the AfD.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*2X6yGBgsb3e878snigNMpQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*L9NNI6O2yqZBZ4zl4haXGg.png" /></figure><p>In 2017, men born between 1958–1972 are the most likely to vote for the AfD in their primary vote for the German federal election. This follows many of the anecdotal studies that show older individuals are more likely to vote for the AfD.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/1*crY6_BXbvCpcF-9J0_eGlw.png" /><figcaption>The darker the color the greater the percent of individuals who chose the AfD with their second vote from those who voted for other parties first.</figcaption></figure><p>In the above image, it is visible that those who voted most for the AfD came from the South and Eastern states. These states are also some of the most popular places for the CDU, which is the major conservative party. Dresden is home to PEGIDA and has been host for several AfD party rallies. While Berlin is home to the AfD’s headquarters, Munich has seen a spike in AfD support. There are also many suggestions on this that relate to issues related to the reunification of Germany after the end of Soviet rule. As discussed earlier, many of those who vote for the AfD also make some claims about how society has changed.</p><p>In a European Commission public opinion poll, Germans have declared a large problem with immigration in Germany.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*_c7ME_mK_UDB2MnU9JfSMw.png" /><figcaption>Issues that German claim are the biggest that they are facing.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*oMVzooCF7RKOjur_3AR6IQ.png" /><figcaption>Issues that German claim are the biggest that Germany is facing.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/987/1*sb-NXdjOMdTshQ7h59Er7w.png" /><figcaption>Issues that German claim are the biggest that the EU is facing.</figcaption></figure><p>After the formation of the AfD in 2013 it is visible that many see immigration as one of the top five issues and by 2015 Germans declare it is the largest issue facing Germany and the European Union. However, when you look at the trends of immigration across Europe, one do not see as significant of a change.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*cEMX9xrR5wE1iEfQBX18Qw.png" /></figure><p>Immigration was on the rise from 2007–2012, but has stayed stagnant since. However, when one looks at refugee data, the narrative changes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*M71KH_Dx3aKMaZFMkMvHCg.png" /></figure><p>While Turkey has taken in the greatest number of refugees since 2011, Germany has also seen a sharp increase. Many perceive the issue within Germany as a xenophobic reaction to the growth in refugees. Those who are opposed to to refugees claim that they cause crime. This number also does not track with the growth of nationalism.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*IHcYR9R7AgkkYdI1qUAwdA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*msrsJ70mhgbTh7Hk88njWw.png" /><figcaption>Left, total crimes in Germany. Right, crimes committed by Non-German Citizens.</figcaption></figure><p>As one can see, the change in crimes committed by non-German citizens moves the same percentage as total crimes, the growth in refugees has not had an impact on criminality. There has been a growth in immigration crimes by non-German citizens, but not when related to other crimes.</p><p>Overall, the only statistic that relates to the growth in nationalism in Germany is strictly the number of refugees and nothing else. This means the opinions of refugees is the most important number needed to look at to predict AfD voting.</p><p>— —</p><p>The graphs were created with data from the Bundestag, German Federal Police, and European Commission. The graphs were created in Tableau and data formatted in Python.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a82a7c576c5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>