<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Thomas Plapinger on Medium]]></title>
        <description><![CDATA[Stories by Thomas Plapinger on Medium]]></description>
        <link>https://medium.com/@taplapinger?source=rss-b825421e07fc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*3vYTkf_UkTldYxJ9QHjLIA.jpeg</url>
            <title>Stories by Thomas Plapinger on Medium</title>
            <link>https://medium.com/@taplapinger?source=rss-b825421e07fc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 15:09:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@taplapinger/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[News and Gun Violence: Bias in the Writing?]]></title>
            <link>https://medium.com/@taplapinger/news-and-gun-violence-bias-in-the-writing-c45a52e651b2?source=rss-b825421e07fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/c45a52e651b2</guid>
            <category><![CDATA[racism]]></category>
            <category><![CDATA[general-assembly]]></category>
            <category><![CDATA[gun-violence]]></category>
            <category><![CDATA[journalism]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Thomas Plapinger]]></dc:creator>
            <pubDate>Mon, 11 Sep 2017 01:01:39 GMT</pubDate>
            <atom:updated>2017-10-02T12:03:15.472Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*qHvCxTb3zpqMIjMMpE1xLQ.jpeg" /></figure><p>Throughout the last 50 years, the United States has experienced multiple mass shootings from the University of Texas Sniper to Columbine High School to Sandy Hook Elementary, tragedies all. As these events appear to have become more frequent in the recent years, we have seen an increase in the amount of news coverage each incident receives has seemingly increased with the explosion of the 24-hour news cycle, social media, and multiple news sources. With each source of information being so readily available, it became ever-present that certain events were covered in certain ways and there is a perception that tragedies, particularly, mass shootings, are covered differently, depending on the race of the shooter. That is the very issue I looked to uncover with my final project for General Assembly’s Data Science Immersive.</p><p>I wanted to know that whether the words used by the New York Times when covering mass shootings was directly related to the ethnicity of the shooter and if it was what words were the strongest indicator of a particular race</p><p>It is important to note before going further to better your understanding of exactly what is classified as a “Mass Shooting”. While there is, no official definition used by the government, non-profits, the NRA, or other organizations the commonly used definition is, “A Shooting where four or more people are indiscriminately, not including the perpetrator(s), are killed or wounded (excluding gang and terrorist-related shootings).” While this classification is accurate most of the time it does lead to incidents that the average person may not consider to be a mass shooting being included in the data such as an armed brawl in a nightclub that results in some injuries and a few killed or a family-murder suicide. While these meet the threshold of the above definition and are tragic events in the public eye they do not match up with events like the Aurora Cinema shooting or the Charleston Church shooting. I believe the current metric may need to be adjusted to classify these incidents among other crimes.</p><p>My two sources of data were the <a href="https://library.stanford.edu/projects/mass-shootings-america">“Stanford Mass Shootings in America, courtesy of the Stanford Geospatial Center and Stanford Libraries”</a> and the New York Times. The Stanford data began to be collected in 2012 and all previous shooters were gathered using historical data. For the sake of my study, I used shootings that were reported on by the New York Times from 1970 through March of 2016. This was my primary source of the dates, names of incidents, and for discovering the race of a given shooter(s). The data from the New York Times was gathered by using the Python Package “Newspaper” for all articles and article titles from the day of a shooting through the next 6 days. This time was selected because a typical news cycle is about seven days. The New York Times was also chosen because it is an established and highly regarded news source that has done an excellent job at maintaining an archive of its articles and made them readily available to the public. By the end of my scraping, I was able to get 1,053 articles.</p><p>Some important caveats about this data are the races used by the Stanford database do have some issues where they generally classify all Asians as the same without a distinction for Far Eastern, South Asian, Middle Eastern or any of the other many ethnicities from the world largest continent.</p><p>The Methodology used for this analysis was Natural Language Processing methods and more specifically was generating features based upon their prevalence words in the Articles taken from the New York Times using countvectorizer. However, several adjustments were made to make sure unnecessary and words seemingly irrelevant to the analysis were excluded. Along with the standard English Stopwords that removes words like is, the, and similar two to three letter words I generated a custom list of words to be excluded which included the names of the locations of incidents, the names of the perpetrators, month names, and the labels given to certain incidents by the Stanford database as these are words that have to be used in an article with this subject matter, but likely do not distinguish one article from another and thus would likely make it hard to distinguish between shooters by race.</p><p>After cleaning up the words and preparing them for processing I used a random forest classifier from the top 1500 words which to generate the important features related to the general race column that included all five races. This could be done because random forest generates multiple bootstrap aggregated decision trees that look for the optimal decisions, in this case the words to related to the Y. I then used logistic regression to find what words maintained their statistical significant. It Generated a 71% Cross Validation score. Using the same group of words I generated five more random forest models in connection with each race to look for distinguishing words related to each.Each race had been given their own dummy variable column meaning if the shooter was of a certain race it would show as 1 if not as 0.</p><p>The General List of indicating words related to the general race column were:</p><pre>Class<br>Reservation<br>Crime<br>Response<br>Professor<br>Mental<br>Attempted<br>Least<br>Specialist<br>Another</pre><p>The list of top 10 words for the included races came out as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SW_sgVcxe_DNb8356y6eeg.png" /></figure><p>I have highlighted some key words I found interesting across the five races.In the White column we can see that the words are primarily related to the broadcast network and more specifically related to school shootings were we see words like Dormitory, Professor, Class, and Student. Rather surprisingly, we also see the words Immigrant and Reservation.For African-American Shooters the most interesting words I see is that this is the only area where there is a criminal or police related distinction made amongst the races with the use of Officer and Crime. We also see that is only race where the word Minority is explicitly used. The Asian column is perhaps the most interesting becuase it appears to be a mix of words from the White column, but with key distinct words like Islamic, Terorrism, Terrorist, and Foreign which while it does hint towards a bias it can not be a direct conclusion because of the generalisation classification of Asian which makes no distinction between Koreans, Vietnamese, Russians, Indians, Iranians, or those of Middle Eastern descent. While it is clear for the Native American or Alaska Native column that the most distinct words and the likely only clear case of words indicating race in this study is the top use of Reservation and Indian. The word that I found particularly surprising is that Nazi was the fourth most powerful indicator. That is something I would very much like to look into in the future. Finally, amongst Latin American shooters we see that above all the others Immigration and military experience are the key indicators.</p><p>My end conclusion of all of this is that while the New York Times uses different words related to the race of the shooter known of them are particularly racial charged. While we do get the appearance of words like Islamic and Terrorist for Asians and Indian for Native Americans or Alaska Natives it is not 100% certain how accurate that is because of the overall grouping of those diverse ethnic groups. .</p><p>Moving forward I would like to gather more news data from papers around the country to get a better idea of political and regional distinctions and see how local papers covered the events. As I continue to improve my analysis I will likely add additional stop words and seek to better classify the current Asian and Latin American population samples.</p><p>If you would like to learn more about my project and how it was completed you can find the current version of the analysis <a href="https://github.com/Tplapinger/DSI-Capstone">here</a>.</p><p><em>This project was completed for personal, noncommercial purposes, utilizing an application program interface (API) furnished by The New York Times and an open source webscraping package. The New York Times articles from which the information included in the project was scraped are protected by copyrights owned or controlled by The New York Times Company, and the webscraped information is used in compliance with the Terms of Service of NYTimes.com.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c45a52e651b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Beginners Guide to Natural Language Processing]]></title>
            <link>https://medium.com/data-science/a-beginners-guide-to-natural-language-processing-e21e3e016f84?source=rss-b825421e07fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/e21e3e016f84</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Thomas Plapinger]]></dc:creator>
            <pubDate>Sun, 03 Sep 2017 01:13:04 GMT</pubDate>
            <atom:updated>2022-09-26T15:10:21.687Z</atom:updated>
            <content:encoded><![CDATA[<p>As I have begun my journey as a data scientist one of the most captivating is that which seeks to understand the meaning and influence of words, Natural Language Processing (NLP).</p><p>One of the greatest aspects of NLP is that is stretches across multiple areas of computational studies from artificial intelligence to computational linguistics all studying the interactions between computers and the language of humans. It is primarily concerned with programming computers to accurately and quickly process large amounts of natural language corpora. What are natural language corpora? It is the study of language as expressed by real-world languages. It is a comprehensive approach to understanding a set of abstract rules from a text and the relationship that language has with another.</p><p>While NLP has become even more present with the computer industrial revolution in the modern era it was actually the brain child of the amazing Alan Turing who along with helping to crack the German Enigma Coding machine also wrote an article titled “<a href="http://loebner.net/Prizef/TuringArticle.html">Computing Machinery and Intelligence</a>” which proposed the first serious use of relating humans language to computers. With the ever present nature of technology in our daily lives we have seen the emergence of just how influential NLP can be to our daily lives through revolutionary tools like Google Translate, IBM Watson, Speech Recognition and generation and sentiment analysis.</p><p>However, like all things there are a few areas of concern and disadvantages of NLP. It struggles to generate language that would naturally flow as a person talks similar to when you read a bad movie script and it sounds like a computer talking. While there are methods to attempt to understand changes in tone NLP continues to struggle with understanding things like sarcasm and detecting things like humor. However, this is an area of much study that I look forward to the day that the “sarcasm breakthrough” occurs. If for nothing else, then to better understand the occasional text or instant message from a friend.</p><p>Some of the most useful computer/data science tools currently available are:</p><p>CountVectorization, Hashing Vectorization, Term Frequency-Inverse Document Frequency (TF-IDF), Lemmatization, Stemming, Parsing, and Sentiment Analysis.</p><p><a href="http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html">CounterVectorization</a> is a SciKitLearn library tool takes any mass of text and returns each unique word as a feature with the count of number of times that word occurs. While this can generate a lot of features that are some extremely useful parameters that help avoid that including stop_words, n_grams, and max_features. Stop words generates a list of words that will not be included as a feature. The primary use of this is the “English” dictionary where it will get rid of insignificant words like “is, the, a, it, as “which can appear quite frequently, but have little to no influence on our end goal. Ngram_range selects how you can group words together. Instead of having NLP return each word separately you can get results like “Hello again” if it equals 2 or “See you later” if it equals 3. Max_features is how many features you choice to create. If you choose it to equal none it means that you will get every and all words as features, but if you set it equal to 50 you will only get the 50 most frequently used words.</p><p><a href="http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.HashingVectorizer.html">Hashing Vectorizer</a> converts text to a matrix of occurrences using the “hashing trick” Each word is mapped to a feature and using the hash function converts it to a hash. If the word occurs again in the body of the text it is converted to that same feature which allows us to count it in the same feature without retaining a dictionary in memory.</p><p><a href="http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.HashingVectorizer.html">TF-IDF</a> reveals what words are the most discriminating between different bodies of text. It is particularly, helpful if you are trying to see the difference between words that occur a lot in one document, but fail to appear in others allowing you interpret something special about that document. It is dependent on term frequency, how often a word appears, and Inverse document frequency, whether it is unique or common among all documents.</p><p><a href="http://scikit-learn.org/stable/modules/feature_extraction.html">Lemmatization</a> is a process by which inflected forms of words are grouped together to be analyzed as a single aspect. It is a way of using the intended meaning of a word to determine the “lemma”. It is largely depending on correctly finding “intended parts of speech” and the true meaning of a word in a sentence, paragraph, or larger documents. Examples of Lemmatization are that “run” is a base form for words like “running” or “ran” or that the word “better” and “good” are in the same lemma so they are considered the same.</p><p><a href="http://scikit-learn.org/stable/modules/feature_extraction.html">Stemming</a> is quite similar to Lemmatization in that it groups words together, but unlike lemmatization it takes a word and refers it back to its base or root form. In fact, on the best examples I’ve come across to describe it involves refereeing Stemming back to its base form. “Stems”, “Stemming”, “Stemmed”, “and Stemtization” are all based on the single word “stem”.</p><p><a href="https://en.wikipedia.org/wiki/Parsing#Computer_languages">Parsing</a> general use is to analyze a string of words to form to the rules of grammar. For NLP it is the analysis of a string of words resulting in a parse tree revealing the syntactic relationship between words, which can contain semantics. However, a drawback of parsing is that when and what is parsed is entirely dependent on the user as any paragraph can be parsed any way they choose from individual characters to whole sentences, and more.</p><p><a href="http://www.nltk.org/howto/sentiment.html">Sentiment Analysis</a> in it simplest definition is the extraction of the interpretation, or subjective meaning, of a word from a document or set of documents to determine the “attitude” of a specific word or set of words. One of the greatest uses of it is in social media in places like Facebook, Twitter, and others to identify trends of public opinion, how often a word appears in a specific tonal context, and interpreting the tone of groups of words.</p><p>Before doing any and all of this analysis I highly suggest you filter and clean whatever document you are dealing with. One of the best functions I have come across too easily do this has been created by one of my teachers, <a href="https://medium.com/u/45af05e64a7d">Mark Mummert</a>. While it is focused on cleaning an html file through the Use of Beautiful soup it can easily be adapted for other NLP cleanings.</p><pre>def review_to_words(raw_review):<br>      #Remove html<br>      review_text=BeautifulSoup(raw_review).get_text()<br>      #Remove non-letters - Utilized Regex Library<br>      letters_only = re.sub(&quot;[^a-zA-Z]&quot;, &quot; &quot;, review_text)<br>      #Conver to lowercase and split into individual words<br>      words = letters_only.lower().split()<br>      #Convert Stop-Words to a set and remove them<br>      stops = set(stopwords.words(&quot;english&quot;))<br>      meaningful_words = [w <strong>for</strong> w <strong>in</strong> words <strong>if</strong> <strong>not</strong> w <strong>in</strong>    stops]<br>      #Join the Words back into one string seperated by a  space and return the result.<br>      <strong>return</strong>( &quot; &quot;.join( meaningful_words ))</pre><p>The world of NLP is large and complicated and continues to be an area of great amounts of study. There are many useful tools all of which have extremely beneficial uses in interpreting and utilizing the world of words.</p><p>For python users, I believe your most useful sources for NLP will be the <a href="http://scikit-learn.org/stable/modules/feature_extraction.html">Sci-Kit Learn library</a> and the<a href="http://www.nltk.org/"> NLTK library</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e21e3e016f84" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-beginners-guide-to-natural-language-processing-e21e3e016f84">A Beginners Guide to Natural Language Processing</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Global Terror and the EDA Visualization Rabbit Hole]]></title>
            <link>https://medium.com/data-science/global-terror-and-the-eda-visualization-rabbit-hole-a953c998b7e9?source=rss-b825421e07fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/a953c998b7e9</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[terrorism]]></category>
            <category><![CDATA[politics]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <dc:creator><![CDATA[Thomas Plapinger]]></dc:creator>
            <pubDate>Tue, 29 Aug 2017 23:41:41 GMT</pubDate>
            <atom:updated>2022-09-26T15:09:33.279Z</atom:updated>
            <content:encoded><![CDATA[<p>One of the greatest challenges a data scientist faces is knowing when to move on. It may be doing more Exploratory Data Analysis(EDA), it may be adjusting your model’s parameters for a better score, or simply trying to make your notebook look as professional as possible. No matter the subject matter it can be extraordinarily easy to lose track of time and go down the rabbit hole of your analysis, but as your deadlines approach you must face the reality that you have to jump out of the whole and make sure that other parts of your project are not getting ignored because if you only one part of your project looks great while the other parts look mediocre at best all your clients will think is “why wasn’t the rest of his/her analysis and presentation as good as [Insert Rabbit Hole item].” and thus decreases the over value and opinion of the work you put out there.</p><p>I recently experienced a prime example of this when I supplied a Global Terrorism database from START when I got lost in the EDA visualizations. The Global Terrorism Database (GTD) is a database on terrorist attacks around the world from 1970 through 2016 containing over 150,000 observations. Researchers at the National Consortium maintain the database for the Study of Terrorism and Responses to Terrorism (START), headquartered at the University of Maryland.</p><p>Having previously studied global and world history at college I clearly, have a strong interest the subject and as such was easily tempted to dive into the EDA. With the project now complete I can now give you the full scope of my visualizations and all the information they do and do not present. Among the primary purposes of this project was to dive into generating visualizations surrounding time and what they demonstrate. It had a focus on upon global bombings between 1970 and 2016, the reason for which you will see below why it is the clear choice for focus.</p><p>When initially looking at the data a pretty immediate question you wanted to answer is how many events, in this case terrorist attacks, occurred in our datasets timeline. Secondly, if you are focusing on an area, in our case on bombings, you want to see the change in number incidents over time and see if there are any noticeable differences between the two trends.</p><p>We can see here that attacks slowly start to increase from 1970 until about 1993 where until 2003 we see a downward global trend in Terrorist incidents despite events like September 11, 2001. We can also see in this second graph that the number of bombings follows roughly the same trend, but that it has a sharper decline of occurrences in the early 90s.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1009/1*huL5VjUoldpj5JAKookFjA.png" /><figcaption>Global Terrorist Attack Incidents 1970–2016</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G9TUIlB0oQK1rIDpNRVUyg.png" /><figcaption>Global Bombing Numbers 1970–2016</figcaption></figure><p>Here it becomes very clear how the overall trend over attacks has changed over the years in different regions across the globe. In the 1970s we can see that most terrorist attacks occurred in Western Europe, followed by Latin and South America in the 1980s, but then there is an overall decline in the number of attacks between the mid-1990s and 2003. We can also see that come we experience the same jump in numbers and see that the Middle East and North Africa along with South Asia are the hardest hit by terrorist attacks. However, it is also important to note that while the amount by which overall trends increase it does appear that no matter the region the occurrence of terrorist attacks has risen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*OFYuTsJVU4xOMMO5uut6kA.png" /><figcaption>Terrorist Attacks by Region 1970–2016</figcaption></figure><p>Here we can see that in the past 47 years the overwhelming favored form of attack has been bombings with it being the choice of terrorists more than twice that of Armed Assaults. However, it should be noted that if there is a distinction between when these two happen separately or together I was unable to find such a data clarify so these numbers may be askew as events like the Mumbai Hotel and Bataclan attacks both involved both bombs and armed assaults. Interestingly, hijacking is in last despite its prevalence in the 1970s and 1980s.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*P41wiZojPTREBq92a6cZIQ.png" /><figcaption>Type of Terror Attacks</figcaption></figure><p>Below, we see a tableau generated chart of the type of targets that are “hit the hardest” by number of injured and by numbers killed with the size of the circle being the number of occurrences. We can see overwhelmingly that the most frequent target is civilians who as a result also suffer the most at the hands of terrorists. However, this graph can be misleading because single events like 9/11, the 1993 WTC bombings, and Bishopsgate can skew that number as all three registered either high numbers of killed, injured or both. It should also be noted that this is a prime example of when you should better organize your data as there are several points in this graph that could be combined. Such as Civilian could encompass Village, Market, Vehicle, Bus, and of course Civilians while a single Military class could include Military Unit, Soldiers, Checkpoint, Patrol, Convoy, Officers, Unit, and Police Station.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*rc49JPBcRHfDzFvWuArleA.png" /></figure><p>As terrorist actions are often a reaction to an action taken by the targets government or country I next wanted to see if there were any similarities in the trends in number of incidents between countries that have historically not gotten along well (to put it nicely).</p><p>The first comparison is the United Kingdom and Ireland where we can see from chart below that the United Kingdom has taken the brunt of all the violence. The peaks largely related to the conflict between these two countries are those between 1970 and the end of the 1990s which encompasses the entire period known as the “<a href="https://en.wikipedia.org/wiki/The_Troubles">The Troubles</a>” where over 3,500 people died and more than 47,500 were injured. An important note here is that though it appears a large portion of the attacks came in the United Kingdom most the incidents did not occur in mainland Britain, but in Northern Ireland so if you were to display the entire Eire (Ireland and Northern Ireland) vs the UK I imagine the numbers would be different.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*yQ72fRiI0-OGrDR0-9VQig.png" /><figcaption>UK vs Ireland Incidents</figcaption></figure><p>Next, we see two countries that since their independence from the British have had a tumultuous history. However, we can see that except for one jump in the mid-1990s the number of terrorist incidents tends to stay below 500 each year and most are in India. However, this all begins to change in 2008 with the Mumbai Terror attacks. From that point, onwards Pakistan has taken overwhelmingly had more terrorist incidents over India, but it appears that in the last year or two that number has begun to dramatically reduce itself. However, India’s number since 2008 have sharply risen since the attack and ever since that appears to be a slow, but steady increase in attacks. In fact, if you zoom in on the very end of the graph you can see that India surpasses Pakistan in number of incidents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*UJ8OcMyoiMl9QHyRrlTsQQ.png" /><figcaption>India vs Pakistan</figcaption></figure><p>Two other countries we have seen lock heads over the 46 years are of course the United States and Russia. While their wars between them were either proxy ways or cold war’s (i.e. sanctions, espionage, etc.) it never reached a full-scale war that would have likely encompassed the globe. However, we can see here that while the United States has experienced some high-profile terrorist attacks by groups like The Weather underground, Timothy McVeigh, the Unabomber, Al-Qaeda, ISIS, and the Tsarnaev brothers it has largely experienced a great deal fewer number of attacks than Russia. This is largely due to the considerably fewer amounts of areas in the US that are as volatile as certain regions of Russia and the Former Soviet Union like Dagestan, Chechnya, or the Ukrainian/Russian border as well as Russia’s conflict with Georgia. Both have suffered, but clearly due to both Russia’s size and violent involvement amongst itself and neighboring countries it has suffered the most at the hands of terrorists.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*bwTH_7xOmNwTdDHDmSDg0A.png" /><figcaption>United States vs Russian Federation/USSR</figcaption></figure><p>On a happier, and a large portion of my focus for this individual project, the situation in South Africa has significantly improved since 1994 with the country’s first free elections to appoint Nelson Mandela as President and unofficially end Apartheid. While, the official end of Apartheid is 1991 with the release of Mandela from prison the elections would not be held until three years later. In fact, we can see from the graph below that between 1991 and 1994 was among the most violent times in South Africa as Afrikaner nationalists took to more violent means to, unsuccessfully, prevent the elections in 1994 which would of course provoke actions from Black South Africans to strike back. Fortunately, we can see the years we have data for following the elections the number of incidents dropped significantly. Sadly, it does appear that the data has begun to disappear after 2000 which for anyone who knows South African history and politics includes the Presidencies of Mbeki and Zuma who have both been less than transparent about the state of the country.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*mNDDsCaudEXHL8q0NrCDNA.png" /></figure><p>While it does appear that the numbers have gone down significantly since 1994 this graph shows us two things that an individual must do his/her best to obtain and/or manage when dealing with data like this for all data. Which is to gain as much knowledge surrounding your data outside of the numbers to obtain some context and to be able to understand or extrapolate from the data you do have to generate possible numbers for the data we don’t.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a953c998b7e9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/global-terror-and-the-eda-visualization-rabbit-hole-a953c998b7e9">Global Terror and the EDA Visualization Rabbit Hole</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Visualizing your Exploratory Data Analysis]]></title>
            <link>https://medium.com/data-science/visualizing-your-exploratory-data-analysis-d2d6c2e3b30e?source=rss-b825421e07fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2d6c2e3b30e</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Thomas Plapinger]]></dc:creator>
            <pubDate>Tue, 22 Aug 2017 02:45:20 GMT</pubDate>
            <atom:updated>2022-09-26T15:10:30.390Z</atom:updated>
            <content:encoded><![CDATA[<p>Perhaps one of the greatest disparities between those who live in the world of data science and those who don’t is the ability to translate the data so that everyone can understand it. One of the best ways for data scientists to present analysis of the data to those outside the industry is by generating visualizations. Through my brief time as a data scientist so far it has become apparent that there are really only two times where visualizations can be generated. The first of those times is during the Exploratory Data Analysis (EDA) where you have initially received the data, cleaned it, sorted it, and created whatever other features you please for a nice clean and full dataset. From that point you can begin to see the relationship between certain variables, the correlation (if any) between features, and generate many many more visualization in order to better display the data at hand. The other usually takes place at the conclusion of your analysis where you may want to see visually how your model prediction compared to the that of your EDA visualizations or to see if the inferences your inferred are now able to make are new discoveries or a continuance of the data displayed in the EDA.</p><p>This posting will be focused on the first opportunity for visualizations, the EDA. Before, we dive into the visualizations, as with all things understanding the definitions of what you are about to do is key. EDA is an approach to data analysis that uses a variety of techniques, largely visual, to: Maximize insight, reveal underlying structure, check for outliers, test assumptions, and determine optimal factors. It is an approach to analysis of your data that delays the usual assumptions of what kind of model should be used and instead allows the data to speak for itself.</p><p>In my experience the best libraries I have come across so far for visualizations are <a href="https://matplotlib.org/">matplotlib</a>, <a href="http://seaborn.pydata.org/">seaborn</a>, and the <a href="https://www.tableau.com/">Tableau</a> application. Each offers a unique set of tools.</p><p>The two sets of data I will be using are from the Kaggle competition to predict <a href="https://www.kaggle.com/c/predict-west-nile-virus">instances of West Nile Virus</a>(WNV) in mosquito traps across the greater Chicago area. I will show you some of the initial EDA that was done on the weather data set using Seaborn and Matplotlib. I would also like to note that while the visualizations you will see are mine, <a href="http://irickman.github.io/">Ira Rickman</a>, Michael Costa, and Kristina Schendt also worked this project on. The other dataset I will be using to display uses of Tableau uses <a href="https://library.stanford.edu/projects/mass-shootings-america">Stanford Universities Library on Mass Shootings in America</a>, which I am currently using for a separate project to which a full blog will later be devoted to in its entirety.</p><p>While fairly simple easy to create some of the most valuable types two charts you can generate when doing EDA are Histograms and Scatter plots. A histogram allows us to see the distribution of a particular variable while a scatter plot allows us to see a relationship between two or more variables. Throughout my exploration of the WNV weather data these plots were vital even when their uses may have seemed redundant the purpose of EDA is to explore the data in every way you know how.</p><p>The two histograms below show us the distribution and frequency of the Average Temperature across the time period of our data. As you can see it ranges from the low 40s to the mid 90s and the most frequent average temperature in the mid-70s which even without knowing anything else about the data allows you to assume that the time period in which the data was collected was the from spring to early fall.</p><p>You also see the vast differences between Matplotlib and Seaborn. The first graph was created using Matplotlib and requires (and also nicely allows) a greater amount of coding to set up and is limited to the parameters you place on it. The second is a Seaborns version of histogram, known as a Density plot. It is far easier to setup, as you can see, as it sets up your bins and other parameters for you as well as providing you with a nice line over the entire distribution of data. However, since it does not exclude any possible outliers it decreases our ability to focus on the main chunk of our data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*iYOJEbhytga8hLOxBvRs6A.png" /><figcaption>Matplotlib Average Temperature Histogram</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*rtDm4BkhFRuASv5DeBtPIg.png" /><figcaption>Seaborn Average Temperature Density Plot</figcaption></figure><p>Below, are four examples of very different scatterplots generated using Seaborn.</p><p>The first chart compares the data points of Average Temperature to the Wind speed. While at first, it appears too large and a disorganized mess of a scatter plot, it still provides us with some key information. Overall, it shows us that there appears to be little to no correlation between the two variables, while also showing that most data points are between 55 and 85 degrees in temperature and between 1 and 12 miles per hour winds. We can also see that as the temperature decreases the number of occurrences of winds over 15 miles per hour increases as opposed to when the temperature heads in the opposite direction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/831/1*Vk7U3El57K_ve2abwGgM-w.png" /><figcaption>Wind Speed vs Average Temperature</figcaption></figure><p>Our second scatter plot compares the Average Temperature with the Total Precipitation and while it also appears to be slightly unclean as far as graphs go it continues to provide valuable information on our data. The two main takeaways from this graph are that it appears that a large portion of the days in this dataset had little to no precipitation and it has revealed to us that there are a few outlying data points where there was close to 7 inches of rain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*tZrCTrnDdNoJUsSHBP6VZg.png" /><figcaption>Precipitation Total vs Average Temperature</figcaption></figure><p>Our final two scatter plots of Dew Point vs. Average Temperature and Average Wind speed vs. Maximum Wind Speed like the others reveal to us important understandings of our data. For the first time, we are seeing two plots where their variables appear to have both a strong correlation along with a positive trend. We can see in the first plot that as the temperature rises so does the dew point which while unsurprising once you learn that Dew Point is directly related to Humidity, but because of this plot my data told me to investigate the subject further. An investigation I may not have done without the plot. Our second plot unsurprisingly shows us our strongest correlation yet, beyond comparing a feature with itself, by comparing average and max wind speed. However, even in its apparent redundancy it shows us a few rare occurrences, possibly outliers, where the wind speed exceeded 20 miles per hour.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*B8gNFy2HDmQooOeZlwhPCg.png" /><figcaption>DewPoint vs Average Temperature</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/1*aNpD3zDnWTki8Nt0o-yB8A.png" /><figcaption>Wind Speed vs Average Wind Speed</figcaption></figure><p>Below, is one of my favorite charts in the Seaborn library, the <a href="https://seaborn.pydata.org/generated/seaborn.heatmap.html">heat map</a>. This not only provides us with a numerical value of the correlation between each variable, but also provides us with an easy to understand visual representation of those numbers with the darker red showing high correlation to dark blue showing none or negative correlation. It is one of the simplest plots to create, but is also one of the most informative and can guide our hand in generating other plots to investigate the numbers it has brought forward. A quick note is that since it is comparing the correlation between every variable in your dataset it does take some time to process depending on the number of variables, observations, and your computers processing power.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*4tAJoAp2zjUaLTOoD9zzNA.png" /><figcaption>Seabron Correlation Heatmap of Weather Variables</figcaption></figure><p>Another wonderful tool that can be used to is Tableau that at its most basic levels is a point and click application that can create a wonderful world of visualizations. I have found it particularly useful when comparing events across multiple geographic areas. As mentioned before I am currently working on a project involving Mass Shootings in the United States and as such am currently running my EDA. Below are some of the wonderfully easy to interpret and create charts of Tableau that I have been able to generate so far.</p><p>The below chart shows us the locations of every shooting in the continental United States from 1966–2016 with the darkness of the color changing based on the number of injured or killed at each location. The darker the color the higher the number. Almost instantly, we can see that most incidences are in or near populated areas and that the areas that have suffered the highest number of dead or wounded are in Southern California, Denver area, Central Texas, Western Virginia, and Connecticut. However, it is important to note that for some of those locations singular events like that in Aurora, Sandy Hook, Virginia Tech, or at the University of Texas have pushed them to that unfortunate position.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*RDzPAKq8OlnAnW6DwrjQHA.png" /><figcaption>Locations of US Shootings 1966–2016, Total Number of Victims</figcaption></figure><p>Our second chart also shows us the locations of each of these events, but instead of a victim count this time we are viewing the density of cases where the shooter had a diagnosed mental health illness. While we can see some similar epicenters of incidence of both shootings and mental illness it also reveals that an alarming number of the perpetrators of these actions are highly likely to have a pre-existing mental health issue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*unfulpyEEQ_nyIQZ3Qllfw.png" /><figcaption>Locations of US Shootings 1966–2016, Number of Shooters with Diagnosed Mental Health Illnesses</figcaption></figure><p>As you have seen from both the WNV and US Shootings EDA visualizations even the simplest or ugliest of charts can provide you with vital information about your data. Whether it is revealing outliers, trends, correlations, or increasing intrigue down the rabbit hole of our data EDA allows the data to reveal itself and for us to analyze it once it has done so.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2d6c2e3b30e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/visualizing-your-exploratory-data-analysis-d2d6c2e3b30e">Visualizing your Exploratory Data Analysis</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tuning a Random Forest Classifier]]></title>
            <link>https://medium.com/@taplapinger/tuning-a-random-forest-classifier-1b252d1dde92?source=rss-b825421e07fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b252d1dde92</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[modeling]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[random-forest]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Thomas Plapinger]]></dc:creator>
            <pubDate>Sat, 12 Aug 2017 21:32:38 GMT</pubDate>
            <atom:updated>2017-09-26T01:57:06.465Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6cu8g7ZLG5lhuWGMdivoYQ.jpeg" /><figcaption>From <a href="http://www.timetravelturtle.com/2014/12/jasmund-national-park-germany-beech-forests/">here</a></figcaption></figure><p>One of the most useful models I have come across in my brief time as a Data Scientist is Random Forests. What are Random Forests you ask? They are an ensemble method for classification and regression models that run by compiling multiple decision trees at taking the mode of the classes in classification models and the mean output in regression models. What makes them so great is that it corrects the overfitting of a single decision tree model by using Bagging, also known as Bootstrap Aggregating. Bagging is when the model repeatedly generates a random sample from your training data and fits it to a tree with replacement. This means that each tree that is created within a Random Forest while having the same original data source is compiled on randomly selected samples of that data with the end results being an aggregate of those trees. You may be asking what is the difference then between Bagged Decision Trees and Random Forests. At each split in the multiple decision trees a Random Forest generates a random subset of features to be used as opposed to a Bagged Decision Tree that does also use Bootstrap Aggregating, but it uses the same features for all of its trees.</p><p>When in python there are two Random Forest models, RandomForestClassifier() and RandomForestRegressor(). Both are from the sklearn.ensemble library. This article will focus on the classifier.</p><p>First, to make your life easier you should import the classifier. The classifier without any parameters included and the import of the sklearn.ensemble library simply looks like this;</p><pre>from sklearn.ensemble import RandomForestClassifier</pre><pre>model = RandomForestClassifier() <br>#for later in your notebook it will be easier to simply refer to it as model or whatever name you deem appropriate</pre><p>While you could simply put that in and fit your model to your X, y variables using .fit(X,y) the classifier will perform much better if you use its many different parameters. It is important to know that most, if not all, of the parameters have default settings, but your model will likely benefit in some shape or form from tuning them.</p><p>As the title of this posts suggests I will now go through each of these parameters explaining what they are and what they do. For the sake of ease of searching for a precise parameter the order in which in which I will show the parameters and attributes will be as follows:</p><pre>Parameters:<br>n_estimators,<br>criterion,<br>max_features,<br>max_depth,<br>min_samples_split,<br>min_samples_leaf,<br>min_weight_leaf,<br>min_weight_fraction_leaf,<br>max_leaf_nodes,<br>min_impurity_decrease,<br>bootstrap,<br>oob_score,<br>n_jobs,<br>random_state,<br>verbose,<br>warm_start,<br>class_weight</pre><p><strong>PARAMETERS</strong>:</p><p><strong>n_estimators</strong>-(<em>integer</em>)- Default=10</p><p>The number of trees your want to build within a Random Forest before aggregating the predictions. The higher the number the better, but it is important to know this is more computationally expensive and will take longer for your code to run. Based on your previous knowledge of your computers processing speed I suggest making a n_estimator proportional to that speed.</p><p><strong>criterion</strong>-(<em>string</em>)-Default =”<em>gini</em>”</p><p>Measures the quality of each split. It can either be “gini” or “entropy”. “gini” uses the Gini impurity while “entropy” makes the split based on the information gain.</p><p><strong>max_features</strong>-(<em>integer, float, string, or None</em>)-Default=”<em>auto</em>”</p><p>The max number of features considered when finding best split. This improves the performance of the model as each node of each tree is now considering a higher number of options. Once again, by increasing the number of features your processing speed will decrease. max_features is one of the more complicated parameters because it is dependent on the type you set it as.</p><p>If it is an <em>integer</em> then you should think carefully about max_features at each split because the number is basically up to you. If it is <em>float</em> it is a percentage (max_features x number of features).</p><p>If it is set to <em>auto </em>or <em>sqrt </em>then it is set to the square root of the number of features (sqrt(n_features)).</p><p>If you set it to <em>log2 </em>it equals<em> log2(n_features).</em></p><p>None as you might expect simply uses the number of features, or n_features.</p><p><strong>max_depth</strong>-(<em>integer or none)- </em>Default=<em>None</em></p><p>This selects how deep you want to make your trees. Do you want to split once, twice, or if you select None until the all leaves are their purist or contain less samples than min_samples_split. I suggest setting your max_depth because if you let it run down to the purist option you risk overfitting your model.</p><p><strong>min_samples_split-</strong>(<em>integer</em>, <em>float</em>)-Default=2</p><p>Sets the minimum number of samples that must be present from your data in order for a split to occur. If it is a float then it is calculated with min_samples_split*n_samples.</p><p><strong>min_samples_leaf-</strong>(<em>integer, float</em>)-Default=1</p><p>This parameter helps you determine minimum size of the end node of each decision tree. The end node is also known as a leaf.</p><p><strong>min_weight_fraction_leaf</strong>-(<em>float</em>)-Default=0</p><p>This is quite similar to min_samples_leaf, but it uses a fraction of the sum total number of observations instead.</p><p><strong>max_leaf_nodes</strong>-(<em>integer</em>, <em>None</em>)-Default=None</p><p>This parameter grows the tree in the best-first fashion resulting in a relative reduction in impurity.</p><p><strong>min_impurity_decrease</strong>-(<em>float</em>)-Default=0</p><p>“A node will be split if this split induces a decrease of the impurity greater than or equal to this value.</p><p>The weighted impurity decrease equation is the following:</p><pre>N_t / N * (impurity - N_t_R / N_t * right_impurity<br>                    - N_t_L / N_t * left_impurity)</pre><p>where N is the total number of samples, N_t is the number of samples at the current node, N_t_L is the number of samples in the left child, and N_t_R is the number of samples in the right child.</p><p>N, N_t, N_t_R and N_t_L all refer to the weighted sum, if sample_weight is passed.” — <a href="http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier">From SkLearn Library</a></p><p>Note — The sklearn library has used this over min_impurity split.</p><p><strong>bootstrap</strong>-(<em>boolean</em>)-Default=True</p><p>Whether or not you should bootstrap your samples when building trees. This is something you should never have to change from the default as your model will perform much better with bootstrapping.</p><p><strong>oob_score</strong>-(<em>boolean</em>)-Default=False</p><p>This is a cross-validation method that is very similar to a leave-one out validation technique where the generalized estimated performance of a model is trained on n-1 samples of the data. However, oob_score is much faster because it grabs all observations used in the trees and finds out the maximum score for each observation base on the trees which did not use that observation to train.</p><p><strong>n_jobs</strong>-(<em>integer</em>)-Default=1</p><p>This lets the computer know how many processors it is allowed to use. The default value of 1 means it can only use one processor. If you use -1 it means that there is no restriction of how much processing power the code can use. Setting your n_jobs to -1 will often lead to faster processing.</p><p><strong>random_state</strong>-(<em>integer</em>, <em>RandomState</em> <em>instance</em>, <em>None</em>)-Default=None</p><p>Since the bootstrapping generates random samples it is often hard to exactly duplicate results. This parameter makes it easy for others to replicate your results if given the same training data and parameters.</p><p><strong>verbose</strong>-(<em>integer</em>)-Default=0</p><p>Verbose means that you are setting the logging output which gives you constant updates about what the model is doing as it processed. This parameter sets the verbosity of the tree’s building process. It is not always useful and may take up an unnecessary space in your notebook.</p><p><strong>warm_start</strong>-(<em>boolean</em>)-Default=False</p><p>At False it fits a new forest each time as opposed to when it is True it adds estimators and reuses the solution of the previous fit. It is mostly used when you are using recursive feature selection. This means that when you drop some features other features will gain in importance and to “appreciate” the trees they must be reused. It is often used with backward elimination in regression models and not often used in classification models</p><p><strong>class_weight</strong>-(<em>dictionary</em>, <em>list of dictionaries, “balanced”</em>)</p><p>None, also known as “balanced subsample” weights the classes. If you do not put something here it will assume that all classes have a weight of 1, but if you have a multi-output problem a list of dictionaries is used as the columns of y.</p><p>When the “balanced” mode is used the y values automatically adjust their “weights inversely proportional to class frequencies” in the data using “n_samples/(n_classes*np.bincount(y))”</p><p>The balanced subsample and the balanced mode are the same except that balanced subsample used bootstrapped weights for each tree.</p><p>I hope you have found this article helpful. For further descriptions, examples, and further steps you can take in tuning your Random Forest Classifier I suggest clicking this <a href="http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html#sklearn.ensemble.RandomForestClassifier">link</a> for the SkLearn library descriptions. If you want to see the parameters for the RandomForest Regressor model please click <a href="http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b252d1dde92" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is a Decision Tree?]]></title>
            <link>https://medium.com/data-science/what-is-a-decision-tree-22975f00f3e1?source=rss-b825421e07fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/22975f00f3e1</guid>
            <category><![CDATA[data-mining]]></category>
            <category><![CDATA[decision-tree]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Thomas Plapinger]]></dc:creator>
            <pubDate>Sat, 29 Jul 2017 20:50:18 GMT</pubDate>
            <atom:updated>2017-09-26T01:54:43.521Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*EFCePNEkqoGmxm5qR-nqrA.gif" /></figure><p>For a bank to consider whether or not to offer someone a loan they often go through a sequential list of questions to figure out if it is safe to give said loan to an individual. Those questions can start as simple as what kind of income does the person have? If it is between $30–70k they move on to the next question. How long have they held their current job? If 1–5 years it leads to their next question of do they make their credit card payments? If yes then they offer the Loan and if no they do not. This process at its most basic form is a Decision Tree.</p><p>A decision tree is a largely used non-parametric effective machine learning modeling technique for regression and classification problems. To find solutions a decision tree makes sequential, hierarchical decision about the outcomes variable based on the predictor data.</p><p>So what does all that mean?</p><p>Hierarchical means the model is defined by a series of questions that lead to a class label or a value when applied to any observation. Once set up the model acts like a protocol in a series of “if this occurs then this occurs” conditions that produce a specific result from the input data.</p><p>A Non-parametric method means that there are no underlying assumptions about the distribution of the errors or the data. It basically means that the model is constructed based on the observed data.</p><p>Decision tree models where the target variable uses a discrete set of values are classified as Classification Trees. In these trees, each node, or leaf, represent class labels while the branches represent conjunctions of features leading to class labels. A decision tree where the target variable takes a continuous value, usually numbers, are called Regression Trees. The two types are commonly referred to together at CART (Classification and Regression Tree).</p><p>Each CART model is a case of a Directed Acyclic Graph. These graphs have nodes representing decision points about the main variable given the predictor and edges are the connections between the nodes. In the Loan scenario above the $30-$7ok would be an edge and the “Years Present in Job” are nodes.</p><p>As the goal of a decision tree is that it makes the optimal choice at the end of each node it needs an algorithm that is capable of doing just that. That algorithm is known as Hunt’s algorithm, which is both greedy, and recursive. Greedy meaning that at step it makes the most optimal decision and recursive meaning it splits the larger question into smaller questions and resolves them the same way. The decision to split at each node is made according to the metric called <strong>purity</strong>. A node is 100% impure when a node is split evenly 50/50 and 100% pure when all of its data belongs to a single class.</p><p>In order to optimize our model we need to reach maximum purity and avoid impurity. To measure this we use the Gini impurity, which measures how often a randomly chosen element is labeled incorrectly if it was randomly labeled according to distribution. It is calculated by adding the probability, pi, of an item with the label, i, being chosen multiplied by the times the probability (1–pi) of a mistake categorizing the time. Our goal is to have it reach 0 where it will be minimally impure and maximally pure falling into one category.</p><p>The other metric used is information gain, which is used to decide what feature to split at each step in the tree. This is calculated in the following way in a nicely laid out equation made by Wikipedia,</p><pre>Information Gain = Entropy(parent) - Weighted Sum of Entropy(Children).</pre><p>While this is a great model it does present a large problem by resulting in a model that only stops when all the information is in a single class or attribute. At the expense of bias the variance for this model is massive and will definitely lead to over fitting. “Decision-tree learners can create over-complex trees that do not generalize well from the training data.” So how do web combat this. We can either set a maximum depth of the decision tree (i.e. how many nodes deep it will go (the Loan Tree above has a depth of 3) and/or an alternative is to specify a minimum number of data points needed to make a split each decision.</p><p>What are the other disadvantages does a Decision Tree have: It is locally optimized using a greedy algorithm where we cannot guarantee a return to the globally optimal decision tree. It is an incredibly biased model if a single class takes unless a dataset is balanced before putting it in a tree.</p><p>While there are disadvantages there are many advantages to Decision Trees.</p><p>They are incredibly simple to understand to understand due to their visual representation, they require very little data, can handle qualitative and quantitative data, it can be validated using statistical sets, it can handle large amounts of data and it is quite computationally inexpensive.</p><p>I hope this article has helped you better understand Decisions Trees. For the coding and more knowledge about them I highly encourage you to check out Scikit-Learns document on Decision Trees.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22975f00f3e1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/what-is-a-decision-tree-22975f00f3e1">What is a Decision Tree?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Collision of Terminal and Github]]></title>
            <link>https://medium.com/@taplapinger/a-collision-of-terminal-and-github-e14bd07a5d0a?source=rss-b825421e07fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/e14bd07a5d0a</guid>
            <category><![CDATA[terminal]]></category>
            <category><![CDATA[github]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[git]]></category>
            <category><![CDATA[repositories]]></category>
            <dc:creator><![CDATA[Thomas Plapinger]]></dc:creator>
            <pubDate>Sat, 29 Jul 2017 17:31:22 GMT</pubDate>
            <atom:updated>2017-09-26T01:52:31.212Z</atom:updated>
            <content:encoded><![CDATA[<p>For any Data Scientist getting their first crack at exploring the world of data must begin with the knowledge of how to properly use two things before even getting started, Terminal and Github. Particularly, since the two go hand in hand in building your data profile on your personal computer and to the greater data science world.</p><p>Before we get to GitHub, we must learn the essentials of terminal:</p><p>When you open Terminal you will get a message of when it was last opened and the root directory. The root directory is the top folder on your laptop. For Mac users that will be your User folder. It should look like this,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*FsnAK-rxJcbxNCdzSUmVEg.jpeg" /></figure><p>To see what is inside a folder you simply have to type “ls” after the $ sign and Terminal will show you what the contents of that folder are.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*ftXogF5ofXY1ND0zNwuz4Q.png" /></figure><p>To dive deeper into any folder you can then use the command cd followed by the name of the folder. For the above list to access the Documents folder I would simply type “cd Documents”. If you wanted to go back a step and exit a folder all you would have to do is enter “cd ..” A nice shortcut when trying to access long named folders is to type the beginning of the file name and, if it is unique to the rest of the items in that folder, simply press the tab button and the rest of the file name will appear.</p><p>Keep in mind that Terminal is case sensitive.</p><p>Now we can start to go deeper and deeper into different directories throughout your computer. However, if you are time conscious and do not want to have to go through each folder everytime you open up Terminal to reach a file you can use what is known as a relative path. This is a considerably shorter piece of code than what is known as an absolute path as shown below.</p><p>Absolute Path to file:</p><pre>open Users/You/Documents/GA/Blogs/data.csv</pre><p>Relative Path to file:</p><pre>open c/data.csv</pre><p>or,</p><pre>open ./c/data.csv.</pre><p>To do the same process for folders you can simply type “cd ~/YourFolder” to skip many a cd and go right to your prime location.</p><p>So what does all of this have to do with Github?</p><p>Before we get to that here is some brief information on what Git is as defined by the wonderful source of Wikipedia —</p><p>“ a version control system (VCS) for tracking changes in computer files and coordinating work on those files among multiple people. It is primarily used for source code management in software development, but it can be used to keep track of changes in any set of files. As a distributed revision control system it is aimed at speed, data integrity, and support for distributed, non-linear workflows”</p><p>Now we dive deeper into the Git world into Github. Github is a hosting service for Git repositories from individual and group accounts compiled of codebases. It allows a sort of social network for programmers and data scientists by allowing you to follow and favourite projects across the site.</p><p>We have finally reached the convergence of Github and Terminal when accessing individual files, loading them to your local computer, and then returning them to Github.</p><p>The first step is in this process is that you selected Github repository you will need to “Fork” the repository to your computer. To do this you simply click the “Fork” button in the upper right hand corner and then select your repository and it will now appear on your Github page. This now allows you to keep track of any changes and new versions of the repository on your page without affecting the primary repository.</p><p>Once you have completed forking the repository you must now clone it by opening the clone or download tab and then highlighting and copying the SSH Key code. Before you grab the repositories files you should go into your terminal and “cd” to the folder you would like the Github project to be located in for sake of easily finding it later. From here we now save the repositories files on our computer using terminal with the easy command of “git clone” shown in the example below:</p><pre>git clone <a href="https://git.generalassemb.ly/taplapinger/1.04-control-flow.git">https://git.generalassemb.ly/taplapinger/1.04-control-flow.git</a></pre><p>Do not clone a repository into another repository.</p><p>Now you are able to do whatever you want with the repositories files whether that be adding new documents or python notebooks or really any other document it can all be added to your local computer.</p><p>Once you have completed your goals with the repo on your local computer you can now upload to your repository on Github. You should be in the folder you wish to upload. I would advise against uploading large datasets as it can result in overloading Github.</p><p>Now to upload the file we start with entering:</p><pre>git add.</pre><p>this tells Git that we are ready to add the folder to the repository before we commit the changes using,</p><pre>git commit -m&quot;Message&quot;</pre><p>Anything can be put in message, but I recommend calling it the name of the folder-your initials so if others access your repository they know which folder is yours.</p><p>Finally, in Terminal we do,</p><pre>git push</pre><p>This will upload all your files to your repository. If you wish to show your progress to the larger repository you “forked” it from earlier simply open the project file on your Github page click the “New pull request” button and it should upload your items to the larger repository for others to view.</p><p>I hope that this guide has been helpful.</p><p>If you would like a more visual version of the entire process on Github I highly suggest <a href="https://medium.com/u/45af05e64a7d">Mark Mummert</a> entitled “Submitting Work in Github.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e14bd07a5d0a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pandas Basics Walkthrough Youth Unemployment]]></title>
            <link>https://medium.com/@taplapinger/pandas-basics-walkthrough-youth-unemployment-128064704782?source=rss-b825421e07fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/128064704782</guid>
            <category><![CDATA[world-bank]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Thomas Plapinger]]></dc:creator>
            <pubDate>Mon, 17 Jul 2017 01:47:48 GMT</pubDate>
            <atom:updated>2017-09-26T01:47:40.084Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eWtxSsrJe2ZRZrTzMHd11A.jpeg" /></figure><p>For anyone using Python to learn Data Science for the first time one of the greatest discoveries you will come across in manipulating and analyzing your data is the Pandas software library.</p><p>Pandas allows you to create data structures and operations for managing numerical tables as well as time series data. It offers a world of features including: Integrated indexing, data aligning, managing missing and duplicated data, data reorganization, data merging and joining, groupy by engine that allows split-apply-combine operations for your dataframe, and a plethora more of functions that reveal whole new worlds of information about your data.</p><p>We will be walking through some of the beginner level steps that can help you managing and manipulate your data. Our dataset will The <a href="https://www.kaggle.com/sovannt/world-bank-youth-unemployment">World Banks Youth Global Unemployment Rate information from 2010–2014</a> from Kaggle. I personally will be using the program Jupyter Notebook to input and analyze the data, but there are many other programs that are able to use the Pandas program.</p><p>Our first step is to import the Pandas Program and the dataset from the downloaded csv.</p><pre>import pandas as pd #imports the Pandas Program</pre><pre>data = pd.read_csv(&quot;../Blog/API_ILO_country_YU.csv&quot;)<br>#Imports csv file from CPU location and labeled as Data</pre><p>Next comes our first chance to look at the data.</p><pre>data        #will show first 30 and last 30 rows<br>data.head() #reveals the first 10 rows data(shown below)<br>data.tail() #will show the last 10 rows data</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*dDU8mOjWF7C_IVik0_e2rg.png" /></figure><p>Now we examine the rest of the Unemployment Data.</p><pre>data.index    #Shows us the Tables Labels as an Index<br>data.columns  #Shows index of Column Names<br>data.dtypes   #Reveals the data types of each Column<br>              #Pandas Datatypes are Object, Int64, float64, and   datetime64. Their native python equivalents are String, Integer, and  Float. Datetime is unique to Pandas.</pre><pre>data.shape    #Number of Rows and Columns</pre><p>A critical tool to know when sifting through data is how to select a column.</p><pre>data[&#39;Country Name&#39;] #selections</pre><p>One of the most useful tools for checking the validity of your data is describing it. This function shows us the count, number of unique values, values with the greatest frequency for categorical data as well as the mean, standard deviation, min and max, and the 25%, 50%(median), and 75% percentiles for the numeric data.</p><pre>data.describe()                  #Describe the Numerical Columns<br>data.describe(include=[&#39;object&#39;])#Describe all Object Columns</pre><pre>data.describe(include=&#39;all&#39;)     #Describes All Columns(Shown Below)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*WvdnPO_XFyH0l43fUgpFvA.png" /></figure><p>Next we will learn how to search through the dataframe for rows containing specific values using Booleans</p><pre>data[data[&#39;2010&#39;] &gt; 30] <br>#All Countries where Youth Unemployment is greater than 30%<br>data[(data[&#39;2011&#39;] &gt; 30) &amp; (data[&#39;2011&#39;] &lt;40)] <br>#All Countries with Youth Unemployment greater than 30% <strong>and</strong> less than 40% (Shown Below)<br>data[(data[&#39;2011&#39;] &gt; 30) | (data[&#39;2011&#39;] &lt;40)]<br>#All Countries with Youth Unemployement greater than 30% <strong>or</strong> less than 40%</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/1*mO3TpLvs6ash52a_PuseMw.png" /></figure><p>You may also find it useful to sort your data in ascending(default) or descending order</p><pre>data[&#39;Country Name&#39;].order() #sorts column<br>data.sort(&#39;Country Name&#39;)    #sorts Dataframe by single column<br>data.sort(&#39;Country Name&#39;, ascending=False) #Descending Order</pre><p>Perhaps one of the most important aspects of Pandas is the ability to Rename, Add and Remove a column(s)</p><pre>#Renaming Columns<br>data.rename(columns=[&#39;2010&#39;:&#39;10&#39;,&#39;2011&#39;:&#39;11&#39;,&#39;2012&#39;:&#39;12&#39;,&#39;2013&#39;:&#39;13&#39;,&#39;2014&#39;:&#39;14&#39;})<br>#make it permanent by adding , inplace=True after }, but before )</pre><pre>#Add Column as Function of Existing Columns<br>data[&#39;Country&#39;]= data[&#39;Country Name&#39;] + data[&#39;Country Code&#39;]<br>#Instead of Armenia and ARM being in separate columns now there are under the Country column as Armenia ARM</pre><pre>#Removing Column <br>#Since Country Column is useless we will now delete it</pre><pre>data.drop(&#39;Country&#39;, axis=1, inplace=True) <br>#Axis=1 for Columns<br>#Axis=0 for Rows</pre><p>The next vital part of Pandas knowledge is managing duplicate and missing data.</p><pre>#Handling Duplicate Rows<br>data.duplicated().sum() #The Number of Duplicate Rows<br>data[data.duplicated()] #Show Only Duplicate Rows<br>data.drop_duplicates()  #Drop All Duplicate Rows</pre><pre>#Managing Missing Data Points<br>data.isnull()                       #Shows Missing Values as True<br>data[data[&#39;Country Code&#39;].isnull()] #Shows Rows Where Country Code   is Missing<br>data.isnull()                   #Finds Missing Values in Dataframe<br>data.dropna()                   #Drops row if <strong>Any </strong>value is missing<br>data.dropna(how=&#39;all&#39;)          #Drops row if <strong>All</strong> values are missing</pre><pre>data.fillna(value=&#39;NA&#39;, inplace=True) <br>             #Fills in Missing Datapoints with &#39;NA&#39;</pre><p>There is a mountain of tools available to you with Pandas beyond my own skills and I encourage you to visit the program’s website for further documentation and functions(<a href="http://pandas.pydata.org/">http://pandas.pydata.org</a>).</p><p>May your journey through the world of Pandas provide you as much information as you desire.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i1vVm3EqqDIkyucD0079wg.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=128064704782" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is the Best Football League in Europe?]]></title>
            <link>https://medium.com/@taplapinger/what-is-the-best-football-league-in-europe-460a32704e05?source=rss-b825421e07fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/460a32704e05</guid>
            <category><![CDATA[premier-league]]></category>
            <category><![CDATA[soccer]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[bundesliga]]></category>
            <category><![CDATA[la-liga]]></category>
            <dc:creator><![CDATA[Thomas Plapinger]]></dc:creator>
            <pubDate>Sun, 16 Jul 2017 01:16:56 GMT</pubDate>
            <atom:updated>2017-09-26T01:45:48.939Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*FO6ajV6T5rUjjL1sfWZ9sw.jpeg" /></figure><p>An almost continuous debate amongst football fans is “What is the best league in Europe?” Is it La Liga (Spain) with Barcelona and Real Madrid battling it out in El Classico? The Bundesliga (Germany) with the master class of Bayern Munich’s dominant play? Or is it the Premier League (England) with an all out battle between at least six clubs for the title each year?</p><p>One of the prime indicators for most football fans is how competitive is the league in the fight for the championship. For the sake of a similar timeline the data used in this article will begin in 1992 with the start of the Premier League replacing Division One in England and two years after the Germany reunification, which created a larger number of teams in the Bundesliga (Though no team from the former E. Germany has ever won the Bundesliga).</p><p>Since 1992 following teams have won the title in what is seen as the three top leagues of Europe:</p><p>Bundesliga:</p><p>FC Bayern Munich (15), Borussia Dortmund (5), VfB Stuttgart (2), SV Werder Bremen (2) VfFC Kaiserslautern (1), and VfL Wolfsburg (1)</p><p>La Liga:</p><p>Barcelona(13), Real Madrid(8), Atletico Madrid(2), Valencia(2), Deportivo La Coruña(1)</p><p>Premier League:</p><p>Manchester United(13), Chelsea(5), Arsenal(3), Manchester City(2), Blackburn(1), Leicester City(1)</p><p>While it is clear that FC Bayern Munich, Barcelona, and Manchester United have a clear lead in Winners Medals the other factor it is important to note is how close were these teams to the 2nd place team or the teams in 2nd to 4th place (often seen as the top teams in a league) when they won their titles.</p><p>My first measurements for determining how competitive a league is will be are what is the points difference between 1st place and 2nd, what is the goal difference between 1st place and 2nd, and what is the points difference between 1st place and 4th place. For those unfamiliar with the points system per game in Football tables are as follows: 0 points = Loss, 1 point = Draw, 3 Points = Win.</p><p>The league with the smallest average gap between 1st and 2nd place is La Liga with an average of 5.29 points or less than a 2 game difference. However, La Liga also has the largest average gap between 1st and fourth place, 18.42 or 6 games. Indicating that though the battle for 1st place may be fierce there is not such veracity between the first and fourth place teams. In La Liga, more often than not, the two teams battling it out are Barcelona and Real Madrid with the occasional appearance by other teams like Atletico Madrid.</p><p>The league with the smallest gap between 1st and 4th place is, rather surprisingly, the Bundesliga with an average gap of 15.625, or just over 5 games. However, the German league finds itself in the opposite situation as La Liga where it has the largest gap between 1st and 2nd of 7.04, or 3 games. That number has largely increased because of the results of the last few seasons where the German Champion, Bayern Munich, has won by at least 10 points and at most 25 each year. In fact, between 2013 and 2017 the average point difference is 15.8(27.2 for 1st and 4th place) while the fifteen years prior the average differential is 5.25 points (12.95 for 1st and 4th place). From this, we can infer that while the Bundesliga is an extremely competitive league in last five seasons it has become a one-team show from Bayern Munich.</p><p>Our other league, the Premier League, has nicely fallen almost directly in between the other two leagues with an average point differential between 1st and 2nd of 6.52 and an average 1st and 4th point differential of 16.72. Our third metric for measuring the competitive nature of the leagues is the goal difference between first and second place. In this category, the Premier league is well ahead of its competitors. The leagues average is 6.88 while La Liga’s is 9.41 and the Bundesliga 12.00. This metric shows us that while the Premier Leagues point differential may not be at the forefront of our other two measurements they may only be that size because a team won or lost a game by a goal or two. I may look into the frequency of high scoring games across the three leagues at a later date, but I encourage others to look into the matter in the meantime.</p><p>Our other and final metric for measuring the “Greatness” of a European League against its competitors is how a countries team performs in the Champions and Europa League. For those unaware, the Champions League is where each League in Europe sends the top 2–4 teams (depending on country and league) to compete against other best teams in Europe in a Group to Knockout Stage competition. While the Europa, formerly the UEFA cup, is compiled over 2–4 teams from each league that placed below those in the Champions League spot. We will be looking at the final and semi-final of both competitions to see what percentage of the time a participant from our three focus countries participated and the amount that each country has won since 1992.</p><p>In the Europa League it there is a clear front-runner for highest participation and highest number of victories. La Liga is the clear victor here winning 32%, 8, of the last 25 Champions Leagues and unsurprisingly has the highest participation rate at 44%. While the Premier League and Bundesliga are way behind with 20% of the victories split between the two. The Premier League teams have participated in slightly more finals than the Bundesliga with participation rates of 28 and 20 respectively. This trend does not fray when we go a stage back in the competition to the semi-finals where of the 100 teams who have participated in the 25 seasons 23% were Spanish, 14% German, and only 10% were English.</p><p>In Europe’s Premier competition, the Champions League, we see even bigger numbers showing support for La Liga. A team from La Liga has won the competition 44%, or 11, and participated in 64% of the finals. While the Premier League and Bundesliga have a matched participation rate of 32% each. However, with a Premier League team winning one more title (4) than a Bundesliga team (3) the English teams just barely edge out the Germans. A rarity for English Football on the international stage. La Liga continues to out participate the other nations in the semi-finals by making up almost 30% of the teams that have participated while teams from the Premier League and Bundesliga makes up 21% and 15%.</p><p>While it is clear that when it comes to performance against each other in the Europa and Champions League competitions that La Liga teams are vastly out performing their European compatriots it is not such a clear picture when comparing leagues side by side. Each league has its own merits to claim that it is the best league in Europe. La Liga can claim the lowest point difference between 1st and 2nd with Barcelona, Real Madrid, and Atletico Madrid battling it out. The Bundesliga can claim the gap between its first and fourth place team is the least, but until Bayern Munich’s dominance over the League returns to previous levels I don’t believe it can be considered for most competitive. The Premier League can claim to have the smallest goal differential between 1st and 2nd and a larger variety of recent winners of the title. In the end, the argument is just banter between fans taking pride in the respective leagues. It appears this debate will continue until the final whistle is blown on the final match.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=460a32704e05" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Convergence of Language with Code]]></title>
            <link>https://medium.com/data-science/a-convergence-of-language-with-code-a3965d8953e3?source=rss-b825421e07fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/a3965d8953e3</guid>
            <category><![CDATA[english]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[northern-sotho]]></category>
            <category><![CDATA[french]]></category>
            <category><![CDATA[language]]></category>
            <dc:creator><![CDATA[Thomas Plapinger]]></dc:creator>
            <pubDate>Mon, 03 Jul 2017 21:09:40 GMT</pubDate>
            <atom:updated>2022-09-26T15:10:47.301Z</atom:updated>
            <content:encoded><![CDATA[<p>Bonjour, Je m’appelle Thomas, Nna ke Thomas, For name(x): return Thomas. In my life, I have learned multiple languages each with their own corks, grammar, practicalities, and each under different learning styles. Studying French for three years in High School in London from teachers, interactive language labs, and a summer immersive program in Southern/Central France. In the wonderful villages of Matlerekeng and Enable, South Africa during my Peace Corps service I learned Sepedi, also known as Northern Sotho, from live interactions and hour long sessions with native speakers in small groups. Now, confronted with a new language and new approach I have begun to learn the language of Data Science (i.e. Python, SQL, and statistics, etc.) at a General Assembly 12-Week Bootcamp. Each of these experiences has been taught to me with different languages come new learning methods and new problems with each offering their own challenges and growing sets of knowledge, but as my repertoire expands the similarities between them all emerge.</p><p>If you were given the choice to learn French, Sepedi, and a computer language you might, at first, think you would have to approach each differently. However, my experiences have shown me that while the individual styles of teaching may be different the overall methods with the best results are almost identical across the board. They all begin with teaching you the smaller details so that when you approach the larger phrases you know what is going on under the hood. In spoken languages this can be knowing the differences between I, me, her, him, us while python code can be learning import x as, statistical math, and the difference between ==, = , or !=. In fact, the most basic method of understanding what is going on in all three languages is being able to translate into your native tongue. For coding that is leaving as many easy to read comments along with your code as possible in order for others to understand your goals and method. For languages like French and Sepedi that is knowing how to say “Je ne comprends pas. Qu’Est-ce que c’est en Anglais ou français traditionnel?”(I don’t understand. What is it in English or traditional French? or “Aowa, ga re kwisisa. ke sekgowa ke __ ke English ne?”(I don’t understand. Is translated ___ in English what? ). All of these provide a way for the learner to relate their new language to their native tongue. After learning the nitty gritty and everyday phrases the next step of learning is to be immersed in the language whether that is only speaking French in the classroom, living with a family that only speaks Sepedi or very fractured English, or surrounding yourself with code from nine to five, Monday to Friday. The immersion not only forces you to learn in order to survive, sometimes literally, but also allow you to pick up the true nature of a language. By “true nature” of a language I mean that most of the time when a language is taught it is done in the textbook/upper class terminology, which often means when taken from the classroom to the real world the terminology changes. When fully immersed you learn the shorthand for things, how tone and context can change the entire meaning of a word, and connects you with the culture around you. The end result of the immersion is that words or code becomes second nature and even if you spend years away from them there is always a part of your brain that remembers when thrown back into the culture again.</p><p>In spoken languages words changes to where the something is spoken depends on gender, connecting words, and tenses. For example, in French to when going from I to we it goes from Je to Nous and in Sepedi it goes from Rena re to Nna ke. While computer code does not change in the same way languages it does change upon similar variables, but by other names. For coding it is list type, a for or while loop, if-else statements, and an array of computer packages each with their specific role to play in interpreting the situation at hand much like spoken language verb tables.</p><p>While Romance, Bantu, and Computer languages would at first seem to be as close as apple and oranges separated by thousands of miles and created centuries apart there are several over lapping features across all three. Perhaps the most common string is that they are quite literally in their direct translations into plain English. An example of this can be something as simple as saying “It’s very hot. It’s 85 degrees!” has to be spelled/spoken out in clear terms step by step in each language.</p><p>For French: C’est très chaud. C’est quatre-vingt-cinq degrés!</p><p>In English=It is very Hot. it is forty-twenty-five degrees</p><p>For Sepedi: Lehono go a fiša fiša. Go na 85 dikagto!</p><p>In English= Today is a hot hot. It is 85 degrees!</p><p>For Python:</p><pre>temperature=85<br>if temperature &gt;= 85:<br>    return &quot;It is hot. It is &quot; +str(tempeture) + &quot;degrees!&quot;<br>else if temperature &lt;85:<br>    return &quot;It is not so hot. It is&quot; +str(tempeture) + &quot;degrees.&quot;<br>Output[] It is hot. It is 85 degrees!</pre><p>In a world filled with thousands of languages from different continents and different terminals, it is often lost on the world how easily and fluidly they are all connected. They may not share the same words or origins, but when you learn each of them you can see their similarities in phrasing, the situational changes, and their want to understand each other’s aspect of their words. In the end, while on the face of it all our communications may appear different they are all attempting to reach out to another to help others understand what is going on in each other’s our worlds.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a3965d8953e3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-convergence-of-language-with-code-a3965d8953e3">A Convergence of Language with Code</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>