<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jacques Coetzee on Medium]]></title>
        <description><![CDATA[Stories by Jacques Coetzee on Medium]]></description>
        <link>https://medium.com/@plan__b?source=rss-549b0f04c64a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*T02CQFb3DYZ4G8TD.jpeg</url>
            <title>Stories by Jacques Coetzee on Medium</title>
            <link>https://medium.com/@plan__b?source=rss-549b0f04c64a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 22:10:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@plan__b/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Tragedy of and hope for the commons: Part 1]]></title>
            <link>https://medium.com/@plan__b/tragedy-of-and-hope-for-the-commons-part-1-850fd77e5178?source=rss-549b0f04c64a------2</link>
            <guid isPermaLink="false">https://medium.com/p/850fd77e5178</guid>
            <category><![CDATA[governance]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[cryptocurrency]]></category>
            <category><![CDATA[politics]]></category>
            <dc:creator><![CDATA[Jacques Coetzee]]></dc:creator>
            <pubDate>Sun, 14 Nov 2021 12:51:57 GMT</pubDate>
            <atom:updated>2021-11-14T13:35:33.062Z</atom:updated>
            <content:encoded><![CDATA[<h4>Self-financed contract enforcement for governance</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hqVssp4jd8rScKtz.jpg" /><figcaption><a href="https://www.nytimes.com/2019/11/12/books/review/the-iliad-a-graphic-novel-adaptation-gareth-hinds.html">Animal Farm — illustrated (2019)</a></figcaption></figure><p><em>This post is the first part of a series on commons-based peer production which applies the framework of economist Elinor Ostrom — specifically with regards to the so-called “self-financed contract enforcement game” — to modern infrastructure technologies such as the blockchain. Part 1 focuses on “contract enforcement” and Part 2 on “self-financed” or the sustainability thereof.</em></p><p><em>I know there’s been quite a bit written about this topic (see below for references), but the main purpose of this is to help consolidate my ideas about the topic.</em></p><p>In the first week of November 2021, the 26th United Nations Climate Change Conference is held in Glasgow, where government representatives from around the globe discuss and agree on solutions to the ensuing climate crisis. That same week, South Africans — an <a href="https://www.ipsos.com/en-za/third-south-africans-will-decide-outcome-local-government-elections">estimated</a> record low of 30% of citizens — went to the election polls to vote for their local government representatives.</p><p>Both these institutions are organised forms of collective decision making regarding common resources: one on a macro-scale (global governance)— dealing with carbon emissions from nation states to help reduce the impact climate change. The other micro (local governance) — dealing with service delivery issues like access to jobs, water, electricity and refuse removal in the streets of <a href="https://www.news24.com/news24/southafrica/news/low-soweto-voter-turnout-as-residents-boycott-elections-embark-on-service-delivery-protests-20211101">Soweto</a> to <a href="https://cer.org.za/virtual-library/judgments/high-courts/kgetlengrivier-concerned-citizens-another-v-kgetlengrivier-local-municipality-others-interim-order-and-agreementcourt-order">Koster</a>.</p><p>These events are examples that employ mechanisms that fits into the centuries-old study of governing the commons — ultimately, asking the question:</p><blockquote>How do we most optimally make decisions concerning complex, uncertain situations that involve a common pool of resources and multiple stakeholders?</blockquote><h3>Tragedy of the commons</h3><p>In 1968, ecologist Garret Hardin released a famously titled paper the <a href="https://www.bu.edu/cgs/files/2010/03/Tragedy-of-Commons-essay.pdf"><em>Tragedy of the Commons</em></a><em>. </em>He put forward this thought experiment of herders sharing a grazing meadow.</p><p>Each herder receives a direct benefit from his own animals and suffers delayed costs from the deterioration of the commons when his and others’ cattle overgraze. Each herder is also motivated to add more and more animals because he receives the direct benefit of his own animals and bears only a share of the costs resulting from overgrazing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pczJ1HyB0vc0tPYUJdBqKA.png" /><figcaption>When each herdsman put two cows on the commons (Common A), the productivity of each cow is 1 (e.g., 1 litre of milk). When the herdsmen introduce more cattle (Common B), the production per animal is reduced (0.8) due to increased density but the total production is higher. When the carrying capacity is approached (Common C), also the total productivity is further reduced (0.6) while being unsustainable.</figcaption></figure><p>The individual herder’s interests to make more profits (and add more cattle) conflicts with that of the group’s, and because they act independently according to their short-term interests, the result of the collective overgrazing depletes the meadow.</p><p>The above scenario can be visualised by the following graph which shows the decrease in “gain per animal” (e.g. milk in litres) as the “gain per acre” (e.g. available meadow in square metres) decreases — a concept not too dissimilar to that of Hardin’s.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/376/0*PKd1rjxvxsY5O2vq.jpg" /><figcaption><a href="https://www.sciencedirect.com/science/article/pii/B9780128144749000050">Lynn et al. (2020)</a></figcaption></figure><p>This is the main point of Hardin’s argument. Resources shared by individuals acting out of self-interest to maximise their own benefit, result in the depletion of the commons.</p><p>This sentiment has been voiced since the time of Aristotle, when he wrote in defence of private property:</p><blockquote>“What is common to the greatest number has the least care bestowed upon it. Everyone thinks chiefly of his own, hardly at all of the common interest.”</blockquote><p>Commons is essentially resources accessible to all members of a society, such as the farming land we use, the air we breath or the fish we catch.</p><p>But even though common resources — or open-access resource systems — are prone to <a href="https://www.nature.com/articles/s41467-019-09246-2">collapse due to overuse</a>, there are many examples where cooperation has led to common benefit.</p><p>Today, most societies tend to protect common resources through <a href="https://onlinelibrary.wiley.com/doi/10.1111/j.1467-8489.2007.00425.x">markets and government regulation</a>. However, these systems are arguably suboptimal. We need only consider the <a href="https://theconversation.com/south-africa-is-paying-a-heavy-price-for-dysfunctional-local-government-102295">dysfunctional municipalities</a> in South Africa in failing their mandate to deliver basic services — such as water and electricity — to the people. Or the <a href="https://www.theguardian.com/environment/2021/nov/06/cop26-week-one-the-impression-of-progress-but-not-nearly-enough">“too-little-too-late” commitments</a> by world leaders to address nations’ impact on climate change.</p><p>One of the biggest contributors to the field of common governance arguably comes from Nobel laureate economist Elinor Ostrom, who’s large portion of work stems from a refutal of Hardin’s findings. She also proposes an alternative to using markets and government regulations to optimise the use of common resources for all stakeholders.</p><h3>Governing the commons</h3><p>In <a href="https://www.actu-environnement.com/media/pdf/ostrom_1990.pdf">Governing the Commons</a>, Ostrom (1990) formalises Hardin’s theoretical puzzle by applying it to the framework of the prisoner’s dilemma. In a prisoner’s dilemma game, each player has a <a href="https://www.sas.upenn.edu/~haroldfs/540/handouts/french/dirigism/pdgloss.htm">dominant strategy</a> in the sense that the player is always better off choosing this strategy — to <em>defect</em> — no matter what the other player chooses. When both players choose their dominant strategy they produce an equilibrium that is the third-best result for both.</p><p>To illustrate, suppose we think of the players as herders using a common grazing meadow. This meadow has a limit to the number of animals that can graze there per season. For a two-person game, the “cooperate” strategy can be thought of as grazing L/2 animals for each herder, where L refers to the limit of animals to graze. The “defect” strategy is for each herder to allow as many animals as possible to graze, assuming that this number is greater than L/2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ve1KSex0gyJ_tfl0tO_iLQ.png" /><figcaption>The potential outcomes can be summarised as follows: 1) If both herders “cooperate” and limit their grazing to L/2, they will obtain 10 units of profit. 2) Whereas if they both choose to “defect”, they will obtain zero profit. 3) If one of them limits his number of animals to L/2, while the other grazes as many as he wants, the “defector” obtains 10 units of profit, while the “sucker” obtains -1.</figcaption></figure><p>Ostrom further explores three more “games” or scenarios with different outcomes and equilibriums— each one sub-optimal to both herders. However, in the fifth and final game a so-called “alternative solution” is discussed.</p><p>The final game is grounded in the idea that the herders commit to a <em>binding contract</em> to achieve a cooperative outcome. A binding contract is interpreted as one that is unfailingly enforced by an external actor.</p><p>In other words, the only cost to bare is that of sharing the upkeep costs of the independent enforcer both herders have committed to — such as an arbitrator.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nr9hBrDw4CgwEMMyQsT2JQ.png" /><figcaption>The equilibrium, in this scenario, is for both herders to share equally the sustainable yield levels of the meadow (cooperate, cooperate) and the costs of enforcing their contract (agree, agree) so long as each herder’s share of the cost of enforcement is less than 10 (10-e/2). Cooperation is only maintained when both parties have agree to the contract.</figcaption></figure><p>Ostrom’s theoretical framework has been applied to case studies regarding common resources such as fisheries or irrigation infrastructure, mostly in the context of laws and regulation.</p><p>And since the advent of the internet, her ideas were subsequently employed to understand how communities develop and maintain digital commons.</p><h3>Smart contracts for community governance</h3><p>Reflecting Ostrom’s commons resource management equilibrium, namely the “self-financed contract enforcement game”, blockchains like Ethereum have introduced the ability to create <em>smart contracts, </em>whereby clauses are encoded in a manner by which they are automatically enforced and executed without the need for a central authority — as opposed to common law.</p><p>For instance, this vending machine is governed by a specific set of rules and conditions which are <em>deterministic</em> — in other words, it will always produce the same output (e.g. expected Y coins in change) from a given starting condition or initial state (e.g. X number of input coins given the cost of drink).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cXA13EwfcIHYWS4h.png" /><figcaption>A typical vending machine is programmed in a way that allows certain actions and state transitions based on the input. If you want to buy a can of coke that costs $2 and you only have $1 no matter how many times you try you won’t be able to get the drink. On the other hand, if you insert $3, the machine will give you a can of coke and appropriate change. (Source: <a href="https://finematics.com/smart-contracts-explained/">finematics</a>)</figcaption></figure><p>Extreme examples of institutions using smart contracts include decentralised autonomous organisations (DAOs). A DAO is a member-owned community that enables people to coordinate and self-govern themselves mediated by a set of self-executing rules deployed on a blockchain, and whose governance is decentralised.</p><p>Popular examples of DAOs (by <a href="https://coinmarketcap.com/view/dao/">token market cap value</a>) include those within the finance industry, also known as “decentralised finance” or DeFi. These include <a href="https://uniswap.org/">Uniswap</a> — a decentralised exchange and the lending protocol <a href="https://aave.com/">Aave</a>.</p><p>The degree of decentralisation of these protocols are <a href="https://insights.glassnode.com/uni-token-is-uniswap-really-decentralized/">arguable</a>, but for the sake of scope in this blog post, we won’t go down that rabbit hole. Ethereum founder Vitalik Buterin clarifies the terminology of DAOs in this <a href="https://blog.ethereum.org/2014/05/06/daos-dacs-das-and-more-an-incomplete-terminology-guide/">post</a> from 2014.</p><p>Here’s an ecosystem map of popular DeFi:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*OzLNm-Pr7qtLx4p9.png" /><figcaption><a href="https://thedefiant.io/defi-projects-map/">The Defiant (2020)</a></figcaption></figure><p>These DeFi “institutions” employ principles of Ostrom’s “self-financed contract enforcement game” to manage common resources (such as the organisations’ governance and associated tokens).</p><p>For example, many blockchain projects rely on a decentralised treasury governance system for development. As defined by <a href="https://www.lancaster.ac.uk/staff/zhangb2/treasury.pdf">Zhang</a> et al., a treasury system is a community-controlled and decentralised collaborative decision-making mechanism for sustainable funding of blockchain development and maintenance.</p><p>A popular implementation of this is that of blockchain prject Cardano’s governance protocol. In August 2021 <a href="https://bit.ly/3ACVEEP">Project Catalyst Fund6</a> launched in which over $4 million worth were distributed as per the community’s votes. People submitted proposals for projects that received funding to address real-world challenges and help to build Cardano’s ecosystem.</p><p>Examples of blockchain projects aiming to contribute to the governance through peer-to-peer communities include the <a href="https://commonsstack.org/">Commons Stack</a> project and the <a href="https://aragon.org/">Aragon DAO</a> platform.</p><p>The Commons Stack project was <a href="https://medium.com/giveth/introducing-the-commons-stack-scalable-infrastructure-for-community-collaboration-6886eb97413e">born</a> out of the research done by <a href="https://medium.com/@michaelzargham">Zargham</a> on alternative funding and governance mechanisms. By building “continuously funded digital economies based around community causes to bring about positive change in the world through cyber-physical commons management.”</p><p>The power of DAOs (and cryptocurrency for that matter) stems from the idea that for the first time in human history, individuals with common interests from around the globe — regardless of they’re geographical area, tax obligations, culture, etc. — can organise, govern and further their specific interests within the bounds of a protocol specified in code.</p><p>This in contrast to top-down hierarchical organisations such as nation states, multinationals and even non-profits, which rely people to operate within the convinces of simple law via constitutions and regulations interpreted by other individuals. Unlike a protocol — which is deterministic by design — these organisations are prone to vagueness, patronage networks, bias and general human-error.</p><p>The financing of these projects furthermore rely on various models, from initial coin offerings (ICOs), bonding curve offerings (BCOs), dynamic coin offering (DCOs) and initial exchange offerings (IEOs) to lotteries and many more.</p><p><em>Part 2 will further explore these funding models and is dedicated to the sustainability of commons-based peer production with regards to blockchain infrastructure.</em></p><p><strong>Further reading and references:</strong></p><ul><li><a href="https://medium.com/r?url=https%3A%2F%2Fwww.bu.edu%2Fcgs%2Ffiles%2F2010%2F03%2FTragedy-of-Commons-essay.pdf">The Tragedy of the Commons, Hardin (1968)</a></li><li><a href="https://www.actu-environnement.com/media/pdf/ostrom_1990.pdf">Governing the commons, Ostrom (1990)</a></li><li><a href="https://www.frontiersin.org/articles/10.3389/fbloc.2021.577680/full">Analysis of the Potentials of Blockchain for the Governance of Global Digital Commons, Rozas (2021)</a></li><li><a href="https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3272329">When Ostrom Meets Blockchain: Exploring the Potentials of Blockchain for Commons Governance, Rozas (2021)</a></li><li><a href="https://medium.com/giveth/introducing-the-commons-stack-scalable-infrastructure-for-community-collaboration-6886eb97413e">Introducing the Commons Stack: Infrastructure for Scalable Community Collaboration, Emmett (2019)</a></li><li><a href="https://medium.com/@wen_xs/crypto-manifesto-682df071d2">8 Nobel-Prize Winning Rules for Governance Design, Schadeck (2019)</a></li><li><a href="http://oscoin.io/oscoin.pdf">Open Source Coin: Trust and Sustainability in Open Source Communities, Sellier (2019)</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=850fd77e5178" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Auto fake news classifier using headlines]]></title>
            <link>https://medium.com/@plan__b/auto-fake-news-classifier-using-headlines-86c98fa5ab6e?source=rss-549b0f04c64a------2</link>
            <guid isPermaLink="false">https://medium.com/p/86c98fa5ab6e</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[fake-news]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Jacques Coetzee]]></dc:creator>
            <pubDate>Sun, 11 Oct 2020 13:05:57 GMT</pubDate>
            <atom:updated>2020-10-15T08:13:29.800Z</atom:updated>
            <content:encoded><![CDATA[<h4>The final model used to classify fake news has a weighted average F₁ score of 85% and above.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3yvPuypsqsZA16TK" /><figcaption>Photo by <a href="https://unsplash.com/@unitednations?utm_source=medium&amp;utm_medium=referral">United Nations COVID-19 Response</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Project Overview</h3><p>As part of the <a href="https://www.udacity.com/course/data-scientist-nanodegree--nd025">Udacity Nanodegree</a> capstone project, the following analysis attempts to distinguish between news headlines that are real and those that are fake.</p><p>This post details insights following data exploration as well as the NLP pipeline setup using custom transformers along with hyperparameter tuning using grid search.</p><p>The Github repo for this project can be found <a href="https://github.com/feeblefruits/fake_news_classifier">here</a>.</p><h3>Problem statement</h3><p>Even though disinformation often disguises itself as legitimate information, the assumption of this project is that there are defining stylistic features (such as the use of capitalisation and pronouns) that can be used to distinguish the two types of content.</p><p>Headlines are the first content web users are exposed to, most prominently via social media. The aim is to use machine learning to distinguish between fake and real content while identifying prominent characteristics associated with disinformation. The scope of this project therefore does not include other metadata such as article content, date of publication and source.</p><blockquote>The problem to be solved is therefore to automatically identify fake news by solely analysing the content headline or title text style. In the context of the project, “fake” news is defined as news that is intentionally meant to mislead based on false, misinterpreted or manipulated facts.</blockquote><p>The final model has a weighted average <strong>F₁ score of 86%</strong> using the Covid-19 misinformation dataset. The same model parameters are used to train on Fake News Corpus article titles which exceed 100,000 and <strong>scores 85%</strong>.</p><h3>Data Overview</h3><p>Two datasets are used in this project. Data exploration and initial model setup testing used a smaller dataset, while a much larger model is ultimately used.</p><h4>Covid-19 misinformation</h4><p>The initial data used for this project was <a href="https://towardsdatascience.com/automatically-detect-covid-19-misinformation-f7ceca1dc1c7">collected</a> by Susan Li in her excellent project earlier this year used to detect Covid-19 misinformation.</p><p>The table consists of the following columns: <em>title</em>, <em>text</em>, <em>source</em> and <em>label</em>. With a total of 1,159 records, there’s an almost-even split of 579 “true” articles and 490 “fake” ones.</p><h4>Fake News Corpus</h4><p>Upon expansion of this project, data from <a href="https://github.com/several27/FakeNewsCorpus">Fake News Corpus</a> was collected. This is an open source dataset composed of millions of news articles mostly scraped from a curated list.</p><p>For purposes of this project, 130,443 titles labeled “reliable” and “fake” were used to further train and test the model.</p><h3>Setting Up</h3><p><em>It should be noted that the following gists in this post, while functional, are included for explanatory purposes only. A lot of repetition and additional analysis are included in the Github repo.</em></p><p>All required libraries are imported, data read into the Notebook environment and some light cleaning are done for the initial setup.</p><p>The libraries used include conventional ones used for analysis such as Pandas and Numpy along with Re and Itertools. For the inevitable text processing, NLTK and Spacy are used. Given the initial focus being data exploration, Plotly and Seaborn are used for data visualisation.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e641b46b8fc58f788330792876ba18d/href">https://medium.com/media/6e641b46b8fc58f788330792876ba18d/href</a></iframe><p>Some example output of “fake” data are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q9ETLuSP7cgT_FqVe750eA.png" /><figcaption>Off the bat, it’s quite prominent to see that there are an over usage of capitalisation for “fake” news</figcaption></figure><p>While “true” labels include the following titles:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pXUE1q0QVt6TkC1pQ-QFwQ.png" /><figcaption>Many of these articles seem like explainer pieces, posing questions. The usage of question marks is likely to be a prominent feature.</figcaption></figure><h3>1. Data Exploration</h3><p>Inspired by and expanded on Li’s initial approach, the following key stylistic features are added to the table and used for analysis — ultimately contributing to distinguishing features between titles rated as “true” and “fake”:</p><ul><li>Ratio of capital letters</li><li>Total number of words</li><li>Ratio of stopwords</li><li>Usage of parts of speech</li></ul><h4>1.1 Capital letters</h4><p>The following code produces two charts, namely the distribution and box plot, that show the usage of capital letters to overall character ratio.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/141c24da84b95fdcfc980ccb646f528e/href">https://medium.com/media/141c24da84b95fdcfc980ccb646f528e/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*WzyCWmUAkFojbW7XVGB1og.png" /><figcaption>The ratio of capital letters to total characters is fewer in the “true” titles.</figcaption></figure><p>As shown in the sample of text, there’s clearly more capital letters used in titles labeled as “fake”. There’s also a distinct long tail of outliers with capital characters over 100.</p><p>A box plot presents similar findings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*CWtw5jJiUV5YyT7zKDDsXw.png" /><figcaption>The box plot shows that “true” article titles have a median of 3 while “fake” titles has 9 — a considerable difference in ratio of capital letters used.</figcaption></figure><p>In terms of the number of words and characters, there was an expectation that “fake” titles will contain more characters given the descriptive nature of this form of content — and need to generate “click bate”.</p><p>As <a href="https://theconversation.com/fake-news-why-people-believe-it-and-what-can-be-done-to-counter-it-70013">pointed out</a> by Simeon Yates, Director Institute of Cultural Capital, University of Liverpool, the “shareability” of titles could be a likely indicator of “fake” news.</p><blockquote>What fuels this and can we address it? First, the economics of social media favour gossip, novelty, speed and “shareability”. They mistake sociability for social value. There is evidence that “fake news” that plays to existing prejudice <a href="https://www.washingtonpost.com/news/the-intersect/wp/2016/11/18/this-is-how-the-internets-fake-news-writers-make-money/">is more likely to be “liked”</a> and so generate more revenue for the creators. This is no different than “celebrity” magazines. Well researched and documented news is far less likely to be widely shared.</blockquote><h4>1.2 Overall character count</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*qkVhSDBn-HdxjxjG7_fFYA.png" /><figcaption>Understandably, “fake” titles are skewed more to the left than “true” titles.</figcaption></figure><p>Given the finding above that “fake” titles contain far more characters and words than “true” titles, it was assumed that the former would also contain many more stopwords.</p><p>However, as the chart below shows, the distribution in ratio in total number of words to stopwords, seems fairly evenly distributed between both labels — with “true” titles having a slightly bigger ratio.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*W_Znv6_4PdwYRWgUp8b9PQ.png" /><figcaption>The stopword usage for “true” titles is much more relative to “fake” titles.</figcaption></figure><p>Further exploration of stopword usage — as detailed in the Jupyter Notebook — shows no other really prominent distinction between the two categories.</p><h4>1.3 Parts of speech usage</h4><p>By using Spacy’s clever POS tagging, nouns, adjectives and other parts of speech types can be identified and analysed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H1eNgcgBVVq4BbdLvtwF7Q.png" /><figcaption>Example of Spacy’s parts of speech recognition, in this case identifying “Coronavirus” and “the World Health Organization” as organisations. The “4.95” is correctly labeled as money.</figcaption></figure><p>The following gist, shows the code used to extract these parts of speech types.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e635653dbf45510de38ca4d3767ee9f6/href">https://medium.com/media/e635653dbf45510de38ca4d3767ee9f6/href</a></iframe><p>Considering the number of pronouns used in titles, there’s a clear distinction between the two categories, with “true” titles including more than “fake” ones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*-KJzGbfbItVLQ5sDLB1xVA.png" /><figcaption>The ratio in pronouns used is slightly larger for “true” titles relative to “fake” titles.</figcaption></figure><h4>1.4 Predicting on current features</h4><p>So far the dataframe includes the following features that could be used to classify the two categories:</p><pre>&#39;n_capital_title&#39;, &#39;n_characters_title&#39;, &#39;n_characters_text&#39;, &#39;n_words_title&#39;, &#39;n_words_text&#39;, &#39;n_stopwords_title&#39;, &#39;n_stopwords_text&#39;, &#39;n_noun&#39;, &#39;n_adjectives&#39;, &#39;n_verbs&#39;, &#39;n_pronouns&#39;, &#39;n_proper_nouns&#39;</pre><p>To get a preliminary indication of how strong these features are in determining the category of any given title, a logistic regression model is used.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c96f4cdf1f30c90ac47b75209d93201d/href">https://medium.com/media/c96f4cdf1f30c90ac47b75209d93201d/href</a></iframe><p>An accuracy score of <strong>78%</strong> isn’t bad at all considering there has been no Natural Language Processing (NLP) used yet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JgQkTtGfLqOFlPaX" /><figcaption>Photo by <a href="https://unsplash.com/@unitednations?utm_source=medium&amp;utm_medium=referral">United Nations COVID-19 Response</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>2. Setup NLP Pipeline</h3><p>The following libraries are imported to setup the NLP pipeline:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a5a7f109e68d69765329e13da1cc922e/href">https://medium.com/media/a5a7f109e68d69765329e13da1cc922e/href</a></iframe><p>The features analysed above during data exploration are extracted and added to the pipeline below.</p><p>A custom tokeniser to clean and lemmatise the text is defined and also included. As an example, the transformer StopWordCounter() is defined.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2012427f256f277bd35a946f93fc79e6/href">https://medium.com/media/2012427f256f277bd35a946f93fc79e6/href</a></iframe><p>Note that the Notebook includes multiple custom transformers for the Random Forest Classifier defined above.</p><p>Next, the model gets initiated with the customised pipeline and results retrieved.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/506f1a97c822688eef50681a9162db2e/href">https://medium.com/media/506f1a97c822688eef50681a9162db2e/href</a></iframe><p>The code above produces a confusion matrix table.</p><p>Using the Covid-19 disinformation dataset, the model produces a weighted average <strong>F₁ score of 81%</strong>.</p><blockquote><strong>Why is F₁ score used as a metric for success?</strong></blockquote><blockquote><strong>Accuracy</strong> can be used when the class distribution is similar while <strong>F₁</strong>-<strong>score</strong> is a better metric when there are imbalanced classes. Even though there’s an almost-even split of 579 “true” articles and 490 “fake” ones in the Covid-19 dataset, as initially found with the Fake News Corpus model, an unequal distribution has a massive impact on producing biased scores. Given the fake news classification problem, an equal or balanced scoring measurement is required.</blockquote><h3>Test multiple classifiers using updated pipeline</h3><p>Finally multiple classifier types are included in the pipeline. Whereas the above models include a Random Forest Classifier, its performance is tested against.</p><p>The following code prints the results of multiple classifiers.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dc5e79212e6a0a6dfae2d24b6ed2c061/href">https://medium.com/media/dc5e79212e6a0a6dfae2d24b6ed2c061/href</a></iframe><p>The output is as follows for each model included:</p><pre>---------------------------------<br>RandomForestClassifier(bootstrap=True, class_weight=None, criterion=&#39;gini&#39;,<br>                       max_depth=None, max_features=&#39;auto&#39;, max_leaf_nodes=None,<br>                       min_impurity_decrease=0.0, min_impurity_split=None,<br>                       min_samples_leaf=1, min_samples_split=2,<br>                       min_weight_fraction_leaf=0.0, n_estimators=&#39;warn&#39;,<br>                       n_jobs=None, oob_score=False, random_state=None,<br>                       verbose=0, warm_start=False)<br>-----------------------------------<br>fit_time  mean  15.30619223912557<br>fit_time  std  1.5044411739878019<br>score_time  mean  6.76354996363322<br>score_time  std  0.39624837654681416<br>test_score  mean  0.7952821990131009<br>test_score  std  0.02125308090319933<br>---------------------------------</pre><p>Besides the Random Forest (80%) classifier, other models that perform well — in terms of test score mean — include Logistic Regression (83%), Gradient Boosting (82%) and Ada Boost (82%).</p><blockquote><strong>Why is Logistic Regression used?</strong></blockquote><blockquote>Given the rather small dataset of 1,000 observations, Random Forest and Logistic Regression should perform well. Given the test score, the model selection for this design is therefore Logistic Regression.</blockquote><blockquote>In general Logistic Regression performs better when the number of noise variables is less than or equal to the number of explanatory variables.</blockquote><h3>Hyperparameter tuning</h3><p>Given the above performance, a focus on refining the Logistic Regression model should increase the accuracy score by at least 2–3%.</p><p>In order to further improve the model’s accuracy, however, a grid search is setup in order to test and identify model performance of the numerous parameter combinations.</p><p>Note that the following defined parameter settings will take a long time to run.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3de4c723fd8a6a214c56997cc9c5f4e2/href">https://medium.com/media/3de4c723fd8a6a214c56997cc9c5f4e2/href</a></iframe><p>The above output should be as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*iraYEehKEblyhPhJ8pPmsg.png" /></figure><p>Upon tuning parameters, an improved weighted average F₁ score of <strong>83% </strong>to<strong> 86%</strong> is achieved for the Logistic Regression model.</p><p>This can further be visualised as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*uc2FjpZf0UNcO4VKG8yq1Q.png" /></figure><h4>Train and test on bigger dataset</h4><p>Given the size of the <a href="https://github.com/several27/FakeNewsCorpus">Fake News Corpus</a> and the scope of this project not being limited to Covid-19-related content, fresh training is done on the 100,000+ article titles.</p><p>The same pipeline is used as defined above. However, an accuracy of <strong>97%</strong> is achieved, which is not too bad at first glance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/407/1*lkVHXCMQNX-eKi_RPx_FqQ.png" /></figure><p>However, as the confusion matrix shows, the low recall score (50%) in predicting “true” articles indicates that that the model has a poor success in finding relevant cases of “true” titles.</p><p>A big reason for this bias in the model could be attributed to the uneven spread of “true” and “fake” labels in the dataset. This is the main reason <strong>F₁</strong> score instead of accuracy is used.</p><p><strong><em>Edit:</em></strong><em> Upon normalising the distribution of our labels, a </em><strong><em>score of 85%</em></strong><em> is made using Logistic Regression. More importantly, the confusion matrix shows a much more balanced prediction.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*IdWgy0sHPws1EWao0nmnqA.png" /></figure><h3>Conclusion</h3><p>Analysis was done to better understand the data while extracting useful features to be used in the classification model. Various parameters and classifiers were evaluated, encouraging the use of Random Forest, Gradient Boosting and Logistic Regression.</p><p>Hyperparameter tuning was implemented on the Logistic Regression model which ultimately improved the weighted average F₁ score of <strong>83% </strong>to<strong> 86%.</strong></p><p>Testing the model on a larger, imbalanced dataset, namely the Fake News Corpus, a 97% accuracy was achieved. However, the distribution of the dataset was cause for concern and the low recall score of 50% show bias in the model’s prediction. Upon normalising the data, an F₁ score of 85% was achieved.</p><p>An F₁ score above 80% is good enough to solve the problem of <em>automatically identifying fake news by solely analysing the content headline or title text style.</em></p><blockquote><strong>How would this model work in the “real world”?</strong></blockquote><blockquote>Unlabelled article headlines can be processed and given a probability of being either “real” and “fake” given the content and stylistic features of the text.</blockquote><p>Expanding on this design, however, the following could be done to further improve the accuracy:</p><ul><li>Use of punctuation should be extracted as a feature</li><li>Add new metadata as features, including date published, source, etc.</li><li>Better feature performance analysis to gain insights into which characteristics have the biggest weight</li><li>Test randomised grid search</li><li>Increase recall for “true” articles by balancing the dataset labels</li></ul><p>The wish list for this project includes the following:</p><ul><li>Web interface for manual test predictions</li><li>Localise data to include South African content</li></ul><p>If you enjoyed this content, please give this post a 👏, share with your friends or leave a comment below. Would love to get any input into the process and overall performance.</p><h3>References and Acknowledgements</h3><p>A lot of the design and thought-processes have expanded on some of the coursework covered in Udacity’s <a href="https://classroom.udacity.com/nanodegrees/nd025-ent/parts/d52fd59b-fa33-4cd0-8d96-11e461669484">Data Science Nanodegree</a>. Additionally, the following sources assisted as reference with the project development.</p><ul><li><a href="http://www.davidsbatista.net/blog/2018/02/23/model_optimization/">Hyperparameter optimization across multiple models in scikit-learn</a></li><li><a href="https://towardsdatascience.com/automatically-detect-covid-19-misinformation-f7ceca1dc1c7">Automatically Detect COVID-19 Misinformation</a></li><li><a href="https://github.com/several27/FakeNewsCorpus">several27/FakeNewsCorpus</a></li><li><a href="https://www.kaggle.com/gautham11/building-a-scikit-learn-classification-pipeline">Building A Scikit Learn Classification Pipeline</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86c98fa5ab6e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The state of Google Play Store — a data approach]]></title>
            <link>https://medium.com/@plan__b/the-state-of-google-play-store-a-data-approach-9402cda5f038?source=rss-549b0f04c64a------2</link>
            <guid isPermaLink="false">https://medium.com/p/9402cda5f038</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Jacques Coetzee]]></dc:creator>
            <pubDate>Tue, 30 Jun 2020 09:24:18 GMT</pubDate>
            <atom:updated>2020-06-30T19:57:00.526Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oyXfmKsddSi4okgA" /><figcaption>Photo by <a href="https://unsplash.com/@williamtm?utm_source=medium&amp;utm_medium=referral">William Hook</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>The state of Google Play Store — a data approach</h3><p>Earlier this year I started learning the basics of app development using Flutter. The goal being able to eventually publish a — not only functional — but also an attractive, useful app on Google Play Store and the iOS App Store.</p><p>This made me inclined to analyse the state of the Play Store using <a href="https://www.kaggle.com/lava18/google-play-store-apps">this dataset</a> published on Kaggle. My main questions of interest are the following:</p><h4>What do high-rated apps have in common?</h4><h4>What kind of apps are the most popular and well-liked by users?</h4><h4>What characteristics distinguish paid and free apps from each other?</h4><h4>Can we predict the rating given the app features this dataset provides?</h4><p>Before we jump into these answers, it’s perhaps worth noting that the data required a lot of cleaning up. Due to it being scraped directly from the Play Store website, there were a couple of inconsistencies that required some spring cleaning to improve data integrity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*iAqmlXLiTmYdmWLv0gGNYQ.png" /></figure><ol><li>Data types needed to be converted ♻️</li><li>Empty values needed to be removed, imputed or handled some way 🧹</li><li>Potential biases in the data should be exposed 🧐</li></ol><p>After preparing the data by converting all columns to appropriate types, I dropped rows that have missing values. Main reason for that is that our dataset has some extreme outliers, which makes imputing a bit trickier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*YEu9HoTUCug5ja6VI1HDNA.png" /><figcaption>Plot of missing values showing Size and Ratings outshining the rest</figcaption></figure><p>The only column I imputed (using the average value of the category) was app size. For many apps, the size is simply labeled “Varies with device” — removing them would’ve meant losing around 16% of all records of the original total of 10,841.</p><p>Even though there are many missing values in the ratings column, I did not impute any values as we need accurate data for our predictor to help answer one of our main questions: <strong>what do high-rated apps have in common?</strong></p><p>Also, the distribution of our data is extreme with many outliers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*soXKke9OSXWzG_c6dngKZA.png" /><figcaption>When looking at our data distribution of our non-categorical columns, most columns are right skewed<br>with a long tail extending towards the right. In particular, Installs, Price and Reviews seem to have a couple<br>extreme outliers</figcaption></figure><p>The distribution in the above plots show that most columns are right skewed. Most columns are right skewed with a long tail extending towards the right. In particular, installs, price and reviews seem to have a couple extreme outliers.</p><p>The shape of our dataset after all modifications is (9360, 13) which means we’ve scrubbed around 1,500 rows from the original set.</p><p><strong>The distribution plots also helps us identify some low hanging fruit 🍎 for our analysis:</strong></p><ul><li>Of the 9,360 apps in the dataset, 93% of all apps are free</li><li>The most expensive app is the Lifestyle app “I am rich” which costs $399.99 with over 100,000 installs. Read more about this <a href="https://en.wikipedia.org/wiki/I_Am_Rich">here</a>.</li><li>In fact, there are numerous copycat “I am rich” apps similarly priced at the store’s maximum and 18 have “rich” in the name</li><li>The average rating for apps in this dataset is 4.2 stars out of 5 with the most common rating being an impressive 4.4</li><li>There are 274 apps that have a perfect score of which Family app <a href="https://www.youtube.com/watch?v=uzQ6RTAxJc4">Ek Bander Ne Kholi Dukan</a> (a Hindi animation and nursery rhymes app) has the most installs with over 10,000 downloads but only 10 reviewers</li><li>The app <a href="https://play.google.com/store/apps/details?id=com.goodbarber.riosdefe&amp;hl=en_ZA">Ríos de Fe</a> (a televangelist app) also has a perfect score but with 141 reviewers — the most of all five-star apps</li><li>Interestingly, <a href="https://play.google.com/store/apps/details?id=com.herzick.houseparty&amp;hl=en_ZA">House party — live chat</a> is rated 1 star by one reviewer and only 10 installs. Today, the app has 4 stars, rated by over 134,000 reviewers and over 10,000,000 installs.</li><li>The most popular categories are Family, Game and Tools</li></ul><p>Okay, now let’s get to our main questions.</p><h3>What do high-rated apps have in common?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*VION6JFXW9sAR8D6hTfUFA.png" /><figcaption>Last year update and Size has the biggest correlation with our Rating column. We can see the correlation in our data distribution as well with both columns sharing similar distribution shapes</figcaption></figure><p>Regular app updates along with size seem like prominent contributors to app ratings. Due to the fact that our data distribution is dominated by 2018, this correlation should not be taken at face value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T_PE4WYHoH4COB4RvSqeuw.png" /><figcaption><strong>Ratings have been rounded up.</strong> Even though most of our apps are last updated in 2018, the relative distribution of the ratings per year shows a steady increase in apps rated 4.5 and above from 2015 onwards</figcaption></figure><p>The concentration of five-star and four-star apps in 2018 for last updated year, indicates that customer prefer products that are maintained over time — a logical conclusion but one that’s worth noting nonetheless.</p><p>Another relatively strong correlation with app ratings is their size.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/1*83SwF6ZwwKFZCdVvC7k1tw.png" /><figcaption>This joint plot below shows that the size of the app is large especially around 4.0 and 4.5 — which is the average rating of our app dataset. However, apps rated above 4.5 are mostly smaller in size.</figcaption></figure><p>Up-to-date and relatively large app size are two indicators that important indicators that are relatively correlated with high-rated apps.</p><h3>What kind of apps are the most popular and well-liked by users?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pAzUuwly-cYt6ACBUJGIcQ.png" /><figcaption><strong>Ratings have been rounded up. </strong>If we consider another category’s app rating ratio — like category — we can see a clear preference for events, parenting, books and reference and art and design.</figcaption></figure><p>In terms of categories even though a handful are rated exceptionally high, not there are only three categories that are very popular: family, game and tools:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yyyiSWawaMCIbNNMi3lpmw.png" /><figcaption>The column chart shows that most apps are labeled as family, game and tools (38%)</figcaption></figure><h3>What characteristics distinguish paid and free apps from each other?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/1*6ZVASyJn8fw8hy7OjsmnHA.png" /></figure><p>The rating ratio by type shows that paid apps are rated much higher than free ones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HbvLS2vvRpmPvo_HnrsWTw.png" /><figcaption><strong>Ratings have been rounded up. </strong>Nearly a third of all paid apps are rated 4.5 or above compared to a quarter of free apps which achieve such high acclaim</figcaption></figure><h3>Can we predict the rating given the app features this dataset provides?</h3><p>Lastly, we use machine learning to enable us to predict the rating of the app based off of the other meta information available to us.</p><p>After testing logistic regression and SVC, we’ve managed to predict about 80% of the ratings using the column features in the dataset. For more information see the notebook in the Github repo which could be found <a href="https://github.com/feeblefruits/google_playstore_analysis">here</a>.</p><p><em>As part of Udacity’s Data Science nanodegree, students are required to write a blog post that takes the reader through data analysis of a chosen dataset. If you enjoyed this post please give it a couple of claps.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9402cda5f038" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[This Airbnb data will make you think twice about Cape Town accommodation costs]]></title>
            <link>https://medium.com/@plan__b/this-airbnb-data-will-make-you-think-twice-about-cape-town-accommodation-costs-7d31e90178eb?source=rss-549b0f04c64a------2</link>
            <guid isPermaLink="false">https://medium.com/p/7d31e90178eb</guid>
            <category><![CDATA[udacity]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[airbnb]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Jacques Coetzee]]></dc:creator>
            <pubDate>Mon, 02 Sep 2019 18:31:32 GMT</pubDate>
            <atom:updated>2019-09-05T05:23:31.288Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*umGJozQ0DWFLz2LUo5vBWw.jpeg" /></figure><p><em>This is my first post in a while. Since I started as a data fellow at the Mail &amp; Guardian, all my writing has gone onto its platforms. I recently started the Data Science track at Udacity and for my first project am encouraged to publish my first data exploration and analysis post. I decided to go with the Cape Town Airbnb dataset because the city’s property prices are known to be high and are always interesting to make better sense of.</em></p><p><em>Code for this project can be found on </em><a href="https://github.com/feeblefruits/airbnb_ct"><strong><em>Github here</em></strong></a><em>.</em></p><p>I decided to explore Airbnb data for this exercise. Essentially exploring three main questions namely: <strong>1) when are the most popular times to hike up the prices of listings, 2) which areas have the most expensive listings on average and 3) what contributes most to the price of a listing.</strong></p><p><a href="http://insideairbnb.com/">InsideAirbnb</a> has tons of information when it comes to our favourite accommodation platform. The main files that I downloaded to explore include <a href="http://data.insideairbnb.com/south-africa/wc/cape-town/2019-07-20/data/listings.csv.gz">listings.csv.gz</a>, <a href="http://data.insideairbnb.com/south-africa/wc/cape-town/2019-07-20/data/calendar.csv.gz">calendar.csv.gz</a>, <a href="http://data.insideairbnb.com/south-africa/wc/cape-town/2019-07-20/data/reviews.csv.gz">reviews.csv.gz</a> and eventually <a href="http://data.insideairbnb.com/south-africa/wc/cape-town/2019-07-20/visualisations/neighbourhoods.geojson">neighbourhoods.geojson</a>.</p><p>What I found from listings.csv.gz is that there’s an enormous amount of information that I didn’t need for the goals of my analysis — there was scraping information and all kinds of technical things that weren’t entirely relevant.</p><p>The file reviews.csv.gz, on the other hand, will have to wait for another day.</p><p>So, based on my pre-defined questions I wanted to explore I cleaned and organised the dataset. For this I separated all the columns that included the following information data: location, categories and numerical.</p><p>The idea behind this is to have a dataset that includes as many features that are useful to me and my questions. I ended up removing almost half of the initial set’s columns.</p><h3>1. When are the most popular times to hike up the price?</h3><p>By grouping the mean of the listings’ price by date and then plotting it, it shows that there’s a massive spike in costs December 2019 and January 2020. The listing prices show rooms available from the present month up until one year onwards.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/915/1*0Ar-nPudEGIlr-Ogum4Lkg.jpeg" /><figcaption>The above shows the average price of listings for each day from August 2019 — August 2020</figcaption></figure><p>So, the above graphic tells us that there’s a peak during the end-of-year festive season — which is not that surprising really. However, costs are lowest the closer we more to the present date (i.e. August and Septmber 2019).</p><p>Perhaps this is an indicator that hosts tend to rake up prices over the long run and bring them down as the date approaches to fill up more rooms.</p><p>The above chart shows all listings in our file. They’re a mix of listings available and unavailable. Let’s plot these two together. Blue is the price of listings available and Red is the price of listings unavailable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*KXq199e-DqKZCnrNIIcGEg.jpeg" /></figure><p>Between August and November 2019, there’s notably a lot of fluctuation between the two lines — which to me at least doesn’t really say much.</p><p>However, what’s notable is the fact that listings that are still available are more expensive than those already booked. There’s a R191 overall difference.</p><h3>2. Which areas are the most expensive?</h3><p>Let’s take a look at the most expensive listings:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CWqIFN0qqAu-IXFeBMmpeA.jpeg" /></figure><p>Ward 74, which is Hout Bay, and ward 54, which is Camps Bay, are unsurprisingly the wards with the most expensive listings.</p><p>The top listings that goes for R300,0000 needed an extra Google search. Lo and behold, the <a href="https://www.airbnb.com/rooms/35597310?source_impression_id=p3_1567443032_DUJrE32iM9F7UCOi">listing’s photos</a> are plastered with “Fake” warning signs.</p><p>And the least expensive:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7_AJ-d7NxloxGVxAgsT1-g.jpeg" /></figure><p>R127 per night? You’ll only find that in Kuilsriver apparently. Which is not bad if you’re looking for a spot between the winelands and Cape Town.</p><p>For a birdseye view on the most expensive areas I decided to plot out the map as <a href="http://insideairbnb.com/get-the-data.html">InsideAirbnb</a> also provides each city’s dataset with a neighbourhoods.geojson file that includes the locations of all the area’s wards.</p><p>The areas are seperated by wards. So I had to find the mean, minimum and max of each ward first.</p><p>Then I imported the geojson file and set up the new dataframe that would show me each ward’s location boundaries.</p><p>The resulting map is as such — showing prices between R4300 and R5200 in the dark blue and between R200 adn R1100 in the dark red.</p><p>The Atlantic Seaboard are the most expensive along with the inner City.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8JB7tx_gQbBtqMQOgTfFWw.jpeg" /></figure><h3>3. What contributes most to the price of listings?</h3><p>In order to find out what contributes most to the listings’ prices, I made a correlation table:</p><p>Which spits out this guy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*OLpy2OagFIysGbzgm8cA1A.jpeg" /><figcaption>For the styling I came across this fantastic post by <a href="https://pbpython.com/styling-pandas.html">Chris Moffett</a>.</figcaption></figure><p>In the table above we see that naturally the prices are highly correlated. But so is the number of bathrooms, accommodates, square feet and bedrooms.</p><p>For an overall view of our database’s correlation, the Python visualisation library Seaborn helps us out with this matrix below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*3z_vS4mOkJVi4Gy1N0Nu4A.png" /></figure><p>Here, we can see that whether or not a host has a license highly relates to review scores, minimum nights stayed and amount of bathrooms, while square feet of the listing correlates negatively.</p><p>This a brief overview of the Cape Town Airbnb dataset.</p><p>Some of the findings include that costs are lowest the closer we more to the present date. This is perhaps an indicator that hosts tend to rake up prices over the long run and bring them down as the date approaches to fill up more rooms.</p><p>The Atlantic Seaboard are the most expensive along with the inner City. Specific neighbourhoods include Camps Bay and Hout Bay.</p><p>Large price correlations include the number of bathrooms, accommodates, square feet and bedrooms.</p><p><em>Code for this project can be found on Github </em><a href="https://github.com/feeblefruits/airbnb_ct"><em>here</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d31e90178eb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DataVis Battle Entry: Getting to know my Pokémon]]></title>
            <link>https://medium.com/data-science/reddit-datavis-entry-getting-to-know-my-pok%C3%A9mon-e0bcf4b4b803?source=rss-549b0f04c64a------2</link>
            <guid isPermaLink="false">https://medium.com/p/e0bcf4b4b803</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[pokemon]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Jacques Coetzee]]></dc:creator>
            <pubDate>Sat, 22 Sep 2018 09:28:42 GMT</pubDate>
            <atom:updated>2018-09-22T12:32:21.034Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AzLWd4nei6AZdTRevdz1AQ.png" /><figcaption>Stacked percentage bar chart sorted by total damage against Grass-type Pokémon (Plotly).</figcaption></figure><p><em>This is my second entry to the DataVis challenge on Reddit. Data source: </em><a href="https://www.kaggle.com/rounakbanik/pokemon"><em>Kaggle</em></a><em> and </em><a href="https://bulbapedia.bulbagarden.net/wiki/Main_Page"><em>Bulbepedia</em></a><em> for images. All the graphics are done in Python, using Plotly and Matplotlib. Also, as a disclaimer, I’m by no means a Pokémon expert, so apologies in advance if some of the data is not accurately interpreted. I understand there are many nuances in their characteristics and history, so I’m open to the die-hards to add their two cents where necessary.</em></p><p><em>These data visuals come in peace </em>✌</p><h3><strong>Exploring the dataset</strong></h3><h4>Pokémon by type</h4><p>The first variable I wanted to look at is the total number of Pokémon types. Many Pokémon are mixed, so there’s a Type 1 and a Type 2:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*aH8NtGqYlLiJdSXP_SUJ1w.png" /><figcaption>Stacked bar chart of Pokémon Type 1 and Type 2 (Plotly).</figcaption></figure><p>As you can see Water type Pokémon are the most common, followed by Normal and Grass. Flying type Pokémon are the most common Type 2, followed by Poison, Ground and Fairy types.</p><h4>Pokémon by BMI</h4><p>As soon as <a href="https://medium.com/u/504c7870fdb6">Medium</a> decides to allow HTML embeds, my posts will stop featuring these GIFs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/987/1*Y-h42kovH1UGQ9aSNV_Xdg.gif" /><figcaption>Scatter plot by Weight, Height and Traces by Type 1 (Plotly).</figcaption></figure><p>Although this visual is relatively informative when interactive, a static image displaying the Pokémon sprites or images, would also work to bring these critters alive.</p><p>After many a struggle importing these images as Matplotlib’s markers, I finally managed to get this chart working :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hDR4-Gy2nBxJ015ZZ7btUQ.png" /><figcaption>Scatter plot by Pokémon weight and height, with markers as sprite images (Matplotlib). You can find the code <a href="https://gist.github.com/feeblefruits/20e7f98a4c6a47075c8bfce7c06749c2">here</a>.</figcaption></figure><p>Cosmoem — the Pokémon in the bottom right-hand corner — is pretty much the same weight as Celesteela directly above it. However, Cosmoem’s BMI dwarves the rest of the Pokémon, as we can see below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*hk_uzWeFotM2Ps6zysiSgQ.png" /></figure><p>Viewing the lowest BMI, on the other hand, Haunter, Gastly and Kartana come out at the top.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*RVH5-HiRMvdgcb9Amvxk4w.png" /></figure><h4>Pokémon by damage</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XjKBSVnOB-gW2PXZ9FCHiw.png" /><figcaption>Scatter plot by Pokémon SP attack and SP defense, with markers as sprite images (Matplotlib).</figcaption></figure><p>The feature category with most variables in the dataset is the attack. More specifically, it’s each Pokémon’s attack damage against another specific type of Pokémon. A Water type Pokémon, for example, does more damage against a Fire, Ground, Rock type Pokémon, but is very vulnerable to Electric and Grass type Pokemon ⚡🔥💧🌿</p><p>Shuckle’s attack makeup will look like this in absolute values, where each colour represents damage against given type:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/51/1*H0Ce_isfHdtP3o5SFAF_Yw.png" /><figcaption>Sum of all damage against type equals 200</figcaption></figure><p>While Heracross’s looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*15c1IspEjKFA_pc_2aNEYA.png" /><figcaption>Sum off all damage against type equals 3977.5</figcaption></figure><p>The bars are calculated with <strong><em>attack damage * damage to type </em></strong>and then stacked. I’m using absolute numbers as it adds an interesting perspective as to damage output in relation to each other. This is not, however, their attack damage total <strong>but rather attack damage per type summed</strong>. I hope that makes sense.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*4zwwQWa0Cw_73LQcoMj_yw.png" /><figcaption>Pokémon damage to type stacked in absolute numbers where <strong>values = <em>attack damage * damage to type </em></strong><em>(Plotly).</em></figcaption></figure><p>Given that the damage above is made up of absolute numbers, viewing percentages is also interesting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ittW7hOAKHf_VG4Ai770gw.png" /><figcaption>Pokémon damage to type stacked in percentages where <strong>values = (<em>attack damage * damage to type) / sum of damages * 100 </em></strong><em>(Plotly).</em></figcaption></figure><p>We see that there are a couple of ‘blocks’ with the same colour makeups within the graphic above, most prominently between Braviary and Hoothoot. Singling this group out, we see that their kits are exactly the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*N2djfSRd7tkRXrUxPuzrMQ.png" /><figcaption>Stacked percentage graph with equal Damage to Type values.</figcaption></figure><p><em>And that’s it! Thank you for giving this post a read. If you enjoyed it, give me a clap or two and share some of your thoughts or comments. Cheers!</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/1*Kbrak5vDUyPZgwFUSGLt9g.gif" /><figcaption>Shuckle… I couldn’t resist.</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e0bcf4b4b803" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/reddit-datavis-entry-getting-to-know-my-pok%C3%A9mon-e0bcf4b4b803">DataVis Battle Entry: Getting to know my Pokémon</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DataViz Battle entry: Analysing airline data from the TSA]]></title>
            <link>https://medium.com/data-science/dataviz-battle-entry-analysing-airline-data-from-the-tsa-74bfad9539e4?source=rss-549b0f04c64a------2</link>
            <guid isPermaLink="false">https://medium.com/p/74bfad9539e4</guid>
            <category><![CDATA[plotly]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Jacques Coetzee]]></dc:creator>
            <pubDate>Mon, 27 Aug 2018 15:52:10 GMT</pubDate>
            <atom:updated>2018-08-29T15:44:41.471Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LxyjVfVRKcy6DGyOCGE1Vw.jpeg" /></figure><p><em>This is my second Medium post on data analytics. Given the nature of the Reddit contest, the below focuses on the visualisation of the data more so than the exploration.</em></p><p>First off, the dataset for the challenge is this <a href="https://www.dhs.gov/tsa-claims-data">TSA Claims Data</a> which comprises of a bunch of PDF documents. The Transportation Security Administration is an agency of the U.S. Department of Homeland Security that has authority over the security of the traveling public in the United States. The data contains information about items lost and claimed in transit.</p><p>I used <a href="https://tabula.technology/">Tabula</a> to parse the PDFs online, which ended up working well enough. <strong>I only looked at data from 2016</strong>. However, the rows were a bit scattered all over the place, which required me to use <a href="http://openrefine.org/">OpenRefine</a> (shoutout to <a href="https://twitter.com/warrenski">@warrenski</a> for the tip). I can’t stress how much OpenRefine helped me! I then used Python for restructuring and visualisation.</p><p>This was the first time that I used <a href="https://plot.ly/python/">Plotly</a> and it really works wonders. I missed the <a href="https://bokeh.pydata.org/en/latest/">Bokeh</a> boat, so as a first time Plotly user it’ll take a lot for me to jump ship.</p><p>All the graphics below are meant to be interactive but, alas, Medium doesn’t support these kind of embeds. Anyways, below are my entries for the DataViz Battle:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nJsQ6Oxu6lQKycldLNgevQ.png" /></figure><p>Strangely, as the above shows, approved items drops towards the end of the year — which begs the question if the department ends up running out of budget. You’ll see that the lines climb towards April, May and June, after which is plummets and dwindles all the way down to zero in December.</p><p>I decided to add a stacked graph which considering all statuses of 2016, like so:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y4KNju-5tqqcYLBM1WDuVg.png" /></figure><p>As you can see, the item approved in terms of total value drops as the volume of those in review spikes. At the same time, those claims deemed insufficient is a clear indicator that budget ends up running out towards the end of the year.</p><p>Airports and departments are not always to blame for lost items or for approving what gets paid out when, so I also analysed the airliners to see if there are some interesting variations in the <strong>type of items claimed</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ds3SruHYMajcLq-hU7uM0g.png" /><figcaption>This is my new favourite graphic :)</figcaption></figure><p>It was interesting to note that when looking at the airliners and their top contribution to <strong>claims in total, </strong>UAL peeks in hunting items and American Airlines is responsible for most medical items claimed. UAL is headquartered in Chicago which is notorious for its gun control, so perhaps there might be a correlation.</p><p>Overall, it’s interesting to play around with the item categories further, looking at them in terms of their overall value and volume.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4RGdEk15a-V2MVFquxcnCQ.png" /><figcaption><em>This graph is best viewed as an interactive element, so please excuse the terrible hover texts</em></figcaption></figure><p>Probably not the best graph for this representation, but I wanted to add some variation to the above. I also really like bubble scatter plots.</p><p>The above graphic shows that in terms of the <strong>volume of items claimed</strong>, Clothing, Other, Baggage and Electronics are super popular. However, Electronics wins in terms of <strong>value claimed </strong>of the items by far.</p><p>The sizes of the bubbles are determined by the average item value relative to the volume — this I thought was rather necessary. And as you’ll see below, it helped reveal that even though Hunting Items are relatively low in volume, they’re worth a lot more than, say, Medical items or Cameras.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c0JONd0tyysCyhT5E9V1nw.png" /><figcaption>A zoomed in version of the above</figcaption></figure><p>Zooming in on the ugly cluster in the previous graph, in the corner you’ll see that the Hunting Items bubble is relatively big, indicating its low volume but high value per item.</p><p><em>That’s it for this post, thanks for the read and hope you found it interesting. If you like the visuals, feel free to give the post a clap or share. Cheers!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=74bfad9539e4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/dataviz-battle-entry-analysing-airline-data-from-the-tsa-74bfad9539e4">DataViz Battle entry: Analysing airline data from the TSA</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analysing SA gov’s wasteful expenditure using Tabula and Pandas]]></title>
            <link>https://medium.com/data-science/analysing-sa-govs-wasteful-expenditure-from-tabula-to-pandas-464d06c3bdd9?source=rss-549b0f04c64a------2</link>
            <guid isPermaLink="false">https://medium.com/p/464d06c3bdd9</guid>
            <category><![CDATA[south-africa]]></category>
            <category><![CDATA[python-pandas]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Jacques Coetzee]]></dc:creator>
            <pubDate>Mon, 13 Aug 2018 14:48:28 GMT</pubDate>
            <atom:updated>2018-08-15T15:46:54.418Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A first post on some data analytics, looking at irregular government expenditure, specifically contract requests for deviations and expansions by public entities. The data is available freely online, I just parse, explore and present it.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*OPzoNc_D8N8D0gPZaWX7YQ.jpeg" /><figcaption>Image source: <a href="https://www.southernvines.co.za/2016/06/21/rand-rollercoaster-understanding-south-africas-volatile-currency/">Southern Vines</a></figcaption></figure><p>South Africa’s government is in some serious debt. Much of this is due to irregular expenditures which can be traced back to deviation and expansion tenders. These tender requests, which are meant for exceptional circumstances, often become the norm.</p><p>In June 2018, public entities and departments <a href="https://pmg.org.za/committee-meeting/26617/">SARS</a>, <a href="https://pmg.org.za/committee-meeting/26629/">SITA</a>, <a href="https://pmg.org.za/committee-meeting/26635/">Department of Defence</a>, <a href="https://pmg.org.za/committee-meeting/26692/">Department of Public Works</a>, <a href="https://pmg.org.za/committee-meeting/26601/">Department of Water and Sanitation &amp; Trans Caledon Tunnel Authority</a> were all brought forward to explain suspicious requests for deviations and expansions. These included inflated contracts for DSTV subscriptions, millions rands worth of video conferencing equipment, food and catering, and so on. All contract requests are published on a quarterly basis by <a href="http://www.treasury.gov.za/">Treasury</a>.</p><h3>Getting the data</h3><p>Each quarter’s contracts are — as with most government records — in PDF format like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fetybST42AZ-LqLKaN0GmA.png" /><figcaption>The nemesis of the internet</figcaption></figure><p>To parse the data for analysis, I tried <a href="https://github.com/chezou/tabula-py">tabula-py</a> — a Python wrapper of tabula-java, which can read and extract tables in PDFs. It’s well-regarded and one of the very few good ones I could find out there. After converting a couple of files into .csv, I realised that its parsing was not as accurate as I would have wanted.</p><p>So I ended up manually selecting the tables for increased accuracy using the <a href="https://tabula.technology/">Tabula GUI</a>, and imported the .csv’s into Pandas. I parsed all the quarters currently available on the Treasury website — from Q2 2016 to Q3 2018. Below are snippets of these:</p><pre>ex1 = pd.read_csv(&#39;expansions_q2_2016.csv&#39;)<br>ex7 = pd.read_csv(&#39;expansions_q3_2018.csv&#39;)</pre><pre>ex1.head(5)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CcyXYKdaUNwBM1Hog5w3TA.png" /></figure><pre>ex7.head(5)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*FzDzqRidw6CGEAC48SaqcQ.png" /></figure><h3>Cleaning</h3><p><em>For the sake of the post and everyone’s sanity, I’ve simplified the cleaning process by a lot — it was much more gruelling than what it looks like.</em></p><p>In the ‘head’ examples above, you might have noticed the tables require a lot of cleaning up. For instance, value entries in column ‘Original Contract Value’ are spaced with a comma in the first import and a space in the second. This was due to two main issues: 1) lack of uniformity of data entry from Treasury’s part and 2) lack of perfect PDF parsing.</p><p>Many entries in the ‘Original Contract Value’ column were used different formats from the rest and included:</p><pre>array([&#39;estimated R95 per\rtransaction&#39;, &#39;various&#39;, &#39;168 849 638.40 -ABB\r20 961 068.87- Zest\r54 271 707.30 - Actom&#39;, &#39;Not stated&#39;)]</pre><p>Some included foreign currencies like €250,455 and US$334 908. These were all removed from the dataset, bringing our set to just over 1600 rows.</p><p>The column headings were also in different orders and had different labels. After reordering the columns and renaming them all, I concatenated all the tables:</p><pre>ex_all = pd.concat((ex1,<br>                    ex2,<br>                    ex3,<br>                    ex4,<br>                    ex5,<br>                    ex6,<br>                    ex7<br>                   ), axis=0, ignore_index=True)<br>df.head(4)</pre><p>Giving us this guy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*0beEh4z3jy1REIlXzrLG6g.png" /></figure><pre>df.info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t_Vc4J9s1nxvqd43fx2YAg.png" /><figcaption>All the objects…</figcaption></figure><p>After some more cleaning, it was time for some exploratory analysis.</p><h3>Exploratory analysis</h3><pre>df.sort_values(&#39;Value of Contract&#39;, ascending=False)[:6]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/1*lpnUXHrgIFaEfD_vaEv0ow.png" /><figcaption>I also tokenised the ‘Project Description’ column, with the aim of doing some text analysis a bit later, but never got around to it.</figcaption></figure><pre>df[&#39;Value of Contract&#39;].sum()/1000000000000)[:7]</pre><p>A total of R19.4539 trillion worth of contracts were requested in the last 10 quarters.</p><p><em>Some contract values seriously looked like typos and if someone simply slapped a bunch of zeros onto a random figure — so I was forced to double check a lot of the entries to make sure it wasn’t an issue on the parsing front.</em></p><p>Looking at all the contracts didn’t make too much sense and were a bit out of context so let’s consider only those awarded or labeled as ‘supported’.</p><pre>df[&#39;Support Status&#39;] = df[&#39;Support Status&#39;].str.lower()</pre><pre>supported_contracts.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/1*7Y7l3Tal_uPBrRbPOoPdkg.png" /></figure><pre>supported_contracts.info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/1*kTaOi31-IquqDzSY074htQ.png" /></figure><p>All our entries were objects, which required some reformatting. I converted all the numerical entries to floats and the date entries to Python’s date.time.</p><pre><br>supported_contracts[‘Value of Contract’] = pd.to_numeric(supported_contracts[‘Value of Contract’], errors=’coerce’)</pre><pre>supported_contracts[&#39;Date Awarded&#39;] = pd.to_datetime(supported_contracts[&#39;Date Awarded&#39;], errors=&#39;coerce&#39;)</pre><pre>supported_contracts[&#39;Contract Start Date&#39;] = pd.to_datetime(supported_contracts[&#39;Contract Start Date&#39;], errors=&#39;coerce&#39;)</pre><pre>supported_contracts[&#39;Contract Expiry Date&#39;] = pd.to_datetime(supported_contracts[&#39;Date Awarded&#39;], errors=&#39;coerce&#39;)</pre><p>Now, only looking at the Supported Contracts, the total contracts awarded value was R954 billion.</p><pre>(supported_contracts[&#39;Value of Contract&#39;].sum() / 1000000000).round()</pre><pre># in billions contracts awarded</pre><p>To call values easier based on dates, I index the ‘Date Awarded’ column:</p><pre>supported_contracts.index = supported_contracts[‘Date Awarded’]</pre><pre>del supported_contracts[‘Date Awarded’]</pre><pre>supported_contracts.head(2)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*X2supLaOcPShlyaSja7_kA.png" /></figure><pre>(supported_contracts[‘2016’][‘Value of Contract’].sum() / 1000000000).round()</pre><pre># in billions contracts awarded in 2016</pre><p>R311 billion was awarded in 2016.</p><p>I cleared up the dataframe to have a bit more focus by removing the following columns:</p><pre>supported_contracts.drop(&#39;Reason&#39;, axis=1, inplace=True)</pre><pre>supported_contracts.drop(&#39;Contract Expiry Date&#39;, axis=1, inplace=True)</pre><pre>supported_contracts.drop(&#39;Contract Start Date&#39;, axis=1, inplace=True)</pre><pre>supported_contracts.drop(&#39;Description Tokens&#39;, axis=1, inplace=True)</pre><p>Then I visualised some of our findings:</p><pre>support_contracts = support_contracts[&#39;2016&#39;:&#39;2019&#39;]</pre><pre>support_contracts.resample(&#39;D&#39;).sum().plot()<br>fig = plt.gcf()</pre><pre>legend_label = mpatches.Patch(color=&#39;blue&#39;, label=&#39;Total Value of Contracts&#39;)</pre><pre>plt.legend(handles=[legend_label])</pre><pre>plt.suptitle(&#39;Contracts Awarded Over Time&#39;, fontsize=16)</pre><pre>plt.rcParams.update({&#39;font.size&#39;: 9})</pre><pre>fig.set_size_inches(15, 8)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*F4ZyAgqX7VChhhH1kGpUCw.png" /></figure><p>And looking at what those spikes were, I sorted all the entries under ‘Value of Contract’ in descending order:</p><pre># top contracts supported ranked in terms of value</pre><pre>supported_contracts.sort_values(&#39;Value of Contract&#39;, ascending = False)[:9]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*2nBV57Y5y_LtIKxe1JXCzA.png" /></figure><p>Correctional Services and Health (or the Department of Health) had the biggest contracts supported in the last two years for “provision and management of food services” and “provision of alternative chronic medication access programme for public sector patients”.</p><p>To get a better sense of each entity’s impact on overall expenditure, I looked at their overall contract value as well as contract volume in relation to the rest:</p><pre>def value_percent(entity):<br>        <br>    entity_df = supported_contracts[supported_contracts[&#39;Entity&#39;].str.match(entity, na=False)]<br>    entity_value_total = entity_df[&#39;Value of Contract&#39;].sum()<br>    percentage = (entity_value_total / supported_contracts[&#39;Value of Contract&#39;].sum())*100<br>    <br>    return percentage.round(2)</pre><pre>def freq_percent(entity):<br>    try:<br>        percentage = (df[&#39;Entity&#39;].value_counts()[entity]/len(df))*100<br>        return percentage.round(2)<br>    except KeyError:<br>        pass</pre><p>And create a new dataframe:</p><pre>impact_df = {&#39;Entity&#39;: supported_contracts[&#39;Entity&#39;].str.title(),<br>             &#39;Frequency of Appearance&#39;: supported_contracts[&#39;Entity&#39;].apply(freq_percent),<br>             &#39;Value of Contracts&#39;: supported_contracts[&#39;Entity&#39;].apply(value_percent)}</pre><pre># only interested in top 10 for now</pre><pre>impact_df = pd.DataFrame(data=impact_df).drop_duplicates(subset=&#39;Entity&#39;)<br>impact_df.sort_values(&#39;Value of Contracts&#39;, ascending=False)[:9]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*QPGIkzVuq8sCGB9MXRLmJA.png" /></figure><p>In terms of overall ‘Value of Contracts’, SARS makes up 21.42% with Eskom making up 16.20% and Sanral 13.88%.</p><p>Let’s visualise the top ten most expensive entities in relation to each other:</p><pre>cmap = plt.get_cmap(&#39;Spectral&#39;)<br>colors = [cmap(i) for i in np.linspace(0, 1, 9)]</pre><pre>impact_value_df.index = impact_value_df[&#39;Entity&#39;]</pre><pre>plot = impact_value_df.plot.pie(y=&#39;Value of Contracts&#39;,<br>                               autopct=&#39;%.0f%%&#39;,<br>                               figsize=(8, 9),<br>                               colors=colors)</pre><pre>plt.suptitle(&#39;Share of Total Contracts by Value&#39;, fontsize=16)</pre><pre>plt.show()</pre><pre># the chart only shows the total share of the top ten most &#39;expensive&#39; entities (it excludes the other)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/487/1*Q4-TGCiTCMbtsIj8iXh_sg.png" /></figure><p>The share of contracts per volume is also an interesting indicator of mismanagement and is visualised below. And to visualise the most frequent contracts, I used a Seaborn graph:</p><pre>impact_freq_df = impact_df.sort_values(&#39;Frequency of Appearance&#39;, ascending=False)[:9]</pre><pre>f, ax = plt.subplots(figsize=(15, 5))<br>sns.set_color_codes(&quot;muted&quot;)</pre><pre>sns.barplot(x = &#39;Entity&#39;,<br>            y = &#39;Frequency of Appearance&#39;,<br>            data = impact_freq_df,<br>            palette=&#39;dark&#39;,<br>            alpha=.6)</pre><pre>plt.suptitle(&#39;Share of Total Contracts by Volume&#39;, fontsize=16)</pre><pre>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*L2FpZyEIAizhE6NQoKMfiA.png" /></figure><p>There’s a lot more that can be done with this dataset, but that’s it for now. Some other suggestions would be to analyse the top suppliers in terms of contract volume and total values or testing out different formula’s that can identify suspicious requests.</p><p>I’m planning on doing at least two datasets a month, preferably in the African context — if you have any datasets, comments, thoughts or suggestions, please feel free to share.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=464d06c3bdd9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/analysing-sa-govs-wasteful-expenditure-from-tabula-to-pandas-464d06c3bdd9">Analysing SA gov’s wasteful expenditure using Tabula and Pandas</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why you should continue to stomp, scream and stand up to make the most of SA’s democracy]]></title>
            <link>https://medium.com/@plan__b/why-south-africans-should-continue-to-stomp-scream-and-stand-up-to-make-the-most-of-our-democracy-980144d414fd?source=rss-549b0f04c64a------2</link>
            <guid isPermaLink="false">https://medium.com/p/980144d414fd</guid>
            <category><![CDATA[activism]]></category>
            <category><![CDATA[protest]]></category>
            <category><![CDATA[africa]]></category>
            <category><![CDATA[south-africa]]></category>
            <category><![CDATA[democracy]]></category>
            <dc:creator><![CDATA[Jacques Coetzee]]></dc:creator>
            <pubDate>Sun, 27 Aug 2017 18:36:20 GMT</pubDate>
            <atom:updated>2017-08-28T07:53:58.960Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D3VNF4qjb4MnsetlE5FbQA.jpeg" /></figure><p>Nelson Mandela is quite notorious today — in <a href="https://theconversation.com/how-compromises-and-mistakes-made-in-the-mandela-era-hobbled-south-africas-economy-52156">some circles</a> at least. Many South Africans feel that they’ve been sold out, with their parents buying into the pie-in-the-sky rainbow nation bullshit. And rightly so, twenty-something years on and an <a href="https://www.dailymaverick.co.za/article/2017-08-23-statssa-poverty-report-2017-policies-fail-the-people-of-south-africa/#.WaL2IJOg_dQ">estimated</a> 30.4-million people are impoverished, there’s barely any growth, <a href="http://www.statssa.gov.za/?cat=22">inequality is soaring</a> and things aren’t all as great as we think it should be. Not enough has changed.</p><p>The same wheel that gave us corrupt politicians with cozy elites <a href="http://opensecrets.org.za/agm/">40 years ago</a> under Apartheid is still turning <a href="http://www.gupta-leaks.com/">today</a> in our fledgling democracy. The same <a href="https://www.citylab.com/equity/2016/06/apartheids-urban-legacy-in-striking-aerial-photographs-south-africa-cities-architecture-racism/487808/">lines between communities</a> that were built before are still standing in 2017.</p><p>The only difference today — and this is a big one — is the fact that we have the capacity to shape the systems we’re born into, whether that be through reading, sharing, writing or <a href="http://i2.cdn.cnn.com/cnnnext/dam/assets/160831181827-south-africa-school-girl-hair-scandal-full-169.jpg">standing up</a> for your identity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*ObRwGEwIART6kJRZkikSGg.png" /><figcaption><em>Johnny Miller’s </em><a href="http://www.unequalscenes.com/"><em>Unequal Spaces</em></a></figcaption></figure><p>Yes, we — especially black people — have reasons to feel betrayed. But at least today we have the right to voice how we feel, and act accordingly. Not to promote excuses, democracy in itself <a href="https://www.brookings.edu/blog/up-front/2016/09/06/democracy-does-not-cause-growth/">rarely helps economies grow</a>, not to mention overcoming centuries of economic trauma’s enduring scars. Instead, democracy gives recipients the <em>capacity</em> to overcome these challenges through activism — and that agency is something we should take most advantage of.</p><p>And on the grander scale of things, it might seem ironic that while the 21st century witnessed the most democracies <a href="https://ourworldindata.org/democracy/">ever established,</a> it’s also seen an unprecedented amount of popular movements against democratically-elected governments. People are pissed all over. And similar to South Africans, the fact that they are faced with grievances certainly isn’t a sign of democratic progress, the fact that they can know about them and stand up for their opinions certainly are.</p><p>From <a href="http://www.theguardian.com/world/2015/aug/27/guatemala-president-impeachment-threat-protest">Guatemala</a>, <a href="http://www.nytimes.com/2013/02/21/world/europe/bulgarian-government-is-reported-set-to-resign.html">Bulgaria</a>, <a href="https://www.youtube.com/watch?v=LTpe7mxZy4k">Venezuela</a> to <a href="http://www.atlanticcouncil.org/publications/articles/how-ukraine-ousted-an-autocrat-the-logic-of-civil-resistance">Ukraine</a>, <a href="http://www.cnn.com/2013/11/26/world/asia/thailand-protests-explainer/">Thailand</a>, <a href="http://www.bbc.com/news/world-europe-32776204">Macedonia</a>, <a href="http://www.theguardian.com/commentisfree/2011/oct/15/spain-15-m-movement-activism">Spain</a>, <a href="http://www.reuters.com/article/iceland-eu-idUSL2N0WH06T20150315">Iceland</a>, <a href="http://www.economist.com/news/europe/21633856-more-anti-government-protests-little-change-prime-minister-opposing-orban">Hungary</a>, <a href="http://www.nytimes.com/2016/01/26/world/europe/oppositions-groups-in-moldova-unite-to-protest-new-government.html">Moldova</a>, <a href="http://www.nytimes.com/2016/03/14/world/americas/brazil-dilma-rousseff-protests.html">Brazil</a> and <a href="http://foreignpolicy.com/2016/02/08/polands-liberals-strike-back/">Poland</a> — all these governments were recently challenged and some of them forced to step down by mass-based popular uprisings.</p><p>But Africa seems to serve as a sort of microcosm of global trends. There were <a href="https://www.afdb.org/fileadmin/uploads/afdb/Documents/Publications/AEO_2016_Report_Full_English.pdf">five times</a> more protests in 2014 than in 2000 on the continent. For better or worse — while each vary in context — when considered collectively, they tend to raise fundamental questions about democracy and what this system means in the contemporary moment, globally.</p><p>As well placed in <a href="http://press.uchicago.edu/ucp/books/book/distributed/A/bo20500251.html">Afr<em>ica Uprising</em></a>,<em> </em>protests are seen as a collective force for political change which has emerged across the continent. In the book, US academics Mampilly and Adam Branch identified the common drivers behind protests in Africa, with case studies ranging from those in Nigeria to Ethiopia and all around the continent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NPZkZHs7xSvAW10HcZRPBg.png" /><figcaption><a href="http://press.uchicago.edu/ucp/books/book/distributed/A/bo20500251.html">Africa Uprising: Popular Protest and Political Change</a> by Adam Branch and Zachariah Cherian Mampilly</figcaption></figure><p>The authors characterised the 21st century wave of protests as the so-called Third Wave, which follows the First Wave of decolonisation and the Second Wave of democratisation. This wave could be characterised by a renewed sense of activism whereby ordinary Africans, often not driven by political parties, are addressing socio-economic injustices from grassroots.</p><p>“Many of these protests, they may be triggered by economic crises or electoral crises, but they are more often part of these broader movements that question the meaning of democracy in the contemporary moment,” Mampilly told me during an interview in Cape Town.</p><p>“You are a young African, who put a lot of faith in the idea that democracy is going to be this road to prosperity, it is coming up rather hollow in this moment.”</p><p>And rightfully so, protest has been a very effective tool for many Africans’ issues. In October 2015 South African students and low-paid university workers won the battle for a 0% fee increase for 2016, <a href="http://www.news24.com/SouthAfrica/News/Zero-increase-in-fees-Zuma-20151023">among other demands</a>. In 2012, the Nigerian government was <a href="http://www.nytimes.com/2012/01/17/world/africa/nigerian-president-rolls-back-price-of-gasoline.html">forced to lower oil cost </a>to after series of strikes nearly paralysed the country for two weeks. In 2014 mass demonstrations <a href="http://www.cbc.ca/news/world/blaise-compaore-burkina-faso-president-resigns-after-violent-protests-1.2819254">overthrew</a> president Blaise Compaoré from Burkina Faso.</p><p>Another post comes to mind from <a href="https://www.pressreader.com/south-africa/the-citizen-gauteng/20170725/281642485236764">Save SA’s Sipho Pityana</a> highlighting the recent accomplishments of activism in South Africa. Civil society has accomplished the following:</p><blockquote>1. Brian Molefe and Ben Ngubane are no longer at the helm of Eskom.</blockquote><blockquote>2. KPMG, McKinsey and Bell Pottinger — Gupta’s professional enablers are in deep trouble.</blockquote><blockquote>3. Our sleepy Parliament has been forced to hold inquiries into state capture.</blockquote><blockquote>4. PRASA saved billions from a corrupt contract.</blockquote><blockquote>5. Ntlemeza, Jiba and Mrwebi are gone.</blockquote><blockquote>6. Even politicians can no longer dispute the evidence of the capture of our state.</blockquote><blockquote>7. You’ve made the removal of Zuma the central discussion in the ANC.</blockquote><blockquote>8. You’ve stopped the nuclear deal.</blockquote><blockquote>9. You’ve ensured South Africa remains part of the International Criminal Court.</blockquote><blockquote>10. You’ve stopped the capture of SABC.</blockquote><blockquote>11. You’ve built consensus on social grants.</blockquote><blockquote>12. You’ve improved the independence of the Hawks and IPID. (We still have to ensure that the right person is appointed to head the Hawks but that’s a battle for another day).</blockquote><blockquote>13. You’ve exposed the SARS rogue unit narrative as a scam.</blockquote><blockquote>14. You’ve inspired Makhosi Khosa.</blockquote><p>According to a recent study by Washington-based Pew Research Centre, many Kenyans, South Africans and Nigerians <a href="http://www.pewglobal.org/2016/10/24/2-africa-majorities-in-kenya-nigeria-and-south-africa-believe-that-ordinary-citizens-can-influence-the-government/">believe</a> that ordinary citizens can have a lot of influence on government. Furthermore, it is more common to attend a political campaign event in these three nations than it is in the US. Mampilly is cautiously optimistic about the future of the continent and what this trend represents about citizens’ character.</p><p>“For too long, there’s been the assumption that democracy simply mean the presence of elections,” Mampilly says. “Simply holding elections is thought to be the gold standard of a democracy. But then you look at what democracy actually means, from a broader perspective — rule of the people. It’s certainly unclear that elections are the best mechanism through which to capture the desire of the population.”</p><p>And while a lot of people should continue to be pissed of at the state of things, we should embrace our sense of activism, and not take that right for granted. We should continue to read up about our country’s problems, listen to challenging opinions of others and speak out against assholes impeding your — and especially those of your fellow South African’s — capacity to do so.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=980144d414fd" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>