<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Bence L. Tóth on Medium]]></title>
        <description><![CDATA[Stories by Bence L. Tóth on Medium]]></description>
        <link>https://medium.com/@benceltoth?source=rss-6d8c19a5042a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*k3vfup_TvFFXuub-KGjq0A.png</url>
            <title>Stories by Bence L. Tóth on Medium</title>
            <link>https://medium.com/@benceltoth?source=rss-6d8c19a5042a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 16:49:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@benceltoth/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Misconceptions of Procurement Analytics: Data strategy is a one-time initiative]]></title>
            <link>https://pub.towardsai.net/misconceptions-of-procurement-analytics-data-strategy-is-a-one-time-initiative-2771b11f3948?source=rss-6d8c19a5042a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2771b11f3948</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-strategy]]></category>
            <category><![CDATA[governance]]></category>
            <category><![CDATA[procurement]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Bence L. Tóth]]></dc:creator>
            <pubDate>Tue, 07 Dec 2021 20:22:34 GMT</pubDate>
            <atom:updated>2021-12-07T20:22:34.222Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsai.net/p/category/data-analysis">Data Analysis</a></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MNCu3DlJypB_9fHH" /></figure><p>Having spent the last couple of years working with purchasing and accounts payable data from IBM and clients across a variety of industries, the IBM Procurement Analytics as a Service team established a data strategy framework with a broad business perspective. We believe this practical framework can be leveraged by any business unit and industry.</p><p>Many factors require careful consideration to bring about an effective and well-crafted data strategy. This article outlines those factors and highlights that departments beyond IT are essential to bringing a successful data governance program to life. Our framework is built around the data lifecycle: from need assessment through consumption. The “breadcrumbs” for applying this scheme can be found in every business.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/0*BF07fOvpUoGe-yuJ" /></figure><h3>Align data strategy to business goals</h3><p>More and more companies view data as a strategic asset, instead of a by-product of their business activities. According to <a href="https://www.mckinsey.com/business-functions/marketing-and-sales/our-insights/five-facts-how-customer-analytics-boosts-corporate-performance">McKinsey</a>, data-driven organizations are 23 times more likely to acquire customers, 6 times as likely to retain them, and 19 times as likely to be profitable. However, according to a <a href="https://c6abb8db-514c-4f5b-b5a1-fc710f1e464e.filesusr.com/ugd/e5361a_7e5120ecf1ab4d77a5c2a3d253663150.pdf">2019 NewVantage survey</a>, 77% of businesses report that “business adoption” of big data and AI initiatives continues to represent a big challenge for them. Why?</p><p>A significant clue is that 42% of companies <a href="https://www.gartner.com/smarterwithgartner/how-data-and-analytics-leaders-can-master-governance">do not assess, measure, or monitor data governance</a>. Data Governance incorporates data, roles, processes, communications, metrics, and tools that help organizations formally manage and gain better control over data assets.</p><p>Data activities are often isolated within the IT branches and processes are often duplicated or redundant, losing scalability and driving up costs. Without data governance, a company cannot realize the true worth of its data. Lacking an objective measure for prioritization results in incoherent decision-making drives up costs and affects revenue. When the budget gets tight, projects without clear business value will experience funding and resource cuts, further increasing the effects of this vicious circle, affecting IT and non-IT departments alike. Successful enterprise-wide data governance can only be achieved when business goals are aligned with the data strategy.</p><p>When business goals and data strategy are aligned, employees at all levels can understand the importance of data and their role to treat data as an important asset in their daily work. Being a data steward becomes part of the company’s culture, ensuring clear roles and responsibilities. The alignment aids in uncovering cause and effect in the operations and leveraging customer insights, which enables data projects to receive business support with the appropriate resources.</p><p>So how do we link data strategy with business goals? Start with a top-down approach to understand the business goals and priorities. Then, examine what, why, and how data is necessary to achieve them. Where do you want to see your company in <em>x</em> years? How can data help you to get there? Simultaneously, from the bottom-up, gather technical domain knowledge by consulting with department leaders to understand the current issues, requirements, and challenges they face, what data is available, and what is necessary to overcome these challenges.</p><p>A few ideas to consider when aligning your data strategy with your business goals:</p><ul><li>Start simple and iterate to refine your processes and define use cases.</li><li>Understand your customers and your internal end-users to build user-friendly, not just “shiny,” tools.</li><li>Define data strategy success factors and KPIs.</li><li>Revise and adapt data strategy to coincide with business strategy changes.</li><li>Analyze the current data ecosystem and discover its innovative potential, creating a “desired state” that supports business goals.</li><li>Identify gaps between your current ecosystem and the desired state and set up a roadmap to unlock the transformative power of data.</li></ul><h3>Acquire and process data</h3><p>The next key consideration is how to process and store data and make it accessible to everyone in the organization who could benefit from it. Businesses have relied on data warehouses and Extract-Transform-Load (ETL) processes for years to get a consolidated view of the data. Today, the ETL method of integrating data from multiple sources is still a core component of an organization’s data integration toolbox. However, today’s data velocity demands an <a href="https://www.ibm.com/blogs/journey-to-ai/2020/12/deploy-a-modern-data-and-ai-platform-on-the-infrastructure-of-your-choice/">infrastructure</a> that enables an exchange with frictionless data flow at its core. Endpoints (applications, processes, people, or algorithms) interact with the data hub, potentially in real-time, to provision data into the hub or receive data from it.</p><p>A data hub is designed for the rapid exchange of information needed by today’s organizations. The hub captures and reconciles any type of data (meta, master, operational, analytical, etc.). Then, it delivers the data in multiple desired formats without necessarily storing it physically in a central place. Using search-based applications and API services, data consumers can easily discover and get instant access to data they can trust. With minor upskilling, employees can easily start taking advantage of the data hub in their daily work.</p><h3>Enrich data</h3><p>Once data is available, it is cleaned, categorized, and combined with external sources. These enhancements provide context and enable companies to make informed decisions.</p><p>Depending on the business goals, different types of data enrichment can be pursued, but in all cases, data must come from a trusted source. Without reliable data, leaders cannot be confident their decisions are rooted in facts and reality.</p><p>However, keeping the data consistent and up-to-date is a continuous, high-maintenance process. Over 50% of businesses <a href="https://tdwi.org/research/2016/07/best-practices-report-improving-data-preparation-for-business-analytics.aspx">spend more time cleaning data than using it</a>. Regular updating is paramount to keeping the data current, and machine learning algorithms are offering a way to streamline that process.</p><p>After all, if the business decision-makers cannot trust the data within their organization, how can stakeholders and customers know they are in good hands? Information that is not correctly maintained and distributed can prove harmful to the integrity of business decisions.</p><p>Another important element is the implementation of an effective procurement taxonomy. Tracking every dollar an organization spends is a seemingly impossible task. With the help of business experts, setting up such taxonomy hierarchies can aid in classifying transactions that allow us to cross-index spend data against vendor categories and monitor progress towards set strategies.</p><p>In an IBM <a href="https://www.ibm.com/downloads/cas/5KJ5AJQM">whitepaper</a>, Marco Romano (then IBM’s S2P Chief Data &amp; Analytics Officer) details the three characteristics of an effective taxonomy that can feed a cognitive engine:</p><ol><li>Flexible — yet global</li><li>Multi-dimensional — with an ability to combine elements from different sources</li><li>Situation-based — or, as Marco wrote: “It is not about how you buy, but rather what you buy. I would argue even further that an appropriate taxonomy is about identifying how you resolve a business problem through products or services”</li></ol><h3>Apply Artificial Intelligence (AI)</h3><p>In recent years, the accessibility to Machine Learning (ML)-powered AI has dramatically increased, yet we find that better data continues to beat better algorithms. Consequently, in AI initiatives, most time is spent designing and managing data, rather than developing algorithms. Failing to focus on extracting relevant fields or cleaning data will diminish the initiative’s overall effectiveness and significantly increase the likelihood of errors. The solution is to involve procurement experts in Data Science projects to introduce domain knowledge in addition to the technical expertise.</p><p>ML models are organized into complex AI systems that feature many interrelated data components. The data strategy must be built on reusable, general-purpose components with well-understood elements. Concentrating efforts solely on technical requirements can be another peril of the initiative. Focusing on the data requirements instead will yield better results for the ML-infused capability.</p><h3>Analyze data consumption</h3><p>Many organizations fail to support decision-making, because they are trying to please everyone in the organization, instead of selecting a few focus areas.</p><p>When designing your data strategy, you need to articulate a clear goal on which the analyses can be built. Focus on four approaches:</p><ul><li>Understand your clients — customer profiles, the scope of activities, industry trends</li><li>Create better products and services — personalized offerings to beat competitors</li><li>Improve processes</li><li>Data monetization — selling or licensing non-sensitive information</li></ul><p>After you define the analytical goal for your data, choose a business case to implement that reflects the processes within the selected segment (procurement, finance, manufacturing, etc.).</p><p>A common pitfall in data strategy implementation is assuming that employees will be able to work with data visualizations and dashboards. For a successful outcome, you need to <a href="https://hbr.org/2018/02/you-dont-have-to-be-a-data-scientist-to-fill-this-must-have-analytics-role">involve data translators</a> in your strategy team to bridge the gap between technical and operational teams.</p><p>A data translator’s primary role is to bridge the expertise of data scientists, engineers, and analysts into business functions. They need to have a thorough understanding of their departments, including the strategic goals to the processes that they are running, and certainly the industry where they are working in. With their help, you are more likely to uncover key pain points, underlying opportunities, and prioritize business problems that will drive the most value when solved. Data translators can deliver insights from your data through analytics. They are also a key player in transforming the company culture, so that everyone is involved in the data strategy, and treats it as their responsibility, not just the CIO.</p><p>Navigating the data ecosystems is undoubtedly taxing. Business and technology both change in unprecedented rhythms, but at the same time, change opens the door for transformation. Avoiding these misconceptions, acknowledging the various challenges, and understanding the key success factors will help you chart a course and make any necessary corrections along the way to a well-crafted and scalable data strategy.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2771b11f3948" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/misconceptions-of-procurement-analytics-data-strategy-is-a-one-time-initiative-2771b11f3948">Misconceptions of Procurement Analytics: Data strategy is a one-time initiative</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to overcome your Bad Habits with Agile? — A data guy’s approach]]></title>
            <link>https://pub.towardsai.net/how-to-overcome-your-bad-habits-with-agile-a-data-guys-approach-879168794fbb?source=rss-6d8c19a5042a------2</link>
            <guid isPermaLink="false">https://medium.com/p/879168794fbb</guid>
            <category><![CDATA[personal-development]]></category>
            <category><![CDATA[agile]]></category>
            <category><![CDATA[happiness]]></category>
            <category><![CDATA[scrum]]></category>
            <category><![CDATA[habits]]></category>
            <dc:creator><![CDATA[Bence L. Tóth]]></dc:creator>
            <pubDate>Wed, 25 Aug 2021 20:02:44 GMT</pubDate>
            <atom:updated>2021-09-22T10:51:07.368Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsai.net/p/category/product-management">Product Management</a></h4><h3>How to Overcome Your Bad Habits with Agile? — A data guy’s approach</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*R8a0kjvRuIyOcAP5" /><figcaption>Photo by badmanproduction on Getty Images/iStockphoto</figcaption></figure><p>How many times have you tried to break bad habits, or make new ones? Have you ever considered eating more healthily, losing some weight, quitting smoking, or finally grabbing that rusty guitar in the corner to learn your favorite Bob Dylan song?</p><p>We all struggle with our habits. Some of us may have it easier on ourselves, while others give up on the first sight of failure. It takes a lot of willpower and determination to achieve our goals, but the method of <em>how </em>we try to achieve them can have an influential result on the outcome. Here’s what you might have been doing wrong.</p><h3>Why Waterfall is a WaterFail</h3><p>Some of us working in data and analytics fields often make this mistake: we have multiple goals, and we plan the heck out of them. We research many articles (what you might be doing right now), consult friends and professionals, lay down a strict schedule, and get confused by confronting ideas. We might be unable to decide what path to take and end up piling up information without actually taking a step towards our goal. We exhaust ourselves with planning every single moment of the execution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*Opj3IDhcr3Pq8oyv.png" /><figcaption>Figure 1: Waterfall model</figcaption></figure><p>The problem with the <em>Waterfall </em>model is exactly this: planning must be finished before we even start the actual work. This phase is prone to be done without fully understanding our goal, to begin with.</p><blockquote>I want to be happy in life!</blockquote><p>You might have similar, vague ideas in mind, but may end up with actions and requirements that do not contribute to <em>your </em>personal happiness. You rather take general steps that you heard might help but are not tailored towards you. Be clear about what you want to achieve, identify steps that lead you there, and <strong>gather just enough information so that you can start</strong> walking the walk!</p><h3>The “21” days myth</h3><p>This myth is attributed to Maxwell Maltz. It states that it takes 21 days to form a new habit. There is a <a href="https://jamesclear.com/new-habit">great article</a> debunking it, so I will just summarize the main points:</p><p>3 weeks is a short enough period to be achievable and a long enough period to be believable. In reality, it takes around <strong>2 months for a new behavior to become second nature</strong>. Most people quit as they do not see results quickly enough.</p><p>The good news is: you don’t have to be perfect as occasional mistakes do not have a big effect on long-term habit-forming. Just find a quick way to get back on track and keep doing what you do, without being harsh to yourself. Be patient and accept your flaws.</p><h3>You are the product of your actions</h3><p>Consider your life a <em>product</em>. You have tasks to deliver new <em>features </em>(habits)<em> </em>to said product. You want to ship ready parts (implemented habits) as quickly as possible to satisfy your product owners (yourself, family, friends, boss, girlfriend/boyfriend, etc.).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*xU6ysX2pfGRBaufm.jpg" /><figcaption>Figure 2: Scrum</figcaption></figure><p>Little by little, you can set up a <em>Life Roadmap</em>, where your long-term goals are defined, and you can link smaller tasks you want to achieve in accordance with said long-term goals. Take into consideration, that this Roadmap can change over time, and, as an actual agile Product Roadmap, it is never fully finished.</p><h3>As a … I need … so that …</h3><p>First, write down your aspirations, what you want to implement or change in your life, creating a <em>Product backlog</em>. These should be prioritized based on importance, but you can change the order if it helps. For example, I enjoy starting a project with some quick wins that give momentum and initial satisfaction that fuels the rest of the work. So you might not want to “<a href="https://medium.com/@alltopstartups/eat-the-frogs-first-thing-in-the-morning-and-other-better-work-habits-7070f9e79822">eat the frog</a>” right away, but generate some quick wins, then focus on the larger challenges.</p><p>Anyway, you should have a prioritized list of <em>stories </em>(detailed actions with associated benefits), that follow the structure mentioned in the title of this paragraph. A few examples might be:</p><blockquote><strong>As a</strong> productive person, <strong>I need</strong> to <a href="https://www.youtube.com/watch?v=3sK3wJAxGfs">make my bed every morning</a>, <strong>so that</strong> I accomplish the first task of the day, giving me a small feeling of pride and gets my surroundings in order</blockquote><p>, or</p><blockquote><strong>As a</strong> resilient individual, <strong>I need</strong> to avoid news and social media in the morning,<strong> so that</strong> I start the day in a good mood that carries on towards the rest of the day.</blockquote><h3>Time for a Sprint</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/0*AIpa_cVtgMgJOaf-" /><figcaption>Photo by <a href="https://unsplash.com/@bradencollum?utm_source=medium&amp;utm_medium=referral">Braden Collum</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Now that you have a list of well-defined user stories, pick some from your product backlog that you will work on during the first <em>sprint</em>. This might be done by order, or you can pick small habits which can complement each other. For example, if you would like to be more knowledgeable and have a better sleep at night, you can switch off your devices 1 hour before going to bed, so that the emitted blue light does not interfere with your sleep. In your — now Netflix-free — one hour, you can read some of those books that you have on your shelf gathering dust for too long now.</p><p><strong>The length of a sprint should be around the time it takes to form a new habit</strong>. Don’t worry if you have no clue about this in the beginning. You can use rules of thumb such as the 66 days, or even the 21 if these are smaller habits that you think can be implemented in shorter times.</p><p>The next step is to <strong>estimate the size of the activities</strong>: how hard do you think it is to incorporate them into your routine? How long does it take to be comfortable with them taking part in your life? Those sizes can be along the lines of a <a href="https://elearningindustry.com/using-the-fibonacci-sequence-makes-better-agile-development-5-reasons">Fibonacci sequence</a> or any numbers that can work for you. With knowing the effort and time requirements of each task, you should be able to start scheduling your first iteration.</p><p>Bear in mind that the way you define activities and sizes should not feel like “work” per se. If you are not at ease with strictly following these principles, pick and choose what suits you and invent your own ways of prioritizing/estimating that you feel most comfortable with.</p><h3>Iterate for perfection</h3><p>Once you have it all settled, it is time to work on your tasks. Start developing your features from the sprint backlog in the time that you estimated for them. This is not the time required to form the actual habit, rather how long it takes for initial success. For example, if your sprint’s goal is to be more physically active for 2 months, one iteration could be 2–3 weeks when you try your best to fit this routine into your schedule. During the next iteration, you might change things up based on your experience: if you did not like running, try TRX, squash, or other activity that might help you achieve your goal easier. Switch up frequency and length of occasions as you see fit.</p><h3>Daily (or weekly) Scrum</h3><p>Try to free up some minutes in your day (or week) to reflect on the signs of progress you made. Many times we only look in the future to see our goal still far away from ourselves. It is healthy to look back and appreciate what we have done so far. <strong>Change is fueled by pride, not shame</strong>.</p><p>Another aspect is to assess what went wrong during the process and examine it carefully. If your goal was to eat more healthily, why did you end up ordering a large pizza and eating it all by yourself last Thursday? What were the circumstances, inner thoughts, and motives behind that action and how can you avoid it next time you have the temptation? <em>(tip: mindfulness exercises can help with craving thoughts).</em></p><p>Last but not least, think about the next time when you take a step towards your goal. <em>When </em>will you do it, <em>what </em>will you do, <em>how </em>will you do it, and is there any <em>blocker </em>in front of you that might prevent it? Try removing all barriers to <strong>make your tasks as effortless as possible</strong>. If you want to go for a run, prepare your shoes and clothes the night before. When the time comes, you just hop into them, not allowing you to have doubting thoughts while you are dressing up.</p><h3>Sprint Review / Retrospective</h3><p>Once you are finished with alliterations in your first sprint, try seeking answers to three questions:</p><ol><li><em>What went well? </em>Recognize your achievements, even if they are small, and be proud of yourself.</li><li><em>What did not go well? </em>Discover why you diverted from your plan and the inner motives behind your actions.</li><li><em>What are the actions to take? </em>Find practices from your <em>step 1</em> features you can leverage to implement for other goals. Understand the “why” behind <em>step 2</em> features. Remove any obstacles, or iterate on the tasks, keeping the original goal in sight.</li></ol><p>Don’t forget to celebrate and treat yourself to completing your first Habit-Sprint! You earned it!</p><p>The only thing you have to do now is to repeat the process: look at new features from your product backlog to work on (or pick delayed ones from your previous sprint), plan the next iterations, and continue changing your life step-by-step!</p><h3>Conclusion</h3><p>Our habits define us, they give structure to our life and they are a staple of how we see ourselves. Some habits create opportunities for growth, friendships, and betterment, while others are destructive.</p><p>All that being said, I think that framing personal development in the context of agile can help people to plan and make progress — and therefore minimize abstraction and anxiety often caused by figuring out how to move towards your desired goals. This way of thinking obviously won’t charm everyone — but that is the case with all self-development techniques anyway :)</p><p>I hope this article can aid you in your life to pick up good habits and get rid of negative ones more easily, using a few ideas cherry-picked from Scrum and Agile project management.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=879168794fbb" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/how-to-overcome-your-bad-habits-with-agile-a-data-guys-approach-879168794fbb">How to overcome your Bad Habits with Agile? — A data guy’s approach</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Correlation & Causation — How Alcohol Affects Life Expectancy]]></title>
            <link>https://medium.com/data-science/correlation-causation-how-alcohol-affects-life-expectancy-a68f7db943f8?source=rss-6d8c19a5042a------2</link>
            <guid isPermaLink="false">https://medium.com/p/a68f7db943f8</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[health]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[causation]]></category>
            <category><![CDATA[correlation]]></category>
            <dc:creator><![CDATA[Bence L. Tóth]]></dc:creator>
            <pubDate>Sat, 08 Jun 2019 13:42:39 GMT</pubDate>
            <atom:updated>2019-07-27T15:42:48.758Z</atom:updated>
            <content:encoded><![CDATA[<h4>Should we drink MORE to live longer?</h4><h3>Correlation &amp; Causation — How Alcohol Affects Life Expectancy</h3><h4>Correlation does not imply causation!</h4><p>We hear this sentence over and over again as beginner statisticians and data scientists.<em> </em>But what does that actually mean? This small analysis uncovers this topic with the help of <strong>R</strong>, and simple regressions, focusing on how alcohol impacts health.</p><h3>Dionysus’ Dilemma</h3><p>There are numerous studies done on the health effects of alcohol and how it can change the life span of people. Just by typing <em>“alcohol life expectancy study”</em> into Google, we end up with around 11,000,000 results. Some may claim that <a href="https://www.acc.org/latest-in-cardiology/articles/2017/08/14/12/54/consuming-alcohol-in-moderation-can-lower-mortality-risks">moderate drinking can actually be beneficial to our health</a>, while most studies suggest that <a href="https://www.reuters.com/article/us-health-alcohol/even-one-drink-a-day-linked-to-lower-life-expectancy-idUSKBN1I42H6">Even one drink a day linked to lower life expectancy</a>.</p><p>This analysis examines the pattern of association between alcohol consumption and the average life expectancy of various countries, explaining the possible correlations and offers insight into why these correlations might be present.</p><p>My initial hypothesis is that countries with higher life expectancy also consume more alcohol. This might sound controversial at first, but it is probably due to other factors like higher quality of life, and only the byproduct of it is access to alcoholic beverages.</p><h3>The Ingredients</h3><p>3 datasets were used for this task:</p><ul><li><em>Drinks.csv</em>: number of alcohol servings per capita per year for 15 years of age or older (for beer, wine, and spirit) across various countries</li><li><em>LifeExpectancy.csv</em>: life expectancy and other health factors across various countries</li><li><em>CountriesOfTheWorld.xlsx</em>: geographical and socio-economic data across various countries.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/242b7a1dd8cb7e4a7ef7f33c3476afd9/href">https://medium.com/media/242b7a1dd8cb7e4a7ef7f33c3476afd9/href</a></iframe><h3>Mixing up the Drinks</h3><p>The drinks data contains servings of beer, wine, and spirit for 193 countries. These figures are stored as characters, so we can convert them into numeric data types.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0aa01961282124e5624cd209466d4cb1/href">https://medium.com/media/0aa01961282124e5624cd209466d4cb1/href</a></iframe><p>Missing values appear as question marks within the dataset, symbols which we shall replace with <em>NA</em> values.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0f7a49011aa539d026c4647b12a5e414/href">https://medium.com/media/0f7a49011aa539d026c4647b12a5e414/href</a></iframe><p>We need a way to aggregate the servings in a meaningful way: French people might consume more wine, while Germans might drink more beer, according to well-known stereotypes. Nevertheless, we need a way to compare these countries on a similar scale.</p><p>I wrote a function that calculates the total litres of pure alcohol for each country, using the following formula:</p><blockquote><em>total</em>_<em>litres</em>_<em>of</em>_<em>pure</em>_<em>alcohol </em>= <em>beer</em>_<em>servings </em>∗ (12 ∗ 0.0295 ∗ 0.05) + <em>wine</em>_<em>servings </em>∗ (5 ∗ 0.0295 ∗ 0.1) + <em>spirit</em>_<em>servings </em>∗ (1.5 ∗ 0.0295 ∗ 0.4)</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/545f33404404bb6c8c4f9b56983f6ad7/href">https://medium.com/media/545f33404404bb6c8c4f9b56983f6ad7/href</a></iframe><p>We end up with the following table:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/731d8c372a8fd4869cf859bc962b1a1a/href">https://medium.com/media/731d8c372a8fd4869cf859bc962b1a1a/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*_AaxUJuBtmnMGd_NtvRAdg.png" /><figcaption>Table 1: Drinks dataset</figcaption></figure><h3>Expectations in Life</h3><p>The life_exp data has various measures on health (life expectancy at birth, at age 60 and healthy life expectancy), showing data gender-wise and combined, spanning from 1990 to 2013.</p><p>I focus on life expectancy at birth for both sexes in 2012 and joined the drinks data by country.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5d30b80dfbe7d57d9ae5577f7366f099/href">https://medium.com/media/5d30b80dfbe7d57d9ae5577f7366f099/href</a></iframe><p>Carrying out a simple correlation test between life expectancy and alcohol consumption, we can see a result of 0.521.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c09c892fc80c198ba7254cd260e224c6/href">https://medium.com/media/c09c892fc80c198ba7254cd260e224c6/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*80aEZKhpHSRz3mrhDpGxPw.png" /></figure><p>This signals a moderately high correlation, suggesting that countries with higher alcohol consumption also have a higher life expectancy. We will further investigate this phenomenon.</p><h3>We are the World</h3><p>The countries data is a bit messier than the other two. It is stored as a <em>.xlsx</em> file. We skipped the first 3 nonimportant lines on sheet 1 during import. The (now) first line contains the headers, and some part of it is left in the second line as well.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6a596cb47b18fb6ac8fe845ca7a27d07/href">https://medium.com/media/6a596cb47b18fb6ac8fe845ca7a27d07/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tBeAd5G7fDxFQG8ua0Lfnw.png" /><figcaption>Table 2: Countries dataset</figcaption></figure><p>I combined the first and second lines into a single header, cleaned column names (from spaces, dots, etc.) and converted the necessary columns to a numeric type. Finally, I merged the geographical and socio-economic data with the previous dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4198cda6ace65d0d30a5a5e704642985/href">https://medium.com/media/4198cda6ace65d0d30a5a5e704642985/href</a></iframe><h3>Correlation and Causation</h3><p>Let’s have a quick look at how different factors are correlated with life expectancy:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2bf383a1230c52805aba9a155b640bdf/href">https://medium.com/media/2bf383a1230c52805aba9a155b640bdf/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*aRJkh6c8R3KfpBCWYYOj0Q.png" /><figcaption>Figure 1: Correlations of numeric features</figcaption></figure><p>As discussed before, alcohol consumption (combined and serving-separated) has a relatively positive correlation with life expectancy. Infant mortality and death rate are of course negatively correlated (the more/quicker people die, the more it will lower the average).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1092b1ebc1f85b66b8b153302286dc33/href">https://medium.com/media/1092b1ebc1f85b66b8b153302286dc33/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/763/1*c1-tGUYiMfXPNynJo9zxLQ.png" /><figcaption>Figure 2: Highest positive and negative correlations</figcaption></figure><p>The number of phones, literacy, and GDP per capita is also very highly correlated with people living longer. This leads to my assumption in the beginning: better educated, richer, technology-equipped countries have a higher life expectancy. These circumstances provide the availability of alcoholic beverages for citizens, and they consume more of them than poorer countries without that level of access.</p><p>We can see the correlations by each region to prove that claim:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d62574eb54700d3a8594b69f0cbb028a/href">https://medium.com/media/d62574eb54700d3a8594b69f0cbb028a/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*hPTY9ygJzl2LR8gEJgVDwA.png" /><figcaption>Figure 3: Alcohol on Life Expectancy</figcaption></figure><p>The hypothesis seems to be right indeed. More developed regions such as <em>Europe</em>, the <em>Americas</em> and the <em>Western Pacific</em> (Australia, New Zealand, etc.) seem to have a positive relationship according to the loess regression. On the other hand, less developed regions like <em>Africa</em> and <em>Eastern Mediterranean</em> tend to consume less alcohol due to economic/cultural/religious reasons, and their life expectancy is not affected by the magnitude of drink intake.</p><p>If we look at the log GDP per capita regressed on life expectancy, it shows the clearest picture:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c9d8c63a26c0018e77162c6cdf3269d1/href">https://medium.com/media/c9d8c63a26c0018e77162c6cdf3269d1/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*Z-l5UdvTmXIfin5EMVVr_g.png" /><figcaption>Figure 4: GDP on Life Expectancy</figcaption></figure><h3>Summary</h3><p>We could see at first how alcohol consumption is correlated with life expectancy, and why it is dangerous to immediately jump to conclusions. After further inspections, we discovered that it is developed countries that have the highest life expectancy and they also tend to consume more alcohol, but one does not necessarily imply the other. This is why it is very dangerous in implying causation from correlation!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*iDG1c9i_6CBvCaL9Mm8YiA.png" /><figcaption>Source: <a href="https://xkcd.com/552/">https://xkcd.com/552/</a></figcaption></figure><h3>Afterword</h3><p>This project was done as a requirement for the Mastering the Process of Data Science course at Central European University in Hungary. The R code along with the datasets can be found in my <a href="https://github.com/BenceToth/ceu_life-expectancy">ceu_life-expectancy</a> repository on GitHub.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a68f7db943f8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/correlation-causation-how-alcohol-affects-life-expectancy-a68f7db943f8">Correlation &amp; Causation — How Alcohol Affects Life Expectancy</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[R for Product Analytics]]></title>
            <link>https://medium.com/data-science/product-analytics-65812e82b2c2?source=rss-6d8c19a5042a------2</link>
            <guid isPermaLink="false">https://medium.com/p/65812e82b2c2</guid>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[product-analytics]]></category>
            <dc:creator><![CDATA[Bence L. Tóth]]></dc:creator>
            <pubDate>Thu, 06 Jun 2019 11:10:14 GMT</pubDate>
            <atom:updated>2019-07-27T02:33:07.806Z</atom:updated>
            <content:encoded><![CDATA[<h4>Deriving Insights from User Behavior with R</h4><h4>How to Derive Insights from User Behavior</h4><p>This analysis is based on the concepts of the book <a href="http://leananalyticsbook.com/">Lean Analytics: Use Data Analytics to Build a Better Startup Faster</a> by Alistair Croll and Benjamin Yoskovitz. It aims to carry out a simple data exploration regarding the topics of Acquisition, Activity and Retention, three key lifecycle stages of a product, according to the book.</p><h3>About the data</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/458e8b090396080eb09f0fef580d1441/href">https://medium.com/media/458e8b090396080eb09f0fef580d1441/href</a></iframe><p>The example is based on a real Software as a Service, subscription-based product. There are two datasets available:</p><ul><li><em>registrations.csv</em>: unique users with basic demographics</li><li><em>activity.csv</em>: which users have been active in which month</li></ul><p>More information about the datasets can be accessed on the project’s <a href="https://github.com/BenceToth/ceu_product-analytics">GitHub repository</a>.</p><h3>Acquisition</h3><blockquote><em>Acquisition is a stage where you should generate attention through a variety of means, both organic and inorganic. Relevant metrics to assess acquisition include traffic, mentions, cost per click, search results, cost of acquisition, open rate etc.</em></blockquote><p>As a first step in the analysis, let’s see the number of registrations in each month:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3382222f6421bb95282b24c8b1c81325/href">https://medium.com/media/3382222f6421bb95282b24c8b1c81325/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*sPiuo38uEWzN8mX-RI4nkg.png" /><figcaption>Figure 1: number of registrations per month</figcaption></figure><p>We can see that year 2 has a lower number of registrations overall (21831 vs. 18295). In the first year, most registrations occurred during the autumn/early winter period (September — November). We have no information on when this product was released to the market, the lower periods in the first half of year 1 might resemble the early adaptors, and the spike in the second part could be a successful marketing campaign. After some lows during the Christmas holidays, there is a second peak until May of year 2, then a sudden, 36.87% drop in registrations. This might be a failed product update, a decrease in market reach or other circumstances.</p><p>To dig deeper into the data, we can see the year-over-year growth of registrations:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/05f49833212ba592df23c03fdf82122f/href">https://medium.com/media/05f49833212ba592df23c03fdf82122f/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/210/1*rZ--qLAuLPTmfZLevq0qhQ.png" /><figcaption>Table 1: Year on Year growth of registrations</figcaption></figure><p>The next plot shows the growth rates for each month, together with the Naive forecast of Month 22 (October of the second year):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/65aac26eb1d595f78e292cd5c8bbb4dd/href">https://medium.com/media/65aac26eb1d595f78e292cd5c8bbb4dd/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*HBmDrXMvY2eQPXQ5atPsmg.png" /><figcaption>Figure 2: Year-on-year growth &amp; Naive Forecast</figcaption></figure><p>The year-over-year data seems to be decreasing, but keeping a higher number of registrations in year 2 up until September. The Naive forecast for October is 0.976.</p><p>We can investigate if there are any differences in registration numbers based on geographic regions:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eb8ada22337f71532f8bcb4b26b18371/href">https://medium.com/media/eb8ada22337f71532f8bcb4b26b18371/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*cgu2l1VnfkxhUJ-NFKCagg.png" /><figcaption>Figure 3: Region differences in year-on-year growth</figcaption></figure><p>Comparing the regions, there seems to be a difference. <em>America</em> underperforms both <em>EMEA</em> and the rest of the world, except at the beginning of year 1. All show a decreasing trend in registration numbers, but the <em>ROW</em> region seems to produce the highest growth. It is likely to drive future growth in registrations, while <em>EMEA</em>, and especially <em>America</em> might have different interest in the product, or they were less affected by the marketing campaigns.</p><p><strong>Activity</strong></p><blockquote><em>Activation is turning the resulting drive-by visitors into users who are somehow enrolled. It can be done by tracking a specific activity milestone, such as enrollments, signups, completed onboarding process, used the service at least once, subscriptions etc.</em></blockquote><p>The plot below shows the number of active users in each month:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/277f4171cdcf9feb3a0603ce97e3dcfa/href">https://medium.com/media/277f4171cdcf9feb3a0603ce97e3dcfa/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*qb0k2PWBvyjAStsBPCAZuA.png" /><figcaption>Figure 4: number of active users per month</figcaption></figure><p>The number of users increased until the summer of the first year, then it dropped, and only returned to the May level in September. It peaked during November and decreased during Christmas and winter. The second year had its highest number of active users between March and May and again decreased for the summer, showing a seasonality trait.</p><p>As we have seen, the region of <em>America</em> struggles with new registrations. Let’s see the percentage of <em>America</em> among active users in each month:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ade65b4d5a4f6e35d4f3369f8fc681c5/href">https://medium.com/media/ade65b4d5a4f6e35d4f3369f8fc681c5/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*xTnpD54cEsQrQnijjEdbzg.png" /><figcaption>Figure 5: number of active users per month</figcaption></figure><p>We can see that the number of users from <em>America</em> is on average between 17–22%. The company has to think about investing in other ways to drive that market or focus on the rest of their clients which seem to have better registration and activity rates.</p><p>It is worth it to also take a look at the users’ activity patterns. The users can be classified as <em>New</em> (registered that month), <em>Retained</em> (was active the previous month as well) or <em>Resurrected</em> (was inactive the previous month and not <em>New</em>).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0bc1c5a4f5cc066b551cfb4dae25f84b/href">https://medium.com/media/0bc1c5a4f5cc066b551cfb4dae25f84b/href</a></iframe><p>To illustrate this, take a look at a randomly chosen user’s activity history:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/52a3ead4e21138e82531eb3626c700f9/href">https://medium.com/media/52a3ead4e21138e82531eb3626c700f9/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*SA2l9XzylkB7Xpsdgik9Kw.png" /><figcaption>Table 2: User activity history</figcaption></figure><p>This user registered in July kept using the service in August then it did not perform any activity in September. He returned in November and was active in December as well. He did not use the product in the second year.</p><p>Let’s now look at the number of <em>Retained</em> active users in each month:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5c6779f6b55e13dd08d3fb8115ec40a0/href">https://medium.com/media/5c6779f6b55e13dd08d3fb8115ec40a0/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*U6mWPnMwt7jM_E2FbGFUHQ.png" /><figcaption>Figure 6: number of Retained users per month</figcaption></figure><h3>Retention</h3><blockquote><em>The main task during the Retention phase is to</em> <em>convince users to come back repeatedly, exhibiting sticky behavior. It can be tracked by metrics such as engagement, time since last visit, daily and monthly active use, churns etc.</em></blockquote><p>In the case of our example, we can calculate the second-month retention rate (registered users in Month 1 who have been active in Month 2).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cfbeb8a7442dd0789e36ea1a5b787f42/href">https://medium.com/media/cfbeb8a7442dd0789e36ea1a5b787f42/href</a></iframe><p>The second-month retention rate is 46.33%. It can be considered pretty good, it usually ranges from 20% to 60%.</p><p>Let’s see the same metric for year two (between Month 13 and 14).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d2eb95bea31c2de2a74ea78fdc900230/href">https://medium.com/media/d2eb95bea31c2de2a74ea78fdc900230/href</a></iframe><p>36.84% of users who registered in January, second-year were active in February. It is still an acceptable value, but what could be behind the 10% drop since last year? It can be because the majority of users who are interested in the product have already signed up, so the pool of potentially interested users for new registrations became smaller. This way, the users who joined in the second year were not that engaged in the product and decided to opt out of it in the upcoming month with a higher percentage.</p><p>We can also calculate the second-month retention rate based on the operating systems of the users:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/afcc61298023493c2a1f598ce7073660/href">https://medium.com/media/afcc61298023493c2a1f598ce7073660/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*Dm4RJ3I0rUtycF5nueUl5A.png" /><figcaption>Table 3: Second-month retention rate by Operating System</figcaption></figure><p>For <em>Unknown</em> operating systems, the retention rate is way lower (38%) than for <em>Mac</em> users (61.5%). One action is to investigate the data collection process and define which operating systems these users are actually using. There is a significant, 20% difference between <em>Mac</em> and <em>Linux</em> users as well. This might signal that the product is not that optimized for <em>Linux</em> as it is for <em>Mac</em>.</p><h3>Conclusion</h3><p>With this basic introduction to different product analytics practices, we could see how little effort and some programming knowledge could lead to interesting hypotheses about user behavior. Methods such as A/B testing could be applied to test these hypotheses, for example: how optimizing the Linux version of the product will influence retention rates, which regions should be targeted with specific advertising campaigns, etc.</p><p>This project was done as a requirement for the Mastering Product Analytics course at Central European University in Hungary. The R code along with the dataset can be found in my <a href="https://github.com/BenceToth/ceu_product-analytics">ceu_product-analytics</a> repository on GitHub.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=65812e82b2c2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/product-analytics-65812e82b2c2">R for Product Analytics</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Metallica to Adele — Text Analysis of successful song lyrics with R]]></title>
            <link>https://medium.com/data-science/text-analysis-of-successful-song-lyrics-e41a4ccb26f5?source=rss-6d8c19a5042a------2</link>
            <guid isPermaLink="false">https://medium.com/p/e41a4ccb26f5</guid>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[music]]></category>
            <category><![CDATA[text-analysis]]></category>
            <category><![CDATA[sentiment-analysis]]></category>
            <dc:creator><![CDATA[Bence L. Tóth]]></dc:creator>
            <pubDate>Sat, 18 May 2019 20:21:36 GMT</pubDate>
            <atom:updated>2019-11-01T13:33:38.322Z</atom:updated>
            <content:encoded><![CDATA[<h3>From Metallica to Adele — Text Analysis of successful song lyrics with R</h3><h3><strong>Nothing Else Matters</strong></h3><blockquote>”De gustibus non est disputandum” — goes the Latin phrase.</blockquote><p>As each of us has our own preferences in music, different backgrounds that make us resonate with different topics and feelings, it might be the truth that <em>“In matters of taste, there can be no disputes”.</em><br>However, highly successful artists can be easily differentiated from mediocre ones. Success can be measured by the number of records sold, plays on Spotify, sold-out venues all around the world, etc.</p><p>The following meme gives an exaggerated perspective on how song lyrics can be entirely different, coming from two highly successful musicians:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*C765h69JD6GayqgP3J0MhA.jpeg" /><figcaption>Figure 1: lyrics depth</figcaption></figure><p>This article does not want to take sides with either Beyoncé or Freddy Mercury, but rather investigate the word choices, lyrical motives, topics and feelings used in highly successful works of music.</p><p>Do song lyrics have any impact on the success of an artist, or is it all about production value, catchiness and nothing else matters?</p><h3><strong>What Do You Do For Money Honey</strong></h3><p>The <em>songlyrics.xlsx</em> dataset contains lyrics for 6 emblematic figures of different genres and times, using their most sold album. It includes release year, artist, album, genre, number of records sold and amount of revenue generated, along with the lyrics of each song from that album. The following artists and albums are used during the analysis:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/1*jEDiqNiCoHs2lW78eypuWw.png" /><figcaption>Table 1: Artists and albums</figcaption></figure><h3><strong>Cleanin’ Out My Closet</strong></h3><p>The dataset only needs slight data cleaning:</p><ul><li>In some songs, brackets are used to signal [Verse], [Chorus] or artists that perform a part of the song, [Dr. Dre] etc. These will be removed.</li><li>It needs to be transformed into tidytext format: one token (word) per row.</li><li>We have to create a custom stop words dictionary for meaningless words like James Hetfield’s <em>“yeah”</em>s, Eminem’s <em>“yo”</em>s and Michael Jackson’s <em>“Ma ma se, ma ma sa, ma ma coo sa”</em>s.</li><li>Remove numbers.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/28a4ddc4abd527293749f12117a3dcea/href">https://medium.com/media/28a4ddc4abd527293749f12117a3dcea/href</a></iframe><h3><strong>Wanna Be Startin’ Somethin’</strong></h3><p>Let’s start by using dplyr’s count() to find the most common words used:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/851f4e7c008b23dea9f7529526716e4e/href">https://medium.com/media/851f4e7c008b23dea9f7529526716e4e/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*_0AzW3RAUI3ElcfERhx21A.png" /><figcaption>Figure 2: most used words in the dataset</figcaption></figure><p>It is not surprising that the word <em>love</em> is the most used word out of all meaningful words in the dataset. <em>Gonna</em> and <em>wanna</em> might be considered as stop words. Words like <em>girl</em>, <em>baby</em> and <em>heart</em> are also connected to the topic of love and relationships.</p><p>The plot below shows how 3 artists use words differently:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d2c70f49acd20f2269c1c57f493304c8/href">https://medium.com/media/d2c70f49acd20f2269c1c57f493304c8/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/789/1*042AVSrqOVvW8VaZJkiOkw.png" /><figcaption>Figure 3: Different word choices</figcaption></figure><p>There are not many common words used by representatives of different genres. They use <em>head</em>, <em>call</em>, and <em>boy</em> in relatively similar proportions, but Metallica sings about being <em>broken</em>, <em>free</em> and <em>dream</em>s way more than Eminem or Adele. To quantify their differences, we can use Pierson’s correlation. Its 0.03 for Adele and Metallica and 0.09 for Metallica and Eminem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eLdYuWCTG_oVTj25jFpy-w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iGRMaJw0PaSWr_4s5A1i8A.png" /></figure><p>This is expected, as the dataset spans over 31 years (1980–2011) and uses 6 different genres. It was expected that different types of music will have different word choices.</p><p>We can also check how the word frequencies are different for each genre:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*klgzEBgHn7Fj6-g8hC_q0Q.png" /><figcaption>Figure 4: Word frequencies by Genre</figcaption></figure><p>Soul seems to be about <em>love</em> and <em>tears</em>, rap has a lot of cursing, metal is about <em>struggle</em> and <em>misery</em> and reggae is about <em>love</em> and the <em>people</em>. Sounds accurate.</p><h3><strong>Is This Love</strong></h3><p>To gain more insight into the success of these songs, let’s investigate how do they work with feelings, by joining a sentiment lexicon.</p><p>First, we can see what is the average sentiment score for each genre. Is it true that reggae is about making us feel happy while metal touches on our negative feelings?</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/86bf937a982f7356016080097f1aa273/href">https://medium.com/media/86bf937a982f7356016080097f1aa273/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*e_4-juq7DeuJi2hScaOITg.png" /><figcaption>Figure 5: average sentiment per Genre</figcaption></figure><p>It turned out as expected for the “happy” genres, but surprisingly, rock has an overall more negative sentiment than metal, and rap surpasses both.</p><p>To dig deeper into this phenomenon, let’s see the most influential words contributing to this average sentiment.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c2cba87279c056f1f898366cc9131820/href">https://medium.com/media/c2cba87279c056f1f898366cc9131820/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*dYREjBB0Jw3Xp6TRNDouVg.png" /><figcaption>Figure 6: Words contributing to sentiments per Genre</figcaption></figure><p>Now we see why rap turned out to be the most negative, looking at the excessive number of swear words. They might be treated as stop words, but they also carry meaning. My hypothesis is that if the song contains many words from the S-F-B triangle, it is probably not about a happy topic.</p><p>There are also some mischaracterized words: <em>shake</em> in rock is about dancing/moving (“<em>Shake a leg</em>”), and it is a rather neutral word. Of course, the word <em>jam</em> in reggae is not negative either, playing music together has more of positive sentiment.</p><h3><strong>One And Only</strong></h3><p>We can quantify what each genre is about using the <em>tf-idf</em> statistic. It shows how important one word is to a genre in a collection of genres for that genre.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c60388b57c686433df213dd1959c599f/href">https://medium.com/media/c60388b57c686433df213dd1959c599f/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*Z97UJ4BboIKou2-gLBbKdw.png" /><figcaption>Figure 7: Highest tf-idf words per Genre</figcaption></figure><p>Metal is pretty straight-forward: we see important words like <em>misery</em> and <em>struggle</em>, which are not that common in other genres. Pop and Rap is less representative: MJ’s highest tf-idf words are mostly his song titles (<em>billie</em>, <em>jean</em>, <em>startin’</em>, <em>somethin’</em>, <em>thriller</em> etc.), while Slim Shady uniquely mentions Dr Dre and himself (<em>marshall</em>). Reggae is using <a href="https://en.wikipedia.org/wiki/Jamaican_Patois">patois</a> slangs like <em>jah</em>, and speaks about <em>movement</em> and <em>exodus</em>. Rock is about <em>hell</em>, <em>drink</em>ing, and singing about Rock’n’<em>roll</em> itself.</p><h3><strong>The Struggle Within</strong></h3><p>Let’s see what different topics lie within the lines of these songs, using Latent Dirichlet Allocation.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bd17dd38f119cde7f647fc1e46dd0282/href">https://medium.com/media/bd17dd38f119cde7f647fc1e46dd0282/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*gtVrmB7xrAbKw73CbZ439Q.png" /><figcaption>Figure 8: Topic modeling using LDA</figcaption></figure><p>These word choices among topics again imply that it mainly depends on the genre of music which words are chosen and which topics are discussed.</p><p>The first topic is mainly about enjoying <em>life</em>, being in <em>love</em> with a <em>girl</em>, and happiness in general. Most of the words are similar to the most highly used words in reggae. Topic 6 is about a breakup, with words like <em>heart</em>, <em>time</em>, <em>met</em> and <em>tears</em>, most of them coming from the soul lyrics.</p><h3><strong>When The Music Stops</strong></h3><p>This analysis only focused on 6 artists and their most sold album to draw conclusions about what successful pieces of music are about. The outcomes definitely have some bias towards the artists themselves, but we can still draw a picture of their word usage and topics.</p><p>What makes a song successful is very much dependent on which genre it belongs to. Rap, rock and metal songs tend to have a more negative message, profane lyrics at times, talking about the pain and struggles of the author. Pop and soul singers tend to focus on love and breakup, using more positive words and they are about relationships.</p><p>This analysis can easily be repeated with more data to drive more general conclusions about the state of the music industry, and including less successful artists can further unfold the topic of what makes a successful song, which word choices and topics lead to higher sales and revenue.</p><h3><strong>Exodus</strong></h3><p>This analysis is based on the works of Julia Silge and David Robinson, with a focus on their book, <a href="https://www.tidytextmining.com/index.html">Text Mining with R</a>. The R code along with the dataset can be found in my <a href="https://github.com/BenceToth/nlp_songlyrics">NLP_songlyrics</a> repository on GitHub.</p><p>Special shout-out to <a href="https://www.linkedin.com/in/earino/">Eduardo Ariño de la Rubia</a> for equipping me with the underlying knowledge for this article at Central European University in Hungary.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e41a4ccb26f5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/text-analysis-of-successful-song-lyrics-e41a4ccb26f5">From Metallica to Adele — Text Analysis of successful song lyrics with R</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Combating Media Bias with AWS Amazon Comprehend]]></title>
            <link>https://pub.towardsai.net/combating-media-bias-with-aws-comprehend-e4f5e1bfe352?source=rss-6d8c19a5042a------2</link>
            <guid isPermaLink="false">https://medium.com/p/e4f5e1bfe352</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[bias]]></category>
            <dc:creator><![CDATA[Bence L. Tóth]]></dc:creator>
            <pubDate>Mon, 19 Nov 2018 22:24:50 GMT</pubDate>
            <atom:updated>2019-07-28T17:03:01.980Z</atom:updated>
            <content:encoded><![CDATA[<h4>Combating Media Bias with NLP | <a href="https://towardsai.net/">Towards AI</a></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TA-7zlUrFfXIwRxN" /><figcaption>Photo by <a href="https://unsplash.com/@randycolasbe?utm_source=medium&amp;utm_medium=referral">Randy Colas</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In the world of fake news and ideology-driven subjective media coverage, it is questionable which sources of journalism can be considered “reliable”. It happens many times that two different news outlets share two completely different takes on the same story. “Experts” point out different consequences of events from the whole spectrum, mostly the ones that suit their own agenda.</p><p>In these times, it is important to identify the biases of different news organizations towards certain topics.</p><p>Can <strong>machine learning</strong> help to discover these biases?</p><h3>AWS</h3><blockquote><strong>Amazon Comprehend</strong> is a natural language processing (NLP) service that uses machine learning to find insights and relationships in text. <em>(Source: aws.amazon.com).</em></blockquote><p>Combining AWS Comprehend with the web scraping capabilities of the <strong>R language</strong><em> (rvest package)</em>, an analysis can be carried out, with two articles that are written on the same topic, from two news sites with supposedly different background ideologies.</p><h3>Trump’s coverage</h3><p>The topic is the evaluation of Donald <strong>Trump’s first 100 days</strong> as President of the United States. In American politics, actions that took place within the first hundred days after the election are often a good approximation of the overall route that the current president is willing to take in his/her term, therefore a highly covered topic from both the left and the right.</p><p>The first article on Trump’s first 100 days is from <strong>Breitbart News</strong>, what is considered to be an alt-right news agency with columnists such as Milo Yiannopoulous and former chairman of Breitbart and White House Chief Strategist, Steve Bannon.</p><p>The second article is from <strong>Buzzfeed</strong>, a considerably left-wing internet media company, funded by Jonah Peretti, the co-founder of The Huffington Post.</p><p>Parsing the HTML content of these articles with web scraping and a CSS selector, the raw text is available. After minor data cleaning, it is possible to run Amazon’s sentiment analysis on each paragraph of the articles. Each of these is assigned an overall sentiment of “Positive”, “Neutral”, or “Negative”, with extra data on the percentage of assuredness. Eg.: a 0.78 positive measure of a paragraph means that Comprehend is 78% sure that that paragraph has a positive sentiment.</p><p>The summary of the two articles can be seen below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*n1Fy8QuX8VxDncndUXNPTA.png" /><figcaption>Statistics for sentiment analysis</figcaption></figure><p>We can see that Breitbart has more neutral statements than Buzzfeed, and has a higher mean of positive sentiments. On the other hand, it almost has the same negative sentiments, while Buzzfeed’s negative value is more than 3 times bigger than that of Breitbart.</p><p>The boxplot below illustrates the difference between negative sentiments:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/1*kRadJ7tZ6F_R3bvzpg5Ygw.png" /><figcaption>Distribution of negative sentiments</figcaption></figure><p>The median of negative sentiments is a bit higher in Buzzfeed paragraphs, but non-substantially. It seems that there are some parts of the text which have a way higher certainty of negative tone which shifts the mean to be bigger in their article. This could mean that Breitbart’s takes on the worse parts of Trump’s first 100 days are not stated that harshly, while Buzzfeed uses strong words to emphasize their negative assessment of certain legislation.</p><h3>Afterword</h3><p>This analysis does not necessarily prove leniency towards the topic from either side as the sentiment analysis of AWS might be off for more sophisticated sentences, cannot understand sarcasm and furthermore, comparing two random articles of the same topic does not infer bias in an entire news agency. However, further analysis can be done with various topics and more similar articles to potentially infer something about certain sites’ stand on broader ideologies.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e4f5e1bfe352" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/combating-media-bias-with-aws-comprehend-e4f5e1bfe352">Combating Media Bias with AWS Amazon Comprehend</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>