<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Data Analytics @ Hult - Medium]]></title>
        <description><![CDATA[Connecting innovators, academics, and practitioners with the ideas and companies that are growing the data science field. - Medium]]></description>
        <link>https://medium.com/hult-data-analytics-club?source=rss----c16eecc1ca30---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Data Analytics @ Hult - Medium</title>
            <link>https://medium.com/hult-data-analytics-club?source=rss----c16eecc1ca30---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 15:04:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/hult-data-analytics-club" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data Scientist vs Data Analyst — A primer]]></title>
            <link>https://medium.com/hult-data-analytics-club/data-scientist-vs-data-analyst-a-primer-d325b816d1af?source=rss----c16eecc1ca30---4</link>
            <guid isPermaLink="false">https://medium.com/p/d325b816d1af</guid>
            <category><![CDATA[business-analyst-training]]></category>
            <category><![CDATA[data-analyst]]></category>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[business-analyst]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Jonathan Kurniawan]]></dc:creator>
            <pubDate>Thu, 31 May 2018 19:58:43 GMT</pubDate>
            <atom:updated>2018-06-06T15:20:24.025Z</atom:updated>
            <content:encoded><![CDATA[<p>So you’re new to the data industry and want to know more and dive in. You’re talking to practitioners, and ‘Data Scientist’ and ‘Data Analyst’ keeps coming up. At first, you think they are interchangeable. Then, you thought “Hmm, looks like Data Analyst is a more junior Data Scientist”. Then more roles like ‘Business Analyst’ and ‘Data Engineer’ comes up and you don’t know what each role is anymore.</p><p>Not to worry, this article will cover some general differences each role has. A note that some responsibilities might crossover depending on specific companies or industries, but in general, they are different.</p><h3>Data Analyst</h3><p>‌A data analyst creates models and dashboards that describe the data, also known as <strong>descriptive statistics</strong>. These include averages, percentages, rankings, and so on. They focus on communication of data, and provide fast, “back-of-the-envelope” answers. The tools they use are Business Intelligence (BI) tools like Tableau, Power BI, and Chartio, and work with SQL. An analyst would work closely with the product team in empowering product managers to make data-driven decisions on what to build. They also help decide how to perform measurements.</p><h3>Business Analyst</h3><p>This term is commonly interchangeable with a data analyst. In some organizations, they also drive business decisions and have a broader scope. With their big picture focus, they often act as a bridge between business and tech roles.</p><h3>Data Scientist</h3><p>A data scientist creates models that predict the data, or <strong>predictive statistics</strong>. These include machine learning models like regression models and classification models (e.g. neural networks). They tend to provide precise answers by tweaking parameters and fine-tuning the accuracy of a model. They mostly work with Python, R etc. to create these models. A data scientist can work as a separate team, or integrate with engineering to build products.</p><h3>Machine Learning Engineer</h3><p>Similar to a data scientist, a machine learning engineer creates machine learning models. They focus more on building the core machine learning components with the support of the data science team.</p><h3>Data Engineer</h3><p>A data engineer creates the pipelines, storage, and ‘plumbing’ that connects the different data sources to enable data scientists and analyst to do their job. They prepare and transform the data to be accessible for analysis. This includes working with data cleaning pipelines, <a href="https://www.informatica.com/services-and-training/glossary-of-terms/data-warehousing-definition.html#fbid=QCiRNUrvo-a">data warehouses</a> (which stores aggregated structured data for a company), <a href="http://datawarehouse4u.info/ETL-process.html">ETLs</a> (extracts, transforms and loads the data into an application), and BI tools. A data engineer will often spend time on A/B testing and benchmark the infrastructure.</p><h3>Data Architect</h3><p>Closely related to Data Engineer, they also work on the management of data. More specifically, they work with their extensive database knowledge and how data is acquired in the business operations. They are in charge of the data architecture and infrastructure.</p><h3>Data Wrangler</h3><p>Sometimes known as Data Munger, they work to transform raw data into a format where it can be useful downstream to a data analyst. Their job is to make unstructured data into more structured data. They commonly use Python, R, and SQL. They work in the ETL layer where they extract the data in a raw form from the data source, sifting through it with algorithms and parsing the data into predefined structures, and provide the processed data either to someone who can work with analyzing the data, or store in a data warehouse.</p><p>Now that you know the differences, you’re ready to talk to practitioners in the data world!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d325b816d1af" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hult-data-analytics-club/data-scientist-vs-data-analyst-a-primer-d325b816d1af">Data Scientist vs Data Analyst — A primer</a> was originally published in <a href="https://medium.com/hult-data-analytics-club">Data Analytics @ Hult</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Key Takeaways from ODSC East 2018]]></title>
            <link>https://medium.com/hult-data-analytics-club/our-key-takeaways-from-odsc-east-2018-ca31986824f8?source=rss----c16eecc1ca30---4</link>
            <guid isPermaLink="false">https://medium.com/p/ca31986824f8</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[ai-governance]]></category>
            <category><![CDATA[odsc]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Jonathan Kurniawan]]></dc:creator>
            <pubDate>Mon, 14 May 2018 21:03:03 GMT</pubDate>
            <atom:updated>2018-05-18T18:34:43.138Z</atom:updated>
            <content:encoded><![CDATA[<p>The Hult Data Analytics club attended the 4th annual Open Data Science Conference (<a href="https://odsc.com/boston">ODSC</a>) East 2018 on May 1–4 in Boston, on the theme of “The Future of AI is Here”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*QCVBu4QM5WkWJPNi6GdbmA.jpeg" /><figcaption>Rafael Cerutti (left) and me (right) at ODSC East 2018</figcaption></figure><p>It was an incredible four days of great keynote speakers and technical workshops around the latest in the world of data and machine learning, with over 4500 attendees and 200 speakers.</p><p>Our key takeaway from the various speakers are:</p><p>(1) AI is moving from ‘research projects’ to ‘infrastructure at scale’. There are an increasing number of tools and middleware to make it easier to get into AI.</p><p>(2) Machine learning and AI is transforming many industries right now on their operations. Having said that, we should not be afraid of AI as there are lots for AI to work on.</p><p>(3) Deep learning is very powerful and is becoming mainstream, but it is not ‘magic’ and has limits. The analogy is of a blacksmith with only a hammer, who looks at all problems like they’re nails — there needs to be a whole suite of tools in the toolbox to solve different problems.</p><p>(4) AI governance, ethics and regulation a current concern.<br>- AI is not a silver bullet for systemic issues.</p><p>(5) There is an increasing number of open source data-science projects focusing on making the world a better place.</p><p>In the article below, I’ll elaborate on each of these takeaways.</p><h3>(1) AI is moving from ‘research projects’ to ‘infrastructure at scale’ thanks to the different tools that are being developed.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HypvpI5aUbpFveeDmfOPsw.jpeg" /><figcaption>One of the research booths at ODSC East</figcaption></figure><p>According to <a href="https://www.linkedin.com/in/manojsaxena/">Manoj Saxena</a>’s speech on Winning with AI, AI in its current state is where the web was back in 1999; with 2017 being the first year AI became mainstream. A couple of years ago, most of the progress in AI were research projects in contained environments, built in some researcher’s bedroom. Now, though, there is a large number of middleware and 3rd party software providers, especially open source ones, that help with various levels of machine learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GoBAjpQz3i-dVqY41rUi5Q.jpeg" /><figcaption>List of famous open source data tools.</figcaption></figure><p>Now there is a focus on bringing data science and software development workflows, such as ‘data ops’ so the AI model could go and reach for the data, and vice versa. This is to allow enterprises to operationalize AI at scale, which reduces skills needed to manage the models that work on distributed data.</p><p>This is increasingly important, as it currently stands there is a large gap in between business ambition and execution, and the time to business value is still quite large, not to mention the large investment of putting up a sizable data science team. The trend of open innovation on these tools will increasingly help close the gap and can allow in-house data science teams to target high-value use cases and reduce time to source, train and scale data and machine learning models.</p><p>Currently most of the work done is on making faster and more accurate classifications and regressions, however, this is mostly building the components, but not assembling a general workable AI to solve all our complex needs. An analogy is like building steering wheels, gearbox, etc. but not assembling an entire car. This is important to know as we do not hype up current AI technology beyond its current capabilities, as we discuss in part 3).</p><blockquote>To go into detail of some of the popular tools, things like Apache MXNet, or Tensorflow library by Google and its Keras API are making it easier for beginners to start with Deep Learning. There are also countless tutorials online on Plotly and Matplotlib to help one get started on using data visualization tools which would compete with Tableau. Kubernetes, Kafka, mongoDB and Hadoop are all commonplace names both within high-growth startups and enterprise companies, and Jupyter notebook has become the de-facto standard of sharing documentation with live code and explanatory text for tutorials and deliverables. This trend will only continue to grow and more open-source tools will continue to power many of the applications we use daily.</blockquote><h3>(2) Machine learning and AI is transforming many industries right now on their operations.</h3><p>Most of the news we see on AI touch on futuristic sci-fi like self-driving cars and beating the best human player at Go. But what isn’t so obvious is how much data analytics and machine learning is already commonplace and used in all sorts of settings, including everyday business operations.</p><p>To have these machine learning analytics be useful, a company must first have a <a href="http://www.seabourneinc.com/what-is-a-big-data-problem-and-how-do-i-know-if-i-have-one/">Big Data problem</a>: Volume (scale of data), Variety (structured or unstructured data), Velocity (making decision from streaming data within fractions of a second) and Veracity (data uncertainty), and ultimately attain value from insights of analyzing the data.</p><h4>Insurance</h4><p>To take one example of an industry, we had <a href="https://www.linkedin.com/in/aleksandar-lazarevic/">Aleksandar Lazarevic</a> from Aetna to touch on how they are using analytics to power decisions in insurance and healthcare, an industry known to be conservative and complex. They are able to detect fraudulent claims from providers that were excessively performing cosmetic surgeries. The approach is to first be hypotheses-driven from business to detect fraud schemes and then proceeded to using supervised models to detect variations of fraud schemes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y399ZjWBvOS5chKx91gQZA.jpeg" /></figure><p>Another example is using social determinant signals to predict readmission, with signals like number of close friends, neighborhood they live in, economic stability, and education level. Using these signals brought up significant differences in readmission for individuals that had roughly same age and physical fitness. There would be a need of being careful of systemic bias though, as we discuss in part 4) below.</p><p>Overall, we should be careful that the right problems get solved with the right tools, making sure that there is indeed a Big Data problem, and not use AI as a blanket solution to a hard problem, just because it’s ‘the cool thing to do’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U8d5KEBywGoQRpNx2GvK1A.jpeg" /></figure><h3>(3) Deep learning is very powerful and is becoming mainstream, but it is not ‘magic’ and has limits.</h3><p>We saw in 2016 of the high-profile matchup of <a href="https://deepmind.com/research/alphago/">AlphaGo</a> from Google’s DeepMind project against world-class Go player Lee Seedol, where AlphaGo won 4–1 and finally putting to rest that Go was impossible for a computer to beat a human player. AlphaGo was able to do this using something called <a href="http://www.inference.vc/deep-learning-is-easy/">Deep Learning</a>, where it analyzed thousands of different amateur and professional games to learn how to play Go and be performing at the top echelon of the game.</p><p>Since then, DeepMind has continued to build out a new program, <a href="https://deepmind.com/blog/alphago-zero-learning-scratch/">AlphaGo Zero</a>, using Reinforced Learning (RNN), which meant it was not even given <em>any</em> existing game data to learn the rules of the game, and instead had to figure out on its own by playing with itself. This iteration then continued to beat the winning AlphaGo model within just 40 days.</p><p>For those of you who see these developments in Deep Learning and Reinforced Learning and fear of robots taking our jobs, fear not — the hype and the power of Deep Learning has its limitations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*asQK-NHUx2a1gou9krULyg.jpeg" /><figcaption>An example of an ‘illusion’ output that current deep learning models struggle with.</figcaption></figure><p>According to <a href="https://medium.com/@GaryMarcus">Gary Marcus</a>’ keynote speech, Deep Learning requires lots and lots of data to begin to be useful, which limits its applications. Deep Learning currently also has limited capacity for <a href="http://ruder.io/transfer-learning/">Transfer Learning</a> to other domains, something that is required for a general purpose AI. It also struggles with open-ended questions and is a black box with how it calculates its output, something that’s becoming more of an issue as we begin questioning how models arrive at an answer, especially with<a href="https://www.financierworldwide.com/gdpr-time-to-explain-your-ai/"> regulations like GDPR</a> which might require a data scientist explain a model’s decisions. There is also the fact that we can’t trust the answer of a model fully, as shown in the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tU_SBcJiD85tofDMqaPlzA.jpeg" /><figcaption>Adversarial Patch that can ‘confuse’ a deep neural network model into thinking the bottom picture is a “toaster” and not a “banana”.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*bnNkz3EHukfButTI6LEbaQ.png" /><figcaption><a href="https://blog.openai.com/adversarial-example-research/">Attacking machine learning classification algorithms</a> into thinking the image is a ‘safe’ or ‘loud speaker’ instead of a washer.</figcaption></figure><p>Having said all this, Deep Learning is a very powerful tool that is becoming more accessible to the general public. Tools such as Google’s machine learning library Tensorflow and its Keras API abstracts away complexities that allow developers to create <a href="https://colab.research.google.com/github/tensorflow/workshops/blob/master/notebooks/1-mnist-with-keras.ipynb">deep neural networks in only a couple of lines.</a> Open source infrastructure and tools, like the one posted earlier, are allowing deep learning to become more mainstream. Projects such as <a href="https://quickdraw.withgoogle.com/">QuickDraw</a> are adding datasets to fuel the data hungry Deep Learning models that Google has.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R1nppHrT9HVOr6z3849HUw.jpeg" /><figcaption>Things we can do currently vs aspirational ones that are beyond current capabilities.</figcaption></figure><p>To dispel the myths, Deep Learning is another way of doing statistics, namely classifying things into categories, or regression for predictive analysis, and works well when the training data is not too far away from the real-world data it will be set upon. It works great at being a perceptual classifier, but less so on natural language and ‘common sense’. We would need to continue to build other tools to make sure we can solve complex real-world problems that current machine learning models are ill-equipped to solve.</p><p>For those who still believe general AI is going to replace us in the near future, consider Apple’s billion dollar investment in Siri:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*yVy92Ye8rTZmXAwoII8IxQ.jpeg" /><figcaption>No, I <strong>DON’T</strong> want McDonalds, silly Siri.</figcaption></figure><h3>(4) AI governance, ethics and regulation a current concern.</h3><p>With the increasing speed of development of AI and other tech innovations such as Blockchain, IoT, etc. we begin to find our society being forced to adapt and keep up with all these changes. Visibility, explainability and governance become a key concern to progressing AI without adverse effects to the society.</p><p><a href="https://www.technologyreview.com/s/608986/forget-killer-robotsbias-is-the-real-ai-danger/">Automation bias</a> is an over-reliance on automated aids and decision support systems. The keynote speaker <a href="https://twitter.com/mathbabedotorg">Cathy O’Neil</a> addressed this head-on with the question of whether algorithms perpetuate human bias, giving examples such as <a href="https://www.information-management.com/opinion/what-if-algorithms-did-the-hiring-at-fox-news">hiring algorithms </a>that look at historically good hires, which in organizations such as Fox News include mostly white men. Moreover, if those hires tend to become problematic, like in Fox News case of sexual harassment allegations from its founder and some of its TV hosts, <a href="https://www.ted.com/talks/cathy_o_neil_the_era_of_blind_faith_in_big_data_must_end">would AI propagate those attributes for new hires?</a></p><p>One of the speakers <a href="https://www.linkedin.com/in/stephanielkim/">Stephanie Kim </a>also touches on how facial recognition systems can be biased based on race, especially due to the lack of images of minorities in the data set. Look no further to Google’s problem of <a href="https://www.theverge.com/2018/1/12/16882408/google-racist-gorillas-photo-recognition-algorithm-ai">labeling an African-American as a gorilla.</a> Objectives of AI algorithms need to be properly analyzed to make sure that we are building tools that have a positive impact.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*5frthFzt-IMCsViyA8kO3Q.jpeg" /></figure><p>Governance of rogue AI will also be another key challenge. We need to have conversations of how to make sure rogue AI does not go into the wild — for instance, what happens if a testbot figures out its best method of ‘testing resiliency’ in a software is to take down an entire data center? These conversations are starting to prop up and should continue to be a concern as ethics and compliance becomes a major concern for AI.</p><p>To learn more about this, my team and I have done a <a href="https://jonathankurniawan.com/portfolio/wef/">consulting presentation for World Economic Forum</a> as part of our MBA project on automation bias in the justice system in the United States. The lesson is not to use AI as a silver bullet to combat systemic issues and hope for an ‘objective’ solution because a computer said so, but to combat the systemic issues with dialog of different stakeholders and ensure that AI will not aggravate the issue.</p><h3>(5) There is an increasing number of open source data-science projects focusing on making the world a better place.</h3><p>There are reasons to be optimistic though, as many of the speakers in the community were sharing various projects that are currently focusing on positive impact. Speakers like <a href="https://twitter.com/rawkintrevo">Trevor Grant</a>, open source AI evangelist at IBM, and <a href="https://www.linkedin.com/in/ericschles/">Eric Scheles</a>, researcher who combats trafficking with machine learning, all illustrated that there are worthwhile projects that are currently available where we all can contribute to and make the world a better place.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e7oqzy3vNKIiPxcsxLsPgA.jpeg" /><figcaption>Some of the data scientists using data science for good.</figcaption></figure><p>Overall, ODSC was a great event filled with insightful and thought-provoking speakers from all over the data field. To know more about their upcoming events and conferences, visit <a href="http://odsc.com/">odsc.com</a>.</p><blockquote>Hult is an international business school focused on the global mindset, with campuses all over the world, from San Francisco, Boston, New York, London, Dubai, and Shanghai.</blockquote><blockquote>Hult offers a number of post-graduate business degrees focused on the changing landscape of the job market, which include a one-year MBA, Masters of Business Analytics and Masters of Disruptive Innovation. To learn more, visit <a href="http://www.hult.edu/en/masters-degree/">http://www.hult.edu/en/masters-degree/</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca31986824f8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hult-data-analytics-club/our-key-takeaways-from-odsc-east-2018-ca31986824f8">Key Takeaways from ODSC East 2018</a> was originally published in <a href="https://medium.com/hult-data-analytics-club">Data Analytics @ Hult</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why SQL is a must for Data Scientists?]]></title>
            <link>https://medium.com/hult-data-analytics-club/why-sql-is-a-must-for-data-scientists-e7f1d49bdaf9?source=rss----c16eecc1ca30---4</link>
            <guid isPermaLink="false">https://medium.com/p/e7f1d49bdaf9</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-scientist]]></category>
            <dc:creator><![CDATA[Rafael Oliveira]]></dc:creator>
            <pubDate>Sat, 28 Apr 2018 17:07:23 GMT</pubDate>
            <atom:updated>2018-04-28T17:07:22.979Z</atom:updated>
            <content:encoded><![CDATA[<p>Data scientists are working hard on projects to predict problems and finding solutions, and the topic is hot right now. The whole analysis process passes through a big mountain of data (public or not), all this information is stored in databases and that makes SQL a must for Data Scientists.</p><p><strong>So, what is SQL?</strong></p><p>SQL (Structured Query Language) is a standard programming language to access relational databases such as Oracle, SQL Server, IBM DB2, MySQL, Sybase, etc. SQL works with something called structured data, which means all the fields are organized in a fixed representation. Although new databases like Cassandra and MongoDB for unstructured data (freeform data with no consistent structure like Wikipedia posts, Tweets, etc.) are getting their use, the relational databases still represent a big and important portion of the total.</p><p><strong>Advantages of using SQL</strong></p><p><strong>Empower your data usage:</strong> SQL allows users to create and manipulate the data, connect more than one database and cross information between them.</p><p><strong>Combine with other programming languages: </strong>You can use SQL commands inside your code, almost all the programming languages have libraries to allow users to use SQL commands and they normally are straightforward. Additionally, you can also use it with XML, JSON and other scripting languages too.</p><p><strong>Easy to learn:</strong> SQL is probably one of the simplest programming languages you will learn, it is easy, and the commands are very intuitive and simple to remember. Furthermore, the core of the language does not change from one platform to another.</p><p><strong>Prepares you for NoSQL:</strong> It is not mandatory to learn SQL before but is easier. Once you have a good understand about SQL, will be clear the differences between it and NoSQL, and will be possible for you to map the pros and cons of each one.</p><p><strong>I want to learn, what should I do?</strong></p><p>As mentioned before SQL is not complex, and you can probably learn by yourself using a free online course. I suggest you check this SQL Tutorial created by W3 Schools.</p><p><strong>Common mistakes about SQL</strong></p><p>If you are a student or a data science enthusiast, you probably used some datasets in excel before, that is a first big mistake. People focus just on learning coding skills like Python or R and forget that in real-life the major part of the data will probably not be accessible using Excel.</p><p>The second mistake you should avoid is thinking that having SQL knowledge means you are a database administrator. The database administrator is a different role, with other complex activities that go far and beyond the SQL knowledge.</p><p>That being said, don’t forget to add SQL to your skill! It is a must for Data Scientist.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7f1d49bdaf9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hult-data-analytics-club/why-sql-is-a-must-for-data-scientists-e7f1d49bdaf9">Why SQL is a must for Data Scientists?</a> was originally published in <a href="https://medium.com/hult-data-analytics-club">Data Analytics @ Hult</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Fallacies to Avoid]]></title>
            <link>https://medium.com/hult-data-analytics-club/data-fallacies-to-avoid-31268faa6480?source=rss----c16eecc1ca30---4</link>
            <guid isPermaLink="false">https://medium.com/p/31268faa6480</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[cognitive-bias]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-fallacies]]></category>
            <dc:creator><![CDATA[Jonathan Kurniawan]]></dc:creator>
            <pubDate>Mon, 16 Apr 2018 01:11:45 GMT</pubDate>
            <atom:updated>2018-04-16T01:11:44.549Z</atom:updated>
            <content:encoded><![CDATA[<p>As powerful as data is in providing insights and inform decisions, every data leader be aware of some ways where data can be misunderstood due to either poor statistics training, or cognitive biases that we already have.</p><p>Here are a couple of examples:</p><h3>Cognitive biases</h3><p><strong>McNamara Fallacy: </strong>Relying solely on metrics in complex situations and losing sight of the bigger picture. A.k.a <em>Quantitative fallacy.</em></p><p><strong>Danger of summary metrics:</strong> Only looking at summary metrics and missing big differences in raw data.</p><p>Example of both McNamara and summary metrics: if a business, through its promotions, have increased their number of email subscribers. On the surface, it looks like a great sign, but not necessarily. What if the number of people of who never open their email increased? Without having the email read, the increase subscriber count would be useless, and measuring the metric blindly could be misleading.</p><p><strong>Publication Bias: </strong>Interesting research findings are more likely to be published, distorting our impression of reality.</p><p>Example: News reports only high-profile crimes, and so perception of safety is skewed to feel more insecure, even though there are many more instances of peaceful events happening in the city.</p><p><strong>Hawthorne Effect: </strong>The act of monitoring someone can affect their behavior, leading to spurious findings. A.k.a <em>Observer effect.</em></p><p>Example: Supervising manufacturing workers during a QA trip influences their quality positively.</p><p><strong>Cobra Effect: </strong>Setting incentive that accidentally produces the opposite result than intended. A.k.a <em>Perverse incentive.</em></p><p>Example: The even-odd carplate scheme in Jakarta, Indonesia was intending to reduce traffic at main streets during work days, but the unintended consequence was Jakartans buying two cars with both even and odd plates, aggravating the traffic issue in the city.</p><p><strong>Survivorship Bias:</strong> Drawing conclusions from incomplete set of data, because that data has ‘survived’ some selection criteria.</p><p>Example: Lets say you just started joining a gym. Every time you are there, you see the same fit and motivated faces. After a few days, you feel depressed because you couldn’t stick to the schedule and motivation, whilst the others in the gym could. You begin to feel like you’re less than the average gym goer, however, you fail to realize that many more of those who fail to come to the gym, and are biased to those who ‘survive’ and come to the gym.</p><h3>Statistics fallacies</h3><p><strong>Cherry Picking:</strong> Selecting results that fit your claim, and excluding those that don’t.</p><p><strong>Data Dredging: </strong>Process of ‘fishing patterns’ in data that can be presented as statistically significant, without first devising a specific hypothesis as to the underlying causality. A.k.a <em>p-hacking.</em></p><p>A really good example of data dredging: <a href="https://fivethirtyeight.com/features/science-isnt-broken/#part1">https://fivethirtyeight.com/features/science-isnt-broken/#part1</a></p><p><strong>False Causality: </strong>Falsely assuming when two events appear related that one must have caused the other.</p><p>Example: “Every day, I eat cereal for breakfast. One time, I had a muffin instead, and there was a major earthquake in my city. I’ve eaten cereal ever since.”</p><p><strong>Gerrymandering:</strong> Manipulating geographical boundaries used to group data in order to change the result. Here’s youtuber CGP Grey explaining it further: <a href="https://www.youtube.com/watch?v=Mky11UJb9AY">https://www.youtube.com/watch?v=Mky11UJb9AY</a></p><p><strong>Sampling Bias: </strong>Drawing conclusions from a set of data that isn’t representative of the population you’re trying to understand.</p><p>Example: Interviewing students in an international school on what their preference of messaging platform, and taking that insight to say ‘students in the U.S. use WeChat more’. This mistakes the fact that the initial sample was not representing the population.</p><p><strong>Gambler’s Fallacy: </strong>Mistakenly believing that because something has happened more frequently than usual, it’s now less likely to happen in future (and vice versa).</p><p>Example: The past eight coin toss has been tails, so its ‘due’ to be heads now. In actuality, probability of the next coin toss to be heads is still 50–50, regardless of previous coin toss.</p><p><strong>Regression Fallacy: </strong>When something happens that’s usually good or bad, over time, it will revert back towards the average.</p><p>Example: Your favorite sports player, with no new trainings or strategies, suddenly win more games than expected. This could be due to ‘luck’, and after a while, will go back to his/her average capability.</p><p><strong>Simpson’s Paradox: </strong>When a trend appears in different subsets of data but disappears or reverses when groups are combined. Here’s a great example of Simpson’s Paradox in the U.S. median salary: <a href="http://blog.revolutionanalytics.com/2013/07/a-great-example-of-simpsons-paradox.html">http://blog.revolutionanalytics.com/2013/07/a-great-example-of-simpsons-paradox.html</a></p><p><strong>Overfitting: </strong>Creating a model that is overly tailored to the data you already have, and not representative of the general trend.</p><p>Example: This is a common problem in data models and machine learning, especially when a model gives a 100% accurate response. This means the machine learning model has truly optimized for the training data set, and not the real world.</p><p>More importantly, it’s important to know that the model’s “concepts” (relationship of input and output) can change over time, and needs to be accounted for (also known as concept drift) <a href="https://machinelearningmastery.com/gentle-introduction-concept-drift-machine-learning/">https://machinelearningmastery.com/gentle-introduction-concept-drift-machine-learning/</a></p><p>So the next time you’re making a data-driven decision, have some of these in mind and mitigate the risk of misleading fallacies.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=31268faa6480" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hult-data-analytics-club/data-fallacies-to-avoid-31268faa6480">Data Fallacies to Avoid</a> was originally published in <a href="https://medium.com/hult-data-analytics-club">Data Analytics @ Hult</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>