<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Divya Naidu on Medium]]></title>
        <description><![CDATA[Stories by Divya Naidu on Medium]]></description>
        <link>https://medium.com/@divyanaidu?source=rss-b5506f236013------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*c38TfjHSV9y7qkMcEZrHZQ.jpeg</url>
            <title>Stories by Divya Naidu on Medium</title>
            <link>https://medium.com/@divyanaidu?source=rss-b5506f236013------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 07:25:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@divyanaidu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Gather Your Data: The “Not-So-Spooky” APIs!]]></title>
            <link>https://medium.com/data-science/gather-your-data-the-not-so-spooky-apis-da0da1a5992c?source=rss-b5506f236013------2</link>
            <guid isPermaLink="false">https://medium.com/p/da0da1a5992c</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[web-analytics]]></category>
            <category><![CDATA[python-programming]]></category>
            <dc:creator><![CDATA[Divya Naidu]]></dc:creator>
            <pubDate>Mon, 31 Aug 2020 02:32:07 GMT</pubDate>
            <atom:updated>2020-08-31T02:32:07.468Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TY6qrpKZwiXpXrR8" /><figcaption>Photo by <a href="https://unsplash.com/@ffstop?utm_source=medium&amp;utm_medium=referral">Fotis Fotopoulos</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>When python plays with the internet files.</h4><p>A data analytics cycle starts with gathering and extraction. I hope <a href="https://towardsdatascience.com/gather-your-data-the-common-ones-6086ed04b905?source=friends_link&amp;sk=de3cac3fa648dd226aea85abe759d67e">my previous blog</a> gave an idea about how data from common file formats are gathered using python. In this blog, I’ll focus on extracting the data from files that are not so common but has the most real-world applications.</p><p>Whether you are a data professional or not, you might have come across the term API by now. Most people have a rather ambiguous or incorrect idea about this fairly common term.</p><h4>Extracting data using API in python</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EpYhDT_Yk0A3fnmh" /><figcaption>Photo by <a href="https://unsplash.com/@yogasdesign?utm_source=medium&amp;utm_medium=referral">Yogas Design</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote><strong>API </strong>is the acronym for<strong> Application Programming Interface</strong>, which is a software intermediary (A Middleman) that allows two applications to talk to each other.</blockquote><p>Each time you use an app like Tinder, send a WhatsApp message, check the weather on your phone, you’re using an API. They allow us to share important data and expose practical business functionality between devices, applications, and individuals. And although we may not notice them, APIs are everywhere, powering our lives from behind the scenes.</p><p>We can illustrate API as a bank’s ATM(Automated Teller Machine). Banks make their ATMs accessible for us to check our balance, make deposits, or withdraw cash. So here ATM is the middleman who is helping the bank as well as us, the customers.</p><p>Similarly, Web applications use APIs to connect user-facing front ends with all-important back end functionality and data. Streaming services like Spotify and Netflix use APIs to distribute content. Automotive companies like Tesla send software updates via APIs. For further instances, you can check out the article <a href="https://nordicapis.com/5-examples-of-apis-we-use-in-our-everyday-lives/">5<strong> </strong>Examples of APIs We Use in Our Everyday Lives</a>.</p><p>In Data Analysis, API’s are most commonly used to retrieve data, and that will be the focus of this blog.</p><p>When we want to retrieve data from an API, we need to make a request. Requests are used all over the web. For instance, when you visited this blog post, your web browser made a requested to the Towards Data Science web server, which responded with the content of this web page.</p><p><a href="https://monashdatafluency.github.io/python-web-scraping/section-3-API-based-scraping/">API requests</a> work in the same way — you request to an API server for data, and it responds to your request. There are primarily two ways to use APIs :</p><ol><li>Through the command terminal using URL endpoints, or</li><li>Through programming language-specific wrappers.</li></ol><p>For example, <strong>Tweepy</strong> is a famous python wrapper for Twitter API whereas <strong>twurl</strong> is a command line interface (CLI) tool but both can achieve the same outcomes.</p><p>Here we focus on the latter approach and will use a Python library (a wrapper) called <strong>wptools</strong> based around the original <strong>MediaWiki API</strong>. The <a href="https://www.mediawiki.org/wiki/API:Main_page">MediaWiki</a> action API is <a href="https://www.simplifiedpython.net/wikipedia-api-python/">Wikipedia’s API</a> that allows access to some wiki-features like authentication, page operations, and search.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BJHTGgfO3st_GKeC" /><figcaption>Photo by <a href="https://unsplash.com/@lukechesser?utm_source=medium&amp;utm_medium=referral">Luke Chesser</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><a href="https://pypi.org/project/wptools/">Wptools</a> make read-only access to the MediaWiki APIs. You can get info about wiki sites, categories, and pages from any Wikimedia project in any language via the good old Mediawiki API. You can extract unstructured data from page Infoboxes or get structured, linked open data about a page via the Wikidata API, and get page contents from the high-performance RESTBase API.</p><p>In the below code, I have used python’s wptools library to access the <strong>Mahatma Gandhi Wikipedia page</strong> and extract an image file from that page. For a Wikipedia URL ‘https://en.wikipedia.org/wiki/Mahatma_Gandhi’we only need to pass the last bit of the URL.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/741b19baf9d245a845fff73380b6f420/href">https://medium.com/media/741b19baf9d245a845fff73380b6f420/href</a></iframe><p>The get function fetches everything including extracts, images, infobox data, wiki data, etc. present on that page. By using the .data() function we can extract all the required information. The response that we get for our request to the API is most likely to be in JSON format.</p><h4>Reading data from JSON in python</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FeCzYzraG8CdYCge" /><figcaption>Photo by <a href="https://unsplash.com/@cgower?utm_source=medium&amp;utm_medium=referral">Christopher Gower</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote><strong>JSON</strong> is an acronym for <strong>JavaScript Object Notation</strong>. It is a lightweight data-interchange format. It is as easy for humans to read &amp; write as for the machines to parse &amp; generate. <a href="https://realpython.com/python-json/">JSON has quickly become the de-facto standard for information exchange</a>.</blockquote><p>When exchanging data between a browser and a server, the data can only be text. JSON is the text, and we can convert any JavaScript object into JSON, and send JSON to the server.</p><p>For example, you can access <a href="https://www.freecodecamp.org/news/what-is-an-api-in-english-please-b880a3214a82/">GitHub’s API</a> directly with your browser without even needing an access token. Here’s the JSON response you get when you visit a GitHub user’s API route in your browser<a href="https://api.github.com/users/divyanitin">https://api.github.com/users/divyanitin</a> :</p><pre>{<br>  &quot;login&quot;: &quot;divyanitin&quot;,<br>  &quot;url&quot;: &quot;https://api.github.com/users/divyanitin&quot;,<br>  &quot;html_url&quot;: &quot;https://github.com/divyanitin&quot;,<br>  &quot;gists_url&quot;: &quot;https://api.github.com/users/divyanitin/gists{/gist_id}&quot;,<br>  &quot;type&quot;: &quot;User&quot;,<br>  &quot;name&quot;: &quot;DivyaNitin&quot;,<br>  &quot;location&quot;: &quot;United States&quot;,<br>}</pre><p>The browser seems to have done just fine displaying a JSON response. A JSON response like this is ready for use in your code. It’s easy to extract data from this text. Then you can do whatever you want with the data.</p><p>Python supports JSON natively. It comes with a json built-in package for encoding and decoding JSON data. JSON files store data within {} similar to how a dictionary stores it in Python. Similarly, JSON arrays are translated as python lists.</p><p>In my above code, wiki_page.data[‘image’][0] access the first image in the image attribute i.e a JSON array. With Python json module you can read JSON files just like simple text files.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/42a23ab91af96692a13df870346b1b7b/href">https://medium.com/media/42a23ab91af96692a13df870346b1b7b/href</a></iframe><p>The read function json.load() returns a JSON dictionary which can be easily converted into a Pandas dataframe using the pandas.DataFrame() function. You can even load the JSON file directly into a dataframe using the pandas.read_json() function.</p><h4>Reading files from the Internet (HTTPS)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yI879REGAoP1Pc2i" /><figcaption>Photo by <a href="https://unsplash.com/@edhoradic?utm_source=medium&amp;utm_medium=referral">Edho Pratama</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote><strong>HTTPS</strong> stands for <strong>HyperText Transfer Protocol Secure</strong>. It is a language that web browsers &amp; web servers speak to each other. A web browser may be the client, and an application on a computer that hosts a web site may be the server.</blockquote><p>We are writing a code that works with remote APIs. Your maps app fetches the locations of nearby Indian restaurants or the OneDrive app starts up cloud storage. All this happens just by making an HTTPS request.</p><p>‘<a href="https://requests.readthedocs.io/en/master/">Requests</a>’ is a versatile HTTPS library in python with various applications. It works as a request-response protocol between a client and a server. It provides methods for accessing Web resources via HTTPS. One of its applications is to download or open a file from the web using the file URL.</p><p>To make a ‘GET’ request, we’ll use the requests.get() function, which requires one argument — the URL we want to request to.</p><p>In the below script, the open method is used to write binary data to the local file. In this, we are creating a folder and saving the extracted web data on the system using the <a href="https://docs.python.org/3/library/os.html">os library</a> of python.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/392df8d08989a56490305ca6f1852342/href">https://medium.com/media/392df8d08989a56490305ca6f1852342/href</a></iframe><p>The json and requests import statements load Python code that allows us to work with the JSON data format and the HTTPS protocol. We’re using these libraries because we’re not interested in the details of how to send HTTPS requests or how to parse and create valid JSON, we just want to use them to accomplish these tasks.</p><p>A popular web architecture style called <a href="https://realpython.com/api-integration-in-python/#appendix-rest-in-a-nutshell"><strong>REST (Representational State Transfer)</strong></a> allows users to interact with web services via <strong>GET</strong> and <strong>POST</strong> calls (two most commonly used).</p><p><strong>GET</strong> is generally used to get information about some object or record that already exists. In contrast, <strong>POST</strong> is typically used when you want to create something.</p><p>REST is essentially a set of useful conventions for structuring a web API. By “web API,” I mean an API that you interact with over HTTP, making requests to specific URLs, and often getting relevant data back in the response.</p><p>For example, Twitter’s REST API allows developers to access core Twitter data and the Search API provides methods for developers to interact with Twitter Search and trends data.</p><p>This blog focuses on the so-called internet files. I introduced the extraction of data using simple APIs. Most of the APIs require authentication, just like my earlier analogy ATM requires us to enter the pin for authenticating our access to the bank.</p><p>You can check out my other article <a href="https://towardsdatascience.com/twitter-analytics-weratedogs-a441be7d4a85?source=friends_link&amp;sk=b1fa174b4808c69f30c299b3e83471fe">Twitter Analytics: “WeRateDogs”</a> which was focused on data wrangling and analysis of twitter API. I have used all the above-mentioned scripts in this project, you can find the code for the same on <a href="https://github.com/divyanitin">my GitHub</a>.</p><p>As known to all, one of the most common internet files is HTML. Extracting data from the internet has the common term Web Scraping. In this, we access the website data directly using the HTML. I would be covering the same, from the “basic” to the “must-knows” in my next blog.</p><p>If you enjoyed this blog post, leave a comment below, and share it with a friend!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da0da1a5992c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/gather-your-data-the-not-so-spooky-apis-da0da1a5992c">Gather Your Data: The “Not-So-Spooky” APIs!</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Gather Your Data: The Common ones!]]></title>
            <link>https://medium.com/data-science/gather-your-data-the-common-ones-6086ed04b905?source=rss-b5506f236013------2</link>
            <guid isPermaLink="false">https://medium.com/p/6086ed04b905</guid>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[data-wrangling]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[data-gathering]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Divya Naidu]]></dc:creator>
            <pubDate>Mon, 27 Jul 2020 23:12:16 GMT</pubDate>
            <atom:updated>2020-07-27T23:12:32.307Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MaDJWGbIfVfrGquK" /><figcaption>Photo by <a href="https://unsplash.com/@viktortalashuk?utm_source=medium&amp;utm_medium=referral">Viktor Talashuk</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>From zip, csv, tsv, xlsx, txt using Python</h4><p>Do you know which is the most popular data scientist meme? <strong><em>“Collecting Data is my Cardio”.</em></strong> Yes, that’s so true! Unlike what the beginner’s in the field of Data Science think, Data is seldom served to you in a clean &amp; tidy tabular format.</p><p>In real world, the data needs to be gathered from multiple sources like flat files, zip files, databases, API’s, websites, etc. Not just sources but they could be in different formats, file structures such as .csv, .tsv, json etc and separated by different delimiters. How to make sense from this mess?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kcrNfnu8ugspquJulIPcdA.png" /><figcaption>Image by Author</figcaption></figure><p>To perform analysis and get accurate results from our data, you should first, be aware of the different formats and second, know how to import them into Python. In this blog, I am going to touch upon a few basic file structures that you might be familiar with and mention different yet common python libraries that are used for importing data.</p><p>Mastering these file formats is critical for your success in the field of data. So where does everyone begin with? Oh yes, the omnipresent csv file.</p><h4>Reading CSV files in python</h4><p>The most common flat file structure for storing data is the Comma Separated Values (CSV) files. It contains tabular data in plain text format separated by comma (,) delimiter. To identify a file format, you can usually look at the file extension. For example, a file saved with name “datafile” in “CSV” format will appear as “datafile.csv”.</p><p>Pandas library is used for reading the data from a csv file in python. The <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html">read_csv()</a> function from the Pandas library is used to load the data and store it in a dataframe.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b0c5bc26b60c8a6f4e42d43ab0fcd1bf/href">https://medium.com/media/b0c5bc26b60c8a6f4e42d43ab0fcd1bf/href</a></iframe><h4>Reading other Flat files in python</h4><p>Flat files are data files that contains records in tabular row-column format without any numbered structure. CSV is the most common flat file. But there are other flat file formats which contains data with a user specified delimiter such as tabs, space, colons, semi-colons, etc.</p><p>The Tab Separated Values (TSV) files are the second most common flat file structure. The same read_csv() function from the Pandas library is used for reading these files, you just need to specify the right separator to the parameter ‘sep’ of the read_csv() function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/91adfb668365e89bf52f431096cb85eb/href">https://medium.com/media/91adfb668365e89bf52f431096cb85eb/href</a></iframe><h4>Reading XLSX files in python</h4><p>A file with the .xlsx file extension is a Microsoft Excel Open XML Spreadsheet (XLSX) file created by Microsoft Excel. These files are files used in Microsoft Excel, a spreadsheet application that uses tables to organize, analyze, and store data.</p><p>Each cell of the spreadsheet is positioned by rows and columns and can contain text or numerical data, including incorporating mathematical formulas. Spreadsheet files made in earlier versions of Excel are saved in the XLS format.</p><p>The <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html">read_excel()</a> function from the pandas library is used to load the excel file data into python.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b5db42ea2bd41e3a3202697b2e1236da/href">https://medium.com/media/b5db42ea2bd41e3a3202697b2e1236da/href</a></iframe><p>XLSX files organize data in cells that are stored in worksheets, which are in turn stored in workbooks (files that contain multiple worksheets). For reading all the sheets, we use pandas <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.ExcelFile.parse.html">ExcelFile()</a> function. Now we can read a particular sheet from the workbook by just specifying the name of that sheet in the parameter sheet_name of the read_excel() function.</p><h4>Extracting ZIP files in python</h4><p>A file with the ZIP file extension is a ZIP Compressed file and is the most widely used archive format you’ll run into.</p><p>A ZIP file, like other archive file formats, is simply a collection of one or more files and/or folders but is compressed into a single file for easy transportation and compression.</p><p>We often get our csv or excel files in zip format as it saves storage space on the server and decreases the time it takes for you to download it to your computer. The <a href="https://docs.python.org/3/library/zipfile.html">zipfile</a> module in python is used to extract the .zip files.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2fdf8f73ebbd379cfdb6cd7dbb31480b/href">https://medium.com/media/2fdf8f73ebbd379cfdb6cd7dbb31480b/href</a></iframe><h4>Reading Text files using python</h4><p>The basic conceptual structure of a plain text format is that the data are arranged in rows, with several values stored on each row. Plain text files (also known by the extension TXT) consist of characters encoded sequentially in some particular character encoding.</p><p>Python provides various functions to read the data from the text files. Each line of text is terminated with a special character called EOL (End of Line), which is the new line character (‘\n’) in python by default.</p><p><a href="https://www.programiz.com/python-programming/file-operation">Python file operations</a>, specifically opening a file, reading a file, closing a file and various other file methods are something you should be aware of for accessing the text files. The built-in Python function open() has arguments such as mode, encoding, etc. which are used to open the text files correctly in python.</p><p>Python provides us with three built-in methods for reading data from a text file, read([n]), readline([n]) and readlines(). Here ’n’ is the number of bytes to be read. If nothing is passed to n, then the complete file is considered to be read.</p><p>The <strong>read()</strong> method just outputs the entire file if the number of bytes (n) is not given in the argument.</p><p>The <strong>readline(n)</strong> outputs at most n bytes of a single line of a file. It does not read more than one line.</p><p>The <strong>readlines()</strong> method maintains a list of each line in the file which can be iterated using a for loop.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/729bbac82513c1c07bf742f094a1fcb2/href">https://medium.com/media/729bbac82513c1c07bf742f094a1fcb2/href</a></iframe><p>The above gives a basic idea about reading the text file in python. But again, data is never placed in front of us in an easy to access format. Suppose you want to find the 10 most popular restaurants as per the reviews near you.</p><p>Assuming there are 1000 restaurants and the data for this is present in 1000 individual text files and you need the restaurant name, review url and review text present in these files. To read the data from multiple files into one dataframe, Python’s glob library is used.</p><p>The Python built-in <a href="https://docs.python.org/2/library/glob.html">glob module</a> is used to retrieve files/pathnames matching a specified pattern. The pattern rules of glob follow standard Unix path expansion rules.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c2c36f4aa895be4f4c1250974fce29ff/href">https://medium.com/media/c2c36f4aa895be4f4c1250974fce29ff/href</a></iframe><p>The /* is the wildcard in the glob statement. An asterisk (*) matches zero or more characters in a segment of a name. You need to always specify the encoding in the open() function for reading the file correctly.</p><p>Here, the text files had restaurant name on the first line, restaurants review url on the second line and rest of the lines had review text. The readline() and read() functions were used accordingly to extract these data.</p><p>In this blog, I have covered the most common and popular file structures that are used by people in the data industry. This should prepare you for your next frustrations when dealing with data gathering and messy file structures. There are many other that I intend to cover in my forthcoming blogs.</p><p>If you enjoyed this blog post, leave a comment below and share it with a friend!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6086ed04b905" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/gather-your-data-the-common-ones-6086ed04b905">Gather Your Data: The Common ones!</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A/B Testing: The Case Study!]]></title>
            <link>https://medium.com/data-science/a-b-testing-the-case-study-1030a94f3c4b?source=rss-b5506f236013------2</link>
            <guid isPermaLink="false">https://medium.com/p/1030a94f3c4b</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistical-analysis]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[marketing]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Divya Naidu]]></dc:creator>
            <pubDate>Sat, 18 Jul 2020 20:53:33 GMT</pubDate>
            <atom:updated>2020-07-18T20:53:33.479Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XGcTq8M5cAQANdvcvWP3xA.jpeg" /><figcaption>Image by Author</figcaption></figure><p>Analyzing the test results using Python.</p><p>In theory, there is no difference between theory and practice, but<br>practice, there is.</p><p>My previous blog gives a basic idea of what exactly is <a href="https://towardsdatascience.com/a-b-testing-the-basics-86d6d98525c9?source=friends_link&amp;sk=86434b44e90841eb1a30e7e7cc2760eb">A/B testing</a>. From the positioning of images on pages, to the checkout process, we are staunch advocates of A/B Testing. Knowledge of a concept is not enough, implementation of this knowledge gives the best exposure of learning.</p><p>This blog is a walkthrough of an A/B testing case study that I have worked on as part of <a href="https://medium.com/u/2929690a28fb">Udacity</a>’s Data Analyst NanoDegree Program. This case study is developed and used by Udacity in many of their programs.</p><p>“<strong>Audacity</strong>” is an online learning platform that offers a wide variety of courses, degrees, and certifications to the students. So, the customer funnel for a typical new user on the Audacity website would be as follows:</p><p><strong>Homepage -&gt; Explore Courses -&gt; View Course Overview page -&gt; Enroll in course -&gt; Complete the Course</strong></p><p>Just like any other website, there would be greater number of users at the top of the funnel than the bottom. Audacity loses users as they go down the stages of this funnel. So, it decided to test features that can increase student engagement. Audacity is performing A/B test to try out changes that will hopefully increase the conversion rate from one stage to the next.</p><p>The <strong>first change</strong> that they decided to bring in was the homepage design. Audacity decided to optimize its homepage so that the new, more engaging design will increase the number of users to explore their courses.</p><p>This basically covers two stages of the funnel, <em>Homepage -&gt; Explore Courses</em></p><p>An experiment was setup to see how many users went to next stage of explore courses in the funnel in both the control and treatment group. The metric <strong>Click through rate(CTR)</strong> for the “<em>Explore Courses</em>” button on the Homepage was then chosen to measure the level of engagement from the users in each group.</p><p>The null hypothesis here is the new CTR is less than or equal to the old CTR.<br><strong><em>H0: CTR_new — CTR_old &lt;= 0</em></strong></p><p>The alternate hypothesis is what we are trying to prove, the new CTR is greater than the old CTR.<br><strong><em>H1: CTR_new — CTR_old &gt; 0</em></strong></p><p>The data was collected for a period of almost 4 months with a control group size of 3332, and a treatment group size of 2996. The homepage actions, the visitors who <em>viewed or clicked </em>the button, were recorded.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/1*ZskNNBDTFzPm-cNuWMLYTw.png" /><figcaption>Sample Data</figcaption></figure><p>With this data we calculated the CTR for the control group and the treatment group.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/73742341154f7e31a195011fda748a88/href">https://medium.com/media/73742341154f7e31a195011fda748a88/href</a></iframe><p>The CTR for the control group was <strong><em>0.2797</em></strong> and for the treatment group was <strong><em>0.3097</em></strong>. An observed difference was then computed between the CTR’s of the both the groups and was found to be <strong><em>0.030</em></strong>.</p><p>We then simulated the sampling distribution using bootstrapping for the difference in CTR between the two pages over <em>10,000 iterations</em>.</p><blockquote>In statistics, <a href="https://www.analyticsvidhya.com/blog/2020/02/what-is-bootstrap-sampling-in-statistics-and-machine-learning/"><strong>Bootstrap Sampling</strong></a> is a method that involves drawing of sample data repeatedly with replacement from a data source to estimate a population parameter.</blockquote><blockquote>A <a href="https://www.investopedia.com/terms/s/sampling-distribution.asp"><strong>Sampling Distribution</strong></a> shows every possible result a statistic can take in every possible sample from a population and how often each result happens.</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d155c940d3cfbd17ba66e0cea1e51ed0/href">https://medium.com/media/d155c940d3cfbd17ba66e0cea1e51ed0/href</a></iframe><p>We used this sampling distribution to simulate the distribution under the null hypothesis, by creating a random normal distribution centered at 0 with the same spread and size.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ee1b8726e2156e4e38f946b1256fb7fc/href">https://medium.com/media/ee1b8726e2156e4e38f946b1256fb7fc/href</a></iframe><p>The red line on the sampling distribution under null hypothesis is the observed difference that we obtained earlier and it seems very small under the null.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Ze7Uj0M_iVzs1_RefSJEg.png" /><figcaption>Sampling Distribution under Bootstrapping | Sampling Distribution under Null Hypothesis</figcaption></figure><p>We then computed the p-value by finding the mean of values in the null distribution that were greater than our observed difference.</p><p>When your p-value is small, you end up moving with the alternate hypothesis. But how small? So, this would be decided on the basis of the threshold value of Type 1 error. The Type 1 error rate alpha for Business &amp; Research is <strong><em>0.05</em></strong>.</p><p>The p-value that we obtained was <strong><em>0.0061</em></strong>, which is less than the error rate alpha. So now we have evidence to reject the null hypothesis. Thus, the A/B testing resulted in the implementation of new Homepage design.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mNhlHvvWzU6vrGLv" /><figcaption>Photo by <a href="https://unsplash.com/@igormiske?utm_source=medium&amp;utm_medium=referral">Igor Miske</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Moving on to the <strong>second change</strong> that Audacity decided to try. They made the course description more career focused on the course overview page. They hope that this change may encourage more users to enroll in the course and finally complete it.</p><p>This basically covers three stage of the funnel, <em>View Course Overview page -&gt; Enroll in course -&gt; Complete the Course.</em></p><p>For this experiment, four metrics were chosen. They were defined as below:</p><ol><li><strong>Enrollment Rate:</strong> The Click through rate for the “<em>Enroll</em>” button was considered for getting this.</li><li><strong>Average Reading Duration:</strong> The average number of seconds spent on the course overview page by the user.</li><li><strong>Average Classroom Time:</strong> The average number of days spent in the classroom for the students that are enrolled in the course.</li><li><strong>Completion Rate:</strong> The completing rate for the students enrolled in the course.</li></ol><p>Each metric was evaluated individually following the same steps as mentioned in the first change. The observed difference was computed for the metrics, then the sampling distribution using bootstrapping was stimulated for the difference between the two pages over 10,000 iterations. Same was done under null hypothesis and the observed difference was then plotted on the obtained normal distribution.</p><p>The p-value was calculated, and we found that all the metrics are statistically significant individually. The more the metric you evaluate, the more likely you are to observe significant differences just by chance. So, we need to determine whether the difference observed in each metric is statistically significant collectively.</p><p>The Bonferroni correction is one of several methods used to counteract the problem of multiple comparisons.</p><blockquote>The <a href="https://www.statisticssolutions.com/bonferroni-correction/"><strong><em>Bonferroni correction</em></strong></a> is a multiple-comparison correction used when several dependent or independent statistical tests are being performed simultaneously.</blockquote><p>While a given Type 1 Error alpha value may be appropriate for each individual comparison, it is not for the set of all comparisons. In order to avoid a lot of false positives, the alpha value needs to be lowered to account for the number of comparisons being performed.</p><p>Using Bonferroni correction, we divided the original alpha value by the number of metrics(tests) and got a new value for the error rate alpha (<strong><em>0.05/4 = 0.0125</em></strong>).</p><p>The p-values of the four metrics that we computed were as below:<br>Enrollment rate- <strong>0.0188</strong><br>Average Reading Duration- <strong>0</strong><br>Average Classroom time- <strong>0.0384</strong><br>Completion Rate- <strong>0.0846</strong></p><p>The new alpha value being <strong><em>0.0125</em></strong>, the only significant metric to be considered is <strong>Average Reading Duration</strong>. The below plot shows the Sampling Distribution under Bootstrapping and Sampling Distribution under Null Hypothesis for the Average reading duration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JMgj1tHCCQ_XopE3s-j8QQ.png" /><figcaption>Sampling Distribution under Bootstrapping | Sampling Distribution under Null Hypothesis</figcaption></figure><p>The p-value should be less than or equal to the alpha for rejecting the null hypothesis. As we have the evidence for the same, we will go ahead with the change. Thus, the A/B testing resulted in the implementation of more career focused course description on the overview page.</p><p>The above A/B testing case study gives an idea of how the test data is analyzed using statistical methods to get the results. You can find the data that was collected from these experiments along with the full python codes on <a href="https://github.com/divyanitin/A-B-Testing-Case-Study">my GitHub</a>.</p><p>If you want improved user engagement, boost your conversion rates and increase your sales, you need to make data driven decisions. With <a href="https://towardsdatascience.com/a-b-testing-the-basics-86d6d98525c9?source=friends_link&amp;sk=86434b44e90841eb1a30e7e7cc2760eb">A/B testing</a> you get to see your results in black &amp; white thus you know what exactly is working for you.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1030a94f3c4b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-b-testing-the-case-study-1030a94f3c4b">A/B Testing: The Case Study!</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A/B Testing: The Basics!]]></title>
            <link>https://medium.com/data-science/a-b-testing-the-basics-86d6d98525c9?source=rss-b5506f236013------2</link>
            <guid isPermaLink="false">https://medium.com/p/86d6d98525c9</guid>
            <category><![CDATA[digital-analytics]]></category>
            <category><![CDATA[website-analytics]]></category>
            <category><![CDATA[marketing]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <dc:creator><![CDATA[Divya Naidu]]></dc:creator>
            <pubDate>Sun, 05 Jul 2020 20:36:30 GMT</pubDate>
            <atom:updated>2020-07-05T20:36:30.914Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nnv4VdmdXxDRyl84" /><figcaption>Photo by <a href="https://unsplash.com/@euwars?utm_source=medium&amp;utm_medium=referral">Farzad Nazifi</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>What, Why, When and How</h4><p>Which color, pink or blue, leads more people to click on the sign-in button of a website<strong>?</strong> Should they have a Gif instead of an image on the Homepage<strong>?</strong> How many menu options, five or seven, will attract the customer<strong>?</strong> Which version works better in leading the customer to the end of the funnel<strong>?</strong></p><p>You can take a wild guess and go ahead with the change Or <strong>A/B Test!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*PB0Fc38bdoW51quVE54mEA.png" /><figcaption>Image by Author</figcaption></figure><p>An A/B test is used to determine which version of something will perform more effectively in the market.</p><p>From startups to large tech firms, companies of all sizes and industries rely on A/B testing to make smarter choices. Even the simplest of tests can help steer big decisions.</p><blockquote><strong>What is A/B Testing?</strong></blockquote><p>A/B testing is a process of showing two versions of the same web page to different segments of website visitors at the same time to determine which one performs well.</p><p>It’s basically confirming which does better — <strong>this version(A) or that version(B)</strong>?</p><p>A/B testing, also known as <strong>split testing</strong> or <strong>bucket testing</strong>, is essentially an experiment where two or more variants of <strong>an ad, marketing email, or web page </strong>are shown to users at random, and then different statistical analysis methods are used to determine which variant drives more conversions.</p><p>Typically in A/B testing, the variant that gives higher conversions is the winning one, and that variant can help you optimize your site for better results.</p><blockquote><strong>Why should you A/B test?</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-EBtEMTeY-ThVUaQ" /><figcaption>Photo by <a href="https://unsplash.com/@campaign_creators?utm_source=medium&amp;utm_medium=referral">Campaign Creators</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In <strong>Email campaigns</strong>, two variants of one campaign are sent to the users. By doing this, the marketing team will know which email was most effective in terms of encouraging opens or clicks.</p><p>But they won’t know what exactly led the user to open the mail. Was it preheader, the subject line, the visuals or the email content? This can be determined by the A/B test, which is the most compelling element of the email?</p><p>It can help you examine visitor and customer behavior on your site before committing to major decisions on the changes and help you increase your chances of success. In short, A/B testing helps you avoid unnecessary risks by allowing you to target your resources for maximum effect and efficiency.</p><blockquote><strong>When to use A/B Test?</strong></blockquote><p>An <strong>Online learning platform</strong> wants to make changes to their homepage, where the new, more engaging design will increase the number of users that explore their courses. Or maybe they want more career focused description on their course overview page, this may encourage more users to enroll and complete their course — A/B testing can confirm this for them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ppkygDcjf6qMGaYo" /><figcaption>Photo by <a href="https://unsplash.com/@halacious?utm_source=medium&amp;utm_medium=referral">Halacious</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Anything on a web page that can affect the behavior of a visitor while browsing on the site can be tested using A/B Testing.</p><p>Here is a list of variations that can be assessed by A/B Testing —<em> the headline, content, page design, images, Sales promotions and offers, Social media mentions, Site navigations and user interface, Call to action (CTA) text and button, Payment options, Delivery Options</em>, etc.</p><p>In A/B test, the comparisons should be kept as simple as possible. For example, don’t compare two completely different versions of your website, because you’ll have no idea what factors actually made a difference. Similarly, if a new module or menu is added to the website it cannot be tested by A/B testing as there is no point of comparison.</p><blockquote><strong>How to conduct an A/B test?</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_T1qET0ENv4fr2ee" /><figcaption>Photo by <a href="https://unsplash.com/@brookelark?utm_source=medium&amp;utm_medium=referral">Brooke Lark</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>An ideal customer funnel for an <em>e-commerce store</em> will be as follows:</p><p><strong>Homepage</strong> -&gt; <strong>Category listing/Search product</strong> -&gt; <strong>View Product Page</strong> -&gt; <strong>Shopping Cart</strong> -&gt;<strong> Checkout</strong></p><p>The store loses its users as they go down the stages of this funnel. An A/B test is then performed to try out changes that will hopefully increase the conversion rate from one stage to the next.</p><p>A/B testing is done in broadly<strong> four</strong> steps:</p><p><strong>Determine the change and the metric</strong></p><p>First, decide what types of information you’ll be able to collect and analyze. We don’t just guess but use proper analytics tools to be sure you have a problem in the first place and to find where exactly it is.</p><p>The analytics will often provide insight into where you can begin enhancing. For example, we decide to change the <strong>Call to action(CTA)</strong> button on the Product page from “<em>Buy Now</em>” to “<em>Shop Now</em>” to increase the number of users that add items to the Shopping cart.</p><p>A <strong>metric</strong> is then chosen to measure the level of engagement from the users. In our example the metric will be click through rate for the “<em>Buy Now</em>” button. <strong>Click through rate (CTR)</strong> is number of clicks by unique users divided by number of views by unique users. You can select as many metrics as you want, but more the metric you evaluate, more likely you are to observe significant differences.</p><p><strong>Define your hypothesis</strong></p><p><strong>Hypothesis testing</strong> in statistics is a way for you to test the results of an experiment to see if you have meaningful results. The most important and confusing aspects of Hypothesis testing is determining <strong>Null and Alternate hypothesis</strong>.</p><p>To put it simply, the statement that is true before we collect any data is the <strong><em>null hypothesis</em></strong>. So, in A/B testing the basic null hypothesis will be that the new version is no better, or even worse than the old version. For our example, the new Click through rate(CTR) is less than or equal to the old CTR.</p><p>The <strong><em>alternate</em></strong><em> </em>is the competing, non-overlapping hypothesis of null. The statement that we are trying to prove always appear in alternate. So, in A/B testing, the alternate hypothesis is that the new version is better than the old version. For our example, the new CTR is greater than the old CTR.</p><p><strong>Design the experiment (<em>sample size, impacting factors, run time, participants</em>)</strong></p><p>Typically, users are randomly selected and assigned to either a <strong>Control group</strong> or a <strong>Treatment group</strong>. We then run the experiment where the control group sees the old version and the treatment group sees the new version.</p><p>Each user sees just one design (A or B), even if they update the interface. This way, the same number of people will view each version and you can measure which version achieved the lift that you would consider meaningful.</p><p>The sample size that you decide on will determine how long you might have to wait until you have collected enough data.</p><p><strong>Analyze the A/B test data</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jO-xLe69VbYgrLTZ" /><figcaption>Photo by <a href="https://unsplash.com/@kmuza?utm_source=medium&amp;utm_medium=referral">Carlos Muza</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Once your experiment is complete, it’s time to analyze the results. This is where analysts should have more focus. We calculate the metric values of both the control and treatment group.</p><p>Different statistical techniques such as Sampling Distribution using bootstrapping, regression and various other machine learning algorithms are then applied to evaluate the metric and show you the difference between how the two versions of your page performed, and whether there is a statistically significant difference.</p><blockquote><strong>Difficulties in A/B testing</strong></blockquote><p>There are many factors to consider when designing an A/B test and drawing conclusions based on its results. Some common ones are:</p><ol><li>Novelty effect and change aversion when existing users first experience a change.</li><li>Sufficient traffic and conversions to have significant and repeatable results.</li><li>Consistency among test subjects in the control and treatment group.</li><li>Best metric choice for making the ultimate decision eg. measuring revenue vs. clicks.</li><li>Practical significance of a conversion rate, the cost of launching a new feature vs. the gain from the increase in conversion.</li><li>Long enough run time for the experiment to account for changes in behavior based on time of day/week or seasonal events.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JynKeLDV1DNejw5l" /><figcaption>Photo by <a href="https://unsplash.com/@nickmorrison?utm_source=medium&amp;utm_medium=referral">Nick Morrison</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I have written this blog as simple as possible so that people with no prior knowledge of A/B testing can also get a basic idea about it. There are various <a href="https://www.ventureharbour.com/best-a-b-testing-tools/">analytical tools</a> that are used for A/B testing in the industries such as Google Analytics and Google Optimize, Vwo, Optimizely, HubSpot’s A/B testing kit, Crazy Egg, etc.</p><p><a href="https://www.julian.com/guide/growth/ab-testing">A/B’s aren’t a luxury for marketers with spare time; they’re the lifeblood of growth hacking.</a></p><p>One of the most popular examples of A/B testing is<strong> </strong><a href="http://www.zeldman.com/2009/03/20/41-shades-of-blue/"><strong>41 Shades of blue</strong></a>, when Google couldn’t decide which of two blue colors they preferred for a certain design element, they reportedly used A/B testing to assess the performance of 41 different shades of blue instead. Also, check out <a href="https://blog.hubspot.com/marketing/a-b-testing-experiments-examples"><strong>7 Incredible Examples of A/B Tests by Real Businesses</strong></a> for industry level A/B testing examples.</p><p>A precise A/B Testing approach can lead to huge benefits — improved user engagement, increased conversion rate, ease of analysis, and increased sales — It’s a <strong>Win-Win!!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86d6d98525c9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-b-testing-the-basics-86d6d98525c9">A/B Testing: The Basics!</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Twitter Analytics: “WeRateDogs”]]></title>
            <link>https://medium.com/data-science/twitter-analytics-weratedogs-a441be7d4a85?source=rss-b5506f236013------2</link>
            <guid isPermaLink="false">https://medium.com/p/a441be7d4a85</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[twitter-data-scraper]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[udacity]]></category>
            <category><![CDATA[data-wrangling]]></category>
            <dc:creator><![CDATA[Divya Naidu]]></dc:creator>
            <pubDate>Sun, 21 Jun 2020 22:17:16 GMT</pubDate>
            <atom:updated>2020-06-21T22:17:16.455Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Data Wrangling and Analysis Blog</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*KinIUBsckpMYvuBganlgsw.png" /><figcaption>Source: WeRateDogs Twitter</figcaption></figure><blockquote>A good data wrangler knows how to integrate information from multiple data sources, solve common transformation problems, and resolve data cleansing and quality issues.</blockquote><p><em>Data Analyst Nanodegree Program</em> from <a href="https://medium.com/u/2929690a28fb">Udacity</a> introduced me to various data wrangling methods that are being used in the industry today. For one of its wrangling projects, I was given an opportunity to go through the whole data analysis process — collecting the data, cleaning the data, analyzing the data and finally visualizing the data.</p><p>The dataset is the tweet archive of Twitter user <a href="http://twitter.com/dog_rates">@dog_rates</a>, also known as WeRateDogs. <em>WeRateDogs is a Twitter account that rates people’s dogs with humorous comments.</em></p><h3></h3><p></p><p>Most of the focus of this project was on data wrangling. So what exactly is data wrangling?</p><blockquote>Data wrangling refers to the process of cleaning, restructuring and enriching the raw data available into a more usable format.</blockquote><p>I have used various python libraries in this project, below are the ones I got started with.</p><pre><strong>import</strong> <strong>numpy</strong> <strong>as</strong> <strong>np</strong><br><strong>import</strong> <strong>pandas</strong> <strong>as</strong> <strong>pd</strong><br><strong>import</strong> <strong>matplotlib.pyplot</strong> <strong>as</strong> <strong>plt</strong><br>%<strong>matplotlib</strong> inline</pre><h3>Gathering the Data:</h3><p>The data for this project was in <em>three</em> different formats:</p><p><strong>1.Twitter Archive File — WeRateDogs:</strong> WeRateDogs downloaded their Twitter archive and shared it exclusively for use in this project. This archive contains basic tweet data (tweet ID, timestamp, text, etc.) for all 5000+ of their tweets as they stood on August 1, 2017.</p><p>This was extracted programmatically by Udacity and provided as a csv file to use.</p><pre>pd.read_csv(&#39;twitter-archive-enhanced.csv&#39;)</pre><p><strong>2. Image Prediction File:</strong> The tweet image predictions, i.e., what breed of dog is present in each tweet according to a neural network is stored in this file. It was hosted on Udacity’s servers in tsv format and had to be downloaded programmatically using the <a href="https://d17h27t6h515a5.cloudfront.net/topher/2017/August/599fd2ad_image-predictions/image-predictions.tsv"><em>Url</em></a>.</p><p>Python’s <em>request</em> library was used to gather this data from the web. <a href="https://requests.readthedocs.io/en/master/"><strong><em>Requests</em></strong></a> is a versatile HTTP library in python with various applications. One of its applications is to download or open a file from web using the file URL.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c1efb6e68675f648fd490316598757a5/href">https://medium.com/media/c1efb6e68675f648fd490316598757a5/href</a></iframe><p><strong>3. Twitter API — JSON File: </strong>By using the tweet IDs in the WeRateDogs Twitter archive, I queried the Twitter API for each tweet’s JSON data using Python’s tweepy library.</p><p><a href="http://docs.tweepy.org/en/v3.5.0/getting_started.html"><strong><em>Tweepy</em></strong></a> is an open source Python package that gives you a very convenient way to access the Twitter API with Python. You can find more details on <a href="https://developer.twitter.com/en/docs/basics/apps/overview"><em>setting up an app in twitter</em></a> and <a href="https://www.earthdatascience.org/courses/use-data-open-source-python/intro-to-apis/twitter-data-in-python/"><em>accessing Twitter API using Python</em></a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e74ebf6fdc138bd16c2d22e983dea501/href">https://medium.com/media/e74ebf6fdc138bd16c2d22e983dea501/href</a></iframe><p>Further, each tweet’s entire set of JSON data was stored in a file called tweet_json.txt.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/11ae6617c136838d211904f3cc6bf01b/href">https://medium.com/media/11ae6617c136838d211904f3cc6bf01b/href</a></iframe><p>Each tweet’s JSON data is written to its own line and then the .txt file is read line by line into a pandas DataFrame.</p><h3>Assessing the Data</h3><p>After gathering the data, the three tables were saved and assessed Visually and Programmatically. With both the assessments I looked for Unclean data in all the three DataFrames, i.e. for Tidiness and quality issues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*A2fGGvpHwaj-8kjR" /><figcaption>Photo by <a href="https://unsplash.com/@franki?utm_source=medium&amp;utm_medium=referral">Franki Chamaki</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong><em>Quality:</em> </strong><em>Low quality data is commonly referred to as dirty data. Dirty data has issues with its content. </em>The Data Quality Dimensions are Completeness, Validity, Accuracy and Consistency.</p><p><strong><em>Tidiness:</em></strong> <em>Untidy data is commonly referred to as “messy” data. Messy data has issues with its structure. </em>Tidy data is where:<br>1. Each variable forms a column.<br>2. Each observation forms a row.<br>3. Each type of observational unit forms a table.</p><p>Visual Assessment provided me with issues such as Non-descriptive column headers and repetitive columns in the three Datasets.<br>Programmatic Assessment actually gave me most of the quality issues such as incorrect data types and duplicate data that were present in the three datasets. I then documented the gist of my assessment in the python notebook.</p><h3>Cleaning the Data</h3><p>Cleaning means acting on the assessments we made to <strong>improve quality and tidiness</strong>.</p><p>Improving quality doesn’t mean changing the data to make it say something different — that’s data fraud. Quality improvement means <em>Correcting when inaccurate, Removing when irrelevant and Replacing when missing</em>.</p><p>Similarly, improving tidiness means transforming the dataset so that <em>each variable is a column, each observation is a row, and each type of observational unit is a table.</em></p><p>I followed the programmatic Data Cleaning process i.e. <strong><em>Define, Code and Test.</em></strong> I converted my observations from the assess step into <strong>defined</strong> problems, translated these definitions to sophisticated <strong>code</strong> to fix these problems, then <strong>tested</strong> the three datasets to make sure the operations worked.</p><h3>Storing the Data</h3><p>After cleaning the data, I found out that there was no need for three data sets. All the data could be easily converted into a single file. So, I concatenated the three DataFrames on a common attribute twitter_id, to create the twitter_archive_master.csv.</p><h3>Analyzing the Data</h3><p>Using this freshly cleaned WeRateDogs Twitter data, interesting and trustworthy analyses and visualizations were created to communicate back the findings.</p><p>WeRateDogs is a Twitter account that rates people’s dogs with humorous comments. These ratings almost always have a denominator of 10. The numerators though is almost always greater than 10. 11/10, 12/10, 13/10, etc. Why? Because “they’re good dogs Brent”. WeRateDogs believes almost all dogs deserve a 10 or sometimes more than that. Each tweet’s retweet count and favorite (i.e. “like”) count are another interesting parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/1*-xx0dgQN2V3rOi-sl_569g.png" /><figcaption>Source: Twitter</figcaption></figure><p>So, what to do with these ratings? With dogs, the best common question has to be which is the most popular dog? Can we get any relationship between Retweets, Favorites and Ratings? I have analyzed the data to get answers to these questions.</p><p>Python libraries such as matplotlib and seaborn have helped me with some cool visualizations for my analysis.</p><h3>Most Popular Dog Breeds</h3><p>Analyzing my data based on the first predictions gave me the <em>top 10 most popular dog breeds</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*cciiGcV2ir-4YWDoL55CqA.png" /><figcaption>Image by Author</figcaption></figure><p><em>Golden Retriever</em> is the most popular dog. These dogs have most tweets on them, which has led them on our most popular list.</p><p>We have the jpg_url of dogs in our data. Pythons io module is used to manage the file-related input output operations. I have used Pillow, the friendly PIL fork — Python Imaging Library to get the image of our most popular dog Golden Retriever that tops the list as per the count.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e677ce42f7b4c1e15ef7b3d1f2d681e4/href">https://medium.com/media/e677ce42f7b4c1e15ef7b3d1f2d681e4/href</a></iframe><p>Below is the result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/1*us8ujaP4MoFYzqWSMnK2fw.png" /><figcaption><em>Image by Author</em></figcaption></figure><h3>Retweets, Favorites and Ratings</h3><p>Amazingly many of the tweets have been retweeted. Some of the tweets have nearly <em>40000</em> retweets.</p><p>I tried to find out if there is any relationship between the Retweets, the ratings, and the favorites through a Correlation plot, one of the coolest plots in python. I used the seaborn library to generate a heatmap.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*SRWpzEm3OzjHWVhGIl2xdQ.png" /><figcaption>Image by Author</figcaption></figure><p>There is no clear relationship between the retweets and the ratings. But there is a strong positive correlation between Favorites and retweets i.e a linear relationship between the retweet counts and the favorite counts. The same can be confirmed by the below scatterplot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*jsX0lHBSz_cCwalwFaTszw.png" /><figcaption>Image by Author</figcaption></figure><p>So, the insight here is retweets usually are the favorites, but we cannot say that all the retweets are favorite or vice a versa.</p><p>During my analysis, I found that there is a strong linear relationship between the Favourites count and the Retweets count of a given Tweet.<br>Golden retriever is the most popular dog and it along with Labrador Retriever, Pembroke, Chihuahua and Pugs complete the top 5 common dog breeds in the data.</p><h3>Conclusion</h3><p>Many other parameters in this Dataset can be analyzed for different insights and visualizations. Since the primary focus of this project was Data Wrangling, more time was spent in that section.</p><p>For <a href="https://medium.com/u/2929690a28fb">Udacity</a>, I had to submit two reports along with <em>the Python notebook</em>. The first report was an internal document, <em>the wrangle_report</em>, that briefly describes the wrangling efforts and the second one was an external document (like a blog post or magazine article, for example), <em>the act_report</em>, that communicates the insights and displays the visualizations produced from the wrangled data.</p><p>A data wrangler knows their data intimately and is always looking for ways to enrich the data. I have done the same using amazing Python Libraries. This project was one of my most fun data experiences, so much that I decided to write my first blog on it. You can view the project, along with my reports on my <a href="https://github.com/divyanitin/TwitterDataAnalysis-WeRateDogs">Github</a>.</p><p>Thank you for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a441be7d4a85" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/twitter-analytics-weratedogs-a441be7d4a85">Twitter Analytics: “WeRateDogs”</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>