<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Hector Rodriguez Dominguez on Medium]]></title>
        <description><![CDATA[Stories by Hector Rodriguez Dominguez on Medium]]></description>
        <link>https://medium.com/@HRD_SILVER?source=rss-6064ffd599af------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*as0drWuIa1-s160GjMX_hA.jpeg</url>
            <title>Stories by Hector Rodriguez Dominguez on Medium</title>
            <link>https://medium.com/@HRD_SILVER?source=rss-6064ffd599af------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 09:05:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@HRD_SILVER/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a Twitter analytic platform with tweepy 4.10 and python]]></title>
            <link>https://medium.com/@HRD_SILVER/building-a-twitter-analytic-platform-with-tweepy-4-10-and-python-32dc340fe57a?source=rss-6064ffd599af------2</link>
            <guid isPermaLink="false">https://medium.com/p/32dc340fe57a</guid>
            <category><![CDATA[twitter-analytics]]></category>
            <category><![CDATA[django]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[sentiment-analysis]]></category>
            <category><![CDATA[tweepy]]></category>
            <dc:creator><![CDATA[Hector Rodriguez Dominguez]]></dc:creator>
            <pubDate>Thu, 05 Jan 2023 15:24:44 GMT</pubDate>
            <atom:updated>2023-01-09T17:46:29.468Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JCNguBngnIdvt13w6QrZCg.png" /></figure><p>Part 1 — of many… hope so</p><p><em>Let’s sum up all the tutorials out there on the wild and make our lives easier T-T</em></p><h3>Introduction</h3><p>In this publication, we’ll see the first baby steps to build a twitter analytics platform, so you can study the behavior of the community on a specific topic, including sentiment analysis and other techniques.</p><h4>*Warning:</h4><p>In this tutorial, we assume you already have a Twitter developer app access, either elevated or academic. If not, you can follow this official guide: <a href="https://developer.twitter.com/en/support/twitter-api/developer-account">https://developer.twitter.com/en/support/twitter-api/developer-account</a></p><h4>Index</h4><ol><li>Twitter metrics</li><li>Django</li><li>Types of authentication</li><li>Twitter api v2 User context authentication</li><li>Data extraction</li><li>Data cleaning</li><li>Basic EDA</li><li>Bibliography</li></ol><h3>Twitter metrics</h3><p>I’ll try to go fast in this section, since almost everything we must know about twitter metrics is really well explained on their documentation page.</p><h4>What are they?</h4><p>The metrics field allows developers to access public and private engagement metrics for Tweet and media objects. Public metrics are accessible by anyone with a developer account, while private metrics are accessible from owned/authorized accounts</p><h4>*Warning:</h4><blockquote>Public metrics can be requested with <a href="https://developer.twitter.com/en/docs/authentication/oauth-2-0">App only Token</a> authentication. Non-public metrics can be requested for owned/authorized Tweets only. This means developers are required to authenticate using <a href="https://developer.twitter.com/en/docs/authentication/oauth-2-0/authorization-code">OAuth 2.0</a> or <a href="https://developer.twitter.com/en/docs/authentication/oauth-1-0a">OAuth 1.0a</a> user context authorization.</blockquote><blockquote>Non-public, organic, and promoted metrics are only available for Tweets that have been created within the last 30 days.</blockquote><h4>Terminology</h4><ul><li><strong>Authorized account</strong>: Twitter account that has authorized your app to perform certain actions by granting it access.</li><li><strong>Owned account</strong>: Twitter account linked to your app.</li><li><strong>Public metrics</strong>: Metrics’ totals that are available for anyone to access in general, such as number of likes and number of Retweets.</li><li><strong>Non-public metrics</strong>: Metrics’ totals that are not available for anyone, but authenticated users, such as number of impressions and video view quartiles. Requires user context authentication.</li><li><strong>Organic metrics</strong>: Grouping of public and non-public metrics attributed to an organic context. Requires user context authentication.</li><li><strong>Promoted metrics</strong>: Grouping of public and non-public metrics attributed to a promoted context (posted or viewed as part of an Ads campaign). Requires user context authentication and that the Tweet was promoted in an Ad.</li></ul><h4>Available metrics</h4><ul><li><strong>Impressions:</strong> A count of how many times the Tweet has been viewed (not unique by user)</li><li><strong>Retweets:</strong> A count of how many times the Tweet has been Retweeted</li><li><strong>Quote Tweets:</strong> A count of how many times the Tweet has been Retweeted with a new comment (message)</li><li><strong>Likes:</strong> A count of how many times the Tweet has been liked</li><li><strong>Replies:</strong> A count of how many times the Tweet has been replied to</li><li><strong>URL Link Clicks:</strong> A count of the number of times a user clicks on a URL link or URL preview card in a Tweet</li><li><strong>User Profile Clicks:</strong> A count of the number of times a user clicks the following portions of a Tweet: display name, username, profile picture</li><li><strong>Video views:</strong> A count of how many times the video included in the Tweet has been viewed</li></ul><h3>DJango</h3><h4><strong>What is DJango?</strong></h4><p>Django is a high-level Python web framework that enables rapid development of secure and maintainable websites. Built by experienced developers, Django takes care of much of the hassle of web development, so you can focus on writing your app without needing to reinvent the wheel.</p><h4>Django Installation</h4><blockquote>Installing an official release with pip</blockquote><p>This is the recommended way to install Django.</p><ol><li>Install <a href="https://pip.pypa.io/">pip</a>. The easiest is to use the <a href="https://pip.pypa.io/en/latest/installation/">standalone pip installer</a>. If your distribution already has pip installed, you might need to update it if it’s outdated.</li><li>Take a look at <a href="https://docs.python.org/3/tutorial/venv.html">venv</a>. This tool provides isolated Python environments, which are more practical than installing packages systemwide.</li><li>After you’ve created and activated a virtual environment, enter the command:</li></ol><pre>python -m pip install Django</pre><h4>Django project creation basics</h4><p>In this section, I’m planning on creating a new entry to describe step by step the creation and configuration of a django project, but since I’m short in time I’ll leave some links to great tutorials for this part, while I write a specific entry for this.</p><ul><li>A really complete Youtube video list by Corey Schafer <a href="https://www.youtube.com/watch?v=UmljXZIypDc&amp;list=PL-osiE80TeTtoQCKZ03TU5fNfx2UY6U4p&amp;ab_channel=CoreySchafer">https://www.youtube.com/watch?v=UmljXZIypDc&amp;list=PL-osiE80TeTtoQCKZ03TU5fNfx2UY6U4p&amp;ab_channel=CoreySchafer</a></li><li>Official Django page tutorial (just the basics) <a href="https://docs.djangoproject.com/en/4.1/intro/tutorial01/">https://docs.djangoproject.com/en/4.1/intro/tutorial01/</a></li><li>Mozilla basic tutorial (In addition to the second link) <a href="https://developer.mozilla.org/en-US/docs/Learn/Server-side/Django/skeleton_website">https://developer.mozilla.org/en-US/docs/Learn/Server-side/Django/skeleton_website</a></li></ul><h3>Types of authentication</h3><p>Here is where the tricky part begins, since many of this information and terminology is scattered all over the internet, but with this publication I hove you can find what you are looking for.</p><h4>Terminology</h4><ul><li><strong>App-only:</strong> Type of authentication where there is no user account validation required, but it only allows us to access public metrics.</li><li><strong>User-context:</strong> Type of authentication where user account validation is required. This type of authentication will allow us to perform actions on behalf of the user and access private metrics.</li><li><strong>Bearer Token:</strong> Bearer tokens are a type of access token; authentication which uses bearer tokens is also known sometimes as application-only authentication or auth-only authentication.</li><li><strong>API key &amp; secret (consumer_key and secret):</strong> These credentials act as the username and password for your Twitter App, and are used by the Twitter API to understand which App requests are coming from.</li><li><strong>Access token &amp; secret:</strong> Access Tokens represent the user that you are making the request on behalf of.</li></ul><h3>Twitter API v2 User context authentication</h3><p>In this section, we’ll see three ways of connecting to different endpoints:</p><ul><li>Endpoint v1 without user context: This version will allow you to use the endpoint v1 to make request for public information, it will not ask for any kind of user authentication and will provide the model as shown on the next link: <a href="https://docs.tweepy.org/en/stable/api.html#get-tweet-timelines">https://docs.tweepy.org/en/stable/api.html#get-tweet-timelines</a></li></ul><pre>import tweepy<br><br># Authentication<br>def connect(object):<br>    consumerKey = &quot;YOUR KEY HERE&quot;<br>    consumerSecret = &quot;YOUR KEY HERE&quot;<br>    accessToken = &quot;YOUR KEY HERE&quot;<br>    accessTokenSecret = &quot;YOUR KEY HERE&quot;<br>    bearer_token = &quot;YOUR KEY HERE&quot;<br><br>    auth = tweepy.OAuthHandler(consumerKey, consumerSecret)<br>    auth.set_access_token(accessToken, accessTokenSecret)<br>    api = tweepy.Client(bearer_token, wait_on_rate_limit=True, wait_on_rate_limit_notify=True)<br><br>    return api</pre><ul><li>Endpoint v2 without user context: This version will allow you to use the endpoint v2 to make request for public information, it will not ask for any kind of user authentication and will provide the model as shown on the next link: <a href="https://docs.tweepy.org/en/stable/v2_models.html">https://docs.tweepy.org/en/stable/api.html#get-tweet-timelines</a></li></ul><pre>import tweepy<br><br># Authentication<br>def connect(object):<br>    apiSecret = &quot;YOUR KEY HERE&quot;<br>    apiKey = &quot;YOUR KEY HERE&quot;<br>    bearerToken = &quot;YOUR KEY HERE&quot;<br><br>    return tweepy.Client(bearer_token=bearerToken, wait_on_rate_limit=True)</pre><ul><li>Endpoint v2 <strong>WITH </strong>user context: This code segment is the one we’ll be using through the whole tutorial. With this, I’m nos saying you won&#39;t be able to follow the tutorial, just that you’ll need to pay close attention we are using private metrics and make the according changes to the code.</li><li>This method <strong>will </strong>ask for the user authentication of the account we are following</li></ul><p>Create the file “twitter_authV2UserContext.py” and place the following code.</p><pre>import tweepy<br>import pandas as pd<br>import os<br><br>#3-legged OAuth<br>client_id = &#39;YOUR KEY HERE&#39;<br>client_secret = &#39;YOUR KEY HERE&#39;<br>consumer_key = &#39;YOUR KEY HERE&#39;<br>consumer_secret = &#39;YOUR KEY HERE&#39;<br>callback_uri = &#39;Any webpage link here&#39;<br><br>#class twitter_auth():<br>def request(object, screen_name):<br><br>    oauth1_user_handler = tweepy.OAuth1UserHandler(<br>        consumer_key=consumer_key,<br>        consumer_secret=consumer_secret,<br>        callback=callback_uri<br>    )<br><br>    print(oauth1_user_handler.get_authorization_url())<br>    authorization_ur = input(&#39;Paste redirect url here:  &#39;)<br>    oauth_verifier = authorization_ur.split(&quot;oauth_verifier=&quot;)<br><br>    access_token, access_token_secret = oauth1_user_handler.get_access_token(<br>        oauth_verifier[1]<br>    )<br><br>    user_keys = []<br>    user_keys = {<br>        &#39;screen_name&#39;: screen_name,<br>        &#39;access_token&#39;: access_token,<br>        &#39;access_token_secret&#39;: access_token_secret<br>    }<br>    user_df = pd.DataFrame([user_keys])<br><br>    file_name = screen_name + &quot;_keys&quot;<br>    file_path = &quot;profiles/&quot; + file_name + &quot;.csv&quot;<br>    user_df.to_csv(file_path, header=True, index=False, encoding=&#39;utf-8-sig&#39;)<br><br>    return &quot;Profile created successfully&quot;<br><br><br>def connect(object, screen_name):<br>    user_keys = pd.read_csv(&quot;profiles/&quot; + screen_name + &quot;_keys.csv&quot;)<br>    client = tweepy.Client(<br>        consumer_key=consumer_key,<br>        consumer_secret=consumer_secret,<br>        access_token=user_keys.access_token[0],<br>        access_token_secret=user_keys.access_token_secret[0]<br>    )<br>    return client</pre><p>On our Jupyter notebook, we can include our connection functions by importing the file we just created.</p><pre>import twitter_authV2UserContext as tw_a2v2uc</pre><p>Once included on the notebook, we’ll create a new cell calling our connection and run cell.</p><pre>tw_a2v2uc.request(None, &#39;@tweeter_username&#39;)</pre><p>The following text will appear, and click the link.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H-AyLZA7F0X0LNTG5lxAvw.png" /></figure><p>Once the link is clicked, you’ll need to log in to the account we want to follow, or you can send it to the manager of the account so he/she can grant the access to the app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*AncZGtOkOkpU99nmSzfPpQ.png" /></figure><p>After authorizing the app, the user will be redirected to the link you provided on the “twitter_authV2UserContext.py” file, if the authorization link was sent to the manager, ask for the link of the page he was redirected to, an example of the link is: <a href="http://testlink.com/?oauth_token=uBDQgAAAAABe2tKAAABhTrWntA&amp;oauth_verifier=aZNycKgmpdICOeqRpSIfFbyXdwuyQ5w"><em>http://testlink.com?oauth_token=uBDQgAAAAABe2tKAAABhTrWntA&amp;oauth_verifier=aZNycKgmpdICOeqRpSIfFbyXdwuyQ5w</em></a></p><p>Paste the link into the redirect URL field as shown in the image and click enter:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FPyf2EYuhHFJ9coXXVeBLw.png" /></figure><p>As you can see, in the link there is an <a href="http://testlink.com/?oauth_token=uBDQgAAAAABe2tKAAABhTrWntA&amp;oauth_verifier=aZNycKgmpdICOeqRpSIfFbyXdwuyQ5w"><em>oauth_token</em></a><em> </em>and <a href="http://testlink.com/?oauth_token=uBDQgAAAAABe2tKAAABhTrWntA&amp;oauth_verifier=aZNycKgmpdICOeqRpSIfFbyXdwuyQ5w"><em>oauth_verifier</em></a>, this information is what we&#39;ll be saving in the file “profiles/@tweeter_username_keys.csv”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/1*WnmJ5CD2UuQyLeDJJ_NGeg.png" /></figure><p>Now that the keys file is created, we’ll connect to the API by executing the next cell on the notebook.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/1*Hg6jqKBLmT3zO2NhntbMXQ.png" /></figure><h3>Data extraction</h3><p>After we have created our connection, we can start making requests to the API, for the effect of making basic testing we’ll use covid as our search term. On a new cell state, the variables to be used on our queries.</p><pre>#Searching for covid in spanish and filtering out retweets<br>query = &quot;covid lang:es -is:retweet&quot;<br><br>#Converts dates, so the api can search through this limits<br>input_string = &quot;2021-10-01 23:59:59&quot;<br>startDate = psn.get_date_object(input_string)<br>input_string = &quot;2021-10-31 23:59:59&quot;<br>endDate = psn.get_date_object(input_string)<br><br>#Max number of tweets per query, this number varies according to the level<br>#of the account permissions<br>noOfTweets = 500 <br><br>#Max number of tweets in total<br>flatLimit = 20000 </pre><p>Once set the parameters for the queries, we can start making the requests by executing the following cell.</p><pre>def paginator_to_df_v2__batch(api, query, noOfTweets, flatLimit, startDate = None, endDate = None ):  # tweets = to_df_v2(tweets_all)<br>    tweet_info_ls = []<br><br>    for response in tweepy.Paginator(api.search_all_tweets,<br>                                     query=query,<br>                                     start_time=startDate,<br>                                     end_time=endDate,<br>                                     tweet_fields=[&quot;id&quot;,&quot;created_at&quot;, &quot;text&quot;, &quot;attachments&quot;, &quot;author_id&quot;<br>                                         , &quot;conversation_id&quot;, &quot;entities&quot;, &quot;geo&quot;, &quot;lang&quot;, &quot;in_reply_to_user_id&quot;<br>                                         , &quot;possibly_sensitive&quot;, &quot;public_metrics&quot;<br>                                         , &quot;referenced_tweets&quot;, &quot;reply_settings&quot;, &quot;withheld&quot;, &quot;source&quot;],<br>                                     max_results=noOfTweets).flatten(limit=flatLimit):<br>        <br>        if response.referenced_tweets is not None:<br>            for elem in response.referenced_tweets:<br>                tweet_info_referenced = {<br>                    &#39;referenced_tweet_id&#39;: elem.id,<br>                    &#39;referenced_tweet_type&#39;: elem.type,<br>                }<br>        else:<br>            tweet_info_referenced = {<br>                &#39;referenced_tweet_id&#39;: None,<br>                &#39;referenced_tweet_type&#39;: None,<br>            }<br><br>        tweet_info = {<br>            &#39;created_at&#39;: response.created_at,<br>            &#39;text&#39;: response.text,<br>            &#39;source&#39;: response.source,<br>            &#39;author_id&#39;: response.author_id,<br>            &#39;context_annotations&#39;: response.context_annotations,<br>            &#39;conversation_id&#39;: response.conversation_id,<br>            &#39;entities&#39;: response.entities,<br>            &#39;geo&#39;: response.geo,<br>            &#39;lang&#39;: response.lang,<br>            &#39;possibly_sensitive&#39;: response.possibly_sensitive,<br>            &#39;public_metrics&#39;: response.public_metrics,<br>            &#39;referenced_tweets&#39;: tweet_info_referenced,<br>            &#39;reply_settings&#39;: response.reply_settings,<br>            &#39;withheld&#39;: response.withheld,<br>            &#39;in_reply_to_user_id&#39;: response.in_reply_to_user_id<br>        }<br>        tweet_info_ls.append(tweet_info)<br><br>    tweets_df = pd.DataFrame(tweet_info_ls)<br><br>    textended_df = expand_column_to_df_v2(tweets_df, tweets_df.entities)<br><br>    textended_df.drop(&#39;entities&#39;, axis=1, inplace=True)<br><br>    return textended_df<br><br>def expand_column_to_df_v2__batch(df, tweets_orig_column, column):  # df = tweets_orig; tweets_orig_column = tw.context_annotations,geo,public_metrics<br>    df_new = tweets_orig_column.apply(pd.Series, dtype=str)<br>    df_1 = pd.concat([df, df_new], axis=1, join=&#39;inner&#39;)<br>    df_1.drop(column, axis=1, inplace=True)<br><br>    return df_1<br><br>def expand_column_to_df_v2(df, tweets_orig_column):  # df = tweets_orig; tweets_orig_column = tw.context_annotations,geo,public_metrics<br>    df_new = tweets_orig_column.astype(&#39;str&#39;)<br>    <br>    df_new = df_new.apply(lambda x: ast.literal_eval(x))<br>    <br>    df_new = df_new.apply(pd.Series)<br><br>    df_1 = pd.concat([df, df_new], axis=1, join=&#39;inner&#39;)<br><br>    return df_1<br><br><br>def reg_to_df_v2__batch(tw):  # tw = reg_to_df_v2(tw_orig.iloc[5].entities)<br>    if isinstance(tw, list):<br>        df = pd.DataFrame.from_records(tw)<br>    else:<br>        df = pd.DataFrame()<br><br>    return df</pre><pre>tweets_df = None<br>tweets_df = psn.paginator_to_df_v2__batch(api, query, noOfTweets, flatLimit, startDate, endDate)</pre><p>This will return a data frame of 20000 tweets and 19 columns as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vIU2GRCu69Ut2zFWtlrnRA.png" /></figure><p>As a final task in this section, we’ll save the data frame for further use on a csv.</p><pre>psn.to_csv_nodate(tweets_df, &quot;covid_20k.csv&quot;)</pre><h3>Data cleaning</h3><p>We’ll start by reading the csv we just created and expanding a few columns that were returned in arrays.</p><pre>tweets_df = pd.read_csv(&#39;covid_20k.csv&#39;)<br><br>df_eda_csv = psn.expand_column_to_df_v2__csv(tweets_df, tweets_df.referenced_tweets, &#39;referenced_tweets&#39;)#Expansión de referenced_tweets<br>df_eda_csv = psn.expand_column_to_df_v2__csv(df_eda_csv, df_eda_csv.public_metrics, &#39;public_metrics&#39;)#Expansión de public_metrics<br><br>df_eda = df_eda_csv</pre><p>Now according to your preferences and the objectives of your project you can delete empty rows and values of the data frame. The following cell will help you with that.</p><pre>print(&quot;source: &quot;+str(df_eda[&#39;source&#39;].isna().sum()))<br>print(&quot;author_id: &quot;+str(df_eda[&#39;author_id&#39;].isna().sum()))<br>print(&quot;context_annotations: &quot;+str(df_eda[&#39;context_annotations&#39;].isna().sum()))<br>#print(&quot;entities: &quot;+str(df_eda[&#39;entities&#39;].isna().sum()))<br>print(&quot;geo: &quot;+str(df_eda[&#39;geo&#39;].isna().sum()))<br>print(&quot;lang: &quot;+str(df_eda[&#39;lang&#39;].isna().sum()))<br>print(&quot;possibly_sensitive: &quot;+str(df_eda[&#39;possibly_sensitive&#39;].isna().sum()))<br>#print(&quot;public_metrics: &quot;+str(df_eda[&#39;public_metrics&#39;].isna().sum()))<br>#print(&quot;private_metrics: &quot;+str(df_eda[&#39;private_metrics&#39;].isna().sum()))<br>#print(&quot;referenced_tweets: &quot;+str(df_eda[&#39;referenced_tweets&#39;].isna().sum()))<br>print(&quot;reply_settings: &quot;+str(df_eda[&#39;reply_settings&#39;].isna().sum()))<br>print(&quot;withheld: &quot;+str(df_eda[&#39;withheld&#39;].isna().sum()))<br>print(&quot;in_reply_to_user_id: &quot;+str(df_eda[&#39;in_reply_to_user_id&#39;].isna().sum()))<br>print(&quot;annotations: &quot;+str(df_eda[&#39;annotations&#39;].isna().sum()))<br>print(&quot;mentions: &quot;+str(df_eda[&#39;mentions&#39;].isna().sum()))<br>print(&quot;hashtags: &quot;+str(df_eda[&#39;hashtags&#39;].isna().sum()))<br>print(&quot;urls: &quot;+str(df_eda[&#39;urls&#39;].isna().sum()))<br>print(&quot;referenced_tweet_id: &quot;+str(df_eda[&#39;referenced_tweet_id&#39;].isna().sum()))<br>print(&quot;referenced_tweet_type: &quot;+str(df_eda[&#39;referenced_tweet_type&#39;].isna().sum()))<br>print(&quot;retweet_count: &quot;+str(df_eda[&#39;retweet_count&#39;].isna().sum()))<br>print(&quot;reply_count: &quot;+str(df_eda[&#39;reply_count&#39;].isna().sum()))<br>print(&quot;like_count: &quot;+str(df_eda[&#39;like_count&#39;].isna().sum()))<br>print(&quot;quote_count: &quot;+str(df_eda[&#39;quote_count&#39;].isna().sum()))</pre><h3>Basic EDA</h3><p>In this section, I’ll show you the basic data analysis, and you’ll determine what will be useful for your project. Now that we got the data frame cleaned, we can start analyzing its information.</p><ul><li><strong>Shape of the Data Frame</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/120/1*_OpzxoaFYWEfri6wBli4kw.png" /></figure><ul><li><strong>Source of the tweets</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*VF8Ee4Y4Q0MtDvrrCpOrsA.png" /></figure><ul><li><strong>Possibly sensitive tweets:</strong> Since we are planning on making a future sentiment analysis, this column might be useful</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1009/1*DGkczwrI0dD3DXuOh_IiWQ.png" /></figure><ul><li><strong>Number of retweets:</strong> here we can see that most of the tweets just got on retweet and slowly decreasing.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*6--89Dn0eaxS-jIOrHBeuA.png" /></figure><p>With this we can establish a range limit to determine if a tweet has a normal, low, medium, high or extraordinary retweet count.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*be5GvCn4f9Z-ioGhTN6vfA.png" /></figure><ul><li><strong>Reply count</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*UZAXhoSOT5k5AZeE6kVKzg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*XsMJWpiRddK1JLMJYF-Luw.png" /></figure><ul><li><strong>Like count</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*KBd47OlEkpOtjjxYlgl6jg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*RB77eaqpzSVMLCFMDyfdwg.png" /></figure><ul><li><strong>Quote count</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*t3FUV4qAORV-l_d_5wGLOg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*CEl6XCgmWP1QgCb6pSGUUg.png" /></figure><ul><li><strong>User id with the most replies</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*cIolZzn0KWdZbOzxaltzRw.png" /></figure><ul><li><strong>Georeferenced tweets:</strong> Not every tweet has a geotag, but the ones that got one can show us where they come from</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*f3agQWK1p248mSFqTIDDMQ.png" /></figure><ul><li><strong>Correlation heatmap:</strong> We apply this tool, so we can see strong relations between variables.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*g7yF3kAlE3S8fk_tfK_17w.png" /></figure><p><strong>Conclusions of this section</strong></p><ul><li>Commonly if the user likes a tweet he also retweets it</li><li>Most of the data is complete</li><li>We can&#39;t use ‘geo’ as a critical variable</li><li>Identical text tweets must be deleted</li></ul><h3><strong>Bibliography</strong></h3><ul><li><a href="https://developer.twitter.com/en/docs/twitter-api/metrics">Metrics</a></li><li><a href="https://docs.tweepy.org/en/stable/authentication.html#oauth-2-0-authorization-code-flow-with-pkce-user-context">Authentication - tweepy 4.11.0 documentation</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=32dc340fe57a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Twitter sentiment analysis using Zookeeper, Kafka and PySpark live-streaming on Windows 10 in 2022]]></title>
            <link>https://medium.com/mcd-unison/twitter-sentiment-analysis-using-zookeeper-kafka-and-pyspark-live-streaming-on-windows-10-in-2022-ada7757097a2?source=rss-6064ffd599af------2</link>
            <guid isPermaLink="false">https://medium.com/p/ada7757097a2</guid>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[sentiment-analysis]]></category>
            <category><![CDATA[kafka-streams]]></category>
            <category><![CDATA[twitter]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Hector Rodriguez Dominguez]]></dc:creator>
            <pubDate>Sun, 08 May 2022 21:47:05 GMT</pubDate>
            <atom:updated>2022-05-08T21:50:57.536Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xCRMXzexN28L5A2nk20bSA.jpeg" /></figure><p><em>Let’s sum up all the tutorials out there on the wild and make our lives easier T-T</em></p><h3><strong>Introduction</strong></h3><p>On this entry, I’ll guide you (at least I’ll try) through this hell of an experience of trying to install and run everything on Windows, so call me “<em>Virgil”</em>.</p><p>Q: Why is it a hell of an experience?</p><p>A: Most of the tutorials out there are meant of Linux users or its information is outdated.</p><p>At the end of this tutorial we aim to have a development environment with Zookeeper, Kafka and PySpark with which we can stream live tweets using the Twitter API and classify them as “positive”, “neutral” or “negative”.</p><h3>Technology used</h3><p><strong><em>Twitter API v2:</em></strong><em> </em>If you got your credentials (consumer_key, consumer_secret, access_token, access_secret) way to go champ! If not, you can go through the slow and tedious process of requesting them in <a href="https://developer.twitter.com/en/docs/twitter-api/getting-started/getting-access-to-the-twitter-api">here</a>.</p><p><strong>Python 3.9:</strong></p><p>What is it, and what is it used for?</p><blockquote>Python is an interpreted, object-oriented, high-level programming language with dynamic semantics. Its high-level built in data structures, combined with dynamic typing and dynamic binding, make it very attractive for Rapid Application Development, as well as for use as a scripting or glue language to connect existing components together.</blockquote><p>You can download it <a href="https://www.python.org/downloads/release/python-3912/">here</a></p><p><strong>Java JRE:</strong></p><p>What is it, and what is it used for?</p><blockquote><em>Java Runtime Environment </em>(JRE) is a piece of software that is designed to run other software. As the runtime environment for Java, the JRE contains the Java class libraries, the Java class loader, and the Java Virtual Machine</blockquote><p>You can download it <a href="https://www.oracle.com/java/technologies/downloads/#jdk18-windows">here</a></p><p><strong><em>Spark 3.2.1 with Hadoop 3.2.1:</em></strong></p><p>What is it, and what is it used for?</p><blockquote>Apache Spark is basically a computational engine that works with huge sets of data by processing them in parallel and batch systems. Spark is written in Scala, and PySpark was released to support the collaboration of Spark and Python.</blockquote><p>You can download Spark 3.2.1 with <a href="https://www.apache.org/dyn/closer.lua/spark/spark-3.2.1/spark-3.2.1-bin-hadoop3.2.tgz">here</a></p><p><strong><em>Apache Kafka 3.1.0:</em></strong></p><p>What is it, and what is it used for?</p><blockquote>Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.</blockquote><p>You can download Kafka 3.1.0 with Scala 2.13 <a href="https://www.apache.org/dyn/closer.cgi?path=/kafka/3.1.0/kafka_2.13-3.1.0.tgz">here</a></p><p><strong><em>Apache Zookeeper 3.7.0 (stable version):</em></strong></p><p>What is it, and what is it used for?</p><blockquote>ZooKeeper is a high-performance coordination service for distributed applications. It exposes common services — such as naming, configuration management, synchronization, and group services — in a simple interface so you don’t have to write them from scratch. You can use it off-the-shelf to implement consensus, group management, leader election, and presence protocols. And you can build on it for your own, specific needs.</blockquote><p>You can download Zookeeper 3.7.0 <a href="https://www.apache.org/dyn/closer.lua/zookeeper/zookeeper-3.7.0/apache-zookeeper-3.7.0-bin.tar.gz">here</a></p><h3>Tools configurations on Windows 10</h3><p>After downloading the files mentioned above, create three folders on C: (ZOOKEEPER, KAFKA, SPARK), each for a different tool.</p><p>Extract the contents of each compressed file to its corresponding folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*jzj-iUPqBTm1TdzxvSm9oA.png" /></figure><ul><li><strong>JRE Installation</strong></li></ul><p>1. Start the JRE installer and press install, by default the installation folder will be “C:\Program Files\Java\jre1.8.0_321”</p><p>2. Now let&#39;s open Windows environment variables menu by opening Control Panel → System → Advanced system settings → Environment Variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/1*HPQqtGEprcPB4Esb8aSaOQ.png" /></figure><p>3. Hit the New User Variable button in the User variables section, then type JAVA_HOME in “<em>Variable name” </em>and give your JRE path in the “<em>Variable value”.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*BFrYmMc4DjnLcNj3zk8Wlw.png" /></figure><p>4. Search for a Path variable in the “System Variable” section in the “Environment Variables” and press edit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*LuvaJMuzzcpsFT3SugwJQw.png" /></figure><p>5. Press “New” and write the path to the bin folder → “%JAVA_HOME%\bin” and press ok.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/1*X218M9LVnHLg-7CCJt60uQ.png" /></figure><p>6. Now to verify the correct installation, open a terminal and type “<em>java –version” </em>the following message should appear<em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/723/1*ooVv2lEobjAYLAvfVBDyKQ.png" /></figure><ul><li><strong>Python 3.9 Installation</strong></li></ul><ol><li>Start the Python installer and press install, by default the installation folder will be “C:\Users\USER_NAME\AppData\Local\Programs\Python\Python39” -Change the USER_NAME, according to your personal folder-</li><li>Open the environment variables window, select the “Path” variable and hit the Edit button in the System Variables section<em>.</em></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*jQ2Iaw1Vun5ELQQKAu21eQ.png" /></figure><p>3. Now add the new environment variables</p><p>→ C:\Users\USER_NAME\AppData\Local\Programs\Python\Python39</p><p>→ C:\Users\USER_NAME\AppData\Local\Programs\Python\Python39\Scripts</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*VZQ8Ow8hzAZaERUqhuJUhA.png" /></figure><p>4. To verify if the installation is correct, open a terminal and type “python — version”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/277/1*rJKFV4MyIKxwM6uyrsB7aQ.png" /></figure><ul><li><strong>ZooKeeper Installation</strong></li></ul><ol><li>First, we need to open the folder containing our zookeeper files and head to the “conf” folder at “C:\ZOOKEEPER\conf”, there we’ll find a file named “zoo_sample.cfg”. Rename the file to “zoo.cfg” and open the file with a text editor.</li><li>Find and change “dataDir=/tmp/zookeeper” → “dataDir=C:/ZOOKEEPER/data”</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/1*Ti7bDKBqLC5Czn4ZAElYtA.png" /></figure><p>3. Now we have to add an Environment Variable named “ZOOKEEPER_HOME” aimed to “C:\ZOOKEEPER_HOME”, as we did before with JRE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*3hoIEFoJKRlr11ozviFOrg.png" /></figure><p>4. Search for a Path variable in the “System Variable” section in the “Environment Variables” and press edit and add “%ZOOKEEPER_HOME%\bin”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/490/1*KGX4yFjzymCGS0MlOQo6DQ.png" /></figure><p>5. To verify the correct installation, open a terminal and type “<em>zkserver” </em>the following message should appear<em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pfOICNtNHFw6vMJSzFMVyg.png" /></figure><p>NOTE: By default zookeeper will run in port 2181, you can change this option on the zoo.cfg file</p><ul><li><strong>Kafka Installation</strong></li></ul><ol><li>First, we need to open the folder containing our Kafka files and head to the “config” folder at “C:\KAFKA\config”, there we’ll find a file named “server.properties”. Change the line “log.dirs = /tmp/kafka-logs” to “log.dir=C:/KAFKA/kafka-logs” and save it.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*worflytbt4SpOYr1ID5F8Q.png" /></figure><p>2. To validate Kafka, correct installation, open a terminal and navigat to the KAFKA folder at C:. Then type “.\bin\windows\kafka-server-start.bat .\config\server.properties”, now the terminal should show a list of text from Kafka processes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bYzALhir2iY1J9weBgbPqA.png" /></figure><p>3. The next step is to create a topic for Kafka streaming (In the next segment I’ll explain what is a topic, consumer and producer). We can do this by opening a new terminal and heading to “C:\KAFKA” and typing “.\bin\windows\kafka-topics.bat — create — bootstrap-server localhost:9092 — replication-factor 1 — partitions 1 — topic TW_ANALYSIS”, where TW_ANALYSIS is the name of the topic we are going to use, you can choose any name as long as you remember to change it on the kafkaConsumer.py, kafkaProducer.py and main.py files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i-Qd2XVH-sngd7MWBMwdbA.png" /></figure><p>NOTE: By default Kafka runs in port 9092, you can change it in the “server.properties” file, and also modify the Zookeeper default connection port with Kafka.</p><ul><li><strong>Spark Installation</strong></li></ul><ol><li>Once we have moved our files to the “C:/SPARK” folder, we need to download the winutils.exe for the Hadoop 3.2.1 version. You can find the winutils versions <a href="https://github.com/cdarlint/winutils">here</a> and our specific version in <a href="https://github.com/cdarlint/winutils/tree/master/hadoop-3.2.1/bin">here</a>.</li><li>Create a folder named “hadoop\bin” inside our “C:\SPARK” folder, so the final route should be “C:\SPARK\hadoop\bin”, and extract the contents of the winutils compressed file we just downloaded.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yCIFj7Kyi71cO8vEBIqFxw.png" /></figure><p>3. Now we should create a new SPARK_HOME and HADOOP_HOME environment variables in the User variables section. They should be aiming as follows:</p><ul><li>HADOOP_HOME → C:\SPARK\hadoop</li><li>SPARK_HOME → C:\SPARK</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/587/1*yDhdlnRWujqbpQeTIb0-3A.png" /></figure><p>4. Edit the path variable to add two new variables:</p><ul><li>%SPARK_HOME%\bin</li><li>%HADOOP_HOME%\bin</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*T6Hfy3NECY6VHMqxv2z5Og.png" /></figure><p>5. At this point, I recommend you to restart your computer so that the environment variables take effect. Now, to verify your spark installation, open a new terminal and type “PySpark”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*r3UT6o7k4WkYcsG8d1LrVQ.png" /></figure><h3>How it works? Producer.py, Consumer.py and Main.py files</h3><p><em>Now we got the good stuff mon’!</em></p><ol><li>To start with the streaming we must first run our servers(Zookeeper, Kafka, Spark), as we did before, we need to open three new terminals and in each, we start a different process.</li></ol><ul><li>Terminal 1: Type → “zkserver”</li><li>Terminal 2: Move to “C:\KAFKA” and type → “.\bin\windows\kafka-server-start.bat .\config\server.properties”</li><li>Terminal 3: Type → “pyspark”</li></ul><p>2. Create four files in your project folder:</p><ul><li>File 1 → “<a href="https://gist.github.com/Silvertongue26/c9590a7a4bdccca4d08fdda4a1758661"><strong>auth_tokens.py</strong></a>” →Here we will store our Twitter API keys to be granted access to the twitter hell.</li><li>File 2 → “<a href="https://gist.github.com/Silvertongue26/8d23bb17e807571bcd28b409094d45b7"><strong>kafkaProducer.py</strong></a><strong>” </strong>→ This will be our Twitter data extractor, that will send the data to our consumer file</li><li>File 3 → “<a href="https://gist.github.com/Silvertongue26/54f0d60151047323b48d44398336d498"><strong>kafkaConsumer.py</strong></a>” → Here we will receive the information sent by our producer and prepare it for processing in our main.py file</li><li>File 4 → “<a href="https://gist.github.com/Silvertongue26/e06011a3f3a2a60d3db025cae8da0483"><strong>main.py</strong></a><strong>” </strong>→ In this file we will make all our processing functions, data cleaning, sentiment analysis and parquet storage.</li></ul><p><strong>auth_tokens.py</strong></p><p>In this file, we’ll declare our twitter keys (Not much to say in this file)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/77ac50efaa1ad3de3e6fd33caf2772c0/href">https://medium.com/media/77ac50efaa1ad3de3e6fd33caf2772c0/href</a></iframe><p><strong>kafkaProducer.py</strong></p><ul><li>We begin by importing the libraries</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/272/1*J24iWTi5ubksiV8TZlYF9Q.png" /></figure><ul><li>We generate the instance of our Kafka producer, our search term, and topic name, which we established up in our tutorial. Since we are using our computer as our host we will configure the Kafka producer with its default port 9092 → ‘localhost:9092&quot;.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/919/1*1sU9EYxzm1_8ydMZeufEJA.png" /></figure><ul><li>We define our tweeter authentication function</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*knPTTIDWS1wKebpN3NMNeQ.png" /></figure><ul><li>We overwrite our tweet listener function in order to send information to our Kafka consumer</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*9Rv7z4UynawgQMj_utwWWQ.png" /></figure><ul><li>Now we write our main function where we call all the other functions we just wrote</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*ZsbN3pIZcpyBJgbw87R3cA.png" /></figure><ul><li>Finished file</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1a3b4e1379e3aea5517601aceaa65399/href">https://medium.com/media/1a3b4e1379e3aea5517601aceaa65399/href</a></iframe><p><strong>kafkaConsumer.py</strong></p><ul><li>With this script, we’ll catch the information that the Producer is sending us, and we’ll make it available to our main.py script</li><li>Import libraries and define our topic variable previously defined</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*ScXnjHD3DAUvNQp7S66MFQ.png" /></figure><ul><li>We create or Kafka consumer instance and configure it so that the information received in hex is transformed to a readable string in JSON.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/917/1*kwZltSlcVAU-BPgueIzTYQ.png" /></figure><ul><li>Finished file</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a11ddf4dbe1f242972ad899f2f94059c/href">https://medium.com/media/a11ddf4dbe1f242972ad899f2f94059c/href</a></iframe><p><strong>main.py (The big one… not really)</strong></p><p>In this file, we will start working with the information received from our Kafka consumer. We’ll go through a process of Data reading → Data preparation → Data cleaning → Subjectivity and polarity calculation (using the TextBlob) → Sentiment classification → Data storage. All in a day&#39;s work!</p><p>Let&#39;s start by importing libraries need for our tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*6iqppUmFueZv7iKONpk4dg.png" /></figure><p><strong>Data Reading</strong></p><ul><li>On our main function, we create our SparkSession instance and Kafka stream reader.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*VnHsyRkoChj8lgcm20AKcA.png" /></figure><p><strong>Data preparation</strong></p><ul><li>In here, we will convert the stream into a readable string and create the schema for the dataframe that will store it.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/1*kv71XiK73-b5mR0LJmq2EA.png" /></figure><p><strong>Functions definition</strong></p><ul><li>Once we have a data frame with which we can work, it&#39;s time to create the function that will clean all the link, emojis, usernames, etc. from the tweets we captured.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/923/1*_Qncr5WsR9J08viLiRfIMQ.png" /></figure><ul><li>Now that we are defining function, we should define our subjectivity, polarity and Sentiment classification functions. For this, we will use Textblob’s predefined functions. For more information on Textblob click <a href="https://textblob.readthedocs.io/en/dev/">here</a>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/910/1*gfv0azyF9ygpidVfWT039A.png" /></figure><ul><li>Once defined our functions, we should convert them to UDF or User Defined Functions in our main function. More information on UDFs click <a href="https://sparkbyexamples.com/spark/spark-sql-udf/">here</a></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*6yehYU__NmNCqOK8djvu4Q.png" /></figure><p><strong>Subjectivity and polarity calculation and Sentiment classification</strong></p><ul><li>Now defined our functions, the next easy step is to calculate the subjectivity and polarity of each tweet, then classify it as positive, negative or neutral. So we add the next code segment to our main function.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/907/1*iJXxTRHOVW_Ud1I9P1ZB8A.png" /></figure><p><strong>Data storage</strong></p><ul><li>Finally! Now that we have calculated the polarity and we have a usefull dataframe we need to store it. For tis we will use the parquet format that can save us arroun a 75% of storage space. Really usefull of we are paying for storage. For this we can add the following segment on our main.py.</li><li>To open parquet files and see its insides, I use ParquetViewer, you can download it <a href="https://github.com/mukunku/ParquetViewer/releases/download/v2.3.6/ParquetViewer.exe">here</a>.</li><li>NOTE: As you will see in the following code segments, i’ve set a wait time of 60 sec, this is for testing purposes, in production you should adjust this time according to your needs.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*nOhJ4K8SjDu6vSZ5bhkxZg.png" /></figure><p><strong>Quick validation</strong></p><ul><li>For a quick validation of what is going on thorugh console, we can add the following segment at the end of our main function.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*FsAVgwa1F6vmBAq4vY0ILA.png" /></figure><p>Final File</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9b6954365ab4834c7d63ebda6c60aadd/href">https://medium.com/media/9b6954365ab4834c7d63ebda6c60aadd/href</a></iframe><p><strong>Let it there be light!!!</strong></p><p>To run the proyect, just follow this simple steps…. DELETE EVERYTHING!… Just joking… Am I???</p><p>STEPS:</p><ol><li>Run our servers in different terminals (Zookeper, Kafka, Spark).</li><li>Run kafkaProducer.py on an independent terminal and let it run!</li><li>Run kafkaConsumer.py on an independent terminal.</li><li>Run main.py and watch your parquet files be generated with our dataframe.</li><li>Kill every process when you are done… KILL THEM WITH FIRE!!!</li></ol><h3><strong>Final Notes</strong></h3><p>Well it’s been quite the trip! If you managed to reach this point i want to thank you for your time! If you got any questions you can write me a message to my medium account.</p><p>The github project is in here → <a href="https://github.com/Silvertongue26/zookeeper_kafka_pyspark_polarity-analysis">https://github.com/Silvertongue26/zookeeper_kafka_pyspark_polarity-analysis</a></p><p>So for now… May the force be with you… .you’ll need it….</p><h3><strong>Bibliography</strong></h3><ul><li><a href="https://sparkbyexamples.com/pyspark/pyspark-find-datatype-column-names-of-dataframe/">https://sparkbyexamples.com/pyspark/pyspark-find-datatype-column-names-of-dataframe/</a></li><li><a href="https://towardsdatascience.com/sentiment-analysis-on-streaming-twitter-data-using-spark-structured-streaming-python-fc873684bfe3">https://towardsdatascience.com/sentiment-analysis-on-streaming-twitter-data-using-spark-structured-streaming-python-fc873684bfe3</a></li><li><a href="https://dzone.com/articles/running-apache-kafka-on-windows-os">https://dzone.com/articles/running-apache-kafka-on-windows-os</a></li><li><a href="https://phoenixnap.com/kb/install-spark-on-windows-10">https://phoenixnap.com/kb/install-spark-on-windows-10</a></li></ul><h3>Special Thanks</h3><p>To <a href="https://medium.com/@lorenagongang">Lorena Gongang</a> who’s guide was really helpful in the elaboration of this entry, please visit her webpage for many great projects → <a href="https://lorenagongang.com/">https://lorenagongang.com/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ada7757097a2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mcd-unison/twitter-sentiment-analysis-using-zookeeper-kafka-and-pyspark-live-streaming-on-windows-10-in-2022-ada7757097a2">Twitter sentiment analysis using Zookeeper, Kafka and PySpark live-streaming on Windows 10 in 2022</a> was originally published in <a href="https://medium.com/mcd-unison">MCD-UNISON</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Installing Hadoop 3.2.1 in Windows 10 + basic word count example]]></title>
            <link>https://medium.com/mcd-unison/installing-hadoop-3-2-1-in-windows-10-basic-text-analysis-4a5717c209ae?source=rss-6064ffd599af------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a5717c209ae</guid>
            <category><![CDATA[hadoop-installation]]></category>
            <category><![CDATA[windows-10]]></category>
            <category><![CDATA[word-count]]></category>
            <dc:creator><![CDATA[Hector Rodriguez Dominguez]]></dc:creator>
            <pubDate>Fri, 11 Feb 2022 16:41:33 GMT</pubDate>
            <atom:updated>2022-02-11T16:43:59.473Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/667/1*GEThvdX9A5mN7vwOS5aMjQ.png" /></figure><ul><li>This article will be part of a series of entries, exploring different data analysis tools and procedures, that aims to illustrate different technologies for data science.</li></ul><p>On this medium entry, we’ll talk about Hadoop, its way to install on Windows, some notes for the installation and making a basic text analysis.</p><h3><strong>What is Hadoop</strong></h3><p>According to AWS page:</p><blockquote>Apache Hadoop is an open source framework that is used to efficiently store and process large datasets ranging in size from gigabytes to petabytes of data. Instead of using one large computer to store and process the data, Hadoop allows clustering multiple computers to analyze massive datasets in parallel more quickly.</blockquote><h3>How it works</h3><p>Hadoop consists of four main modules:</p><ul><li>Hadoop Distributed File System (<a href="https://en.wikipedia.org/wiki/Apache_Hadoop#HDFS">HDFS</a>): HDFS provides a distributed file system with a better data throughput and support of large datasets, than traditional file systems.</li><li>Yet Another Resource Negotiator (<a href="https://yarnpkg.com/getting-started">YARN</a>) — Manages and monitors cluster nodes and resource usage.</li><li><a href="https://en.wikipedia.org/wiki/MapReduce">MapReduce </a>— A tool that helps programs do the parallel computation on data.</li><li>Hadoop Common — Java libraries that can be used on all modules.</li></ul><p>In Hadoop, applications place data into the Hadoop cluster, by connecting to the NameNode. The NameNode tracks the file directory structure and the placement of “chunks” for each file, replicated across DataNodes.</p><p>To query the data, Hadoop creates a MapReduce job, made up of map and reduce tasks that run against the data spread across the DataNodes. Map tasks run on each node against the input files supplied, and reducers run to aggregate and organize the final output.</p><h3>Installing Hadoop</h3><p>Now that we know the basics of Hadoop, we will proceed to install it on a Windows 10 environment.</p><p>Starting notes:</p><ul><li>The language of the OS in which Hadoop will be installed, should be english.</li></ul><h4>Prerequisites</h4><p>First, we should have some programs installed.</p><ul><li>Java 8 Runtime Environment (JRE): <a href="https://www.java.com/en/download/windows_offline.jsp">Download here</a></li><li>Java 8 Development Kit (JDK): <a href="https://www.oracle.com/java/technologies/downloads/#java8">Download here</a></li><li><a href="https://www.win-rar.com/download.html?&amp;L=6">WinRar </a>o <a href="https://www.7-zip.org/">7Zip</a></li></ul><h3>1. Downloading Hadoop binaries</h3><p>We should download Hadoop binaries from the official site: <a href="https://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-3.2.1/hadoop-3.2.1.tar.gz">Download here</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1PFxwXFt4wre_C8eQwy2Og.png" /><figcaption>Click the link to download the Hadoop</figcaption></figure><p>Once downloaded the file, notice the file has a “.tar” extension. We should extract the files by using Winrar o 7zip</p><p>The following folder will appear: “hadoop-3.2.1.tar”, copy it and paste it on your “C:” drive, so the route to the folder will be → “C:\hadoop-3.2.1”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*imZNrd5HIm_z5ktmJVOTsQ.png" /><figcaption>Hadoop folder on “C:”</figcaption></figure><p>Since we already extracted Hadoop folder on C:, now we should download the windows libraries for Hadoop from this GitHub repository: <a href="https://github.com/cdarlint/winutils/tree/master/hadoop-3.2.1/bin">https://github.com/cdarlint/winutils/tree/master/hadoop-3.2.1/bin</a> and copy them into the “C:\hadoop-3.2.1\bin” directory.</p><h3><strong>2. Configuring environment variables</strong></h3><p>After installing Hadoop, we should configure the environment variables to define Hadoop and Java paths.</p><p>To edit environment variables, go to Control Panel &gt; System and Security &gt; System (or right-click &gt; properties on My Computer icon) and click on the “Advanced system settings” link.</p><p>When the “Advanced system settings” dialog appears, go to the “Advanced” tab and click on the “Environment variables” button located on the bottom of the dialog.</p><p>In the “Environment Variables” dialog, press the “New” button to add a new variable.</p><p>Now we need to configure two variables:</p><ul><li>JAVA_HOME: This route should aim to the JDK installation folder</li></ul><p>Note: The default installation folder for java JDK is: “C:\Program Files\Java\jre1.8.0_321”, I suggest using the wildcard “Progra~1” to avoid problems with the blank space on “Program Files”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*EAulApWICJdL3CQEB9Yi9w.png" /></figure><ul><li>HADOOP_HOME: This should aim to the Hadoop folder we created on C:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*Y_E5r8JfhFrmwn7qtSFXWw.png" /></figure><p>Now, on the environment variable window, we need to select the path variable and click the “Edit” button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/1*AF4XuSrxG1ZM86yQUk5MEQ.png" /><figcaption>Environment variables</figcaption></figure><p>Once inside the path variable configuration window, click “new” button and add the following variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/246/1*Mld_iAMlGxk4scQ8iymw9w.png" /></figure><h3>3. Configuring Hadoop cluster</h3><p>Once we configured the environment variables, we should create two folders for the Name Node (Where all the master node data will be stored) and Data node (Where all data will be stored). So create the following folders on the specified routes:</p><ul><li>C:\Program_files\hadoop-3.2.1\data\dfs\datanode</li><li>C:\Program_files\hadoop-3.2.1\data\dfs\namenode</li></ul><p>Next, we must configure 4 internal files:</p><ul><li>1.- C:\Program_files\hadoop-3.2.1\etc\hadoop\hdfs-site.xml</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/1*u5j0OuquNNkJQS5QJMitow.png" /><figcaption>hdfs-site.xml configuration</figcaption></figure><blockquote>Note that the replication factor is set to 1 since we are creating a single node cluster.</blockquote><ul><li>2.- C:\Program_files\hadoop-3.2.1\etc\hadoop\core-site.xml</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*cnhTwSeAJDKUTVUH2OHYCQ.png" /><figcaption>core-site.xml configuration</figcaption></figure><ul><li>3.- C:\Program_files\hadoop-3.2.1\etc\hadoop\mapred-site.xml</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*1yx9oKbSTmrdNIRQ-sAhgw.png" /><figcaption>mapred-site.xml configuration</figcaption></figure><ul><li>4.- C:\Program_files\hadoop-3.2.1\etc\hadoop\yarn-site.xml</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*b6jwme2Q-dGc__J87C5T6A.png" /><figcaption>yarn-site.xml configuration</figcaption></figure><h3>4. Formatting the Name node</h3><p>After we finished configuration the above files, we execute the following command in order to format the name node:</p><blockquote>Note: We must open a command prompt as an administrator</blockquote><pre>hdfs namenode -format</pre><p>Due to a <a href="https://issues.apache.org/jira/browse/HDFS-14890">bug in the Hadoop 3.2.1 release</a>, you will receive an error:</p><blockquote>ERROR namenode.NameNode: Failed to start namenode.</blockquote><p>To fix this error, we’ll have to download files from this <a href="https://github.com/FahaoTang/big-data/blob/master/hadoop-hdfs-3.2.1.jar">link</a> and Copy the downloaded hadoop-hdfs-3.2.1.jar to folder “C:\Program_files\hadoop-3.2.1\share\hadoop\hdfs”.</p><p>Once we replaced the file mentioned above, we run again the command:</p><blockquote>Note: Remember to run the command prompt as administrator</blockquote><pre>hdfs namenode -format</pre><p>If the command prompt asks for authorization, just accept and type “Y”</p><p>5. Starting Hadoop</p><p>Finally, we have configured Hadoop, the only thing missing is to start the services, for that, we’ll need to navigate to “C:\Program_files\hadoop-3.2.1\sbin” and run the following command:</p><pre>.\start-dfs.cmd</pre><p>After running this command, two command prompts will pop up, one for the name node and the other for the data node. Next we will start the Yarn service with the folowing command:</p><pre>./start-yarn.cmd</pre><p>After running this command, two command prompts will pop up, one for the resource manager and the other for the node manager.</p><p>To verify everything is ok, we’ll run the following command:</p><pre>jps</pre><p>If everything is ok, we should get a similar message to the following:</p><p><em>19840 DataNode<br>14856 NodeManager<br>17288 Jps<br>15644 ResourceManager<br>5308 NameNode</em></p><p>Now we can access three pages</p><ul><li>1.- Name node web page: <a href="http://localhost:9870/dfshealth.html">http://localhost:9870/dfshealth.html</a></li><li>2.- Data node web page: <a href="http://localhost:9864/datanode.html">http://localhost:9864/datanode.html</a></li><li>3.- Yarn web page: <a href="http://localhost:8088/cluster">http://localhost:8088/cluster</a></li></ul><blockquote><em>NOTE: If after you end services and start again, the command prompt shows a problem on the \temp folder, open a command prompt as administrator and run the following command: (The problem is due to permissions of the Hadoop folders)</em></blockquote><pre><em>cacls hadoop-3.2.1 /t /p everyone:f</em></pre><h3>Basic word count example using Hadoop’s Mapreduce</h3><p>In this section, we’ll analyze the most common words in the novel “<a href="https://es.wikipedia.org/wiki/Don_Quijote_de_la_Mancha">Don Quijote de la Mancha</a>” by <a href="https://es.wikipedia.org/wiki/Miguel_de_Cervantes">Miguel de Cervantes Saavedra</a>.</p><p>For this purpose, we’ll need to download the novel as a plain text file, which you can find on this link: <a href="https://gist.githubusercontent.com/jsdario/6d6c69398cb0c73111e49f1218960f79/raw/8d4fc4548d437e2a7203a5aeeace5477f598827d/el_quijote.txt">https://gist.githubusercontent.com/jsdario/6d6c69398cb0c73111e49f1218960f79/raw/8d4fc4548d437e2a7203a5aeeace5477f598827d/el_quijote.txt</a></p><p>Once downloaded, move the file to “C:”.</p><p>Open a command prompt as administrator and run the following command to create an input and output folder on the Hadoop file system, to which we will be moving the novel for our analysis.</p><pre>hadoop fs -mkdir /input_dir<br>hadoop fs -mkdir /output_dir</pre><p>We can verify the folder we just created on the following link: <a href="http://localhost:9870/explorer.html#/">http://localhost:9870/explorer.html#/</a></p><p>Now let’s move the file “el_quijote.txt”, from C: to /input_dir</p><pre>hadoop fs -put C:/el_quijote.txt /input_dir</pre><p>We can verify the file has moved successfully on the following link:</p><p><a href="http://localhost:9870/explorer.html#/input_dir">http://localhost:9870/explorer.html#/input_dir</a></p><p>or with the following command line:</p><pre>hadoop fs -ls /input_dir</pre><p>For the next step, we’ll need the “hadoop-mapreduce-examples-3.2.1.jar”, found in “C:\hadoop-3.2.1\share\hadoop\mapreduce”.</p><p>We’ll execute the following command in order to call the function wordcount inside the mapreduce.</p><pre>hadoop jar C:\hadoop-3.2.1\share\hadoop\mapreduce\hadoop-mapreduce-examples-3.2.1.jar wordcount /input_dir/el_quijote.txt /output_dir</pre><p>Once the script is executed, the following message will appear on the command prompt</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*qbbri-GnzhCkiNW73sekaA.png" /><figcaption>Command prompt message</figcaption></figure><p>Now you can find the results of the wordcount on the output_dir we created.</p><p>Open up the output_dir folder inside the Hadoop interface on the following link: <a href="http://localhost:9870/explorer.html#/output_dir">http://localhost:9870/explorer.html#/output_dir</a> and click the file named “part-r-00000”<strong><em>(1)</em></strong>, and a small window will show up, click the “download”<strong><em>(2)</em></strong> option, and there you will be able to see the number of appearances for each word in the novel.</p><p><strong>Now we are done!</strong></p><h3>References</h3><ul><li><a href="http://www.ics.uci.edu/~shantas/Install_Hadoop-2.6.0_on_Windows10.pdf">http://www.ics.uci.edu/~shantas/Install_Hadoop-2.6.0_on_Windows10.pdf</a></li><li><a href="http://tecadmin.net/hadoop-running-a-wordcount-mapreduce-example">http://tecadmin.net/hadoop-running-a-wordcount-mapreduce-example</a></li><li><a href="https://muhammadbilalyar.github.io/blogs/How-to-Run-Hadoop-wordcount-MapReduce-Example-on-Windows-10/">https://muhammadbilalyar.github.io/blogs/How-to-Run-Hadoop-wordcount-MapReduce-Example-on-Windows-10/</a></li><li><a href="https://kontext.tech/column/hadoop/377/latest-hadoop-321-installation-on-windows-10-step-by-step-guide">https://kontext.tech/column/hadoop/377/latest-hadoop-321-installation-on-windows-10-step-by-step-guide</a></li><li><a href="https://towardsdatascience.com/installing-hadoop-3-2-1-single-node-cluster-on-windows-10-ac258dd48aef">https://towardsdatascience.com/installing-hadoop-3-2-1-single-node-cluster-on-windows-10-ac258dd48aef</a></li><li><a href="https://aws.amazon.com/es/emr/details/hadoop/what-is-hadoop/">https://aws.amazon.com/es/emr/details/hadoop/what-is-hadoop/</a></li><li><a href="https://en.wikipedia.org/wiki/Apache_Hadoop">https://en.wikipedia.org/wiki/Apache_Hadoop</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a5717c209ae" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mcd-unison/installing-hadoop-3-2-1-in-windows-10-basic-text-analysis-4a5717c209ae">Installing Hadoop 3.2.1 in Windows 10 + basic word count example</a> was originally published in <a href="https://medium.com/mcd-unison">MCD-UNISON</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Pushshift API for data analysis on Reddit]]></title>
            <link>https://medium.com/mcd-unison/using-pushshift-api-for-data-analysis-on-reddit-b08d339c48b8?source=rss-6064ffd599af------2</link>
            <guid isPermaLink="false">https://medium.com/p/b08d339c48b8</guid>
            <category><![CDATA[reddit]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[pushshift]]></category>
            <dc:creator><![CDATA[Hector Rodriguez Dominguez]]></dc:creator>
            <pubDate>Tue, 14 Sep 2021 03:46:51 GMT</pubDate>
            <atom:updated>2021-09-14T03:46:51.266Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qAnIkNDNjd2rpsFrWNAVwQ.jpeg" /><figcaption>Image credit(Reddit)</figcaption></figure><p><em>On this entry, we will learn how to mine, clean and analyze data from the social network Reddit, by using a python library named “Pushshift”.</em></p><h3>Introduction</h3><p>Let’s begin by defining what Reddit and Pushshift are…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/321/1*i4OhaNq6mE3AMHyWT9dwhw.png" /></figure><p><strong>Reddit: </strong>According to themselves, “<em>It’s a network of communities where people can dive into their interests, hobbies and passions. There’s a community for whatever you’re interested in</em>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*39APQycCSjz-bxKQjRbFlQ.jpeg" /></figure><p><strong>Pushshift</strong>: Is a social media data collection, analysis, and archiving platform that has collected Reddit data and made it available to researchers. Pushshift’s Reddit dataset is updated in real-time, and includes historical data.</p><p>Now that we have defined our tools of the trade, we can begin with the “good stuff”.</p><p>In the last year, the Covid-19 epidemic changed our life in every aspect, we started using more and more technology in our daily routines, people increased their usage of social networks creating a huge amount of data ready por the picking.</p><p>In this publication, we will learn how to gather information from Reddit using python libraries and we will see how to collect data from a particular topic or subreddit, how to explore and visualize it by plotting the dataframes, that will leave a bedrock for further data analysis.</p><h3>Setting up the environment</h3><p>Reddit is organized in communities called <em>subreddits</em>. Each subreddit is filled of submissions posted by users. Each submission can be commented by other users and can be upvoted or downvote.</p><p>In order to analyze Reddit, we need to access all of its submissions, comments and users’ information. To do this, we’ll use an API called “pushshift”.</p><p>To setup our environment, first we need to install some tools. So open up a terminal and type the following commands:</p><pre>pip install psaw                 #Installing pushshift library</pre><p>Next, we need to import some tools. So, open up a python script and type the following commands:</p><pre>from psaw import PushshiftAPI    #library Pushshift<br>import datetime as dt            #library for date management<br>import p                         #library for data manipulation<br>import matplotlib.pyplot as plt  #library for plotting</pre><p>Once our tools are installed and imported, on the python script, we need to create an instance of the pushshift API.</p><pre>api = PushshiftAPI()              #Object of the API</pre><p>Now that we are all set up! We’ll explore two of the main methods of the API.</p><p>“<strong>search_submissions</strong>”: Which will return a list of posts from the selected subreddit.</p><p>“<strong>search_comments</strong>”: Which will return a list of comments associated with a search term.</p><p>This methods use some parameters in common to filter the search results:</p><p>“<strong>after</strong>”: Lower date limit for search</p><p>“<strong>before</strong>”: Upper date limit for search</p><p>“<strong>filter</strong>”: Column names we want to retrieve (suggested: ‘id’, ‘author’, ‘created_utc’, ‘domain’, ‘url’, ‘title’, ‘num_comments’)</p><p>“<strong>limit</strong>”: Number of rows to return</p><p>“<strong>subreddit</strong>”: The name of the subreddit to crawl (case sensitive)</p><p>“<strong>q</strong>”: Search term.</p><p>We have defined the parameters for the functions, so we will add the following code to our script.</p><pre>&quot;&quot;&quot;FOR POSTS&quot;&quot;&quot;<br>def data_prep_posts(subreddit, start_time, end_time, filters, limit):<br>    if(len(filters) == 0):<br>        filters = [&#39;id&#39;, &#39;author&#39;, &#39;created_utc&#39;,<br>                   &#39;domain&#39;, &#39;url&#39;,<br>                   &#39;title&#39;, &#39;num_comments&#39;]                 <br>                   #We set by default some useful columns<br><br>    posts = list(api.search_submissions(<br>        subreddit=subreddit,   #Subreddit we want to audit<br>        after=start_time,      #Start date<br>        before=end_time,       #End date<br>        filter=filters,        #Column names we want to retrieve<br>        limit=limit))          ##Max number of posts<br><br>    return pd.DataFrame(posts) #Return dataframe for analysis<br><br><br>&quot;&quot;&quot;FOR COMMENTS&quot;&quot;&quot;<br>def data_prep_comments(term, start_time, end_time, filters, limit):<br>    if (len(filters) == 0):<br>        filters = [&#39;id&#39;, &#39;author&#39;, &#39;created_utc&#39;,<br>                   &#39;body&#39;, &#39;permalink&#39;, &#39;subreddit&#39;]<br>                   #We set by default some usefull columns <br><br>    comments = list(api.search_comments(<br>        q=term,                 #Subreddit we want to audit<br>        after=start_time,       #Start date<br>        before=end_time,        #End date<br>        filter=filters,         #Column names we want to retrieve<br>        limit=limit))           #Max number of comments<br>    return pd.DataFrame(comments) #Return dataframe for analysis</pre><p>Let’s create our main function, where we will define our parameters and do some data cleaning before we can start visualizing the data on plots.</p><pre>def main():<br>    subreddit = &quot;Darkestdungeon&quot;     #Subreddit we are auditing<br>    start_time = int(dt.datetime(2021, 1, 1).timestamp())  <br>                                     #Starting date for our search<br>    end_time = int(dt.datetime(2021, 1, 31).timestamp())   <br>                                     #Ending date for our search<br>    filters = []                     #We don´t want specific filters<br>    limit = 1000                     #Elelemts we want to recieve<br><br>    &quot;&quot;&quot;Here we are going to get subreddits for a brief analysis&quot;&quot;&quot;</pre><pre>    #Call function for dataframe creation of comments<br>    df_p = data_prep_posts(subreddit,start_time,<br>                         end_time,filters,limit) <br><br>    #Drop the column on timestamp format<br>    df_p[&#39;datetime&#39;] = df_p[&#39;created_utc&#39;].map(<br>        lambda t: dt.datetime.fromtimestamp(t))<br>    df_p = df_p.drop(&#39;created_utc&#39;, axis=1) <br>    #Sort the Row by datetime               <br>    df_p = df_p.sort_values(by=&#39;datetime&#39;)  <br>    #Convert timestamp format to datetime for data analysis               <br>    df_p[&quot;datetime&quot;] = pd.to_datetime(df_p[&quot;datetime&quot;])</pre><pre>&quot;&quot;&quot;Here we are going to get comments for a brief analysis&quot;&quot;&quot;<br>    term = &#39;bitcoin&#39;            #Term we want to search for<br>    limit = 10                  #Number of elelemts <br>    df_c = data_prep_comments(term, start_time,<br>                     end_time, filters, limit)<br>                                #Call function for dataframe creation of comments</pre><p>Now that we have created our dataframe <em>df_p and df_c </em>and converted the timestamp to a datetime format, we can start creating functions for different purposes and add them to our main function, and run them. I’ll add some screenshots of the resulting graphs.</p><p><strong>Posts per day on a specific subreddit</strong></p><pre>def count_posts_per_date(df_p, title, xlabel, ylabel):<br>    df_p.groupby([df_p.datetime.dt.date]).count().plot(y=&#39;id&#39;, rot=45, kind=&#39;bar&#39;, label=&#39;Posts&#39;)<br>    plt.xlabel(xlabel)<br>    plt.ylabel(ylabel)<br>    plt.title(title)<br>    plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q2F52iRz-dqqgUmMog_OFg.png" /></figure><p><strong>Mean of comments on a specific subreddit</strong></p><pre>def mean_comments_per_date(df_p, title, xlabel, ylabel):<br>       df_p.groupby([df_p.datetime.dt.date]).mean().plot(y=&#39;num_comments&#39;, rot=45, kind=&#39;line&#39;, label=&#39;Comments&#39;)<br>    plt.xlabel(xlabel)<br>    plt.ylabel(ylabel)<br>    plt.title(title)<br>    plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LleRgNrUI81CQErJgV2AeA.png" /></figure><p><strong>Most active users on a specific subreddit</strong></p><pre>def most_active_author(df_p, title, xlabel, ylabel, limit):<br>    df_p.groupby([df_p.author]).count()[&#39;id&#39;].nlargest(limit).sort_values(ascending=True).plot(y=&#39;id&#39;, rot=45, kind=&#39;barh&#39;, label=&#39;Users&#39;)<br>    plt.xlabel(xlabel)<br>    plt.ylabel(ylabel)<br>    plt.title(title)<br>    plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RuNJvl3_PZztXVUx9a19vw.png" /></figure><p><strong>Origin of crosspostings</strong></p><pre>def get_posts_orign(df_p, title, xlabel, ylabel, limit, subreddit):<br>    domains = df_p[(df_p.domain != &#39;reddit.com&#39;) &amp; (df_p.domain != f&#39;self.{subreddit}&#39;) &amp; (df_p.domain != &#39;i.redd.it&#39;)]<br>    domains.groupby(by=&#39;domain&#39;).count()[&#39;id&#39;].nlargest(limit).sort_values(ascending=True).plot(kind=&#39;barh&#39;, rot=45, x=&#39;domain&#39;, label=&#39;# of posts&#39;, legend=True, figsize=(8,13))<br>    plt.xlabel(xlabel)<br>    plt.ylabel(ylabel)<br>    plt.title(title)<br>    plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NsSjuQaQe0YnQfvN2M15Sg.png" /></figure><p><strong>Most active subreddit according to search term</strong></p><pre>def get_subreddits(df_p, title, xlabel, ylabel, limit):<br>    df_p.groupby(by=&#39;subreddit&#39;).count()[&#39;id&#39;].nlargest(limit).sort_values(ascending=True).plot(kind=&#39;barh&#39;, x=&#39;subreddit&#39;, label=&#39;Subreddit&#39;, legend=True, figsize=(8,13))<br>    plt.xlabel(xlabel)<br>    plt.ylabel(ylabel)<br>    plt.title(title)<br>    plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*84Vy01N3vARgex78Fvj-Og.png" /></figure><p>Well, we are almost at the end of this brief tutorial. The only thing left to show you is the full script in case there are still some questions.</p><p><strong>Full Script</strong></p><pre><em>&quot;&quot;&quot;<br>BEGIN - Script preparation<br>Section for importing libraries and setting up basic environment configurations<br>&quot;&quot;&quot;<br></em>from psaw import PushshiftAPI                               #Importing wrapper library for reddit(Pushshift)<br>import datetime as dt                                       #Importing library for date management<br>import pandas as pd                                         #Importing library for data manipulation in python<br>import matplotlib.pyplot as plt                             #Importing library for creating interactive visualizations in Python<br>from pprint import pprint                                   #Importing for displaying lists in the &quot;pretty&quot; way (Not required)<br><br>pd.set_option(&quot;display.max_columns&quot;, None)                  #Configuration for pandas to show all columns on dataframe<br>api = PushshiftAPI()                                        #We create an object of the API<br>&quot;&quot;&quot;<br>END - Script preparation <br>&quot;&quot;&quot;<br><br><br><br>&quot;&quot;&quot;<br>BEGIN - DATAFRAME GENERATION FUNCTIONS<br><br>Here we are going to make a request to through the API<br>to the selected subreddit and the results are going <br>to be placed inside a pandas dataframe<br>&quot;&quot;&quot;<br><br>&quot;&quot;&quot;FOR POSTS&quot;&quot;&quot;<br>def data_prep_posts(subreddit, start_time, end_time, filters, limit):<br>    if(len(filters) == 0):<br>        filters = [&#39;id&#39;, &#39;author&#39;, &#39;created_utc&#39;,<br>                   &#39;domain&#39;, &#39;url&#39;,<br>                   &#39;title&#39;, &#39;num_comments&#39;]                 #We set by default some columns that will be useful for data analysis<br><br>    posts = list(api.search_submissions(<br>        subreddit=subreddit,                                #We set the subreddit we want to audit<br>        after=start_time,                                   #Start date<br>        before=end_time,                                    #End date<br>        filter=filters,                                     #Column names we want to get from reddit<br>        limit=limit))                                       #Max number of posts we wanto to recieve<br><br>    return pd.DataFrame(posts)                              #Return dataframe for analysis<br><br><br>&quot;&quot;&quot;FOR COMMENTS&quot;&quot;&quot;<br>def data_prep_comments(term, start_time, end_time, filters, limit):<br>    if (len(filters) == 0):<br>        filters = [&#39;id&#39;, &#39;author&#39;, &#39;created_utc&#39;,<br>                   &#39;body&#39;, &#39;permalink&#39;, &#39;subreddit&#39;]        #We set by default some columns that will be useful for data analysis<br><br>    comments = list(api.search_comments(<br>        q=term,                                             #We set the subreddit we want to audit<br>        after=start_time,                                   #Start date<br>        before=end_time,                                    #End date<br>        filter=filters,                                     #Column names we want to get from reddit<br>        limit=limit))                                       #Max number of comments we wanto to recieve<br>    return pd.DataFrame(comments)                           #Return dataframe for analysis<br><br>&quot;&quot;&quot;<br>END - DATAFRAME GENERATION FUNCTIONS<br>&quot;&quot;&quot;<br><br><br><br>&quot;&quot;&quot;<br>BEGIN - FUNCTIONS<br>&quot;&quot;&quot;<br>###Function to plot the number of posts per day on the specified subreddit<br>def count_posts_per_date(df_p, title, xlabel, ylabel):<br>    df_p.groupby([df_p.datetime.dt.date]).count().plot(y=&#39;id&#39;, rot=45, kind=&#39;bar&#39;, label=&#39;Posts&#39;)<br>    plt.xlabel(xlabel)<br>    plt.ylabel(ylabel)<br>    plt.title(title)<br>    plt.show()<br><br>###Function to plot the mean of comments per day on the specified subreddit<br>def mean_comments_per_date(df_p, title, xlabel, ylabel):<br>    df_p.groupby([df_p.datetime.dt.date]).mean().plot(y=&#39;num_comments&#39;, rot=45, kind=&#39;line&#39;, label=&#39;Comments&#39;)<br>    plt.xlabel(xlabel)<br>    plt.ylabel(ylabel)<br>    plt.title(title)<br>    plt.show()<br><br>###Function to plot the most active users on the subreddit<br>def most_active_author(df_p, title, xlabel, ylabel, limit):<br>    df_p.groupby([df_p.author]).count()[&#39;id&#39;].nlargest(limit).sort_values(ascending=True).plot(y=&#39;id&#39;, rot=45, kind=&#39;barh&#39;, label=&#39;Users&#39;)<br>    plt.xlabel(xlabel)<br>    plt.ylabel(ylabel)<br>    plt.title(title)<br>    plt.show()<br><br>###Function to que the orgin form the crossposting<br>def get_posts_orign(df_p, title, xlabel, ylabel, limit, subreddit):<br>    domains = df_p[(df_p.domain != &#39;reddit.com&#39;) &amp; (df_p.domain != f&#39;self.{subreddit}&#39;) &amp; (df_p.domain != &#39;i.redd.it&#39;)]<br>    domains.groupby(by=&#39;domain&#39;).count()[&#39;id&#39;].nlargest(limit).sort_values(ascending=True).plot(kind=&#39;barh&#39;, rot=45, x=&#39;domain&#39;, label=&#39;# of posts&#39;, legend=True, figsize=(8,13))<br>    plt.xlabel(xlabel)<br>    plt.ylabel(ylabel)<br>    plt.title(title)<br>    plt.show()<br><br>###Gets most active subrredits according to search term<br>def get_subreddits(df_p, title, xlabel, ylabel, limit):<br>    df_p.groupby(by=&#39;subreddit&#39;).count()[&#39;id&#39;].nlargest(limit).sort_values(ascending=True).plot(kind=&#39;barh&#39;, x=&#39;subreddit&#39;, label=&#39;Subreddit&#39;, legend=True, figsize=(8,13))<br>    plt.xlabel(xlabel)<br>    plt.ylabel(ylabel)<br>    plt.title(title)<br>    plt.show()<br><br><br>&quot;&quot;&quot;<br>END - FUNCTIONS<br>&quot;&quot;&quot;<br><br><br>def main():<br>    subreddit = &quot;darkestdungeon&quot;                           #Name of the subreddit we are auditing<br>    start_time = int(dt.datetime(2021, 1, 1).timestamp())  #We define the starting date for our search<br>    end_time = int(dt.datetime(2021, 1, 31).timestamp())   #We define the ending date for our search<br>    filters = []                                           #We don´t want specific filters<br>    limit = 1000                                           #Number of elelemts we want to recieve<br><br>    &quot;&quot;&quot;Here we are going to get subreddits for a brief analysis&quot;&quot;&quot;<br>    df_p = data_prep_posts(subreddit,start_time,<br>                         end_time,filters,limit)           #Call function for dataframe creation of comments<br><br>    df_p[&#39;datetime&#39;] = df_p[&#39;created_utc&#39;].map(<br>        lambda t: dt.datetime.fromtimestamp(t))<br>    df_p = df_p.drop(&#39;created_utc&#39;, axis=1)                #Drop the column on timestamp format<br>    df_p = df_p.sort_values(by=&#39;datetime&#39;)                 #Sort the Row by datetime<br>    df_p[&quot;datetime&quot;] = pd.to_datetime(df_p[&quot;datetime&quot;])    #Convert timestamp format to datetime for data analysis<br><br><br>    df_p.to_csv(f&#39;dataset_{subreddit}_posts.csv&#39;, sep=&#39;,&#39;, # Save the dataset on a csv file for future analysis<br>                header=True, index=False, columns=[<br>            &#39;id&#39;, &#39;author&#39;, &#39;datetime&#39;, &#39;domain&#39;,<br>            &#39;url&#39;, &#39;title&#39;, &#39;num_comments&#39;<br>        ])<br><br>    count_posts_per_date(df_p, &#39;Post per day&#39;, &#39;Days&#39;,     #Function to plot the number of posts per day on the specified subreddit<br>                         &#39;posts&#39;)<br>    mean_comments_per_date(df_p,                           #Function to plot the mean of comments per day on the specified subreddit<br>                           &#39;Average comments per day&#39;,<br>                           &#39;Days&#39;, &#39;comments&#39;)<br>    most_active_author(df_p, &#39;Most active users&#39;,          #Function to plot the most active users on the subreddit<br>                       &#39;Posts&#39;, &#39;Users&#39;, 10)<br>    get_posts_orign(df_p, &#39;Origin of crosspostings&#39;,       #Function to que the orgin form the crossposting<br>                    &#39;Crossposts&#39;, &#39;Origins&#39;, 10,<br>                    subreddit)<br><br>    &quot;&quot;&quot;Here we are going to get comments for a brief analysis&quot;&quot;&quot;<br>    term = &#39;bitcoin&#39;                                        #Term we want to search for<br>    limit = 10                                              #Number of elelemts we want to recieve<br>    df_c = data_prep_comments(term, start_time,             #Call function for dataframe creation of comments<br>                         end_time, filters, limit)<br><br>    get_subreddits(df_c, &#39;Most active subreddit&#39;, &#39;Posts&#39;,  #Gets most active subrredits according to search term<br>                   &#39;Subreddits&#39;, 10)<br><br><br><br><br>if __name__== &quot;__main__&quot; : main()</pre><p>Also, you can download it from <a href="https://github.com/Silvertongue26/reddit_api/blob/main/reddit6.py">https://github.com/Silvertongue26/reddit_api/blob/main/reddit6.py</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b08d339c48b8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mcd-unison/using-pushshift-api-for-data-analysis-on-reddit-b08d339c48b8">Using Pushshift API for data analysis on Reddit</a> was originally published in <a href="https://medium.com/mcd-unison">MCD-UNISON</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>