<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tonya Chernyshova on Medium]]></title>
        <description><![CDATA[Stories by Tonya Chernyshova on Medium]]></description>
        <link>https://medium.com/@tonya-chernyshova?source=rss-3b1e1ccfeed6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*V04RqlWfroZskUyYJNvgUQ.jpeg</url>
            <title>Stories by Tonya Chernyshova on Medium</title>
            <link>https://medium.com/@tonya-chernyshova?source=rss-3b1e1ccfeed6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 03:05:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tonya-chernyshova/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Beyond Airflow: 6 Great Alternatives for Data Orchestration]]></title>
            <link>https://blog.dataengineerthings.org/beyond-airflow-6-great-alternatives-for-data-orchestration-537ab0696ae3?source=rss-3b1e1ccfeed6------2</link>
            <guid isPermaLink="false">https://medium.com/p/537ab0696ae3</guid>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Tonya Chernyshova]]></dc:creator>
            <pubDate>Tue, 11 Mar 2025 18:47:07 GMT</pubDate>
            <atom:updated>2025-03-11T18:47:07.072Z</atom:updated>
            <content:encoded><![CDATA[<p>Data orchestration plays a pivotal role in modern data engineering, empowering teams to automate and optimize their data workflows. While <strong>Apache Airflow</strong> has long been a dominant player due to its flexibility and vibrant community, there are several other tools available that offer distinctive features and can better align with specific project needs.</p><p>In this post, we’ll explore six compelling alternatives to Apache Airflow for managing your data workflows. These tools span a range of use cases and provide unique capabilities suited for data engineering, machine learning workflows, and beyond. By examining these platforms, you’ll be better equipped to choose the right orchestration tool for your needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u-r-pyLoaO8O3XhK2-2K1A.jpeg" /><figcaption>Generated by <a href="https://ideogram.ai/">https://ideogram.ai/</a></figcaption></figure><h3>1. Flyte</h3><p><a href="https://flyte.org/">Flyte</a> is an open-source platform specifically designed for orchestrating complex workflows, with a strong emphasis on scalability and cloud-native operations. It’s particularly well-suited for machine learning pipelines and data engineering tasks that require large-scale execution.</p><p>Flyte integrates seamlessly with Kubernetes and major cloud providers, enabling teams to scale their workflows with ease. It offers features like versioned workflows, data lineage, and robust monitoring, making it ideal for teams managing sophisticated workflows across various environments.</p><p><strong>Key Features:</strong></p><ul><li>Kubernetes-first and cloud-native design</li><li>Scalable infrastructure for large data workflows</li><li>Built-in data lineage, versioning, and monitoring</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R6VqAIvxsa2IUot1f5HVjA.png" /><figcaption>Flyte UI (<a href="https://docs.flyte.org/">https://docs.flyte.org/</a>)</figcaption></figure><h3>2. Prefect</h3><p><a href="https://www.prefect.io/">Prefect</a> is an open-source tool that simplifies the process of building and managing workflows. Its hybrid execution model allows workflows to run both in the cloud and on-premises, providing users greater flexibility and control over their infrastructure.</p><p>With Prefect, teams benefit from an intuitive UI that makes monitoring and debugging workflows easier, alongside a powerful set of APIs for deeper integration. It is especially popular for data pipelines that require clear observability and robust error handling.</p><p><strong>Key Features:</strong></p><ul><li>Hybrid execution model (cloud or on-prem)</li><li>Intuitive user interface for real-time monitoring</li><li>Rich API support for integration and flexibility</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s_Vph6rkCYDuQ8iGnwv1ng.png" /><figcaption>Prefect UI (<a href="https://orion-docs.prefect.io/">https://orion-docs.prefect.io/</a>)</figcaption></figure><h3>3. Dagster</h3><p><a href="https://dagster.io/">Dagster</a> is an open-source data orchestrator focused on simplifying the lifecycle management of data assets. Its Python-centric approach allows users to define their data assets as functions and manage them through schedules or event triggers.</p><p>Dagster is designed for cloud-native environments and emphasizes data lineage and observability, making it an excellent choice for teams building complex data workflows. It also offers a user-friendly development environment that accelerates debugging and testing.</p><p><strong>Key Features:</strong></p><ul><li>Integrated data lineage and observability</li><li>Python-based asset definition</li><li>Cloud-native and scalable architecture</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3VFji_RH2IKSbANj4Stczg.png" /><figcaption>Dagster UI (<a href="https://dagster.io/">https://dagster.io/</a>)</figcaption></figure><h3>4. Mage AI</h3><p><a href="https://www.mage.ai/">Mage AI</a> is a flexible and user-friendly platform for building data transformation pipelines. Combining the flexibility of interactive notebooks with the power of modular code, Mage AI streamlines the process of data extraction, transformation, and loading (ETL).</p><p>Mage AI also includes built-in tools for monitoring, alerting, and real-time observability, making it ideal for teams managing large-scale data pipelines. With dbt integration, it provides an easy way to manage data models in your workflows.</p><p><strong>Key Features:</strong></p><ul><li>Hybrid framework with notebooks and modular code</li><li>Real-time monitoring and alerting</li><li>dbt integration for streamlined data transformation</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uutg_tNd6w452RuSE_ipLQ.png" /><figcaption>Mage AI UI <a href="https://www.mage.ai/">https://www.mage.ai/</a></figcaption></figure><h3>5. Kedro</h3><p><a href="https://kedro.org/">Kedro</a> is a Python-based framework focused on building reproducible, maintainable, and modular data and machine learning pipelines. It follows software engineering best practices, ensuring that projects are structured in a way that makes them easy to test, maintain, and scale.</p><p>Kedro’s standardized project templates and coding conventions help maintain consistency across teams, and it provides tools for version control, dependency management, and flexible deployment, making it an excellent choice for both data engineers and data scientists.</p><p><strong>Key Features:</strong></p><ul><li>Standardized templates for consistent project structure</li><li>Strong version control and dependency management</li><li>Flexible deployment across different platforms</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OFp2SEJ3ziMX14Y8OvTWuw.png" /><figcaption>Kedro UI <a href="https://kedro.org/">https://kedro.org/</a></figcaption></figure><h3>6. Luigi</h3><p><a href="https://luigi.readthedocs.io/en/stable/">Luigi</a>, developed by Spotify, is a Python framework designed for building complex batch pipelines with many interdependent tasks. Its focus is on reliability and scalability, allowing teams to manage intricate workflows that involve multiple data processing tasks.</p><p>Luigi handles dependency resolution, task scheduling, and visualization, ensuring tasks are executed in the correct order. It’s a great option for teams that work with Hadoop, Python scripts, or other batch processes.</p><p><strong>Key Features:</strong></p><ul><li>Dependency resolution and task scheduling</li><li>Scalable infrastructure for batch job pipelines</li><li>Built-in task visualization</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*exsKLDynrnKe_7pZluliLw.png" /><figcaption>Visualiser page <a href="https://luigi.readthedocs.io/">https://luigi.readthedocs.io/</a></figcaption></figure><h3>Conclusion</h3><p>While <strong>Apache Airflow</strong> remains one of the most widely used tools for data orchestration, these six alternatives each offer unique advantages that may better suit your specific use case. Whether you’re managing machine learning pipelines, dealing with batch jobs, or looking for flexibility in cloud environments, there’s a tool on this list that can meet your needs.</p><p>By understanding the strengths and capabilities of these platforms, you can select the best orchestration solution to optimize your data workflows and enhance the efficiency of your data engineering and machine learning operations.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=537ab0696ae3" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/beyond-airflow-6-great-alternatives-for-data-orchestration-537ab0696ae3">Beyond Airflow: 6 Great Alternatives for Data Orchestration</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[K-means clustering from scratch in Python implementation]]></title>
            <link>https://tonya-chernyshova.medium.com/k-means-clustering-from-scratch-in-python-implementation-b8ce69c37800?source=rss-3b1e1ccfeed6------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8ce69c37800</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[clustering]]></category>
            <dc:creator><![CDATA[Tonya Chernyshova]]></dc:creator>
            <pubDate>Fri, 13 Jan 2023 20:44:13 GMT</pubDate>
            <atom:updated>2023-01-13T20:44:13.911Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LJicaHqe9urGjIKod0MYGg.png" /><figcaption>Image by the author</figcaption></figure><p>Nowadays there are many libraries and frameworks available that make it easier for data scientists and machine learning developers to solve complex problems without having to write everything from scratch. These libraries and frameworks provide pre-built algorithms and functions working like magic, saving time and effort. However, writing a machine learning algorithm from scratch can still be an extremely rewarding learning experience, as it allows for a deeper understanding of how the algorithm works and how it can be implemented. Additionally, writing an algorithm from scratch can give developers more control over the details of the implementation and allow for customization to fit specific needs. While it may require more effort and time, the learning experience and increased control can be well worth it.</p><p>In this post, I’ll walk you through the <strong>k-means</strong> algorithm step-by-step. To understand how the libraries we use work under the hood we’ll develop the code for the algorithm from scratch using Python.</p><p><strong>K-means algorithm</strong> is one of the simplest and most popular unsupervised machine learning algorithms that is used for clustering. It is called unsupervised because it does not require a target variable, as is the case in supervised learning. Instead, the algorithm works by grouping similar data points together into clusters based on their feature values. The k-means algorithm has a variety of applications, including customer segmentation, image segmentation, and finding patterns in data. It is a simple and effective algorithm that is widely used in many fields.</p><p><strong>K-means clustering</strong> begins with <strong>k</strong> randomly placed <strong>centroids</strong> (points in space that represent the center of the cluster) and assigns every item to the nearest one. After the assignment, the centroids are moved to the average location of all the nodes assigned to them, and the assignments are redone. This process repeats until the assignments stop changing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y1tbAW0YgO7v1EO3LO6Q9A.png" /><figcaption>Image by the author</figcaption></figure><p><strong>K-means cycle:</strong></p><ol><li>Pick a value for k: The first step in running the k-means algorithm is to choose the number of clusters you want to create (k). This value should be chosen based on the structure of the data and the goals of your analysis.</li><li>Initialize centroids: The next step is to initialize the centroids. This is typically done by selecting k random points from the dataset as the initial centroids.</li><li>Create clusters: The algorithm then assigns each data point to the nearest centroid, creating k clusters.</li><li>Update centroids: The centroids are then updated to the average location of all the data points assigned to them.</li><li>Repeat until convergence: Steps 3 and 4 are repeated until the assignments of the data points to the centroids stop changing. At this point, the algorithm has converged and the final clusters have been determined.</li></ol><p>Let’s implement this logic <strong>in the code</strong>. The first step is to generate a dataset.</p><pre># Import packages<br>from copy import deepcopy<br>import numpy as np<br>import pandas as pd<br>import plotly.express as px<br>import plotly.graph_objects as go<br>import kaleido<br>from sklearn.datasets import make_blobs<br>import imageio<br><br># Generate Dataset<br>X, y = make_blobs(<br>    n_samples=2000, <br>    n_features=2, <br>    centers=3, <br>    cluster_std=1.35, <br>    random_state=11<br>)<br># Put Dataset to DataFrame<br>data = pd.DataFrame(dict(x=X[:,0], y=X[:,1], label=y))<br># Pring data size and the first 3 rows<br>print(&#39;Data size&#39;)<br>print(data.shape)<br>data.head(3)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*zo1eWoV_Tmnbb7pgT6_yoA.png" /><figcaption>Output generated by the code above</figcaption></figure><p>For visualization I’m using Plotly library, I’m defining a function that I’m going to use for all future plots.</p><pre># Define Plot function<br><br>def plot_data(df, centroids_list=None, labels=False, show=True, save=True):<br>    if labels:<br>        color_plot = df[&#39;label&#39;]<br>    else:<br>        color_plot = &#39;#FFB266&#39;<br>    fig = go.Figure(data=go.Scatter(<br>                x=df[&#39;x&#39;],<br>                y=df[&#39;y&#39;],<br>                mode=&#39;markers&#39;,<br>                marker=dict(<br>                     color=color_plot,<br>                     colorscale = &#39;Rainbow&#39;,<br>                     size=10,<br>                     line=dict(<br>                        color=&#39;DarkSlateGrey&#39;,<br>                        width=1<br>                      )<br>               ), showlegend=False<br>        ))<br>    fig.update_layout(<br>    autosize=False,<br>    width=700,<br>    height=600)<br>    <br>    # plot Centroids<br>    if not centroids_list is None:<br>        fig.add_trace(go.Scatter(x=[i[0] for i in centroids_list], <br>                         y=[i[1] for i in centroids_list], mode=&#39;markers&#39;,marker=dict(color=&#39;#100101&#39;,size=14,<br>                                                                                      symbol = &#39;x&#39;),<br>        showlegend=False))<br>    <br>    ## Save files for GIF generation<br>    if save:<br>        try:<br>            fig.write_image(f&#39;./images_kmeans/img_{t}.png&#39;,engine=&#39;orca&#39;)<br>        except NameError:<br>            pass<br>    <br>    # Parameter for displaying files    <br>    if show:<br>        fig.show()</pre><p>First, let’s plot the generated dataset:</p><pre>plot_data(data,centroids_list=None, labels=False, show=True, save=False)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QhxTYQW_y1KwIchiZAEyPA.png" /><figcaption>The plot generated by Plotly Python library</figcaption></figure><p>Selecting the number of clusters and initializing the centroids are the first two steps in the k-means algorithm. I picked <strong><em>k=3</em> </strong>and used a random function to place <strong><em>3 initial centroids.</em></strong></p><pre># K value, number of clusters<br>k = 3<br># Coordinates for random initial centroids <br>centroidX = np.random.randint(data.x.min(), data.x.max(), size=k)<br>centroidY = np.random.randint(data.y.min(), data.y.max(), size=k)<br>centroids_list = np.array(list(zip(centroidX, centroidY)), dtype=np.float32)<br>print(&#39;Initial Centroids&#39;)<br>print(centroids_list)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/394/1*ZWthBt8u7in_F5q8yUQkzw.png" /><figcaption>Output generated by the code above</figcaption></figure><p>Now let’s plot initials centroids:</p><pre>plot_data(data, centroids_list, labels=False, show=True, save=False)</pre><figure><img alt="The plot generated by Plotly Python library" src="https://cdn-images-1.medium.com/max/1024/1*dkLKsl7jE5FZaKeNdeF7Qw.png" /><figcaption>The plot generated by Plotly Python library</figcaption></figure><p>In the k-means algorithm, the Euclidean distance is often used to measure the distance between data points and centroids. The Euclidean distance between two points in n-dimensional space is defined as the square root of the sum of the squares of the differences between the coordinates of the points:</p><blockquote>distance = sqrt((x1 — x2)² + (y1 — y2)² + … + (zn — zn)²)</blockquote><p>This distance measure is used to determine which centroid is closest to each data point, and the data point is then assigned to that centroid.</p><p>Let’s create a function to use the distance between centroids and each data point:</p><pre># Euclidean Distance calculation function<br>def distance(a, b, ax=1):<br>    return np.linalg.norm(a - b, axis=ax)</pre><p>It’s time to start implementing the algorithm. At first, small data preparation is needed:</p><pre># Data prep<br>x = data[&#39;x&#39;].values<br>y = data[&#39;y&#39;].values<br>dt = np.array(list(zip(x, y)))</pre><p>Create lists to store centroid values and clusters labels:</p><pre># List for storing old values of centroids when they change location, fill with 0s<br>centroids_list_old = np.zeros(centroids_list.shape)<br># Create an empty list for each element of clusters labels, exp. (1,0,2,1,2,0...2,1)<br>clusters = np.zeros(len(dt))</pre><p>Calculate the distance between new centroids and old centroids, we assume that centroids are found when <em>error = 0</em></p><pre># Distance between new centroids and old centroids, we&#39;ll find our centroid when error = 0 <br>error = distance(centroids_list,centroids_list_old, None)</pre><p>So, while the error is not equal to 0, we assign a centroid to every point in the input dataset. Then centroids are moved to the center of their clusters.<br>I use a function defined above to save a plot at each iteration of the loop, then I combine all of the plots into an attractively animated GIF.</p><pre>#for GIF generation<br>t=0<br>frames = []<br><br>while error != 0:<br>    #for each point find centroid<br>    for i in range(len(dt)):<br>        distances =  distance(dt[i], centroids_list)<br>        cluster = np.argmin(distances) #returns index of min value<br>        clusters[i] = cluster # ger cluster for i element<br>    #copy current centroid list to &#39;old&#39; list, for the first loop = centroids_list -&gt; random, old -&gt; empty<br>    centroids_list_old = deepcopy(centroids_list) <br>    <br>    #recalculate centroids <br>    for i in range(k):<br>        points = [dt[x] for x in range(len(dt)) if clusters[x] == i]<br>        centroids_list[i] = np.mean(points, axis=0)<br>    error = distance(centroids_list, centroids_list_old, None)<br>    #plot<br>    df_plot = pd.DataFrame()<br>    for i in range(k):<br>        points = [np.append(dt[j],i) for j in range(len(dt)) if clusters[j] == i]<br>        df_temp = pd.DataFrame(points, columns=[&#39;x&#39;,&#39;y&#39;,&#39;label&#39;])<br>        df_plot = pd.concat([df_plot,df_temp])<br>     <br>    plot_data(df_plot, centroids_list, labels=True, show=False, save=True)<br>    t += 1</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*K-CUnHkeXkwcO6EUrPZROw.gif" /><figcaption>GIF generated by imageio library</figcaption></figure><p>Below is the code to generate GIF using imageio library:</p><pre># Gif generation<br>frames = []<br>for t in range(t):<br>    image = imageio.v2.imread(f&#39;./images_kmeans/img_{t}.png&#39;)<br>    frames.append(image)<br>    <br>imageio.mimsave(&#39;./example.gif&#39;, frames, fps = 2)</pre><p>Here is an example of how I might compare my implementation of the k-means algorithm with the implementation provided by <strong>scikit-learn</strong> to ensure that my implementation is correct:</p><pre>from sklearn.cluster import KMeans<br><br>kmeans = KMeans(n_clusters=k)<br>kmeans = kmeans.fit(dt)<br>labels = kmeans.predict(dt)<br>centroids = kmeans.cluster_centers_<br>print(&quot;My code&quot;)<br>print(centroids_list) <br>print(&quot;sklearn&quot;)<br>print(centroids)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*TJgo5yoV9bvgf_AcedPGVA.png" /><figcaption>Output generated by code above</figcaption></figure><p>My implementation and scikit-learn produce the same results so the code above is working correctly.</p><p>I hope this post has helped you understand the mechanics of the algorithm and how it works. Thanks for reading!</p><p><a href="https://www.linkedin.com/in/a-chernyshova/">Connect with me on LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8ce69c37800" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Loading Reddit posts using AWS Lambda and CloudWatch events]]></title>
            <link>https://medium.com/data-science/loading-reddit-posts-using-aws-lambda-and-cloudwatch-events-9edb03ba4c14?source=rss-3b1e1ccfeed6------2</link>
            <guid isPermaLink="false">https://medium.com/p/9edb03ba4c14</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[lambda]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[reddit]]></category>
            <dc:creator><![CDATA[Tonya Chernyshova]]></dc:creator>
            <pubDate>Thu, 31 Oct 2019 21:32:42 GMT</pubDate>
            <atom:updated>2019-12-17T18:28:56.985Z</atom:updated>
            <content:encoded><![CDATA[<p>Last month I finished Reddit Classification project where I collected data by scraping Reddit website and then built a binary classifier to identify where a given post came from. Now when I have some free time I’ve decided to go over that project and make some improvements. Previously, I was running a python script on AWS EC2 instance using cron. But this time I’ve decided to explore more AWS services and turn that script into a Lambda that is triggered by CloudWatch.</p><p>So, let’s begin. We are going to create an AWS lambda that will load Reddit posts using Reddit api and store them into S3 bucket.</p><p>First of all, we define a bucket name and a list of topics to load. Then we define a lambda’s main method — “lambda_handler” that accepts two arguments — event and context. This method is an entry point of our lambda, CloudWatch is going to trigger it every 15mins. This method will go through the list of topics and load reddit posts for each of them. In case of an exception we catch it, log it and move on to the next topic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/0*1NTWuTYKWpSpofwU.png" /></figure><p>Next, let’s look at the main method that loads reddit posts. First of all we check if a csv file with previously loaded posts already exists in S3. If so, we load those posts into a list. We extract an id of the last saved post from the list to continue loading all newer posts after that id using “before” parameter. If there is no existing file in S3 it means that we have not loaded any posts yet. In this case we are going to load all available posts through Reddit api using “after” parameter. You can read more about how to use Reddit api by following this link: <a href="https://www.reddit.com/dev/api/">https://www.reddit.com/dev/api/</a> It explains how to use “before”/”after” parameters for pagination. Finally, we convert all posts to csv string and save it to S3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*buMFjhquKFwDT6PQ.png" /></figure><p>Here is how we load saved posts from S3. We use AWS boto3 client to access S3. We read a file from S3, split it to separate lines and pass them to python csv reader library. Then we use list comprehension to convert each parsed line into a tuple.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WSGOSFHfy-tlPUOt.png" /></figure><p>Here are the methods that load posts using python requests library and extract required fields from json response. The load_posts method uses a while loop to keep loading reddit posts. At first it passes pagingId as None to load the very first page. After that it extracts a next pagingId from the loaded page and passes it with the next request. It keeps doing it until pagingId becomes None. It means that reddit api doesn’t have any pages left.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6SycT8sNVsSVomX2.png" /></figure><p>You can find the full code on my <a href="https://github.com/achernyshova/Reddit-NLP-Classification/tree/master/AWS">Github</a></p><p>Now, when we have our lambda code ready, we need to create a zip file that we are going to deploy to AWS. When I tried to run this code for the first time on AWS it didn’t work because lambda didn’t have a requests python library. It turned out that we have to add all required dependencies into the archive along with the lambda script before deploying it to AWS. Let’s run the following commands in cmd:</p><pre>$ <strong>mkdir lambda</strong><br>$ <strong>cd lambda</strong><br>$ <strong>pip install --target ./ requests .</strong>  //install requests library<br>$ <strong>zip -r ../reddit_lambda.zip .</strong>   //archive the folder<br>$ <strong>cd ..</strong><br>$ <strong>zip -g reddit_lambda.zip reddit_lambda.py</strong>  //add lambda script</pre><p>This creates an archive called reddit_lambda.zip with requests dependency and our lambda script.</p><p>Now we are going to practice using AWS Cli to deploy our lambda function and to trigger it every 15mins. Be prepared to run many commands in command line.</p><p>First of all we have to create an execution role with name <em>reddit_lambda_role</em> for our lambda:</p><pre>$ <strong>cat lambda_assume_role_policy.json</strong><br>{<br> &quot;Version&quot;: &quot;2012-10-17&quot;,<br> &quot;Statement&quot;: [{<br>     &quot;Effect&quot;: &quot;Allow&quot;,<br>     &quot;Principal&quot;: {<br>         &quot;Service&quot;: &quot;lambda.amazonaws.com&quot;<br>     },<br>     &quot;Action&quot;: &quot;sts:AssumeRole&quot;<br> }]<br>}<br>$ <strong>aws iam create-role --role-name reddit_lambda_role --assume-role-policy-document file://lambda_assume_role_policy.json</strong></pre><p>This assume policy document allows the role to be assumed by any lambda in our account. Currently this role has no permissions attached to it — it mean that even if lambda assumes it will not have the rights to do anything useful, like access S3 to save an object. To fix it let’s create a policy:</p><pre><strong>$ cat lambda_role_policy.json<br></strong>{<br> &quot;Version&quot;: &quot;2012-10-17&quot;,<br> &quot;Statement&quot;: [{<br>   &quot;Effect&quot;: &quot;Allow&quot;,<br>   &quot;Action&quot;: [<br>    &quot;s3:GetObject&quot;,<br>    &quot;s3:PutObject&quot;,<br>    &quot;s3:PutObjectAcl&quot;,<br>    &quot;s3:GetObjectAcl&quot;<br>   ],<br>   &quot;Resource&quot;: &quot;arn:aws:s3:::<strong>your-bucket-name</strong>/*&quot;<br>  },{<br>   &quot;Effect&quot;: &quot;Allow&quot;,<br>   &quot;Action&quot;: [<br>    &quot;logs:CreateLogStream&quot;,<br>    &quot;logs:PutLogEvents&quot;<br>   ],<br>   &quot;Resource&quot;: &quot;arn:aws:logs:eu-west-1:<strong>your-account-id</strong>:*&quot;<br>  },{<br>   &quot;Effect&quot;: &quot;Allow&quot;,<br>   &quot;Action&quot;: &quot;logs:CreateLogGroup&quot;,<br>   &quot;Resource&quot;: &quot;*&quot;<br>  }]<br>}$ <strong>aws iam create-policy --policy-name reddit_lambda_policy --policy-document file://lambda_role_policy.json</strong></pre><p>This policy with name <em>reddit_lambda_policy</em> gives permissions to access S3 to load and save files and to save logs to CloudWatch. Now let’s attach this policy to a role:</p><pre>$ <strong>aws iam attach-role-policy --role-name reddit_lambda_role --policy-arn arn:aws:iam::<em>your-account-id</em>:policy/reddit_lambda_policy</strong></pre><p>Now let’s create a lambda with name <em>RedditLoader</em> in AWS:</p><pre>$ <strong>aws lambda create-function --region us-east-1 --function-name RedditLoader --zip-file fileb://reddit_lambda.zip --role arn:aws:iam::<em>your-account-id</em>:role/reddit_lambda_role --handler reddit.lambda_handler --timeout 900 --runtime python3.7</strong></pre><p>At this point, you can go to the AWS console, open Lambda section and run this lambda manually. It should be able to load reddit posts and store them in S3. But our aim is to trigger this lambda every 15mins using CloudWatch events.</p><p>Let’s create a scheduled event rule with name <em>RedditLoadingRule</em>:</p><pre>$ <strong>aws events put-rule --name RedditLoadingRule --schedule-expression &#39;rate(15 minutes)&#39;</strong></pre><p>Now we add permissions to our lambda to be triggered by event from the rule:</p><pre>$ <strong>aws lambda add-permission --function-name RedditLoader --statement-id RedditScheduleEvent --action &#39;lambda:InvokeFunction&#39; --principal events.amazonaws.com --source-arn arn:aws:events:us-east-1:<em>your-account-id</em>:rule/RedditLoadingRule</strong></pre><p>Add a target to our rule so it knows what to trigger every 15 minutes:</p><pre>$ <strong>aws events put-targets --rule RedditLoadingRule --targets &quot;Id&quot;: &quot;1&quot;,&quot;Arn&quot;:&quot;arn:aws:lambda:us-east-1:<em>your-account-id</em>:function:RedditLoader&quot;</strong></pre><p>And that’s it. Now CloudWatch will send an event every 15mins which will trigger our lambda.</p><p>All AWS commands may look intimidating, but in reality once you learn how to use basic commands through AWS cli, you will quickly get comfortable interacting with AWS using only cli. It is a very good skill to learn that opens doors to powerful automation.</p><p>In the next post we will look at how to use SageMaker to train and deploy models that use our reddit posts from S3.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9edb03ba4c14" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/loading-reddit-posts-using-aws-lambda-and-cloudwatch-events-9edb03ba4c14">Loading Reddit posts using AWS Lambda and CloudWatch events</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Athena helps to find the worst place to park your car in Portland.]]></title>
            <link>https://medium.com/data-science/aws-athena-helps-to-find-the-worst-place-to-park-your-car-in-portland-ffb5195cfd9?source=rss-3b1e1ccfeed6------2</link>
            <guid isPermaLink="false">https://medium.com/p/ffb5195cfd9</guid>
            <category><![CDATA[geolocation]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[athena]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Tonya Chernyshova]]></dc:creator>
            <pubDate>Wed, 30 Oct 2019 03:30:36 GMT</pubDate>
            <atom:updated>2021-02-08T16:09:51.534Z</atom:updated>
            <content:encoded><![CDATA[<h3>AWS Athena helps to find the worst place to park your car in Portland</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HdZYtEq9cEOBYXblyre9kg.png" /><figcaption><a href="https://public.tableau.com/profile/achernyshova#!/vizhome/PortlandCrime_16126542652560/Map">My Tableau Public</a></figcaption></figure><p>After visiting Portland, OR last weekend I’ve decided to explore some publicly available datasets about the city. In this post, we are going to calculate the number of incidents related to vehicles (theft from or theft of a vehicle) and the number of parking spots in each Portland neighborhood using<em> </em>Athena geo queries. After that, we will calculate the number of incidents per a parking spot to identify dangerous neighborhoods.</p><p>We are going to use three datasets:</p><ul><li>A police report with all the incidents that occurred in Portland in 2019. You can download this dataset <a href="https://www.portlandoregon.gov/police/71978">here</a> (click on “Download Open Data” tab)</li><li>Portland parking spots with coordinates, you can download this dataset <a href="http://gis-pdx.opendata.arcgis.com/datasets/ddd0f467686b422e88e40eb34aa9202f_210">here</a></li><li>Portland neighbourhoods boundaries in JSON, you can download it <a href="http://gis-pdx.opendata.arcgis.com/datasets/neighborhoods-regions">here</a> (select GeoJSON format)</li></ul><p>A quick and easy way to start exploring a dataset with SQL is to use AWS Athena database and S3. Create an S3 bucket (I called it portland-crime-score).</p><p>For incidents file, create a folder “crime_data” in the bucket. Import the downloaded CSV file into the folder. After that go to AWS console, open Athena service and run the following query to create a table (replace S3 bucket name with yours):</p><pre>CREATE EXTERNAL TABLE IF NOT EXISTS sampledb.incidents (<br>  `Address` string,<br>  `CaseNumber` string,<br>  `CrimeAgainst` string,<br>  `Neighborhood` string,<br>  `OccurDate` string,<br>  `OccurTime` string,<br>  `OffenseCategory` string,<br>  `OffenseType` string,<br>  `OpenDataLat` float,<br>  `OpenDataLon` float,<br>  `OpenDataX` float,<br>  `OpenDataY` float,<br>  `ReportDate` string,<br>  `OffenseCount` int <br>)<br>ROW FORMAT DELIMITED FIELDS TERMINATED BY &#39;,&#39; STORED AS TEXTFILE LOCATION &#39;s3://portland-crime-score/crime_data/&#39;<br>TBLPROPERTIES (&#39;skip.header.line.count&#39;=&#39;1&#39;);</pre><p>For parking spots create a folder called “parking_data” and import corresponding file. Run the following query to create a table:</p><pre>CREATE EXTERNAL TABLE IF NOT EXISTS sampledb.parking (<br> `X` float,<br> `Y` float,<br> `OBJECTID` string,<br> `ModifiedBy` string,<br> `ModifiedOn` string,<br> `ModifiedUsing` string,<br> `Comments` string,<br> `NonAssetID` string,<br> `Status` string,<br> `Owner` string,<br> `MaintResp` string,<br> `LocationID` string,<br> `ImagePath` string,<br> `Metered` string,<br> `APPZone` string,<br> `UseCode` string,<br> `ParkingDuration` string,<br> `Rotation` string<br>)<br>ROW FORMAT DELIMITED FIELDS TERMINATED BY &#39;,&#39; STORED AS TEXTFILE LOCATION &#39;s3://portland-crime-score/parking_data/&#39;<br>TBLPROPERTIES (&#39;skip.header.line.count&#39;=&#39;1&#39;);</pre><p>For the neighborhood boundaries file, create a folder called “neighborhoods_data” and import corresponding file. Neighborhoods data is stored in nested JSON file, that’s why the table schema looks different this time:</p><pre>CREATE EXTERNAL TABLE neighborhoods (<br>    type string,<br>    features array&lt;<br>        struct&lt;type: string,<br>            properties: struct&lt;OBJECTID: string,<br>                  NAME: string, <br>                  COMMPLAN: string,<br>                  SHARED: string,<br>                  COALIT: string,<br>                  HORZ_VERT: string,<br>                  MAPLABEL: string,<br>                  ID: string,<br>                  Shape_Length: string,<br>                  Shape_Area: string<br>            &gt;,<br>            geometry: struct&lt;type: string,<br>              coordinates: array&lt;array&lt;array&lt;string&gt;&gt;&gt;<br>            &gt;<br>        &gt;<br>    &gt;<br>)<br>ROW FORMAT SERDE &#39;org.openx.data.jsonserde.JsonSerDe&#39;<br>LOCATION &#39;s3://portland-crime-score/neighborhoods_data/&#39;;</pre><p>Let’s look at a table <strong>incidents </strong>and<strong> </strong>run a few SQL queries to explore the data. First of all, let’s see how many records we’ve got:</p><pre>select count(*) <br>from <strong>incidents;</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*d6GY5btxJvP15fcw5ldjeA.png" /></figure><pre>select * <br>from <strong>incidents</strong> <br>limit 10;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ci-d4wAzLWLVV3akZCB6AQ.png" /></figure><p>As we can see, some of the records don’t have coordinates of an incident. If we check the metadata of the dataset on the website, it says that they are removed on purpose:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PI6VprLuSSkQh1J4gpa71w.png" /></figure><p>Let’s see how many records we have without coordinates:</p><pre>select count(*) as count <br>from <strong>incidents</strong> <br>where OpenDataLat is Null or OpenDataLon is Null;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*kT9Da6JjIXrYULQAWG7UcQ.png" /></figure><p>We are going to exclude those records from further exploration. Now let’s see what types of incidents we have in the file:</p><pre>select distinct OffenseType <br>from <strong>incidents</strong>;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*x3P3heYk-dp6AJJ4xJzLeA.png" /></figure><p>There is a total of 46 incident types. Let’s focus on those related to a vehicle:</p><pre>select distinct OffenseType <br>from <strong>incidents</strong> <br>where OffenseType like &#39;%Vehicle%&#39;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/1*TWRjSW6KziKN1aUt5Hrpog.png" /></figure><p>Now let’s create a table that contains only Athena geo points of vehicle incidents and see how many rows we have after applying all these filters:</p><pre><strong>incident_points</strong> as (<br>select ST_POINT(opendatalon, opendatalat) <br>from <strong>incidents</strong> <br>where OffenseType in (<br>  &#39;Motor Vehicle Theft&#39;, <br>  &#39;Theft From Motor Vehicle&#39;, <br>  &#39;Theft of Motor Vehicle Parts or Accessories&#39;) <br>  and OpenDataLat is not NULL <br>  and OpenDataLon is not NULL<br>)<br>select count(*) as incident_count <br>from <strong>incident_points</strong>;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/1*2imK5Q3dlRG8QGL2lESDHw.png" /></figure><p>Now let’s create a table with geo points of all parking spots — this time we don’t have to filter anything:</p><pre><strong>parking_points</strong> as (<br>  select ST_POINT(x, y) as point <br>from <strong>parking</strong><br>)<br>select count(*) as parking_count<br>from <strong>parking_points</strong>;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*hJc2jbE3Xq54x_yNKwtUQA.png" /></figure><p>Now, let’s look at a neighborhood dataset. Let’s parse JSON to extract boundaries coordinates and create objects of type Polygon that are supported by Athena. I could not find an easy way to parse GeoJSON in Athena. <a href="https://docs.aws.amazon.com/athena/latest/ug/geospatial-input-data-formats-supported-geometry-types.html">Documentation</a> said nothing about supporting it. That’s why I had to perform a series of manipulations with json to extract the required coordinates.</p><p>First of all, let’s see what we’ve got in the table neighborhoods:</p><pre>select * <br>from <strong>neighborhoods</strong>;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gKPnWsjBlt04geZd9kSryw.png" /></figure><p>The whole file was parsed into a single row. We need to extract data from features columns that has a type of array. Let’s unnest the array like this:</p><pre>select type, feature <br>from <strong>neighborhoods</strong><br>CROSS JOIN UNNEST(features) AS t(feature)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aNa3YoLVfehL2948plCu2A.png" /></figure><p>Each feature represents a neighborhood. Now we can access its values like this:</p><pre>select feature.properties.Name as name, feature.geometry.coordinates[1] as shape <br>from <strong>neighborhoods</strong><br>CROSS JOIN UNNEST(features) AS t(feature)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P57UNIlMXR_ju7qCsK0aYg.png" /></figure><p>Here we have a neighborhood name and an array of arrays with boundary coordinates. Our aim is to convert these coordinates to a varchar format that Athena can parse into Polygon object like that: <em>SELECT ST_POLYGON(‘polygon ((1 1, 1 4, 4 4, 4 1))’). </em>To achieve this we should concatenate coordinates into a string. Also, two neighborhoods with names ‘MC UNCLAIMED #13’ and ‘CRESTWOOD’ had strange coordinates format that were not parsable so I excluded them from the list. Let’s run the following query:</p><pre>with <strong>neighborhood_coords</strong> as (<br>  select feature.properties.Name as name,<br>  feature.geometry.coordinates[1] as shape <br>  from <strong>neighborhoods</strong><br>  CROSS JOIN UNNEST(features) AS t(feature)<br>  where feature.properties.Name != &#39;MC UNCLAIMED #13&#39; AND    feature.properties.Name != &#39;CRESTWOOD&#39;<br>),<br><strong>unnest_nodes</strong> as (<br>  select name, array_join(node, &#39; &#39;) as node_str <br>  from <strong>neighborhood_coords</strong><br>  CROSS JOIN UNNEST(shape) AS t(node)<br>)<br>select * from <strong>unnest_nodes</strong>;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gkUztfGCbWzTymN41YuNUg.png" /></figure><p>We extracted neighborhood coordinates into multiple columns. Now all we need to do is aggregate them back with string concatenation:</p><pre><strong>suburbs</strong> as (<br>  select name as suburb_name, ST_POLYGON(&#39;polygon ((&#39; || array_join(array_agg(node_str), &#39;, &#39;) || &#39;))&#39;) as poly from <strong>unnest_nodes</strong> group by name <br>)<br>select * from <strong>suburbs</strong>;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ImVEZU0j21ifbvdj6AUIXw.png" /></figure><p>Now let’s join our neighborhoods with parking spots to count the number of spots in each neighborhood. We are going to use an Athena function ST_CONTAINS to check if a parking spot Point lies inside a boundary Polygon:</p><pre><strong>suburb_parking</strong> as (<br>  select suburb_name, count(*) as parking_count from <strong>suburbs</strong> <br>  join parking_points on ST_CONTAINS(poly, point)<br>  group by suburb_name order by parking_count desc<br>)<br>select * from <strong>suburb_parking</strong> limit 10;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VxLHT-0YM-bphUyNUnrQhg.png" /></figure><p>As we can see, downtown has the highest number of parking spots.</p><p>Now let’s do the neighborhoods with the incidents to count the number of incidents in each neighborhood:</p><pre><strong>suburb_incident</strong> as (<br>  select suburb_name, count(*) as incident_count from <strong>suburbs</strong> <br>  join incident_points on ST_CONTAINS(poly, incident_point)<br>  group by suburb_name order by incident_count desc<br>)<br>select * from <strong>suburb_incident</strong> limit 10;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*74wyDRSk5VexU5NWQr0tiQ.png" /></figure><p>The highest number of incidents is recorded in Hazelwood, and Downtown is in second place.</p><p>Finally, let’s calculate a ratio between the number of incidents and the number of parking spots:</p><pre>select suburb_parking.suburb_name, CAST(incident_count as double)/parking_count as crime_score <br>from <strong>suburb_incident</strong><br>join suburb_parking on suburb_incident.suburb_name = suburb_parking.suburb_name<br>order by incident_count/parking_count desc;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*miIr_XQs1sVUcUkXkEtyCg.png" /></figure><p><strong>And to our surprise, the neighborhood with such a nice name as Richmond has the highest number of incidents per parking spot, when Northwest district is the safest place with 0,056 incidents per parking spot.</strong></p><p>The whole query looks like this:</p><pre>with <strong>regions_coords</strong> as (<br>  select feature.properties.Name as name, feature.geometry.coordinates[1] as shape from <strong>neighborhoods</strong><br>  CROSS JOIN UNNEST(features) AS t(feature)<br>  where feature.properties.Name != &#39;MC UNCLAIMED #13&#39; AND feature.properties.Name != &#39;CRESTWOOD&#39;<br>),<br><strong>unnest_nodes</strong> as (<br>  select name, array_join(node, &#39; &#39;) as node_str from <strong>regions_coords</strong><br>  CROSS JOIN UNNEST(shape) AS t(node)<br>),<br><strong>suburbs</strong> as (<br>  select name as suburb_name, ST_POLYGON(&#39;polygon ((&#39; || array_join(array_agg(node_str), &#39;, &#39;) || &#39;))&#39;) as poly from <strong>unnest_nodes</strong> group by name <br>),<br><strong>parking_points</strong> as (<br>  select ST_POINT(x, y) as point from <strong>parking</strong><br>),<br><strong>incident_points</strong> as (<br>  select ST_POINT(opendatalon, opendatalat) as incident_point from <strong>incidents</strong><br>  where OffenseType in (<br>    &#39;Motor Vehicle Theft&#39;, <br>    &#39;Theft From Motor Vehicle&#39;, <br>    &#39;Theft of Motor Vehicle Parts or Accessories&#39;) <br>  and OpenDataLat is not NULL <br>  and OpenDataLon is not NULL<br>),<br><strong>suburb_parking</strong> as (<br>  select suburb_name, count(*) as parking_count from <strong>suburbs</strong> <br>  join parking_points on ST_CONTAINS(poly, point)<br>  group by suburb_name order by parking_count desc<br>),<br><strong>suburb_incident</strong> as (<br>  select suburb_name, count(*) as incident_count from <strong>suburbs</strong> <br>  join incident_points on ST_CONTAINS(poly, incident_point)<br>  group by suburb_name order by incident_count desc<br>)<br>select suburb_parking.suburb_name, CAST(incident_count as double)/parking_count as crime_score <br>  from <strong>suburb_incident</strong><br>  join suburb_parking on suburb_incident.suburb_name = suburb_parking.suburb_name<br>  order by incident_count/parking_count desc;</pre><p>Next time when I go to Portland I will check this information to find the best place to stay there.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ffb5195cfd9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/aws-athena-helps-to-find-the-worst-place-to-park-your-car-in-portland-ffb5195cfd9">AWS Athena helps to find the worst place to park your car in Portland.</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reddit posts classification using NLP, Pandas and S3]]></title>
            <link>https://medium.com/analytics-vidhya/reddit-posts-classification-using-nlp-pandas-and-s3-68045246a90e?source=rss-3b1e1ccfeed6------2</link>
            <guid isPermaLink="false">https://medium.com/p/68045246a90e</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Tonya Chernyshova]]></dc:creator>
            <pubDate>Sun, 13 Oct 2019 17:23:48 GMT</pubDate>
            <atom:updated>2019-12-17T18:22:25.034Z</atom:updated>
            <content:encoded><![CDATA[<p>In my last <a href="https://medium.com/@tonya.chernyshova/loading-reddit-posts-using-aws-lambda-and-cloudwatch-events-7e58cbddd000">post</a> I automated gathering reddit posts on S3 using AWS lambda and CloudWatch events. In this post we are going to take posts from two subreddits and build a ML classification model to identify which subreddit a given post belongs to.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1oRk5XfC8x7kcn8sfB8XaA.png" /></figure><p>For the project I’ve chosen highly correlated subreddits:</p><ul><li>Dog lovers subreddit — <a href="https://www.reddit.com/r/dogs/">https://www.reddit.com/r/dogs/</a></li><li>Dog haters subreddit — <a href="https://www.reddit.com/r/Dogfree">https://www.reddit.com/r/Dogfree</a></li></ul><p>Let’s open our Jupyter notebook and start with loading posts from two subreddits. Pandas has a convenient method for reading a file from S3.</p><pre><strong>import</strong> pandas <strong>as</strong> pd<br><strong>import</strong> numpy <strong>as</strong> np</pre><pre>#name of the bucket with reddit posts<br>bucket_name = &#39;nlp-reddit-posts&#39;</pre><pre>df_dogs = pd.<strong>read_csv</strong>(&#39;s3://&#39; + bucket_name + &#39;/dogs.csv&#39;)<br>df_dogs = df_dogs[[&#39;subreddit&#39;, &#39;title&#39;, &#39;selftext&#39;]]<br>df_dogs.<strong>head</strong>()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BTipILSI5hwsyKrWItTu-g.png" /></figure><pre>df_dogfree = pd.<strong>read_csv</strong>(&#39;s3://&#39; + bucket_name + &#39;/Dogfree.csv&#39;)<br>df_dogfree = df_dogfree[[&#39;subreddit&#39;,&#39;title&#39;,&#39;text&#39;]]<br>df_dogfree.<strong>head</strong>()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4IFVyn25cx5PX-eXaogXNw.png" /></figure><p>As we can see, some posts have no text. Such posts consist of a title and a link to some external resource. Let’s concatenate both dataframes and see some stats:</p><pre>df = pd.<strong>concat</strong>([df_dogs, df_dogfree]) <br>df.<strong>shape<br></strong>(2245, 3)</pre><pre>df.<strong>isnull</strong>().sum().sort_values(ascending=<strong>False</strong>)<br>selftext     618<br>title        0  <br>subreddit    0  <br>dtype: int64</pre><pre>df[&#39;subreddit&#39;].<strong>value_counts</strong>()<br>dogs       1127<br>Dogfree    1118<br>Name: subreddit, dtype: int64</pre><p>Let’s replace null values with empty text and concatenate titles and selftexts. Also, let’s create a column subreddit_class that represents a subreddit as a binary value 0 or 1.</p><pre>df.<strong>replace</strong>(np.nan, &#39;&#39;, inplace=<strong>True</strong>)<br>df[&#39;text&#39;] = df[&#39;title&#39;]  + &#39; &#39; + df[&#39;selftext&#39;]</pre><pre>df[&#39;subreddit_class&#39;] = [1 <strong>if</strong> i == &#39;dogs&#39; <strong>else</strong> 0 <strong>for</strong> i <strong>in </strong>df[&#39;subreddit&#39;]]</pre><pre>df.sample(frac=1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wd9EPbZumhQn7ZeLef86WA.png" /></figure><p>Let’s create nice word clouds in shapes of a dog and a cat:</p><pre><strong>from</strong> wordcloud <strong>import</strong> WordCloud, STOPWORDS, ImageColorGenerator<br><strong>from</strong> nltk.corpus <strong>import</strong> stopwords</pre><pre># exclude stop words from the cloud<br>stop_words = stopwords.<strong>words</strong>(&#39;english&#39;) + [&#39;help&#39;, &#39;discussion&#39;]</pre><pre><strong>def</strong> world_cloud(text, filename, mask=<strong>None</strong>):<br>    wordcloud = WordCloud(max_font_size=72, max_words=150,<br>                      background_color=&quot;white&quot;,<br>                      stopwords = stop_words,<br>                      mask=mask).<strong>generate</strong>(text)<br>    plt.<strong>figure</strong>(figsize=(10,10))<br>    plt.<strong>imshow</strong>(wordcloud, interpolation=&quot;bilinear&quot;);<br>    plt.<strong>axis</strong>(&quot;off&quot;);<br>    plt.<strong>show</strong>();</pre><p>Dog lovers cloud can be created like this:</p><pre># break down titles of &quot;dogs&quot; subreddit and concatenate them<br>text_dogs = &quot; &quot;.<strong>join</strong>(text <strong>for</strong> text <strong>in</strong> df[&#39;title&#39;][(df[&#39;subreddit&#39;]==&#39;dogs&#39;)]) </pre><pre># image with a shape of a dog<br>url_dog=&#39;https://render.fineartamerica.com/images/rendered/default/print/8.000/6.375/break/images-medium-5/golden-retriever-dog-vector-silhouette-helgamariah.jpg&#39;</pre><pre>mask_dog=np.<strong>array</strong>(Image.<strong>open</strong>(requests.<strong>get</strong>(url_dog,stream=<strong>True</strong>).<strong>raw</strong>))world_cloud(text_dogs, &#39;dogs&#39;, mask_dog)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*04pq46TynW0uhk1-EheZrA.png" /></figure><p>Similarly, dog hater cloud can be created like this:</p><pre># break down titles of &quot;dogfree&quot; subreddit and concatenate them<br>text_dogfree = &quot; &quot;.<strong>join</strong>(text <strong>for</strong> text <strong>in</strong> df[&#39;title&#39;][(df[&#39;subreddit&#39;] == &#39;Dogfree&#39;)]) </pre><pre># image with a shape of a cat<br>url_cat=&#39;<a href="https://images-na.ssl-images-amazon.com/images/I/31CaLdDFlKL.jpg&#39;">https://images-na.ssl-images-amazon.com/images/I/31CaLdDFlKL.jpg&#39;</a></pre><pre>mask_cat=np.<strong>array</strong>(Image.<strong>open</strong>(requests.<strong>get</strong>(url_cat,stream=True).<strong>raw</strong>))<br>world_cloud(text_dogfree, &#39;dog_free_shape&#39;, mask_cat)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*WOFYLsuCC9IhDrqDWJmcwg.png" /></figure><p>Let’s tokenize and stem text from posts using <a href="https://www.nltk.org/">Natural Language Toolkit</a>:</p><pre><strong>from</strong> nltk.tokenize <strong>import</strong> RegexpTokenizer <br><strong>from</strong> nltk.stem.porter <strong>import</strong> PorterStemmer</pre><pre><strong>def</strong> steam_text(text):    <br>    stemmer = PorterStemmer()<br>    tokenizer = RegexpTokenizer(r&#39;\w+&#39;)<br>    text_token = tokenizer.<strong>tokenize</strong>(text.<strong>lower</strong>())  <br>    <strong>return</strong> &#39; &#39;.<strong>join</strong>(map(<strong>lambda</strong> x: stemmer.<strong>stem</strong>(x), text_token))</pre><pre>df[&#39;stemmed_text&#39;] = df[&#39;text&#39;].<strong>apply</strong>(steam_text)</pre><p>Create our features matrix (X) and a target vector (y) and split to train and test datasets:</p><pre>X = df[&#39;stemmed_text&#39;] <br>y = df[&#39;subreddit_class&#39;]<br>X_train, X_test, y_train, y_test = train_test_split(X, y, <br>                                                    stratify=y, <br>                                                    random_state=42)</pre><p>For a baseline estimator let’s take the class that has the most observations and use that class as the result for all predictions. In this case our baseline model accuracy will be 0.5018 or 50.18%:</p><pre>y_test.<strong>value_counts</strong>(normalize=True)<br>1    0.501779<br>0    0.498221<br>Name: subreddit_class, dtype: float64</pre><p>Lets train 3 models using Gridsearch for hyperparameters tuning. First will be logistic regression:</p><pre>pipe_lr_tf = Pipeline([<br>    (&#39;tf&#39;, TfidfVectorizer()),<br>    (&#39;lr&#39;, LogisticRegression())<br>])<br><br>params_lr_tf = {<br>    &#39;tf__stop_words&#39;:   [None, &#39;english&#39;],<br>    &#39;tf__max_features&#39;: [1000, 2000, 3000, 4000], <br>    &#39;tf__ngram_range&#39;:  [(1, 1), (1, 2), (1, 3)], <br>    &#39;lr__penalty&#39;:      [&#39;l2&#39;, &#39;l1&#39;],<br>    &#39;lr__C&#39;:            [.5, .01, 1],<br>    &#39;lr__random_state&#39;: [42] <br>}<br>gs_lr_tf = GridSearchCV(pipe_lr_tf, <br>                        param_grid=params_lr_tf,<br>                        cv=5,<br>                        n_jobs=2,<br>                        verbose=1)<br><br>gs_lr_tf.fit(X_train, y_train)<br><br><strong>print</strong>(f&#39;GridSearch Best Params: <strong>{gs_lr_tf.best_params_}</strong>&#39;)<br><strong>print</strong>(&#39;Train Score:&#39;, gs_lr_tf.<strong>score</strong>(X_train, y_train))<br><strong>print</strong>(&#39;Test Score:&#39;, gs_lr_tf.<strong>score</strong>(X_test, y_test))<br><br>GridSearch Best Params: {&#39;lr__C&#39;: 1, &#39;lr__penalty&#39;: &#39;l2&#39;, &#39;lr__random_state&#39;: 42, &#39;tf__max_features&#39;: 1000, &#39;tf__ngram_range&#39;: (1, 2), &#39;tf__stop_words&#39;: &#39;english&#39;}<br>Train Score: 0.976232917409388<br>Test Score: 0.9537366548042705</pre><p>Next let’s try Naive Bayes:</p><pre>pipe_multinomial_nb_cvec = Pipeline([<br>    (&#39;cvec&#39;, CountVectorizer()),<br>    (&#39;multinomial_nb&#39;, MultinomialNB())<br>])<br><br>params_multinomial_nb_cvec = {<br>    &#39;cvec__stop_words&#39;:           [None, &#39;english&#39;],<br>    &#39;cvec__max_features&#39;:         [100, 500, 1000, 2000, 3000, 4000], <br>    &#39;cvec__ngram_range&#39;:          [(1, 1), (1, 2), (1, 3)],<br>    &#39;multinomial_nb__alpha&#39;:     [0], <em># 2, 5, .5, 1</em><br>    &#39;multinomial_nb__fit_prior&#39;: [<strong>True</strong>, <strong>False</strong>]<br>}<br><br>gs_multinomial_nb_cvec = GridSearchCV(pipe_multinomial_nb_cvec, <br>                        param_grid=params_multinomial_nb_cvec,<br>                        cv=5,<br>                        n_jobs=2,<br>                        verbose=1)<br>    <br>gs_multinomial_nb_cvec.<strong>fit</strong>(X_train, y_train)<br><br><strong>print</strong>(f&#39;GridSearch Best Params:{gs_multinomial_nb_cvec.best_params_}&#39;)<br><strong>print</strong>(&#39;Train Score:&#39;, gs_multinomial_nb_cvec.<strong>score</strong>(X_train, y_train))<br><strong>print</strong>(&#39;Test Score:&#39;, gs_multinomial_nb_cvec.<strong>score</strong>(X_test, y_test))</pre><pre>GridSearch Best Params: {&#39;cvec__max_features&#39;: 1000, &#39;cvec__ngram_range&#39;: (1, 1), &#39;cvec__stop_words&#39;: &#39;english&#39;, &#39;multinomial_nb__alpha&#39;: 0, &#39;multinomial_nb__fit_prior&#39;: False}<br>Train Score: 0.9477124183006536<br>Test Score: 0.9234875444839857</pre><p>And finally, Random Forest:</p><pre>pipe_rf_tf = Pipeline([<br>    (&#39;tf&#39;, TfidfVectorizer()),<br>    (&#39;rf&#39;, RandomForestClassifier())<br>])<br><br>params_rf_tf={<br>    &#39;tf__stop_words&#39;:   [None, &#39;english&#39;],<br>    &#39;tf__max_features&#39;: [1000, 2000, 3000],<br>    &#39;tf__ngram_range&#39;:  [(1,1), (1, 2)],<br>    &#39;tf__smooth_idf&#39;:   [<strong>True</strong>, <strong>False</strong>],<br>    &#39;rf__n_estimators&#39;: [25, 50, 100], <br>    &#39;rf__criterion&#39;:    [&#39;gini&#39;], <em>#&#39;entropy&#39;,</em><br>    &#39;rf__max_depth&#39;:    [3, 5], <br>    &#39;rf__random_state&#39;: [42]<br>}<br>gs_rf_tf = GridSearchCV(pipe_rf_tf,<br>                        param_grid=params_rf_tf, <br>                        cv=5,<br>                        n_jobs=2,<br>                        verbose=1)<br><br>gs_rf_tf.<strong>fit</strong>(X_train, y_train)<br><br><strong>print</strong>(f&#39;GridSearch Best Params: {gs_rf_tf.best_params_}&#39;)<br><strong>print</strong>(&#39;Train Score:&#39;, gs_rf_tf.<strong>score</strong>(X_train, y_train))<br><strong>print</strong>(&#39;Test Score:&#39;, gs_rf_tf.<strong>score</strong>(X_test, y_test))</pre><pre>GridSearch Best Params: {&#39;rf__criterion&#39;: &#39;gini&#39;, &#39;rf__max_depth&#39;: 5, &#39;rf__n_estimators&#39;: 100, &#39;rf__random_state&#39;: 42, &#39;tf__max_features&#39;: 1000, &#39;tf__ngram_range&#39;: (1, 1), &#39;tf__smooth_idf&#39;: False, &#39;tf__stop_words&#39;: &#39;english&#39;}<br>Train Score: 0.9548425430778372<br>Test Score: 0.9199288256227758</pre><p>Let’s evaluate our models using the following methods:</p><pre><strong>def</strong> conf_matrix(model, X_test, y_test):<br>    y_hat = model.predict(X_test) <br>    <em># get confusion matrix data</em><br>    cm = confusion_matrix(y_test, y_hat)  <br>    <em># extract elements of confusion matrix</em><br>    tn, fp, fn, tp = cm.ravel()           <br>    <br>    <em># Convert Confusion Matrix to DataFrame</em><br>    cm_df = pd.DataFrame(cm, columns = [&#39;Predict Dog Haters (y=0)&#39;, &#39;Predict Dog Lovers (y=1)&#39;], index = [&#39;Actual Dog Haters (y=0)&#39;, &#39;Actual Dog Lovers (y=1)&#39;])<br>    <strong>return</strong> cm_df<br>    <br><strong>def</strong> conf_matrix_metrics(model, X_test, y_test):<br>    y_hat = model.predict(X_test) <br>    cm = confusion_matrix(y_test, y_hat)  <br>    tn, fp, fn, tp = cm.ravel()           <br>    <br>    <em>#calculate metrics </em><br>    accuracy = (tp + tn) / (tn + fp + fn + tp)<br>    sensitivity = tp / (tp + fn)<br>    specificity = tn/ (tn + fp)<br>    precision = tp /(tp + fp)<br>    <br>    <strong>return</strong> {&#39;accuracy&#39;: np.round(accuracy,3),<br>            &#39;sensitivity&#39;: np.round(sensitivity,3),<br>            &#39;specificity&#39;: np.round(specificity,3),<br>            &#39;precision&#39;: np.round(precision,3)}</pre><p>Logistic regression:</p><pre><strong>print</strong>(conf_matrix_metrics(gs_lr_tf, X_test, y_test))<br><strong>conf_matrix</strong>(gs_lr_tf, X_test, y_test)</pre><pre>{&#39;accuracy&#39;: 0.954, &#39;sensitivity&#39;: 0.961, &#39;specificity&#39;: 0.946, &#39;precision&#39;: 0.948}</pre><pre>+-------------------+---------------------+--------------------+<br>|                   | Predict Dog Haters  | Predict Dog Lovers |<br>|                   | (y=0)               | (y=1)              |<br>+-------------------+---------------------+--------------------+<br>| Actual Dog Haters |                 265 |                 15 |(y=0)<br>| (y=0)             |                     |                    |      <br>| Actual Dog Lovers |                  11 |                271 |<br>| (y=1)             |                     |                    |<br>+-------------------+---------------------+--------------------+</pre><p>Naive Bayes:</p><pre><strong>print</strong>(conf_matrix_metrics(gs_multinominal_nb_cvec, X_test, y_test))<br><strong>conf_matrix</strong>(gs_multinominal_nb_cvec, X_test, y_test)</pre><pre>{&#39;accuracy&#39;: 0.923, &#39;sensitivity&#39;: 0.947, &#39;specificity&#39;: 0.9, &#39;precision&#39;: 0.905}</pre><pre>+-------------------+---------------------+--------------------+<br>|                   | Predict Dog Haters  | Predict Dog Lovers |<br>|                   | (y=0)               | (y=1)              |<br>+-------------------+---------------------+--------------------+<br>| Actual Dog Haters |                 252 |                 28 |<br>| (y=0)             |                     |                    |      <br>| Actual Dog Lovers |                  15 |                267 |<br>| (y=1)             |                     |                    |<br>+-------------------+---------------------+--------------------+</pre><p>Random forest:</p><pre><strong>print</strong>(conf_matrix_metrics(gs_rf_tf, X_test, y_test))<br><strong>conf_matrix</strong>(gs_rf_tf, X_test, y_test)</pre><pre>{&#39;accuracy&#39;: 0.92, &#39;sensitivity&#39;: 0.901, &#39;specificity&#39;: 0.939, &#39;precision&#39;: 0.937}</pre><pre>+-------------------+---------------------+--------------------+<br>|                   | Predict Dog Haters  | Predict Dog Lovers |<br>|                   | (y=0)               | (y=1)              |<br>+-------------------+---------------------+--------------------+<br>| Actual Dog Haters |                 263 |                 17 |<br>| (y=0)             |                     |                    |      <br>| Actual Dog Lovers |                  28 |                254 |<br>| (y=1)             |                     |                    |<br>+-------------------+---------------------+--------------------+</pre><p>In spite of the fact that I’ve chosen highly correlated subreddits all three models (Logistic Regression, Naive Bayes and Random Forest) showed at least 1.7x better accuracy score than the baseline.</p><p>In this experiment we got the best results with Logistic Regression. We can predict with an accuracy of about 95.4% where a given post came from. Let’s see feature importance of the model. For dog lovers we sort coefficients descending:</p><pre>pd.DataFrame(<br>list(zip(gs_lr_tf.best_estimator_[0].<strong>get_feature_names</strong>(),(gs_lr_tf.best_estimator_[1].coef_[0]))),<br>columns=[&#39;word&#39;,&#39;coef&#39;]).<strong>sort_values</strong>(&#39;coef&#39;,ascending=<strong>False</strong>).<strong>head</strong>(5)</pre><pre>+---------+----------+<br>|  Word   |   Coef   |<br>+---------+----------+<br>| help    | 5.765303 |<br>| parti   | 4.812514 |<br>| discuss | 4.729457 |<br>| vet     | 2.615551 |<br>| ani     | 2.071909 |<br>+---------+----------+</pre><p>And for dog haters we sort coefficients ascending:</p><pre>pd.DataFrame(<br>list(zip(gs_lr_tf.best_estimator_[0].<strong>get_feature_names</strong>(),(gs_lr_tf.best_estimator_[1].coef_[0]))),<br> columns=[&#39;word&#39;,&#39;coef&#39;]).sort_values(&#39;coef&#39;,ascending=<strong>True</strong>).<strong>head</strong>(5)</pre><pre>+--------+-----------+<br>|  Word  |   Coef    |<br>+--------+-----------+<br>| attack | -2.942621 |<br>| dog    | -2.639061 |<br>| anim   | -2.497448 |<br>| hate   | -2.313820 |<br>| shit   | -1.950404 |<br>+--------+-----------+</pre><p>Our top stem predictors:</p><ul><li>for Dog lovers: help, parti, discuss, vet, ani</li><li>for Dog haters: attack, dog, anim, hate, shit</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68045246a90e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/reddit-posts-classification-using-nlp-pandas-and-s3-68045246a90e">Reddit posts classification using NLP, Pandas and S3</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>