<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Chris Doenlen on Medium]]></title>
        <description><![CDATA[Stories by Chris Doenlen on Medium]]></description>
        <link>https://medium.com/@scrapfishies?source=rss-2d189adedc19------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Q0tu4kTrV55feS-8YIdHPg.jpeg</url>
            <title>Stories by Chris Doenlen on Medium</title>
            <link>https://medium.com/@scrapfishies?source=rss-2d189adedc19------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 00:13:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@scrapfishies/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Finding the Narrative with Natural Language Processing]]></title>
            <link>https://scrapfishies.medium.com/finding-the-narrative-with-natural-language-processing-47177d20f743?source=rss-2d189adedc19------2</link>
            <guid isPermaLink="false">https://medium.com/p/47177d20f743</guid>
            <category><![CDATA[twitter]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[topic-modeling]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <dc:creator><![CDATA[Chris Doenlen]]></dc:creator>
            <pubDate>Sun, 27 Dec 2020 19:38:15 GMT</pubDate>
            <atom:updated>2021-01-01T02:05:02.957Z</atom:updated>
            <content:encoded><![CDATA[<h4>A framework for topic modeling with tweets in Python</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HduxR_6bObdpuYYIw3GYZw.jpeg" /></figure><p>When I first started studying data science, one of the areas I was most excited to learn was natural language processing. “Unsupervised machine learning” certainly has a mystical ring to it, and, coming from finance roles in startups, working with text data felt like the polar opposite from what I’m used to.</p><p>My experience from working in finance roles in startups did, however, teach me the importance of process and storytelling with data. Because finance touches everything, you get a sense of how all the different parts of the business fit together, how data flows between them, and how to effectively communicate performance and strategy to various, non-technical audiences.</p><p>There’s no shortage of articles detailing how various topic modeling algorithms work and tutorials on how to code them up (<em>and I relied heavily on them while working through this project</em>) — what felt missing, however, was a high-level framework for implementing them into a comprehensive workflow and how to distill findings and insights into a compelling narrative.</p><p>With this in mind, I’ll be outlining my process for topic modeling and creating a few insightful visualizations using the <a href="https://www.kaggle.com/vikasg/russian-troll-tweets">Russian Troll Tweets dataset</a>:</p><ol><li>Text preprocessing</li><li>Topic discovery with pyLDAvis</li><li>“Semi-Supervised” topic modeling with CorEx</li><li>Creating visualizations</li></ol><p>If you’d like to see exactly how I put this together, you can find <a href="https://github.com/scrapfishies/russian-troll-tweets-nlp">the code for this project on my GitHub</a>.</p><h3>Text Preprocessing</h3><p>In general, text pre-processing should include lowercasing all words, removing punctuation and stop words, and stemming or lemmatization. When working with tweets, in addition to the normal text-preprocessing tasks we also have to consider hashtags, acronyms, re-tweet syntax (‘RT @scrapfishies:…’), emojis, and other elements.</p><p>Should hashtags be be segmented (divided into their unique words) or kept as a single concatenated string? Well, I’d argue that it depends on the hashtag. As an example, the <em>#blacklivesmatter</em> hashtag was used frequently in this corpus — segmenting would give us 3 distinct tokens: <em>‘black’, ‘lives’,</em> and <em>‘matter’. </em>Unless we use bigrams or trigrams when tokenizing, we might lose some of the hashtag’s <em>potency</em> which could affect the themes that emerge when topic modeling. What about a <em>#donaldtrump </em>or<em> #hillaryclinton </em>hashtag? Should these be segmented or left concatenated? Instead of drawing hard lines, my guideline is this:</p><p><strong>The more <em>salient</em> a hashtag is in the corpus, the stronger the case for leaving it intact.</strong></p><p>This is where domain knowledge and exploratory data analysis will be your guide. Spend time reviewing hashtags and consider segmenting less salient ones with the help of a <a href="https://pypi.org/project/wordsegmentation/">word segmentation library</a> or by using a dictionary and pandas’ replace() method for a more manual approach.</p><p>A text preprocessing pipeline for a corpus of tweets could include the following:</p><ol><li>Lowercase and remove punctuation, emojis, URLs, and retweet prefixes</li><li>Segment hashtags where appropriate</li><li>Remove stop words</li><li>Lemmatize (<em>I really like </em><a href="https://spacy.io/api/lemmatizer"><em>spaCy’s lemmatizer</em></a><em> for this!</em>)</li></ol><p>Remember: this will be an iterative process! As you get into the topic modeling phase, you may find that you need to go back and make tweaks to your pipeline. For example, spaCy’s lemmatizer changed <em>‘[Mike] Pence’ </em>to <em>‘penny’</em>, and so I had to account for this in my workflow. I also found that making a basic dictionary of word frequencies was a quick way to get a feel for the prevalence of words and hashtags in the corpus which informed my text preprocessing pipeline.</p><h3>Topic Discovery with pyLDAvis</h3><p><a href="https://github.com/bmabey/pyLDAvis">pyLDAvis</a> is an interactive <a href="https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation">latent Dirichlet allocation</a> (LDA) visualization library for Python that can be a great foundational framework for topic modeling.</p><p>Below is a static sample output with 12 topics from the Russian troll tweets dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lOcMaLMQ85eiCmFtmEQ_0w.png" /><figcaption>12-topic pyLDAvis visualization with the Russian troll tweet dataset</figcaption></figure><p>On the right, we have the 30 most salient terms for a given topic — as you click through the numbered topics or hover over their corresponding bubbles on the left, these words and their frequencies will change. As you pan through the topics, you may be able to identify the theme by the keywords.</p><p>On the left, we can see numbered bubbles of varying sizes placed on a grid. These topics have been projected onto the two-dimensional plane using principal component analysis (PCA). The size of the bubble gives a sense of the topics prevalence in the corpus, and the distances between bubbles gives a sense of topic similarity. As an example, we can see a lot of overlap in topics 7, 8, and 9, while topic 12 sits alone in the bottom left quadrant (<em>topic 12 represented a bunch of German tweets — makes sense!</em>).</p><p>By adjusting the number of topics with pyLDAvis, we can start to get a feel for the different themes in the corpus. Eventually, you should find a ‘sweet spot’ where topics are distinct enough from one another while maintaining coherent themes.</p><h3>“Semi-Supervised” Learning with CorEx</h3><p>Another powerful topic modeling framework is <a href="https://github.com/gregversteeg/corex_topic">CorEx</a> (a portmanteau of <em>‘Correlation Explanation’</em>), which allows users to integrate their domain knowledge through the use of “anchor” words. Using these anchor words, we can highlight smaller topics that might be hidden by larger ones as well as tease them out further to get subtopics.</p><p>As an example, from pyLDAvis, a few topics emerged: Donald Trump, Hillary Clinton, police violence, race and the Black Lives Matter movement, and tweets in German. But I also remember seeing were tweets about Barrack Obama and Islam, and I want to get a sense of these as possible themes, too. I can set the following anchors (anchors can be a single word or a list) :</p><pre>anchor_topics = [[&#39;donald&#39;, &#39;trump&#39;],                                <br>                [&#39;hillary&#39;, &#39;clinton&#39;],                              <br>                [&#39;merkel&#39;, &#39;muss&#39;, &#39;die&#39;, &#39;ist&#39;, &#39;ich&#39;, &#39;das&#39;],<br>                [&#39;blacklivesmatter&#39;],<br>                [&#39;police&#39;, &#39;officer&#39;, &#39;shoot&#39;]<br>                [&#39;obama&#39;, &#39;barack&#39;],<br>                [&#39;muslim&#39;, &#39;islam&#39;]]</pre><p>Next, I’ll tell CorEx to use these 7 anchor words as topics and then find 3 more (n_hidden=10):</p><pre>topic_model = ct.Corex(<strong>n_hidden=10</strong>, <br>                       words=words,<br>                       max_iter=100, <br>                       seed=42)<br><br>topic_model.fit(doc_word, words=words, docs=proc_tweets, <br>                <strong>anchors=anchor_topics, anchor_strength=8</strong>)</pre><p>Plotting their correlation scores, I can see that the Trump and Clinton topics were the strongest in my group of anchored topics, that Black Lives Matter, police violence, and Islam were maybe less prevalent that I suspected, and that tweets about Barrack Obama could be a topic worth exploring more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uYRM6L3j8-VDvwbZG-DBdg.png" /></figure><p>Similarly, knowing that Donald Trump and Hillary Clinton are the dominant topics in this corpus, I can set several anchors for each to see what subtopics emerge:</p><pre>anchor_topics = [[&#39;donald&#39;, &#39;trump&#39;],<br>                 [&#39;donald&#39;, &#39;trump&#39;],<br>                 [&#39;donald&#39;, &#39;trump&#39;],<br>                 [&#39;donald&#39;, &#39;trump&#39;],<br>                 [&#39;hillary&#39;, &#39;clinton&#39;],<br>                 [&#39;hillary&#39;, &#39;clinton&#39;],<br>                 [&#39;hillary&#39;, &#39;clinton&#39;],<br>                 [&#39;hillary&#39;, &#39;clinton&#39;]]</pre><p>Which gives me the following subtopic keywords:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3PSa2R26WJ5H2IQcZ9lVPw.png" /></figure><p>Depending on the scope of the project, I might expand the lists of anchor words for each topic to further tease them out, or continue to explore smaller themes in the corpus.</p><h3>Creating Visualizations</h3><p>Creating visualizations for natural language processing projects is not the most intuitive task. Word clouds or bar charts depicting word frequencies can be a neat addition to a suite of visualizations, but leave much to be desired in terms of crafting a compelling narrative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZjxSt0yxzAKrTYN3mkfaDA.png" /><figcaption>Word cloud depicting vocabulary frequency in Russian troll tweets corpus</figcaption></figure><p>In the example of the Russian troll tweets dataset, we have timestamp details about when tweets were made. Plotting tweet frequency over time piques our interest: we can see a period of relatively low activity leading up to May 2016, followed by a sharp spike lasting about a year.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fs2EJrLchESdQfkvmGO1iQ.png" /></figure><p>What’s going on during this highly active period? What are they tweeting about? If we align the tweets’ topics by the date they were created, we can start to answer these questions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oEGjMb-_X6S7up7ZHp4Vew.png" /></figure><p>From pyLDAvis, I was able to identify half a dozen or so distinct, relevant topics in the corpus. From there, I labeled each tweet with its dominant topic (LDA assumes documents are a mixture of topics and provides a probability distribution of each). Plotting the topic frequencies over time starts to give us some meaningful insights: we can see that the huge spike mid-2016 can be mostly attributed to tweets about Trump leading up to the US Presidential Election that year.</p><p>Let’s say our thesis about this corpus is that these Russian accounts were being used to influence the US election in support of Trump, plotting topic frequency over time doesn’t tell the entire story. We’re missing tone — sure, there are tweets about Trump, but are they actually in support of his candidacy?</p><p>Applying sentiment analysis to the tweets might give us some insight:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JjL7ClyM7vtXCWuiMmCgdg.png" /></figure><p><a href="https://github.com/cjhutto/vaderSentiment">VADER sentiment analysis</a> is the preferred library for social media corpora and was used above on three topics: Clinton and Trump (our focal points), and the “General Twitter” topic (tweets about music, games, holidays, etc.) as a type of baseline or reference point. We quickly see the sharp contrast in tone between the tweets about Clinton and Trump, especially in our period of interest: mid-2016 through the election.</p><p>These two charts together help us construct a narrative that the US presidential candidates were the dominant focus of these accounts during the election year, and that they clearly favored of one in particular.</p><p>Another great visualization tool <a href="https://github.com/JasonKessler/scattertext">Scattertext</a>, which leverages the spaCy language model to provide rich, interactive scatterplots depicting word frequencies with regards to a label. In the example below, we have our 2016 presidential candidates on either axis and the most prevalent terms associated with each plotted, colored, and labeled in the space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wX8Y_ZXviMuJnIRI5YPI5Q.png" /><figcaption>ScatterText provides a more interactive way to view word frequencies and context by label</figcaption></figure><p>Clicking on a word or searching for it in the chart reveals the reference documents — in this case, the tweets containing the selected term.</p><p>The clusters of words in the scatterplot can also be revealing. If we look at the most salient terms used in the Clinton topic, we see a very specific theme and tight clustering. The vocabulary is concerned with the various ‘scandals’ surrounding Clinton during this time and some negative hashtags.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W8UXaMGSg9OzZxgGvmLX2w.png" /><figcaption>Cluster of most salient terms used in the Clinton topic</figcaption></figure><p>Conversely, the most salient terms used in the Trump topic are less densely clustered, and more general and slogany: ‘<em>#trumpforpresident’, ‘Republican’, </em>and ‘<em>rally’.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aRQtGWuOtoZTAx4ZQv_EWw.png" /><figcaption>Cluster of most salient terms used in the Trump topic</figcaption></figure><p>Again, we can get a sense of the tone of the tweets for these topics, albeit without a sentiment score.</p><h3>Final thoughts</h3><p>There are a lot of exciting libraries and frameworks out there for topic modeling and natural language processing — I’m barely scratching the surface. But hopefully some of you will find this as a useful overview for how to put some of them together and gives you some ideas for how to present your findings in a coherent and compelling narrative.</p><p>I still think unsupervised learning and NLP are pretty magical, and I’m even more excited to continue building my data science toolkit and finding meaning in chaos.</p><p>Again, <a href="https://github.com/scrapfishies/russian-troll-tweets-nlp">you can find all of the code on my GitHub</a>. If you found this article helpful, please 👏!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47177d20f743" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Twitter Bot or Not]]></title>
            <link>https://scrapfishies.medium.com/twitter-bot-or-not-cc2ad52b36ba?source=rss-2d189adedc19------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc2ad52b36ba</guid>
            <category><![CDATA[flask]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[twitter]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[twitterbots]]></category>
            <dc:creator><![CDATA[Chris Doenlen]]></dc:creator>
            <pubDate>Sun, 01 Nov 2020 15:54:14 GMT</pubDate>
            <atom:updated>2020-11-04T15:01:41.583Z</atom:updated>
            <content:encoded><![CDATA[<h4>Twitter bot detection using supervised machine learning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aSzD6b5zGS_S1PfMkW6Stg.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@rayhennessy?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Ray Hennessy</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Whether you’re on <a href="https://twitter.com/">Twitter</a> or stay away from social media altogether, these platforms affect us all — from shaping public discourse to entertainment to spreading information.</p><p>The existence of bots on these platforms has gained a lot of attention in recent years, and yet many people are still unaware of or misunderstand their presence and purpose on platforms like Twitter.</p><p>And so it’s important that we start with a <a href="https://en.wikipedia.org/wiki/Twitter_bot">simple working definition of a bot</a>:</p><blockquote><em>A </em><strong><em>Twitter bot is</em></strong><em> a software bot </em><strong><em>that controls a Twitter account via the Twitter API</em></strong><em>. It may autonomously perform actions like </em><strong><em>tweeting</em></strong><em>, </em><strong><em>retweeting</em></strong><em>, </em><strong><em>liking</em></strong><em>, </em><strong><em>following</em></strong><em>, </em><strong><em>unfollowing</em></strong><em>, or </em><strong><em>direct messaging</em></strong><em> other users.</em></blockquote><p>Bots are designed for a variety of purposes: they can be creative, helpful, informative, and even funny.</p><p>There are, of course, more <em>nefarious</em> bots — bots that can spread misinformation or scam other users. The presence of these bots can degrade our experience on these platforms and worse: our trust in one another.</p><p>Instead of bucketing all bots into good or bad, I think developing <strong>bot <em>awareness</em> </strong>is key to preserving social trust and the integrity of these platforms. And so with that in mind, I wanted to create a tool that could help Twitter users and spectators alike become more bot aware: <a href="https://twitter-bot-or-not.herokuapp.com/">Twitter Bot or Not</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*NSJpda1GyPBZWV6Fi1XYIw.png" /><figcaption><a href="https://twitter-bot-or-not.herokuapp.com/">Twitter Bot or Not</a></figcaption></figure><p>Users can enter in any Twitter handle and see the probability of that account being a bot based on a dozen or so account-level features (more on that to come). These predictions are intended to provide the user with peace of mind with regards to who they’re following or interacting with.</p><p>I developed a supervised machine learning classification model using Python, the <a href="https://scikit-learn.org/">sklearn</a> library, and <a href="https://xgboost.readthedocs.io/en/latest/">XGBoost</a>, and then built a simple app using Flask that I deployed on Heroku. In this post, I’ll walk through a few of the steps that went into developing the model and the app. All of the <a href="https://github.com/scrapfishies/twitter-bot-detection">code and notebooks can be found on my GitHub</a>.</p><h3>Dataset and model features</h3><p>For this project, I used the <a href="https://www.kaggle.com/davidmartngutirrez/twitter-bots-accounts">Twitter Bot Accounts dataset on Kaggle</a>, which has account-level information approximately 37,000 Twitter users, labeled as ‘bot’ or ‘human’. <em>Account-level information</em> means we’re not actually looking at tweets, but rather activity information like number of tweets and likes, network information like number of follows and friends, whether or not the user is verified, etc.</p><p>When exploring the dataset, right off the bat we can see that bot and human accounts <em>behave</em> differently — for example, humans tweet more frequently than their robot counterparts:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*FGPOTcBq9O8lZKrstCzz5w.png" /><figcaption>Average Tweets per Day by Account Type</figcaption></figure><p>Their <em>networks</em> also look differently, too — humans have bigger networks, both in terms of accounts they follow and that follow them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3P2YiD8hjeoYPlaYQmNy8g.png" /><figcaption>Distributions of Followers and Friends by Account Type</figcaption></figure><p>I used these account-level details, along with a few other engineered metrics such as overall network size and reach as the features in the model.</p><h3>Model scoring and evaluation</h3><p>I wanted this model to accurately label bots as such, but not at the expense of labeling everything a bot — so I sought to find and tune a model that would achieve a balance between both <a href="https://en.wikipedia.org/wiki/Precision_and_recall">precision and recall</a>.</p><p>I tried a number of classification models from the sklearn library: Logistic Regression, K-Nearest Neighbors, Naive Bayes, Decision Tree, Random Forest, and XGBoost. Recognizing the non-linear nature of the data and the number of features, it wasn’t a surprise that the last two performed best ‘out of the box’ with regards to balanced precision and recall scores.</p><p>With some additional parameter tuning, XGBoost stood out as the best model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*SZwrBoahJi5x8mvP8Ut18Q.png" /><figcaption>KFolds Cross-Validation Precision-Recall Curves for XGBoost Model</figcaption></figure><h3>Building the Flask app</h3><p>After training on the full dataset, I pickled the final model to be used in a Flask app that would allow others to the check bot-likeliness of any Twitter users.</p><p>The app itself would be minimal — just a search bar where users could type in the Twitter account handle and the results. The structure of the app was easily coded up with a few lines of HTML, and I used the <a href="https://bulma.io/">Bulma</a> CSS framework to help with a simple and clean style.</p><p>To pull live Twitter information via their API, I <a href="https://developer.twitter.com/en">set up a developer account</a>, which is a straightforward application of providing a few details of who you are and your intentions. Within a few hours of submission, I had my access keys.</p><p>With the help of <a href="https://www.tweepy.org/">Tweepy</a>, a Python library for interfacing with the Twitter API, I was able to pull the same account-level information I needed to make predictions on new, live users.</p><p>If you’re new to Tweepy, here’s a nice chunk of starter code that’ll allow you to access your home timeline, create a Pandas DataFrame of tweets by a specific user or topic, and actually send tweets from Python.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0f956fd6ff97175b461cdd66859a4775/href">https://medium.com/media/0f956fd6ff97175b461cdd66859a4775/href</a></iframe><p>This was my first time creating a Flask app, let alone deploying anything. Fortunately, <a href="http://heroku.com/">Heroku</a> is pretty straightforward for new developers like myself to get <a href="https://twitter-bot-or-not.herokuapp.com/">their first projects</a> live.</p><h3>Final thoughts</h3><p>I’m quite pleased with how this project turned out. I think the model does a reasonable job of making predictions on live data and I found myself lurking Twitter feeds playing ‘bot or not’: looking for trollish or bot-like comments and running a search on their handle with the app.</p><p>Of course there’s no way to be sure, but it’s pretty fun when your hunch is validated by AI.</p><p>I’m extremely interested in using Twitter more in the future and I’m excited to add natural language processing to my toolkit to dig deeper into the data. I see this project as the first piece in a larger body of work that explores networks, bubbles, and the nature of our curated realities on these platforms.</p><p>Thanks for reading — please <a href="https://twitter-bot-or-not.herokuapp.com/">check out the app </a>and let me know <em>your</em> bot-likelihood!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc2ad52b36ba" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to find good apartment deals Craigslist]]></title>
            <link>https://medium.com/analytics-vidhya/how-to-find-good-apartment-deals-craigslist-f21967420059?source=rss-2d189adedc19------2</link>
            <guid isPermaLink="false">https://medium.com/p/f21967420059</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[predictive-analytics]]></category>
            <category><![CDATA[linear-regression]]></category>
            <dc:creator><![CDATA[Chris Doenlen]]></dc:creator>
            <pubDate>Sun, 11 Oct 2020 19:14:25 GMT</pubDate>
            <atom:updated>2020-10-13T12:54:26.094Z</atom:updated>
            <content:encoded><![CDATA[<h4>Predictive modeling</h4><h4>Predicting rental prices with linear regression</h4><p>A year ago my partner and I were looking for a new apartment. We weren’t looking for anything fancy, just a one bedroom that allowed dogs, with an outdoor space for a garden, and didn’t cost an arm-and-a-leg.</p><p>If you live in the Bay Area, or likely any hub city in the U.S., you know where this story is going: multiple home viewings in which you wait in a line of prospective renters all vying for the landlord’s attention and favor.</p><p>I have never hated perfectly nice strangers more than in this moment.</p><p>The competitive nature of apartment hunting means we as seekers need to be ready to act immediately to secure a lease. Speed comes at the expense of careful evaluation and assurance that what we’re getting is a fair deal. With this in mind, I wanted to create a predictive model that would help renters like myself better and more quickly assess their options.</p><p>For this <a href="https://github.com/scrapfishies/CL-housing-rent-predictions">project</a>, I scraped a little over 3,000 San Francisco apartment/housing listings on Craigslist and used linear regression to predict rental prices based on a set of standard features available on most postings. I’ll share a high level outline of my web scraping and data cleaning and preparation processes, and then get into my approach for building and evaluating the model.</p><p>All of the code and details of this project can be found on my <a href="https://github.com/scrapfishies/CL-housing-rent-predictions">GitHub repository</a> .</p><h3>Anatomy of a Craigslist post</h3><p>For those new to apartment hunting on Craigslist, let’s take a quick look at a pretty typical post:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RUM7bkUU3elL6tkZyWftSA.png" /><figcaption>Typical Craigslist post for an apartment in San Francisco</figcaption></figure><p>Craigslist listings can vary wildly as posters are only required to provide a title, rent amount, and fill out an open field text box. However, there are additional standard fields that can be utilized: location (for most major cities, there are neighborhood options), square footage, number of bedrooms and bathrooms, and then some basic information about amenities offered like laundry facilities, whether or not pets are allowed, and parking.</p><p>The consistency of these fields, both in phrasing and placement on the webpage, means they can be easily scraped and used as features in a model.</p><h3>Web scraping Craigslist</h3><p>When I first started this project and was searching for information on scraping Craigslist, I came across <a href="https://towardsdatascience.com/web-scraping-craigslist-a-complete-tutorial-c41cea4f4981">this great tutorial for scraping the results listings pages</a>. This article was a helpful start, but I knew I’d need to go into each individual posting to get additional features like number of bathrooms and amenities.</p><p>The scraping process:</p><ul><li>Scrape the <a href="https://sfbay.craigslist.org/search/sfc/apa?bundleDuplicates=1&amp;availabilityMode=0&amp;sale_date=all+dates">first page of listings results</a> for date posted, title, url, rent amount, square footage, number of bedrooms, and neighborhood (location) and add it to a pandas DataFrame. There are usually about 120 listings on each page.</li><li>Scrape each of those 120 urls from that listing results page to extract number of bathrooms and the amenities, and append those features to the DataFrame.</li><li>Move to the next results listing page and repeat until all the results pages and individual posts have been scraped.</li></ul><p>Here’s one of the helper functions to compile a results page worth of listings:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/69a0dcdb06a0772b8fc77cfb5021bf6d/href">https://medium.com/media/69a0dcdb06a0772b8fc77cfb5021bf6d/href</a></iframe><h3>Data cleaning and preparation</h3><p>With the listings safe and sound in a DataFrame, the next step is to clean and prepare the data for modeling:</p><ul><li>remove duplicates (in this scrape, there were a little over 600)</li><li>format bedrooms and bathrooms (e.g. convert bathroom entries like ‘splitBa’ to 1)</li><li>parse through the list of amenities and create distinct categories for laundry, pets, and parking</li><li>reduce number of locations to a workable number of categories</li><li>and finally, drop rows with missing or bad data (e.g. wildly incorrect square footage figures, listings for apartments that are not in San Francisco, etc.), and outliers (e.g. a listing for an apartment that was over $25k/month! Who are you and why are you looking on Craigslist anyways?)</li></ul><p>Here’s an example of searching through the amenities list to parse for pet restrictions:</p><pre>def pets_allowed(amen_list):<br>    if (&#39;dogs are OK - wooof&#39; and &#39;cats are OK - purrr&#39;) in amen_list:<br>        return &#39;(a) both&#39;<br>    elif &#39;dogs are OK - wooof&#39; in amen_list:<br>        return &#39;(b) dogs&#39;<br>    elif &#39;cats are OK - purrr&#39; in amen_list:<br>        return &#39;(c) cats&#39;<br>    else<strong>:</strong><br>        return &#39;(d) no pets&#39;</pre><pre>sf[&#39;pets&#39;] = sf[&#39;amens_list&#39;].apply(lambda amen_list pets_allowed(amen_list))</pre><p>As I mentioned earlier, Craigslist allows posters to assign the location of their offering from a list of standard neighborhoods. For San Francisco, there are about 35 unique neighborhoods to choose from. Posters also have the option of using an open field text box to provide their own location details. As a result, there were over a 100 unique values for location in the DataFrame — many of them duplicates (e.g. ‘soma’, ‘SoMa’, ‘SOMA’, and ‘South of Market’), and many of them not in San Francisco (sometimes not even in the Bay Area).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*Szev83uDjvLt46sAGJHeuA.png" /><figcaption>SFAR District Map</figcaption></figure><p>The San Francisco Association of Realtors has provided a map dividing the city into 10 districts, and <a href="https://data.sfgov.org/Geographic-Locations-and-Boundaries/Realtor-Neighborhoods/5gzd-g9ns">Data SF created an interactive version</a> that allows users to easily find a district based on smaller neighborhood designations.</p><p>With this map as my guide, I was able to reduce the number of unique location values to these 10 districts, which would significantly reduce the number of dummy variables used in the model later.</p><p>After a thorough scrub, the dataset was whittled down to just under 1,000 rows and 8 features that could be used to help predict rental prices.</p><h3>Model build, results, and evaluation</h3><h4>Exploratory data analysis</h4><p>Before modeling, it’s important to explore the data and understand the relationships between possible features and the target variable.</p><p>Below, we can see that the laundry facilities offered in these listings is positively correlated with price, with more ideal or preferred options associated with higher rents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*xCQx1u_fJ0Rf4zlOvefqZQ.png" /><figcaption><a href="https://seaborn.pydata.org/generated/seaborn.catplot.html">Catplot</a> depicting price range by laundry facilities</figcaption></figure><p>Similarly, parking availability may also command a premium. As anyone in the Bay Area knows, it’s not a matter of <em>if</em> your car gets broken into — it’s <em>when</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*Si-HyKzoXX_sxfTEzrFJ2w.png" /><figcaption>Catplot depicting price by parking situation</figcaption></figure><p>Surprisingly, building type — whether it was a single residence home or multi (e.g. duplex, apartment buildings) — were not strongly correlated with price. Thus, this feature didn’t make it into the model.</p><h4>Feature selection and engineering</h4><p>The final features used in the model could be categorized as:</p><ul><li>continuous (sqft)</li><li>discrete (number of bedrooms, bathrooms)</li><li>ordinal (parking, laundry), and</li><li>categorical (pet restrictions, neighborhood district)</li></ul><p>To improve the model’s predictive power, I used s<a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PolynomialFeatures.html">klearn’s PolynomialFeatures</a>, which would generate interactions among the different features. Then I applied <a href="https://towardsdatascience.com/ridge-and-lasso-regression-a-complete-guide-with-python-scikit-learn-e20e34bcbf0b">Ridge and Lasso regularization</a> to reduce complexity, prevent overfitting, and tease out which features had the biggest impact on rental price.</p><p>After a rigorous <a href="https://machinelearningmastery.com/k-fold-cross-validation/">cross-validation process using K-folds</a>, both scored similarly on R-squared, however the Ridge model scored most consistently between training and validation datasets on root mean squared error (RMSE) and mean absolute error (MAE) metrics, and so I moved forward with Ridge for the final model.</p><h4>Final model and results</h4><p>After fitting the model with the full training set, the model’s test scored reasonably well:</p><pre>R-squared: 0.77<br>Root Mean Squared Error (RMSE): $477<br>Mean Absolute Error (MAE): $366</pre><p>The plotted prediction prices compared with actual values is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*GYb_VRLRJWhqDkx3LsouIw.png" /></figure><p>The fitted prediction line almost aligns perfectly with the identity line, starting to break away somewhere around the $4,500/month range. The residual plot provides a better view:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*i_t9OrZoSYxNlOaxaOhvCA.png" /></figure><p>As most of the listings in the sample fell in the $2,500 to $4,000/month range, it seems reasonable that the model performs best in this window. The distribution of prices for the full sample can be seen below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*gl1M--CgluavYei4-EGe9Q.png" /><figcaption>Distribution of rental prices for full Craigslist sample</figcaption></figure><h3>Final thoughts and suggestions for future work</h3><p>Ultimately, I think the model does a pretty good job of predicting rental prices up to around $4,000/month. Tolerance for the ~$400 swing (via RMSE and MAE) should be considered in relation to the rental price and will be subjective to the seeker’s budget.</p><p>For example, a $400 delta might be significant for a single person, but may be completely acceptable when splitting between two or more people.</p><p>Still, there are likely other factors influencing rental prices:</p><ul><li>rent control and history (especially in the Bay Area)</li><li>other amenities offered but not standard in Craigslist postings (e.g. fitness facilities, common spaces, bike storage, etc)</li><li>other environmental or economic factors</li></ul><p>On the last point, <a href="https://www.sfchronicle.com/bayarea/article/Bay-Area-rents-keep-plummeting-especially-in-15613722.php">we’re currently seeing an unprecedented decline in San Francisco rents</a> due to COVID migration. Those moving out of the city may be people who have the means to move and work remotely during the pandemic. Thus, the sample of the listings scraped in early October might not be the best representation of Craigslist’s normal offerings.</p><h4>Suggestions for future work</h4><p>I would love to take this project farther and improve the model’s predictive power. Here are some ideas for future work:</p><ul><li>sample again once COVID’s influence subsides</li><li>increase scope of scraping methods to find/create additional features (search body of text for key words like ‘gym’ or ‘backyard’ or find information about security deposit and leasing requirements)</li><li>implement image processing tools to evaluate a unit’s quality through posted photos</li><li>scam post identification</li></ul><p>I’d also love to hear from you if you have additional ideas for making <a href="https://github.com/scrapfishies/CL-housing-rent-predictions">this model</a> more precise or if you’re interested in adapting it to your own city or town — let’s talk!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f21967420059" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/how-to-find-good-apartment-deals-craigslist-f21967420059">How to find good apartment deals Craigslist</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Does MTA Turnstile Data Follow Benford’s Law?]]></title>
            <link>https://medium.com/swlh/does-mta-turnstile-data-follow-benfords-law-e236028bd5b5?source=rss-2d189adedc19------2</link>
            <guid isPermaLink="false">https://medium.com/p/e236028bd5b5</guid>
            <category><![CDATA[benfords-law]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Chris Doenlen]]></dc:creator>
            <pubDate>Sun, 27 Sep 2020 23:00:51 GMT</pubDate>
            <atom:updated>2020-09-29T19:35:54.256Z</atom:updated>
            <content:encoded><![CDATA[<p>I recently learned about <a href="https://en.wikipedia.org/wiki/Benford%27s_law"><strong>Benford’s Law</strong></a><strong> </strong>through the Netflix series <a href="https://www.netflix.com/title/81031737#:~:text=Digits&amp;text=Latif%20explores%20a%20law%20of,and%20perhaps%20the%20entire%20universe.">Connected, with Latif Nasser</a>. Episode 4: ‘Digits’ explores how this phenomenon applies to music, social media, tax fraud, and more. It’s a fun show and I’d highly recommend checking it out if you’ve got the time.</p><p>A quick primer on Benford’s Law from Wikipedia:</p><blockquote><strong>Benford’s law</strong>, also called the <strong>Newcomb–Benford law</strong>, the <strong>law of anomalous numbers</strong>, or the <strong>first-digit law</strong>, is an observation about the frequency distribution of leading digits in many real-life sets of numerical data. <strong>The law states that in many naturally occurring collections of numbers, the leading digit is likely to be small.</strong> For example, in sets that obey the law, the number 1 appears as the leading significant digit about 30% of the time, while 9 appears as the leading significant digit less than 5% of the time. If the digits were distributed uniformly, they would each occur about 11.1% of the time. Benford’s law also makes predictions about the distribution of second digits, third digits, digit combinations, and so on.</blockquote><p>This frequency distribution is visualized below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*dBpyRAfYktBK36xKT6HpxA.png" /><figcaption>Benford’s Law describes the frequency distribution of leading digits in observed data</figcaption></figure><p>Since watching the episode, I’ve had it in the back of my mind that I’d like to explore a dataset’s <em>Benfordness</em>. Last week I started <a href="https://www.thisismetis.com/">Metis’ Data Science Bootcamp</a> and our first project was to conduct an exploratory analysis on <a href="http://web.mta.info/developers/turnstile.html">MTA turnstile data</a>. With the project behind me, I decided to dig a little more into the data and see how well NYC’s subway traffic patterns fit Benford’s Law.</p><p>In this post, I’ll outline my process for data collection and preparation and share some of my findings. I’ll also provide some snippets of the code used along the way. You can find the full details in <a href="https://github.com/scrapfishies/mta_benfords">my Jupyter Notebook on GitHub</a>.</p><h3>Data Cleaning &amp; Preparation</h3><p>I decided to import the last four weeks of available data which, at the time, were the weeks of Sept 5, Sept 12, Sept 19, and Sept 26, 2020.</p><pre>def get_data(week_nums):<br>    url =     &quot;http://web.mta.info/developers/data/nyct/turnstile/turnstile_{}.txt&quot;<br>    dfs = []     <br>    for week_num in week_nums:         <br>        file_url = url.format(week_num)<br>        dfs.append(pd.read_csv(file_url))<br>    return pd.concat(dfs)</pre><pre><br>import_weeks = [200926, 200919, 200912, 200905]<br><br>mta_raw = get_data(import_weeks)</pre><p>Below you can see a view of the raw data set in a Pandas data frame:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nu_sPt5JH53jyXs7LfOMDQ.png" /><figcaption>Raw dataframe from MTA’s turnstile data</figcaption></figure><p>The MTA turnstile counters provide cumulative figures for each time period provided (usually 4 hours). For example, we might see 15,470,879 at 4am and then 15,470,894 at 8am. To get the actual number of entries or exits in a given time period, I needed to take the difference between rows.</p><p>Kaggle provided a simple approach in their <a href="https://www.kaggle.com/nieyuqi/mta-turnstile-data-analysis">MTA Turnstile Data Analysis</a> article which I used here and added the ‘abs()’ method to account for the few instances where turnstiles are actually counting <em>backwards</em>.</p><pre>mta_sorted = mta_raw.sort_values([&#39;turnstile&#39;, &#39;timestamp&#39;])<br>mta_sorted = mta_sorted.reset_index(drop = True)<br><br>turnstile_grouped = mta_sorted.groupby([&#39;turnstile&#39;])<br><br>mta_sorted[&#39;entries&#39;] = <br>    turnstile_grouped[&#39;entries_cum&#39;]<br>    .transform(pd.Series.diff).abs()</pre><pre>mta_sorted[&#39;exits&#39;] = <br>    turnstile_grouped[&#39;exits_cum&#39;]<br>    .transform(pd.Series.diff).abs()</pre><p>When counters reset, it creates some pretty big outliers. I decided to toss out anything above 10,000 entries or exits per turnstile per time period. This works out to about 40 people per minute, which isn’t unreasonable for some stations.</p><p>After scrubbing the data, I set up a few data frames for Benford’s testing.</p><h3>Finding the leading digit for each entry/exit</h3><p>Here I set up new columns that would hold the value of the leading digit for each entry, exit, and total value. Values were first converted to strings, spliced to return the first character, and then converted back to integers.</p><pre>mta[&#39;ent_dig&#39;] = mta[&#39;entries&#39;].astype(str)<br>mta[&#39;ent_dig&#39;] = (mta[&#39;ent_dig&#39;]<br>                  .apply(lambda x: x[0])).astype(int)<br><br>mta[&#39;ex_dig&#39;] = mta[&#39;exits&#39;].astype(str)<br>mta[&#39;ex_dig&#39;] = (mta[&#39;ex_dig&#39;]<br>                 .apply(lambda x: x[0])).astype(int)<br><br>mta[&#39;total_dig&#39;] = mta[&#39;total&#39;].astype(str)<br>mta[&#39;total_dig&#39;] = (mta[&#39;total_dig&#39;]<br>                    .apply(lambda x: x[0])).astype(int)</pre><p>Some turnstiles had zero entries or exits for some time periods. Because we’re only interested in getting the leading digits for 1 through 9, I excluded those rows from subsequent data frames by applying a simple filter.</p><pre>benford_ents = mta[mta[&#39;ent_dig&#39;] &gt; 0] <br>benford_exits = mta[mta[&#39;ex_dig&#39;] &gt; 0] <br>benford_total = mta[mta[&#39;total_dig&#39;] &gt; 0]</pre><p>Below you can see a cleaned up data frame with leading digits in the last three columns:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ls07o1tTlydId4BvcXn0Fg.png" /><figcaption>Cleaned up dataframe with leading digit columns</figcaption></figure><p>From here, I set up slimmer data frames to check <em>Benfordness</em>.</p><h3>Setting up the Entries DataFrame</h3><p>With the zeroes gone, I set up some lighter data frames to work with. The following creates a DataFrame for entries with the value counts of each leading digit.</p><pre>benford_ents = benford_ents[[<br>                            &#39;station&#39;, &#39;date&#39;, &#39;time&#39;,<br>                            &#39;timestamp&#39;, &#39;weekday&#39;, <br>                            &#39;weekday_num&#39;, &#39;ent_dig&#39;]]</pre><pre>nums = list(range(1, 10))<br>benford_ents_count = list(benford_ents.ent_dig.value_counts())<br>benford_ents_df = pd.DataFrame({&#39;digit&#39;: nums, <br>                              &#39;counts&#39;: benford_ents_count},<br>                              index=nums)</pre><p>Next, I calculated the total number of counts so I could take the percent frequency of each digit and compare it to Benford’s Law.</p><pre>total_entries = benford_ents_df.counts.sum()</pre><pre><em># Creating % Frequency Column</em><br>benford_ents_df[&#39;percent&#39;] = benford_ents_df[&#39;counts&#39;]<br>    .apply(lambda x: (x / total_entries) * 100)<br>    .round(decimals=1)<br><br><em># Adding Benford&#39;s Law %s, <br># and some calculation columns to see the difference</em><br>benford_ents_df[&#39;benfords&#39;] = benfords[&#39;percent&#39;]</pre><pre>benford_ents_df[&#39;diff_abs&#39;] = <br>    (benford_ents_df[&#39;percent&#39;] - benford_ents_df[&#39;benfords&#39;]).abs()</pre><pre>benford_ents_df[&#39;diff_perc&#39;] = <br>    (benford_ents_df[&#39;diff_abs&#39;] / benford_ents_df[&#39;percent&#39;]) * 100</pre><p>This provided a simple view of the digit counts, percent frequency, and how well it compares to Benford’s distribution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_BjlWQXfNoc3YkZ7bfmNgw.png" /></figure><p>From the table view above, we can already see that total entries is pretty <em>Benfordy</em><strong><em>. </em></strong>Plotting the results side by side gives a better view.</p><h3>Plotting the results</h3><p>Here we can see Benford’s distribution next to total entries:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*k9XcHPTmUMHoAOJVnDIeig.png" /><figcaption>Total entries for this time period lines up nicely with Benford’s distribution</figcaption></figure><p>I’d say entries lines up quite nicely with Benford’s distribution!</p><p>Let’s take a look at exits:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*s0SlF6B7VVv31BGJksKAvg.png" /><figcaption>Total exits compared with Benford’s distribution</figcaption></figure><p>Pretty spot on.</p><p>Curious to see if high traffic stations in Manhattan were somehow biasing the data, I decided to compare a high traffic station in Manhattan with a more residential station in a different borough: Times Square 42nd Street and Prospect Av in Brooklyn.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*_YVBOzEN0Xves-6ttEc8zw.png" /><figcaption>Comparing Benford’s law with TmSq-42nd St and Prospect Av stations</figcaption></figure><p>These both seem to fit Benford’s to a T! We can see a few instances where Times Square fits better than Prospect Av, and vice versa. But just eyeballing the differences, I’d call it a wash as to which station fits best.</p><p>Most importantly, the overall principle remains true for each station: frequency of smaller leading digits is greater than larger ones. In no instance is the frequency of a 4, for example, less than any larger number.</p><h3>Final Thoughts and Suggestions for Further Analysis</h3><p>Every <a href="https://github.com/scrapfishies/mta_benfords/tree/master/images">view</a> I took of the MTA turnstile data seemed to fit Benford’s Law. This raises a beautiful idea that could be better supported with further analysis: the entire MTA subway system shares the same properties as Fibonacci numbers, stock market values, income tax data, and other data observed in the natural sciences.</p><p>It makes me think of the NYC subway system as this living organism that obeys certain natural laws, just like any other, because the data ultimately reflects our humanness.</p><h4><strong>Suggested further analysis</strong></h4><p>Please feel free to use my <a href="https://github.com/scrapfishies/mta_benfords/blob/master/mta_benfords.ipynb">notebook</a> as a starting point!</p><ul><li>Compare other stations for <em>Benfordness</em></li><li>Sample other time periods, both larger and shorter than 4 weeks</li><li>Test for <em>Benfordness</em> at the turnstile level</li><li>Are there any views of the MTA turnstile data that disobey Benford’s Law?</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e236028bd5b5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/does-mta-turnstile-data-follow-benfords-law-e236028bd5b5">Does MTA Turnstile Data Follow Benford’s Law?</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>