<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by M.M. on Medium]]></title>
        <description><![CDATA[Stories by M.M. on Medium]]></description>
        <link>https://medium.com/@melissamunz?source=rss-c1db7fc8d19d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*UGIYqJdwrTF9aRKZyfk4ow.png</url>
            <title>Stories by M.M. on Medium</title>
            <link>https://medium.com/@melissamunz?source=rss-c1db7fc8d19d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 02:45:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@melissamunz/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Exploratory Data Visualizations with Cornell Ornithology’s eBird Data]]></title>
            <link>https://medium.com/@melissamunz/exploratory-data-visualizations-with-cornell-ornithologys-ebird-data-7ad3b90cb4bd?source=rss-c1db7fc8d19d------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ad3b90cb4bd</guid>
            <dc:creator><![CDATA[M.M.]]></dc:creator>
            <pubDate>Mon, 25 May 2020 02:35:55 GMT</pubDate>
            <atom:updated>2020-05-25T02:35:55.869Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mn6zGy-5btubxcD7-ajAIw.png" /></figure><p>I wrote a blog earlier detailing the features contained within Cornell Ornithology’ eBird dataset. Today I’ll discuss some of the preliminary exploratory analysis on those features.</p><h4>Quick data overview:</h4><ul><li>The eBird API dataset for Manhattan had 1,699,064 unique sightings and 47 features.</li><li>It encompasses all Manhattan bird sightings in the eBird records from 1880–2020.</li><li>There are 7,102 distinct contributors to the dataset who have logged observations during that time.</li><li>All sightings in this dataset have been marked as ‘Approved’, meaning essentially that they were peer reviewed — thus more credible.</li></ul><h4>Visualizing Missing Data:</h4><p>The first thing I wanted to know what how much data I really had, which would help me to zoom in on the features that are worth keeping. I used an amazing tool for visualizing missing data called <a href="https://github.com/ResidentMario/missingno">missingno</a>. The following visualizations show which columns are robust (gray) and which have a lot of NaN values (white).</p><pre>!conda install --yes -c conda-forge missingno</pre><pre>import missingno as msno<br>%matplotlib inline</pre><pre># How many missing values are there in our dataset?<br>msno.matrix(df, figsize = (30,5))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lHNvCyZnZSAYj4MAcawiPQ.png" /></figure><p>I also looked at them in ascending order:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AwJfdz36QkR_idNC292Lkw.png" /></figure><p>Right off, I can see that some columns are missing too much data to be useful in this analysis. I all columns with less than 20% of data present, as none seemed candidates for any kind of meaningful imputation. However, I did keep the comment columns for natural language processing.</p><p>I also dropped all columns having to do with location besides the latitude and longitude (all the data is from Manhattan).</p><p>Finally, I also used the missingno correlation heatmap, which measures nullity correlation: how strongly the presence or absence of one variable affects the presence of another. Basically, nullity correlation ranges from -1 (if one variable appears the other definitely does not) to 0 (variables appearing or not appearing have no effect on one another) to 1 (if one variable appears the other definitely also does). Note that the heatmap does not include features with full completion.</p><pre>msno.heatmap(df)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PP07RIVC5XU7fN4sBpNiRg.png" /></figure><p>A lot of these make some sense — those who reported when their observations started also reported the duration of those observations. However, they weren’t terrible helpful in my feature selection efforts.</p><p>My final missing data visualization is the dendrogram, which more specifically shows relationships between a the impact of a features presence on another.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6noiDXI8GG2sl8pYUF2IXg.png" /></figure><h3>Initial Feature Visualizations:</h3><p>To look at the initial spread of data for each, the following function performs EDA visualizations for categorical and numeric data:</p><pre># Let’s plot the distribution of each feature</pre><pre>def plot_distribution(dataset, cols=5, width=20, height=15, hspace=0.2, wspace=0.5):<br>    <br>    #set up the seaborn style. <br>    plt.style.use(&#39;seaborn-whitegrid&#39;)<br>    fig = plt.figure(figsize=(width,height))<br>    <br>    fig.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=wspace, hspace=hspace)<br>    <br>    #math.ceil() : Python number method ceil() returns ceiling value of x - the smallest integer not less than x.<br>    rows = math.ceil(float(dataset.shape[1]) / cols)<br>    <br>    #Determine visualization type based on the data types!<br>    for i, column in enumerate(dataset.columns):<br>        ax = fig.add_subplot(rows, cols, i + 1)<br>        ax.set_title(column)<br>        if dataset.dtypes[column] == np.object:<br>            g = sns.countplot(y=column, data=dataset)<br>            substrings = [s.get_text()[:18] for s in g.get_yticklabels()]<br>            g.set(yticklabels=substrings)<br>            plt.xticks(rotation=25)<br>        else:<br>            g = sns.distplot(dataset[column])<br>            plt.xticks(rotation=25)<br>    <br># plot_distribution(df_subset, cols=3, width=20, height=20, hspace=0.45, wspace=0.5)</pre><h4>Categorical Features:</h4><pre>df_cat = df[[&#39;CATEGORY&#39;, &#39;LOCALITY TYPE&#39;]]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*FIcMabrUpZKLK4mRHs1G5Q.png" /></figure><p>CATEGORY: Righ off, we can see the most common sightings are ‘species’, followed by a much smaller number of ‘domestic’ (distinctly-plumaged domestic varieties)and issf (identifiable subspecies). However, two things are important to note here:</p><ul><li>On closer examination, the form/slash/hybrid’s are not actually 0, but in fact there are ~1000+ sightings for each;</li><li>In the world of birding, rare is desirable! Birders WANT to see these rare hybrids and form(sports).</li></ul><p>LOCALITY TYPES: Additionally, and surprisingly, most of the ‘Locality types’ are ‘H’ or Hotspots , followed by ‘P’ or Personal. The eBird website states that ‘<em>Hotspots represent a set of public locations that people regularly visit for birding’. </em>Thus, after looking at the other features, I determined this is likely because locations like Central Park and Inwood Park, which both account of a huge percentage of bird sightings also just tend to be the places people go to look at birds.</p><h4>Numeric Features:</h4><pre>df_numeric = df[[&#39;OBSERVATION_COUNT&#39;, &#39;MONTH&#39;, &#39;YEAR&#39;, &#39;NUMBER OBSERVERS&#39;]]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*zTezvNMUmNmX3p42N731dQ.png" /></figure><ul><li>OBSERVATION COUNT : Overall, most sightings are of a single bird. However, we should investigate <strong>outliers </strong>— some counted up to 12,000 birds?!</li><li>MONTH: Most sightings are April-June, followed by Sept-Oct which makes perfect sense — those are peak migration times.</li><li>YEAR: This is an important one. There is a huge spike in overall sightings around the year 2000 — likely the result of greater adoption of cell phone and ubiquitous personal computers. The spike is so big that it may be difficult to compare data before 2000 and after.</li><li>NUMBER OBSERVERS : Mostly small groups of 1–2. Should investigate <strong>outliers </strong>again, some of these groups appear to be 1000+ people!</li></ul><h4>Boolean Features:</h4><pre>sns.countplot(df[&#39;HAS MEDIA&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*AzWK5hDjj2R7roGSQs5Slw.png" /></figure><ul><li>HAS MEDIA: This is an ebird specific feature, ‘Has Media’ refers to whether or not that entry was accompanied by a photograph or video that was uploaded onto the eBird site. Huge imbalance in those who do, but worth looking into how that may have changed over time.</li></ul><h4>Other features:</h4><p>However, I did encounter a couple issues. Immediately, it was clear that several of my features were far too large to visualized in this way. Those included:</p><ul><li>COMMON NAME: With 442 distinct species, it’s difficult to visualize overall spread. I took a look at the top 20 species — all are extremely common in this area. The house sparrow, a ubiquitous invasive species, was spotted over 75,000 times!</li></ul><pre>df[&#39;COMMON NAME&#39;].value_counts()[:20].plot(kind=&#39;barh&#39;, figsize=(20,8))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s-uLmbxlWH9yttOvuxP9Ug.png" /></figure><ul><li>OBSERVER ID: With 7,102 distinct contributors, I wanted to look closer at those top observers. Obviously we don’t know much about the person’s behind these observer id’s but it is noteworthy that our top observer (obsr338513) has logged 72,980 bird sightings!!! Definitely need to investigate the time frame and frequency — perhaps its a group?</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1W67gnkBH-58jxwIEuip_A.png" /></figure><ul><li>TRIP and SPECIES COMMENTS: In preparation for doing some nlp analysis of the comments sections, I spent a little time investigating what the comments look like.</li></ul><pre>def print_trip_comments(index):<br>    &quot;&quot;&quot;Returns a single row sample from the comment column with corresponding unique identifier&quot;&quot;&quot;<br>    example = df[df.index == index][[&#39;GLOBAL UNIQUE IDENTIFIER&#39;, &#39;TRIP COMMENTS&#39;]].values[0]<br>    if len(example) &gt; 0:<br>        print(example[0])<br>        print(&#39;TRIP COMMENTS:&#39;, example[1])</pre><p>Output for a specific Trip Comment:</p><pre>URN:CornellLabOfOrnithology:EBIRD:OBS35685993</pre><pre>TRIP COMMENTS: Report not submitted to eBirds NYC.----Date: Tue, 26 Jul 2005 (3:35p-8:00p)--Location: Central Park - reservoir, The Pond &amp; Heckscher Ballfields ----** 27 species</pre><p>Output for a specific Species Comment:</p><pre>URN:CornellLabOfOrnithology:EBIRD:OBS345270913</pre><pre>SPECIES COMMENTS: very dull</pre><p>Most follow along these same lines — about the location or appearance of the birds.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ad3b90cb4bd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NYC eBird Migration Project]]></title>
            <link>https://medium.com/@melissamunz/nyc-ebird-migration-project-44fc2741f1b1?source=rss-c1db7fc8d19d------2</link>
            <guid isPermaLink="false">https://medium.com/p/44fc2741f1b1</guid>
            <dc:creator><![CDATA[M.M.]]></dc:creator>
            <pubDate>Fri, 01 May 2020 21:10:00 GMT</pubDate>
            <atom:updated>2020-05-01T21:10:00.535Z</atom:updated>
            <content:encoded><![CDATA[<h4>Exploratory Data Analysis</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*h1sriun_n8wPS_gNm6XBfA.jpeg" /></figure><p>To counter my cabin fever and to fulfill my desire to view the bird migration during a worldwide pandemic, I started doing some analysis on eBird’s amazing datasets.</p><p>On the Cornell Ornithology Lab’s <a href="https://ebird.org/home">ebird.org</a> site you can request access to their data and explain why you want it. It’s takes some time to be granted access, but once you have it a century of bird data opens up to you. For this project, I filtered my dataset by New York City on the ebird site, and a few hours later I was emailed an enormous .txt file with my data!</p><h4><strong>Overview of NYC bird data</strong>:</h4><p>To provide some context on the data, every row on ebird is an individual bird observation. A single bird watcher records where, when and what they saw with a rather detailed text description to accompany the 47 meta features. Incredibly, much of this data has been extremely well recorded over the past century, with only ten of the columns containing a majority missing values.</p><p>Additionally, the data comes in a package that also contains a kind of Readme guide in the form of a pdf which provides a thorough overview of the file format and each feature of the dataset.</p><h4><strong>Important Feature Information</strong>:</h4><ul><li><strong>Global Unique Identifier</strong>: Each bird sighting has a unique key.</li><li><strong>Common Name</strong>: The bird species. There are a total of 442 distinct species in this data.</li><li><strong>Category</strong>: A taxonomy detailing hybrids and non-natives, etc.</li><li><strong>Latitude/Longitude</strong>: Will be useful for mapping the bird sightings.</li><li><strong>Observation Date</strong>: The date the bird(s) were sighted, which I converted to a Datetime object.</li><li><strong>Observation Count</strong>: The number of birds of that species spotted on that particular sighting. Unfortunately, if this was not filled in by the birdwatcher it has been filled with an ‘X’.</li><li><strong>Locality Type</strong>: State (S), County (C ), Postal/Zip Code (PC), Town (T), Hotspot (H), Personal (P). Mainly I think the interested part here is the ‘hotspot’ to see if there really tend to be more sightings in those areas.</li><li><strong>Month</strong>: A column I created from Observation Date to be able to better track monthly movements.</li><li><strong>Year</strong>: A column I created from Observation Date to be able to better track monthly movements.</li><li><strong>Observer ID</strong>: To track specific birders.</li><li>N<strong>umber of Observers</strong>: Perhaps an intersting way to track habits of birders.</li><li><strong>Duration Minutes</strong>: How long did the bird actually spend there being observed?</li><li><strong>Trip Comments/Species Comments</strong>: A string of text about the experience written by the person who logged it.</li></ul><p><strong>Interesting Questions for this Dataset:</strong></p><p>After an initial exploration of the features, there are a number of things this data can tell us that I’m planning to explore going forward.</p><ol><li><strong>Explore change in bird migration populations over time</strong>. However, right off, there’s a a couple big issues. As smart phones and internet have become more common, everyday people have been reporting more bird sightings. This means that in the years since 2005, overall sightings have skyrocketed. Scaling the data then will be a big issue when looking at changes over time — I don’t think the white-throated sparrow population has increased 10x from the early 1900s!</li><li><strong>When is the best month to see specific birds?</strong> Obviously many people have spent time thinking about and visualizing this particular aspect of bird migrations already, but it would be a great way to make some interesting subplot visualizations comparing when each bird species arrives, peaks, and leaves NYC.</li><li><strong>Mapping bird sighting locations</strong>: Mapping bird sightings based on the latitude and longitude provides a great chance to do some visualizations over time. Might be a good time to brush up on those Plotly skills and using a slider to make the visualizations more interactive.</li><li><strong>Meta-information about the birders themselves!</strong> One thing I love about this dataset is that there is actually a number of ways to find out information about the birders — do they travel in groups or solo? How long do they spend on each sighting? Are there any super sighters (aka people who log a lot of bird sightings). Finally, my personal favorite, the comments! This seems like a great opportunity to do a little natural language processing to see what birders are really paying attention to.</li></ol><p>Next week, the analysis begins!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44fc2741f1b1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rec Sytems for Beginners: Part 2]]></title>
            <link>https://medium.com/@melissamunz/rec-sytems-for-beginners-part-2-c3ee9762c0e6?source=rss-c1db7fc8d19d------2</link>
            <guid isPermaLink="false">https://medium.com/p/c3ee9762c0e6</guid>
            <dc:creator><![CDATA[M.M.]]></dc:creator>
            <pubDate>Fri, 20 Mar 2020 15:57:05 GMT</pubDate>
            <atom:updated>2020-03-20T15:59:50.654Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Setting up a baseline model with Surprise</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j7-Mr2KBB1XCqeqUpu6YSg.png" /></figure><p>This is a continuation from a <a href="https://medium.com/@melissamunz/how-to-make-a-recommender-system-for-a-beginner-aadb2ed51f59">previous blog on recommendation systems</a>. Last week’s covered how to think about and start thinking about creating a rec system and what the dataset looks like (in this case, users and articles they have interacted with), and this week we will dive into making a baseline model!</p><p>This exercise will use the <a href="http://surpriselib.com/">surprise</a> python library. Fortunately, if you’ve used sklearn before, surprise operates very similarly both in terms of imports and usage. It contains similar modules within the whole package that relate back to different sections of the project process (see ‘prediction_algorithms’ or ‘model_selection’) and the documentation is very similar.</p><p>Before diving into the models, there are a couple things to consider:</p><h4>Weighting interactions types:</h4><p>First off, we needed to decide how to rate each individual interaction from the event_type column. We basically have to do this by considering what we think matters most — Is a comment of equal value to a like? What about a starred review? Should we always rate a purchase highest?</p><p>Here is what I came up with:</p><pre>event_type_strength = {<br>   &#39;VIEW&#39;: 1.0,<br>   &#39;LIKE&#39;: 3.0, <br>   &#39;BOOKMARK&#39;: 3.5, <br>   &#39;FOLLOW&#39;: 4.0,<br>   &#39;COMMENT CREATED&#39;: 2.0,  <br>}</pre><pre>df[&#39;event_strength&#39;] = df[&#39;eventType_y&#39;].apply(<strong>lambda</strong> x: event_type_strength[x])</pre><h4>Mitigate User Cold Start:</h4><p>It it impossible to recommend products to users who have had few product interactions so far because we don’t yet know their preferences. We can mitigate this ‘cold start’ issue by limiting to only users who have had more than 5 interactions.</p><p>First lets explore our total number of users:</p><pre>interactions_count_df = df.groupby([&#39;personId&#39;, &#39;contentId&#39;]).size().groupby(&#39;personId&#39;).size()  print(f&#39;Total Users: {len(interactions_count_df)}&#39;) interactions_count_df.head()</pre><pre>Total Users: 1895</pre><p>Limit to those with 5 or more interactions:</p><pre>limited_interactions_count_df = interactions_count_df[interactions_count_df &gt;= 5].reset_index()[[&#39;personId&#39;]]</pre><p>Explore those users:</p><pre>print(f&#39;Total Users 5 or More Interactions: {len(limited_interactions_count_df)}&#39;)  limited_interactions_count_df.head()</pre><pre>Total Users 5 or More Interactions: 1139</pre><p>And finally, I’ll merge those users with the other table to get my full dataset of users with over five interactions and their ratings:</p><pre>selected_user_interactions_df = df.merge(limited_interactions_count_df, how = &#39;right&#39;, left_on = &#39;personId&#39;, right_on = &#39;personId&#39;)<br><br>print(&#39;# of interactions from users with at least 5 interactions: <strong>%d</strong>&#39; % len(selected_user_interactions_df))</pre><pre># of interactions from users with at least 5 interactions: 69820</pre><h4>Mitigate issues with outliers:</h4><p>The final thing to consider is the impact of users that act outside the norm who might skew the results. To mitigate the fact that some users are likely to interact with specific articles much more heavily than others we need to normalize by applying a log transformation.</p><p><a href="https://docs.python.org/2/library/math.html">math.log</a> take two arguments: <em>(a, Base). </em><strong>The </strong><em>‘</em>a’ is a numeric value and the ‘Base’ parameter refers to the base which the log has to be computed (here, we used 2).</p><pre><strong>import</strong> <strong>math</strong><br><br><strong>def</strong> log_user_performance(x):<br>    <strong>return</strong> math.log(1+x, 2)</pre><p>Then we apply the log transformation to the full dataframe. It will contain only the columns below with the formatted event strength column</p><pre>total_interactions_df = selected_user_interactions_df.groupby([&#39;personId&#39;, &#39;contentId&#39;])[&#39;event_strength&#39;].sum().apply(log_user_performance).reset_index()</pre><pre>print(&#39;# of unique user/item interactions: <strong>%d</strong>&#39; % len(total_interactions_df))</pre><pre># of unique user/item interactions: 39075</pre><p>So we ended up with a lot fewer interactions but hopefully ones that are more representative of the regular user.</p><p>Before moving on to modeling, I saved this dataset as a csv:</p><pre>total_interactions_df.to_csv(r&#39;full_interactions_dataset.csv&#39;)</pre><h3>Cross Validation:</h3><p>Cross validation for rec systems with surprise looks a little different from the norm, and needs some specific formatting in order to be used.</p><p>This can be achieved using the <a href="https://surprise.readthedocs.io/en/stable/reader.html">Reader method</a>, which parses files containing ratings, and which is passed in as a parameter in the <a href="https://surprise.readthedocs.io/en/stable/dataset.html">Dataset module</a> where the set is broken into the three columns necessary for surprise’s version of cross validation:</p><pre>reader = Reader(rating_scale=(0, 5))<br>data = Dataset.load_from_df(interactions_df[[&#39;personId&#39;, &#39;contentId&#39;, &#39;event_strength&#39;]], reader)</pre><p>First, we need to cross validate the set in order to be able to test if our rec system is working. For rec systems using surprise, you don’t need to break the data into a x-train, x-test, y-train, y-test 80/20 split the way you would for say, a classification model. Instead, the surprise module only uses the ratings of other users and items, and so only produces a train-set and a test-set.</p><pre>from surprise.model_selection import train_test_split</pre><pre>train_set, test_set = train_test_split(data, test_size = 0.2, random_state = 42)</pre><p>Additionally, if you inspect the type of the output you’ll see that the train_set datatype is now a special surprise datatype that is supposed to be more computationally friendly, but the test_set remains a regular list.</p><pre>type(train_set)<br>surprise.trainset.Trainset</pre><pre>type(test_set)<br>list</pre><h4>Modeling: Start with K-Nearest Neighbors</h4><p>Here, we’ll start with using K-nearest neighbors which is a basic neighbor based model. It’s a good starting place or baseline option for a rec system because it’s simple.</p><p>First let’s import what we’ll need:</p><pre>from surprise.prediction_algorithms import knns<br>from surprise.similarities import cosine, msd, pearson<br>from surprise import accuracy</pre><p>Here we have to make a decision about user-user similarity or item-item similarity. A good rule of thumb is to calculate the similarity between whichever is fewer, users or items. Note that in this case, the items are actually our articles. For our training set, we can see that:</p><pre>print(&#39;Number of users: &#39;, train_set.n_users, &#39;\n&#39;)<br>print(&#39;Number of items: &#39;, train_set.n_items, &#39;\n&#39;)</pre><pre>Number of users:  1139 <br>Number of items:  2904</pre><p>Looks like there are more items than users, so we’ll keep that in mind as we move forward because it will be more computationally efficient to calculate user-user interactions:</p><p>Now let’s get to modeling!</p><p>First we’ll set up the hyperparameters we want to start with. We are using user-based cosine similarity.</p><pre>sim_cos = {&#39;name&#39;:&#39;cosine&#39;, &#39;user_based&#39;:True}</pre><p>Now we will train the model. This is the part where choosing the smaller group (items or users) becomes important — the smaller group will take less time to train!</p><pre>basic = knns.KNNBasic(sim_options=sim_cos)<br>basic.fit(train_set)</pre><p>Let’s look at the output — the similarity of users to one another — using the <strong>basic.sim</strong> method. The output is a sparse array.</p><p>Finally, lets get our predictions using the test set:</p><pre>predictions = basic.test(test_set)</pre><p>…And evaluate it using accuracy.rmse! Remember that RMSE is root mean squared error, the standard deviation of the residuals (prediction errors) and a common evaluation method used to determining how successful or accurate your model is at its predictions.</p><p>RMSE is measured on a scale of 0 to infinity, and the score needs to be evaluated in the context of the data that you are using. We’re going to have to do some exploring to see what went wrong — A great question for next week’s blog!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c3ee9762c0e6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What makes a passing bill? — Topic Modeling Congressional bills]]></title>
            <link>https://medium.com/@melissamunz/what-makes-a-passing-bill-topic-modeling-congressional-bills-b6975467980a?source=rss-c1db7fc8d19d------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6975467980a</guid>
            <dc:creator><![CDATA[M.M.]]></dc:creator>
            <pubDate>Fri, 13 Mar 2020 15:20:45 GMT</pubDate>
            <atom:updated>2020-03-13T15:24:54.513Z</atom:updated>
            <content:encoded><![CDATA[<h3>What makes a passing bill? — Topic Modeling Congressional bills</h3><h3>Predicting Bill Passage: Part One</h3><h3>Overview:</h3><p>Can you predict if a Congressional bill will pass the House of Representatives?</p><p>This is obviously a huge question that many people have tried to answer. In this project, I attempted to answer it using a combination of natural language processing, topic modeling, and classification. This blog details my process topic modeling bill text using Natural Language Processing.</p><p><strong>The data: </strong>For this project, I used the congress.gov <a href="https://www.congress.gov/about/data">api</a> to obtain congressional bill data from 110th to 114th Congresses, which lasted from 2011 to 2017, including urls that linked the the bill summaries. I used those urls to web scrape the bill summaries for all the bills using Beautiful Soup 4. Joint resolutions and resolutions were removed from the dataset, maintaining only bills.</p><p>In total, I ended up with a total of 51,067 bills, of which only 4,025 passed. This discrepancy represented a severe class imbalance that I attempted to mitigate throughout the project.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/0*YS-YyzQs_BpAJB0L" /></figure><p>The data contained primarily categorical features about:</p><ul><li>The bill itself (passage date, introduction date)</li><li>The proposing congressperson (state, party, name)</li><li>The Congress at the time of proposal (congress number, majority)</li><li>Text features (title, bill summary).</li></ul><p>The data spanned two Democratic majority congresses, two split congresses, and ended with a Republican majority congress. The proportion of bills proposed by each party in each congress also followed this trend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eFy0YER-PUc9behxXdVYtQ.png" /></figure><p>The <strong>text features </strong>— bill title and bill summary — were used for topic modeling using natural language processing, and those topics were included as a feature in the final classification model.</p><h3>Natural Language Processing:</h3><p>I started out the text analysis by graphically modeling the text features — bill titles and summaries. This gave me a good jumping off point for seeing which words were prevalent in each class. Additionally, because bills are written in similar formats, it provided me with a starting list of stop words — such as ‘bill’, ‘purpose’, ‘amend’, etc. Below are the word clouds for passed bill and not passed bills, respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*03gJcvZtRy_lKJO-" /></figure><p><strong>Creating a Tokenizer</strong>: Before diving deeply into the topic modeling, I first needed to get the words into a cleaned format which would allow me to perform the topic modeling. Tokenization is the process of breaking sequence of strings into pieces such as words, phrases, symbols and other elements called tokens.</p><p>I created a custom <strong>Spacy tokenizer</strong>, which means that I took the basic functionality of Spacy’s tokenization software and built on it using a combination of regex, and domain knowledge of my subject (see removing bill specific stop words above).</p><ul><li><strong><em>Removed all punctuation</em></strong>. Bills are written in dry, formal language which means that punctuation not necessary to understanding bill meanng in the way that it might be for, say, analyzing tweets or text containing any kind of sentiment. ‘</li><li><strong><em>Maintained all parts of speech</em></strong>. After some trial and error, I determined that all parts of speech are necessary for bill comprehnsion. Few adjectives are used, and proper nouns actually tend to important to the bill type — the name of a national forest, state names, etc all tended to be linked to the bill topic or meaning.</li><li><strong><em>Lemmatization</em></strong>: Bills tend toward similar language and tending towards words that are easy to understand and get the point across simply. Lemmatization helped break these kinds of oft used words into their root — meaning that the word veterans and veteran would be considered of the same</li><li><strong><em>Stop words</em></strong>: Created a custom list of stop words commonly used in bill language such as ‘amend’, ‘bill’, etc. and removed them from the corpus. Note that some of the stop words were retroactively removed after a few tests of the topic modeling when they were found to account for too high a portion of specific topics.</li><li><strong><em>Word Limit</em></strong>: Limited to 10,000 words.</li></ul><p>The tokenizer was passed into a <strong>CountVectorizer</strong> before topic modeling.</p><h3>Topic Modeling:</h3><p>The processed summaries and titles were then used for topic modeling using <strong>Latent Dirichlet Allocation (LDA).</strong></p><p>Optimal parameters were tested first using a wide spread in Gridsearch — starting with n-components (the actual number of topics) as low as 5 and as high as 25. Human review was used once the number of topics had been narrowed between 10–15, with 13 topics being optimal for human topic comprehension — meaning they made the most sense to a human observer when observing the top 15 words from each topic.</p><p>Using the word groupings, the topics were modeled based on theme, with a final grouping of top 15 words:</p><ul><li><strong>Homeland Security/Immigration</strong> [‘foreign’, ‘immigration’, ‘homeland’, ‘require’, ‘criminal’, ‘enforcement’, ‘person’, ‘prohibit’, ‘security’, ‘law’]</li><li><strong>Energy</strong> [‘exchange’, ‘establish’, ‘use’, ‘production’, ‘require’, ‘security’, ‘renewable’, ‘policy’, ‘commission’, ‘oil’, ‘fuel’, ‘gas’, ‘product’, ‘consumer’, ‘energy’]</li><li><strong>Veterans Affairs </strong>[‘family’, ‘title’, ‘va’, ‘affair’, ‘facility’, ‘health’, ‘affairs’, ‘service’, ‘medical’, ‘child’, ‘veteran’, ‘program’, ‘department’, ‘veterans’, ‘care’]</li><li><strong>Entitlements</strong> [‘amends’, ‘hospital’, ‘benefit’, ‘insurance’, ‘medicaid’, ‘payment’, ‘service’, ‘xviii’, ‘program’, ‘small’, ‘business’, ‘medicare’, ‘title’, ‘social’, ‘security’]</li><li><strong>Defense</strong> [‘require’, ‘war’, ‘benefit’, ‘employee’, ‘compensation’, ‘title’, ‘code’, ‘defense’, ‘pay’, ‘armed’, ‘veteran’, ‘military’, ‘service’, ‘force’, ‘member’]</li><li><strong>Appropriations</strong> [‘emergency’, ‘management’, ‘committee’, ‘disaster’, ‘president’, ‘year’, ‘government’, ‘report’, ‘establish’, ‘budget’, ‘office’, ‘congress’, ‘national’, ‘agency’, ‘require’]</li><li><strong>Internal Revenue</strong> [‘business’, ‘extend’, ‘deduction’, ‘taxpayer’, ‘certain’, ‘increase’, ‘property’, ‘allow’, ‘income’, ‘credit’, ‘internal’, ‘code’, ‘tax’, ‘revenue’]</li><li><strong>Tariffs</strong> [‘version’, ‘repeat’, ‘temporary’, ‘suspension’, ‘summary’, ‘harmonized’, ‘extend’, ‘tariff’, ‘suspend’, ‘schedule’, ‘measure’, ‘temporarily’, ‘duty’]</li><li><strong>Federal Grants</strong> [‘direct’, ‘mortgage’, ‘loan’, ‘service’, ‘national’, ‘authorize’, ‘training’, ‘community’, ‘development’, ‘establish’, ‘housing’, ‘education’, ‘assistance’, ‘grant’, ‘program’]</li><li><strong>Transportation</strong> [‘public’, ‘regulation’, ‘air’, ‘direct’, ‘prohibit’, ‘use’, ‘requirement’, ‘service’, ‘safety’, ‘loan’, ‘certain’, ‘vehicle’, ‘require’, ‘transportation’]</li><li><strong>Education </strong>[‘child’, ‘award’, ‘agency’, ‘high’, ‘local’, ‘educational’, ‘grant’, ‘elementary’, ‘program’, ‘drug’, ‘secondary’, ‘student’, ‘food’, ‘school’, ‘education’]</li><li><strong>US Interior</strong> [‘county’, ‘forest’, ‘conservation’, ‘river’, ‘environmental’, ‘management’, ‘park’, ‘authorize’, ‘system’, ‘project’, ‘interior’, ‘area’, ‘water’, ‘land’, ‘national’]</li><li><strong>Healthcare</strong> [‘treatment’, ‘patient’, ‘coverage’, ‘drug’, ‘plan’, ‘information’, ‘individual’, ‘protection’, ‘insurance’, ‘human’, ‘require’, ‘public’, ‘care’, ‘service’, ‘health’]</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4S0rsvauuQWSmXdGKIpspw.png" /></figure><p>Other past topic modeling that I have done has sometimes been difficult to interpret, but it worked extremely well for Congressional bills which do tend to follow some particular topic structure that somewhat mimicked the structure of Congressional subcommittees.</p><p>Final topics were also evaluated using Log Likelihood and Perplexity, but overall these had little effect on the actual parameters chosen.</p><p>Reviewing passed bill by topic from 110th to 114th Congresses:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TgCqOGfecjBVpTFuzFF4YA.png" /></figure><p>It’s clear here that <strong>tariff</strong> related bills make up a huge amount of passed bills because they are often proposed individually. Additionally, Note<em> </em>a decrease in <strong>US Interior</strong> (environmental bills) as congress grows more conservative, and a slow increase towards <strong>Homeland Security/</strong>I<strong>mmigration</strong>.</p><h4><strong>Next Steps</strong>:</h4><p>The topics in this project were incorporated into a final classification model along with the other meta features. For more about the classification part of this project, stay tuned for the next blog!</p><h4><strong>Sources:</strong></h4><p><a href="https://github.com/mellymillionz/Congressional_Bill_Passage_Analysis">mellymillionz/Congressional_Bill_Passage_Analysis</a></p><p>Congress.gov</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6975467980a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>