<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sky B.T. Williams on Medium]]></title>
        <description><![CDATA[Stories by Sky B.T. Williams on Medium]]></description>
        <link>https://medium.com/@skybtw?source=rss-7f376e2745bc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*m4LVR3gnMbUV0MCVnKA29g.jpeg</url>
            <title>Stories by Sky B.T. Williams on Medium</title>
            <link>https://medium.com/@skybtw?source=rss-7f376e2745bc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 07:57:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@skybtw/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Natural Language Processing of Social Media Content]]></title>
            <link>https://medium.com/data-science/natural-language-processing-analysis-of-social-media-content-55518ded990c?source=rss-7f376e2745bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/55518ded990c</guid>
            <category><![CDATA[twitter]]></category>
            <category><![CDATA[disney]]></category>
            <category><![CDATA[k-means]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sky B.T. Williams]]></dc:creator>
            <pubDate>Wed, 21 Mar 2018 04:42:53 GMT</pubDate>
            <atom:updated>2022-08-02T16:12:47.767Z</atom:updated>
            <content:encoded><![CDATA[<h4>Alternate Title: Is the world talking about The Walt Disney Company? Yes, yes it is.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LBmTBV4kH3HjbLOEVkkzDA.jpeg" /><figcaption>Photo by <a href="https://pixabay.com/users/geralt-9301/">geralt </a>on <a href="https://pixabay.com/photos/people-google-polaroid-pinterest-3175027/">pixabay</a></figcaption></figure><h3>The Set-up</h3><p>The Walt Disney Company is a prolific powerhouse of a multinational corporation, with about 200,000 employees, and what I estimate to be a total worth to be around $150-$160 billion, based solely on the stock value at the moment of writing this post, and the number of outstanding shares (<a href="https://www.nasdaq.com/symbol/dis/stock-report">DIS</a>). As with most companies, you better believe that they have a vested interest in gauging how their customers and the public at large feel about the services that they provide, and the products that they create, but collecting that information, whether through in-person, over the phone, email, or mail-in customer surveys and questionnaires, cost money and time to create, disseminate, collect, and analyze. I wanted to see if I could use Natural Language Processing (NLP) tools and unsupervised machine learning to gauge public opinion of Disney in real-time, at minimal cost.</p><h3>Getting the Data</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pk2DZcgYpuN09JiC8nDE8Q.jpeg" /><figcaption>Photo by <a href="https://pixabay.com/users/geralt-9301/">geralt </a>on <a href="https://pixabay.com/illustrations/twitter-keyboard-social-media-3075846/">pixabay</a></figcaption></figure><p>First, I needed data in the form of text, which is publicly posted, and likely to contain topic-specific opinions; I turned to Twitter. Using the Twitter API (Application Program Interface) I wrote a script to download tweets related to keyword and hashtag ‘disney’ as they are posted online. Yes, absolutely anybody can freely do this for any tweets, and they can even be queried from a back catalogue of storage. In addition to the text of the tweet, you can also download a plethora of data and metadata related to that tweet and the user who tweeted or retweeted that status, including, but <em>absolutely not </em>limited to: time, date, location, language, number of followers, number of accounts following, date of account creation, profile picture, and username of who made the original tweet and who retweeted the status. I streamed the real-time tweets into a MongoDB collection, and since I had kind of a lot of tweets to analyze, I stored them on a personal AWS (Amazon Web Services) instance. If anyone is interested in any of these tools, or even just looking into what kinds of information the entire computer-owning world has access to about your Twitter account, I’ll include some links at the bottom of this post that you can check-out.</p><h3>Preliminary Stuff</h3><p>I collected somewhere around 100,000 to 120,000 tweets each day, in over 30 different languages, all related to Disney. As I explored the data, I discovered my first source of bias. While I searched for many permutations of keyword disney (i.e. “disney”, “disney’s”, “#disney”, etc.), this will not collect all possible tweets related to my topic, but only tweets that mention one of the words in my search list. For example, my search criteria would have collected a tweet that said something like “I love Disney World!”, but not “I love EPCOT!”, since the second tweet does not contain a word similar to ‘disney.’ I could have expanded my search list to include each theme park specifically, but it would have biased my tweets towards theme parks, and I simply can’t include the title of every Disney movie, TV show, and character ever created. In order to keep my analysis as objective as possible, I kept my search criteria generalized, which biased my tweets towards over-representing properties that had the word ‘Disney’ in the title, such as Walt Disney World and Disney Channel.</p><p>I took my ~500,000 Disney tweets, cut them down to the 320,000 that were in English and I could actually read, and I ran them through a Python library called TextBlob, which analyzes each data point of text, in this case a single tweet, and calculates the Polarity to be somewhere between -1 (unfavorable) and 1 (favorable), with 0 meaning that the sentiment was neutral. The distribution of all of the tweets together had a mean of about 0.115, or just positive of neutral, as can be seen below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JkprCDLn7kZ32otw2dvKBg.png" /><figcaption>Image by Author</figcaption></figure><p>And I guess that that is pretty neat, but the truth is that The Disney Company is actually a massive conglomerate of other companies, including Marvel, Lucas Films, ESPN, ABC, Disney Parks and Resorts, Disney Cruise Line, and many more, as well as a very diverse portfolio of the associated intellectual properties. I don’t think that knowing overall sentiment would be nearly as useful to a company as would knowing sentiment of specific intellectual properties or brands. To achieve grouping these tweets into separate categories, without looking at each individually, we need to turn to clustering through unsupervised learning, but first we need to explore vectorization and dimensionality reduction! Exciting stuff.</p><h3>Data Cleaning and Vectorization</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ANj661EFCQvGSzrapucDAQ.jpeg" /><figcaption>Photo by <a href="https://pixabay.com/users/lobostudiohamburg-13838/">LoboStudioHamburg</a> on <a href="https://pixabay.com/photos/phone-display-apps-applications-292994/">pixabay</a></figcaption></figure><p>In general I find data cleaning pretty interesting to do, but very tedious to read about. Unfortunately it is also super important, so I have to mention it. The biggest issue with working with tweets is duplicate tweets. I couldn’t eliminate collecting retweets because a retweeted positive tweet, I feel, still indicates a new positive sentiment on the topic and may be useful later, but you can’t cluster data with duplicate data points; I’ll touch on this later, but for now just take me at my word that having a ton of tweets with the exact same text is really bad for my analysis. Therefore I removed any url, symbol, or retweet tag in any of the tweets, then I kept only a unique set of each text data point to clean out anything that was either a retweet or an automated bot, resulting in about 98,000 tweets.</p><p>The input for all algorithms is numbers. There are many different ways to turn other forms of data into a numerical format, such as deconstructing a picture into values for red, green, and blue for each pixel, or in my case, turning a tweet of words into counts for each word in the tweet, called Tokenization and Vectorization. First you define what exactly a token is, which could be each unique word, combinations of adjacent words, or even any combination of a certain number of adjacent letters. These techniques can certainly be used separately or in combination, depending on your needs. Next you vectorize your text, which assigns a value for each of these tokens for each data point. The most basic example would be taking the text “<em>I like soup</em>”, which count vectorizes to [<em>i</em>:1, <em>like</em>:1, <em>soup</em>:1]. We now have a numerical representation of our text! Next, in order to have all of our tweets in the same table, we can’t just have a row for “<em>I like soup</em>” and 1 column for each ‘<em>i</em>’, ‘<em>like</em>’, and ‘<em>soup</em>’, but we need a column for each word that is contained in any of our data points. For instance, if our second data point is “<em>I do not like flying</em>”, the vectorization of “<em>I like soup</em>” now has to expand to specify that it <em>does not</em> contain the new words, such that its new value is [<em>i</em>:1, <em>like</em>:1, <em>soup</em>:1, <em>do</em>:0, <em>not</em>:0, <em>flying</em>:0], while the value of “<em>I do not like flying</em>” is [<em>i</em>:1, <em>like</em>:1, <em>soup</em>:0, <em>do</em>:1, <em>not</em>:1, <em>flying</em>:1], so that both of our data points have different values, but the exact same format. You can see how with 98,000 unique tweets, the number of counts for each text data point will quickly become huge, even though most counts will be 0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IYWr-hYEwCbz0VJfN55qEA.jpeg" /><figcaption>Photo by <a href="https://pixabay.com/users/geralt-9301/">geralt </a>on <a href="https://pixabay.com/illustrations/smartphone-hand-photomontage-faces-1445489/">pixabay</a></figcaption></figure><p>Still with me? Excellent! So now we have a beautiful table/dataset of 98,000 rows, each representing a tweet, and around 10,000 columns, each representing the number of times a unique word appeared in that tweet. This table is mostly 0s, with some 1s, 2s, and occasionally 3s mixed in, which is now actually a different kind of problem. If someone (such as a computer program) were to just look at the numbers in the table, and decide whether these tweets were similar or different, they would all look about the same, because pretty much every row is made up of around 10,000 0s, with a handful of 1s, 2s, and 3s. This is an example of the <em>curse of dimensionality</em>, which is a problem many fields of analysis have to contend with everyday.</p><p>Before moving on to dimensionality reduction, I also want to mention that I removed stop words, such as ‘<em>a’</em>, ‘<em>the’</em>, and ‘<em>if’</em>, which carry very little analytical value, as well as weighting the words so that if a word like ‘disney’ appears in a lot of the data points, then it carries less value in determining differences between the topics of the tweets.</p><h3>Dimensionality Reduction</h3><p>Wow, this post is getting long, so let’s charge ahead and try to cover some ground. In case you took a break and came back, here’s where we are: we want to group a bunch of tweets by Disney topic, we’ve converted our text into a numerical format, and now we have a dataset with ~10,000 columns of mostly zeros, which is accurate, but bad for analysis. In order to reduce the number columns I have in my dataset, but still contain lots of descriptive information, I used Singular-Value Decomposition (SVD). Basically what this does is it takes all of the information (variability) within the tweets, and kind of distills it down into a smaller number of newly created columns. This process is abstract, heavy in linear algebra, and complicated. What is important to take away is that we can get a lot of the information originally contained in the 10,000 token columns compressed down into just a few dozen new columns, which is great.</p><p>We have now created what’s called an LSI space (Latent Semantic Indexing) that can help us determine the usefulness of each new column in separating the tweets into topic groups that have words in common. Unfortunately, because of the ‘distilled’ nature of the new columns, this data is a lot harder to interpret than the straight word counts, so let’s feed these columns to a clustering algorithm. Side note: The new columns will put tweets closer together that have words in common, which is why retweets and bots are bad. If the exact same 20 words appear in identical tweets 15,000 times, then it artificially creates a relationship between those words that confuses the model.</p><h3>Clustering</h3><p>I would love to go into the ins and outs of the K-Means unsupervised clustering algorithm, but I won’t, and instead I will just say two things. First off, the algorithm (or model) assigns each data point to a cluster number so that tweets that are textually similar are clustered together, ideally into some sort of logical theme or topic. Secondly, what makes it unsupervised is that you <strong>do not </strong>use it to create predictions, then test it against known data, which would be supervised. An example of supervised learning would be to give a model a bunch of information, like day of the week, weather conditions, time of day, etc., and have the model predict whether it thinks a bus will be on time or late. Once you have a prediction, you then compare it to what actually happened and decide if it was right or wrong; pretty straight forward. With an unsupervised model like k-means, I give it a bunch of data, such as vectorized tweets, then the algorithm groups the data points into clusters, but since we don’t know the content of the tweets to see how well the model performed, I need to sift through the tweets for each grouping to see if the clusters follow some useful theme. To cut to the chase, in this case, they did not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*vMVMn3K3QcfqomGVzNDuRg.png" /><figcaption>Image by Author</figcaption></figure><p>Above is a silhouette plot, which basically shows that the majority of my tweets are in a giant, fairly diffuse cluster (12), while the other clusters are much smaller in comparison. When looking at the tweets contained in each grouping, in some cases there were the makings of clear themes, though there was almost always a lot of overlap between topics and confusion in the signal. In other words, I was not able to pull out a single cluster and trust that it contained all tweets on a specific topic, such as brand, character, movie, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fWKjg9bypTAQDZvgipdHjA.jpeg" /><figcaption>Image by Author</figcaption></figure><h3>Why Disney tweets are hard to work with</h3><p>So after a lot of model tuning and trying different things to get my clustering algorithm to produce something really exciting and insightful (shoot me a message and I would be happy to go into all of it in painful detail), there came a time when I had to conclude that the tweets just didn’t want to work with me. That said, I think that there is still something that can be learned, and you may even find it interesting, so let’s dig into a couple of the clusters and see why Disney tweets don’t like to be separated into groups.</p><p>First off, this is a common example of a tweet in my data set:<br>- <em>“I wanna go to Disney!”<br></em>As much as I appreciate, and maybe even agree with, the sentiment, if you can imagine what the vectorized version of this tweet would be, there isn’t much there for the model to work with. If you had to label this tweet with a topic, what would it be? Maybe we could infer that they are talking about a theme park, but there isn’t really mention of a specific park, or a movie, or even a character for the model to use as a topic label. There is definitely a lack of signal, or usable information, in this tweet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*GolDDKpKLm9M1Zv7H9kmJA.jpeg" /><figcaption>Image by Author</figcaption></figure><p>When digging into another identified cluster, I would see the following tweets:<br>- <em>“Ariel is the only Disney Princess to have had a child.”<br>- “How is Pocahontas not a Disney Princess?”<br></em>From these I could see some type of ‘Disney princess’ theme starting to form, which would be great! But when I dug deeper I could see that mentions of princesses were in many other clusters as well, indicating confusion between my topics.<br>- <em>“Disney owns both Marvel and Lucas Films, idc what you say Shuri and Leia are Disney Princesses in my eyes”<br></em>This is an example of a tweet in the same group as the first two, but the third undeniably includes material related to Marvel and Lucas Films. If I had a cluster for each Marvel and Star Wars/Lucas Films, where would the model place this tweet?</p><p>One final example:<br><em>- “Disney Is Not Only Disney </em><strong><em>Parks</em></strong><em>! #Disney</em><strong><em>smmc</em></strong><em> @Disney</em><strong><em>Cruise </em></strong><em>@Disney</em><strong><em>Aulani</em></strong><em> @</em><strong><em>DVC</em></strong><em>News @Disneymoms”<br></em>In this single tweet I see ‘Parks,’ ‘smmc’ (a special event at Disney — Social Media Moms Celebration), ‘Cruise,’ ‘Aulani’ (Disney resort in Hawaii), and ‘DVC’ (Disney’s timeshare program — Disney Vacation Club). As with the previous example, I interpret this as a lot of potential signal in a tweet, but not enough distinct signal for the model to be able to place it in a single category with textually similar tweets of the same topic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JKRfCaUx9ORQ3a7DTRjNrQ.jpeg" /><figcaption>Image by Author</figcaption></figure><h3>Closing thoughts</h3><p>So maybe I wasn’t able to complete the efficient and informative, brand-focused sentiment analysis on behalf of Disney like I had hoped, but I did take you on a quick, or not so quick, tour of how a simple NLP project is structured, as well as how and why certain steps are done along the way. As it turns-out, The Disney Company is really good about taking an intellectual property, like a single successful character, and spreading it across many of its businesses. It would not be at all uncommon for a character like Aladdin to appear in movies, TV series, theme parks, cruise ships, merchandise, and even more, at times even appearing with seemingly unrelated characters. Not only that, but it seems that when somebody loves a Disney movie, they will probably also love the parks, cruise ships, and maybe even have a Disney wedding, mentioning it all in a single 280 character tweet. These are just a couple of reasons that Disney is so incredibly successful, but also why my model had such a hard time separating these tweets into distinct topics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OzBWYCQRYcqSAsWp3VZA9w.jpeg" /><figcaption>Image by Author</figcaption></figure><p>Thanks for hanging in there for the whole thing, or at least skipping past the technical bits to read the end! I hope that you found something interesting, and feel free to reach-out here or on LinkedIn to let me know what you think! (<a href="https://www.linkedin.com/in/sky-williams/">https://www.linkedin.com/in/sky-williams/</a>)</p><h3>Resources</h3><p>Twitter API: <a href="https://developer.twitter.com/en/docs">https://developer.twitter.com/en/docs</a><br>Amazon Web Services: <a href="https://aws.amazon.com/">https://aws.amazon.com</a><br>MongoDB: <a href="https://www.mongodb.com/">https://www.mongodb.com</a><br>If you want details about any of the techniques, algorithms, approahces, or libraries that I used: <a href="https://www.linkedin.com/in/sky-williams/">https://www.linkedin.com/in/sky-williams/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55518ded990c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/natural-language-processing-analysis-of-social-media-content-55518ded990c">Natural Language Processing of Social Media Content</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Wildfire Destruction — A Random Forest Classification of Forest Fires]]></title>
            <link>https://medium.com/data-science/wildfire-destruction-a-random-forest-classification-of-forest-fires-e08070230276?source=rss-7f376e2745bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/e08070230276</guid>
            <category><![CDATA[random-forest]]></category>
            <category><![CDATA[wildfires]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[logistic-regression]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Sky B.T. Williams]]></dc:creator>
            <pubDate>Sat, 24 Feb 2018 08:17:15 GMT</pubDate>
            <atom:updated>2022-07-29T14:50:56.222Z</atom:updated>
            <content:encoded><![CDATA[<h4>Can we predict how large a wildfire will become?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VO9nuzrJ9XseMyxdtPh_vg.jpeg" /><figcaption>Image by <a href="https://baynature.org/article/bay-area-filmmaker-looks-for-answers/rim-fire-photo-credit-usfs-mike-mcmillan/">Mike McMillan</a> on <a href="https://baynature.org/wp-content/uploads/2018/09/Rim-Fire-photo-credit-USFS-Mike-McMillan.jpg">baynature</a></figcaption></figure><p>Believe it or not, each year there are about 80,000 wildfires in the United States. Most of these are very small and go mostly unnoticed, only affecting a handful of acres of uninhabited countryside, but some become a roaring, un-containable forest fire that destroys everything in its path. From October to December, 2017, wildfire-related insurance claims reached nearly $12 billion in Northern California alone (<a href="http://www.insurance.ca.gov/">www.insurance.ca.gov</a>). Unsurprisingly, wildfires can also be incredibly dangerous, claiming a reported 1,114 lives between 1920 and 2015 (<a href="http://www.nifc.gov/">www.nifc.gov</a>). Anything that we can learn about what causes wildfires, what drives their size and spread, and what contributes to their behavior in any way can be a benefit to fire suppression and prevention efforts, so for this project I built a model to predict the size of wildfires using information for ~1.88 million fires from 1992–2015, found in the US Forest Service Archive, and integrated historical air temperature data from Berkeley Earth, Lawrence Berkeley National Laboratory (links provided below).</p><h3>Features, Exploratory Data Analysis, and Baselining</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZUTbrnrkT1OE0lblpKQ6Vw.jpeg" /><figcaption>Photo by <a href="https://pixabay.com/users/skitterphoto-324082/">Skitterphotoon</a> on <a href="https://pixabay.com/photos/easter-fire-bonfire-fire-flames-384602/">pixabay</a></figcaption></figure><p>The United States uses a classification system to group wildfires by total acreage burned, with a category ‘A’ burning less than 1/4 of an acre, and a category ‘G’ exceeding 5000 acres of burned land. I collected features for my analysis, such as the state where the wildfire was discovered, who owned the land where the fire took place (federal, state, tribal, private, etc.), the initial cause of the fire (lightning, fireworks, campfire, etc.), longitude, latitude, and air temperature; then I engineered a few additional features from the information already obtained, such as total number of fires per state and average annual air temperature. With this information loaded via SQL into Python3, I isolated 20% of my overall data (~367,000 fires) as a sub-sample to reduce computational load, making sure that it was randomly stratified so that each of the target fire size categories was as proportionately represented in my sub-sample as they were in my complete dataset. Next I fit both a distance-based Logistic Regression and a tree-based Random Forest Classifier (RFC) to my sub-sample, using a 5-fold cross-validation technique with an 80% Train, 20% Validate split for baselines, and the results were underwhelming.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vvI5gl5JlFBDXfRRMnAd8A.png" /><figcaption>Image by Author</figcaption></figure><p>You can see above that the Logistic Regression yielded a 0.00 recall for categories ‘C’ through ‘G’, basically saying that it didn’t successfully classify any fires into those groups, which isn’t good. The Random Forest Classifier on the right didn’t do much better.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplayer.vimeo.com%2Fvideo%2F257034352&amp;dntp=1&amp;url=https%3A%2F%2Fvimeo.com%2F257034352&amp;image=https%3A%2F%2Fi.vimeocdn.com%2Fvideo%2F684856967_1280.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=vimeo" width="1652" height="1080" frameborder="0" scrolling="no"><a href="https://medium.com/media/273733fd11181b12b522ee51ee6de356/href">https://medium.com/media/273733fd11181b12b522ee51ee6de356/href</a></iframe><p>To understand why my models were so poorly predicting the larger fires, I went back to the full dataset and explored the distribution of the fire classes, and it turns out that the two size categories with fires smaller than 10 acres, ‘A’ and ‘B’, made up the vast majority of my data points; additionally, the number of category ‘G’ fires was extremely low when compared to most of the other classes. This imbalance in terms of quantity of fires was causing very noticeable bias in both my models. Another very important imbalance can be seen when, instead of quantifying by how many fires are in each category, I looked at how many acres were cumulatively burned by each class. In this way the data shows that category ‘G’ fires, while only making up about 0.2% of all wildfires in the country, have actually burned almost 75% of all wildfire acreage. Above is a video of a D3 animation I used to illustrate this relationship, with the first graph showing the imbalance in Number of Fires, with the graphic then transitioning to show the imbalance in terms of Total Acreage Burned. The section in red represents the most destructive, most dangerous, and most rare fires in the country, and for these reasons, it is predicting these category ‘G’ fires that are the priority for my model.</p><h3>Tuning the Model</h3><p>So I went back to my baseline models and implemented weight classes to compensate for the imbalance in data points, which instructs the models to place a greater importance on the characteristics of data points belonging to the rarer fire sizes while training. After re-fitting and re-testing both models with the new parameters, the Random Forest Classifier once again outperformed the Logistic Regression, as well as other distance-based models, so I moved-forward with the RFC. One aspect to note here is that since RFC is a tree-based model, and does not depend on distances between points, I was able to include some features I had originally collected, but excluded until now because the data was incomplete, making them incompatible with the Logistic Regression. Since I wanted a true comparison of the two model types, I needed to use identical data for my baselines. Now that I have decided on the RFC, and tree-based methods are not only robust to handling Null values, but can actually glean information contained in the fact that data is missing, I can reintegrate the excluded data. In addition to adding features that contained Null values, I was also able to use the Feature Importance Scores from the RFC model to eliminate some of the features that were rarely used and had no signal for differentiating fire size class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I8LiLQJnnPL38551gMkEpg.jpeg" /><figcaption>Photo by <a href="https://pixabay.com/users/ylvers-337353/">Ylvers </a>on <a href="https://pixabay.com/photos/forest-fire-trees-burning-forest-432870/">pixabay</a></figcaption></figure><p>When tuning my RFC, the hyperparameters that ended-up making a noticeable difference for the recall of the highest priority target class were: the number of features that the model considered when deciding where to make a split in the data, the maximum number of splits that I allowed within each tree of my ensemble “forest” of tree models, the minimum number of samples needed to create an internal split in the data, and the minimum number of samples required for there to be the creation of a terminating leaf node on each tree. Once I dialed-in my model on the 20% stratified sub-sample, I initialized an instance with Amazon Web Services with enough RAM and processing power to run these ensemble models on my 1.88 million rows, employed another 80%-20% split for training and testing, and re-tuned my model, once again using the 5-CV K-Fold method to protect against misleading error metrics from random data patterns, and finally I evaluated my model against the test data held-out before the final tuning.</p><h3>Final Evaluation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*jkrJEutGS8Odz5teEH9Nyg.png" /><figcaption>Image by Author</figcaption></figure><p>With a final ‘G’ class recall score of 0.62, this model has come a long way from where it started. The ‘G’ precision measures how many fires from other categories were incorrectly labeled ‘G’, which the recall measures how many ‘G’ fires were correctly placed in ‘G’. Unfortunately, the precision is very poor, but this was just a decision that I had to make, and a trade-off that I had to accept. For this model, I determined that I would much rather have some small fires that maybe would get extra attention in being contained and ended-up being false alarms, in order to minimize the number of huge, destructive fires that potentially slipped through the cracks.</p><h3>What the model can teach us</h3><p>Aside from the base predictive power of my Random Forest Classifier model, as evaluated via recall of the most dangerous size category, there is one last important point to bring-up, and that is that when looking at the feature importance scores from my finalized model, three of the features that my model used the most, and offered the most gain when learning how to differentiate between the fire size classes, were: longitude, latitude, and air temperature. This is important because longitude and latitude can both be proxies for true environmental conditions, such as air temperature, humidity, rainfall, wind, and elevation, and my model determined these to be more impactful than features like initial cause and property owner, when predicting the eventual burn size of a wildfire. From the beginning of this project I had wanted to include more environmental elements into my analysis, but time constraints limited how much attention I could devote to data acquisition and data wrangling. With these results, if I were to continue to improve this model, I would definitely work to integrate additional environmental features into my Random Forest Classifier, hopefully increasing its predictive power.</p><p>A truly accurate, well-maintained model could identify under what conditions wildfires are most likely to spread to dangerous and destructive sizes in real-time, thus assisting fire-suppression professionals in efficiently allocating their resources, and channeling their efforts towards the most potentially destructive fires, preventing property damage and saving lives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3R87sw0LyHVOf5pb-8mvEw.jpeg" /></figure><h3><strong>Resources</strong></h3><p>Wildfires Data:<br>Short, Karen C. 2017. Spatial wildfire occurrence data for the United States, 1992–2015 [FPA_FOD_20170508]. 4th Edition. Fort Collins, CO: Forest Service Research Data Archive. <a href="https://doi.org/10.2737/RDS-2013-0009.4">https://doi.org/10.2737/RDS-2013-0009.4</a><br><a href="https://www.kaggle.com/rtatman/188-million-us-wildfires">https://www.kaggle.com/rtatman/188-million-us-wildfires</a></p><p>Air Temperature Data:<br><a href="http://berkeleyearth.org/data/">http://berkeleyearth.org/data/</a><br><a href="https://www.kaggle.com/berkeleyearth/climate-change-earth-surface-temperature-data">https://www.kaggle.com/berkeleyearth/climate-change-earth-surface-temperature-data</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e08070230276" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/wildfire-destruction-a-random-forest-classification-of-forest-fires-e08070230276">Wildfire Destruction — A Random Forest Classification of Forest Fires</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Magic of Theme Park Crowd Prediction — A Narrative]]></title>
            <link>https://medium.com/@skybtw/the-magic-of-theme-park-crowd-prediction-a-narrative-eaf311b48f96?source=rss-7f376e2745bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/eaf311b48f96</guid>
            <category><![CDATA[disney-world]]></category>
            <category><![CDATA[disney]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[disneyland]]></category>
            <dc:creator><![CDATA[Sky B.T. Williams]]></dc:creator>
            <pubDate>Sun, 04 Feb 2018 02:08:41 GMT</pubDate>
            <atom:updated>2018-03-24T04:48:33.295Z</atom:updated>
            <content:encoded><![CDATA[<h4>When should you plan your next theme park vacation to avoid big crowds?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dJklWnjzo4QysJaeeXDE4Q.jpeg" /></figure><p>I admit that growing up, while I enjoyed Disney movies as much as the typical American child, and I was fortunate enough to go experience both Disney parks and cruises, I wasn’t necessarily a giant Disney fan. That changed after I moved to Saint Petersburg, FL for school in 2010. A friend of mine said that she wanted to visit me, and that she wanted to go to Disney World, so my response was “Sure, let me see how far away it is.” After that, things moved quickly. I discovered that Walt Disney World was less than an hour and a half away from my house, my friend flew down to visit, we went to the parks, and I instantly could not get enough of all things Disney. I signed-up for a Walt Disney World annual pass and went any opportunity that I could; I convinced friends whom I had lost touch with to come and visit so that we could go on the cheap, and I started reading any material that I could find on Disney movies, theme parks, cruise ships, and anything related to the company. I was lucky enough to have parents that were Disney Vacation Club members and were generous with their hotel stays, and I even found a very close friend that came to love the parks as much as I did (the two of us have gone many, many, <em>many</em> times, and we haven’t had a bad or repeat experience yet). All of this culminated in several years of annual pass ownership, and the number of days that I have spent at Walt Disney World breaking into the triple digits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*FhQxU5SLUZDCpTP2N_nyqg.jpeg" /></figure><p>After not too long, my newfound Disney addiction became common knowledge among those that were close to me, and whether I was free and able to go to the parks with them or not, I was the go-to Disney vacation planner, which of course I loved, and I continue to do whenever I am able and asked. So going in to week 2 and project 2 of my data science training at Metis, when I was tasked with creating a predictive linear regression model using any material that I wanted, my immediate thought was to go Disney. So I set out to see if I could use a computer model to predict the crowd levels at a Disney theme park.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*A7jLWc-ZdgedDviyeyHzaQ.jpeg" /></figure><p>I started by selecting features that I thought could be important for determining how busy a park would be, such as the day of the week, the month, whether or not there was an in-park event or national holiday, the park’s hours of operation, and weather. In addition to that information, I also wanted to include data that the model could use to differentiate between the parks, such as the number of dining options and cost of food, the intensity of the rides, and the age of each park. Finally, I managed to track-down an archive of historical crowd levels for all 6 US-based Disney theme parks stretching back to 2009. And with all of that information in-hand, I was ready to start modeling!! [I feel that it may be prudent at this time to say that this data was used solely for educational purposes during my studies at an accredited institution, and was not, nor ever will be used for the financial gain of myself or others. Please don’t sue me!]</p><p>My first modeling step was to take all of the data that I had collected and look for false information, the distribution of crowd levels, and any relationships between features that could mess up my model (specifically I was eliminating data points that were potential data entry or data collection errors, errors that could have been an artifact of my scraping scripts, ensuring that my target variable was normally-distributed, and confirming that there wasn’t any co-linearity between inputs, but that truly is all of little importance). Once I had clean, reliable data, I plugged it into a basic ordinary least squares linear regression model and found that my unrefined predictive accuracy was okay, but not great (adjusted R² = 0.346, for those that truly would like to know).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Prtqf0CJMBxYGXiXbLHokA.png" /><figcaption>Residuals plot showing relatively poor predictive power</figcaption></figure><p>Next I used the LASSO method with 5-fold cross-validation to help me eliminate features that it deemed weren’t very important in predicting crowd levels, and re-ran my data, excluding the features isolated by LASSO, back through the basic linear regression as well as LASSO and Ridge regressions to apply normalization, but the overall improvement was minimal (leading to a final R² = 0.353 against test data held-out from before I fit the model, 20% of my overall data).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kBPtIWOMrwhmAVjRDlqhjg.png" /></figure><p>Okay, now I have a model that tries its best to predict crowds, and doesn’t do it all that well, so I went back to the information that LASSO gave me, including the graph on the left (LARS PATH), which basically shows how important the model thinks that each feature is in predicting crowd level. Some of the inputs that it eliminated or put very low value on were expected, such as a certain day of the week or a certain month of the year, but there were some low values (called coefficients) that I was very surprised by. At this point I took a step back and re-evaluated by model and my approach to the problem. If I am able to look at the data and come up with a better prediction in my head than my model can, I need to ask myself what exactly am I using to estimate crowds that my model isn’t? In effect, what am I as the data scientist failing to provide my model?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*QLDAmkh2HJGjzIIPdjyMjQ.jpeg" /></figure><p>The answer that I came up with was Domain Knowledge. I realized that when I look at the data, I am actually changing how I use each of the features depending on which park I am looking at; I change how much weight I give to each input, whereas the model is forced to choose a single relationship that is best suited for all parks (it assigns a single fixed coefficient to each feature). For example, I know through experience and research that attendance at the parks in California is largely driven by locals, or people that drive to the parks, whereas the population of visitors at Walt Disney World is absolutely dominated by tourists that fly in from all over the world. So if you were planning on driving to Disneyland in Anaheim for a day trip and heard that it was going to rain, maybe you would still go, but maybe you would postpone a day, or a week, or even a month until the weather was going to be nicer. But if you were flying to Orlando from Australia to go to the Magic Kingdom or Epcot, and you heard that it was going to be pouring, you’re probably not going to cancel your flight, hotel, and park tickets, most likely you would still go and make the best of it. In other words, it actually makes intuitive sense that my predictive model was getting mixed signals from the data I was feeding it.</p><p>So I went back to the beginning. I took my 1 large data set of 16,000 historical crowd levels, split it into 6 separate data sets, and developed 6 separate predictive models, 1 for each park. Instantly I saw an improvement. When my model no longer needed to differentiate between parks and was able to focus solely on Disneyland, my predictive accuracy went up quite a bit (R² = 0.561 vs. the previous 0.353 from the 6 parks together). With additional time, I am incredibly confident that I will be able to refine my models further and that the predictive capabilities will increase through experimentation with different model structures, as well as potential transformations and regularization.</p><p>So what does all of this mean? First off, it means that if you are lucky enough to have access to a complicated computer model to aid in the planning of your theme park vacation, my efforts suggest that you should look at each park individually to maximize your predictive accuracy. Another rather interesting take-away is that in working with all 6 of my single theme park models, I discovered that the most important factor in determining crowd levels in every case was the park’s number of operational hours for the day of interest. So the next time that you’re planning your theme park day, and want to figure out when to go in order to avoid long waits to get on rides, instead of crawling through the archive of historical crowd levels, looking through all of the in-park even calendars, and researching the long-term weather forecasts, your first step, and maybe even only step, should probably be to scroll over to the official Disney website and look-up what their posted hours will be, because if they have extended hours, then they are probably expecting large crowds, and then so should you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bANU1qZuaW6rG_Rqc05GgA.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eaf311b48f96" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Maximizing food truck sales as a function of New York City foot traffic]]></title>
            <link>https://medium.com/@skybtw/maximizing-food-truck-sales-as-a-function-of-new-york-city-foot-traffic-c213a1339a6f?source=rss-7f376e2745bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/c213a1339a6f</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[new-york]]></category>
            <category><![CDATA[food-truck]]></category>
            <category><![CDATA[subway]]></category>
            <dc:creator><![CDATA[Sky B.T. Williams]]></dc:creator>
            <pubDate>Wed, 24 Jan 2018 05:33:26 GMT</pubDate>
            <atom:updated>2022-07-29T14:27:17.846Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9iVcNGCDRvvO6UQXOb2UFA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@josephyates_">Joe Yates</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>By the year 2020 the American food truck industry is projected to be valued at nearly $1,000,000,000; which would be one Billion with a “B” (<a href="https://www.statista.com/statistics/444924/industry-value-us-food-trucks/">statista.com</a>). If a brand new Manhattan-based company was intending to break into that industry, or if an existing mobile food distribution franchise was looking to increase their profits without investing in additional trucks, how would they know where to start? One way would be to determine where on the bustling streets of New York City they could position their carts to maximize foot traffic during the times of day when people are looking for a quick bite on the go.</p><h3>Where to start?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*MaOGxH_u7-JNoIeUw8yG-Q.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@punttim">Tim Gouw </a>on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>To answer this question, my group and I turned to turnstile data for New York City subway stations (MTA). Locations to board and depart public transit naturally concentrate pedestrian foot traffic, and are therefore logical locations to target for selling food. First I acquired several months of recent foot traffic data, then I completed a cleaning process to ensure high data quality, which entailed either removing or repairing counts that were deemed to be software or mechanical errors in the turnstiles. Finally I took counts of both entries and exits grouped by station, and separated them into 3 groups: Breakfast/Morning commute, Lunch, and Dinner/Evening commute. My lunch block also integrated exit counts from the morning period in order to incorporate potential customers that will potentially purchase lunch within walking distance from their place of work a few hours after their morning commute. [All analysis and visualization was completed using Python3]</p><h3>The Data</h3><p>The following graph shows average combined entries and exits for the entire workday added together and grouped by New York City subway station, though it only shows the top-10 stations in terms of overall traffic, since those are the best candidates to maximize customer exposure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mhtrdnglwp-80HR0b42k9Q.png" /><figcaption>Average overall daily foot traffic through busiest NYC subway stations | Image by the author</figcaption></figure><p>It would be straight forward and certainly logical to simply target the stations with the greatest overall traffic for the entire day, starting from the left at Grand Central Station and moving to the right with each additional food truck you had available, but let’s see what happens when the foot traffic is broken down by time of day.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3QxnRhA-c2A80K5ZBXq4Gw.png" /><figcaption>Average foot traffic broken down by meal time through the busiest NYC subway stations | Image by the author</figcaption></figure><p>This is the exact same data, so the stations are still ordered from greatest total daily traffic on the left to the least of the top-10 on the right, but now you can see that there are stations ranked very highly in terms of overall daily traffic, but are actually less busy during a single meal period when compared to a station ranked lower overall, such as the #2-ranked 34 St.-Herald Sq. station, which has lower breakfast numbers than the #3-ranked 34th St-Union Sq station.</p><p>Let’s take a look at this data one more way. Below are 3 tables, each lists the top-10 foot traffic stations for a specific meal time, with greatest on the top and least on the bottom. The lines help track a single station as it moves up or down through the relative rankings, with the top 3 overall busiest stations accented with color.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xjXU7ungZ7NcJDzGLbi3ZQ.png" /><figcaption>Image by the author</figcaption></figure><p>You can see right away that the overall rankings definitely do not stay fixed throughout the day, and in fact in some cases the rankings can change dramatically depending on the meal. For example 34th St-Penn Station in red is the #3 busiest station overall, and while it starts the day in the #2 position, it drops down to 6th over lunch, being beaten out by the #4, #5, and #6 overall during that time period. If a company was only targeting the busiest overall stations, they could potentially be missing-out on large amounts of sales everyday.</p><h3>What does this all really mean?</h3><p>The true goal is to optimize the potential path of a food truck on the street of New York City, so we need to not just look at where there are the most amount of people, but also take into account how long it will take for the vendor to travel between subway stations, and what kind of potential sales they may be missing while in transit. To do this, we needed to make a couple of assumptions about the business’s rate of sale, as well as rate of travel between breakfast and lunch, and lunch and dinner. We assumed that about 0.2% of all passersby would make a purchase at the food truck, and through investigating different routes and different times of day on google maps, we developed the following estimates:</p><p>The most profitable route would be to park at Grand Central Station, and stay all day. Grand Central is ranked #1 overall, as well as #1 is each time period, so this is a clear and smart choice. Since it is unwise to compete again other trucks within the same company, Grand Central is then excluded from other possible routes.<br>The #2 most profitable route would be to start at the 34th St-Penn Station for breakfast, then relocate to 34th St-Herald Sq around 11:30 am, which should provide a very large bump in foot traffic over 34th St-Penn Station for Lunch and dinner, while only requiring about a block of travel with minimal time lost.<br>The #3 most profitable route would actually be to park at 34th St-Herald Sq for the entire day, but since this completes with the #2 option listed above, if a food vendor business had access to 3 food trucks, the best option would be to have one truck at Grand Central, one at 34th St-Herald Sq, and one at 34th St-Penn, each staying for the entire work day, so as to maximize business-wide sales, while eliminating travel time and avoiding intra-company competition.<br>Finally, if a franchise had access to a 4th food truck, the next best route would be to start the day on 23rd St, then relocate to 14th-Union before the lunch rush and stay through dinner. This provides access to the greatest amount of remaining potential customers, without interfering with any of the above routes.</p><h3>Additional Thoughts</h3><p>There is always more work that can be done. In this case, if I were to continue to develop this model, I would run a similar analysis focused on the weekend, excluding workday commutes, and examined tourism. Also, I would try to incorporate a seasonality component, maybe emphasizing stations with protection from inclement weather. Finally, I would pull in additional sources of foot traffic proxy data, such as public transit buses and rideshare usage.</p><p>Through this project we have explored a couple of the different ways that a single data set can be viewed, and how deconstructing that data can tell a very different story than before. We strove to optimize the sales route of a mobile food vendor, balancing an effort to seek-out the highest traffic street corners with the highest potential for food sales, while minimizing sales lost while in transit between stations. I hope that you found something in here useful, informative, or interesting. <br>Please feel free to contact me to discuss this further!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zNrG0VgN_D6bSBN7SQeSVw.jpeg" /><figcaption>Photo by Anonymous on <a href="https://www.wallpaperflare.com/people-standing-beside-food-truck-burger-street-food-city-wallpaper-zmssz">Wallpaper Flare</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c213a1339a6f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>