While other students presented really interesting projects on forecasting Tesla stock prices, simulating a self-driving car, or using generative adversarial networks to draw images, the problem I focused on was a simple one. In this post, I’m going to give a quick summary of my process, plus comment on a few things I learned along the way. It may be useful for future students considering what kind of projects they’d like to complete during their data science courses.

When I first moved to DC, I was an unpaid intern with a number of part-time jobs. I started working for Bookstore Movers in January of 2012, and within a few months I was out driving a moving truck and supervising moving crews every day. The company was small back then, and I’ve become lifelong friends with some of the folks I met while working there. I enjoyed not having to sit at a desk so much that I ended up working at Bookstore Movers off and on for a number of years.

For my final data science project at GA, I wanted to show off a diverse set of skills, address a well-defined problem statement, and make something that can actually be deployed and used. Doing a project using data from Bookstore Movers had the additional benefit of synthesizing my past with my future career, and I thought it would give me an opportunity to discuss the leadership and management experience I gained working at Bookstore Movers. So, what’s the data science question?

When a potential client seeks a quote for a moving job, they fill out an online estimate form on our website. That form contains their origin and destination addresses, some categorical data about the type of place they’re moving out of (and into), how many stairs are at each location, the number of boxes they have, plus some text fields for clients to tell us what furniture they have and anything else they see fit to mention. Currently, a team of admin staff at Bookstore Movers responds to these forms with a moving estimate. That estimate tells the client whether a big or a small truck will be required, how many hours the job will likely take, and information on our hourly rate and travel fee, if applicable. Using data from tens of thousands of completed jobs, I built a series of models on this data.

In a real data science project, the problem statement is not obvious. Although I ultimately wanted to provide an accurate estimate of how long a job would take by comparing client-submitted data to the real length of each move, I could just as easily try and predict the human-estimated time for each job. Lastly, it was also possible to perform a binary classification to predict whether (and why) a job would tend to take longer than estimated. I decided that our admin team didn’t need their own thinking process explained to them, and that training a model to actually predict how long a job would take would be the most useful path forward, if also the hardest. I also built a binary classifier to pinpoint factors that explained a job taking longer than expected.

My dataset contained about 17,000 observations on 175 variables, going all the way back to the early days, when Bookstore Movers used Google Docs to track jobs. I began by constructing lists of my predictor variables, divided according to whether they represented numeric, categorical, ordinal, or text data. Then I gathered all the target variables, including the size of the truck that was sent out, the actual number of hours the job took, the hourly rate, whether there was a travel fee or not, and a few other variables that, even if I didn’t end up modeling them, certainly didn’t belong to the list of predictor variables. As it turns out, identifying target variables isn’t always as simple as it seems. For example, is the number of movers assigned to a job a predictor or a target? If my model is meant to replicate the job of the human admin team, then it should be treated as a target variable to be spit out by the model. But in order to predict how long a job will take, the number of movers scheduled can be treated as a given.

Next, I went about narrowing down the number of features in my dataset. Some variables turned out to be useless ID type variables which I could drop, while others just duplicated information that was found elsewhere. Many variables were simply null for most observations. The column for the type of elevator at the fourth location was N/A for the majority of observations, for example. To make my job simpler, I decided only to model the typical job that goes from point A to point B, and to drop from my dataset any job with more than two locations. I also dropped out commercial jobs, which are significantly different from residential jobs and which are a minority in the dataset. Data cleaning is often thought of as a mindless and cumbersome process. In fact, data cleaning demands countless small decisions that may significantly affect the outcome of analysis.

To give one example, clients moving in or out of storage units usually give some dimensions for the size of the unit, but that data isn’t collected in as organized a way as the number of rooms in a house or apartment. I treated any observation containing the word ‘storage’ for the origin or destination as coming from or going to a storage unit, regardless of size. Obviously, I gave up important information here, but it saved me time spent writing regex code and deciding whether a numerical, categorical, or ordinal encoding would better suit the data. Keeping track of these kinds of decisions makes it easy to revisit the process at a later date to improve the performance of the model.

After all my data was cleaned, I had addresses, text, numeric, and categorical data to work with. Diversity in my data allowed me to bring a lot of different skills into play. I encoded half a dozen categorical variables as dummy variables. Using Google Maps’ APIs, I calculated the driving time between each location to use as a predictor variable. Using regex and basic string manipulation, I engineered a number of features including how many rooms each location had. Using Sci-kit Learn’s CountVectorizer function, I built a vocabulary of common furniture items from a training set, so that I could count how often each term appeared. I used the TextBlob package to run sentiment analysis on my text fields.

I would recommend anyone looking for a data science project to identify a problem that involves a variety of data types. That way, a student can show off a range of data science skills without putting all their eggs into one technical basket. Performing a time series analysis or doing NLP are worthy tasks, but they might not give you much to talk about if you’re interviewing at a company that doesn’t do time series or NLP.

After getting my data into good shape, random forest models tended to perform the best for both classification and regression. I built one classifier to determine whether a job should be done with a small moving truck or a large one. Because a big truck can accomplish a smaller job, but the reverse is not true, a model to assign jobs to a truck should be very careful not to assign a very large job to a small truck. Therefore, that model was scored and tuned to maximize sensitivity, treating ‘small truck jobs’ as the positive class.

Another two models were trained to predict how long a job would take, given that it was a big truck or a small truck job. I used R-squared to evaluate those models, and I compared their score against the R-squared of the human admin team whose job it is to provide estimates. I counted it as initial success when my models performed nearly as well as human estimators, instead of relying on an arbitrary definition of what counts as a good R-squared score.

Lastly, I trained a binary classifier to determine whether a job would take more or less time than estimated by our admin team. Because in this case there was no real-world cost of misclassification one way or another, and I just wanted to know what features were associated with human estimators being wrong, I optimized this model using AUC-ROC, which is a threshold-agnostic way of scoring a classifier for its ability to distinguish between two classes.

Obviously, I used different model evaluation metrics for each of the models I built. Choosing the correct evaluation metric will depend on subject matter expertise, knowing the ‘cost’ of being wrong, and understanding exactly what your model is meant to deliver.

Because there is so much unstructured data in this dataset, I could go down practically as many rabbit holes as time allows. In a field where aspiring practitioners are constantly told to have side projects, a project that keeps on giving really isn’t a bad thing. And so, for the purposes of my data science bootcamp, this project is complete. But since I plan to deploy the system of models I’ve developed, the project is just a rough draft. A good data science project is one that you can keep on working on.

Without revealing any of the results of my analysis, I’ve tried to share a few lessons I learned completing my final project. I would encourage any student looking for a data science project to find subject matter that permits many different questions to be asked, that combines structured and unstructured data, and that allows a wide range of skills to be demonstrated. Embrace the complexity in real world data science problems, and don’t be afraid to bring a suite of relatively simple methods into play. Not every problem requires the newest cutting edge technique. Lastly, try and find a data science project that could actually be useful to someone.

A Markov chain is a simulated sequence of events. Each event in the sequence comes from a set of outcomes that depend on one another. In particular, each outcome determines which outcomes are likely to occur next. In a Markov chain, all of the information needed to predict the next event is contained in the most recent event. That means that knowing the full history of a Markov chain doesn’t help you predict the next outcome any better than only knowing what the last outcome was.

Markov chains aren’t generally reliable predictors of events in the near term, since most processes in the real world are more complex than Markov chains allow. Markov chains are, however, used to examine the long-run behavior of a series of events that are related to one another by fixed probabilities.

For any sequence of non-independent events in the world, and where a limited number of outcomes can occur, conditional probabilities can be computed relating each outcome to one another. Often this simply takes the form of counting how often certain outcomes follow one another in an observed sequence.

To generate a simulation based on a certain text, count up every word that is used. Then, for every word, store the words that are used next. This is the distribution of words in that text conditional on the preceding word.

In order to simulate some text from Donald Trump, let’s use a collection of his speeches from the 2016 campaign available here.

First import numpy and the text file containing Trump’s speeches:

import numpy as np
trump = open('speeches.txt', encoding='utf8').read()

Then, split the text file into single words. Note we’re keeping all the punctuation in, so our simulated text has punctuation:

corpus = trump.split()

Then, we define a function to give us all the pairs of words in the speeches. We’re using lazy evaluation, and yielding a generator object instead of actually filling up our memory with every pair of words:

def make_pairs(corpus):
    for i in range(len(corpus)-1):
        yield (corpus[i], corpus[i+1])
        
pairs = make_pairs(corpus)

Then we instantiate an empty dictionary, and fill it words from our pairs. If the first word of the pair is already a key in the dictionary, simply append the next word to the list of words that follow that word. Otherwise, initialize a new entry in the dictionary with the key equal to the first word and the value a list of length one:

word_dict = {}

for word_1, word_2 in pairs:
    if word_1 in word_dict.keys():
        word_dict[word_1].append(word_2)
    else:
        word_dict[word_1] = [word_2]

Finally we pick some random word to kick off the chain, and choose the number of words we want to simulate:

first_word = np.random.choice(corpus)

chain = [first_word]

n_words = 30

After the first word, every word in the chain is sampled randomly from the list of words which have followed that word in Trump’s actual speeches:

for i in range(n_words):
    chain.append(np.random.choice(word_dict[chain[-1]]))

The final join command returns the chain as a string:

' '.join(chain)

When I run this code, my first result is:

'I will be able to vote. Got them back. We’re going to make a total lie, proven out right after. "During the opposite. We have some turnout. My patients are really'

The nice thing here is that we’re using a dictionary to actually look up the next word in the chain. Of course, you can wrap this all up in a function, which I leave as an exercise to the reader.

There are a lot of tools are there to ‘Markovify’ text, and I encourage you to look them up. But for someone just learning Markov chains, the code here is an easy place to start. But there are endless possibilities for improvement. For example, you might require the first word to be capitalized, so your text doesn’t begin mid-sentence:

while first_word.islower():
    first_word = np.random.choice(corpus)

I hope this is helpful for those of you getting started in the wide world of Markov chains. If this code can be improved without sacrificing clarity, leave a comment!

Simulating Text With Markov Chains in Python was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

So, what are Markov chain Monte Carlo (MCMC) methods? The short answer is:

MCMC methods are used to approximate the posterior distribution of a parameter of interest by random sampling in a probabilistic space.

In this article, I will explain that short answer, without any math.

First, some terminology. A parameter of interest is just some number that summarizes a phenomenon we’re interested in. In general we use statistics to estimate parameters. For example, if we want to learn about the height of human adults, our parameter of interest might be average height in in inches. A distribution is a mathematical representation of every possible value of our parameter and how likely we are to observe each one. The most famous example is a bell curve:

In the Bayesian way of doing statistics, distributions have an additional interpretation. Instead of just representing the values of a parameter and how likely each one is to be the true value, a Bayesian thinks of a distribution as describing our beliefs about a parameter. Therefore, the bell curve above shows we’re pretty sure the value of the parameter is quite near zero, but we think there’s an equal likelihood of the true value being above or below that value, up to a point.

As it happens, human heights do follow a normal curve, so let’s say we believe the true value of average human height follows a bell curve like this:

Clearly, the person with beliefs represented by this graph has been living among giants for years, because as far as they know, the most likely average adult height is 6'2" (but they’re not super confident one way or another).

Lets imagine this person went and collected some data, and they observed a range of people between 5' and 6'. We can represent that data below, along with another normal curve that shows which values of average human height best explain the data:

In Bayesian statistics, the distribution representing our beliefs about a parameter is called the prior distribution, because it captures our beliefs prior to seeing any data. The likelihood distribution summarizes what the observed data are telling us, by representing a range of parameter values accompanied by the likelihood that each each parameter explains the data we are observing. Estimating the parameter value that maximizes the likelihood distribution is just answering the question: what parameter value would make it most likely to observe the data we have observed? In the absence of prior beliefs, we might stop there.

The key to Bayesian analysis, however, is to combine the prior and the likelihood distributions to determine the posterior distribution. This tells us which parameter values maximize the chance of observing the particular data that we did, taking into account our prior beliefs. In our case, the posterior distribution looks like this:

Above, the red line represents the posterior distribution. You can think of it as a kind of average of the prior and the likelihood distributions. Since the prior distribution is shorter and more spread out, it represents a set of belief that is ‘less sure’ about the true value of average human height. Meanwhile, the likelihood summarizes the data within a relatively narrow range, so it represents a ‘more sure’ guess about the true parameter value.

When the prior the likelihood are combined, the data (represented by the likelihood) dominate the weak prior beliefs of the hypothetical individual who had grown up among giants. Although that individual still believes the average human height is slightly higher than just what the data is telling him, he is mostly convinced by the data.

In the case of two bell curves, solving for the posterior distribution is very easy. There is a simple equation for combining the two. But what if our prior and likelihood distributions weren’t so well-behaved? Sometimes it is most accurate to model our data or our prior beliefs using distributions which don’t have convenient shapes. What if our likelihood were best represented by a distribution with two peaks, and for some reason we wanted to account for some really wacky prior distribution? I’ve visualized that scenario below, by hand drawing an ugly prior distribution:

As before, there exists some posterior distribution that gives the likelihood for each parameter value. But its a little hard to see what it might look like, and it is impossible to solve for analytically. Enter MCMC methods.

MCMC methods allow us to estimate the shape of a posterior distribution in case we can’t compute it directly. Recall that MCMC stands for Markov chain Monte Carlo methods. To understand how they work, I’m going to introduce Monte Carlo simulations first, then discuss Markov chains.

Monte Carlo simulations are just a way of estimating a fixed parameter by repeatedly generating random numbers. By taking the random numbers generated and doing some computation on them, Monte Carlo simulations provide an approximation of a parameter where calculating it directly is impossible or prohibitively expensive.

Suppose that we’d like to estimate the area of the follow circle:

Since the circle is inside a square with 10 inch sides, the area can be easily calculated as 78.5 square inches. Instead, however, we can drop 20 points randomly inside the square. Then we count the proportion of points that fell within the circle, and multiply that by the area of the square. That number is a pretty good approximation of the area of the circle.

Since 15 of the 20 points lay inside the circle, it looks like the circle is approximately 75 square inches. Not too bad for a Monte Carlo simulation with only 20 random points.

Now, imagine we’d like to calculate the area of the shape plotted by the Batman Equation:

Here’s a shape we never learned an equation for! Therefore, finding the area of the bat signal is very hard. Nevertheless, by dropping points randomly inside a rectangle containing the shape, Monte Carlo simulations can provide an approximation of the area quite easily!

Monte Carlo simulations aren’t only used for estimating the area of difficult shapes. By generating a lot of random numbers, they can be used to model very complicated processes. In practice, they’re used to forecast the weather, or estimate the probability of winning an election.

The second element to understanding MCMC methods are Markov chains. These are simply sequences of events that are probabilistically related to one another. Each event comes from a set of outcomes, and each outcome determines which outcome occurs next, according to a fixed set of probabilities.

An important feature of Markov chains is that they are memoryless: everything that you would possibly need to predict the next event is available in the current state, and no new information comes from knowing the history of events. A game like Chutes and Ladders exhibits this memorylessness, or Markov Property, but few things in the real world actually work this way. Nevertheless, Markov chains are powerful ways of understanding the world.

In the 19th century, the bell curve was observed as a common pattern in nature. (We’ve noted, for example, that human heights follow a bell curve.) Galton Boards, which simulate the average values of repeated random events by dropping marbles through a board fitted with pegs, reproduce the normal curve in their distribution of marbles:

Pavel Nekrasov, a Russian mathematician and theologian, argued that the bell curve and, more generally, the law of large numbers, were simply artifacts of children’s games and trivial puzzles, where every event was completely independent. He thought that interdependent events in the real world, such as human actions, did not conform to nice mathematical patterns or distributions.

Andrey Markov, for whom Markov chains are named, sought to prove that non-independent events may also conform to patterns. One of his best known examples required counting thousands of two-character pairs from a work of Russian poetry. Using those pairs, he computed the conditional probability of each character. That is, given a certain preceding letter or white space, there was a certain chance that the next letter would be an A, or a T, or a whitespace. Using those probabilities, Markov was ability to simulate an arbitrarily long sequence of characters. This was a Markov chain. Although the first few characters are largely determined by the choice of starting character, Markov showed that in the long run, the distribution of characters settled into a pattern. Thus, even interdependent events, if they are subject to fixed probabilities, conform to an average.

For a more useful example, imagine you live in a house with five rooms. You have a bedroom, bathroom, living room, dining room, and kitchen. Lets collect some data, assuming that what room you are in at any given point in time is all we need to say what room you are likely to enter next. For instance, if you are in the kitchen, you have a 30% chance to stay in the kitchen, a 30% chance to go into the dining room, a 20% chance to go into the living room, a 10% chance to go into the bathroom, and a 10% chance to go into the bedroom. Using a set of probabilities for each room, we can construct a chain of predictions of which rooms you are likely to occupy next.

Making predictions a few states out might be useful, if we want to predict where someone in the house will be a little while after being in the kitchen. But since our predictions are just based on one observation of where a person is in the house, its reasonable to think they won’t be very good. If someone went from the bedroom to the bathroom, for example, its more likely they’ll go right back to the bedroom than if they had come from the kitchen. So the Markov Property doesn’t usually apply to the real world.

Running the Markov chain for thousands of iterations, however, does give the long-run prediction of what room you’re likely to be in. More importantly, this prediction isn’t affected at all by which room the person began in! Intuitively, this makes sense: it doesn’t matter where someone is in the house at one point in time in order to simulate and describe where they are likely to be in the long-term, or in general. So Markov chains, which seem like an unreasonable way to model a random variable over a few periods, can be used to compute the long-run tendency of that variable if we understand the probabilities that govern its behavior.

With some knowledge of Monte Carlo simulations and Markov chains, I hope the math-free explanation of how MCMC methods work is pretty intuitive.

Recall that we are trying to estimate the posterior distribution for the parameter we’re interested in, average human height:

We know that the posterior distribution is somewhere in the range of our prior distribution and our likelihood distribution, but for whatever reason, we can’t compute it directly. Using MCMC methods, we’ll effectively draw samples from the posterior distribution, and then compute statistics like the average on the samples drawn.

To begin, MCMC methods pick a random parameter value to consider. The simulation will continue to generate random values (this is the Monte Carlo part), but subject to some rule for determining what makes a good parameter value. The trick is that, for a pair of parameter values, it is possible to compute which is a better parameter value, by computing how likely each value is to explain the data, given our prior beliefs. If a randomly generated parameter value is better than the last one, it is added to the chain of parameter values with a certain probability determined by how much better it is (this is the Markov chain part).

To explain this visually, lets recall that the height of a distribution at a certain value represents the probability of observing that value. Therefore, we can think of our parameter values (the x-axis) exhibiting areas of high and low probability, shown on the y-axis. For a single parameter, MCMC methods begin by randomly sampling along the x-axis:

Since the random samples are subject to fixed probabilities, they tend to converge after a period of time in the region of highest probability for the parameter we’re interested in:

After convergence has occurred, MCMC sampling yields a set of points which are samples from the posterior distribution. Draw a histogram around those points, and compute whatever statistics you like:

Any statistic calculated on the set of samples generated by MCMC simulations is our best guess of that statistic on the true posterior distribution.

MCMC methods can also be used to estimate the posterior distribution of more than one parameter (human height and weight, say). For n parameters, there exist regions of high probability in n-dimensional space where certain sets of parameter values better explain observed data. Therefore, I think of MCMC methods as randomly sampling inside a probabilistic space to approximate the posterior distribution.

Recall the short answer to the question ‘what are Markov chain Monte Carlo methods?’ Here it is again as a TL;DR:

MCMC methods are used to approximate the posterior distribution of a parameter of interest by random sampling in a probabilistic space.

I hope I’ve explained that short answer, why you would use MCMC methods, and how they work. The inspiration for this post was a talk I gave as part of General Assembly’s Data Science Immersive course in Washington, DC. The goals of that talk were to explain Markov chain Monte Carlo methods to a non-technical audience, and I’ve tried to do the same here. Leave a comment if you think this explanation is off the mark in some way, or if it could be made more intuitive.

A Zero-Math Introduction to Markov Chain Monte Carlo Methods was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

In my last post, I used Python’s requests and Beautiful Soup packages to write two web scraping functions to grab “Hot Network Questions” from Stack Exchange and extract the text of a Stack Exchange question based on its URL. Additionally, I wrote a few helper functions to identify a question’s category and its ID from its URL, and (conversely) to reconstruct a Stack Exchange URL from a given category and ID.

Provided we are in the same working directory as before, we can import those functions with

from SEData.data import *

As a first step in my web scraping process, I periodically ran a script that would check Stack Exchange’s top 50 “Hot Network Questions” using the populate_question_links function I defined in the last post. For each of these links, I would decompose the URL into the Stack Exchange forum, or category, that it came from, and the question’s unique ID. Every time I ran this script, I was populating a running list of the different question categories, plus a recent ID for each category.

Importing this list of recent questions in a separate script, we can create a list of links to scrape. To create a new data set of 2000 observations, for example, we divide 2000 by 50, our number of categories, to arrive at 40: the number of links we’ll need to scrape for every category. Note that we’re guaranteeing balanced classes this way. We’ll next write a function to generate a list of 40 new links from the list of newest questions above:

After scraping once or twice following this procedure, I realized there was nothing to stop me from scraping the same links twice. So I cross reference my list of links to scrape against those I’ve already scraped:

data_links = back_generate_links(questions['url'], n = 40)

data_links = [i for i in data_links if i not in list(links_already_scraped)]

Now we’re finally ready to scrape Stack Exchange:

Thanks to the power of Python list comprehensions, our scraping is performed in just one line of code. Note we have enabled our pause argument: by default, this pauses the function for a random amount of time between 1 and 5 seconds. This to avoid being rate-limited by Stack Exchange, and because its probably best practice to be kind to the bandwidth of a website you’re trying to scrape.

We append our data to a .csv using mode = 'a', and we update our list of previously scraped links. This way, whenever want to scrape Stack Exchange, we can be sure we’re not over-writing our previous data or wasting time scraping links we’ve already processed.

As I’ve written it here, this procedure works just fine. But as we continue to update our running list of recent “Hot Network Questions,” our list of categories will grow from below 50 categories towards 75 categories or more. This is because at any given time some categories are represented more than once in Stack Exchange’s trending questions. Others will show up only rarely, so it will take us a while to catch them. If we begin scraping Stack Exchange using a list of 40 categories, for example, we’ll have an equal number of observations for each category. But when new categories show up on our radar, and we be begin scraping them, they’ll be under-represented in our data set moving forward.

As I’ve learned in the course of this project, writing a single function to do some web scraping is easy. The hard part is iterating over many pages while storing data effectively and avoiding the types of error messages, missing data, and duplicates that make your final data set very ugly.

To someone teaching himself the basics of Data Science, web scraping is a seductive tool. Done right, it promises to yield practically limitless amounts of clean, well-formatted text data. But even if HTTP error messages, pages with no data, and messy punctuation have been removed, certain data collection considerations should not be ignored.

Most websites a Data Scientist would want to scrape contain way more data than an individual can (or should) ever capture. Therefore, sampling methods need to be taken into consideration. Put another way, knowing what to scrape and when scrape it become very important.

In the case of Stack Exchange questions, an ever-growing list of the more active fora can lead to unbalanced classes in your final data set, and create a moving goalpost for your modelling efforts. If your goal is to accurately classify questions by category, the more values your target variable can take on, the harder your classification task will be.

Since beginning this project, I’ve begun a 12-week full-time Data Science immersive course at General Assembly in Washington, DC. In the last week, we’ve begun discussing web scraping. “Scrape early, scrape often” is oft-repeated advice in this context. But scraping intermittently, at arbitrary times of the day, over an undefined period of time, is not a systematic approach to sampling. Depending on the website you are targeting, you may be sampling only trending posts, rather than all posts, or data that are particular to certain times of the day. Scraping Twitter during the day, for example, won’t fetch much data from Twitter users on the other side of the planet. Scraping the front page of Reddit won’t tell you much about the posts that get downvoted or are never seen.

Although a Data Scientist would never normally want to give up precious data, web scraping presents a case in which more data is always a few (thousand) iterations of a web scraping script away. Throwing away a lot of data that was gathered intermittently and arbitrarily will almost always strengthen the integrity of the analysis. After testing out a certain web scraping procedure for the sampling biases or messy data challenges it represents, my recommendation is to begin again, forget about all the data that has been gathered so far, and begin scraping with a more systematic approach.

I.

If your work involves any kind of coding, you likely spend a lot of time on Stack Overflow troubleshooting your error messages and looking for answers to your “how to” questions. In fact, Stack Overflow’s annual developer survey reveals that more experienced developers consult Stack Overflow just as much, if not more, than less experienced developers.

When I consult Stack Overflow, I often become distracted by the “Hot Network Questions” in Stack Exchange’s other fora. These are trending questions displayed on the sidebar that give you an idea of the kinds of questions people are asking (and answering) about all the other subjects for which Stack Exchange maintains a forum.

Inevitably, I can’t help clicking on questions like “Is each of the 18 Mahapurans related to a different Kalpa?” or “Why do pens write better on pages that have pages below them?” In the first case, I have absolutely no idea what that question is even about; in the second case, I’m curious what kind of forum exists for questions like that. (The first question comes from the Hinduism forum, the second was posed to the experts at the Physics forum.)

(Sidenote: My favorite questions are almost always from the Worldbuilding forum, a place for video game designers and writers to test out concepts for their stories. My favorite recent question was “How to safely check if you are immortal?”.)

After clicking through to many Stack Exchange questions, and being consistently surprised by the forum to which a particular question belonged, I decided to see if I could train a model to accurately predict the forum, or category, of each question.

Over the next few blog posts, I’ll walk through my process of scraping Stack Exchange for question texts, transforming those texts into a numerical format suitable for modelling, and doing the modelling itself.

II.

To begin, we import the necessary Python packages:

import requests
from bs4 import BeautifulSoup

We’ll use the requests package to easily send out HTTP requests and return the output of those requests as an HTML document. We’ll use the BeautifulSoup function from the Beautiful Soup 4 (bs4) package to parse and make sense of those HTML documents.

url = 'https://stackexchange.com/questions?pagesize=50'

r = requests.get(url)

html_doc = r.text 

soup = BeautifulSoup(html_doc, 'lxml')

In one line of code, the requests package sends out an HTTP request, receives the response, and stores as the variable r. We call the .text attribute to access the HTML document, and store it. Finally we pass that HTML document to the BeautifulSoup function to create a navigable “parse tree” to extract the information we’re looking for.

(We also tell BeautifulSoup to use Python’s xlml parser on the HTML document. Beautiful Soup’s documentation contains more information on the pros and cons of different HTML parsers.)

III.

Beautiful Soup provides a useful API for navigating the parse tree generated from our HTML document. We can examine individual tags by name by calling them as an attribute of the Beautiful Soup object (soup.head), or we can use the find() or find_all() functions to filter tags by their name, attributes, or text.

We want to scrape Stack Exchange’s “Hot Network Questions” page for all of the URLs it contains. First, we initialize an empty list to store the URLs. Then we search the parse tree for all <a> tags, which define a hyperlink. For each of those links, we extract and append the href attribute, which gives the URL for that hyperlink.

links = []
for link in soup.find_all('a'):
    links.append((link.get('href')))

The next few lines of code filter and refine my list of links to ensure I’m only capturing links to Stack Exchange questions. Note that I make use of the re module for regular expressions. I also drop questions which are linked to more than once.

import re

question_links = [k for k in links if k and 'questions' in k]

pattern = re.compile('questions/\d')

question_links = filter(pattern.search, question_links)

question_links = list(set(question_links))

Now we can wrap up all of the above code in a function we can call whenever we want to grab the links to the top 50 Stack Exchange “Hot Network Questions.”

IV.

We’ve now written a helper function to grab the links to 50 trending Stack Exchange questions. Later on, we’ll use it to populate a list of recent questions by category, and then we’ll iteratively build our dataset from that list.

Briefly, lets define a function to return the category of a question based on its URL. Because Stack Exchange URLs take the form https://category.com/questions/questionID/, we can simply use Python string splitting to grab the category:

Now, we’re ready to define our workhorse web scraping function. This function will take a link and return the text of the corresponding Stack Exchange question, plus the forum where it was asked. As before, we send and catch an HTTP request using requests.get(), store the HTML document, and parse it using Beautiful Soup.

This time, instead of extracting the URL from all hyperlink tags, we want to find the tag containing the text of the post, and extract that text. After digging around in the parse tree in order to understand the structure of our HTML document, it turns out that every question and answer on a Stack Exchange page has a tag of the same class. Since we know that the first time that tag appears is always the question itself, we use Beautiful Soup’s find() function to return the first (or only) appearance of the tag with the class attribute equal to post-text:

text = soup.find(attrs={'class':'post-text'}).get_text()

(If we wanted to return the question and all of the responses, we would use find_all().)

Now that we’ve fiddled with the parse tree and sorted out how to extract just the text of a question, we want to wrap it all up in a function.

If we wrap just the lines above in a function get_text(url) we quickly discover that our function fails as we iterate through different question URLs. It turns out that there are valid Stack Exchange links that do not have any tags of class post-text. That’s because those questions were removed either by the user or by Stack Exchange moderators.

To deal with this case, we do our web-scraping inside a try block. If any error occurs, our function returns np.NaN for the question label and question text.

Another error that occurs when quickly iterating over many URLs is that you will exceed Stack Exchange’s rate limit. In this case, you will be denied further HTTP requests and your code will stall out. To prevent this, I import and use the sleep and randint functions from the time and random packages, respectively. If the pause argument is True, the function will pause between 1 and 5 seconds (unless sleep_max is set to some other value). This slows down the rate of HTTP requests to something more closely approximating the speed that a user might normally click through a website.

Finally, the function by default removes punctuation, digits, and line breaks.

V.

We’ve defined our two main web scraping functions. We’ll define three additional helper functions below which will be useful as we iteratively construct our data set.

Since we won’t be messing about with these functions much now that they’re defined, we can create a Python package to store them in:

$ mkdir SEData
$ cd SEData
$ touch __init__.py
$ touch data.py

Inside our data.py script, we’ll save these function definitions along with their package dependencies. Then, we can import them all at once in our Jupyter Notebooks as we begin our analysis. Boom!

In my next post, I’ll use these functions to iteratively update a running list of recent Stack Exchange questions and create a data set of questions texts and labels.