Multithreading with Web Scraping using Beautiful Soup 4 and pandas

Photo by Anand Thakur on Unsplash

I was recently scraping ticket information for a project I was working on. I wanted to get all of the information that I could from a bus ticketing website. My eventual goal was to analyze trends in bus tickets and availability. To the information I want, I need to send a request for each possible permutation of origin and destination, for about 2 weeks. This ends up being more than 30,000 requests. My script was taking over 10 hours to run! In comes multithreading.

The problem is that each request takes a small amount of time to go and come back. Each of these trips takes a little bit of time, which means that your script is just sitting and waiting for the information to come back. Instead of waiting for each request to return, multithreading lets you send many requests in parallel, which means less waiting and more scraping.

Here is the basic gist of the script:

from multiprocessing.pool import ThreadPool
with ThreadPool(10) as pool:
    for result in pool.map(df_loops, _from_to_date):
        df = df.append(result, ignore_index=True, sort=True)

First, we import the function ThreadPool from the module multiprocessing.pool. We open the pool with 10 threads, and each thread does the function df_loops with the inputs from the list _from_to_date, which as you probably guessed, is a list of lists with 3 keys: from, to, and date.

Get Tyler Tresslar’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The df_loops function looks like this:

def df_loops(_list):
 # first I unpack the list
 origin, destination, date = _list
 # then I fetch the url with to use beautiful soup 
 soup = get_soup(origin,destination,date)
 # the function get_ticket_info uses bs4 to parse everything 
 # into a usableformat
 new_df = pd.DataFrame(get_ticket_info(soup))
 return new_df

each time I get a new data frame and append it to the old one. Once it’s finished running, save it all in a CSV:

csv_name = "scraped_" + datetime.datetime.today().strftime("%Y-%m-%d")+ ".csv"
df.to_csv(csv_name)

I hope you found this useful, the full script is available on GitHub!