Member-only story
How to Scrape Multiple URLs with Python: Tutorial
Because the data you want is not always the data you have
In this article, I will show you three ways to scrape data from multiple URLs. More specifically, I will show how to loop over the page number, loop over a manually created list of URLs and finally, loop over a scraped list of URLs.
I am assuming in this tutorial that you have some super basic knowledge of web scraping. If you need a quick refresher on how to inspect and scrape a website, check this out.
I will be scraping data from hostels in the beautiful city of Barcelona from Hostelworld, the best website to find hostels anywhere in the world. Alright, now let’s begin!
Looping over the page number
This is the simplest, most straightforward way of scraping multiple pages. Let’s begin by looking at the end of the URL we are scraping the hostels from (full URL available at the end of the article):
