How to Scrape Websites with Playwright and Python
If you’ve decided to try Playwright for scraping dynamic websites instead of Selenium or Pyppeteer, this post might be useful. It offers pretty much the same features, but with a few extras, like generating code based on your actions.
Setup: Install Python + Playwright
To make sure the scripts from this article work as expected, you’ll need Python 3.11 or higher. I haven’t tested them on older versions, so can’t guarantee anything there.
If you don’t have Python yet, just download it from the official website.
You can write the script in any text editor that supports syntax highlighting — no need to install a full IDE unless you really want to.
To install Playwright, open your terminal or command prompt (depends on your OS) and run:
pip install playwrightThat installs the library itself. To get the actual browsers, you’ll need to run one more command:
playwright installOnce that’s done, you’re good to go with Chromium, Firefox, and WebKit.
MWE: Scrape title from a real site
Before we jump into a more complex example, here’s a basic script that shows how to open a page and get the title using Playwright:
import asyncio
from playwright.async_api import async_playwright
async def run():
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
page = await browser.new_page()
await page.goto("https://example.com")
h1 = await page.text_content("h1")
print("H1:", h1)
await browser.close()
asyncio.run(run())As you can see, we’re using the asyncio library here because Playwright is asynchronous. If you’re not into async code, you might want to take a look at Selenium instead.
Scrape the Data
Let’s take some random online store as an example. In general, the data scraping principles stay the same, the only things that change are the elements we pull data from and the ways we deal with the site’s protection.
Locate & Parse Elements
Go to the target website and open DevTools to find out the tags of the elements that contain the data you need:
Find all of elements you need and create selectors and XPath expressions for them. Let’s put all that into a table:
Each product card has the class .product, so for the scraper to work properly, we first need to collect all elements with this class, and only then extract data from each individual card.
Get HasData’s stories in your inbox
Join Medium for free to get updates from this writer.
Write a simple script that launches a headless browser and opens the page:
import asyncio
from playwright.async_api import async_playwright
async def main():
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
page = await browser.new_page()
await page.goto("https://amundsensports.com/sweater-women/")
#Here will be code
await browser.close()
asyncio.run(main())Add a wait until the page finishes loading (at least until the product cards appear), and collect all the cards:
await page.wait_for_selector("li.product")
products = await page.query_selector_all("li.product")Extract the data from each card using the selectors we created earlier:
for product in products:
title = await (await product.query_selector(".woocommerce-loop-product__title")).inner_text()
gender = await (await product.query_selector(".woocommerce-loop-product__gender")).inner_text()
price = await (await product.query_selector(".price .amount")).inner_text()
link = await (await product.query_selector("a.woocommerce-loop-product__link")).get_attribute("href")
color_elements = await product.query_selector_all(".color-variable-item")
size_elements = await product.query_selector_all(".button-variable-item")
colors = [await el.get_attribute("title") for el in color_elements]
sizes = [await el.get_attribute("title") for el in size_elements]If we print out the results, they’ll look something like this:
But honestly, printing data to the screen isn’t really useful in real projects, so let’s go ahead and save it to a file instead.
Save the Data
The most common formats for storing scraped data are CSV and JSON. So, let’s import the libraries we need for that:
import json
import csvBefore scraping the page, create a variable to store all the collected data. We’ll use it later to export the results in the needed format:
results = []Inside the loop where we process each card, add the data to that variable:
results.append({
"title": title,
"gender": gender,
"price": price,
"link": link,
"colors": colors,
"sizes": sizes
})Save the results in JSON format:
with open("products.json", "w", encoding="utf-8") as f:
json.dump(results, f, ensure_ascii=False, indent=2)Or in CSV format:
with open("products.csv", "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=["title", "gender", "price", "link", "colors", "sizes"])
writer.writeheader()
for item in results:
writer.writerow({
"title": item["title"],
"gender": item["gender"],
"price": item["price"],
"link": item["link"],
"colors": ", ".join(item["colors"]),
"sizes": ", ".join(item["sizes"]),
})If you need to save the data in some other format, you can use the Pandas library. It lets you create dataframes and export them to the most popular formats — from databases to Excel files.
Full Example: Scrape e-commerce page
Here’s the full code of the scraper:
import asyncio
import json
import csv
from playwright.async_api import async_playwright
async def main():
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
page = await browser.new_page()
await page.goto("https://amundsensports.com/sweater-women/")
await page.wait_for_selector("li.product")
products = await page.query_selector_all("li.product")
results = []
for product in products:
title = await (await product.query_selector(".woocommerce-loop-product__title")).inner_text()
gender = await (await product.query_selector(".woocommerce-loop-product__gender")).inner_text()
price = await (await product.query_selector(".price .amount")).inner_text()
link = await (await product.query_selector("a.woocommerce-loop-product__link")).get_attribute("href")
color_elements = await product.query_selector_all(".color-variable-item")
size_elements = await product.query_selector_all(".button-variable-item")
colors = [await el.get_attribute("title") for el in color_elements]
sizes = [await el.get_attribute("title") for el in size_elements]
results.append({
"title": title,
"gender": gender,
"price": price,
"link": link,
"colors": colors,
"sizes": sizes
})
with open("products.json", "w", encoding="utf-8") as f:
json.dump(results, f, ensure_ascii=False, indent=2)
with open("products.csv", "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=["title", "gender", "price", "link", "colors", "sizes"])
writer.writeheader()
for item in results:
writer.writerow({
"title": item["title"],
"gender": item["gender"],
"price": item["price"],
"link": item["link"],
"colors": ", ".join(item["colors"]),
"sizes": ", ".join(item["sizes"]),
})
asyncio.run(main())You can use this script to scrape most of sites built with WooCommerce.
Final thoughts
Scraping with Playwright works fine — until you hit a site with solid anti-bot protection. If that happens, you’ll either need to tweak your script, deal with captchas using external services, set up proxies, or just switch to a scraping API that handles all that for you.
Here’s an example of the same script we used earlier with Playwright, but rewritten to use HasData’s web scraping API instead.
Since the API handles the request, we can drop Playwright and use plain requests and BeautifulSoup. There’s no need to mimic real user behavior anymore. The data parsing and saving part stays pretty much the same, just adjusted for the new libraries:
import requests
import json
from bs4 import BeautifulSoup
import csv
api_key = "YOUR-API-KEY"
target_url = "https://amundsensports.com/sweater-women/"
api_url = "https://api.hasdata.com/scrape/web"
payload = json.dumps({
"url": target_url,
"proxyType": "datacenter",
"proxyCountry": "US"
})
headers = {
'Content-Type': 'application/json',
'x-api-key': api_key
}
response = requests.post(api_url, headers=headers, data=payload)
html = response.json()["content"]
soup = BeautifulSoup(html, "html.parser")
products = soup.select("li.product")
results = []
for product in products:
title_tag = product.select_one(".woocommerce-loop-product__title")
gender_tag = product.select_one(".woocommerce-loop-product__gender")
price_tag = product.select_one(".price .amount")
link_tag = product.select_one("a.woocommerce-loop-product__link")
color_tags = product.select(".color-variable-item")
size_tags = product.select(".button-variable-item")
results.append({
"title": title_tag.get_text(strip=True) if title_tag else "",
"gender": gender_tag.get_text(strip=True) if gender_tag else "",
"price": price_tag.get_text(strip=True) if price_tag else "",
"link": link_tag['href'] if link_tag else "",
"colors": [tag.get("title", "") for tag in color_tags],
"sizes": [tag.get("title", "") for tag in size_tags],
})
with open("products.json", "w", encoding="utf-8") as f:
json.dump(results, f, ensure_ascii=False, indent=2)
with open("products.csv", "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=["title", "gender", "price", "link", "colors", "sizes"])
writer.writeheader()
for item in results:
writer.writerow({
"title": item["title"],
"gender": item["gender"],
"price": item["price"],
"link": item["link"],
"colors": ", ".join(item["colors"]),
"sizes": ", ".join(item["sizes"]),
})To run this script, you’ll need a HasData API key, which you can get after signing up.
This setup is just easier. You skip the whole mess with anti-bot defenses, and the API works for pretty much any site. But if you’re targeting something specific, like Google SERP, Amazon, Zillow, etc., you’re better off using a dedicated API that returns clean, structured JSON instead of raw HTML.
