Selenium Interactions and Data Extraction in Python

This is the last post in the Selenium scraping series.

Get HasData’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Here, we’ll look at the most useful parts of scraping: how to move around pages, handle tabs and popups, fill forms, scroll, and extract data into Python.

Navigation and Page States

Basic navigation is simple but important. These methods help you move through pages like a real user:

driver.get("https://example.com")   # Open a page
driver.refresh()                    # Reload
driver.back()                       # Go back
driver.forward()                    # Go forward

To check the current state:

print(driver.title)
print(driver.current_url)

Handling Tabs and Popups

Modern sites often open links in new tabs or popups. Here’s how to switch between them:

driver.execute_script("window.open('https://example.com')")
tabs = driver.window_handles
 
driver.switch_to.window(tabs[1])  # Switch to new tab
print(driver.title)
 
driver.close()                    # Close tab
driver.switch_to.window(tabs[0])  # Back to main

This same logic works for popups and modal dialogs too.

Scrolling and Infinite Scroll

Some pages load content only when you scroll. You can scroll using JavaScript:

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

For infinite scroll pages, loop until no new data appears:

import time
 
last_height = driver.execute_script("return document.body.scrollHeight")
 
while True:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(2)
    new_height = driver.execute_script("return document.body.scrollHeight")
    if new_height == last_height:
        break
    last_height = new_height

Filling Forms and Sending Keys

Use send_keys() to type text into inputs and Select for dropdowns.

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select
 
driver.find_element(By.NAME, "email").send_keys("test@example.com")
 
select = Select(driver.find_element(By.ID, "country"))
select.select_by_visible_text("Canada")

You can also press Enter or Tab using special keys.

Using By

Selenium 4.10+ encourages using the By:

from selenium.webdriver.common.by import By
 
driver.find_element(By.CSS_SELECTOR, ".btn-submit").click()
driver.find_elements(By.XPATH, "//div[@class='item']")

Working with Shadow DOM

Some modern websites use Shadow DOM, a special DOM layer that hides elements. You can reach inside it using JavaScript:

shadow_host = driver.find_element(By.CSS_SELECTOR, "custom-element")
shadow_root = driver.execute_script("return arguments[0].shadowRoot", shadow_host)
button = shadow_root.find_element(By.CSS_SELECTOR, "button.submit")
button.click()

Getting Clean Text and Attributes

Text can be overloaded, always strip it and handle missing attributes safely.

el = driver.find_element(By.CSS_SELECTOR, ".product")
name = el.text.strip()
link = el.get_attribute("href") or ""

Extracting Lists and Tables

If you need multiple rows or elements:

rows = driver.find_elements(By.CSS_SELECTOR, "table tr")
for row in rows:
    cols = [td.text for td in row.find_elements(By.TAG_NAME, "td")]
    print(cols)

For lists:

items = [el.text for el in driver.find_elements(By.CSS_SELECTOR, ".list-item")]

Exporting Data to CSV or JSON

Once you have structured data, export:

import csv, json
 
data = [
    {"name": "Keyboard", "price": "$50"},
    {"name": "Mouse", "price": "$20"}
]
 
# CSV
with open("data.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=data[0].keys())
    writer.writeheader()
    writer.writerows(data)
 
# JSON
with open("data.json", "w", encoding="utf-8") as f:
    json.dump(data, f, indent=2)

Final Thoughts

If you want to take your scraping even further, you can scale it with an API like HasData.

import asyncio
import aiohttp
import requests
 
api_key = "YOUR_API_KEY"
 
# Get available concurrency for your API key
def get_available_concurrency():
    url = "https://api.hasdata.com/user/me/usage"
    headers = {
        "Content-Type": "application/json",
        "x-api-key": api_key
    }
    response = requests.get(url, headers=headers)
    if response.status_code == 200:
        data = response.json()
        # return available concurrency, fallback to 1
        return data.get("data", {}).get("availableConcurrency", 1)
    return 1
 
# Build payload for scraping a single URL
def make_payload(url):
    return {
        "url": url,
        "proxyType": "datacenter",
        "proxyCountry": "US",
        "js_rendering": True,
        "aiExtractRules": {
            "companyName": {"description": "Company name", "type": "string"},
            "email": {"description": "Email addresses", "type": "string"}
        }
    }
 
# Async function to scrape one site
async def scrape_site(session, url):
    payload = make_payload(url)
    async with session.post(
        "https://api.hasdata.com/scrape/web",
        headers={
            "x-api-key": api_key,
            "Content-Type": "application/json"
        },
        json=payload
    ) as response:
        if response.status == 200:
            data = await response.json()
            ai_resp = data.get("aiResponse", {})
 
            company = ai_resp.get("companyName", "-")
            emails = ai_resp.get("email", "")
 
            return {
                "url": url,
                "company": company,
                "emails": emails
            }
        else:
            return {"url": url, "error": response.status}
 
# Async scrape for multiple sites
async def scrape_all(urls):
    concurrency = get_available_concurrency()
    # limit concurrent connections to API's availableConcurrency
    connector = aiohttp.TCPConnector(limit=concurrency)
    async with aiohttp.ClientSession(connector=connector) as session:
        tasks = [scrape_site(session, url) for url in urls]
        results = await asyncio.gather(*tasks)
    return results
 
# Run scraper
if __name__ == "__main__":
    urls_to_scrape = [
        "https://example.com",
        "https://hasdata.com",
        # add more URLs
    ]
    all_results = asyncio.run(scrape_all(urls_to_scrape))
    for r in all_results:
        print(r)

This lets you run multiple scrapes in parallel without opening a browser each time. For a more detailed guide and advanced techniques, check out our The Complete Guide to Web Scraping with Selenium in Python.