Scraping Google Search Results with Python

Google’s search pages are not like static websites. They’re dynamic, protected with anti-bot systems, and selectors change often. A simple requests + BeautifulSoup script won’t work here.

Get HasData’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Here are three ways to scrape organic results, from simplest to more reliable.

Requests + BeautifulSoup (won’t work on Google)

This is the “classic” approach. You send an HTTP request, parse the HTML, and try to extract results. It looks simple, but Google’s dynamic content usually breaks it. Still, it’s useful to see how scraping works in theory.

import requests
from bs4 import BeautifulSoup
import pandas as pd
import json
import urllib.parse


query = "web scraping"
url = f"https://www.google.com/search?q={urllib.parse.quote_plus(query)}"
headers = {"User-Agent": "Mozilla/5.0"}
resp = requests.get(url, headers=headers)
soup = BeautifulSoup(resp.text, "html.parser")


results = []
for block in soup.select("div.MjjYud"):
    title = block.select_one("h3")
    link = block.select_one("a")
    snippet = block.select_one("div.VwiC3b")
    if title and link and snippet:
        results.append({"Title": title.text, "Link": link['href'], "Snippet": snippet.text})


pd.DataFrame(results).to_csv("organic_results_bs4.csv", index=False)
with open("organic_results_bs4.json", "w", encoding="utf-8") as f:
    json.dump(results, f, ensure_ascii=False, indent=2)
print(f"Saved {len(results)} organic results (BS4).")

Most of the time, this returns nothing because Google loads content with JavaScript.

Scrape Google SERP with Selenium

Selenium controls a real browser. That means it can load all the dynamic content and gives you access to the elements like a normal user would see them. This method works reliably but is slower than requests.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
import time
import pandas as pd
import json
import urllib.parse


query = "web scraping"


options = Options()
driver = webdriver.Chrome(options=options)
results = []


for page in range(1):  # change to more pages if needed
    driver.get(f"https://www.google.com/search?q={urllib.parse.quote_plus(query)}&start={page*10}")
    time.sleep(3)
    for block in driver.find_elements(By.CSS_SELECTOR, "div.MjjYud"):
        try:
            title = block.find_element(By.CSS_SELECTOR, "h3").text
            link = block.find_element(By.CSS_SELECTOR, "a").get_attribute("href")
            snippet = block.find_element(By.CSS_SELECTOR, "div.VwiC3b").text
            results.append({"Title": title, "Link": link, "Snippet": snippet})
        except:
            continue


driver.quit()
pd.DataFrame(results).to_csv("organic_results_selenium.csv", index=False)
with open("organic_results_selenium.json", "w", encoding="utf-8") as f:
    json.dump(results, f, ensure_ascii=False, indent=2)
print(f"Saved {len(results)} organic results (Selenium).")

SeleniumBase + UC mode (undetected Chrome)

SeleniumBase with undetected Chrome mode is the “next level.” It reduces the chance of being flagged as a bot. Everything else is the same as regular Selenium: open page, find result blocks, extract title/link/snippet.

from seleniumbase import BaseCase
import pandas as pd
import json
import urllib.parse
import time


query = "web scraping"


# Open Google search page
BaseCase().open(f"https://www.google.com/search?q={urllib.parse.quote_plus(query)}")
time.sleep(2)


# Find all organic result blocks
results = []
blocks = BaseCase().find_elements("div.MjjYud")
for block in blocks:
    title = block.find_element_by_css_selector("h3").text
    link = block.find_element_by_css_selector("a").get_attribute("href")
    snippet = block.find_element_by_css_selector("div.VwiC3b").text
    results.append({"Title": title, "Link": link, "Snippet": snippet})


# Save to CSV and JSON
pd.DataFrame(results).to_csv("organic_results_sb.csv", index=False)
with open("organic_results_sb.json", "w", encoding="utf-8") as f:
    json.dump(results, f, ensure_ascii=False, indent=2)


print(f"Saved {len(results)} organic results (SeleniumBase).")

Conclusion

For serious scraping, use Selenium or a Google SERP API. If you just want a quick look or a demo, Requests with BeautifulSoup is fine. For a complete guide on how to scrape Google SERP, check out our full blog post.