Scrape OpenSea Data with Python

In this post, we’ll show you how to scrape OpenSea data, collection pages, and individual NFT detail pages using Python and the Crawlbase Crawling API. Let’s get started!

Originally published at Crawlbase.com.

What Data Can You Extract From OpenSea?

When scraping OpenSea, it’s essential to know what data to focus on. The platform has a ton of information about NFTs (Non-Fungible Tokens), and extracting the right data will help you track performance, analyze trends, and make decisions. Here’s what to extract:

OpenSea Scraping with Crawlbase Crawling API

The Crawlbase Crawling API makes OpenSea data scraping easy. Since OpenSea uses JavaScript to load its content, traditional scraping methods will fail. But the Crawlbase API works like a real browser so you can get all the data you need.

Crawlbase Python Library

Crawlbase also has a Python library using which you can easily use Crawlbase products in your projects. You’ll need an access token, which you can get by signing up with Crawlbase.

Here’s an example to send a request to Crawlbase Crawling API:

from crawlbase import CrawlingAPI
# Initialize Crawlbase API with your access token
crawling_api = CrawlingAPI({'token': 'CRAWLBASE_JS_TOKEN'})
def make_crawlbase_request(url):
 response = crawling_api.get(url)
if response['headers']['pc_status'] == '200':
 html_content = response['body'].decode('utf-8')
 return html_content
 else:
 print(f"Failed to fetch the page. Crawlbase status code: {response['headers']['pc_status']}")
 return None

Note: Crawlbase provides two types of tokens: a Normal Token for static sites and a JavaScript (JS) Token for dynamic or browser-rendered content, which is necessary for scraping OpenSea. Crawlbase also offers 1,000 free requests to help you get started, and you can sign up without a credit card. For more details, check the Crawlbase Crawling API documentation.

In the next section, we’ll set up your Python environment for scraping OpenSea effectively.

Setting Up Your Python Environment

Before scraping data from OpenSea, you need to set up your Python environment. This setup will ensure you have all the necessary tools and libraries to make your scraping process smooth and efficient. Here’s how to do it:

Installing Python and Required Libraries

Install Python: Download Python from the official website and follow the installation instructions. Make sure to check “Add Python to PATH” during installation.

Set Up a Virtual Environment (optional but recommended): This keeps your project organized. Run these commands in your terminal:

cd your_project_directory
python -m venv venv
venv\Scripts\activate # Windows
# or
source venv/bin/activate # macOS/Linux

Install Required Libraries: Run the following command to install necessary libraries:

pip install beautifulsoap4 crawlbase pandas

• beautifulsoup4: For parsing and extracting data from HTML.
• crawlbase: For using the Crawlbase Crawling API.
• pandas: For handling and saving data in CSV format.

Choosing an IDE

Select an Integrated Development Environment (IDE) to write your code. Popular options include:

• Visual Studio Code: Free and lightweight, with Python support.
• PyCharm: A feature-rich IDE for Python.
• Jupyter Notebook: Great for interactive coding and data analysis.

Now that your Python environment is set up, you’re ready to start scraping OpenSea collection pages. In the next section, we will inspect the HTML for CSS selectors.

Scraping OpenSea Collection Pages

In this section, we will scrape collection pages from OpenSea. Collection pages show various NFTs grouped under specific categories or themes. To do this efficiently, we will go through the following steps:

Inspecting the HTML for CSS Selectors

Before we write our scraper we need to understand the structure of the HTML on the OpenSea collection pages. Here’s how to find the CSS selectors:

1. Open the Collection Page: Go to the OpenSea website and navigate to any collection page.
2. Inspect the Page: Right-click on the page and select “Inspect” or press Ctrl + Shift + I to open the Developer Tools.

1. Find Relevant Elements: Look for the elements that contain the NFT details. Common data points are:

• Title: In a <span> with data-testid="ItemCardFooter-name".
• Price: Located within a <div> with data-testid="ItemCardPrice", specifically in a nested <span> with data-id="TextBody".
• Image URL: In an <img> tag with the image source in the src attribute.
• Link: The NFT detail page link is in an <a> tag with the class Asset--anchor.

Writing the Collection Page Scraper

Now we have the CSS selectors, we can write our scraper. We will use the Crawlbase Crawling API to handle JavaScript rendering by using its ajax_wait and page_wait parameters. Below is the implementation of the scraper:

from crawlbase import CrawlingAPI
import pandas as pd
# Initialize Crawlbase API with your access token
crawling_api = CrawlingAPI({'token': 'CRAWLBASE_JS_TOKEN'})
def make_crawlbase_request(url):
 options = {
 'ajax_wait': 'true',
 'page_wait': '5000'
 }
response = crawling_api.get(url, options)
if response['headers']['pc_status'] == '200':
 html_content = response['body'].decode('utf-8')
 return html_content
 else:
 print(f"Failed to fetch the page. Crawlbase status code: {response['headers']['pc_status']}")
 return None
def scrape_opensea_collection(html_content):
 soup = BeautifulSoup(html_content, 'html.parser')
 data = []
# Find all NFT items in the collection
 nft_items = soup.select('div.Asset - loaded > article.AssetSearchList - asset')
for item in nft_items:
 title = item.select_one('span[data-testid="ItemCardFooter-name"]').text.strip() if item.select_one('span[data-testid="ItemCardFooter-name"]') else ''
 price = item.select_one('div[data-testid="ItemCardPrice"] span[data-id="TextBody"]').text.strip() if item.select_one('div[data-testid="ItemCardPrice"] span[data-id="TextBody"]') else ''
 image = item.select_one('img')['src'] if item.select_one('img') else ''
 link = item.select_one('a.Asset - anchor')['href'] if item.select_one('a.Asset - anchor') else ''
# Add the extracted data to the list
 data.append({
 'title': title,
 'price': price,
 'image_url': image,
 'link': f"https://opensea.io{link}" # Construct the full URL
 })
return data

Here we initialize the Crawlbase Crawling API and create a function make_crawlbase_request to get the collection page. The function waits for any AJAX requests to complete and waits 5 seconds for the page to fully render before passing the HTML to the scrape_opensea_collection function.

Get Crawlbase’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

In scrape_opensea_collection, we parse the HTML with BeautifulSoup and extract details about each NFT item using the CSS selectors we defined earlier. We get the title, price, image URL and link for each NFT and store this in a list which is returned to the caller.

Handling Pagination in Collection Pages

OpenSea uses scroll-based pagination, so more items load as you scroll down the page. We can use the scroll and scroll_interval parameters for this. This way we don’t need to manage pagination explicitly.

options = {
 'ajax_wait': 'true',
 'scroll': 'true',
 'scroll_interval': '20' # Scroll for 20 seconds
}

This will make the crawler scroll for 20 seconds so we get more items.

Storing Data in a CSV File

After we scrape the data, we can store it in a CSV file. This is a common format and easy to analyze later. Here’s how:

def save_data_to_csv(data, filename='opensea_data.csv'):
 df = pd.DataFrame(data)
 df.to_csv(filename, index=False)
 print(f"Data saved to {filename}")

Complete Code Example

Here’s the complete code that combines all the steps:

from crawlbase import CrawlingAPI
import pandas as pd
from bs4 import BeautifulSoup
# Initialize Crawlbase API with your access token
crawling_api = CrawlingAPI({'token': 'CRAWLBASE_JS_TOKEN'})
def make_crawlbase_request(url):
 options = {
 'ajax_wait': 'true',
 'scroll': 'true',
 'scroll_interval': '20' # Scroll for 20 seconds
 }
response = crawling_api.get(url, options)
if response['headers']['pc_status'] == '200':
 html_content = response['body'].decode('utf-8')
 return html_content
 else:
 print(f"Failed to fetch the page. Crawlbase status code: {response['headers']['pc_status']}")
 return None
def scrape_opensea_collection(html_content):
 soup = BeautifulSoup(html_content, 'html.parser')
 data = []
# Find all NFT items in the collection
 nft_items = soup.select('div.Asset - loaded > article.AssetSearchList - asset')
for item in nft_items:
 title = item.select_one('span[data-testid="ItemCardFooter-name"]').text.strip() if item.select_one('span[data-testid="ItemCardFooter-name"]') else ''
 price = item.select_one('div[data-testid="ItemCardPrice"] span[data-id="TextBody"]').text.strip() if item.select_one('div[data-testid="ItemCardPrice"] span[data-id="TextBody"]') else ''
 image = item.select_one('img')['src'] if item.select_one('img') else ''
 link = item.select_one('a.Asset - anchor')['href'] if item.select_one('a.Asset - anchor') else ''
# Add the extracted data to the list
 data.append({
 'title': title,
 'price': price,
 'image_url': image,
 'link': f"https://opensea.io{link}" # Construct the full URL
 })
return data
def save_data_to_csv(data, filename='opensea_data.csv'):
 df = pd.DataFrame(data)
 df.to_csv(filename, index=False)
 print(f"Data saved to {filename}")
if __name__ == "__main__":
 url = "https://opensea.io/collection/courtyard-nft"
 html_content = make_crawlbase_request(url)
if html_content:
 data = scrape_opensea_collection(html_content) # Extract data from HTML content
 save_data_to_csv(data)

opensea_data.csv Snapshot:

Optimize OpenSea NFT Data Scraping

Scraping OpenSea opens up a whole world of NFTs and market data. Throughout this blog, we covered how to scrape OpenSea using Python and Crawlbase Crawling API. By understanding the layout of the site and using the right tools, you can get valuable insights while keeping ethics in mind.

When you get deeper into your scraping projects, remember to store the data in human-readable formats, like CSV files, to make analysis a breeze. The NFT space is moving fast, and being aware of new trends and technologies will help you get the most out of your data collection efforts. With the right mindset and tools you can find some great insights in the NFT market.

If you have any questions or want to give feedback, our support team can help you with web scraping. Happy scraping!

Frequently Asked Questions

Q. Why should I web scrape OpenSea?

Web scraping is a way to extract data from websites automatically. By scraping OpenSea, you can grab important information about NFTs, such as their prices, descriptions, and images. This data helps you analyze market trends, track specific collections, or compare prices across NFTs. Overall, web scraping provides valuable insights that can enhance your understanding of the NFT marketplace.

Q. Is it legal to scrape data from OpenSea?

Web scraping is a gray area when it comes to legality. Many websites, including OpenSea, allow data collection for personal use but always read the terms of service before you start. Make sure your scraping activities comply with the website’s policies and copyright laws. Ethical scraping means using the data responsibly and not flooding the website’s servers.

Q. What tools do I need to start scraping OpenSea?

To start scraping OpenSea, you’ll need a few tools. Install Python and libraries like BeautifulSoup and Pandas for data parsing and manipulation. You’ll also use Crawlbase Crawling API to handle dynamic content and JavaScript rendering on OpenSea. With these tools in place, you’ll be ready to scrape and analyze NFT data.

Originally published at Crawlbase.com.