Unlocking Precision and Efficiency: A Step-by-Step Guide to Crafting Custom Scrapy Spider Templates for Web Scraping Projects

Welcome to the third blog in the series-Understanding and Implementing Scrapy Spider Templates Powered by Zyte’s AI Scraping Tool.

If you stumble upon this blog first, read the first two blogs of the series-
1. A Brief History of the Evolution of the Web Scraping Tech Stack at Zyte.
2. AI Web Scraping Revolution: Your Guide to Scraping 30 E-Commerce Sites in 30 Minutes

Also, these blogs are supporting articles for the hands-on workshop-How to Build 30 Scrapy Spiders to scrape product data in under 30 minutes. You can watch the recording, here.

In this blog, I will show you how you can customise the open-source Scrapy Spider Templates for your needs.

Table of Contents

1. How does the customisation work?
2. Demo website and defining the customisation on this website
3. How to customise the metadata and parameters.
4. How to customise the product schema- add a new data field to the product schema.

How does the customisation work?

You can use these spiders however you prefer:

• You can use them in Scrapy Cloud UI ( which is no code solution)
• Or you can start a Scrapy project just like in the previous blog.
  You clone the Zyte Spider Template Project and start data extraction from the URL of your choice, Data will be extracted as per this Product Schema.
• Or you can create your template too by Extending a Scrapy project (covered in the web scraping tutorial) if you want additional data fields or to change the crawling behaviour, you can do customizations by creating page objects.

The thumb rule is :
1. If you want to customize metadata, parameters, and crawling logic, you do so by subclassing the spider templates.
2. If you want to override the parsing logic for all or some websites, both for navigation and item detail data, it is done by customizing the page objects.

Let me show you how :)

Demo website and defining the customisation on this website.

In this tutorial, I will use `ao.com`. We will customise the e-commerce template for this website

1. by adding new metadata and a new parameter min_price , with which we will only extract the product with a price higher than the min_price.
2. by adding a new data field epc rating. In this tutorial, we will extract this rating from the product name.

For your Curiosity- An EPC rating typically refers to an Energy Performance Certificate, which is a measure used for buildings rather than individual electric appliances. An Energy Performance Certificate assesses the energy efficiency of a building and gives it a rating from A (most efficient) to G (least efficient). The certificate also contains suggestions on ways to improve the building’s energy performance.

How to customise the metadata and parameters.

1. Create a file named ao.pyunder the spiders folder.

from decimal import Decimal
from scrapy_poet import DummyResponse
from scrapy_spider_metadata import Args
from zyte_spider_templates import EcommerceSpider
from zyte_spider_templates.spiders.ecommerce import EcommerceSpiderParams
from zyte_common_items import Product

class ao_Params(EcommerceSpiderParams):
    min_price: str = "0.00"

class ao_Custom(EcommerceSpider, Args[ao_Params]):
    name = "ao_com"
    metadata = {
        **EcommerceSpider.metadata,
        "title": "ao.com",
        "description": "Template for ao.com",
    }

    def parse_product(self, response: DummyResponse, product: Product):
        if Decimal(product.price) >= Decimal(self.args.min_price):
            yield product

The spider is created using the EcommerceSpider class from the zyte_spider_templates package and the Product class from the zyte_common_items package. The scrapy-spider-metadata is used for spider and argument management. And, the scrapy-poet is used to support web-poet page objects.

The code defines two classes:

ao_Params(AoComSpiderParams): This class inherits from the EcommerceSpiderParams class and defines the minimum price for the products to be extracted. The default value is set to 0.00.

ao_Custom(AoComSpider): This class inherits from the EcommerceSpider and Args classes and is the primary spider class. The spider is named `ao_com.py` and provides metadata information about the spider.

The key functionality of the spider is implemented in the parse_product method, which accepts a response object (a webpage) and a product object (the extracted product data). The method checks if the product’s price is greater than or equal to the minimum price defined in the ao_Params class. If the condition is satisfied, the spider yields the product object for further processing.

Run the spider with this command in your terminal.

scrapy crawl ao_com -a url="https://ao.com/l/freezers/1/32/" -a min_price="<enter the min price>" -o output.jsonl

How to customise the product schema- add a new data field to the product schema.

If you have to add this new field to the data schema or override the data schema, you can do so by implementing Page Objects.

What are Page Objects?
Page Objects were introduced by Martin Fowler

When you write tests against a web page, you need to refer to elements within that web page to click links and determine what’s displayed. However, if you write tests that manipulate the HTML elements directly your tests will be brittle to changes in the UI. A page object wraps an HTML page, or fragment, with an application-specific API, allowing you to manipulate page elements without digging around in the HTML.

Learn more about Page Objects for Web Data Extraction by joining this discussion on Extract Data Discord Community.

1. In `items.py`, paste the following code-
   This code defines a new class called CustomProduct using the @attrs.define decorator from the attrs library. The CustomProduct the class inherits from the Product class from the zyte_common_items package. The attrs library provides a simple and convenient way to define classes. The CustomProduct class has a single attribute epc (External Product Code) of type str. The Optional type hint indicates that the epc attribute can be None (i.e., it is optional).

from typing import Optional

import attrs
from zyte_common_items import Product


@attrs.define
class CustomProduct(Product):
    epc: Optional[str]

2. Under the pages folder create ao.py

pages/ao.py

This code imports the re module from the standard package, the AutoProductPage class from the zyte_common_items package, the CustomProduct class from the local items module, and the handle_urls, Returns, and field functions from the web_poet package.

The handle_urls decorator is used to define the URL(s) that the crawling should be performed on, in this case, the ao.com domain.

The AoComPage class inherits from AutoProductPage and Returns[CustomProduct], indicating that it is a web page that returns a CustomProduct an object defined in the items.py`

The @field decorator is used to define a field on the AoComPage class, which is used to extract data from the web page. In this case, the epc field is defined to extract the epc rating from the product name-name attribute using the re module.

The AoComPage class does not contain a parse_product method, as it is not required in this case. By inheriting from AutoProductPage, self.product gets automatically filled with a Product instance from Zyte API, and every field not overridden uses that data. In this case, the CustomProduct class is used instead of the Product class, which is specified by the Returns[CustomProduct] type hint. The combination of handle_urls and Returns[CustomProduct] is what makes that, when defining product: CustomProduct in our callback, scrapy-poet knows to use AoComPage to build the requested instance of CustomProduct.

Overall, this code sets up a Scrapy spider that crawls the ao.com domain and extracts information from web pages, returning CustomProduct objects for each product found. The epc the field is defined to extract the epc rating from the product name-name attribute using the re regular expression module.

import re
from web_poet import Returns, field, handle_urls
from zyte_common_items import AutoProductPage

from ..items import CustomProduct

@handle_urls("ao.com")
class AoComPage(AutoProductPage, Returns[CustomProduct]):
    @field
    async def epc(self):
        match = re.search(r"\b[A-F]\+?\+?(?=[\s-]*[Rr]at(ed|ing)\b)", self.name)
        return match[0] if match else None

3. Under the `spiders folder` create ao.py

spiders/ao.py

from decimal import Decimal
from scrapy_poet import DummyResponse
from scrapy_spider_metadata import Args
from zyte_spider_templates import EcommerceSpider
from zyte_spider_templates.spiders.ecommerce import EcommerceSpiderParams
from ..items import CustomProduct

class ao_Params(EcommerceSpiderParams):
    min_price: str = "0.00"

class ao_Custom(EcommerceSpider, Args[ao_Params]):
    name = "ao_com"
    metadata = {
        **EcommerceSpider.metadata,
        "title": "ao.com",
        "description": "Template for ao.com",
    }

    def parse_product(self, response: DummyResponse, product: CustomProduct):
        if Decimal(product.price) >= Decimal(self.args.min_price):
            yield product

Run the spider with this command in your terminal.

scrapy crawl ao_com -a url="https://ao.com/l/freezers/1/32/" -a min_price="<enter the min price>" -o output.jsonl

The output will have an additional field ` “epc”: “F”`

{"aggregateRating": {"bestRating": 5.0, "ratingValue": 5.0, "reviewCount": 3}, "availability": "InStock", "brand": {"name": "CDA"}, "breadcrumbs": [{"name": "Home", "url": "https://ao.com/"}, {"name": "Freezers", "url": "https://ao.com/l/freezers/1/32/"}, {"name": "Under Counter Freezer", "url": "https://ao.com/l/under_counter_freezer/1/32-35/"}, {"name": "CDA FW284"}], "currency": "GBP", "currencyRaw": "\u00a3", "description": "Delivery From Tue 26th Mar \u25b3From Tue 26th March \u25b3From Tue 26th March \u25b3From Tue 26th March \u25b3 Check delivery dates and prices for your area\n\nServices Connection (\u00a3115), Remove & Recycle (\u00a325), Unpack (\u00a35)\n\nReturns Period 30-day returns", "descriptionHtml": "<article>\n\n<p>Delivery\n                    \nFrom Tue 26th Mar \u25b3From Tue 26th March \u25b3From Tue 26th March \u25b3From Tue 26th March \u25b3                    \n                        Check delivery dates and prices for your area</p>\n\n<p>Services\n                    \n                        Connection (\u00a3115), Remove &amp; Recycle (\u00a325), Unpack (\u00a35)</p>\n\n<p>Returns Period\n              \n                    \n                        30-day returns</p>\n\n</article>", "mainImage": {"url": "https://assets.ao.com/brand-logos/en-GB/cda.png"}, "metadata": {"dateDownloaded": "2024-03-21T09:11:03Z", "probability": 0.9953565001487732}, "name": "CDA FW284 Integrated Under Counter Freezer with Sliding Door Fixing Kit - F Rated", "price": "379.0", "regularPrice": "419.0", "sku": "CDA FW284", "url": "https://ao.com/product/fw284-cda-under-counter-freezer-white-78207-35.aspx", "epc": "F"}

Next

Join the Extract Data Community- We’ve established a vibrant Discord community for web scraping enthusiasts like yourself, dedicated to sharing insights, learning new technologies, and advancing in web scraping.

Ask your queries, and doubts, and share your projects on Discord. Looking forward to seeing you there :)

This article introduces web scraping developers to an innovative and efficient system for constructing large-scale projects with minimal code. It offers a faster, more cost-effective, and easily scalable alternative to other AI tools and LLMs like ChatGPT.

There are multiple AI tools and plenty of articles promising to automate web scraping using AI tools and LLMs like ChatGPT but can you fully automate large-scale web scraping? Here’s a blunt truth!

Currently, you cannot completely automate large-scale web scraping projects with AI unless you are okay
1. with sacrificing data quality.
2. to give up the entire control of your web data extraction projects to black-box AI solutions, which sometimes hallucinate and when they do, you can’t do anything about them.

You should do neither! And the good news is: you don’t have to with Zyte’s AI Tech Stack for Web Scraping!

In this article, I will demonstrate a new way of approaching {any}-scale web scraping projects with the Spider Template System using Zyte’s AI Tech Stack without losing control, without compromising the data quality or without the overhead maintenance of managing multiple tools. This system is

• Self-healing and resilient towards the website layout changes and antiban upgrades.
• effortlessly scales to accommodate hundreds or thousands of websites.

The objective of this blog is to demonstrate Zyte’s AI Scraping Tool to scrape product data from various e-commerce websites with a brief introduction to the Scrapy Spider Template System. But if you want to understand this new system in depth, I recommend reading this blog- A Brief History of the Evolution of the Web Scraping Tech Stack at Zyte.

In my view, the best way to proceed with this blog would be to go to the second heading-How to set up a product data extraction project from 30 e-commerce websites using Zyte’s AI tech stack and Try it :)

Also, this blog is a supporting article for the hands-on workshop-How to Build 30 Scrapy Spiders to scrape product data in under 30 minutes. If you prefer learning through video, You can watch the recording, here.

Table of contents

1. Introduction to Zyte’s AI tech stack and Scrapy spider template system for web scraping.
2. Setting up a product data extraction project from 30 e-commerce websites using Zyte’s AI tech stack.
3. Surprise!
4. Conclusion.
5. Next Steps- Tailoring the Solution: Customization Possibilities and Implementation Guide For Spider Templates.

Introduction to Zyte’s AI tech stack and scrapy spider template system for web scraping.

Zyte’s AI tech stack combines Scrapy, Zyte API with AI capabilities, and Scrapy Cloud into a predefined Scrapy Spider Template(currently only available for product data). This template simplifies and automates the traditional complexities of a web scraping project.

Smart Spider Templates equip you to scrape websites rapidly and build self-healing spiders that adapt to changes without writing the crawling logic or Xpath/CSS selectors. These templates have no limitations and allow for extensive customization.

1. Scrapy Spider Templates
   zyte-spider-templates: This library contains Scrapy spider templates or automatic crawlers. They can be used out of the box with the Zyte features such as Zyte API or modified to be used standalone. The AI-powered Spider Templates automate and streamline the web scraping process. They enable quick and easy setup by automatically recognizing navigation patterns and handling pagination. Whatever you can achieve with Scrapy, you can accomplish within a template, including altering crawling patterns, filtering results, adding arguments, processing data, and exporting it.
   
   zyte-spider-templates-project : This is a starting template for a Scrapy project, with built-in integration with Zyte technologies (scrapy-zyte-api, zyte-spider-templates). We will use this template in the project.
2. Self-Healing AI capabilities of the Spider template system :
   Crawling
   Zyte API’s AI capabilities eliminate the burden of writing and maintaining custom crawling code for each website, enabling scalability. The AI is trained to recognize both common and uncommon navigation patterns, relieving developers from the task of figuring out site navigation, URL structures, and pagination. It works out of the box and can be easily customized using Python Scrapy code if desired. Zyte’s Automated ban management is another benefit of AI crawling. The system automatically avoids bans by utilizing the minimum infrastructure required.
   Extraction
   AI also solves the challenge of parsing and extracting data by learning to read pages like a human in real-time. It autonomously finds the required data without explicit instructions, making the process fast and immune to site changes. By providing an e-commerce page URL, the AI delivers product details in a legally compliant schema, which can be further extended with manual rules if desired.

So, how does the template system work? Let’s understand through this small project.

Prerequisites

1. Sign up for Zyte’s AI Scraping Tool, here
2. Take a note of Zyte_API_Key.
3. Understanding of Scrapy

Setting up a product data extraction project from 30 e-commerce websites using Zyte’s AI Tech Stack.

You can also download the entire code here. {insert a github}

But here’s a step-by-step guide to running the e-commerce spider template

1. Open the terminal and clone the zyte-spider-templates-project :

git clone https://github.com/zytedata/zyte-spider-templates-project

2. Rename the zyte_spider_templates_project folder to a valid Python module name (e.g. use _ instead of - or spaces).

3. cd <your-project-name>

4. update these files with your project name.

• scrapy.cfg
• settings.py
  BOT_NAME, SPIDER_MODULE, NEWSPIDER_MODULE, SCRAPY_POET_DISCOVER

5. Add in settings.py `ZYTE_API_KEY`

6. Remove or replace the LICENSE and README.rst files.

7. Delete .git, and start a fresh Git repository:

rm -rf .git
git init
git add -A
git status
git commit -m "Initial commit"

8. Create a Python virtual environment and install requirements.txt into it:

python3 -m venv venv
. venv/bin/activate
pip install -r requirements.txt

9. You can now run the e-commerce spider with a URL of your choice.

scrapy crawl ecommerce -a url="<enter the url >"

10. Time to Set up 30 e-commerce spiders.

Create two files- domains.txt and run30sites.py and paste the following code into the respective files.
The code in the run30sites.py reads URLs from the domain.txt(containing 30 different e-commerce domain URLs) and runs a scrapy e-commerce template spider.

domain.txt

wayfair.ca
kleinanzeigen.de
lowes.com
tesco.com
shein.com
allegro.pl
lowes.com
adidas.com
otto.de
idealo.de
homedepot.com
xxxlutz.at
kroger.com
ubereats.com
direct.leaseplan.co.uk
target.com
farfetch.com
shein.co.uk
leroymerlin.fr
tesco.ie
xxxlutz.ch
wayfair.com
118118.com
kick.com
justeat.com
sainsburys.co.uk
dba.dk
shopee.sg
rappi.com.br
myvue.com
ubereats.com
doordash.com
cars.com
finde-offen.de
tori.fi
grubhub.com

run30sites.py

import subprocess
import os

def main():
    processes = []
    os.makedirs("output",exist_ok=False)
    with open("domains.txt") as input_file:
        for domain in input_file:
            domain = domain.strip()
            processes.append(
                subprocess.Popen(
                    [
                        "scrapy", 
                        "crawl",
                        "ecommerce",
                        "-a",
                        "max_requests=50",
                        "-a",
                        f"url=https://{domain}/",
                        "-o",
                        f"output/{domain}.jsonl",
                    ]
                )
            )
    for process in processes:
        process.wait()

if __name__ == "__main__":
    main()

How to scale this project to hundreds or thousands of e-commerce websites.
With the provided code, you can effortlessly set up an impressive number of spiders — whether it’s 30, 100, or even 1000 — in just 10 minutes(Just add the URLs in `domain.txt`). The speed, quality and efficiency of this process are truly remarkable, enabling you to enhance your productivity in web scraping tasks.

This code empowers developers to save valuable time and swiftly deploy spiders for extracting product data from various sources. The streamlined spider creation process and faster development cycles make it an exhilarating tool to work with.

Conclusion

Zyte’s mission is to connect data enthusiasts with quality reliable web data at any scale without coding hassles, getting banned or breaking spiders.

With this Spider Template System supercharged with Zyte’s AI Stack, Zyte is one more step closer to fulfilling its mission. Of course, we won’t stop inventing and optimising, so stay tuned :P

Get ready to revolutionize your web scraping endeavours with Zyte’s AI Scraping Tool, and let us know what you think about this new approach to web scraping projects or reach out to us if you need coding help on our discord- Join and connect with 1000s of industry leaders on Extract Data Community!

Happy Scraping and Love to all :)
Neha, Developer Advocate, Zyte

Read the Next Blog- Tailoring the Solution: Customization Possibilities and Implementation Guide For Spider Templates.

If you truly want to understand the present or yourself, you must begin in the past. You see, history is not simply the study of the past. It is an explanation of the present — Paul Hunham from the movie- The Holdovers.

Let me tell you the story of a developer named Lucas. Lucas has approximately ten years of experience in the web scraping industry. He broadly went through three broad phases in his entire web scraping experience namely- the manual phase, the automation phase, and the optimisation phase.

If I divide this evolution into three broad phases-:

1. The Manual Phase using Scrapy
   In this phase, Lucas diligently wrote every line of code for web scraping projects. This included crafting the crawling logic, implementing rotating proxy solutions, and utilizing browser automation tools like Splash, Puppeteer, and Selenium. Lucas also had to invest time in researching and manually upgrading the solution to tackle advanced antiban software.
   However, this manual approach posed numerous challenges. Writing the spiders alone consumed several days of work, not to mention the additional time required to write the crawling code, selectors, and unblock bans. Moreover, the constant need to adjust both selectors and bans whenever the website layout or anti-bot measures changed added to the complexity. The arduous process of creating site-specific code and integrating multiple tools not only made scaling difficult but also increased costs and maintenance overhead.

1. The Automation Phase using Scrapy and Zyte API
   Lucas experienced a significant reduction in workload thanks to the integration of Zyte API into the web scraping system. With Zyte API automatically handling bans, proxy management, and supporting browser automation through its actions capability, Lucas was able to focus primarily on writing the crawling and extraction logic using scrapy spiders. This eliminated the need for configuring the spider to avoid bans or managing rotating proxies, resulting in streamlined operations.
   Furthermore, integrating Zyte API into the system eliminated the need for multiple tools, reducing maintenance overhead and improving monitoring capabilities. However, despite these advancements, challenges remained.
   Writing spiders manually still posed a hurdle, and hence, scaling the system proved to be difficult. While the system was equipped to self-heal in terms of antiban software upgrades, it lacked the adaptability to seamlessly handle website layout changes.

• The Automation Phase uses Scrapy and Zyte API with AI-powered extraction.
  With the integration of patented machine learning into Zyte API, Lucas could now enjoy a range of benefits. The system offers supervised machine learning that surpasses the performance of large language models in terms of speed, accuracy, and cost-effectiveness. The high accuracy of the system is further enhanced by the ‘quick fixes’ process, where the support team applies specialized hints for optimizing the model on a per-site basis.
  Moreover, the economical nature of running the system on every page at runtime makes it self-healing when website layouts change, which means no more writing selectors manually. Hence, effectively addressing the maintenance challenges.
  Additionally, the system provides compliance confidence out of the box by employing standard schemas that automatically exclude sensitive data such as personally identifiable information (PII) and copyrighted content. These features ensure that Lucas can extract data quickly, accurately, and in a compliant manner.

The Optimisation Phase using the Scrapy Spider Template System composed of Zyte AI Stack-Scrapy + Zyte API (with AI-powered extraction capabilities)

During the Optimization Phase, Lucas leverages the Scrapy Spider Template System powered by the Zyte AI Stack, which combines Scrapy and Zyte API with AI-powered extraction capabilities. This integrated template system revolutionizes the web scraping process for Lucas.

With the template system, Lucas no longer needs to write spiders, and selectors, or manually configure them for bans, site upgrades, or proxy management. Instead, Lucas runs the template using the command
"Scrapy crawl ecommerce url='enter your URL'"
to effortlessly extract product data. The template eliminates the need for manual coding and streamlines the entire data extraction process.

Next Steps

1. Sign up for AI For Web Scraping- Try it and Let us know what you think on Extract Data Discord.
2. Read the Next Blog in the Series- AI web scraping-web scraping product data from 30 e-commerce websites in under 30 minutes
3. Join the Extract Data Community- We’ve established a vibrant Discord community for web scraping enthusiasts like yourself, dedicated to sharing insights, learning new technologies, and advancing in web scraping. I am looking forward to seeing you there :)

Zyte SmartProxy Selenium library is a client library built on top of Selenium — an open-source framework for web automation across Chromium, Firefox, and WebKit, with a single API, written to work seamlessly with Zyte Smart Proxy Manager.

With this library, you will be able to make the best of the headless browser capabilities of Selenium and manage bans by unlocking the powerful proxy management tool — Zyte Smart Proxy Manager in your web scraping projects.

Read the detailed Blog on how to integrate selenium with Zyte Smart Proxy Manager

I will be hosting a webinar on the 22nd of June, to demonstrate the true power of this new integration. This webinar will be a good opportunity for you to interact with our web scraping experts and clarify your doubts on the fly.

Register here!

#selenium #webscraping #webautomation #headlessbrowser #dataextraction #zyte

Read this post and more on my Typeshare Social Blog

Zyte SmartProxy Puppeteer library is a client library built on top of Puppeteer — a high-level API to control headless chrome, written to work seamlessly with Zyte Smart Proxy Manager.

With this library, you will be able to make the best of the headless browser capabilities of Puppeteer and manage bans by unlocking the powerful proxy management tool — Zyte Smart Proxy Manager in your web scraping projects.

Read the detailed blog on How to use Playwright with Zyte Smart Proxy Manager.

Also, will be hosting a webinar on the 22nd of June, to demonstrate the true power of this new integration. This webinar will be a good opportunity for you to interact with our web scraping experts and clarify your doubts on the fly.

Don’t forget to Register here!

#puppeteer #webscraping #webautomation #headlessbrowser #dataextraction #zyte

Read this post and more on my Typeshare Social Blog

Zyte SmartProxy Playwright library is a client library built on top of Playwright — an open-source framework for web automation across Chromium, Firefox, and WebKit, with a single API, written to work seamlessly with Zyte Smart Proxy Manager.

With this library, you will be able to make the best of the headless browser capabilities of Playwright and manage bans by unlocking the powerful proxy management tool — Zyte Smart Proxy Manager in your web scraping projects.

Read the detailed blog on How to use Playwright with Zyte Smart Proxy Manager.

Also, will be hosting a webinar on the 22nd of June, to demonstrate the true power of this new integration. This webinar will be a good opportunity for you to interact with our web scraping experts and clarify your doubts on the fly.

Don’t forget to Register here!

#playwright #webscraping #webautomation #headlessbrowser #dataextraction #zyte

Read this post and more on my Typeshare Social Blog

Just like your fingerprints uniquely identify you, your Digital Fingerprint- A collection of various parameters attribute to your unique identity on the web. This information is then used by the websites to know your preferences and interests on the web.

A rule of thumb to master the art of crawling is to avoid being identified as a bot or standing out from the crowd, which implies train your spiders to behave like a human. If website identify you as a bot, they might block you. How do you make sure you leave a real-user like fingerprint and not a spider’s?

To answer, let’s dive deep into browser fingerprinting.

The state of the art websites interact with users and render graphics through javascript. To render javascript elements, the websites call the browser APIs. While doing so they inject fingerprinting scripts which look just like any other script running on a website. These scripts collect the attributes — device specifications, OS, browser settings and plug-ins, user agents, audio and video capabilities, timezone, language, screen resolution and more — that can be compiled into a browser fingerprint.

To know more about what information does a website gather through browser fingerprinting or to test Your browser’s fingerprinting?

Check this website, https://amiunique.org/

Here are some of the advanced techniques for the browser fingerprinting

• Canvas fingerprinting.
• WebGL fingerprinting and rendering fingerprinting.
• Device fingerprinting.
• Audio fingerprinting.

So, how to look like a human while crawling?

While we cannot delete browser fingerprint or block javascript as the website will break. What we can do is confuse the fingerprinting scripts. The trick is to blend-in with the crowd by masking and modifying your identity.

Masking and modifying refer to manipulating your unique attributes and. changing them periodically so that your fingerprint is constantly changing and you can’t be reliably identified. You can do so by

1. rotating proxies.
2. rotating user-agents.

Read more about rotating proxies on this blog and user-agent on this Twitter thread.

3. Invest in a Tool that does it for you- Introducing Zyte API

Zyte API is an all-in-one solution for reliable web data extraction, providing the necessary tools to extract data from any website with the leanest setup and at the lowest cost. It uses state-of-the-art techniques to counter virtually every anti-scraping method in use, making it a genuine breakthrough for data scientists and professionals.

Zyte API is a powerful API that consolidates virtually every known web scraping technology and technique into a simple, yet powerful API to collect web data at scale. It automatically adapts to any site changes, ensuring users are never banned and always get their data even if the target site changes its code. The solution saves money and reduces valuable developer hours on maintenance and tasks better suited to algorithms and specialist API developers.

How do I start?

1. Sign up here and get $5 in free credits.
2. Use your web browser to send your first request to a website so you check what the cost is.
3. Follow the Zyte API tutorial to learn how to get more data.

Check out Zyte API tutorial and get started in minutes.

#antibot #anitban #webscraping #webcrawling #digitalfingerprinting #browserfingerprinting

Read this post and more on my Typeshare Social Blog

Are you experiencing 403 Forbidden status code while crawling a website? Well, It indicates that the server understands the request but refuses to authorise it. Here’s one of the reasons you might be experiencing it.

Introducing request headers…

When it comes to food, everyone has a different palate. That’s the reason why people pass on additional information about their preferences when ordering online. Request Headers are no different when you request a website from your browser. They provide added information about the website requested by the user so the server can tailor the response accordingly. To view the request header in your browser, follow this tutorial.

The following fields are consistently used by most major browsers when initiating any connection:

• Host request header specifies the host and port number of the server to which the request is being sent.
• Connection controls whether the network connection stays open after the current transaction finishes. For example, Connection: keep-alive, the value keep-alive indicates that the client would like to keep the connection open for the subsequent requests.
• Accept is used to specify which media types, the client is able to understand. For example, Accept: text/html, Accept: image/*
• Accept-Encoding indicates the content encoding (usually a compression algorithm) that the client can understand. For example, Accept-Encoding: gzip, Accept-Encoding: compress
• Accept-Language indicates the natural language and locale that the client prefers. For example: Accept-Language: en-US
• User-Agent represents you by telling the server about your application, operating system, vendor, and/or version. For example:
• user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.88 Safari/537.36
• Without user agents, a web server would serve the same content to all website visitors. However, with the information contained in a user agent, a web server can custom-tailor the content for every visitor.

But What is the Role of Request Headers in Web Scraping?

Typically, the only field that matters in web scraping is User-Agents. The sole purpose of a scraping project is to crawl a website politely without being identified as a bot. To ensure you are a human, the website inspects the user-agents field.

Scraping Bots take the default value for the user-agent set by the python library making the request. For example, urllib the library will set UA as Python-urllib/3.4. With the default UA value, the website identifies you as a bot and blocks you. The trick is to set this value human-like.

So, How To Behave Human-like With User-Agents?

1. Use quality user-agents
   Stick to user agents that match the browser you’re using for scraping to make them match the default behaviour of this browser. One of the many ways to set the user-agent is using Request Library. It allows complete customisation of the HTTP headers.
2. Rotate user-agents
   Rotate User-agents to make the server believe the requests are from different users. If you use the same user-agent for multiple requests, you will be blocked. You can perform user-agent rotation in Python and Selenium. If using scrapy, you can use `scrapy-useragents` middleware.

Pro-tip

If you ever encounter an extremely suspicious website, populating one of the commonly used but rarely checked headers such as Accept-Language might be the key to convincing it you’re a human.

To conclude, one of the many things on the checklist for seamless crawling is to make sure you are choosing reputed user-agents and keep rotating them. If you don’t want to do it yourself, look for an all in one solution like Zyte (formerly Scrapinghub)’s Zyte API that automatically adjust all these parameters according to the website.

How do I start?

1. Sign up here and get $5 in free credits.
2. Use your web browser to send your first request to a website so you check what the cost is.
3. Follow the Zyte API tutorial to learn how to get more data.

Check out Zyte API tutorial and get started in minutes.

#webscraping #antiban #403 #useragents #scrapyuseragents #scrapy #webcrawling

Consider integrating a headless browser with the scraping logic…headless? Wait, What? but why?

Let me explain…

Websites have moved on from static HTML and CSS barebone structures to interactive and responsive Javascript structures that are rendered by browsers to dynamically inject HTML and CSS into websites when demanded.

Regular HTML scrapers do not include functionality to render the page full of ajax and javascript elements like a real user in an automated web scraping. Introducing Headless Browser…

A headless browser is a browser with no Graphical User Interface(GUI).

Headless browsers exhibit human-like behavior in an automated task to scrape the required data embedded in javascript elements without the additional overhead of loading and processing visual elements of a website. They are controlled programmatically using command-line tools.

There are many web scraping tools that can be used for headless browsing. The most popular ones are Puppeteer, Selenium, and Playwright. Let’s talk about Puppeteer…

Puppeteer is a Node.js library that supports controlling headless Chrome and Firefox. It allows downloading data, using proxies, and more. The library is maintained by the Chrome DevTools team with an active open-source community which is why Puppeteer has become one of the most popular options. For this reason, @zytedata has been working on the Zyte SmartProxy Puppeteer library to work in conjunction with the Smart Proxy Manager to crawl javascript-heavy websites with ease.

Zyte SmartProxy Puppeteer library is a client library built on top of Puppeteer — a high-level API to control headless chrome, written to work seamlessly with Smart Proxy Manager.

With Zyte SmartProxy Puppeteer library, there is

• No need to manage an additional headless-proxy tool running in the background.
• No need to run a separate piece of software to connect Puppeteer with Smart Proxy Manager

Resources:

The library is open-source and available here:

GitHub - zytedata/zyte-smartproxy-puppeteer: A wrapper over Puppeteer to provide Zyte Smart Proxy Manager specific functionalities.

To use Smart Proxy Manager, Sign up for a 14-day free trial.

Automated Rotating Proxy Solution | Smart Proxy Management | Zyte

and follow this hands-on tutorial to integrate Smart proxy Manager with puppeteer

Using Smart Proxy Manager with Puppeteer - Zyte documentation

Read this post and more on my Typeshare Social Blog

Check if it can …

• Detect different types of bans so that you can troubleshoot and fix the underlying problem — i.e. captchas, redirects, blocks, cloaking, etc.
• Retry errors and timeouts with different proxies.
• Manage Request headers by rotating user agents.
• Manage sessions and Cookies.
• configure geographical targeting for proxies.
• Automatically randomise delays and change request throttling based on the real-time feedback.
• works well with the chosen headless browser libraries to scrape dynamic websites seamlessly.

Introducing Zyte API that does all of the above so you can focus on extracting quality data at large scale .

How to start?

1. Sign up here and get $5 in free credits.
2. Use your web browser to send your first request to a website so you check what the cost is.
3. Follow the Zyte API tutorial to learn how to get more data.

#developers #webscraping #dataextraction #ZyteAPI #antiban #zyte