Integrating Google Custom Search Engine with Python.

Data is Everything. Today’s world is all about Data. We search for what we want and we get what we search for. This is a white box working of data, but what if i say we always tell more about ourselves when we search for something. Suppose we search for what we want, and the browser owner has all our data, like what are our needs, at what pace we search for those results, at what rate we search, and many more. This all data helps to describe our characteristics. And we can call it a Black box working of Data.

Data Search

This blog is about searching for the results on one of the Popular search engines “GOOGLE”. Searching data on google is our daily routine, which makes Google data-rich. Searching for small results is fine if we do it manually. But what if we require a huge amount of data in less time?

Google provides one service “Google custom search engine “ (GCSE). It provides the API service to search for customized results.

Working with it manually will be also time-consuming, so we will be integrating the GSCE with python to make it programmatic.

Let's start with the API URL.

How to get an API Key?

1. Go to https://developers.google.com/custom-search/v1/introduction
2. Click on the Get Key button.
3. Create a new Project and Generate an API key.

Using REST to Invoke the API:
The format for the Custom Search JSON API URI is “https://www.googleapis.com/customsearch/v1?[parameters]".
Three query [parameters] are required with each search request:
1. API key — Use the key query parameter to identify your application.
2. Programmable Search Engine ID — Use cx to specify the Programmable Search Engine you want to use to perform this search. The search engine must be created with the Control Panel Note: The Search Engine ID (cx) can be of different format (e.g. 8ac1ab64606d234f1)
3. Search query — Use the q query parameter to specify your search expression.

What if we need to search results related to specific sites? Google provides one more API named “SiteRestrict”, which allows us to add the sites for which we need to search results.

About the Custom Search Site Restricted JSON API
Making a request to Custom Search Site Restricted JSON API is similar to making a request to Custom Search JSON API; however, the URI is different. The format for the Custom Search Site Restricted JSON API is
“https://www.googleapis.com/customsearch/v1/siterestrict?[parameters]”.
The [parameters] are the same as the Custom Search JSON API parameters

Above in parameters we came across Programmable Search Engine ID, to get it, go to https://programmablesearchengine.google.com/controlpanel/all, Click on the project you created, here you will get all the required information of the project credentials and settings. Consider it as the control panel of your project.

Till now we have looked for the API and the credentials required. Now, let's move toward how to integrate it into python.

Get TEJAS BIDWAI’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The simple way is to call the API using the requests library. As follow:

import requests

# get the API KEY here: https://developers.google.com/custom-search/v1/overview
API_KEY = "<INSERT_YOUR_API_KEY_HERE>"
# get your Search Engine ID on your CSE control panel
SEARCH_ENGINE_ID = "<INSERT_YOUR_SEARCH_ENGINE_ID_HERE>"
# the search query you want
query = "python"
# using the first page
page = 1
# constructing the URL
# doc: https://developers.google.com/custom-search/v1/using_rest
# calculating start, (page=2) => (start=11), (page=3) => (start=21)
start = (page - 1) * 10 + 1
url = f"https://www.googleapis.com/customsearch/v1?key={API_KEY}&cx={SEARCH_ENGINE_ID}&q={query}&start={start}"
# make the API request
data = requests.get(url).json()
# get the result items
search_items = data.get("items")
# iterate over 10 results found
for i, search_item in enumerate(search_items, start=1):
    try:
        long_description = search_item["pagemap"]["metatags"][0]["og:description"]
    except KeyError:
        long_description = "N/A"
    # get the page title
    title = search_item.get("title")
    # page snippet
    snippet = search_item.get("snippet")
    # alternatively, you can get the HTML snippet (bolded keywords)
    html_snippet = search_item.get("htmlSnippet")
    # extract the page url
    link = search_item.get("link")
    # print the results
    print("="*10, f"Result #{i+start-1}", "="*10)
    print("Title:", title)
    print("Description:", snippet)
    print("Long description:", long_description)
    print("URL:", link, "\n")

Another library python provides is “google-api-python-client”.

This library provides more flexibility and easy for executing the query.

pip install google-api-python-client

from googleapiclient.discovery import build
import pprint
 
#define key
api_key = "Your API Key"
cse_key = "Your Custom search Engine Key"
 
resource = build("customsearch", 'v1', developerKey=api_key).cse()
result = resource.list(q='simplified python', cx=cse_key).execute()
 
print(result)

Next, we can scrape the information according to our needs.

Custom Search Site Restricted JSON API, we need to add the specific sites in our project's control panel. and the limit for the sites is max 10. We will discuss it in detail in Billing and Limitation part.

It Looks Awesome, Right!!!

But does it cost any amount? We use this approach for getting a huge amount of data, thus google charges a good amount for using their APIs.

Let's discuss Billing and the Limitations of the service.

Billing and Limitations

So for both the API ie. Custom Search JSON API and Custom Search Site Restricted JSON API, Google provides 100 API calls per day for free trials. One API call can fetch information on one page ie. one page contains 10 results. To extract results from the next page we need to change the start parameter and hit the API again. Thus in the free trial, we can scrape about 1000 results.

What if “Dil mange more!!” (need more)?

Google makes it paid service if we need more API calls per day.
For Custom Search JSON API:

It cost $5 per 1000 API calls. And has a limit of 10k requests per day. ie. using Custom search JSON API we can scrape 1 lakh of results per day by paying $50.

But if we need the results from specific sites ie. site restricted, which are less than or equal to 10, then we can use Custom Search Site Restricted JSON API.
Note: We can use Custom Search Site Restricted JSON API only if the count of specific sites is less than or equal to 10.

For Custom Search Site Restricted JSON API:

It cost the same $5 per 1000 API calls. But it does not have a day limit. We can request as many API calls as possible by paying the required amount.

Thank You!!