Google Map Scraper: Identify key buildings within a region

Introduction

When a person travels to a foreign country, the first sensible thing to do was reconnaissance as they needed to know what places they planning on visit in advance or keep note of important structures such as embassies if things went south during their travels.

While simply searching for the location on Google will suffice for some, others would collect information in advance and plan ahead of time. However, manually inputting every important building within a town or even a city would be time-consuming.

Research

Google Map Scraper by gosom makes use of another GitHub repository called scrapemate. Scrapemate is a web crawling and scraping framework written in Golang. It is designed to be simple and easy to use, yet powerful enough to handle complex scraping tasks.

Using that as a base for the tool, Google Map Scraper scrapes data directly from Google Maps. The tool itself not only extracts geo-location data such as latitude & longitude, but it also extracts details related to the place such as the name of the building, reviews and contact information.

Installation & Usage

The tool is usable via Docker or manually cloning the git repository. To install via GitHub repository cloning:

#clone Github repository
git clone https://github.com/gosom/google-maps-scraper.git

#Change directory & initial repository build
cd google-maps-scraper
go mod download
go build

#Initiate Google Map Scraper
./google-maps-scraper -input example-queries.txt -results restaurants-in-cyprus.csv -exit-on-inactivity 3m

Based on the code above, after cloning the GitHub repository the user must change to the “google-map-scraper” directory. From there, the user has to download the dependencies for the Go project before compiling it.

The “go build” command allow the terminal to read the source files in the current directory and its subdirectories, compiles them, and produces an executable binary. The output binary will have the same name as the directory if no specific output name is provided.

Get CyberRaya’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

From there, the user must make a list of queries for the project to find before initiating the tool. The query itself uses a .txt format for ease of search.

Example of a simple search query for Google Map Scraper.

Once the query is set, the user must specify the following:

• -input <name>.txt : Selects which .txt file and data required for scraping
• -results <name>.csv : Where to save the scraped data into. The tool will create a CSV file and save the data in there.
• -exit-on-inactivity: Automatically exits the project after a certain period of time.

Once everything is set, the project will start and it will automatically scrape any data based on the .txt file. First it will show the total amount of locations found within the search radius, followed by individually scraping every available data based on the Google Maps information field.

Google Map Scraper progress.

Once the tool finishes scraping, you can review the scraped data within the newly created CSV file.

Scraped Google Map Data

From here, the user can proceed to do data cleaning to remove unnecessary data and data validation to prepare for visualization using geo-vizualization tools such as ArcGIS or QGIS.

However, the most necessary data from the scraped data needed for visualization are:

1. Latitude & Longitude
2. Category
3. Name of location
4. Contact Information

Closure

Overall, Google Map Scraper is very useful in identifying key locations in a matter of minutes and requires the user to only clean unnecessary data from the scraped results.

Additionally, the tool itself can help anyone to do reconnaissance and plan strategies or routes ahead of time.

Reference

1. https://github.com/gosom/google-maps-scraper

Mind Map

Mind map for Google Map Scraper