Case: Scraping Data using Selenium

One of my python playgrounds, VOILA, it’s worked

I worked with dirty data every day, the source could be anywhere among websites or any sources from offline. After you got these kinds of data, you only have 2 options left:

1. Stay comfortable with manual working, it’s less thinking but much effort to work on.
2. Try to grasp ideas on how to automate those kinds of stuff, it’s really a pain in the ass, but you can work smart after.

I believe we’re all taking second option for sure.

I admit Python is changing my whole working life. I just learned it a couple of months ago thoroughly and applied it to working life. I’m type of person that very enthusiastic in terms of learning and actually, I can’t imagine how my life is if without python. Just like Gandhi said:

Live as if you were to die tomorrow. Learn as if you were to live forever.

Python is so handy programming language, quite easy to learn and understandable compared to other languages. If you’re a newbie at coding, I recommend you to learn Python first and understand the concepts.

WHAT IS DATA SCRAPING?

an action of scraping at the website page to get the valuable data or information. This is usually needed when you’re working with public datasets, where it isn’t available yet in Kaggle, etc, then you need to take the data from a website.

Get Karl Christian’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

For saving time and effort, you should scrap that website by running python script. In this case, the script is using Selenium framework.

WHY SELENIUM?

For scraping websites, we can use several well-known frameworks, such as BeautifulSoup, lxml, etc. In fact, Selenium isn’t built mainly for scraping websites, it’s testing purpose framework to automate web browsers. But personally, I prefer Selenium than others, because we can do more interactive with the websites (login, click, etc) just with few additional lines code and the built-in function is more straight-forward.

STEPS TO DO

1. Install Chromedriver package into jupyter realm (I’m using chrome as my browser)

pip install chromedriver_binary

2. Import necessary package into your scripts

from selenium import webdriver
import chromedriver_binary
import pandas as pd
import datetime
import numpy as np

3. Set URL you wanna go

url = "https://www.bps.go.id/dynamictable/2015/03/10/812/jumlah-penumpang-yang-berangkat-pada-penerbangan-domestik-di-bandara-utama-indonesia-2006-2019-orang-.html"

4. Create a web driver to start scraping

driver = webdriver.Chrome()
driver.get(url)

5. Click on the data you want to capture, run inspect element, and copy XPATH

website and inspect elements

6. Create function to get data from other rows also, only by changing value

def parse_table(val):
    value = driver.find_elements_by_xpath('//*[@id="tableRightBottom"]/tbody/tr[{}]/td'.format(str(val)))
added_value = []
for item in value:
        member = item.text
        member = "".join(member.split())
        added_value.append(member)
        
    return added_value
polonia = parse_table(1)
soeta = parse_table(2)
juanda = parse_table(3)
ngurah_rai = parse_table(4)
hasanudin = parse_table(5)

7. Create monthly period, starting from 2006 until 2019

start_year = 2006
end_year = 2020
month_key = pd.DataFrame({"month": pd.period_range(start_year, freq='M', periods=(end_year-start_year)*12)
                         })

8. Lastly, prepare the data in the form of one dataframe by concating all, and clean it, then create total value as a sum of 5 existing airports for each month

complete_data = pd.concat([month_key, 
           pd.DataFrame({"polonia": polonia,
                         "soeta": soeta,
                         "juanda": juanda,
                         "ngurah_rai": ngurah_rai,
                         "hasanudin": hasanudin
                        })
                        ], axis=1)
complete_data = complete_data.dropna()
def cast_as_numeric(df, col):
    df[col] = pd.to_numeric(df[col])
    return print("cast as numeric done")
cast_as_numeric(complete_data, "polonia")
cast_as_numeric(complete_data, "soeta")
cast_as_numeric(complete_data, "juanda")
cast_as_numeric(complete_data, "ngurah_rai")
cast_as_numeric(complete_data, "hasanudin")
complete_data["total"] = complete_data.iloc[:,1:].sum(axis=1)

output data

Next time, if you want to run additional months, you just run all in your jupyter and the script will automatically get the data from the website.

Thank you for clapping this article, I just want to share what I’ve done so far with my struggle to work smartly. My motto is doing whatever it needs and doesn’t do those that aren’t efficient.

I have other articles also that maybe is interesting for you.

• How to Schedule Query using Airflow and Google BigQuery
• Creating Time Comparison Trend using Tableau

I will always learn a lot further to keep improving my pieces of knowledge and collaborate on what tools I have learned so far, my next agenda could be automating data scraping using airflow. STAY TUNED !!