Using Python to Access Web Data
I started Python for Everyone (PY4E) “Using Python To Access Web Data” the first week of August. I completed all but two assignments in mid-August, but I became ‘ILL in the second week. I required six weeks to recover. My desire is to give weekly updates while progressing through the PY4E program. But on this course, I will offer a general view of the whole course.
Access Web Data is broken up into six weeks. Each week includes specific aspects of accessing web data with Python. The six weeks are as follows:
· Getting Started
· Regular Expressions
· Networks and Sockets
· Programs that Surf the Web
· Web Services and XML
· JSON and the REST Architecture
Get The Gray Coder’s stories in your inbox
Join Medium for free to get updates from this writer.
The class, in reality, is a five-week course, as “Getting Started” is repeated in every PY4E course. It is essentially a beginner’s guide to installing Python, Atom (a text editor for Python), and how to use the text editor with Python via the terminal window for both IOS and Windows. I recognize why this is done as all these courses can be signed up independently of the specialization for PY4E. You can enter the program at any class within the specialization. But it becomes monotonous.
In the second week I learned about “Regular Expressions” (RE). Prior to attending this course, I would see strange python code formatted in a way that made sense but in other ways it seemed unintelligible. The unusual code I was looking at were “Regular Expressions.” One source I found on YouTube that explains RE is Corey Schafer. Corey does an excellent job of explaining RE and working through specific Python code in his “Regular Expressions (Regex) Tutorial: How to Match Any Pattern of Text”. The second week’s assignment was straightforward in that you had to pull out numbers from a short unformatted text file, subsequently converting the numbers from text format to integer format and summing the numbers.
The third week was “Networks and Sockets”, it was not difficult. You are given specific URL and you were to write code pinging socket 80 and printing out the headers and data.
The fourth week was “Programs that Surf the Web”. This is the week that I found most interesting. During this week I was introduced to BeautifulSoup (BS4) and learned how to scrape a web page. Retrieving data and scraping out specific data. My go to as a reference was Corey Schafer and his video on Python Tutorial: Web Scraping with BeautifulSoup and Requests. There were two assignments for this week. The first was not difficult, you scrape a URL and sum a set of number associated with a “class”. The second assignment is a bit more challenging in that you are given a URL and within it there are a number of contacts with their specific URL’s to their contacts. You are to cycle through the contact URL’s an established number of times and then scrape a set contact and then print their name. I have to say that the knowledge of BeautifulSoup is the most valuable thing that I have acquired thus far in this class.
Week five was “Web Services and XML”. This section I used XML to retrieve and parse XML data. I found that the exercise for this section was difficult. Not in understanding the code but understanding how the code is applied. In the beginning I could not even get my code to open the given URL. You know that putting in a code that says to open and read the XML helps a lot. Once I included urlib.request.urlopen into my code I was able to get the program to open up the XML. Once I was able to open and read the file it went smoothly, and I was able to complete the assignment.
The last week, six, “JSON and the REST Architecture”. This week was spent mostly learning about JSON. Which along with BeautifulSoup (BS4) make a very powerful tool for scraping web pages. JSON basically allows for formatting of JavaScript on a web page. Once formatted then you can parse the JSON with BS4 to retrieve the data needed. The assignment for this week was straight forward as it was built on the previous week’s assignments. It was easily completed.
Generally speaking, this class started out slowly, but I ended up learning some very good tools. I can say that BeautifulSoup and JSON are powerful tools that will allow me to gather information from web sites and aggregate them into useable data. It will be interesting as to what I will be able to do with this knowledge going forward. Now on to PY4E “Using Databases with Python” a five-week course.
Thank you.
TheGrayCoder
Winners are not people who never fail but are people who never quit.
