What are Regular Expressions:

Yep, when your pattern is not working right, it can make you feel like the baby above. However, that is not what we are talking about here. Focusing back on web scraping… Everything can be thought of as a character. By writing patterns to match a sequence of characters, we are creating a regular expression. Based on wikipedia, regular expressions “is a sequence of characters that define a search pattern. Usually this pattern is used by string searching algorithms for “find” or “find and replace” operations on strings, or for input validation. It is a technique that developed in theoretical computer science and formal language theory. “

During the first initial chapters on the web scraping book I obtained (i.e. Web Scraping Book with Python — Ryan Mitchell), I started noticing multiple instances of regular expressions (i.e. regex). These seemed like a flexible and scalable way to filter information. After the last entry, I felt it was time to brush up on my skills.

Step 1:
~ Find a resource to quickly learn the basics.

After a quick google search, I came across RegexOne. This is a great site that incorporates quick, short lessons to practice what you are learning. After a few examples, I started feeling better and understanding the flow more. Now it was time to practice what was preached on a live site.

What are possible Use Cases:

~ Filtering text in resume
~ Searching for specific entries in medical notes
~ Web scraping
~ etc.

Below is a four step process I used to target a id tag within a <div>, on the NPR site.

Process to creating your First Regular Expression:

Step 1 — Review the https://www.npr.org/robots.txt file for NPR

Step 2 — Navigate to the desired page and inspect for respective div tag

Here, I am focused on targeting the <div> tag with id “storytext”. Once that has been located, the goal is to print the first paragraph starting with “The Medicaid expansion…”.

Step 3 — Take the id and paste into regex platform to try patterns to match sequence

Here, I tested out various patterns to get the test string to highlight all of the characters. Is this the best pattern every created…No…Could this capture other sequence of characters…Yes…However, for the html being reviewed this pattern avoids these pitfalls.

Step 4 — Add to program
Here we run the program with our regular expression in step 2.

Code:

On line 15 we see the re.compile with the pattern created to find “storytext”.

~ ^[s]+ -> finds a string that begins with the character ‘s’, matching at least once.

~ [t]+ -> adds to that string the character ‘t’, matching at least once.

~ [a-z]{1,7} -> adds to the string any character within the range of a to z, matching a repetition of a minimum of no less than 1 and a maximum of no more than 7.

~ $ -> suggests the end of the pattern.

Lessons Learned:
~ Find the tag in the html that needs to be selected
~ Practice regex prior to adding to code to test
~ Insert regex into code
~ Repeat

Next Steps:
~ It would be nice to learn more about logging errors, instead of using the print() method.

Food For Thought:
~ (Plain English Explanation of Regex — Just Paste) -> Highly Recommend!

“Wherever there is a human being, there is an opportunity for a kindness.”

— Seneca

The baby search engine…

Cute picture, but we are not talking about that type of crawling. At a high level, a search engine can be thought of in two parts. Part A, concerns multiple data warehouses to store all the data collected. Part B, is the crawler to scrape the internet. However, what we see in our preferred browser is not all of the internet. Apparently, there is a thing called the deep web and then another called the dark net. The deep web includes any part of the web that is not indexed by search engines, which is a good bit. Some suggest this makes of 90% of the internet. The Dark web is intentionally hidden from search engines using Tor and other elements. This deep web may have a great deal of information, but it could be difficult finding without a web scraper. But, first things first…Let’s figure out how to create a crawler to jump from one external link to another.

[Caution: Building web crawlers that are not contained within a page, may quickly lead to Non-PG material. So, be careful when creating.]

As with most things, a little planning never hurts in the long run. Below are a few questions to reference below plugging away at your code.

Before you start scraping, ask the following:

~ What information do I need to gather and why?
a) Are there pre-defined websites that hold this data?
b) Will this crawler need to discover new sites or not?

~ Does the crawler reach a website and drill down into the various pages or does it quickly follow the next outbound link?

~ What are specific websites that should not be scraped (i.e. those depending on local languages)?

~ What are the legal implications if my crawler draws the attention of a webmaster?

In the below example, we are crawling the web with no specific target in mind at this time. The bot will need to discover new sites and will use set() method to hold an unordered collections of unique urls. At this time, we have no limits on the bot regarding specific websites. And lastly, we have reviewed the https://hbr.org/robots.txt and should be ok. However, I did see a User-Agent: 008 had a disallow rule. So, it is always worth notifying the webmaster just in case.

Below is one example of how to use a bot to crawl from one external link to another.

(5) Step Process:

• Step 1 — Import Libraries

• Step 2 & 3 — Set variable to unique links and randomizer to randomly select links

This uses the current time of the device to set the initial value to generate a random value, in step 3.

• Step 4 — Retrieve list of Links

The try/ except section uses the urlparse() method to break up the url. Below is taken from the documentation. In our case, the conditional finds internal links by using the pattern that they begin with ‘/’ and consist of the current domain network location.

>>> from urlparse import urlparse
>>> o = urlparse('http://www.cwi.nl:80/%7Eguido/Python.html')
>>> o   # doctest: +NORMALIZE_WHITESPACE
ParseResult(scheme='http', netloc='www.cwi.nl:80', path='/%7Eguido/Python.html',
            params='', query='', fragment='')

This stores our external links in a python list. The regex searches for <a> tags that begin with either http or www; however, it excludes internal network locations. If it finds an external link it adds it to the external links list.

This part focuses on finding if any external links already exist within the page with the conditional “if len(external_links) ==0” section. If none found, the bot proceeds to look within the current domain until one is found. If there exists an external link, it follows that trail.

This takes the initial site and calls the function to retrieve the file and parse the information using BeautifulSoup, prints a random external link that has been scraped and then searches for the next in the list of external links.

• Step 5 — Trigger Program

This triggers the program to start at the Harvard Business Review website.

Well it appears that I have hit a HTTP 403 — Forbidden Error…

A great tip when planning the initial phase is to draw out a flow diagram as well, like below.

Lessons Learned:

~ .random() -> Pseudo-random number generator.
~ .seed() -> Seeding a pseudo-random number generator gives it its first “previous” value to perform some operation on.
~ .datetime() -> Provides the ability to manipulate dates and times in both simple and complex ways
~ urlparse() -> Breaks URL strings up in components (addressing scheme, network location, path etc.), to then combine back into a URL string later.
~ .robots.txt -> Examples below
a) https://www.google.com/robots.txt -> there are a lot here
b) https://www.yahoo.com/robots.txt
c) https://www.bloomberg.com/robots.txt -> Found a nice apply here message with this one.

Next Steps:

~ Regular expressions -> Still struggling to consistently make my own regex from scratch. Need more practice.

Food for Thought:

~ Exception Handling:
~ Always draw out the Flow of Program
~ Polite Web Crawler Article
~ Disallow Bots: (Example: https://hbr.org/robots.txt)

It’s time you realized that you have something in you more powerful and miraculous than the things that affect you and make you dance like a puppet.— Marcus Aurelius

For some students, being called on is a thrill and a privilege. For others…Not so much. This is the time to prove that all the teacher has been instructing has not gone in one ear and out the other. Well, after writing my first scraper I thought it would be good to find another website to test. So, in the words of Stephen Covey, let us start with the begin in mind. Let us build a scraper to find online courses from one of the many education platforms.

Options:
~ Udemy
~ Coursera
~ Udacity

Goal:
1) Create a web scraper to pull online course titles and their rankings from three websites.
a) Udemy
b) Coursera
c) Udacity

I’m not afraid of Errors…

After successfully completing the first program, the second should be easier. Well, that is what I thought. First I ran into errors, specifically “HTTP Error 403: Unauthorized”. In the back of my mind the Ghostbuster’s song came to mind “I ain’t afraid of no ghost”. Or better yet, I am not afraid of a little error. After a google search, I learned that this error pertains to something that is forbidden and authorization will not help. That was enough for me to stay away. However, without “handling exceptions” I would have never known that was tripping up the program. So on to the next site. Using coursera, I was able to avoid the HTTP errors and access the html. I was off to the races, woot woot! Using a combination of .find(), .find_all(), and html tags, I was able to navigate to the title and ratings. Below is an easy five step process to create your own scraper.

The Setup:
* The example uses Python 3.6 within Jupyter lab with the below dependencies
~ BeautifulSoup

Warning:
* Feel free to review the docs for additional arguments for the methods.
* Or pick up these books for additional details on the math behind these topics:
~ Web Scraping Book with Python — Ryan Mitchell

(5) Step Process:
~ Step 1 — Import libraries
~ Step 2 — Create Generic URL retrieval function
~ Step 3 — Assign variable to Generic URL
~ Step 4 — Navigate to list of courses
~ Step 5 — Print data

Step 1 — Import libraries

This is the basic setup where urlopen allows us to retrieve a specific url from the server. HTTPError allows for us to handle potential errors during the process of obtaining the web file from the server. BeautifulSoup allows us to parse the data for insights.

Step 2 — Create Generic URL retrieval function

It is important to use handling exceptions here to alert of potential issues.

Step 3 — Assign variable to Generic URL

During the troubleshooting phase, I used page_content.prettify() to initially try and see if the content would display. This I switched over and added a try/ except to this section to better understand the error being received.

Step 4 — Navigate to list of courses

Here we use the .find() method from BeautifulSoup to navigate to the specific section where the course list is located.

Step 5 — Print data

Here we loop over the list of <li> tags that contain the course specifics and extract the title and current rating.

Lessons Learned:

The use of exception handling was a big advantage to understand what was going on during the initial steps of the program. Once I understood the error, it was easier to identify the next steps to take. Breaking out the code with multiple exceptions made the process of finding out where specifically the error occurred. Next, the .find() and .find_all() are very useful methods that have been great for me as a beginner. And finally, f-strings make reading the final result easier on the eyes.

Next Step:

Regular expressions may have more flexibility than the current method, so that would be interesting to add to future projects. Also, I was at a local meetup and they mentioned using logging instead of the print() method. I will need to dig more into this.

Resources:

~ BeautifulSoup
~ HTTP Status Codes
~ Calling Tags

“Difficulties strengthen the mind, as labor does the body.” — Lucius Annaeus Seneca

Note: The information below is based on the book, “Web Scraping with Python” by Ryan Mitchell. This blog is a way for me to refer back to my notes and assist with my learning.

When you have a hammer, sometimes everything looks like a nail. Einstein once suggested, “Everything should be made as simple as possible, but not simpler.” This is all to say layering techniques may be a way to impress others or a way that leads to fragile code that is hard to debug. Below are five steps to assist.

(5) Ways to Avoid Advanced HTML Parsing:

1) When code heavily relies on the structure of a website, this may lead to vulnerabilities if the site ever changes.
2) There may be some sites that have a “Print this Page” option that has a cleaner HTML format.
3) Depending on the comfort level with Javascript, there may be ways to find hidden dictionaries with tons of information.
4) Depending on the development, useful information may be contained in the url of the page.
5) If the content desired is not unique to a specific page, feel free to investigate other sources that may be easier to access.

Assuming all of the above has been exhausted, there are times when Advanced HTML parsing is helpful (i.e. lets get out our small hammer).

Topics:
~ Creative Ways to Select Content
1) BeautifulSoup options
2) Regular Expressions: Define; Common Symbols; ID things about tag; sites;
~ Use cases
~ Anatomy of Creative Scraper
# Step 1 — Import libraries
# Step 2 — Create Generic URL retrieval function
# Step 3 — Assign variable to Generic URL
# Step 4 — Parse Data

The Setup:
* The example uses Python 3.6(https://www.python.org/downloads/) within Jupyter notebook(http://jupyter.org/) with the below dependencies
~ BeautifulSoup

Warning:
* Feel free to review the docs for additional arguments for the methods.
* Or pick up these books for additional details on the math behind these topics:
~ (Author of Web Scraping Book)

Creative Ways to select content:

A quick review of the BeautifulSoup website and it is clear that there are many ways filter content. We could use the .find_all() method and iterate through the list we retrieve back. We could pass in a parameter to the .find_all() method of limit, to limit the amount of items we retrieve in our list. We could use the .find() method, to search for the first instance of a tag, attribute, position or content. The possibilities feel like that are endless. Below are a few methods from BeautifulSoup.

Calling Tags:
1) .find() -> returns list containing single result; If it can’t find anything it returns None.
2) .find_all() -> scans an entire document for results; If it can’t find anything it returns an empty list.
3) .find_next_siblings() -> this iterates over the element’s siblings in a navigation tree and returns all matches

Regular Expressions:

What is a regular expression? They are a sequence of characters that define a search pattern…Or think of them as tools to identify regular strings which can be generated by a series of linear rules. An example would be a program that validates/ matches an email, telephone number, etc. If trying to target unique image tags, this would be a great resource. See example below in step four.

Anatomy of Creative Scraper of Item Page and Price:

Below is a process to scrape the information from a Craigslist page.

Step 1 covers importing the necessary libraries for requesting urls from the server, handling exceptions regarding server errors or page errors, using beautifulsoup to parse the html and using regular expressions to filter the content.

Step 2 covers creating a generate function to retrieve the desired url and find the unordered list tag within the html. This focuses on just the items for sale on the craigslist page.

Step 3 defines the url we wish to scrape and assigns a variable to the generic site file function.

Step 4 uses regular expression to target only a tags with a class of “result-image” and then proceeds to locate the item page link and the cost of the item. F-strings are used to make the content look better.

Below is the finished result.

Well, that is all for today.

Quote:

“Luck is what happens when preparation meets opportunity.” — Seneca

Note: The information below is based on the book, “Web Scraping with Python” by Ryan Mitchell. This blog is a way for me to refer back to my notes and assist with my learning.

What is the End Goal of this Journey?

Stephen Covey always started “Begin with the End in Mind”, regarding one of the 7 Habits. A while ago, I completed a course on Data Science with Python. It was exciting to learn the new tools, but everywhere I turned I kept hearing “You need to access messy data”. As my interest are focused on data engineering and data science, I quickly learned it is critical to have clean data. The whole concept, “Trash in Trash out” is real. I thought…What better way to obtain messy data than to go to the internet. However, there was a problem. I knew nothing about this thing called Web Scraping. The end goal is to learn more about this technique and find what works best for me. Below are a few of the elements that will be covered along the adventure.

1) Databases
2) Web Servers
3) HTTP
4) HTML
5) Internet Security
6) Image Processing
7) And a little Data Science

Topics:
~ What is Web Scraping
~ What are Use Cases
~ Anatomy of a Basic Web Scraper
1) Import Libraries
2) Create generic function
3) Assign variable
4) Handle Exceptions

What is web scraping?:

Is it a tool construction workers give to the intern and say, “Class is now in session, Wax On…Scrape Off”? Is it a device required to tear up the bathroom floor for remodeling? Well, those are all good guesses, but we are discussing something else. We are referring to the process of aggregating data by means other than having a program interact with an API (i.e. application programming interface). At a high level, a web scraper does the following.

1) Queries a web server automatically
2) Sends request for data
3) Extracts information by parsing data

What are some use Cases?

Have you ever had a project where large quantities of data needed to be gathered and processed quickly? The recommended way would normally be to access an API; however, sometimes that will not work due to certain limitations. Enter stage left Web Scraping with a few, yet not all, use cases below.

1) Machine Learning Training Data
2) Medical Diagnostics
3) Forecasting

Anatomy of Basic Web Scraper:

A web scraper may consist of, Step (1), importing Libraries to allow for sending requests to the server, handling server and page errors as well as formatting of HTML content. Step (2) Consists of creating a generic function to retrieve a file, given a url. During this phase, we will need to account for page, server and/ or missing tag errors. Step (3) assigns the variable to the object created from the previous function to allow for further parsing. Step (4) allows for parsing the data to extract useful information.

The Setup:
* The example uses Python 3.6 within Jupyter lab with the below dependencies.
~ BeautifulSoup

Warning:
* Feel free to review the docs for additional arguments for the methods.
* Or pick up these books for additional details on the math behind these topics:
~ (Author of Web Scraping Book) https://github.com/REMitchell/python-scraping

1) Import Libraries

The urllib is a standard Python library that allows for us to request urls from the server for scraping. There are numerous errors to account for; however, HTTPError allows for checking if a page is not found on a server. While, URLError validates that the server is found. BeautifulSoup is a none standard library that helps organizing messy data from the internet.

2) Generic URL Retrieval Function

Handling exceptions allows for us to check for potential errors that may negatively affect our program.

3) Assign variable

This variable is assigned to the function that will target the python.org events page for data retrieval.

4) Handle Exceptions

If there are errors we will receive the “Sorry the file…” message. However, if we are successful there will be a list of events. The for loop takes the BeautifulSoup object we created and drills down the page to the <ul> tag, specifically with the class ‘list-recent-events’. We quickly reviewed the structure of the site’s HTML to identify this was where the information was located. The .find_all() function allows us to target only the <li> tags. Once we have focused on this area we create variables for the event name and date to print.

Congratulations, we have a list of events.

Well, that is our first web scraper! Next we get to play with a Web Crawler.

“Think of the life you have lived until now as over and, as a dead man, see what’s left as a bonus and live it according to Nature. Love the hand that fate deals you and play it as your own, for what could be more fitting?” — Marcus Aurelius

Resources:
~ (Documentation on urllib) https://docs.python.org/3/library/urllib.html
~ (BeautifulSoup) https://www.crummy.com/software/BeautifulSoup/bs4/doc/#

With any new venture, it feels like the hardest thing to do is start. Questions float around like…What should be learned first? Where is the best value training? What platform offers the best value? And many more…So, what is one option to start learning Data Engineering? Having struggled with other skills, reinforcements were needed this time.

Agenda:
~ How to Learn fast
~ Rapid Skill Acquisition
~ (10) Principles of Rapid Skill Acquisition
~ (10) Principles of Effective Learning

Warning:
* This is just one path and there are many ways to attain a desired skill.

Learning Fast:

To organize this initial process, there is a very helpful book by Josh Kaufman titled “The First 20 Hours, How to Learn Anything Fast”. In a way the secret sauce can be thought of as follows…

Recipe:

1) Know what you are getting into

• Focus on desconstructing the skill into the smallest possible subskills.

2) Learn the fundamentals

• Learn about the most important subskills to ensure practice is conducted intelligently and there is a self-correct element during the progression. The goal is to be able to be self-aware and adapt quickly if something is not working.

3) Remove Obstacles

• Learn to remove physical, mental and emotional barriers.

4) Identify a Consistent Practice routine

• Dedicate self to practicing subskills for at least 20 hours before deciding to move on to another venture, if the skill is no longer interesting.

Rapid Skill Acquisition:

The book focuses on Rapid Skill Acquisition, which is the process of breaking a skill down into the smallest possible parts, by identifying the most important areas of the skill and practicing these first. This allows for creativity, flexibility and freedom to experiment during the experience. The process typically passes through three stages to achieve skill acquisition.

(3) Stages -> Using infants’ walking experience as example…

1) Cognitive (Early) Stage

• Think of an infant watching toddlers walking and beginning their first steps to do the same thing.

2) Associative (Intermediate) Stage

• Think of this as an infant discovering their balance and coordination, by taking the first few steps unassisted.

3) Autonomous (Late) Stage

• Finally think of this as freedom, or the time to child proof additional areas of the house. Quick…Baby on the run!

This is nice and all, but how can it be used to help learn Data Engineering? Enter the 10 principles of Rapid Skill Acquisition. This is the what to do.

10 Principles of Rapid Skill Acquisition:

These will be our roadmap to successfully acquiring our intended skills.

(10) Principles:

1. Choose a lovable project
   ~ By putting all focus on the prime skill, we should be able to acquire it in less time.
2. Focus your energy on ONE skill at a time
   ~ This is like the dog being distracted by the squirrel…Stop That! Creating a “Someday/ Maybe” list is very helpful to log things of interest to be reviewed at a later date.
3. Define target performance level
   ~ Take the time to write a brief statement of what the end looks like. What does it look like when the skill has been obtained to the desired level? A good reference is Steven Covey’s 7 Habits of Highly Effective People (i.e. Begin with the end in mind).
4. Deconstruct skill into subskills (Very Important)
   ~ Learning to Snowboarding Example
   Lock boots into bindings
   Stand up
   Practice falling
   Leaf down the slopes
   Practice front side and then back side
5. Obtain critical tools
   ~ Finding these prior to starting practice is critical.
6. Eliminate barriers to practice
   ~ The key is to alter the environment as much as possible to make it easy to initiate practice.
7. Make dedicated time for practice
   ~ Dedicate at least 90 minutes/ day to practice
   ~ Pre-commit to at least 20 hours of practice before stopping the skill acquisition
8. Create Fast Feedback loops
   ~ Create opportunities to get feedback on performance as quickly as possible. Possibly from friends, family, industry experts or others who are knowledgeable.
9. Practice by the clock in short bursts
   ~ Use phone timer, or alternate device, to practice for 20 minutes sessions at a time.
   ~ Break practice sessions into smaller chunks with 3 to 5 per day
10. Emphasize quantity and speed
    ~ It is critical to focus on practicing as much as possible, as quickly as possible in the beginning. Be sure the practice is not sloppy however.

Think or these elements as a checklist to get started. Now we need to learn the how to do. Enter 10 Principles of Effective Learning.

10 Principles of Effective Learning:

“No problem can withstand the assault of sustained thinking.” ~ Voltaire

1. Research Skill and related topics
   ~ Spend 20 minutes, or more, to find resources on the skill.
   ~ Find at least 3 references (i.e. books, dvds, course material, etc.).
   ~ Identify trends and patterns regarding the most important subskills.
   ~ Skimming is fine.
2. Jump in over your head
   ~ Move forward, even though the content may be confusing (i.e. this is going to be hard for some).
   ~ Strive for an intimidating learning pace.
3. Identify mental models and mental hooks
   ~ Mental Models -> This is like putting labels on things that come up over and over again in the reviewing of reference materials.
   ~ Mental Hooks -> This is using analogies and metaphors to remember new concepts. Think of how a wonderful Librarian assists users to resources the same way a server helps users to a website of interest.
4. Imagine opposite of what you want
   ~ Think about the worst possible result, from the experience.
   ~ Ask…What if everything goes wrong? What is the worst possible outcome?
   ~ This should help to point out the most important skills to acquire.
5. Talk to practitioners to get expectations
   ~ Ask what does reasonable performance look like for a beginner?
   ~ This should help prevent discouragement, or at least limit the instances.
6. Eliminate distractions in your environment
   ~ Off with the digital devices…
   ~ Find a quiet space from friends and family…And even enemies.
7. Use spaced repetition and reinforcement for memorization concepts
   ~ Ideas that are harder to remember are reviewed often, while easier ones are reviewed less often.
   ~ Try using flash cards.
8. Create scaffolding and checklists
   ~ Checklists -> Helps to remember the setup for every practice
   ~ Scaffolding -> Structured approach to practicing skill…Think about how players set up to conduct free throws in basketball. Yes, even Shaq probably had some process. : )
9. Make and test predictions
   ~ Try new things and see what works…Be the mad scientist…
   ~ Try to guest the outcome for the experiment is started…
   (4) Steps
   ~ a) Observations — What currently observing?
   ~ b) Knowns — What known about topic?
   ~ c) Hypothesis — What will improve performance?
   ~ d) Tests — What to do next?
   ~ Record the experiments for future reference.
10. Honor your biology
    ~ Break up study sessions to prevent burn-out.

Again, these are just helpful checklists to stay focused on the task at had and provide a structured process for learning.

Well, time to start building that road map.

“A problem well stated is a problem half-solved.” — Charles Kettering

Potential Food for Thought (Books):
~ Mindset: The New Psychology of Success (Dr. Dweck)
~ The First 20 Hours (Josh Kaufman)
~ 7 Habits (Stephen Covey)

Until next time…

This is where Data Engineering comes into play. Well, this is the start! Let’s see what happens.

Disclaimer: I took a front-end engineering course at a bootcamp in 2016. Since then, the majority of my training has been through online courses, youtube and reading various industry books. This is my personal journey, which may be different for you. However, I am sure there will be some great lessons along the way.

“Get busy living, or get busy dying” ~ Andy Dufresne…

This article covers work from the Python for Data Science and Machine Learning Bootcamp course on Udemy by Jose Portilla and helpful tips along the way. This course was very helpful in gaining a base understanding of the topic.

What is this KNN (K Nearest Neighbors) all about?:
Who is KNN? Is it some underground band that is bringing a new sound with the banjo that lives next door? Steve Martin, watch out now. What is it? Does it make a pie better than Patti LaBelle’s sweet potato pie? Well, no. KNN is a classification algorithm. It uses the k value and distance metric(Euclidean distance) to measure the distance of new points to nearest neighbors. The pros include that it is: simple to implement, training is easier and it has few parameters. The cons include: high prediction cost, doesn’t play well with numerous features and doesn’t like categorical features. Sorry qualitative features, only quantitative welcome here. The figure below displays a new point — identified by the star — that we want to classify. If we use a k value of 3, we would classify the new point with the gray elements. However, if we use a k value of 10, we would then classify the new point with the orange elements. The smaller the k value the greater the noise with the data; however, we can smooth this out by increasing the value of k.

What are some use Cases?:
There are times where data may be given that is anonymized and the goal is to attempt to classify it, without really knowing the context of the data. Think group customer personalities to recommend certain products. Some of the ‘use cases’ of this are for the following:

• Recommender Systems
• Theft prevention in the modern retail business
• Detect patterns in credit card usage and much more

Topics:
~ What is KNN
~ How do we use it
1) Import Libraries
2) Read in Dataset
3) Standardize scale to prep for KNN Algorithm
4) Split data into training and test sets
5) Create and Train the Model
6) Make predictions with the Model
7) Evaluate the predictions
8) Evaluate alternative K-values for better predictions
9) Plot Error Rate
10) Adjust K value per error rate evaluations

The Setup:
* The example uses Python 3.6within Jupyter notebook with the below dependencies
Matplotlib 2.1.2
Numpy 1.14.1
Pandas 0.20.3
Seaborn 0.7.1
Scikit Learn 0.19.1

Warning:
* Feel free to review the docs for additional arguments for the methods.
* Books for additional details on the math behind these topics:
~ Book (An Introduction to Statistical Learning — The Math)
~ Book (The Elements of Statistical Learning)

Step 1: Import the necessary Libraries
Similar to previous discussions, we will need to import libraries that allow for data analysis and data visualization to get acclimated to the dataset. We will be using pandas, numpy, matplotlib and seaborn to conduct this.

Data Exploration libraries

import pandas as pd
import numpy as np

Data Visualization libraries

import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

Step 2: Read in the dataset
We will use the pandas .read_csv() method to read in the dataset. Then we will use the .head() method to observe the first few rows of the data, to understand the information better. In our case, the feature(column) headers tell us pretty little. This is fine because we are merely trying to gain insight via classifying new data points by referencing it’s neighboring elements.

# Use pandas .read_csv() method to read in classified dataset
# index_col -> argument assigns the index to a particular column

df = pd.read_csv('Classified Data', index_col=0)

# Use the .head() method to display the first few rows
df.head()

Step 3: Standardize (normalize) the data scale to prep for KNN algorithm
Because the distance between pairs of points plays a critical part on the classification, it is necessary to normalize the data to minimize this(helpful link). This will generate an array of values. Again, KNN depends on the distance between each feature.

# Import module to standardize the scale
from sklearn.preprocessing import StandardScaler

# Create instance (i.e. object) of the standard scaler
scaler = StandardScaler()

# Fit the object to all the data except the Target Class
# use the .drop() method to gather all features except Target Class
# axis -> argument refers to columns; a 0 would represent rows
scaler.fit(df.drop('TARGET CLASS', axis=1))

We should obtain the following confirmation below.

# Use scaler object to conduct a transforms
scaled_features = scaler.transform(df.drop('TARGET CLASS',axis=1))

# Review the array of values generated from the scaled features process
scaled_features

Here we have the normalized dataset, minus the target column

Step 4: Split the normalized data into training and test sets
This step is required to prepare us for the fitting (i.e. training) the model later. The “X” variable is a collection of all the features. The “y” variable is the target label which specifies the classification of 1 or 0 based. Our goal will be to identify which category the new data point should fall into.

# Import module to split the data
from sklearn.model_selection import train_test_split

# Set the X and ys
X = df_feat
y = df['TARGET CLASS']

# Use the train_test_split() method to split the data into respective sets
# test_size -> argument refers to the size of the test subset
# random_state -> argument ensures guarantee that the output of Run 
# 1 will be equal to the output of Run 2, i.e. your split will be always the same

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=101)

This allows for use to train our model on the training set and evaluate the built model against the test set to identify errors.

Step 5: Create and Train the Model
Here we create a KNN Object and use the .fit() method to train the model. Upon completion of the model we should receive confirmation that the training has been completed.

# Import module for KNN
from sklearn.neighbors import KNeighborsClassifier

# Create KNN instance
# n_neighbors -> argument identifies the amount of neighbors used to ID classification
knn = KNeighborsClassifier(n_neighbors=1)

# Fit (i.e. traing) the model
knn.fit(X_train, y_train)

Step 6: Make Predictions
Here we review where our model was accurate and where it misclassified elements.

# Use the .predict() method to make predictions from the X_test subset
pred = knn.predict(X_test)

# Review the predictions
pred

Step 7: Evaluate the predictions
Evaluate the Model by reviewing the classification report or confusion matrix. By reviewing these tables, we are able to evaluate how accurate our model is with new values.

# Import classification report and confusion matrix to evaluate predictions
from sklearn.metrics import classification_report, confusion_matrix

Classification Report -> This tells us our model was 92% accurate…

# Print out classification report and confusion matrix
print(classification_report(y_test, pred))

Confusion Matrix

# Print out confusion matrix
cmat = confusion_matrix(y_test, pred)

#print(cmat)
print('TP - True Negative {}'.format(cmat[0,0]))
print('FP - False Positive {}'.format(cmat[0,1]))
print('FN - False Negative {}'.format(cmat[1,0]))
print('TP - True Positive {}'.format(cmat[1,1]))
print('Accuracy Rate: {}'.format(np.divide(np.sum([cmat[0,0],cmat[1,1]]),np.sum(cmat))))
print('Misclassification Rate: {}'.format(np.divide(np.sum([cmat[0,1],cmat[1,0]]),np.sum(cmat))))

Step 8: Evaluate alternative K-values for better predictions
To simplify the process of evaluating multiple cases of k-values, we create a function to derive the error using the average where our predictions were not equal to the test values.

# Generate function to add error rates of KNN with various k-values
# error_rate -> empty list to gather error rates at various k-values
# for loop -> loops through k values 1 to 39
# knn -> creates instance of KNeighborsClassifier with various k
# knn.fit -> trains the model
# pred_i -> conducts predictions from model on test subset
# error_rate.append -> adds error rate of model with various k-value, using the average where prediction not
# equal to the test values

error_rate = []

for i in range(1,40):
    
    knn = KNeighborsClassifier(n_neighbors=i)
    knn.fit(X_train, y_train)
    pred_i = knn.predict(X_test)
    error_rate.append(np.mean(pred_i != y_test))

Step 9: Plot Error Rate

# Configure and plot error rate over k values

plt.figure(figsize=(10,4))
plt.plot(range(1,40), error_rate, color='blue', linestyle='dashed', marker='o', markerfacecolor='red', markersize=10)
plt.title('Error Rate vs. K-Values')
plt.xlabel('K-Values')
plt.ylabel('Error Rate')

Here we see that the error rate continues to decrease as we increase the k-value. A picture tells a thousand words. Or at least here, we are able to understand what value of k leads to an optimal model. The k-value of 17 seems to give a decent error rate without too much noise, as we see with k-values of 28 and larger.

10) Adjust K value per error rate evaluations
This is just fine tuning our model to increase accuracy. We will need to retrain our model with the new k-value.

# Retrain model using optimal k-value

knn = KNeighborsClassifier(n_neighbors=17)
knn.fit(X_train, y_train)
pred = knn.predict(X_test)

Classification Report -> This tells us our model was 95% accurate…

# Print out classification report and confusion matrix
print(classification_report(y_test, pred))

Confusion Matrix

# Print out confusion matrix
cmat = confusion_matrix(y_test, pred)

#print(cmat)
print('TP - True Negative {}'.format(cmat[0,0]))
print('FP - False Positive {}'.format(cmat[0,1]))
print('FN - False Negative {}'.format(cmat[1,0]))
print('TP - True Positive {}'.format(cmat[1,1]))
print('Accuracy Rate: {}'.format(np.divide(np.sum([cmat[0,0],cmat[1,1]]),np.sum(cmat))))
print('Misclassification Rate: {}'.format(np.divide(np.sum([cmat[0,1],cmat[1,0]]),np.sum(cmat))))

Well, until next time. Still trying to find a better way to consolidate the information as I learn new content. The goal is to get back to publishing a post a day, per each new machine learning algorithm studied.

Tips:

~ Currently reading Naked Statistics by Charles Wheelan. Great perspective on statistics and how it is used in everyday life. If you hated your undergrad courses because you saw know purpose, this is the book for you.

Quote
It’s easy to lie with statistics, but it’s hard to tell the truth without them. ~ Andrejs Dunkels

This article covers work from the Python for Data Science and Machine Learning Bootcamp course on Udemy by Jose Portilla and helpful tips along the way. This course was very helpful in gaining a base understanding of the topic.

What is Logistic Regression:
Unlike the discussion on linear regression, not all problems in the world are continuous. There are times when the solution to our problems is of a binary state. This model helps with classifying a new data point and by specifying which category it most closing aligns. These categories can be thought of as qualitative elements. To predict qualitative responses, we use classification. This is different from linear regression, as linear regression is concerned with the prediction of continuous values. Logistic regression models the probability that a response falls into a specific category. Once trained we may use the confusion matrix to evaluate the classification. The four values are true positive (predicted yes and actually was yes), true negative (predict no and actually was no), false positive (predicted yes and actually was no) and false negative (predicted no and actually was yes).

False Positives(FP) are type I errors (i.e. think or these as doctor informing a man he is pregnant). False Negatives(FN) are type II errors (i.e. think of a doctor telling a pregnant women that she is not indeed pregnant).

What are some use Cases?:
A logistic regression model helps us solve, via the Sigmoid function, for situations where the output can take but only two values, 1 or 0. This process can be used to identify spam email vs. non-spam emails, whether or not that loan offer approves an application or the diagnosis of a particular disease.

We will be working with the Titanic Dataset:
To learn about logistic regression, we will be working with the Titanic dataset from Kaggle.

Topics:
~ What is Logistic Regression*
~ What are use cases for Logistic Regression*
~ Example with Titanic Dataset
a) Exploratory Data Analysis
b) Cleaning Data to handle missing and categorical data
c) Training the Logistic Regression Model

The Setup:
* The example uses Python 3.6within Jupyter notebook with the below dependencies
Matplotlib 2.1.2
Numpy 1.14.1
Pandas 0.20.3
Seaborn 0.7.1
Scikit Learn 0.19.1

Warning:
* Feel free to review the docs for additional arguments for the methods.
* Books for additional details on the math behind these topics:
~ Book (An Introduction to Statistical Learning — The Math)
~ Book (The Elements of Statistical Learning)

Exploratory Analysis:
It is always beneficial to get to know your data before jumping too far into analysis. It has been stated many times before, using the .info() and .describe() methods are great tools to quickly get a feel for your data. Please refer to images below after the dataset has been loaded in via .read_csv() method.

They can tell you the data types for your features, if any features are missing information, what the total rows and columns may be and so much more. Planning ahead saves significant time down the road.

Step 1 — Import data analysis and visualization libraries

# Import data analysis modules
import numpy as np
import pandas as pd

# Import visualization modules

import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

Step 2 — Load dataset in with pandas

# Use pandas to read in csv file
train = pd.read_csv(‘titanic_train.csv’)
train.head()

Step 3 — Data Analysis

# Use the .isnull() method to locate missing data
missing_values = train.isnull()
missing_values

We use seaborn to conduct heatmap to identify missing data, via the isnull() method on the dataset. The yellow dashes suggest the data is missing.

# Use seaborn to conduct heatmap to identify missing data
# data -> argument refers to the data to creat heatmap
# yticklabels -> argument avoids plotting the column names
# cbar -> argument identifies if a colorbar is required or not
# cmap -> argument identifies the color of the heatmap

sns.heatmap(data = missing_values, yticklabels=False, cbar=False, cmap=’viridis’)

When reviewing the dataset, it is helpful to use a .countplot() on the target label to get a quick reference of the ratios. Then we can begin assessing relationships with the target label and the various features in the dataset. The value of 0 are individuals who did not survive.

# Use the countplot() method to identify ratio of who survived vs. not
# (Tip) very helpful to get a visualization of the target label
# x -> argument referes to column of interest
# data -> argument refers to dataset

sns.countplot(x='Survived', data=train)

# Use the countplot() method to identify ratio of who survived vs. not with interest in Passenger class
# x -> argument referes to column of interest
# data -> argument refers to dataset
# hue -> allows another level to subdivide data
# palette -> argument refers to plot color

sns.countplot(x='Survived', data=train, hue='Pclass')

To provide for even more engagement, we can use plot.ly to create interactive figures.

Cleaning Data: (Most of the time is spent here*)

Step 1 — Assess missing values to impute
There are times when data is missing, but we can impute (i.e. fill in) the values based on the existing content available. To investigate this concept we will use the .boxplot() on the ‘Pclass’ and ‘Age’ features to identify the average age per passenger class level.

Step 2 — Create function to update missing values
Defining a function using if statements allows for us to iterate over the data and update cells that contain missing values. Upon completion we are able to use the .heatmap() method to visualize our efforts to clean up the information.

# Create function to impute the age value if it is null

def impute_age(cols):
    Age = cols[0]
    Pclass = cols[1]
    
    if pd.isnull(Age):
        
        if Pclass == 1:
            return 37
        elif Pclass == 2:
            return 29
        else:
            return 24
    else:
        return Age

# Apply function to impute the age for missing values
# The age column is at position 0
# The pclass column is at position 1
# axis -> argument refers to columns instead of rows to apply the impute_age function

train['Age'] = train[['Age', 'Pclass']].apply(impute_age, axis=1)

Heatmap

Step 3 — Drop features that are missing too much information or insignificant rows

Step 4 — Convert Categorical features with Dummy variables
(Critical) -> Ensure avoidance of multicollinearity issue by passing in the argument ‘drop_first’. This issue may undermine the statistical significance of an independent variable.

# Use the .get_dummies() method to convert categorical data into dummy values
# train['Sex'] refers to the column we want to convert
# drop_first -> argument avoids the multicollinearity problem, which can undermines
# the statistical significance of an independent variable.

sex = pd.get_dummies(train['Sex'], drop_first=True)
embark = pd.get_dummies(train['Embarked'], drop_first=True)

# Use  .concat() method to merge the series data into one dataframe

train = pd.concat([train, sex, embark], axis=1)

Step 5 — Drop any further insignificant features from the dataset

# Drop columns with categorical data

train.drop(['Sex','Embarked','Ticket','Name','PassengerId'], axis=1, inplace=True)

Training the Logistic Regression Model:

Step 1 — Split dataset into “X” features and “y” labels
This step is required to prepare us for the fitting (i.e. training) the model later. The “X” variable is a collection of all the features. Think of this as ‘Pclass’, ‘Sex’, ‘Age’, etc.. The “y” variable is the target label which specifies whether the individual survives or not. Our goal will be to identify which category the data point should fall into.

# Split data into 'X' features and 'y' target label sets

X = train[['Pclass', 'Age', 'SibSp', 'Parch', 'Fare', 'male', 'Q',
       'S']]
y = train['Survived']

Step 2 — Split the data into a training and test set
This allows for use to train our model on the training set and evaluate the built model against the test set to identify errors.

# Import module to split dataset
from sklearn.model_selection import train_test_split

# Split data set into training and test sets

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=101)

Step 3 — Create and Train the Model
Here we create a LogisticRegression Object and use the .fit() method to finally train the model. Upon completion of the model we should receive confirmation that the training has been completed.

# Import module for fitting
from sklearn.linear_model import LogisticRegression

# Create instance (i.e. object) of LogisticRegression
logmodel = LogisticRegression()

# Fit the model using the training data
# X_train -> parameter supplies the data features
# y_train -> parameter supplies the target labels
logmodel.fit(X_train, y_train)

Step 4 — Evaluate the Model by reviewing the classification report or confusion matrix. By reviewing these tables, we are able to evaluate the model. Below we are able to identify that the model has a precision of 81% accuracy. To improve this we could gather more data, conduct further feature engineering and more to continue to adjust.

Well, until next time…

We are more often frightened than hurt; and we suffer more from imagination than from reality. ~ Seneca the Younger

This article covers work from the Python for Data Science and Machine Learning Bootcamp course on Udemy by Jose Portilla and helpful tips along the way. This course was very helpful in gaining a base understanding of the topic.

We’re Back!!! But, this will not be a dinosaur story. We left off with learning how to use choropleth maps with plotly. It is time to get our hands dirty with this thing called machine learning. One way to view machine learning is as a method that allows computers to discover insights regarding a topic without being explicitly programmed. It is a method of data analysis that automates analytical model building. Some of the ‘use cases’ of this are for the following:

• Fraud detection
• Equipment failure prediction
• Sentiment analysis
• Image recognition and much more.

Overview of Machine Learning Process:
This process may be split up into five phases. The first is the acquisition of data. Think of the old saying “Trash in, Trash out”.

There are free and paid services that offer datasets. Kaggle has some great datasets to get started and competitions to obtain great hands-on learning. The next step is data cleaning. Here we can use skills from pandas, numpy and others to prepare the data for training. Once the data has been cleaned we can split it into a training and a test data set. We then proceed to fit (i.e. train) the model on the training data set portion. Once the model has been trained we use the test data set to evaluate the model and iterate over the process until the desired results are achieved. At the completion of a satisfactory model, we deploy it.

Main types of Machine Learning:
These include supervised, unsupervised and reinforcement learning. Supervised consists of having labeled data to train and trying to predict a label from known features. The algorithm learns by comparing actual output with the correct outputs to identify errors. Some methods are classification, regression and prediction. Use cases are where historical data predicts a future event (i.e. stock prices). Unsupervised is having unlabeled data and trying to group similar data points together to gain insights. Here the the right answer is not provided. The main purpose is to explore the data and find attributes that separate segments. Typical techniques include self-organizing maps, nearest neighbor mapping and single value decomposition (i.e. think if one guy with red shirt bought this, maybe another will also — very simple example I know…). While Reinforcement is where an algorithm learns an action from experience (i.e. practice). This is used for robotics, gaming and navigation. The algorithm discovers which actions yield the best results through trial and error.

Linear Regression:
The purpose of regression is to draft a line that is as close as possible to every data point. The residual is the difference between the data point (i.e. observation) and the line of best fit.

Topics:
* How to conduct Data Exploration
* How to training our model using linear regression

The Setup:
* The example uses Python 3.6within Jupyter notebook with the below dependencies
Matplotlib 2.1.2
Numpy 1.14.1
Pandas 0.20.3
Seaborn 0.7.1
Scikit Learn 0.19.1

Warning:
* Feel free to review the docs for additional arguments for the methods.

How to conduct Data Exploration:
Before we start splitting data and training models we will need to do a little data exploration to understand the information that we have. Obtaining a clear understanding of the data we are working with provides a better foundation to analyze and apply machine learning techniques later.

Step 1 — Conduct Imports:
To begin we must conduct the necessary imports. We will use pandas and numpy for data manipulation and apply aggregate methods. We will use seaborn and matplotlib for data visualization.

Step 2 — Obtaining Data
We will use the pandas method .read_csv() to read in the data. This data resides in the same directory as my working jupyter notebook.

Step 3 — Investigating Data
To investigate we will use the .info() method to identify the rows and column details. The .describe() method provides statistical insights on the dataset. While the .columns method returns a list of all the column names.

Step 4 — Visualize the Data
Seaborn’s .pairplot() method is used to display the numeric correlations of the data from a quick high level point. The seaborn .distplot() method displays the distribution of a desired feature (i.e. column). While the seaborn .heatmap() method provides a clearer picture of feature correlations.

How to training our model using linear regression:

Step 1 — Split dataset into “X” features and “y” labels:
This step is required to prepare us for the fitting (i.e. training) the model later. The “X” variable is a collection of all the features. Think of this as a house’s age, number of bedrooms, number of rooms and etc. The “y” variable is the target label which specifies the price. Our goal will be to identify the best value to set a feature to obtain the maximum price.

Step 2 — Split the data into a training and test set:
This allows for use to train our model on the training set and evaluate the built model against the test set to identify errors. Still trying to fully understand the random_state argument, but here is some help(https://stackoverflow.com/questions/45089858/what-does-the-random-state-parameter-do-in-sklearns-parametersampler).

Step 3 — Create and Train the Model:
Here we create a LinearRegression Object and use the .fit() method to finally train the model. Upon completion of the model we should receive confirmation that the training has been completed.

Step 4 — Evaluate the Model by reviewing the coefficients:
By reviewing the coefficients, we are able to evaluate the model. The Avg. Area Income element in the figure below could be translated as for every 1 unit that that feature increases, the price of the house increases as well by 21.53. Obviously, this is a test dataset and these do not pass the real world test. However, there are others out there to gain practice.

Tips, Tricks and Lessons Learned:
- Web scraping — Great for creating your own custom datasets…
- (Book) An Introduction to Statistical Learning — The Math — great for understanding the math behind the algorithms
- (Book) The Elements of Statistical Learning
- (Data Exploration Tools) Using the .info() method to investigate data
- (Data Exploration Tools)Using the .describe() method to investigate data
- (Data Exploration Tools)Using Distribution Plots
- (Data Exploration Tools)Using pairplot() method for correlations
- (Data Exploration Tools)Using heatmaps() method for correlations
~ Guilty Pleasures
a) Transcending Time
b) Easy Listening

Well, until next time…

It’s not because things are difficult that we dare not venture. It’s because we dare not venture that they are difficult. ~ Seneca the Younger