Principle Component Analysis
A recap of Andrew Ng course (Machine Learning Specialization) on Coursera
Course 3 Week 2
It is hard to visualize the data when there are hundreds or thousands of features, and PCA is commonly used by data scientists to visualize high dimensional data.
For example, if we have some data about cars such as length and width, and we need to pick just one to represent the car, we may choose length because most of the cars have similar width due to the width of the roads. This process is similar to what PCA does. In addition to just picking one feature from the dataset, we can also create or find a new axis, such as z axis, on the flat 2D plane (this is not the z axis that sticks out in the third dimension) to represent the original features. In practice, PCA is normally used to convert thousands of features into just 2 or 3 features so that we can visualize them.
PCA Algorithm (Unsupervised learning algorithm)
- Preprocess/scale the features first if they have different scales.
2. Choose an axis
Option 1: choose a z axis as below and then project all the examples onto the axis with the line segment at 90 degrees. This method still captures the distribution of the examples.
Option 2:
If we choose another z axis as below, the examples are all squished together and we lose the original variance. (With these two examples, you can notice that there are projections that place different points in almost the same point, and there are projections that keep the points as separated as they were in the plane.)
Option 3:
Another potential z axis may look as below, and this is the best axis to choose from because this one is the principal component, which is the axis that when you project the data onto it, you end up keeping the largest possible amount of variance (information).
We can pick more than one principal components. If we pick a second principal component, it will be at 90 degree to the first principal component. And the third principal component will also be at 90 degree to the first and second principal components.
3. After getting the z axis, we will start to do the projection.
Get Aimyo’s stories in your inbox
Join Medium for free to get updates from this writer.
We first find the length one vector pointing in the direction of this z axis, and then take the dot product of the original vector and the length one vector. The result is 3.55, meaning that if we were to use one number to represent the original vector on the z axis, that number will be 3.55.
Note: if we were given the value on the z axis, we can find a vector that is similar to the original vectors by multiplying the value by the length one vector [0.71, 0.71]
PCA in code
If we pick two components for a 2D feature dataset, the resulting two principal components will cover all the variance. The first principal component will explain 0.992 percent of the variance and the second will explain 0.008 because they have to sum up to 1. (If you had 2 features and choose 2 principal components, then you’ll keep all the information and the data will end up the same as the original.)
PCA was once used to reduce the number of features for supervised training like support vector machine but not used that often recently. Because we normally pass the raw features into deep learning algorithms.
