Unsupervised Learning: PCA
PCA (Principal Components Analysis)
PCA is a linear dimensionality reduction technique with use in exploratory data analysis, visualisation, and data processing.
Generally, in unsupervised learning, it refers to selecting which principal components to use and the subsequent process of using them to understand the data.
Why PCA?
“ When there are p predictors, and we want to draw plots of n observations as a part of data exploratory analysis. We can do it by examining two-dimensional scatterplots of the data. However, there will be p*(p — 1)/2 number of pairs of scatter plots.
The better method of the n observations with p predictors visualisations, especially when p is large, requires a low-dimensional representation of data that holds as much information as it possibly holds.
PCA is the tool that finds the low-dimensional visualisation of data, and it also contains as much variation as it can. ”
Example: USA arrest data (in R)
For each of the 50 states of the USA, the USArrests data set contains the number of arrests for 3 crime: Assault, Murder, and Rape (per 100,000 residents). Also includes the UrbanPop ( the percentage of the population in each of the states living in the urban areas).
names(USArrests)
dim(USArrests)PCA is performed after standardisation.
# scale=TRUE standardizes features to mean=0, sd=1
pr.out <- prcomp(USArrests, scale=TRUE)The principal components scores and the loading vectors.
Get Aram’s stories in your inbox
Join Medium for free to get updates from this writer.
The loading vectors are coefficients or weights that indicate how much each original variable contributes to the principal components.
# Loading vectors are coeffiecnts and measure the weights of importance
PC1 PC2 PC3 PC4
Murder -0.5358995 -0.4181809 0.3412327 0.64922780
Assault -0.5831836 -0.1879856 0.2681484 -0.74340748
UrbanPop -0.2781909 0.8728062 0.3780158 0.13387773
Rape -0.5434321 0.1673186 -0.8177779 0.08902432“The first loading vectors are closely located, approximately equal for PC1. They are less weight than UbranPop.”
“ The second loading vectors put the most weight on UrbanPop.”
“ Overall, you can observe that the 3 main crimes are relatively close to each other, the UbanPop is a bit further from the other 3 variables. This indicats the 3 craim variables are highly correlated, otherwise the Ubarnpop is less correlated with others.”
“ Most of the loading vectors are unique; even if you flip the loading vectors, even the sign of the loading vector may differ. “
biplot(pr.out, scale=0)
The black name represents the principal components.
PC1: the first principal component: crime rates
PC2: the second principal component: level of urbanisation
The red arrows represent the first two principal loading vectors.
Staying close to the zero on both components means they are close to average levels for both crime and urbanisation. Because they are standardised, centred at zero. In some cases, when all components are measured in the same units, no need to do standardisation.
Proportion of Variance Explained (PVE)
We are interested in finding the smallest number of principal components to obtain a good understanding of the data.
Q: How many components do we need??
The two plots above explain the proportion of variance explained by each of the four principal components in the USArrests data.
As a subjective approach, the first few PCS were used to find an interesting relationship in the data. We continue to find subsequent PCs until we find an interesting relationship.
As an objective approach, we compute PCs for the supervised approach; we can treat the number of principal components as a score vector to be used in the regression as a tuning parameter to be selected by cross-validation or other approaches.
