What is PCA and why is it used?
Principal Component Analysis (PCA) is a statistical technique used to reduce the number of variables (dimensions) in a dataset while preserving as much information as possible.
Intuition
Imagine you have data with many features:
- Height
- Weight
- Age
- Income
- Education
- Spending score
- Credit score
Many of these variables are correlated. PCA finds a new set of variables, called principal components, that summarize the original data with fewer dimensions.
Think of it like taking a photo of a 3D object:
- A good camera angle captures most of the object’s shape.
- PCA finds the “best angles” from which the data can be viewed while losing the least amount of information.
How PCA works
Suppose you have two correlated variables:
HeightWeight17065175701807518580
Since taller people generally weigh more, these variables contain similar information.
PCA creates new variables:
- Principal Component 1 (PC1): captures the direction of the greatest variation in the data.
- Principal Component 2 (PC2): captures the remaining variation while being perpendicular (orthogonal) to PC1.
Typically:
- PC1 explains most of the information.
- PC2 explains much less.
Instead of storing both height and weight, you might only keep PC1.
Why is PCA used?
1. Dimensionality reduction
If you have 100 features, PCA might reduce them to 10 while retaining 95% of the information.
Benefits:
- Faster algorithms
- Less memory
- Easier visualization
2. Remove redundancy
Many features are correlated.
For example:
- Temperature (°C)
- Temperature (°F)
These contain essentially the same information.
Get Karthikanivas’s stories in your inbox
Join Medium for free to get updates from this writer.
PCA combines correlated variables into fewer components.
3. Data visualization
Humans cannot visualize data beyond three dimensions.
PCA lets you reduce:
- 100 dimensions → 2 dimensions
- 500 dimensions → 3 dimensions
This makes it easier to identify clusters or outliers.
4. Noise reduction
Small principal components often represent random noise rather than meaningful patterns.
By discarding them, PCA can improve data quality.
5. Improve machine learning performance
Many algorithms struggle with high-dimensional data (the curse of dimensionality).
Using PCA can:
- Reduce overfitting
- Speed up training
- Sometimes improve prediction accuracy
PCA might reduce those 10,000 features to 100 principal components while retaining most of the important information.
The model trains much faster with little loss in performance.
Simple analogy
Imagine a student who has 20 pages of notes before an exam.
Instead of memorizing every page, they create a 2-page summary that captures the most important concepts.
- Original notes = original features
- Summary = principal components
- Important ideas retained = explained variance
PCA performs a similar type of summarization for data.
Advantages
- Reduces dimensionality
- Removes correlated features
- Speeds up computation
- Helps visualize high-dimensional data
- Can reduce noise
- May improve machine learning models
Limitations
- Principal components are combinations of the original variables, so they are often harder to interpret.
- PCA assumes that directions with the greatest variance contain the most useful information, which is not always true.
- It captures linear relationships only, not complex nonlinear patterns.
- Features should usually be standardized first if they are on different scales.
In one sentence
PCA transforms a dataset into a smaller set of new, uncorrelated variables (principal components) that retain as much of the original variation as possible, making analysis, visualization, and machine learning more efficient.
