Big Data Analysis Techniques (Linear Regression, Dimensionality Reduction)

15 min readFeb 16, 2026

In the world of Big Data, raw information is only as valuable as the insights we can extract from it. Whether it’s managing petabytes of storage or predicting crop yields, the tools and techniques we use to analyze data are fundamental to decision-making.

This post breaks down the core methodologies of Big Data analysis, ranging from quantitative analysis to a deep dive into linear regression with a practical example.

Big Data Analysis Techniques (Linear Regression, Dimensionality Reduction)

1. Quantitative Analysis

Quantitative analysis focuses on quantifying patterns and correlations found within a dataset. It involves statistical techniques to analyze a large number of observations.

Goal: To interpret data through numerical results.

Key Metrics:

Classification Accuracy: How often a model is correct.
Correlation: The relationship between variables.
Signal-to-Noise Ratio (SNR): The level of the desired signal compared to the background noise.
Root Mean Squared Error (RMSE): A measure of the differences between values predicted by a model and the values observed.

2. Data Mining

Data mining refers to automated, software-based techniques that sift through massive datasets to identify hidden or unknown patterns and trends.

Role: It forms the basis for predictive analytics and Business Intelligence (BI).
Also known as: Pattern Recognition or Knowledge Discovery in Databases (KDD).
Function: It extracts patterns that are not immediately obvious, allowing for data-driven predictions.

3. Statistical Analysis

Statistical analysis uses mathematical formulas to analyze data, typically to infer patterns and relationships. It is quantitative in nature and includes several key methods:

A/B Testing (Split Testing)

A technique used to compare two versions of an element to determine which is superior based on a pre-defined metric.

Control Version (Group A): The current version of the element (e.g., an existing ad).
Treatment Version (Group B): The modified version (e.g., the ad with a slightly altered layout).
Process: Both versions are subjected to experiments, and observations are recorded to determine success.
Use Case: Most often used in marketing to optimize email campaigns or ads (e.g., measuring which “Buy Now” button design gets more clicks).

Correlation

A technique used to determine if and how two variables are related.

Purpose: To discover data patterns, hidden anomalies, and relationships that explain a phenomenon.

Measurement: The Pearson’s correlation coefficient ($r$) measures the degree of correlation.

Range: -1 (Strong negative correlation) to +1 (Strong positive correlation).
0: No correlation.
+1: Perfect positive linear relationship (as X increases, Y increases).
-1: Perfect negative linear relationship (as X increases, Y decreases).

Regression

Techniques that explore how a dependent variable (Y-axis) is related to an independent variable (X-axis).

Goal: To understand why a phenomenon occurred and to make predictions about future values.

Types:

Linear Regression: Represents a constant rate of change (a straight line).
Non-linear Regression: Represents a variable rate of change (a curved line).

4. Deep Dive: Linear Regression

Linear regression is one of the most fundamental algorithms in data analysis.

Assumptions

For linear regression to be valid, certain assumptions must be met:

The relationship between X and Y is linear.
Y is distributed normally at each value of X.
Homogeneity of variances: The variance of Y at every value of X is the same (constant rate).
The observations are independent.

The Equation

The relationship is expressed as a linear function:

Y: Dependent Variable (what we are trying to predict).
X: Independent Variable (the input).
α (Alpha): The Y-Intercept (value of Y when X = 0).
β (Beta): The Slope (rate of change in Y for a unit change in X).
ε (Epsilon): Random error (difference between actual and predicted values).

Least Squares Regression

The goal is to find the “Best Fit” line.

Because the line cannot pass through every point, there are “residuals” (differences between the actual Y and the predicted Y).
Method: We minimize the sum of squared errors (SSE).

Prediction Equation: ŷᵢ = β̂xᵢ + α̂
(y = mx + c)

Calculating Parameters

Slope (β̂):

(Alternatively: Covariance(x,y) / Variance(x))

Y-Intercept (α̂):

Y-Intercept

5. Practical Example: Optimizing Crop Yield

Let’s apply linear regression to a real-world scenario.

Problem Statement:
You are a Data Scientist helping farmers assess the amount of pesticide to be sprayed for better corn crop yield.

The Data:

Hypothesis: Is there a correlation between the amount of pesticide sprayed and crop yield?

Parameter Estimation Table:
To solve this, we calculate X², Y², and XY for each data point:

Row 1: X = 4, Y = 3.0 → X² = 16, Y² = 9.0, XY = 12
Row 2: X = 6, Y = 5.5 → X² = 36, Y² = 30.25, XY = 33
Row 3: X = 10, Y = 6.5 → X² = 100, Y² = 42.25, XY = 65
Row 4: X = 12, Y = 9.0 → X² = 144, Y² = 81.0, XY = 108

Totals (Sum):
∑X = 32
∑Y = 24.0
∑X² = 296
∑Y² = 162.5
∑XY = 218

Results: Using the formulas for slope and intercept, we derive the following model:

Slope (β): 0.65
- Interpretation: Crop yield is expected to increase by 0.65 lb for each 1 oz increase in pesticide sprayed.
Y-Intercept (α): 0.8
- Interpretation: The average crop yield is expected to be 0.8 lb when the pesticide sprayed is 0 oz.

Final Prediction Equation: Yield = 0.8 + 0.65(Pesticide)

In the realm of Big Data, we often deal with massive datasets containing hundreds or thousands of features. Managing this complexity requires robust techniques to reduce dimensionality without losing critical information. This post explores Principal Component Analysis (PCA), a mathematical transformation used to simplify complex datasets.

Principal Component Analysis (PCA)

1. The Challenge of High-Dimensional Data

Imagine a census of 10,000 dogs where you collect data on three attributes: Height, Length, and Weight.

These variables are often correlated (e.g., as height increases, weight usually increases).
If you know the weight, measuring the length might not provide much new information.
The Goal: Perform a mathematical transformation to turn these related variables into a set of unrelated variables (x, y, z), effectively reducing the data’s dimensions.

2. Understanding PCA

Principal Component Analysis (PCA) is an unsupervised learning technique used for dimensionality reduction.

It linearly projects high-dimensional data into lower dimensions while preserving important details.
It can be visualized as rotating existing axes to new locations in space defined by eigenvectors.

PCA chooses a projection that minimizes the mean squared error between the data and the projection, which simultaneously maximizes the variance of the projected data.

Eigenvectors (Principal Components (PC)): These define the new axes. They are mutually orthogonal (perpendicular) and represent directions of maximum variability.
PC #1: Points in the direction of the largest variance.
Subsequent PCs: Point in the direction of the largest variance of the residual subspace and are orthogonal to previous PCs.

To compute the Principal Components (PC), we follow a mathematical process:

Center the Data: Subtract the mean along each dimension so the axis system is centered at the centroid.
Covariance Matrix (C): Calculate C = x·xᵀ. This matrix captures the similarities of the original axes based on data projections.
Maximize Variance: We look for a unit vector u that maximizes uCuᵀ.
Eigenvalues (λ): The maximization problem leads to Cu = λu, where u is the principal eigenvector and λ (eigenvalue) denotes the amount of variability captured along that dimension.

3. A Computational Example

Consider data x = {x₁, …, xₘ}ᵀ, compute the principal components of the data.

Step 1: Find the Covariance Matrix (C)

Step 2: Solve for Eigenvalues (λ)
Find the determinant of (C — λ I) = 0:

Step 3: Solve for Eigenvectors (u)

Result:
The matrix $U$ containing the eigenvectors is:

4. Singular Value Decomposition (SVD)

SVD is closely related to PCA and provides an alternative way to decompose the data.

Formula: X = UΣVᵀ

U and V give the left and right singular vectors (eigenvectors).
Σ (Sigma) gives the eigenvalues (scaling factor).

Optimality: By the Eckart–Young theorem, SVD is optimal in the mean squared error sense.

Complexity: The computational complexity is O(MN²) (assuming N < M).

5. Real-World Applications

PCA is not just theoretical; it has powerful practical applications.

Face Recognition (Eigenfaces)

Eigenvectors (V₁ … Vₖ) span the “space of faces”.
A face image is converted into eigenface coordinates:
x → ((x − x̄) · v₁, …, (x − x̄) · vₖ)

This allows a face to be reconstructed as an average face plus a weighted sum of eigenfaces.

Hyperspectral Image Classification

Scenario: Analyzing images of the Washington Mall from the HYDICE sensor, which contains 191 bands of data.

Results: PCA significantly reduces data volume while maintaining accuracy.

Original (50 bands): 100 seconds to classify, 46% correct rate.
PCA (Top 25 bands): 16 seconds to classify, 92.05% correct rate.
PCA (Top 5 bands): 5 seconds to classify, 77.24% correct rate

6. Choosing ‘K’ and Shortcomings (Elbow Method)

How many Principal Components should wekeep?

Get Prajun Trital’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Strategy: Choose the first K eigenvectors based on their highest eigenvalues. If the eigenvalues for dropped dimensions are small, we don’t lose much information.

Plot for Principal Components and Variance. The **elbow point** is the number of components after which adding more PCs yields **diminishing returns** in variance explained. For this data, the elbow is at **PC3**, so we can reduce the dataset from *n dimensions* to **3 principal components** safely.

The elbow method helps to decide how many PCs to keep:

Look at the plot.
Find the point where the slope of the curve changes sharply (the “elbow”). Here for the above plot, PC1 and PC2 explain a large portion of the variance. After PC3, adding more components doesn’t improve the explained variance much.
Keep all components up to that point because after that, each extra component adds little extra explained variance. So, in practice, we might choose 3 principal components (for the above figure) to reduce the dimensionality of the dataset while still keeping most of the information.

Shortcomings:

Class Labels: PCA does not distinguish between class labels; it is purely unsupervised.
Nonlinear Structures: PCA cannot capture nonlinear data structures.
Assumption: It implicitly assumes that the data distribution is Gaussian.

PCA: Conclusion
1. Assumes data is approximately Gaussian.
2. Finds an orthonormal basis for the data.
3. Orders dimensions (principal components) by variance.
4. Low-significance dimensions/components can be discarded.
Advantages
1. Compacts data while preserving most information.
2. Reduces noise (denoising).
3. Improves data separability → better classification.
Disadvantages
1. Ignores class labels (unsupervised).
2. Cannot capture nonlinear relationships in data.
3. Computationally expensive: O(min(N³, M³))

Learn more about PCA, Dual PCA, and Kernel PCA here.

Curvilinear Component Analysis (CCA)

When data isn’t linear, PCA fails. This is where CCA comes in.

Definition: CCA is a non-linear extension of PCA.
Key Function: It looks for a non-linear projection of the data.
Topology Preservation: It preserves the proximity between points in the input space (the local topology of the distribution).
Use Case: It enables unfolding varieties in input data, such as unrolling a “Swiss roll” or “Horseshoe” shape.

Independent Component Analysis (ICA)

While PCA looks for uncorrelated components (orthogonal coordinates), ICA looks for independent components (non-orthogonal coordinates).

The Cocktail Party Problem

ICA is best explained via the “Cocktail Party Problem”.

Scenario: Multiple sound sources in a room (speakers) are independent. Multiple sensors (microphones) receive signals that are mixtures of these original signals.

Goal: Estimate the original source signals from the mixture, also known as Blind-Source Separation (BSS) because the mixing parameters are unknown.

Mathematical Model: x = As.

s: Vector containing independent source signals.
A: The mixing matrix.
x: The observed (mixed) signals.

How Fast ICA Works

The algorithm (FastICA) relies on the Central Limit Theorem, which suggests that a mixture of independent variables is more Gaussian than the original variables. Therefore, to separate them, we must find the projection that is the least Gaussian.

Step 1: Remove correlations (whiten the data) via rotation and scaling.
Step 2: Find the transformation y = Vx.
Step 3: Maximize the marginal densities (non-Gaussianity) to recover the original signals.
Complexity: O(N²Mα), where α is the number of iterations, N is the dimensions and M is the samples.

Linear Discriminant Analysis (LDA)

Unlike PCA and ICA, LDA is a supervised method. It cares about class labels.

Goal: Project data in directions that are efficient for discriminating between different classes.
Mechanism: LDA maximizes the Between-Class Scatter (S_B) and minimizes the Within-Class Scatter (S_W).
It wants examples from the same class projected very close together.
It wants the means of different classes projected as far apart as possible.

The Fisher Criterion

To find the optimum projection, LDA maximizes the Fisher criterion function J(w):

Solving it: This becomes a generalized eigenvalue problem:
S_W^{-1} S_B w — Jw = 0.

Extension: LDA can be extended to C classes, producing at most C-1 feature projections.

PCA vs. LDA: Which one to use?

The choice between PCA and LDA depends heavily on your dataset and your goal.

Principal Component Analysis (PCA):

Focus: Preserves as much variance in the high-dimensional space as possible.
Type: Unsupervised (ignores class labels).
Best For: Small training sets.
Statistics: Relies on Second-order statistics (uncorrelated, Gaussian).

Linear Discriminant Analysis (LDA):

Focus: Preserves as much class discriminatory information as possible.
Type: Supervised (uses class labels).
Best For: Large number of samples.
Complexity: O(N²M) (assuming N < M; N: dimensions, M: samples).

Limitations of LDA

Non-Gaussian Distributions: LDA is parametric and assumes unimodal Gaussian likelihoods. If the distribution is significantly non-Gaussian, LDA may fail to capture the complex structure.
Variance vs. Mean: LDA fails when the discriminatory information is in the variance of the data rather than the mean. If two classes have the same mean but different variances (e.g., one ring inside another), LDA cannot separate them.

The Challenge of Data Sampling

Traditional signal processing relies on establishing a specific sampling rate to ensure a signal can be perfectly reconstructed. However, in the era of Big Data, adhering to these traditional limits can result in unmanageably large datasets.

Nyquist’s Rate: The traditional theorem states that for perfect signal recovery, the sampling frequency (fₛ) must be at least twice the highest frequency of the signal (f_b).
- Formula: fₛ ≥ 2f_b
The Big Data Problem: Adhering to the Nyquist rate generates massive amounts of data (Petabytes/Terabytes) that are difficult to store, search, and analyze.
The Solution: Compressed Sensing (CS) allows for perfect signal recovery with a sampling rate significantly lower than the Nyquist limit (fₛ < 2f_b) by using convex optimization.

Compressed Sensing (CS) Fundamentals

Compressed Sensing is a technique where a high-dimensional signal is reduced to a lower dimension using a measurement matrix, yet the original signal can still be recovered perfectly.

Dimensionality Reduction: A signal x with N dimensions is reduced to a signal y with K dimensions, where K ≪ N.

Conditions for Validity

For Compressed Sensing to work, the signal must meet specific criteria regarding its structure and the way it is sampled.

1. Sparsity

The signal must be compressible or sparse. If the signal x is not sparse in the time domain (e.g., an ECG signal), it can be transformed into a domain where it is sparse (for example, using the Discrete Cosine Transform (DCT)).

Transformation: x = Ψs, where Ψ is the basis matrix, and s is the sparse representation.
Combined Equation: y = ΦΨs

2. Incoherence

The sampling matrix Φ and the basis matrix Ψ must be dissimilar (low coherence).

Coherence Formula: μ(Φ, Ψ) = √n · max |⟨φₖ, ψⱼ⟩|
Requirement: Low coherence (value close to 1) is desired. High coherence indicates similarity, which is bad for Compressed Sensing.

The Sampling Matrix (Φ)

The choice of the sampling matrix is critical. It projects the data into a lower-dimensional space while preserving essential information.

Randomness is Key:
Deterministic bases often fail the incoherence test.
Random matrices generally provide low coherence with deterministic bases.
Types of Matrices:
- Sparse Measurement Matrix: Example: Hadamard matrix. Computationally more efficient.
- Dense Measurement Matrix: Example: Gaussian matrix. More general but computationally heavier.
Recovery Condition:
A signal can be exactly recovered with high probability if the number of samples K satisfies: K ≥ C · μ²(Φ, Ψ) · S · log N , where S is sparsity, and N is signal length

Signal Recovery Algorithms

Once the data is compressed into y, algorithms are required to solve the underdetermined system and recover x.
Since there are fewer equations than unknowns, the sparsest solution is sought.

L1 Minimization (Basis Pursuit)

Optimizes the L1 norm (sum of absolute values) instead of the L2 norm (energy).
Objective:
Minimize ‖x*‖₁ subject to Ax* = y
Solvable in polynomial time using linear programming.

Greedy Algorithms (Matching Pursuit)

Iterative methods that approximate the signal by selecting the most significant components step by step.
Examples: OMP, ROMP, StOMP, CoSaMP
Process: In each iteration:
- Compute approximation error Δ
- Sparsify (set small coordinates to 0)
- Update the estimate
Popular Software / Methods:
- GPSR (Gradient Projection for Sparse Reconstruction)
- L1-magic
- SparseLab

Applications of Random Projections (RP)

Beyond signal recovery, Random Projections are used for dimensionality reduction in machine learning and hardware design.

Single Pixel Camera: Uses a Digital Micromirror Device (DMD) to encode an image using a random basis, allowing for image capture with a single photodiode.
Hyperspectral Classification: Projecting high-dimensional spectral data into lower dimensions for faster classification using Support Vector Machines (SVM).
Data Stream Algorithms & Pooling: Efficiently handling massive streams of data.

Random Projection vs. Transform Methods

Random Projections (data-independent) are compared with transform-based methods such as Singular Value Decomposition (SVD) (data-dependent).

Transform-Based Methods (e.g., SVD)

Pros: Highly accurate representation.
Cons: Data-dependent, computationally expensive with complexity O(MN²).

Random Projections (RP)

Pros: Data-independent, computationally efficient, simple implementation.
Cons: Approximate representation (usually sufficient).
Mechanism: Preserves Euclidean distance (E(yyᵀ) = xxᵀ). This allows algorithms like SVM to operate directly on compressed data.

Hybrid Approach: Fast Approximate SVD (FSVD)

Concept: Combines Random Projections with Feature Selection.
Benefit: Achieves higher classification performance than standard SVD while significantly reducing computational cost.

Computational Complexity & Matrix Types

The efficiency of dimensionality reduction depends heavily on the random matrix type used.

Gaussian Matrix (GM)

Type: Dense matrix
Entries: Independent and identically distributed 𝒩(0, 1) random variables
Complexity: O(MNK)

Hadamard Matrix (HM)

Type: Sparse / Structured matrix
Definition:
H₁ = [1], and

Complexity:
O(MN log K) (much faster than Gaussian)

Experimental Results: Hyperspectral Imaging

The document presents a case study comparing these methods using two major datasets: Indian Pines (AVIRIS sensor) and Pavia (ROSIS sensor).

Datasets

Indian Pines: 145 × 145 pixels, 224 spectral bands, 16 classes
Pavia: 610 × 340 pixels, 103 spectral bands, 9 classes

Performance Comparison (Accuracy & Speed)

Spectral Angle: FSVD methods (both GM and HM) approximate the original eigenvectors very closely (spectral angle near 0 degrees) compared to pure random sampling.
Accuracy (Indian Pines):
- HM-FSVD: Achieves ~86.36% accuracy.
- GM-FSVD: Achieves ~84.59% accuracy.
- Pure HM: ~84% (lower than FSVD).
- SVD: ~84.86%.
Computation Time (Pavia Dataset Example):
- SVD: Takes ~3300+ seconds.
- HM/GM Methods: Take fractions of a second (e.g., 0.003s for HM, 0.07s for GM-FSVD).

This represents a massive speedup.

Conclusion of Experiments

Random Projection using Feature Selection (FSVD) offers a “structured two-fold dimensionality reduction”.
It provides classification accuracy comparable to or better than SVD while being orders of magnitude faster.

🔙 Back to all notes
𝕏 Let’s Connect

Backlinks:
Intro to Big Data Computing