Supervised Learning Techniques
In the era of Big Data, we aren’t just dealing with massive storage capacities (petabytes and terabytes); we are dealing with complex analytics, social networks, and real-time sensor data. One of the most fundamental tools in the data scientist’s toolkit for making sense of this data is Classification.
This guide breaks down the core concepts of supervised classification, probabilistic modeling, and validation techniques, moving from intuitive examples to mathematical algorithms.
1. The Intuition: Sorting Fish (Classification in Action)
To understand classification, let’s look at a practical problem: automating the sorting of fish on a conveyor belt into two categories: Salmon and Sea Bass.
The Goal: Distinguish species based on visual data captured by a camera.
The Process:
- Capture Image: Get raw data.
- Segmentation: Isolate the fish from the background.
- Feature Extraction: Calculate measurements like length, width, lightness, and shape.
- Decision: Assign a class label (Salmon or Sea Bass).
Challenges:
Noise: Observation errors can occur due to lighting, camera noise, or the fish’s position.
Feature Overlap:
- Length: Using length alone is imperfect. While Sea Bass are generally longer, there is a significant overlap in the histograms of the two populations.
- Lightness: Sea Bass are generally lighter/brighter, but overlap still exists.
Decision Boundaries:
- An ideal boundary would perfectly separate the classes, but this is impossible in practice.
- Complex boundaries might fit the training data perfectly, but fail on new data (overfitting).
- Real-world boundaries are always a trade-off between performance (accuracy) and the complexity of the classification model.
2. The Math: Supervised Probabilistic Classification
To move beyond simple thresholds, we use probability theory.
Probability Basics
- Prior Probability (P(x)): The probability of an event occurring before seeing any new evidence.
- Conditional Probability (P(x│c)): The probability of x given that c has occurred.
- Joint Probability (P(x, c)): The probability of both occurring together.
- Independence: If two variables are independent, P(x₁, x₂) = P(x₁)·P(x₂).
Bayes Rule
The foundation of probabilistic classification is Bayes Rule
- Posterior (P(c|x)): The probability of the class c given the feature x.
- Likelihood (P(x|c)): How likely the feature x is seen in class c.
- Prior (P(c)): How common class c is overall.
- Evidence (P(x)): The probability of the feature occurring generally.
Maximum A Posteriori (MAP) Classification
- For an input x, we want to find the class c that maximizes the posterior probability.
- We assign x to the class c* if P(c*|x) is the largest among all possible classes.
- Since the denominator P(x) is common to all classes, we only need to maximize the numerator: P(x|c)P(c).
3. The Naïve Bayesian Classifier
Calculating the full joint probability P(x₁, …, xₙ │ c) is computationally infeasible because it requires massive amounts of data to represent every possible combination of features.
The “Naïve” Assumption
We assume that all input features are conditionally independent given the class label.
This reduces the problem to estimating simple 1D distributions for each feature independently.
The Algorithm
- Learning Phase (Training):
- Estimate the prior probability P(cᵢ) from the fraction of each class in the training set.
- Estimate the conditional probabilities P(xⱼ │ cᵢ) for every feature value given every class.
- Output: A set of “Look-up tables” (Generative models).
2. Test Phase (Prediction):
- For a new instance, look up the probabilities for each of its features in the tables.
- Multiply them together with the class prior.
- Pick the class with the highest score.
Learn more about the example here.
4. Validation: How Good is the Model?
We cannot evaluate a model on the same data used to train it. We need specific splits.
The Data Split
- Training Set: Used to learn the parameters.
- Validation Set: Used to tune the architecture and estimate error during development.
- Test Set: Used only once at the end to assess performance. It provides an unbiased estimate of generalization error.
- Note on Overfitting: As training continues, training error drops, but validation error may eventually rise. We stop where the validation error is lowest.
Cross-Validation
To make the best use of limited data, we use cross-validation.
k-fold Cross-Validation:
- Split data into k equal subsets.
- Use one subset for testing and the rest for training.
- Repeat k times and average the results.
Leave-One-Out (LOO):
- Set k equal to the number of training instances (n).
- Build the classifier n times.
- Pros: Makes best use of data; no random subsampling.
- Cons: Very computationally expensive.
Confusion Matrix
When validating, we categorize predictions into four buckets:
- True Positive (TP): Correctly identified a positive (e.g., correctly found an edge).
- True Negative (TN): Correctly identified a negative.
- False Positive (FP): Incorrectly labeled as positive.
- False Negative (FN): Incorrectly labeled as negative
Learn more about model evaluation, ROC Analysis, Cross-Validation, formulas for sensitivity, and specificity here.
Motivation: Dimensionality Reduction Revision
Earlier dimensionality reduction techniques include:
- Principal Component Analysis (PCA)
- Linear Discriminant Analysis (LDA)
Common Trait
Both PCA and LDA:
- Rely on linear combinations of input features
- - Use a model of the form:
f(x) = wᵀx
Limitation
- Linear methods may fail when data is not linearly separable
- This motivates the extension to nonlinear spaces
Understanding Feature Space
To handle nonlinear patterns, data is mapped into a feature space (F) using a nonlinear transformation.
Types of Feature Spaces
- High-dimensional space
- Infinite-dimensional (countable) space
- Function space (Hilbert space)
Nonlinear Mapping
A mapping function: Φ : ℝᵈ → F
Example transformation:
- Input: (x, y)
- Feature mapping: (x², y², xy)
This allows linear algorithms to operate on nonlinear data.
Kernel Methods: Core Idea
Instead of explicitly computing Φ(x), kernel methods compute: k(x, y) = ⟨Φ(x), Φ(y)⟩
Benefit
- Avoids explicit computation in high-dimensional space
- Reduces computational complexity
- Enables nonlinear learning using linear algorithms
What Is a Valid Kernel?
A kernel is a symmetric, positive semi-definite function such that:
- For any finite set of points, the resulting Gram matrix is positive semi-definite
- For all vectors α:
αᵀKα ≥ 0
Kernel Representation Theorem
A function k(x, y) is a valid kernel if and only if: k(x, y) = ⟨Φ(x), Φ(y)⟩
Importance
- Validity can be checked without knowing the feature map
- Only properties of the kernel function are required
Mercer’s Theorem
Conditions
If:
- The input space X is compact
- k(x, y) is symmetric and continuous
- The kernel operator is positive semi-definite
Then:
- There exists an orthonormal eigenfunction expansion:
Any function satisfying Mercer’s conditions is a valid kernel.
Enables implicit work in infinite-dimensional spaces.
Kernel Methods Framework
Kernel methods consist of:
- Choosing a kernel
- Applying a kernel-based algorithm
Modularity Property
- Any valid kernel can be used with any kernel algorithm
- Kernel choice and learning algorithm are independent
Popular Kernel Algorithms
- Kernel PCA (KPCA)
- Kernel Discriminant Analysis (KDA)
- Support Vector Machines (SVMs)
The Kernel (Gram) Matrix
The kernel matrix K is constructed as: Kᵢⱼ = k(xᵢ, xⱼ)
Properties
- Symmetric
- Positive semi-definite (if kernel is valid)
- The only information used by kernel-based learning algorithms
How to Choose a Kernel?
There are no absolute rules. Kernel choice depends on the problem.
Design Considerations
- Incorporate prior (domain) knowledge
- Preserve meaningful data structure
- Balance between:
- Simpler structure (e.g., linear separability)
- Information preservation
Practical Insight
- Kernel methods are often robust to kernel choice
- Experimental validation is essential
Building New Kernels
Valid kernels can be constructed using:
- Linear combinations
- Products
- Scalar multiplication
- Functional transformations
Validity Preservation
- Operations preserve symmetry and positive definiteness
- Enables flexible kernel engineering
Support Vector Machines (SVMs)
SVMs are powerful classifiers that find the optimal hyperplane to separate different classes of data.
Get Prajun Trital’s stories in your inbox
Join Medium for free to get updates from this writer.
Linear SVM Model: g(x) = wᵀΦ(x) + w₀
Classification Rule
- g(x) > 0 → Class w₁
- g(x) < 0 → Class w₂
Linearly Separable Case
Margin
Defined as the minimum distance from the decision boundary to the nearest data point
Optimization Objective
Minimize: ½‖w‖²
Subject to: tᵢ(wᵀxᵢ + w₀) ≥ 1
Key Insight
- Minimizing ‖w‖ is equivalent to maximizing the margin
- Points on the margin are support vectors
Nonlinearly Separable Case
Slack Variables
Introduced to allow misclassification: tᵢ(wᵀxᵢ + w₀) ≥ 1 − ξᵢ
Where: 0 ≤ ξᵢ ≤ 1
New Objective
Minimize: ½‖w‖² + C Σᵢ₌₁ᴺ ξᵢ
Role of C
- Controls trade-off between margin size and classification error
- Larger C → stricter classification
- Smaller C → more flexibility
Kernel Trick in SVMs
Challenge
- Finding a high-dimensional space where data is linearly separable
- Avoiding computational complexity
Solution
Using kernels: K(xᵢ, xⱼ) = ⟨Φ(xᵢ), Φ(xⱼ)⟩
Result
- Benefits of high-dimensional space
- Computation remains in the original space
Learn more deatil about the math behind the SVM here.
Experimental Case Study: The “Adult” Dataset
To demonstrate these concepts, SVMs were applied to the UCI Adult Dataset (Census Income), which predicts if an individual’s income exceeds $50K/year.
Observations
- Polynomial and RBF kernels often outperform linear kernels
- Higher CCC generally increases complexity
- RBF kernel shows strong performance, but a higher SV count
- Linear kernel remains competitive with fewer support vectors
Key Takeaways:
- Robustness: There is often experimental robustness in kernel choice when the kernel strikes a trade-off between simplicity (linear separability) and preserving the information structure.
- Higher Dimensions: Data points generally become more separable in higher dimensions, reducing complex problems to linearly separable cases.
- Kernel methods enable nonlinear learning using linear models
- Valid kernels must satisfy positive semi-definiteness
- Mercer’s theorem provides theoretical grounding
- SVMs maximize margin for robust classification
- Kernel choice significantly affects performance and complexity
- Real-world datasets benefit from nonlinear kernels
Attention Models
In most deep learning systems, spatial and spectral features extracted from data are treated equally — every feature gets the same weight in the final analysis. But does that make sense? Not all features are created equal. Some carry rich, discriminative information, while others contribute very little. This is the central problem that Attention Models aim to solve. This note walks through the motivation for attention, two attention module architectures, a combined system, and two sets of experimental results.
Key Drivers for Adopting Attention Models:
- Selective Discrimination: Not all spatial (where things are) and spectral (what they are made of) features contribute equally to an analysis.
- Informative Feature Prioritization: Methodology is required to distinguish between highly informative and less informative data points.
- Complexity Management: Attention mechanisms allow models to focus on specific parts of a large dataset, mirroring human cognitive focus.
Part 1: Spectral Attention Module
The Spectral Attention Module is designed to selectively weight spectral bands based on their informative value. It is built on a Bidirectional LSTM (Long Short-Term Memory) architecture.
Architecture Overview:
- Input: A hyperspectral cube of shape 11 × 11 × P (spatial patch with P spectral bands).
- 3D Convolution (3DConv): The input is first processed through a 3D convolutional layer, reshaping the data into a sequence of shape L × 1 (one spectral vector per band).
- Bidirectional LSTM Layer 1: Processes the spectral sequence in both forward and backward directions to capture dependencies across bands. Each time step’s output is multiplied (⊗) by an attention weight.
- Bidirectional LSTM Layer 2: A second BiLSTM layer processes the weighted features further.
- Softmax Activation: Generates attention weights (values between 0 and 1) that indicate the importance of each spectral band.
- Attention Mechanism: The softmax output is multiplied (⊗) with the original features, and the result is added (⊕) via a residual/skip-style connection.
- Output: A 50-Dimensional Embedding produced by a fully connected layer — a compact, attention-weighted spectral representation.
“The architecture of the proposed bi-directional LSTM-based spectral attention module”
Part 2: Spatial Attention Module
The Spatial Attention Module focuses on where in the spatial domain the most informative regions are located. It uses a 3D-CNN-based attention mechanism combined with a ResNet-based feature extractor.
Architecture Overview:
- Input: A hyperspectral patch of shape 11 × 11 × P.
- 3DCNN-Based Attention Mechanism:
- A Conv3D layer processes the input.
- A Softmax activation generates an Attention Map — a spatial grid highlighting which pixels deserve more focus.
- The Attention Map is multiplied (⊗) with the original input, and the result is added (⊕) — again using a residual connection to preserve original information. - ResNet Block 1 (with Skip Connection):
- Passes through multiple Conv2D layers and Batch Normalization layers.
- Includes a skip connection that adds the input directly to the block’s output, helping prevent vanishing gradients. - ResNet Block 2 & ResNet Block 3:
- Further deep feature extraction using residual connections.
- Followed by ReLU activation and Average Pooling. - Output: A 50-Dimensional Embedding via a fully connected layer — a compact, attention-weighted spatial representation.
“The architecture of the proposed 3D-CNN-based spatial attention and ResNet-based feature extraction module”
Proposed System Architecture: SPAT-SPEC-HYP-ATTN
The two modules above are combined into a unified framework called SPAT-SPEC-HYP-ATTN (Spatial-Spectral Hyperspectral Attention Network).
End-to-End Pipeline:
- Input: A raw Hyperspectral Cube of shape M × N × D (spatial dimensions M×N, D spectral bands).
- PCA (Principal Component Analysis): Dimensionality reduction is applied to compress the spectral dimension from D to P components, yielding a reduced cube of shape M × N × P.
- Dual-Branch Architecture: The PCA output is fed simultaneously into two parallel modules:
- Spatial Attention Module → produces a 50-Dimensional Embedding capturing spatially-weighted features.
- Spectral Attention Module → produces a 50-Dimensional Embedding capturing spectrally-weighted features. - Feature Fusion: The two 50-dimensional embeddings are concatenated (⊕) into a single 100-dimensional combined feature vector.
- Fully Connected Neural Network: The fused embedding is passed through a fully connected network for final land cover / class prediction.
“The proposed spatial-spectral attention-based classification framework (SPAT-SPEC-HYP-ATTN)”
Experimental Results: Classification Maps — Indian Pines Dataset
The SPAT-SPEC-HYP-ATTN framework was evaluated on the Indian Pines hyperspectral benchmark dataset and compared against several state-of-the-art methods.
- The combined spatial+spectral attention model (SPAT-SPEC-HYP-ATTN) achieves the highest accuracy, visually producing maps closest to the Ground Truth.
- Using only spatial attention (SPAT-HYP-ATTN) still performs well at 94.06%, while spectral-only (SPEC-HYP-ATTN) drops significantly to 83.86% — highlighting that spatial features are particularly important for this dataset.
Proposed System Architecture: HYPER-VIT
Beyond attention-based CNNs, the lectures also introduce HYPER-VIT — a Vision Transformer (ViT)-based framework for hyperspectral image classification.
End-to-End Pipeline (Part a):
- Input: Hyperspectral cube of shape M × N × D.
- PCA: Reduces spectral dimensionality to P components → shape M × N × P.
- Windowed Inputs: The PCA output is divided into overlapping spatial patches of shape 9 × 9 × P.
- Data Oversampler Module: Addresses class imbalance common in hyperspectral datasets by oversampling underrepresented classes.
- Data Augmentation Layer: Applies transformations to increase data diversity and improve generalization.
- Segmented Image Patches with Positional Embeddings: Each 9×9 patch is tokenized and assigned a positional embedding (numbered 1–9 in a 3×3 grid) so the transformer can understand spatial relationships.
- Three Transformer Modules (in sequence): Each module processes the patch tokens via self-attention.
- Layer Normalization: Applied after the transformer stack to stabilize training.
- Fully Connected Neural Network: Final classification head producing class predictions.
Internal Transformer Module Structure (Part b):
Each of the three transformer modules follows this internal structure:
- Final Preprocessed Input → feeds into the module.
- Layer Normalization → normalizes the input.
- Multi-Head Attention → the core self-attention mechanism; learns relationships between all patch tokens simultaneously across multiple attention heads.
- Residual (Skip) Connection (+) → adds the attention output back to the original input.
- Layer Normalization → normalizes again.
- MLP (Multi-Layer Perceptron) → a feed-forward network for non-linear transformation.
- Residual (Skip) Connection (+) → adds the MLP output back.
- Output → passed to the next transformer module or final classification head.
Resources
For further reading and hands-on practice with neural networks and deep learning:
- TensorFlow Playground — interactive browser-based neural network visualizer: playground.tensorflow.org
- PyTorch Tutorial — official tutorials for getting started with PyTorch
- Multi-Layer Perceptron using FastAI and PyTorch — practical MLP implementation guide
- Multilayer Perceptron — conceptual deep dive into MLP architectures
- Implementing Deep Neural Networks Using PyTorch — hands-on implementation walkthrough
