Unsupervised Learning Techniques

In the realm of Big Data, we often find ourselves with massive datasets that lack predefined labels. While supervised learning relies on “known” data to train algorithms, the sheer volume and variety of modern data frequently require Unsupervised Learning — where the algorithm must discover intrinsic structures and statistics on its own.

Unsupervised Learning Techniques

1. The Learning Spectrum

Understanding the different approaches to machine learning is the first step in Big Data processing:

• Supervised Learning: Uses labeled pairs $(x, y)$ to learn a function that maps inputs to outputs.
• Unsupervised Learning: Operates without prior training or labels, exploring data to find underlying structures.
• Semi-supervised Learning: A middle ground where an algorithm is trained on a small set of labeled data and then applied to a much larger unlabeled dataset, often because labels are expensive to obtain.

2. What is Clustering?

Clustering is the unsupervised grouping of data points and is often considered synonymous with unsupervised learning in the context of Big Data Analytics.

Bezdek’s example

Bezdek: Number of clusters for different initializations

• The Goal: Group data instances that are “similar” or “near” to each other in one cluster, while ensuring instances in different clusters are “very different” or “far away”.
• The Subjective Nature: As demonstrated by Bezdek’s examples, the “correct” number of clusters often lies in the “eye of the beholder”. Depending on the initialization or the specific question being asked, 30 unlabeled points could be seen as two, three, or even six distinct clusters.
• Numerical Taxonomy: Historically, clustering has been used for “Convenience” (organizing data) or “Natural Classification” (based on similarity or phylogenetic relationships).

Cluster lies in the eye of the beholder!!!

3. The Three Fundamental Questions

When approaching a dataset X, there are three critical questions to ask:

1. Does X have a cluster substructure? This involves a pre-clustering assessment of “clustering tendency” using tools like VAT or coVAT.
2. How can we find the clusters? This is the cluster analysis phase using algorithms like K-Means (HCM), Fuzzy C-Means (FCM), or MeanShift.
3. How can we validate them? This post-clustering phase uses validity indices to ensure the results are meaningful

4. Mathematical Foundations: Distance and Membership

To perform clustering, we must define how data points relate to one another.

Proximity Metrics

• Dissimilarity (Distance): Common metrics include Euclidean distance, Manhattan distance, Max distance, and Mahalanobis distance.
• Similarity: Often measured via Inner Product/Cosine similarity or the Tanimoto coefficient.
• Metric Properties: A valid metric must satisfy non-negativity, identity (d(x,y) = 0 iff x=y), symmetry, and the triangle inequality.
• Clustering quality:
  Inter-cluster distance => maximized
  Intra-cluster distance => minimized

The quality of a clustering result depends on the algorithm, the distance function, and the application.

The Membership Matrix (U)

A membership matrix U tracks which sample M belongs to which cluster C.

The Membership Matrix (U)

• Crisp/Hard: A point belongs to exactly one cluster (uᵢⱼ ∈ {0, 1}).
• Fuzzy/Probabilistic: A point has a degree of membership in multiple clusters, summing to 1.
• Possibilistic: Membership is greater than 0 but does not necessarily sum to 1.

This Membership Matrix U represents a crisp or hard partition of a dataset X containing five samples (x₁ through x₅) into three distinct clusters (C₁, C₂, C₃). In this format, each entry uᵢⱼ is binary: a 1 indicates that a specific sample belongs to a cluster, while a 0 means it does not. When we sum across a row, the result represents the total number of elements contained within that specific cluster. Conversely, because every sample must belong to exactly one cluster in a hard partition, the sum of each column must always equal 1, ensuring that no data point is left unassigned or duplicated across multiple groups.

5. K-Means Clustering: The Partitional Approach

K-Means is an exclusive, partitional algorithm that searches for compact, well-separated clusters.

A cluster is represented by a single point, known as Centroid (or cluster center) of the cluster.

The Process:

1. Initialize: Randomly choose K cluster centroids.
2. Assign: Every data point finds its closest centroid and is assigned to that cluster.
3. Update: Re-estimate the centroids by calculating the mean of all points currently in that cluster.
4. Repeat: Loop until the centroids converge

Pros and Cons

• Advantages: Simple and fast to compute.
• Disadvantages: Very sensitive to initial points and outliers. It only detects spherical clusters and requires the user to manually specify K.

6. Hierarchical Clustering: Trees of Data

Hierarchical clustering builds relations between clusters, often visualized through a Dendrogram.

• Agglomerative (Bottom-up): Starts with every element as its own cluster and merges the most similar pairs until only one root cluster remains.
• Divisive (Top-down): Starts with all points in one root cluster and recursively splits them until only singleton clusters remain.

Pros: Excellent for visualization and capturing concentric clusters. Cons: Difficult to define the “correct” level to cut the tree, and often outperformed by other techniques in accuracy.

7. Cluster Validity: Are the Results Correct?

Validity indices provide a quantitative way to evaluate clustering results. Challenges include dealing with clusters of different sizes or varying levels of separation.

Ultimately, no single index is always “correct.” The choice depends entirely on the specific question the analyst is trying to answer.

Get Prajun Trital’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

🔙 Back to all notes
𝕏 Let’s Connect

Backlinks:
Intro to Big Data Computing