Unsupervised learning: Clustering
Unsupervised learning does not require response variables. It aims to explore the data structure and discover potential structures without a reference to an outcome. We don’t aim to predict or forecast.
Therefore, it is very hard to evaluate the performance of unsupervised learning. Since there are no response variables, unsupervised learning does not have simple accuracy measures. However, it can be assessed with internal criteria, visualisations and domain knowledge.
The article focuses on two key ideas: exploration and subjectivity.
- Exploration: How to visualise high-dimensional data, how to summarise complex relationships
- Subjectivity: how the analytical choices I make influence the patterns from the exploration.
Exploration
Clustering: finding subgroups
Partition the data into distinct groups, within which observations in each group are homogeneous, while the other groups are heterogeneous.
The goal of clustering:
- Cluster the observations with basis features to find subgroups among observations.
- Cluster the features based on observations to get subgroups with the same features.
The two commonly used clustering methods are:
K-means clustering
Hierarchical clustering
- K-means clustering: partition the observations into a pre-specified number of clusters.
1st step: to define the most desirable number of cluster K
Let be the containing the indices of the observations of each cluster, it satisfies two properties:
(1) The observations belong to at least one of the K clusters.
(2) The clusters are non-overlapping ( no observation belongs to more than one cluster)
The objective is to minimise the total within-cluster variation.
The optimisation sometimes depends on the choice of the objective functions, distance metric or predetermining the number of cluster K.
Algorithm :
Since the K-means algorithm seeks a local minimum, this does not necessarily mean a global minimum. This is why we need to perform multiple K-means clustering in practice with different random starts and choose the minimum objective value.
K- means is easy and fast, but the result depends on where you start.
2. Hierarchical clustering: produces a tree-like visual representation of the observations (dendrogram), which shows the clusters obtained for each possible cluster. The most common ways are bottom-up or agglomerative ( single linkage) clustering. In hierarchical clustering, “nested” means that clusters are organised like a set of Russian nesting dolls.
Hierarchical clustering is an alternative approach that does not require committing to a particular choice of K.
Hierarchical clustering considers full Euclidean distance, not only horizontal distance.
The attractive feature of hierarchical clustering is that a single dendrogram can be used to get multiple clusters. The number of clusters is determined by making a horizontal cut across the dendrogram at a chosen height.
However, the ‘nesting’ assumptions of hierarchical clustering are not always realistic. For example, you want to differentiate groups by gender and nationality. If the first split group of people is evenly split by gender (Men/Women), then the later split should occur along the gender groups. But more likely, the second split is done by nationality (American/Japanese/French). Therefore, the true structure is ‘cross-cutting (partitioning)’ rather than ‘nesting’.
When the algorithm tries to find the best division into two groups, it will likely split them by gender. If it is, then forced to find the best division into three groups, it will divide the data again. This results in not ‘nesting’, this is ‘cross-cutting’.
Algorithm :
Initialise: Start with n observations and a measure (e.g., Euclidean distance) among n(n-1)/2 pairwise dissimilarities (n observations). Treat each observation as a cluster itself.
Get Aram’s stories in your inbox
Join Medium for free to get updates from this writer.
Calculate Distance: Examine all pairwise inter-cluster dissimilarities among each cluster that are the least dissimilar (most similar). Fuse these two clusters; the height of the dendrogram indicates the difference between the two clusters.
Merges: Find the two clusters with the minimum distance between their closest clusters and merge them into one cluster. The sequence of merging implies significance. For example, early merges show significant similarity between the clusters. Later merges indicate less significant similarity.
Compute the new pairwise inter-cluster dissimilarities among the remaining clusters. Iteratively repeat until all points meet at one single, overarching cluster.
- The most commonly used linkage type in hierarchical clustering.
The concept of dissimilarity between a pair of groups of observations.
Complete: record the largest dissimilarity between all the pairwise dissimilarities (Maximum inter-cluster dissimilarities).
Single: record the smallest dissimilarities (Minimal inter-cluster dissimilarities).
Average: record the average dissimilarities. (Mean inter-cluster dissimilarities).
Centroid: dissimilarities between the centroid of cluster A and the centroid of cluster B.
- Choice of dissimilarity Measure
Euclidean distance.
Correlation-based distance: focus on the shape of observation profiles rather than their magnitude.
How to choose a dissimilarity measure???
For example, to analyse the online shopping mall customers' purchase patterns.
Euclidean distance: infrequent service users vs fewer product buyers.
Correlation-based distance: Customer bought item A and B vs customers who never bought item A and B.
When variables should be scaled to get standardised deviation?
Items are more frequently purchased (e.g. socks) than others (e.g. computers).
When the variables are measured on different scales.
Surprisingly, it depends on the application at hand!
Practical issues in clustering
Small decisions cause big differences!!
? Standardised required???
In a hierarchical clustering, what dissimilarity measure should be used?
What type of linkage should be used?
Where should we cut a dendrogram to obtain clusters?
In K-means clustering, how many clusters do we look for in the data?
Validating clustering; there is no single best method to determine whether we found the true clustering or clustering noise.
Sometimes it is not possible to cluster outliers into subgroups. Maybe a mixture model is required.
When Actuaries use unsupervised learning methods:
To understand and communicate the high-dimensional data.
To identify the meaningful subgroups and to produce meaningful results.
When there is no single correct unsupervised solution, interpretation and context are more important.
From the Exploration methods, to guide subsequent modelling steps.
Reference: An Introduction to Statistical Learning, with applications in R, chapter 12.1–12.2, 12.4.
