Introduction to Representation Learning
The performance of a model is heavily dependent on the choice of data representation (or features) on which they are applied. For that reason, most of the actual effort in deploying models goes into the design of preprocessing pipelines and data transformations that result in a representation of the data that can support effective machine learning.
Feature engineering is a way to take advantage of human ingenuity and intuition, but it’s labor-intensive and highlights a gap in current learning algorithms: An AI must fundamentally understand the world around it, by learning to identify and disentangle the underlying explanatory factors hidden in the observed low-level sensory data. It would be highly desirable to make learning algorithms less dependent on feature engineering.
Problem statement
Before diving in, let’s present the basic problem statement and the notation we will be using throughout. The goal of representation learning is to learn to map from data points x ∈ X to abstract representations, z ∈ Z:
We call this mapping an encoding and learn a function f : X → Z. Typically both x and z are high-dimensional vectors, and z is called a vector embedding of x. The mapping f is trained so that z has certain desirable properties:
- z be lower dimensional than x
- The distribution of z values, that is, p(z), has a simple structure (such as being the unit normal distribution);
- The dimensions of z be independent factors of variation, that is, the representation should be disentangled.
In this way Z is a simpler, more abstracted, or better organized representational space than X.
What makes a good representation?
Priors for Representation Learning
Good representations can reflect many general “priors” (assumptions that are not task-specific), about the world around us. Let’s look at some of these priors:
Smoothness: Assumes the function to be learned f is s.t. x ≈ y generally implies f(x) ≈ f(y). However, traditional models (e.g., linear models or kernel machines) aren’t enough for complex tasks such as vision or language because they rely too heavily on smoothness.
An image of a cat and an image of pure noise may appear close to each other in the embedding space, due to their raw pixel values being similar). Therefore, we instead need richer priors.
Multiple explanatory factors: The data is generated by several independent or semi-independent factors. A good representation disentangles these factors so the model can learn each one independently. This assumption is behind the idea of distributed representations.
A hierarchical organization of explanatory factors: The concepts that are useful for describing the world around us can be defined in terms of other concepts, in a hierarchy, with more abstract concepts higher in the hierarchy. This assumption is exploited with deep representations.
Semi-supervised learning: The structure of the input X contains information useful for predicting the label Y. For example, learning the structure of images (unsupervised) helps later with classification (supervised). Hence representations that are useful for P (X) tend to be useful when learning P (Y | X).
Shared factors across tasks: Many learning tasks or Y’s of interest, share underlying causes and therefore learning one improves learning another.
Manifolds: Probability mass concentrates near regions that have a much smaller dimensionality than the original space where the data lives. This is explicitly exploited in some of the autoencoder algorithms and other manifold-inspired algorithm.
Natural clustering: Different values of categorical classes (labels) are associated with separate manifolds. More precisely, the local variations on the manifold tend to preserve the value of a category, and a linear interpolation, more precisely, P(X|Y = i) for different i tend to be well separated and not overlap much.
Temporal and spatial coherence: Sequentially consecutive or spatially nearby observations tend to be associated with the same value of relevant categorical concepts. When attempting to capture such categorical variables, this prior can be enforced by making the associated representations slowly changing, i.e., penalizing changes in values over time or space.
Sparsity: For any given observation x, only a small fraction of the possible factors are relevant. In terms of representation, this could be represented by features that are often zero, or by the fact that most of the extracted features are insensitive to small variations of x.
Get Chamuditha Kekulawala’s stories in your inbox
Join Medium for free to get updates from this writer.
Simplicity of Factor Dependencies: In good high-level representations, the factors are related to each other through simple, typically linear dependencies. This is assumed when plugging a linear predictor on top of a learned representation.
Distributed Representations
Good representations are expressive, meaning that a reasonably-sized learned representation can capture a huge number of possible input configurations. A simple way to assess the expressiveness of a representation is: how many parameters does it require compared to the number of input configurations it can distinguish?
Learners of one-hot representations, such as traditional clustering, kNN, Gaussian mixtures, decision trees, or Gaussian SVMs basically assign one feature per region/class. So to distinguish N features, they require O(N) parameters to distinguish O(N) input regions. This doesn’t scale well.
However, Restricted Blotzmann Machines (RBMs), sparse coding, auto-encoders or neural networks can all represent up to O(2ᵏ) input regions using only O(N) parameters.
k is the number of non-zero elements in a sparse representation, and k = N in dense representations. If k features can turn on/off independently → they can encode exponentially many patterns.
The generalization of clustering to distributed representations is multi-clustering, where either several clusterings take place in parallel or the same clustering is applied on different parts of the input (e.g., clustering small image patches and then combining those clusters into a higher-level feature). This is the idea behind many modern vision systems: each image patch is “clustered,” and the results are combined hierarchically.
Depth and abstraction
Depth is a key aspect to representation learning strategies. Deep architectures are often challenging to train effectively and this has been the subject of much recent research and progress. However, despite these challenges, they carry two significant advantages:
- Deep architectures promote the re-use of features
- Deep architectures can potentially lead to progressively more abstract features at higher layers of representations
Feature re-use: Deep learning works well because features can be reused across many layers. In a deep network, each layer builds on the previous one. The number of ways features can be combined grows exponentially with depth. This means deep models can represent very complex functions using fewer parameters. If a function can be represented with fewer parameters, it usually requires fewer training examples to learn.
Abstraction and invariance: Deep networks also work because deeper layers learn more abstract and more invariant features. Lower layers detect simple patterns (edges, textures). Higher layers combine these into more abstract concepts (object parts, whole objects). Abstract features tend to stay unchanged despite small variations in the input (shift, lighting, noise, etc.). This is especially clear in CNNs, where pooling helps enforce invariance. Abstract, invariant features generalize better, recognize categories despite changes, capture the underlying concepts that matter for prediction.
Deep Representations
In 2006, a breakthrough in feature learning and deep learning was initiated by Geoffrey Hinton. A central idea, referred to as greedy layerwise unsupervised pre-training, was to learn a hierarchy of features one level at a time, using unsupervised feature learning to learn a new transformation at each level to be composed with the previously learned transformations; essentially, each iteration of unsupervised feature learning adds one layer of weights to a deep neural network.
Finally, the set of layers could be combined to initialize a deep supervised predictor, such as a neural network classifier, or a deep generative model, such as a Deep Boltzmann Machine. It was empirically observed that layerwise stacking of feature extraction often yielded better representations, in terms of classification error, quality of the samples generated or invariance properties of the learned features.
After greedy layerwise unsuperivsed pre-training, the resulting deep features can be used either as input to a standard supervised machine learning predictor (such as an SVM) or as initialization for a deep supervised neural network (e.g., by appending a logistic regression layer or purely supervised layers of a multi-layer neural network).
Single-layer Learning Modules
In representation learning, single-layer learning modules are the simplest building blocks that learn a transformation from input → representation in one step (one layer). They are the basic units used to build deeper models.
It is a function that maps an input x to a representation h = f(x) using one layer of parameters (weights + bias), often with a nonlinearity, trained with some learning objective (reconstruction, sparsity, contrastive loss, etc.). These modules can be trained alone or stacked to form deep networks.
For these modules, there has developed 2 broad paradigms:
- Probabilistic graphical models
- Neural networks.
Fundamentally, the difference between these two paradigms is whether the layered architecture of a deep learning model is to be interpreted as describing a probabilistic graphical model or as describing a computation
graph. In short, are hidden units considered latent random variables or as computational nodes?
This is likely a function of the fact that recent progress in both of these areas has focused on single-layer greedy learning modules and the similarities between the types of single-layer models that have been
explored: mainly, the RBM and Sparse Coding on the probabilistic side, and the auto-encoder variants on the neural network side.
Additionally, we can consider PCA to be an unsupervised single-layer representation learning algorithm spanning all three views: probabilistic, auto-encoder, and manifold learning.
That’s it for this article! Thanks for reading 🎉
