Learning to generate videos with uncertain futures
A summary of the paper “Stochastic Video Generation with a Learned Prior”
TL;DR: This post provides a high-level overview of the video generation model described in Stochastic Video Generation with a Learned Prior, which is capable of generating video sequences with multiple futures.
Video Generation as Self-Supervised Learning task
Supervised deep learning models have proven to yield groundbreaking results in the recent past on hard tasks like real-time object segmentation, detection, machine translation & speech recognition. The high-fidelity performance of these models have enabled their use in real-world applications to bring a positive impact on the world (but sometimes negative too).
The next phase of challenges that lay in unsupervised learning have also been addressed successfully to a certain extent in domains like domain adaption and machine translation. Learning from raw, noisy real-world data is much cheaper and faster operationally since a huge chunk of manual pre-processing like cleaning and annotating can be skipped.
A sub-domain of unsupervised learning is self-supervised learning, which has been talked about a lot recently by the AI community. In this scenario, the model is provided input data and is tasked with completing the story around that data. A particularly interesting application in this domain is to generate future events of any given video sequence.
The caveat with this task, which makes this problem hard, is unlike other tasks where the inputs have fixed outputs, the future events in a video can unravel with multiple possibilities, e.g., a bouncing ball hitting the ground can have multiple possible trajectories, spin etc.
Stochastic Video Generation with a Learned Prior
“Stochastic Video Generation with a Learned Prior”, published in ICML 2018, presents a framework to deal with this situation by proposing a prediction model where the “uncertain” part of the future is handled by a latent variable sampled from a prior distribution. To get an understanding of how that works, let’s dive right into the system design.
The system consists of the following components:
- Prediction Model
- Inference Model
- Learned Prior
Prediction Model
The prediction model is the core predictor/generator with the other models providing supplementary information to aid the prediction model. It consists of an Encoder that takes an image frame at time t-1 from a video and encodes it as a dense representation in the latent space. This representation vector is then passed onto a LSTM (frame predictor), which is the memory component of the prediction model. It helps the model to take into account the information from past image frames in the sequence. The frame predictor is responsible for predicting a dense representation of the next frame in the future. This is then passed onto a decoder/generator network which converts this to the predicted image at time t. To further aid the decoder, skip connections are made between the encoder and decoder so that the static portions of the video can simply be copied directly.
The frame predictor is also conditioned on an auxiliary variable which can be seen as representing information about a particular future. We will discuss this in the next steps.
The training of the model so far involves a very simple objective of minimizing the reconstruction error or the L2 distance between the predicted frame and ground truth.
Inference Model
The inference model is tasked with learning the distribution which contains the information about the event at time t. A sample z is drawn from this distribution which is passed onto the frame predictor in the prediction model.
Get Anant Gupta’s stories in your inbox
Join Medium for free to get updates from this writer.
The input to the inference model is image frame at time t and thus this model is only used during training. The architecture of the inference model uses the same encoder as in the prediction model because the type of image encoding we want is same in both the stages. Re-using the model or weight sharing makes the learning process a bit easier. The encoded vector is then passed onto a different LSTM model (called posterior in this paper) which has a similar objective as frame predictor, but in the context of learning to represent future information. The posterior might learn to look at the hidden state from the past and avoid capturing redundant information in the learnt distribution.
One may wonder that the inference model may simple memorise the entire image and reproduce it through the decoder. To ensure that only the concerned information is captured, we make use of the learned prior model.
Learned Prior
To prevent the inference model from simply copying the entire target frame, we impose the restriction that the learned distribution from posterior should closely match a prior distribution. This is ensured by minimizing the KL-divergence between the two multivariate distributions that the posterior and prior predict. This prior distribution is learned through another model that is similar to the inference model but takes the frame at time t-1 as input. The idea here is that the prior is also time-dependent, i.e., the prior distribution predicted at time t is a function of the past frames till t-1.
An alternative approach is to fix this prior distribution as Gaussian with zero mean and identity covariance.
Results
Inference is performed by feeding first t-1 frames of ground truth to the prediction and prior model, sampling z from the learned prior distribution and generating the frame for timestep t. This is then fed back in the model as input to generate the frame at t+2.
Apart from smaller datasets like Moving MNIST and KTH action, this model showed impressive results on the BAIR robot pushing dataset. Given how simple the loss objectives, it was indeed a surprise that the model is able to perform so well, i.e., generate non-blurry motions of the robot arm, for multiple timesteps into the future.
For more videos of results, checkout https://sites.google.com/view/svglp/.
Current Drawbacks
If the reader looks at the enlarged results in the provided link, it is easy to notice that the predicted frames exhibit blurriness when the robot arm interacts with other objects lying around. In other words, this model has learned to represent future stochastic movement of the arm, but fails to capture the dynamics between the arm and it’s interaction with other objects.
One way to address this is to make use of hierarchical latent modelling, such that the broader concepts (like overall arm movement) are captured by prior on a lower lever, and intricate dynamics are captured by models in the next level. Earlier this year, I spent some time looking into this, applying various techniques based on the hierarchical concept. Some of my findings can be checked out here.
Final Thoughts
This post provides a brief overview of the SVG-LP model and some of the results it obtains. For a more formal explanation from a probabilistic point of view, refer the paper linked above.
Video generation models not only serve as an immensely useful application for video understanding and synthesis, but also help in planning and reinforcement learning. Later half of 2018 has seen an incredible rise in model-based reinforcement learning. A foundational component of these methods is the forward prediction model that are tasked with predicting the next states given the current and the action. Checkout David Ha’s https://worldmodels.github.io/ to get a glimpse how this type of model can be used in reinforcement learning.
