Deep Dive into Video Understanding with CNNs

Till now, we have seen how CNNs are applied on images, be it classification, recognition or segmentation. But standalone images processed with these algorithms only give limited information. To truly unlock computer vision we need to process videos as we view the real world as videos or set of images not single image.

Consider this, there are two videos. One is of a person standing up from chair while second video is of same person sitting down on the chair. You take a central frame. Can you make an image recognition CNN which can tell you that wether the person is sitting down or standing up from these central images?

This perfectly illustrates the limitation of static images versus the rich information in videos. The real world isn’t static. It’s dynamic, flowing and temporal. To truly understand visual reality, our neural networks need to make the leap from processing individual frames to interpreting sequences of images unfolding over time.

In this article, we’ll explore how researchers have cleverly adapted CNNs to capture this temporal dimension, enabling them to recognize actions, track motion and understand activities in videos.

We’ll focus exclusively on CNN-based approaches, leaving transformer-based and other architectures for future articles.