<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Binu M. Nair on Medium]]></title>
        <description><![CDATA[Stories by Binu M. Nair on Medium]]></description>
        <link>https://medium.com/@nair.binum?source=rss-a2467c8c2259------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*W3CnD6PpDiRQyuKVaBdDag@2x.jpeg</url>
            <title>Stories by Binu M. Nair on Medium</title>
            <link>https://medium.com/@nair.binum?source=rss-a2467c8c2259------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 13:44:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nair.binum/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Deep Dive into Convolutional 3D features for action and activity recognition (C3D)]]></title>
            <link>https://medium.com/@nair.binum/quick-overview-of-convolutional-3d-features-for-action-and-activity-recognition-c3d-138f96d58d8f?source=rss-a2467c8c2259------2</link>
            <guid isPermaLink="false">https://medium.com/p/138f96d58d8f</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[activity-recognition]]></category>
            <category><![CDATA[video-recognition]]></category>
            <dc:creator><![CDATA[Binu M. Nair]]></dc:creator>
            <pubDate>Mon, 23 Jul 2018 02:42:49 GMT</pubDate>
            <atom:updated>2018-07-24T14:33:01.287Z</atom:updated>
            <content:encoded><![CDATA[<p>In this blog, I would provide a brief overview of the Convolutonal 3D or C3D model widely used in video recognition applications and recent challenges in Kaggle (<a href="https://research.google.com/youtube8m/workshop2018/">ECCV YouTube 8M challenge</a>). My motivation for this blog is that recently at CVPR (2018), I came across a number of models especially in action and activity recognition problems where C3D have been used as a de-facto feature extraction technique. So for those who are new or currently exploring computer vision especially in the video domain, this blog can be beneficial where I will explore its architecture and give some technical insights into this model. For readers interested in knowing concrete details about the experimentation, here is the reference given below.</p><blockquote><a href="https://www.cv-foundation.org/openaccess/content_iccv_2015/papers/Tran_Learning_Spatiotemporal_Features_ICCV_2015_paper.pdf">Learning spatiotemporal features with 3d convolutional networks ; D Tran, L Bourdev, R Fergus, L Torresani, M Paluri , 2015 IEEE International Conference on Computer Vision (ICCV), 4489–4497</a></blockquote><h3>What are C3D features?</h3><p>C3D are deep 3-dimensional convolutional neural networks with a homogenous architecture containing 3 x 3 x 3 convolutional kernels followed by 2 x 2 x 2 pooling at each layer. They are trained on a large scale supervised video dataset such as UCF-101 and Sports 1M. The key properties that make it stand against traditional pre-trained models such as ResNets, AlexNet are the following:</p><ol><li><strong>Generic</strong> feature extraction: The 3D convolutions extracts both spatial and temporal components relating to motion of objects, human actions, human-scene or human-object interaction and appearance of those objects, humans and scenes. Thus it is not limited to appearance representation as in ResNets or AlexNet. This makes it a very generic video feature representation for various video related tasks such as action localization, and event detection without the need for fine-tuning for each task.</li><li><strong>Compact </strong>representation: The C3D model is given an input video segment of 16 frames (after downsampling to a fixed size which depends on dataset used) and the outputs a 4096-element vector. It gives you a compact representation of a video segment which can be further fed to either a temporal network for action localization task [3] , action recognition [1], and action-word mining [4]. Thus, it makes storage and retrieval of videos highly scalable.</li><li><strong>Fast, efficient inference: </strong>Due to its homogenous architecture with small kernel sizes of 3 x 3 x 3, it can facilitate for optimized implementation of these architectures for embedded platforms. This can have huge implications for smart camera environments where processing of real-time feed and extraction of meta data is critical for surveillance and smart city applications.</li></ol><p>Before we dwell into the network architecture, I will focus first on the concept of 3D convolution on a spatial temporal cube of size <em>b </em>x <em>L </em>x <em>n </em>x <em>n </em>where <em>c</em> is the number of channels, <em>L</em> is the number of frames.</p><h3>3D Convolution Operator — Basics</h3><p>There are typically four types of convolution that we commonly see in computer vision:</p><ol><li>2D convolution on gray scale image</li><li>2D convolution on RGB image</li><li>2D convolution on L-frame RGB video segment</li><li>3D convolution on L-frame RGB segment.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wr3uBvKucVPa3IwSC1pI5Q.png" /><figcaption>Illustration of 3D convolution on L-frame RGB video segment</figcaption></figure><p>A 2D convolution of an <em>n</em> x <em>n </em>image with a kernel of size <em>k</em> x <em>k </em>results in another 2D image. Similar is the case with the RGB image ( <em>3</em> x <em>n</em> x <em>n</em>) where the kernel of size <em>3 </em>x <em>k</em> x <em>k </em>is convolved with the image resulting in another 2D image (Figures (a),(b)). Now, when to comes to an <em>L</em>-frame RGB video segment, the effective input channel length will be <em>3 </em>x <em>L</em>. Therefore, a 2D convolution operation will result in convolving with a kernel of size <em>3</em> x L x <em>k</em> x <em>k </em>resulting in again a 2D image ( as shown sub figure c). The issue in a 2D convolution on a video segment is that the temporal features at this point gets squashed by this operation resulting in a temporally averaged appearance feature map without any motion representation. This can be circumvented by using a 3D convolution operation where we explicity define a kernel of size <em>d </em>x <em>k</em> x <em>k </em>where now, the convolution operation is between a RGB video segment of size <em>3 </em>x <em>L </em>x <em>n</em> x <em>n </em>with an effective kernel of size <em>3 </em>x <em>d </em>x <em>k</em> x <em>k </em>resulting in an output cuboid which preserves the temporal information.</p><h3>C3D Architecture</h3><p>The network architecture contains 8 convolutional, 5 pooling layers and 2 fully connected layers as shown in Figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EkGOxBq8_HJh5mwfmKWLcA.png" /><figcaption>C3D Architecture [1]</figcaption></figure><p>The first convolution layer of size <em>1</em>x<em>3</em>x<em>3 </em>followed by a pooling layer of size <em>1</em>x<em>2</em>x<em>2. </em>This is to preserve the temporal information in the first layer and build higher level representation of the temporal information at subsequent layers of the network. Every other convolution layer and pooling layer would have a size of <em>3</em>x<em>3</em>x<em>3 </em>and <em>2</em>x<em>2</em>x<em>2 </em>where the strides are 1 and 2 respectively. The fully connected layers have a size of <em>4096 </em>dimensions with softmax outputs reflecting either 101 classes of UCF-101 dataset or 487 classes of the Sports 1M dataset. For those who wish to play with the C3D model, apre-trained model trained on Sports 1M dataset is available as open-source (<a href="http://vlg.cs.dartmouth.edu/c3d/">link</a>). An implementation in Keras which ports a trained model from Caffe is also available (<a href="https://gist.github.com/albertomontesg/d8b21a179c1e6cca0480ebdf292c34d2">link</a>).</p><h3>Key Takeaways</h3><h4>Temporal information representation</h4><p>One of the key takeaways in this architecture design is the use of 3D convolutions which retains the temporal information across the feature maps for a video sequence. The figure below shows the deconvolution of the Conv5b feature maps with the highest activation. We see that in the first few frames, the features focus more on the appearance of the actor. As the action proceeds, the feature focus more on the salient motion near the pole.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dHCO97e87AYG31xcSc62yA.png" /><figcaption>Pole Vault Action : Emphasis on human appearance (Feature map from Conv5b for first few frames)[1]</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6pWXSDk9joowjTUURv-Gzw.png" /><figcaption>Pole Vault Action : Emphasis is on salient motion on pole (Feature map from Conv5b for last few frames)[1]</figcaption></figure><p>Similarly, for the action sequence “apply makeup” shown below, the first few frames focus on the facial appearance and the last few frames on the salient motion occuring near the eye brows and eyes. Thus, C3D filters selectively focus on appearance and motion at different instants of a video segment. This is the core differentiator with standard 2D ConvNets such as AlexNet or ResNet where the filters focus only on appearance and the temporal information averaged out across the frames.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v05VyUqtIrnHLL-UpD_JIg.png" /><figcaption>Apply Makeup : Emphasis is on facial appearance (Feature map from Conv5b for first few frames)[1]</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*5nO6qPgi1p31l0ltaNh6Rw.png" /><figcaption>Apply Makeup : Emphasis is on salient motion near eyebrows (Feature map from Conv5b for last few frames)[1]</figcaption></figure><h4>Better separability across action classes</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v1WF37Nj9Fwmuvii5tTKrg.png" /><figcaption>Video Feature representation in low dimensional space[1]</figcaption></figure><p>Another key takeway from this model is the genetic nature of the features for video representation. The figure above shows the distribution of the video features extracted from ImageNET-based network and the C3D network in low dimensional space. The video features represent the fully connected layer ouputs averaged across the 16-frame segments, and the 2-dimensional embedded space is obtained by applying t-SNE dimensionality reduction technique [2]. Note that these features are not assigned class labels and are projected in an unsupervised manner. We see that the features from C3D have much better seperability than those extracted from networks trained on ImageNet. This shows that for videos, C3D provides both a generic and separable representation for video sequences.</p><h4>Spatio-temporal feature abstraction</h4><p>The C3D network also represents spatio-temporal information at different levels of abstraction ranging from detection of moving edges, tracking corner movements, and emphasis on people and objects relevant to the action. An illustration of the information learned by the first layer (Conv1b) is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L_I-ocHeAZd9-EmJHWdRGQ.png" /><figcaption>Filter visualization using DeConvolution method from Conv1b[1]</figcaption></figure><p>The set of filters in first two rows corresponds to moving edges and blob detectors of varying sizes, thickness and flexibility. We see that the deconvolution of such filters provides the kind of patches that gives the highest response for the filters in an image. Similarly, in the third row, the filters are more inclined to give highest responses to different color and texture variations. For example, in the last row and column of the figure above, the filters provide highest responses to image patches that are reddish or yellowish in color with some sutle responses to the bluish regions. We can also see the filters can detect specific combination of colors or a combination of a color and an edge (eg: blue planar region imposed on a black region). This particular type of filter representation provides only a single level of spatio-temporal feature abstraction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*blU_vvwtJgAoapWxgB4S9g.png" /><figcaption>Filter visualization using DeConvolution method from Conv3b[1]</figcaption></figure><p>In the figure above, a different level of spatio-temporal feature abstraction is provided and these are from the deconvolution outputs of Conv3b filters. In the first row, the filters detect moving smaller body parts such as arms and legs associated with the motion and texture like features corresponding to the context in the action. The second row represents filters which detects much larger body regions of the human performing various types of actions. In the third row and first column, the filters capture various movements associated with object trajectories. However in the second column, the filters characterize objects which are circular in nature and captures the context of an action. Thus, through these visualizations, we see that C3D network captures spatio-temporal features at various abstract levels which represents both motion and appearance (context) necessary to represent a video segment.</p><h3>How and why is C3D important?</h3><p>C3D is a very common video segment descriptor which embeds both motion and temporal characteristics. Thus, it is very intuitive to use C3D features as a first step before further processing for various applications.</p><ol><li>Video representation [1]: For representing a long video of 2 sec, the video is broken down in 16-frame temporal chunks with 50% overlap between consecutive chunks. The C3D features are then averaged across the temporal segments/chunks using L-2 norm to give a robust compact description of the long video.</li><li>Action-Word Embedding [3] : Given a video segment, a network can be trained to model semantic relationship between a video feature and a corresponding word vector representation of the action class. This helps in building action word vectors. Using these action-word vectors, videos can be queried in an unsupervised manner based on this embedding and helps in zero-shot action recognition applications.</li><li>Temporal Localization [4] : A Faster R-CNN type architecture has been used to segment and localize video sequences into specific human actions in long continous videos. For this approach, C3D features are used along with multi temporal scale R-CNN model.</li><li>Video Generation [5] : For generating videos from a set of frames, C3D features are used during the comparison stage between the generated frames and true frames during training.</li></ol><h3>To conclude</h3><p>In this article, I provide a quick overview of the C3D model and describe some of its applications in current research. In my next blog, I will provide details on the current state of art video descriptor known as I3D [6] where this model has been trained on a very large scale dataset known as Kinetics 600.</p><h3>References</h3><p>[1] D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, <strong><em>Learning Spatiotemporal Features with 3D Convolutional Networks</em></strong>, ICCV 2015</p><p>[2] Maaten, Laurens van der and Geoffrey E. Hinton. “<strong><em>Visualizing Data using t-SNE</em></strong><em>.</em>” (2008).</p><p>[3] Meera Hahn, Andrew Silva, James M. Reh, “<strong><em>Action2Vec: A Crossmodal Embedding Approach to Zero Shot Action Learning</em></strong>”, Computer Vision and Pattern Recognition (CVPR) 2018 Deep Vision Workshop</p><p>[4] Y. W. Chao, S. Vijayanarasimhan, B. Seybold, D. Ross, J. Deng, R. Sukthankar. “<strong><em>Rethinking the Faster R-CNN Architecture for Temporal Action Recognition</em></strong>”. Computer Vision and Pattern Recognition (CVPR), 2018</p><p>[5] De-An Huang, Vignesh Ramanathan, Dhruv Mahajan, Lorenzo Torresani, Manohar Paluri, Li Fei-Fei, Juan Carlos Niebles, “<strong><em>What Makes a Video a Video: Analyzing Temporal Information in Video Understanding Models and Dataset</em></strong>”, Computer Vision and Pattern Recognition (CVPR) 2018</p><p>[6] J Carreira, A Zisserman, “<strong><em>Quo vadis, action recognition? a new model and the kinetics dataset” </em></strong>Computer Vision and Pattern Recognition (CVPR), 2017</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=138f96d58d8f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>