<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Itamar Tsayag on Medium]]></title>
        <description><![CDATA[Stories by Itamar Tsayag on Medium]]></description>
        <link>https://medium.com/@itamartsayag?source=rss-8595d195ee2d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*t_DRNZ1CuJqxKFor.</url>
            <title>Stories by Itamar Tsayag on Medium</title>
            <link>https://medium.com/@itamartsayag?source=rss-8595d195ee2d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 00:43:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@itamartsayag/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[TubeR — Video Action Detection]]></title>
            <link>https://medium.com/@itamartsayag/tuber-video-action-detection-1e345721e7a0?source=rss-8595d195ee2d------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e345721e7a0</guid>
            <category><![CDATA[video-recognition]]></category>
            <category><![CDATA[tuber]]></category>
            <category><![CDATA[video-analysis]]></category>
            <category><![CDATA[action-detection]]></category>
            <category><![CDATA[detr]]></category>
            <dc:creator><![CDATA[Itamar Tsayag]]></dc:creator>
            <pubDate>Mon, 20 Feb 2023 14:35:22 GMT</pubDate>
            <atom:updated>2023-02-20T14:43:19.751Z</atom:updated>
            <content:encoded><![CDATA[<h3>TubeR — Human Action detection Action in Videos</h3><h4><em>An End-to-End Spatio-temporal human action detection in videos.</em></h4><p><em>We will explore a neat and original adaptation of the DETR framework to perform spatiotemporal human action detection in videos.</em></p><h4>Action detection is a compound task, requiring:</h4><ul><li>Localization of per-frame person instances</li><li>Linking of these detected person instances into action tubes</li><li>Prediction of these action tube action class labels</li></ul><p><em>A tubelet is defined as a sequence of bounding boxes and their action labels.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lPP9bPqrgu3FN987ugH92g.png" /></figure><p>There are roughly 2 main approaches in video action detection:</p><ul><li><strong>Frame-level Detection</strong>: localizing bounding boxes in the temporal areas around a keyframe (incorporating extra streams of data to improve tracking is possible, such as optical flow), and then applying a 3D CNN classifier network to capture temporal information in the tubelet.</li><li><strong>Tubelet-level action detection: </strong>Using 3D cuboid proposals that progressively refined the cuboid proposals across time. This method has difficulties processing long video clips.</li></ul><blockquote><strong>TubeR</strong> reformulates the action detection task as a sequence-to-sequence learning problem and explicitly models the temporal correlations within a tubelet. TubeR directly predicts the tubelet and its’ class label.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CXZEnqx4KWG0CJ6k4geCzQ.png" /></figure><p>TubeR is built on the <a href="https://github.com/facebookresearch/detr">DETR</a> framework with a tubelet-transformer for localizing and recognizing actions from a single representation.</p><h3>Architecture:</h3><p>An input clip I of size T_in× H × W × 3 enters a CNN backbone and a feature map F_b of size T’× H’ × W’ × C’.</p><ul><li>I3D was chosen in the paper as the backbone, but lower-weight models such as the Video Swin Transformer) can work as well.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RMBYZ00I4O62sQ2teKFbCA.png" /><figcaption>TubeR architecture</figcaption></figure><p>The F_b enters the Transformer encoder-decoder, each stacked 6 times.<br>The transformer encoder is very similar to the one in DETR, with the addition of the 3D positional embedding, resulting in F_enc of size T’H’W’×C’.</p><p>The transformer decoder is an interesting component. It is similar to the DETR decoder, but with the following changes:</p><h4>1. Learnt Tubelet-Query:</h4><p><strong><em>N</em></strong> learnt tublets locations are used as queries in the decoder. The tubelets are sequences of T_out bounding boxes, as follows</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/307/0*BWg-tzgxR6oZd6hn.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/381/0*v2vm4rzJQHddza8u.png" /></figure><h4>2. Tubelet Attention:</h4><p>The Tubelet-Attention consists of self-attention layers:</p><blockquote><strong>Spatial Self-Attention Layer — </strong>processes the spatial relations between box query embeddings within a frame, i.e. for each temporal index t, apply self-attention between bounding boxes of different tubelets.</blockquote><blockquote><strong>Temporal Self-Attention Layer — </strong>models the correlations between box query embeddings across time within the same tubelet, i.e., for each tubelet (b is fixed), the self-attention is across all the temporal axis. This helps the TubeR track actors and generate action tubelets that focus on single actors instead of a fixed area in the frame.</blockquote><p>The Transformer encoder output, F_enc, and the decoder F_q are incorporated using a Cross-Attention, as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/0*hEYuQu5q97iM2t-z.png" /></figure><p>The output of the decoder is then:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/0*bcdz_-mzPfsgGlk-.png" /></figure><h3>TubeR Heads:</h3><p>The Transformer-decoder output, F_tub, is served to multiple heads to produce accurate tubelets:</p><ul><li><strong>Bounding Box Head </strong>— outputs a sequence of 4-point coordinates defining the spatial area of the tubelet (part of the regression head).</li><li><strong>Action Switch Head</strong> — outputs a binary sequence in which the tubelet is valid. This head helps limit the temporal length of the tubelet.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*EZ7CWh0vO2xfhnqLr9bDpw.png" /></figure><ul><li><strong>Action Head</strong>: Classifier head for the tubelet action. This head can incorporate a long/short-term context component that can help in determining the class in using its context (specifically work well in long videos)</li></ul><h4>Losses are accordingly to the head types:</h4><ul><li>Bounding Box Head — Detection-style losses, i.e. L1 regression on the bounding boxes, and GIoU loss per frame bounding box</li><li>Action Switch Head — Binary Cross Entropy</li><li>Action Head: Categorial Cross Entropy</li></ul><h3>Evaluation:</h3><p>The model is tested on the following datasets:</p><ul><li><strong>UCF101–24</strong>: 24 sport classes in 3207 untrimmed videos</li><li><strong>JHMDB51–21</strong>: 21 action categories in 928 trimmed videos</li><li><strong>AVA</strong>: 299 15-minute movies, 235 for training, and the remaining 64 for validating.</li></ul><p>The Evaluation Criteria are similar to detection, <strong>Video-mAP</strong> at different IoUs.</p><p>The results, below, show that TubeR achieves the highest mAP relative to all other categories of tubelet detectors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fmUImBkiO9XAuH56S4mhQw.png" /></figure><p>TubeR at action is <strong>amazing</strong>! and can help us reduce long videos to actions for future uses. For example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iut4BIc3X4lCiZBJ1Q6jFw.png" /></figure><p>Research was done by: <strong>Jiaojiao Zhao, Yanyi Zhang, Xinyu Li, Hao Chen, </strong><a href="https://www.amazon.science/author/bing-shuai"><strong>Bing Shuai</strong></a><strong>, Mingze Xu, </strong><a href="https://www.amazon.science/author/chunhui-liu"><strong>Chunhui Liu</strong></a><strong>, </strong><a href="https://www.amazon.science/author/kaustav-kundu"><strong>Kaustav Kundu</strong></a><strong>, </strong><a href="https://www.amazon.science/author/yuanjun-xiong"><strong>Yuanjun Xiong</strong></a><strong>, </strong><a href="https://www.amazon.science/author/davide-modolo"><strong>Davide Modolo</strong></a><strong>, Ivan Marsic, Cees G.M. Snoek, </strong><a href="https://www.amazon.science/author/joseph-tighe"><strong>Joseph Tighe</strong></a></p><h4>References:</h4><p><a href="https://www.amazon.science/publications/tuber-tubelet-transformer-for-video-action-detection">TubeR: Tubelet transformer for video action detection</a></p><p>End-to-End Object Detection with Transformers — <a href="https://arxiv.org/abs/2005.12872">https://arxiv.org/abs/2005.12872</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e345721e7a0" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>