<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rafael Toledo on Medium]]></title>
        <description><![CDATA[Stories by Rafael Toledo on Medium]]></description>
        <link>https://medium.com/@rafaeltol?source=rss-4ebee3139190------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*sDI0QCtu6NaduSMZcFatAg.jpeg</url>
            <title>Stories by Rafael Toledo on Medium</title>
            <link>https://medium.com/@rafaeltol?source=rss-4ebee3139190------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 03:59:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rafaeltol/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Understanding and Coding DETR (DEtection TRansfomer)]]></title>
            <link>https://medium.com/@rafaeltol/understanding-and-coding-detr-detection-transfomer-80e4c206fbc8?source=rss-4ebee3139190------2</link>
            <guid isPermaLink="false">https://medium.com/p/80e4c206fbc8</guid>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[detr]]></category>
            <dc:creator><![CDATA[Rafael Toledo]]></dc:creator>
            <pubDate>Tue, 21 May 2024 14:20:53 GMT</pubDate>
            <atom:updated>2025-04-13T15:07:38.065Z</atom:updated>
            <content:encoded><![CDATA[<p>AI is on fire these days, and transformer-based architectures are one of the key factors that leveraged the last AI success cases. In the computer vision field, <strong>DEtection TRansformer (DETR)</strong> from Carion et al. (2020) was one of the first works to demonstrate that transformers could also be used for vision tasks, reaching competitive results with other baseline works.</p><p>Another meaningful DETR contribution was the introduction of a new solution paradigm for the object detection task that handles it as a set prediction problem. The new paradigm introduced a new element called object queries and freed the object detection solutions of hand-designed mechanisms like anchors or non-maximum suppression (NMS). This paradigm was later applied to segmentation tasks and state-of-the-art models like Mask2Former (Cheng et al., 2022) and OneFormer (Jain et al., 2023).</p><p>This article will expose the main mechanisms of the DETR implementation, and, in the end, you’ll be able to train the network on the COCO dataset without any external scripts but PyTorch methods.</p><p>You can follow this story running on <a href="https://colab.research.google.com/drive/1-TVkq7PrXnlu1IObXjhnT1h6HKzbq2eI?usp=sharing">Colab</a>. For the jupyter notebook and checkpoints, check this tutorial <a href="https://github.com/tldrafael/understanding-and-coding-detr">repo</a>.</p><p><strong>Disclaimer:</strong> Many code snippets below were copied or adapted from the original DETR <a href="https://github.com/facebookresearch/detr">repo</a>. My approach is to make everything as explicit as possible.</p><p><strong>Updates 2025.03.05: </strong>I’d like to thank <a href="https://medium.com/u/ee1c924472ce">Mitsos Z</a> for pointing out two things that I missed from the original DETR work: the frozen BatchNorm layers during training, and the argument batch_first=True in the TransformerEncoderLayer . The text and code here are updated now, although, I’m still re-running the experiments with the new additions, hence, the results will be updated soon.</p><p><strong>Updates 2025.04.13: </strong>The loss graphs and the checkpoints were updated to accomplish the fixes raised in the previous note. Qualitatively, the performance looked the same, whereas quantitatively, it changed: the loss for the full training converged, but this final loss was higher than the previous version. I suspect the frozen BatchNorm may have constrained the network to reach a lower loss value.</p><h3>Dependencies</h3><p>Import all required dependencies:</p><pre>import torch<br>from torch import nn<br>from torch.nn import functional as F<br>from torch.utils.data import Dataset, DataLoader<br>from torch.optim import AdamW, lr_scheduler<br>import torchvision.transforms as T<br>from torchvision import datasets, ops<br>from torchvision.models.feature_extraction import create_feature_extractor<br>from einops import rearrange<br><br>import matplotlib.pyplot as plt<br>import numpy as np<br>from scipy.optimize import linear_sum_assignment</pre><h3>COCO Dataset</h3><p>First, let’s prepare our dataset because, as we all know, machine learning is all about data. Fortunately, the repo below has a tiny version of 50 samples of the COCO dataset that allows a fast start.</p><pre>git clone https://github.com/lizhogn/tiny_coco_dataset.git </pre><p>Noteworthy points for our data preparation are:</p><ol><li>All images are resized to (480, 480); this is an arbitrary decision to facilitate this didactic work. The original work applied independent resizing for each sample, varying between 480 and 1333.</li><li>The target labels are the 4 box coordinates and the box class. The box coordinates are their centers (cx, cy) and their dimensions (w, h). The box classes are the original 91 coco classes, plus the “empty” class.</li><li>The bounding box coordinates are all normalized to represent a relative size with regard to the image input, thus, their values will be in the range [0, 1].</li></ol><p>Now, we declare all methods required to handle the COCO dataset.</p><pre>CLASSES = [<br>    &#39;N/A&#39;, &#39;person&#39;, &#39;bicycle&#39;, &#39;car&#39;, &#39;motorcycle&#39;, &#39;airplane&#39;, &#39;bus&#39;,<br>    &#39;train&#39;, &#39;truck&#39;, &#39;boat&#39;, &#39;traffic light&#39;, &#39;fire hydrant&#39;, &#39;N/A&#39;,<br>    &#39;stop sign&#39;, &#39;parking meter&#39;, &#39;bench&#39;, &#39;bird&#39;, &#39;cat&#39;, &#39;dog&#39;, &#39;horse&#39;,<br>    &#39;sheep&#39;, &#39;cow&#39;, &#39;elephant&#39;, &#39;bear&#39;, &#39;zebra&#39;, &#39;giraffe&#39;, &#39;N/A&#39;, &#39;backpack&#39;,<br>    &#39;umbrella&#39;, &#39;N/A&#39;, &#39;N/A&#39;, &#39;handbag&#39;, &#39;tie&#39;, &#39;suitcase&#39;, &#39;frisbee&#39;, &#39;skis&#39;,<br>    &#39;snowboard&#39;, &#39;sports ball&#39;, &#39;kite&#39;, &#39;baseball bat&#39;, &#39;baseball glove&#39;,<br>    &#39;skateboard&#39;, &#39;surfboard&#39;, &#39;tennis racket&#39;, &#39;bottle&#39;, &#39;N/A&#39;, &#39;wine glass&#39;,<br>    &#39;cup&#39;, &#39;fork&#39;, &#39;knife&#39;, &#39;spoon&#39;, &#39;bowl&#39;, &#39;banana&#39;, &#39;apple&#39;, &#39;sandwich&#39;,<br>    &#39;orange&#39;, &#39;broccoli&#39;, &#39;carrot&#39;, &#39;hot dog&#39;, &#39;pizza&#39;, &#39;donut&#39;, &#39;cake&#39;,<br>    &#39;chair&#39;, &#39;couch&#39;, &#39;potted plant&#39;, &#39;bed&#39;, &#39;N/A&#39;, &#39;dining table&#39;, &#39;N/A&#39;,<br>    &#39;N/A&#39;, &#39;toilet&#39;, &#39;N/A&#39;, &#39;tv&#39;, &#39;laptop&#39;, &#39;mouse&#39;, &#39;remote&#39;, &#39;keyboard&#39;,<br>    &#39;cell phone&#39;, &#39;microwave&#39;, &#39;oven&#39;, &#39;toaster&#39;, &#39;sink&#39;, &#39;refrigerator&#39;, &#39;N/A&#39;,<br>    &#39;book&#39;, &#39;clock&#39;, &#39;vase&#39;, &#39;scissors&#39;, &#39;teddy bear&#39;, &#39;hair drier&#39;,<br>    &#39;toothbrush&#39;, &#39;empty&#39;<br>]<br><br># Colors for visualization<br>COLORS = [[0.000, 0.447, 0.741], [0.850, 0.325, 0.098], <br>          [0.929, 0.694, 0.125], [0.494, 0.184, 0.556],<br>          [0.466, 0.674, 0.188], [0.301, 0.745, 0.933]]<br>COLORS *= 100<br><br><br>revert_normalization = T.Normalize(<br>    mean=[-.485/.229, -.456/.224, -.406/.225],<br>    std=[1/.229, 1/.224, 1/.225]<br>)<br><br><br>def plot_im_with_boxes(im, boxes, probs=None, ax=None):<br>    <br>    if ax is None:<br>        plt.imshow(im)<br>        ax = plt.gca()<br>        <br>    for i, b in enumerate(boxes.tolist()):<br>        xmin, ymin, xmax, ymax = b<br>        <br>        patch = plt.Rectangle(<br>            (xmin, ymin), xmax - xmin, ymax - ymin,<br>            fill=False, color=COLORS[i], linewidth=2)<br>        <br>        ax.add_patch(patch)<br>        if probs is not None:<br>            if probs.ndim == 1:<br>                cl = probs[i].item()<br>                text = f&#39;{CLASSES[cl]}&#39;<br>            else:<br>                cl = probs[i].argmax().item()<br>                text = f&#39;{CLASSES[cl]}: {probs[i,cl]:0.2f}&#39;<br>        else:<br>            text = &#39;&#39;<br>            <br>        ax.text(xmin, ymin, text, fontsize=7,<br>                bbox=dict(facecolor=&#39;yellow&#39;, alpha=0.5))<br>    <br>    <br>def preprocess_target(anno, im_w, im_h):<br>    <br>    anno = [obj for obj in anno <br>            if &#39;iscrowd&#39; not in obj or obj[&#39;iscrowd&#39;] == 0]<br><br>    boxes = [obj[&quot;bbox&quot;] for obj in anno]<br>    boxes = torch.as_tensor(<br>        boxes, dtype=torch.float32).reshape(-1, 4)<br>    <br>    # xywh -&gt; xyxy<br>    boxes[:, 2:] += boxes[:, :2]<br>    boxes[:, 0::2].clamp_(min=0, max=im_w)<br>    boxes[:, 1::2].clamp_(min=0, max=im_h)<br>    keep = (boxes[:, 3] &gt; boxes[:, 1]) &amp; (boxes[:, 2] &gt; boxes[:, 0])<br>    boxes = boxes[keep]<br><br>    classes = [obj[&quot;category_id&quot;] for obj in anno]<br>    classes = torch.tensor(classes, dtype=torch.int64)<br>    classes = classes[keep]<br><br>    # scales boxes to [0,1]<br>    boxes[:, 0::2] /= im_w<br>    boxes[:, 1::2] /= im_h<br>    boxes.clamp_(min=0, max=1)<br><br>    boxes = ops.box_convert(boxes, in_fmt=&#39;xyxy&#39;, out_fmt=&#39;cxcywh&#39;)<br>    return classes, boxes<br><br><br>class MyCocoDetection(datasets.CocoDetection):<br><br>    def __init__(self, *args, **kwargs):<br>        super().__init__(*args, **kwargs)<br>        self.edge = 480<br><br>        self.T = T.Compose([<br>            T.ToTensor(),<br>            T.Normalize(mean=[.485, .456, .406],<br>                        std=[.229, .224, .225]),<br>            T.Resize((self.edge, self.edge), antialias=True)<br>        ])<br><br>        self.T_target = preprocess_target<br><br>    def __getitem__(self, idx):<br>        img, target = super().__getitem__(idx)<br>        # PIL image<br>        w, h = img.size<br><br>        input_ = self.T(img)<br>        classes, boxes = self.T_target(target, w, h)<br><br>        return input_, (classes, boxes)<br><br><br>def collate_fn(inputs):<br>    input_ = torch.stack([i[0] for i in inputs])<br>    classes = tuple([i[1][0] for i in inputs])<br>    boxes = tuple([i[1][1] for i in inputs])<br>    return input_, (classes, boxes)</pre><p>Then, we import the tiny set.</p><pre>train_ds = MyCocoDetection(<br>    &#39;tiny_coco_dataset/tiny_coco/train2017/&#39;,<br>    &#39;tiny_coco_dataset/tiny_coco/annotations/instances_train2017.json&#39;,<br>)<br><br>train_loader = DataLoader(<br>    train_ds, batch_size=4, shuffle=False, collate_fn=collate_fn)<br><br>print(f&#39;\nNumber of training samples: {len(train_ds)}&#39;)<br># Number of training samples: 50</pre><p>The tiny COCO repo has only 50 samples; that’ll be enough for a fair presentation of how DETR trains and predicts. Let’s check what this data looks like:</p><pre>input_, (target) = next(iter(train_loader))<br>fig = plt.figure(figsize=(10, 10), constrained_layout=True)<br><br>for ix in range(4):<br>    t_cl = target[0][ix].cuda()<br>    t_bbox = target[1][ix].cuda()<br><br>    t_bbox = ops.box_convert(<br>        t_bbox*480, in_fmt=&#39;cxcywh&#39;, out_fmt=&#39;xyxy&#39;)<br><br>    im = revert_normalization(input_)[ix].\<br>        permute(1,2,0).cpu().clip(0,1)<br><br>    ax = fig.add_subplot(2, 2, ix+1)<br>    ax.imshow(im)<br>    plot_im_with_boxes(im, t_bbox, t_cl, ax=ax)<br>    ax.set_axis_off()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ri-f0jFzClud6tegFBrAvQ.jpeg" /></figure><h3>The DETR Architecture</h3><p>The DETR architecture is well posted on the diagram below of the original paper.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/0*RRsVDPRH8PyKa3Jv" /><figcaption>The DETR diagram [1]</figcaption></figure><p>DETR presents four main components:</p><ol><li><strong>CNN Backbone</strong> extracts the image features and compacts the original image dimensions, similar to an embedding layer.</li><li><strong>Transformer’s encoder</strong> adds the PE information to the tokens and processes it through multiple layers, applying the self-attention mechanism and, hence, producing high-level features.</li><li><strong>Transformer’s decoder</strong> receives objects’ queries as input and applies cross-attention between the queries and the encoder’s feature map. Object queries are plain positional embedding information.</li><li><strong>FFN Heads </strong>are composed of two heads despite the diagram above showing four. One head has 4 output units to predict the 4 box coordinates, and the other has 92 output units to predict the box classes. The FFN heads are just a sequence of linear layers and activation functions.</li></ol><p>Now, we delve into each component’s mechanism and illustrate them in the code. Here are some abbreviations to help us cope with the tensors’ dimensions: B (Batch), C (Channels), H (Height), W (Width), T (Tokens), T’ (squared root of T), and Q (Queries).</p><h4>1. CNN Backbone</h4><p>The backbone extracts image features and shrinks the original image dimensions, turning patches into embeddings. When using a standard ResNet-50, the dimension is shrunk by a factor of 32, which is equivalent to using a patch size of 32 on a vanilla ViT.</p><p>Consider an image input of (480, 480, 3), which translates to an input tensor with dimensions (1, 3, 480, 480). After passing the ResNet backbone, the input tensor becomes a coarse feature map of dimensions (1, 2048, 15, 15). The spatial information is compressed, but the depth dimension is enlarged.</p><p>The large number of channels makes the computational load heavy for the transformer’s encoder. This load is alleviated by applying a subsequent 1x1 convolutional layer to reduce the channel dimension from 2048 to 256. <br>In this stage, the data tensor changed from (B, 3, H, W) to (B, C, T&#39;, T&#39;). The number of channels of the embeddings (C = 256) is held through the entire transformer process, and this value is called the “model width”.</p><p>See the code:</p><pre><br>backbone = create_feature_extractor(<br>    torch.hub.load(&#39;pytorch/vision:v0.10.0&#39;, &#39;resnet50&#39;, pretrained=True),<br>    return_nodes={&#39;layer4&#39;: &#39;layer4&#39;}<br>)<br><br>d_model = 256<br>conv1x1 = nn.Conv2d(2048, d_model, kernel_size=1, stride=1)<br><br>x = torch.randn((1, 3, 480, 480))<br>embeddings = backbone(x)[&#39;layer4&#39;]<br>embeddings = conv1x1(embeddings)<br><br>print(x.shape, embeddings.shape)<br># torch.Size([1, 3, 480, 480]) torch.Size([1, 256, 15, 15])</pre><h4>2. Transformer’s encoder</h4><p>Some actions are required before feeding the embeddings information to the encoder: 1) rearrange the image embeddings to a 1-D sequence length, and 2) add positional encoding information.</p><p>Rearranging the embeddings to a 1-D sequence length implies to flatten the tensor’s height and width information, and to move the channel axis to the last dimension, such that the tensor dimensions change from (B, C, T’, T’) to (B, T, C).</p><p>While we’re not using tokens in the same way as language models, they serve a similar purpose. The tokens’ length here means the number of elements from the embeddings. In this example, the number of tokens is 15 * 15 = 225, which is the number of spatial units resulting from the backbone.</p><pre>embeddings_old = embeddings.clone()<br>embeddings = rearrange(embeddings, &#39;b c h w -&gt; b (h w) c&#39;)<br><br>print(embeddings_old.shape, embeddings.shape)<br># torch.Size([1, 256, 15, 15]) torch.Size([1, 225, 256])</pre><p>The second required action is to add positional encoding. As the self-attention mechanism does not regard position information when crossing information between the embeddings, the PE helps to make spatial information relevant.</p><p>Some forms of adding PE into the embeddings exist, such as modeling the <em>x </em>and <em>y</em> patch coordinates into sine functions or even learning it along with the training. Both were tried on the DETR work. In our didactic example, we will also learn the PE parameters; the weights are initialized from a uniform distribution.</p><pre># the original implementation uses sine spatial PE<br>n_tokens = 225<br><br>inp_pe = nn.Parameter(<br>    torch.rand((1, n_tokens, d_model)),<br>    requires_grad=False<br>)<br><br>embeddings += inp_pe<br><br>print(inp_pe.shape, embeddings.shape)<br># torch.Size([1, 225, 256]) torch.Size([1, 225, 256])</pre><p>After completing the required pre-steps, we instantiate the encoder itself. The tensor dimensions (B, T, C), in our case (1, 255, 256), are held constant throughout the transformer pass, including the decoder.</p><pre>n_heads = 8<br>encoder_layer = nn.TransformerEncoderLayer(<br>    d_model=d_model, <br>    nhead=n_heads,<br>    dim_feedforward=4*d_model,<br>    batch_first=True<br>)<br><br>n_layers = 6<br>transformer_encoder = nn.TransformerEncoder(<br>    encoder_layer, num_layers=n_layers)<br><br>out_encoder = transformer_encoder(embeddings)<br><br>print(out_encoder.shape)<br># torch.Size([1, 225, 256])</pre><h4>3. Transformer’s decoder</h4><p>A highlight concept introduced by DETR is object queries. The queries represent the asking for a box prediction; hence, Q object queries return Q number of box predictions. In a practical outlook, the queries are plain PE that feed the decoder.</p><p>One caveat: when running B instances simultaneously, the Q information is replicated for the B length; the same happens with the encoder’s PE. The queries tensor has the shape of (B, Q, C).</p><pre>n_queries = 100<br><br>queries = nn.Parameter(<br>    torch.rand((1, n_queries, d_model)),<br>    requires_grad=False<br>)<br><br>print(queries.shape, embeddings.shape)<br># torch.Size([1, 100, 256]) torch.Size([1, 225, 256])</pre><p>The decoder has a symmetric configuration with the encoder, i.e., the same number of heads and layers.</p><pre>decoder_layer = nn.TransformerDecoderLayer(<br>    d_model=d_model,<br>    nhead=n_heads,<br>    dim_feedforward=4*d_model,<br>    batch_first=True<br>)<br><br>transformer_decoder = nn.TransformerDecoder(<br>    decoder_layer, num_layers=n_layers)<br><br>out_decoder = transformer_decoder(queries, out_encoder)<br><br>print(out_decoder.shape, queries.shape, out_encoder.shape)<br># torch.Size([1, 100, 256]) torch.Size([1, 100, 256]) torch.Size([1, 225, 256])</pre><h4>4. FFN Heads</h4><p>Finally, at the top of the architecture, two FFNs receive the decoder’s outcomes as input. One FFN predicts the box classes, and the other predicts the box coordinates. In terms of dimensions, the decoder’s outcome of shape (B, T, Q) is turned into (B, Q, 92) by the FFN in charge of the class predictions and into (B, Q, 4) by the FFN in charge of the coordinates predictions.</p><pre>n_classes = 92<br><br>linear_class = nn.Linear(d_model, n_classes)<br>linear_bbox = nn.Linear(d_model, 4)<br><br>pred_classes = linear_class(out_decoder)<br>pred_bboxes = linear_bbox(out_decoder)<br><br>print(pred_classes.shape, pred_bboxes.shape)<br># torch.Size([1, 100, 92]) torch.Size([1, 100, 4])</pre><h4>Putting it all together</h4><p>After joining all the components, we define the DETR architecture class. Some additional details: dropout layers with a probability of 0.1 are added to the transformers’ encoder and decoder layers; and, BatchNorm layers from the backbone have frozen statistics, ie., they are not trained together with the new task — the DETR authors say this is a widely adopted practice in object detection.</p><pre><br>def get_hook(outs, name):<br>    def hook(self, input, output):<br>        outs[name] = output<br>    return hook<br><br><br>class DETR(nn.Module):<br><br>    def __init__(<br>        self, d_model=256, n_classes=92, n_tokens=225,<br>        n_layers=6, n_heads=8, n_queries=100<br>    ):<br>        super().__init__()<br><br>        self.backbone = create_feature_extractor(<br>            torch.hub.load(&#39;pytorch/vision:v0.10.0&#39;, &#39;resnet50&#39;, pretrained=True),<br>            return_nodes={&#39;layer4&#39;: &#39;layer4&#39;}<br>        )<br><br>        self.conv1x1 = nn.Conv2d(<br>            2048, d_model, kernel_size=1, stride=1)<br><br>        self.pe_encoder = nn.Parameter(<br>            torch.rand((1, n_tokens, d_model)),<br>            requires_grad=True<br>        )<br><br>        encoder_layer = nn.TransformerEncoderLayer(<br>            d_model=d_model, <br>            nhead=n_heads,<br>            dim_feedforward=4*d_model,<br>            dropout=0.1,<br>            batch_first=True,<br>        )<br>        <br>        self.transformer_encoder = nn.TransformerEncoder(<br>            encoder_layer, num_layers=n_layers)<br><br>        self.queries = nn.Parameter(<br>            torch.rand((1, n_queries, d_model)),<br>            requires_grad=True<br>        )<br><br>        decoder_layer = nn.TransformerDecoderLayer(<br>            d_model=d_model,<br>            nhead=n_heads,<br>            dim_feedforward=4*d_model,<br>            dropout=0.1, <br>            batch_first=True<br>        )<br>        <br>        self.transformer_decoder = nn.TransformerDecoder(<br>            decoder_layer, num_layers=n_layers)<br><br>        self.linear_class = nn.Linear(d_model, n_classes)<br>        self.linear_bbox = nn.Linear(d_model, 4)<br><br>        # Add hooks to get intermediate outcomes<br>        self.decoder_outs = {}    <br>        for i, L in enumerate(self.transformer_decoder.layers):<br>            name = f&#39;layer_{i}&#39;<br>            L.register_forward_hook(<br>                get_hook(self.decoder_outs, name))<br>        <br>    <br>    def train(self, mode=True):<br>        def freeze_batchnorm(module):<br>            # Switch to evaluation mode (uses running stats)<br>            if isinstance(module, nn.BatchNorm2d):<br>                module.eval()<br>                for param in module.parameters():<br>                    param.requires_grad = False<br>  <br>        super().train(mode=mode)<br>        self.backbone.apply(freeze_batchnorm)<br><br>    def forward(self, x):<br>        tokens = self.backbone(x)[&#39;layer4&#39;]<br>        tokens = self.conv1x1(tokens)<br>        tokens = rearrange(tokens, &#39;b c h w -&gt; b (h w) c&#39;)<br><br>        out_encoder = self.transformer_encoder(<br>            tokens + self.pe_encoder)<br>        <br>        out_decoder = self.transformer_decoder(<br>            self.queries.repeat(len(out_encoder), 1, 1),<br>            out_encoder)<br><br>        # Compute outcomes for all intermediate<br>        # decoder&#39;s layers<br>        outs = {}<br>        for n, o in self.decoder_outs.items():<br>            outs[n] = {<br>                &#39;cl&#39;: self.linear_class(o),<br>                &#39;bbox&#39;: self.linear_bbox(o)<br>            }<br>        <br>        return outs</pre><pre>detr = DETR(<br>    d_model=256, n_classes=92, n_tokens=225, <br>    n_layers=6, n_heads=8, n_queries=100<br>)<br><br>x = torch.randn((1, 3, 480, 480))<br>outs = detr(x)<br>pred_cl, pred_boxes = outs[&#39;layer_5&#39;].values()<br><br>print(pred_cl.shape, pred_boxes.shape)<br># torch.Size([1, 100, 92]) torch.Size([1, 100, 4])</pre><p>The code above has forward hooks due to auxiliary losses, which will be explained in the next session.</p><h3>Training</h3><p>Our next step is to understand the particularities of the DETR training. We break this session into three parts: 1) the set prediction problem, 2) the compound loss function, and 3) auxiliary losses.</p><h4>The Set Prediction Problem</h4><p>An innovative approach in the DETR project was framing the detection task as a set prediction task. This paradigm involves matching elements between the set of predictions and the set of labels, such that the sum of the errors between the matches is minimized rather than the sum of individual box errors. The optimal set of matches is computed by a bipartite matching algorithm called the Hungarian algorithm.</p><p>The Hungarian algorithm operates on a cost matrix that stores the dissimilarity between all elements of both sets. It’s important to note that not every element matches its best correspondence, as the bipartite algorithm optimizes the sum of the matches.</p><p>The cost matrix has dimensions (Q, len(target)), in which each cell (i, j) stores a dissimilarity metric of the pair (the prediction “i” and the label “j”). The dissimilarity metric is the weight summation of the generalized intersection over union (GIoU) over the boxes, L1-norm between the box coordinates (xy, xy), and the L1-norm between the box class probabilities. The respective weights are 1, 5, and 2. The Hungarian algorithm is applied with the method linear_sum_assignment.</p><p>A critical point during training is dealing with the varied number of boxes in each COCO image. Among the COCO images, the number of boxes varies between 1 and 63 boxes. On the other hand, DETR assumes a fixed number of predictions, 100, which is the query size Q. This problem is solved by assigning an additional “empty class” to all predictions without a correspondent match.</p><p>See the code below:</p><pre>train_loader = DataLoader(<br>    train_ds, batch_size=1, shuffle=False, collate_fn=collate_fn)<br><br>input_, (tgt_cl, tgt_bbox) = next(iter(train_loader))<br><br>outs = detr(input_)<br>out_cl, out_bbox = outs[&#39;layer_5&#39;].values()<br><br>train_loader = DataLoader(<br>    train_ds, batch_size=1, shuffle=False, collate_fn=collate_fn)<br><br>input_, (tgt_cl, tgt_bbox) = next(iter(train_loader))<br><br>outs = detr(input_)<br>out_cl, out_bbox = outs[&#39;layer_5&#39;].values()<br><br>out_bbox = out_bbox.sigmoid()<br><br>o_bbox = out_bbox[0]<br>t_bbox = tgt_bbox[0]<br>o_cl = out_cl[0]<br>t_cl = tgt_cl[0]<br>    <br>o_probs = o_cl.softmax(dim=-1)<br><br># Negative sign here because we want the maximum magnitude<br>C_classes = -o_probs[..., t_cl]<br><br># Positive sign here because we want to shrink the l1-norm<br>C_boxes = torch.cdist(o_bbox, t_bbox, p=1)<br><br># Negative sign here because we want the maximum magnitude<br>C_giou = -ops.generalized_box_iou(<br>    ops.box_convert(o_bbox, in_fmt=&#39;cxcywh&#39;, out_fmt=&#39;xyxy&#39;),<br>    ops.box_convert(t_bbox, in_fmt=&#39;cxcywh&#39;, out_fmt=&#39;xyxy&#39;)<br>)<br><br>C_total = 1*C_classes + 5*C_boxes + 2*C_giou<br><br># Convert the tensor to numpy array<br>C_total = C_total.cpu().detach().numpy()<br><br># Find the optimum pairs that produces the minimum summation.<br># The method returns the pair indices<br>o_ixs, t_ixs = linear_sum_assignment(C_total)<br><br>print(t_bbox.shape, o_bbox.shape, C_total.shape)<br>out_bbox = out_bbox.sigmoid()<br><br>o_bbox = out_bbox[0]<br>t_bbox = tgt_bbox[0]<br>o_cl = out_cl[0]<br>t_cl = tgt_cl[0]<br>    <br>o_probs = o_cl.softmax(dim=-1)<br><br># Negative sign here because we want the maximum magnitude<br>C_classes = -o_probs[..., t_cl]<br><br># Positive sign here because we want to shrink the l1-norm<br>C_boxes = torch.cdist(o_bbox, t_bbox, p=1)<br><br># Negative sign here because we want the maximum magnitude<br>C_giou = -ops.generalized_box_iou(<br>    ops.box_convert(o_bbox, in_fmt=&#39;cxcywh&#39;, out_fmt=&#39;xyxy&#39;),<br>    ops.box_convert(t_bbox, in_fmt=&#39;cxcywh&#39;, out_fmt=&#39;xyxy&#39;)<br>)<br><br>C_total = 1*C_classes + 5*C_boxes + 2*C_giou<br><br># Convert the tensor to numpy array<br>C_total = C_total.cpu().detach().numpy()<br><br># Find the optimum pairs that produces the minimum summation.<br># The method returns the pair indices<br>o_ixs, t_ixs = linear_sum_assignment(C_total)<br><br>print(t_bbox.shape, o_bbox.shape, C_total.shape)<br># torch.Size([26, 4]) torch.Size([100, 4]) (100, 26)</pre><h4>The Loss Function</h4><p>The loss computation is close to the dissimilarity metric previously computed for the Hungarian algorithm, but adjustments exist. For example, the standard cross-entropy between the class distributions is adopted instead of using the bare probability values for the class component. Besides, all predictions that did not match any target are assigned as empty classes, and a cross-entropy loss is computed apart for them and added into the final loss summation. The complete training code is shown below:</p><pre># Transform indices to tensors<br>o_ixs = torch.IntTensor(o_ixs)<br>t_ixs = torch.IntTensor(t_ixs)<br><br># Reorder o_ixs to naturally align with target_cl length, such<br># the pairs are {(o_ixs[0], t[0]), {o_ixs[1], t[1]}, ...}<br>o_ixs = o_ixs[t_ixs.argsort()]<br><br># Average over the number of boxes, not the number of coordinates<br>num_boxes = len(t_bbox)<br>loss_bbox = F.l1_loss(<br>    o_bbox[o_ixs], t_bbox, reduce=&#39;sum&#39;) / num_boxes<br><br># Vectorize the operation<br>target_gIoU = ops.generalized_box_iou(<br>    ops.box_convert(o_bbox[o_ixs], in_fmt=&#39;cxcywh&#39;, out_fmt=&#39;xyxy&#39;),<br>    ops.box_convert(t_bbox, in_fmt=&#39;cxcywh&#39;, out_fmt=&#39;xyxy&#39;)<br>)<br># Get only the matrix diagonal that contains the bipartite pairs<br># and transform gIoU into a loss<br>loss_giou = 1 - torch.diag(target_gIoU).mean()<br><br># Assign empty class for the outside predictions<br>queries_classes_label = torch.full(o_probs.shape[:1], 91)<br>queries_classes_label[o_ixs] = t_cl<br>loss_class = F.cross_entropy(o_cl, queries_classes_label)<br><br>loss_total = 1*loss_class + 5*loss_bbox + 2*loss_giou<br>print(loss_total)<br># tensor(5.5788, grad_fn=&lt;AddBackward0&gt;)</pre><h4><strong>Auxiliary losses</strong></h4><p>Now that we know how to compute loss, an important detail to speed up convergence and stabilize training is auxiliary losses. In this case, auxiliary losses only applies the same loss function to the outcome of all the intermediate layers of the transformer’s decoder.<br> <br>To do so, all the intermediate features are obtained with the get_hook function declared in the DETR class snippet. Then, the same FFN heads are used to predict box coordinates and classes for each decoder layer. It becomes clearer after checking the forward method of class DETR defined above.</p><h4>Putting it all together</h4><p>We join all the pieces above and make it run over the short train set of 50 samples. To speed up the training, we do not train the backbone together.</p><p>Beforehand, I create a specific function to compute the sample loss.</p><pre><br>def compute_sample_loss(<br>    o_bbox, t_bbox, o_cl, t_cl, n_queries=100):<br>    <br>    # If  the example has none box, we just feed it with empty classes.<br>    if len(t_cl) &gt; 0:<br>        <br>        t_bbox = t_bbox.cuda()<br>        t_cl = t_cl.cuda()<br><br>        o_probs = o_cl.softmax(dim=-1)<br><br>        # Negative sign here because we want the maximum magnitude<br>        C_classes = -o_probs[..., t_cl]<br>        <br>        # Positive sign here because we want to shrink the l1-norm<br>        C_boxes = torch.cdist(o_bbox, t_bbox, p=1)<br>        <br>        # Negative sign here because we want the maximum magnitude<br>        C_giou = -ops.generalized_box_iou(<br>            ops.box_convert(o_bbox, in_fmt=&#39;cxcywh&#39;, out_fmt=&#39;xyxy&#39;),<br>            ops.box_convert(t_bbox, in_fmt=&#39;cxcywh&#39;, out_fmt=&#39;xyxy&#39;)<br>        )<br>        <br>        C_total = 1*C_classes + 5*C_boxes + 2*C_giou<br>        <br>        # Convert the tensor to numpy array<br>        C_total = C_total.cpu().detach().numpy()<br>        <br>        # Find the optimum pairs that produces the minimum summation.<br>        # the method returns the pair indices<br>        o_ixs, t_ixs = linear_sum_assignment(C_total)            <br>        <br>        # Transform indices to tensors<br>        o_ixs = torch.IntTensor(o_ixs)<br>        t_ixs = torch.IntTensor(t_ixs)<br>        <br>        # Reorder o_ixs to naturally align with target_cl length, such<br>        # the pairs are {(o_ixs[0], t[0]), {o_ixs[1], t[1]}, ...}<br>        o_ixs = o_ixs[t_ixs.argsort()]<br>        <br>        # Average over the number of boxes, not the number of coordinates<br>        num_boxes = len(t_bbox)<br>        loss_bbox = F.l1_loss(<br>            o_bbox[o_ixs], t_bbox, reduce=&#39;sum&#39;) / num_boxes<br>        <br>        # vectorize the operation<br>        target_gIoU = ops.generalized_box_iou(<br>            ops.box_convert(o_bbox[o_ixs], in_fmt=&#39;cxcywh&#39;, out_fmt=&#39;xyxy&#39;),<br>            ops.box_convert(t_bbox, in_fmt=&#39;cxcywh&#39;, out_fmt=&#39;xyxy&#39;)<br>        )<br>        # get only the matrix diagonal that contains the bipartite pairs<br>        # and transform gIoU into a loss<br>        loss_giou = 1 - torch.diag(target_gIoU).mean()<br>        <br>        # assign empty class for the outside predictions<br>        queries_classes_label = torch.full(o_probs.shape[:1], 91).cuda()<br>        queries_classes_label[o_ixs] = t_cl<br>        loss_class = F.cross_entropy(o_cl, queries_classes_label)<br><br>    else:<br>        queries_classes_label = torch.full((n_queries,), 91).cuda()<br>        loss_class = F.cross_entropy(o_cl, queries_classes_label)<br>        loss_bbox = loss_giou = torch.tensor(0)<br>    <br>    return loss_class, loss_bbox, loss_giou</pre><p>Then, we instantiate the model and set the optimizer.</p><pre># Ensure saving directory exists<br>![ ! -d ckpts-tiny ] &amp;&amp; mkdir ckpts-tiny<br><br>detr = DETR(d_model=256, n_classes=92, n_tokens=225, <br>            n_layers=6, n_heads=8, n_queries=100)<br>detr.cuda()<br><br>backbone_params = [<br>    p for n, p in detr.named_parameters() if &#39;backbone.&#39; in n]<br><br>for p in detr.backbone.parameters():<br>    p.requires_grad = False<br>    <br>transformer_params = [<br>    p for n, p in detr.named_parameters() if &#39;backbone.&#39; not in n]<br><br>optimizer = AdamW([<br>    {&#39;params&#39;: transformer_params, &#39;lr&#39;: 1e-5},<br>], weight_decay=1e-4)<br><br><br>nparams = sum([p.nelement() for p in detr.parameters()]) / 1e6<br>print(f&#39;DETR params: {nparams:.1f}M&#39;)<br># DETR params: 35.2M</pre><p>Finally, the training iteration script:</p><pre><br>torch.set_grad_enabled(True)<br>detr.train()<br><br><br>batch_size = 16<br>train_loader = DataLoader(<br>    train_ds, batch_size=batch_size,<br>    shuffle=True, collate_fn=collate_fn)<br><br><br>niters = 40000<br>print_every_n = 1000<br>save_every_n = 10000<br>losses = []<br><br>hist = []<br>iters = 1<br><br><br>while iters &lt;= niters:<br>    <br>    for input_, (tgt_cl, tgt_bbox) in train_loader:<br>        input_ = input_.cuda()<br>        <br>        outs = detr(input_)<br>        <br>        loss = torch.Tensor([0]).cuda()<br>        for name, out in outs.items(): <br>            <br>            out[&#39;bbox&#39;] = out[&#39;bbox&#39;].sigmoid()<br>            <br>            for o_bbox, t_bbox, o_cl, t_cl in zip(<br>                out[&#39;bbox&#39;], tgt_bbox, out[&#39;cl&#39;], tgt_cl):<br>        <br>                loss_class, loss_bbox, loss_giou = compute_sample_loss(<br>                    o_bbox, t_bbox, o_cl, t_cl)<br>                <br>                sample_loss = 1*loss_class + 5*loss_bbox + 2*loss_giou<br>                <br>                loss += sample_loss / batch_size / len(outs)<br>            <br>        optimizer.zero_grad(set_to_none=True)<br>        loss.backward()<br>        <br>        # clip gradient norms<br>        nn.utils.clip_grad_norm_(detr.parameters(), .1)<br>        optimizer.step()<br>        <br>        losses.append(loss.item())<br><br>        if iters % print_every_n == 0:<br>            loss_avg = np.mean(losses[-10:])<br>            print_text = f&#39;iters: {iters},\tloss: {loss_avg:.4f}&#39;<br>            print(print_text)<br>            print(f&#39;loss_class: {loss_class.item():.4f}\tloss_bbox: {loss_bbox.item():.4f}\tloss_giou: {loss_giou.item():.4f}\t&#39;)<br>            <br>            hist.append(loss_avg)<br>            losses = []<br><br>        if iters % save_every_n == 0 and iters &gt; 0:<br>            str_iters = str(iters)<br>            str_iters = &#39;0&#39;*(6-len(str_iters)) + str_iters<br>            torch.save(detr.state_dict(), f&#39;ckpts-tiny/model_it{str_iters}.pt&#39;)<br>            np.save(f&#39;ckpts-tiny/hist_it{str_iters}.npy&#39;, hist)<br><br>        iters += 1<br>        if iters &gt; niters:<br>            break</pre><p>I ran the training scripts for 40k iterations. It took a large number of iterations for the model be able to fit only 50 training samples. The long run is because we are training a transformer network without any pre-training.</p><p>After seeing the log-loss curve below, it’s clear that there is still room for many more iterations, we just wanted a fast test, though.</p><pre>plt.plot(np.log(hist))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*DK7349eRHCIOYEMbijdKfw.jpeg" /><figcaption>Log-scaled loss captured during training.</figcaption></figure><h3>Inference</h3><p>In the prediction mode, we can disregard all the bipartite matching stuff for plain inference and only apply the neural network forward pass. Besides, every predicted box classified as “empty class” is discarded. See the figure below; the predictions are on the left column, and the labels are on the right one.</p><pre>torch.set_grad_enabled(False)<br>detr.eval()<br><br>eval_loader = DataLoader(<br>    train_ds, batch_size=2, shuffle=True,<br>    collate_fn=collate_fn)<br><br>input_, (tgt_cl, tgt_bbox) = next(iter(eval_loader))[:2]<br><br># out_cl, out_bbox = detr(input_.cuda())<br>out_cl, out_bbox = detr(input_.cuda())[&#39;layer_5&#39;].values()<br>out_bbox = out_bbox.sigmoid().cpu()<br>out_cl = out_cl.cpu()<br><br><br>fig, axs = plt.subplots(2, 2, figsize=(10, 10), constrained_layout=True)<br>[a.set_axis_off() for a in axs.ravel()]<br>    <br>for ix in range(len(out_bbox)):<br>    o_cl = out_cl[ix]<br>    t_cl = tgt_cl[ix]<br>    o_bbox = out_bbox[ix]<br>    t_bbox = tgt_bbox[ix]<br>    <br>    o_probs = o_cl.softmax(dim=-1)<br>    o_bbox = ops.box_convert(o_bbox*480, in_fmt=&#39;cxcywh&#39;, out_fmt=&#39;xyxy&#39;)<br>    t_bbox = ops.box_convert(t_bbox*480, in_fmt=&#39;cxcywh&#39;, out_fmt=&#39;xyxy&#39;)<br>    <br>    im = revert_normalization(input_)[ix].permute(1,2,0).cpu().clip(0,1)<br><br>    o_keep = o_probs.argmax(-1) != 91<br><br>    axs[ix, 0].imshow(im)<br>    plot_im_with_boxes(im, o_bbox[o_keep], o_probs[o_keep], ax=axs[ix, 0])<br>    axs[ix, 1].imshow(im)<br>    plot_im_with_boxes(im, t_bbox, t_cl, ax=axs[ix, 1])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*HdzrnFdXQRJllWJaa3m6uA.jpeg" /></figure><p>The results are good for the tiny training set, proving that our code works and the model learns despite needing more iterations to completely overfit the samples. As pointed by (Zhu et al., 2020), transformers take way longer than CNNs when trained from scratch.</p><p>For curiosity, we can also check the results on the validation set by changing the eval_loader dataset object; we do not expect the model could generalize after just seeing 50 samples.</p><pre><br>val_ds = MyCocoDetection(<br>    &#39;tiny_coco_dataset/tiny_coco/val2017/&#39;,<br>    &#39;tiny_coco_dataset/tiny_coco/annotations/instances_val2017.json&#39;,<br>)<br><br>eval_loader = DataLoader(<br>    val_ds, batch_size=2, shuffle=False, collate_fn=collate_fn)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*tRkvi9XztmBP1vZ9_sLciw.jpeg" /></figure><p>Unsatisfied with the validation performance of our model. I ran our code with the complete COCO dataset to see how it goes.</p><h3>The Complete COCO Set</h3><p>To download the original COCO dataset, check the code below. The original zip file is 19 GB in size.</p><pre>wget http://images.cocodataset.org/annotations/annotations_trainval2017.zip<br>wget http://images.cocodataset.org/zips/train2017.zip<br><br>unzip -d. annotations_trainval2017.zip<br>unzip -d. train2017.zip</pre><p>We instantiate a new train_ds.</p><pre>train_ds = MyCocoDetection(<br>    &#39;coco/train2017/&#39;,<br>    &#39;coco/annotations/instances_train2017.json&#39;,<br>)<br>print(f&#39;\nNumber of training samples: {len(train_ds)}&#39;)<br># Number of training samples: 118287<br><br><br>val_ds = MyCocoDetection(<br>    &#39;coco/val2017/&#39;,<br>    &#39;coco/annotations/instances_val2017.json&#39;,<br>)<br>print(f&#39;\nNumber of val samples: {len(val_ds)}&#39;)<br># Number of val samples: 5000</pre><p>In the original DETR work, they trained 300 epochs with a batch size of 64. To get a glimpse of this network capacity, we will do a shorter experiment. We ran 150 epochs with a batch size of 16. The reduced batch size is due to GPU constraints, and the number of epochs was the minimum needed for the model to present reasonable results. The results were quite bad until epoch 100.</p><p>The training code follows the same setup above. This time, the backbone parameters are trained together with the transformer parameters. We also adapt the code to run and evaluate based on epochs; one COCO epoch equals 7393 iterations of batch size 16.</p><p>Our current setup of resized inputs with a size of 480px requires 16GB of GPU VRAM to fit the 16 samples simultaneously.</p><pre># Ensure saving directory exists<br>![ ! -d ckpts ] &amp;&amp; mkdir ckpts<br><br><br>detr = DETR(<br>    d_model=256, n_classes=92, n_tokens=225, <br>    n_layers=6, n_heads=8, n_queries=100<br>)<br>detr.cuda()<br><br>backbone_params = [<br>    p for n, p in detr.named_parameters() if &#39;backbone.&#39; in n]<br><br>remaining_params = [<br>    p for n, p in detr.named_parameters() if &#39;backbone.&#39; not in n]<br><br>    <br>optimizer = AdamW([<br>    {&#39;params&#39;: remaining_params, &#39;lr&#39;: 1e-4},<br>    {&#39;params&#39;: backbone_params, &#39;lr&#39;: 1e-5},<br>], weight_decay=1e-4)<br><br><br>nparams = sum([p.nelement() for p in detr.parameters()]) / 1e6<br>print(f&#39;DETR params: {nparams:.1f}M&#39;)<br># DETR params: 35.2M</pre><p>The training loop based on epochs:</p><pre>torch.set_grad_enabled(True)<br>detr.train()<br><br>batch_size = 16<br>train_loader = DataLoader(<br>    train_ds, batch_size=batch_size, shuffle=True, collate_fn=collate_fn)<br><br><br>nepochs = 2000000<br>losses = []<br><br>hist = []<br><br><br>for i_epoch in range(len(hist)+1, nepochs):<br><br>    for input_, (tgt_cl, tgt_bbox) in train_loader:<br>        input_ = input_.cuda()<br><br>    <br>        outs = detr(input_)<br>        <br>        loss = torch.Tensor([0]).cuda()<br>        for name, out in outs.items(): <br>            out[&#39;bbox&#39;] = out[&#39;bbox&#39;].sigmoid()<br>            <br>            for o_bbox, t_bbox, o_cl, t_cl in zip(<br>                out[&#39;bbox&#39;], tgt_bbox, out[&#39;cl&#39;], tgt_cl):<br>                <br>                loss_class, loss_bbox, loss_giou = compute_sample_loss(o_bbox, t_bbox, o_cl, t_cl)<br>                sample_loss = 1*loss_class + 5*loss_bbox + 2*loss_giou<br>                # sample_loss = loss_bbox<br>                loss += sample_loss / batch_size #/ len(outs)<br>        <br>        optimizer.zero_grad(set_to_none=True)<br>        loss.backward()<br>        <br>        # clip gradient norms<br>        nn.utils.clip_grad_norm_(detr.parameters(), .1)<br>        optimizer.step()<br>        <br>        losses.append(loss.item())<br>        <br>    if i_epoch % 1 == 0:<br>        loss_avg = np.mean(losses)<br>        print_text = f&#39;epoch: {i_epoch},\tloss: {loss_avg:.4f}&#39;<br>        print(print_text)<br>        print(f&#39;loss_class: {loss_class.item():.4f}\tloss_bbox: {loss_bbox.item():.4f}\tloss_giou: {loss_giou.item():.4f}\t&#39;)<br>        <br>        hist.append(loss_avg)<br>        losses = []<br>    <br>    if i_epoch % 1 == 0:<br>        str_epoch = str(i_epoch)<br>        str_epoch = &#39;0&#39;*(3-len(str_epoch)) + str_epoch<br>        <br>        torch.save(<br>            {&#39;state&#39;:detr.state_dict(),<br>             &#39;opt&#39;: optimizer.state_dict()},<br>            f&#39;ckpts/model_ep{str_epoch}.pt&#39;)<br>        <br>        np.save(f&#39;ckpts/hist-ep{str_epoch}.npy&#39;, hist)</pre><p>After seeing the log-loss curve, the loss seems to have converged already and no more running is needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*rpDFww0jj4VLl-U_S_VfcA.jpeg" /><figcaption>Log scaled loss captured during training.</figcaption></figure><p>Then, let’s see the performance of validation samples: predictions on the left column and labels on the right one. The code is the same as seen above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*ELx3tr1XOP7b7LR4uwQ68A.jpeg" /></figure><p>Finally, we start to perceive some generalization capability.</p><h3>Takeaways</h3><ul><li>DETR was one of the pioneers of transformer-based work for computer vision tasks that presented a competitive result. It fostered more investigation of this architecture in computer vision, eventually becoming a state-of-the-art paradigm.</li><li>DETR introduced new elements for deep learning projects like object queries and the set prediction problem, which were fundamental to other works like Mask2Former and OneFormer, which achieve state-of-the-art results.</li><li>DETR can take a very long training time to start presenting fair results.</li></ul><h3>Final Words</h3><p>Thanks if you followed it until here. I hope this post helped you better understand the internal mechanisms of some of the last state-of-the-art deep learning works.</p><p>Please let me know if you like this post and also reach out for comments or feedback.</p><h3>Keep Tuned</h3><p>You can find the notebook related to this post <a href="https://github.com/tldrafael/understanding-and-coding-detr">here</a>.</p><p>This work is part of my PhD research done in the <a href="https://lapix.ufsc.br/">LAPIX</a> labs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/240/1*3L-KkGOK4pGKEbOO_67HZg.png" /><figcaption><a href="https://lapix.ufsc.br/">https://lapix.ufsc.br/</a></figcaption></figure><p>You can also follow a visual transformer lecture <a href="https://www.youtube.com/watch?v=yUQCvrI4xIo">here</a> (in Portuguese).</p><p>More of my works in <a href="http://github.com/tldrafael">github.com/tldrafael</a> and <a href="http://rtoledo.me/">rtoledo.me</a>.</p><h3>References</h3><p>Carion, Nicolas, et al. “End-to-end object detection with transformers.” <em>European conference on computer vision</em>. Cham: Springer International Publishing, 2020.</p><p>Cheng, Bowen, et al. “Masked-attention mask transformer for universal image segmentation.” <em>Proceedings of the IEEE/CVF conference on computer vision and pattern recognition</em>. 2022.</p><p>Jain, Jitesh, et al. “Oneformer: One transformer to rule universal image segmentation.” <em>Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition</em>. 2023.APA</p><p>Zhu, Xizhou, et al. “Deformable detr: Deformable transformers for end-to-end object detection.” <em>arXiv preprint arXiv:2010.04159</em> (2020).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80e4c206fbc8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Editing Face Attributes using VAE]]></title>
            <link>https://medium.com/@rafaeltol/editing-face-attributes-using-vae-733b47a76436?source=rss-4ebee3139190------2</link>
            <guid isPermaLink="false">https://medium.com/p/733b47a76436</guid>
            <category><![CDATA[face-manipulation]]></category>
            <category><![CDATA[variational-autoencoder]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Rafael Toledo]]></dc:creator>
            <pubDate>Mon, 31 May 2021 00:00:43 GMT</pubDate>
            <atom:updated>2023-12-10T20:52:33.861Z</atom:updated>
            <content:encoded><![CDATA[<p>You can play with the model and face editions in <a href="https://colab.research.google.com/drive/1auI8GsWtazP_FHHOX0iAZ6cyC2NX3cZE?usp=sharing">Colab</a> or check more about the code in the GitHub <a href="https://github.com/tldrafael/ManipulateFacialAttributesUsingVAE">repo</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Zktgq8tcapc4DmaL.jpg" /></figure><h3>1. Introduction</h3><p>This article shows a quick overview of a recent project that I developed while working on my master’s about how to edit attributes from faces using Variational Autoencoder (VAE)<a href="https://rtoledo.me/post/2021-05-31-edit-face-attributes-using-vae/edit-face-attributes-using-vae/#1">[1]</a><a href="https://rtoledo.me/post/2021-05-31-edit-face-attributes-using-vae/edit-face-attributes-using-vae/#5">[5]</a>.</p><p>In a practical view, VAE architecture is similar to a vanilla Autoencoder. However, at the Neural Network bottleneck their neurons learn a Multivariate Gaussian distribution such that their outcomes represent means and variances of the distribution.</p><p>This Multivariate Gaussian distribution is called a <strong>latent distribution</strong> $Z$, which means they are variables that cannot be observed directly. Thus the distribution of the latent variables is approximated using a Variational Bayesian (VB) approach, also called Variational Inference. As a matter o fact, what is approximated is the posterior distribution $p(Z|X)$. The prior distribution $Z$ is assumed to be a normal $N(0, 1)$.</p><p>Check out the next post for more information about VAE and its objective goal.</p><p>To realize this experiment. I used the <a href="http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html">celebA</a><a href="https://rtoledo.me/post/2021-05-31-edit-face-attributes-using-vae/edit-face-attributes-using-vae/#2">[2]</a> dataset, which contains around 200 thousand images labeled with 40 binary attributes.</p><h3>2. The Solution</h3><h4>2.1 The Architecture</h4><p>The proposed architecture has an additional decoder branch that predicts face masks. During training, the labels of the face masks are used too, which replaces the background of the reconstructed image such that the loss function is applied only over the face pixels. On the other hand, in the prediction mode, the background replacement is done by the predicted mask itself, not requiring any extra input but a sample image.</p><blockquote>On training</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/0*pzR9ZN1LC-wT6rU1.png" /></figure><blockquote>On prediction</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/0*Rh4XOb_4oJBivgOF.png" /></figure><p>This is, in fact, a deep model containing 145 layers and 16.6M parameters.</p><h4>2.2 Objective function</h4><p>The objective function is broken into three parts:</p><ol><li>Reconstruction of the image: I used the SSIM (Structural Similarity Index), which is a perceptual loss, and MAE (Mean Absolute Error) here. SSIM helps the perceptual quality like contrast and helps to avoid the usual blur from L1 and L2 losses. MAE helps to keep the original colors from the images.</li><li>Prediction of the face mask: I used the Binary Crossentropy and Dice loss.</li><li>Kullback-Leibler divergence: This is a specific loss function that comes from the variational inference approach, which minimizes the divergence and allows us to maximize the <em>Evidence Lower Bound</em> (ELBO) of the VB equation <a href="https://rtoledo.me/post/2021-05-31-edit-face-attributes-using-vae/edit-face-attributes-using-vae/#3">[3]</a>. This divergence measures the dissimilarity between the approximated posterior $q_\phi(z|x)$ and the prior $p(z)$ distribution. As we assume the prior are multivariate normals, we want to make the posterior distributions as normal as possible; therefore it also acts as a regularizer term.</li></ol><p>To not overextend this article, code implementation can be directly seen on the <a href="https://github.com/tldrafael/ManipulateFacialAttributesUsingVAE/blob/main/training.py">training.py</a> script.</p><h4>2.3 Training metrics</h4><p>The training ran for 50 hours on Colab Pro, it did 113 iterations of 4000 steps and a batch size of 32. The training metrics can be seen below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*06r3d3Gio1m5OEzx1-HQ3g.png" /></figure><p>SSIM and MAE decreased along the training and not suffering variance. The losses (BCE + Dice) on the mask show variance. Although the generalizing capacity looked visually good, the variance seems to be caused by slight gaps in the ground truth mask between the face, ears, and hair.</p><p>The KL divergence curve had an odd behavior, initially increasing on the first fifteen iterations and then decreasing afterward. That happened because I scaled the KL divergence factor to 1e-3 such its regularization effect doesn’t inhibit the reconstruction learning at the beginning of the training; however, it does not avoid the divergence either. After the reconstruction loss is low enough, the NN starts to “learn” the KL divergence too.</p><h4>2.4 Results</h4><p>As the main purpose of this architecture is to reconstruct the image, let’s see how it performs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XKYv-iBO6w4VMHgT.png" /><figcaption>Validation set examples</figcaption></figure><p>It does well, although some high-frequency details are lost on the skin and on the hair. Make them look polished.</p><h3>3. Editing Attributes</h3><p>One of the VAE advantages is its continuous latent space. Making it easy to interpolate vectors on the latent space, allowing us to edit attributes, or even generate artificial images from random sampling on the latent space <strong><em>Z</em></strong>.</p><h4>3.1 Attribute Spectrum</h4><p>As was seen in <a href="https://rtoledo.me/post/2021-05-31-edit-face-attributes-using-vae/edit-face-attributes-using-vae/#4">[4]</a>, you can use simple arithmetic to create an attribute vector and edit a sample. I took 10k positive and negative examples for each attribute to compute an attribute vector. Afterward, this vector can be added to the latent sample features. The vector values can be scaled in any factor or direction, expressing different attribute levels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Od5JVorLefV98YQZ.jpg" /><figcaption>Spectrum example</figcaption></figure><h4>3.2 Beards</h4><p>Another example is adding beards of different colors. For that, I created a vector joining the beard presence and the hair color. It is interesting to notice different correlated attributes that follow each color.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kC3NAaNNXDy7RUTg.jpg" /></figure><p>I used the same vector scale for all colors, but you can see that some beards were shallow as the blond ones, and others were long as the black ones. Besides, you have some slight changes in skin colors; for instance, the black beard examples got a darker skin tone.</p><h4>3.3 More Attributes</h4><p>In this example, I like to notice how the edition doesn’t affect “much” the pose and expression of the faces. On the second row, the smiling expression kept practically the same across different editions. Besides the eyeglasses of all three fitted on the correct position of the face, regarding the top of the nose and the head pose.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z2d7uEP3DHmMizo8.jpg" /></figure><h3>4. Conclusions</h3><p>We just checked that is possible to synthetically edit generic attributes on a face using VAE, still preserving the head pose and face expressions. However, it does have losses when reconstructing the skin and hair nuances of the original photos, making them smooth on everyone.</p><h4>References</h4><p>[1] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.</p><p>[2] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celeb faces attributes (celebA) dataset. Retrieved August, 15(2018):11, 2018.</p><p>[3] Pattern Recognition and Machine Learning, by Cristopher M. Bishop, Springer-Verlag New York, 2006, pp. 463–463.</p><p>[4] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks, 2016.</p><p>[5] Rezende, D. J., Mohamed, S., &amp; Wierstra, D. (2014, June). Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning (pp. 1278–1286). PMLR.</p><p><em>Originally published at </em><a href="https://rtoledo.me/post/2021-05-31-edit-face-attributes-using-vae/edit-face-attributes-using-vae/"><em>https://rtoledo.me</em></a><em> on May 31, 2021.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=733b47a76436" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>