<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sieun Park on Medium]]></title>
        <description><![CDATA[Stories by Sieun Park on Medium]]></description>
        <link>https://medium.com/@sieunpark77?source=rss-8a8e55663707------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*xGbGH-FyE5XeZN36g8ZPsA.png</url>
            <title>Stories by Sieun Park on Medium</title>
            <link>https://medium.com/@sieunpark77?source=rss-8a8e55663707------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 13:03:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sieunpark77/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[CANINE — Transformers without Tokenization ]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://sieunpark77.medium.com/canine-transformers-without-tokenization-b2451e4fdc3f?source=rss-8a8e55663707------2"><img src="https://cdn-images-1.medium.com/max/1452/1*bywMRqSCzCCL3b-Li3Uzww.png" width="1452"></a></p><p class="medium-feed-snippet">How to train Transformers directly on Characters</p><p class="medium-feed-link"><a href="https://sieunpark77.medium.com/canine-transformers-without-tokenization-b2451e4fdc3f?source=rss-8a8e55663707------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://sieunpark77.medium.com/canine-transformers-without-tokenization-b2451e4fdc3f?source=rss-8a8e55663707------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2451e4fdc3f</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[tokenization]]></category>
            <dc:creator><![CDATA[Sieun Park]]></dc:creator>
            <pubDate>Tue, 18 Mar 2025 17:31:04 GMT</pubDate>
            <atom:updated>2025-03-18T17:31:04.043Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[What are Hash Embeddings?]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://sieunpark77.medium.com/what-are-hash-embeddings-217759886d0c?source=rss-8a8e55663707------2"><img src="https://cdn-images-1.medium.com/max/1558/1*euM2vqf-r_JI8OSoR1Ru_w.png" width="1558"></a></p><p class="medium-feed-snippet">When embeddings are too big.</p><p class="medium-feed-link"><a href="https://sieunpark77.medium.com/what-are-hash-embeddings-217759886d0c?source=rss-8a8e55663707------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://sieunpark77.medium.com/what-are-hash-embeddings-217759886d0c?source=rss-8a8e55663707------2</link>
            <guid isPermaLink="false">https://medium.com/p/217759886d0c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[tokenization]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[embedding]]></category>
            <dc:creator><![CDATA[Sieun Park]]></dc:creator>
            <pubDate>Sat, 15 Mar 2025 18:12:42 GMT</pubDate>
            <atom:updated>2025-03-18T12:01:59.278Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Bringing NLP to life with streamlit]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://sieunpark77.medium.com/bringing-nlp-to-life-with-streamlit-98069ccc85b1?source=rss-8a8e55663707------2"><img src="https://cdn-images-1.medium.com/max/833/1*L0UI9znSGdmybez9mXFzmg.png" width="833"></a></p><p class="medium-feed-snippet">Presenting your new model with a slick demo!</p><p class="medium-feed-link"><a href="https://sieunpark77.medium.com/bringing-nlp-to-life-with-streamlit-98069ccc85b1?source=rss-8a8e55663707------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://sieunpark77.medium.com/bringing-nlp-to-life-with-streamlit-98069ccc85b1?source=rss-8a8e55663707------2</link>
            <guid isPermaLink="false">https://medium.com/p/98069ccc85b1</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[chatgpt]]></category>
            <dc:creator><![CDATA[Sieun Park]]></dc:creator>
            <pubDate>Sat, 15 Mar 2025 12:20:30 GMT</pubDate>
            <atom:updated>2025-03-15T12:20:30.171Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Publication Data Reveals Shifting Reader Preferences in 2025]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://sieunpark77.medium.com/publication-data-reveals-shifting-reader-preferences-in-2025-422231c4fea9?source=rss-8a8e55663707------2"><img src="https://cdn-images-1.medium.com/max/1400/0*PwGfKzlrakxDP2Oj.png" width="1400"></a></p><p class="medium-feed-snippet">Which publications are winning in 2025 and why</p><p class="medium-feed-link"><a href="https://sieunpark77.medium.com/publication-data-reveals-shifting-reader-preferences-in-2025-422231c4fea9?source=rss-8a8e55663707------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://sieunpark77.medium.com/publication-data-reveals-shifting-reader-preferences-in-2025-422231c4fea9?source=rss-8a8e55663707------2</link>
            <guid isPermaLink="false">https://medium.com/p/422231c4fea9</guid>
            <category><![CDATA[publication]]></category>
            <category><![CDATA[medium]]></category>
            <category><![CDATA[writing-tips]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[writing]]></category>
            <dc:creator><![CDATA[Sieun Park]]></dc:creator>
            <pubDate>Tue, 11 Mar 2025 12:50:33 GMT</pubDate>
            <atom:updated>2025-03-11T12:50:33.307Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Pics of Europe trip — Germany]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://sieunpark77.medium.com/pics-of-europe-trip-germany-aa09aa72c981?source=rss-8a8e55663707------2"><img src="https://cdn-images-1.medium.com/max/2600/1*h356r7u3ej39gPQv4x2FxQ.jpeg" width="2992"></a></p><p class="medium-feed-snippet">Real pics from my 3 months&#x200A;&#x2014;&#x200A;Koblenz, Cologne, Rothenburg ob de Tauber, Munich, Fussen, Berlin.</p><p class="medium-feed-link"><a href="https://sieunpark77.medium.com/pics-of-europe-trip-germany-aa09aa72c981?source=rss-8a8e55663707------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://sieunpark77.medium.com/pics-of-europe-trip-germany-aa09aa72c981?source=rss-8a8e55663707------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa09aa72c981</guid>
            <category><![CDATA[travel]]></category>
            <category><![CDATA[backpacking]]></category>
            <category><![CDATA[europe]]></category>
            <category><![CDATA[germany]]></category>
            <dc:creator><![CDATA[Sieun Park]]></dc:creator>
            <pubDate>Tue, 11 Mar 2025 10:14:57 GMT</pubDate>
            <atom:updated>2025-03-11T10:14:57.087Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[A late review of OpenAI’s “Training Verifiers to Solve Math Word Problems”]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://sieunpark77.medium.com/a-late-review-of-openais-training-verifiers-to-solve-math-word-problems-0d457eb706e3?source=rss-8a8e55663707------2"><img src="https://cdn-images-1.medium.com/max/1340/1*9ym7uzMTw5x3XVzgGPS7Ag.png" width="1340"></a></p><p class="medium-feed-snippet">Solving math problems with GPT. Verifiers, GSM-8K dataset, mathematical reasoning</p><p class="medium-feed-link"><a href="https://sieunpark77.medium.com/a-late-review-of-openais-training-verifiers-to-solve-math-word-problems-0d457eb706e3?source=rss-8a8e55663707------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://sieunpark77.medium.com/a-late-review-of-openais-training-verifiers-to-solve-math-word-problems-0d457eb706e3?source=rss-8a8e55663707------2</link>
            <guid isPermaLink="false">https://medium.com/p/0d457eb706e3</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[math]]></category>
            <category><![CDATA[openai]]></category>
            <dc:creator><![CDATA[Sieun Park]]></dc:creator>
            <pubDate>Thu, 14 Mar 2024 07:42:20 GMT</pubDate>
            <atom:updated>2024-03-14T08:02:05.532Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text…]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/codex/sentencepiece-a-simple-and-language-independent-subword-tokenizer-and-detokenizer-for-neural-text-ffda431e704e?source=rss-8a8e55663707------2"><img src="https://cdn-images-1.medium.com/max/699/1*hflYLQ_iObcXQPXAJDnQGA.png" width="699"></a></p><p class="medium-feed-snippet">The tokenization method for Alpaca, LLaMA, T5, XLNet</p><p class="medium-feed-link"><a href="https://medium.com/codex/sentencepiece-a-simple-and-language-independent-subword-tokenizer-and-detokenizer-for-neural-text-ffda431e704e?source=rss-8a8e55663707------2">Continue reading on CodeX »</a></p></div>]]></description>
            <link>https://medium.com/codex/sentencepiece-a-simple-and-language-independent-subword-tokenizer-and-detokenizer-for-neural-text-ffda431e704e?source=rss-8a8e55663707------2</link>
            <guid isPermaLink="false">https://medium.com/p/ffda431e704e</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Sieun Park]]></dc:creator>
            <pubDate>Fri, 19 May 2023 17:27:23 GMT</pubDate>
            <atom:updated>2023-05-19T17:27:23.504Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[An Introduction to Multiprocessing Using Python]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/codex/an-introduction-to-multiprocessing-using-python-165f51f83c0d?source=rss-8a8e55663707------2"><img src="https://cdn-images-1.medium.com/max/1120/0*FeLd62OxdwLpZ6op.png" width="1120"></a></p><p class="medium-feed-snippet">Is multithreading or multiprocessing faster than another?</p><p class="medium-feed-link"><a href="https://medium.com/codex/an-introduction-to-multiprocessing-using-python-165f51f83c0d?source=rss-8a8e55663707------2">Continue reading on CodeX »</a></p></div>]]></description>
            <link>https://medium.com/codex/an-introduction-to-multiprocessing-using-python-165f51f83c0d?source=rss-8a8e55663707------2</link>
            <guid isPermaLink="false">https://medium.com/p/165f51f83c0d</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[multiprocessing]]></category>
            <category><![CDATA[multithreading]]></category>
            <dc:creator><![CDATA[Sieun Park]]></dc:creator>
            <pubDate>Sun, 16 Apr 2023 18:21:33 GMT</pubDate>
            <atom:updated>2023-04-16T19:22:18.610Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Subword Regularization]]></title>
            <link>https://sieunpark77.medium.com/subword-regularization-6aace5e6a165?source=rss-8a8e55663707------2</link>
            <guid isPermaLink="false">https://medium.com/p/6aace5e6a165</guid>
            <category><![CDATA[tokenization]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[sentencepiece]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Sieun Park]]></dc:creator>
            <pubDate>Sun, 05 Mar 2023 09:12:28 GMT</pubDate>
            <atom:updated>2023-03-05T09:14:10.999Z</atom:updated>
            <content:encoded><![CDATA[<h3>Improving subword tokenization with Subword Regularization</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xy-6dbZer7-4bQhn" /><figcaption>Photo by <a href="https://unsplash.com/@sigmund?utm_source=medium&amp;utm_medium=referral">Sigmund</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><ul><li><a href="https://arxiv.org/abs/1804.10959">Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates</a> [ACL 2018]</li><li>Publish month(arxiv): 2018.04</li><li>Tags: nlp, tokenization</li><li>✰ Best viewed in [<a href="https://deep-learning.notion.site/Subword-Regularization-Improving-Neural-Network-Translation-Models-with-Multiple-Subword-Candidates-7f23ecbb8d904733816e65b805d87248">Notion</a>]</li></ul><p><a href="https://deep-learning.notion.site/Subword-Regularization-Improving-Neural-Network-Translation-Models-with-Multiple-Subword-Candidates-7f23ecbb8d904733816e65b805d87248">Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates</a></p><h3>Overview</h3><ul><li>A sentence can be represented in multiple subword sequences even with the same vocabulary. Previous methods don’t consider this problem.</li><li>Instead of maximizing the log likelihood on a single subword sequence with the largest likelihood, the authors propose an efficient subword regularization method that maximizes the likelihood of all possible combinations.</li><li>The method improves machine translation performance. The results are even more significant on the OOD corpus.</li></ul><h3>A sentence can be represented in multiple subword sequences even with the same vocabulary</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/455/0*_d10Rv0zD392YLH_.png" /></figure><p>BPE encodes a sentence into a unique subword sequence. Suppose in an NMT scenario where X is the source language passage and Y is the target language passage. Expressing P(Y | X) as the formula above for the sequence of subwords X, Y can be incorrect when X, Y can actually be multiple subword sequences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/0*jme-032sjo3gaVS6.png" /></figure><p>Subword regularization acknowledges the issue and addresses it by randomly sampling subword sequences.</p><h3>Overview: A new regularization method for open-vocabulary NMT</h3><ol><li>Standard NMT is trained to maximize the log-likelihood L(θ)</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/0*_hmll9o7x9u7XoP2.png" /></figure><p>2. To consider that X, Y can be segmented into multiple subword sequences, we should like to optimize L_marginal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/0*p_BUtfIMN7pBppRu.png" /></figure><p>3. Since exact optimization of L_marginal is impossible, we sample k subword sequences from P(x | X) and P(y | Y) and optimize the log-likelihood for these sequences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/822/0*hCk7rucQOkn7lJTe.png" /></figure><p>4. In particular, a single randomized segment is sampled for x, y (i.e. k = 1).</p><h3>Unigram language model for subword segmentation</h3><p><strong>Motivation</strong></p><p>Using BPE-based tokenizers to compute P(x | X) for subword normalization is non-trivial because it is based on a greedy and deterministic encoding method. A unigram language model that assumes that each subword occurs independently is needed to model the probability of each subword sequence P(x | X).</p><p><strong>Definition</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/0*rOz3EizxFZNs58yV.png" /></figure><p>The authors propose a unigram language model that assumes that each subword occurs independently. Here, you can write P(x) of a subword sequence x as the product of the probability of each subword occurring..</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/0*3P77Q6aYJCWZdvyS.png" /></figure><p>The most probable sequence x* from S(X) a set of segmentation candidates built from X can be obtained efficiently using the <a href="https://ratsgo.github.io/data%20structure&amp;algorithm/2017/11/14/viterbi/">Viterbi algorithm</a>.</p><p><strong>Training unigram model</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/0*ooPg1FDa7IvWgfbV.png" /></figure><p>The unigram model should be trained to maximize the EM likelihood L above. The algorithm jointly optimizes vocabulary set V and occurrence probabilities:</p><ol><li>Make a reasonably big seed vocabulary from the training corpus.</li></ol><ul><li>All single characters are included.</li><li>Most frequent substrings in the corpus which is similar to WordPiece.</li></ul><p>2. Repeat the following steps until |V| reaches the desired vocabulary size.</p><ul><li>Optimize p(x) with the EM likelihood above, given current V.</li><li>For each subword x_i, compute how much the likelihood L is reduced when x_i is removed from the current vocabulary.</li><li>Keep the top η% of subwords (η=80%).</li></ul><h3>Sampling subwords</h3><p>*<em>segmentation means subword sequence</em></p><ol><li>Sample l-best segmentations according to the probability P(x | X)</li><li>Single(because k=1) segmentation is sampled from the l-best segmentations weighted on smoothed probabilities.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*EpHW-aRhHPpdD5VI.png" /></figure><ul><li>l-best search is performed in linear time with the Forward-DP Backward-A* algorithm</li><li>To sample all possible segmentations(for l → ∞ experiments) for exact computation, the authors describe a Forward-Filtering and Backward-Sampling(FFBS) algorithm.</li></ul><h3>n-best decoding</h3><p>During decoding, the authors suggest an n-best decoding algorithm:</p><ol><li>Given n-best subword sequences of X (x_1, … x_n),</li><li>the best translation y* should have the largest length-normalized log-likelihood during decoding. In particular, y* should maximize the following score for some x.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/0*RAW28S2q4603SZeT.png" /></figure><h3>Results</h3><p><strong>Main result on NMT</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tP2d43ohMBkqiQWV.png" /></figure><ul><li>Unigram LM without subword regularization(l=1) and BPE show similar performance.</li><li>l=64 shows significant BLEU improvements.</li><li>n-best decoding provides further gains for l ≠ 1 , but degrades the performance for l = 1. This indicates that the model is confused about multiple segmentations when they are not explored at training time.</li></ul><p><strong>Results on OOD corpus</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/0*dFCm3UEZZW-JzpAv.png" /></figure><ul><li>The results are even more significant on the OOD corpus.</li></ul><p><strong>Hyperparameter</strong>:</p><ul><li>subword regularization has two hyperparameters: l: size of sampling candidates, α: smoothing constant.</li><li>Tuning the value of α &lt; 1 was crucial.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/0*NlTmmGF4marlD6ZP.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6aace5e6a165" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Details of Faster, Mask, Cascade R-CNN ]]></title>
            <link>https://sieunpark77.medium.com/details-of-faster-mask-cascade-r-cnn-7b9c34326cdf?source=rss-8a8e55663707------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b9c34326cdf</guid>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[image-processing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Sieun Park]]></dc:creator>
            <pubDate>Tue, 07 Jun 2022 17:30:38 GMT</pubDate>
            <atom:updated>2022-09-08T18:46:29.085Z</atom:updated>
            <content:encoded><![CDATA[<h3>Fundamentals of Object Detection: Faster, Mask, Cascade R-CNN 🔥</h3><p>Faster R-CNN is still an actively cited benchmark for comparing object detection performance of modern network architectures. In this post, we will discuss the insights and contributions of the papers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*G-xU8gmUlxGOKjnM" /><figcaption>Photo by <a href="https://unsplash.com/@davisuko?utm_source=medium&amp;utm_medium=referral">davisuko</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Introduction — Why do we even care in 2022?</h4><p>Faster R-CNN was a huge breakthrough in computer vision because it enabled real-time object detection and enabled end-to-end training of the object detection pipeline. After 7 years, in 2022, its variants are still used as robust starting points for practical applications and a benchmark for comparing network architectures.</p><ul><li>The paper has significant research contributions, as it is conceptually a foundation of every two-stage object detection scheme which is a popular and active branch of research(superficially, the paper is also cited 40K+ times).</li><li>Although not s.o.t.a(state-of-the-art) anymore, the pipeline is still extremely effective and architecture agnostic. Faster R-CNN is always a robust starting point for your object detection application 🙂</li><li>Papers that propose <strong>network architectures routinely compare performance</strong> for object detection on the “cascade-mask-rcnn” framework(He et al., 2017; Cai et al., 2018), a popular variation of faster rcnn.</li><li>Despite its popularity and significance, the framework is <strong>very complex</strong> and there are many important details that should be modified for different use cases, which motivates this post. Many hyperparameters and details are involved in the training, which is sometimes not so obvious.</li></ul><h4>Overview</h4><p>We will start by briefly reviewing the Fast R-CNN and Faster R-CNN pipelines. Then, we will discuss the popular cascade-mask rcnn framework and how they improve the performance of Faster R-CNN. This post is mainly based on the <em>interesting</em> insights from the paper instead of the nitty-gritty details of hyperparameters etc. However, we will discuss the details involved in the implementation in the next post, so keep tuned if you are interested!</p><p>While writing this post, I referenced the implementations of popular public object detection frameworks, but the post is mainly focused in the torchvision implementation[R1].</p><p>[R1]<em> torchvision.models.detection.faster_rcnn.FasterRCNN — </em>[<a href="https://github.com/pytorch/vision/blob/main/torchvision/models/detection/faster_rcnn.py">github</a>][<a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html">tutorial</a>]</p><p>[R2]<em> facebookresearch/maskrcnn-benchmark </em>and <em>Detectron2 — </em><a href="https://github.com/facebookresearch/detectron2">[github]</a></p><p>[R3] <em>faster-rcnn.pytorch </em>— [<a href="https://github.com/jwyang/faster-rcnn.pytorch">github</a>]</p><p>References to the papers mentioned are provided at the end of the article.</p><h4>A summary and list of keywords</h4><p>You may skip this section and come back to it later:)</p><p><strong>~Fast R-CNN (Girshick et al., 2014; Girshick et al., 2015)</strong></p><ul><li>Two-stage object detection: Generate region proposals → compute features for each region of interest(RoI) → classify each RoI as either some object or background, regress precise box coordinates.</li><li>Selective search: Classic approach to generate region proposals.</li><li>RoI Pooling was proposed in Fast R-CNN to receive CNN features of different objects that have various shapes and reduce into constant shape.</li><li>Background and foreground RoIs are balanced while sampling data because the majority of RoIs are background and random sampling will result in too much easy background samples. The <em>image-centric sampling </em>of Fast R-CNN picks N images and extracts (batch_size / N) RoIs from each image in a balanced manner.</li><li>Classification of object classes is trained with an extra <em>background </em>class and softmax &amp; cross-entropy loss. Bounding box regression is trained with smoothed L1 loss, as proposed by Girshick et al., 2015. Two objects are jointly learned with a combined loss.</li></ul><p><strong>Faster R-CNN (Ren et al., 2015)</strong></p><ul><li>Region proposals based on selective search were the main bottleneck for inference speed. Faster R-CNN proposed a learning-based approach based on region proposal networks for accurate and cheap region proposals.</li><li>Region proposal networks(RPN) slides across CNN feature maps to predict objectness scores — the probability of object or background and box regressions — exact box coordinate refinements of <em>k</em> reference <em>anchor boxes</em>.</li><li>RPN share features with the original Fast R-CNN detection network and is jointly trained. The authors also discuss 3 schemes for jointly training RPN and Fast R-CNN.</li></ul><p><strong>Mask R-CNN (He et al., 2017)</strong></p><ul><li>Summarize a better baseline for training Faster R-CNN, which involves better data augmentation, hyperparameters, and model architectures.</li><li>Proposed a drop-in replacement of RoIPool named RoIAlign, which allows pixel-to-pixel alignment of RoI.</li><li>Proposed a framework capable of dense predictions based on the region proposals of RPN, and demonstrated the effectiveness on keypoint estimation and instance segmentation.</li></ul><p><strong>Cascade R-CNN (Cai et al., 2018)</strong></p><ul><li>Investigate and observe interesting challenges and phenomenons in the IoU overlap threshold problem.</li><li>Training with small IoU threshold results in noisy samples, and training with large thresholds degrades the performance because of overfitting and low-quality region proposals at inference compared to training.</li><li>Models trained using specific IoU thresholds might be suboptimal when evaluated on other IoU thresholds or region proposals of different level of IoU threshold is given.</li><li>Suggest a cascade of specialized detectors to resolve the challenges and was able to significantly improve performance.</li></ul><h3>Quick overview of Faster R-CNN</h3><h4>Two-stage object detection</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UjT_TnaffUQQOpc7FbLqJg.png" /><figcaption>Girshick et al., 2014, an overview of R-CNN, a popular 2-stage object detection method</figcaption></figure><p>In two-stage object detection(Girshick et al., 2014), an array of region proposals are used to represent the location of potential objects, which are represented as yellow boxes in the figure above. Each region proposal is then further evaluated so the class and refine the precise location of the object is determined.</p><p>Previous approaches such as R-CNN(Girshick et al., 2014) and Fast R-CNN(Girshick et al., 2015) rely on heuristics such as <a href="https://pyimagesearch.com/2020/06/29/opencv-selective-search-for-object-detection/">selective search</a> to compute pre-defined region proposals from the image. These initial region proposals are then processed by learning-based methods which decide whether each proposal are actual object(i.e. no background), assign classes to proposals, and refine the bounding box.</p><h4>Fast R-CNN Pipeline</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-FUTOXkVNzfkxSl4op8CgQ.png" /><figcaption>Girshick et al., 2015, an overview of the Fast R-CNN framework</figcaption></figure><p>In the Fast R-CNN framework, the image and precomputed region of interests are provided to the model. The image is reduced to a feature map using a CNN, and the feature for each RoI is cropped based on the position of the RoI. This cropped feature represents the high-level information of that RoI.</p><p>Then a <em>RoIPooling</em> layer is applied to process each cropped feature into the same size so it can be processed by an MLP. A series of fully-connected layers create RoI feature vectors for each RoI. Finally, the feature vector is used to 1) assign object classes to the RoI and 2) refine the initial box coordinates of the RoI.</p><h4>Fast R-CNN Training</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*1pwQBFNSrVZtxzpFsOLiRQ.png" /><figcaption>Image by author, example output of selective search</figcaption></figure><p>The objective of Fast R-CNN training is to accurately predict object classes and bounding box refinements from an RoI. Region proposal methods such as selective search generate ~2,000 RoIs for each image. There are usually much fewer objects in the image, thus an overwhelming number of the region proposals contain background (negative) RoIs compared to foreground RoIs that contain actual objects of interest. Therefore, most methods train the model with a balanced batch of RoI where the ratio of positive and negative RoIs are strictly controlled.</p><p>In training Fast R-CNN with a batch size of K=128, we read N=2 images from the dataset of images and select 64=K/N RoIs from each image, with 16 positive and 48 negative RoIs. Object proposals that have intersection over union (IoU) overlap with a ground-truth bounding box of at least 0.5 are considered positive and negative if that value is between [0.1, 0.5].</p><p>For each RoI, the model predicts 1) the object class and 2) bounding box refinements. For class prediction, negative RoIs are labeled as the “0” class and positive RoIs are labeled as the corresponding object class. The model is trained using the cross-entropy between the prediction and ground truth. Bounding box refinement predictions try to predict the xywh offsets between the region proposal and the exact location of the related g.t. bounding box. This is done by minimizing a smoothed MSE loss (Girshick et al., 2015) between the prediction and ground truth. We will later precisely define how the offsets are calculated together with the code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZE3YXm4WMhd4lorETLtFxA.png" /><figcaption>Girshick et al., 2015, smoothed MSE loss</figcaption></figure><p>The two branches of the “Fast R-CNN RoI head” are jointly trained with a combined loss function. Note that the bounding box regression loss is only computed when the object proposal is a <em>foreground</em> class. This is described as “[u ≥ 1]=1 if u≥1 else 0” in the equation below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MU2AhZMQMHz8l5wsby_giA.png" /><figcaption>Girshick et al., 2015, Joint Multi-task loss function of Fast R-CNN.</figcaption></figure><h4>Faster R-CNN</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jDr_adQD2FcqpIBpBiooqw.png" /><figcaption>Ren et al., 2015, Faster R-CNN can efficiently generate proposals.</figcaption></figure><p>The table above describes the time needed for each step of inference. The first row(SS+Fast R-CNN) that describes the Fast R-CNN framework shows that instead of the CNN feature extractor, the major bottleneck is the “proposal” stage, which performs selective search (SS) to find region proposals. To replace selective search, Faster R-CNN(Ren et al., 2015) suggests region proposal networks (RPN) that share features with the convolutional feature extractor to efficiently generate region proposals. The results demonstrate rich but nearly cost-free region proposals.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oRva18DrBo52MJJg1yRWPg.png" /><figcaption>Ren et al., 2015, an overview of Faster R-CNN</figcaption></figure><p>The overall pipeline of Faster R-CNN is very similar to Fast R-CNN. The CNN network extracts features from the image. Whereas Fast R-CNN performs selective search on the image to generate region proposals, the region proposal network (RPN) takes the CNN features to generate region proposals for Faster R-CNN. The RPN outputs ~2K region proposals. The region proposals and CNN features are then fed to the Fast R-CNN head that predicts object classification and bounding box regression. Lines 8~10 in the snippet below describe the high-level implementation.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9bf94182ef20bf2ddfa13ab66f7c3540/href">https://medium.com/media/9bf94182ef20bf2ddfa13ab66f7c3540/href</a></iframe><h4>Region Proposal Networks</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*elcA0o7khL8tky9FRF7hRw.png" /><figcaption>Ren et al., 2015, an overview of region proposal network(RPN)</figcaption></figure><p>To generate region proposals, a region proposal network is slid across the convolutional feature map. This small network receives part of the feature map and predicts multiple region proposals. At maximum at each sliding-window location, k region proposals can be made. These k proposals are based on k reference anchor boxes, which are defined as a combination of scales={128, 256, 512} and aspect ratios={1:1, 1:2, 2:1}, k=9 is used in Faster R-CNN.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*3YwySodD2QU_CeuAz4PXBw.png" /></figure><p>Specifically, the feature map is projected into an intermediate layer which is used to make 2 predictions: 1) the objectness score(<em>cls</em>) predicts the probability of a region proposal based on each k anchor box centered on the window to be an object or not object and 2) the regression layer(<em>reg</em>) predict the coordinates of k boxes. The regression layer predicts box refinements that are relative to the corresponding reference anchor boxes. By combining the reference anchor, box coordinate refinements from the regression layer, and filtering the proposals by predicted objectness score, the model can predict accurate region proposals.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a1b745086c40297871508abe85daa196/href">https://medium.com/media/a1b745086c40297871508abe85daa196/href</a></iframe><p>Interestingly, the sliding window that parses the convolutional feature map and generates the intermediate layer can be represented as a single convolution layer with a kernel size of n×n and no padding. Then a 1×1 convolution layer can be used to predict the objectness score or box refinements from the intermediate features. This is indeed demonstrated in the code above(note that self.conv can have multiple layers in this implementation).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*QAF7XNppuNAAhR3Ss7TmhA.png" /><figcaption>Ren et al., 2015, training loss of RPN in Faster R-CNN</figcaption></figure><p>The objectness score can be trained by assigning a binary class label (of being an object or not) to each anchor and measuring the cross-entropy loss. Similar but slightly different to the “background” class for Fast R-CNN training, an anchor is considered positive if it has an IoU overlap higher than 0.7, and is considered negative IoU overlap is lower than 0.3 for all ground-truth boxes. Anchors with IoU overlap in between are not used for training objectness score.</p><p>Finally, there are some details that have to be taken care of during training and inference using RPN. During training, all cross-boundary anchors are ignored so they do not contribute to the loss, but all anchors are kept during inference. The author suggests clipping cross-boundary boxes to the image boundary at test time. Non-max suppression(NMS) with a threshold of 0.7 is applied to region proposals in order to remove redundant proposals. At test time, the top-N ranking proposals based on objectness scores are used for detection.</p><h4>Faster R-CNN Training</h4><p>Faster R-CNN is a combination of Fast R-CNN and learnable region proposals using the region proposal network. Therefore, there are two main training objectives for training:</p><ul><li>Fast R-CNN module — box regression(smoothed L1), object classification(cross-entropy)</li><li>Region proposal network—objectness score(cross-entropy), box regression</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oRva18DrBo52MJJg1yRWPg.png" /><figcaption>Ren et al., 2015, an overview of feature sharing between RPN and Fast R-CNN</figcaption></figure><p>Both Fast R-CNN and RPN can be trained independently, but as training one module will alter the useful weights for the other module, Ren et al., 2015 discuss several approaches for learning a unified network for both Fast R-CNN and RPN by sharing features such as described in the figure above. The authors discuss 3 training schemes as follows:</p><ul><li><em>Alternating training</em> approach to first train RPN, then use the proposals from RPN to train Fast R-CNN. The network tuned by Fast R-CNN is then used to initialize RPN, and this process is iterated.</li><li><em>Approximate joint training </em>approach to predict region proposals and use region proposals to compute the RPN loss and use these region proposals to also train the Fast R-CNN detector. This is efficient because only one forward/backward pass is made while training both RPN and Fast R-CNN.</li><li><em>The non-approximate joint training </em>approach is considered because the region proposals used in RoIPooling of Fast R-CNN are indeed functions of the input image. In other words, gradients from the Fast R-CNN detector must also flow into the RPN and convolutional features involved in making region proposals. However, RoIPooling is <em>approximate joint training</em> because the gradients are cut in between.</li></ul><p>The authors of Ren et al., 2015 adopt the alternative training approach by default but notes that the approximate joint training approach is much faster while demonstrating close results empirically. Later experiments in Mask R-CNN(He et al., 2017) actually adopt the approximate joint training strategy. Indeed, the code example GeneralizedRCNN above demonstrates this approach. The authors however describe the non-approximate joint training approach as nontrivial and don’t provide experiments regarding it.</p><p>Now we will dive into the cascade-mask rcnn variants that improve the performance of Faster R-CNN!! 🔥</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8I7p9esf_oerDYRqsq9a1Q.png" /><figcaption>He et al., 2017, Mask R-CNN results on instance segmentation</figcaption></figure><h4>Improving Faster R-CNN</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*1DKvHILEkX-BPRzgXPciTg.png" /><figcaption>He et al., 2017, Ablation on enhancing Faster R-CNN</figcaption></figure><p>Mask R-CNN(He et al., 2017) mainly focuses on proposing a powerful baseline system for instance segmentation, but the authors also propose a significantly better baseline for object detection based on Faster R-CNN. We will first discuss the hyperparameters and algorithms to improve object detection performance that is described in the Appendix of He et al., 2017.</p><ul><li>updated baseline: increase training steps from 120k → 180k, increase NMS post-processing threshold from 0.3 → 0.5.</li><li>end-to-end(e2e) training: Use the approximate joint training strategy instead of the original alternate training strategy, which actually improves performance.</li><li>ImageNet-5k: Pre-train the feature extractor on ImageNet-5k instead of the 1k subset.</li><li>train-time augmentation: Randomly scale images as data augmentation, further increase the training budget to 260k</li><li>deeper: go deeper by using the ResNext152 model</li><li>Non-local: Improve model architecture by using non-local blocks</li><li>Test-time augmentation: Use <a href="https://github.com/qubvel/ttach">TTA</a> on image scales of [400, 1200] pixels with a step of 100 and on their horizontal flips.</li></ul><p>There are also some modifications adapted from Lin et al., 2017 that are not mentioned in this ablation:</p><ul><li>Images are resized such that their scale (shorter edge) is 800 pixels</li><li>15 reference anchors are used with scales={32, 64, 128, 256, 512} and aspect ratios={1:1, 1:2, 2:1}.</li></ul><h4>RoIAlign</h4><p>RoIAlign is a drop-in replacement of RoIPooling, that allows pixel-to-pixel alignment without quantization. The authors consider this the <em>main missing piece of Fast/Faster R-CNN</em> for dense prediction tasks such as for instance segmentation and find this update crucial for instance segmentation. The layer still does provides subtle improvements to bounding box-level performance. Let&#39;s first have a look at the problems of RoIPooling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*Peb1stsxklVqkMkK_GKeUQ.png" /><figcaption>Original image by He et al., 2017, Illustration of the quantization of RoIPool</figcaption></figure><p>RoIPool is a standard operation for extracting a small feature map (e.g., 7×7) from each RoI which is given as something like the black box above. RoIPool first quantizes a floating-number RoI to the discrete granularity of the feature map(i.e. the red box). This quantized RoI is then subdivided into spatial bins which are themselves quantized(i.e. blue points) and finally feature values covered by each bin are aggregated (usually by max-pooling).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*PTOIYGh9O-mxKEo1UtWxKA.png" /><figcaption>He et al., 2017, Overview of RoIAlign</figcaption></figure><p>Instead of quantizing coordinates to index the nearest discrete value of the feature map, RoIAlign uses bilinear interpolation to compute the values of 4 nearby features and aggregate them. No quantization is performed.</p><h4>Instance Segmentation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1009/1*ks3NaHJYRBH_yapXfoGUjw.png" /><figcaption>He et al., 2017, Bounding box object detection results of Mask R-CNN</figcaption></figure><p>Instance segmentation is a task to detect the bounding box of objects while also precisely segmenting each instance. Although the topic might seem irrelevant, I would like to briefly mention that the authors showed that jointly learning the mask and bounding box benefits the detector. In the table above, performance degraded when the mask branch was removed in the <em>Faster R-CNN, RoIAlign</em> experiment.</p><p>Moreover, please also note that although we viewed the paper as a variant of Faster R-CNN from the perspective of object detection, Mask R-CNN is an amazing baseline that can solve many other tasks involving localization using the methodologies of two-stage detectors and RPN.</p><blockquote>We show top results in all three tracks of the COCO suite of challenges, including instance segmentation, bounding-box object detection, and person keypoint detection. Without bells and whistles, Mask R-CNN outperforms all existing, single-model entries on every task, including the COCO 2016 challenge winners.we predict a binary mask for each class independently, without competition among classes</blockquote><p>At last, we will discuss our final paper called Cascade R-CNN(Cai et al., 2018) which is based on a very insightful observation about thresholds. 🔍</p><h4>IoU Threshold</h4><p>We have seen that in two-stage object detection, an IoU threshold is defined to determine whether a region proposal is positive (object) or negative (background). A region proposal is considered positive if the IoU overlap with a ground truth object is larger than the threshold, or negative elsewise. We also discussed that Fast R-CNN and later methods use an IoU threshold of 0.5. Seems fair enough, but in Cai et al., 2018, the authors suggest that the threshold issue might not be as trivial.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*Gs_cq7LE2skGjCIfi0hiKg.png" /><figcaption>Cai et al., 2018, a threshold of 0.5 permits noisy detections!</figcaption></figure><p>First, the authors show the commonly used IoU threshold of 0.5 could allow many detections that are considered false positives to humans, as shown in the figure above. OK, then if we train with a higher threshold like 0.7, <em>problem solved</em>? Figure (d) below illustrates the object detection performance for 3 different thresholds used for training, where the x-axis is the IoU threshold used for measuring AP. We can see that using a higher threshold has worse object detection performance on most parts of the curve.</p><p>The authors point out 2 problems that make it very challenging to increase the threshold and allow only high-quality boxes.</p><ol><li>Overfitting, since the diversity of the dataset exponentially decreases as the IoU overlap threshold increase.</li><li>The inference-time mismatch between IoUs: although we might be able to artificially sample such good region proposals during training, the RPN is not always able to generate region proposals with high IoU overlap.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ThEYjC-WxNAoaEWlLmIZ9g.png" /><figcaption>Cai et al., 2018, the performance of box regressor and detector based on IoU threshold used to train model</figcaption></figure><p>Is there even anything we can do 🙁? The plots (c, d) present two interesting pieces of evidence that we can work on. First, plot(c) shows the IoU overlap measured before and after bounding box regression of the Fast R-CNN head.</p><ul><li>Note that the precision of the bounding box tends to improve(e.g. an initial box with 0.7 IoU overlap is refined to have ~0.8 IoU overlap) since most of the curves are over the grey line.</li><li>Especially when the input has similar IoU thresholds to what was used during training. A bounding box regressor trained with region proposals with &gt;0.5 IoU overlap is likely to make good refinements when similar images of 0.5~0.6 IoU overlap are given.</li></ul><p>Next, in plot (d), the detector trained with u = 0.5 outperforms the detector of u = 0.6 for low IoU examples, while underperforming it at higher IoU levels. Crucially, we note that in general, a detector optimized at a single IoU level is not necessarily optimal at other levels. To summarize the challenges observed regarding the IoU overlap threshold that should be used,</p><ul><li>Using a high IoU overlap threshold for training leads to 1) overfitting due to reduced samples and 2) inference-time mismatch due to poor region proposals.</li><li>A detector optimized for a certain IoU threshold might not be optimal for other IoU thresholds. An optimizer trained with a different threshold is needed to achieve optimal performance on another threshold.</li></ul><blockquote>…we define the quality of an hypothesis as its IoU with the ground truth, and the quality of the detector as the IoU threshold u used to train it. The basic idea is that a single detector can only be optimal for a single quality level.</blockquote><h4>Cascade R-CNN</h4><p>So how can we optimize for all IoU overlap levels? We will first start by discussing two trivial approaches to the issue and suggest that a combination of them might be the most effective. The Faster R-CNN framework is shown in figure (a) below. The convolutional features are given to the RPN(H0) and object proposals B0 are created. These proposals are then used to parse the convolutional feature map and the RoI head predicts the final bounding box refinements B1 and objects class C1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VpAMgnbNMzhm2tA2cRtf2Q.png" /></figure><p>The first approach is iterative bounding box refinement, described in figure (b). The refined object proposals(or bounding boxes) C1 are used to once more parse the convolutional feature map. The same Fast R-CNN network predicts new and possibly improved object class C2 and bounding boxes B2. This procedure may be repeated iteratively. However, the authors suggest that this idea ignores two problems.</p><ol><li>The IoU could increase as the refinement continues. As shown previously, a regressor trained at u = 0.5, is suboptimal for higher IoU overlap.</li><li>The distribution of bounding boxes changes significantly after each iteration. While the regressor is optimal for the initial distribution it can be quite suboptimal after that.</li></ol><p>Another approach is to develop an ensemble of classifiers that are each optimized with a different IoU threshold. This is described in figure (c) and the loss is the equation below, where U={0.5, 0.55, · · · , 0.75} is an increasing set of IoU thresholds. This approach is flawed because the set of positive samples decreases quickly as u increases due to the poor region proposals of RPN as seen in the most left histogram below that illustrates the quality of initial RPN region proposals. This means detectors with high IoU thresholds are very prone to overfitting and low quality at inference, which makes them pointless.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/399/1*KA_EMvftJgZLQLQ1fvKr6Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/676/1*ludet4t8wbLkyZao6hLrYA.png" /></figure><p>Finally, the cascade R-CNN framework is described in figure (d), which uses a cascade of <em>specialized</em> regressors, which are trained with increasing IoU overlap thresholds. Unlike iterative box refinement of (b) which is applied as post-processing, the framework is used both for training and inference. Using the previous observation that bounding box regression generally improves the quality of the next object proposal, we can jointly increase the IoU threshold used for training.</p><p>The authors suggest refining the outputs of RPN 3 times with Fast R-CNN heads trained using IoU overlap threshold {0.5, 0.6, 0.7} in order. The histogram above illustrates how the quality of boxes is improved throughout the regression layers. You might think that the proposed method is very similar to the iterative box refinement strategy of (b), but surprised to see the large improvements from Cascade R-CNN.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/491/1*qRrRcIsrGad9uNde5E2YJg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tZjjI_hVuufvkFIS5uu7KQ.png" /></figure><p>Indeed, the ablations are very compelling as it was able to improve the performance of different detection frameworks and backbone architectures by a significant margin.</p><h4>Remarks</h4><p>The cascade-mask-rcnn benchmark is a combination of Mask R-CNN and Cascade R-CNN, there the RoIAlign layer and improved hyperparameters of Mask R-CNN are used together with the cascading bounding box regression modules of Cascade R-CNN. The common implementation can be found in <a href="https://github.com/open-mmlab/mmdetection/tree/master/configs/cascade_rcnn">mmDetection</a>.</p><p>This post was a journey to relieve my technical debt, but I really enjoyed looking at the insights of legendary papers in computer vision. In particular, I was thrilled to read about the neet RoIAlign layer and the concept of reference anchor boxes+slight refinement which were a great abstraction to represent the infinite possibilities of bounding boxes centered at some point.</p><p>One thing I felt was that there is room for improvement in the workaround of two-stage detectors to the class imbalance problem between foreground and background RoIs. The 1:3 sampling strategy was used since the very first proposal of R-CNN(Girshick et al., 2014).</p><p>Also, the <em>non-approximate joint training </em>approach to training networks also seemed very promising to me.</p><p>I haven’t included more recent strategies for object detection in this article, but I am looking forward to reviewing many papers in object detection, so stay tuned if you are interested 😇!</p><h3>References</h3><p>Girshick, R., Donahue, J., Darrell, T., &amp; Malik, J. (2014). Rich feature hierarchies for accurate object detection and semantic segmentation. In <em>Proceedings of the IEEE conference on computer vision and pattern recognition</em> (pp. 580–587).</p><p>Girshick, R. (2015). Fast r-cnn. In <em>Proceedings of the IEEE international conference on computer vision</em> (pp. 1440–1448).</p><p>Ren, S., He, K., Girshick, R., &amp; Sun, J. (2015). Faster r-cnn: Towards real-time object detection with region proposal networks. <em>Advances in neural information processing systems</em>, <em>28</em>.</p><p>Lin, T. Y., Dollár, P., Girshick, R., He, K., Hariharan, B., &amp; Belongie, S. (2017). Feature pyramid networks for object detection. In <em>Proceedings of the IEEE conference on computer vision and pattern recognition</em> (pp. 2117–2125).</p><p>He, K., Gkioxari, G., Dollár, P., &amp; Girshick, R. (2017). Mask r-cnn. In <em>Proceedings of the IEEE international conference on computer vision</em> (pp. 2961–2969).</p><p>Cai, Z., &amp; Vasconcelos, N. (2018). Cascade r-cnn: Delving into high quality object detection. In <em>Proceedings of the IEEE conference on computer vision and pattern recognition</em> (pp. 6154–6162).</p><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b9c34326cdf" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>