<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ruth Crasto on Medium]]></title>
        <description><![CDATA[Stories by Ruth Crasto on Medium]]></description>
        <link>https://medium.com/@crastoru?source=rss-f11d5eb7e93d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Qwdm47olKjE5cty9XqHS6w.jpeg</url>
            <title>Stories by Ruth Crasto on Medium</title>
            <link>https://medium.com/@crastoru?source=rss-f11d5eb7e93d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 09:42:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@crastoru/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Geographic Position Encoders]]></title>
            <link>https://medium.com/data-science/geographic-position-encoders-59dafebf6f2d?source=rss-f11d5eb7e93d------2</link>
            <guid isPermaLink="false">https://medium.com/p/59dafebf6f2d</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[geospatial]]></category>
            <dc:creator><![CDATA[Ruth Crasto]]></dc:creator>
            <pubDate>Tue, 01 Oct 2024 23:10:48 GMT</pubDate>
            <atom:updated>2024-10-04T05:46:51.919Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understanding modern techniques for encoding geographic coordinates in a neural network</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OylUR3M_nIom9REh" /><figcaption>Photo by <a href="https://unsplash.com/@chuttersnap?utm_source=medium&amp;utm_medium=referral">CHUTTERSNAP</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>An inductive bias in machine learning is a constraint on a model given some prior knowledge of the target task. As humans, we can recognize a bird whether it’s flying in the sky or perched in a tree. Moreover, we don’t need to examine every cloud or take in the entirety of the tree to know that we are looking at a bird and not something else. These biases in the vision process are encoded in convolution layers via two properties:</p><ul><li><strong>Weight sharing</strong>: the same kernel weights are re-used along an input channel’s full width and height.</li><li><strong>Locality</strong>: the kernel has a much smaller width and height than the input.</li></ul><p>We can also encode inductive biases in our choice of input features to the model, which can be interpreted as a constraint on the model itself. Designing input features for a neural network involves a trade-off between expressiveness and inductive bias. On one hand, we want to allow the model the flexibility to learn patterns beyond what we humans can detect and encode. On the other hand, a model without any inductive biases will struggle to learn anything meaningful at all.</p><p>In this article, we will explore the inductive biases that go into designing effective position encoders for geographic coordinates. Position on Earth can be a useful input to a wide range of prediction tasks, including image classification. As we will see, using latitude and longitude directly as input features is under-constraining and ultimately will make it harder for the model to learn anything meaningful. Instead, it is more common to encode prior knowledge about latitude and longitude in a nonparametric re-mapping that we call a positional encoder.</p><h3>Introduction: Position Encoders in Transformers</h3><p>To motivate the importance of choosing effective position encoder more broadly, let’s first examine the well-known position encoder in the transformer model. We start with the notion that the representation of a token input to an attention block should include some information about its position in the sequence it belongs to. The question is then: how should we encode the position index (0, 1, 2…) into a vector?</p><p>Assume we have a position-independent token embedding. One possible approach is to add or concatenate the index value directly to this embedding vector. Here is why this doesn’t work well:</p><ol><li>The similarity (dot product) between two embeddings — after their position has been encoded — should be independent of the total number of tokens in the sequence. The two last tokens of a sequence should record the same similarity whether the sequence is 5 or 50 words long.</li><li>The similarity between two tokens should not depend on the absolute value of their positions, but only the relative distance between them. Even if the encoded indices were normalized to the range [0, 1], two adjacent tokens at positions 1 and 2 would record a lower similarity than the same two tokens later in the sequence.</li></ol><p>The original “Attention is All You Need” paper <a href="https://arxiv.org/pdf/1706.03762">[1]</a> proposes instead to encode the position index <em>pos </em>into a discrete “snapshot” of <em>k </em>different sinusoids, where <em>k </em>is the dimension of the token embeddings. These snapshots are computed as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*59lniHSZp3nnA5H51PnI-Q.png" /></figure><p>where <em>i = </em>1, 2, …, <em>k </em>/ 2. The resulting <em>k</em>-dimensional position embedding is then added elementwise to the corresponding token embedding.</p><p>The intuition behind this encoding is that the more snapshots are out of phase for any two embeddings, the further apart are their corresponding positions. The absolute value of two different positions will not influence how out of phase their snapshots are. Moreover, since the range of any sinusoid is the interval [-1, 1], the magnitude of the positional embeddings will not grow with sequence length.</p><p>I won’t go into more detail on this particular position encoder since there are several excellent blog posts that do so (see <a href="https://kazemnejad.com/blog/transformer_architecture_positional_encoding/">[2]</a>). Hopefully, you can now see why it is important, in general, to think carefully about how position should be encoded.</p><h3>Geographic Position Encoders</h3><p>Let’s now turn to encoders for geographic position. We want to train a neural network to predict some variable of interest given a position on the surface of the Earth. How should we encode a position (λ, ϕ) in spherical coordinates — i.e. a longitude/latitude pair — into a vector that can be used as an input to our network?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*UbV4kyi-yOsnEdnsT9iuXw.png" /><figcaption>By Peter Mercator, <a href="https://commons.wikimedia.org/w/index.php?curid=12226167">Public Domain</a>.</figcaption></figure><h4>Simple approach</h4><p>One possible approach would be to use latitude and longitude values directly as inputs. In this case our input feature space would be the rectangle [-<em>π, π</em>] × [0<em>, π</em>], which I will refer to as lat/lon space. As with position encoders for transformers, this simple approach unfortunately has its limitations:</p><ol><li>Notice that as you move towards the poles, the distance on the surface of the Earth covered by 1 unit of longitude (λ) decreases. Lat/lon space does not preserve distances on the surface of the Earth.</li><li>Notice that the position on Earth corresponding to coordinates (λ, ϕ) should be identical to the position corresponding to (λ + 2<em>π, </em>ϕ). But in lat/lon space, these two coordinates are very far apart. Lat/lon space does not preserve periodicity: the way spherical coordinates wrap around the surface of the Earth.</li></ol><p>To learn anything meaningful directly from inputs in lat/long space, a neural network must learn how to encode these properties about the curvature of the Earth’s surface on its own — a challenging task. How can we instead design a position encoder that already encodes these inductive biases? Let’s explore some early approaches to this problem and how they have evolved over time.</p><h3>Early Position Encoders</h3><h4>Discretization-based (2015)</h4><p>The first paper to propose featurizing geographic coordinates for use as input to a convolutional neural network is called “Improving Image Classification with Location Context” <a href="https://arxiv.org/pdf/1505.03873">[3]</a>. Published in 2015, this work proposes and evaluates many different featurization approaches with the goal of training better classification models for geo-tagged images.</p><p>The idea behind each of their approaches is to directly encode a position on Earth into a set of numerical features that can be computed from auxiliary data sources. Some examples include:</p><ul><li>Dividing the U.S. into evenly spaced grids in lat/lon space and using a one-hot encoding to encode a given location into a vector based on which grid it falls into.</li><li>Looking up the U.S ZIP code that corresponds to a given location, then retrieving demographic data about this ZIP code from ACS (American Community Survey) related to age, sex, race, living conditions, and more. This is made into a numerical vector using one-hot encodings.</li><li>For a chosen set of Instagram hashtags, counting how many hashtags are recorded at different distances from a given location and concatenating these counts into a vector.</li><li>Retrieving color-coded maps from Google Maps for various features such as precipitation, land cover, congressional district, and concatenating the numerical color values from each into a vector.</li></ul><p>Note that these positional encodings are not continuous and do not preserve distances on the surface of the Earth. In the first example, two nearby locations that fall into different grids will be equally distant in feature space as two locations from opposite sides of the country. Moreover, these features mostly rely on the availability of auxiliary data sources and must be carefully hand-crafted, requiring a specific choice of hashtags, map features, survey data, etc. These approaches do not generalize well to arbitrary locations on Earth.</p><h4>WRAP (2019)</h4><p>In 2019, a paper titled “Presence-Only Geographical Priors for Fine-Grained Image Classification” <a href="https://arxiv.org/pdf/1906.05272">[4]</a> took an important step towards the geographic position encoders commonly used today. Similar to the work from the previous section, this paper studies how to use geographic coordinates for improving image classification models.</p><p>The key idea behind their position encoder is to leverage the periodicity of sine and cosine functions to encode the way geographic coordinates wrap around the surface of the Earth. Given latitude and longitude (λ, ϕ), both normalized to the range [-1, 1], the WRAP position encoder is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/1*DoXZgI0kuFaYO2ck--NUxg.png" /></figure><p>Unlike the approaches in the previous section, WRAP is continuous and easily computed for any position on Earth. The paper then shows empirically that training a fully-connected network on top of these features and combining them with latent image features can lead to improved performance on fine-grained image classification benchmarks.</p><h3>The Double Fourier Sphere Method</h3><p>The WRAP encoder appears simple, but it successfully encodes a key inductive bias about geographic position while remaining expressive and flexible. In order to see why this choice of position encoder is so powerful, we need to understand the Double Fourier Sphere (DFS) method <a href="https://en.wikipedia.org/wiki/Double_Fourier_sphere_method">[5]</a>.</p><p>DFS is a method of transforming any real-valued function <em>f </em>(<em>x</em>, <em>y</em>, <em>z</em>) defined on the surface of a unit sphere into a 2<em>π</em>-periodic function defined on a rectangle [-<em>π</em>, <em>π</em>] × [-<em>π</em>, <em>π</em>]. At a high level, DFS consists of two steps:</p><ol><li>Re-parametrize the function <em>f </em>(<em>x</em>, <em>y</em>, <em>z</em>) using spherical coordinates, where (λ, ϕ) ∈ [-<em>π</em>, <em>π</em>] × [0<em>, π</em>]</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*I4gbCHQ9Gc4fTLsjkkrIRg.png" /></figure><p>2. Define a new piece-wise function over the rectangle [-<em>π</em>, <em>π</em>] × [-<em>π</em>, <em>π</em>] based on the re-parametrized <em>f </em>(essentially “doubling it over”).</p><p>Notice that the DFS re-parametrization of the Earth’s surface (step 1.) preserves the properties we discussed earlier. For one, as ϕ tends to 0 or ± <em>π </em>(the Earth’s poles), the distance between two points (λ, ϕ) and (λ’, ϕ) after re-parametrization decreases. Moreover, the re-parametrization is periodic and smooth.</p><h4>Fourier Theorem</h4><p>It is a fact that any continuous, periodic, real-valued function can be represented as a weighted sum of sines and cosines. This is called the Fourier Theorem, and this weighted sum representation is called a Fourier series. It turns out that any DFS-transformed function can be represented with a finite set of sines and cosines. They are known as <strong>DFS basis functions</strong>, listed below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*C4u5HOQ8Tp3E7WmqDZH9zw.png" /></figure><p>Here, ∪ denotes union of sets, and <em>S </em>is a collection of scales (i.e. frequencies) for the sinusoids.</p><h4>DFS-Based Position Encoders</h4><p>Notice that the set of DFS basis functions includes the four terms in the WRAP position encoder. “Sphere2Vec” <a href="https://arxiv.org/pdf/2201.10489">[6]</a> is the earliest publication to observe this, proposing a unified view of position encoders based on DFS. In fact, with this generalization in mind, we can construct a geographic position encoder by choosing any subset of the DFS basis functions — WRAP is just one such choice. Take a look at <a href="https://arxiv.org/pdf/2310.06743v2">[7]</a> for a comprehensive overview of various DFS-based position encoders.</p><h4>Why are DFS-based encoders so powerful?</h4><p>Consider what happens when a linear layer is trained on top of a DFS-based position encoder: each output element of the network is a weighted sum of the chosen DFS basis functions. Hence, the network can be interpreted as a <strong>learned Fourier series</strong>.<strong> </strong>Since virtually any function defined on the surface of a sphere can be transformed using the DFS method, it follows that a linear layer trained on top of DFS basis functions is powerful enough to encode arbitrary functions on the sphere! This is akin to the universal approximation theorem for multilayer perceptrons.</p><p>In practice, only a small subset of the DFS basis functions is used for the position encoder and a fully-connected network is trained on top of these. The composition of a non-parametric position encoder with a neural network is commonly referred to as a <strong>location encoder</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D48gnx3ETxbpTN-8t9JsLg.png" /><figcaption>A depiction of a geographic location encoder. Image by author.</figcaption></figure><h3>Geographic Location Encoders Today</h3><p>As we have seen, a DFS-based position encoder can effectively encode inductive biases we have about the curvature of the Earth’s surface. One limitation of DFS-based encoders is that they assume a rectangular domain [-<em>π</em>, <em>π</em>] × [-<em>π</em>, <em>π</em>]. While this is mostly fine since the DFS re-parametrization already accounts for how distances get warped closer to the poles, this assumption breaks down at the poles themselves (ϕ = 0, ± <em>π</em>), which are lines in the rectangular domain that collapse to singular points on the Earth’s surface.</p><p>A different set of basis functions called spherical harmonics have recently emerged as an alternative. Spherical harmonics are basis functions that are natively defined on the surface of the sphere as opposed to a rectangle. They have been shown to exhibit fewer artifacts around the Earth’s poles compared to DFS-based encoders <a href="https://arxiv.org/pdf/2310.06743v2">[7]</a>. Notably, spherical harmonics are the basis functions used in the SatCLIP location encoder <a href="https://arxiv.org/pdf/2311.17179">[8]</a>, a recent foundation model for geographic coordinates trained in the style of CLIP.</p><p>Though geographic position encoders began with discrete, hand-crafted features in the 2010s, these do not easily generalize to arbitrary locations and require domain-specific metadata such as land cover and demographic data. Today, geographic coordinates are much more commonly used as neural network inputs because simple yet meaningful and expressive ways of encoding them have emerged. With the rise of web-scale datasets which are often geo-tagged, the potential for using geographic coordinates as inputs for prediction tasks is now immense.</p><h3>References</h3><p>[1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser &amp; I. Polosukhin, <a href="https://arxiv.org/pdf/1706.03762">Attention Is All You Need</a> (2017), 31st Conference on Neural Information Processing Systems</p><p>[2] A Kazemnejad, <a href="https://kazemnejad.com/blog/transformer_architecture_positional_encoding/">Transformer Architecture: The Positional Encoding</a> (2019), Amirhossein Kazemnejad’s Blog</p><p>[3] K. Tang, M. Paluri, L. Fei-Fei, R. Fergus, L. Bourdev, <a href="https://arxiv.org/pdf/1505.03873">Improving Image Classification with Location Context</a> (2015)</p><p>[4] O. Mac Aodha, E. Cole, P. Perona, <a href="https://arxiv.org/pdf/1906.05272">Presence-Only Geographical Priors for Fine-Grained Image Classification</a> (2019)</p><p>[5] <a href="https://en.wikipedia.org/wiki/Double_Fourier_sphere_method">Double Fourier Sphere Method</a>, Wikipedia</p><p>[6] G. Mai, Y. Xuan, W. Zuo, K. Janowicz, N. Lao <a href="https://arxiv.org/pdf/2201.10489">Sphere2Vec: Multi-Scale Representation Learning over a Spherical Surface for Geospatial Predictions</a> (2022)</p><p>[7] M. Rußwurm, K. Klemmer, E. Rolf, R. Zbinden, D. Tuia, <a href="https://arxiv.org/pdf/2310.06743v2">Geographic Location Encoding with Spherical Harmonics and Sinusoidal Representation Network</a> (2024), ICLR 2024</p><p>[8] K. Klemmer, E. Rolf, C. Robinson, L. Mackey, M. Rußwurm, <a href="https://arxiv.org/pdf/2311.17179">SatCLIP: Global, General-Purpose Location Embeddings with Satellite Imagery</a> (2024)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=59dafebf6f2d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/geographic-position-encoders-59dafebf6f2d">Geographic Position Encoders</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Zero-Shot Localization with CLIP-Style Encoders]]></title>
            <link>https://medium.com/data-science/zero-shot-localization-with-clip-style-encoders-2ac3cea172a8?source=rss-f11d5eb7e93d------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ac3cea172a8</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[zero-shot]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Ruth Crasto]]></dc:creator>
            <pubDate>Tue, 24 Sep 2024 04:14:15 GMT</pubDate>
            <atom:updated>2024-10-21T18:25:23.135Z</atom:updated>
            <content:encoded><![CDATA[<h4>How can we see what a vision encoder sees?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8xOQHi9Cy5ccbeEi" /><figcaption>Photo by <a href="https://unsplash.com/@stewi?utm_source=medium&amp;utm_medium=referral">Stephan Widua</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Think of your favorite pre-trained vision encoder. I’m going to assume you’ve chosen some variant of a CNN (Convolutional Neural Network) or a ViT (Visual Transformer). The encoder is a function that maps an image into a <em>d</em>-dimensional vector space. In the process, the image is transformed into a sequence of feature maps:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f4r6sN2PaUcLuo-U_yS-Ww.png" /><figcaption>Image by author.</figcaption></figure><p>A feature map (<em>w</em> × <em>h</em> × <em>k</em>) can be thought of as a collected 2D array of <em>k</em>-dimensional patch embeddings, or, equivalently, a coarse image (<em>w</em> × <em>h</em>) with <em>k</em> channels <em>f</em>₁, … <em>fₖ</em>. Both CNNs and ViTs, in their respective ways, are in the business of transforming an input image into a sequence of feature maps.</p><p>How can we see what a vision encoder sees as an image make its way through its layers? Zero-shot localization methods are designed to generate human-interpretable visualizations from an encoder’s feature maps. These visualizations, which can look like heatmaps or coarse segmentation masks, discriminate between semantically related regions in the input image. The term “zero-shot” refers to the fact that the model has not explicitly been trained on mask annotations for the semantic categories of interest. A vision encoder like CLIP, for instance, has only been trained on image-level text captions.</p><p>In this article, we begin with an overview of some early techniques for generating interpretable heatmaps from supervised CNN classifiers, with no additional training required. We then explore the challenges around achieving zero-shot localization with CLIP-style encoders. Finally, we touch on the key ideas behind<strong> </strong>GEM (Grounding Everything Module) [<a href="https://arxiv.org/pdf/2312.00878">1</a>], a recently proposed approach to training-free, open-vocabulary localization for the CLIP ViT.</p><h3>1. Localization with supervised CNN classifiers</h3><h4>Class Activation Maps (2016)</h4><p>Let’s build some intuition around the concept of localization by considering a simple vision encoder trained for image classification in a supervised way. Assume the CNN uses:</p><ol><li>Global average pooling (GAP) to transform the final feature map channels <em>f</em>₁(<em>x, y</em>), …,<em> fₖ</em>(<em>x, y</em>) into a <em>k</em>-dimensional vector. In other words, each <em>fᵢ</em> is averaged along the width and height dimensions.</li><li>A single linear layer <strong>W</strong> to map this <em>k</em>-dimensional vector into a vector of class logits.</li></ol><p>The logit for a given class <em>c</em> can then be written as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/295/1*ED6UH_3rg5OyeJtIJFDGqA.png" /></figure><p>where <strong>W</strong><em>ᵢ</em>(<em>c</em>) denotes the (scalar) weight of feature channel <em>i </em>on logit <em>c</em>, and <em>Zᵢ</em> is a normalizing constant for the average pooling.</p><p>The key observation behind Class Activation Maps [<a href="https://arxiv.org/pdf/1512.04150">2</a>] is that the above summation can be re-written as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/295/1*NuRNnvQ_0hrxU8liVbwqLQ.png" /></figure><p>In other words, the logit can be expressed as a weighted average of the final feature channels<em> </em>which is then averaged across the width and height dimensions.</p><p>It turns out that the weighted average of the <em>fᵢ</em> ’s alone gives an interpretable heatmap for class <em>c</em>, where larger values match regions in the image that are more semantically related to the class. This coarse heatmap, which can be up-sampled to match the dimensions of the input image, is called a Class Activation Map (CAM):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/298/1*Ov-aCDEmVCiGZyYJ6E-TOA.png" /></figure><p>Intuitively, each <em>f</em>ᵢ is already a heatmap for some latent concept (or “feature”) in the image — though these do not necessarily discriminate between human-interpretable classes in any obvious way. The weight <strong>W</strong><em>ᵢ</em>(<em>c</em>) captures the importance of <em>f</em>ᵢ in predicting class <em>c.</em> The weighted average thus highlights which image features are most relevant to class <em>c. </em>In this way, we can achieve discriminative localization of the class <em>c</em> without any additional training.</p><h4>Grad-CAM (2017)</h4><p>The challenge with class activation maps is that they are only meaningful under certain assumptions about the architecture of the CNN encoder. Grad-CAM [<a href="https://arxiv.org/pdf/1610.02391">3</a>], proposed in 2019, is an elegant generalization of class activation maps that can be applied to any CNN architecture, as long as the mapping of the final feature map channels <em>f</em>₁, …, <em>fₖ</em> to the logit vector is differentiable.</p><p>As in the CAM approach, Grad-CAM computes a weighted sum of feature channels <em>fᵢ </em>to generate an interpretable heatmap for a class <em>c</em>, but the weight for each <em>fᵢ </em>is computed as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/177/1*_4Pkx_Lc-Kz1T7IOzn53bg.png" /></figure><p>Grad-CAM generalizes the idea of weighing each <em>f</em>ᵢ<em> </em>proportionally to its importance for predicting the logit for class <em>c, </em>as<em> </em>measured by the average-pooled gradients of the logit with respect to elements <em>fᵢ</em>(<em>x</em>, <em>y</em>). Indeed, it can be shown that computing the Grad-CAM weights for a CNN that obeys assumptions 1–2 from the previous section results in the same expression for CAM(<em>c</em>) we saw earlier, up to a normalizing constant (see [<a href="https://arxiv.org/pdf/1610.02391">3</a>] for a proof).</p><p>Grad-CAM also goes a step further by applying ReLU on top of the weighted average of the feature channels <em>fᵢ. </em>The idea is to only visualize features which would strengthen the confidence in the prediction of class <em>c </em>should their intensity be <em>increased</em>.<em> </em>Once again, the output can then be up-sampled to give a heatmap that matches the dimensions of the original input image.</p><h3>2. Localization with CLIP</h3><p>Do these early approaches generalize to CLIP-style encoders? There are two additional complexities to consider with CLIP:</p><ol><li>CLIP is trained on a large, open vocabulary using contrastive learning, so there is no fixed set of classes.</li><li>The CLIP image encoder can be a ViT or a CNN.</li></ol><p>That said, if we could somehow achieve zero-shot localization with CLIP, then we would unlock the ability to perform zero-shot, <em>open-vocabulary </em>localization: in other words, we could generate heatmaps for arbitrary semantic classes. This is the motivation for developing localization methods for CLIP-style encoders.</p><p>Let’s first attempt some seemingly reasonable approaches to this problem given our knowledge of localization using supervised CNNs.</p><p>For a given input image, the logit for a class <em>c </em>can be computed as the cosine similarity between the CLIP text embedding of the class name and the CLIP image embedding. The gradient of this logit with respect to the image encoder’s final feature map is tractable. Hence, one possible approach would be to directly apply Grad-CAM — and this could work regardless of whether the image encoder is a ViT or a CNN.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uN_Tp24jkzdNBdG6uGVN3w.png" /><figcaption>Image by author.</figcaption></figure><p>Another seemingly reasonable approach might be to consider alignment between image patch embeddings and class text embeddings. Recall that CLIP is trained to maximize alignment between an <em>image-level </em>embedding (specifically, the CLS token embedding) and a corresponding text embedding. Is it possible that this objective implicitly aligns a <em>patch </em>in embedding space more closely to text that is more relevant to it? If this were the case, we could expect to generate a discriminative heatmap for a given class by simply visualizing the similarity between its text embedding and each patch embedding:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W1NpAaWSwKJmaP4PdZZDxA.png" /><figcaption>Image by author.</figcaption></figure><h4>Opposite Visualizations</h4><p>Interestingly, not only do both these approaches fail, but the resulting heatmaps turn out to be the <em>opposite</em> of what we would expect. This phenomenon, first described in the paper “Exploring Visual Explanations for Contrastive Language-Image Pre-training” [<a href="https://arxiv.org/pdf/2209.07046">4</a>], has been observed consistently across different CLIP architectures and across different classes. To see examples of these “opposite visualization” with both patch-text similarity maps and Grad-CAM, take a look at page 19 in the pre-print “A Closer Look at the Explainability of Contrastive Language-Image Pre-training” [<a href="https://arxiv.org/pdf/2304.05653">5</a>]. As of today, there is no single, complete explanation for this phenomenon, though some partial hypotheses have been proposed.</p><h4>Self-Attention Maps</h4><p>One such hypothesis is detailed in the aforementioned paper [<a href="https://arxiv.org/pdf/2304.05653">5</a>]. This work restricts its scope to the ViT architecture and examines attention maps in the final self-attention block of the CLIP ViT. For a given input image and text class, these attention maps (<em>w </em>× <em>h</em>) are computed as follows:</p><ol><li>The patch embedding (a <em>d-</em>dimensional vector — the same as the output dimension of the image-level embedding) with highest cosine similarity to the class text embedding is selected as an anchor patch.</li><li>The attention map is obtained by computing the query-key attention weights for the anchor patch query embedding <em>Q </em>and all key embeddings <em>K</em>, which can be reshaped into a heatmap of size <em>w </em>× <em>h</em>. The attention weights are computed as:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/268/1*J_PpMtgFc_pF75dIhutpHg.png" /></figure><p>You might expect the anchor patch to be attending mostly to other patches in the image that are semantically related to the class of interest. Instead, these query-key attention maps reveal that anchor patches consistently attend to unrelated patches just as much. As a result, query-key attention maps are blotchy and difficult to interpret (see the paper [<a href="https://arxiv.org/pdf/2304.05653">5</a>] for some examples). This, the authors suggest, could explain the noisy patch-text similarity maps observed in the CLIP ViT.</p><p>On the other hand, the authors find that value-value attention maps are more promising. Empirically, they show that value-value attention weights are larger exclusively for patches near the anchor that are semantically related to it. Value-value attention maps are not complete discriminative heatmaps, but they are a more promising starting point.</p><h3>3. Grounding Everything Module (2024)</h3><p>Hopefully, you can now see why training-free localization is not as straightforward for CLIP as it was for supervised CNNs — and it is not well-understood why. That said, a recent localization method for the CLIP ViT called the Grounding Everything Module (GEM) [<a href="https://arxiv.org/pdf/2312.00878">1</a>], proposed in 2024, achieves remarkable success. GEM is essentially a training-free method to correct the noisy query-key attention maps we saw in the previous section. In doing so, the GEM-modified CLIP encoder can be used for zero-shot, open-vocabulary localization. Let’s explore how it works.</p><h4>Self-Self Attention</h4><p>The main idea behind GEM is called self-self attention, which is a generalization of the concept of value-value attention.</p><p>Given queries <em>Q, </em>keys <em>K </em>and values <em>V, </em>the output of a self-self attention block is computed by applying query-query, key-key, and value-value attention iteratively for <em>t = </em>0, …, <em>n</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/272/1*r5gkWPwyw2dBVKP0QrbpyQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/156/1*OyfIaAQ-ayiHizXAAsU2gQ.png" /></figure><p>where <em>p</em>₀ ∈<em> </em>{<em>Q, K, V</em>} and <em>n, </em>the number of iterations, is a hyperparameter. This iterative process can be thought of as clustering the initial tokens <em>p</em>₀ based on dot-product similarity. By the end of this process, the resulting tokens <em>p</em>ₙ is a set of cluster “centers” for the initial tokens <em>p</em>₀.</p><p>The resulting self-self attention weights are then ensembled to produce the output of the self-self attention block:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*SNBY9AJ8aYx-QSMLytA8xQ.png" /></figure><p>where:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/265/1*UE89Ha3ookCsuUi_EHNE-g.png" /></figure><p>This is in contrast to a traditional query-key attention block, whose output is computed simply as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/268/1*OP962kjC__fqnVD7dV7TSQ.png" /></figure><h4>Grounding Everything Module</h4><p>Now consider our method for generating value-value attention maps in the previous section, where we first chose an anchor patch based on similarity to a class text embedding, then computed value-value attention map. GEM can be thought of as the reverse of this process, where:</p><ol><li>The first step is to apply <em>qkv-</em>ensembled self self-attention instead of regular attention for the last <em>m</em> attention blocks in the ViT (<em>m </em>is another hyperparameter). Intuitively, this is a way to compute ensembled cluster assignments for value embeddings <em>V</em>, thereby correcting the original query-key attention maps.</li><li>The second step is to generate a heatmap by computing the cosine similarity between patch embeddings output from the modified ViT and the class text embedding. This effectively gives a class logit for each cluster.</li></ol><p>This set of logits can then be reshaped to produce a discriminative heatmap for the chosen class, which can take the form of any arbitrary text! Below are some examples of GEM heatmaps for various class prompts (red indicates higher similarity to the class prompt):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*utkyUVJskvc735vSDfpt9w.png" /><figcaption>GEM heatmaps for different text classes generated by the author | (<strong>Top</strong>) Photo by <a href="https://unsplash.com/@nirzar">Nirzar Pangarkar</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a> | (<strong>Bottom</strong>) Photo by <a href="https://unsplash.com/@arnavdas">Arnav Das</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Discriminative localization can transform an image-level encoder into a model that can be used for semantic segmentation, without the need for notoriously expensive mask annotations. Moreover, training-free localization is a powerful approach to making vision encoders more explainable, allowing us to see what they see.</p><p>For supervised vision models, zero-shot localization began with class activation maps, a technique for a specific kind of CNN architecture. Later, a generalization of this approach, applicable to any supervised CNN architecture, was proposed. When it comes to CLIP-style encoders, however, training-free localization is less straightforward: the phenomenon of opposite visualizations remains largely unexplained and exists across different CLIP encoder architectures. As of today, some localization techniques for the CLIP ViT such as GEM have proven successful. Is there a more generalized approach waiting to be discovered?</p><h3>References</h3><ol><li>W. Bousselham, F. Petersen, V. Ferrari, H. Kuehne, <a href="https://arxiv.org/pdf/2312.00878">Grounding Everything: Emerging Localization Properties in Vision-Language Transformers</a> (2024), 2024 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)</li><li>B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, A. Torralba, <a href="https://arxiv.org/pdf/1512.04150">Learning Deep Features for Discriminative Localization</a> (2016), 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)</li><li>R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, D. Batra, <a href="https://arxiv.org/pdf/1610.02391">Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization</a> (2017), 2017 IEEE International Conference on Computer Vision (ICCV)</li><li>Y. Li, H. Wang, Y. Duan, H. Xu, X. Li, <a href="https://arxiv.org/pdf/2209.07046">Exploring Visual Explanations for Contrastive Language-Image Pre-training</a> (2022)</li><li>Y. Li, H. Wang, Y. Duan, J. Zhang, X. Li, <a href="https://arxiv.org/pdf/2304.05653">A Closer Look at the Explainability of Contrastive Language-Image Pre-training</a> (2024)</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ac3cea172a8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/zero-shot-localization-with-clip-style-encoders-2ac3cea172a8">Zero-Shot Localization with CLIP-Style Encoders</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>