<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Romain Futrzynski on Medium]]></title>
        <description><![CDATA[Stories by Romain Futrzynski on Medium]]></description>
        <link>https://medium.com/@fromain?source=rss-20e58a37c7c8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*8UmBUhHQET1zh3UEKWU0Dg.png</url>
            <title>Stories by Romain Futrzynski on Medium</title>
            <link>https://medium.com/@fromain?source=rss-20e58a37c7c8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 15:10:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@fromain/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Image Similarity With Deep Learning Explained]]></title>
            <link>https://pub.towardsai.net/image-similarity-with-deep-learning-explained-ba7e57e51d3?source=rss-20e58a37c7c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba7e57e51d3</guid>
            <category><![CDATA[explained]]></category>
            <category><![CDATA[similarity-search]]></category>
            <category><![CDATA[image-recognition]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Romain Futrzynski]]></dc:creator>
            <pubDate>Thu, 04 Feb 2021 22:38:28 GMT</pubDate>
            <atom:updated>2021-02-06T15:17:27.135Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V_j1eGhGt8gYfn-gZrIMJg.jpeg" /><figcaption><strong>Image similarity with deep learning to find similar-looking products.</strong> | <a href="https://unsplash.com/photos/_ztjWhn51lc">Sketch</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>, images <a href="https://www.zalando.se/pullandbear-vardagsklaenning-light-brown-puc21c0j8-o11.html">1</a>,<a href="https://www.zalando.se/ivy-and-oak-vardagsklaenning-purple-iv321c08o-j12.html">2</a>,<a href="https://www.zalando.se/glamorous-mid-dress-vardagsklaenning-red-gl921c0mp-g11.html">3</a> from <a href="https://www.zalando.se/">zalando.se</a> shown with kind permission</figcaption></figure><h4><a href="https://towardsai.net/p/category/machine-learning/deep-learning">Deep Learning</a></h4><p>Recently I gave <a href="https://youtu.be/7L2ql9YLJBY?t=45">a Meetup talk</a> where I discussed two of my favorite topics: neural networks and images. Since my company is launching a <a href="https://peltarion.com/image-similarity">feature for image similarity</a>, I thought I would sum up some ideas from this talk and give my take on a few questions:</p><p>Why work with similarity? <br>How does deep learning help? <br>Why is cosine similarity so widespread? <br>How is it different from image classification?</p><h3>What is image similarity?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*a-gVzEB6wgbszqg3DEtmRg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*oVw6qpKJboiKiVv-MQb3eA.jpeg" /><figcaption><strong>Are these images similar? Sure, why not.</strong> | Images <a href="https://unsplash.com/photos/pRBNd8uuku0">1</a>, <a href="https://unsplash.com/photos/YDcp0OLex8E">2</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Similarity is by definition a <strong>vague</strong> concept. It works well when two humans communicate, but it isn’t the easiest thing to deal with for a deterministic computer program. Think of the two images above: you could easily make anyone agree that they look similar (person in a leather jacket, contemplative, facing left), although some aspects are clearly different (standing vs. sitting, ocean vs. street background, man vs. woman).</p><p>If your goal is to evaluate specific aspects of an image then, by all means, design, and train a model for that task. However, if your criteria for “similarity” are visual but not well defined, or if you lack finely labeled data for training, this is where <em>image similarity with deep learning</em> can give you a boost.</p><p>Deep learning opens a path to consistently <strong>quantify</strong> the similarity between images, enabling the automation of even ill-defined tasks.</p><h3>What can you do with image similarity?</h3><figure><img alt="Find similar dresses online from a reference picture" src="https://cdn-images-1.medium.com/max/1024/1*pzDgros7UdxkeGCVL_j1wg.png" /><figcaption><strong>Dress concept and top 3 matches on Zalando, ranked by similarity from an </strong><a href="https://medium.com/analytics-vidhya/image-classification-with-efficientnet-better-performance-with-computational-efficiency-f480fdb00ac6"><strong>EfficientNet B0</strong></a><strong>. Thousands of images compared in the blink of an eye, free from the rigidity of keywords. </strong>| <a href="https://unsplash.com/photos/_ztjWhn51lc">Sketch</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a>, images <a href="https://www.zalando.se/pullandbear-vardagsklaenning-light-brown-puc21c0j8-o11.html">1</a>,<a href="https://www.zalando.se/ivy-and-oak-vardagsklaenning-purple-iv321c08o-j12.html">2</a>,<a href="https://www.zalando.se/glamorous-mid-dress-vardagsklaenning-red-gl921c0mp-g11.html">3</a> from <a href="https://www.zalando.se/">zalando.se</a> shown with kind permission</figcaption></figure><p>Arguably the most obvious application is reverse searching. I’m somehow old enough to have known a time when keywords ruled the internet, and good luck finding anything without spelling them exactly right.</p><p>Reverse searching lets you start with the content and gives you the associated keywords. Nowadays, of course, search engines are a lot more flexible and you may not be interested in finding the keywords, but you might still start from an example and <strong>explore</strong> by browsing from like to like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J8Y0mm7jPTFE1TMvuwEOJw.png" /><figcaption><strong>Extreme classification: </strong><a href="https://ai.googleblog.com/2019/05/announcing-google-landmarks-v2-improved.html"><strong>Google releases datasets</strong></a><strong> with hundreds of thousands of landmarks to recognize.</strong></figcaption></figure><p>Deep learning similarity techniques are also behind some of <a href="https://www.kaggle.com/c/landmark-recognition-2020/discussion/200506">the best methods</a> for <strong>landmark recognition</strong>, if you need help remembering where your vacation pictures were taken. And when you can recognize landmarks all around the world, you might as well <a href="https://medium.com/towards-artificial-intelligence/the-strengths-weaknesses-of-face2vec-facenet-612a28a17826">check how well it works</a> for <strong>face recognition</strong> too.</p><p>Finally, if you like to have things well organized, image similarity can be applied to <strong>data clustering</strong>. This allows you to leverage a combination of explicit information, like the type of clothes, and visual features that are learned implicitly by a deep learning model.</p><figure><img alt="Data clustering mixing labeled categories and visual similarity" src="https://cdn-images-1.medium.com/max/1024/1*ohbki9Ou1qVcpu0WSZQ3gQ.png" /><figcaption><strong>Clustering items from an online retail shop, colored by category. Categories like the shoes are visually unique, while other categories are more diverse and may visually overlap with their neighbors. </strong>| Images <a href="https://unsplash.com/photos/dPCx8JOIGuk">1</a>,<a href="https://unsplash.com/photos/2lQ9lvIAxWY">2</a>,<a href="https://unsplash.com/photos/_fOL6ebfECQ">3</a>,<a href="https://unsplash.com/photos/8hAsLeE6Fbo">4</a>,<a href="https://unsplash.com/photos/BAoncvnAl8I">5</a>,<a href="https://unsplash.com/photos/SCIJBrxWn_g">6</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>How does image similarity work?</h3><p>I can’t emphasize enough that similarity is a vague concept. If we want to make any progress without fixing it to a specific definition, we have to think abstractly for a moment.</p><p>Let’s say that for any data item, specifically here an image, we want a representation of this data that is easy to compare with other items. If this representation is small in size, <strong>searching through thousands of items will be fast</strong> (if only by virtue of loading less data from disk). That’s essentially the same idea as creating a <a href="https://en.wikipedia.org/wiki/Hash_table">hash table</a> of your data, an angle detailed further in <a href="https://medium.com/towards-artificial-intelligence/deep-hashing-for-similarity-search-9273aac054db">Deep Hashing for Similarity Search</a>.</p><p>However, unlike a typical hash function, the compressed representations should be comparable to directly indicate the degree of similarity between their corresponding items. Since we haven’t defined what we mean by similar anyway, let’s start with the idea that the data representation should <strong>preserve information</strong>. This can be guaranteed if there is a way to reconstruct the data from its representation.</p><p>So what we’re looking for is a way to compress the data in a meaningful way. <br>This is where <strong>deep neural networks come in handy</strong>. In particular, the autoencoder structure checks a lot of useful boxes:</p><ul><li>Preserves information by reconstructing the input</li><li>Can handle various types of data</li><li>Fixed-size compressed representation</li><li>More or less continuous function of input</li><li>Trainable for noise robustness</li><li>Fine-tunable for specific meaning of “similar”</li></ul><figure><img alt="Autoencoder architecture" src="https://cdn-images-1.medium.com/max/1024/1*qvcnWg2uOa0oFHUzsywxjg.png" /><figcaption><strong>The autoencoder structure: data is compressed to a fixed size by an encoder network, yet information is preserved since the input can be recovered by a decoder network.</strong></figcaption></figure><p>The fact that autoencoders are (more or less) continuous functions of their input, and that they can be trained for noise robustness, is very good news for similarity applications. It means that inputs that look the same should have nearly identical compressed representations.</p><p>However, it could be argued that if the decoder is allowed to be arbitrarily complex, it could possibly behave more like a hash function. This would make it difficult to compare the compressed embeddings directly.</p><p>That’s why in practice, the base architecture of similarity models resembles classifier networks more than autoencoders. Arguably this preserves much less information, but it allows to make comparisons with the cosine similarity. Not to mention that classifier networks are much more widespread and easier to train than autoencoders.</p><h3>Why use cosine similarity?</h3><p>When it comes to similarity, “<a href="https://deepai.org/machine-learning-glossary-and-terms/cosine-similarity">cosine similarity</a>” is on everyone’s lips as it’s the operation that is almost always used to measure how close two items are. So when you read that two images are 90% similar, it most likely means that the cosine similarity of their compressed representation is equal to 0.90, or equivalently that they are 25 degrees apart.</p><p>Bringing in trigonometry might seem unnecessary when dealing with deep neural networks, but it comes rather naturally from the way networks calculate. To understand why, let’s have a look at a simple classifier network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kq0qmoCmLRoC8nzpohu4SQ.png" /><figcaption><strong>A classification network. A CNN like VGG acts as the encoder and compresses images into coordinates. A single dense layer classifies the coordinates by calculating their distance from the category lines.</strong></figcaption></figure><p>Here’s a classifier that I trained on 3 classes from <a href="https://cvml.ist.ac.at/AwA2/">animals with attributes 2</a>. The encoder does the heavy part of image processing and compresses the input into coordinates in an N-dimensional space (here N=2).</p><p>The “decoder”, however, is reduced to a single layer and can merely act as a classifier, as opposed to reconstructing the entire input image. To decide if a point coming from the encoder belongs to a given category, it can only do a very simple thing: draw a line (or plane) in the N-dimensional space. If a point is on the right side of this line, it belongs to the category. If the point is on the wrong side, it doesn’t belong to the category from this line.</p><p>Both the encoder that calculates the coordinates and the classifier that draws the lines are trained together so that every image from a known category can be correctly classified. As we increase the number of known categories (especially if the categories are mutually exclusive), we can see that the points created by the encoder organize themselves in clusters spread around a circle.</p><p>As a result, the angle between two points becomes a natural way to compare them. The <strong>cosine similarity</strong>, which maps an angle between 0° and ±180° to a nice value between 1 and -1, offers an intuitive measure of similarity.</p><h3>How is image similarity different from classification?</h3><p>It’s true that image similarity is closely related to image classification, if only because it uses the same classifier networks as processing workhorse. However, there are a couple of points that make image similarity its own thing.</p><h4>Image similarity considers many dimensions</h4><p>In an image classification problem, we are only concerned with figuring out whether or not an image belongs to one (or possibly several) discrete categories, which means that a lot of information is discarded from the final output.</p><p>By using the representation of the data before the final classification layer, more aspects of the data can be used.</p><p>In the example below, I give a twist on the <a href="https://youtu.be/oKzNUGz21JM">good old MNIST dataset</a>. I trained a network to recognize handwritten digits, and surely enough the points get clustered according to their category — the digit between 0 and 9. However, I also defined 2 more dimensions that the network had to learn:</p><ul><li>The parity: whether a digit is even or odd</li><li>The magnitude of the digit</li></ul><figure><img alt="Cosine similarity combines MNIST digit class, digit parity, and magnitude" src="https://cdn-images-1.medium.com/max/1024/1*-BQD_b9Xi7d2TvLlM3Lq5Q.png" /><figcaption><strong>A network can be explicitly trained to recognize the digit on an image, the digit parity, and the digit magnitude. The cosine similarity measures the similarity between two images by considering all of these dimensions at once.</strong></figcaption></figure><p>The cosine similarity between two images combines all these dimensions and returns a single value that is minimal for the same digit, slightly bigger for consecutive digits of same parity, and largest for digits of different parity.</p><h4>Dimensions can be learned implicitly</h4><p>If you trained a classifier network to recognize dolphins, tigers, and horses, these animals are the training domain and you shouldn’t count on classifying anything else. According to the image below, the network I trained previously would classify most images of zebras as being a tiger, which is simply <em>wrong</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yz2ueHc6EU-xRuGM4AoLcw.png" /><figcaption><strong>This network calculates that zebras, which it wasn’t trained to recognize, look like something between horses and tigers. Perhaps due to the stripes, the average zebra is seen as cos(25°) = 0.90 similar to tigers.</strong></figcaption></figure><p>The cosine similarity combines many dimensions that are learned implicitly (presumably here, the striped pattern and the presence of legs or grass) into a fluid numeric value. As a result, you get from the same network that zebras look like something between horses and tigers, even though you didn’t train explicitly to recognize the class of zebras.</p><p>This particular network might find zebras a bit more <em>similar</em> to tigers than you would, but thankfully the network does so using deterministic operations that can be tuned.</p><h4>Image similarity can be trained differently</h4><p>If you’re willing to let the network learn implicit dimensions entirely on its own, you might not even need any labeled categories to train on.</p><p>Using labels only indicating if two images are similar or not, networks can be trained thanks to <a href="https://towardsdatascience.com/contrasting-contrastive-loss-functions-3c13ca5f055e">contrastive losses</a>. This works if you know the similarity relationships between every pair of images in a dataset, or only between some of the pairs.</p><p>Such loss functions are rather different from the <a href="https://peltarion.com/knowledge-center/documentation/modeling-view/build-an-ai-model/loss-functions">loss functions</a> traditionally used to train classification networks. Indeed, their purpose is not to push an input image towards a fixed category but towards another image known to be similar.</p><h3>Going further</h3><p><strong>Tutorials:</strong></p><ul><li><a href="https://medium.com/pytorch/image-similarity-search-in-pytorch-1a744cf3469">Image Similarity Search in PyTorch</a></li><li><a href="https://peltarion.com/knowledge-center/documentation/tutorials/find-similar-images-of-fruits">Find similar images of fruits</a></li></ul><p><strong>Meetup talk:</strong> <a href="https://youtu.be/7L2ql9YLJBY?t=45">Image similarity with deep learning</a> — digital meetup with Solita &amp; Romain Futrzynski/Peltarion</p><p><strong>Literature:</strong></p><ul><li><a href="https://arxiv.org/abs/2002.05709">A Simple Framework for Contrastive Learning of Visual Representations</a>, Ting Chen et al., 2020</li><li><a href="https://arxiv.org/abs/1905.08139">Self-Supervised Similarity Learning for Digital Pathology</a>, Jacob Gildenblat, Eldad Klaiman, 2020</li><li><a href="https://arxiv.org/abs/2010.01650">Supporting large-scale image recognition with out-of-domain samples</a>, Christof Henkel, Philipp Singer, 2020</li><li><a href="https://arxiv.org/abs/1804.06655">Deep Face Recognition: A Survey</a>, Mei Wang, Weihong Deng, 2020</li><li><a href="https://arxiv.org/abs/1805.01978">Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination</a>, Zhirong Wu et al., 2018</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba7e57e51d3" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/image-similarity-with-deep-learning-explained-ba7e57e51d3">Image Similarity With Deep Learning Explained</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Meaning from Text: Self-attention Step-by-step Video]]></title>
            <link>https://pub.towardsai.net/getting-meaning-from-text-self-attention-step-by-step-video-7d8f49694f89?source=rss-20e58a37c7c8------2</link>
            <guid isPermaLink="false">https://medium.com/p/7d8f49694f89</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[attention]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[bert]]></category>
            <dc:creator><![CDATA[Romain Futrzynski]]></dc:creator>
            <pubDate>Tue, 01 Sep 2020 08:54:36 GMT</pubDate>
            <atom:updated>2021-02-07T11:41:43.596Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Key/Query interaction in the multi-head self-attention" src="https://cdn-images-1.medium.com/max/1024/1*3XvVEQwBftk6ZiqgsXXQig.png" /></figure><h4><a href="https://towardsai.net/p/category/nlp">Natural Language Processing</a></h4><p>In October 2019, <a href="https://blog.google/products/search/search-language-understanding-bert/">Google announced</a> that it would process search queries with the BERT model that its researchers have developed. This model can grasp difficult nuances of language: in the search <em>2019 brazil traveler to the USA need a visa</em>, it is understood that the traveler is Brazilian, and the destination is the USA. From now on, Google says, this search returns the page of the U.S. embassy in Brazil, and no longer shows a page about U.S. citizens traveling to Brazil.</p><p>Remarkably, most of the attention mechanism at the core of <a href="https://peltarion.com/blog/data-science/a-year-in-review-nlp-in-2019">many transformer models like BERT</a> relies on just a few basic vector operations.</p><p><strong>Let’s see how it works.</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F-9vVhYEXeyQ%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D-9vVhYEXeyQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F-9vVhYEXeyQ%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a60d14709b77330f6b635c24f6b5bc3a/href">https://medium.com/media/a60d14709b77330f6b635c24f6b5bc3a/href</a></iframe><h3>What’s the matter with context?</h3><p>If you see the word <em>bank</em>, you might think about a financial institution, the office where your advisor works, the portable battery that charges your phone on the move, or even the edge of a lake or river.<br>If you’re given more context, as in <em>It’s a pleasant walk by the river bank</em>, you can realize that <em>bank</em> goes well with the <em>river</em>, so it must mean the land next to some water. You could also realize that you can <em>walk by</em> this bank, so it must look like a footpath along the river. The whole sentence <strong>adds up</strong> to create a mental picture of the <em>bank</em>.</p><p>Self-attention seeks to do the very same thing.</p><h3>Word embedding</h3><p>A word like <em>bank</em> is called a <strong>token</strong> when it represents a fundamental piece of text, is commonly encoded as a vector of real, continuous values: the <strong>embedding vector</strong>.</p><p><a href="http://jalammar.github.io/illustrated-word2vec/">Determining the values</a> inside the embedding vector of a token is a large part of the heavy lifting in text processing. Thankfully, with hundreds of dimensions available to organize the vocabulary of known tokens, embeddings can be pre-trained to <a href="https://www.youtube.com/watch?v=gQddtTdmG_8">relate numerically in ways that reflect how their tokens relate in natural language</a>.</p><h3>How to contextualize the embeddings?</h3><p>The key to the state of the art performance in Natural Language Processing (NLP) is to transform the embeddings to create the right numerical picture from the tokens in any given sentence.</p><p>This is what the <strong>scaled dot-product self-attention</strong> mechanism does elegantly with (mostly) a few operations of linear algebra.</p><figure><img alt="Self-attention" src="https://cdn-images-1.medium.com/max/1024/1*iHhebIuMqG6bYnUhyacl5g.png" /><figcaption>Self-attention mechanism. Image by the author.</figcaption></figure><ul><li><strong>Token relationships<br></strong>The words in a sentence sometimes relate to each other, like <em>river</em> and <em>bank, </em>and sometimes they don’t. To determine how related two tokens are, attention simply calculates the <strong>scalar product</strong> between their embeddings.<br>We can imagine that the embeddings of <em>bank</em> and <em>river</em> are more similar since they should both encode the aspect of nature so that their scalar product should be higher than if the tokens were completely unrelated.</li><li><strong>Keys, queries, and values</strong><br>Unfortunately, calculating the scalar product directly on the embeddings would simply tend to give higher values when two tokens are the same and give smaller values otherwise. But what grammatical analysis taught us is that important relationships can happen between words that can be completely different: a subject and a verb, a preposition, and a complement, etc.<br>To have more flexibility, the embeddings go through different <strong>linear projections</strong> so that one embedding creates a <strong>key</strong>, a <strong>query, </strong>and a<strong> value </strong>vector. The projections allow us to select which components of the embeddings to focus on and to orient them so that the scalar products between the keys and the queries represent the relationships that matter.</li><li><strong>Activations</strong><br>The scalar products between a query and the keys, which give the level of relationship between the query’s token and every other token, are typically <strong>scaled</strong> down for numerical stability, then passed through a <strong>softmax</strong> <a href="https://peltarion.com/knowledge-center/documentation/modeling-view/build-an-ai-model/activations">activation function</a>.<br>The softmax makes large relationships exponentially more significant. Since this operation is non-linear, it also means that self-attention can be re-applied several times to achieve more and more complex transformations, making the process deep learning.</li><li><strong>Linear combinations</strong><br>New <strong>contextualized embeddings</strong> are created by combining the <strong>values</strong> corresponding to every input token, in proportions given by the results of the softmax function: if the query of the <em>river</em> token has a strong relationship with the key of the <em>bank</em> token, the value of <em>bank</em> is added in large part to the contextualized embedding for <em>river</em>.</li></ul><h3>Multi-head attention and BERT</h3><p>A single sequence of input embeddings can be projected using <strong>many</strong> <strong>different</strong> sets of <strong>key</strong>, <strong>query</strong>, and <strong>value</strong> projections in what is called <strong>multi-head attention</strong>. Each projection set can focus on calculating different types of relationships between the tokens and create specific contextualized embeddings.</p><p>The contextualized embeddings coming from different attention heads are simply concatenated together.</p><figure><img alt="Multi-head attention: several self-attention heads are applied in parallel, and the output from each head is concatenated" src="https://cdn-images-1.medium.com/max/987/1*_E2Ycw7oHpIJkp_OeOOTNg.png" /><figcaption>Multi-head attention. Image by the author.</figcaption></figure><p>Deep learning models for Natural Language Processing typically apply many layers of multi-head attention and mix in extra operations to get robust results.</p><figure><img alt="Illustration of a complete NLP model, BERT, transforming embeddings to get more representative values" src="https://cdn-images-1.medium.com/max/845/1*X80E2bByAXJ2kEWuqwJSzw.png" /><figcaption>BERT processes a sentence to output contextualized embeddings that are more representative of the true meaning. Image by the author.</figcaption></figure><p>For instance, the <a href="https://peltarion.com/knowledge-center/documentation/modeling-view/build-an-ai-model/blocks/bert-encoder">BERT Encoder</a> uses the WordPiece embeddings of tokens but always begins by adding them to <strong>positional</strong> <strong>embeddings</strong>. This step gives information about the order of the tokens in the input sentence, which self-attention would not consider otherwise.</p><p>Additional linear projections, normalization, and feed-forward layers give the whole model more flexibility and stability.</p><p>The result is a model that can remove ambiguity from natural text and reduce it to precise values that you can use to automatically search, classify, or even annotate text content.</p><h3>Read more</h3><p>Literature</p><ul><li><a href="https://arxiv.org/abs/1706.03762">Attention is All You Need</a></li><li><a href="https://arxiv.org/abs/1810.04805">BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding</a></li><li><a href="https://arxiv.org/abs/1609.08144">Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation</a></li></ul><p>Online resources</p><ul><li><a href="http://jalammar.github.io/illustrated-word2vec/">The Illustrated Word2vec</a></li><li><a href="http://jalammar.github.io/illustrated-transformer/">The illustrated transformer</a></li><li><a href="https://www.youtube.com/watch?v=gQddtTdmG_8">Vectoring words (word embeddings)</a></li><li><a href="https://peltarion.com/webinars/nlp-and-bert">Webinar: How NLP and BERT will change the language game</a></li><li><a href="https://peltarion.com/blog/data-science/a-year-in-review-nlp-in-2019">A year in review: NLP in 2019</a></li><li><a href="https://blog.google/products/search/search-language-understanding-bert/">Understanding searches better than ever before</a></li><li><a href="https://nlp.seas.harvard.edu/2018/04/03/attention.html">The Annotated Transformer</a></li><li><a href="https://github.com/google-research/bert/blob/eedf5716ce1268e56f0a50264a88cafad334ac61/modeling.py#L558-L751">Lines of the official Google attention implementation for BERT</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d8f49694f89" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/getting-meaning-from-text-self-attention-step-by-step-video-7d8f49694f89">Getting Meaning from Text: Self-attention Step-by-step Video</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>