<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aleksandr Movchan on Medium]]></title>
        <description><![CDATA[Stories by Aleksandr Movchan on Medium]]></description>
        <link>https://medium.com/@movchan74?source=rss-a15137937199------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*BZgWmPVZADXSYQp5IPCHPg.jpeg</url>
            <title>Stories by Aleksandr Movchan on Medium</title>
            <link>https://medium.com/@movchan74?source=rss-a15137937199------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 21:37:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@movchan74/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Apply Distance Metric Learning to Street-to-Shop Problem]]></title>
            <link>https://blog.mlreview.com/how-to-apply-distance-metric-learning-for-street-to-shop-problem-d21247723d2a?source=rss-a15137937199------2</link>
            <guid isPermaLink="false">https://medium.com/p/d21247723d2a</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[fashion]]></category>
            <category><![CDATA[keras]]></category>
            <dc:creator><![CDATA[Aleksandr Movchan]]></dc:creator>
            <pubDate>Sun, 14 Jan 2018 22:04:05 GMT</pubDate>
            <atom:updated>2018-01-18T17:00:53.331Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*-D6nLYnhPBBWhcZIpPCdIA.png" /></figure><p>Let’s start with a definition of street-to-shop problem — identifying a fashion item in a user image and finding it in an online shop. Have you ever seen somebody in the street and thought “Wow, this is a nice dress, I wonder where I can buy it?” I haven’t. But for me, it was the cool task to try distance metric learning techniques. I hope that you will find it interesting too.</p><h3>Dataset</h3><p>Firstly, we need a dataset for it. Actually, I came to this idea after I found out that there are tons of images taken by users on <a href="http://aliexpress.com/">Aliexpress</a>. And I thought “Wow, I can make a search by image using this data, just for fun of course”. I have decided to focus on women’s top clothing for simplicity.</p><p>Below is the list of the categories I used for scrapping:</p><ul><li>Dresses</li><li>Blouses &amp; Shirts</li><li>Hoodies &amp; Sweatshirts</li><li>Sweaters</li><li>Jackets &amp; Coats</li></ul><p>I used <a href="https://pypi.python.org/pypi/requests">requests</a> and <a href="https://pypi.python.org/pypi/beautifulsoup4">BeautifulSoup</a> for scrapping. Seller images can be obtained from the main page of the item, but for user’s images, we need to go through feedback pages. There is a thing called “colors” on item’s page. Color can be just item of another color or even completely other items. So we will consider different colors as different items.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-c1F0MZS9b72fZOSJHEnNA.png" /><figcaption>“Colors” from item’s page</figcaption></figure><p>You can find the code that I have used to get all information about one item (it scraps even more than we need for our task) by link <a href="https://github.com/movchan74/street_to_shop_experiments/blob/master/get_item_info.py">https://github.com/movchan74/street_to_shop_experiments/blob/master/get_item_info.py</a>.</p><p>All we need is to go through search pages by each category, take URLs of all items and use the function above to get the info about each item.</p><p>Finally, we will have two sets of images for each item: images from a seller ( field url for each element initem[&#39;colors&#39;]) and images from users (field imgs for each element initem[&#39;feedbacks&#39;]).</p><p>For each color, we have only one image from a seller, but it can be more than one image for each color from users (sometimes there are no images for color at all).</p><p>Great! We got data. However, the collected dataset is noisy:</p><ul><li>There are noisy images from users (photos of package boxes, photos of texture or only some part of an item, unpacked items, unrelated photos).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*cQelr2jTLRKjsOiOdH5QHQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*mpFxtolik2rlRo5sZucynA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*rSt8Sh1Fp6cG2z2hfgn3Xw.jpeg" /><figcaption>Examples of noise in user’s images.</figcaption></figure><p>To mitigate this problem I have labeled 5000 images into two categories: good images and noise images. In the beginning, my plan was to train a classifier for two categories and use it to clean dataset. But later I decided to leave this idea for future work and just added cleaned images to the test and validation sets.</p><ul><li>The second problem is there are items that sold by several sellers. Sellers even have the same images sometime (or slightly edited images). But how to deal with it? The easiest way is to do nothing and use a robust algorithm for distance metric learning. It can however affect validation because we can have the same item in the validation and training data. So it leads to a data leak. Another way is to use something to find similar (or even identical images) and merge them into one item. We can use perceptual hashing to find identical images (like <a href="http://www.hackerfactor.com/blog/index.php?/archives/432-Looks-Like-It.html">phash</a> or <a href="https://fullstackml.com/wavelet-image-hash-in-python-3504fdd282b5">whash</a>), or we can train a model on noisy data and apply the model to find similar images. I chose the second option because it allows merging even slightly edited images.</li></ul><h3>Distance Metric Learning</h3><p>One of the most popular distance metric learning method is the triplet loss:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*3YfyfyBhJDLRZ_bCYDPNlg.png" /></figure><p>where <em>max(x, 0)</em> is the hinge function, <em>d(x, y)</em> is the distance function between <em>x</em> and <em>y, F(x)</em> is deep neural network, <em>M</em> is the margin, <em>a</em> is the anchor, <em>p</em> is the positive point, <em>n</em> is the negative point.</p><p><em>F(a), F(p), F(n)</em> are points in high dimensional space (embeddings) produced by a deep neural network. It is worth mentioning that the embeddings often needs to be normalized to have unit length, i.e., <em>||x|| = 1</em>, in order to be robust to illumination and contrast changes and for training stability. The anchor and the positive samples belong to the same class, the negative sample is the instance of another class.</p><p>So the main idea of the triplet loss is to separate embeddings of the positive pair (anchor and positive) from embeddings of the negative pair (anchor and negative) by a distance margin <em>M</em>.</p><p>But how to select the triplet (<em>a</em>, <em>p, n)</em>? We can just randomly select samples as a triplet but it causes following problems. Firstly, there are N³ possible triplets. It means that we need a lot of time to go through all possible triplets. But actually, we don’t need to do it, because after few iterations of training there will be many triplets which don’t violate the triplet constraint (give zero loss). It means that these triplets are useless for a training.</p><p>One of the most common way of triplet selection is hard negative mining:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/234/1*Cu83aP0ZHHhtA27YfzMJsw.png" /></figure><p>Selecting the hardest negatives can in practice lead to bad local minima early on in training. Specifically, it can result in a collapsed model (i.e. <em>F(x) = 0</em>). In order to mitigate this we can use semi-hard negative mining.</p><p>Semi-hard negative samples are further away from the anchor than the positive sample but they are still hard (violate triplet constraint) because they lie inside the margin M.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/1*RqniuIWysdU8_St1o3r3hw.png" /><figcaption>Conditions for a triplet with semi-hard negative sample</figcaption></figure><p>There are two way to generate semi-hard (and hard) negative samples: online and offline.</p><ul><li>Online means that we randomly select samples from the train dataset as a mini-batch and select triplets from samples inside it. However, we need to have a large mini-batch size for the online method. It is not possible in my case because I have only one GTX 1070 with 8Gb RAM.</li><li>In the offline method we need to stop training after some time, predict embeddings for some amount of samples, select triplets and train model with these triplets. It’s mean that we need to do forward pass two times but it’s price for the offline method.</li></ul><p>Good! We already can start train the model with the triplet loss and offline semi-hard negative mining. But! There is always a “but” in this imperfect world. We need one more trick to successfully solve street-to-shop problem. Our task is to find seller’s image most similar to user’s image. However, usually seller’s images have much better quality (in terms on lighting, camera, position) than user’s images so we have two domains: seller’s images and user’s images. In order to get efficient model we need to reduce a gap between these two domains. This problem is called domain adaptation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*s5uc_wrt6-hckTkanrtVzA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*Ng1HkFJWpdlsDDwG6BvbFA.jpeg" /><figcaption>Left: user’s image, right: seller’s image</figcaption></figure><p>I propose a really simple technique to reduce domain gap: let’s select anchors from seller’s images, positive and negative samples from user’s images. That’s all! Simple yet effective.</p><h3>Implementation</h3><p>To implement my ideas and to do fast experimenting I have used <a href="http://keras.io/">Keras</a> library with Tensorflow backend.</p><p>I chose Inception V3 model as base CNN for my model. As usual, I initialized CNN with ImageNet weights. I have added two fully connected layers after global pooling with L2-normalization at the end of the network. The size of embedding is 128.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/22890a04ca3665f04cce284712802b18/href">https://medium.com/media/22890a04ca3665f04cce284712802b18/href</a></iframe><p>We also need to implement the triple loss function. We pass the anchor, the positive/negative samples as single mini-batch and divide it into 3 tensors inside the loss function. The distance function is squared euclidean distance.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9b826454cfc271fe479eb7ecf0ccb296/href">https://medium.com/media/9b826454cfc271fe479eb7ecf0ccb296/href</a></iframe><p>And compile model:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/87e616ce9d36d13523965522563402f4/href">https://medium.com/media/87e616ce9d36d13523965522563402f4/href</a></iframe><h3>Experimental Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JHjUJc79KsbarB_RLmcAZQ.png" /><figcaption>Retrieval results. First column — query (user’s image), next 5 — most similar seller’s images.</figcaption></figure><p>Performance is measured in terms of recall at K (R@K).</p><p>Let’s take a look how to calculate R@K. Each user’s image from validation set was used as a query and we need to find the corresponding seller’s image. We take one query image, calculate embedding vector and search nearest neighbors of this vector among vectors of all seller’s images. We use not only seller’s images from the validation set but images from the train set too because it allows to increase the number of distractors and makes our task more challenging.</p><p>So we have a query image and a list of the most similar seller’s images. If there is a corresponding seller image in the K most similar images then we return 1 for this query else return 0. Now we need to make it for each user’s image in the validation set and find an average of scores from each query. It will be R@K.</p><p>As I said before I have cleaned the small amount of user’s images from noisy images. So I have measured a quality of the model on two validation datasets: full validation set and a subset of only clean images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*YhDV_RHaczsuxugNzm-uyQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*UVscCdnZL_-UGwZTBg0-Mg.png" /><figcaption>R@K for validation data</figcaption></figure><p>Results are far from ideal, there are many things to do:</p><ul><li>Clean user’s images from noise. I already did first step into this direction by cleaning a small set.</li><li>Merge items more accurately (at least in the validation set).</li><li>Decrease domain gap. I suppose it can be done by domain-specific augmentation (for example lighting augmentation) and by using specialized methods (like this one <a href="https://arxiv.org/abs/1409.7495">https://arxiv.org/abs/1409.7495</a>).</li><li>Apply another distance metric learning technique. I have tried this one <a href="https://arxiv.org/abs/1703.07464">https://arxiv.org/abs/1703.07464</a>, but it works worse in my case.</li><li>Collect more data of course.</li></ul><h3>Demo, code and trained model</h3><p>I have made a demo of the model. You can check it out here: <a href="http://vps389544.ovh.net:5555/">http://vps389544.ovh.net:5555/</a>. You can upload your own image for search or use random image from validation set.</p><p>Code and trained model: <a href="https://github.com/movchan74/street_to_shop_experiments">https://github.com/movchan74/street_to_shop_experiments</a></p><p>Thanks for reading. If you’re enjoying the article, please let me know by clapping. If you want more information, you can connect with me on <a href="https://www.linkedin.com/in/aleksandr-movchan-6b8838b7/">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d21247723d2a" width="1" height="1" alt=""><hr><p><a href="https://blog.mlreview.com/how-to-apply-distance-metric-learning-for-street-to-shop-problem-d21247723d2a">How to Apply Distance Metric Learning to Street-to-Shop Problem</a> was originally published in <a href="https://blog.mlreview.com/">ML Review</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>