<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by NeuroHive on Medium]]></title>
        <description><![CDATA[Stories by NeuroHive on Medium]]></description>
        <link>https://medium.com/@neurohive?source=rss-c5715faa7cc7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*QkwHhMJFcpYGM17yLpDIaw.png</url>
            <title>Stories by NeuroHive on Medium</title>
            <link>https://medium.com/@neurohive?source=rss-c5715faa7cc7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 02:52:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@neurohive/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[EXAM — State-of-The-Art Method for Text Classification]]></title>
            <link>https://medium.com/@neurohive/exam-state-of-the-art-method-for-text-classification-75647834299?source=rss-c5715faa7cc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/75647834299</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[natural-language-process]]></category>
            <category><![CDATA[text]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[classification]]></category>
            <dc:creator><![CDATA[NeuroHive]]></dc:creator>
            <pubDate>Mon, 24 Dec 2018 13:53:25 GMT</pubDate>
            <atom:updated>2018-12-25T16:07:22.541Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*owUPnr6pHrcZ3BPfO9ZImg.jpeg" /></figure><h3>EXAM — State-of-The-Art Method for Text Classification</h3><p>One of the widely used Natural Language Processing &amp; Supervised Machine Learning (ML) task in different problems and used cases is the so-called <strong>Text Classification</strong>. It is an example of Supervised Machine Learning task since a labeled dataset containing text documents and their labels is used for training a text classifier.</p><h3>Previous Works on Text Classification</h3><p>Historically, the problem of text classification was addressed by feature engineering and training simple classifiers such as linear regression models or Support Vector Machines (SVMs). As a classification task, text classification can be the multi-class — unique category for each text sample out of multiple classes and multi-label — multiple categories.</p><p>In recent years, deep neural networks have shown remarkable success also in the task of text classification. From a Natural Language Processing (NLP) perspective, text classification was tackled by learning a word-level representation using word embeddings and later learning a text-level representation used as the feature for classification. Although successful, this kind of encoding-based methods ignores fine-grained details and clues for classification (because a general text-level representation is learned by compressing the word-level representations).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*EwtVsMM6OHpW-kcCZRC1Hw.png" /><figcaption>Encoding-based methods for text classification with text-level matching</figcaption></figure><h3>State-of-the-art Idea</h3><p><strong>Researchers from Shandong University and the National University of Singapore, </strong><a href="https://arxiv.org/abs/1811.09386"><strong>have proposed a new model</strong></a><strong> for text classification that incorporates word-level matching signals into the text classification task. </strong>Their method uses an interaction mechanism to inject fine-grained, word-level matching clues into the classification process.</p><h3>EXAM</h3><p>To address the problem of incorporating finer word-level matching signals, the researchers propose to calculate the matching scores between the words and classes explicitly. The key idea is to compute an interaction matrix out of word-level representation which will carry the matching clues on a word-level. Each entry of this matrix is the matching score between a word and a particular class.</p><p>The proposed framework for text classification called EXAM — EXplicit interAction Model, contains three major components: a word-level encoder, interaction layer, and the aggregation layer. This three-level architecture allows to encode and classify text using both fine- and coarse-grained signals and clues. The whole architecture is given in the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*r0I-oUywpSHCJf_zJRBESA.png" /><figcaption>The architecture of the proposed method</figcaption></figure><p>In the past, word-level encoders have been explored extensively within the NLP community, and there exist very powerful word-level encoders. The authors use an existing method as a word-level encoder, and in their paper, they elaborate on the other two components of their architecture: the interaction and aggregation layer.</p><p>The interaction layer the main contribution and novelty in the proposed method are based on the well-known interaction mechanism. In this part, the researchers’ first project classes into real-valued latent representations. They use a trainable representation matrix to encode each of the classes for later to be able to compute word-to-class interaction scores. The final scores are estimated simply using dot product as the interaction function between the target word and each class. More complex functions have not been considered due to an increase in computational complexity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*u6O78CF6NdTqvnLOKQCG7A.png" /><figcaption>The visualization of interaction features of EXAM</figcaption></figure><p>Finally, as an aggregation layer, they employ a simple fully-connected two-layer MLP. They also mention that here it is possible to use more complex aggregation layer incorporating CNN or LSTM. The MLP is used to compute the final classification logits taking the interaction matrix and the word-level encodings. Cross-entropy is used as a loss function for the optimization.</p><h3>Evaluation and Comparison</h3><p>To evaluate the proposed framework for text classification, the researchers did extensive experiments both in multi-class and multi-label settings. They show that their method outperforms current corresponding state-of-the-art methods remarkably.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*ZwPoRsAtscmxSfiyWr2Yvg.png" /><figcaption>Statistics of the used datasets for evaluation</figcaption></figure><p>For the evaluation, they set three different baseline model types: 1) models based on feature engineering; 2) Char-based deep models, and 3) Word-based deep models. They used publicly available benchmark datasets from (Zhang, Zhao, and LeCun 2015) to evaluate the proposed method. There are in total six text classification datasets, corresponding to sentiment analysis, news classification, question-answer, and ontology extraction tasks, respectively. In the paper, they show that EXAM achieves the best performance over the three datasets: AG, Yah. A., and DBP. The evaluation and comparison to other methods can be seen in the tables below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZvYYdNCXIi2VlySFK22V7w.png" /><figcaption>Test Set Accuracy [%] on multi-class document classification tasks and comparison with other methods</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*Rg1UO8EAFGISiqHy6mPPfg.png" /></figure><h3>Conclusion</h3><p>To conclude, this work represents an important contribution in the field of Natural Language Processing (NLP). It is the first work that introduces finer, word-level matching clues in text classification with deep neural network models. The proposed model achieves state-of-the-art performance on several datasets.</p><p><em>Originally published at </em><a href="https://neurohive.io/en/state-of-the-art/exam-text-classification"><em>neurohive.io</em></a><em> on December 24, 2018.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=75647834299" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[True Face Super-Resolution Upscaling]]></title>
            <link>https://medium.com/neurohive-computer-vision/true-face-super-resolution-upscaling-636f4ee031b2?source=rss-c5715faa7cc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/636f4ee031b2</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[super-resolution]]></category>
            <category><![CDATA[scaling-up]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[NeuroHive]]></dc:creator>
            <pubDate>Mon, 01 Oct 2018 15:03:45 GMT</pubDate>
            <atom:updated>2018-10-02T14:23:05.696Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/885/0*KlhWlesdn5H0rE58.png" /></figure><p>The performance of the most facial analysis techniques relies on the resolution of the corresponding image. Face alignment or face identification is not going to work correctly when the resolution of a face is adversely low.</p><h3>What’s Face Super-Resolution?</h3><p><strong>Face super-resolution (FSR) or face hallucination</strong>, provides a viable way to recover a high-resolution (HR) face image from its low-resolution (LR) counterpart. This research area has attracted increasing interest in the recent years, and the most advanced deep learning methods achieve state-of-the-art performance in face super-resolution.</p><p>However, even these methods often produce the results with the distorted face structure and only partially recovered facial details. Deep learning based FSR methods fail to super-resolve LR faces under large pose variations.</p><p>How can we solve this problem?</p><ul><li>Augmenting training data with large pose variations still leads to suboptimal results where facial details are missing or distorted.</li><li>Directly detecting facial components or landmarks in LR faces is also suboptimal and may lead to ghosting artifacts in the final result.</li></ul><p>But what about a method that super-resolves LR faces images while collaboratively predicting face structure? Can we use heatmaps to represent the probability of the appearance of each facial component?</p><p>We are going to discover this very soon, but let’s first check the previous approaches to the problem of face super-resolution.</p><h3>Related Work</h3><p>Face hallucination methods can be roughly grouped into three categories:</p><ul><li><strong>‘Global model’ based approaches</strong> aim at super-resolving an LR input image by learning a holistic appearance mapping such as PCA. For instance, <a href="https://ieeexplore.ieee.org/document/1487591/">Wang and Tang</a> reconstruct an HR output from the PCA coefficients of the LR input; <a href="https://pdfs.semanticscholar.org/c0e5/ebcfa7fd20fa688e6623fdd0dfd5eefc2fad.pdf">Liu et al.</a> develop a Markov random field (MRF) to reduce ghosting artifacts caused by the misalignments in LR images; <a href="https://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Kolouri_Transport-Based_Single_Frame_2015_CVPR_paper.pdf">Kolouri and Rohde</a> employ optimal transport techniques to morph an HR output by interpolating exemplary HR faces.</li><li><strong>Part based methods</strong> are proposed to super-resolve individual facial regions separately. For instance, <a href="https://people.csail.mit.edu/celiu/pdfs/ECCV12-ImageHallucination.pdf">Tappen and Liu</a> super-resolve HR facial components by warping the reference HR images; <a href="http://faculty.ucmerced.edu/mhyang/papers/cvpr13_hallucination.pdf">Yang et al.</a> localize facial components in the LR images by a facial landmark detector and then reconstruct missing high-frequency details from similar HR reference components.</li><li><strong>Deep learning techniques</strong> can be very different: <a href="http://faculty.ucmerced.edu/mhyang/papers/iccv2017_gan_super_deblur.pdf">Xu et al.</a> employ the framework of generative adversarial networks to recover blurry LR face images; <a href="http://personal.ie.cuhk.edu.hk/~ccloy/files/eccv_2016_hallucination.pdf">Zhu et al.</a> present a cascade bi-network, dubbed CBN, to localize LR facial components first and then upsample the facial components.</li></ul><h3>State-of-the-art-idea</h3><p>Xin Yu and his colleagues propose a multi-task deep neural network that not only super-resolves LR images but also estimates the spatial positions of their facial components. Their convolutional neural network (CNN) has two branches: one for super-resolving face images and the other — for predicting salient regions of a face coined facial component heatmaps.</p><p>The whole process looks like this:</p><ol><li>Super-resolving features of input LR images.</li><li>Employing a spatial transformer network to align the feature maps.</li><li>Estimating the heatmaps of facial components with the upsampled feature maps.</li><li>Concatenating estimated heatmaps of facial components with the upsampled feature maps.</li></ol><p>This method can super-resolve tiny unaligned face images (16 x 16 pixels) with the upscaling factor of 8x while preserving face structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/0*UQQtLis3s3uuJxc1.png" /><figcaption>(a) LR image; (b) HR image; (c) Nearest Neighbors; (d) CBN, (e) TDAE, (f) TDAE trained on better dataset, (g) suggested approach</figcaption></figure><p>Now let’s learn the details of the proposed method.</p><h3>Model overview</h3><p>The network has the following structure:</p><ol><li><strong>A multi-task upsampling network</strong> (MTUN):</li><li>an upsampling branch (composed of a convolutional autoencoder, deconvolutional layers, and a spatial transformer network);</li><li>a facial component heatmap estimation branch (HEB).</li><li><strong>Discriminative network</strong>, which is constructed by convolutional layers and fully connected layers.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/0*eFCIVK7jlgwCmrP2.png" /><figcaption>The pipeline of the suggested network</figcaption></figure><p><strong>Facial Component Heatmap Estimation.</strong> Even the state-of-the-art facial landmark detectors cannot accurately localize facial landmarks in very low-resolution images. So, the researchers propose to predict facial component heatmaps from super-resolved feature maps.</p><p>2D photos may exhibit a wide range of poses. Thus, to reduce the number of training images required for learning HEB, they suggest employing a spatial transformer network (STN) to align the upsampled features before estimating heatmaps.</p><p>In total, four heatmaps are estimated to represent four components of a face: eyes, nose, mouth, and chain (see the image below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/0*Ohdqklz9dOhmLS8i.png" /><figcaption>Visualization of estimated facial component heatmaps: (a) Unaligned LR image; (b) HR image; © Heatmaps; (d) Result; (e) The estimated heatmaps overlying the results</figcaption></figure><p><strong>Loss Function.</strong> The results of using different combinations of losses are provided below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/0*Pw-5pU4N5lhPPaWh.png" /><figcaption>Comparison of different losses</figcaption></figure><p>On the above image:</p><ol><li>unaligned LR image,</li><li>original HR image,</li><li>pixel-wise loss only,</li><li>pixel-wise and feature-wise losses combined,</li><li>pixel-wise, feature-wise, and discriminative losses,</li><li>pixel-wise and face structure losses,</li><li>pixel-wise, feature-wise, and face structure losses,</li><li>pixel-wise, feature-wise, discriminative, and face structure losses.</li></ol><p>In training their multi-task upsampling network, the researchers have selected to use the last option (h).</p><h3>Qualitative and Quantitative Comparisons</h3><p>See the qualitative comparison of the suggested approach with the state-of-the-art methods:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/0*rzqfl2p9tF5cGyfG.png" /><figcaption>Comparisons with the state-of-the-art methods: (a) Unaligned LR image; (b) HR image; (c) Bicubic interpolation; (d) VDSR; (e) SRGAN; (f) Ma et al.’s method; (g) CBN; (h) TDAE; (i) Suggested approach</figcaption></figure><p>As you can see, most of the existing methods fail to generate realistic face details, while the suggested approach outputs realistic and detailed images, which are very close to the original HR image.</p><p>Quantitative comparison with the state-of-the-art methods leads us to the same conclusions. All methods were evaluated on the entire test dataset by the average PSNR and the structural similarity (SSIM) scores.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/0*TeRfK981r6edSMYg.png" /><figcaption>Quantitative comparisons on the entire test dataset</figcaption></figure><p>The results in the table show that the approach presented here outperforms the second best with a large margin of <strong>1.75</strong> dB in PSNR. This confirms that estimating heatmaps helps in localizing facial components and aligning LR faces more accurately.</p><h3>Bottom Line</h3><p>Let’s summarize the <strong>contributions</strong> of this work:</p><ul><li>It presents a novel multi-task upsampling network that can super-resolve very small LR face images (16 x 16 pixels) by an upscaling factor of 8x.</li><li>The method not only exploits image intensity similarity but also estimates the face structure with the help of facial component heatmaps.</li><li>The estimated facial component heatmaps provide not only spatial information of facial components but also their visibility information.</li><li>Thanks to the aligning of feature maps before heatmap estimation, the number of images required for training the model is largely reduced.</li></ul><p>The method is good at super-resolving very low-resolution faces in different poses and generates realistic and detailed images free from distortions and artifacts.</p><p><em>Originally published at </em><a href="https://neurohive.io/en/state-of-the-art/realistic-face-resolution-upscaling/"><em>neurohive.io</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=636f4ee031b2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/neurohive-computer-vision/true-face-super-resolution-upscaling-636f4ee031b2">True Face Super-Resolution Upscaling</a> was originally published in <a href="https://medium.com/neurohive-computer-vision">Neurohive - CV papers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Identity Verification with Deep Learning: ID-Selfie Matching Method]]></title>
            <link>https://medium.com/coinmonks/identity-verification-with-deep-learning-id-selfie-matching-method-be56d72be632?source=rss-c5715faa7cc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/be56d72be632</guid>
            <category><![CDATA[identity]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[verification]]></category>
            <category><![CDATA[face-recognition]]></category>
            <dc:creator><![CDATA[NeuroHive]]></dc:creator>
            <pubDate>Mon, 24 Sep 2018 16:39:38 GMT</pubDate>
            <atom:updated>2020-08-31T17:41:55.507Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3YU0QHDcHnNa7deSH3N5hQ.png" /></figure><p>A large number of daily activities in our lives require <strong>identity verification</strong>. Identity verification provides a security mechanism starting from access control to systems all the way to at border crossing and bank transactions. However, in many of the activities that require identity verification, the process is done manually, and it is often slow and requires human operators.</p><p>An automated system for identity verification will significantly speed up the process and provide a seamless security check in all those activities where we need to verify our identity. One of the simplest ways to do this is to design a system that will match ID photos with selfie pictures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kEkO23rL1ItgeWpwEE8pug.png" /><figcaption>Examples of automatic ID document photo matching systems at international borders</figcaption></figure><h3>Previous works</h3><p>There have been both successful and unsuccessful attempts in the past to employ an automated system for identity verification. A successful example is Australia’s SmartGate. It is an automated self-service border control system operated by the Australian Border Force and located at immigration checkpoints in arrival halls in eight Australian international airports. It uses a camera to capture a verification picture and tries to match it to a person’s ID. Also, China has introduced such systems at train stations and airports.</p><p>While there have also been attempts to match ID Documents and selfies using traditional computer vision techniques, the better-performing methods rely on deep learning. Zhu et al. proposed the first deep learning approach for a document to selfie matching using Convolutional Neural Networks.</p><h3>State-of-the-art idea</h3><p><strong>In their new paper, researchers from Michigan State University</strong><a href="https://arxiv.org/pdf/1809.05620.pdf"><strong> proposed an improved version of their DocFace — a deep learning approach for document-selfie matching.</strong></a></p><p>They show that gradient-based optimization methods converge slowly when many classes have very few samples — like in the case of existing ID-selfie datasets. To overcome this shortcoming, they propose a method, called <strong>Dynamic Weight Imprinting (DWI)</strong>. Additionally, they introduce a new recognition system for learning unified representations from ID-selfie pairs and an open-source face matcher called <strong>DocFace+</strong>, for ID-selfie matching.</p><h3>Method</h3><p>There are a large number of problems and constraints in building an automated system for ID-selfie matching. Speaking about ID-selfie matching, numerous challenges are different from general face recognition.</p><p>The two main challenges are low quality of document (as well as selfie) photos due to compression and the large time gap between the document issue time and the verification moment.</p><p>The whole method is based on transfer learning. A base neural network model is trained on a large-scale face dataset <strong>(MS-Celeb 1M)</strong>, and then features are transferred to the target domain of ID-selfie pairs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ysj9FxMiPfpWgpfY.png" /></figure><p>Arguing that the convergence is very slow and very often the training can get stuck in local minima when dealing with many classes having very few samples, the researchers propose to use <strong>Additive Margin Softmax (AM-Softmax)</strong> loss function alongside with a novel optimization method that they call Dynamic Weight Imprinting (DWI).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/0*mRYdxx-MjGPCd6bo.png" /><figcaption>Generalization performance of different loss functions.</figcaption></figure><h3>Dynamic Weight Imprinting</h3><p>Since Stochastic Gradient Descent updates the network with mini-batches, in a two-shot case (like the one of ID-selfie matching), each weight vector will receive signals only twice per epoch. These sparse attraction signals make little difference to the classifier weights. To overcome this problem, they propose a new optimization method where the idea is to update the weights based on sample features and therefore avoid underfitting of the classifier weights and accelerate the convergence.</p><p>Compared with stochastic gradient descend and other gradient-based optimization methods, the proposed DWI only updates the weights based on genuine samples. It only updates the weights of classes that are present in the mini-batch, and it works well with extensive datasets where the weight matrix of all classes is too large to be loaded, and only a subset of weights can be sampled for training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/0*Beme9a7RzJxDyyMi.png" /><figcaption>Comparison of AM-Softmax loss and the proposed DIAM loss.</figcaption></figure><p>The researchers trained the popular Face-ResNet architecture using stochastic gradient descent and AM-Softmax loss. Then they fine-tune the model on the ID-selfie dataset by binding the proposed Dynamic Weight Imprinting optimization method with the Additive Margin Softmax. Finally, a pair of sibling networks is trained for learning domain-specific features of IDs and selfies sharing high-level parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/523/0*r87wh3BsbV8df8Wx.png" /><figcaption>Workflow of the proposed method. A base model is trained on a large scale unconstrained face dataset. Then, the parameters are transferred to a pair of sibling networks, who have shared high-level modules.</figcaption></figure><h3>Results</h3><p>The proposed ID-selfie matching method achieves excellent result obtaining true acceptance rate TAR to 97.51 ± 0.40%. The authors report that their approach using the MS-Celeb-1M dataset and the AM-Softmax loss function achieves 99.67% accuracy on the standard verification protocol of LFW and a Verification Rate (VR) of 99.60% at False Accept Rate (FAR) of 0.1% on the BLUFR protocol.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/0*hza9mNyd7DLkjW5S.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/0*pE9bTZU-eMbIOAHm.png" /><figcaption>Examples of falsely classified images by our model on the Private ID-selfie dataset.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/0*pDob8UK9jH3ZZQIg.png" /><figcaption>The mean performance of constraining different modules of the sibling networks to be shared</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/0*25wSLpoxkEnYFQ-c.png" /><figcaption>Comparing Static and Dynamic Weight Imprinting regarding TAR</figcaption></figure><h3>Comparison with other state-of-the-art</h3><p>The approach was compared with other state-of-the-art general face matches since there are no existing public ID-selfie matching methods. The comparison with these methods is given concerning TAR — true accept rate and FAR — false accept rate and shown in the tables below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/0*W5q6oQcX_ik-vdzT.png" /><figcaption>The mean (and s.d. of) performance of different matches on the private ID-selfie dataset</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/0*38F35rdbPC2nEFoq.png" /><figcaption>Evaluation results were compared with other methods on Public-IvS dataset</figcaption></figure><h3>Conclusion</h3><p>The proposed DocFace+ method for ID-selfie matching shows the potential of transfer learning, especially in tasks where not enough data is available. The proposed method is achieving high accuracy in selfie to ID matching and has potential to be employed in identity verification systems. Additionally, the proposed novel optimization method — Dynamic Weight Imprinting shows improved convergence and better generalization performance and represents a significant contribution to the field of machine learning.</p><blockquote><a href="https://coincodecap.com/?utm_source=coinmonks">Get Best Software Deals Directly In Your Inbox</a></blockquote><figure><a href="https://coincodecap.com/?utm_source=coinmonks"><img alt="" src="https://cdn-images-1.medium.com/max/584/0*OJ-qb5G6i863msBB.png" /></a></figure><p><em>Originally published at </em><a href="https://neurohive.io/en/state-of-the-art/identity-verification-with-deep-learning-id-selfie-matching-method/"><em>neurohive.io</em></a><em> on September 24, 2018.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be56d72be632" width="1" height="1" alt=""><hr><p><a href="https://medium.com/coinmonks/identity-verification-with-deep-learning-id-selfie-matching-method-be56d72be632">Identity Verification with Deep Learning: ID-Selfie Matching Method</a> was originally published in <a href="https://medium.com/coinmonks">Coinmonks</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deforming Autoencoders (DAEs) — Learning Disentangled Representations]]></title>
            <link>https://medium.com/neurohive-computer-vision/deforming-autoencoders-daes-learning-disentangled-representations-7ef7d71a00a1?source=rss-c5715faa7cc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ef7d71a00a1</guid>
            <category><![CDATA[generative-adversarial]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[autoencoder]]></category>
            <dc:creator><![CDATA[NeuroHive]]></dc:creator>
            <pubDate>Fri, 21 Sep 2018 15:35:44 GMT</pubDate>
            <atom:updated>2018-09-24T09:30:34.286Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fBisbEJ4LrLp02jaCx9ySQ.png" /></figure><p>Generative Models are drawing a lot of attention within the Machine Learning research community. This kind of models has practical applications in different domains. Two of the most commonly used and efficient approaches recently are Variational Autoencoders (VAE) and Generative Adversarial Networks (GAN).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ruID_Ekd-Ix63lUsefMzJw.png" /></figure><p>While vanilla autoencoders can learn to generate compact representations and reconstruct their inputs well, they are pretty limited when it comes to practical applications. The fundamental problem of the standard autoencoders is that the latent space (in which they encode the input data distribution) may not be continuous therefore may not allow smooth interpolation. A different type of autoencoders called Variational Autoencoders (VAEs) can solve this problem, and their latent spaces are, <em>by design, </em>continuous, allowing easy random sampling and interpolation. This allowed VAEs to become very popular and used for many different tasks, especially in Computer Vision.</p><p>However, controlling and understanding deep neural networks, especially deep autoencoders is a difficult task and being able to control what the networks are learning is of crucial importance.</p><h3>Previous works</h3><p>The problem of feature disentanglement has been explored in the literature, for image and video processing and text analysis. Disentangling factors of variation are necessary for the goal of controlling and understanding deep networks and many attempts have been made to solve this problem.</p><p>Past work has explored the separation of the latent image representations into dimensions that account for different factors of variation. For example identity, illumination and spatial support, then low-dimensional transformations, such as rotations, translation, or scaling or more descriptive levels of variation such as age, gender, wearing glasses.</p><h3>State-of-the-art idea</h3><p><strong>Recently, </strong><a href="https://arxiv.org/abs/1806.06503"><strong>Zhixin Shu et al. introduced Deforming Autoencoders or shortly DAEs</strong></a><strong> — a generative model for images that disentangles shape from the appearance in an unsupervised manner.</strong> In their paper, researchers propose a way to disentangle shape and appearance by assuming that object instances are obtained by deforming a prototypical object or ‘template’. This means that the object’s variability can be separated into variations associated with spatial transformations linked to the object’s shape, and variations that are associated with appearance. As simple as the idea sounds, this kind of disentanglement using deep autoencoders and unsupervised learning proved to be quite powerful.</p><h3>Method</h3><p>The proposed method can disentangle shape and appearance as factors of variation in a learned lower-dimensional latent space. The technique employs a deep learning architecture comprising of an encoder network that encodes the input image into two latent vectors (one for each shape and appearance) and two decoder networks taking the latent vectors as input and outputting generated texture and deformation, respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jHn8XcesuWYnwq2cw_t4lQ.png" /><figcaption>The proposed Deforming Autoencoder architecture comprising of one encoder and two decoder networks</figcaption></figure><p>Independent decoder networks learn the appearance and deformation functions. The generated spatial deformation is used to warp the texture to the observed image coordinates. In this way, the Deforming Autoencoder can reconstruct the input image and at the same time disentangle the shape and appearance as different features. The whole architecture is trained in an unsupervised manner using only simple image reconstruction loss.</p><p>In addition to Deforming Autoencoders (DAEs), the researchers propose Class-aware Deforming Autoencoders, which learn to reconstruct an image while at the same time disentangle the shape and appearance factors of variation conditioned by the class. To make this possible, they introduce a classifier network that takes a latent vector (third latent vector used to encode the class, besides the latent vectors for shape and appearance). This kind of architecture allows learning a mixture model conditioned on the class of the input image (rather than a joint multi-modal distribution).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uZlLveBf9jrfGHN9bhASwQ.png" /><figcaption>The proposed class-aware Deforming Autoencoder</figcaption></figure><p>They show that introducing class-aware learning drastically improves the performance and stability of the training. Intuitively, this can be explained as the network learning to separate the spatial deformation that is different among different classes.</p><p>Also, the researchers propose a Deforming Autoencoder to learn to disentangle albedo and shading (a widespread problem in computer vision) from facial images. They call this architecture Intrinsic Deforming Autoencoder, and it is shown in the picture below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NqAZHRMeHTKSQpiHxdxsQQ.png" /><figcaption>Intrinsic Deforming Autoencoder (Intrinsic DAE)</figcaption></figure><h3>Results</h3><p>It is shown that this method can successfully disentangle shape and appearance while learning to reconstruct an input image in an unsupervised manner. They show that class-aware Deforming Autoencoders provide better results in both reconstruction and appearance learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J-mChMPqlKjcef0t8kxAqg.png" /><figcaption>Results of the image reconstruction of MNIST images using Deforming Autoencoder</figcaption></figure><p>Besides the qualitative evaluation, the proposed Deforming Autoencoder architecture is evaluated quantitatively concerning landmark localization accuracy. The method was evaluated on</p><ol><li>unsupervised image alignment/appearance inference;</li><li>learning semantically meaningful manifolds for shape and appearance;</li><li>unsupervised intrinsic decomposition</li><li>unsupervised landmark detection.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aNd--U1ZwnqlPY-M6-Gxlg.png" /><figcaption>Results of the image reconstruction of MNIST images using Class-aware Deforming Autoencoder</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-S2BgAyvplfBV_ad3qHlLw.png" /><figcaption>Unsupervised alignment on images of palms of left hands. (a) The input images; (b) reconstructed images; © texture images warped with the average of the decoded deformation; (d) the average input image; and (e) the average texture</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C3TuBnqAPXWKs0nqLc0Xrg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ACh6c7-VB9k30eG6aklIhw.png" /><figcaption>Smooth interpolation of the latent space representation</figcaption></figure><p>Watch the video:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FhwVGD6BYZd0%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DhwVGD6BYZd0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FhwVGD6BYZd0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3530bf47636e1ae2cd8b688d8ce3b79e/href">https://medium.com/media/3530bf47636e1ae2cd8b688d8ce3b79e/href</a></iframe><h3>Comparison with other state-of-the-art</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*Z8K80hlJgoZ4OE5PvW0kpA.png" /><figcaption>Mean error on unsupervised landmark detection on the MAFL test set</figcaption></figure><p>The proposed method was evaluated on the MAFL test — mean error on unsupervised landmark detection. It outperforms the self-supervised approach proposed by Thewlis et al.</p><h3>Conclusion</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MJXxi5tpbvfaFo8RiKprLg.png" /><figcaption>Lighting interpolation with Intrinsic-DAE</figcaption></figure><p>As I mentioned previously, being able to disentangle factors of variation can be of crucial importance for many tasks. Disentanglement allows complete control and understanding of deep neural network models, and it may be a key to solving problems. This approach introduced Deforming Autoencoders as a specific architecture able to disentangle particular factors of variation (in this case shape and appearance). The results show that this method can successfully disentangle the factors of variability by employing an autoencoder architecture.</p><p><em>Originally published at </em><a href="https://neurohive.io/en/state-of-the-art/deforming-autoencoders/"><em>neurohive.io</em></a><em> on September 21, 2018.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ef7d71a00a1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/neurohive-computer-vision/deforming-autoencoders-daes-learning-disentangled-representations-7ef7d71a00a1">Deforming Autoencoders (DAEs) — Learning Disentangled Representations</a> was originally published in <a href="https://medium.com/neurohive-computer-vision">Neurohive - CV papers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep Clustering Approach for Image Classification Task]]></title>
            <link>https://medium.com/good-audience/deep-clustering-approach-for-image-classification-task-68fb73bfe601?source=rss-c5715faa7cc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/68fb73bfe601</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[NeuroHive]]></dc:creator>
            <pubDate>Thu, 20 Sep 2018 15:59:30 GMT</pubDate>
            <atom:updated>2018-09-20T22:23:51.402Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4f6ERa1H6lXscRtZXE1pew.png" /></figure><p>Clustering of images seems to be a well-researched topic. But in fact, little work has been done to adapt it to the end-to-end training of visual features on large-scale datasets.</p><p>The existence and usefulness of <strong>ImageNet</strong>, a fully-supervised dataset, has contributed to pre-training of convolutional neural networks. However, <a href="http://academictorrents.com/details/564a77c1e1119da199ff32622a1609431b9f1c47">ImageNet</a> is <strong>not so large by today standards</strong>: it “only” contains a million images. Now we need to move to the next level and build a bigger and more diverse dataset, potentially consisting of <strong>billions</strong> of images.</p><h3>No Supervision Required</h3><p>Can you imagine the number of manual annotations required for this kind of dataset? This is huge! Replacing labels by raw metadata is also a wrong solution as this leads to biases in the visual representations with unpredictable consequences.</p><p>So, it looks like we need methods that can be trained on internet-scale datasets with no supervision. That’s precisely what a Facebook AI Research team suggests. DeepCluster is a novel clustering approach for the large-scale end-to-end training of convolutional neural networks.</p><p>The authors of this method claim that the resulting model outperforms the current state of the art by a significant margin on all the standard benchmarks. But let’s first discover the previous works in this research area.</p><h3>Previous Works</h3><p>All the related work can be arranged into three groups:</p><ul><li><strong>Unsupervised learning of features</strong>: for example, <a href="https://arxiv.org/abs/1604.03628">Yang et al.</a> iteratively learn Сonvnet features and clusters with a recurrent framework, <a href="https://arxiv.org/abs/1704.05310">Bojanowski and Joulin</a> learn visual features on a large dataset with a loss that attempts to preserve the information flowing through the network.</li><li><strong>Self-supervised learning</strong>: for instance, <a href="https://arxiv.org/abs/1505.05192">Doersch et al.</a> use the prediction of the relative position of patches in an image as a pretext task, <a href="https://arxiv.org/abs/1603.09246">Noroozi and Favaro</a> train a network to rearrange shuffled patches spatially. These approaches are usually domain dependent.</li><li><strong>Generative models</strong>: for example, <a href="https://arxiv.org/abs/1605.09782">Donahue et al.</a> and <a href="https://arxiv.org/abs/1606.00704">Dumoulin et al.</a> have shown that using a GAN with an encoder results in visual features that are pretty much competitive.</li></ul><h3>State-of-the-art idea</h3><p><a href="https://research.fb.com/publications/deep-clustering-for-unsupervised-learning-of-visual-features/">DeepCluster</a> is a clustering method presented recently by a Facebook AI Research team. The method iteratively groups the features with a standard clustering algorithm, k-means, and uses the subsequent assignments as supervision to update the weights of the network. For simplicity, the researchers have focused their study on k-means, but other clustering approaches can also be used, like for instance, Power Iteration Clustering (PIC).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Jg4X5cmTm1dXcMGd.jpg" /><figcaption>Images and their 3 nearest neighbors: query –&gt; results from randomly initialized network –&gt; results from the same network after training with DeepCluster PIC</figcaption></figure><p>Such an approach has a significant advantage over the self-supervised methods as it doesn’t require specific signals from the output or extended domain knowledge. As we will see later, DeepCluster achieves significantly higher performance than previously published unsupervised methods.</p><p>Let’s now have a closer look at the design of this model.</p><h3>Method overview</h3><p>The performance of random convolutional networks is intimately tied to their convolutional structure which gives a strong prior on the input signal. The idea of DeepCluster is to exploit this weak signal to bootstrap the discriminative power of a Сonvnet.</p><p>As illustrated below, the method implies iterative clustering of deep features and using the cluster assignments as pseudo-labels to learn the parameters of the Сonvnet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sq6ByFJc4_hmkPYm.jpg" /><figcaption>Illustration of the proposed method</figcaption></figure><p>This type of alternating procedure is prone to <strong><em>trivial solutions</em></strong>, which we’re going to discuss briefly right now:</p><ul><li><strong>Empty clusters. </strong>Automatic reassigning of empty clusters solve this problem during the k-means optimization.</li><li><strong>Trivial parametrization.</strong> If the vast majority of images is assigned to a few clusters, the parameters will exclusively discriminate between them. The solution to this issue lies in sampling images based on a uniform distribution over the classes, or pseudo-labels.</li></ul><p>DeepCluster is based on a standard AlexNet architecture with five convolutional layers and three fully connected layers. To remove color and increase local contrast, the researchers apply a fixed linear transformation based on Sobel filters.</p><p>So, the model doesn’t look complicated, but let’s check its performance on the <em>ImageNet </em>classification and transfer tasks.</p><h3>Results</h3><p>The results of preliminary studies are demonstrated below:</p><ul><li>(a) the evolution of the Normalized Mutual Information (NMI) between the cluster assignments and the <em>ImageNet</em> labels during training;</li><li>(b) the development of the model’s stability along the training epochs;</li><li>(c) the impact of the number of clusters <em>k </em>on the quality of the model (k = 10,000 gives the best performance).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*eH9ECQVVvzkHlRwiOzNjfQ.png" /><figcaption>Preliminary studies</figcaption></figure><p>To assess the quality of a target filter, the researchers learn an input image that maximizes its activation. The figure below shows these synthetic filter visualizations and the top 9 activated images from a subset of 1 million images from <em>YFCC100M</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cTJnq3NLADoTEr9j.jpg" /><figcaption>Filter visualization and top 9 activated images for target filters in the layers conv1, conv3, and conv5 of an AlexNet trained with DeepCluster</figcaption></figure><p>Deeper layers in the network seem to capture larger textural structures. However, it looks like some filters in the last convolutional layers merely replicate the texture already captured in the previous layers.</p><p>Check below the results from the last convolutional layers but this time using VGG-16 architecture instead of AlexNet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*97anD_YoAEs1lQRD.jpg" /><figcaption>Filter visualization and top 9 activated images for target filters in the last convolutional layer of VGG-16 trained with DeepCluster</figcaption></figure><p>As you can see, the filters learned without any supervision, can capture quite complex structures.</p><p>Next figure shows the top 9 activated images of some filters that seem to be semantically coherent. The filters in the top row reflect the structures that are highly correlated with object class, while the filters in the bottom row seem to trigger on style.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_XUjPsWu5unswhHp.jpg" /><figcaption>Top 9 activated images for target filters in the last convolutional layer</figcaption></figure><h3>Comparison</h3><p>To compare DeepCluster to other methods, the researchers train a linear classifier on top of different frozen convolutional layers. The table below reports the classification accuracy of different state-of-the-art approaches on the <em>ImageNet</em> and the <em>Places</em> dataset.</p><p>On <em>ImageNet</em>, DeepCluster outperforms state of the art from conv2 to conv5 layers by 1–6%. Poor performance in the first layer is probably due to the Sobel filtering discarding color. Remarkably, the difference of performance between DeepCluster and a supervised AlexNet is only around 4% at conv2-conv3 layers, but rises to 12.3% at conv5, showing where the AlexNet stores most of the class level information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*lG1BeZrwNggrlmzkNfv_Iw.png" /><figcaption>Table 1. Linear classification on ImageNet and Places using activations from the convolutional layers of an AlexNet as features</figcaption></figure><p>The same experiment on the <em>Places</em> dataset reveals that DeepCluster yields conv3-conv4 features that are comparable to those trained with the ImageNet labels. This implies that when the target task is sufficiently far from the domain covered by the ImageNet, labels are less important.</p><p>The next table summarizes the comparisons of DeepCluster with other feature learning approaches on the three tasks: classification, detection, and semantic segmentation. As you can see, DeepCluster outperforms all previous unsupervised methods on all three tasks with the most substantial improvement in semantic segmentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*b1XCb4TYDUafceiEwZyZbA.png" /><figcaption>Table 2: Comparison of the proposed approach to state-of-the-art unsupervised feature learning on classification, detection, and segmentation on Pascal VOC</figcaption></figure><h3>Bottom Line</h3><p>DeepCluster proposed by the Facebook AI Research team achieves performance that is significantly better than the previous state of the art on every standard transfer task. What is more, when tested on the <em>Pascal VOC 2007</em> object detection task with fine-tuning, DeepCluster is only 1.4% below the supervised topline.</p><p>This approach makes a little assumption about the inputs and doesn’t require much domain-specific knowledge, which makes it a good candidate for learning deep representations specific domains where labeled datasets are not available.</p><p><em>Originally published at </em><a href="https://neurohive.io/en/state-of-the-art/deep-clustering-approach/"><em>neurohive.io</em></a><em> on September 20, 2018.</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2F7b0181%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2F7b0181%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/748a06e9019942f2677bba917ef55804/href">https://medium.com/media/748a06e9019942f2677bba917ef55804/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68fb73bfe601" width="1" height="1" alt=""><hr><p><a href="https://medium.com/good-audience/deep-clustering-approach-for-image-classification-task-68fb73bfe601">Deep Clustering Approach for Image Classification Task</a> was originally published in <a href="https://medium.com/good-audience">Good Audience</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Realistic Exemplar-Based Image Colorization]]></title>
            <link>https://medium.com/neurohive-computer-vision/realistic-exemplar-based-image-colorization-6c5697ac1a81?source=rss-c5715faa7cc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c5697ac1a81</guid>
            <category><![CDATA[black-and-white]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[image-processing]]></category>
            <category><![CDATA[coloring]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[NeuroHive]]></dc:creator>
            <pubDate>Tue, 18 Sep 2018 14:11:42 GMT</pubDate>
            <atom:updated>2018-09-20T16:15:27.721Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kTW789MViKeyPWpSfPsOWw.jpeg" /></figure><p>Image colorization is a widespread problem within computer vision. The ultimate objective of image colorization is to map a gray-scale image to a visually plausible and perceptually meaningful color image.</p><p>It is important to mention that image colorization is an ill-posed problem. The definition itself says that the goal is obtaining “visually plausible and realistic image colorization,” which is conditioned as a result of the multi-modal nature of the problem — various colorizations are possible for a gray-scale image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hOCZK3b9I--D3sUToby67g.jpeg" /></figure><p>Image decolorization (color-to-grayscale) is non-invertible image degradation, and starting from this point it is clear that multiple meaningful and plausible results are possible for a single input image. Moreover, the expected “visually plausible” result is subjective and differs pretty much from different people.</p><h3>Previous works</h3><p>However, in the past researchers have proposed some solutions to the ill-conditioned problem of realistic image colorization. As I mentioned multiple times, the result highly subjective and in many of the proposed methods, human intervention was included to obtain better colorization results. In the past few years, several colorization approaches exploiting the power of deep learning have been proposed. In this way the colorization is learned from large amounts of data, hoping for an improved generalization in the process of colonization.</p><p>More recently, the new approaches have examined the trade-off between controllability from interaction and robustness from learning.</p><p><strong>In </strong><a href="https://arxiv.org/pdf/1807.06587.pdf"><strong>novel approach</strong></a><strong> researchers from Microsoft Research Asia, introduce the first deep learning approach for exemplar-based local colorization.</strong></p><h3>State-of-the-art idea</h3><p>The idea of the proposed approach is to introduce a reference color image, besides the input gray-scale image to the method that will output plausible colorization. Given a reference color image, potentially semantically similar to the input image, a convolutional neural network maps a gray-scale image to an output colorized image in an end-to-end manner.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B1aUupdmoUUh_AvzFDl7RA.png" /><figcaption>The proposed idea: given an input gray-scale image and a reference image, the method outputs a realistic colorized image</figcaption></figure><h3>Method</h3><p>Besides the proposed, first deep learning based exemplar colorization method, in their paper the researchers provide a few more contributions: A reference image retrieval algorithm for reference recommendation, with which a fully automatic colorization can also be obtained, a method capable of transferability to unnatural images and an extension to video colorization.</p><p>The main contribution: the colorization method is, in fact, able to colorize an image according to a given semantically “similar” reference image. This image is given a user’s input, or it can be obtained with the image retrieval system given as the second contribution. This system is trying to find semantically similar image to the input image to reuse the local color patches and provide more realistic and plausible output colorization.</p><p>The colorization method employs two deep convolutional neural networks: <strong>similarity sub-network</strong> and <strong>colorization sub-network</strong>. The first network is a pre-processing network that measures the semantic similarity between the reference and the target using a VGG-19 network. The VGG network was pre-trained on gray-scale image object recognition task. This network gives the input to the colorization network, and it provides a robust and more meaningful similarity metric.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FdkWy40cp4SOa109lWa_FA.png" /><figcaption>The two branches in the colorization network</figcaption></figure><p>The second network — the colorization sub-network is an end-to-end Convolutional Neural Network that can learn selection, propagation, and prediction of colors simultaneously. This network takes as input the output of the pre-processing done with the similarity sub-network as well as the input grayscale image. More precisely, the data to this network is the target gray-scale image, the aligned reference image and bidirectional similarity maps (between the input and the reference image).</p><p>To control both: true reference colors (from the reference image or similar) to be used and natural colorization when no reliable reference color is available, the authors propose a branch learning scheme for the colorization network. This multi-task network involves two branches, Chrominance branch, and Perceptual branch. The same network is used and trained while taking different input depending on the branch as well as different loss function (again depending on which branch is used).</p><p>In the Chrominance branch, the network learns to selectively propagate the correct reference colors, which depends on how well the target and the reference are matched. While this network is trying to satisfy the chrominance consistency the other branch through “Perceptual loss,” enforces a close match between the result and the exact color image of high-level feature representations. Both of the networks are shown in the picture below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CHXxTaJ-dDlidEK-Q3h6Gg.png" /><figcaption>The proposed deep learning architecture for image colorization</figcaption></figure><h3>Comparison with other state-of-the-art</h3><p>The proposed method gives more than satisfactory results in realistic image colorization. The evaluation is divided into few groups: comparison with exemplar-based methods, comparison with learning based methods and comparison with interactive methods.</p><p>To compare against exemplar-based methods, the authors collected around 35 image pairs from all the papers of the comparison methods and compared quantitatively and qualitatively the results. They show that this method outperforms other existing exemplar-based methods yielding better visual results. They argue that the success comes from the sophisticated mechanism of color sample selection and propagation that are jointly learned from data rather than through heuristics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g0fJrNE_B5kvdithbMfgLg.jpeg" /><figcaption>Comparison with other exemplar-based methods</figcaption></figure><p>By sending the colorized results into VGG19 or VGG16 pre-trained on image recognition task the authors try to measure the indication of more or less natural generated images and compare against other learning-based methods. Using this evaluation, they show that the method outperforms the existing methods. However, some of the ways yield better PSNR than the proposed method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nvGivcCZOKF3O8MnL_SUow.jpeg" /><figcaption>Comparison with other learning-based methods</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NPJat1GNBRkZixdC2JFZXQ.png" /><figcaption>Evaluation of learning-based methods</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*jRc2ILdjS347Z4a-IwBGOw.jpeg" /><figcaption>The assessment based on user’s colorization preference. Comparison against the ground truth that represents the user’s colorization choice</figcaption></figure><h3>Conclusion</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PDlJHFVskuGPh7hqceCMUw.png" /><figcaption>The color reference recommendation pipeline</figcaption></figure><p>In this novel colorization approach, the authors show the power of deep learning again to solve an important although ill-conditioned problem. They try to leverage the flexibility and potential of deep convolutional neural networks and provide a robust and controllable image colorization method. They also offer a whole system for automatic image colorization based on most-similar reference image that is also extended to unnatural images as well as videos. In conclusion, a whole deployable system is proposed incorporating innovative deep-learning based method for colorizing gray-scale images in a robust, and realistic way.</p><p><em>Originally published at </em><a href="https://neurohive.io/en/state-of-the-art/image-colorization/"><em>neurohive.io</em></a><em> on September 18, 2018.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c5697ac1a81" width="1" height="1" alt=""><hr><p><a href="https://medium.com/neurohive-computer-vision/realistic-exemplar-based-image-colorization-6c5697ac1a81">Realistic Exemplar-Based Image Colorization</a> was originally published in <a href="https://medium.com/neurohive-computer-vision">Neurohive - CV papers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AlphaGAN: Natural Image Matting]]></title>
            <link>https://medium.com/neurohive-computer-vision/alphagan-natural-image-matting-4dd6ede06382?source=rss-c5715faa7cc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/4dd6ede06382</guid>
            <category><![CDATA[generative-adversarial]]></category>
            <category><![CDATA[image-processing]]></category>
            <category><![CDATA[dataset]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[NeuroHive]]></dc:creator>
            <pubDate>Tue, 11 Sep 2018 10:23:26 GMT</pubDate>
            <atom:updated>2018-09-11T13:32:23.804Z</atom:updated>
            <content:encoded><![CDATA[<p>Many image-editing and film post-production applications rely on natural image matting as one of the processing steps. The task of the matting algorithm is to estimate the opacity of a foreground object in an image or video sequence accurately. Researchers from Trinity College Dublin propose the AlphaGAN<strong> </strong>architecture for natural image matting.</p><p>In mathematical terms, every pixel <strong><em>i</em></strong> in the image is assumed to be a linear combination of the foreground and background colors:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*s1jh7vDLjBuvPAm3PzULPg.png" /></figure><p>where <em>ai</em> is a scalar value that defines the foreground opacity at pixel <strong><em>i</em> </strong>and is referred to as the alpha value.</p><p>So, how to solve this equation with so many unknown values? Let’s first discover the current <a href="https://neurohive.io/en/state-of-the-art/">state-of-the-art approaches</a> to solving this problem…</p><h3>Previous Works</h3><p>Lots of current algorithms aim to solve the matting equation by treating it as a color-problem following either <em>sampling</em> or <em>propagation</em> approaches.</p><p><em>Sample-based image matting</em> assumes that true foreground and background colors of the unknown pixel can be derived from the known foreground and background samples that are near that pixel. Methods that follow this assumption include:</p><ul><li><a href="https://www.computer.org/csdl/proceedings/cvpr/2001/1272/02/127220264-abs.html">bayesian matting</a>;</li><li>iterative matting;</li><li><a href="https://pdfs.semanticscholar.org/074e/39a1c533993dcc829d9996c6518608d01e49.pdf?_ga=2.215891991.1846355347.1536150091-466379679.1530186298">shared sampling matting</a>;</li><li><a href="http://www.eccv2016.org/files/posters/P-1B-17.pdf">sparse coding</a>.</li></ul><p><em>Propagation image matting</em> works by propagating the known alpha value between known local foreground and background samples to the unknown pixels. The examples include:</p><ul><li><a href="http://www.cse.cuhk.edu.hk/leojia/all_project_webpages/Poisson%20matting/poisson_matting.html">poisson matting</a>;</li><li><a href="http://wwwcg.in.tum.de/fileadmin/user_upload/Lehrstuehle/Lehrstuhl_XV/Research/Publications/2005/VIIP05.pdf">random walk</a>;</li><li><a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.695.658&amp;rep=rep1&amp;type=pdf">geodesic matting</a>;</li><li><a href="http://www.cs.toronto.edu/~kyros/courses/2530/papers/Lecture-08/Levin2008.pdf">spectral matting</a>;</li><li><a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.478.6701&amp;rep=rep1&amp;type=pdf">close-form matting</a>;</li><li><a href="https://ieeexplore.ieee.org/document/4587455/">fuzzy connectedness matting</a>.</li></ul><p>However, the over-dependency on color information can lead to artifacts in images where the foreground and background color distributions overlap.</p><p>Thus, recently, several <em>deep learning approaches</em> to the natural image matting were introduced, including:</p><ul><li>a two-stage network consisting of an encoder-decoder stage and a refinement stage by <a href="https://arxiv.org/pdf/1703.03872.pdf">Xu et al.</a>;</li><li>end-to-end <a href="https://neurohive.io/vidy-nejrosetej/glubokaya-svertochnaja-nejronnaja-set/">CNN</a> for deep automatic portrait matting by <a href="http://xiaoyongshen.me/papers/deepmatting.pdf">Shen et al.</a>;</li><li>end-to-end CNN that utilizes the results deduced from local and non-local matting algorithms by <a href="https://www.researchgate.net/publication/308277494_Natural_Image_Matting_Using_Deep_Convolutional_Neural_Networks">Cho et al.</a>;</li><li>granular deep learning (GDL) architecture by <a href="https://www.deepdyve.com/lp/elsevier/image-matting-in-the-perception-granular-deep-learning-nkAlnLW0J8">Hu et al</a>.</li></ul><p>But is it possible to improve further the performance of these algorithms by applying GANs? Let’s find out now!</p><h3>State-of-the-art Idea</h3><p>Lutz, Amplianitis, and Smolić from Trinity College Dublin are the first to <a href="https://arxiv.org/pdf/1807.10088.pdf">propose generative adversarial network (GAN) for natural image matting</a>. Their generator network is trained to predict visually appealing alphas, while the discriminator is trained to classify well-composited images.</p><p>The researchers build their approach by improving the network architecture of <a href="https://arxiv.org/pdf/1703.03872.pdf">Xu et al</a>. to better deal with the spatial localization issues inherent in CNNs. In particular, they use dilated convolutions to capture global context information without downscaling feature maps and losing spatial information.</p><p>We are now ready to move on to the details of AlphaGAN — this is how Lutz and his colleagues call their image matting algorithm.</p><h3>Network Architecture</h3><p><strong>AlphaGAN</strong> architecture consists of one generator G and one discriminator D.</p><p>The <em>generator G</em> is a convolutional encoder-decoder network that is trained both with the help of the ground-truth alphas as well as the adversarial loss from the discriminator. It takes an image composited from the foreground, alpha and a random background appended with the trimap as 4th-channel as input and attempts to predict the correct alpha. Resnet50 architecture is used for the encoder.</p><p>As you can see from the figure below, the decoder part of the network includes skipping connections from the encoder to improve the alpha prediction by reusing local information to capture fine structures in the image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IWBB3EjglBsQuGMzgRO7LA.png" /><figcaption>The generator of the AlphaGAN</figcaption></figure><p>The <em>discriminator D</em> tries to distinguish between real 4-channel inputs and fake inputs where the first three channels are composited from the foreground, background and the predicted alpha. PatchGAN introduced by <a href="https://arxiv.org/pdf/1611.07004.pdf">Isola et al.</a> is used for the discriminator in this network.</p><p>The full objective of the network includes alpha-prediction loss, compositional loss, and adversarial loss:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9RTeVbe7nJP3qnaPhd7l4w.png" /></figure><h3>Experimental Results</h3><p>The proposed method was evaluated based on two datasets:</p><ul><li>the <em>Composition-1k</em> dataset, which includes 1000 test images composed of 50 unique foreground objects;</li><li>the <em>alphamatting.com</em> dataset<em>,</em> which consists of 28 training images and 8 test images; for each set, three different sizes of trimaps are provided, namely, “small” (S), “large” (L) and “user” (U).</li></ul><h3>The Composition-1k Dataset</h3><p>The metrics used include <em>the sum of absolute differences</em> (SAD), <em>mean square error</em> (MSE), <em>gradient</em> and <em>connectivity</em> errors.</p><p>The researchers compared their method with several state-of-the-art approaches where there is public code available. For all methods, the original code from the authors was used, without any modifications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mKwqEwkeVdNAs2LIIhNhFw.png" /><figcaption>Quantitative results on the Composition-1k dataset. AlphaGAN’s results are shown in parenthesis</figcaption></figure><p>As it can be observed from the table, AlphaGAN delivers noticeably better results than other image matting algorithms selected for comparison. There is only one case (gradient error from the comprehensive sampling approach), where they do not achieve the best result.</p><p>See also some qualitative results of this comparison in the next set of pictures:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9QX2sEm-Lm-_i44OdBpOSw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*40YT6faR7NiOE0M0HpHhTg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*svKYjVJAfNQs86Yh5_NYaw.png" /></figure><h3>The Alphamatting.com Dataset</h3><p>The researchers submitted results generated by AlphaGAN to <em>alphamatting.com</em> benchmark and got to the top positions for some of the images:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VEL1SU0WJ_6Xg7E6KHoa8g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3P7lfi6YOQCx00akcHJWPA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UhQzdFIjFPqspTvUiS1BhA.png" /><figcaption>Alpha matting predictions for the “Troll” and “Doll” images (best results) and the “Net” image (worst result) taken from the alphamatting.com dataset. From left to right: DCNN [7], IF [1], DI [33], ‘Ours’</figcaption></figure><p>Specifically, they achieved the best results for <em>Troll</em> and <em>Doll</em> images, and the first place overall on the gradient evaluation metric. Their high results on these particular images demonstrate the advantage of using the adversarial loss from the discriminator to correctly predict the alpha values for such fine structures as hair.</p><p>The worst results of the proposed method come from the <em>Net</em> image. However, even though AlphaGAN approach appears low in the rankings for this image, the results still look very close to the top-performing approaches:</p><h3>Bottom Line</h3><p>AlphaGAN is the first algorithm that uses GANs for natural image matting. Its generator is trained to predict alpha mattes from input images while the discriminator is trained to distinguish good images composited from the ground-truth alpha from images composited with the predicted alpha.</p><p>Such network architecture produces visually appealing compositions with state-of-the-art or comparable results for the primary metrics. Exceptional performance is achieved for the images with such fine structures as hair. That is of great importance in practical matting applications, including film and TV production.</p><p><em>Originally published at </em><a href="https://neurohive.io/en/state-of-the-art/alphagan-natural-image-matting"><em>neurohive.io</em></a><em> on September 11, 2018.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4dd6ede06382" width="1" height="1" alt=""><hr><p><a href="https://medium.com/neurohive-computer-vision/alphagan-natural-image-matting-4dd6ede06382">AlphaGAN: Natural Image Matting</a> was originally published in <a href="https://medium.com/neurohive-computer-vision">Neurohive - CV papers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning 3D Face Morphable Model Out of 2D Image]]></title>
            <link>https://medium.com/neurohive-computer-vision/learning-3d-face-morphable-model-out-of-2d-image-f02cabf6ce4e?source=rss-c5715faa7cc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/f02cabf6ce4e</guid>
            <category><![CDATA[3d-modeling]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-neural-network]]></category>
            <category><![CDATA[convolutional-network]]></category>
            <category><![CDATA[face-recognition]]></category>
            <dc:creator><![CDATA[NeuroHive]]></dc:creator>
            <pubDate>Wed, 05 Sep 2018 19:16:13 GMT</pubDate>
            <atom:updated>2018-09-17T10:37:45.795Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w2arHIjagMLOzX4K_dp7mA.png" /></figure><p>The <strong>3D Morphable Model (3DMM)</strong> is a statistical model of 3D facial shape and texture. 3D Morphable Models have various applications in many fields including computer vision, computer graphics, human behavioral analysis, craniofacial surgery.</p><p>In essence, <strong>3D Morphable Models</strong> are used to model facial shapes and textures and modeling human faces is not a trivial task at all. Different identities, highly-variable face shapes, and postures make the modeling of the human face a challenging task. In this context, a 3D Morphable Model is trying to learn a model of facial shape and texture in a space where there are explicit correspondences. This means, there has to be a point-to-point correspondence between the reconstruction and all other models, enabling morphing, and second, it has to model the underlying transformations between types of faces (male to female, neutral to smile, etc.).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sig5UOAV-Pk19MTVHhDvKA.png" /></figure><p>Researchers from Michigan State University propose a novel Deep Learning-based approach to learning a 3D Morphable Model. Exploiting the power of Deep Neural Networks to learn non-linear mappings, they suggest a method for learning 3D Morphable Model out of just 2D images from in-the-wild (images not taken in a controlled environment like a lab).</p><h3>Previous Approaches</h3><p>A conventional 3DMM is learned from a set of 3D face scans with associated well-controlled 2D face images. Traditionally, 3DMM is learned through supervision by performing dimension reduction, typically Principal Component Analysis (PCA), on a training set of co-captured 3D face scans and 2D images. By employing a linear model such as PCA, non-linear transformations and facial variations cannot be captured by the 3D Morphable Model. Moreover, large amounts of high-quality 3D data are needed to model highly variable 3D face shapes.</p><h3>State of the art idea</h3><p>The idea of the <a href="https://arxiv.org/pdf/1808.09560.pdf">proposed approach</a> is to leverage the power of Deep Neural Networks or more specifically Convolutional Neural Networks (which are more suitable for the task and less expensive than multilayer perceptrons) to learn the 3D Morphable Model with an encoder network that takes a face image as input and generates the shape and albedo parameters, from which two decoders estimate shape and albedo.</p><h3>Method</h3><p>As mentioned before a linear 3DMM has the problems such as the need of 3D face scans for supervised learning, unable to leverage massive in-the-wild face images for learning, and the limited representation power due to the linear model (PCA). The proposed method learns a nonlinear 3 DMM model using only large-scale in-the-wild 2D face images.</p><h3>UV Space Representation</h3><p>In their method, the researchers use an unwrapped 2D texture (where 3 D vertex v is projected onto the UV space) as a texture representation for the shape and the albedo. They argue that keeping the spatial information is very important as they employ Convolutional Networks in their method and frontal face-images contain little information about the two sides. Therefore their choice falls on UV-space representation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZW0oc5fEvMcuSNu7Mdcg1A.png" /><figcaption>Three albedo representations. (a) Albedo value per vertex, (b) Albedo as a 2D frontal face, © UV space 2D unwarped albedo</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iOZ6K3p1ulD1ys4-AhOOMA.png" /><figcaption>UV space shape representation. x, y, z, and a combined shape representation</figcaption></figure><h3>Network architecture</h3><p>They designed an architecture that given an input image it encodes it into shape, albedo and lightning parameters (vectors). The encoded latent vectors for albedo and shape are decoded using two different Decoder networks (again Convolutional Neural Networks) to obtain face skin reflectance, image (for the albedo) and 3D face mash (for the shape). Then a differentiable rendering layer was designed to generate the reconstructed face by fusing the 3D face, albedo, lighting, and the camera projection parameters estimated by the encoder. The whole architecture is nicely presented in the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7JAne0TKHoYcQT8WOlQbAA.png" /><figcaption>The proposed method’s architecture for learning a non-linear 3DMM</figcaption></figure><p>The presented robust, learning of a non-linear 3D Morphable Model is applied to 2D Face Alignment, and 3D Face Reconstruction problems. It can also have many applications since it represents a model learning method, which can solve different problems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_eA2MK49uhd_eRoh6K4cuw.png" /><figcaption>The proposed rendering layer</figcaption></figure><h3>Comparison with other methods</h3><p>The method was evaluated against other methods on the following tasks: 2D Face Alignment, 3D Face Reconstruction and Face Editing. The suggested technique outperforms other state-of-the-art methods on these tasks. Some of the results of the evaluation are presented below.</p><h3>2D Face Alignment</h3><p>One of the critical applications of this kind of approach can become face alignment. Alignment naturally should improve facial analysis in a range of tasks (for example face recognition). However, alignment is not a straightforward task, and this method proves successful in face alignment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FeWTwqfuZ_MYWue2j7ydmA.png" /><figcaption>2D face alignment results. Invisible landmarks are marked as red. The technique can well handle extreme pose, lighting, and expression</figcaption></figure><h3>3D Face Reconstruction</h3><p>The approach was also evaluated on another task: 3D Face Reconstruction, yielding outstanding results compared to other methods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BQfz4V2zHXC0uhTrW8yeDA.png" /><figcaption>Quantitative evaluation of the 3D reconstruction</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hI6NY2R1WOdyg1tO-8QhtQ.png" /><figcaption>3D reconstruction results comparison to Sela et al. The proposed method handles facial hair and occlusions far better than this method</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JzzFi3_wvgzkYc-eXgt51Q.png" /><figcaption>3D reconstruction results comparison to VRN by Jack- son et al. on the popular CelebA dataset</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1SLJwZzahEeVCRPi89k2Ug.png" /><figcaption>3D reconstruction results comparison to Tewari et al. This result shows that the proposed method overcomes the problem of face shrinking when dealing with a different texture (like facial hair)</figcaption></figure><h3>Face Editing</h3><p>A method that learns a model and decomposes a face image into individual components allows the image to be modified and the face to be edited by manipulating different elements. The method was also evaluated on face editing tasks such as relighting and attribute manipulation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U_tv8Kn03Oj0DVEyyrKuKw.png" /><figcaption>Growing mustache editing results. The first column shows original images, the following columns show edited images with increasing magnitudes.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xZHVt1rcBmlX3_aVwexing.png" /><figcaption>Comparing to Shu et al. results (last row), the proposed method produces more realistic images, and the identity is better preserved</figcaption></figure><h3>Conclusions</h3><p>In conclusion, the proposed method will have a potentially high impact since it improves the way of learning a 3D Morphable Model. This kind of model has been widely adopted in the past since its introduction, but there was not an efficient, robust way of learning this model from in-the-wild data.</p><p>The proposed approach exploits the power of deep neural networks as very good function approximator to model the highly variable human face robustly. The unusual path of learning a 3DMM allows different manipulations and many applications of this method, some of which are presented in the paper, and many others are expected.</p><p><em>Originally published at </em><a href="https://neurohive.io/en/state-of-the-art/learning-3d-face-morphable-model-out-of-2d-images/"><em>neurohive.io</em></a><em> on September 5, 2018.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f02cabf6ce4e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/neurohive-computer-vision/learning-3d-face-morphable-model-out-of-2d-image-f02cabf6ce4e">Learning 3D Face Morphable Model Out of 2D Image</a> was originally published in <a href="https://medium.com/neurohive-computer-vision">Neurohive - CV papers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Facial Surface and Texture Synthesis via GAN]]></title>
            <link>https://medium.com/neurohive-computer-vision/facial-surface-and-texture-synthesis-via-gan-3334b2c47d10?source=rss-c5715faa7cc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/3334b2c47d10</guid>
            <category><![CDATA[generative-adversarial]]></category>
            <category><![CDATA[3d-modeling]]></category>
            <category><![CDATA[landmark]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[NeuroHive]]></dc:creator>
            <pubDate>Mon, 03 Sep 2018 15:28:27 GMT</pubDate>
            <atom:updated>2018-09-06T11:12:20.181Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*porjOERbGo4aTtZdjr5P4A.png" /></figure><p>Deep networks can be extremely powerful and effective in answering complex questions. But it is also well-known that in order to train a really complex model, you’ll need lots and lots of data, which closely approximates the complete data distribution.</p><p>With the lack of real-world data, many researchers choose data augmentation as a method for extending the size of a given dataset. The idea is to modify the training examples in such a way that keeps their semantic properties intact. That’s not an easy task when dealing with human faces.</p><p>The method should account for such complex transformations of data as pose, lighting and non-rigid deformations, yet create realistic samples that follow the real-world data statistics.</p><p>So, let’s see how the latest state-of-the-art methods approach this challenging task…</p><h3>Previous approaches</h3><p>Generative adversarial networks (GANs) have demonstrated their effectiveness in making synthetic data more realistic. Taking the simulated data as input, <a href="https://arxiv.org/pdf/1804.03675.pdf">GAN produces samples that appear more realistic</a>. However, the semantic properties of these samples might be altered, even with a loss penalizing the change in the parameters of the output.</p><p><a href="https://cseweb.ucsd.edu/~ravir/6998/papers/p187-blanz.pdf">3D morphable model (3DMM)</a> is the most commonly used method for representation and synthesis of geometries and textures, and it was originally proposed in the context of 3D human faces. By this model, the geometric structure and the texture of human faces are linearly approximated as a combination of principal vectors.</p><p>Recently, the <a href="https://arxiv.org/pdf/1703.10131.pdf">3DMM model was combined with the convolutional neural networks</a> for data augmentation. However, the generated samples tend to be smooth and unrealistic in appearance as you can observe in the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nmSuhaaZliyWvdH2hjCIgg.png" /><figcaption>Faces synthesized using the 3DMM linear model</figcaption></figure><p>Moreover, 3DMM generates samples following a Gaussian distribution, which rarely reflects the true distribution of the data. For instance, see below the first two PCA coefficients plotted for real faces vs the synthesized 3DMM faces. This gap between the real and synthesized distributions may easily result in non-plausible samples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/489/1*lovnUBhW_DgQbc418vPiIw.png" /><figcaption>First two PCA coefficients of real (left) and 3DMM generated (right) faces</figcaption></figure><h3>State-of-the-art idea</h3><p>Slossberg, Shamai, and Kimmel from Technion — Israel Institute of Technology <a href="https://arxiv.org/pdf/1808.08281.pdf">propose a new realistic data synthesis approach for human faces</a> by combining GAN and 3DMM model.</p><p>In particular, the researchers employ a GAN to imitate the space of parametrized human textures and generate corresponding facial geometries by learning the best 3DMM coefficients for each texture. The generated textures are mapped back onto the corresponding geometries to obtain new generated high-resolution 3D faces.</p><p>This approach produces realistic samples, and it:</p><ul><li>doesn’t suffer from indirect control over such desired attributes as pose and lighting;</li><li>is not limited to producing new instances of existing individuals.</li></ul><p>Let’s have a closer look at their data processing pipeline…</p><h3>Data processing pipeline</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rPgF9GJeES9B8WZ4YwRTeg.png" /><figcaption>Data preparation</figcaption></figure><p>The process includes aligning 3D scans of human faces vertex to vertex and mapping their textures onto a 2D plane using a predefined universal transformation.</p><p>The data preparation pipeline contains four main stages:</p><ul><li><strong>Data acquisition</strong>: the researchers collected about 5000 scans from a wide variety of ethnic, gender, and age groups; each subject was asked to perform five distinct expressions including a neutral one.</li><li><strong>Landmark annotation</strong>: 43 landmarks were added to the meshes semi-automatically by rendering the face and using a pre-trained facial landmark detector on the 2D images.</li><li><strong>Mesh alignment</strong>: this was conducted by deforming a template face mesh according to the geometric structure of each scan, guided by the previously obtained facial landmark points.</li><li><strong>Texture transfer</strong>: the texture is transferred from the scan to the template using a ray casting technique built into the animation rendering toolbox of Blender; then, the texture is mapped from the template to a 2D plane using the predefined universal mapping.</li></ul><p>See the resulting mapped textures below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jc1Mpak6hNOzctrzmWcy5Q.png" /><figcaption>Flattened aligned facial textures</figcaption></figure><p>The next step is to train GAN to learn and imitate these aligned facial textures. For this purpose, the researchers use a <a href="https://arxiv.org/pdf/1710.10196.pdf">progressive growing GAN</a> with the generator and discriminator constructed as symmetric networks. In this implementation, the generator progressively increases the resolution of the feature maps until reaching the output image size, while the discriminator gradually reduces the size back to a single output.</p><p>See below the new synthetic facial textures generated by the aforementioned GAN:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vQyfVuz0PfSFo87DKV4yBg.png" /><figcaption>Facial textures synthesized by GAN</figcaption></figure><p>The final step is to synthesize the geometries of the faces. The researchers explored several approaches to finding plausible geometry coefficients for a given texture. You can observe the qualitative and quantitative (<em>L2</em> geometric error) comparison between the various methods in the next figure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-FfJa3VxIsugZQo-rbbqnw.png" /><figcaption>Two synthesized textures mapped onto different geometries</figcaption></figure><p>Apparently, the least squares approach produces the lowest distortion results. Considering also its simplicity, this method was chosen for all the subsequent experiments.</p><h3>Experimental results</h3><p>The proposed method can generate many new identities, and each one of them can be rendered under varying pose, expression, and lighting. Different expressions are added to the neutral geometry using the <a href="https://liris.cnrs.fr/Documents/Liris-6534.pdf">Blend Shapes model</a>. The resulting images with different pose and lighting are shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/426/1*0STFF905rjjuTNyLUQCg7w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/817/1*7ONy_eMD1nOb-jl-DAcEjQ.png" /></figure><p>For quantitative evaluation of the results, the researchers used the sliced Wasserstein distance (SWD) to measure distances between distributions of their training and generated images in different scales:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*66Qq38ymDC-wOxlUnCduBQ.png" /></figure><p>The table demonstrates that the textures generated by the proposed model are statistically closer to the real data than those generated by 3DMM.</p><p>The next experiment was designed to evaluate if the proposed model is capable of generating samples that diverge significantly from the original training set and resemble previously unseen data. Thus, 5% of the identities were held out for evaluation. The researchers measured the <em>L2</em> distance between each real identity from the test set to the closest identity generated by the GAN, as well as to the closest real identity from the training set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*WytUSYxkhp-ATkjIsmPpIw.png" /><figcaption>The distance between generated and real identities</figcaption></figure><p>As it can be seen from the figure, the test set identities are closer to the generated identities than the training set identities. Moreover, the “Test to fake” distances are not significantly larger than the “Fake to real” distances. That implies that generated samples do not just produce IDs that are very close to the training set, but also novel IDs that resemble previously unseen examples.</p><p>Finally, a qualitative evaluation was performed to check if the proposed pipeline can generate original data samples. Thus, facial textures generated by the model were compared to their closest real neighbors regarding L2 norm between identity descriptors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WDNeJ1SZfmDXEDRI3tyg8A.png" /><figcaption>Synthesized facial textures (top) vs. corresponding closest real neighbors (bottom)</figcaption></figure><p>As you can see, the nearest real textures are far enough to be visually distinguished as different people, which confirms the model’s ability to produce novel identities.</p><h3>Bottom Line</h3><p>The suggested model is probably the first to synthesize both the texture and geometry of human faces realistically. It can be useful for training face detection, face recognition or face reconstruction models. Also, it can be applied in cases where many different realistic faces are needed like for instance, film industry or computer games. Moreover, this framework is not limited to synthesizing human faces but can be employed to other classes of objects where alignment of the data is possible.</p><p><em>Originally published at </em><a href="https://neurohive.io/en/state-of-the-art/facial-surface-and-texture-synthesis-via-gan/"><em>neurohive.io</em></a><em> on September 3, 2018.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3334b2c47d10" width="1" height="1" alt=""><hr><p><a href="https://medium.com/neurohive-computer-vision/facial-surface-and-texture-synthesis-via-gan-3334b2c47d10">Facial Surface and Texture Synthesis via GAN</a> was originally published in <a href="https://medium.com/neurohive-computer-vision">Neurohive - CV papers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vid2Vid — Conditional GANs for Video-to-Video Synthesis]]></title>
            <link>https://medium.com/good-audience/vid2vid-conditional-gans-for-video-to-video-synthesis-7be64f68f54?source=rss-c5715faa7cc7------2</link>
            <guid isPermaLink="false">https://medium.com/p/7be64f68f54</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[vid2vid]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[NeuroHive]]></dc:creator>
            <pubDate>Thu, 30 Aug 2018 16:56:50 GMT</pubDate>
            <atom:updated>2018-09-01T20:23:55.875Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KGX9yo52M3eds0HoLyAfbw.jpeg" /></figure><p>Researchers from NVIDIA and MIT’s Computer Science and Artificial Intelligence Lab have proposed a novel method for <strong>video-to-video synthesis</strong>, showing impressive results. <a href="https://tcwang0509.github.io/vid2vid/paper_vid2vid.pdf">The proposed method</a> — <strong>Vid2Vid</strong> — is able to synthesize high-resolution, photorealistic, temporally coherent videos on a diverse set of input formats including segmentation masks, sketches, and poses.</p><h3>Previous works</h3><p>Due to the inherent complexity of the problem of video-to-video synthesis, this topic remained relatively unexplored in the past. Compared to its image counterpart — image-to-image synthesis, much fewer studies have explored and tackled this problem.</p><p>Arguing that a general-purpose solution to video-to-video synthesis has not yet been explored in the prior work (as opposed to the image counterpart — image-to-image translation), the researchers compare and benchmark this approach with a strong baseline that combines a state-of-the-art video style transfer algorithm with a state-of-the-art image-to-image translation approach.</p><h3>State-of-the-art idea</h3><p>The general idea is to learn a mapping function that can convert an input video to a realistic output video. They frame this problem as a distribution matching problem, and they leverage the generative adversarial learning framework to produce a method that can generate photorealistic videos given an input video (such as a sequence of segmentation masks, poses, sketches, etc.).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*tIIZJdOwWmzjqdJ6.gif" /></figure><h3>Vid2Vid Method</h3><p>As I mentioned before, the authors proposed a method based on Generative Adversarial Networks. They tackle the complex problem of video-to-video translation or video-to-video synthesis in a really impressive way by carefully designing an adversarial learning framework.</p><p>Their goal is to learn a mapping function that will map a sequence of input (source) images to a series of realistic output images, where the conditional distribution of the generated sequence given the source sequence is identical to the distribution of the original sequence given the source sequence:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/1*9Ofk19PNItpGSr9JzJ-Ghw.png" /></figure><p>The matching of the distributions enforces the method to learn to output realistic and temporally coherent output videos. In a generative adversarial learning context, a Generator-Discriminator framework is designed to learn the mapping function. The generator is trained by solving an optimization problem — minimizing the Jensen-Shannon divergence between the two distributions. A minimax optimization is applied to a defined objective function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X5pW_9VEff1ym9ds5mLO4w.png" /></figure><p>As mentioned in their paper, also widely known, optimizing for an objective function as the given one is a very challenging task. Often, the training of a generator and discriminator models becomes very unstable or even impossible depending on the optimization problem being solved. Therefore, in their research, they propose a simplified sequential generator making a few assumptions. They make the Markov property assumption to factorize the conditional distribution and decouple the dependencies across frames in the sequences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*MrqZCOOTk72DM3khVys37g.png" /></figure><p>In simple words, they simplify the problem by assuming that the video frames can be generated sequentially and the generation of the t-th frame only depends on three things: 1) the current source image, 2) the past L source images, and 3) the past L generated images.</p><p>From this point, they design a feed-forward network F to learn a mapping from the past L source images, and the past L-1 generated images to a newly generated output image.</p><p>To model such a network, researchers make another assumption based on the fact that if the optical flow from the current frame to the next frame is known, it can be used to warp the current frame to generate an estimation of the next frame. Arguing that this will be largely true except for occluded areas (where it is unknown what is happening), they are considering a specific model.</p><p>The estimation network F is modeled in such a way to take into account a given occlusion mask, the estimated optical flow between the previous and the current image (which is given by an optical flow estimation function) and a hallucinated image (generated from scratch). The hallucinated image is necessary to fill the areas under occlusions.</p><p>Similarly as in images, wherein the context of generative adversarial learning many methods exploit local discriminators besides a global one, here the authors propose an interesting approach utilizing conditional image discriminator and conditional video discriminator.</p><p>Additionally, to make their method even better and more impressive, the researchers extend their approach to multimodal synthesis. They propose a generative method based on feature embedding scheme and using a Gaussian Mixture Models, to output several videos with different visual appearances depending on sampling different feature vectors.</p><h3>Results</h3><p>The proposed method yields impressive results. It was tested on several datasets such as Cityscapes, Apolloscape, Face video dataset, Dance video dataset. Moreover, two strong baselines were used for comparison: pix2pix method and modified video style transfer method (CONVST) where they changed the stylization network to pix2pix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*P-POiwh7uYjQgYbY.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*Po1Tqy89bxfH3JUR.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*MQQ5PhKH6Yq6AW1D.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*HLCZq55KjGmMIHCR.gif" /></figure><h3>Comparison with other state-of-the-art</h3><p>They use both subjective and objective evaluation metrics for performance evaluation: Human preference score, Fréchet Inception Distance (FID). A comparison between the proposed and other methods is given in the tables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BGk7RRIqKYZ8xQViupPBMA.png" /><figcaption>Future video prediction results. Top left: ground truth. Top right: PredNet. Bottom left: MCNet. Bottom right: proposed</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qW5DSf0ZIc24R-NVGnLEXg.png" /><figcaption>Example output from the multimodal video synthesis.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oIywxkjSBllCSwK7QqCkZg.png" /></figure><h3>Conclusion</h3><p>A new state-of-the-art method in video synthesis has been proposed. The conditional GAN-based approach shows impressive results in several different tasks within the scope of video-to-video translation. There are numerous applications of this kind of methods in computer vision, robotics, and computer graphics. Using a learned video synthesis model, one can generate realistic videos for many different purposes and applications.</p><p><em>Originally published at </em><a href="https://neurohive.io/en/state-of-the-art/vid2vid-conditional-gans-for-video-to-video-synthesis/"><em>neurohive.io</em></a><em> on August 30, 2018.</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2F7b0181%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2F7b0181%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/748a06e9019942f2677bba917ef55804/href">https://medium.com/media/748a06e9019942f2677bba917ef55804/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7be64f68f54" width="1" height="1" alt=""><hr><p><a href="https://medium.com/good-audience/vid2vid-conditional-gans-for-video-to-video-synthesis-7be64f68f54">Vid2Vid — Conditional GANs for Video-to-Video Synthesis</a> was originally published in <a href="https://medium.com/good-audience">Good Audience</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>