<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Neurohive - CV papers - Medium]]></title>
        <description><![CDATA[Machine Learning and Neural Networks - Medium]]></description>
        <link>https://medium.com/neurohive-computer-vision?source=rss----d720a0e44949---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Neurohive - CV papers - Medium</title>
            <link>https://medium.com/neurohive-computer-vision?source=rss----d720a0e44949---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 03:04:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/neurohive-computer-vision" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[PIFR: Pose Invariant 3D Face Reconstruction]]></title>
            <link>https://medium.com/neurohive-computer-vision/pifr-pose-invariant-3d-face-reconstruction-b2228daaef42?source=rss----d720a0e44949---4</link>
            <guid isPermaLink="false">https://medium.com/p/b2228daaef42</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[3d-modeling]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Kate Koidan]]></dc:creator>
            <pubDate>Tue, 25 Dec 2018 15:58:37 GMT</pubDate>
            <atom:updated>2018-12-25T15:58:37.340Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*jbjY0l_UNZFGzfitUH-Njg.png" /></figure><p>3D face geometry needs to be recovered from 2D images in many real-world applications, including face recognition, face landmark detection, 3D emoticon animation, etc. However, this task remains challenging especially under the large pose, when much of the information about the face is unknowable.</p><p>Jiang and Wu from Jiangnan University (China) and Kittler from University of Surrey (UK) <a href="https://arxiv.org/abs/1811.05295">suggest a novel 3D face reconstruction algorithm</a>, which significantly improves the accuracy of reconstruction even under extreme pose.</p><p>But let’s first shortly review the previous work on 3D face models and 3D face reconstruction.</p><h3>Related Work</h3><p>The research mentions four publicly available 3D deformation models:</p><p>· <a href="https://gravis.dmi.unibas.ch/publications/2009/BFModel09.pdf">BFM model</a> proposed by the University of Basel;</p><p>· <a href="https://arxiv.org/pdf/1209.6491.pdf">3DMM models</a> developed by Brunton et.al;</p><p>· a <a href="http://epubs.surrey.ac.uk/809478/">multi-resolution 3D face model</a> provided by University of Surrey (UK);</p><p>· the <a href="https://link.springer.com/article/10.1007/s11263-017-1009-7">large-scale face model (LSFM)</a> built by Imperial College.</p><p>This paper uses a BFM model, which is the most popular.</p><p>There are several approaches to reconstructing 3D model from 2D images, including:</p><p>· a <a href="https://arxiv.org/abs/1509.06161">cascaded</a> regression method;</p><p>· combining face key point detection and 3D face reconstruction and using 3d-shape-feature-indexing to construct the <a href="http://openaccess.thecvf.com/content_iccv_2015/papers/Tulyakov_Regressing_a_3D_ICCV_2015_paper.pdf">tree-based</a> regression model;</p><p>· 3D morphable model based on <a href="https://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Zhu_High-Fidelity_Pose_and_2015_CVPR_paper.pdf">Pose and Expression normalization method</a>;</p><p>· an extended 3DMM (<a href="https://gravis.dmi.unibas.ch/publications/2008/FG08_Amberg.pdf">E-3DMM</a>), which accounts for the variation of facial expression;</p><p>· performing a <a href="https://arxiv.org/pdf/1808.05399.pdf">weighted landmark 3DMM fitting</a> based on the traditional regression method.</p><h3>State-of-the-art idea</h3><p>The paper by Jiang, Wu, and Kittler proposes a novel Pose-Invariant 3D Face Reconstruction (PIFR) algorithm based on 3D Morphable Model (3DMM).</p><p>Firstly, they suggest generating a <strong>frontal image</strong> by normalizing a single face input image. This step allows restoring additional identity information of the face.</p><p>The next step is to use a <strong>weighted sum</strong> of the 3D parameters of <strong>both images</strong>: the frontal one and the original one. This allows to preserve the pose of the original image, but also enhance the identity information.</p><p>The pipeline for the suggested approach is provided below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*8hxIjUdvGedpbfQX7jPyjg.png" /><figcaption>Overview of the Pose-Invariant 3D Face Reconstruction (PIFR) method</figcaption></figure><p>The experiments show that PIFR algorithm has significantly improved the performance of 3D face reconstruction compared to the previous methods, especially in the extreme pose cases.</p><p>Let’s now have a closer look at the suggested model…</p><h3>Model details</h3><p>PIFR method is largely relying on the 3DMM fitting process, which can be expressed as minimizing the error between the 2D coordinates of the 3D point projection and the ground truth. However, the face generated by the 3D model has about 50,000 vertices, and thus iterative calculations result in the slow and ineffective convergence. To overcome this problem, the researchers suggest using the <strong>landmarks</strong> (e.g., eye center, mouth corner, and nose tip) as the ground truth in the fitting process. Specifically, they use a weighted landmark 3DMM fitting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/1*_Z9uvmNRlqINr2HH90m5sA.png" /><figcaption>Top row: the original image and landmark. Bottom row: 3D face model and its alignment to the 2D image</figcaption></figure><p>The next challenge is to reconstruct 3D faces in large poses. To solve this problem, the researchers use High-Fidelity Pose and Expression Normalization (<a href="https://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Zhu_High-Fidelity_Pose_and_2015_CVPR_paper.pdf">HPEN</a>) method, but only for normalization of the pose and not expression. Also, <a href="http://leyvand.com/adv-graphics/pie2003.pdf">Poisson Editing</a> is used to recover the occluded area of the face due to the angle.</p><h3>Performance Comparison with Other Methods</h3><p>The performance of PIFR method was evaluated for the face reconstruction:</p><p>· in small and medium poses;</p><p>· large poses;</p><p>· extreme poses (±90 yaw angles).</p><p>For this purpose, the researchers used three publicly available datasets:</p><p>· <strong>AFW</strong> dataset, which was created using Flickr images, contains 205 images with 468 marked faces, complex backgrounds and face pose.</p><p>· <strong>LFPW</strong> dataset, which has 224 face images in the test set and 811 face images in the training set; each image is marked with 68 feature points; 900 face images from both sets were selected for testing in this research.</p><p>· <strong>AFLW</strong> dataset is a large-scale face database, which contains around 250 million hand-labeled face images, and each image is marked with 21 feature points. This study used only extreme pose face images from this dataset for qualitative analysis.</p><p><strong>Quantitative analysis.</strong> Using the Mean Euclidean Metric (MEM), the study compares the performance of PIFR method to <a href="https://gravis.dmi.unibas.ch/publications/2008/FG08_Amberg.pdf">E-3DMM</a> and <a href="https://arxiv.org/pdf/1808.05399.pdf">FW-3DMM</a> on AFW and LFPW datasets. Cumulative errors distribution (CED) curves look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0jLw4TbKuKNY5D5YROGhUg.png" /><figcaption>Comparisons of cumulative errors distribution (CED) curves on AFW and LFPW datasets</figcaption></figure><p>As you can see from these plots and the tables below, PIFR method shows superior performance compared to the other two methods. Its reconstruction performance in large poses is particularly good.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*mwejH4tcdgzmN216Yi2pyw.png" /></figure><p><strong>Qualitative analysis.</strong> The method was also assessed qualitatively based on the face images in extreme poses from AFLW dataset. The results are shown in the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*y9dkQhxqLi8kpn6havwdJw.png" /><figcaption>Comparison of 3D face reconstruction: (a) Input image; (b) FW-3DMM; © E-3DMM; (d) Suggested approach</figcaption></figure><p>Even though half of the landmarks are invisible due to extreme posture, which leads to large errors and failures of other methods, the PIFR method still performs quite well.</p><p>Here are some additional examples of the PIFR method performance based on the images from the AFW dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*6Xx74w925Q4TyHWlD2INuQ.png" /><figcaption>Top row: Input 2D image. Middle row: 3D face. Bottom row: Align to a 2D image.</figcaption></figure><h3>Bottom Line</h3><p>A novel 3D face reconstruction framework PIFR demonstrates good reconstruction performance even in extreme poses. By taking both the original and the frontal images for weighted fusion, the method allows restoring enough face information to reconstruct the 3D face.</p><p>In the future, the researchers plan to restore even more facial identity information to improve the accuracy of reconstruction further.</p><p>Originally published at: <a href="https://neurohive.io/en/state-of-the-art/pifr-pose-invariant-3d-face-reconstruction/">https://neurohive.io/en/state-of-the-art/pifr-pose-invariant-3d-face-reconstruction/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2228daaef42" width="1" height="1" alt=""><hr><p><a href="https://medium.com/neurohive-computer-vision/pifr-pose-invariant-3d-face-reconstruction-b2228daaef42">PIFR: Pose Invariant 3D Face Reconstruction</a> was originally published in <a href="https://medium.com/neurohive-computer-vision">Neurohive - CV papers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[New Datasets for 3D Object Recognition]]></title>
            <link>https://medium.com/neurohive-computer-vision/new-datasets-for-3d-object-recognition-30039fa34116?source=rss----d720a0e44949---4</link>
            <guid isPermaLink="false">https://medium.com/p/30039fa34116</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[3d]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[dataset]]></category>
            <dc:creator><![CDATA[Kate Koidan]]></dc:creator>
            <pubDate>Mon, 19 Nov 2018 10:48:57 GMT</pubDate>
            <atom:updated>2018-11-19T10:48:56.675Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*9oX-gOF7aWga1I1onrlT6w.png" /></figure><p>Robotics, augmented reality, autonomous driving — all these scenarios rely on recognizing 3D properties of objects from 2D images. This puts 3D object recognition as one of the central problems in computer vision.</p><p>Remarkable progress has been achieved in this field after the introduction of several databases that provide 3D annotations to 2D objects (e.g., IKEA, Pascal3D+). However, these datasets are limited in scale and <a href="http://cvgl.stanford.edu/papers/xiang_eccv16.pdf">include only about a dozen object categories</a>.</p><p>This is not even close to the large-scale image datasets such as ImageNet or Microsoft COCO, while these are these huge datasets that stay behind the significant progress in image classification task in recent years. Consequently, large-scale datasets with 3D annotations are likely to significantly benefit 3D object recognition.</p><p>In this article, we present one large-scale dataset, <strong>ObjectNet3D</strong>, and also several specialized datasets for 3D object recognition: <strong>MVTec ITODD</strong> and <strong>T-LESS</strong> — for industry settings and <strong>Falling Things</strong> dataset — for object recognition tasks in the context of robotics.</p><h3>ObjectNet3D</h3><p><strong>Number of images:</strong> 90,127</p><p><strong>Number of objects:</strong> 201,888</p><p><strong>Number of categories:</strong> 100</p><p><strong>Number of 3D shapes: </strong>44,147</p><p><strong>Year:</strong> 2016</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*nqofII4HIkDGL3_k5i1gWw.png" /><figcaption>An example image from ObjectNet3D with 2D objects aligned with 3D shapes</figcaption></figure><p><a href="http://cvgl.stanford.edu/projects/objectnet3d/">ObjectNet3D</a> is a large-scale database, where objects in the images are aligned with the 3D shapes, and the alignment provides both accurate 3D pose annotation and the closest 3D shape annotation for each 2D object. The scale of this dataset allows for significant progress with such computer vision tasks as recognizing 3D pose and 3D shape of objects from 2D images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*DBRX9VFss1o8AuFvAZBGxQ.png" /><figcaption>Examples of 3D shape retrieval. Green boxes indicate the selected shape. Bottom row illustrates two cases where a similar shape was not found among the top 5 shapes</figcaption></figure><p>To construct this database, <a href="http://cvgl.stanford.edu/papers/xiang_eccv16.pdf">researchers from Stanford University</a> resort to images from existing image repositories and propose an approach to align 3D shapes (available from existing 3D shapes repositories) to the objects in these images.</p><p>In their work, the researchers consider only rigid object categories, for which they can collect a large number of 3D shapes from the web. Here is the full list of categories:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*xwrGDHI-mP_r_bcgv1TTVg.png" /><figcaption>Object categories in ObjectNet3D</figcaption></figure><p>2D images were collected from the <a href="http://www.image-net.org/">ImageNet</a> dataset and additionally, through Google Image Search for categories that are not sufficiently covered by the ImageNet dataset. 3D shapes were acquired from <a href="https://3dwarehouse.sketchup.com/">Trimble 3D Warehouse</a> and <a href="https://www.shapenet.org/">ShapeNet</a> repository. Then, objects in the image were aligned with the 3D shapes using a camera model, which is described in detail in the corresponding <a href="http://cvgl.stanford.edu/papers/xiang_eccv16.pdf">paper</a>. Finally, 3D annotations were provided to objects in 2D images.</p><p>The resulting dataset can be used for object proposal generation, 2D object detection, joint 2D detection and 3D object pose estimation, and image-based 3D shape retrieval.</p><h3>MVTec ITODD</h3><p><strong>Number of scenes:</strong> 800</p><p><strong>Number of objects:</strong> 28</p><p><strong>Number of 3D transformations:</strong> 3500</p><p><strong>Year:</strong> 2017</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*k8OuR2ESkfmlSdpmIZVUog.png" /><figcaption>Example scene of the dataset from all sensors. Top row: grayscale cameras. Bottom row: Z and grayscale image of the High-Quality (left) and Low-Quality (right) 3D sensor</figcaption></figure><p><a href="https://www.mvtec.com/company/research/datasets/mvtec-itodd/">MVTec ITODD</a> is a dataset for 3D object detection and pose estimation with a strong focus on industrial settings and applications. It contains 28 objects arranged in over 800 scenes and labeled with their rigid 3D transformation as ground truth. The scenes are observed by two industrial 3D sensors and three grayscale cameras, allowing to evaluate methods that work on 3D, image, or combined modalities. The dataset’s <a href="https://www.mvtec.com/fileadmin/Redaktion/mvtec.com/company/research/mvtec_itodd.pdf">creators from MVTec Software GmbH</a> have chosen to use grayscale cameras because they are much more prominent in industrial setups.</p><p>As mentioned in the <a href="https://www.mvtec.com/fileadmin/Redaktion/mvtec.com/company/research/mvtec_itodd.pdf">dataset description</a>, the objects were selected such that they cover a range of different values with respect to surface reflectance, symmetry, complexity, flatness, detail, compactness, and size. Here are the images of all objects included to the MVTec ITODD along with their names:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*hBrGVxa6Zg17bLptZFoybQ.png" /><figcaption>Images of 28 objects used in the dataset</figcaption></figure><p>For each object, scenes with only a single instance and scenes with multiple instances (e.g., to simulate bin picking) are available. Each scene was acquired once with each of the 3D sensors, and twice with each of the grayscale cameras: once with and once without a random projected pattern.</p><p>Finally, for all objects, manually created CAD models are available for training the detection methods. The ground truth was labeled using a semi-manual approach based on the 3D data of the high-quality 3D sensor.</p><p>This dataset provides a great benchmark for the detection and pose estimation of 3D objects in the industrial scenarios.</p><h3>T-LESS</h3><p><strong>Number of images: </strong>39K training + 10K test images from each of three sensors</p><p><strong>Number of objects:</strong> 30</p><p><strong>Year:</strong> 2017</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*em4S1kSEQfE1kzoGYB1qHQ.png" /><figcaption>Data Examples of T-LESS test images (left) overlaid with colored 3D object models at the ground truth 6D poses (right). Instances of the same object have the same color</figcaption></figure><p><a href="http://cmp.felk.cvut.cz/t-less/">T-LESS</a> is a new public dataset for estimating the 6D pose, i.e. translation and rotation, of texture-less rigid objects. This dataset includes 30 industry-relevant objects with no significant texture and no discriminative color or reflectance properties. Another unique property of this dataset is that some of the objects are parts of others.</p><p><a href="https://arxiv.org/pdf/1701.05498.pdf">Researchers behind T-LESS</a> have chosen different approaches to the training images and test images. Thus, training images in this dataset depict individual objects against a black background, while test images originated from twenty scenes with varying degree of complexity. Here are the examples of training and test images:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*lGdVV75MhX3BmqcPPg3RcQ.png" /><figcaption>Top: training images and 3D models of 30 objects. Bottom: test images of 20 scenes overlaid with colored 3D object models at the ground truth poses</figcaption></figure><p>All the training and test images were captured with three synchronized sensors, including a structured-light and a time-of-flight RGB-D sensor and a high-resolution RGB camera.</p><p>Finally, two types of 3D models are provided for each object: 1) manually created CAD model, and 2) a semi-automatically reconstructed one.</p><p>This dataset can be very useful for evaluating approaches to 6D object pose estimation, 2D object detection and segmentation, 3D object reconstruction. Considering the availability of images from three sensors, it is also possible to study the importance of different input modalities for a given problem.</p><h3>Falling Things</h3><p><strong>Number of images:</strong> 61, 500</p><p><strong>Number of objects:</strong> 21 household objects</p><p><strong>Year:</strong> 2018</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/1*sm99bUMZwjafUcBfCaa1-Q.png" /><figcaption>A sample from FAT dataset</figcaption></figure><p>The <a href="https://research.nvidia.com/publication/2018-06_Falling-Things">Falling Things (FAT)</a> dataset is a synthetic dataset for 3D object detection and pose estimation, created by <a href="https://arxiv.org/pdf/1804.06534.pdf">NVIDIA team</a>. It was generated by placing 3D household object models (e.g., mustard bottle, soup can, gelatin box, etc.) in virtual environments.</p><p>Each snapshot in this dataset consists of per-pixel class segmentation, 2D/3D bounding box coordinates for all objects, mono and stereo RGB images, dense depth images, and of course, 3D poses. Most of these elements are illustrated in the above image.</p><p>The FAT dataset includes the variety of object poses, backgrounds, composition, and lighting conditions. See some examples below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*tEEgOeRZrujkrHcHsxOfrQ.png" /><figcaption>Sample images from the FAT dataset</figcaption></figure><p>For more details on the process of building the FAT dataset, check <a href="https://neurohive.io/en/datasets/falling-things-a-synthetic-dataset-by-nvidia-for-pose-estimation/">our article</a> dedicated entirely to this dataset.</p><p>The Falling Things dataset provides a great opportunity to accelerate research in object detection and pose estimation, as well as segmentation, depth estimation, and sensor modalities.</p><h3>Bottom Line</h3><p>3D object recognition has multiple important applications, but progress in this field is limited by the available datasets. Fortunately, there were several new 3D object recognition datasets introduced in recent years. While they have different scale, focus and characteristics, each of these datasets makes a significant contribution to the improvement of current 3D object recognition systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30039fa34116" width="1" height="1" alt=""><hr><p><a href="https://medium.com/neurohive-computer-vision/new-datasets-for-3d-object-recognition-30039fa34116">New Datasets for 3D Object Recognition</a> was originally published in <a href="https://medium.com/neurohive-computer-vision">Neurohive - CV papers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[New Datasets for 3D Human Pose Estimation]]></title>
            <link>https://medium.com/neurohive-computer-vision/new-datasets-for-3d-human-pose-estimation-45cd320e37bd?source=rss----d720a0e44949---4</link>
            <guid isPermaLink="false">https://medium.com/p/45cd320e37bd</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[dataset]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[pose-estimation]]></category>
            <dc:creator><![CDATA[Kate Koidan]]></dc:creator>
            <pubDate>Mon, 19 Nov 2018 10:48:10 GMT</pubDate>
            <atom:updated>2018-11-19T10:48:10.134Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*jh7Nig9I3O4pmeP9fv131g.png" /></figure><p>Human pose estimation is another fundamental problem in computer vision. Computer’s ability to recognize and understand humans in images and videos is crucial for multiple tasks including autonomous driving, action recognition, human-computer interaction, augmented reality and robotics vision.</p><p>In recent years, significant progress has been achieved in 2D human pose estimation. The crucial factor behind this success is the availability of large-scale annotated human pose datasets that allow training networks for 2D human pose estimation. At the same time, advances in 3D human pose estimation remain limited because obtaining ground-truth information on the dense correspondence, depth, motion, body-part segmentation, occlusions is a very challenging task.</p><p>In this article, we present several recently created datasets that attempt to address the shortage of annotated datasets for 3D human pose estimation.</p><h3>DensePose</h3><p><strong>Number of images:</strong> 50K</p><p><strong>Number of annotated correspondences:</strong> 5M</p><p><strong>Year:</strong> 2018</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*p5FWOfhiuUp5k3mmlAI8VA.png" /></figure><p><a href="http://densepose.org/">DensePose</a> is a large-scale ground-truth dataset with image-to-surface correspondences manually annotated on 50K COCO images. To build this dataset Facebook AI Research team involved human annotators, who were establishing dense correspondences from 2D images to surface-based representations of the human body using a specifically developed annotation pipeline.</p><p>As shown below, in the first stage annotators define regions corresponding to visible, semantically defined body parts. In the second stage, every part region is sampled with a set of roughly equidistant points and annotators are requested to bring these points in correspondence with the surface. The researchers wanted to avoid manual rotation of the surface and for this purpose, they provide annotators with six pre-rendered views of the same body part and allow users to place landmarks on any of them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*m74lyVU5csBb_v0b6d1olQ.png" /><figcaption>Annotation pipeline</figcaption></figure><p>Below are visualizations of annotations on images from the validation set: Image (left), U (middle) and V (right) values for the collected points.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*qiQvbliN8RM4eyKVVuE8iA.png" /><figcaption>Visualizing of annotations</figcaption></figure><p>DensePose is <a href="https://arxiv.org/pdf/1802.00434.pdf">the first manually-collected ground truth dataset</a> for the task of dense human pose estimation.</p><h3>SURREAL</h3><p><strong>Number of frames:</strong> 6.5M</p><p><strong>Number of subjects:</strong> 145</p><p><strong>Year:</strong> 2017</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/658/1*_0Opa0tI7amU8M-x6K069Q.png" /><figcaption>Generating photorealistic synthetic images</figcaption></figure><p><a href="https://www.di.ens.fr/willow/research/surreal/">SURREAL</a> (Synthetic hUmans foR REAL tasks) is a new large-scale dataset with synthetically-generated but realistic images of people rendered from 3D sequences of human motion capture data. It includes over 6 million frames accompanied with the ground-truth pose, depth maps, and segmentation masks.</p><p>As described in the original <a href="https://arxiv.org/pdf/1701.01370.pdf">research paper</a>, images in SURREAL are rendered from 3D sequences of MoCap data. The realism of synthetic data is usually limited. So, to ensure the realism of human bodies in this dataset, the researchers decided to create synthetic bodies using <a href="http://files.is.tue.mpg.de/black/papers/SMPL2015_fixed.pdf">SMPL body model</a>, whose parameters are fit by the <a href="http://files.is.tue.mpg.de/black/papers/MoSh.pdf">MoSh method</a> given raw 3D MoCap marker data. Moreover, creators of SURREAL dataset ensured a large variety of viewpoints, clothing, and lighting.</p><p>A pipeline for generating synthetic human is demonstrated below:</p><p>· a 3D human body model is posed using motion capture data;</p><p>· a frame is rendered using a background image, a texture map on the body, lighting and a camera position;</p><p>· all the “ingredients” are randomly sampled to increase the diversity of the data;</p><p>· generated RGB images are accompanied with 2D/3D poses, surface normal, optical flow, depth images, and body-part segmentation maps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*RKiPc0NKVBOYvYqMm7G2gw.png" /><figcaption>Pipeline for generating synthetic data</figcaption></figure><p>The resulting dataset contains 145 subjects, more than 67.5K clips and over 6.5M frames:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*ZQu0CQQpazL-NktmNeD2qA.png" /></figure><p>Even though SURREAL contains synthetic images, the researchers behind this dataset demonstrate that <a href="https://arxiv.org/pdf/1701.01370.pdf">CNNs trained on SURREAL allow for accurate human depth estimation</a> and human part segmentation in real RGB images. Hence, this dataset provides new possibilities for advancing 3D human pose estimation using cheap and large-scale synthetic data.</p><h3>UP-3D</h3><p><strong>Number of subjects: </strong>5,569</p><p><strong>Number of images: </strong>5,569 training images and 1208 test images</p><p><strong>Year:</strong> 2017</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*V57MChJQT_su8bR1zLl-vw.png" /><figcaption>Bottom: Validated 3D body model fits on various datasets constitute the initial UP-3D dataset. Top: improved 3D fits can extend the initial dataset</figcaption></figure><p><a href="http://files.is.tuebingen.mpg.de/classner/up/">UP-3D</a> is a dataset, which “Unites the People” of different datasets for multiple tasks. In particular, using the recently introduced <a href="https://arxiv.org/pdf/1607.08128.pdf">SMPLify method</a> the researchers obtain high-quality 3D body model fits for several human pose datasets. Human annotators only sort good and bad fits.</p><p><a href="https://arxiv.org/pdf/1701.02468.pdf">This dataset</a> combines two LSP datasets (11,000 training images and 1,000 test images) and the single-person part of the MPII-HumanPose dataset (13,030 training images and 2622 test images). While it was possible to use an automatic segmentation method to provide foreground silhouettes, the researchers decided to involve human annotators for reliability. They have built an interactive annotation tool on top of the <em>Opensurfaces</em> package to work with Amazon Mechanical Turk (AMT) and have been using the interactive Grabcut algorithm to obtain image consistent silhouette borders.</p><p>So, the annotators were asked to evaluate fit for:</p><p>· foreground silhouettes;</p><p>· six body part segmentation.</p><p>While the average foreground labeling task was solved in 108s on the LSP and 168s on the MPII datasets respectively, annotating the segmentation for six body parts took on average more than twice as long as annotating foreground segmentation: 236s.</p><p>The annotators were sorting good and bad fits and here are the percentages of accepted fits per dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*-kfJyRqhan9GsIOzvNk2Yg.png" /></figure><p>Thus, the validated fits formed the initial UP-3D dataset with 5,569 training images and 1,208 test images. After the experiments on semantic body part segmentation, pose estimation and 3D fitting, the improved 3D fits can extend the initial dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*1TadaAnH7hIGYlyk8e_vCg.png" /><figcaption>Results from various methods trained on labels generated from the UP-3D dataset</figcaption></figure><p>The presented dataset allows for a holistic view on human-related prediction tasks. It sets a new mark in terms of levels of detail by including high-fidelity semantic body part segmentation in 31 parts and 91 landmark human pose estimation. It was also demonstrated that training the pose estimator on the full 91 keypoint dataset helps to improve the state-of-the-art for 3D human pose estimation on the two popular benchmark datasets HumanEva and Human3.6M.</p><h3>Bottom Line</h3><p>As you can see, there are many possible approaches to building a dataset for 3D human pose estimation. The datasets presented here focus on different aspects of recognizing and understanding humans in images. However, all of them can be handy for estimating human poses in some of the real-life applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45cd320e37bd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/neurohive-computer-vision/new-datasets-for-3d-human-pose-estimation-45cd320e37bd">New Datasets for 3D Human Pose Estimation</a> was originally published in <a href="https://medium.com/neurohive-computer-vision">Neurohive - CV papers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[New Datasets for Object Tracking]]></title>
            <link>https://medium.com/neurohive-computer-vision/new-datasets-for-object-tracking-d5765bbeb37e?source=rss----d720a0e44949---4</link>
            <guid isPermaLink="false">https://medium.com/p/d5765bbeb37e</guid>
            <category><![CDATA[dataset]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[tracking]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Kate Koidan]]></dc:creator>
            <pubDate>Mon, 19 Nov 2018 10:47:10 GMT</pubDate>
            <atom:updated>2024-07-09T13:32:18.842Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bBiFqDOw6YDzKTz0frsf0w.png" /><figcaption>Samples from the Need for Speed (NfS) dataset</figcaption></figure><p>Object tracking in the wild is far from being solved. Existing object trackers do quite a good job on the established datasets (e.g., VOT, OTB), but these datasets are relatively small and do not fully represent the challenges of real-life tracking tasks. Deep learning is at the core of the most state-of-the-art trackers today. However, a dedicated large-scale dataset to train deep trackers is still lacking.</p><p>In this article, we discuss three recently introduced datasets for object tracking. They differ in scale, annotations and other characteristics but all of them can contribute something to solving object tracking problem: <strong>TrackingNet</strong> is a first large-scale dataset for object tracking in the wild, <strong>MOT17</strong> is a benchmark for multiple object tracking, and <strong>Need for Speed</strong> is the first higher frame rate video dataset.</p><h3>TrackingNet</h3><p><strong>Number of videos:</strong> 30,132 (train) + 511 (test)</p><p><strong>Number of annotations: </strong>14,205,677 (train) + 225,589 (test)</p><p><strong>Year:</strong> 2018</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*VNdFcduvZvmE6ylksC9Ycw.png" /><figcaption>Examples from TrackingNet test set</figcaption></figure><p><a href="https://tracking-net.org/">TrackingNet</a> is a first large-scale dataset for object tracking in the wild. It includes over 30K videos with an average duration of 16.6s and more than 14M dense bounding box annotations. The dataset is not limiting to a specific context but instead <a href="https://arxiv.org/abs/1803.10794">covers a wide selection of object classes</a> in a broad and diverse context. TrackingNet has a number of notable advantages:</p><p>· a large scale of this dataset enables the development of deep design specific for tracking;</p><p>· by being specifically created for object tracking, the dataset enables model architectures to focus on the temporal context between the consecutive frames;</p><p>· the dataset was sampled from YouTube videos and thus, represents real-world scenarios and contains a large variety of frame rates, resolutions, context and object classes.</p><p>TrackingNet training set was derived from the <a href="https://arxiv.org/abs/1702.00824">YouTube-Bounding Boxes (YT-BB)</a>, a large-scale dataset for object detection with roughly 300K video segments, annotated every second with upright bounding boxes. To build TrackingNet, the researchers filtered out 90% of the videos by selecting the videos that a) are longer than 15 seconds; b) include bounding boxes that cover less than 50% of the frame; c) contain a reasonable amount of motion between bounding boxes.</p><p>To increase annotation density from 1fps provided by YT-BB, the creators of TrackingNet rely on a mixture of state-of-the-art trackers. They claim that any tracker is reliable on a short interval of 1 second. So, they have densely annotated 30,132 videos using a weighted average between a forward and a backward pass using the <a href="https://arxiv.org/abs/1404.7584">DCF tracker</a>. Furthermore, the code for automatically downloading videos from YouTube and extracting the annotated frames is also available.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*NtBkzvpLC0M86qc7XKpc3w.png" /><figcaption>Comparison of tracking datasets across the number of videos, the average length of the videos, and the number of annotated bounding boxes (reflected with the circle’s size)</figcaption></figure><p>Finally, TrackingNet dataset comes with a new benchmark composed of 511 novel videos from <em>YouTube with Creative Commons</em> license, namely YT-CC. These videos have the same object class distribution as the training set and are annotated with the help of Amazon Mechanical Turk workers. With the tight supervision in the loop, TrackingNet team has ensured the quality of the annotations after a few iterations, discouraged bad annotators and incentivized the good ones.</p><p>Thus, by sequestering the annotation of the test set and maintaining an online evaluation server, the researchers behind TrackingNet provide a fair benchmark for the development of object trackers.</p><p><strong>MOT17</strong></p><p><strong>Number of videos:</strong> 21 (train) + 21 (test)</p><p><strong>Number of annotations:</strong> 564,228</p><p><strong>Year:</strong> 2017</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*urTV3DTWKcy2Cws-q6ieoQ.png" /><figcaption>Examples from the MOT17 dataset</figcaption></figure><p><a href="https://motchallenge.net/data/MOT17/">MOT17</a> (Multiple Object Tracking) is an extended version of the MOT16 dataset with new and more accurate ground truth. As evident from its name, the specific focus of this dataset is on multi-target tracking. It should be also noted that the context of <em>MOTChallenge</em> datasets, including this last MOT17 dataset, is limited to the street scenes.</p><p>The new MOT17 benchmark includes a set of 42 sequences with crowded scenarios, camera motions, and weather conditions. <a href="https://arxiv.org/abs/1603.00831">The annotations for all sequences have been carried out by qualified researchers</a> from scratch following a strict protocol. Even more, to ensure the highest annotations accuracy, all the annotations were double-checked. Another thing that distinguishes this dataset from the earlier versions of <em>MOTChallenge</em> datasets is that here not only pedestrians are annotated, but also vehicles, sitting people, occluding objects, as well as other significant object classes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*3ovGbkb9LU2VURkcieIj1A.png" /><figcaption>An overview of annotated classes and example of an annotated frame</figcaption></figure><p>The researchers have defined some classes as the target ones — they are depicted with orange in the above image; these classes are the central ones to evaluate on. The red classes include ambiguous cases such that both recovering and missing them will not be penalized in the evaluation. Finally, the classes in green are annotated for training purposes and for computing the occlusion level of all pedestrians.</p><p>An exemplar of an annotated frame demonstrates how partially cropped objects are also marked outside of the frame. Also, note that the bounding box encloses the entire person but not the white bag of the pedestrian.</p><p>Rich ground truth information provided within the MOT17 dataset can be very useful for developing more accurate tracking methods and advancing the field further.</p><h3>NfS</h3><p><strong>Number of videos:</strong> 100</p><p><strong>Number of annotations:</strong> 383,000</p><p><strong>Year:</strong> 2017</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*DeCypglkG6TxVHLmsbIm5w.png" /><figcaption>The effect of tracking higher frame rate videos</figcaption></figure><p><a href="http://ci2cv.net/nfs/index.html">NfS (Need for Speed)</a> is the first higher frame rate video dataset and benchmark for visual object tracking. It <a href="https://arxiv.org/abs/1703.05884">includes</a> 100 videos comprised out of 380K frames and captured with 240 FPS cameras, which are now often used in real-world scenarios.</p><p>Particularly, 75 videos were captured using the iPhone 6 (and above) and the iPad Pro, while 25 videos were taken from YouTube. The tracking targets include vehicles, humans, faces, animals, aircraft, boats and generic objects such as sport balls, cups, bags etc.</p><p>All frames in NfS dataset are annotated with axis-aligned bounding boxes using the <a href="http://www.cs.columbia.edu/~vondrick/vatic/ijcv.pdf">VATIC toolbox</a>. Moreover, all videos are manually labeled with nine visual attributes: occlusion, illumination variation, scale variation, object deformation, fast motion, viewpoint change, out of view, background clutter, and low resolution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/1*62nuZu2MJrOcc7mHOCpyqw.png" /><figcaption>Comparing lower frame rate (green boxes) to higher frame rate (red boxes) tracking. Ground truth is shown by blue boxes</figcaption></figure><p>NfS benchmark provides a great opportunity to evaluate state-of-the-art trackers on higher frame rate sequences. Actually, some <a href="https://arxiv.org/abs/1703.05884">surprising results</a> were already revealed thanks to this dataset: apparently, at higher frame rates, simple trackers such as correlation filters outperform complex deep learning algorithms.</p><h3>Bottom Line</h3><p>The scarcity of the dedicated large-scale tracking datasets leads to the situation when object trackers based on the deep learning algorithms are forced to rely on the object detection datasets instead of the dedicated object tracking ones. Of course, this limits advances in object tracking field. Fortunately, the object tracking datasets introduced recently, especially the large-scale <strong>TrackingNet</strong> dataset, provide data-hungry trackers with the great opportunities for significant performance upgrades.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5765bbeb37e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/neurohive-computer-vision/new-datasets-for-object-tracking-d5765bbeb37e">New Datasets for Object Tracking</a> was originally published in <a href="https://medium.com/neurohive-computer-vision">Neurohive - CV papers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[New Datasets for Action Recognition]]></title>
            <link>https://medium.com/neurohive-computer-vision/new-datasets-for-action-recognition-4610c5101823?source=rss----d720a0e44949---4</link>
            <guid isPermaLink="false">https://medium.com/p/4610c5101823</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[dataset]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Kate Koidan]]></dc:creator>
            <pubDate>Mon, 22 Oct 2018 13:09:09 GMT</pubDate>
            <atom:updated>2018-10-25T13:16:42.208Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LV2NfdsTkLvJC4_IsmfrBw.png" /></figure><p>Action recognition is vital for many real-life applications, including video surveillance, healthcare, and human-computer interaction. What do we need to do to classify video clips based on the actions being performed in these videos?</p><p>Original article: <a href="https://neurohive.io/en/computer-vision/new-datasets-for-action-recognition/">New Datasets for Action Recognition</a></p><p>We need to <a href="http://blog.qure.ai/notes/deep-learning-for-videos-action-recognition-review">identify different actions from video clips where the action may or may not be performed throughout the entire duration of the video</a>. This looks similar to the image classification problem, but in this case, the task is extended to multiple frames with further aggregation of the predictions from each frame. And we know that after the introduction of ImageNet dataset, deep learning algorithms are doing a pretty good job in image classification. But do we observe the same progress in video classification or action recognition tasks?</p><p>Actually, there is a number of things that turn action recognition into a much more challenging task. This includes huge computational cost, capturing long context, and of course, a need for good datasets.</p><p>A good dataset for action recognition problem should have a number of frames comparable to ImageNet and diversity of action classes that will allow for generalization of the trained architecture to many different tasks.</p><p>Fortunately, several such datasets were presented during the last year. Let’s have a look.</p><h3>Kinetics-600</h3><p><strong>Number of videos:</strong> 500,000</p><p><strong>Number of action classes:</strong> 600</p><p><strong>Year:</strong> 2018</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*YoFXoTxalNg3oxn-iFgiiQ.png" /><figcaption>Samples from the Kinetics-600 dataset</figcaption></figure><p>We start with the dataset introduced by Google’s DeepMind team. This is a <a href="https://deepmind.com/research/open-source/open-source-datasets/kinetics/">Kinetics</a> dataset — a large-scale, high-quality dataset of YouTube URLs created to advance models for <strong><em>human action recognition</em></strong>. Its last version is called Kinetics-600 and includes around 500,000 video clips that cover 600 human action classes with at least 600 video clips for each action class.</p><p>Each clip in Kinetics-600 is taken from a unique YouTube video, lasts around 10 seconds and is labeled with a single class. The clips have been through multiple rounds of human annotation. A single-page web application was built for the labeling task, and you can see the labeling interface below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*CiALCWjEaFu_6VlB3Gpxag.png" /><figcaption>Labeling Interface</figcaption></figure><p>If a worker responded with ‘Yes’ to the initial question “Can you see a human performing the action <em>class-name</em>?”, he was also asked the follow-up question “Does the action last for the whole clip?” in order to use this signal later during model training.</p><p>The creators of Kinetics-600 have also checked if the dataset is gender balanced and discovered that <a href="https://arxiv.org/pdf/1705.06950.pdf">approximately 15% of action classes are imbalanced</a> but this doesn’t lead to a biased performance.</p><p>The actions cover a broad range of classes including human-object interactions such as playing instruments, arranging flowers, mowing a lawn, scrambling eggs and so on.</p><h3>Moments in Time</h3><p><strong>Number of videos:</strong> 1,000,000</p><p><strong>Number of action classes:</strong> 339</p><p><strong>Year:</strong> 2018</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*wDCZq1HKvLtNK8-d-mpHcw.png" /><figcaption>Samples from the Moments in Time dataset</figcaption></figure><p><a href="http://moments.csail.mit.edu/">Moments in Time</a> is another large-scale dataset, which was developed by the MIT-IBM Watson AI Lab. With a collection of one million labeled 3-second videos, it is not restricted to human actions only and includes people, animals, objects and natural phenomena, that capture the gist of a dynamic scene.</p><p>The dataset has a significant intra-class variation among the categories. For instance, video clips labeled with the action “opening” include people opening doors, gates, drawers, curtains and presents, animals and humans opening eyes, mouths and arms, and even a flower opening its petals.</p><p>That’s natural for the humans to recognize that all of the above-mentioned scenarios belong to the same category “opening” even though visually they look very different from each other. So, as <a href="https://arxiv.org/pdf/1801.03150.pdf">pointed out by the researchers</a>, the challenge is to develop deep learning algorithms that will be also able to discriminate between different actions, yet generalize to other agents and settings within the same action.</p><p>The action classes in Moments in Time dataset are chosen such that they include the most commonly used verbs in the English language, covering a wide and diverse semantic space. So, there are 339 different action classes in the dataset with 1,757 labeled videos per class on average; each video is labeled with only one action class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*l_FtTdwDrzu1rRTLOtP4UQ.png" /><figcaption>Labeling interface</figcaption></figure><p>As you can see from the image, the annotation process was very straightforward: workers were presented with video-verb pairs and asked to press a Yes or No key responding if the action is happening in the scene. For the training set, the researchers run each video through annotation at least 3 times and required a human consensus of at least 75%. For the validation and test sets, they increased the minimum number of rounds of annotation to 4 with a human consensus of at least 85%.</p><h3>SLAC</h3><p><strong>Number of videos:</strong> 520,000 videos –&gt; 1.75M 2-second clips</p><p><strong>Number of action classes:</strong> 200</p><p><strong>Year:</strong> 2017</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*w3tjoPj2uGRpieN0ZZsGfA.png" /><figcaption>Data collection procedure</figcaption></figure><p>A <a href="http://slac.csail.mit.edu/">Sparsely Labeled ACtions Dataset (SLAC)</a> is introduced by the group of researchers from MIT and Facebook. The dataset is focused on human actions, similarly to Kinetics, and includes over 520K untrimmed videos retrieved from YouTube with the average length of 2.6 minutes. <a href="https://arxiv.org/abs/1712.09374">2-second clips were sampled from the videos</a> by a novel active sampling approach. This resulted in 1.75M clips, including 755K positive samples and 993K negative samples as annotated by a team of 70 professional annotators.</p><p>As you see, the distinctive feature of this dataset is the presence of negative samples. See the illustration of negative samples below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*An78kbomQLgOxqk8yRL02Q.png" /><figcaption>Negative samples from the SLAC dataset</figcaption></figure><p>The dataset includes 200 action classes taken from the <a href="http://activity-net.org/">ActivityNet</a> dataset.</p><p>Please note that even though the paper introducing this dataset was released in December 2017, the dataset is still not available for download. Hopefully, this will change very soon.</p><h3>VLOG</h3><p><strong>Number of videos:</strong> 114,000</p><p><strong>Year:</strong> 2017</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*3FvN5yZOd03rPI68PHRvLw.jpeg" /><figcaption>Samples from the VLOG dataset</figcaption></figure><p>The <a href="https://people.eecs.berkeley.edu/~dfouhey/2017/VLOG/index.html">VLOG dataset</a> differs from the previous datasets in the way it was collected. The traditional approach to getting data starts with a laundry list of action classes and then searching for the videos tagged with the corresponding labels.</p><p>However, such an approach runs into trouble because everyday interactions are not likely to be tagged on the Internet. Could you imagine uploading and tagging video of yourself opening a microwave, opening a fridge, or getting out of bed? The people tend to tag unusual things like for example, jumping in a pool, presenting the weather, or playing the harp. As a result, available datasets are often imbalanced with more data featuring unusual events and less data on our day-to-day activities.</p><p>To solve this issue, the <a href="https://people.eecs.berkeley.edu/~dfouhey/2017/VLOG/paper.pdf">researchers from the University of California</a> suggest starting out with a superset of what we actually need, namely interaction-rich video data, and then annotating and analyzing it after the fact. They start data collection from the lifestyle VLOGs — an immensely popular genre of video that people publicly upload to YouTube to document their lives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*uBo79UNbuOUpjVquAOHKcw.png" /><figcaption>Illustration of the automatic gathering process</figcaption></figure><p>As the data was gathered implicitly, it represents certain challenges for annotation. The researchers decided to focus on the crucial part of the interaction, the hands, and how they interact with the semantic objects at a frame level. Thus, this dataset can also make headway on the difficult problem of understanding hands in action.</p><h3>Bottom Line</h3><p>The action recognition problem requires huge computational costs and lots of data. Fortunately, several very good datasets have appeared during the last year. Together with the previously available benchmarks (<a href="http://activity-net.org/">ActivityNet</a>, <a href="http://crcv.ucf.edu/data/UCF101.php">UCF101</a>, <a href="http://serre-lab.clps.brown.edu/resource/hmdb-a-large-human-motion-database/">HMDB</a>), they build a great foundation for significant improvements in the performance of the action recognition systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4610c5101823" width="1" height="1" alt=""><hr><p><a href="https://medium.com/neurohive-computer-vision/new-datasets-for-action-recognition-4610c5101823">New Datasets for Action Recognition</a> was originally published in <a href="https://medium.com/neurohive-computer-vision">Neurohive - CV papers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[New Datasets for Disguised Face Recognition]]></title>
            <link>https://medium.com/neurohive-computer-vision/new-datasets-for-disguised-face-recognition-27cd97be0363?source=rss----d720a0e44949---4</link>
            <guid isPermaLink="false">https://medium.com/p/27cd97be0363</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[dataset]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[face-recognition]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Kate Koidan]]></dc:creator>
            <pubDate>Wed, 17 Oct 2018 10:17:14 GMT</pubDate>
            <atom:updated>2018-10-25T13:15:45.851Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-q6nbuPMp56KkAnpZwYWPg.png" /></figure><p>Face recognition is a common task in deep learning and convolutional neural networks (CNNs) are doing a pretty good job here. I guess Facebook usually performs great at recognizing you and your friends in the uploaded images.</p><p>Original article: <a href="https://neurohive.io/en/datasets/disguised-face-recognition/">New Datasets for Disguised Face Recognition</a></p><p>But is this really a solved problem? What if the picture is obfuscated? What if the person impersonates somebody else? Can heavy makeup trick the neural network? How easy is it to recognize a person who wears glasses?</p><p>In fact, disguised face recognition is still quite a challenging task for neural networks and primarily due to the lack of corresponding datasets. In this article, we are going to feature several face datasets presented recently. Each of them reflects different aspects of face obfuscation but their goal is the same — to help developers create better models for disguised face recognition.</p><h3>Disguised Faces in the Wild</h3><p><strong>Number of images:</strong> 11,157</p><p><strong>Number of subjects:</strong> 1,000</p><p><strong>Year:</strong> 2018</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*j2r_HbeC5rXxMOYkR9IEaA.png" /><figcaption>Sample <em>genuine</em>, <em>cross-subject impostor</em>, <em>impersonator</em>, and <em>obfuscated</em> face images for a single subject</figcaption></figure><p>We will start with the most recent dataset presented earlier this year — <a href="http://iab-rubric.org/papers/2018_CVPRW_disguised-faces-wild.pdf">Disguised Faces in the Wild (DFW)</a>. It primarily contains images of celebrities of Indian or Caucasian origin. The dataset focuses on a specific challenge of face recognition under the disguise covariate.</p><p>According to the DFW’s description, it covers disguise variations with respect to hairstyles, beard, moustache, glasses, make-up, caps, hats, turbans, veils, masquerades and ball masks. This is coupled with other variations with respect to pose, lighting, expression, background, ethnicity, age, gender, clothing, and camera quality.</p><p>There are four types of images in the dataset:</p><p>· <strong>Normal Face Image:</strong> each subject has a non-disguised frontal face image.</p><p>· <strong>Validation Face Image:</strong> 903 subjects have an image, which corresponds to a non-disguised frontal face image and can be used for generating a non-disguised pair within a subject.</p><p>· <strong>Disguised Face Image:</strong> each subject has from 1 to 12 face images with intentional or unintentional disguise.</p><p>· <strong>Impersonator Face Image:</strong> 874 subjects have from 1 to 21 images pertaining to the impersonators. An impersonator of a subject refers to an image of any other person (intentionally or unintentionally) pretending to be the subject’s identity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*C818FxMq-5emgCKU5njzug.png" /><figcaption>Sample images of 3 subjects from the DFW dataset. Each row corresponds to one subject, containing the normal (gray), validation (yellow), disguised (green), and impersonated (blue) images.</figcaption></figure><p>In total, the DFW dataset contains 1,000 normal face images, 903 validation face images, 4,814 disguised face images, and 4,440 impersonator images.</p><h3>Makeup Induced Face Spoofing</h3><p><strong>Number of images:</strong> 642</p><p><strong>Number of subjects:</strong> 107 + 107 target subjects</p><p><strong>Year:</strong> 2017</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*xDpV0M8_3A5bq1h08qn1sA.png" /><figcaption>An attempt of one subject to spoof multiple identities</figcaption></figure><p><a href="http://www.antitza.com/makeup-datasets.html">Makeup Induced Face Spoofing dataset (MIFS)</a> is also about impersonating but with the specific focus on makeup. The researchers extracted the images from the YouTube videos where female subjects were applying makeup to transform their appearance in order to resemble celebrities. It should be noted, though, that the subjects were not trying to deliberately deceive an automated face recognition system but rather, their intention was to impersonate a target celebrity from a human vision perspective.</p><p>The dataset consists of 107 makeup-transformations with two before-makeup and two after-makeup images per subject. Additionally, two face images of the target identity were taken from the Internet and included to the dataset. However, it is important to point out that the target images are not necessarily those used by the spoofer as a reference during the makeup transformation process. The celebrities sometimes change their facial appearance drastically, and so, the researchers were trying to select target identity images that most resembled the after-makeup image.</p><p>And finally, all the acquired images were subjected to face cropping. This routine eliminates hair and accessories. The examples of the cropped images are provided below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*mTnMgJ-gw9aoPLtgm5Zisw.png" /><figcaption>Examples of images in the MIFS dataset after cropping: before makeup — after makeup — target identity</figcaption></figure><p>So, in total, the MIFS dataset contains 214 images of a subject before makeup; 214 images of the same subject after makeup with the intention of spoofing; and 214 images of the target subject who is being spoofed. It should be also noted that there are subjects attempting to spoof multiple target identities resulting in duplicate subject identities and also there are multiple subjects attempting to spoof the same target identity resulting in duplicate target identities.</p><h3>Specs on Faces dataset</h3><p><strong>Number of images:</strong> 42,592</p><p><strong>Number of subjects:</strong> 112</p><p><strong>Year:</strong> 2017</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*YQ3ex46NzBIBue-rkRv9Fg.png" /><figcaption>Samples from SoF dataset: metadata for each image includes 17 facial landmarks, a glass rectangle, and a face rectangle</figcaption></figure><p>It looks like glasses as a natural occlusion threaten the performance of many face detectors and facial recognition systems. That’s why such a dataset with all the subjects wearing glasses is of particular importance. <a href="https://sites.google.com/view/sof-dataset">Specs on Faces dataset (SoF)</a> comprises 2,662 original images of size 640 × 480 pixels for 112 persons (66 males and 46 females) from different ages. The glasses are the common natural occlusion in all images of the dataset. This original set of images consists of two parts:</p><p>· 757 unconstrained face images in the wild that were captured over a long period in several locations under indoor and outdoor illumination environments;</p><p>· 1905 images that are specifically dedicated to challenging harsh illumination changes: 12 persons were filmed under a single lamp located in arbitrary locations to emit light rays in random directions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*6FkvSBbe972FvdI3fS8djg.png" /><figcaption>Images captured under different lighting directions</figcaption></figure><p>Then, for each image of the original set, there are:</p><p>· 6 extra images generated by the synthetic occlusions — nose and mouth occlusion using a white block;</p><p>· 9 extra images generated with the image filters: Gaussian noise, Gaussian blur, and image posterization using fuzzy logic.</p><p>So, in total, the SoF dataset includes 42,592 images of 112 persons and a huge bonus — handcrafted metadata that contains subject ID, view (frontal/near-frontal) label, 17 facial feature points, face and glasses rectangle, gender and age labels, illumination quality, and facial emotion for each subject.</p><h3>Large Age-Gap Face Verification</h3><p><strong>Number of images:</strong> 3,828</p><p><strong>Number of subjects:</strong> 1,010 celebrities</p><p><strong>Year:</strong> 2017</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*Tmd2ps5nJPvlrvmShq48gA.png" /><figcaption>Examples of face crops for matching pairs in the LAG dataset</figcaption></figure><p>Another challenge is a large age gap. Can the algorithm recognize a celebrity based on her picture from the early childhood? <a href="http://www.ivl.disco.unimib.it/activities/large-age-gap-face-verification/">Large-age gap dataset (LAG)</a> was created to help developers with solving this challenging task.</p><p>The dataset is constructed with photos of celebrities discovered through the Google Image Search and YouTube videos. The large age gap may have different interpretations: from one side it refers to images with extreme difference in age (e.g., 0 to 80 years old) but on the other side, it also refers to a large difference in appearance due to the aging process. For instance, <a href="https://www.researchgate.net/publication/301844520_Large_Age-Gap_face_verification_by_feature_injection_in_deep_networks">as pointed out by the dataset author</a>, “0 to 15 years old is a relatively small difference in age but has a large change in appearance”.</p><p>The LAG dataset reflects both aspects of a large-age gap concept. It contains 3,828 images of 1,010 celebrities. For each identity, at least one child/young image and one adult/old image are present. Starting from the collected images, a total of 5051 matching pairs has been generated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*6h9nmiTj2MHvFO_-JQL7jA.png" /><figcaption>Additional examples of matching pairs in the LAG dataset</figcaption></figure><h3>Bottom Line</h3><p>The face recognition problem is still topical. There are lots of challenging tasks that significantly threaten the performance of the current facial recognition systems — it turns out that even glasses are a huge problem. Fortunately, the new face image datasets appear constantly. While each of them is focusing on the different aspects of the problem, together they build a great foundation for significant improvements in the performance of the facial recognition systems.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27cd97be0363" width="1" height="1" alt=""><hr><p><a href="https://medium.com/neurohive-computer-vision/new-datasets-for-disguised-face-recognition-27cd97be0363">New Datasets for Disguised Face Recognition</a> was originally published in <a href="https://medium.com/neurohive-computer-vision">Neurohive - CV papers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Has MS Voxel Deep Network Managed to Improve 3D Objects Recognition Using Cloud Map Only]]></title>
            <link>https://medium.com/neurohive-computer-vision/how-has-ms-voxel-deep-network-managed-to-improve-3d-objects-recognition-using-cloud-map-only-76fb8f223661?source=rss----d720a0e44949---4</link>
            <guid isPermaLink="false">https://medium.com/p/76fb8f223661</guid>
            <category><![CDATA[3d-modeling]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[image-recognition]]></category>
            <dc:creator><![CDATA[NeuroHive]]></dc:creator>
            <pubDate>Mon, 15 Oct 2018 10:08:45 GMT</pubDate>
            <atom:updated>2018-04-26T16:40:19.539Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xdYaHGTQubjUg9rw." /><figcaption>Example of classified point cloud on Semantic3D test set (blue: man-made terrain, cerulean blue: natural terrain, green: high vegetation, light green: low vegetation, chartreuse green: buildings, yellow: hard scape, orange: scanning</figcaption></figure><p>Mobile Laser Scanning (MLS) systems can now scan large areas, like cities or even countries. The produced 3D point clouds can be used as maps for autonomous systems. To do so, the automatic classification of the data is necessary and is still challenging, regards to the number of objects present in an urban scene. Xavier Roynard, Jean-Emmanuel Deschaud and François Goulette <a href="https://arxiv.org/pdf/1804.03583v1.pdf">proposes</a> both a training method that balances the number of points per class during each epoch, and a 3D CNN capable of effectively learning how to classify scenes containing objects at multiple scales.</p><p>MS Voxel Deep Network — A new convolutional neural network (CNN) to classify 3D point clouds of urban or indoor scenes. On the reduced-8 Semantic3D benchmark, this network, ranked second, beats the state of the art of point classification methods (those not using a regularization step).</p><h3>Network Learning Difficulties</h3><p>Training on scenes point cloud leads to some difficulties. For the point classification task, each point is a sample, so the number of samples per class is very unbalanced (from thousands of points for the class “pedestrian” to tens of millions for the class “ground”). Also with the training method of deep-learning, an Epoch would be to pass through all points of the cloud, which would take a lot of time. Indeed, two very close points have the same neighborhood, and will therefore be classified in the same way.</p><p>Authors propose a training method that solves these two problems. We randomly select N (for example 1000) points in each class, then we train on these points mixed randomly between classes, and we renew this mechanism at the beginning of each Epoch.</p><p>Once a point p to classify is chosen, we compute a grid of voxels given to the convolutional network by building an occupancy grid centered on p whose empty voxels contain 0 and occupied voxels contain 1. We only use NxNxN cubic grids where n is pair, and we only use isotropic space discretization steps ∆.</p><h3>Network Training</h3><p>Some classic data augmentation steps are performed before projecting the 3D point clouds into the voxels grid:</p><p>• Flip x and y axis, with probability 0.5</p><p>• Random rotation around z-axis</p><p>• Random scale, between 95% and 105%</p><p>• Random occlusions (randomly removing points), up to 5%</p><p>• Random artefacts (randomly inserting points), up to 5%</p><p>Random noise in position of points, the noise follows a normal distribution centered in 0 with standard deviation 0.01m</p><p>The cost function used is cross-entropy, and the optimizer used is ADAM with a learning rate of 0.001 and ε = 10−8, which are the default settings in most deep-learning libraries.</p><h3>Architecture — Layers</h3><p>3D Essential Layers</p><ul><li>Conv(n, k, s, p) a convolutional layer that transforms feature maps from previous layer into n new feature maps, with a kernel of size k × k × k and stride s and pads p on each side of the grid.</li><li>DeConv(n, k, s, p) a transposed convolutional layer that transforms feature maps from previous layer into n new feature maps, with a kernel of size k × k × k and stride s and pads p on each side of the grid.</li><li>FC(n) a fully-connected layer that transforms the feature maps from previous layer into n feature maps.</li><li>MaxPool(k) a layer that aggregates on each feature map every group of 8 neigboring voxels.</li><li>MaxU nPool(k) a layer that computes an inverse of MaxPool(k).</li><li>ReLU, LeakyReLU and PReLU common non-linearities used after linear layers as Conv and FC. ReLU (x) returns the positive part of x, and to avoid null gradient if x is negative, we can add a slight slope which is fixed (LeakyReLU ) or can be learned (PReLU ).</li><li><strong>SoftMax</strong> a non-linearity layer that rescales a tensor in the range [0, 1] with sum</li><li>BatchNorm a layer that normalizes samples over a batch.</li><li>DropOut(p) a layer that randomly zeroes some of the elements of the input tensor with probability p.</li></ul><p>The chosen network architecture is inspired from <a href="https://arxiv.org/pdf/1409.1556.pdf">this one</a> that works well in 2D.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lisDCBkzIOVi5R2E." /><figcaption>Multi-Scale Voxel Network architecture: MS3_DeepVoxScene (all tensors are represented as 2D tensors instead of 3D for simplicity).</figcaption></figure><h3>Datasets</h3><p>Authors compared 3 different datasets. Paris-Lille-3D contains 50 classes but for our experimentations we keep only 9 coarser classes. In brackets is indicated the number of points after subsampling at 2 cm.</p><p>Among the 3D point cloud scenes datasets, these are those with the most area covered and the most variability</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GCzIJ2Yb2fgVmjJD." /></figure><p>The covered area is obtained by projecting each cloud on an horizontal plane in pixels of size 10cm × 10cm, then summing the area of all occupied pixels.</p><p>A finer resolution of 5 cm was added to better capture the local surface near the point, and a coarser resolution of 15 cm to better understand the context of the object to which the point belongs. This method achieves better results than all methods that classify cloud by points (i. e. without regularization). Even better results could probably be achieved by adding for example a CRF after classification.</p><p><strong>Example of classified point cloud on S3DIS dataset:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/0*FSh0AN0gktWqQyP2." /><figcaption>Classified with MS3_DVS</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/0*LMSRxGOx_-seX3K9." /><figcaption>Right: ground truth (blue: ground, cerulean blue: buildings, dark green: poles, green: bollards, light green: trash cans, yellow: barriers, dark yellow: pedestrians, orange: cars, red: natural).</figcaption></figure><p>Results are very close to the truth. This is achieved by both focusing on the local shape of the object around a point and by taking into account the context of the object.</p><p>Quote:</p><blockquote><em>We observe a confusion between the classes wall and board (and more slightly with beam, column, window and door), this is explained mainly because these classes are very similar geometrically and we do not use color. To improve these results, we should not sub-sample the clouds to keep the geometric information thin (such as the table slightly protruding from the wall) and add a 2 cm scale in input to the network, but looking for neighborhoods would then take an unacceptable amount of time.</em></blockquote><p>For a comparison with the state-of-the-art methods on S3DIS 5th fold see table below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fIl-7IxydLFPB6nn." /></figure><p>To evaluate architecture choices, this classification task was tested by one of the first 3D convolutional networks: VoxNet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/0*PqW3deLIhtA_r5oQ." /><figcaption>Comparison to VoxNet</figcaption></figure><p>Comparison with the state-of-the-art methods <a href="http://www.semantic3d.net/view_results.php?chl=2">on reduced-8 Semantic3D benchmark</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*j1jE5CSz1pOr1zw5." /></figure><p>Comparison per class between MS1_DeepVoxScene and MS3_DeepVoxScene on Paris-Lille-3D dataset. This shows that the use of multi-scale networks improves the results on some classes, in particular the buildings, barriers and pedestrians classes are greatly improved (especially in Recall), while the car class loses a lot of Precision.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DIa4LG2chzoonO1W." /><figcaption>Comparison per class between MS1_DeepVoxScene and MS3_DeepVoxScene on Paris-Lille-3D dataset</figcaption></figure><h3>Conclusion</h3><p>Proposed training method — MS3_DVS, balances the number of points per class seen during each epoch, as well as a multi-scale CNN that is capable of learning to classify point cloud scenes. You may follow this on Semantic3D benchmark. Now it’s number 2 among all — very good result. This is achieved by both focusing on the local shape of the object around a point and by taking into account the context of the object</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76fb8f223661" width="1" height="1" alt=""><hr><p><a href="https://medium.com/neurohive-computer-vision/how-has-ms-voxel-deep-network-managed-to-improve-3d-objects-recognition-using-cloud-map-only-76fb8f223661">How Has MS Voxel Deep Network Managed to Improve 3D Objects Recognition Using Cloud Map Only</a> was originally published in <a href="https://medium.com/neurohive-computer-vision">Neurohive - CV papers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Neural Network Has Learned to Separate Individuals’ Speech on Video]]></title>
            <link>https://medium.com/neurohive-computer-vision/neural-network-has-learned-to-separate-individuals-speech-on-video-bad8f38974ca?source=rss----d720a0e44949---4</link>
            <guid isPermaLink="false">https://medium.com/p/bad8f38974ca</guid>
            <category><![CDATA[voice-recognition]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[NeuroHive]]></dc:creator>
            <pubDate>Mon, 15 Oct 2018 10:08:39 GMT</pubDate>
            <atom:updated>2018-04-26T16:40:52.409Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3OjEbbR6i6QyDI_t.png" /></figure><p>The fact that our brain in a noisy environment can effectively focus on a particular speaker, “turning off” background sounds — is no secret. This phenomenon even received the popular name “cocktail party effect”. But, despite the good study of the phenomenon, the automatic separation of speech of a particular speaker is still a difficult task for machines.</p><p>So Looking to Listen project with a combined audio-visual model was created. The project’s technology makes it possible to select one person’s speech from the soundtrack (including background noise and other people’s voices) and to mute all other sounds.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FNzZDnRni-8A%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DNzZDnRni-8A&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FNzZDnRni-8A%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/bdbb508a5d2167ab557c9a70e2d5bdb6/href">https://medium.com/media/bdbb508a5d2167ab557c9a70e2d5bdb6/href</a></iframe><p>The method works on totally ordinary videos with single audio stream. All that is required from users is to select face of the person they want to hear on the video.</p><p>Possible application of this technology is extremely wide — from speech recognition to hearing aids, which today work poorly if several people speak simultaneously.</p><p>The new technology uses a combination of audio and video to separate speech. Thanks to the human lips movement and its correlations with the sounds spoken, it is possible to determine which part of the audio stream is associated with a particular person. This significantly improves the quality of speech allocation (in comparison with systems that process only audio), especially in situations where there are several speakers at once.</p><p>More importantly, the technology makes it possible to recognise which of the people on the video says what, by linking the already separated speech with specific speakers.</p><h3>How does it work?</h3><p>As training examples for a neural network, a database of 100,000 videos of lectures and conversations on YouTube was used. Of these, fragments with “pure speech” (without background music, audience sounds or other people’s speeches) and just one speaker in the frame with a total duration of about 2000 hours were singled out.</p><p>Then from these “pure” data “synthetic cocktail parties” were created — videos in which faces of the speakers were mixed, as well as their pre-separated speech and background noises, which were taken on <a href="https://research.google.com/audioset/">AudioSet</a>.</p><p>As a result, it was possible to train the convolutional neural network to allocate from the “cocktail party” a separate audio stream for each person speaking on the video.</p><p><strong>The architecture of the neural network</strong> can be found on the diagram below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QCrD1YJNCMWZ1GJf." /><figcaption>The model of the multistream neural network used in the Looking to Listen: the video stream uses as the original data fragments with faces recognized in each frame, and the audio stream uses soundtrack of the video clip with both speech and background noises.</figcaption></figure><p>First, the recognised faces are extracted from the video stream, after which the convolutional neural network looks at the features of each face. From the audio stream, in turn, the system first obtains a spectrogram using STFT, and then it processes it via a similar neural network. The combined audiovisual representation is obtained by fusing the processed audio and video signals and is further processed using a bi-directional LSTM and three layers of a deep convolutional neural network.</p><p>The network creates a complex spectrogram mask for each speaker, which is multiplied with “noisy” original data and again converted into a waveform to obtain an isolated speech signal for each speaker.</p><p>More information about the technology and projects results can be found in <a href="https://arxiv.org/abs/1804.03619">the project documentation</a> and on its <a href="https://looking-to-listen.github.io/">Github page</a>.</p><h3>The Results</h3><p>Below are the results of applying this technology to several videos. All sounds, except for the speech of the selected person, can be either turned off at all or muffled to the required level.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FZ_ogAiVoE1g%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DZ_ogAiVoE1g&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FZ_ogAiVoE1g%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/479985d5abd3ac77db8ac4f6758e47df/href">https://medium.com/media/479985d5abd3ac77db8ac4f6758e47df/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FuKwUL7vt03M%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DuKwUL7vt03M&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FuKwUL7vt03M%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d4444e2f2e314ae05eaca8548446eb59/href">https://medium.com/media/d4444e2f2e314ae05eaca8548446eb59/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FOBLiJ0N9ac4%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DOBLiJ0N9ac4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FOBLiJ0N9ac4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/26eb65a69cd62871059b409f1879ace5/href">https://medium.com/media/26eb65a69cd62871059b409f1879ace5/href</a></iframe><p>This technology can be useful for speech recognition and automatic captions. Existing systems do not cope with the situation when the speech of several people overlap. Separation of sound “by source” allows you to get more accurate and easier to read captions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bad8f38974ca" width="1" height="1" alt=""><hr><p><a href="https://medium.com/neurohive-computer-vision/neural-network-has-learned-to-separate-individuals-speech-on-video-bad8f38974ca">Neural Network Has Learned to Separate Individuals’ Speech on Video</a> was originally published in <a href="https://medium.com/neurohive-computer-vision">Neurohive - CV papers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Gfycat Solves the Problem of Recognizing Faces of Asians and Africans]]></title>
            <link>https://medium.com/neurohive-computer-vision/how-gfycat-solves-the-problem-of-recognizing-faces-of-asians-and-africans-59b08357a299?source=rss----d720a0e44949---4</link>
            <guid isPermaLink="false">https://medium.com/p/59b08357a299</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[face-recognition]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[NeuroHive]]></dc:creator>
            <pubDate>Mon, 15 Oct 2018 10:08:33 GMT</pubDate>
            <atom:updated>2018-04-25T14:26:23.592Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*6Dj39DtOqmDV_3hZR6qtZw.png" /></figure><p>Is it correct to ask such a question at all? The latest study showed that racial differences have to be taken into account by developers of neural networks to improve the accuracy of face recognition. But in society it is formally considered that all people are equal and bias in this matter should be avoided.</p><p>Gfycat software engineer Gurney Gan said that last summer his software successfully identified most of his colleagues, but “stumbled” on one group.</p><p>“It got some of our Asian employees mixed up. Which was strange because it got everyone else correctly”, — says Gan.</p><p>However, even the largest companies have similar problems. For example, Microsoft and IBM face analysis services are <a href="https://www.wired.com/story/photo-algorithms-id-white-men-fineblack-women-not-so-much/">95 percent more accurate</a> when recognizing white men than women with darker skin.</p><p>The Google Photos service does not respond to requests “gorilla”, “chimpanzee” or “monkey”. Thus, the risk of repetition of the embarrassment of 2015 is eliminated. The search engine back then mistakenly took black people in photos for monkeys.</p><p>A universal standard for testing and eliminating bias in AI systems has not been developed yet.</p><p>“Lots of companies are now taking these things seriously, but the playbook for how to fix them is still being written”, — says Meredith Whittaker, co-director of AI Now.</p><p>Gfycat started face recognition development to create a system that would allow people to find the perfect GIFs to use in messengers. The search system of the company works with about 50 million GIF-files — from cats to presidents’ faces. With the help of face recognition, developers wanted to facilitate the search for famous personalities — from politicians to stars</p><h3>“Asian detector”</h3><p>The company used open source software based on Microsoft research and trained it on millions of photos from collections issued by the Universities of Illinois and Oxford.</p><p>But the neural network could not distinguish Asian celebrities, such as Constance Wu and Lucy Liu, and did not distinguish reliably people with dark skin.</p><p>First, Gan tried to solve the problem by adding more photos of “problematic” faces to machine learning examples. But adding a large number of photos of black and asian celebrities to the data array helped only partially.</p><p>It was possible to solve the problem only by creating a sort of “Asian detector”. Finding an asian face, the system goes into a mode of hypersensitivity.</p><p>According to Ghana, this was the only way to make the program distinguish Asians from each other.</p><p>The company says that the system is now 98 percent accurate when identifying white people, and 93 percent when dealing with Asians.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgfycat.com%2Fifr%2FMasculineFavoriteKob&amp;url=https%3A%2F%2Fgfycat.com%2Fgifs%2Fdetail%2FMasculineFavoriteKob&amp;image=https%3A%2F%2Fthumbs.gfycat.com%2FMasculineFavoriteKob-size_restricted.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=gfycat" width="640" height="368" frameborder="0" scrolling="no"><a href="https://medium.com/media/67869f6f6c02cef274cb72328eba20e9/href">https://medium.com/media/67869f6f6c02cef274cb72328eba20e9/href</a></iframe><p>Intentional search for racial differences may seem a strange way to combat prejudice. But this idea was supported by a number of scientists and companies. In December, Google published a <a href="https://arxiv.org/abs/1712.00193">report</a> on improving the accuracy of the smile recognition system. This was achieved by determining whether the person in the photo is a man or a woman, and to which of the four racial groups he belongs. But the document also says that artificial intelligence systems should not be used to determine the person’s race, and that using only two gender and four racial categories is not sufficient in all cases.</p><p>Some researchers have suggested forming industry standards for transparency and decreasing AI bias. Thus, Eric Learned-Miller, professor of the University of Massachusetts, suggests that organizations utilizing face recognition, such as Facebook and the FBI, should disclose the accuracy of their systems for different gender and racial groups.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=59b08357a299" width="1" height="1" alt=""><hr><p><a href="https://medium.com/neurohive-computer-vision/how-gfycat-solves-the-problem-of-recognizing-faces-of-asians-and-africans-59b08357a299">How Gfycat Solves the Problem of Recognizing Faces of Asians and Africans</a> was originally published in <a href="https://medium.com/neurohive-computer-vision">Neurohive - CV papers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Neural Network Compresses Images Without Loss Of Quality]]></title>
            <link>https://medium.com/neurohive-computer-vision/neural-network-compresses-images-without-loss-of-quality-b97108054491?source=rss----d720a0e44949---4</link>
            <guid isPermaLink="false">https://medium.com/p/b97108054491</guid>
            <category><![CDATA[image]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[compression]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[NeuroHive]]></dc:creator>
            <pubDate>Mon, 15 Oct 2018 10:08:16 GMT</pubDate>
            <atom:updated>2018-04-25T14:11:57.616Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gDKHARvD4rHTxeWZAt1P2A.png" /></figure><p>Graphical information requires a lot of storage resources, and the main task is to learn how to compress images without losing quality.</p><h3>Computer Vision Laboratory Solution</h3><p>A group of researchers from the Swiss Computer Vision laboratory <a href="https://arxiv.org/pdf/1804.02958.pdf">offered</a> a way to process images and video, allowing to significantly reduce the amount of memory required for storing graphical information. They reported that this approach is intended for images with a small bit rate (the number of bits for storing one pixel of the image) and can save up to 67% of memory compared to the BPG method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/0*wy3NeBM0AFLugH2L." /></figure><p>Developers decided not to store the entire image, but only the most significant parts of it. The rest will be reconstructed when data is extracted.</p><p>For example, if we are watching a video with a walking person, we primarily focus on that person. The background does not really matter if there is nothing extraordinary.</p><p>Generation of insignificant areas cannot occur from scratch. To do this, a map of semantic marks is created from the original image. It notes that in one part of the image, for example, green foliage is located, in other part — asphalt, etc.</p><p>A solution became possible due to the use of deep neural networks (DNNs) as image compression systems. We are faced with them searching for information and translating text using Google and Yandex.</p><h3>GANs Modes of Operation</h3><p>Generative adversarial networks (GANs) developed in the Computer Vision laboratory can work in two modes:</p><p>a) global generative compression (GC);</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/0*yj_F0mfsa3SC3NlV." /></figure><p>b) selective generative compression (SC).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/0*8IvJtmvsyvkXP9dF." /></figure><p>In the scheme, the original image is designated as <strong><em>x</em></strong>, and the map of semantic marks is <strong><em>s</em></strong>. In the GC mode, the semantic map can be used optionally. In the SC mode, its application is compulsory.</p><p>Other marks on the diagram:</p><ul><li><strong><em>E</em></strong> — image encoder;</li><li><strong><em>w</em></strong> — image code;</li><li><strong><em>w(^)</em></strong> — the code after the quantization procedure (the second level of discretization of information);</li><li><strong><em>G</em></strong> — the generator of the compressed image;</li><li><strong><em>x(^)</em></strong> — compressed image.</li></ul><p>An <strong><em>F</em></strong> device is added to the SC circuit , which extracts data from the semantic map and indicates their place to the generator.</p><p>The system allows efficient processing and indexing of any compressed images.</p><h3>Neural Networks</h3><p>Serious errors in the compressed image occur at bitrates below 0.1 bit per pixel (bpp). When bpp aspires to zero, it is impossible to preserve the full content of the image.</p><p>Therefore, it is important to go beyond the standard peak-to-noise ratio (PSNR). Adversarial losses function is considered a promising tool. It is used to capture global semantic information and local texture. They allow you to obtain high-resolution images from a semantic tag map.</p><p>Two types of neural networks are used to work with this technology: autoencoders and recursive neural networks. They convert the incoming image into a bitstream, which is compressed by mathematical coding or the Huffman method. The image quality remains the same.</p><p>Generative adversarial networks (GANs) have become a popular tool for neural networks training. They allow you to create more stable and clear images in comparison with previous technologies.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b97108054491" width="1" height="1" alt=""><hr><p><a href="https://medium.com/neurohive-computer-vision/neural-network-compresses-images-without-loss-of-quality-b97108054491">Neural Network Compresses Images Without Loss Of Quality</a> was originally published in <a href="https://medium.com/neurohive-computer-vision">Neurohive - CV papers</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>