<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lee Cohn on Medium]]></title>
        <description><![CDATA[Stories by Lee Cohn on Medium]]></description>
        <link>https://medium.com/@lncohn?source=rss-244ab10f73e9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*5SCx_0P598q7ArK5CN_KFA.png</url>
            <title>Stories by Lee Cohn on Medium</title>
            <link>https://medium.com/@lncohn?source=rss-244ab10f73e9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 15:12:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lncohn/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Artificial “Multispectralization” of Color Satellite Imagery via GANs]]></title>
            <link>https://medium.com/the-downlinq/artificial-multispectralization-of-color-satellite-imagery-via-gans-f7816770dd54?source=rss-244ab10f73e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/f7816770dd54</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[gans]]></category>
            <category><![CDATA[multispectral]]></category>
            <dc:creator><![CDATA[Lee Cohn]]></dc:creator>
            <pubDate>Tue, 07 Nov 2017 17:06:34 GMT</pubDate>
            <atom:updated>2017-11-07T17:06:34.972Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XFNamlLayl3AhNH8Iqg3cQ.png" /><figcaption>GANLeft: color R-G-B satellite image over Las Vegas | Right: artificial multispectral image over Las Vegas visualized with NIR1-G-B</figcaption></figure><h4>Introduction</h4><p>In a previous <a href="https://medium.com/the-downlinq/artificial-colorization-of-grayscale-satellite-imagery-via-gans-part-1-79c8d137e97b">blog post</a>, we outlined an approach to artificially colorize grayscale satellite imagery using <a href="https://arxiv.org/abs/1406.2661">Generative Adversarial Networks</a> (GANs).</p><p>In this blog post, we discuss the analogous task of “multispectralization”: predicting 8-band visible and near-infrared (VNIR) multispectral images from 3-band RGB color images. The intuition behind this project is that contextual information in the image may allow deep learning algorithms to deduce the object type of a given pixel, thereby enabling the inference of additional multispectral values for this pixel since many objects have documented distinct spectral signatures.</p><p>Although our multispectralization GAN achieves a high average peak signal-to-noise ratio (PSNR) score, the failures of our multispectralization GAN also prove intriguing. That is, certain objects in our satellite imagery confuse the GAN and hence attain very low local PSNR scores. These objects have “interesting” spectral signatures from the viewpoint of the GAN.</p><p>We conclude this post with ideas on how to further pursue the interaction between multispectral satellite imagery and deep learning.</p><h3>SpaceNet Data</h3><p>The dataset used in this project is the second <a href="https://aws.amazon.com/public-datasets/spacenet/">SpaceNet dataset</a>, which provides satellite imagery for four different cities (Las Vegas, Paris, Shanghai, and Khartoum). However, we restrict our attention here to Las Vegas and Khartoum. The imagery is comprised of grayscale 30 cm <a href="https://en.wikipedia.org/wiki/Ground_sample_distance">GSD</a>, as well as 30 cm 3-band RGB color imagery, and 30 cm 8-band VNIR multispectral imagery.</p><h3>Generative Adversarial Networks and Multispectralization</h3><p>In a previous blog post on <a href="https://medium.com/the-downlinq/artificial-colorization-of-grayscale-satellite-imagery-via-gans-part-1-79c8d137e97b">colorization via generative adversarial networks</a>, we described the basic definitions and ideas of GANs as well as the specific architecture we chose to address the problem of artificial colorization. Since we use the same architecture for multispectralization (except changing the number of input and output bands from 1 &amp; 3 to 3 &amp; 8), we refer the reader to the <a href="https://medium.com/the-downlinq/artificial-colorization-of-grayscale-satellite-imagery-via-gans-part-1-79c8d137e97b">colorization blog post</a> for a technical discussion of GANs in this context.</p><p>For the remainder of this post, the multispectralization GAN will be treated as a black box, whose input is a 3-band RGB image and whose output is an 8-band multispectral image, and we focus on the output and the applications of our multispectralization GAN.</p><h3>The Artificial NDVI index and the Artificial NDWI index</h3><p>The normalized difference vegation index (NDVI) is a well-established mechanism for identifying and visualizing vegetation in multispectral remote sensing images. It relies on the principle that the chlorophyll in living plant material strongly absorbs visible light, and strongly reflects near-infrared light. The NDVI is calculated from multispectral data according to the following formula: [ NDVI = (Xnir — Xred)/(Xnir + Xred) ], where Xnir refers to the near-infrared band and Xred refers to the red band.</p><p>We can use the GAN to generate the NDVI index from 3-band imagery. Given a 3-band RGB color satellite image <em>x</em>, one can artificially generate an 8-band multispectral image<em> G(x)</em> using the multispectralization GAN, and then compute the NDVI <em>G(x)</em>. We will call this the artificial NDVI.</p><p>The following visualization is produced by taking an 8-band multispectral image over Las Vegas <em>y</em>,<em> </em>its corresponding 3-band color image <em>u</em>,<em> </em>and comparing the NDVI of <em>y</em> and <em>G(u)</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*scLKieezJjErOwQxSzFnQA.png" /></figure><p>Figure: Left is the real NDVI and right is the artificial NDVI</p><p>The real and artificial NDVIs produce strikingly similar visual results. Both indicate high (green) values for vegetation as can be seen in the golf course and various house lawns.</p><p>Similar to NDVI, the normalized difference water index (NDWI) is a standard index for monitoring changes in water content of leaves or a body of water. The NDWI is computed from multispectral data as follows: [ NDWI = (Xgreen — Xnir)/(Xgreen + Xnir) ], where Xgreen refers to the green band and Xnir refers to the NIR band . This formulation of NDWI produces an image in which the positive data values are typically open water areas; while the negative values are typically non-water features.</p><p>As with NDVI, we can compute NDWI via 3-band imagery and the GAN. The following visualization compares the real NDWI and artificial NDWI of an image over Las Vegas.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*Sgz33u4VVxA5t82PUmCm_w.png" /></figure><p>Figure: Left is the real NDVI and right is the artificial NDVI</p><p>Both the real NDWI and the artificial NDWI show high (blue) values for backyard pools. There are also some high NDWI values for shadows in both images, but this is a well-documented property of the NDWI because dark water bodies and shadows have similar spectral properties.</p><h3>The PSNR Heat Map as a Measure of Image Reconstruction Quality</h3><p>We now explore the limitations and failures of the multispectralization GAN. That is, given an 8-band multispectral image <em>y</em>, and its underlying 3-band RGB image <em>u</em>, when do<em> y</em> and <em>G(u)</em> significantly differ? And can one speculate as to why they differ?</p><p>The <a href="https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio">Peak signal-to-noise ratio</a> (PSNR), which is used in a variety of applications including image compression, can also be used to describe the quality measurement between an original and reconstructed image. For example, CosmiQ has used the PSNR metric previously to assess the performance of a <a href="https://medium.com/the-downlinq/super-resolution-on-satellite-imagery-using-deep-learning-part-2-c9ce41dc0ee0">super resolution algorithm</a>. (Chao Dong also used the PSNR metric in the paper “<a href="https://arxiv.org/pdf/1501.00092.pdf">Image Super-Resolution Using Deep Convolutional Networks</a>”.) PSNR is usually expressed in terms of the logarithmic decibel scale, and the higher the PSNR, the better the quality of the reconstructed image. The precise definition and formulas for PSNR can be found <a href="https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio">here</a>.</p><p>The PSNR heat map is simply an image that maps the PSNR computed locally around each pixel in an artificially generated 8-band image. In our example, the PNSR corresponding to a pixel in an artificial 8-band image is the average PSNR of the 8 x 8 square image surrounding that pixel.</p><p>The following two visualizations are PSNR heat maps over Las Vegas and Khartoum.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*3JrdMGdIWNrb2dBPZXLILQ.png" /></figure><p>There are a few observations to make about these two PSNR heat maps.</p><ul><li>Overall, the GAN recovers a majority of the multispectral information in both images. Many areas in Las Vegas and Khartoum are achieving PSNR scores in the 40’s, which exceed our initial expectations based upon prior work including that of Dong mentioned above. That said, our initial findings should be considered preliminary only.</li><li>No single PSNR number can characterize the quality of an entire, large image. Some areas and objects in an image are reconstructed with higher accuracy than others.</li><li>Given limited training data, cities with more homogenous landscapes like Las Vegas achieve higher PSNR scores than cities with heterogeneous landscapes like Khartoum.</li></ul><h3>Objects with “Interesting” Spectral Signatures from the Viewpoint of GANs</h3><p>The PSNR heat maps introduced in the previous section demonstrate that the mulitspectralization GAN reconstructs multispectral image information more accurately in some areas that in others. In this section, we take a closer look at examples of this phenomenon. Furthermore, we interpret the areas and objects where the PSNR score is low as “interesting” from the viewpoint of the GAN.</p><p>We now visualize five areas/objects together with their PSNR heat maps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*caJ7lczxY1o36nHy8cBu4w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*XpiBWQcqKfu5z01M34We2w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*gkTHuC2GhkKbKVO6Gr4Crw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*5rB5_ML2_zrxQ15XkG1vKw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*zXLLD-ZGY1mYlQe1X-clKA.png" /></figure><p>In general, areas or objects with unexpected spectral properties (from the viewpoint of the GAN) will light up as red and orange.</p><p>For example, given a pixel belonging to the green tennis courts in image scene D, the GAN might be mistaking this pixel for grass and hence predict multispectral values that correspond to grass in the training data. The PSNR heat map compares these values to the original image and obtains a low PSNR score displayed as bright orange over the tennis courts.</p><p>For another example, given a pixel belonging to shallow and muddy water in image scene A, which is Khartoum’s White Nile river, the GAN might be mistaking this pixel for deeper water and predicting multispectral values that correspond to deeper water in the training data.</p><h3>Future Research Plans</h3><p>We intend to continue to explore the potential value of multispectral information for deep learning object detection algorithms, as well as the extent to which GANs might be used as a pre-processing step to enhance their performance. We are considering developing a data set that will include labeled objects with known unique VNIR spectral signatures. In the recent research paper entitled <em>Detection of Classification of Buildings in Multispectral Satellite Imagery</em> by Ishii et al. (<a href="http://www.f.waseda.jp/hfs/IshiiICPR2016.pdf">link</a>), the authors demonstrated significant improvement of their deep learning object detection algorithm when using seven of the eleven Landsat 8 bands compared to when using only the three RGB bands. Based on their results, we believe imagery focused on solar farms would be a reasonable starting point for creating a new data set consisting of low-resolution images from NASA’s Landsat 8 and very high resolution images from DigitalGlobe’s Worldview 2 or Worldview 3.</p><p>Given such a labeled data set, we could explore and quantify the performance of several popular deep learning object detection algorithms (e.g., <a href="https://medium.com/the-downlinq/you-only-look-twice-multi-scale-object-detection-in-satellite-imagery-with-convolutional-neural-38dad1cf7571">YOLT</a> and <a href="https://medium.com/the-downlinq/object-segmentation-on-spacenet-via-multi-task-network-cascades-mnc-f1c89d790b42">MNC</a>) against both 3-band and 8-band data for low and high resolution imagery. We could then apply our “multispectralization GAN” as a pre-processing step to determine whether any of the performance degradation in going from 8-band to 3-band could be recovered, as we demonstrated in our prior post with the MNC algorithm applied to grayscale imagery (<a href="https://medium.com/the-downlinq/artificial-colorization-of-grayscale-satellite-imagery-via-gans-part-2-ea382c9da8d">link</a>). Such work could further illustrate the potential utility of the GAN as a pre-processing step for object detection algorithms in particular scenarios, for example, in bandwidth-limited applications.</p><p>Attribution: Special thanks to Adam, Dave, Jake, Lisa, and Ryan.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f7816770dd54" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-downlinq/artificial-multispectralization-of-color-satellite-imagery-via-gans-f7816770dd54">Artificial “Multispectralization” of Color Satellite Imagery via GANs</a> was originally published in <a href="https://medium.com/the-downlinq">The DownLinQ</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Artificial Colorization of Grayscale Satellite Imagery via GANs: Part 2]]></title>
            <link>https://medium.com/the-downlinq/artificial-colorization-of-grayscale-satellite-imagery-via-gans-part-2-ea382c9da8d?source=rss-244ab10f73e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/ea382c9da8d</guid>
            <category><![CDATA[space]]></category>
            <dc:creator><![CDATA[Lee Cohn]]></dc:creator>
            <pubDate>Wed, 06 Sep 2017 13:16:25 GMT</pubDate>
            <atom:updated>2017-09-06T13:16:25.856Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JKYSYB0j470BIlQphtguTA.png" /><figcaption>Left : MNC on a grayscale image over Las Vegas | Right: MNC on the associated artificially colored image (white outline = true positive, yellow outline = false positive)</figcaption></figure><h3>Introduction</h3><p>In <a href="https://medium.com/the-downlinq/artificial-colorization-of-grayscale-satellite-imagery-via-gans-part-1-79c8d137e97b">Part 1</a> of this blog series, we outlined a method to artificially color grayscale satellite images using a generative adversarial network (GAN).</p><p><a href="https://arxiv.org/abs/1512.04412">Multi-task Network Cascades</a> (MNC) is a popular object segmentation algorithm that won first place in the MS COCO 2015 competition. In a previous blog post, <a href="https://medium.com/the-downlinq/object-segmentation-on-spacenet-via-multi-task-network-cascades-mnc-f1c89d790b42">Object Segmentation on SpaceNet via Multi-task Network Cascades (MNC)</a>, we discussed the results of applying MNC to building footprints in 3-band color satellite imagery. In this blog post we show that MNC is better at extracting building footprints in artificially colored satellite images than in their underlying grayscale images. Thus, colorization may be interpreted as a preprocessing step to grayscale satellite images that adds value for building footprint detection with MNC.</p><h3>Performance as a Function of Imaging Bands</h3><p>In a previous <a href="https://medium.com/the-downlinq/panchromatic-to-multispectral-object-detection-performance-as-a-function-of-imaging-bands-51ecaaa3dc56">blog post</a>, we reengineered <a href="https://medium.com/the-downlinq/object-segmentation-on-spacenet-via-multi-task-network-cascades-mnc-f1c89d790b42">MNC</a> and <a href="https://medium.com/the-downlinq/building-extraction-with-yolt2-and-spacenet-data-a926f9ffac4f">YOLT</a> to train on all three SpaceNet image types: grayscale panchromatic imagery, pansharpened 3-band RGB color imagery, and pansharpened 8-band multispectral imagery. This resulted in 12 trained models.</p><p>The F1 scores of these 12 trained models (evaluated on test SpaceNet images) demonstrate that MNC performs better on pansharpened 3-band RGB color imagery than on 1-band grayscale imagery. The following chart summarizes these findings for MNC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*W0k0Y-ns0Ajr6yc3rHewJw.png" /></figure><p>The small black bars in this chart are error bars for the F1 scores computed via bootstrapping. A more detailed description of these findings are described in <a href="https://medium.com/the-downlinq/panchromatic-to-multispectral-object-detection-performance-as-a-function-of-imaging-bands-51ecaaa3dc56">Panchromatic to Multispectral: Object Detection Performance as a Function of Imaging Bands</a>.</p><h3>MNC Results on Artificially Colored Images</h3><p>The experiment we perform is the following. For each city, we possess an MNC model trained on grayscale images and a second model trained on real 3-band RGB color images. Using the Colorization GAN introduced in <a href="https://medium.com/the-downlinq/artificial-colorization-of-grayscale-satellite-imagery-via-gans-part-1-79c8d137e97b">Part 1</a> of this series, we produce inferred (or fake) color images from the grayscale images using the generative network.</p><p>We compare the results of the following three combinations of trained MNC models and test data sets:</p><ul><li>MNC trained on grayscale images and tested on grayscale images.</li><li>MNC trained on real color images and tested on artificially colored images.</li><li>MNC trained on real color images and tested on real colored images.</li></ul><p>The following bar chart and table summarize the results of these experiments:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*5cg399zkKUGIbbgj1_eHDg.png" /><figcaption>The small black bars are error bars computing via bootstrapping.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*J3882FvJfPjfLt_Ol_6uOw.png" /></figure><p>In summary, we find that MNC performance improves significantly from grayscale images to artificially colored images. However, MNC still performs better on the original color satellite images than on artificially colored images. That is, we cannot fully recover all of the building footprint detections by colorizing grayscale images.</p><h3>Visualizations of MNC</h3><p>We now include <a href="http://www.topcoder.com/contest/problem/BuildingDetector2/visualizer_readme.html">visualizations</a> of MNC building footprint detections on grayscale, fake color images, and real color images. Each row of images represents a single 200 m x 200 m area over a single city: Las Vegas, Paris, Shanghai, and Khartoum.</p><p>In the pictures below, a white outline represents a true positive for MNC, and a yellow outline represents a false positive. A more detailed description of the SpaceNet metric can be found <a href="https://medium.com/the-downlinq/the-spacenet-metric-612183cc2ddb">here</a>. We choose not to outline false negatives in these images for visual simplicity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3BepXPLursoqEwMkxyySAQ.png" /><figcaption>Left: MNC on a Las Vegas grayscale image | Middle: MNC on a Las Vegas fake color image | Right: MNC on a Las Vegas real color image</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zBtJUb6EXk0tvhqCN4_5Yg.png" /><figcaption>Left: MNC on a Paris grayscale image | Middle: MNC on a Paris fake color image | Right: MNC on a Paris real color image</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Lfq3sJemVCNx8mr9MrnrQ.png" /><figcaption>Left: MNC on a Shanghai grayscale image | Middle: MNC on a Shanghai fake color image | Right: MNC on a Shanghai real color image</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d8ABfFFjdzprFPFTkACUbA.png" /><figcaption>Left: MNC on a Khartoum grayscale image | Middle: MNC on a Khartoum fake color image | Right: MNC on a Khartoum real color image</figcaption></figure><h3>Conclusions</h3><p>Given a large corpus of grayscale satellite images over a city (say in storage) and a small amount of color satellite images over that same city (we used one 2500 m x 2500 m color image to train our GAN), artificially coloring these grayscale images could significantly boost the performance of building footprint detection with MNC.</p><p>Analogously, in a bandwidth limited environment, one could transmit a small amount of color imagery and transmit the rest as grayscale imagery, relying on the GAN to artificially color the grayscale imagery.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea382c9da8d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-downlinq/artificial-colorization-of-grayscale-satellite-imagery-via-gans-part-2-ea382c9da8d">Artificial Colorization of Grayscale Satellite Imagery via GANs: Part 2</a> was originally published in <a href="https://medium.com/the-downlinq">The DownLinQ</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Artificial Colorization of Grayscale Satellite Imagery via GANs: Part 1]]></title>
            <link>https://medium.com/the-downlinq/artificial-colorization-of-grayscale-satellite-imagery-via-gans-part-1-79c8d137e97b?source=rss-244ab10f73e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/79c8d137e97b</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Lee Cohn]]></dc:creator>
            <pubDate>Wed, 09 Aug 2017 13:15:06 GMT</pubDate>
            <atom:updated>2017-08-09T13:15:06.511Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vEdFnEoymNkXYzbFwOA6-w.png" /><figcaption>Example artificial colorization of a satellite image over Las Vegas</figcaption></figure><h3>Introduction</h3><p>In this blog post we introduce a new method to artificially colorize grayscale satellite imagery.</p><p>Our approach uses <a href="https://arxiv.org/abs/1406.2661">Generative Adversarial Networks</a>, and thus differs from most other recent deep learning colorization techniques [<a href="http://richzhang.github.io/colorization/">1</a>,<a href="https://arxiv.org/abs/1605.00075">2</a>,<a href="http://hi.cs.waseda.ac.jp/~iizuka/projects/colorization/en/">3</a>,<a href="http://people.cs.uchicago.edu/~larsson/colorization/">4</a>,<a href="http://tinyclouds.org/colorize/">5</a>]. We explore the use of GANs for colorizing grayscale satellite imagery because, unlike other methods, such an architecture generalizes to the similar problem of generating the additional bands found in 8-band multispectral satellite images from their underlying 3-band RGB color images.</p><p>We will first motivate why one may want to colorize satellite imagery, and then provide a high-level technical overview of our algorithm.</p><p>In an upcoming blog post, we will show that <a href="https://medium.com/the-downlinq/object-segmentation-on-spacenet-via-multi-task-network-cascades-mnc-f1c89d790b42">MNC</a>, an object detection algorithm, is better at extracting building footprints in artificially colored images than in their underlying grayscale images. Specifically, we will show that MNC trained on real color satellite images and evaluated on artificially colored satellite images has a significantly higher F1 score for building footprints than MNC trained on grayscale satellite images and evaluated on grayscale satellite images.</p><h3>SpaceNet Data</h3><p>The dataset used in this project is the second <a href="https://aws.amazon.com/public-datasets/spacenet/">SpaceNet dataset</a>, which provides satellite imagery for four different cities (Las Vegas, Paris, Shanghai, Khartoum) with attendant GeoJSON labels for building footprints. The imagery is comprised of grayscale 30 cm <a href="https://en.wikipedia.org/wiki/Ground_sample_distance">GSD</a>, as well as 30 cm 3-band RGB color imagery, and 30 cm <a href="https://www.spaceimagingme.com/downloads/sensors/datasheets/DG_WorldView3_DS_2014.pdf">8-band VNIR multispectral</a> imagery.</p><h3>Why Colorize Grayscale Satellite Imagery?</h3><p>Colorizing grayscale satellite images can provide value by:</p><ul><li>Increasing the performance of object detection algorithms like MNC in satellite imagery. Thus, if a grayscale satellite image is colorized, then its potential value for use in machine learning algorithms increases.</li><li>Making digital maps created from grayscale images easier to see with the human eye.</li><li>Augmenting small training data sets of color satellite images with additional artificially colored images.</li><li>Suggesting that one can intelligently compress color satellite images with deep learning. Given a bandwidth-limited remote sensing situation, one could transmit grayscale images and recreate color images from the received grayscale images.</li><li>Providing motivation for generating 8-band multispectral satellite imagery from 3-band RGB color satellite imagery by using a similar algorithm. We will refer to this process as “multispectralizing” color satellite imagery.</li></ul><h3>Recent Research on Colorization with Deep Learning</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ehp6pxvv90b4BsNc7UR7FQ.png" /><figcaption>Example artificial colorizations from <a href="http://richzhang.github.io/colorization/">Colorful Image Colorization</a></figcaption></figure><p>Artificially coloring grayscale images using deep learning has produced several compelling results. <em>It is important to note that the goal of colorization is not to recover the actual ground truth color, but rather, to produce a plausible colorization that the user finds useful even if the colorization differs from the ground truth color.</em></p><p>Colorization can seem like a daunting task because so much information is lost (two out of three color dimensions) in converting a color image to its underlying grayscale representation. Because grass is usually green and the sky is usually blue, the semantics of an image scene provide many clues for a plausible colorization. It also seems reasonable that deep learning has the potential to be a successful tool for colorization because it already takes advantage of scene semantics for image classification and object detection.</p><p>We quickly describe three recent papers on colorization using deep learning:</p><ol><li><a href="https://arxiv.org/pdf/1605.00075.pdf">Deep Colorization</a> of Cheng, Yang, and Sheng (2016) describes a neural network for colorization with input a grayscale image and output a color image in <a href="https://en.wikipedia.org/wiki/YUV">YUV color space</a>. This paper formulates colorization as a regression problem, as the loss function for this neural network is framed as a least squares minimization problem between the predicted and ground-truth color pixel values.</li><li><a href="http://richzhang.github.io/colorization/">Colorful Image Colorization</a> of Zhang, Isola, and Efros (2016) describes a convolutional neural network (CNN) whose input is a lightness grayscale image (L) and whose output is a distribution over quantized color values in <a href="https://en.wikipedia.org/wiki/Lab_color_space">CIE Lab color space</a>, thus framing colorization as a classification task. Lab color space is an alternative system for representing pixel colors versus the standard RGB values. It is useful because the L channel is statistically independent from the pure color a-b channels.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lRBlejZF6Ap-JxeM_73HAA.png" /><figcaption>Neural network architecture for Colorful Image Colorization</figcaption></figure><p>3. <a href="https://arxiv.org/abs/1611.07004">Image-to-Image Translation with Conditional Adversarial Networks</a> of Isola, Zhu, Zhou, and Efros introduces conditional adversarial networks as a general-purpose solution to image-to-image translation problems like reconstructing objects from edge maps, converting a daytime photo to a nighttime photo, and even colorizing of grayscale photos. Our approach to colorization has philosophical similarities to <a href="https://arxiv.org/abs/1611.07004">Image-to-Image Translation with Conditional Adversarial Networks</a>, but we use somewhat different network architectures.</p><p>There are many other great papers on colorization. Many of them can be found in the references of the three papers above.</p><p>In the next section we describe our colorization algorithm. Our approach differs from the three colorization papers reviewed above because we seek a colorization algorithm whose architecture is easily generalizable to “multispectralization”: the related task of generating an 8-band multispectral image from a 3-band RGB image. For instance, we are unaware of multispectral analogs of YUV or Lab color spaces.</p><h3>Generative Adversarial Networks and Colorization</h3><p><a href="https://arxiv.org/abs/1406.2661">Generative Adversarial Networks</a> (or GANs) are composed of two neural networks: a generative model G that tries to model the real data distribution, and a discriminative model D that estimates the probability that a data sample is real data rather than fake. By fake data we mean data that is the output of our generative model G.</p><p>In our case, think of the generative model G as a colorization algorithm: i.e., as trying to produce a 3-band RGB color image from a 1-band grayscale image. Think of the discriminative model D as trying to model the probability that a color image is real or the result of applying G to a grayscale image.</p><p>Given the underlying grayscale image <em>x</em> of a training color RGB color image <em>y</em>, <a href="https://arxiv.org/abs/1605.00075">Deep Colorization</a> would try to minimize the mean squared error of G(<em>x</em>) and <em>y</em>. Meanwhile, <a href="http://richzhang.github.io/colorization/">Colorful Image Colorization</a> frames this as a classification problem to create even more vibrant colorizations. In fact, the <a href="http://richzhang.github.io/colorization/">Colorful Image Colorization</a> algorithm is evaluated using a “colorization Turing test,” asking human participants to choose between a generated and ground truth color image. This method successfully fools humans on 32% of the trials.</p><p>A natural next question to ask is if this human <a href="https://en.wikipedia.org/wiki/Turing_test">Turing test</a> can itself be addressed by a neural network. The discriminative network D in a GAN is defined precisely for this task! It is a neural network whose output is the probability that the discriminator D thinks the input data is real or fake. This gives a training goal for G that is aligned with the human Turing test: the generator network G tries to maximize the probability that D cannot distinguish between real and fake data.</p><p>A comparison of <a href="https://arxiv.org/abs/1605.00075">Deep Colorization</a> and a GAN approach to colorization leads to a comparison of the following two training categories:</p><ol><li>Training G to minimize the mean squared error of G(<em>x</em>) and <em>y</em> where <em>x</em> is a grayscale image and <em>y</em> is a ground-truth color image.</li><li>Training G to maximize the probability that the discriminator network D cannot tell the difference between real and fake color images.</li></ol><p>We conjecture that training a GAN is correlated to minimizing the mean squared error between G(<em>x</em>) and <em>y</em>, but a detailed study of this correlation is beyond the scope of this blog post.</p><h3>Our Approach to Colorization</h3><p>We now describe our approach to colorizing satellite imagery. Our generative network for colorization is similar to <a href="https://medium.com/@Hagerty">Patrick Hagerty’s</a> super-resolution neural network [<a href="https://medium.com/the-downlinq/super-resolution-on-satellite-imagery-using-deep-learning-part-1-ec5c5cd3cd2">1</a>,<a href="https://medium.com/the-downlinq/super-resolution-on-satellite-imagery-using-deep-learning-part-2-c9ce41dc0ee0">2</a>,<a href="https://medium.com/the-downlinq/super-resolution-on-satellite-imagery-using-deep-learning-part-3-2e2f61eee1d3">3</a>].</p><h4>Our GAN Architecture</h4><p>We first describe the architecture of our generative network G. Our training network takes as input <em>x</em> (a 64x64 pixel sliding window of a grayscale image) and is comprised of three outer layers. Each outer layer contains two convolutional layers, one deconvolutional layer, and one ReLU layer. The input to an outer layer is a convex combination of the previous outer layer and the 2nd previous outer layer. A picture of generative network is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XzoC1DQKKPhwHBWFzQqbmQ.png" /><figcaption>Image of generative network for colorization</figcaption></figure><p>Next, we describe the discriminator network D. Our discriminator network takes an input <em>y </em>(a 64x64 sliding window of a real or fake color image)<em> </em>and passes this color image through five convolutional layers with 64, 128, 256, 512, 1024 filters respectively. Each convolutional layer is followed by a 2x2 max pooling layer. The last layer of the discriminator network is a fully connected layer (<em>without</em> an activation function) whose output is a single number. This single number is interpreted as the probability that the discriminator network D believes the input color image is real or generated (i.e fake). A picture of the discriminator network is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7_SKoTWDdTSY9LV5UlQB7w.png" /><figcaption>Image of discriminator network for colorization</figcaption></figure><h4>Training our GAN</h4><p>We employ the following to train the networks G and D.</p><p>Let <em>y </em>be a real color image and let <em>x</em> be a grayscale image, so that G(<em>x</em>) is a generated (i.e. fake) color image. Then the discriminator tries to maximize the following function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*ZkGA4DKhg0iM-Bj0jfOODg.png" /></figure><p>Note that trying to maximize this function is equivalent to trying to maximize the probability that D assigns the value 1 to real color images and the value 0 to fake color images.</p><p>The generator tries to minimize the following function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/298/1*24g2h1bDrYEHERHa_aoYiw.png" /></figure><p>Minimizing this function is equivalent to trying to <em>maximize </em>the probability that D confuses a fake color image for a real color image; i.e., the generative model is trying to produce images that are as “realistic” as possible through the lens of D.</p><p>GAN training is implemented using alternating training steps (with stochastic gradient descent) between the discriminator and the generator.</p><p>Since <a href="https://arxiv.org/pdf/1606.03498.pdf">GAN training can be unstable</a>, we attempt to stabilize training by including a regularization term in both the discriminator loss function and the generator loss function. The goal of the regularization term in the discriminator loss function is to keep the discriminator’s weights small, while the goal of the regularization term in the generator loss function is to keep the generated colorizations somewhat close to the ground truth colorizations during early training iterations.</p><p>We train four different colorization GANs on Las Vegas, Paris, Shanghai, and Khartoum. We now display the resulting colorizations of these four models on large GeoTIFF images. Each of the four images below are split down the middle with the real RGB color satellite image on the left and the artificial colorization of a grayscale image on the right. If the difference is difficult to see, that means the GAN is producing a realistic colorization.</p><p>Note that if one were to <em>instead</em> show a real color image and an artificial color image over the same exact area side-by-side, the human eye would try to pick out the differences between the two images. Since the goal of a colorization to produce a plausible colorization, and not necessarily recover the ground truth, this visualization would be counter-productive.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p-RnUKTZRayNetN0UGN6-Q.png" /><figcaption>A satellite image over Las Vegas split down the middle. The left side is a real pansharpened RGB image, while the right side is an artificially colored grayscale (panchromatic) image. If you look closely, you might be able to see small deviations in color down the centerline.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*83NKGFQjIJInQYNPTstRbw.png" /><figcaption>A satellite image over Paris split down the middle. The left side is a real pansharpened RGB image, while the right side is an artificially colored grayscale (panchromatic) image. If you look closely, you might be able to see small deviations in color down the centerline.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uz89IkbR4j_x4VwEKcCDAQ.png" /><figcaption>A satellite image over Shanghai split down the middle. The left side is a real pansharpened RGB image, while the right side is an artificially colored grayscale (panchromatic) image. If you look closely, you might be able to see small deviations in color down the centerline.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dva25ovPb4isRvsb3Ay5KQ.png" /><figcaption>A satellite image over Khartoum split down the middle. The left side is a real pansharpened RGB image, while the right side is an artificially colored grayscale (panchromatic) image. If you look closely, you might be able to see small deviations in color down the centerline.</figcaption></figure><p><strong>Warning:</strong> Although the four large images above look very compelling, some of the finer details of these images have non-naturalistic colorings. For example, below is an example image over Las Vegas whose artificial colorization is clearly non-naturalistic: some of the streets have a strange reddish color and the building rooftops are not uniformly colored.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*Bv_G0ttTXTLAvIB4sJuqeA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/1*mWaXxzp-i_sZrx7c5JTbFQ.png" /><figcaption>Example Las Vegas image with clearly non-naturalistic colorization</figcaption></figure><p>In summary, our colorizations tend to have an airbrushed effect and tend to be somewhat biased towards grayish hues due to an averaging effect.</p><h3>Going Further</h3><p>This blog post represents a first attempt to colorize grayscale satellite imagery via a method that is potentially generalizable to multispectralizing 3-band RGB satellite imagery.</p><p>Because training a GAN is computationally expensive, we have several possible directions for improvement.</p><ul><li>We currently train our colorization GAN on 64x64 pixel-sized sliding windows. Training on larger image windows would allow the network to see even more semantic image information (like an entire building rather than just a piece of a building). This would decrease the effect of the airbrushed quality shown above.</li><li>We stopped training on each city at approximately 200,000 iterations (2–3 days of training on our Nvidia DevBox). We may see significant improvement if we allow our networks to train for millions of iterations.</li><li>It is possible that applying a classification approach to colorization (as described in <a href="http://richzhang.github.io/colorization/">Colorful Image Colorization</a>) would be helpful in creating more vibrant colorizations.</li><li>The methods described to here for generating 3-band color images from 1-band grayscale images should be directly related to problem of generating 8-band multispectral images from 3-band color images. This will be explored in more detail in an upcoming blog post.</li></ul><p><strong>Acknowledgements:</strong> We thank Patrick Hagerty and Adam Van Etten for numerous helpful conversations.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79c8d137e97b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-downlinq/artificial-colorization-of-grayscale-satellite-imagery-via-gans-part-1-79c8d137e97b">Artificial Colorization of Grayscale Satellite Imagery via GANs: Part 1</a> was originally published in <a href="https://medium.com/the-downlinq">The DownLinQ</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Object Segmentation on SpaceNet via Multi-task Network Cascades (MNC)]]></title>
            <link>https://medium.com/the-downlinq/object-segmentation-on-spacenet-via-multi-task-network-cascades-mnc-f1c89d790b42?source=rss-244ab10f73e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/f1c89d790b42</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Lee Cohn]]></dc:creator>
            <pubDate>Tue, 18 Apr 2017 18:46:54 GMT</pubDate>
            <atom:updated>2017-05-19T16:22:11.236Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HW_O5KQlkXny_uDjIh6Mxg.png" /></figure><p>This blog post shares the results of applying the Multi-task Network Cascades (MNC) object segmentation algorithm (available <a href="https://github.com/daijifeng001/MNC">here</a>) to the SpaceNet challenge (available <a href="https://crowdsourcing.topcoder.com/spacenet">here</a>).</p><p>Please note that the algorithm trained here is for research purposes only, and is not actively competing in the current SpaceNet competition.</p><h3>The SpaceNet Competition</h3><p>The second SpaceNet competition asks its participants to submit an algorithm that inputs satellite images (of Las Vegas, Shanghai, Khartoum, and Paris) and outputs polygons of building footprints.</p><p>An algorithm’s proposed building footprint is considered a true positive if its IOU score with a ground truth building footprint is at least 0.5, and if no other proposal has a higher IoU with the ground truth building. For more precise information on how the SpaceNet competition is scored, see <a href="https://medium.com/the-downlinq/the-spacenet-metric-612183cc2ddb">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*pp7YqHgKY1A5IYfhhEMlHw.png" /><figcaption>A SpaceNet satellite image over Paris together with its labels visualized</figcaption></figure><h3>Motivation</h3><p>The <a href="https://arxiv.org/abs/1512.04412">MNC</a> algorithm is a state-of-the-art object segmentation algorithm. MNC won 1st place in the MS COCO 2015 image segmentation competition, garnering wide popularity.</p><p>Segmentation means that in addition to detecting the location of objects in an image, the algorithm further classifies which pixels belong to an object within an image. Two examples of segmentations produced by MNC are provided in the image below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K8ILSNEyFhrOdeiOJ_hfmQ.png" /><figcaption>Image from the original MNC Paper</figcaption></figure><p>Furthermore, at least two submissions to the first SpaceNet competition used MNC, motivating further inquiry into the application of this algorithm to SpaceNet.</p><h3>A Technical Overview of MNC</h3><p>In this section, we include a high-level, technical overview of MNC for the interested reader. None of the ideas in this section are my own, and further details can be found in the original <a href="https://arxiv.org/abs/1512.04412">MNC paper</a>.</p><h4>Shared Convolutional Features Maps</h4><p>The first step of MNC is to run the original image through a convolutional neural network (CNN). The outputs of this CNN are referred to as <em>shared convolutional features maps</em>, and will be the inputs into all later steps of the MNC algorithm.</p><p><em>Aside</em>: The word <em>maps</em> in convolutional features maps means we are explicitly encoding the location of the feature as well as the feature itself.</p><p>A previous generation object detection algorithm (<a href="https://arxiv.org/abs/1311.2524">R-CNN</a>) generates region proposals on the original image, and subsequently runs each region proposal through a CNN. MNC (building on <a href="https://arxiv.org/abs/1504.08083">Fast R-CNN</a>) takes the reverse approach: the whole original image is first run through a CNN, and region proposals are generated using the output features of this CNN. Thus, the convolutional features need only be computed once, which saves time.</p><h4>Multi-Task Learning in MNC</h4><p>Instance-aware semantic segmentation can generally be decomposed into three different but related subtasks: finding bounding boxes of objects, producing pixel-level masks within bounding boxes, and categorizing the masked object. One novel insight of MNC is a twist on multi-task learning that allows for a later stage to depend on the output of an earlier stage.</p><p>Below is an image of 5-stage learning for MNC.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*oMP1YGPMSAcCYuT70DRz_Q.png" /><figcaption>Image taken from the original <a href="https://arxiv.org/abs/1512.04412">MNC paper</a>.</figcaption></figure><p>The inputs to stage 1 are the shared convolutional features, the outputs of a CNN whose input is the original image as discussed above. The outputs of stage 1 are proposed bounding boxes of image objects (B). Producing bounding boxes from convolutional features is via a <a href="https://arxiv.org/abs/1506.01497">Region Proposal Network (RPN)</a>.</p><p>The bounding boxes (B) produced by stage 1 together with the shared convolution features are the inputs into stage 2 which produce pixel-level masks (M).</p><p>The pixel-level masks (M) together with the share convolutional features are the inputs into stage 3 which produce the categories of masked objects as well as updated bounding boxes (B’), from a box regression layer.</p><p>The new bounding boxes (B’) together with the shared convolutional features are used to produce new masks (M’) and then finally new category scores (C’).</p><h3>MNC Results on SpaceNet</h3><p>We trained and evaluated the MNC algorithm separately on each of the four SpaceNet cities (Las Vegas, Paris, Shanghai, and Khartoum) obtaining an average F1 score of 0.57. Our average F1 score should be compared with 0.60, the average F1 score of the current frontrunner of the second SpaceNet competition.</p><p>Our performance score is visualized below in more detail.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TKTLhlp5a10L7BnUJb60lg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F_OOs7sMJOSzZqT8uqVCWA.png" /></figure><h3>A Few Example Images of MNC on SpaceNet</h3><p>We now include several images of MNC applied to test satellite images over Las Vegas. The color white represents a true positive for an IOU threshold of 0.5, the color yellow a false positive, and the color blue a false negative. The numbers printed inside the proposed polygons are the IOU scores for the proposal footprint.</p><p>These images are produced using Topcoder’s visualizer for the SpaceNet challenge linked <a href="http://www.topcoder.com/contest/problem/BuildingDetector2/visualizer_readme.html">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*dj4lDiXZH5rrMRo5Wj-v5A.png" /><figcaption>Example test Las Vegas image</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*PfgTdeJ0FhvM1JropZmaTQ.png" /><figcaption>Example test Las Vegas image</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*G-pkCZxUvT54olP4aW6Y4w.png" /><figcaption>Example test Las Vegas image</figcaption></figure><p><em>Warning</em>: These are example test images of Las Vegas, our highest performing city. Satellite images over other cities look less promising, and there is clearly much work ahead for those of us working with messy satellite data.</p><h3>Limitations of MNC on SpaceNet</h3><p>MNC struggles with satellite images containing:</p><ul><li>Small buildings. Indeed, many false negatives are due to small buildings significantly decreasing the recall score of MNC.</li><li>L-shaped or concave buildings. This trained version of MNC seems to predict very smooth, convex building footprints.</li><li>Buildings with dark rooftop colors that can sometimes be confused with the surrounding area.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*RWf4QzewFppKelYeIhVwVg.png" /><figcaption>MNC struggles with L-shaped buildings on SpaceNet</figcaption></figure><h3>Training MNC on SpaceNet</h3><p>To train MNC on SpaceNet data, one must first convert the SpaceNet tif images to jpg images, and convert the SpaceNet geoJson labels to Pascal VOC SBD labels. This step is discussed in a previous blog post, linked <a href="https://medium.com/the-downlinq/spacenet-labels-to-pascal-voc-sbd-benchmark-release-labels-a7eb9218157a">here</a>. The code for converting SpaceNet labels to Pascal VOC SBD labels is slightly different for the second SpaceNet competition, and is linked <a href="https://github.com/lncohn/spacenet2_to_pascal_voc_sbd">here</a>.</p><p>After this data conversion, one can train the 5-stage MNC model by following the instructions in MNC’s <a href="https://github.com/daijifeng001/MNC">README.md</a>. A docker container suitable for training MNC can be found <a href="https://hub.docker.com/r/lncohn/mnc_updated/">here</a>.</p><p>We train MNC on SpaceNet data for 25,000 iterations with stochastic gradient descent and a base learning rate of 0.0001. Note that the default learning rate for MNC is 0.01. It is possible that a better solution may arise from further adjusting the learning rate and the number of training iterations.</p><h3>Testing MNC on SpaceNet</h3><p>We wish to run our test SpaceNet images through the trained MNC model to generate building footprints and evaluate our algorithm. See the above section “A Few Example Images of MNC on SpaceNet” for pictures of this output.</p><p>First we define the network using the trained model discussed above.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/46c7a1743a1130ee862e24230f23cde0/href">https://medium.com/media/46c7a1743a1130ee862e24230f23cde0/href</a></iframe><p>We run each image in the SpaceNet test directory through the net defined in the gist above. The output of this is 600 proposed bounding boxes, proposed masks, and proposed category scores.</p><p>MNC further processes this list of 600 boxes, masks, and category scores using non-maximal suppression (NMS) to remove redundant object detections. That is, a set of bounding boxes with high overlap are most likely detecting the same ground truth object. NMS picks one bounding box to represent this set.</p><p>We are left with a new, shorter list of bounding boxes and masks, which we take to be the output of the trained MNC algorithm.</p><h4>Post-Processing SpaceNet Test Data</h4><p>Finally, one can convert this new, shorter list of bounding boxes and masks to geoJson format using <a href="http://www.gdal.org/gdal_polygonize.html">gdal.Polygonize</a>.</p><p>One then converts the entire directory of geoJson proposals into a CSV file using the SpaceNet utility <a href="https://github.com/SpaceNetChallenge/utilities/blob/master/python/createCSVFromGEOJSON.py">createCSVFromGEOJSON.py</a>.</p><p>Our proposals are now in the correct format to submit to the SpaceNet competition!</p><p>We include a github repository for the test-time inference and post-processing <a href="https://github.com/lncohn/pascal_to_spacenet">here</a>.</p><h3>Going Further</h3><p>We include several ideas for further research with MNC on SpaceNet:</p><ul><li>MNC generates region proposals at three different scales and three different aspect rations. One could add a fourth, smaller scale to possibly aid in detecting small objects, currently false negatives of MNC.</li><li>One could increase the number of regional proposals before applying non-maximal suppression to possibly increase the recall score for MNC.</li><li>One could vary the total number of training iterations and the base learning rate to optimize both of these parameters. This is computationally very expensive.</li><li>Include 8-band satellite imagery into the training of MNC. A priori, 8-band imagery has more information than standard RGB imagery. Hence, 8-band imagery has potentially more information to exploit during machine learning.</li></ul><h3>Summary</h3><p>In this blogpost we outlined the pre-processing, training, inference, post-processing, and the results of MNC applied to the second SpaceNet competition. Our results appear to be competitive with current SpaceNet competition submissions.</p><h3><strong>Acknowledgements</strong></h3><p>We thank Adam Van Etten, Patrick Hagerty, and David Lindenbaum for many helpful conversations regarding SpaceNet and object detection.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f1c89d790b42" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-downlinq/object-segmentation-on-spacenet-via-multi-task-network-cascades-mnc-f1c89d790b42">Object Segmentation on SpaceNet via Multi-task Network Cascades (MNC)</a> was originally published in <a href="https://medium.com/the-downlinq">The DownLinQ</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SpaceNet Labels To Pascal VOC SBD Benchmark Release Labels]]></title>
            <link>https://medium.com/the-downlinq/spacenet-labels-to-pascal-voc-sbd-benchmark-release-labels-a7eb9218157a?source=rss-244ab10f73e9------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7eb9218157a</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Lee Cohn]]></dc:creator>
            <pubDate>Tue, 21 Mar 2017 15:51:40 GMT</pubDate>
            <atom:updated>2017-05-19T16:17:13.916Z</atom:updated>
            <content:encoded><![CDATA[<p>This blogpost serves as an introduction to the Pascal VOC SBD benchmark release MATLAB labels (linked <a href="http://home.bharathh.info/pubs/codes/SBD/download.html">here</a>), as well as one approach to converting the SpaceNet geoJson vector labels into the Pascal VOC SBD benchmark release MATLAB format. My hope is that this blogpost and GitHub repository (linked at the bottom of the post) facilitates the application of algorithms developed for the Pascal VOC SBD image competition to the 1st SpaceNet competition linked <a href="https://crowdsourcing.topcoder.com/spacenet">here</a>.</p><p>Two such algorithms of interest are <em>Instance-aware Semantic Segmentation via Multi-task Network Cascades</em> (linked <a href="https://github.com/daijifeng001/MNC">here</a>) and <em>Semantic Countours from Inverse Detectors</em> (linked <a href="http://home.bharathh.info/pubs/pdfs/BharathICCV2011.pdf">here</a>).</p><p><em>Warning</em>: This conversion is for the first SpaceNet competition. For the second SpaceNet competition, see <a href="https://github.com/lncohn/pascal_to_spacenet_2">here</a>.</p><h4>Downloading the Data</h4><p>The SBD benchmark release is available for download <a href="http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/semantic_contours/benchmark.tgz">here</a>, and the SpaceNet data is available for download <a href="https://aws.amazon.com/public-datasets/spacenet/">here</a>.</p><h4>Exploring the Pascal VOC SBD Labels</h4><p>Let us first recall some general information regarding the Pascal VOC SBD benchmark release.</p><p>The SBD benchmark release is bundled together in a directory called ‘dataset’. There are three subdirectories:</p><ul><li><strong>img</strong></li></ul><p>This directory contains all of the images in jpg format.</p><p><em>Aside</em>: Since the SpaceNet images are in tiff format, I used the ImageMagick mogrify command available <a href="http://www.imagemagick.org/Usage/basics/#mogrify">here</a> to batch convert all of the tiff files to jpg files.</p><ul><li><strong>cls</strong></li></ul><p>This directory contains category-specific segmentation and boundary information for each image.</p><p><em>Aside</em>: The Pascal VOC competition labels objects from 20 different categories: ‘aeroplane’, ‘bicycle’, ‘bird’, ‘boat’, ‘bottle’, ‘bus’, ‘car’, ‘cat’, ‘chair’, ‘cow’, ‘diningtable’, ‘dog’, ‘horse’, ‘motorbike’, ‘person’, ‘pottedplant’, ‘sheep’, ‘sofa’, ‘train’, and ‘tvmonitor’.</p><p>There is one .mat file for each image containing class information, and each .mat file contains a MATLAB struct called GTcls with 3 fields:</p><ul><li>GTcls.Segmentation is a single two-dimensional matrix containing the class segmentation information. Matrix entries that correspond to pixels that belong to category k have value k, and matrix entries corresponding to pixels that do not belong to any category have the value 0.</li><li>GTcls.Boundaries is a MATLAB cell array. GTcls.Boundaries[k] is a MATLAB sparse matrix that contains the boundaries of the k-th category.</li><li>GTcls.CategoriesPresent is a row vector of all the categories that are present in the jpg image.</li><li><strong>inst</strong></li></ul><p>This directory contains instance-specific segmentations and boundaries. There is one .mat file for each image containing instance information, and each .mat file contains a MATLAB struct called GTinst with 3 fields:</p><ul><li>GTinst.Segmentation is a single two-dimensional matrix containing segmentation information. Matrix entries that correspond to pixels belonging to the i-th instance have value i.</li><li>GTinst.Boundaries is a cell array. GTinst.Boundaries[i] is a MATLAB sparse matrix that contains the boundary of the i-th instance.</li><li>GTinst.Categories is a <em>column</em> vector with as many entries as there are instances. Each entry is the category label of the i-th instance.</li></ul><p>There are in addition two text files, train.txt and val.txt, containing the names (<em>without the extension</em>) of the training images and validation images respectively.</p><p><em>Aside</em>: I do not own or have access to a MATLAB license. I did my data exploration using Octave, which has the ability to load .mat files. All of my data manipulation was done in python using scipy.io.loadmat and scipy.io.savemat to load and save .mat files into python numpy arrays. Thus, one does not need MATLAB to engage with these labels.</p><p>Here is an example of loading an instance .mat file and retrieving the three fields of the GTinst struct in python.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lNDHLCZhsnKEzipNkCoHbQ.png" /></figure><p><em>Remark</em>: Observe that the entries of matrices (sparse or not) are of type unsigned 8-bit integers.</p><h4>SpaceNet geoJson Labels</h4><p>Let us now take a look at an example SpaceNet geoJson file, as this (together with its corresponding image) will be what we wish to transform into the MATLAB data structures presented above.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4558ba191cb13561513d0572280554c6/href">https://medium.com/media/4558ba191cb13561513d0572280554c6/href</a></iframe><p>The building label is stored as a Polygon with five pairs of latitudes and longitudes giving the vertices of the label. This is fundamentally different than the labels used in the Pascal VOC competition. In particular, the SpaceNet geoJson labels record the vertices of a building label in terms of latitude and longitude, while the Pascal VOC labels record object labels in a variety of pixel-level matrices.</p><p>Given a latitude and longitude building vertex, one can try to convert this vertex into its corresponding matrix location in a pixel-level representation of a satellite image. But what if the longitude and latitude map to somewhere in between two pixels in the satellite image? Which pixel should represent the vertex of the building? Such problems always arise when converting continuous data (like latitudes and longitudes) to discrete data (like finite matrices representing pixel locations).</p><p>Thus, there is no canonical way to transform a SpaceNet geoJson label to its corresponding Pascal VOC .mat labels. That is, this transformation of labels will depend on choices.</p><p>To solve this problem (that is, make one choice), we will use a distance transform that takes a raster source (tif) and vector source (geoJson file) and returns a matrix whose values are positive if they are inside a building segmentation label and negative if they are outside a building segmentation label. The matrix value is zero if it is on the boundary of a building segmentation label.</p><p>The distance transform has appeared in other CosmiQ projects including the SpaceNet utilities repository available <a href="https://github.com/SpaceNetChallenge/utilities">here</a>, Adam Van Etten’s blog on getting started with SpaceNet available <a href="https://medium.com/the-downlinq/getting-started-with-spacenet-data-827fd2ec9f53#.7vxrt4h7s">here</a>, and Patrick Hagerty’s blog on object detection in SpaceNet available <a href="https://medium.com/the-downlinq/object-detection-on-spacenet-5e691961d257#.84vfixm83">here</a>.</p><p>The code below shows the distance transform as well as how to build class and instance segmentation functions given the distance transform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*LVr125vr42xbp52yZ_FEyQ.tiff" /><figcaption>Image to be processed below</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q7YFK9UdLEdEjDrsBvFrFw.png" /></figure><h4>GitHub Repository</h4><p>We also include a GitHub repository containing the file ‘<em>spacenet_labels_dir_to_voc_labels_dir.py</em>’</p><p>One can run the command:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/77feb13b67ac06debfe2c79285066223/href">https://medium.com/media/77feb13b67ac06debfe2c79285066223/href</a></iframe><p>in a directory containing a folder of raster tif files, a folder of vector geoJson files, an empty class labels directory, and another empty instance labels directory. This will process all of the labels at once.</p><p>This GitHub repository is available at: <a href="https://github.com/lncohn/spacent_to_pascal">https://github.com/lncohn/spacent_to_pascal</a></p><p>In summary, this allows for any algorithm developed for the Pascal VOC SBD benchmark release competition to be applied to the SpaceNet data competition.</p><p>Acknowledgements: The author thanks Patrick Hagerty, David Lindenbaum, and Adam Van Etten for helpful discussions regarding SpaceNet and the distance transform.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7eb9218157a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-downlinq/spacenet-labels-to-pascal-voc-sbd-benchmark-release-labels-a7eb9218157a">SpaceNet Labels To Pascal VOC SBD Benchmark Release Labels</a> was originally published in <a href="https://medium.com/the-downlinq">The DownLinQ</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>