<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Daniel Hogan on Medium]]></title>
        <description><![CDATA[Stories by Daniel Hogan on Medium]]></description>
        <link>https://medium.com/@daniel_hogan?source=rss-ecbbd25b9fd2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*8XpNVL18hghcUMPCVhtN1Q.jpeg</url>
            <title>Stories by Daniel Hogan on Medium</title>
            <link>https://medium.com/@daniel_hogan?source=rss-ecbbd25b9fd2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 17:14:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@daniel_hogan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Closer Look at the SpaceNet Change and Object Tracking (SCOT) Metric]]></title>
            <link>https://medium.com/the-downlinq/a-closer-look-at-the-spacenet-change-and-object-tracking-scot-metric-44c6b5c07791?source=rss-ecbbd25b9fd2------2</link>
            <guid isPermaLink="false">https://medium.com/p/44c6b5c07791</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Daniel Hogan]]></dc:creator>
            <pubDate>Mon, 08 Feb 2021 17:41:15 GMT</pubDate>
            <atom:updated>2021-02-08T17:53:45.529Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Preface: </em><a href="https://spacenet.ai/"><em>SpaceNet LLC</em></a><em> is a nonprofit organization dedicated to accelerating open source, artificial intelligence applied research for geospatial applications, specifically foundational mapping (i.e., building footprint &amp; road network detection). SpaceNet is run in collaboration by co-founder and managing partner </em><a href="http://www.cosmiqworks.org/"><em>CosmiQ Works</em></a><em>, co-founder and co-chair </em><a href="https://www.maxar.com/"><em>Maxar Technologies</em></a><em>, and our partners including </em><a href="https://aws.amazon.com/"><em>Amazon Web Services (AWS)</em></a><em>, </em><a href="https://www.capellaspace.com/"><em>Capella Space</em></a><em>, </em><a href="https://www.topcoder.com/"><em>Topcoder</em></a><em>, </em><a href="http://www.grss-ieee.org/"><em>IEEE GRSS</em></a><em>, the </em><a href="https://www.nga.mil/"><em>National Geospatial-Intelligence Agency</em></a><em> and </em><a href="https://www.planet.com/"><em>Planet</em></a><em>.</em></p><p>The recent SpaceNet 7 Challenge introduced the <a href="https://medium.com/the-downlinq/the-spacenet-change-and-object-tracking-scot-metric-5339b0c904c">SpaceNet Change and Object Tracking (SCOT) metric</a>, an evaluation metric for geospatial time series. In this post, we’ll use the results of SpaceNet 7 to explore how the metric acts with real-world data. We want to see whether the different parts of the metric really measure what they claim to measure. (Short answer: they do!)</p><p>The <a href="https://medium.com/the-downlinq/the-spacenet-7-multi-temporal-urban-development-challenge-dataset-release-9e6e5f65c8d5">SpaceNet 7 dataset</a> contains data for more than 100 distinct regions (“areas of interest” or AOIs) around the world. For each AOI, there is a deep stack of time series data, with one image per month for a period of about two years. The imagery is accompanied by building footprint labels that include an ID number for each building that persists from one month to the next, making it possible to track buildings over time and identify any new buildings each month. In the <a href="https://medium.com/the-downlinq/announcing-spacenet-7-the-multi-temporal-urban-development-challenge-53f68e24b9fa">SpaceNet 7 Challenge</a>, participants were asked to find the building footprints in each image and also to provide their own set of internally-consistent ID numbers, with the same building ideally getting the same ID number in every image where it appears.</p><p>Participants’ submissions were scored using the aforementioned SCOT metric. The metric compares a proposed set of footprints (with ID numbers) against ground truth, and it evaluates the proposal using two different terms: a tracking term and a change detection term. The tracking term measures how well the proposal tracks buildings from month to month. This term is basically an F1 score, but with a built-in penalty for inconsistent ID numbers. On the other hand, the change detection term measures how well the proposal identifies newly-built buildings, the most common change in this dataset. This term is also basically an F1 score, but it only uses building footprints making their chronological first appearance. Once the tracking and change detection terms are calculated for an AOI, their weighted harmonic mean gives the overall SCOT score for that AOI. To get the score for a collection of AOIs, a simple average of their individual SCOT scores is used. So that’s the math, but what does it mean in real life? Let’s find out.</p><h3>Change versus Tracking</h3><p>To explore the relationship between SCOT’s tracking and change detection terms in a model-independent way, we’ll use six different models: the <a href="https://medium.com/the-downlinq/the-spacenet-7-multi-temporal-urban-development-challenge-announcing-the-winners-847a9a53004c">five prizewinners</a> in SpaceNet 7 plus the SpaceNet 7 <a href="https://medium.com/the-downlinq/the-spacenet-7-multi-temporal-urban-development-challenge-algorithmic-baseline-4515ec9bd9fe">algorithmic baseline</a> model. For each model, the SCOT metric is used to measure the model’s performance on each of the 21 AOIs in SpaceNet 7’s held-back private test set. That gives 126 comparison points (6 models x 21 AOIs), and Figure 1 shows the change score versus tracking score for each one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*hpOLvFCOZo7EomLy8Zy0qw.png" /><figcaption>Figure 1. SCOT change detection score versus tracking score for each of 6 models and 21 AOIs. The color of each point indicates the model, as shown in the legend. Contour lines are lines of constant SCOT score for β=2.</figcaption></figure><p>In Figure 1, the contour lines are lines of constant SCOT score. A careful inspection shows that the contours are not quite symmetric with respect to the two axes. That’s because the SCOT score is a weighted harmonic mean, and SpaceNet 7 used a non-equal weight (β=2, defined <a href="https://medium.com/the-downlinq/the-spacenet-change-and-object-tracking-scot-metric-5339b0c904c">here</a>) to prioritize the tracking score, thereby creating the observed asymmetry.</p><p>Turning our attention from the contour lines to the data itself, the first observation is evident: The median tracking score is much higher than the median change detection score, by a factor of about four. This may relate to the fact that a model can “recover” from a tracking error by being consistent in the subsequent time steps. But if a model misidentifies the time of construction, there are no opportunities for second chances.</p><p>The variation around the mean shows that, on average, higher tracking scores correlate with higher change detection scores. That’s not surprising — on the whole some models are better than others and some AOIs are easier than others, and both tracking and change detection can be expected to benefit from that. One difference among the models stands out in the plot: the five winning models noticeably outperform the baseline algorithm in both change and tracking. But even though the first-place model gets a SCOT score 33% higher than the fifth-place model, the data points from the prizewinners do not cluster by model. In other words, the variation among the AOIs for each model is not small compared to the variation among the prizewinning models’ averages. This same phenomenon was previously seen in, for example, <a href="https://medium.com/the-downlinq/spacenet-5-results-deep-dive-part-2-apls-to-apls-2b1fc7307629">SpaceNet 5</a>.</p><p>Given the first-order correlation between change and tracking, do we really need both? When it comes to identifying the winner of SpaceNet 7, the differences do matter. Ranking the top five models by just the change detection score, instead of the complete SCOT score, would have swapped the order of 3rd and 4th place. The overall competition winner, lxastro0, rode to victory on the strength of a strong change detection score. If the winner had been decided by tracking alone, that submission would have instead come in 3rd. And the situation gets more interesting when we start looking at individual AOIs.</p><h3>Regional Variation</h3><p>Next, the data can be grouped by region, in other words, by AOI. We assign an arbitrary integer to each of the 21 AOIs in the private test set, and Figure 2 shows the performance of the five winning models, with each point labeled not by model but by AOI. To emphasize the distribution, a shaded polygon marks the convex hull of each AOI’s points. We leave out the baseline model here because its across-the-board lower scores obscure the inter-AOI variation we’re interested in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*Vw18g4NSQyI6NmGMHcAZbw.png" /><figcaption>Figure 2. SCOT change detection score versus tracking score for the five winning models. Points are labeled by AOI, and shaded polygons surround the five points (one per model) for each AOI.</figcaption></figure><p>Despite a variety of <a href="https://medium.com/the-downlinq/the-spacenet-7-multi-temporal-urban-development-challenge-announcing-the-winners-847a9a53004c">neural network architectures</a> and <a href="https://medium.com/the-downlinq/spacenet-7-results-tis-the-season-fadc722fdff0">data processing approaches</a>, all five models often produce tracking and change detection scores in the vicinity of other models for the same AOI. That suggests that the data’s departure from a strict one-to-one relationship between tracking score and change detection score is reflecting something intrinsic to the challenges posed by each AOI, rather than anything strictly model-dependent.</p><p>Figure 2 also gives a hint at where we can look next to get some clues about what’s going on. AOI 20 (in the upper left part of the figure) has consistently high change detection scores, despite having some of the lowest tracking scores. For AOI 12 (in the lower right), the opposite is true.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*exJnanT2JfLW5QdXFJCXAw.png" /><figcaption>Figure 3. Part of AOI 20 (a) just before and (b) just after a burst of construction activity. Building footprints identified by the winning model are shaded in white, false negatives in blue, and false positives in yellow.</figcaption></figure><p>In the case of AOI 20, a closer inspection shows that many buildings fall into one of two extremes: those that are detected every month starting exactly when they were built, and those that are never detected at all. Figure 3 shows that immediately after construction, some buildings are detected while others aren’t. Over the following months, this distribution of true positives and false negatives persists in much the same way. As a result, the change detection score is almost as large as the tracking score, since so many of the tracked buildings get their time of first appearance correctly identified.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/1*LES0Hd6Wqrhwn1peoswzKA.png" /><figcaption>Figure 4. The same detail in back-to-back images from AOI 12. Images (a) and (b) show just the imagery from adjacent months, while images (c) and (d) overlay building footprints using the same color scheme as Figure 3. Five proposal footprints have no ground truth match in (c) but gain a match in (d).</figcaption></figure><p>AOI 12, however, is a different story. In this region, the appearance of new buildings is frequently preceded by one or more months of what may be site preparation and/or construction activity. This often causes colors and textures that appear indistinguishable from those of a finished building. As a result, models frequently jump the gun and generate a building too early, as shown in Figure 4. (It is also possible that the ground truth labels are simply wrong, in which case the difficulty stems from this AOI’s labels, not its imagery.)</p><p>As these examples illustrate, difficulties related to the timing of construction most heavily drag down the change detection score, while difficulties that persist throughout the time series can weigh down the tracking score. The ability of the change detection score to identify time-sensitive problems around first appearance, and the corresponding ability of the tracking score to identify problems persisting in time, makes them complimentary measures of model success that both contribute to the holistic measure that is the SCOT metric.</p><p>This concludes our series analyzing the results of SpaceNet 7. Previous articles are <a href="https://medium.com/the-downlinq/the-spacenet-7-multi-temporal-urban-development-challenge-announcing-the-winners-847a9a53004c">here</a>, <a href="https://medium.com/the-downlinq/spacenet-7-results-tis-the-season-fadc722fdff0">here</a>, and <a href="https://medium.com/the-downlinq/spacenet-7-results-overachieving-pixels-6d03947b8a05">here</a>, and look for the open-sourcing of the winning models coming soon.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44c6b5c07791" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-downlinq/a-closer-look-at-the-spacenet-change-and-object-tracking-scot-metric-44c6b5c07791">A Closer Look at the SpaceNet Change and Object Tracking (SCOT) Metric</a> was originally published in <a href="https://medium.com/the-downlinq">The DownLinQ</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deploying the SpaceNet 7 Baseline on AWS]]></title>
            <link>https://medium.com/the-downlinq/deploying-the-spacenet-7-baseline-on-aws-df756a4d9695?source=rss-ecbbd25b9fd2------2</link>
            <guid isPermaLink="false">https://medium.com/p/df756a4d9695</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Daniel Hogan]]></dc:creator>
            <pubDate>Fri, 25 Sep 2020 13:43:27 GMT</pubDate>
            <atom:updated>2020-09-25T13:43:27.296Z</atom:updated>
            <content:encoded><![CDATA[<p>Adapted by <a href="https://medium.com/@daniel_hogan">Daniel Hogan</a> from a post by <a href="https://medium.com/@avanetten">Adam Van Etten</a> and <a href="https://medium.com/@nweir">Nick Weir</a></p><p><em>Preface: </em><a href="https://spacenet.ai/"><em>SpaceNet LLC</em></a><em> is a nonprofit organization dedicated to accelerating open source, artificial intelligence applied research for geospatial applications, specifically foundational mapping (i.e., building footprint &amp; road network detection). SpaceNet is run in collaboration by co-founder and managing partner </em><a href="http://www.cosmiqworks.org/"><em>CosmiQ Works</em></a><em>, co-founder and co-chair </em><a href="https://www.maxar.com/"><em>Maxar Technologies</em></a><em>, and our partners including </em><a href="https://aws.amazon.com/"><em>Amazon Web Services (AWS)</em></a><em>, </em><a href="https://www.capellaspace.com/"><em>Capella Space</em></a><em>, </em><a href="https://www.topcoder.com/"><em>Topcoder</em></a><em>, </em><a href="http://www.grss-ieee.org/"><em>IEEE GRSS</em></a><em>, the </em><a href="https://www.nga.mil/"><em>National Geospatial-Intelligence Agency</em></a><em> and </em><a href="https://www.planet.com/"><em>Planet</em></a><em>.</em></p><p>The <a href="https://medium.com/the-downlinq/announcing-spacenet-7-the-multi-temporal-urban-development-challenge-53f68e24b9fa">SpaceNet 7 Challenge</a> centers on a unique time-series <a href="https://medium.com/the-downlinq/the-spacenet-7-multi-temporal-urban-development-challenge-dataset-release-9e6e5f65c8d5">dataset</a>, with monthly collects over an approximately-two-year period for more than 100 locations spanning the globe. Challenge participants are asked to identify building footprints in the imagery and, furthermore, to track buildings from month to month revealing the temporal structure of the data. The challenge is currently underway. With a Challenge end date of October 28, now is a great time to <a href="https://www.topcoder.com/challenges/21ba3d19-3f7a-4abc-8d28-7c932887f0f6">sign up</a>!</p><p>Deep learning techniques, which have increasingly become the norm for complex tasks like this, require powerful hardware usually in the form of graphical processing units (GPUs). To help democratize access to the tools needed for the SpaceNet 7 Challenge, this blog post shows step-by-step how to run our <a href="https://medium.com/the-downlinq/the-spacenet-7-multi-temporal-urban-development-challenge-algorithmic-baseline-4515ec9bd9fe">baseline (i.e., example) solution</a> online with <a href="https://aws.amazon.com/">Amazon Web Services</a> (AWS). No special hardware on the user’s part is required, and the total cost to train the model is around $60. To further democratize the process, the SpaceNet team is providing AWS compute credits to SpaceNet 7 competitors. Specifically, the first 100 competitors to reproduce (or exceed) the score of the open source baseline model will receive a $250 credit, thus allowing a large number of competitors to freely experiment and improve upon their submissions.</p><p><strong>I. Introduction</strong></p><p>To lower the barrier to entry for SpaceNet 7 participation, the CosmiQ Works team created an Amazon Machine Image (AMI) pre-loaded with the <a href="https://github.com/CosmiQ/solaris">Solaris</a> software suite, SpaceNet 7 <a href="https://github.com/CosmiQ/CosmiQ_SN7_Baseline">baseline</a> algorithm, and the <a href="https://spacenet.ai/sn7-challenge/">SpaceNet 7</a> dataset. This post walks through how to use that AMI to train and test the baseline model in the cloud. No hardware beyond an ordinary laptop is required. Challenge participants have the option of using the baseline as a starting point as they set off to develop their own innovative solutions to the SpaceNet 7 Challenge.</p><p><strong>II. Loading the AMI</strong></p><p>In order to load and prepare the AMI for model training and testing, simply execute the following steps:</p><ol><li><strong>In a web browser navigate to:</strong></li></ol><pre><a href="https://console.aws.amazon.com/ec2">https://console.aws.amazon.com/ec2</a></pre><p>If you don’t have an AWS account, create one as a “Root user” and login.</p><p><strong>2. In the top-right of the page, ensure you are in the “N. Virginia” region:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WS42qBz4vlZ8o4KW.png" /></figure><p><strong>3. Select “Launch Instance” (orange button)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5KyE4DE6hcllHTCI.png" /></figure><p><strong>4. Search for the pre-built AMI</strong></p><p>Type “CosmiQ_SpaceNet7_Baseline_v1” in the search bar, click the “Community AMIs” tab, and hit “Select.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SFPB8cucni9KU74sNIZePQ.png" /></figure><p><strong>5. Select the appropriate instance</strong></p><p>We recommend the p3.2xlarge instance, which includes one NVIDIA P100 GPU and costs $3.06 per hour. Hit “Review and Launch.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_DkOO3OUbShThmnL.png" /></figure><p><strong>6. Initiate launch</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uzxH3aTlhMuy-Dlw1eQKtg.png" /></figure><p><strong>7. Create a new Key Pair</strong></p><p>Create a new key pair (<em>e.g</em>. cosmiq_sn7_baseline), and save to your local machine. Hit “Launch Instances.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PTrcYKixEN4ECIY_gOJyKA.png" /></figure><p>Note: It is possible that AWS may throw an error stating that you have requested more vCPU capacity than allowed (this is an intentional feature put in place to keep users from unwittingly accumulating large bills). In this case, open a ticket for a service limit increase at: <a href="https://console.aws.amazon.com/support/home#/case/create">https://console.aws.amazon.com/support/home#/case/create</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Iejo7diletLClCif.png" /></figure><p><strong>8. Record the address</strong></p><p>The instance is now running! Back at the Instance Dashboard you should write down the Public DNS (IPv4) for this instance (of form: ec2-XX-XXX-X-XX.compute-1.amazonaws.com), you will need this later to access the instance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yZW-u5L_z-bqk9Rs.png" /></figure><p><strong>9. Access the instance via SSH</strong></p><p>On your local machine, <a href="https://en.wikipedia.org/wiki/Secure_Shell">ssh</a> into the AMI from a terminal. You may need to change permissions of the key (<em>.pem</em>) file first.</p><pre>chmod 400 /path_to_keys/cosmiq_sn7_baseline.pem</pre><pre>ssh -i &quot;/path_to_keys/cosmiq_sn7_baseline.pem&quot; ubuntu@ec2-XX-XX-XX-XX.compute-1.amazonaws.com</pre><h3>III. Train and Test the Baseline Model</h3><p>The GPU instance is now up and running, so we can train and test the SpaceNet 6 baseline algorithm, which is pre-installed in a <a href="https://www.docker.com/">Docker</a> container (entitled <em>cqw_sn6_bl</em>) in the AMI. The command below should be run in the ssh terminal of your instance.</p><ol><li><strong>Attach the docker container</strong></li></ol><pre>nvidia-docker start cqw_sn7_bl</pre><pre>nvidia-docker attach cqw_sn7_bl</pre><p>This puts us in the the <em>/root </em>directory within the docker container where we can execute the train.sh and test.sh scripts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*roqNJFevMmEaz-JXHG3qiQ.png" /></figure><p><strong>2. Train the model</strong></p><p>By default the script runs for 300 epochs (~20 hours). To modify the training time, simply edit the “epochs” line in the <em>yml/sn7_baseline_train.yml</em> file.<em> </em>The training script first pre-processes the data, then applies a VGG-16 + U-Net deep learning model. Training is launched by simply invoking the <em>train.sh</em> script:</p><pre>./train.sh /data/train</pre><p>Training will proceed either until completion of all epochs, or training is terminated by the user (<em>ctrl+c</em>). The best model is saved as training progresses, so early termination still saves the best model trained to date.</p><p><strong>3. Test the Model</strong></p><p>Testing can be executed either when training is complete (Step 2 above), or using the existing weights file <em>models/sn7_baseline/xdxd_final.pth</em> in the docker container. Testing is invoked with the following command (<em>sn7_baseline_predictions.csv </em>is the name of the output file):</p><pre>./test.sh /data/test_public sn7_baseline_predictions.csv</pre><p><strong>4. Copy outputs back locally</strong></p><p>Inference is now complete, so we now transfer the outputs out of the AMI back to our local machine. First, switch out of the Docker container (ctrl+p ctrl+q) then copy the results from the docker container back to the AMI:</p><pre>mkdir /home/ubuntu/transfer</pre><pre>docker cp cqw_sn7_bl:/root/sn7_baseline_predictions.csv /home/ubuntu/transfer/</pre><pre>docker cp cqw_sn7_bl:/root/output /home/ubuntu/transfer/</pre><pre>docker cp cqw_sn7_bl:/root/inference_out /home/ubuntu/transfer/</pre><p>Now copy results back to your local machine (for example, to location: <em>/path_to_results/sn7_baseline_aws/</em>). To just get the CSV file of predictions, the command is:</p><pre>scp -i &quot;/path_to_keys/cosmiq_sn7_baseline.pem&quot; -r ubuntu@ec2-XX-XX-XX-XX.compute-1.amazonaws.com:/home/ubuntu/transfer/sn7_baseline_predictions.csv /path_to_results/sn7_baseline_aws/</pre><p>Or, to get all the intermediate and output files (masks, etc.), use:</p><pre>scp -i &quot;/path_to_keys/cosmiq_sn7_baseline.pem&quot; -r ubuntu@ec2-XX-XX-XX-XX.compute-1.amazonaws.com:/home/ubuntu/transfer /path_to_results/sn7_baseline_aws/</pre><p><strong>5. Inspect Results</strong></p><p>Among the files generated by the SpaceNet 7 baseline are images identifying which pixels are most likely to be parts of buildings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*JBfNNUj7neVW6RDnfxIoaw.png" /><figcaption>An image from the public_test data (left) and the model’s assessment of which pixels are most likely to be parts of buildings (right). Applying a threshold to the image at right generates a prediction mask.</figcaption></figure><p>The baseline also creates a CSV file with a geometry and a tracking ID number for each building prediction (see below) — this file yields a score of ~0.158 when entered onto the <a href="https://www.topcoder.com/spacenet">challenge website</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/797/1*3OlA6lggpPXfAShyU5ObcA.png" /><figcaption>SpaceNet baseline building footprint coordinates in challenge entry format.</figcaption></figure><p><strong>Side Note</strong></p><p>As an aside, the AMI was built using a <a href="https://github.com/dphogan/CosmiQ_SN7_Baseline">fork</a> of the <a href="https://github.com/CosmiQ/CosmiQ_SN7_Baseline">original</a> baseline repo. The original repo runs many of the computations within Jupyter Python notebooks, which the fork repackages as Python scripts. But under the hood, both versions do the same computations in the same way.</p><h3>IV. Conclusion</h3><p>The SpaceNet 7 Challenge focuses on the difficulties and opportunities of working with time series data. To make this task as approachable as possible, we have previously released a working example solution in the form of the algorithmic baseline. In this post, we’ve released an Amazon Machine Image with everything necessary to use the baseline, along with step-by-step instructions to undertake the deep learning process in the cloud. The baseline and this platform can be an effective way to get started on SpaceNet 7. We encourage the interested reader to <a href="https://go.topcoder.com/spacenet/">register for the challenge</a>, implement this code, and apply for the free AWS compute credits to experiment with and improve upon the baseline model.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df756a4d9695" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-downlinq/deploying-the-spacenet-7-baseline-on-aws-df756a4d9695">Deploying the SpaceNet 7 Baseline on AWS</a> was originally published in <a href="https://medium.com/the-downlinq">The DownLinQ</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The SpaceNet Change and Object Tracking (SCOT) Metric]]></title>
            <link>https://medium.com/the-downlinq/the-spacenet-change-and-object-tracking-scot-metric-5339b0c904c?source=rss-ecbbd25b9fd2------2</link>
            <guid isPermaLink="false">https://medium.com/p/5339b0c904c</guid>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Daniel Hogan]]></dc:creator>
            <pubDate>Thu, 13 Aug 2020 19:37:41 GMT</pubDate>
            <atom:updated>2020-08-13T19:37:41.485Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dihjVuYDeWaSvQcwNZAa1w.png" /></figure><p><a href="https://medium.com/@daniel_hogan">Daniel Hogan</a> and <a href="https://medium.com/@avanetten">Adam Van Etten</a></p><p><em>Preface: </em><a href="https://spacenet.ai/"><em>SpaceNet LLC</em></a><em> is a nonprofit organization dedicated to accelerating open source, artificial intelligence applied research for geospatial applications, specifically foundational mapping (i.e. building footprint and road network detection). SpaceNet is solely managed by co-founder, </em><a href="https://www.iqt.org/"><em>In-Q-Tel </em></a><a href="http://www.cosmiqworks.org/"><em>CosmiQ Works</em></a><em>, in collaboration with co-founder and co-chair, </em><a href="https://maxar.com/?utm_source=partner_blog&amp;utm_medium=medium&amp;utm_campaign=spacenet4th"><em>Maxar Technologies</em></a><em>, and the other partners:</em> <a href="https://aws.amazon.com/"><em>Amazon Web Services (AWS)</em></a><em>, </em><a href="https://www.capellaspace.com/"><em>Capella Space</em></a><em>, </em><a href="https://www.topcoder.com/"><em>Topcoder</em></a><em>, </em><a href="http://www.grss-ieee.org/"><em>Institute of Electrical and Electronics Engineers (IEEE) Geoscience and Remote Sensing Society (GRSS),</em></a> <em>the </em><a href="https://www.nga.mil/"><em>National Geospatial-Intelligence Agency</em></a> (NGA)<em> and </em><a href="https://www.planet.com/"><em>Planet</em></a><em>.</em></p><p>The SpaceNet 7 Multi-Temporal Urban Development Challenge has the ambitious goal of tracking precise building addresses and urban change from satellite imagery. As detailed in our <a href="https://medium.com/the-downlinq/announcing-spacenet-7-the-multi-temporal-urban-development-challenge-53f68e24b9fa">announcement blog</a>, the goal of SpaceNet 7 is relevant to numerous human development and disaster response applications. Furthermore, the unique <a href="https://medium.com/the-downlinq/the-spacenet-7-multi-temporal-urban-development-challenge-dataset-release-9e6e5f65c8d5">SpaceNet 7 dataset</a> poses a challenge from a computer vision standpoint because of the small pixel area of each object, the high object density within images, and the dramatic image-to-image difference compared to frame-to-frame variation in video object tracking.</p><p>How exactly one should measure performance for the SpaceNet 7 task is a tricky question given the multiple dimensions of the dataset. For one, each building footprint is assigned a unique identifier (i.e. address), which we would like to track over time. Secondly, there is significant construction activity in the SpaceNet 7 data cubes, with new buildings appearing (or disappearing) throughout the time series. Ideally, we would like to quantify the ability of machine learning algorithms to correctly predict when and where the change takes place. These two competing priorities (tracking and change) are not fully captured in existing metrics, and lead us to develop a new metric: the <strong>SpaceNet Change and Object Tracking (SCOT) metric</strong>. This blog details the advantages and specifics of this new metric that will be used to score the impending SpaceNet 7 challenge.</p><h3>1. The SCOT Metric</h3><p>For SpaceNet 7, the ground truth and the model-generated proposals both consist of a set of building footprints for each month. Each footprint is assigned an ID number, with the idea being that each reappearance of the same ID in subsequent months corresponds to a new observation of the same building. Measuring the concordance between the ground truth and proposals is the job for an evaluation metric.</p><p>We originally intended to use the <a href="https://cvhci.anthropomatik.kit.edu/~stiefel/papers/ECCV2006WorkshopCameraReady.pdf">Multiple Object Tracking Accuracy</a> (MOTA) metric for SpaceNet 7, which is a commonly used metric for object tracking in video. Yet this metric is not a good choice for difficult sequences, since errors are compounded additively (rather than being “averaged” out via a harmonic mean with the historical SpaceNet metric). The end result is that for difficult scenes with few true positives and many false positives, it is possible to achieve a negative MOTA score. Since we are tackling a very hard problem in SpaceNet 7, MOTA is therefore a poor choice. In fact, all existing metrics that we investigated unfortunately proved to be a poor fit for our dataset and challenge task, leading us to develop our own metric.</p><p>The SpaceNet Change and Object Tracking (SCOT) metric combines two terms: a tracking term and a change detection term. The tracking term evaluates how often the proposal correctly tracks the same buildings from month to month with consistent ID numbers. In other words, it measures the model’s ability to characterize what stays the same as time goes by. The change detection term evaluates how often the proposal correctly picks up on the construction of new buildings. In other words, it measures the model’s ability to characterize what changes as time goes by.</p><h4>1.1 Matching Footprints</h4><p>For both terms (tracking and change detection) in the SCOT metric, we start by finding “matches” between ground truth and proposal footprints for each month, just like in the original <a href="https://medium.com/the-downlinq/the-spacenet-metric-612183cc2ddb">SpaceNet Metric</a> used for previous SpaceNet building footprint challenges. To be matched, a ground truth footprint and a proposal footprint must have an intersection-over-union (IOU) exceeding a given threshold. In previous challenges, a threshold of 0.5 was used, but for SpaceNet 7 we relax the threshold to 0.25 due to the extra difficulty of interpreting lower-resolution imagery. Using a threshold below 0.5 makes it possible for more than one proposal footprint to qualify to be matched to the same ground truth footprint, or vice versa. To resolve that ambiguity, the SCOT metric introduces an optimized way of selecting matches when such choices arise. The metric uses the set of matches that minimizes the number of unmatched ground truth footprints plus the number of unmatched proposal footprints. If there’s more than one way to achieve that goal, then as a tiebreaker the set of matches with the highest sum of IOUs is selected. This is an example of a long-studied problem in combinatorics (the “unbalanced linear assignment problem”), and an algorithmic solution is available in SciPy and other places.</p><p>For the original SpaceNet Metric, just one step remains after finding the matches. That’s calculating the F1 score, treating every match as a true positive:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/1*uAYgNQyCgKsVbv7PRnocGQ.png" /><figcaption>F1 = F1 score, tp = true positives, fp = false positives, fn = false negatives</figcaption></figure><p>The SCOT metric consists of two terms (tracking and change detection) that are both F1 scores that follow this same general procedure, but with one small (though significant) tweak in each case.</p><h4>1.2 The Tracking Term</h4><p>To measure how well a proposed set of footprints tracks IDs from month to month, the tracking term calculation uses the same formula as above but applies a more stringent definition of what counts as a true positive.</p><p>A match between a ground truth footprint and a proposal footprint is considered a “mismatch” if the ground truth footprint was most recently matched with a different proposal footprint ID or if the proposal footprint was most recently matched with a different ground truth ID. (This is inspired by, but slightly different from, the MOTA definition of a mismatch.) When calculating the F1 score for tracking, <em>only those matches that are not mismatches count as true positives</em>. If a ground truth footprint is matched to a proposal footprint but it’s a mismatch, then the ground truth footprint is considered a false negative and the proposal footprint is considered a false positive, the same outcome as if they’d never been matched at all. The result is an F1 score that penalizes even correctly-located proposal footprints if their ID numbers are not consistent across time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*m0B7VolLOewk8Rx9689gqg.png" /><figcaption>Figure 1: Tracking. An example time series of four buildings over five months. Solid brown polygons are ground truth building footprints and outlines are proposal footprints. ID numbers for ground truth are centered over the buildings, and ID numbers for the proposals are to their upper right.</figcaption></figure><p>Figure 1 shows a made-up example of how this works. In this example, a small area with four buildings (ground truth IDs 1–4) is imaged each month for five months. The outlines show proposal footprints, labeled by their ID numbers. Buildings 2 and 4 are newly constructed during this time, and buildings 3 and 4 temporarily disappear in April, which can happen when an image is partially occluded by clouds. The model in this example does a good job of generating proposal footprints that are matched with ground truth building footprints, but it makes some mistakes with the proposals’ ID numbers, causing some of the matches to be mismatches. Notice that in April, proposal 12 shifts from being matched with building 2 to being matched with building 1. That leads to two mismatches in April: The match of building 1 and proposal 12 is a mismatch because proposal 12 had been most recently matched with a different building, while the match of building 2 and proposal 14 is a mismatch because building 2 had been most recently matched with a different proposal. On the other hand, the match of building 3 and proposal 11 in May is <em>not</em> a mismatch, because the last time this building and proposal were matched with anything — even though it was several months earlier — it was with each other.</p><p>What all these details have in common is that proposals that stay on target with the same building are not mismatches, while ID mistakes count as mismatches during the specific months when those mistakes occur, lowering the value of the tracking term.</p><h4>1.3 The Change Detection Term</h4><p>Like the tracking term, the change detection term is an F1 score that’s similar to the SpaceNet Metric with one small modification. This time, we don’t worry about mismatches. Instead, once the matching is complete, we simply ignore every building and proposal that isn’t making its chronological first appearance. That amounts to dropping any footprint (ground truth or proposal) with an ID number that has appeared in any previous month. Rather than measuring performance on all buildings, <em>the result is an F1 score concerned only with new buildings (i.e., changes), requiring the model not only to find them but to identify them as such.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/1*tVlx2wNuouxXGeyKCR69Bg.png" /><figcaption>Figure 2: Change detection. The same time series as Figure 1, showing new ground truth building footprints in brown, new proposal footprints in blue, and graying out all ground truth and proposal footprints with previously-seen IDs.</figcaption></figure><p>Figure 2 shows how this works for the same made-up example considered above. There are two appearances of new buildings: building 2 in February and building 4 in March. (Buildings already present in the first month of data don’t count.) Building 2’s February arrival is matched with proposal 12, which is also making its first appearance, so that counts as a true positive for the change detection term. Building 4’s arrival in March is matched with proposal 13, but that proposal ID already appeared in February so its subsequent appearance in March is ignored. Building 4’s arrival is therefore a false negative.</p><p>To summarize the change detection term, only new footprints are used for the purpose of calculating the F1 score. One important property of this term is that a set of static proposals that do not vary from one month to another will receive a change detection score of 0, even for a data cube with very little new construction.</p><h4>1.4 Putting It All Together</h4><p>To combine the tracking term with the change detection term, a weighted harmonic mean of these two F1 scores is used:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/1*XxZLHfQZZ1ZdXXjqtwbi5w.png" /></figure><p>This makes it possible to tune the relative weight of tracking and change detection by setting a value for β. For SpaceNet 7 we use a value of β=2, which emphasizes the tracking term.</p><p>In SpaceNet 7 we have multiple locations (a.k.a. areas of interest), and so the aggregate reported score is a simple mean of each unique location’s SCOT score.</p><h3>2. Conclusions</h3><p>The uniqueness of the SpaceNet 7 dataset (namely: dynamic objects and geolocated addresses) demand careful consideration of an appropriate metric for the Multi-Temporal Urban Development Challenge. Existing metrics (e.g. MOTA) are not adequate for our proposed task of tracking uniquely identified building footprints through a dynamic time series. Accordingly, we created the SpaceNet Change and Object Tracking (SCOT) metric that combines an object tracking and change term into a singular score. One benefit of this metric is its continuity with past SpaceNet challenges: the SCOT tracking term reduces to the original SpaceNet Metric when evaluated at a single time step. Another benefit of the metric is explicitly quantifying both change detection, as well as object tracking performance. An upcoming post will provide further details on the implementation of SCOT, including the complete code, via the scoring of the SpaceNet 7 baseline algorithm.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5339b0c904c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-downlinq/the-spacenet-change-and-object-tracking-scot-metric-5339b0c904c">The SpaceNet Change and Object Tracking (SCOT) Metric</a> was originally published in <a href="https://medium.com/the-downlinq">The DownLinQ</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing the Solaris Multimodal Preprocessing Library]]></title>
            <link>https://medium.com/the-downlinq/introducing-the-solaris-multimodal-preprocessing-library-a05dba39dd8c?source=rss-ecbbd25b9fd2------2</link>
            <guid isPermaLink="false">https://medium.com/p/a05dba39dd8c</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Daniel Hogan]]></dc:creator>
            <pubDate>Thu, 16 Jul 2020 19:59:25 GMT</pubDate>
            <atom:updated>2020-07-16T22:28:44.147Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*c2sLGRAAa08vrw-dvn6V-A.png" /></figure><p>Within the latest release of <a href="https://github.com/CosmiQ/solaris">Solaris</a>, available now, is a new resource for geospatial deep learning: the Solaris Multimodal Preprocessing Library. This library eases the work of imagery preprocessing, i.e., undertaking data cleaning and image analysis to get the data to a final form for deep learning. It does so by providing building blocks for constructing powerful image-processing workflows.</p><p>Sometimes building a dataset is as simple as tiling ready-to-go images and their accompanying vector labels, a task for which Solaris’s tile subpackage is ably suited. But sometimes the imagery itself must also be modified in more substantial ways. This is especially true for multimodal datasets, where tasks like applying masks from one modality to another are the norm. But it’s also true anytime one wants to process imagery with techniques like pansharpening, noise reduction, etc. The analysis of synthetic aperture radar (SAR), in particular, is replete with image processing algorithms that are needed to convert complex SAR data to more readily-interpretable imagery.</p><p>To support workflows like those, Solaris’s new preproc <a href="https://github.com/CosmiQ/solaris/tree/master/solaris/preproc">subpackage</a> contains a library of more than 60 Python classes, each of which does a distinct data manipulation task. Although these classes may be useful individually, their real value is that they can easily be combined into user-defined classes to execute image-processing workflows of arbitrary complexity.</p><p>Suppose that a user has in mind a flowchart describing some preprocessing workflow. To turn that idea into a <a href="https://docs.python.org/3/tutorial/classes.html">class</a>, the first step is to instantiate an object for each task in the flowchart. Many typical tasks (e.g., load an image, apply a mask, do a calculation with the image’s bands) are available among the 60+ options already in the library. The second step is to specify how the inputs and outputs of all those tasks are connected to each other. This can be done in a single line of code using a concise notation. (Compared to similar programs that require the user to draw a flowchart in a GUI or re-type the task names every time their output is used, this approach is much less tedious.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SEF8FoaKEeOPj_BlBRn6Bg.png" /><figcaption>Figure 1: Flowchart outlining a pansharpening algorithm, along with a class definition and example output. These items can all be seen in a larger size <a href="https://solaris.readthedocs.io/en/latest/tutorials/notebooks/preprocessing_branching.html">here</a>.</figcaption></figure><p>The reason this works is that the classes in the preprocessing library all inherit from a base class, called PipeSegment, that handles much of the functionality in the background. For example, the user does not need to keep track of hard disk file paths for all the intermediate results — intermediate results are instead stored in RAM and deleted as soon as they’re no longer needed. The user doesn’t even need to figure out the order in which to do the calculations; that’s also handled automatically. In this functional programming approach, the user’s only job is to clarify exactly what data processing should be done. Figuring out the implementation details is a job for the computer.</p><p>In addition to the release of the code itself, a three-part tutorial has been added to the <a href="https://solaris.readthedocs.io/en/latest/tutorials/index.html#the-python-api">Solaris tutorials page</a>. <a href="https://solaris.readthedocs.io/en/latest/tutorials/notebooks/preprocessing_pipelines.html">Part 1</a> of the tutorial shows how to create a simple image-processing pipeline, using the real-world example of a normalized difference vegetation index (NDVI). <a href="https://solaris.readthedocs.io/en/latest/tutorials/notebooks/preprocessing_branching.html">Part 2</a> shows how to handle more complicated workflows where the data flow branches apart and recombines. This is shown using the example of a pansharpening algorithm. Finally, <a href="https://solaris.readthedocs.io/en/latest/tutorials/notebooks/preprocessing_sar.html">Part 3</a> shows how to use the preprocessing library to work with SAR. The flexibility of the Solaris Multimodal Preprocessing Library makes it an effective tool for use cases like these and many others.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a05dba39dd8c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-downlinq/introducing-the-solaris-multimodal-preprocessing-library-a05dba39dd8c">Introducing the Solaris Multimodal Preprocessing Library</a> was originally published in <a href="https://medium.com/the-downlinq">The DownLinQ</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The SpaceNet 6 Baseline]]></title>
            <link>https://medium.com/the-downlinq/the-spacenet-6-baseline-3b8ae8068351?source=rss-ecbbd25b9fd2------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b8ae8068351</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Daniel Hogan]]></dc:creator>
            <pubDate>Thu, 05 Mar 2020 22:01:10 GMT</pubDate>
            <atom:updated>2020-03-05T22:05:58.617Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/691/1*5P0BUbBUKe0LPTsSvBiIuQ.png" /></figure><p><em>Preface: </em><a href="https://spacenet.ai/"><em>SpaceNet LLC</em></a><em> is a nonprofit organization dedicated to accelerating open source, artificial intelligence applied research for geospatial applications, specifically foundational mapping (i.e. building footprint &amp; road network detection). SpaceNet is run in collaboration with </em><a href="http://www.cosmiqworks.org/"><em>CosmiQ Works</em></a><em>, </em><a href="https://www.maxar.com/"><em>Maxar Technologies</em></a><em>, </em><a href="https://www.intel.ai/#gs.xjmhqz"><em>Intel AI</em></a><em>, </em><a href="https://aws.amazon.com/"><em>Amazon Web Services (AWS)</em></a><em>, </em><a href="https://www.capellaspace.com/"><em>Capella Space</em></a><em>, </em><a href="https://www.topcoder.com/"><em>Topcoder</em></a><em>, and </em><a href="http://www.grss-ieee.org/"><em>IEEE GRSS</em></a><em>.</em></p><h3>Introduction</h3><p>The <a href="https://spacenet.ai/sn6-challenge/">SpaceNet 6 Challenge</a> is the first multimodal SpaceNet challenge, inviting participants to work with optical imagery as well as synthetic aperture radar imagery to design a foundational mapping algorithm. This post describes a newly-released baseline algorithm, which could be used as an example or starting point for SpaceNet 6 participants.</p><p>The goal of the challenge is to produce an algorithm that takes a synthetic aperture radar (SAR) image and returns a list of building footprints (i.e., outlines), expressed as vector polygons instead of just pixel maps. To enable supervised learning methods such as deep learning, training data has been provided. The training data includes tiles of SAR imagery, along with the corresponding building footprints for each tile. The training data also includes optical imagery of those same tiles. Although it’s possible to train a SAR→footprint algorithm without using the optical data at all, having it available builds intuition and opens up additional possibilities for how to approach this problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*JencNJ7yxZw4ynEKjWfI7w.png" /><figcaption>SpaceNet 6 baseline example. Top row shows optical and SAR views of the same tile. This tile includes the edge of a SAR strip, which is why the imagery is cut off instead of being a perfect square. The bottom left image is inferred building footprints based on only the SAR image. (The model gets to see optical imagery during training but not inference.) The bottom right image is the ground truth labels.</figcaption></figure><p>The baseline algorithm presented here is built using <a href="https://github.com/CosmiQ/solaris">Solaris</a>, an end-to-end Python framework for geospatial deep learning. The code for this baseline can be found on CosmiQ’s Github under “<a href="https://github.com/CosmiQ/CosmiQ_SN6_Baseline">CosmiQ_SN6_Baseline</a>.” In some ways this baseline draws on the baseline and winning submission from SpaceNet 4. However, it also has features that are specific to the new dataset. SAR imagery is affected by the direction from which the data was collected (through effects such as <a href="https://medium.com/the-downlinq/sar-101-an-introduction-to-synthetic-aperture-radar-2f0b6246c4a0">layover</a>). Therefore, the baseline rotates tiles so the model is trained on SAR images of matching directionality. Also, the baseline takes advantage of the optical imagery through transfer learning. Training the model on the optical imagery first and the SAR imagery second leads to higher performance than training the model on the SAR imagery alone. Lastly, since small buildings (&lt;20m²) are not used for evaluation, they are removed from the training data. This baseline earns a <a href="https://medium.com/the-downlinq/the-spacenet-metric-612183cc2ddb">SpaceNet metric</a> score of just above 0.2, and we look forward to seeing challenge participants improve on that score by trying a variety of techniques.</p><h3>Baseline Details</h3><p>At the heart of the baseline algorithm is a neural network, trained to take an image and return a pixel segmentation map. The neural net used here is a U-Net with a VGG-11 encoder. This architecture, also called a TernausNet, is the same one used for the SpaceNet 4 baseline. The only difference here is that the neural network takes four-band images as input. That’s because our quad-pol SAR imagery contains <a href="https://medium.com/the-downlinq/sar-201-an-introduction-to-synthetic-aperture-radar-part-2-895beb0b4c0a">four polarization channels</a>.</p><p>To make transfer learning as simple as possible, the pan-sharpened RGB optical imagery is converted into four-channel images so it can be treated as ersatz SAR imagery. (Red is used for the HH polarization, green for VV, and blue for both HV and VH.) That way, the model can be trained on the optical and then the SAR with no swapping out of layers required.</p><p>Training is done for 150 epochs on optical imagery followed by 50 epochs on SAR imagery. Optical imagery is augmented by random rotations of 0, 90, 180, or 270 degrees, but no such augmentation is done with the SAR data because of the directionality mentioned above. About 1/5 of the training data is set aside for validation, with this data being drawn from the easternmost part of the training data region. Instead of just using the model weights after the final epoch, the model weights that result in the highest performance on the validation data are used. Because the model is susceptible to overtraining with this data, the “best” version sometimes comes well before the final epoch.</p><p>Training is done with an AdamW optimizer. Model performance is evaluated using a loss function that combines Dice and focal loss, with the latter being given ten times the weight of the former. This choice of optimizer and loss function is based on the winning SpaceNet 4 algorithm. Training is completed in about ten hours on 4 Titan Xp GPUs; testing in about half an hour.</p><p>It should be emphasized that there are many ways to approach this challenge, with the given baseline providing only one such approach. For example, a completely different way to use the optical data is to train a deep learning model to convert SAR images to optical-style images (see <a href="https://phillipi.github.io/pix2pix/">here</a> and <a href="https://arxiv.org/abs/1802.10036">here</a>), effectively turning a SAR→footprint task into an optical→footprint task.</p><h3>Baseline Performance</h3><p>Using the SpaceNet metric, the baseline model achieves a score of 0.21±.02 for the SpaceNet 6 Challenge. If transfer learning with optical data is not used, the score drops to 0.135±.002. If transfer learning is not used and furthermore the SAR imagery is augmented with random rotation (thereby failing to respect directionality), the scores sinks slightly lower to 0.12±.03. During training, the lowest loss when training on optical data was almost three times smaller than the lowest loss training on SAR, suggesting that interpreting SAR is a fundamentally harder problem than optical imagery, at least when the optical imagery is nearly on-nadir. Several years ago, the baseline for the very first SpaceNet challenge, which concerned finding building footprints from optical imagery, clocked in at a <a href="https://arxiv.org/abs/1807.01232">score of 0.21</a>, roughly equal to the current baseline for SAR. The results of that first challenge, and subsequent building footprint challenges with optical data, saw ever-increasing performance. If this trend holds true for the realm of SAR, then achievable performance may far exceed what’s shown in this baseline.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b8ae8068351" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-downlinq/the-spacenet-6-baseline-3b8ae8068351">The SpaceNet 6 Baseline</a> was originally published in <a href="https://medium.com/the-downlinq">The DownLinQ</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SAR 201: An Introduction to Synthetic Aperture Radar, Part 2]]></title>
            <link>https://medium.com/the-downlinq/sar-201-an-introduction-to-synthetic-aperture-radar-part-2-895beb0b4c0a?source=rss-ecbbd25b9fd2------2</link>
            <guid isPermaLink="false">https://medium.com/p/895beb0b4c0a</guid>
            <category><![CDATA[remote-sensing]]></category>
            <category><![CDATA[satellite-imagery]]></category>
            <category><![CDATA[satellite]]></category>
            <category><![CDATA[multimodal]]></category>
            <category><![CDATA[geospatial]]></category>
            <dc:creator><![CDATA[Daniel Hogan]]></dc:creator>
            <pubDate>Thu, 13 Feb 2020 21:56:49 GMT</pubDate>
            <atom:updated>2020-02-13T21:56:49.824Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By Daniel Hogan (In-Q-Tel CosmiQ Works) and Jason Brown (Capella Space) with the CosmiQ Works and Capella teams.</em></p><p><em>Preface: </em><a href="https://spacenet.ai/"><em>SpaceNet LLC</em></a><em> is a nonprofit organization dedicated to accelerating open source, artificial intelligence applied research for geospatial applications, specifically foundational mapping (i.e. building footprint &amp; road network detection). SpaceNet is run in collaboration with </em><a href="http://www.cosmiqworks.org/"><em>CosmiQ Works</em></a><em>, </em><a href="https://www.maxar.com/"><em>Maxar Technologies</em></a><em>, </em><a href="https://www.intel.ai/#gs.xjmhqz"><em>Intel AI</em></a><em>, </em><a href="https://aws.amazon.com/"><em>Amazon Web Services (AWS)</em></a><em>, </em><a href="https://www.capellaspace.com/"><em>Capella Space</em></a><em>, </em><a href="https://www.topcoder.com/"><em>Topcoder</em></a><em>, and </em><a href="http://www.grss-ieee.org/"><em>IEEE GRSS</em></a><em>.</em></p><p>Synthetic aperture radar (SAR) provides all-weather ground imaging, but SAR images are quite different from optical images. This post gives an overview of data analysis methods used with SAR and what can be learned from SAR imagery. This concludes a discussion begun in <a href="https://medium.com/the-downlinq/sar-101-an-introduction-to-synthetic-aperture-radar-2f0b6246c4a0">a previous post</a>, which looked at how SAR images are produced. SAR data is featured in the soon-to-begin <a href="https://medium.com/the-downlinq/announcing-spacenet-6-multi-sensor-all-weather-mapping-b5d923e39974">SpaceNet 6 Challenge</a>.</p><h3>SAR Data</h3><p>As introduced in the previous post, SAR is a coherent imaging method. This enables techniques like interferometry, covered in the next section, to be achieved. Since SAR is a coherent imaging method the returns have two components, the intensity and the phase.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*in2Uf2Rqk3wUNrWdO9Nf9g.png" /><figcaption>Figure 1. Intensity (left) and phase (right) components of SAR data (Courtesy: Capella)</figcaption></figure><p>The intensity component of SAR data is the part that looks like an image after SAR image formation processing has occurred. The radio waves in the SAR beam are aligned in space and time (coherent) upon transmission, or in other words they are “in phase.” The phase component of SAR data measures how much the radio waves have shifted “out of phase” after they interact with the scatterers on the surface. These phase components are useful, because phase differences between different channels or different collects can reveal information about the geometry and composition of the scene.</p><h4>Single Look Complex</h4><p>SAR data can be delivered as Single Look Complex (SLC) data where the intensity and phase components are represented as complex numbers for each pixel. Intensity and phase values can be subsequently computed from the complex numbers. Intensity and phase can be viewed as images, as seen in Figure 1, though only the intensity component is recognizable as an image. Additionally, SLC data is in the radar-image (slant) plane projection, so its pixels do not correspond to geo-coordinates.</p><h4>Ground Range Detected</h4><p>SAR data can also be delivered as Ground Range Detected (GRD) data, in which processing has been done to project the image onto the ground plane with geo-coordinates such as latitude and longitude, creating a terrain map.</p><h4>Speckle Reduction</h4><p>There are various methods to achieve speckle reduction and make a SAR image easier to interpret and analyse. Multi-looking is the process of splitting the radar beam into several narrower sub-beams. Each of these sub-beams is a “look” and is subject to speckle. However all the “looks” can be averaged and the amount of speckle in the final averaged image will be reduced. Another method of speckle reduction is spatial filtering, in which a “moving window” technique is used to calculate a weighted average of a group of pixels and replace the window’s center pixel with that value. This technique produces a smoothing effect in which the amount of speckle is reduced (Figure 2).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*r80nw2T-Eo-lir54" /><figcaption>Figure 2. Comparison of SAR imagery before (left zoom detail) and after (right zoom detail) speckle filtering (Courtesy: Capella)</figcaption></figure><h3>Multichannel Analysis Techniques</h3><p>The methods seen so far are everything needed to generate a high-quality SAR image. But SAR becomes an even more powerful tool with multichannel techniques, where multiple SAR scans of the same area are used together to learn more than is possible with one image alone. Three main multichannel approaches are polarimetry, interferometry, and tomography.</p><h4>Polarimetry</h4><p>SAR polarimetry exploits the fact that radio waves, just like visible light waves, can be polarized. For example, a radar pulse might be horizontally- or vertically-polarized, and similarly the radar can be configured to measure only the horizontally- or vertically-polarized part of the echo. That gives up to four polarization combinations. A SAR instrument that toggles among these combinations as it travels will gain more information about the scene, because radar echoes from different types of objects have different relative strengths depending on polarization (Fig. 2). For example, if HH scattering is larger than VV scattering, then the scatterer is likely more horizontally-oriented than vertically-oriented. In polarimetry, even the difference in phase angle between the different polarizations can help distinguish features on the ground. SpaceNet 6 features quad-pol data, meaning all four linear polarizations were measured and are included in the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/0*GRpV9wdUO3AXP6yK" /><figcaption>Figure 3: In this false-color SAR polarimetry image, each color channel (blue, green, or red) shows the reflection strength of a different combination of polarizations. (Courtesy of <a href="https://www.sandia.gov/radar/imagery/index.html">Sandia National Laboratories, Radar ISR</a>)</figcaption></figure><h4>Interferometry and Tomography</h4><p>For completeness, we mention two other approaches that are different from the polarimetry being used in SpaceNet 6.</p><p>In SAR interferometry, two images are taken of the same area, and the phase angle difference between them is calculated for each point on the ground. If the distance from the object to the radar is different for the two images, the difference will show up as a phase shift. That applies whether the difference is due to a change in the radar’s position or due to motion of the ground itself. There’s just one catch — a full 360-degree change in phase (or any exact multiple thereof) is indistinguishable from having no phase shift at all. So very small distance changes can be measured directly, but figuring out larger changes requires more work to overcome that ambiguity. Interferometry has two main applications. In across-track interferometry, two passes are made with a SAR radar along slightly different paths. The phase differences can then be used to construct a map of surface height (called a digital elevation model, or DEM). In differential SAR interferometry, the two passes are made at different times and can be used to figure out how the surface height has changed in the interim due to, for example, ground subsidence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/0*Gc8CTgdnwJTqYbSX" /><figcaption>Figure 4: This SAR interferometry image shows an elevation map of a volcano in Hawaii. The colors repeat in bands because of the 360-degree ambiguity. (Credit: <a href="https://photojournal.jpl.nasa.gov/catalog/PIA01762">NASA/JPL</a>)</figcaption></figure><p>Finally, in SAR tomography, many passes (maybe twenty or so) are made over the same area along different paths. From that data, a 3D model of the area is constructed using the same mathematical techniques that underlie Computed Tomography (CT) scans.</p><h3>Applications</h3><h4>Change Detection</h4><p>Because of the periodic nature of satellite remote sensing, change detection is one of its fundamental applications. For example, the satellite imaging startup Capella Space plans to launch a constellation of 36 SAR satellites which will achieve a 1 hour revisit time. This will allow the collection of time series data on the order of hours to years, enabling every imaginable type of change detection. Shipping ports can be monitored for patterns of life and commerce, land cover/land use can be monitored over weeks and months, and the health of forests can be monitored year over year.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JXQIVmhuWirv5VQJ" /><figcaption>Figure 5. Detail of shipping activity at three different times during August 23rd, 2019 at Eemhaven, near Pernis, Netherlands. An image from 12:31pm was loaded into the red color gun, 1:56pm into the green color gun, and 3:11pm into the blue color gun (Courtesy: Capella).</figcaption></figure><h4>Additional Applications</h4><p>The uses of SAR imagery are numerous. SAR has been shown to detect oil spills from ships, perform ice monitoring, and detect “dark ships” (vessels with their identification signals turned off). SAR has utility in disaster management and response in helping create flood maps, assess deformation of the ground due to earthquakes, as well as mapping wildfire severity and recovery. SAR has been used to detect subsidence due to oil/water extraction and tunneling, as well as for monitoring deforestation and reforestation. Additionally SAR data has been used to generate digital elevation models and three-dimensional land use models. The use of high resolution SAR for foundational mapping purposes holds promise as well, particularly when combined with machine learning.</p><h3>SAR and Machine Learning</h3><p>Techniques for analyzing SAR data have been steadily advancing ever since the synthetic aperture concept was developed in the 1950s. One major advance, however, has taken place in just the past few years: the application of deep learning to SAR.</p><h4>Deep Learning for SAR</h4><p>Deep learning is supervised machine learning using a deep (many-layered) neural net. Analyzing SAR images with a neural net is <a href="https://ieeexplore.ieee.org/abstract/document/499786">not new</a>, but deep neural nets are a more recent development, and their first application to SAR data was less than <a href="https://spie.org/Publications/Proceedings/Paper/10.1117/12.2176558">five years ago</a>. A major task for SAR-related algorithms has long been object detection, which for historical reasons is called automatic target recognition (ATR) when working with SAR data. In perhaps the <a href="https://spie.org/Publications/Proceedings/Paper/10.1117/12.2176558">first published exploration</a> of deep learning with SAR, a deep neural net was shown to work as well as other methods for the final classification stage of a typical ATR pipeline.</p><p>That work used the <a href="https://www.sdms.afrl.af.mil/index.php?collection=mstar">MSTAR dataset</a>, a publicly-available collection of SAR images of different types of military vehicles. <a href="https://arxiv.org/abs/1801.08558">Subsequent work</a> with the same dataset shows how much deep learning for SAR has advanced. Classification accuracy rose from 92.3% to 99.6% in three years.</p><p>At the same time, deep learning is being used with SAR data in an ever-growing <a href="https://www.capellaspace.com/deep-learning-and-sar-applications">variety of ways</a>, including change detection and land cover classification. Researchers have even applied a generative adversarial network (GAN) to the task of <a href="https://arxiv.org/abs/1802.10036">image “translation”</a> — taking a single-polarization SAR image and generating a simulated full-color optical image of the same area.</p><p>When it comes to identifying building footprints (outlines) in SAR, which is the topic of the upcoming SpaceNet 6 Challenge, <a href="https://arxiv.org/abs/1808.06155">various ideas</a> have been pursued. Many of these methods do not use deep learning at all, instead reconstructing building footprints in an unsupervised way from carefully-selected features in the imagery. Efforts towards building footprint extraction with deep learning have so far only begun to scratch the surface. There have been studies of various algorithms for both polarimetric and single-image data. But as <a href="https://arxiv.org/abs/1808.06155">a recent paper</a> observed with regard to currently-available SAR training data, “lack of such annotated datasets is one of the major issues” for SAR deep learning.</p><h4>Closing Thoughts</h4><p>This blog post series has covered a lot of ground in its crash course introduction to synthetic aperture radar. To read more about SAR theory and techniques, <a href="https://www.nrcan.gc.ca/maps-tools-publications/satellite-imagery-air-photos/remote-sensing-tutorials/microwave-remote-sensing/9371">this website</a> (less technical) or <a href="https://ieeexplore.ieee.org/document/6504845/versions">this article</a> (more technical) are great places to go next. However, nothing beats actually getting ahold of SAR data and trying out deep learning on it yourself! This week, <a href="https://spacenet.ai/">SpaceNet</a> released a new open source multimodal dataset with SAR data from Capella Space, optical imagery from Maxar, and building footprint labels. This dataset will be the basis of the <a href="https://medium.com/the-downlinq/announcing-spacenet-6-multi-sensor-all-weather-mapping-b5d923e39974">SpaceNet 6 Challenge</a>.</p><p>In closing, SAR has tremendous potential, and that’s more true now than ever before. On the hardware side, the deployment of constellations of small SAR satellites could offer unprecedented access to the technology. On the software side, the application of deep learning has the potential to yield insights from data on a correspondingly large scale. For someone who’s only worked with optical imagery, the strange properties of SAR images can take some getting used to, but if you’ve made it through this blog post series then you know the key ideas already. Ultimately, it is those same unusual properties of SAR that make it so useful for providing a new perspective on the world.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=895beb0b4c0a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-downlinq/sar-201-an-introduction-to-synthetic-aperture-radar-part-2-895beb0b4c0a">SAR 201: An Introduction to Synthetic Aperture Radar, Part 2</a> was originally published in <a href="https://medium.com/the-downlinq">The DownLinQ</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SAR 101: An Introduction to Synthetic Aperture Radar]]></title>
            <link>https://medium.com/the-downlinq/sar-101-an-introduction-to-synthetic-aperture-radar-2f0b6246c4a0?source=rss-ecbbd25b9fd2------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f0b6246c4a0</guid>
            <category><![CDATA[multimodal]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[remote-sensing]]></category>
            <category><![CDATA[satellite]]></category>
            <category><![CDATA[satellite-imagery]]></category>
            <dc:creator><![CDATA[Daniel Hogan]]></dc:creator>
            <pubDate>Wed, 05 Feb 2020 22:47:54 GMT</pubDate>
            <atom:updated>2020-02-13T22:15:55.219Z</atom:updated>
            <content:encoded><![CDATA[<p><em>By Daniel Hogan (In-Q-Tel CosmiQ Works) and Jason Brown (Capella Space) with the CosmiQ Works and Capella teams.</em></p><p><em>Preface: </em><a href="https://spacenet.ai/"><em>SpaceNet LLC</em></a><em> is a nonprofit organization dedicated to accelerating open source, artificial intelligence applied research for geospatial applications, specifically foundational mapping (i.e. building footprint &amp; road network detection). SpaceNet is run in collaboration with </em><a href="http://www.cosmiqworks.org/"><em>CosmiQ Works</em></a><em>, </em><a href="https://www.maxar.com/"><em>Maxar Technologies</em></a><em>, </em><a href="https://www.intel.ai/#gs.xjmhqz"><em>Intel AI</em></a><em>, </em><a href="https://aws.amazon.com/"><em>Amazon Web Services (AWS)</em></a><em>, </em><a href="https://www.capellaspace.com/"><em>Capella Space</em></a><em>, </em><a href="https://www.topcoder.com/"><em>Topcoder</em></a><em>, and </em><a href="http://www.grss-ieee.org/"><em>IEEE GRSS</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xy2SdtM-UVjB-WZf" /></figure><p>When asked to picture a “satellite image,” most people envision something like the left side of the figure above. It’s an optical image — a photograph — albeit one taken by a very powerful camera. But optical images are not the only way to visualize the earth’s surface from a satellite or an airplane.</p><p>Synthetic aperture radar, or SAR, is a completely different way to generate a picture by actively illuminating the ground rather than utilizing the light from the sun as with optical images. The right side of the image above shows how very different SAR images look from optical images. These differences present challenges but also create new capabilities. One major advantage of SAR is simple: Even the best aircraft-mounted or satellite-mounted optical camera is less useful at night and useless when clouds or smoke are present. SAR can capture images at night and see right through clouds and smoke. It is a 24-hour, all-weather technology.</p><p>SAR data will be featured in <a href="https://medium.com/the-downlinq/announcing-spacenet-6-multi-sensor-all-weather-mapping-b5d923e39974">SpaceNet 6</a>. In this two-part blog series, we’ll cover the basics of how SAR works, what makes it unique, and what makes it useful. Even (or especially) if you’ve never heard of SAR before, this series is for you!</p><h3>How Does Synthetic Aperture Radar Work?</h3><p>Synthetic aperture radar is a way of creating an image using radio waves. The radio waves used in SAR typically range from approximately 3 cm up to a few meters in wavelength, which is much longer than the wavelength of visible light, used in making optical images. These wavelengths fall within the microwave part of the spectrum in the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/0*wK5PY4PoJKHNjgYE" /><figcaption>Figure 1. Comparison of wavelength, frequency, and energy for the electromagnetic spectrum. (Credit: NASA’s <a href="https://imagine.gsfc.nasa.gov/science/toolbox/emspectrum1.html">Imagine the Universe</a>)</figcaption></figure><p>RADAR is an acronym for Radio Detection And Ranging. Radar is an active system, which generates its own radio waves and transmits them from its antenna, toward a target. Depending on the target properties and the imaging geometry, the radar antenna will receive all, some, or none of the radio wave’s energy (this is the Detection part of RADAR). This received signal will travel for an amount of time proportional to the target’s distance from the antenna (this is the Ranging part of RADAR).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/0*bQyGKQmY9tEbwQhx" /><figcaption>Figure 2. Imaging Radar Geometry (Credit: NASA)</figcaption></figure><h4>Real Aperture Radar (RAR)</h4><p>If a radar antenna is carried in an airplane or an orbiting satellite, it can be used to make an image of the ground below. Such a radar image is formed by transmitting pulses of radio frequency (RF) energy towards the ground to the side of the aircraft, and measuring the strength of the return (sometimes called an “echo”) and the length of time it takes to make the round trip back to the antenna. In this manner, the ground is “scanned” in two dimensions. One dimension is the “range” dimension. Objects are placed along this dimension according to their distance from the radar. The second dimension is the “along-track” (or “cross-range” or “azimuth”) dimension. In this dimension, the ground is scanned by the beam moving across the ground at a rate equal to the speed of the platform (aircraft or satellite), and objects are placed along this dimension according to the position of the aircraft along the track. An image is built up from the reflected signals in both dimensions. This arrangement is shown in Figure 2.</p><p>Spatial resolution, the ability to resolve objects on the ground, differs in the range direction (perpendicular to the flight direction) compared to the azimuth direction (parallel to the flight direction). In “real aperture radar,” the range resolution is defined by the width of the pulses transmitted from the antenna. The azimuth resolution is determined by the width of the beam’s footprint on the ground, and the width of the beam is inversely proportional to the antenna length. A short antenna length corresponds to a wide beam-width (beam footprint on the ground). Because flying an antenna large enough to generate a reasonable azimuth resolution, in space, is prohibitive, this limits the spatial resolution in the azimuth direction. The development of advanced processing algorithms solved this problem, leading to a new generation of imaging radars called Synthetic Aperture Radar.</p><h4>Synthetic Aperture Radar (SAR)</h4><p>To mitigate the undesired effects of real aperture radar’s poor azimuth resolution, the motion of the antenna along the azimuthal direction is utilized to “synthetize” or give the effect of a long antenna as shown in Figure 3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*QiMpww_5rZsz2VF-ESz3Eg.png" /><figcaption>Figure 3. Synthetic Aperture Generation (Credit: NASA)</figcaption></figure><p>This synthetizing process is possible because a scatterer (target) on the ground remains within the real-aperture radar beam for many radar pulses. Adding up the reflections from all these pulses appropriately allows one to synthesize a large antenna with a much narrower beam-width, resulting in a better spatial resolution in the azimuth direction. This technique is applicable for aerial systems as well as space-borne systems.</p><h3>SAR Image Interpretation</h3><p>While the images created by SAR can be rendered into a recognizable terrain map, there are important differences between optical imagery and SAR imagery. SAR imagery is considered a non-literal imagery type because it does not look like an optical image which is generally intuitive to humans. These aspects must be understood for accurate image interpretation to be performed.</p><h4>Shadowing</h4><p>Shadowing is caused for the same reasons that shadows are formed in optical imagery: an object blocks the path of direct radiation — visible light in the case of optical imaging and the radar beam in the case of SAR. However, unlike optical imagery in which objects in shadows can be seen due to atmospheric scattering, there is no information in a SAR shadow because there is no return signal.</p><h4>Foreshortening</h4><p>Because SAR is a side-looking, ranging instrument, the backscattered returns will be arranged in the image based on how far the target is from the antenna <em>along the slant plane</em> (radar-image plane). This causes some interesting geometrical distortions in the imagery, such as foreshortening. As seen in Figure 4, the slope A-B is compressed in the slant plane because the radar signal reaches point B shortly after reaching point A in time. This causes a tall object with a slope, such as a mountain, to appear steeper, with a thin bright “edge” appearance. Note that the sensor’s look angle affects foreshortening; a larger look angle will decrease the effect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/411/1*HtHQ7LWE4Wm9yMqzX3aUDg.png" /><figcaption>Figure 4. Foreshortening Geometry (Credit: <a href="https://history.nasa.gov/JPL-93-24/p48.htm">NASA</a>)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/409/1*yKSxnNj-L5ujE4Fgh-8nDA.png" /><figcaption>Figure 5. Layover Geometry (Credit: <a href="https://history.nasa.gov/JPL-93-24/p48.htm">NASA</a>)</figcaption></figure><h4>Layover</h4><p>Layover is an extreme example of foreshortening where the object is so tall that the radar signal reaches point B before it reaches point A. This causes the returns from point B to be placed on the image closer to the sensor (near range) and obscure point A, as if the top has been overlaid on the foot of the mountain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/0*-r80JHPToOqejiiS" /><figcaption>Figure 6. Examples of Geometric Effects in SAR Imagery (Image Credit: ERS, ESA 2011. Retrieved from <a href="https://search.asf.alaska.edu/#/">ASF DAAC</a> 20 January 2020.)</figcaption></figure><p>The effect of these phenomena are altered depending on the sensor’s look angle. A larger look angle increases the effect of shadows (lengthening the shadow), while minimizing the effect of layover (less layover). A smaller look angle has the opposite effect. Figure 6 gives examples of these effects on rugged terrain. Figure 7, meanwhile, shows an example of how buildings in an urban setting are distorted by the same effects. All tall buildings appear laid out horizontally because of layover.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/237/0*OVL_-IjLnsJ4R-Gc" /><figcaption>Figure 7. Examples of layover in urban SAR imagery. In this detail of Rotterdam, there is land to the north (top) and water to the south (bottom). This image was recorded by aircraft from the south. The three skyscrapers near the riverbank appear to jut into the water because of layover. (Courtesy: Capella)</figcaption></figure><h3>Pixel Brightness</h3><p>While a radar image can look like a monochrome optical image, this impression is deceptive. The intensity of pixels in a radar image are not indicative of the color of the object (as in a color photograph). Instead, the intensity depends on the amount of energy the SAR sensor transmitted (like the brightness of the illumination source), the material properties of the object, the physical shape of the object, and the angle from which the object is viewed.</p><h4>Sensor Parameters</h4><p>Design and operation parameters allow engineers to have control over the reflected return signal (called backscatter). Engineers design and model the system and operating parameters to maximize radar returns and therefore the information collected against specific targets. During the design, the system’s wavelength and polarization (discussed below) are chosen and once launched, cannot be changed. These fixed sensor parameters dictate at some level the resulting brightness of a pixel in a particular image.</p><p>The <strong>wavelength</strong> affects the azimuth resolution but it also has important implications for penetration, see Figure 8. In general, radar penetration increases with wavelength.</p><p>The <strong>look angle</strong> affects layover and shadow as described above but can also have an effect on pixel brightness because it changes how the radar beam interacts with the object.</p><p>The <strong>polarization</strong> on transmit and on receive also affect the pixel brightness as described in the following section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QIlFkvU-Pb-LgIyv" /><figcaption>Figure 8. Radar Penetration by Frequency</figcaption></figure><h4>Surface Parameters</h4><p>The surface parameters that affect pixel brightness are the surface roughness of the material, with respect to the system wavelength and the scattering material (the dielectric constant of the object). If the <strong>surface roughness</strong> of the material is smooth with respect to the system wavelength, the radar beam is reflected (Figure 9) according to the law of reflection. This is called specular reflection. If the surface is rough with respect to the system wavelength, the radar beam is scattered in all directions. This is called diffuse scattering. Varying surface roughness results in varying amounts of diffuse scattering and varying pixel brightness.</p><p>The <strong>dielectric constant</strong> of the scattering material is a physical property of a material that determines how reflective that material is to electromagnetic waves. Metallic objects and water have a higher dielectric constant and are more reflective, however since they are smooth with respect to the system wavelength, and usually flat, the radar beam is specularly reflected, away from the sensor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/0*ax6rRSsn4PpDqgon" /><figcaption>Figure 9. Surface Roughness Scattering (Credit: <a href="https://history.nasa.gov/JPL-93-24/p53a.htm">NASA</a>)</figcaption></figure><p>Additionally, certain surface features will cause a specular reflection back toward the sensor, by bouncing off multiple surfaces. A <strong>double-bounce</strong> reflection is called a dihedral return and a Triple-bounce return is called a trihedral return. These are caused by smooth surfaces oriented at 90 degree angles to each other as seen in figure 10.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/294/0*OVh0fvi1v_n5lNAV" /><figcaption>Figure 10. Dihedral Scattering</figcaption></figure><h4>Speckle</h4><p>SAR is a coherent imaging method because the radio waves in a radar beam are aligned in space and time. This coherency provides many advantages (it’s required for the synthetic aperture process to work), but it leads to a phenomenon called speckle. Speckle is a “salt and pepper” variation in the pixel brightness that degrades the quality of the SAR images, making image interpretation more difficult. Speckle occurs because there are often many individual scatters in a given pixel, which leads to positive (salt) and negative (pepper) interference across pixels with an otherwise constant backscatter return.</p><h3>What’s Next</h3><p>This post has described how SAR images are produced. By using clever signal processing, SAR creates radar images of higher resolution than would otherwise be possible. SAR imagery provides information about what’s on the ground, but distortions and speckle make these images very different from optical images. Understanding how these unique data proprieties impact deep learning models is one of the motivations for incorporating SAR data into the SpaceNet 6 Challenge which is scheduled to launch in March.</p><p>In the second part of this two-part series, we’ll look at some of the special data analysis methods used with SAR and explore what makes SAR useful over optical imagery. That will be in our upcoming installment, “<a href="https://medium.com/the-downlinq/sar-201-an-introduction-to-synthetic-aperture-radar-part-2-895beb0b4c0a">SAR 201</a>.”</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f0b6246c4a0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-downlinq/sar-101-an-introduction-to-synthetic-aperture-radar-2f0b6246c4a0">SAR 101: An Introduction to Synthetic Aperture Radar</a> was originally published in <a href="https://medium.com/the-downlinq">The DownLinQ</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Robustness of Limited Training Data: Part 5]]></title>
            <link>https://medium.com/the-downlinq/robustness-of-limited-training-data-part-5-5a0b0518add9?source=rss-ecbbd25b9fd2------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a0b0518add9</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Daniel Hogan]]></dc:creator>
            <pubDate>Wed, 16 Oct 2019 16:45:27 GMT</pubDate>
            <atom:updated>2019-10-16T16:45:27.494Z</atom:updated>
            <content:encoded><![CDATA[<p>Deep learning models for interpreting satellite imagery show increasing performance as the amount of training data is increased. We’ve looked at the details of <a href="https://medium.com/the-downlinq/robustness-of-limited-training-data-for-building-footprint-identification-part-1-8c55810b5ef9">how this plays out</a>, but our previous studies of this topic have suffered from a weakness: they’ve all used just one model architecture, meaning the layout of artificial neurons and other details of the algorithm were the same each time. In this post, we’ll recreate our first analysis with a whole new model architecture, to see what changes and what stays the same.</p><p>This is the last post in the Robustness of Limited Training Data series, so we’ll close with some overall conclusions. You can read all the previous posts here: <a href="https://medium.com/the-downlinq/robustness-of-limited-training-data-for-building-footprint-identification-part-1-8c55810b5ef9">part 1</a>, <a href="https://medium.com/the-downlinq/robustness-of-limited-training-data-part-2-f51eb783823f">part 2</a>, <a href="https://medium.com/the-downlinq/robustness-of-limited-training-data-part-3-9df24c58c2">part 3</a>, <a href="https://medium.com/the-downlinq/robustness-of-limited-training-data-part-4-3b9fb5033abc">part 4</a>. However, they are not prerequisites for what follows.</p><h3>Two Model Architectures</h3><p>To explore the dependence of model performance on the amount of training data, we train a model to identify building footprints in satellite imagery. Our imagery comes from the <a href="https://spacenet.ai/off-nadir-building-detection/">SpaceNet 4 data set</a>, which contains 27 views of Atlanta taken with Maxar’s WorldView-2 satellite. The imagery, with a ground sample distance of about half a meter, is chipped into tiles of about 450m on a side. The model architecture is a lightly-modified version of the <a href="https://github.com/SpaceNetChallenge/SpaceNet_Off_Nadir_Solutions/tree/master/XD_XD">5th-place SpaceNet 4 prize winning model</a> submitted by challenge participant XD_XD. This model features fast inference, and the version used here replaces the original ensemble of three neural networks with a single neural network to speed up training. For brevity, we will refer to this model architecture as “architecture A.” Performance is evaluated with an object-wise F1 score (the <a href="https://medium.com/the-downlinq/the-spacenet-metric-612183cc2ddb">SpaceNet metric</a>) for approximately-correct building footprints. Figure 1 shows how that performance varies with the amount of training data. The red, green, and blue lines show the results broken down by viewing angle, while the black line is an overall result. Details on this can be found in <a href="https://medium.com/the-downlinq/robustness-of-limited-training-data-for-building-footprint-identification-part-1-8c55810b5ef9">previous</a> <a href="https://medium.com/the-downlinq/robustness-of-limited-training-data-part-2-f51eb783823f">posts</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*soVsgEolsn_5R6MBTq2sIQ.png" /><figcaption>Figure 1: Performance of model architecture A, as measured by F1 score, versus number of images used for training. Dotted lines are fitted curves.</figcaption></figure><p>Next, we repeat this process with a different model architecture to see if the results in Figure 1 hold up. Our second architecture is a lightly-modified version of the <a href="https://github.com/SpaceNetChallenge/SpaceNet_Off_Nadir_Solutions/tree/master/cannab">1st-place SpaceNet 4 submission</a>, from challenge participant cannab. The original submission’s ensemble of 28 neural nets is pared down to an ensemble of four for faster training. Each neural net uses a U-net layout with a SE-ResNeXt50 encoder for pixel segmentation. Possible positives are subjected to further winnowing prior to generating vector building footprints. We’ll refer to this model architecture as “architecture B.”</p><p>Architectures A and B both use U-nets to create pixel maps from which building footprint polygons are generated, but there the similarities end. The two architectures use different encoders and different amounts of post-processing. Architecture B is an ensemble while A is not. Two model architectures clearly can’t span the space of everything that’s possible, but these are different enough to be a reasonable test of any highly-model-specific tendencies.</p><p>With that in mind, we measure performance as a function of training data quantity for model architecture B. We follow the procedure in <a href="https://medium.com/the-downlinq/robustness-of-limited-training-data-for-building-footprint-identification-part-1-8c55810b5ef9">post 1</a>, except for using the method to calculate error bars introduced in <a href="https://medium.com/the-downlinq/robustness-of-limited-training-data-part-3-9df24c58c2">post 3</a>. Figure 2 shows the result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*DwbZD1tNU9JIpEW8WL5rGQ.png" /><figcaption>Figure 2: Performance of model architecture B, as measured by F1 score, versus number of images used for training. Dotted lines are fitted curves.</figcaption></figure><p>The results of model architectures A and B are broadly similar. In both cases, a rapid performance increase with low amounts of training data gives way to diminishing returns with higher amounts of data. Performance initially rises more quickly with architecture B, getting closer to its maximum value with less training data. Architecture B also shows more consistent performance, with shorter error bars (even after taking into account a factor of two from a change in the analysis procedure).</p><p>As <a href="https://medium.com/the-downlinq/robustness-of-limited-training-data-part-2-f51eb783823f">previously shown</a> for architecture A, architecture B results can be roughly fit by curves scaling as a constant minus an inverse power law. Such fitted curves for architecture B are included in Figure 2. In this case, the actual results do deviate slightly from the simple model, with the curves underpredicting the measured results at 67 tiles and overpredicting the results at 266 tiles. This deviation is too big to ascribe to a chance statistical fluctuation. In fact, a chi-squared test rejects the hypothesis that the simple curves alone can fully explain the variation in the observed performance. Nevertheless, the deviations from the fitted curves are still small compared to the dynamic range of the performance, so the model is still useful even though it is something of an oversimplification.</p><p>Finally, we ask what seems like a simple question: which architecture is better? Figure 3 shows the architecture B results again, which the architecture A results overlaid in lighter colors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*FRcKVNf0LKYsp27wOk_m9Q.png" /><figcaption>Figure 3: F1 score versus number of training images for architecture A (lighter shades, from Figure 1) and architecture B (darker shades, from Figure 2).</figcaption></figure><p>At around 1000 training images, architecture B shows much higher performance than architecture A. By around 20,000 training images, architecture B only barely outperforms A. Extrapolating with the fitted curves indicates that, with enough data, architecture A overtakes architecture B.</p><p>What this demonstrates is that the model architectures cannot be absolutely ranked. It might not be possible to answer the question “which architecture gets better performance?” — unless one specifies how much training data is to be used. Deep learning papers often pronounce a model to be a success if it outperforms a previous best model on some canonical data set. But a canonical data set necessarily has a fixed size. The results with architectures A and B give a clean example of a case where a comparison using a single quantity of training data cannot tell the full story.</p><h3>Project Conclusions</h3><p>To close this series on the robustness of models with limited training data, we revisit some of the key points we’ve come across along the way.</p><ul><li>Foundational mapping model performance rises with data rapidly when data is scarce and slowly when data is abundant. This holds across model architectures and across geographies. Having a whole city’s worth of data is not a requirement for geospatial deep learning. A large fraction of that performance can be achieved with a small fraction of the data.</li><li>Various statistical methods enable more thorough analyses. Error bars are critically important for deep learning research, revealing when results are repeatable and when they’re even real.</li><li>To get the maximum performance across multiple cities with a fixed (and equal) amount of data for each one, building a specific model for each city gives worse results than piling all that data together to train one general-use model.</li><li>A simple functional form works well for fitting building footprint model performance across a variety of architectures and geographies. Differences in the fitted parameters point to poorly-understood differences in the cities themselves.</li><li>A deep learning architecture that performs the best with low training data might not be the best given lots of training data.</li></ul><p>But in the end, the most important lesson is this: Geospatial deep learning works surprisingly well with surprisingly little training data. Millions of images are <em>not</em> required — the training data barrier to entry for geospatial deep learning is lower than many people think it is.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a0b0518add9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-downlinq/robustness-of-limited-training-data-part-5-5a0b0518add9">Robustness of Limited Training Data: Part 5</a> was originally published in <a href="https://medium.com/the-downlinq">The DownLinQ</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Robustness of Limited Training Data: Part 4]]></title>
            <link>https://medium.com/the-downlinq/robustness-of-limited-training-data-part-4-3b9fb5033abc?source=rss-ecbbd25b9fd2------2</link>
            <guid isPermaLink="false">https://medium.com/p/3b9fb5033abc</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[maps]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Daniel Hogan]]></dc:creator>
            <pubDate>Wed, 25 Sep 2019 13:50:15 GMT</pubDate>
            <atom:updated>2019-09-25T13:50:15.294Z</atom:updated>
            <content:encoded><![CDATA[<p>When it comes to the relationship between geospatial neural network performance and the amount of training data, do geographic differences matter? In a <a href="https://medium.com/the-downlinq/robustness-of-limited-training-data-part-3-9df24c58c2">previous post</a>, we examined this question by training the same building footprint model using various amounts of data from four different cities: Las Vegas, Paris, Shanghai, and Khartoum. That led to a plot (Figure 1) of performance for each city, either using a model trained on the city in question or using a model trained on the combined data of all four cities. In this post, we’ll take a closer look at two questions that went unanswered in the previous post. First, what happens if we take a model trained on one city and apply it to a different one it’s never seen before? And second, why does just one of the cities — Khartoum, Sudan — respond to a training data deficit more resiliently than the others?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*6QRwHFOFUb55uEFdzWewyg.png" /><figcaption>Figure 1: Average F1 score versus number of training images per city. “Individual model” denotes models trained only on the city for which their F1 score is shown, while “combined model” denotes models trained on all four cities (i.e., with four times as much training data). Reprinted from <a href="https://medium.com/the-downlinq/robustness-of-limited-training-data-part-3-9df24c58c2">here</a>.</figcaption></figure><h3>Transferability</h3><p>To assess model transferability, models trained on one city are tested on the others. The training is repeated four times for each training city, using 759 randomly selected tiles each time. Figure 2 shows the resulting performance, as measured by average F1 score.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*qz_5c231DaadO2sOcjcAMQ.png" /><figcaption>Figure 2: Mean F1 score for models trained on one city (horizontal axis) and tested on another (vertical axis). Training is done with 759 tiles that are 200m on a side.</figcaption></figure><p>Not surprisingly, the best results are achieved when the model is tested on the same city it’s been trained on. Looking at the off-diagonal terms, none exceeds those on the diagonal. Beyond that, the matrix is strongly asymmetric in places. For example, a model trained on Khartoum and tested on Vegas does far better (F1=.44) than one trained on Vegas and tested on Khartoum (F1=.07). This illustrates that transferability is not commutative.</p><p>We can also use Figure 2 as a way of understanding which cities are most similar in the appearance of their imagery, at least as concerns traits relevant to building footprint identification. For each pair of cities we assign a similarity score as defined in Figure 3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*nQXwkqlyJ9aL-1YAxGi3zw.png" /><figcaption>Figure 3: Similarity score for two cities, denoted A and B. S is similarity, F is an F1 score, and the subscript <em>A→B</em> means the model is trained on city <em>A</em> and tested on city <em>B</em>.</figcaption></figure><p>Then, we can make a “map” by plotting the network of cities in such a way that the more dissimilar cities are pushed further apart, as shown in Figure 4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/323/1*5OWGocF5nu1UE-8D5_bq0A.png" /><figcaption>Figure 4: Graph of cities, with proximity based on similarity score.</figcaption></figure><p>Khartoum and Paris are the two most disparate cities; Paris is the most unique overall. An important point is that this might be due to meaningful features on the ground (e.g., Paris’s abundance of trees or distinctive architecture), but it could just as easily be due to incidental details of this collect (e.g., low light at the time the Paris photo was taken, or the choice to include a larger amount of rural terrain in the area of interest). Using more cities could help elucidate which factors are actually important.</p><h3>Geography and the Low-Data Falloff</h3><p>We’ll now switch gears and consider an anomalous feature back in Figure 1. As the amount of training data is decreased, model performance declines along with it. By the time there’s very little data, on the order of 12 tiles per city, performance plunges with even a small reduction in data. Three of the cities seem to fall off almost in unison, but the results for Khartoum in Figure 1 show a somewhat more gradual decline in performance as training data is reduced. This qualitative observation can be seen quantitatively in the numeric parameters of the fitted curves. Those curves (shown as dotted lines in Figure 1) follow a constant minus an inverse power law term. The power law exponents for the six curves from testing on cities other than Khartoum vary from city to city, but their average is 0.48±.10 (taking their standard deviation to be the uncertainty). The two curves from testing on Khartoum, however, have an average exponential term of only 0.20±.02, reflecting a less-rapid change in performance in the low-data regime. Additionally, the uncertainty for the model trained on Khartoum with 12 tiles is notably lower than the other individual cities’ 12-tile models.</p><p>In short, Khartoum is more resilient in a data-constrained situation. And since Khartoum is neither the best- nor worst-performing city in this regime, the issue appears not to be a simple matter of overall difficulty. Instead, something specific to Khartoum is going on.</p><p>So why is Khartoum different? We can rule out a chance statistical fluctuation. Here, we are rewarded for our patience in running enough trials to generate error bars. Comparing the one-standard-deviation error bars on the curves for Khartoum and Shanghai in Figure 1 shows that it is quite unlikely that their diverging behavior at low training data levels is due to chance alone.</p><p>We can also rule out an unusual distribution of building sizes in Khartoum. Figure 5 shows that the city’s distribution of building sizes is not particularly different from the other cities. Even the bimodal peak in Khartoum’s building size distribution is not unique, as Las Vegas shows the same feature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*m6Vfp_BAxypdUlImzpzQXQ.png" /><figcaption>Figure 5: Normalized histogram of building sizes in each city, as a function of the square root of their area.</figcaption></figure><p>Khartoum ranks neither first nor last in median building size, average count of buildings per tile, or percentage of land covered by buildings. As for the collect, the Khartoum imagery is not unusually dark or bright. It is the most off-nadir, but only barely so (compare Khartoum’s 25.7 degrees off nadir to Shanghai’s 20.5, both too low to see the <a href="https://medium.com/the-downlinq/the-good-and-the-bad-in-the-spacenet-off-nadir-building-footprint-extraction-challenge-4c3a96ee9c72">largest effects</a> of off-nadir angles).</p><p>What, then, can explain the difference? A visual inspection shows just how different Khartoum looks compared to the other cities. Figure 6 shows typical tiles from Khartoum and Vegas. We’ll compare these two cities in terms of low-level features likely to play a role in the first layers of the neural net: colors and edges.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*YSJazsF1UlzGBWlI4k-g_w.png" /><figcaption>Figure 6: Typical tiles from (a) Vegas and (b) Khartoum.</figcaption></figure><p>Figure 7 shows the range of colors in eight randomly selected tiles from Vegas and Khartoum. The colors are represented in the <a href="https://en.wikipedia.org/wiki/HSL_and_HSV">HSV color space</a>, with hue (color of the rainbow) on the x-axis and value (darkness) on the y-axis. The saturation (intensity) of each point indicates how frequently pixels with the given combination of hue and value occur in the imagery. Vegas is a colorful place, with recognizable features such as the distinctive light green of backyard swimming pools. In contrast, the color palette of Khartoum is more restrained. (Another look at the role of color using this same data set can be found <a href="https://medium.com/the-downlinq/panchromatic-to-multispectral-object-detection-performance-as-a-function-of-imaging-bands-51ecaaa3dc56">here</a>.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*yKKKUlQ0VYDLL5F9nVqk4A.png" /><figcaption>Figure 7: Distribution of colors across eight randomly-selected tiles from each city. The saturation of each pixel in the histogram scales logarithmically with how often the corresponding hue+value combination occurs in the tiles. Hues from 100 to 180 have been left off because the imagery includes almost no pixels in this range. Example objects of different colors are indicated in the Vegas plot, but this list is not exhaustive.</figcaption></figure><p>As for edges, we can get a feel for the distribution of edges in the sample tiles by applying a simple edge detector with image editing software. Figure 8 shows the result. Most edges in this Khartoum tile are associated with buildings, whereas edges are prevalent in both building and non-building parts of the Vegas image. That’s due to Khartoum having less variation in color and also due to other aspects of the scenery, such as the lower variation in ground cover type in Khartoum.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*F-BVdbkRv2Uz4rOJChb08g.png" /><figcaption>Figure 8: Edges in the tiles of (a) Vegas and (b) Khartoum shown in Figure 6. Images are processed with a difference-of-gaussians filter followed by an increase in brightness and contrast.</figcaption></figure><p>All of this suggests a hypothesis for why building footprint identification holds up better with little training data in Khartoum. It may be that a rule of thumb that “edges imply buildings” works exceptionally well in Khartoum, and such a rule would require very little training data to learn. The high variety of scenery in a city like Vegas (for which the high variety of colors and edges serve as simplistic proxies) would permit no such rule for low training data. Such variation, despite being a liability with low training data, nevertheless becomes advantageous given enough training data for the model to learn it.</p><p>While further evidence for this hypothesis could possibly come from hand-engineering an edge-based building footprint detector, actually proving it is a more difficult matter. One route would be to generate synthetic data, building synthetic city views with mixes of attributes to find out which ones lead to reduced performance degradation at the lowest training data levels. Another route would be to investigate the actual features being learning by the models with an explainable AI approach. Short of that, an understanding of this issue, like the issue of transferability discussed above, would benefit from studying a greater number of cities.</p><h3>Next Step</h3><p>In this post, we’ve investigated model performance in two especially challenging scenarios. First, we looked at models trained on a different city from where they were being put to the test. The performance matrix was shown not to be symmetric, and a way to visualize similarity was proposed. Second, we looked at models trained on very little data (12 tiles = half a square kilometer) to understand why some face steeper performance declines than others in the low-training-data limit. Comparing Vegas and Khartoum, the latter shows lower variation in hue, as well as a stronger relationship between edges and buildings. This inquiry demonstrated the challenges of uncovering what’s happening under the hood in deep learning.</p><p>In the next blog post in this series, we’ll look at the effects of changing a different variable. Instead of changing geographic location, it’s time to head back to Atlanta and try changing the model architecture itself.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3b9fb5033abc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-downlinq/robustness-of-limited-training-data-part-4-3b9fb5033abc">Robustness of Limited Training Data: Part 4</a> was originally published in <a href="https://medium.com/the-downlinq">The DownLinQ</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Robustness of Limited Training Data: Part 3]]></title>
            <link>https://medium.com/the-downlinq/robustness-of-limited-training-data-part-3-9df24c58c2?source=rss-ecbbd25b9fd2------2</link>
            <guid isPermaLink="false">https://medium.com/p/9df24c58c2</guid>
            <category><![CDATA[mapping]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Daniel Hogan]]></dc:creator>
            <pubDate>Tue, 17 Sep 2019 12:43:16 GMT</pubDate>
            <atom:updated>2019-09-17T12:43:16.013Z</atom:updated>
            <content:encoded><![CDATA[<p>When training a deep neural network to identify building footprints in satellite imagery, having more training data never hurts. But how much does more data help, and when is it worth the cost and difficulty of procuring it? <a href="https://medium.com/the-downlinq/robustness-of-limited-training-data-for-building-footprint-identification-part-1-8c55810b5ef9">We’ve seen</a> that performance rises rapidly with training data quantity when data is limited, but huge amounts of data give diminishing returns. In this post, we will explore how <em>geography</em> affects that trend, by studying performance versus training data set size with satellite images of four cities from around the world. We’ll see that the overall pattern of diminishing returns holds across the board, but that there are also some city-specific characteristics. In a follow-up post, we’ll look more closely at these city-specific features and see what happens when you train a model on one city and test it on another.</p><p>This post is the third in a series on the robustness of model performance with limited training data. The previous posts, which looked at imagery collected in just one place (Atlanta), are <a href="https://medium.com/the-downlinq/robustness-of-limited-training-data-for-building-footprint-identification-part-1-8c55810b5ef9">here</a> and <a href="https://medium.com/the-downlinq/robustness-of-limited-training-data-part-2-f51eb783823f">here</a>, but they’re not prerequisites for what follows.</p><h3>Method</h3><p><a href="https://spacenet.ai/spacenet-buildings-dataset-v2/">The SpaceNet 2 Challenge</a> saw the release of electro-optical imagery of four world cities from the WorldView-3 satellite, along with accompanying building footprint labels. We’ll use that data here, allowing for a comparison of the diverse urban environments of Las Vegas, USA; Paris, France; Shanghai, China; and Khartoum, Sudan.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*JFMEBmF3hCIRPTpSLKInRA.png" /><figcaption>Figure 1: Scenic views of (clockwise from top left) Vegas, Paris, Shanghai, and Khartoum.</figcaption></figure><p>The SpaceNet 2 imagery is chipped into tiles of 650 pixels on a side. At a ground sample distance of around 0.3m, that corresponds to about 200m. Weighting the four cities equally, the average tile has 21 buildings.</p><p>The deep learning model architecture used here for identifying building footprints is a lightly-modified version of a prizewinning <a href="https://github.com/SpaceNetChallenge/SpaceNet_Off_Nadir_Solutions/tree/master/XD_XD">model architecture </a>submitted to the <a href="https://medium.com/the-downlinq/the-spacenet-challenge-off-nadir-buildings-introducing-the-winners-b60f2b700266">SpaceNet 4 Challenge</a> by user “XD_XD.” The modified version, which was also used in the <a href="https://medium.com/the-downlinq/robustness-of-limited-training-data-for-building-footprint-identification-part-1-8c55810b5ef9">previous data robustness study</a> of Atlanta, uses one neural net instead of an ensemble.</p><p>For each city, models are trained with four different quantities of training data. These quantities — 12, 48, 192, and 759 tiles — are separated by multiplicative factors of about four. In addition, a fifth set of models is trained using tiles from all four cities at once. These combined models use the same number of tiles <em>per city</em> as the individual city models, so the combined model with the most data is trained on 3036 tiles, or 759 per city. Most one-city models are trained for approximately 100,000 image impressions and most combined models are trained for approximately 400,000 image impressions. After training, every model is tested against four test datasets, one for each city. This study uses the same <a href="https://medium.com/the-downlinq/the-spacenet-metric-612183cc2ddb">evaluation metric</a> as SpaceNet building footprint challenges: an F1 score describing how often predicted building footprints are similar to ground truth building footprints.</p><p>For each of the twenty combinations of geography/training data amount, the training and testing procedure is repeated four times with a different randomly-selected subset of the data each time. The reported F1 score is the average of the four trials, and its error bar is half their standard deviation. (It’s a half because the error of a mean falls as the square root of the number of samples.) This method represents an improvement over the technique used in the earlier Atlanta data robustness study. The averaging makes the data less jittery and the error bars smaller. Having a more precise measure of the average model performance allows us to perceive small geographic differences that might otherwise get lost in the random performance fluctuations that occur each time a model is trained anew.</p><h3>Results and Conclusions</h3><p>The main results from this campaign of training and testing 80 models can be found in Figure 2. The figure shows model performance, measured by average F1 score, versus the amount of training data used. The solid lines with error bars are the actual results, and the accompanying dotted lines are curves fitted to those results. The color of each graph indicates the city where performance is being evaluated (as specified in the figure’s legend). For each city, two scenarios are shown. The lower graph, shown in a darker shade, is the performance of a model trained only on data from that city itself. The higher graph, shown in a lighter shade, is the performance of the “combined model” trained on data from all four cities. Note that the x-axis is the amount of training data per city, so the combined-model results are based on four times more training data than the individual city results alongside them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*6QRwHFOFUb55uEFdzWewyg.png" /><figcaption>Figure 2: Average F1 score versus number of training images per city. “Individual model” denotes models trained only on the city for which their F1 score is shown, while “combined model” denotes models trained on all four cities (i.e., with four times as much training data).</figcaption></figure><p>There are a lot of conclusions to be drawn from this, so let’s give it a closer look!</p><p>To start, the plot exhibits the trend noted at the start of this post: When data is limited, performance rises rapidly with increased data, but that rate of growth cannot possibly be sustained once data becomes abundant. The logarithmic x-axis of Figure 2 visually disguises just how extreme the change is. To take the results trained on only Paris, for example, the slope of a line connecting the two lowest data points is sixty times greater than the slope of a line connecting the two highest ones. The trend can be seen more clearly in Figure 3, which shows the same data with a linear x-axis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*D5VYeqvjpPtfdHhUKe9CvA.png" /><figcaption>Figure 3: Identical to Figure 2, but with a linear x-axis.</figcaption></figure><p>This has a fortunate consequence. Compared to labeling a large portion of a city or of a collect, one can get most of the performance with a small fraction of the training data. For all eight scenarios shown in Figure 2, training on 48 tiles per city instead of 759 gives more than 3/4 of the performance using a mere 1/16 of the data. The total area of 48 tiles is less than two square kilometers.</p><p>The dotted lines in Figure 2 are empirical fits to the data points, and here we see another result from our previous analysis of Atlanta that holds up across a variety of geographies. These curves follow a “learning curve” functional form: a constant minus an inverse power law term, with three free parameters. Having a simple empirical fit like this is useful for interpolation and extrapolation, as shown with the Atlanta data set <a href="https://towardsdatascience.com/predicting-the-effect-of-more-training-data-by-using-less-c3dde2f9ae48">elsewhere</a>.</p><p>Although the different scenarios can all be fit with the same type of function, clearly they are not the same. The most noticeable difference is the wide range of F1 scores among cities given the same amount of training data. When it comes to identifying building footprints, some cities are more challenging than others.</p><p>Among these training scenarios, there’s a more subtle trend that also has implications for a real-world use case. Suppose I have a fixed amount of data for each city, and I want to get the highest performance possible. Should I train a separate bespoke model for each city, or should I pool all the data together to train one general-purpose model? In other words, does the benefit from having four times as much data outweigh the challenge of trying to create one generalized model for four cities from around the world? As Figure 2 shows, a model trained on four cities’ worth of data performs no worse than a model trained on the quarter of that data from the specific city where the model is being tested. That means the model architecture’s parameter space is big enough to hold four cities’ worth of learning, so it does not force any trade-off between performance and generality. In fact, it’s better than that: the model trained on the larger combined data set of all four cities consistently gets slightly <em>higher</em> performance than the city-specific models.</p><p>Figure 4 gives an example of reading Figure 2 to see this effect in action. If my only goal is to make the best possible model for, say, Paris, and all I have are 12 imagery tiles of Paris and 12 tiles apiece for three other cities, then a model trained on all 48 tiles from the four cities will, on average, outperform a city-specific model trained on the 12 tiles from Paris alone. Of course, if I can increase my Paris training data from 12 to 48 tiles, that will bring about an even greater benefit to the Paris model. But the key conclusion is that increasing training data by incorporating different geographies not only makes a more generalized model— it also makes a model that in every case outperforms city-specific models trained on less data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/1*km7cwxpcMVJh8xniBP6RxQ.png" /><figcaption>Figure 4: Detail of Figure 2, showing the change in Paris F1 score from quadrupling the amount of training data. Quadrupling the training data by getting four times more data from the same city (diagonal line) shows the larger improvement, but quadrupling the training data by incorporating equal amounts of data from three other cities (vertical line) still helps some.</figcaption></figure><p>Finally, since we’ve been comparing the results of this analysis to a previous analysis of Atlanta, it’s worth seeing the results side by side, as shown in Figure 5. The Atlanta data shows lower and more steeply rising F1 scores for the same number of training images. The cause is not a geographic difference, but rather a difference in how the data sets are structured. Although the Atlanta tiles are larger (~450m on a side instead of ~200m), they are lower-resolution and also geographically redundant: each Atlanta location is shown in 27 different tiles from different angles. As a result, five times more Atlanta tiles are needed to cover the same amount of area. This, along with the intrinsic difficulty of lower-resolution and off-nadir imagery, affects the shape of the curves and increases the number of tiles needed to achieve performance comparable to the SpaceNet 2 cities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/1*VshmxObSg8ca4bQNo0Algg.png" /><figcaption>Figure 5: F1 score versus amount of training data for the SpaceNet 4 Off-Nadir Atlanta data set and the SpaceNet 2 data set of world cities.</figcaption></figure><p>This discussion of training models on different geographic areas has focused on overall trends. These include the utility of small amounts of data and a simple empirical fitting function. The value of combining training data to build a global model has also been shown. In the next post in this series, we’ll take a closer look at two specific questions about the variations between cities: First, what happens if we train a model on one city and test it on a completely different city? Second, why does the performance seem to differ among cities when using the very lowest amounts of data? Stay tuned.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9df24c58c2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-downlinq/robustness-of-limited-training-data-part-3-9df24c58c2">Robustness of Limited Training Data: Part 3</a> was originally published in <a href="https://medium.com/the-downlinq">The DownLinQ</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>