<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Patrick Hagerty on Medium]]></title>
        <description><![CDATA[Stories by Patrick Hagerty on Medium]]></description>
        <link>https://medium.com/@Hagerty?source=rss-387385dac9e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*z3oAqz_dyrJRA0AISNSkDQ@2x.jpeg</url>
            <title>Stories by Patrick Hagerty on Medium</title>
            <link>https://medium.com/@Hagerty?source=rss-387385dac9e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 16:46:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@Hagerty/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Ensemble Models]]></title>
            <link>https://medium.com/@Hagerty/ensemble-models-8f5cf862ff09?source=rss-387385dac9e------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f5cf862ff09</guid>
            <dc:creator><![CDATA[Patrick Hagerty]]></dc:creator>
            <pubDate>Wed, 19 Aug 2020 18:37:00 GMT</pubDate>
            <atom:updated>2020-08-19T18:37:00.115Z</atom:updated>
            <content:encoded><![CDATA[<p>Credit card receipts, footfall traffic, and web scrapes are early examples of alternative data. New, diverse data sources are rapidly becoming available, expanding the possibilities for analysis. Datamonster™ is a platform for discovering, quantifying, and visualizing relationships amongst the data, while managing the entire <a href="https://en.wikipedia.org/wiki/Extract,_transform,_load">ETL process</a> — Datamonster™ makes working with the data easy.</p><p>Diverse data sources provide the opportunity to explore, compare, and combine data. Different models, that produce different prediction, based on the data sources can be confusing. To provide a singular, coherent analysis from multiple data sources, we employ an averaging algorithm called the <strong>Ensemble Model</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7V4A52xq_pGzBHf2xuxHHw.png" /><figcaption>Like the <a href="https://en.wikisource.org/wiki/The_poems_of_John_Godfrey_Saxe/The_Blind_Men_and_the_Elephant">parable of a blind man misinterpreting parts of an elephant</a>, models based on one data source can lead to biased predictions. The ensemble approach combines multiple perspectives to produce a more accurate prediction. Analogously, financial models built from foot traffic, credit card receipts, and website visits may each have their own biases but in combination become more reliable.</figcaption></figure><p>The Ensemble Model combines multiple models into one. In this post, we discuss the construction of the ensemble model and analyze its advantages in accuracy and in precision over using multiple, member models.</p><h3>When to use an Ensemble Model</h3><p>There are two extremes for creating models from multiple data sources:</p><ul><li><strong>Single, Complex Model:</strong> if there is enough data, one can create a single complex model that inputs the data sources and produces a single output. Optimizing the parameters in the model (making sure the model fits the data) can be difficult and computational expensive. If data is scarce, complex models are at risk of being overfit and lack predictive power.</li><li><strong>Ensemble of Multiple, Simple Models:</strong> if data is scarce, then it is advisable to create a simpler model. Multiple models are required to incorporate multiple data sources., and afterwards we combine those simple models into a single, Ensemble Model.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wewxMSDj0QQuSg7Lk-3z5w.png" /><figcaption>Some models are complex like the neural network called <a href="https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf">AlexNet</a> (left). Optimizing the large number of parameters in AlexNet requires extreme amounts of data and computing powers. In contrast, linear regression (right) has two parameters. The linear model may be too simple to describe the nuances of the data like seasonality, but it greatly reduces the risk of overfitting.</figcaption></figure><p>By judiciously combining simple models, the Ensemble Model increases accuracy, tightens confidence levels, and mitigates overfitting. The Ensemble Model is a good choice when data is scarce. The limitations of the Ensemble Model include the inability to model complex interdependencies of the data sources.</p><p>For the modeling application of investors, data is often scarce — think of quarterly earning reports. This scarcity imposes restrictions on the complexity of models. The Ensemble Model benefits from multiple data sets, mitigates overfitting, and provides one coherent prediction.</p><h3>Weights of the Ensemble Model</h3><p>The Ensemble Model is a weighted average of the member (simple) models where more accurate models are given more weight and noisier models are given less weight. If the noise in the member models are independent then the optimal choice of weights is proportional to the inverse of the variance of the model— or <a href="https://en.wikipedia.org/wiki/Inverse-variance_weighting">inverse-variance weighting.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*84AxI5j7v114j68E8w2Ofg.png" /><figcaption>The Ensemble Model combines multiple models together to produce an averaged prediction. The variance of the prediction is represented by the radius of the prediction circles. The ensemble prediction theoretically has smaller variance than the those of the member models.</figcaption></figure><h4>Ensemble Weights of correlated weights</h4><p>For each member model that is part of the ensemble, we have a prediction of a future value and a variance associated with that prediction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*zqGF3GywS320iT0gi0A80A.png" /><figcaption>The Ensemble is a weighted average of the member models. The weights in the Ensemble sum to one.</figcaption></figure><p>If the noise model for each member model is independent, then the optimal choice of weights in the Ensemble Model construction is inversely proportional to the variance of the member models:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6pAt2hpnr-TXIgduEZWOZQ.png" /></figure><p>In the case of correlation in the noise of the member models, the weights are chosen to minimize the variance in the Ensemble’s prediction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EzyCeXgWkMEC8hVBgzG3hQ.png" /><figcaption>The weights are the solution to the optimization problem.</figcaption></figure><p>The optimal weights can be difficult to compute exactly, but can be numerically approximated using a gradient descent method starting at the optimal weight assuming independent noise.</p><p>The variational choice of weights ensures the variance in the ensemble’s prediction decreases with more models — the Ensemble Model benefits from including more models.</p><h3>Choice of Member Models in the Ensemble</h3><p>To prevent overfitting when data is scarce, member models are typically on the simpler end of the spectrum: meaning the models have a very small number of parameters.</p><h4>Simpler Models: Linear Regression</h4><p>Linear regression models are less likely to overfit but have explicit weaknesses including:</p><ul><li><strong>No growth over time.</strong> Each data point is equally weighted whether the point was from 20 quarters ago or last quarter. For linear regression, the error tends to be smaller near the middle of the time window with larger errors near the ends.</li><li><strong>Fails to capture acceleration (or deceleration). </strong>The relationship between the data source and the metric may be non-linear.</li><li><strong>Seasonality.</strong> Some relationships change with seasons. Modeling each seasonal component separately may require more data than is available and result in overfitting. A single linear regression will average the seasonal variations.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DXRXT6UUmfXRIyEG_QdxqA.png" /><figcaption>The same linear regression data is plotted in time series (left) and scatter plot (right). The scatter plot does not show the time dependency of data, aside from the quarter label. The time series shows the seasonal pattern to the errors and perhaps an acceleration in growth.</figcaption></figure><h4>More Complex Models: LASSO and Seasonal ARIMA(X)</h4><p>More complex models such as <a href="https://en.wikipedia.org/wiki/Lasso_(statistics)">LASSO</a> or <a href="https://en.wikipedia.org/wiki/Autoregressive_integrated_moving_average">ARIMA(X)</a> can be used either alone or as member models in the ensemble. These algorithms generally have more parameters and consequently require significantly more data to build than simpler linear models.</p><p><a href="https://en.wikipedia.org/wiki/Lasso_(statistics)">LASSO (Least Absolute Shrinkage and Selection Operator)</a> is a regression model akin to linear regression (and sharing the same major weakness — a lack of time dependence). The difference is that the parameters in the model are not only chosen to minimize the mean-squared error (like linear regression) but also try to make the parameters small. LASSO relies on regularization to learn the order of importance of each data source included in the model, but LASSO is extremely susceptible to overfitting.</p><p>ARIMA(X) is collection of time-dependent models that generally have more parameters than simpler models. It is possible to integrate data sources as exogenous variables— the X in ARIMA(X) —but doing so increases the risk of overfitting.</p><p>While both of these models are alternatives to the Ensemble Model, they can also be included as member models in the Ensemble Model. We can improve accuracy and reduce variance in our predictions by including a simple seasonal ARIMA model (no exogenous variables) into the Ensemble Model. This choice benefits from the advantages of time dependent models while mitigating the risk of overfitting: including exogenous variables in the ARIMA model did not improve the accuracy of the model. LASSO is similar in spirit to the weighted averaging of additional data but chooses the weights differently. The choice of weights accounts for more complex relationship between different data sources but in our financial models has inferior results to the choise of weights in the Ensemble Model.</p><h3>How does the Ensemble Model compare to choosing the best of multiple models?</h3><p>Choosing the best of multiple models ignores less accurate models entirely. The Ensemble Model tempers, but still includes, the contribution from less accurate models.</p><p>A direct comparison is difficult because there are a large number of variables including: the accuracy of the simple models, the number of models, the number of data points, and the diversity of the simple models. For a wide range of these variables, the Ensemble is demonstrably more accurate than the best of multiple models. However, the ensemble has diminishing returns against more complex models as the data become less scarce, but computationally the ensemble still scales better than more complex models.</p><h4>Abstract Example</h4><p>The first example that we discuss is to compare the Ensemble Model, created from five single-variable linear regression models, with two alternatives models:</p><ul><li>Choosing the linear regression model with the lowest (best) Mean Squared Error (MSE). This computational complexity scales linearly with the number of data sources.</li><li>Choosing the two-variable multi-linear regression model with the lowest MSE. This computational complexity scales quadratically with the number of data sources.</li></ul><p>For this experiment, we simulate a noisy, linear relationship between a dependent variable and five independent data sources. We build five linear regression models and all ten multi-linear regression models. All of the data sources are constructed using a normal distributed variance to model the relation between the dependent and independent variables. We partition the data into two halves: one for training and and one for testing. We compare the Ensemble Model with the best linear and multi-linear models. The plot shows the median ratio of mean squared error (MSE) for the Ensemble Models and the best individual models over 10,000 trials.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/1*fh3XfIIkePOuHM4Qaxar5A.png" /><figcaption>Comparing Mean Squared Error (MSE) between Ensemble and Best Single Model of Ordinary Least Squares (OLS). Values less than one mean the ensemble is better. The median ratio of MSE of the ensemble model over the test data is compared to the MSE of the regression models. The blue points compare the single variable regression models and the red points compare the two variable regression models.</figcaption></figure><p>The results of the experiment support the utility of the Ensemble Model over choosing the best of linear regression model. The computational cost of the Ensemble Model scales linearly with the number of models. The best two-variable models starts to consistently match the Ensemble Model’s accuracy when there are forty or more data points (for the noise models of our data sources) in our experiment.</p><h4>Datamonster™ Example</h4><p>In DataMoster, our clients often build models based on several different data sources to help predict a quarterly reported value. The scarcity in the data limits the complexity of models and fits well with the Ensemble Model. The member models comprise single variable linear models. In addition to the linear models, we include a seasonal ARIMA (without exogenous variable) model into the Ensemble. This ARIMA model is not based on data sources but rather gives a prediction of the metric based solely on historical values of the metric.</p><p>One of the challenges with working with real data is that models are not perfect — the relationship between the data source and what you want to predict may have non-linear components and/or time dependencies that are not captured by the linear models. We include a seasonal ARIMA model as part of the ensemble to capture some time dependency. Another time dependency included in the Datamonster™ ensemble is the computation of weights is based on a recent model performance.</p><p><strong>Datamonster™ Ensemble:</strong></p><ul><li>Step 1: Compute seasonal ARIMA model for metric, using no exogenous data, including the variance for the next two predictions. Adjust for the length of the quarter if appropriate.</li><li>Step 2: Compute linear regression models between data sources and metric, using all data available.</li><li>Step 3: Compute the means and covariance of linear model residuals over the last twelve quarters. The choice of twelve quarters is a compromise between enough data to build a good model and having the data be relevant to a prediction.</li><li>Step 4: Compute the weights proportional to the inverse of the variance in the models residuals over the last twelve quarters. The weight for the seasonal ARIMA model is inversely proportional to the variance in the model’s prediction. Normalize the sum of the weights to one.</li><li>Step 5: The ensemble prediction is the weighted sum of the models’ predictions (shifted for the linear models’ biases over the restricted time window).</li><li>Step 6: The variance of the prediction is the prediction variance of the weighted sum accounting for covariance of residuals.</li></ul><p>For this particular example, we build a model to predict quarterly revenues of a company using several data sets. The company and data sources are anonymized throughout the sequence of graphs that demonstrate the ensemble.</p><p>First we create a seasonal ARIMA model to predict future values of the metric — we use the term <strong>KPI</strong> to represent this metric representing key performance indicator. The ARIMA model depends only on historic values of the metric.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*9l1PYWIkWhVDYGUCzTLtXg.png" /><figcaption>Seasonal ARIMA model</figcaption></figure><p>The seasonal ARIMA model serves as good baseline of comparison. Comparing the ARIMA model to other member model is a good sanity check. If another member model’s prediction deviates significantly from the ARIMA model’s prediction then the member model maybe expressing valuable insight of the metric or exposing an issue with the member model (lacking time dependency) or with the underlying data source.</p><p>The next graph shows the predictions of the ARIMA model and the linear regression model for the “best” data source, determined by the variance of the residuals over the specified time window. We see during the holdout period, there is a significant difference between the ARIMA model and the best linear regression. This deviation is partially explained by the shifted 53-week year and shifted Q3 period that is part of the linear regression models but does not affect the ARIMA model. During the holdout period during this example, the linear regression model of the data source is in the right correctional direction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*aNVCD8ip8d-2mA_eOCTgNg.png" /><figcaption>Seasonal ARIMA and Linear Regression Models.</figcaption></figure><p>We plot the regressions for each of the ten “best” alternative data sources in DataMonster™. This plot provides visual context for the models and the deviations from the ARIMA model. During the holdout period, we can see that the “best” data source is on the extreme side of the range of regressions and that other “good” data sources effectively balance this prediction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*Dz0yHwgGxBsM3D7ohlVr8w.png" /><figcaption>Seasonal ARIMA Model and Linear Regression Models. Some of the member models are visually more accurate than others.</figcaption></figure><p>When we create the ensemble, we compute the weights using the residuals from the last twelve quarters. The weights are inversely proportional to the variance of these residuals; the ensemble is the weighted average of each of the models shifted by the respective mean residual error over the last twelve quarters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*Ntmi4M6RINnauWYsYY3xgg.png" /><figcaption>Seasonal ARIMA model with Linear Regressions and Ensemble.</figcaption></figure><p>The weights in the ensemble model provide a more accurate prediction than the best regression model or the seasonal ARIMA model. The actual improvement in accuracy and variance depends on the metric and comprising member models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*DkkgplAzO5ma8jQ0rSbuUw.png" /><figcaption>Seasonal ARIMA model with Linear Regressions and Ensemble. The blue region is plus/minus one standard deviation from the predicted mean. The light blue region is plus/minus two standard deviations from the ensemble mean. When the regression models are tighter then the variance is lower.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*YUr8sSnN2xJ9x9C2UBtKBA.png" /><figcaption>Seasonal ARIMA, Best Linear Regression, and Ensemble Models.</figcaption></figure><p>There are cases, unlike the presented one, where the Ensemble model is inferior (measured by larger variance) to a linear member model. Possible reasons for these cases include:</p><ul><li>Inaccurate estimation of member model’s variance.</li><li>Dependence in residuals of member models lead to non-optimal choice of ensemble weights.</li><li>All the member models are noisy, including the ARIMA model.</li></ul><p>The choice of weights in the Ensemble is typically a robust model if at least one of the member models is reasonably accurate.</p><h3>Conclusions</h3><p>The Ensemble Model combines multiple data sources and mitigates overfitting. Statistical calculations show that ensemble model improves with with more data sources (with some independence assumptions) in accuracy and variance. The robust nature of the ensemble properly accommodates noisy (and perhaps superfluous) models.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f5cf862ff09" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Entropic Ghosts]]></title>
            <link>https://medium.com/@Hagerty/entropic-ghosts-35670292bc87?source=rss-387385dac9e------2</link>
            <guid isPermaLink="false">https://medium.com/p/35670292bc87</guid>
            <category><![CDATA[facial-recognition]]></category>
            <category><![CDATA[entropy]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[privacy]]></category>
            <dc:creator><![CDATA[Patrick Hagerty]]></dc:creator>
            <pubDate>Fri, 01 Mar 2019 16:07:10 GMT</pubDate>
            <atom:updated>2019-03-01T16:07:10.009Z</atom:updated>
            <content:encoded><![CDATA[<p>In this post, we investigate methods of redacting information from a machine learning classifier. The topic is inspired by the collection of privacy attacks on neural networks contained in <a href="https://github.com/Lab41/cyphercat">Cyphercat</a> that exposes information that may reside in the structure of a classifier:</p><ul><li>membership inference: was a data sample used in training the network,</li><li>model inversion: recover training data from the network.</li></ul><p>Suppose the creator of a classifier is aware of the Cyphercat attacks and wants to redact or remove information about an entire class from the classifier. Possible motivations include:</p><ul><li>sharing a classifier that has been trained on proprietary/sensitive training data,</li><li>governmental privacy requirements, e.g. removal of identities from facial recognition machine learning applications.</li></ul><p>Attacks and countermeasures go hand-in-hand — neural networks can be hardened against such privacy attacks in a variety of ways including with generative adversarial networks. Successful hardening often obscures the informational theoretic foundation of the countermeasure. To better understand how a privacy countermeasure works, we compare two simple privacy countermeasures using an informational theoretic metric.</p><h4>Example</h4><p>An example of this application is as follows. Suppose a social network uses a neural network to perform facial recognition on shared photos. Now suppose a court orders the social networking company to “forget” information on how to identify Person A. It is reasonable that the company has spent thousands of machine hours training the neural network for facial recognition and doesn’t want to incur the cost of complete or partial retraining the classifier. Instead the company removes all nodes in the last layer (or layers if the last layer is a Softmax layer).</p><ul><li>How much information about Person A still resides in the redacted network?</li><li>Can you tell if this algorithm was used over retraining?</li></ul><p>While these are simplistic questions have merit individually, the real utility would be using this in combination with other privacy attacks or to provide insight into when another privacy attack is likely to be successful: to provide a metric for a countermeasure to a privacy attack.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*ChvWMOTUdRjzA4F2I_YUeg.png" /><figcaption>Cartoon of Facial Recognition Redaction. In this cartoon, Pac-man is redacted from the neural network, but information (ghosts) related to Pac-man is latent in the structure of the network. We use entropy to measure the amount of latent structure of the network: higher entropy means more latent structure.</figcaption></figure><h4>Neural Network Details</h4><p>The approach to assess if a class was redacted relies on information theoretic measurements of the output of the neural network. The last output layer of a neural network is a <a href="https://en.wikipedia.org/wiki/Softmax_function">Softmax</a> layer that normalizes the output into a probability distribution on classes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o7edw02F9r5CCF6UByECxA.png" /><figcaption>Example neural network for image classification. The input image is processed by the neural network and the last layer of processing is the Softmax layer. The output is a probability vector where each dimension corresponds to the probability of the label assigned to the image. Typical metrics measure the actual label against the labels of the largest (or largest k) probabilities.</figcaption></figure><p>The thesis of this investigation is that this output probability distribution (the values in the red circles) will look different depending on how a class is redacted from a neural network. This post will present methods of redaction and metrics for comparing the methods.</p><h3>Redacted Classes</h3><p>A neural network used in image classification will take as input an image and produce a probability vector of length N, representing the probability that the image is classified into each of N classes. We explore two different ways to redact or remove as a possible output of the tool.</p><ul><li>Retrain. The more expensive way is to retrain the neural network without using images in the redacted class and remove the redacted label as a possible output. This approach requires more computation to train than the subsequent alternative. More importantly, the absence of the redacted training data may make the network inferior at identifying the remaining classes.</li><li>Clip and Renormalize. A cheaper (computationally) approach is to remove all connections from the redacted dimension in the penultimate layer of the neural network. The last Softmax layer renormalizes the probabilities into an N-1 long probability vector.</li></ul><p>In each of the two cases, a shorter probability vector is the output of the redaction. We investigate if one can distinguish the redaction methods given the two outputs. Additionally, we measure the difference in accuracy for the remaining classes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tj4HJu26fpZWmxQ_UcHc2w.png" /><figcaption>Retrained Redaction. In this neural network, the images that have the dog label are removed prior to training. The weight of the last layer (and likely interior layers) are different than the network trained with the dog label included.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g0PCSo6F0Y_aS0fjNJR-zQ.png" /><figcaption>Renormalized Redaction. In this neural network, the dog label is removed from penultimate and the final layer. The other weights have the same values as the original neural network and are renormalized by the Softmax layer.</figcaption></figure><h3>Metrics: entropy</h3><p>The information embedded in a probability vector is measured by <strong>entropy</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*xci99i7NB9IDlXJmNXQyBg.png" /><figcaption>Shannon Entropy of a discrete probability distribution/vector.</figcaption></figure><p>When applied to the output of an image classifier, a smaller value of Shannon entropy of the output vector represents more certainty in the classification — if the Shannon entropy is zero then the classifier is completely certain of the assigned label. As the output vector becomes flatter (the values of each dimension becomes the same value) then the Shannon entropy is maximized as is the uncertainty of the label.</p><p>Classification usually is dependent on the dimension with the largest probability — this corresponds to min-entropy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/315/1*yOGmfprk9fsIWaSYCx3ACw.png" /><figcaption>Min-entropy measures how likely the most likely label is.</figcaption></figure><p>The entropies (Shannon and min-entropy) are more sensitive to changes than other typical metrics. There is a spectrum on entropy called Rényi Entropy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/394/1*m3FcB50OCrCif_ylPeBfSQ.png" /><figcaption>Rényi Entropy for parameter alpha. Shannon and min-entropy are limiting cases where alpha tends to 1 and to infinity respectively.</figcaption></figure><p>In general, the better classifier will have a smaller output entropy.</p><h3>Examples</h3><p>We use the <a href="https://github.com/Lab41/cyphercat">Cyphercat</a> code base to build examples of redacted neural networks. The two neural networks are the<a href="https://github.com/Lab41/cyphercat/blob/master/Utils/models.py#L106"> mlleaks_cnn</a> trained and tested against the <a href="https://www.cs.toronto.edu/~kriz/cifar.html">cifar10</a> data set. There are ten classes in the <a href="https://www.cs.toronto.edu/~kriz/cifar.html">cifar10</a> data set. We perform a redaction for each of the classes and compare the entropy and the recall for the specific remaining (unredacted) classes. The comparison is made between the retrained method and the renormalized method of redaction.</p><p>The recall of the simple neural network maxes out around 80%. Since each class may have distinct features that affect the difficulty of classification, we average the recall for each class over different redacted classes. For example, we compute the recall for the “bird” class for when we redact the “cat” class, the “deer” class, etc. separately.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*lZREFb9RrqK2gZwdm00CHA.png" /><figcaption>Recall for different classes. The blue bars represent the performance of the neural network trained excluding the redacted class (retrained) and the orange bars represent the neural network trained with the redaction after training (renormalization). Performance measured by recall are similar for the two approaches and is averaged over the each redacted class.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*OHJVuKOwhp5CZ27yDBUgDw.png" /><figcaption>Recall for different classes. The blue bars represent the performance of the neural network trained excluding the redacted class (retrained), the orange bars represent the neural network trained with the redaction after training (renormalization), and the green bars represent the original neural network. A slightly worse recall in the original neural network may be a result of the increased number of original classes.</figcaption></figure><p>The two methods of redacting classes appear to have similar recall rates for each of the remaining classes. Using this performance criterion alone was insufficient in distinguishing if a class had been redacted at all.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*e01A6qdPFZa9ece2FKPpyw.png" /><figcaption>Mean Shannon Entropy of Two Redaction Methods. Actual choice of alpha for Rényi entropy is 1.001 to simplify computation. The blue bars represent the entropy of the neural network trained excluding the redacted class (retrained) and the orange bars represent the neural network trained with the redaction after training (renormalization). Entropy is higher for the renormalization method.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*dr3O8pwQKZqz-pY2DHkSkg.png" /><figcaption>Mean Shannon Entropy of Two Redaction Methods with Entropy of Original Classifier. Actual choice of alpha for Rényi entropy is 1.001 to simplify computation. The blue bars represent the entropy of the neural network trained excluding the redacted class (retrained) , the orange bars represent the neural network trained with the redaction after training (renormalization), and the green bars are the entropy of the output of the original classifier. Entropy is higher for the original than the entropy of the redaction methods — this is expected because there is an additional class.</figcaption></figure><p>The Shannon entropy of the output probability vectors appears to distinguish between the two redaction methods . Retraining may not affect the recall performance significantly, but a decreased entropy demonstrates that the neural network is more confident in the predicted label.</p><p>The simpler method of renormalizing the probability vector and removing unused connections in the neural network retains information about the redaction in the entropy of the output: an entropic ghost.</p><p>Optimizing over the spectrum of Rényi entropies will improve our ability to distinguish between the two redaction methods. For the purposes of this post, Shannon entropy proved sufficient.</p><h3>Acknowledgements and Future Directions</h3><p>Identifying a redacted class is the first step in recovering more information about the class. Future research will consider how to adapt the inversion and membership attacks to a redacted class.</p><p>We would like to thank the <a href="https://www.lab41.org/">Lab41</a> for valuable discussions and insight into hardening neural networks. Code used in this experiment will be contributed to the <a href="https://github.com/hagerty/cyphercat">cyphercat</a> repository.</p><h4>Lab41 is a Silicon Valley challenge lab where experts from the U.S. Intelligence Community (IC), academia, industry, and In-Q-Tel come together to gain a better understanding of how to work with — and ultimately use — big data.</h4><h4>Learn more at <a href="http://www.lab41.org/">lab41.org</a> and follow us on Twitter: <a href="https://twitter.com/_lab41">@_lab41</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=35670292bc87" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Time Series Analysis for Alternative Data]]></title>
            <link>https://medium.com/@Hagerty/time-series-analysis-for-alternative-data-c5ebd39b2f69?source=rss-387385dac9e------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5ebd39b2f69</guid>
            <category><![CDATA[timeseries]]></category>
            <category><![CDATA[alternative-data]]></category>
            <category><![CDATA[forecast]]></category>
            <category><![CDATA[investment]]></category>
            <dc:creator><![CDATA[Patrick Hagerty]]></dc:creator>
            <pubDate>Wed, 23 May 2018 14:48:56 GMT</pubDate>
            <atom:updated>2018-05-23T16:14:51.068Z</atom:updated>
            <content:encoded><![CDATA[<p>As Chief Data Scientist at <a href="http://www.adaptivemanagement.com/">Adaptive Management</a>, my goal is to make data simple for our clients. Alternative data sources (such as credit card receipts, footfall traffic, satellite imagery, etc.) are non-traditional data sources that offer a scalable, quantified measure of performance to support theses on a current or future investment. Time series analysis is essential for estimating the values of fragmented, missing, or future data sources — these situations are common in modeling key performance indicators.</p><p>While time series analysis may appear as a black box, it is our goal to bring some transparency to the algorithms that are available in DataMonster™. We highlight the integral steps of time series analysis but avoid going into the technical details of parameter optimization.</p><h3>Time Series Analysis</h3><p>Time series analysis aims to understand the structures and patterns in time series data. These patterns are useful to describe trends and seasonal properties of the data as well as to forecast future values of the data. Simplicity is prefered over complexity in the analysis for the following reasons:</p><ul><li><strong>Reduce the risk of over-fitting.</strong> A model has a number of parameters with optimized values over a subset (training set) of the time series — the remainder of the time series becomes the validation set. Over-fitting occurs when the model accurately describes the training set but fails to accurately describe the validation set. For time series, the validation set is often a hold out period. If the model fails to accurately describe the training set then the model may be too simple or the training set may be too small.</li><li><strong>Minimize the computational cost of the analysis.</strong> When computing forecasts for a large number of time series, computation costs and time are critical components. It is common to train neural networks for weeks or months — this may be too long to be applicable to the problem. When working with alternative data sets, we often analyze a large number of time series. This number can be further increased by subsampling the time series on another data field. In all cases, you should expect an increase in accuracy for extra computational cost.</li><li><strong>Tractable error rates.</strong> Error rates in complex models are often computed using Monte-Carlo simulations. Statistically properties of some simpler models can be computed analytically (as opposed to numerically), resulting in easier to compute error bounds and lessening the data requirements. More complex models may require significantly more computation and data to provide reasonable confidence intervals for associated forecasts.</li><li><strong>Interpretability.</strong> Some complex models are essentially black boxes with parameters that are difficult to relate to real world phenomena. The simpler model has to be more judicious in its choice of parameters. To support a thesis, it is helpful to translate the value of a parameter to a real world metric. Insights from subject matter experts may constrain the parameters, increasing the relevance of the model.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ituqy-jSyeM_fKF81R9bhQ.png" /><figcaption>Simplicity and Complexity of Models. If one is attempting to forecast a national election result then state-wide fidelity may be too simplistic for accurate predictions. In the absurd extreme, too much information (e.g. genetics) would result in an overly complicated model. Finding the correct number of parameters (perhaps county-wide resolution) is the challenge of creating a good model.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kwzq-u23u61PIGMH1IXFAA.png" /><figcaption>Underfitting and Overfitting Model. The line does not fully describe the structure of the true function. The degree 15 polynomial describes the deviations more than the underlying true function.</figcaption></figure><p>The right amount of complexity/parameters in a model balances extracting structures and of patterns from the data with overfitting. The goal is have the deviations from the model be well-modeled by a stochastic process. For example, the flip of a coin is deterministic with complex enough physical model that includes aerodynamics and rigid body mechanics. For simplicity, it is usually sufficient to model the coin flip as a Bernouli random process, where the probability of heads and of tails are included in a model but all of the physics is absorbed into the noise model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*uQpRyZg44wCjZSud4UUE5A.jpeg" /><figcaption>Coin Flip: deterministic process modeled as a random event.</figcaption></figure><h3>The Path to Stationarity</h3><p>For our forecasts, we structure the time series model to include the general trend of growth, annual (and weekly) seasonal components, and influence from holidays. Our forecast is powerful if the residuals of the model, the components not included in the model, have nice statistical properties. For time series analysis, the goal is to have the statistical properties of the excluded components be well modeled by a stationary process, meaning the the statistical properties do not change over time.</p><p>We will describe the transformations of the time series that define our model; each additional transformation brings the residuals closer to the desired stationary state.</p><h4><strong>Trend or Linear Growth</strong></h4><p>Trend is the long term growth of a time series without any seasonal or holiday components. We compute a rolling mean over a year to model the trend of a time series. To simplify the rolling mean, we compute a piecewise-linear model with optimal choice of change points. The distributions of the times of the change points and of the linear coefficients of the linear models allows us to forecast the trend into the future. More complex algorithms can be employed to describe the trend at an increased computational cost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nhmWbrzNlyq4H_L64G9wbw.png" /><figcaption>Extracting Trend. The linear growth is extracted from a moving average. The noise model from the forecast is derived from the distribution of change points in the piecewise-linear approximation. The validity of the linear extrapolation may be limited in time.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XdI2Yz56URMupAquEAn3ng.png" /><figcaption>Removal of Trend. The red points are the original time series and the blue points are after the removal of the linear trend removed. It is visually evident that the residuals are not stationary.</figcaption></figure><h4><strong>Seasonality</strong></h4><p>Seasonality is the periodic component to a time series. For many data sources, the natural periods are yearly or weekly. To extract the seasonal components, we filter the top Fourier modes (a calculation that sorts the amplitudes of different frequencies of the time series).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*pp82pbNUXRCHhkPSZt1BYA.png" /><figcaption>Filtration of Time Series to Find Seasonality. The largest amplitude frequencies are used to forecast seasonality in the model. The filtration process starts in the time domain (upper left). Next we transform to the frequency domain (upper right). We remove weaker frequencies (lower left). The remaining frequencies (the ones that pass through our filter) are used to model seasonality in the forecast (lower right).</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R1SAD2lsq8ZXblY9dshYdA.png" /><figcaption>Seasonality Filter. The red point are the residuals of the time series after removals of the linear growth and seasonal component. It is close to appearing stationary but there are still outliers that may be accounted for by a holiday analysis.</figcaption></figure><h4><strong>Holidays</strong></h4><p>Holidays are points in the series where we anticipate large deviations from the model. Sometimes these holidays are on a fixed date such as Christmas and sometimes the holidays move around such as Thanksgiving. We generalize holidays to describe days that are also near typical holidays such as Black Friday. We extract the average value of the time series on the holidays from the signal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K1Jq1vWZBrSdEKoke5qajA.png" /><figcaption>Filter out Holiday Deviations. The red points are the residuals after extracting the linear growth, seasonality, and holiday components. There are still some large deviations that are likely a result of unknown holiday-like events or noise in the data. One can balance overfitting with removal of the larger deviations. But for our purposes, we treat the blue time series as stationary.</figcaption></figure><p>This decomposition is not perfect. There may be holiday-like events, e.g. large deviations from the model, that we do not include, such as an Amazon Prime day. The goal is to arrive at a stationary distribution after extracting trend, seasonality, and holidays. By stationary, we mean that the remainder is well modeled as a random process with constant mean and constant variance.</p><h4><strong>Forecasting</strong></h4><p>Putting the components together, we can forecast the time series into the future. Assuming the noise in each component is normally distributed and independent, the variance of the forecast is the sum of the variances of each component:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c-e7SppvK_W-1XaMnZ8myw.png" /><figcaption>Forecast of Time Series.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*Hq9NHSKngw0KzNe1PjzR5Q.png" /><figcaption>Forecast of Time Series. The uncertainty grows as one tries to forecast further into the future.</figcaption></figure><p>Each of these components along the path to stationarity are part of the complete model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xsaq5NKzkveBEIyLhjARRw.png" /><figcaption>Formula for Time Series Forecasting.</figcaption></figure><p>The complexity ignored by the model is modeled by the Gaussian noise. After extracting the trend, seasonality, and holiday components, we can compute the statistical parameters of the residual noise.</p><p>To fit the model to training data in the time series, we optimize the choice of parameters to minimize the mean squared error (MSE) between the model predictions and the training data.</p><h4><strong>Test for Stationarity</strong></h4><p>There are multiple ways to determine if the model is sufficiently descriptive for forecasting. The primary test is a backtest that holds out a subset of the data and uses the model to forecast the held out values.</p><p>Even without backtesting, we can estimate how good the model is by the statistical properties of the residuals. The model is well-suited for forecasting when the residuals are described by a stationary process. There are several tests (e.g. <a href="https://en.wikipedia.org/wiki/Dickey%E2%80%93Fuller_test">Dickey-Fulle</a>r or <a href="https://en.wikipedia.org/wiki/Neyman%E2%80%93Pearson_lemma">Neyman-Pearson</a>) to determine if this is indeed the case. The gist of the tests is to present two models — one non-stationary and one stationary — and determine which model the data fit better.</p><h3>Alternative forecasts and ARIMA</h3><h4>Bayesian Inference</h4><p>Having prior assumptions on the distributions of the parameters of the model enables us to employ Bayesian inference to improve the optimization. While fitting the model, we jointly optimize the trend, seasonality, and holiday parameters. However, we obtain similar quality results with a fraction of the computational cost by judiciously prescribing the location of changepoints in the trend and subsequently decomposing the optimization by component.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3UFzv6U8i0nYl46JC92Bsw.png" /><figcaption>Order of Magnitude of More Complex Time Series Analysis Methods. The accuracy levels of other time series analysis techniques did not improve accuracy of forecasts. However, the computational time is significantly cheaper for the presented method (labeled Time Series Analysis). Furthermore, our analysis is parallelizable across GPUs and have very modest memory requirements.</figcaption></figure><h4>ARIMA</h4><p>One powerful collection of algorithms for computing forecasts are grouped together and called ARIMA (AutoRegression, Integration, and Moving Average). While ARIMA algorithms have some potential advantages, many of the techniques one would use to get to stationarity for ARIMA are precisely what we compute in our decomposition of time series into trend, seasonality, and holiday components. Several reasons why we choose not to employ ARIMA at scale are the following:</p><ul><li>typically more expensive (with respect to the number of calculations and time to compute) than the decomposition described above especially for yearly seasonality on daily data,</li><li>have hyperparameters that need to be optimized,</li><li>typically good for short-term forecasts but start to drift after that,</li><li>computation of confidence intervals and uncertainty is more difficult,</li><li>lacks the ability to understand the parameters/decomposition.</li></ul><p>A more technical discussion that compares ARIMA to a Bayesian approach can be found <a href="https://multithreaded.stitchfix.com/blog/2016/04/21/forget-arima/">here</a>.</p><h3>Summary</h3><p>The strength of our model is the ability parallelize its computation across a large number of time series since each step is a relatively simple computation. However, every model is limited by the training data. If the training data is not representative of future values, then the model will be lacking. The variance in the forecast is computed under the assumptions that the patterns and trends extracted from training data are similar to future data.</p><p>Time Series Analysis is not a black box. The goal is to extract enough useful structure from your data to be predictive. After the extraction/decomposition of the structures, the remainder is often well-modeled by a common statistical model. Forecasts and confidences intervals are based on the statistical model and the extracted structures.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5ebd39b2f69" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Super-Resolution on Satellite Imagery using Deep Learning, Part 3]]></title>
            <link>https://medium.com/the-downlinq/super-resolution-on-satellite-imagery-using-deep-learning-part-3-2e2f61eee1d3?source=rss-387385dac9e------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e2f61eee1d3</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[remote-sensing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Patrick Hagerty]]></dc:creator>
            <pubDate>Wed, 08 Mar 2017 12:47:26 GMT</pubDate>
            <atom:updated>2017-03-08T12:47:26.402Z</atom:updated>
            <content:encoded><![CDATA[<p>In previous posts (<a href="https://medium.com/the-downlinq/super-resolution-on-satellite-imagery-using-deep-learning-part-1-ec5c5cd3cd2#.bhbulz2g9">Part 1</a> and <a href="https://medium.com/the-downlinq/super-resolution-on-satellite-imagery-using-deep-learning-part-2-c9ce41dc0ee0#.726i7nh7w">Part 2</a>), we trained a neural network to perform image enhancement using <a href="https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio">Peak Signal-to-Noise Ratio (PSNR)</a> as a cost function and hypothesized that the enhancement will also improve object detection in satellite imagery. In this post, we show that our hypothesis appears to be correct.</p><p>As a general rule, when the resolution of imagery decreases, the accuracy of detection algorithms decreases, see <a href="https://medium.com/the-downlinq/the-effect-of-resolution-on-deep-neural-network-image-classification-accuracy-d1338e2782c5#.p3hwuibpy">here</a> and <a href="https://medium.com/the-downlinq/quantifying-the-effects-of-resolution-on-image-classification-accuracy-7d657aca7701#.8ede7gwpj">here</a> for more details. Our strategy is to pre-process imagery using our super-resolution technique to effectively improve resolution and subsequently improve accuracy of the detection algorithms. We quantify the effect of our super-resolution process on the performance of several detection algorithms that have been developed for the <a href="http://crowdsourcing.topcoder.com/spacenet">SpaceNet competition</a>.</p><h3>CosmiQNet</h3><p>The initial release of SpaceNet has two types of imagery: 3-band (RGB) images at approximately 50 cm GSD and 8-band (MSI) at approximately 2 m GSD. We train two neural networks to enhance the 3-band imagery and the 8-band imagery separately. The neural networks are trained using PSNR as a cost function (actually the Mean Squared Error since the derivative is cheaper to compute) and consist of 5 perturbative layers with each perturbative layer having 3 convolutional layers. We scaled our imagery by a factor of 0.5 and by a factor of 0.25. (In other words, we increased the GSD by a factor of about 2 and 4 respectively.) We reduce the resolution of the image using bilinear interpolation to a smaller image and bilinear interpolation back to the original image size.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S1xJq9-KQLN2q1Dnj1mWhg.png" /><figcaption>CosmiQNet</figcaption></figure><p>Good starting weights significantly affect the training process. The choice of 3 convolutional layers per perturbative layer increased the risk of being trapped in a local optimum (much more so than 2 convolutional layers per perturbative layer). Using several initial seeds, we are able to train a neural network with fewer layers and similar enhancement capability as measured by PSNR.</p><p>The spatial distribution of PSNR gains on similar imagery is presented in Images 5 and 6 of <a href="https://medium.com/the-downlinq/super-resolution-on-satellite-imagery-using-deep-learning-part-2-c9ce41dc0ee0#.drbo4z421">Part 2</a>. Experimenting with better starting weights, hyper-parameters, and training strategies has resulted in increased PSNR gains.</p><p>It is difficult to translate the dB gain directly into the ability of a human to perceive different sizes and types of objects — the <a href="https://fas.org/irp/imint/niirs.htm">National Image Interpretability Rating Scale (NIIRS</a>) lists features that are visually perceptible at different resolutions. One example is that cars are identifiable at 50 cm GSD and become difficult to identify at 2 m GSD.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*H731CeZcozRFnp2BtPFZfg.tiff" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*4ektFLe4v-zYrdp9DdYw0Q.tiff" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*vopgvia-I1r0UVEENdZ_8Q.tiff" /><figcaption>Super Resolution on 3-band Imagery: Left is an original SpaceNet image roughly 50 cm GSD. Middle is a blurred version of the image by a factor of 0.5, roughly 1 m GSD. Right is the results of the blurred image enhanced by CosmiQNet.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*H731CeZcozRFnp2BtPFZfg.tiff" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*DWLhSVqLaYn7aIUB9fBVeQ.tiff" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*ijuVnGuu-6yvqUU91LQhCw.tiff" /><figcaption>Super Resolution on 3-band Imagery: Left is an original SpaceNet image roughly 50 cm GSD. Middle is a blurred version of the image by a factor of 0.25, roughly 2m GSD. Right is the results of the blurred image enhanced by CosmiQNet.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ddnh_IpeaEm8ETgbPEG78Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r6Yp6iUzTU2q6XxVkw7mVA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jqv15Y6OQO8at3xg16U36Q.png" /><figcaption>Super Resolution on 8-band Imagery: We assign 3 bands (coastal, blue, and green) to pseudo-colorize the image. Left is an original SpaceNet image roughly 2 m GSD. Middle is a blurred version of the image by a factor of 0.5, roughly 4 m GSD. Right is the results of the blurred image enhanced by CosmiQNet.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ddnh_IpeaEm8ETgbPEG78Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EUrdPtICXjPhCoJcw3qCNA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6vRB7v9k6iNJgeN5ytCN2w.png" /><figcaption>Super Resolution on 8-band Imagery: We assign 3 bands (coastal, blue, and green) to pseudo-colorize the image. Left is an original SpaceNet image roughly 2 m GSD. Middle is a blurred version of the image by a factor of 0.25, roughly 8 m GSD. Right is the results of the blurred image enhanced by CosmiQNet.</figcaption></figure><p>The images above illustrate that the super-resolution process recovers details of the original imagery from a blurred image. To the untrained eye it is difficult to distinguish the enhanced image from the original at the 0.5 x scale. At the 0.25 x scale, the super-resolution process lacks the detail of the original imagery — the details resemble the 0.5 x scale more than the original imagery.</p><p>Source code for training CosmiQNet is available at <a href="https://github.com/CosmiQ/super-resolution">github/CosmiQ/super-resolution.</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2c759442066f2f0ede81d8f38a2c97fa/href">https://medium.com/media/2c759442066f2f0ede81d8f38a2c97fa/href</a></iframe><h3>The Experiment</h3><p>The experiment is to test the hypothesis that PSNR is a relevant metric for image enhancement for the purposes of automated object detection. We perform this test using three entries from the SpaceNet competition (1st place <a href="https://github.com/SpaceNetChallenge/BuildingDetectors/tree/master/wleite">wleite</a>, 2nd place <a href="https://github.com/SpaceNetChallenge/BuildingDetectors/tree/master/marek.cygan">marek</a>, and 4th place <a href="https://github.com/SpaceNetChallenge/BuildingDetectors/tree/master/fugusuki">takahashi</a>) and two neural networks from CosmiQ Works (<a href="https://medium.com/the-downlinq/object-detection-on-spacenet-5e691961d257#.f3fq62jde">CosmiQNet</a> and <a href="https://medium.com/the-downlinq/building-extraction-with-yolt2-and-spacenet-data-a926f9ffac4f#.gsuz3l90t">YOLT2</a>).</p><p>Details of the algorithms are available at the linked locations. The wleite entry is the winning entry in the competition and uses manual crafted features and random forests in the algorithm. The marek and takahashi entries are convolution neural networks with some post-processing. YOLT2 is a customized version of YOLO2 neural network discussed in this<a href="https://medium.com/the-downlinq/building-extraction-with-yolt2-and-spacenet-data-a926f9ffac4f#.wtnkoywo4"> blog post</a>. The CosmiQNet entry, discussed in<a href="https://medium.com/the-downlinq/object-detection-on-spacenet-5e691961d257#.892rcjeeq"> this blog post</a>, adapted the super-resolution network to perform object detection and requires significant post-processing to be competitive with the other algorithms in the comparison.</p><p>Each of the algorithms is tested against 5 datasets: the original imagery, imagery blurred by a factor of 0.5, imagery blurred by a factor of 0.5 and then enhanced using super-resolution, imagery blurred by a factor of 0.25, and imagery blurred by a factor of 0.25 and then enhanced using super-resolution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p60gnVoQ114XlZD7-1g8aw.png" /><figcaption>F1 scores for Object Detection Algorithms on datasets. Blue is the F1 score of the algorithm on the original dataset. Green is the F1 score of the algorithm on the original dataset blurred by a factor of 0.5. Light blue is the F1 score of the algorithm on the original dataset blurred by a factor of 0.25. Red is the F1 score of the algorithm on the original dataset blurred by a factor of 0.5 and processed by the super-resolution neural network. Purple is the F1 score of the algorithms on the original dataset blurred by a factor of 0.25 and processed by the super-resolution neural network.</figcaption></figure><h3>Interpretation</h3><p>The improvement in the <a href="https://en.wikipedia.org/wiki/F1_score">F1</a> scores after enhancement supports the hypothesis that the PSNR was a relevant metric for training the super-resolution process. The enhancement recovered most of the accuracy loss incurred with a 0.5 x resolution blur, across five different object detection algorithms: 97%±4% of the original F1 score up from 68%±11% of the original F1 score. The recovery from the 0.25 x resolution blur was also significant:55%±16% of the original F1 score up from 24%±14% of the original F1 score.</p><p>To caveat these results, each of the algorithms is trained to perform building footprint detection on the original imagery at its original resolution. Some algorithms are more robust to changes in resolution. The F1 score is the harmonic mean of precision and recall. The improvement of F1 score by the super-resolution pre-processing is mainly a result of improved recall.</p><p>CosmiQNet has demonstrated a capability to recover details in blurred images and significantly improve object detection. The natural application is to increase the utility of lower cost sensors, especially satellite imagery sensors, using advanced analytic post-processing.</p><h3>Future</h3><p>In this post, we have limited the experiments to enhance imagery based on synthetic blurring. We plan to explore this impact of super-resolution on other commercial sources of imagery.</p><p>Current training uses DigitalGlobe’s 50 cm GSD imagery from Woldview-2. We also plan to investigate training on aerial imagery to improve satellite imagery.</p><p>Keep following <a href="https://medium.com/the-downlinq">The DownlinQ</a> for updates on these projects.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e2f61eee1d3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-downlinq/super-resolution-on-satellite-imagery-using-deep-learning-part-3-2e2f61eee1d3">Super-Resolution on Satellite Imagery using Deep Learning, Part 3</a> was originally published in <a href="https://medium.com/the-downlinq">The DownLinQ</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Super-Resolution on Satellite Imagery using Deep Learning, Part 2]]></title>
            <link>https://medium.com/the-downlinq/super-resolution-on-satellite-imagery-using-deep-learning-part-2-c9ce41dc0ee0?source=rss-387385dac9e------2</link>
            <guid isPermaLink="false">https://medium.com/p/c9ce41dc0ee0</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Patrick Hagerty]]></dc:creator>
            <pubDate>Tue, 06 Dec 2016 13:54:58 GMT</pubDate>
            <atom:updated>2016-12-06T13:54:58.226Z</atom:updated>
            <content:encoded><![CDATA[<p>In <a href="https://medium.com/the-downlinq/super-resolution-on-satellite-imagery-using-deep-learning-part-1-ec5c5cd3cd2#.n3ifl6620">Part 1</a>, we trained the neural network CosmiQNet to enhance 3-band satellite imagery. In this post, we continue this investigation with Multi-Spectral Imagery (MSI) available in the satellite imagery of <a href="https://aws.amazon.com/public-datasets/spacenet/">SpaceNet</a>.</p><p>Our thesis, in <a href="https://medium.com/the-downlinq/super-resolution-on-satellite-imagery-using-deep-learning-part-1-ec5c5cd3cd2#.n3ifl6620">Part 1</a>, is that geographically relevant imagery of higher quality can be used to train a machine learning algorithm to enhance satellite imagery. We redesign CosmiQNet to test new hypotheses:</p><ul><li>Using all the channels of MSI can improve visual enhancement,</li><li>Training can occur on imagery that is similar but does not necessarily overlap as in <a href="https://medium.com/the-downlinq/super-resolution-on-satellite-imagery-using-deep-learning-part-1-ec5c5cd3cd2#.n3ifl6620">Part 1</a>.</li></ul><p>The latter hypothesis was necessary since SpaceNet does not include multiple images over the same region at the time of this post.</p><p>The metric that we use to measure enhancement is the distribution of <a href="https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio">Peak Signal-to-Noise Ratio</a> (PSNR). Since satellite imagery is large and enhancement is not uniform, we present the measure of enhancement as an overlay on the image instead of assigning one number to the process.</p><p>The enhancement that is performed is essentially an intermediate step in imagery analysis. A more appropriate metric is the impact of enhancement on automated analysis — we plan to explore this in part 3.</p><p>By using MSI to improve super-resolution, we demonstrate a novel application of MSI. One common use of MSI data is to combine the bands to form an <a href="http://earthobservatory.nasa.gov/Features/MeasuringVegetation/">index to infer various environmental conditions</a> (e.g., vegetation density, drought). It would be interesting, but beyond the scope of this post, to explore how a DNN like CosmiQNet might be useful to enhance the performance of NDVI-type analyses.</p><p><strong>Multi-Spectral Imagery</strong></p><p>Multi-spectral imagery (MSI) consists of data collected by sensors of multiple bands of the electromagnetic spectrum. The filtering of photons results in lower (worse) resolution of MSI compare to panchromatic imagery. The lower resolution make MSI a natural candidate for super-resolution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*X6_FdodJY_B8OBQOtjZj0w.png" /><figcaption>Image 1: Spectral Bands from WorldView-2 and WorldView-3, see <a href="http://content.satimagingcorp.com.s3.amazonaws.com/static/satellite-sensor-specification/WorldView-2-PDF-Download.pdf">WorldView-2 Specification</a> and the <a href="http://content.satimagingcorp.com.s3.amazonaws.com/static/satellite-sensor-specification/WorldView-3-PDF-Download.pdf">WorldView-3 Specification</a> for details.</figcaption></figure><p>While the bands are relatively disjoint, objects seen from satellites typically have significant overlap within bands. The super-resolution process in our experiment enhances only the Coastal, Blue, and Green bands. We chose these three bands arbitrarily from the eight — Coastal, Blue, and Green are the first three channels of the GeoTIFFs in SpaceNet. In hindsight, this choice maximizes the spectral distance of the bands and possibly increases the difficulty of enhancement.</p><p>Two SpaceNet images over Rio that we analyze are displayed in Image 2 (training) and Image 3 (testing). A subset of the bands are displayed in these images. Unlike the experiment in part 1 where enhancement of boats is compared to the enhancement of the entire image, we do not isolate a region of structure to compare.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NG8ujh7jwyh3kWcrAl88Qg.png" /><figcaption>Image 2: The training image of Rio from SpaceNet. This image is a mosaic from multiple WorldView 2 images and shows only three of the eight bands. Water regions are distinguishable from city regions.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d2BW-4VLfmlgWghwXlij2g.png" /><figcaption>Image 3: The test image from SpaceNet. While this is an image over Rio, it does not overlap geographically with the training image.</figcaption></figure><h3>CosmiQNet</h3><p>CosmiQNet is a deep neural network that we have developed to perform several different tasks from <a href="https://medium.com/the-downlinq/object-detection-on-spacenet-5e691961d257#.iq8spkz46">building detection</a> to <a href="https://medium.com/the-downlinq/super-resolution-on-satellite-imagery-using-deep-learning-part-1-ec5c5cd3cd2#.y04adx9xf">3-band super-resolution</a>. The DNN is architected as a sequence of perturbative layers and trained a layer at a time. The application determines the number of convolutional sublayers appropriate for each perturbative layer. The perturbative layers are an extension of a <a href="http://image-net.org/challenges/talks/ilsvrc2015_deep_residual_learning_kaiminghe.pdf">ResNet layer</a> that gained fame for winning the <a href="http://image-net.org/challenges/LSVRC/2015/">2015 Large Scale Visual Recongnition Challenge</a>.</p><p>Our experiment is to demonstrate the impact that 8-bands of MSI has on the super-resolution capability of CosmiQNet. For this experiment, we present two versions of CosmiQNet:</p><ul><li>(3-band CosmiQNet) Each perturbative layer takes input and produces output in the Coastal, Blue, and Green spectral bands. The other 5 bands are ignored in this DNN.</li><li>(8-band CosmiQNet) Each perturbative layer takes input and produces output in all 8 bands. This network requires more weights than the 3-band version which results in slightly longer training times.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bx9EZAiKHpKc6ZgvAYkQog.png" /><figcaption>Image 4: CosmiQNet adapted to support MSI. In one version, only three bands (Coastal, Blue, and Green) are used to enhance those bands. In the other version, all eight bands are used to enhance the Coastal, Blue, and Green bands. The green circle represents the trainable bypass parameters of the blue perturbation layers.</figcaption></figure><p>CosmiQNet is constructed in <a href="https://www.tensorflow.org/">TensorFlow</a> and trained using an <a href="https://developer.nvidia.com/devbox">Nvidia DevBox</a> with 4 (Maxwell) Titan X GPUs. Each layer was trained for roughly 12 hours — optimizing this training was not fully investigated. For both versions of CosmiQNet, the optimization function is the same: the <a href="https://en.wikipedia.org/wiki/Mean_squared_error">mean squared error</a> (MSE) between the original image and the enhanced image in the Coastal, Blue, and Green channels. It is computationally easier for gradient descent based training algorithms to optimize MSE than PSNR, even though PSNR is essentially a normalized version of MSE.</p><h3>Results</h3><p>Similar to the plots in Part 1, we quantify performance by the distribution of the PSNR gain (compared to linear interpolation) overlaying the image. The red areas represent increased enhancement compared to linear interpolation, while the blue areas represent decreased enhancement compared to linear interpolation.</p><p>While it may be desirable to have the entire region red (meaning better than linear interpolation), the linear interpolation process results in a very high PSNR in regions with little structure like water, grassy areas, and forests, and it results in significantly lower PSNR in regions without structure. Hence, the impact of super-resolution is amplified when gain occurs in regions with structure. Similarly, the negative impact of poor super-resolution performance in unstructured regions is offset by the high PSNR from linear interpolation in these regions.</p><p>Often, the areas with structure are more likely to contain an object of interest to the analyst and thus have increased value. We do not partition the MSI into regions with more structure and regions with less structure. Comparing the plots in Images 5 and 6 with the SpaceNet imagery in Images 2 and 3 clearly shows enhancement within the city areas.</p><p>The key results are in presented in Image 5 (for training) and Image 6 (for testing):</p><ul><li>The 8-band version of CosmiQNet outperformed the 3-band version. The increase in red regions in the city areas is visually apparent. Further research is required to determine whether this PSNR performance enhancement translates to performance enhancement in object detection and/or image classification tools.</li><li>The 8-band version’s outperformance is validated by the test SpaceNet image. This demonstrates a degree of transferability of CosmiQNet beyond overlapping regions to geographically similar regions.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*aayBYvFmiKS_Op1aUevsaQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*fhzK0I1fFBZjXNhJn-Qmbw.png" /><figcaption>Image 5: Distribution of PSNR gain by CosmiQNet compared to linear interpolation on the training SpaceNet image. The left image corresponds to the 3-band version of CosmiQNet and the right image corresponds to the 8-band version of CosmiQNet. Red corresponds to improved PSNR while blue corresponds to worsened PSNR. The PSNR and its gain are measured using the Coastal, Blue, and Green bands of the SpaceNet image. The large blue region corresponds to water area where the linear interpolation performs roughly 20 dB better than it does in the non-water regions.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*4fwvKTT2MYE8mHkPX1_AfA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*vmNkdjDd7l79WoDpavCcdA.png" /><figcaption>Image 6: DistrIbution of PSNR gain by CosmiQNet compared to linear interpolation on the test SpaceNet image. The left image corresponds to the 3-band version of CosmiQNet and the right image corresponds to the 8-band version of CosmiQNet. Red corresponds to improved PSNR while blue corresponds to worsened PSNR. The PSNR and its gain are measured using the Coastal, Blue, and Green bands of the SpaceNet image. The test image does not overlap with the training image but rather is a different part of the city of Rio.</figcaption></figure><h3>What’s next?</h3><p>We have demonstrated the potential value of MSI in image enhancement using deep learning. Future research is required to determine how much better the ehancement process can become and to quantify the enhancement’s impact on automated analysis. We organize future research directions into the following categories:</p><ul><li><strong>Algorithm development. </strong>We plan to integrate algorithmic advances into future versions of CosmiQNet. <a href="https://arxiv.org/pdf/1406.2661v1.pdf">Adversarial networks</a> appear promising for image enhancement and for improving the quantifying metric.</li><li><strong>Enhancement thresholding. </strong>At some resolution scale, PSNR decreases even in the structured areas. Quantifying these limits and their relevance to applications can provide perspective on the utility of image enhancement.</li><li><strong>Impact on object detection.</strong> In previous posts, we demonstrated the relationship of resolution on the performance of object detection algorithms. A natural extension of this work is to quantify the effect of image enhancement 0n similar object detection problems.</li><li><strong>Synthesis applications.</strong> Labeled data is a scarce resource. We would like to investigate how CosmiQNet can be used to synthetically increase the size and diversity of a labeled data set.</li><li><strong>Pan-sharpening.</strong> CosmiQNet can be retrained and adapted to improve pan-sharpening of satellite imagery using MSI.</li></ul><p>We plan to present more details of our research in future posts in this series.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9ce41dc0ee0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-downlinq/super-resolution-on-satellite-imagery-using-deep-learning-part-2-c9ce41dc0ee0">Super-Resolution on Satellite Imagery using Deep Learning, Part 2</a> was originally published in <a href="https://medium.com/the-downlinq">The DownLinQ</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Super-Resolution on Satellite Imagery using Deep Learning, Part 1]]></title>
            <link>https://medium.com/the-downlinq/super-resolution-on-satellite-imagery-using-deep-learning-part-1-ec5c5cd3cd2?source=rss-387385dac9e------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec5c5cd3cd2</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Patrick Hagerty]]></dc:creator>
            <pubDate>Fri, 18 Nov 2016 16:12:28 GMT</pubDate>
            <atom:updated>2016-11-18T16:12:28.026Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://www.facebook.com/yann.lecun/posts/10153426023477143">Yann Lecun</a> has compared supervised learning to the icing on a cake, and unsupervised learning as the cake, asserting that we know how to make the icing but do not know how to make the cake. In this post, we provide a “cake” recipe for training an unsupervised learning algorithm to enhance satellite imagery.</p><p>This research is motivated by the increase in access to lower cost satellite imagery in the emerging commercial space industry. In this emerging industry, there is a trade-off amongst sensor quality, revisit rate, and price. We investigate advanced image processing’s potential to reduce the trade-off and to improve the imagery from a lower quality sensor at the same price point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lKYt4h1s10NwOItSR-CO9w.png" /><figcaption>Figure 1: Remote Sensing via Airplane, Commercial Exquisite, and Space 3.0. This figure is not to scale but it meant to convey the potential overlap of remote sensing activities. Aerial remote sensing can be used to enhance commercially exquisite satellite imagery. Commercially exquisite satellite imagery can be used to enhance lower-resolution satellite imagery.</figcaption></figure><p>We embed imagery details from a higher resolution image in a deep neural network (DNN) and extract the details to enhance geographically similar imagery. As part of this research, we develop a novel architecture for DNNs using perturbative layers that is well-suited to the enhancement task.</p><h3>Super-Resolution</h3><p>There are many forms of image enhancement, including noise-reduction and color adjustments. For satellite imagery, one common measure of image quality is Ground Sampling Distance (GSD), the physical measure represented by one pixel in the image. Enhancement in this post refers to decreasing (better) the GSD in satellite imagery, also called super-resolution. The super-resolution process synthesizes sub-pixel information in imagery to increase the resolution of the image. Typical synthesis techniques include:</p><ul><li>interpolation of nearby pixels within the image,</li><li>interpolation of nearby frames within a video,</li><li>frequency filtering, to reduce noise.</li></ul><p>In this investigation, we extend these techniques to include:</p><ul><li>deep learning structures from geographically relevant imagery.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IIzJxmPzaClecwvxp4pShQ.png" /><figcaption>Figure 2: Super-Resolution. To transform super-resolution from an ill-posed optimization problem into a well-posed inverse problem, we must start with higher resolution imagery, degrade that imagery, and optimize the super-resolution algorithm to reconstuct the original imagery from the degraded imagery. Peak Signal-to-Noise ratio measures the difference between the original image and the reconstructed one.</figcaption></figure><p>To quantify the effectiveness of our enhancement techniques, we compare the <a href="https://en.wikipedia.org/wiki/Peak_signal-to-noise_ratio">Peak Signal-to-Noise Ratio</a> (PSNR) before the enhancement to the PSNR after the enhancement. Moreover, we show the geographic distribution of PSNR over the image and its relevance to further analysis.</p><p>PSNR is a natural choice to measure the generative ability of the super-resolution algorithm. We plan to present a future post on using a <a href="https://arxiv.org/pdf/1406.2661v1.pdf">Generative-Adversarial Network</a> to learn a better cost function for performing super-resolution.</p><h3>Fully Convolutional Neural Networks with Perturbation Layers</h3><p>Before going directly to the results, we take a detour to discuss the architecture developed to perform the super-resolution process. Standard DNNs such as <a href="https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf">AlexNet</a>, <a href="https://arxiv.org/pdf/1512.03385v1.pdf">ResNet</a>, <a href="https://arxiv.org/pdf/1409.1556.pdf">VGG</a>, and <a href="https://www.cs.unc.edu/~wliu/papers/GoogLeNet.pdf">GoogLeNet</a> are great architectures for image classification and object detection on low resolution imagery but are ill-suited for the exponentially large output space in super-resolution.</p><p>Inspired by <a href="http://image-net.org/challenges/talks/ilsvrc2015_deep_residual_learning_kaiminghe.pdf">ResNet</a>, we decided to design a new DNN as a sequence of perturbations of the identity map, since super-resolution is essentially a perturbation of the lower resolution image. The network is extended a layer at a time by optimizing a convex combination of the previous layer and the current layer, producing a trainable weight (bypass parameter) for the new layer that measures its contribution to the final output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O-Wq-Cr2mvN8EFTWrZfEgQ.png" /><figcaption>Figure 3: Our convex perturbation layer compared to a ResNet Layer. In both architectures there is a combination of a convolutional layer with an identity function. The convex perturbation allows for one to train to the optimal combination. As the beta values decrease, the layer’s contribution to the enhancement decreases.</figcaption></figure><p>There are several benefits of this structure:</p><ul><li>The network architecture fits well into modern training strategies for training extremely deep neural networks including <a href="https://arxiv.org/pdf/1603.05027.pdf">skip connections</a> and <a href="https://arxiv.org/pdf/1603.09382.pdf">stochastic depth</a>.</li><li>The bypass parameters measure the contribution for each layer, giving feedback on how deep the network should be.</li><li>Each layer performs a near-identity transformation that enhances the image using different structures.</li></ul><p>Within each perturbation layer, we include at least two convolutional layers and a non-linear ReLU layer between each convolutional layer. More convolutional layers within a perturbative layer increases the ability of the perturbative layer to enhance the image but becomes more difficult for training to converge. Alternatively, additional perturbative layers have similar enhancement potential without the convergence issues.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*i2FVIskyRyBQpw1xDucqfw.png" /><figcaption>Figure 4: A Deep Neural Network with perturbative layers.</figcaption></figure><p>The bypass parameters give direct feedback on the impact of each perturbative layer. This feedback helps answer the question of how deep the network has to be.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a47tziq2DUR-spm7RFs4gw.png" /><figcaption>Figure 5: Bypass Parameters during training. The weights of the bypass parameters are plotted during the training process. For this particular training algorithm, training occurred in two stages per layer: firstly the parameters of the layer were trained and secondly all previous trained parameters were jointly optimized with the new layer. The bypass parameter decreases as the network grows. Eventually the impact of the new layer will not affect the integral value of the pixels in an enchanced image (without being aggregated with other layers) — this defines a sub-pixel threshold.</figcaption></figure><h3>The Experiment</h3><p>The initial experiment that we perform measures the ability for the DNN to enhance degraded, 3-band GeoTIFFs over the Panama Canal. We use two GeoTIFFs (very large satellite images) courtesy of DigitalGlobe in the experiment: one for training and one for testing. We do not enhance the entire image through one pass of the DNN, but rather we enhance a 27-pixel by 27-pixel region at a time. Since the GeoTIFFs are very large images, sampling 27-pixel by 27-pixel regions provides sufficient training data for our DNN. Access to more training imagery should improve the results. Using the two GeoTIFFs, we procede with training the DNN:</p><ul><li>The two GeoTIFFs are rescaled to effectively reduce the resolution of the images.</li><li>Regions from the first GeoTIFF are randomly sampled to train the DNN a layer at a time. We train the weights of the DNN to maximize the PSNR of the output of the DNN.</li><li>The DNN is used to enhance both degraded GeoTIFFs.</li><li>Results are compared to interpolation based enhancement algorithms.</li></ul><p>We use the TensorFlow framework to construct, train, and infer the DNN on a 2015 <a href="https://developer.nvidia.com/devbox">Nvidia Devbox</a> with 4 Titan X GPUs, but we used only one GPU to train the DNN. To train the neural network we used the <a href="https://arxiv.org/pdf/1412.6980.pdf">ADAM optimization</a> algorithm; ADAM has associated parameters that impact training time and convergence rates. We did not fully explore the optimal choice of the ADAM parameters, but spent about 12 hours (on one Titan X GPU) training per perturbative layer. The rate at which the bypass parameters converged ( as shown in Figure 5) assisted in our choice of the ADAM parameters and subsequently the training time.</p><h3>Results</h3><p>In this experiment, we have two GeoTIFF images around the Panama Canal, one for training and one for testing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rmriaIJOWfGH6dkSMTmmFw.png" /><figcaption>Figure 6: Satellite Image of the Panama Canal. This is the original training image for the DNN.</figcaption></figure><p>The first step is to create training data by degrading a GeoTIFF. By resizing the GeoTIFF, the resulting degraded image has an effective reduction of GSD, or resolution. Using linear interpolation as a starting point, we can plot the distribution of PSNR throughout the degraded image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9S3n7-7Df6DqWC7AEUEwfA.png" /><figcaption>Figure 7: The Distribution of PSNR in the Input to the DNN. The input to the DNN is a degraded satellite image that is resized (by a factor of 2 using linear interpolation) to match the dimensions of the original GeoTIFF. This plot shows the location of noise introduced by the degrading process. Blue regions have more noise introduced by the degrading process, while red regions have less. The blue regions tend to be areas with fine structure (like boats), while red regions tend to have more coarse features (like open water).</figcaption></figure><p>Figure 7 demonstrates that one number to represent PSNR is insufficient to describe the noise within a satellite image. Regions with more structure, such as boats, have lower PSNR in the degraded image than regions with less structure, such as water. When we train the super-resolution algorithm to enhance the degraded image, we want to enhance regions that we care about, which are often regions with structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4ZyLB0gYtXjvhgDP-2ezgA.png" /><figcaption>Figure 8: The PSNR Gain after Enhancement using the DNN. The distribution of the PSNR gain is plotted for the test image — the DNN was not trained on this image. Most regions benefited from enhancement. The blue regions are general regions where there was significantly less noise in the original image. The enhancement is the PSNR compared against the initial linear interpolation.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nOelqVbnKFhekgAT9oU3XQ.png" /><figcaption>Figure 8: PSNR Gain versus Bicubic Interpolation. On the test GeoTIFF, we plot the difference in PSNR compared to bicubic interpolation. Regions with initially more noise also benefit.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*O2h2JCEpuxWYsIo7_gvIRg.png" /><figcaption>Figure 9: PSNR Change in dB from Linear and Bicubic Interpolation to DNN-based Enhancement. The PSNR change is computed over the entire GeoTIFFs and over sub-regions of the GeoTIFFs that contains boats. The enhancement in the regions with structure is significantly higher than in the water regions.</figcaption></figure><p>The results in Figure 9 are evidence that the DNN-based enhancement has improved performance in the regions with more structure. Even though the test image and the training image had the same GSD, different atmospheric conditions and cloud coverage impact the enhancement, partially explaining the improved performance on the testing image over the performance on the training image. Image clarity also affects the ability to label regions containing boats; less accurate labeling includes more water regions and would likely lower the dB gain in that region. Experiments that isolate these phenomena are beyond the scope of this post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K3hBpwgPfryreQ_3bWPmHg.png" /><figcaption>Figure 10: Example of Enhancement of Boat plus Water region. This figure shows the enhancement of a degraded boat. Since the region contains a large portion of water the PSNR is lower than a region containing just a boat.</figcaption></figure><h3>Alternative Research Directions</h3><p>There are examples, such as <a href="https://arxiv.org/pdf/1501.00092.pdf">SRCNN</a>, of super-resolution performed on non-satellite images that demonstrate similar dB gains when trained on ImageNet. These approaches may have viability in the enhancement of satellite imagery but lack a fundamental advantage of our approach: location information of the image. Our approach is different for several other reasons:</p><ul><li>Satellite imagery tends to be a corner case for many DNN-based machine learning algorithms.</li><li>Over-training may not be as detrimental to our algorithm as it is for a more diverse image set.</li><li>Perturbative layers provide insight into the required depth of the DNN and the marginal performance improvement expected by increasing the depth.</li><li>GeoTIFFs have the potential to contain more than just Red, Green, and Blue channels. Our approach is easily modified to take advantage of additional channels (such as 8-band imagery).</li></ul><p>Finally, we have experimented with increasing the number of convolutional layers within each perturbation layer with improved performance. We will present this in part two with a specific focus on 8-band images and <a href="https://aws.amazon.com/public-datasets/spacenet/">SpaceNet</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec5c5cd3cd2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-downlinq/super-resolution-on-satellite-imagery-using-deep-learning-part-1-ec5c5cd3cd2">Super-Resolution on Satellite Imagery using Deep Learning, Part 1</a> was originally published in <a href="https://medium.com/the-downlinq">The DownLinQ</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The SpaceNet Metric]]></title>
            <link>https://medium.com/the-downlinq/the-spacenet-metric-612183cc2ddb?source=rss-387385dac9e------2</link>
            <guid isPermaLink="false">https://medium.com/p/612183cc2ddb</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Patrick Hagerty]]></dc:creator>
            <pubDate>Thu, 10 Nov 2016 18:16:00 GMT</pubDate>
            <atom:updated>2016-11-14T17:17:54.742Z</atom:updated>
            <content:encoded><![CDATA[<p>In <a href="http://crowdsourcing.topcoder.com/spacenet">SpaceNet Challenge</a>, the metric for ranking entries is based on the Jaccard Index, also called the Intersection-over-Union (IoU). This post describes in detail the metric as well as some of its benefits and limitations. For a great introduction to IoU, Adrian Rosebrock has a excellent introductory <a href="http://www.pyimagesearch.com/2016/11/07/intersection-over-union-iou-for-object-detection/">post on IoU.</a></p><h3>Object Detection and Image Segmentation</h3><p>Object detection is an application of computer vision algorithms to locate and identify objects of interest within digital imagery/video. Typically the region that is required to locate a target object is a bounding box: the vertices (in pixel coordinates) of a rectangular region containing the target object. Object detection also requires accurate classification of the bounding box into the associated category.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/489/1*_IiRznaSfH8dDc8NsOxAEQ.png" /><figcaption>Image 1: Example of object detection (from<a href="https://arxiv.org/pdf/1405.0312v3.pdf"> Microsoft COCO: Common Objects in Context, by T. Lin, et al.</a>)</figcaption></figure><p>Image segmentation is an application of computer vision algorithms that classifies each pixel/voxel in a digital image/video into a category. Image segmentation results in more accurate distinction of different categories within an image, but less distinction of different objects of the same category.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*0DsGPEDQxxN4rsAgeJqjYg.png" /><figcaption>Image 2: Example of image segmentation (from<a href="https://arxiv.org/pdf/1405.0312v3.pdf"> Microsoft COCO: Common Objects in Context, by T. Lin, et al.</a>)</figcaption></figure><p>The labels in the SpaceNet challenge are not bounding boxes, but rather polygons in a vector layer, or <a href="http://geojson.org/">geojson</a>. The flexibility of the polygonal label allows for more accurate localization than a bounding box but retains the ability to distinguish different objects of the same category.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*k_kIMQ0Rd-Ook6Z2pTjmmg.png" /><figcaption>Image 3: Object labels in SpaceNet. The blue regions are non-bounding box labels for building footprints.</figcaption></figure><h3>Distance between two Regions</h3><p>To train and evaluate computer vision algorithms, one needs a measure that can relate the distance between two regions. There are several candidates for this measure that have viability for certain applications:</p><ul><li><a href="https://en.wikipedia.org/wiki/Euclidean_distance"><strong>Euclidean distance between centroids.</strong></a><strong> </strong>For large, isolated regions Euclidean distance may be appropriate, but lacks the capability to value any detail of the regions.</li><li><a href="https://en.wikipedia.org/wiki/Hausdorff_distance"><strong>Hausdorff distance.</strong></a> The Hausdorff distance may overvalue the boundary of a region compared to the interior. Additionally, the Hausdorff distance is computationally expensive to compute.</li><li><strong>I</strong><a href="https://en.wikipedia.org/wiki/Jaccard_index"><strong>ntersection-over-Union.</strong></a><strong> </strong>The IoU presents a normalized (scale-invariant) measure that focuses on the areas of the regions. IoU shines at distinguishing regions that overlap but lacks detail on non-overlapping regions.</li></ul><p>The <a href="http://image-net.org/challenges/LSVRC/">LSVRC competitions</a> associated to ImageNet use IoU as a metric largely because the scale-invariance works well with diverse object sizes. The SpaceNet competition leverages the familiarity of IoU to attract participation from the machine learning community. As SpaceNet evolves, IoU can scale with expected diversity of object sizes.</p><h3>Some IoU Details</h3><p>The IoU is a measure of how close two regions are two each other on a scale between 0 and 1 — a value of 0 means the regions do not overlap and a scale of 1 means that the regions are exactly the same. Explicitly,</p><p>IoU(A,B) = area(A intersection B) / area( A union B)</p><p>IoU can be converted into a true, mathematical metric but it is often preferable to use IoU directly as opposed to the related metric. Since IoU is scale invariant, IoU can be computed in world coordinates or in pixel coordinates with the caveat that the conversion may need to accommodate fractional values for pixels.</p><p>Scale invariance is generally desirable but may result in low IoU scores when automating detection of small objects for various reasons including, inaccurate labeled training data, pixelization in imagery, and sensitivity to occlusions.</p><h3>Implications of GIS Imagery</h3><p>Satellite imagery comes in a variety of resolutions depending on the satellite and sensor. Using the geospatial coordinate system to label the objects of interest allows for a resolution independent description of the location of an object of interest. We often call the GIS (short for Geographic Infomation System) coordinate system “world coordinates” when comparing to image-specific “pixel coordinates”. Conversion between the two coordinate systems is not difficult but could be a barrier-of-entry to working with GIS imagery.</p><p>One significant advantage that GIS imagery has over other imagery is the known scale. When working in world coordinates, object detection algorithms can be optimized to use the known scale and reduce the search space.</p><h3>Thesholds and Detections</h3><p>With an object detection algorithm, the performance of an algorithm should depend on how many objects the algorithm detects (true positives), how many objects it fails to detects (true negatives), and how many non-objects it detects (false positives). Just using IoU is insufficient to define a detection. SpaceNet defines a threshold of the IoU score of 0.5, above which is considered a detection and below which is not a detection.</p><p>Even though IoU is scale-invariant, resolution limits in an image make the 0.5 threshold challenging for small objects. The first instance of SpaceNet works with building footprint labels, for which the chosen threshold gives ample room to differentiate algorithms.</p><p>One additional feature that SpaceNet adopts from LSVRC is the notion that each labeled region can have at most one true positive associated with that labeled region. This feature is implemented by a sequential search for a true positive sorted by decreasing IoU values. If a true positive is found, then the pair — the label and the proposed region — are removed from the sequence and the search continues. The following flowchart removes ambiguity based on the order the proposals are submitted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0q4LlVULnEtOlRyD_qZOZw.png" /><figcaption>Chart 1: Flow chart for the detection in SpaceNet. A list of proposals is generated by the detection algorithm and compared to the ground truth in the list of labels.</figcaption></figure><h3>Precision, Recall, and F1</h3><p>After having defined detection based on Chart 1, we summarize the performance using precision, recall and F1 score.</p><p>Precision is the fraction of proposals that are true positives. Precision does not measure the number of objects that the algorithm has failed to detect. If the number of objects in an image is known, then precision is more valuable.</p><p>Recall is the fraction of labeled objects that are detected (true positives) by the algorithm. A low precision algorithm can still have high recall, usually implying many false positives. And an algorithm with low recall can still have high precision, usually implying few but accurate guesses.</p><p>SpaceNet chooses the harmonic average of precision and recall, essentially giving equal weight to each; this average is called the F1 score and is defined explicitly as follows:</p><p>F1= 2 * precision * recall / (precision + recall).</p><h3>Aggregation</h3><p>The count of true positives and false positives is aggregated over all of the (test) images in SpaceNet. This is in contrast to other algorithms where precision, recall, or F1 is evaluated per image. There are many alternatives for averaging to obtain an F1 score, but this choice resolves some confusing conditions where images have no objects to detect.</p><h3>Examples</h3><p>Examples of the IoU computation on SpaceNet imagery are available on another <a href="https://medium.com/the-downlinq/object-detection-on-spacenet-5e691961d257#.5jgt1tqyx">DownlinQ blog</a>. IoU values are plotting with region proposals on SpaceNet imagery using QGIS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*kDgZbtnc7LVCquE4Kz-5lw.png" /><figcaption>Image 4: Example of IoU computed for SpaceNet labels using bounding boxes. This is Image 11 from <a href="https://medium.com/the-downlinq/object-detection-on-spacenet-5e691961d257#.72vzofsli">Object Detection on SpaceNet</a>. Blue polygons are ground truth building footprints. Green rectangles are bounding boxes derived from ground truth footprints and have an IoU with the corresponding footprint greater than the 0.5 threshold (true positives). Yellow rectangles are bounding boxes derived from the ground truth footprints but have an IoU less than 0.5 (false positives). The SpaceNet metric for the entire public SpaceNet imagery results in an F1 score of 0.729.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Mzd9Gq_kU1ai1ToFjgvPBA.png" /><figcaption>Image 5: CosmiQNet-v1 Proposal with Labeled IoU. This is Image 10 from <a href="https://medium.com/the-downlinq/object-detection-on-spacenet-5e691961d257#.72vzofsli">Object Detection on SpaceNet</a>. The red proposal in the Northeast region that is being counted as a false positive is a result of a ground truth labeling mistake. Yellow regions are false positives that are close to the threshold and green regions are true positives.</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=612183cc2ddb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-downlinq/the-spacenet-metric-612183cc2ddb">The SpaceNet Metric</a> was originally published in <a href="https://medium.com/the-downlinq">The DownLinQ</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Object Detection on SpaceNet]]></title>
            <link>https://medium.com/the-downlinq/object-detection-on-spacenet-5e691961d257?source=rss-387385dac9e------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e691961d257</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Patrick Hagerty]]></dc:creator>
            <pubDate>Tue, 11 Oct 2016 16:54:17 GMT</pubDate>
            <atom:updated>2016-10-11T16:54:17.940Z</atom:updated>
            <content:encoded><![CDATA[<p>This post presents a walk through of an object detection process applied to <a href="https://aws.amazon.com/public-data-sets/spacenet/">SpaceNet</a> imagery. The strategy employed is far from optimal but rather is meant to illustrate a complete workflow based on a previous <a href="https://medium.com/the-downlinq/establishing-a-machine-learning-workflow-530628cfe67#.3rpbn716k">post</a>. Sample Python code is included to demonstrate basic GIS functionality within Python and neural network design in TensorFlow.</p><p>Similar to many other computer vision problems, we desire to find objects (in particular, building footprints) in images (satellite imagery from <a href="https://aws.amazon.com/public-data-sets/spacenet/">SpaceNet</a>). There are several properties of this challenge that distinguish it from other computer vision problems:</p><ul><li>The classifier uses 11 bands within the satellite imagery.</li><li>We pre-process the labeled imagery using a distance transform to create a richer labeled data set.</li><li>We train a weighted fully convolutional neural network to label the output; the weights give feedback on the importance of bands and layers.</li><li>We post-process the output to generate non-rectangular region proposals and output the proposed regions into geoJSON files.</li><li>We implement the metric derived from Intersection-over-Union to compute an F1-score for the machine learning algorithm.</li></ul><h3>License</h3><p>The code snippets in this blog are shared under the following license:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8fd1656ebf184dda6f1963e4fc8a0d20/href">https://medium.com/media/8fd1656ebf184dda6f1963e4fc8a0d20/href</a></iframe><h3>SpaceNet</h3><p>More information on SpaceNet is available <a href="https://aws.amazon.com/public-data-sets/spacenet/">here</a>. For each sub-region, there are two images (GeoTIFFs) and one label (geoJSON):</p><ul><li>The 3-band raster image, at roughly 0.5 m ground sampling distance, contains Red, Green, and Blue color channels with 8-bit values.</li><li>The 8-band raster image, at roughly 2 m ground sampling distance, contains both visible spectrum channels and near infrared channels with 16-bit values. The three-band image is derived from a panchromatic image and a subset of the three channels — the details of the derivation are available from the <a href="http://www.satimagingcorp.com/satellite-sensors/worldview-2/">WorldView 2 specifications</a>.</li><li>The label contains a list of polygons contained within the sub-region stored in a geoJSON format. Each polygon (or multi-polygon for non-simply connected regions) represents a building footprint.</li></ul><h3>Pre-processing.</h3><p>Since geoJSONs can be more descriptive than just bounding boxes, we employ a strategy that is more flexible with boundary descriptions than a rectangular box, see [<a href="https://arxiv.org/pdf/1602.06564v1.pdf">Yuan2016</a>]. This strategy transforms a label of building footprints into a raster image where each pixel has the value of the distance that pixel is to the boundary of a building — exterior pixels have a negative distance, interior pixels have a positive distance, the boundary of the footprint has a distance of zero. Provided the building footprints do not overlap, this transform is invertible. The following unoptimized Python code performs the relabeling (for an implementation that uses the more efficient GDAL library see <a href="https://github.com/SpaceNetChallenge">SpaceNetChallenge</a>):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f45f897ea842a95bd2a038787d6d17f0/href">https://medium.com/media/f45f897ea842a95bd2a038787d6d17f0/href</a></iframe><p><a href="http://qgis.org/">QGIS</a> is an open-source tool for managing and editing GeoTIFFs and geoJSON files. In Image 1, we present a 3-band image from the SpaceNet collection overlaid with the geoJSON for building footprints as red polygons.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*vwJjbtw8kKxAk_JUk_VGTA.png" /><figcaption>Image 1: geoJSON Labels for buildings overlayed on 3-band GeoTIFF. The image was created using QGIS.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*bV-Wnwbhe-fi0I0mfU6ZiQ.png" /><figcaption>Image 2: Distance to the Boundary (Saturated). The image was created in Python using matplotlib. Since we are interested in regions near the boundary, we saturate the interior pixels to Red and exterior pixels to Blue — the boundary is yellow.</figcaption></figure><p>Applying the distance transform to the geoJSON features, we obtain a continuously valued label. The transform is useful in separating buildings that are close together — this separation improves object detection based on Intersection-over-Union described below.</p><h3>Architecting the Classifier.</h3><p>We build a fully convolutional neural network (FCNN), that we call CosmiQNet, to classify a pair of input images, one 3-band and one 8-band. The output of the classifier is a distance transform.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NtD7BfyCNhhf0iaAiHKATA.png" /><figcaption>Figure 1: Blackbox Fully Convolution Neural Network: CosmiQNet. The inputs are at two resolutions and the output distance transform matches the lower of the input resolutions. The resolution of the 8-band GeoTIFF is roughly one quarter (in each dimension) the resolution of the the 3-band GeoTIFF; the difference in resolution is depicted by the scale of the GeoTIFFs.</figcaption></figure><p>We include bypass parameter weights in each layer of CosmiQNet to facilitate training and to provide feedback on the necessary required depth. This architecture was developed for another application but seems to perform reasonably in this application. If you want to hear more about these FCNNs, I plan to present at <a href="http://dc.gputechconf.com/">GTC DC 2016</a> for a different application.</p><p>The philosophy of CosmiQNet is that each additional layer refines the output of the network. The refinement is integrated into the training algorithms: where each additional layer is locally optimized, then all the previously layers are optimized together with the new layer. The network is extended by optimizing a convex combination of the previous layer and the current layer, producing a weight for the new layer that measures its contribution to the final output. As the network grows (training deeper layers), the weights decrease — each deeper layer is a slight perturbation of the identity map.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i0_OZoZdmDS6AXnfappczQ.png" /><figcaption>Figure 2: CosmiQNet v1.</figcaption></figure><p>Below is a code snippet written in Python using TensorFlow to create the FCNN we call CosmiQNet. The code includes the architecture, the cost function, and the optimizers used in training.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8a82c6183ebf4a0ee859a1d1e65470db/href">https://medium.com/media/8a82c6183ebf4a0ee859a1d1e65470db/href</a></iframe><p>This architecture loses very little information during each layer. However, repeated perturbations of the distance transform increase the dependence on the first layer of CosmiQNet to integrate all 11 imagery bands. Alternatively, we can feed the original imagery into each additional layer along with the output of the previous layer, which we call CosmiQNet v2. For these architectures, the deep layers empirically (and perhaps theoretically, as well) converge to be perturbations of the identity map. CosmiQNet v2 performs slightly better than the original CosmiQNet at the cost of additional training time and increasing the parameter space roughly 12-fold.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1eaf099aa06bba492966b20861f3afbe/href">https://medium.com/media/1eaf099aa06bba492966b20861f3afbe/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mtOWrLQVR16xVmb1vIA4fQ.png" /><figcaption>Figure 3: CosmiQNet v2</figcaption></figure><p>The cost function is the square of the Euclidean distance between the output and the distance transform derived from the label. Since we care mostly about the boundary, there are likely better cost functions that focus more on the boundary.</p><h3>Testing the Classifier.</h3><p>CosmiQNet is constructed to be trained a layer at a time. Each successive layer refines the previous layer. To train a layer, first the parameters specific to that layer are optimized, and second all of the previous parameters are optimized as well.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/69824db707e4787c58b362050f7dce18/href">https://medium.com/media/69824db707e4787c58b362050f7dce18/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4gdAxMQEAV8-PRfckuRoKw.png" /><figcaption>Figure 5: Layer Weights for CosmiQNet v1. The weights show the contribution of each successive layer of CosmiQNet v1.</figcaption></figure><h3>Post-processing</h3><p>The output of the CosmiQNet is a heatmap of predicted distances to the boundary of buildings. The post-processing converts this heatmap into a geoJSON of proposed building footprints. We employ a greedy algorithm for cluster growing (<a href="https://gist.github.com/hagerty/724b84ad69897d1fe6d241bbca9e2781">Python code</a>):</p><ol><li>Define B to be the set of points in the heatmap that are predicted to be inside the building (positive distance from the boundary). Let b1 be the point in the heatmap with the largest value.</li><li>Define the cluster, c1, as the largest connected component of B containing b1 such that there is a path of decreasing values between b1 and every point in c1. The cluster can be computationally identified with a nearest neighbor search.</li><li>Remove c1 from B and repeat to find b2 and c2.</li></ol><p>The greedy algorithm for clustering can certainly be improved. Building footprints of long, narrow buildings or non-convex buildings create erroneous output from the greedy algorithm.</p><p>We then convert the array of clusters into a geoJSON using Python GDAL commands.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/68291c2e8063a6ead790c3c5c7337858/href">https://medium.com/media/68291c2e8063a6ead790c3c5c7337858/href</a></iframe><p>In the creation of the vector layer, GDAL produces some extraneous polygons. These polygons can be removed with GDAL commands that select features — the feature selection can be used for further post-processing.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cea58a49216b502fbb52c82440f06b19/href">https://medium.com/media/cea58a49216b502fbb52c82440f06b19/href</a></iframe><h4>Intersection over Union</h4><p>Finally, we want to compare the Jaccard Index to the labeled data to assess the classifier. Recall that the Jacquard Index (Intersection over Union, IoU) between two sets, <em>A</em> and <em>B</em>, is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/326/1*6u2I3HIvDkAjIMga0l6F3g.png" /></figure><p>The scoring algorithm that we implement is based on Algorithm 2 from the <a href="http://arxiv.org/pdf/1409.0575v3.pdf">ILSVRC documentation</a>. To compute the metric based on the Jaccard Index, we compute true positives, false positives, and false negatives based on the IoU between a proposal and a label being above a threshold of 0.5. Code to implement the evaluation of the metric is available of the <a href="https://github.com/SpaceNetChallenge/utilities/blob/master/python/evaluateScene.py">SpaceNetChallenge github repository.</a></p><h3>Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*2QPpb2xKeg_BJK3pVkZOSw.png" /><figcaption>Image 3: Example image from SpaceNet</figcaption></figure><p>Image 3 is an example image from SpaceNet that has not been included in the training imagery for CosmiQNet. Image 4 show the same SpaceNet image with the labeled building footprints. These images demonstrate that real-world labeling is imperfect: footprints are slightly mis-aligned or even missing. Trees and shadows partially obstruct several footprints. These complexities increase the difficulty of hand labeling and necessitate robust machine learning algorithms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*k_kIMQ0Rd-Ook6Z2pTjmmg.png" /><figcaption>Image 4: SpaceNet Image with Ground Truth Overlays (blue).</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/1*5eAs62TMIiOFl9iSr2MUsQ.png" /><figcaption>Image 5: Heat map example of ground truth distance transform. Red is interior, yellow is the boundary, and blue is exterior.</figcaption></figure><p>For the distance transform to be a good predictor of buildings, we must assume that the footprints of buildings do not overlap. The slightest overlap essentially merges the two distinct buildings into one. Also, the strategy assumes simple connectivity of buildings (no holes) — this may produce some errors for non-simply connected buildings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*NVCp6bzxMtN-yRej6X7lTg.png" /><figcaption>Image 6: Output of CosmiqNet v1 to predict distance transform. Red is interior, yellow is the boundary, and blue is exterior.</figcaption></figure><p>Since the output layer of CosmiQNet is a distance transform, training the network optimizes the Euclidean distance to the derived distance transform from the label. This cost function may not be optimal for finding footprints. Further analysis of the distance transforms provides a measure of how much noise there is in the output layer — as the network trains, the measure of noise in the output layer provides valuable feedback without having to perform more expensive post-processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*NaYUGP322FJkyjXnbtNttQ.png" /><figcaption>Image 7: Output of clustering algorithm applied to CosmiQNet v1. Different colors correspond to the prediction of different building footprints.</figcaption></figure><p>The Greedy Algorithm for growing clusters from the distance transform is dependent on a threshold; we have selected a 0.5 exterior/interior threshold. Also, it is possible to dilate the clusters to grow them as well. Since the scale is known and some assumptions can be made about the size of the building footprints, one could choose these parameters even on a per cluster basis. We did not investigate the optimization of these parameters at this time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zCtrPrh-KUhbKY0hdB3Z5A.png" /><figcaption>Image 8: CosmiQNet-v1 Proposals above IoU Threshold of 0.5 (green)</figcaption></figure><p>Finally, we have some IoUs. The green polygons are the true positives of the algorithm with the IoU label on the polygon.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1007/1*L8sx9oWY3r0lnpWWa004QA.png" /><figcaption>Image 9: CosmiQNet-v1 Proposals above IoU Threshold 0.25 (yellow). The true positives with respect to the 0.5 threshold are also plotted in green.</figcaption></figure><p>We also show an overlay with near-misses: proposed building footprints with IoU below the threshold that could become true positives with refinement of post-processing and better training of CosmiQNet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LPKXNgBp92OWGZwNgTAFUw.png" /><figcaption>Image 10: CosmiQNet-v1 Proposal with Labeled IoU. The red proposal in the Northeast region that is being counted as a false positive is a result of a ground truth labeling mistake. Yellow regions are false positives that are close to the threshold and green regions are true positives.</figcaption></figure><p>All proposed building footprints are shown with their respective IoUs. Even the significant misses have relevant information that could be useful for secondary testing or counting. The greedy clustering algorithm does not factor into account the rectangular (for the most part) shape of the labeled building footprints. Post-processing that includes this ansatz could improve the accuracy of the algorithm.</p><h3>Scores</h3><p>CosmiQNet v2 predicts the distance transform better than CosmiQNet v1, but does not significantly differ in performance with respect to the IoU metric at a threshold of 0.5. There are several parameters to be optimized and post-processing that can be done to improve the performance of CosmiQNet.</p><p>A bounding box is the smallest vertically aligned rectangular region that contains the building footprint. In image 11, the bounding boxes are shown in yellow and green for the building footprint in blue. The area of the building footprint is always less than the area of the bounding box — in some cases the difference in areas is large enough to have the IoU between the bounding box and the ground truth footprint to fall below the 0.5 threshold. The F1-score for using bounding boxes as proposals is 0.729.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*_lcvyphHDzppQKsqumCipg.png" /><figcaption>Image 11: Example image with ground truth derived bounding boxes. Blue polygons are ground truth building footprints. Green rectangles are bounding boxes derived from ground truth footprints and have an IoU with the corresponding footprint greater than the 0.5 threshold (true positives). Yellow rectangles are bounding boxes derived from the ground truth footprints but have an IoU less than 0.5 (false positives).</figcaption></figure><p>Another measure of comparison is the application of the greedy clustering algorithm to the ground truth distance transform. The F1-score for this procedure is 0.54, indicating that improving the post-processing should have a large impact on the accuracy of the classifier. The F1-scores for CosmiQNet version 1 and version 2 are 0.19 and 0.20 respectively. A future blog post will discuss the details of the metric and methods to improve performance of the classifiers.</p><p>The F1-scores above may seem low compaerd to the touted accuracies of the top performers from the <a href="http://image-net.org/challenges/LSVRC/2016/">ImageNet Large Scale Visual Recognition Competition</a>. But it is important to keep in mind the following:</p><ul><li>The algorithms from the ILSVRC have been optimized for the ImageNet data sets.</li><li>ImageNet uses bounding boxes for object detection as opposed to the polygonal building footprint labels in SpaceNet.</li><li>Some SpaceNet images have a large number of tightly packed buildings; in these cases, it is easier to label a region as being in a building than it is to separate the buildings from each other.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e691961d257" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-downlinq/object-detection-on-spacenet-5e691961d257">Object Detection on SpaceNet</a> was originally published in <a href="https://medium.com/the-downlinq">The DownLinQ</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Establishing a Machine Learning Workflow]]></title>
            <link>https://medium.com/the-downlinq/establishing-a-machine-learning-workflow-530628cfe67?source=rss-387385dac9e------2</link>
            <guid isPermaLink="false">https://medium.com/p/530628cfe67</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[satellite-technology]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Patrick Hagerty]]></dc:creator>
            <pubDate>Fri, 02 Sep 2016 21:40:57 GMT</pubDate>
            <atom:updated>2016-09-04T19:09:41.416Z</atom:updated>
            <content:encoded><![CDATA[<p>While the rapid advancements in neural networks offer enormous promise for the field of imagery analytics, application of these technologies against an ever-increasing volume of commercially available satellite imagery remains in its early stages. CosmiQ Works has developed and tested a feasible technical workflow to quantitatively assess the performance of open-source machine learning (ML) algorithms against satellite images.</p><p>The workflow, illustrated below, includes all of the necessary steps to test the performance of ML algorithms against satellite imagery. At the highest level, the workflow can be classified into two parts: human intensive activities; and computationally intensive activities. It is important to determine the necessary human and computational resources before beginning an ML experiment.</p><p>We will explore each part of the workflow in greater detail in upcoming blog posts. We plan to apply a majority of our focus to the most resource intensive activities: data labeling, training the imagery classifier and testing the imagery classifier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UzHbNGFuFneTwSkDU2MzyQ.png" /></figure><p>Several logos appear in the workflow to represent products that CosmiQ Works used during each of the steps of the workflow. The list of logos is not meant to be exhaustive but rather suggestive of products that facilitate the ML process, some with special support for satellite imagery.</p><h3>Labeling:</h3><p>The human intensive aspect of ML is the labeling of the data. Proper labels often require subject matter expertise in the objects of interest and in the desired labels to be associated with the objects. Satellite imagery has attributes that complicate the labeling process. For example, satellite imagery is large, multi-channeled, geo-tagged, and each pixel may store more than eight bits per channel. Common image libraries typically do not handle the imagery very well. CosmiQ Works used the Open Source Geographic Information Systems <a href="http://qgis.org/">QGIS</a> as a framework for managing, displaying, and labeling the satellite imagery. The label is stored as a vector layer with geo-references and is typically output as either a <a href="http://geojson.org/">GeoJSON</a> file or as an <a href="http://www.esri.com/">Esri</a> Shapefile. For feature extraction, <a href="http://opencv.org/">OpenCV</a> supports standard 3-band imagery while <a href="https://www.orfeo-toolbox.org/">Orfeo Toolbox</a> is specifically designed for remote sensing applications. <a href="http://gdal.org/">GDAL</a> is a useful library for processing the geospatial data from the command line or Python scripts.</p><h3>Pre-process Data &amp; Feature Extraction:</h3><p>Automated analysis is usually optimized for a specific type of input, e.g., size of input image, contrast within the image, and physical dimension of the image. To improve accuracy, imagery is pre-processed by a variety of techniques. CosmiQ Works performed only minimal processing in terms of pixel normalization and image scaling using a standard computer vision library <a href="http://opencv.org/">OpenCV</a>.</p><p>Feature extraction differentiates deep learning from classical machine learning. In classical machine learning features are determined a priori to the training. Sometimes these features are extracted manually (hand-crafted) and can be extremely effective for identifying regions and objects of interest, but often require expertise. More frequently, however, features are extracted automatically by a pre-determined suite of algorithms. On the other hand deep learning algorithms have the capability of learning a hierarchy of features during the training process, which can greatly increase the flexibility of the models; some learned features are optimized for the intended task and some are general enough to be used across multiple tasks.</p><p>While OpenCV is good for manipulating 3-band imagery, it does not support the geographic data embedding in the imagery. GDAL is a useful software library for working with satellite imagery, supporting multi-band, 16-bit color depth, geotagged imagery. Many GIS frameworks rely on GDAL for basic functionality. Classical feature extraction (such as SIFT/SURF/Canny edge detection) is supported by OpenCV as well as the Orfeo Toolbox specifically designed for remote sensing.</p><h3>Dividing Data into Test Data and Training Data:</h3><p>The reason for dividing data into training and testing data sets is to provide independent validation in order to prevent overfitting model parameters during the training process. While this may seem straightforward for computer vision, geographic and temporal correlations are unavoidable in satellite imagery. For our experiments, we wrote Python scripts to randomly divide labeled sub-regions of satellite imagery into either a testing set or a training set. The ML algorithm only optimizes using the training set. The testing set is used to provide a measure of overfitting of the training data. For additional validation, we reserved separate imagery to validate the trained classifiers.</p><h3>Synthesize New Labeled Imagery:</h3><p>When there is a significant difference in accuracy of a trained classifier on a test set compared to the accuracy on a training set, overfitting is the likely culprit. Increasing the amount of labeled data helps mitigate overfitting. For satellite imagery, the objects of interest may be either limited or expensive to find and label. Using symmetries in the imagery, one can simulate new images from the initial training set. For example, some common symmetries include image rotations, increased noise, change in lighting, and scaling of imagery. Generating new images using these symmetries introduces correlations in training data but the benefits usually outweigh the newly introduced correlations.</p><h3>Import parameters:</h3><p>In deep learning, one can reduce training time and data requirements by importing parameters from previously trained neural networks. Even if the applications are different, the features represented by the lower layers of a trained neural network are often sufficient features for training a new classifier. AlexNet, GoogLeNet, and ResNet are previous ImageNet winners that have publicly available neural networks and trained weights. While these networks have not been optimized for satellite imagery, we have used the pre-trained weights as starting points for the training process.</p><h3>Training and Testing:</h3><p>The computational intensive part of ML is the optimization of the parameters of the classifier. Neural networks commonly have millions of parameters and can only be optimized using special hardware, software libraries, and starting values for the parameters.</p><p>Hardware options for deep learning include:</p><p>· Specially designed, energy efficient ASICs;</p><p>· Reprogrammable FPGAs;</p><p>· Highly parallelized GPUs; and</p><p>· Large memory supported CPUs.</p><p>We used high-end consumer hardware to train algorithms for image classification and object detection; our computational server is the NVIDIA DevBox with four Maxwell GeForce Titan X GPUs. Advantages for different hardware choices depend on the application, available budget, and the desired training time.</p><p>There are several software frameworks for performing deep learning; most support parallelization on NVIDIA GPUs. Initially, we chose to work with the deep learning framework Caffe because of the Python support and the access to pre-trained networks. For programmers comfortable with Python or C++, Tensorflow is a well-documented framework with a growing developer base. We generally design new network architectures in Tensorflow, but use Caffe to fine-tune pre-trained networks. NVIDIA Digits is a polished frontend to labeled data management and Caffe-based model training.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=530628cfe67" width="1" height="1" alt=""><hr><p><a href="https://medium.com/the-downlinq/establishing-a-machine-learning-workflow-530628cfe67">Establishing a Machine Learning Workflow</a> was originally published in <a href="https://medium.com/the-downlinq">The DownLinQ</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>