<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kendallfrimodig on Medium]]></title>
        <description><![CDATA[Stories by Kendallfrimodig on Medium]]></description>
        <link>https://medium.com/@kendallfrimodig?source=rss-a5438bda42ef------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*fdoRKWUtdlos3ySY</url>
            <title>Stories by Kendallfrimodig on Medium</title>
            <link>https://medium.com/@kendallfrimodig?source=rss-a5438bda42ef------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 16:42:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kendallfrimodig/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Efficient Object Detection Within Satellite Imagery Using Python]]></title>
            <link>https://medium.com/@kendallfrimodig/efficient-object-detection-within-satellite-imagery-using-python-85331d71ff69?source=rss-a5438bda42ef------2</link>
            <guid isPermaLink="false">https://medium.com/p/85331d71ff69</guid>
            <category><![CDATA[rasterio]]></category>
            <category><![CDATA[qgis]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[gis]]></category>
            <dc:creator><![CDATA[Kendallfrimodig]]></dc:creator>
            <pubDate>Thu, 28 Jul 2022 19:37:24 GMT</pubDate>
            <atom:updated>2022-07-28T19:41:14.900Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Part 1:</strong> Data Acquisition &amp; Pre-Processing</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*AmTvhARicjexC0lTe4Aypw.png" /></figure><p>June 23, 2022</p><p>Kendall Frimodig</p><h4>Overview of Series</h4><p>Today I’ll be introducing a series of technical walkthroughs, for applying an object detection algorithm, such as YOLO or Mask-R-CNN, to satellite imagery with the ultimate goal of detecting specific objects, and maintaining the formatting necessary to re-project such findings to a map. The focus will be on data-engineering tricks I’ve found along the way or developed, to ultimately make running the project on a laptop possible, which was essential for me.</p><p>Given the complex nature of the task at hand, I plan to break the workflow into a 5 part series</p><ul><li>[ 1 ] Data reduction and preprocessing <strong>todays topic</strong> (test/train image creation)</li><li>[ 2 ] Creating annotations with QGIS for Mask-R-CNN algorithm (separating test/train images)</li><li>[ 3 ] Creating annotations with python for YOLOR algorithm (w/o use of roboflow, label tool)</li><li>[ 4 ] Running YOLO-R algorithm in Google Colab</li></ul><p>On to todays topic, data acquisition, and preprocessing</p><h4>Outline for Today</h4><p>The method I want to emphasize most as having scalable utility for dealing with big data such as satellite images, is the combined use of an area of interest filter, and a tile slicing function in python. By combining the two, the total pixel count can be greatly reduced.</p><p><strong>If you have a specific region where your objects will appear, then all pixels within this area could be candidates for test or training images.</strong></p><p>Today will explain creation of all candidates — later posts will explain delineating test and training images into their respective folders</p><p>The Area of Interest (AOI), sometimes referred to as Region of Interest (ROI), is simply a geo-formatted polygon representing the locations where your target could potentially exist.</p><p>This will differ between analysis, but given the vast public repository of spatial data — it is likely there exists a GeoJSON or SHP file which can narrow down where your hunt.</p><p>Lets say were looking to count the number of vehicles on the highway. Since our objects will only be present within a 30 feet of the road centerline, we can use a road shapefile, create a buffer around the line, and clip the composite image.</p><p>Here is a crude example I’ve put together for demonstrating this data-reduction technique.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*8RF2gAE2t2TrSmmQRfD8jA.png" /></figure><p>The remainder of this post will outline these 4 steps to convert the bulky composite, into a manageable amount of data for a neural net to process</p><h4>Step 1: Data Acquisition</h4><h4>Downloads</h4><p>If you’ve done some poking around, you will know the main sources of public imagery are the satellites LANDSAT, MODIS, and if you’re in the US the National Agriculture Imagery Project takes aerial photos for each state every two years. For object detection, we want the best resolution possible, so the NAIP imagery is ideal at a &lt; 1m resolution.</p><p>27 TB of NAIP data can be downloaded by US county, for many years as a .sid format. This permits temporal analysis, for example counting households for a given county in 2015 and then in 2019 for example. You can source smaller JPEG images of the county composites, however I’ve found the quality to be lower, and only 2015 is available on the national map. Google Earth’s public image collections for NAIP are similarly restricted to 2015.</p><p><a href="https://nrcs.app.box.com/v/naip/folder/17936490251">Source Data Link</a></p><h4>Compression</h4><p>The challenge today is the file size of high quality imagery (in python). My analysis, performed on a single county in Colorado, consisted of a roughly 1 GB download. The first step is to convert this strange .sid format into a GeoTiff, as the .sid format is incompatible with pythons raster analysis libraries</p><pre>gdal_translate -co COMPRESS=JPEG -co TILES=YES source-file-name.sid  output-file-name.tif</pre><p>If not compressed, the image will be roughly 40 times the size on disk. I’m no expert when it comes to compression, but comparing the 40 gb uncompressed and 1.2 GB compressed GeoTiff, I could not tell there was any loss of pixel information.</p><p>Can you tell the difference?</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*07hpsE83mz5DDRBg1kJ0qg.png" /></figure><p>However as mentioned before, when this 1.2 gb image is loaded into python as a 3 band numpy array, it required 30gb in memory so I ultimately found the best approach was to slice the composite image into many small pieces.</p><h4>Step 2: Define Area of Interest, Mask Image</h4><p>In my case, I used national forest roads as a mask. Using QGIS I converted the road centerlines to a ‘buffer’ which creates a polygon centered on the roads, including anything within 500 feet of the road. The buffer tool in QGIS is a easy and approachable task.</p><p>Here’s the entire county composite with area of interest (AOI — buffered roads) shown</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*izlVh97wqENY8G4fzoM42g.png" /></figure><p>And here’s with the ‘buffer’ applied as a mask — or in other words the composite clipped by the AOI. You can mask an image with the following GDAL command (using prebuilt QGIS toolbox or command line).</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*9p7V_GQih45L3zqGwYuwrQ.png" /></figure><p>Using command line with GDAL installed</p><pre>gdalwarp -overwrite -of GTiff -cutline &quot;path-to-mask&quot; -cl  -crop_to_cutline &quot;path-to-raster&quot; output-file-path</pre><p>with QGIS</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*YXTsrAu0eS2g1vOWQwp8ww.png" /></figure><h4>Step 3: Slice Composite Into TILES</h4><p>Now we can start working in python. Required libraries as follows</p><pre>import rasterio <br>import numpy as np<br>import pandas as pd<br>import shutil<br>import os</pre><p>Here are the functions which break the composite into many smaller pieces. On a 3-band, 101,464 by 116,937 pixel composite this took just over an hour.</p><p><strong>After I explain the following step I took, to reduce the data-set size for modeling, I’ll insert an exception to the final function (embed the second step) so that the processing time can be greatly reduced on the front end</strong>.</p><p>this will be included in part 2, where bounding boxes are created and test/train images are defined from the set of valid images created here</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a8d827c45ae36bc5dd435425c6decab5/href">https://medium.com/media/a8d827c45ae36bc5dd435425c6decab5/href</a></iframe><ul><li>the masked, satellite image (composite image)</li><li>data directory of your repository or project folder</li><li>destination folder for your sliced tiles</li></ul><p>You will also need to create a ‘data/tiles’ folder for now. Create this in your home directory (or wherever you have space) as only a subset will be transferred to your project folder</p><pre>img = rasterio.open(&quot;F:/chafee_clipped_compressed.tif&quot;)<br><br>dst_image_dir = &quot;F:/tiles/&quot;</pre><p>Slices the image composite for the county into 12,000 1,000 x 1,000 pixel tiles which retain original projection information</p><pre>generate_tiles(img, 1024, dst_image_dir)</pre><pre>100%|█████████████████| 100/100 [1:16:49&lt;00:00, 46.10s/it]</pre><p>The result — nearly 12,000 tiles, most containing ‘no data’ as they fell outside the region of interest:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*lmWvCvqWiv-elOKSZ6P34g.png" /></figure><p>For now I&#39;m leaving the functions above intact — in case you want to perform a sub setting of a composite image but aren’t filtering by a region of interest</p><p>The following section sifts through the 12,000 tile slices to obtain a list of valid training and test images (having any pixels - <strong>AKA</strong> within the region of interest the original image was masked upon)</p><p>Once we have this list of train/test images — we use ‘shututil’ to transfer the valid set of images to our project folder ‘tiles’ for now. Test and train will be created in later posts.</p><h4>Step 4: Filtering Unnecessary Data</h4><p>Once we slice the composite into a series of 1024 by 1024 pixel tiles however, we can use the python ‘rasterio’ library to filter out tiles containing no-data. Since our tiles were named accordingly in a numerical index fashion from the top left corner of the composite, given the image name we can back engineer where each tile belongs in the composite.</p><p><strong>If tile contains no pixel data, the RGB array sum will be 0 for each band, though only one needs to be checked</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/434747f28294c54b91e2f460e7361de7/href">https://medium.com/media/434747f28294c54b91e2f460e7361de7/href</a></iframe><p>Here we move the valid tiles (based on list above) to our repo’s data folder</p><pre>for each in valid_tiles:<br><br>    shutil.copy(f&quot;{dst_img_dir}{each}&quot;,f&quot;./data/valid_tiles/{each}&quot;)</pre><p>The result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*flNAIkoT9CiEux7vuHLJzQ.png" /></figure><h3>In Conclusion</h3><p>Using QGIS, we applied a polygon mask, referred to as a region of interest, to narrow down the potential pixels where our object might exist. We then used python to slice the large satellite image into smaller pieces, images which size are suitable for modeling. We used rasterio, to filter this complete subset of tiles to only those containing pixels, thus reducing our dataset from 33GB to 500MB — simply by applying a mask to narrow down our locations.</p><p>Next, we’ll discuss creating training labels (bounding boxes) with QGIS, and how to save a companion composite image, containing pixel locations of these boxes.</p><p>The steps introduced here will be modified to separate the set of valid tiles created here, into test and training folders (rather than one blob of all valid tiles).</p><p>We’ll also modify the tile slicing code, to skip saving the invalid tiles altogether. Thanks for reading, and I look forward to continuing on this series. Feel free to reach out if you have any questions!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85331d71ff69" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>