<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by SeanGorman on Medium]]></title>
        <description><![CDATA[Stories by SeanGorman on Medium]]></description>
        <link>https://medium.com/@seangorman?source=rss-e7b227aa2213------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*UlGY50vQk2wymTR4.jpeg</url>
            <title>Stories by SeanGorman on Medium</title>
            <link>https://medium.com/@seangorman?source=rss-e7b227aa2213------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 13:30:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@seangorman/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[NeRF World (+3DiM)]]></title>
            <link>https://medium.com/@seangorman/nerf-world-3dim-d8ba3a028ac4?source=rss-e7b227aa2213------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8ba3a028ac4</guid>
            <category><![CDATA[nerf]]></category>
            <category><![CDATA[diffusion]]></category>
            <category><![CDATA[maps]]></category>
            <category><![CDATA[photogrammetry]]></category>
            <category><![CDATA[3d]]></category>
            <dc:creator><![CDATA[SeanGorman]]></dc:creator>
            <pubDate>Mon, 10 Oct 2022 21:50:28 GMT</pubDate>
            <atom:updated>2022-10-11T15:33:32.677Z</atom:updated>
            <content:encoded><![CDATA[<p>My various social media feeds have been alight with 3D models. Some of these are models of the real world and others are more ayahuasca infused hallucinations. They are all pretty mind blowing and in part driven by two relatively new computational techniques “neural radiance fields” (NeRF) and “3D diffusion models” (3DiM). Both techniques use amazingly clever techniques to address challenges of using traditional photogrammetry for creating detailed 3D scenes from 2D images. We’ll start with a quick overview of both techniques and how they work and then talk how all this is applicable to geospatial imagery and 3D models. As a bonus we’ll dig a bit into how Google is using all this for Immersive View and how the rest of us could be doing the same soon.</p><h3>NeRF</h3><p>Over the last few years we’ve seen the power and limitations of deep learning for solving a variety of tasks with earth observation imagery — classifying scenes, detecting objects and even semantic segmentation. You can think of NeRF as an extention of this work but in the 3D realm instead of 2D. As with previous deep learning work the innovations began with non-spatial photographs. Specifically, reconstructing a 3D shape from 2D images by synthesizing new views of an object not seen in the sample 2D photos.</p><p>In the geospatial world we are pretty familiar with the traditional photogrammetric process of turning 2D aerial and satellite images into 3D models. A big problem with our traditional approach is if we often don’t have a view of all the angles in our 2D imagery set, and we end up with holes or melty artifacts in our reconstruction. NeRF helps solves this problem by synthesizing new views that are missing in the data set to fill in the gaps. Magically, it can also solve problems like glass and other reflective surfaces not reconstructing in a photogrammetry model. How does all this sorcery work? Let’s take a shallow dive in.</p><p>NeRF starts its life just as any standard photogrammetry project would. We’d collect a set of photos of a scene from multiple perspectives, but we don’t have to adhere to the typical photogrammetry rules of needing 50% overlap. Allowing us to use sparser photo collections. Image collections rules aside, we need to determine camera positions just as we would with photogrammetry. In fact new techniques like Nvidia’s <a href="https://developer.nvidia.com/blog/getting-started-with-nvidia-instant-nerfs/">InstaNeRF</a> use <a href="https://colmap.github.io/">COLMAP</a>’s traditional photogrammety pipeline to do just that. You can see the shift from input images to camera positions in the drum kit example from <a href="https://www.matthewtancik.com/nerf">Mildenhal</a>l et. al. (2020).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iFr7l42_dglmhbcQXRsMLw.png" /><figcaption>NeRF Formation</figcaption></figure><p>The novel part of NeRF comes in how the images and camera positions are then rendered into new views. Instead of the tradition <a href="https://en.wikipedia.org/wiki/Scale-invariant_feature_transform">SIFT</a> or <a href="https://en.wikipedia.org/wiki/Oriented_FAST_and_rotated_BRIEF">ORB</a> approach of finding common pixels between images to create key descriptors; NeRF instead iterates camera rays through the entire scene to hit every (x; y; z). The results is an output that can be described as a 5D vector-valued function whose input is a 3D location x = (x; y; z) and 2D viewing direction (θ; Φ), and whose output is an emitted color c = (r; g; b) and volume density (α) (<a href="https://medium.com/swlh/nerf-neural-radiance-fields-79531da37734">Bhaseen 2020</a>). As with most things 3D, visuals are a great aid and Mildenhall’s seminal paper has a great video laying out the concepts.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FJuH79E8rdKc%3Fstart%3D56%26feature%3Doembed%26start%3D56&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DJuH79E8rdKc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FJuH79E8rdKc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/19e4bdbe89d3c684c9cf2676b4b46fe9/href">https://medium.com/media/19e4bdbe89d3c684c9cf2676b4b46fe9/href</a></iframe><p>An additional benefit, that isn’t entirely obvious in the video, is that the NeRF approach generally does a much better job reconstructing a full 3D scene vs. just the target object, as result of the ray based approach. While NeRF has gotten most of the attention for amazing new 3D reconstruction techniques there has also been some amazing advancements in diffusion, specifically 3D diffusion models (3DiM).</p><h3>3DiM</h3><p>Diffusion models have really jumped into the popular consciousness through impressive generative art projects like <a href="https://openai.com/blog/dall-e/">DALL-E</a>. More broadly we can define this class of models as:</p><blockquote><strong>Diffusion Models</strong> are generative models, which means that they are used to make data that looks like the data they were trained on. Diffusion Models work by destroying training data by adding Gaussian noise to it over and over again, and then learning how to get the data back by reversing this process of adding noise (<a href="https://medium.com/mlearning-ai/enter-the-world-of-diffusion-models-4485fb5c5986">Gross 2019</a>).</blockquote><p>Another way to think of it is with diffusion models you are increasing image sample fidelity and the cost of sample diversity. Similar to NeRF this is potentially great for geospatial use cases where the sample size for specific locations on the earth can often be quite limited (satellite, aerial, drone, street view etc.). In the case of 3DiM scientists use a diffusion model to reconstruct a 3D scene from a limited number of images including just a single image.</p><p>While NeRF has exploded with a wide range of related methods 3DiM is still quite new and revolves around one <a href="https://3d-diffusion.github.io/">core paper</a> out of Google Research.</p><blockquote>3DiM takes a single reference view and a relative pose as input and uses diffusion to create a new view. After our new stochastic conditioning sampler, 3DiM can then make a whole 3D scene that is consistent.</blockquote><p>A visual, again, goes a long way describing what is going on with the new 3D diffusion based modeling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*QgMFiHBsrdUP-SE5npAbFg.gif" /><figcaption>Watson et. al. 2022</figcaption></figure><p>To what extent 3DiM will be applicable to geospatial data in not yet known. Given the work is coming out of Google who has already applied NeRF in spectacular fashion to their new <a href="https://blog.google/products/maps/three-maps-updates-io-2022/">Immersive View</a> we’ll probably find out sooner rather than later.</p><h3>Geospatial NeRF in Action</h3><p>Bringing up Immersive View is a good excuse to dig into just how impactful these new techniques can be for geospatial 3D modeling. What follows is a bit of reverse engineering on my part, so take it with a grain of salt. At the end of the Spring this year Google announced Immersive view, which showed of amazingly realistic real world views of iconic landscapes around the globe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*jgX2JZRlh2AibEf6rJ0X5A.gif" /><figcaption>Google’s Immersive View</figcaption></figure><p>There is a great tip off at the beginning of this view as to how Google does part of this magic. The multi-colored point cloud is a sematically segmented set of key descriptors derived from Google street view imagery (buildings = blue, vegetation = green, roads = yellow etc.). The key descriptors aren’t only the common pixels between images for doing photogrammetry, they are also the features used for augmented reality localization that Google utilized for Live View.</p><p>The reconstruction of the street view imagery can create a wonderful 3D model of the first two stories of buildings, but the Immersive View has the whole city — roof included. This is where Google’s aerial imagery comes to bear. In a previous <a href="https://blog.google/products/maps/google-maps-101-how-imagery-powers-our-map/">blog post</a> Google highlights how they co-register their aerial and street view imagery to create blended photogrammetric models.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*RN_MNGli2Z371SSI4G589w.gif" /><figcaption>Google Co-registration</figcaption></figure><p>Once you’ve co-registered your street view and aerial imagery, and run something like COLMAP to get a common set of camera positions for both sets of imagery, you are all set to NeRF your world. Without doubt Google has done something amazing here, but it is not out of the realm of mere mortal like us.</p><h3>DIY NeRF World</h3><p>If you want to do something super cool like Google most of the ingredients are at hand with some caveats. The caveats being: not all the software pieces are open source and currently you need a street view data source with telemetry (e.g. video off commodity mobile phones doesn’t quite cut it). That said let’s roll through what is available.</p><p>On the photogrammetry front there are some many great packages available, both open source and proprietary. Back at Pixel8earth we used <a href="https://github.com/alicevision">AliceVision</a> and most of the big tech players use the aforementioned <a href="https://github.com/colmap/colmap">COLMAP</a>. I’d be remiss also not calling out <a href="https://github.com/OpenDroneMap/ODMdata">OpenDroneMap</a> from the geospatial community. For inputs you can run with aerial, satellite, drone, street view and action cameras (GoPro et. al.). Then comes the hard part co-registration of your aerial and terrestrial point clouds.</p><p>This was at the core of what we did at <a href="https://pixel8earth.medium.com/">Pixel8earth</a> and there are a good number of blog posts laying out <a href="https://pixel8earth.medium.com/industrial-and-commodity-data-agreeing-on-x-y-z-57372afd2ea">how we did it</a> if not the code. In the grand plan we’d like to revisit an open source option, likley built around NeRF, that steers clear of our old IP. That said there are existing proprietary co-registration solution that are very nice, including <a href="https://www.maxar.com/products/precision-3d-registration">Maxar’s P3DR</a> and <a href="https://edgybees.com/">Edgybees</a>.</p><p>Once you’ve co-registered your resulting point clouds and imagery it opens up the door to a whole bunch of interesting NeRF models. There is Waymo’s <a href="https://waymo.com/research/block-nerf/">Block-NeRF</a> for street view reconstructions. Also, we have <a href="https://github.com/city-super/BungeeNeRF">Bungee-NeRF</a> (nee City-NeRF) for city scale reconstructions from aerial imagery. Not to mention <a href="https://centreborelli.github.io/satnerf/">Sat-NeRF</a> and <a href="https://github.com/Kai-46/SatelliteNeRF">Satellite-NeRF</a>. For a community scale project nothing would be better that leveraging something like <a href="https://mobile-nerf.github.io/">Mobile-NeRF</a> to allow anyone with a mobile device to contribute.</p><p>This was always the dream for Pixel8earth — allowing anyone with a phone to map the world in 3D. Alas the vagaries of commodity mobile phone GPS foiled us every time. This why in the short term we’ve committed ourselves to seeing if we can fix that problem. That, though, is another blog post for another time. I’ll add one last observation in regards to additional directions and potential. While I love Immersive View I don’t think the future is photorealistic representations of the earth. The underlying data is semantically segmented and we could go further adding doors, windows, awnings etc. — there are lots of models to support more features. In total it would create a fantastic substrate for procedurally rendering all sorts of fun worlds that would still be rooted in a geographic reality.</p><p>We’ve covered a good bit in this post and I’ve skipped over lots of thorny problems. Some I’m aware of like what is the NeRF model that works for a multisource data set? Everyone I listed is bespoke to the source. Another problem that comes up frequently is the lack of ETL for NeRF outputs. How do I get a mesh or other formats from a NeRF rendering. Maybe something like <a href="https://github.com/gmum/points2nerf">Points2NeRF</a> begins to lay the foundation. Then there are all the thorny issues I’m oblivious to or got wrong in this post, and I’ll learn as it all evolves. Hopefully this post helps push in the direction that for the geospatial community there are lots of good reason to start exploring!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8ba3a028ac4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Going Digital to Analog: Turning Webcams into Thing Counters]]></title>
            <link>https://medium.com/@seangorman/going-digital-to-analog-turning-webcams-into-thing-counters-3d038624d05a?source=rss-e7b227aa2213------2</link>
            <guid isPermaLink="false">https://medium.com/p/3d038624d05a</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[algorithms]]></category>
            <dc:creator><![CDATA[SeanGorman]]></dc:creator>
            <pubDate>Thu, 07 Apr 2016 14:56:18 GMT</pubDate>
            <atom:updated>2016-04-07T17:05:04.100Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<p>-</p><p>At Timbr.io we’ve been plugging away at building a platform to share reusable algorithms and data sources. There are so many fantastic algorithms emerging in academic papers and as open source libraries we think it is a good time to help organize the work we are collectively creating. Instead of having to decipher and implement a new algorithm from scratch, we want to enable the community to build on each other’s work. That way we can avoid reinventing the wheel and can focus on creating new ideas.</p><p>The concept can get a bit esoteric, so we figured an example would help tell the story. We recently did an internal hack day to build a demo for a start up pitch event. The team kicked around several cool ideas, and the winner was a pipeline that scrapes images from a webcam and an algorithm that counts transient things in those images. We really liked the idea of turning a digital image stream into an analog counter. Below is a video showing that work being repurposed by our product manager, Bill:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplayer.vimeo.com%2Fvideo%2F156446273&amp;url=https%3A%2F%2Fvimeo.com%2F156446273&amp;image=http%3A%2F%2Fi.vimeocdn.com%2Fvideo%2F557463832_1280.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=vimeo" width="1920" height="1080" frameborder="0" scrolling="no"><a href="https://medium.com/media/b7bac82a68ed1bffa3521dc1b25a8146/href">https://medium.com/media/b7bac82a68ed1bffa3521dc1b25a8146/href</a></iframe><p>The fun bit with the video is Bill doesn’t code but was able to repurpose the team’s work to create his analytic.</p><h4>How it Works</h4><p>One of our integration points for the platform is Github. Specifically, the ability to share your algorithms as gists and post your pipeline data snapshots to Github. Since we are in the middle of our beta I’ll share this example workflow through those publicly exposed points. We’ll start things off with the code for the webcam scraper</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a1edc2ad002abe22ffa5fe0c9acef270/href">https://medium.com/media/a1edc2ad002abe22ffa5fe0c9acef270/href</a></iframe><p>Using this source we can build a data snapshot. For the purposes of portability to a Jupyter notebook I used Timbr.io’s native HDF5 format to share the data <a href="https://github.com/Seagor/Web-Cam-Counter/blob/master/steamboat_christiecam.h5">here</a>. With a nicely structured data snapshot we can dive into a Jupyter <a href="https://github.com/Seagor/Web-Cam-Counter/blob/master/Steamboat%20Webcam%20People%20Counter.ipynb">notebook</a> for some analysis. With a new data source I like to check to make sure I’ve configured my polling interval correctly first. <a href="http://timbr.io/aboutus/">Pramukta</a> wrote a nice unique timeline visualization (<a href="http://stackoverflow.com/questions/32619424/is-it-possible-to-plot-timelines-with-matplotlib">inspired by</a>) that is super handy to check this.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ac23e72bdb703aa74a84ca02f9502d5b/href">https://medium.com/media/ac23e72bdb703aa74a84ca02f9502d5b/href</a></iframe><p>The output ends up looking like so:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o_tJVF-RBo8DC4SgtCywIg.png" /><figcaption>Unique Image Timeline</figcaption></figure><p>Each unique color represents a unique image — till you run out of colors and it repeats. Good news is we have all unique images for our snapshot, so we are all set to do a bit analysis with it.</p><h4>Detecting Objects in Images</h4><p>During a previous hack day <a href="http://timbr.io/aboutus/">Winnie</a> had tested out the facial recognition library in <a href="https://github.com/Itseez/opencv">OpenCV</a> to see if it would detect cat faces in an Imugr stream. Pramukta took that example and started evolving it into a people counter. He tuned it specifically for the Steamboat resort webcam we’d collected data for and ended up with this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/acf87f535df22e61232274274b54d234/href">https://medium.com/media/acf87f535df22e61232274274b54d234/href</a></iframe><p>He added an image slider in the snippet so you can scroll through the image collection and get people counts for each one — example below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vt1zUP-3Dq-0eXDCeqo8ow.png" /><figcaption>Steamboat People Counter in Action</figcaption></figure><p>The trick with getting most algorithms to work well is tuning them to the data. Sometimes we can do this through parameterization or often with training data. This is what makes generalizing algorithms to broad use cases so difficult, and why with Timbr.io we tackle algorithm sharing with a search, iterate and publish approach.</p><h4>Repurposing an Algorithm</h4><p>A good example of this was when one of our other data scientists, <a href="http://timbr.io/aboutus/">Jamie</a>, wanted to create a car counter for a border crossing webcam he found. Instead of reinventing the wheel he did a search for object detection algorithms and found Pramukta’s work. Using the shared code he tuned the detection parameters and added a couple of new methods for his car <a href="https://www.dropbox.com/s/t8vh3gzqgzax06k/laredo_ib1.h5?dl=0">snapshot</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3af7811d1c4f2b3fb5adfe6a72b07ed4/href">https://medium.com/media/3af7811d1c4f2b3fb5adfe6a72b07ed4/href</a></iframe><p>In the original ski area counter Pramukta took an image subtraction approach then detected contours. Since the background in ski area images was pretty consistent this worked well. For the border crossing webcam the movement of the sun and shadows over time made this trickier. So, Jamie added a sliding window average to the method as well as opening the morphologies in the image to reduce noise then closing them, so cars did not get joined into one object. Resulting in the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tEQAQZdLBjAycn_aoFHhFw.png" /></figure><p>Once Jamie had it well tuned he published it to Timbr.io and now the next person has a car counter and person counter to choose from. We hope this approach will create a virtuous cycle of contributions. Each new algorithm or data source makes the community that much smarter, while creating new opportunities for novel analysis. Our job will be recommending the best algorithms and giving you the best environment to iterate your custom solution.</p><p>While Github is an awesome place to post up snippets and data sets we think there is a biggish missing piece of fusing data+algorithms together seamlessly. An environment that lets you easily collect and structure data then mesh algorithms with it in one spot. If you’d like to check it out — hit up Timbr.io to sign up for early access. We’ll be posting more examples as we keep building the platform.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d038624d05a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>