<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Adam Rodnitzky on Medium]]></title>
        <description><![CDATA[Stories by Adam Rodnitzky on Medium]]></description>
        <link>https://medium.com/@adamrodnitzky?source=rss-e03ec3123736------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*gGkepDFWlNdJAudc.jpg</url>
            <title>Stories by Adam Rodnitzky on Medium</title>
            <link>https://medium.com/@adamrodnitzky?source=rss-e03ec3123736------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 18:19:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@adamrodnitzky/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Tangram Vision Blog is Moving!]]></title>
            <link>https://medium.com/tangram-visions/the-tangram-vision-blog-is-moving-bb8b0f0ce004?source=rss-e03ec3123736------2</link>
            <guid isPermaLink="false">https://medium.com/p/bb8b0f0ce004</guid>
            <dc:creator><![CDATA[Adam Rodnitzky]]></dc:creator>
            <pubDate>Wed, 03 Mar 2021 18:26:24 GMT</pubDate>
            <atom:updated>2021-03-03T18:26:24.860Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4Xih5AFmnklHStWd1NppCw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@handiworknyc?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Handiwork NYC</a> on <a href="https://unsplash.com/s/photos/moving?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Thanks for joining us so far here on Medium! We have decided to self-host our blog at tangramvision.com. All of our existing posts have been ported over, and no more new posts will be added here.</p><p><strong>Visit the new Tangram Vision blog at </strong><a href="http://www.tangramvision.com/blog"><strong>www.tangramvision.com/blog</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bb8b0f0ce004" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tangram-visions/the-tangram-vision-blog-is-moving-bb8b0f0ce004">The Tangram Vision Blog is Moving!</a> was originally published in <a href="https://medium.com/tangram-visions">Tangram Visions</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sensors 101: 3D Sensing]]></title>
            <link>https://medium.com/tangram-visions/sensors-101-3d-sensing-66e528f276c4?source=rss-e03ec3123736------2</link>
            <guid isPermaLink="false">https://medium.com/p/66e528f276c4</guid>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[3d-sensor]]></category>
            <category><![CDATA[sensors]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[lidar]]></category>
            <dc:creator><![CDATA[Adam Rodnitzky]]></dc:creator>
            <pubDate>Wed, 27 Jan 2021 22:19:38 GMT</pubDate>
            <atom:updated>2021-01-27T22:58:05.918Z</atom:updated>
            <content:encoded><![CDATA[<p>Over the past decade, 3D sensors have emerged to become one of the most versatile and ubiquitous types of sensor used in robotics. In many robotic applications, 3D sensing has become the de facto choice for tasks such as near-field object detection and collision avoidance, surface and object inspection, and map creation.</p><p>There are a number of 3D sensing modalities available. In this roundup, we’ll be covering the three most common: Stereo CMOS (both active and passive), structured light, and time of flight. We won’t be covering LiDAR, despite the fact that LiDAR data is also three dimensional. LiDAR users, fret not…we’ll be covering LiDAR as its own separate class of sensors in a separate post in the future!</p><p><strong>Stereo 3D Sensors</strong></p><p>These ubiquitous sensors bring a combination of good overall performance coupled with low cost. Stereo 3D sensors come in two basic types: passive and active.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*V_HVgpEqMXnVdYE5jFEOBQ.jpeg" /><figcaption>Structure Core Active Stereo 3D Sensor. Image attribution: Occipital, Inc.</figcaption></figure><p>Passive stereo 3D sensors are the least expensive of all 3D sensors, as they use off-the-shelf components that are inexpensive to source and manufacture. In addition, they are available in a wide range of baselines, giving users a choice of sensing ranges that are best adapted to their use case. However, most rely on visible light to operate effectively, which means they struggle to perform in low-light or unlit conditions.</p><p>Active stereo 3D sensors add an infrared (IR) pattern projector which both increases the fidelity of 3D data captured, as well as adding more reliable operation in lower light conditions. However, the IR projectors on these sensors are limited in range, which relegates these sensors to near- and mid-range applications, regardless of baseline. The addition of an IR pattern projector effectively turns these sensors into hybrid stereo/structured light sensors (more on structured light 3D sensors in the next paragraph!).</p><p>Examples of stereo 3D sensors:</p><p><a href="https://www.stereolabs.com/">Stereolabs Zed</a> — Passive stereo 3D sensors available in large and small form factors, with wide and narrow baselines. The Zed Mini and Zed 2 feature integrated IMUs for kinetic data streams.</p><p><a href="https://www.ensenso.com/">Ensenso</a>- Industrial-grade stereo 3D sensors with ruggedized construction and Gigabit Ethernet connections.</p><p><a href="https://structure.io/structure-core">Occipital Structure Core</a> — Active stereo 3D sensor with bespoke IR projector and built-in IMU for kinetic data. Available as an embeddable module or as an encased standalone sensor.</p><p><a href="https://www.intelrealsense.com/">Intel RealSense</a> — The D series (D415, D435, D435i, D455) are a diverse range of low cost active stereo 3D sensors. The D435i adds a built-in IMU for kinetic data, while the newest D455 adds extended range that is suitable for mid-range robotic navigation.</p><p><strong>Structured Light Sensors</strong></p><p>Structured light is the most ubiquitous 3D sensing modality used in robotics, thanks to the popularity of the original PrimeSense 3D sensors, which included the first generation Microsoft Kinect.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M1gWNLda_romiApSlP17fg.png" /><figcaption>Microsoft Kinect for Xbox 360. By Evan Amos — Own work, Public Domain, <a href="https://commons.wikimedia.org/w/index.php?curid=16059165">https://commons.wikimedia.org/w/index.php?curid=16059165</a></figcaption></figure><p>Structured light 3D sensors combine low cost with high-fidelity 3D data capture, as well as performance in a wide set of lighting conditions, with one notable exception: direct sunlight, or bright indirect sunlight. This is because the infrared light source used by structured light sensors gets overpowered by infrared light in the same bandwidth that is naturally present in sunlight.</p><p>Like stereo 3D sensors, structured light sensors are available in an array of small to large baselines that are suited to tasks that occur at different ranges from the sensor.</p><p>Examples:</p><p><a href="https://orbbec3d.com/product-astra-pro/">Orbbec Astra</a> — A low-cost structured light 3D sensor that is often used as a drop-in substitute for the original Microsoft Kinect.</p><p><a href="https://www.kickstarter.com/projects/occipital/structure-sensor-capture-the-world-in-3d">Occipital Structure Sensor</a> (No longer available new) — The first standalone 3D sensor designed specifically to be used with mobile devices like Apple iPad and iPhone models. Compatible with <a href="https://structure.io/openni">OpenNI</a> for depth-powered application development. Now replaced by the <a href="https://structure.io/structure-sensor">Occipital Structure Sensor Mark II </a>active stereo sensor.</p><p><a href="https://en.wikipedia.org/wiki/Kinect#Kinect_for_Xbox_360_(2010)">Microsoft Kinect V1</a> (No longer available new) — The most ubiquitous consumer-grade 3D sensor ever produced. Originally launched to support Xbox gaming, the first Kinect became popular with roboticists and hackers for adding robust 3D sensing at low cost to robots, drones and other platforms. Compatible with <a href="https://structure.io/openni">OpenNI</a> for depth-powered application development.</p><p><a href="http://xtionprolive.com/primesense-carmine-1.09">PrimeSense Carmine</a> (No longer available new) — The Carmine series is near-identical in specification to the original Microsoft Kinect, and was available in short- and mid-range models. Unlike the original Microsoft Kinect, the Carmine does not include a microphone array. Compatible with <a href="https://structure.io/openni">OpenNI</a> for depth-powered application development.</p><p><a href="https://www.photoneo.com/phoxi-3d-scanner/">Photoneo</a> — Ruggedized 3D sensors for industrial applications. The Photoneo phoXi line is available in multiple baselines.</p><p><a href="https://www.zivid.com/">Zivid</a> — A range of structured light 3D sensors available in multiple baselines, with USB3 connection for the Zivid One series, and CAT 6A Ethernet connection for the Zivid Two.</p><p><strong>Time of Flight Sensors</strong></p><p>Time of flight (ToF) sensors are effectively LiDAR sensors — aka, light radar. They send out packets of infrared light, and record the time it takes for the infrared light sent out to return. Despite using infrared light sources like structured light sensors, the manner in which ToF sensors transmit and receive light make them less susceptible to interference from bright and indirect sunlight.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*wgZI3QJuKbagvd831m87wg.png" /><figcaption>PMD Time of Flight (ToF) 3D sensor. ToFExpert, <a href="https://creativecommons.org/licenses/by-sa/3.0">CC BY-SA 3.0</a>, via Wikimedia Commons</figcaption></figure><p>ToF sensors often have wider range capabilities than structured light sensors, and can operate accurately for longer distance use cases. However, they aren’t able to return 3D images with the fidelity offered by structured light or stereo 3D sensors. Therefore, they are better suited for tasks like navigation assistance versus surface inspections.</p><p>Examples:</p><p><a href="https://azure.microsoft.com/en-us/services/kinect-dk/">Microsoft Azure Kinect</a> — The newest member of the Microsoft Kinect family of 3D sensors is a compact ToF module that adds a 12MP RGB video camera that is registered to the depth stream for high quality color imaging and textures. The Azure Kinect includes a built-in IMU for kinetic data.</p><p><a href="https://pmdtec.com/picofamily/">PMD Pico</a> — Simple, compact ToF 3D sensors that can be purchased as embeddable modules, or as enclosed standalone sensors.</p><p><a href="https://www.asus.com/ch-en/Networking-IoT-Servers/Smart-Home/Security-Camera/Xtion-2/">ASUS Xtion 2</a> — Increasingly hard-to-find. However, like the original Microsoft Kinect, PrimeSense Carmine and original Occipital Structure Sensor 3D sensors, the Xtion 2 is compatible with <a href="https://structure.io/openni">OpenNI</a> for depth-powered application development.</p><p><strong>Comparing Stereo Sensors to Structured Light to ToF Perception Sensors</strong></p><p>So why choose one modality over another? Well, we provided hints in the above descriptions for each modality, but let’s get more concrete on why you might choose one over another.</p><p>At the highest level, the primary factors that determine 3D sensing selection are:</p><ul><li>Sensor Performance</li><li>Impact on Host</li><li>Environmental Limitations</li></ul><p>Let’s explore each of these factors in detail, and then show how each sensing modality compares.</p><p><strong>Sensor Performance</strong></p><p>The main parameters we’ll be exploring here are range, resolution and reliability.</p><p>Range is perhaps the most important parameter, because different robot tasks are defined by the range in which they occur. For example, mapping environments at high speed for autonomous vehicles is typically left to rotating LiDAR units, as they have ranges that can exceed 100 meters, which allows them to see distant obstacles well before the autonomous vehicle is upon those obstacles. Those same LiDAR units, however, are poorly suited for close visual inspections, as these tasks benefit from sensors with sub-meter ranges that can support enhanced detail capture and accuracy.</p><p>For stereo and structured light 3D sensors, range is a function of the baseline between the optical elements on the sensor itself. The rule of thumb is that the wider the baseline, the longer the range. Similarly, a narrower baseline creates a shorter range. So why not just spec a 3D sensor with the widest baseline possible? Well, for one, form factor limitations can preclude this approach. But, more importantly, wider baseline sensors’ sensing ranges start further away from the sensor. That means that a wide baseline sensor might ignore objects that are less than two meters from the sensor. A shorter baseline sensor might not have as long of a range, but it might be able to start sensing within .25 meters of the sensor. Therefore, baseline and the resultant range becomes a tradeoff of how near you need to sense versus how distant you need to sense. You get one or the other, but you can’t get both.</p><p>Most 3D sensors have ranges that start at around 0.5–1m from the sensor, and extend to 3–6m at best. Keep in mind that at the furthest reaches of a sensor’s range, resolution and accuracy can start to diminish rapidly. ToF 3D sensors typically have much greater range than structured light or stereo 3D sensors, and can often receive reliable signals from objects that are up to 25m away. However, ToF sensors often suffer from excessive noise, and resolution can be much lower than similarly priced 3D modalities.</p><p>Speaking of resolution, this is both a function of range, as well as the nature of the optics used by the sensor. Structured light sensors’ resolution is limited by the density of the pattern that they project (which gets less dense as distance increases), and the resolution of the IR camera that reads the pattern as it falls on objects and environments to be sensed. Stereo sensors’ resolution is limited by the native resolution of the cameras used for the sensor.</p><p>For tasks such as navigation, high resolution isn’t strictly necessary, unless objects that might cause an obstruction are very fine (an extension cable, for instance, may be on the edge of detectability for some low-resolution 3D sensors). For tasks such as surface inspections and 3D modeling, high resolution is much more desirable. Because these tasks often occur at close range, many structured light and stereo sensors are well suited to this kind of task. ToF sensors are best suited for mid-distance navigation tasks, or skeletal tracking. These tasks don’t require high-resolution sensing, but benefit from the range and operational speed of ToF sensors.</p><p>Lastly, let’s talk about reliability. For sensors, this comes in two forms: data acquisition reliability, and mechanical reliability. Data acquisition reliability comes down to the inherent limitations of each sensor modality. Stereo sensors, for instance, need sufficient ambient light in the visible spectrum to work properly. Conversely, infrared-based sensors like structured light and ToF suffer when exposed to too much ambient light, causing sensor washout and data integrity issues. These sensors also struggle with black surfaces (they absorb IR wavelengths) and shiny surfaces (they scatter projected IR light). Active stereo sensors with infrared pattern projectors offer a “best of both worlds” approach that broaden sensor reliability over a number of environments and lighting conditions.</p><p>Mechanical reliability is often the result of compromises inherent to a specific modality, or result from manufacturer cost cutting. Many consumer-grade sensors use USB for data and power transfer; this is fine in a lab or test environment. However, in real-world use, USB can be flaky and error-prone, resulting in intermittent sensor failures and compromised data. Ethernet is much more stable, but is rarely found on consumer-grade sensors. It is much more likely to be found on much more expensive industrial-grade sensors, where it is a prerequisite for a purchase decision.</p><p><strong>Impact on Host</strong></p><p>The main parameters we’ll be exploring here are power consumption, compute use, heat dissipation, data transmission and form factor.</p><p>All robotic hosts have finite amounts of compute and power available. Robots, drones and other vision-enabled devices are often resource balancing acts where multiple sensors, motors, actuators, and other components fight for scarce system resources. Different sensors have different impacts. Sensors with active IR components are much more power hungry than passive stereo 3d sensors, for instance. But passive stereo sensors with high dynamic range (HDR) cameras may use much more compute than a structured light sensor, as the images they capture are much more data rich. Thus, sensor decisions must be made holistically in context of the other components that require system resources (and that subject is complex enough to deserve its own blog post).</p><p>Similarly, data transmission and storage can often be limited on robots and drones. Again, the example of using an HDR-equipped 3D sensor can strain even robust WiFi networks given the sheer size of the datastreams being captured and transmitted.</p><p>Lastly, there are industrial design considerations that can limit what sensor options can be specified. These considerations include the size and weight of a sensor (particularly important for lightweight devices like drones), as well as a sensor’s ability to dissipate heat generated. The latter is of particular importance to IR-equipped sensors. The laser-based IR components in these sensors rely on operating within a specific temperature range to maintain the bandwidth at which they operate. Deviating by even a few degrees Fahrenheit outside of the operating temperature range can cause significant variations in sensor readings, corrupting the data generated enough to make it useless. Therefore, considerations must be made to ventilate or insulate these sensors, depending on the environments in which they will be used.</p><p><strong>Environmental Limitations</strong></p><p>The main parameters we’ll be exploring here are ambient lighting, and connectivity. We mentioned above that many of these sensors are sensitive to ambient lighting, but it bears repeating. Lighting challenges are one of the most common causes of 3D sensor failures for robots. A robot that worked exceptionally well in a laboratory environment with consistent lighting conditions will suddenly experience drastic and intermittent failures when placed in a real world context with changing, inconsistent lighting conditions. Therefore, it is important to know all the lighting permutations that may exist upon deployment, and account for those during sensor selection.</p><p>Connectivity is also important. If sensor data is to be processed and used locally, then connectivity isn’t an issue. However, for robots that will be deployed in an environment with low bandwidth, then it is either necessary to compress or filter sensor data on the host before transmitting, or it is necessary to choose a low bandwidth sensor type (for instance, ToF). Again, it is important to test for packet size thresholds under real world bandwidth constraints, as laboratory environments may not sufficiently reveal deficiencies.</p><p><strong>Synthesizing the Data and Deciding</strong></p><p>We’ve put together a handy chart showing different <a href="https://medium.com/tangram-visions/sensors-for-hire-e337aa64972c">robot tasks to be done</a>, and the 3D sensors that work best for those scenarios. We’ve noted what key decision factors should be considered for each application, and added notes to further clarify why specific choices should be made.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*cTZOwRPET3MFYqyUn1C3vQ.png" /></figure><p>We hope that you’ve found this guide to 3D sensors useful! Our next sensor roundup will cover LiDAR sensors, so stay tuned.</p><p>If you’re currently working on a vision-enabled product like a robot or a drone, please take a look at what we do at <a href="https://www.tangramvision.com/">Tangram Vision</a>. We can help you get your product to market faster, and keep it functioning more reliably upon deployment, too.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=66e528f276c4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tangram-visions/sensors-101-3d-sensing-66e528f276c4">Sensors 101: 3D Sensing</a> was originally published in <a href="https://medium.com/tangram-visions">Tangram Visions</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Event Cameras: Where Are They Now?]]></title>
            <link>https://medium.com/tangram-visions/event-cameras-where-are-they-now-293343754bfd?source=rss-e03ec3123736------2</link>
            <guid isPermaLink="false">https://medium.com/p/293343754bfd</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[neuromorphic-computing]]></category>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[sensors]]></category>
            <category><![CDATA[event-camera]]></category>
            <dc:creator><![CDATA[Adam Rodnitzky]]></dc:creator>
            <pubDate>Wed, 30 Dec 2020 21:10:06 GMT</pubDate>
            <atom:updated>2021-03-12T17:37:47.579Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="A Portrait Of The Author As An Event Camera Owner" src="https://cdn-images-1.medium.com/max/640/1*rvbTwEEcIZJtRFLdpWNiCg.jpeg" /><figcaption><strong>A Portrait Of The Author As An Event Camera Owner.</strong> Image captured by a Samsung SmartThings Vision camera, June 2020.</figcaption></figure><p>Two years ago, the computer vision community was abuzz with anticipation for a new type of sensor that promised a revolution in speed and efficiency.</p><p>These new sensors are called <strong>event cameras</strong>, and also know as dynamic vision sensors (DVS) or neuromorphic cameras.</p><p>Yet two years later, many of us are now wondering why event cameras seem to have joined <a href="http://www.spinaltapfan.com/">Spinal Tap</a> in the “<a href="https://memes.getyarn.io/yarn-clip/5a6e923e-55af-4dc2-b182-a9f40c97d5bf">Where Are They Now?</a>” file.</p><h3>Event Cameras: What Are (Were) They?</h3><p>The idea behind event cameras is compelling: a camera that works very much like the human eye. In doing so, it can react quicker, under more circumstances, while transmitting much less data. But before we can understand how an event camera works, let’s first review traditional cameras.</p><p>The fundamental manner in which most cameras work hasn’t changed much since the very first <a href="https://en.wikipedia.org/wiki/Daguerreotype">daguerreotype</a> images were captured in the mid 19th century. A shutter would open, and light would hit a substrate that would then capture an image. In the 19th century, that substrate was silver-plated copper. In 2020, that substrate is a light-sensitive chip.</p><p>In a traditional camera, the shutter opens intermittently to let light in, which the substrate then captures as an image. In the context of video cameras, this is done multiple times per second to capture motion. The speed with which a motion camera can open and close its shutter becomes a limiting (or, if you’re a cup half full kind of person, enabling) factor that determines camera performance.</p><p>Once an image is captured on the substrate, something must be done to it. From 1840 to around 1960, that “something” was typically using a noxious bath of chemicals on the substrate to develop the images that subsequently become photographic prints and movies. Starting in the mid 1960s, digital recording formats allowed the substrate to record in ones and zeroes. Enter the second limiting (or enabling) factor: data transmission and storage. The earliest 2-inch magnetic video tape from the 1960s required quite a lot of space to store an hour’s worth of color video:</p><figure><img alt="2-inch video tape reel" src="https://cdn-images-1.medium.com/max/1024/1*2uzj6iMHcoGy59o7q7Z_wQ.jpeg" /><figcaption>An hour’s worth of two-inch videotape on a reel. One-hour DVC tape on top for comparison. CC BY-SA 3.0, Image attribution: <a href="https://commons.wikimedia.org/w/index.php?curid=880661">https://commons.wikimedia.org/w/index.php?curid=880661</a></figcaption></figure><p>Today, we’re not as concerned with local storage as we are with cloud data transfer and remote storage. True, compression has allowed us to lower the amount of data that a typical camera transmits. On the flip side, this has led system designers to increase the number of cameras, increase their data quality, and increase their usage. Back to square one with data constraints!</p><p>So, to recap, typical cameras:</p><ul><li>Open a shutter to record images onto a substrate</li><li>Are speed limited by how frequently that shutter can open or close to record motion</li><li>Are data limited by how much bandwidth they are able to use</li><li><em>Bonus points: </em>there are many other factors that enhance or limit a camera’s performance, such as its dynamic range.</li></ul><p><strong>Now how do event cameras compare?</strong></p><p>To start with, event cameras have no shutter. But, like a CMOS camera, they do have a chip with hundreds of thousands of tiny, light-sensitive pixels. Each one of those pixels is programmed to record data under a simple set of conditions:</p><ul><li>Did the amount of light received go up?</li><li>Did the amount of light received go down?</li><li>Did the amount of light received stay the same?</li></ul><p>If either of the first two conditions occurs, that pixel transmits the new value. If the third condition occurs, it sends no new value.</p><p>Each pixel in an event camera operates independently. That means that event cameras have <strong>microsecond responsiveness</strong>. That is one millionth of a second. <em>One millionth</em>. That 240fps camera in your iPhone doesn’t seem so fast now, huh?</p><p>On top of that laser-fast speed, the data that an event camera sends is, by default, compressed. Any pixel that doesn’t register a change sends no value, just like a typical image compression algorithm. But, better yet, those pixels that do send data can send a minuscule packet that barely even registers.</p><p>So, to recap, event cameras:</p><ul><li>Have no shutter and are always exposed to light</li><li>Are microsecond responsive</li><li>Transfer negligible amounts of data continuously</li><li><em>Bonus points:</em> the manner in which event cameras work make them suitable in conditions that typically require high dynamic range cameras, such as rapid motion and low light.</li></ul><h3>So Why Aren’t There Event Cameras Everywhere?</h3><p>Great question. Where are they? Well, they still exist. But it appears that those companies who may choose to implement them in a mass-produced product are still in “wait and see” mode.</p><p>As a result, there are very few companies still actively producing event cameras. Our current contenders are:</p><p><a href="https://inivation.com/">iniVation</a></p><p><a href="https://www.celepixel.com/#/Home">CelePixel</a></p><p><a href="https://www.prophesee.ai/">Prophesee</a></p><p>There has also been activity by two of the larger consumer electronics companies, Samsung and Sony. Samsung is the only company to date that has tried to mass market an event-camera-powered product to consumers, with the <a href="https://www.samsung.com/au/smartthings/camera/smartthings-vision-gp-u999gteeaac/">SmartThings Vision</a> camera.</p><figure><img alt="Samsung SmartThings Vision Event Camera" src="https://cdn-images-1.medium.com/max/289/1*NZTerUQ1_9GOeegCVNh2iA.jpeg" /><figcaption>Samsung SmartThings Vision Event Camera. Image attribution: Samsung Electronics Australia Pty Ltd</figcaption></figure><p>The SmartThings Vision camera was marketed briefly by Samsung Australia as a home surveillance camera that offered greater levels of privacy. It was pulled from the market after just a short time on sale.</p><figure><img alt="Samsung SmartThings Vision Event Camera" src="https://cdn-images-1.medium.com/max/720/1*91sKszAfJTscmlL48avpBg.jpeg" /><figcaption>Samsung SmartThings Vision Event Camera. Image attribution: Samsung Electronics Australia Pty Ltd</figcaption></figure><p>With the first consumer event camera product a failure, event camera companies are turning to industrial use cases to find a market. Robotic vision. Object scanning. Security. So far, there have been few takers who are willing to change their system designs from the current industry-standard CMOS designs to event camera designs. So what gives?</p><h3>Why Event Cameras Have Not Yet Succeeded</h3><h4>Supply Chain</h4><p>We’ll start with the most obtuse factor. If you’d like to manufacture a custom CMOS-based camera, there is an embarrassment of contract manufacturers ready to develop that camera to your precise specifications. Chances are, the specific components you need are already being mass produced to a high specification of quality and performance.</p><p>Want to develop an event camera based system? Good luck funding a manufacturer with the willingness to transfer a profitable CMOS semiconductor assembly line to what they’ll believe to be a brand new, commercially unproven technology. And, if you do, it’ll cost you an arm and a leg (or, in camera terms, an eyeball?) to get them manufactured, and those costs will need to be transferred to your customers.</p><p>In consumer markets, this will make your product price-prohibitive. In industrial markets, it will relegate your product to niche applications, which will suppress demand, thereby keeping costs perpetually high, which continues to suppress demand, which…well, you get the idea. Until a radical breakthrough in demand occurs, this vicious cycle will limit supply chain participants’ interest in producing the components that are unique to event cameras.</p><h4>Price</h4><p>To build on one of the key points from the previous factor, event cameras simply aren’t cheap. The least expensive models that are broadly available cost in the thousands of dollars each. That severely limits applications to those products that can assume such a massive additional cost into an aggregate bill of materials.</p><p>Our focus market of <a href="https://www.tangramvision.com/">robotics</a> already struggles to justify the addition of a single multi-thousand-dollar LiDAR unit, and can only do so because of the sheer utility that these long-range, 360° sensors add to robotic platforms. Event cameras will absolutely outperform even high dynamic range (HDR) CMOS cameras in robotic contexts — but not to the degree required to justify their extra cost.</p><h4>Exoticism</h4><p>Industries like robotics are at the cutting edge of technology. However, the hardware design decisions that robotics companies make tend to be conservative.</p><p>Because of the complexity of robotic platforms, proven and reliable components are of paramount importance in system design. Event cameras are simply too new and exotic to be seriously considered for a robotic platform, for example.</p><p>This is also reflected in a key leading indicator for what imaging technologies will receive the most industry attention in the near future: academic research. Peruse the paper submissions for key computer vision conferences like <a href="http://cvpr2021.thecvf.com/">CVPR</a> or <a href="https://iccp-conference.org/">ICCP</a>, and you’ll see very few focused on event cameras. And a plethora focused on other modalities like CMOS and depth.</p><h4>Competition</h4><p>While few CMOS cameras (even HDRs) can come close to the levels of performance offered by a typical event camera, the performance gap in key areas has been closing.</p><p>Modern CMOS cameras offer ever-faster shutters, higher pixel density and higher dynamic ranges that make them “good enough” for applications where event cameras shine. And they are inexpensive enough that multiple CMOS cameras can be utilized in single purpose applications, collectively overcoming some of the deficiencies that would otherwise occur in a single CMOS deployment.</p><p>So why spend thousands of dollars on a single event camera when you could design a good enough CMOS-based system that costs…$150? $30? Even under $10?</p><h4>Environment</h4><p>Some of event cameras’ inherent advantages have been whittled away by factors that are exogenous to camera design. Consider data usage.</p><p>A decade ago, available bandwidth was scarce in wired scenarios (for instance, an IP camera used for security in a retail store) and even less so for wireless scenarios (a drone with an inspection camera transmitting over a 3G network). Today, data pipes are much larger, data transmission speeds have radically accelerated with LTE and 5G rollout, and data compression algorithms have progressed as well. For robotic platforms that process data locally, hosts have gotten much more powerful, too. Therefore, data efficiency is no longer as great an advantage for event cameras as it once was.</p><p>True, greater amounts of data transmission impact other areas of system performance (for instance, battery life and available compute). However, in most applications, the aggregate data requirements for above average application performance is now aligned with transmission and processing thresholds that preserve battery life and compute resources.</p><h3>Future Events</h3><p>So will we ever see broad-based deployment of event cameras? In our estimation, the answer is somewhere between <em>not for another few years</em> to <em>not at all</em>. Current CMOS technology is simply good enough (and getting better still) to win the argument against justifying the massive investments that would be required for a large shift in markets to event cameras.</p><p>That’s not to say that event cameras can’t find a market at all. They are still incredibly powerful, as evidenced by some incredible demos that show just what they are capable of:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FsbJAi6SXOQw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DsbJAi6SXOQw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FsbJAi6SXOQw%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/315b381ff6f36e28a6feed005c716304/href">https://medium.com/media/315b381ff6f36e28a6feed005c716304/href</a></iframe><p>But will they? Or, like our friends <a href="https://www.youtube.com/watch?v=g7-5io1muSQ">Nigel</a>, <a href="https://www.youtube.com/watch?v=xsz71luPVuE">David</a>, and <a href="https://www.youtube.com/watch?v=JtvGR8UX1L0">Derek</a> from Spinal Tap, will it be <a href="http://www.spinaltapfan.com/atozed/TAP00471.HTM">shark sandwiches</a> for event cameras?</p><p>Here at <a href="https://www.tangramvision.com/">Tangram Vision</a>, we’ll continue to track whether these powerful cameras become adopted into the industries we serve with our <a href="https://www.tangramvision.com/tangram-vision-sdk">SDK</a>(robotics, drones, industrial automation and autonomous vehicles). What do you think? Let us know in the comments below, or <a href="https://twitter.com/TangramVision">tweet at us</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=293343754bfd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tangram-visions/event-cameras-where-are-they-now-293343754bfd">Event Cameras: Where Are They Now?</a> was originally published in <a href="https://medium.com/tangram-visions">Tangram Visions</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Calibration Matters for Artificial Intelligence and Machine Learning]]></title>
            <link>https://medium.com/tangram-visions/how-calibration-matters-for-artificial-intelligence-and-machine-learning-621ebc5a8a18?source=rss-e03ec3123736------2</link>
            <guid isPermaLink="false">https://medium.com/p/621ebc5a8a18</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[autonomous-cars]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Adam Rodnitzky]]></dc:creator>
            <pubDate>Thu, 17 Dec 2020 23:12:02 GMT</pubDate>
            <atom:updated>2020-12-17T23:12:02.936Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Evolution of Image-Based Inputs for Machine Learning</h3><p>In the beginning, there were pictures. Simple, static pictures of all sorts of scenes and objects: Dogs. Cars. Houses. People. Fed into systems like <a href="https://deepdreamgenerator.com/">Deep Dream</a>, they trained classifiers which then generated their own hallucinated outputs from those inputs. Some of the results were predictably hilarious. But some were also incredibly realistic, and hard to discern from the real-world inputs from which they were derived.</p><figure><img alt="Deep Dream Hallucination" src="https://cdn-images-1.medium.com/max/1024/1*YT_-_NP_TNRLvCmCaynpJA.jpeg" /><figcaption>Photo: Public Domain / via Wikipedia</figcaption></figure><p>Next, came pre-recorded video. In this phase of image-based machine learning (hereafter referred to as “ML”), these video streams were often used to train systems to separate objects from environments, or to discern different environmental states based on multiple observed factors. These formed some of the first data sets to be used to train machine learning systems used for autonomous vehicles (aka, an “AV”). But, and this is important to note, they were not used in real-time during an AV’s operation.</p><p>Now, however, the applications that rely on imagery-trained ML systems have significantly leveled up. These new applications are low-latency and they <em>are</em> real-time. They span a range of industries and use cases, from <a href="https://www.tesla.com/autopilotAI">self driving cars</a> to <a href="https://plusonerobotics.com/">supply chain robots</a>.</p><h3>Real-Time Image-Based Machine Learning Relies on Reliable Sensing Inputs</h3><p>The real-time nature of these applications presents a critical challenge that must be properly managed by the sensor systems that feed them: the data that they receive must be accurate, as it is processed and analyzed in real-time to determine the immediate and subsequent actions of the application.</p><p>Sensors, as we know, can fail in multiple ways during deployment:</p><ul><li>Intermittent communication faults (for instance, a flaky USB connection)</li><li>External impediments (for instance, road dust occluding a camera lens)</li><li>Unnoticed or inconsistent shifts in sensor calibration (for instance, a sensor with an insecure mounting)</li><li>Improperly configured sensors (for instance, a camera that does not operate properly in typical daylight settings)</li></ul><p>Worse yet, sensors often fail silently. They still emit a data stream, and the ML system assumes that that means the sensors are functioning 100% properly. However, if any of the above scenarios are occurring, they will be sending anomalous data into the ML system. This can create corrupted data sets that lead to immediate errors, or compounding errors that increase over time.</p><h3>The Impact of Silent Sensor Failures on ML Systems: An Example</h3><p>Consider the below scenario:</p><ul><li><strong>The System:</strong> A new autonomous taxi designed to operate in urban settings</li><li><strong>The Task:</strong> Recognize when the autonomous taxi is too close to a curb while in motion</li><li><strong>A Key Parameter:</strong> Distances under 1.2 m present an unsafe condition due to the presence of bike lanes and cyclists in urban settings</li><li><strong>The Inputs:</strong> GPS data; wheel encoders to measure distance traveled; LiDAR sensors and HDR CMOS sensors to capture scene geometry</li><li><strong>The Problem:</strong> Silent LiDAR sensor calibration fault due to a loosened sensor mount; extrinsic calibration shifts by 3 cm in two axes when AV is in forward motion; returns to near-calibrated state at rest. AV driven during rain results in water occlusion on CMOS sensors.</li><li><strong>The Result:</strong> Undetected anomalous data that becomes part of the training set causes the AV to operate within 0.7 m of the curb, which is 0.5 m too close.</li></ul><h3>Ensuring Reliable, Accurate Sensor Inputs for ML</h3><p>Some of the current thinking around sensor failures in machine learning systems suggests that the errors themselves are a valid input for training. This is true, to a degree. The challenge, however, is that the system needs to be able to recognize that a sensor is failing and properly classify that input as such. This is easier to do under controlled circumstances. In the real world, relying on these processes can become a riskier proposition.</p><p>Therefore, the best practice remains ensuring that sensors operate accurately and reliably as much as possible. Achieving this requires effort. It involves:</p><ul><li>Capturing and analyzing statistically significant subsets of sensor logs to identify and properly classify common failure states</li><li>Visually assessing sensors for external impediments after sessions when failures are detected</li><li>Confirming sensor calibration and recalibrating sensors frequently</li><li>Testing sensors with multiple configurations, and then capturing and analyzing sensor logs to identify operational deficiencies before determining the optimal configuration</li></ul><p>This is not an insignificant amount of work. It requires data analysis, sensor expertise, calibration expertise and copious amounts of engineering time. But, as is the case with any data-driven system, it’s a matter of garbage in, garbage out. The investment in proper sensor operation pays off when data sets deliver the expected outcomes, and the ML-powered agent can operate successfully in the real world.</p><p>Of course, we can help ease this burden. <a href="https://www.tangramvision.com/">Tangram Vision</a> automates sensor management, ensuring that otherwise silent failures are detected and handled efficiently, and uptime with optimal performance is maximized.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=621ebc5a8a18" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tangram-visions/how-calibration-matters-for-artificial-intelligence-and-machine-learning-621ebc5a8a18">How Calibration Matters for Artificial Intelligence and Machine Learning</a> was originally published in <a href="https://medium.com/tangram-visions">Tangram Visions</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[On Sensoriums & Self-Organizing Networks]]></title>
            <link>https://medium.com/tangram-visions/on-sensoriums-self-organizing-networks-c08d9c97f060?source=rss-e03ec3123736------2</link>
            <guid isPermaLink="false">https://medium.com/p/c08d9c97f060</guid>
            <category><![CDATA[sensors]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[sensorium]]></category>
            <dc:creator><![CDATA[Adam Rodnitzky]]></dc:creator>
            <pubDate>Wed, 28 Oct 2020 19:38:59 GMT</pubDate>
            <atom:updated>2020-10-28T20:18:06.428Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0NOue_j8ffzOfPAp5rVhzA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@joshriemer?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Josh Riemer</a> on <a href="https://unsplash.com/s/photos/brain?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>One of the perks of working with cutting-edge technology is that it tends to open doors for you that you alone couldn’t open. Over the course of my career, I’ve been able to meet some pretty fascinating people because of the products I worked with. Like <a href="https://en.wikipedia.org/wiki/Tom_Wheeler">Tom Wheeler</a>, the former chairman of the FCC, who dropped by for a demo of the <a href="https://structure.io/">Structure Sensor</a> at CES. Or <a href="https://en.wikipedia.org/wiki/Family_of_Barack_Obama#Malia_Obama_and_Sasha_Obama">Malia Obama</a>, who I had the chance to speak with while she was an intern on the set of <a href="https://www.cbs.com/shows/extant/">Extant</a>. And <a href="https://en.wikipedia.org/wiki/Tim_O%27Reilly">Tim O’Reilly</a>, the legendary technology conference and publishing luminary, who provided an on-the-spot testimonial for the Structure Sensor as part of our preparation for our <a href="https://www.kickstarter.com/projects/occipital/structure-sensor-capture-the-world-in-3d">Kickstarter campaign</a>.</p><p>It was Tim who first introduced me to the concept of a <strong>sensorium</strong>.</p><h3>So What is a Sensorium?</h3><p>The concept of a <a href="https://en.wikipedia.org/wiki/Sensorium">sensorium</a> begins with biology. Most sentient creatures have more than one sense that they rely on to understand their environment. And these senses typically do not operate independently, but rather in concert with each other to inform the organism that they serve.</p><p>But how does the organism interpret the signals it receives from different senses? How does it prioritize one over others, or combine sensory inputs to deduce information? In what manner does its learned behaviors influence which senses it pays more attention to? And in what way does its social environment impact how it uses its senses? The answer to these questions lies within that organism’s sensorium.</p><p>Summed up, the sensorium is the manner in which an organism interprets and prioritizes what it senses. It is influenced by past behaviors and its social environment in ways that elevate some senses while deprecating others.</p><p>In the context of 3D sensing, Tim O’Reilly suggested that the broader deployment of this sensing modality into computing platforms could fundamentally change the sensorium of sensor-equipped devices.</p><p>Nowhere is this more apparent, now, than in the world of robotics.</p><h3>Robotic Sensoriums</h3><p>Robots and their capabilities have evolved in lockstep with the sensory inputs available to their designers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JSFybJd2cQtjGyGJPWYSkw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@photoshobby?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Photos Hobby</a> on <a href="https://unsplash.com/s/photos/robot-arm?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Initially, industrial robotic arms had almost no sensors, and operated according to pre-programmed routines. In the next evolution of robotics, devices like <a href="https://fetchrobotics.com/fetch-robotics-blog/amrs-vs-agvs-whats-the-difference/">automated guided vehicles</a> (AGVs) gained basic obstacle sensors to ensure they didn’t run over unmarked items (or people!) as they performed their pre-programmed routines. The arrival of more sophisticated perception sensors and feedback systems has allowed for the creation of autonomous mobile robots (AMRs) and collaborative robots (or <a href="https://en.wikipedia.org/wiki/Cobot">cobots</a>) that can work in close proximity to people, with significantly reduced risk of danger to their human counterparts.</p><p>A modern AMR might have a dozen or more sensors onboard. And, like any other multi-sensory organism, it must employ its sensorium to optimize how its uses those sensory inputs to operate. Until recently, that sensorium was effectively programmed by a software engineer, using algorithms to guide the decision making process within the robot. Emerging deep learning techniques are now letting the robot itself evolve its sensorium, much in the same way a biological organism might also evolve its behavior based on sensory inputs and feedback.</p><h3>Sensoriums and Self Organizing Networks</h3><p>Everyday consumers often seamlessly accept technology into their lives without realizing how fully a particular approach has become embedded in their world.</p><p>Take computer vision, for instance. Consumers are blissfully ignorant of how and why Facebook is able to automatically tag them in their friends’ photos, and their ability to drive through a toll booth with no need to stop, and how they can now unlock their iPhone by just looking at it.</p><p>In the same way, self organizing networks are blossoming as a technology input that is positively impacting our day to day lives. At home, the clearest example of this are mesh networked WiFi systems, which can optimize delivery of bandwidth to a diverse set of devices in a diverse set of locations. In industry, the concept of self organizing networks is finding a home with robots, and specifically around how they share sensory data.</p><p>Consider this scenario: a logistics facility deploys 10 AMRs to perform a range of tasks, such as <a href="https://www.adaptalift.com.au/blog/2011-12-23-what-is-cross-docking-understanding-the-concept-definition#:~:text=Cross%20docking%20is%20a%20logistics,no%20handling%20or%20storage%20time.">cross-docking</a>, delivering remote goods to packing stations, and waste removal.</p><p>To set up those AMRs, an initial map of the facility is created, and downloaded to each robot. However, no facility is static. And the map itself can’t show the presence of moving and movable objects, like workers, equipment and inventory. Thus, each AMR must also rely on its active senses to safely and effectively navigate while it does its job.</p><p>But what if those 10 AMRs could cooperate and share sensory data with each other? They would then have greatly enhanced sensing capabilities that would boost safety and efficiency.</p><p>This self organizing behavior is effectively a result of the robots’ social environment. And each robot, then, utilizes its live senses, its learned data, and the sensing and learning from the other robots in the network to interpret the world around it. In other words, these robots have a sophisticated sensorium to work with.</p><h3>Making More Sense for Robots</h3><p>The more sensing and data robots can receive, the more complete their sensoriums. And, as Tim O’Reilly told us back in 2013, the addition of new sensing modalities to a sensorium makes new capabilities possible.</p><p>Our goal at Tangram is to make it simple to add more sensors (and multiple sensors) to any vision-enabled platform, whether it be a robot, a drone or a new vison-enabled device the world has yet to experience. If you’d like to enhance your own robot’s sensorium, <a href="https://www.tangramvision.com/contact">get in touch</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c08d9c97f060" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tangram-visions/on-sensoriums-self-organizing-networks-c08d9c97f060">On Sensoriums &amp; Self-Organizing Networks</a> was originally published in <a href="https://medium.com/tangram-visions">Tangram Visions</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Future Is Robots…As Long As We Solve These Problems]]></title>
            <link>https://medium.com/tangram-visions/the-future-is-robots-as-long-as-we-solve-these-problems-c980d7c7008f?source=rss-e03ec3123736------2</link>
            <guid isPermaLink="false">https://medium.com/p/c980d7c7008f</guid>
            <category><![CDATA[sensors]]></category>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[perception]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[Adam Rodnitzky]]></dc:creator>
            <pubDate>Fri, 09 Oct 2020 21:14:04 GMT</pubDate>
            <atom:updated>2020-10-09T21:14:04.356Z</atom:updated>
            <content:encoded><![CDATA[<p>For as long as Hollywood has been making films about the future, the popular imagination has been captured by visions of a world served by robots.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/955/1*zIVXtBAlYm43LB7OI02tKg.jpeg" /></figure><p>Yet, even though it has been almost a century since Fritz Lang’s landmark film <a href="https://en.wikipedia.org/wiki/Metropolis_(1927_film)">Metropolis</a>, we still don’t live in a world where robots and humans regularly occupy the same spaces, whether public or private. Perhaps the only exception might be my adopted hometown of San Francisco, where multiple fleets of <a href="https://www.theverge.com/2020/4/29/21241122/cruise-self-driving-car-deliveries-food-banks-sf">self-driving cars</a> roam the streets, <a href="https://www.theverge.com/2019/8/8/20791926/postmates-robots-sidewalk-delivery-san-francisco-permit">delivery robots</a> trundle down sidewalks delivering sushi to software engineers, and <a href="https://www.doublerobotics.com/">telepresence robots</a> let remote workers (aka, <em>everyone</em> now) interact with others in the home office.</p><p>So, why then, is the case?</p><h3>Robots Still Aren’t Reliable</h3><p>It’s true. Even though brilliant minds from the world’s best universities are tasked with building these robotic platforms, a robot is a non-trivial device to engineer. The complexity of a robot that appears simple (like a delivery robot, for instance) is, in fact, staggering. There are multiple subsystems, from control to locomotion to perception to more esoteric areas like power management and compute resource allocation. All of them need to work all of the time. All of them need to work all of the time <em>at the same time</em>. And if one system fails, <strong>the whole robot fails</strong>. This actually happens a lot.</p><p>Some of these systems have become hardened through the collective efforts of thousands of roboticists and robotics startups over the past couple of decades. But, for the most part, the real world is a harsh place, and it does its best to flummox a robot however it can.</p><p>Take our particular area of robotics, perception sensors. In terms of performance and reliability, the <a href="https://structure.io/structure-core">sensors available to roboticists today</a> are leagues ahead of what was available just a few years ago. Yet these sensors can still fail for myriad reasons. Passing a poorly shielded electrical box on a delivery route can introduce EMF surges that suddenly compromise communications. A toddler might want to give a cute robot a hug, knocking a sensor off calibration in the process. Or a sudden burst of direct sunlight will exceed the dynamic range of a vision sensor, rendering it inoperable until the sun has continued its path from dawn til dusk. Or perhaps the sensors work perfectly fine, but the perception software can’t discern a flat floor from a staircase, which then gives you this:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F0T9v9FUE7ms%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D0T9v9FUE7ms&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F0T9v9FUE7ms%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/b17bedc2cc6139f3e8094eb6df3308fe/href">https://medium.com/media/b17bedc2cc6139f3e8094eb6df3308fe/href</a></iframe><p>For each failure mode, there is a solution. However, these failure modes typically don’t occur in the lab, which means that these real world failures send robotic teams back to the drawing board for what they assumed to be solved problems (or they can just use <a href="https://www.tangramvision.com/">Tangram Vision</a> to manage sensor performance during deployment).</p><h3>Development Timelines Kill Robotic Companies</h3><p>Even with multiple rounds of venture capital and incredible development teams, we’ve seen cutting edge robotic companies die again and again because development timelines exceeded their financial runway.</p><p>As we noted above, robots can appear to be robust and reliable in laboratory environments, only to fall to pieces when confronted with real world conditions. And given the inherent complexity of robots, even getting to a completed prototype to test in a lab setting can be a monumental task.</p><p>As a result, many promising companies have had to shut down because they failed to raise enough capital to account for delays that occur during development and pilot phases.</p><p>Of course, this is one of the reasons we’re building a <a href="https://www.tangramvision.com/product/integrate">common integration layer for perception sensors</a>. As we’ve spoken with dozens of robotics companies, we now understand that perception sensor integration can add anywhere from six to 18 months of unplanned time to development timelines. Six to 18 months! Delays like that kill companies, no matter what the industry. It’s as simple as that.</p><h3>People Don’t Trust Robots (Even Though They Should)</h3><p>Videos like those showing Boston Dynamics robots performing amazing, superhuman-like feats, often serve to reinforce narratives around the dangers of releasing robots into the world. After all, what mere mortal could compete against the capabilities of robots like these, whether in work or in warfare?</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FuhND7Mvp3f4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DuhND7Mvp3f4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FuhND7Mvp3f4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a562b709191be86b4febd5e97da8b58f/href">https://medium.com/media/a562b709191be86b4febd5e97da8b58f/href</a></iframe><p>For the uninformed, the idea of robot proliferation initially conjures these thoughts of job destruction. That couldn’t be further from the truth! In fact, study after study has shown that the onset of robotics and automation isn’t a job killer, but in fact is a job creator.</p><p>The most thorough study we’ve come across was done by the Brookings Institute. It concedes that robots and automation does destroy some jobs — but it then overcompensates by creating even more jobs than existed before. <a href="https://www.brookings.edu/blog/up-front/2019/03/18/robots-kill-jobs-but-they-create-jobs-too//">You can learn more here by reading their report.</a></p><p>Yet these studies aren’t enough. This message also needs to reach and be understood by those outside of the robotics industry. As an industry, robotics and automation still does a poor job of making the layperson understand the real impact of robots in terms of economic output and job creation.</p><p>A goal of Tangram is to help society at large better understand these benefits, and in turn, accept robotics and automation as a positive force for the world.</p><h3>Governments Don’t Understand How To Optimally Legislate Around Robots and Automation</h3><p>This may not come as a surprise. But this challenge is compounded by the fact that the newest wave of mobile robots and autonomous vehicles are operating in public spaces, and those public spaces are legislated by overlapping mandates from local, state and federal governments.</p><p>As a result, progress in one jurisdiction does not necessarily set a precedent for another. In the reverse case, statewide or federal mandates can preclude more progressive legislation at a local level. <a href="https://cal.streetsblog.org/2020/05/04/california-readying-rules-for-automated-vehicle-ride-hailing/">The situation for autonomous vehicle fleet deployment in California</a>, for instance, demonstrates the complexity of managing local expectations with statewide mandates.</p><p>As an industry, robotics and autonomy companies need to work with elected officials from the local through national level to create sensible legislation that ensures public safety while simultaneously allowing for innovation that improves quality of life and economic output.</p><h3>We Need Robot Insurance</h3><p>OK, truth time. I wrote this entire Medium post because this is my favorite Saturday Night Live commercial spoof of all time:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fg4Gh_IcK8UM%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dg4Gh_IcK8UM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fg4Gh_IcK8UM%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/950ce7f1ddbf251138d765e550686d68/href">https://medium.com/media/950ce7f1ddbf251138d765e550686d68/href</a></iframe><p>In all seriousness, the existence of this sketch is in fact evidence of society’s skepticism of robots and their safety…really! The more positive interactions that people have with robots, the sooner they’ll be accepted as productive and positive.</p><h3>In Conclusion</h3><p>Robotics and automation are inevitable; all of the above challenges (even robot insurance!) will become solved problems in due time. The question is…when?</p><p>At <a href="http://www.tangramvision.com/">Tangram Vision</a>, we’re doing our part to <a href="https://www.tangramvision.com/product/integrate">shorten robot development timelines</a> and <a href="https://www.tangramvision.com/product/operate">increase robot reliability</a> through more sound sensor deployment. And we’ll add our voice to our industry’s to support sensible legislation that benefits all.</p><p>Are you developing a robot or autonomous platform that uses perception sensors? <a href="https://www.tangramvision.com/contact">Get in touch so we can see how we can help.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c980d7c7008f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tangram-visions/the-future-is-robots-as-long-as-we-solve-these-problems-c980d7c7008f">The Future Is Robots…As Long As We Solve These Problems</a> was originally published in <a href="https://medium.com/tangram-visions">Tangram Visions</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Computer Vision For All]]></title>
            <link>https://medium.com/tangram-visions/computer-vision-for-all-7518b3104401?source=rss-e03ec3123736------2</link>
            <guid isPermaLink="false">https://medium.com/p/7518b3104401</guid>
            <category><![CDATA[machine-vision]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[sensors]]></category>
            <dc:creator><![CDATA[Adam Rodnitzky]]></dc:creator>
            <pubDate>Wed, 16 Sep 2020 20:40:20 GMT</pubDate>
            <atom:updated>2020-09-16T20:48:06.649Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*Wnp_tqwN7Nsd_XrIRGZiWw.png" /></figure><p>A decade ago, as I was starting <a href="https://www.crunchbase.com/organization/retel-technologies">my last computer vision company</a>, one of our investors announced that they had funded another deep tech company. Except this deep tech company seemed to be focused on decidedly old-fashioned tech compared to the futuristic vision-based system we were working on.</p><p>At the time (and this was at the beginning of the 2010s), we had just gone through a transformation from Web 1.0 to Web 2.0, where Internet-powered businesses had gone from static to dynamic functionality. The Web was no longer just a mechanism to distribute information; it now had a service layer being built where it could be used to address and control devices and take previously offline processes online.</p><p>With that foundation in place, a set of entrepreneurs set out to harness these capabilities to expand what was possible with the Internet. One of these entrepreneurs was a former Amazon engineer who happened to know quite a bit about old-school PSTN technology. If you’re not familiar with what PSTN is, it stands for <a href="https://en.wikipedia.org/wiki/Public_switched_telephone_network">Public Switched Telephone Network</a>, aka, the world’s already established telephony network.</p><p>PSTN was its own arcane technology backwater, and there were very few engineers outside of those who worked at the traditional telecoms who knew how to work with it.</p><p>So when <a href="https://twitter.com/jeffiel?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor">Jeff Lawson</a>, that former Amazon engineer, realized that he could combine the power of the new Internet with the capabilities of PSTN, a new company was created: <a href="https://www.twilio.com/">Twilio</a>.</p><p>The brilliance of Twilio wasn’t simply that it connected PSTN to the Internet; it was that it lowered the bar for working with PSTN from telecom industry insider to any software or web engineer. Now, anyone who understood how to build with web services could rapidly add PSTN-powered functionality to the product that they were working on. That enabled the creation of tools we now take for granted, like text-message-based two-factor-authentication, and “call me now” customer support services.</p><p>The ethos behind Twilio — democratizing access to an otherwise challenging or arcance technology — brings us to my new company, <a href="https://www.tangramvision.com/">Tangram Vision</a>. Over my past decade of working in computer vision, I’ve witnessed significant progress in the development of tools that are designed to make it simpler than ever to build a vision-powered product or service. Things like:</p><ul><li>Lower cost <a href="https://structure.io/">perception sensors</a> that make it cheaper and simpler to prototype vision-powered products</li><li>Open-source libraries like <a href="https://opencv.org/">OpenCV</a> and <a href="https://www.ros.org/">ROS</a> that provide many of the foundational libraries for building vision-powered products</li><li>Powerful algorithmic tools like <a href="https://www.tensorflow.org/">TensorFlow</a> and <a href="https://openslam-org.github.io/">OpenSLAM</a> that provide building blocks for higher-level product functionality</li></ul><p>Yet, despite these advancements, building a vision-enabled product is still complicated, and requires specialized understanding of computer vision techniques and engineering approaches. That’s because computer vision has not yet had its Twilio moment. And that is what we intend to change with Tangram Vision.</p><p>Our focus at Tangram Vision is to make those sensors and algorithms simple to integrate, and painless to maintain over time. We aim to let any software or web engineer build a computer vision powered product. And, for those who already understand computer vision, we aim to relieve them from the duty of building infrastructure, so they can focus on those higher level tasks that make their products unique.</p><p>The <a href="https://www.tangramvision.com/">Tangram Vision Platform</a> is currently in development; we’ll be launching our private beta in the next few months. Interested at getting a first look at the platform? <a href="https://share.hsforms.com/1EHGAYO-WS4ew6GWVAkK4DQ4q1r0">Sign up for our notification list today</a>, and we’ll let you know as soon as its ready for release.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7518b3104401" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tangram-visions/computer-vision-for-all-7518b3104401">Computer Vision For All</a> was originally published in <a href="https://medium.com/tangram-visions">Tangram Visions</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>