Until recently, the high-quality experiences that modern AR and VR systems enable at home were only possible with expensive, custom solutions in research labs or commercial setups. The new wave of AR and VR began when Oculus and HTC released their Oculus Rift and HTC Vive products in 2016. Suddenly, technology that was once extremely expensive became more broadly accessible, with some users able to use these systems in their own homes. Still, these tethered systems rely on high-end computers and the use of “outside-in tracking,” meaning that stationary devices must be installed in the environment in order to build the tracking infrastructure for measuring user position and orientation.

Since then, newer systems such as Microsoft Mixed Reality, Oculus Go, and HTC Vive Focus have progressed to inside-out tracking. These newer systems do not rely on prepared environments, which significantly reduces the burden of setting the system up. Similarly, on the AR side, systems like DAQRI Smart Glasses® and Microsoft HoloLens leverage inside-out tracking to enable true mobility. All these systems use some form of visual-inertial fusion to achieve a level of latency that remains relatively unnoticeable to humans.

In this article, we’ll take a deep dive into the technologies used by these AR and VR systems and see how the three key components (inertial measurement units (IMUs), prediction, and late correction) are combined into pipelines with end-to-end latencies of just a few milliseconds. Due to the significantly higher challenges mobile AR systems must overcome in comparison to stationary VR systems, we’ll focus exclusively on mobile AR systems.

LATENCY

Latency, or “motion to photon latency,” is a critical property of all AR and VR systems. It describes the length of time between the user performing a motion (e.g., turning the head to the left) and the display showing the appropriate content for that particular motion (e.g., the content on the head mounted display (HMD) subsequently moving to the right). The term “photon” indicates that all parts of the display system are involved in this process, up until the point when photons are emitted from the display.

Humans are highly perceptive to latency. In VR, perception of misalignment originates from both muscle motion and the vestibular system (within the ear), as the user only sees virtual content. In these systems, up to 20 milliseconds of lag can remain undetected by the user. In AR, optical see-through (OST) systems are commonly used to overlay virtual content onto the real-world. Since there is no lag in the real world, latency in such AR systems becomes visually apparent in the form of misregistration between virtual content and the real world. Since the human visual system is extremely sensitive to lag, 5 milliseconds of latency or less is required in OST systems in order to remain unnoticed by the user.

Achieving such low latencies is more complicated than one might think, especially since each step taken by the system contributes to the overall latency. At a high level, the pipeline of measuring a motion to showing updated content on the display consists of the following main blocks:

• Sensors (camera and IMU)
• Tracking (motion estimation)
• Rendering (image generation)
• Display (image presentation)

Let’s assume these four main blocks apply to a well-designed and implemented AR/VR system, where external factors such as copying or locking buffers and queueing at the operating system level do not add unnecessary latency.

After investigating each block, we will see that a naïve implementation of this pipeline can easily result in over 100ms of latency. After separately evaluating the latency of each block, we’ll discuss how the end-to-end latency can be decreased to meet the 5ms requirement of AR systems.

SENSORS

Most mobile AR and VR devices today use a combination of cameras and inertial measurement units (IMUs) for motion estimation (inside-out pose tracking). We will see that the IMU is important for both robust pose (position and orientation of the device) estimation and low end-to-end latency.

AR and VR systems typically run tracking cameras at 30Hz, which means that an image can be read out and passed on for processing every 33ms. Further, a frame rate of 30Hz puts an upper limit of 33ms exposure time per frame.

Let’s assume the exposure is fixed at 20ms. For later processing, we must first assign a timestamp to each sensor measurement. For the actual exposure, the start and end times are defining factors. From a processing point of view, however, it is more convenient to use the point in time right in the middle of exposure. Therefore, when exposure ends, the age of the image is equal to half the exposure time. Thus, in our example, the image is already 10ms old. Once exposure completes, the image must be read out and forwarded to the image processing system. For our system, this takes another 5ms, making our image 15ms old at that time, even if the tracker starts processing the image immediately.

While cameras typically run at 30–60Hz, IMUs operate at a much higher rate of hundreds or even thousands of Hz. For our system, let’s assume the IMU runs at 1000 Hz and consequently generates a new sample every millisecond. IMU data is very compact (typically a few dozen bytes) and has very low latency, so it’s reasonable to assume that the latest IMU samples are approximately 1ms old at any given time.

TRACKING

Tracking consumes the sensor data to calculate a 6DOF motion estimation (pose), which describes how the device has moved and rotated. As mentioned above, the camera image that arrives at the tracking system is at least 15ms old. Alternatively, the IMU has much lower latency, with 1ms old IMU data readily available. However, IMU data that is newer than the camera frame is usually not processed at this stage.

Since our example assumes the camera runs at 30Hz, the tracking system should operate at the same rate, providing a processing budget of up to 33ms. While it is possible to execute the processing of sequential camera frames in parallel to lend more processing time to each frame, this approach results in additional latency and therefore will not be used here. Instead, we’ll assume that the tracking system takes 25ms to finish processing the camera image plus IMU data.

Since camera frame processing can only begin once the frame becomes available to the tracker, the duration of sensor data acquisition and tracking are added together, resulting in 15ms + 25ms = 40ms of latency at the time we have a pose estimate. This pose is then passed on to the application for rendering.

RENDERING

Rendering is the step that generates a 2D image that is then sent to the display in the form of a frame buffer. To generate this image, the renderer requires several inputs, such as the 3D content that shall be displayed and the pose (view port) it is rendered from. Rendering operates at a different rate from pose estimation (typically at 60, 75 or 90Hz). As a result, it is fairly common to perform rendering asynchronously from tracking. With this in mind, let’s assume rendering operates at 60Hz.

When a new frame is generated, we can assume the renderer uses the latest camera pose estimated by the tracker. Since the tracker runs at 30Hz, this pose can be up to 33ms old. At 60Hz, the renderer produces a new frame approximately every 16.7ms. Only at the end of that 16.7ms period does the process of transmitting the frame to the display begin. This means it takes 16.7ms from the start of rendering until the frame buffer read-out starts.

In summation, the rendering block adds another 17–50ms in addition to the 40ms from previous blocks, putting us at 57–90ms of latency, far greater than the 5ms requirement.

DISPLAY

Most computer scientists don’t spend much time thinking about what happens to the frame buffer generated by the GPU once rendering has finished. For AR and VR, however, the way the display works is critical, as it adds a significant amount of additional highly-visible latency that must be addressed. As a result, understanding how displays work is fundamental to achieving low latency.

When frame buffer read-out (also known as “scanout”) begins, data is sent pixel-by-pixel and line-by-line to the display. The specific display protocol (HDMI, DisplayPort, MIPI-DSI) is not important for our discussion, since all of these protocols behave similarly by sending the frame buffer only as quickly as needed for the given frame rate. As a result of running the display at 60Hz, it takes 1/60 of a second to transmit the frame to the display, adding another 17ms of latency.

The AR and VR pipeline we have discussed so far has been fairly generic, but leads us to distinguish between different display types. Let’s discuss line-sequential displays, (e.g., LCD or OLED) and color-sequential displays (e.g.,LCOS).

LINE SEQUENTIAL DISPLAYS

Line-sequential displays are integrated into PC monitors, TVs, and VR headsets, and are the most common display type today. Each pixel in these systems stores the three primary colors (red, green and blue), so a frame’s data looks as follows: RGBRGBRGB…

This pattern indicates that the frames are organized as scanlines, where each scanline contains pixels and each pixel contains red, green and blue components. OLED and LCD displays do not typically store the received pixel data in a temporary buffer, unless they have to perform extra processing, such as rescaling. Instead, as data arrives, the display directly updates the corresponding pixel cells. Therefore, the top-most scanlines of the display (whose data arrives first) are updated around 16ms earlier than the bottom-most scanlines (whose data arrives last) of the same frame.

Similarly, the bottom-most scanline of one frame is updated just before the top-most scanlines of the next frame are updated. Hence, latency of the top-most scanlines is very small (let’s assume 1ms), whereas the latency of the bottom-most scanlines is almost the length of a full frame time (16.7ms). The following figure illustrates the timings of this “rolling” update mechanism of a line sequential display.

COLOR SEQUENTIAL DISPLAYS

Today, LCOS are less common in consumer devices, but they are very common in mobile AR devices, such as DAQRI Smart Glasses and Microsoft HoloLens. LCOS can only show one color at a time. In order to display a full RGB frame, the LCOS display matrix is first updated for the red components of all pixels before the red LED is turned on to actually show the red image. Then, the red LED is turned off and the display matrix is reconfigured for the green components, activating the green LED and displaying the green image. This process is repeated for blue and (optionally) green again.

This behavior has two important consequences. First, data sent to the LCOS needs to be organized in subframes of different colors rather than interleaved RGB frames. GPUs produce interleaved RGB frames by default, and display standards (such as HDMI and DisplayPort) work with interleaved RGB frames as well, so an additional color separation step (usually in the form of a hardware block) is needed before subframes can be passed on to the LCOS. However, since a color separated subframe isn’t available until the complete RGB frame has been received, one frame of latency is added as a consequence of this separation step — before even a single subframe has been displayed.

While HDMI and DisplayPort are slow by standard protocols, LCOS typically utilize fast, proprietary protocols. For example, it is common to take only ~1ms to send a separated subframe to an LCOS. As a result of using subframes, each color is displayed separately, meaning latency varies by color. Typical subframe sequences are red, green, blue, green, with an offset of 4ms each. Overall, at 60Hz it takes ~17ms to send the RGB frame from the GPU to the color separator and around 1ms to send the first subframe from the color separator to the LCOS. Each successive subframe is delayed by an additional offset, which we assume to be 4ms here. So, we get display latencies of 18ms for red, 22ms for green, 26ms for blue and 30ms for an optional second green subframe.

OVERALL LATENCY OF A NAÏVE AR/VR PIPELINE

Now that we’ve covered all four blocks, let’s see how their latencies sum up:

It is clear that neither the overall latency in the OLED case nor the overall latency in LCOS case is sufficient for an AR or VR system. In fact, each block alone takes longer than the required 5ms for AR systems. Furthermore, the latency of the LCOS system is a factor of 15–24x away from those 5ms. How can we fix this?

As evident in the table above, both rendering and display contribute to more than 50% of overall latency. Even more challenging, these variable latencies would create very unpleasant effects, such as jerky tracking and animations.

The three key technologies to reduce latency include IMU, prediction, and late correction. All three come into play in the rendering and display stage. This seems odd, as we have already accumulated 40ms of latency at that point in the process, but we’ll see how this latency can be “undone” several times allowing us to reach our 5ms goal.

OVERCOMING SENSORS AND TRACKING LATENCY

As mentioned above, when the renderer picks up the latest pose from the tracker, that pose can be anywhere between 40ms and 73ms old. Fortunately, a number of different methods can be used to reduce this latency. As we’ll see below, however, some of these solutions are accompanied by unfavorable consequences.

First, we could reduce camera exposure time, which would also have a positive effect on motion blur. However, small, low-cost cameras are typically unable to operate with lower exposure times without introducing severe noise artefacts. As a result, attaining less than 20ms of exposure is typically only possible in very well-lit environments, such as outdoors, disqualifying this as a viable solution.

Second, we could reduce the time it takes to process an image in the tracker by using faster tracking methods or processing units. However, faster methods generally equal less accuracy and/or robustness — two factors we don’t want to compromise. Additionally, faster processing units often require more power, which may or may not be deemed acceptable either. In this case, we can assume that a low-power CPU (e.g., a DSP or an ARM CPU) is used for tracking that does not have the power to run the tracker faster.

Fortunately, the IMU comes to the rescue. In contrast to the camera, the IMU can run at nearly arbitrary sampling rates (compared to camera rates). Here we assume 1000Hz, which results in every sample being 1ms old at maximum. With proper visual inertial fusion, one can use the IMU data to calculate a relative motion path from one time stamp to another by integrating all IMU samples within that time span. Such a relative motion can be applied on top of an absolute pose to calculate a new absolute pose. Therefore, by taking the last pose from the tracker and all IMU samples since that point in time, we can calculate a new pose that is only around 1ms old. Integrating IMU data does have its limits, though, due to noise and accumulated drift. Nonetheless, for our purposes, it is a great solution that results in a roughly 1ms old pose (rather than 40–73ms) for the renderer.

OVERCOMING RENDERING AND DISPLAY LATENCY

Using an up-to-date pose is undoubtedly preferred over an old pose, but it is not sufficient, as the pose will again be outdated by the time the rendered frame appears on the display. Fortunately, at the point in time when rendering begins, we do have a good estimate of how long it will take until the rendered frame appears on the display (when photons are emitted), since the display runs in sync with the GPU. So, rather than using the current pose, we should instead use the pose at that point of time in the future. Though we are unable to foresee the future, we can use models such as constant velocity, constant acceleration, and higher order motion curve fitting to make predictions based upon recent motion.

Motion prediction works well when the time to predict forward is short — such as 20ms — but quickly degrades for longer periods. It is therefore common to post-correct the render output by warping the frame buffer after the render step with an even newer pose prediction. Since this warping is a pure 2D operation, it is much faster than the 3D scene rendering and its speed is independent of scene complexity. As a result, this can be executed in just a few milliseconds on a modern GPU, right before the frame buffer is sent to the display. This late correction (also called “late warping”) has been known in VR research since the 90s, but it was recently re-invented under the name “time warping”. While conceptually simple, late warping requires highly accurate timing and ideally the ability to interrupt the GPU during its current operation (also called preemption). Companies such as Oculus and Valve have been working with GPU manufacturers for years so that all modern GPUs support this feature very well today. The following figure shows how rendering and warping use different forward prediction times.

At this point, we have reached the last step of our pipeline — optimizing the transfer of pixel data from GPU memory to the display. This step is highly dependent on the actual display technology. Let’s discuss the common solutions for OLED and LCOS, the two dominant mobile AR and VR display types.

OLED DISPLAYS

OLEDs are commonly used in consumer VR displays. Compared to LCDs, OLEDs have lower pixel persistence. This leads to reduced latency and motion blur, which is essential for high quality AR and VR experiences. Today, most consumer VR devices, such as from Oculus, HTC, and Sony, use OLED displays as a result. On the AR side, the ODG R9 is a well-known device that uses an OLED display. At a high level, these displays function similarly to normal desktop monitors: pixel data is sent row-by-row from GPU memory to the display, and per default, the display updates the respective pixel cells directly without any further buffering. Transferring the framebuffer is done in “real-time,” meaning it takes the duration of one frame to transmit one framebuffer. Assuming the framebuffer is sent top to bottom, the top rows of the display will therefore be visible on the display almost immediately after the read-out started, whereas the bottom rows will not become visible until almost a full frame later.

This behavior is the same for all the standard protocols, such as DisplayPort and HDMI in the PC domain and MIPI-DSI in the embedded space. On a desktop computer, this isn’t a problem, but in an AR or VR scenario, this behavior has noticeable consequences. Due to the time offset between top and bottom pixel rows, the rendered scene can suffer from shearing or other effects during motion, similar to a rolling shutter camera.

The solution to this problem is to perform the late warping operation more than just once for the whole image. The image is instead segmented vertically into multiple tiles, where each tile is warped separately with a different pose. This staggered warping needs to be synchronized accurately with the framebuffer read-out. Right before the next tile is read-out, that same tile is generated by warping with the latest available pose. Conceptually, this is very similar to raster interrupt programming, which was popular in the 80s on home computers and game consoles.

For example, let’s say the display runs at 60Hz, meaning it takes about 16ms for a full frame to read out. If the framebuffer is split into 16 tiles, then each tile will be read-out and displayed around 1ms after the previous tile. Using this technique, we can reduce latency from GPU memory to photons to just a few milliseconds.

LCOS DISPLAYS

In contrast to OLEDs, there is no standard protocol for LCOS, and there are multiple LCOS manufacturers who provide quite different proprietary protocols. Let’s consider the HIMAX LCOS that is used in DAQRI Smart Glasses and apparently in Microsoft HoloLens as well.

LCOS displays work very differently than OLED displays for multiple reasons. First, their mechanism for updating pixels operates much closer to that of DLP projectors than regular LCD or OLED displays. Further, the LCOS pixel matrix is updated while the LED is off so that the actual updating step isn’t visible. Once the pixel matrix has been fully updated, the LED is turned on, showing the entire new image at once.

Second, while traditional displays show full RGB images, LCOS-based displays usually work color sequential: Only a single-color is shown at a time, typically in the order of red, green, blue, and green again. Between each shown color, the pixel matrix is updated to present the respective components of the pixels. This alternative mode is advantageous in that it prevents rolling updates as OLEDs do. However, it also has its disadvantages, such as color breakup (rainbow effect). Under motion, the time-separated colors lead to spatially separated colors, which is highly noticeable. This effect can be reduced by conducting pose prediction per color, but this subsequently creates some overhead and cannot perfectly reduce the breakup. Fortunately, the color sequential nature means the display operates at a much higher rate. Further, with the aforementioned RGBG sequence, the actual update rate of the display is not 60 full frames per second, but 240 sub frames per second. Since each subframe has only a single-color component, it is much more compact and can be transmitted to the display in less time than a full RGB frame. Let’s discuss how this technique can be used to reach extremely low latencies.

LCOS, such as the one from HIMAX, do not use standard protocols such as HDMI or DisplayPort. One reason behind this is that GPUs produce color sequential frame buffers per default, so some extra processing needs to occur to provide the framebuffer in the color sequential format the LCOS requires. LCOS manufacturers typically offer companion chips that receive framebuffers via common display standards, perform the required color separation step, and then pass the data on using a proprietary protocol. However, before an RGB frame can be color separated, it must be fully received by the companion chip, which takes a full frame time and adds significant delay.

Instead, it is more effective to execute the color separation on the host side (CPU and/or GPU), where it is done as part of the late warping step, and then transmit the single-color component frame buffers directly to the LCOS. The host would then warp the red components from the RGB frame buffer and transmit it to the LCOS for direct display. With the right time offset, the same operation is then repeated for green, blue and potentially green again for the same RGB frame. This approach is challenging to implement, as it requires both custom silicon for connecting the GPU to the LCOS directly, as well as the ability to warp the current frame while the next one is being rendered. Nonetheless, optimal implementations of this technique can achieve latencies of 1–2ms per color component from GPU to photons.

LATENCY OF AN OPTIMIZED AR/VR PIPELINE

So, how low can we get the motion-to-photon latency when considering all of the above techniques? To answer this question, we must first distinguish between display technologies (OLED vs LCOS), since they each require different methods and create different effects. Secondly, we must refine our definition of latency.

Motion-to-photon latency is about two points in time: the point in time when a motion occurs, and the point in time when the resulting image becomes visible to the user. On the motion side, we must consider how we can use the measured motion. We initially wanted to use it as pose input to render the 3D scene, but once we realized this results in excessively high latencies, we extended our system with late warping. While this late warping does use pose input, it can only perform a very limited set of corrections, since it is applied on an already created 2D framebuffer. Fortunately, this limited set of corrections are sufficient in practice, so we will consider the time stamp of the warping pose for the motion side.

On the display side, we saw that both OLED and LCOS produce artefacts due to the sequential nature of updating the display. We introduced warping per vertical tile to reduce latency of the scanline sequential displays, such as OLED, and we introduced warping per color component to reduce latency of color sequential displays, such as LCOS. Interestingly, the achievable latencies are similar in both cases, but one is in the spatial domain and the other is in the color domain.

Taking this all into account, let’s look at an optimized AR pipeline with an LCOS-based display. We’ll use the numbers from the examples above, with just a few changes. We will assume that renderer and display run at 90Hz, as DAQRI Smart glasses do. We’ll also assume RGB subframes rather than RGBG subframes for the sake of simplicity. Finally, we’ll assume the LCOS is being connected to the GPU in such a manner that it receives subframes directly (we will not go into detail how such a connection could be implemented).

The figure below shows how all components work together in this system. As declared earlier, at the time the tracker picks up the camera image (see marker [1] in the figure below), it is already ~15ms old. The tracker adds another 25ms of latency, so by the time the pose becomes available [2], we already have a latency of ~40ms. When the renderer picks up the latest pose from the tracker, it uses all available IMU samples to calculate a pose plus camera motion that has only ~1ms of latency [3]. The renderer further estimates a pose that is forward predicted ~17 milliseconds into the future: 11ms for render time plus ~6ms for the display to show the middle (green) subframe.

In order to further improve our forward prediction, the system repeats this process in the late-warping step [4]. This warping is performed a total of three times — once for each subframe. Assuming 2ms for warping plus 2ms for readout and transfer into the LCOS, we must therefore forward predict only 4ms for each subframe. This reduces latency, leads to more accurate predictions, and introduces corrections per color channel to ultimately reduce color-breakup.

Overall, what is the final latency we can achieve with such a system? The answer is: It depends. If we consider rendering using the latest 6DOF pose estimates as the last step in our pipeline that produces fully correct augmentations then achieving a latency of ~17ms is our best-case scenario.

However, late warping can correct most noticeable artefacts, so it makes sense to treat it as valid part of the pipeline. Without using forward prediction for late warping, we can achieve a physical latency of 4 milliseconds per color channel in our example system. However, 4 milliseconds is such a short length of time, making forward prediction almost perfect and pushing perceived latency towards zero. Further, we could actually predict beyond 4ms into the future to achieve a negative perceived latency. However, negative latency is just as unpleasant as positive latency, so this would not make sense for our scenario.

CONCLUSIONS

In this article, we described the challenges and approaches in dealing with latency in mobile AR and VR systems today. We saw that the required hardware and software solutions for these systems are dissimilar to gaming systems for monitors, where latency is of less concern. Additionally, mobile solutions have significantly less processing power than stationary systems — a problem that is usually addressed with more specialized hardware blocks in the form of DSPs, FPGAs or ASICs in order to successfully resolve the conflicting requirements for low power, low latency, and high throughput.

The overview presented here is largely conceptual, while real commercial systems must deal with numerous low-level details, from connecting a standard GPU to an LCOS to achieving highly accurate timings required for warping with tiling on a general-purpose architecture. Similarly, there are a number details on the software side that we did not discuss, such as enabling generic applications to take advantage of the low latency pipeline.

Lacking the required standards, these details are usually implemented in proprietary forms today. Display and display protocol standardization groups are already discussing extensions for improved AR and VR support, but it will take years for these extensions to mature and become available in products. Similarly, software like OpenVR and UnityXR continue to aim for standardization, but they have not reached product level yet. Until then, AR and VR device manufacturers will continue developing and promoting their proprietary solutions.

Though this article primarily focuses on depth cameras for mobile AR, expect to learn about a variety of depth cameras, if only for the benefit of better understanding the restrictions imposed by mobile AR use cases. Additionally, we’ll cover the whole range of depth cameras — those based on structured light (SL), passive stereo (PS), active stereo (AS) and time of flight (ToF). We will only consider solutions at a commercial level, thereby excluding research topics such as depth from a single image that may deliver impressive results, but are not up to the requirements of mobile AR.

This article is written by Daniel Wagner, Chief Scientist at DAQRI. Before coming to DAQRI to lead the DAQRI Austria Research Center, Daniel was Senior Director of Technology at the Qualcomm Austria Research Center. He received his MSc from Vienna University of Technology and his Ph.D. from Graz University of Technology, and is well-known for pioneering the research area of AR on mobile phones.

DEPTH CAMERAS FOR MOBILE AR

Though depth cameras — cameras with the ability to capture 3D images — have been around for several decades, the general public was first exposed to this technology in 2010 with the introduction of the Microsoft Kinect as an accessory for the XBOX 360. Unfortunately, as with many cutting-edge products, and despite its impressive features, extensive press coverage, and significant financial backing, adoption of the Kinect was limited, leading to its recent discontinuation. More recently, we’ve seen similar technology integrated into Apple’s iPhone X as a front sensor bar that enables robust facial tracking and device unlocking.

While the above examples are clearly novel and entertaining, they are often considered to be somewhat gimmicky by those more familiar with the vast capabilities of depth cameras for mobile augmented reality (AR). More appropriately, high-end wearables such as Microsoft Hololens, Meta 2 and DAQRI Smart Glasses® include depth sensors for reasons outside of mere entertainment: the environmental understanding required by AR wearables is difficult to achieve with other types of sensors, and they are essential for natural, hands-free user input (such as gestures). I must admit, though, I haven’t seen a really convincing usage of gestures as an input mechanism.

With that being said, it’s important to appreciate and understand the surprisingly large variety of depth cameras, ranging from consumer solutions, as in the aforementioned iPhone X, to large, highly precise and expensive industrial solutions, as with the Mantis Vision F6 or the Ensenso cameras from IDS Imaging. While this white paper primarily focuses on depth cameras for mobile AR, expect to learn about a variety of depth cameras, if only for the benefit of better understanding the restrictions imposed by mobile AR use cases. What’s more, we’ll cover the whole range of depth cameras- those based on structured light (SL), passive stereo (PS), active stereo (AS) and time of flight (ToF). We will only consider solutions at a commercial level, thereby excluding research topics such as depth from a single image that may deliver impressive results, but are not up to the requirements of mobile AR.

DEPTH CAMERA BASICS

At present, the terms structured light and active stereo aren’t 100% clearly defined. However, for the purpose of this paper, let’s refer to structured light as a combination of a single-camera and a single-projector, where the projector projects a known pattern; as opposed to active stereo, which uses two cameras and a projector projecting an unknown (random) pattern.

Both structured light and active stereo, as well as passive stereo, are all stereo-based solutions — meaning that depth estimations are based on identifying features from two different viewpoints so that they can be triangulated. It is this which implicitly gives us depth, the distance of a point from the camera along the camera viewing direction. Note that the depth estimation is stereo because of the two viewpoints needed to estimate depth, not the number of cameras (a common misconception). In the case of structured light, the projector represents one viewpoint and, due to the known projected pattern, features have to be found only in one camera view. Active and passive stereo solutions, on the other hand, need to match features from one camera to the other. Only Time of Flight does not rely on stereo principles and measures depth directly (although most often not based on time, as we will see below).

REQUIREMENTS FOR MOBILE AR DEPTH CAMERAS

So what are the specific requirements for depth cameras in the case of mobile AR? First, sensors need to be very small in order to integrate into headsets of comparably restricted size. For AR headsets, small can be defined as “mobile phone class sensors” in size,e.g., a camera module no more than 5mm thick. Second, the depth camera should use as little power as possible, ideally something noticeably lower than 500 mW, since the overall heat dissipation capability of the average headset is just a few watts. Third, in order to further save power, the depth camera should not require intensive processing of the sensor output since that would result in further power consumption.

Two prominent use cases for depth cameras in mobile AR today include environmental scanning (e.g. reconstruction) and user input (e.g. gesture recognition). These two use cases have largely different requirements. For environmental scanning, the depth camera needs to see as far as possible — in practice roughly a range of around 60 cm to 5 meters. In contrast, user input needs to work at only arm’s length, hence a range of around 20 to 100 cm. Thus, in total, we’d like to cover a range of 20 cm to 5 meters, a 25x ratio. Unfortunately, such a vast range is problematic for most depth cameras today. For example, the emission power of a projector that reaches as far as 5 meters would oversaturate the camera for objects at a distance of 20 cm. Similarly, a stereo-based solution providing meaningful accuracy at 5 meters requires a large baseline, which simultaneously makes it highly challenging to find correspondences close up. Fortunately, these two ranges usually don’t have to be supported simultaneously; although it can be advantageous for a depth camera to support pseudo-simultaneous modes alternating between short and long range modes on a per frame basis.

Lastly, there is the matter of calibration. As with any sensor, depth cameras need to be calibrated to function properly. Mobile devices undergo a lot of physical stress over several years of usage, which can invalidate factory calibration. As automatic built-in self-calibration is not yet available, they rely on the factory calibration to remain valid over their lifetime, which can be a problem.

STRUCTURED LIGHT

As mentioned above, structured light (SL) projects a known pattern using a projector. The projector and camera form the stereo system, hence no second camera is required. Additionally, SL systems usually work in the infrared spectrum so that the pattern is not visible to the human eye. For safety reasons, the projector needs to be limited in projection power, usually resulting in a projection distance of up to 4–5 meters. It’s clear that there is a trade off between robustness and accuracy. For example, even though the Kinect v1 delivered VGA depth maps (~300,000 depth values), it projected a pattern with 30,000 dots only. Hence, it could also calculate only around 30,000 depth values, but used clever filtering to populate the other 90% of the depth map entries. On the positive side, by projecting only a small number of dots the projector could use a relatively large amount of energy per dot. Similarly a low number of dots allows each one to be relatively large and easily detectable in a camera image. As a result, this sensor was quite robust to difficult materials and lighting conditions. For instance, Mantis Vision — another depth sensor maker — uses a much more detailed pattern in their depth cameras that results in more detailed and more accurate depth maps. However, as users of the 2nd Tango developer device (codename Yellowstone) noticed, this depth camera was less robust than the Kinect.

Primesense — the company that provided the IP for the Kinect v1 — had a successor with codename Capri in development that was used in the first Tango device (codename Peanut). Capri was significantly smaller and more energy efficient than it’s precursor. Apple acquired Primesense and it’s believed the technology was shrunk even further to fit into the front sensor bar of the iPhone X.

Mantis Vision’s focus is on professional structured light cameras. Their handheld scanner products F5 and the more recent F6, are the most accurate structured light cameras on the market at present. Yet form factor and cost of these devices are not suitable for integration into mobile AR headsets.

With the demise of Primesense other companies are stepping in to fill the gap. Orbbec seems to be on the rise of becoming the successor in robotics, still their sensors remain too large for integration into mobile AR devices.

As with all stereo-based solutions, structured light can suffer from structural integrity: If the baseline is modified — such as due to pressure on the device — depth quality can quickly deteriorate to an unacceptable level. In order to achieve high quality measurements up to 5m, a baseline of ~8 cm is required. This fits tablets well, but can be challenging for mobile phones as well as for AR headsets. Also, structured light solutions optimized for environment scanning usually have a minimum sensing range of 50–70 cm preventing usage for hand tracking.

To summarize, structured light doesn’t measure depth directly, but relies on stereo matching, which is an expensive operation requiring special hardware units such as an ASIC or a DSP for real-time, low-power processing.

ACTIVE AND PASSIVE STEREO

Active and passive stereo are quite similar in that they both rely on matching features from one camera to another. Passive stereo suffers from not being able to match uniformly colored surfaces such as white walls or empty desks that simply lack features, such as texture, to match. Active stereo overcomes this problem by projecting a random pattern onto the environment such that even textureless surfaces can be matched between the two views. However, it also means that active stereo is the most hardware component intensive solution requiring two cameras, a projector and, same as SL, most often an ASIC or DSP for real-time processing. Since the matching principle is the same, active stereo cameras have the advantage that they can usually fall back to passive stereo in situations where the pattern cannot be projected successfully, such as in outdoor lighting or over large distances.

Intel’s RealSense sensor, enabling depth and IR sensing in DAQRI Smart Glasses, is a well known implementation of an active and optionally passive stereo system. Occipital recently announced their Structure Core depth sensor. While their older Structure IO sensor still uses Primesense IP, the new Structure Core sensor is a complete in-house development based on active stereo.

One often hears arguments that passive stereo works well outdoors where untextured surfaces are rare. While that is true for use cases such as self-driving cars, a small baseline of less than 10 cm limits the distance up to which matched features can be triangulated accurately, which is not practical in most AR use cases. While it is, of course, possible to employ more than two cameras, such approaches are not suitable for mobile AR due to increased cost, size and computational complexity.

A strength of all stereo-based solutions (including structured light) is that they usually have a pretty good x/y resolution (e.g. VGA or higher resolution depth maps). If very small objects need to be identified, then that can be a significant advantage. On the other hand, all stereo-based solutions suffer from shadowing effects, which arise from the one view not observing exactly the same part of the scene as the other view: At large distances both cameras see mostly the same, but close up the two cameras can see very different parts of objects. This is like holding your finger close up between your eyes: The left eye will see the finger from the left side and the right eye will see the finger from the right side, but the overlap and hence the part for which depth could be calculated is minimal. Since depth maps are usually calculated from one camera’s point of view, this looks as if a close-up object casts a shadow on itself where no depth data is available.

TIME OF FLIGHT

While stereo-based solutions estimate depth by triangulating matched features, Time of Flight (ToF) cameras measure distance directly, meaning that the device measures the time offset between a signal sent out by the emitter until it arrives back at the sensor. Like stereo-based solutions, most ToF cameras today emit infrared light at 850nm wavelength, rendering it invisible to human eyes and suffer less from interference with most artificial light sources found indoors.

Based on the measured time offset and using the known speed of light, distance can be calculated. However, due to the vast speed of light, sub-nanosecond timing accuracy is required to get accurate results. Due to technical challenges, only a few devices actually use this principle: one well-known device class are LIDAR sensors, which are popular in self-driving car research today.

Most ToF cameras used in AR and VR today are not measuring the time of the signal, but its phase shift instead, from which distance can also be estimated (aka “continuous wave”). In order to measure phase, the emitted signal needs to be modulated (e.g. at 80MHz). However, phase cannot be measured directly but instead must be calculated from at least four measurements of the modulated infrared signal. As a result, ToF cameras record at least four so called sub-frames. These four measurements at each pixel are then combined to calculate the signal’s phase at that pixel, from which in turn distance can then be estimated.

There is an issue with this elegant method: To achieve accurate phase measurements, the signal’s phase needs to be short, yet short phases can wrap around meaning that a specific phase shift can be the result of an object at distance N, but also at distances 2N, 3N, etc. This issue in known as phase ambiguity or integer ambiguity.

For short range measurements, that is not a problem: In such a working mode the emitted infrared light is weak enough so that a signal reflected from a distance resulting in wrapping around would be too weak to be picked up by the sensor. In long range mode, a phase wrap-around can easily happen. The usual solution is to then use multiple phases to disambiguate the measurement. However, since each phase measurement requires 4 sub-frames, using two phases requires 8 sub-frames. The long time it takes to capture 8 sub-frames can lead to motion artifacts if either the depth camera or measured objects move.

ToF depth cameras usually support mixed modes e.g. a 35Hz mode short range mode that runs simultaneously to a 5Hz long range mode by cleverly interleaving them, thereby supporting environment scanning and gesture tracking at the same time. A clear advantage of ToF cameras is that they can be very small: The measurement principle does not rely on triangulation, so there is no baseline required and emitter and sensor can be put as closely together as integration allows for. Another benefit of not requiring a baseline is that calibration is simpler and less likely to invalidate over time. Also, calculating the phase and hence the distance from four measurements requires much less processing than matching features as in stereo-based solutions.

Yet, ToF cameras are not without their shortcomings. Most obviously, ToF cameras today typically have a relatively low resolution of 100k pixels and often even lower. Also, ToF cameras suffer from multipath effects, which occurs if signals do not only reflect directly back to the sensor, but take multiple turns. Similarly, at object boundaries some of the sub-frames might measure the foreground, while others measure background surfaces resulting in artifacts. Finally, stray light (light reflecting back from very close objects such as the headset itself) can be problematic. All these issues can be dealt with up to a certain extent, but require special care when positioning the sensor or extra processing thereby reducing the computational advantage of ToF cameras.

THE FUTURE

At present, there is no perfect depth camera — a fact that is unlikely to change for the foreseeable future. However, we have seen enormous progress over the last decade. The Kinect v1 was released only 7 years ago. Its technology has since been miniaturized to fit in the front sensor bar of the iPhone X. Similarly, ToF cameras have made significant progress delivering same or better quality than stereo based solutions today.

Still, all popular sensors are limited primarily to indoors usage. While ToF has a slight advantage here due to using modulated light, in strong, direct sunlight all active solutions fail. ToF camera manufacturers such as PMD have recently announced that their sensors will also support 940nm, which is known to work better outdoors than the 850nm dominantly used today. Samsung has shown similar improvements with a 940nm based structured light depth sensor.

Another important factor is cost. Due to the huge success of color cameras in mobile phones, top quality camera modules cost only ~10$ today. Until recently, there has been no market that would support large scale and low cost production of depth sensors, but it seems this is changing and I expect depth sensor prices to come down too.

DAQRI WORKSENSE WILL INCLUDE FIVE APPS:

DAQRI Show. A natural combination of video, voice and 3D collaboration, DAQRI Show enables you to bring others into your work situation. Your audience can observe, direct, or annotate your real-world view with digital tools and instructions. DAQRI Show is perfect for hands-free interaction with remote experts who can help you diagnose next steps, or to deliver the ultimate product support experience to your customers. DAQRI Show has evolved from the DAQRI Remote Expert app that has been used by customers for over a year.

DAQRI Tag. Manage your operations at a glance. With DAQRI Tag, your critical information or multimedia is visually attached to real-world spaces, facilities and assets. DAQRI Tag can also link to live data from existing IoT systems such as IBM Maximo or Honeywell BMS — making tasks like maintenance and compliance both simpler and hands-free.

DAQRI Scan. Designed to let anyone critically assess an actual facility or piece of equipment, DAQRI Scan captures environments as photorealistic, shareable 3D models. DAQRI Scan is ideal for retrofits, installations, and change detection. DAQRI Scan models can be easily used by platforms from software vendors such as Autodesk or Siemens as well as popular 3D tools and engines like SketchUp or Unity.

DAQRI Model: BIM Edition. For professionals in the design studio or in the field, DAQRI Model transforms high resolution 3D Objects from Autodesk BIM 360 Docs into immersive, full-scale walkthroughs. With DAQRI Model, teams can compare designs to work in progress anytime and keep the job site and the head office in sync with a fully digital workflow.

DAQRI Guide. With DAQRI Guide, simple references or training are brought into the field of view. Your policy documents and media help users confirm their workflow. DAQRI Guide helps you deliver the right content to people in the context of their jobs and environment.

The DAQRI Worksense Suite will be available for your preview over the upcoming weeks and months via the DAQRI Worksense Early Adopter Program. DAQRI Show is available to you now. Apply now to the DAQRI Worksense Early Adopter program.

THE DAQRI WORKSENSE SUITE WILL ULTIMATELY BE AVAILABLE AS TWO EDITIONS:

DAQRI Worksense Standard, included with every pair of DAQRI Smart Glasses and containing all Standard Edition Worksense apps (Show, Tag, Scan, Model, BIM Edition, and Guide). As a customer, you will receive a perpetual license to DAQRI Worksense Standard when it releases later in 2018.

DAQRI Worksense Pro, offering all Worksense apps plus AR capabilities and features for enterprise needs such as team collaboration and analytics. DAQRI Worksense Pro will be offered as an optional subscription later in 2018. As a DAQRI customer, you will receive a no-cost trial of DAQRI Worksense Pro to help determine the right edition of Worksense for your needs.

We will let you know when new Worksense applications are available via the Worksense Early Adopter Program, or as released apps. Stay tuned!

DAQRI WORKSENSE PRO SUBSCRIPTION AND LAUNCH PROMOTION

DAQRI also announced the DAQRI Worksense Pro Subscription, a subscription service that delivers a complete augmented reality device and software package. Customers can sign up for a 12 or 24-month DAQRI Worksense Pro Subscription to receive:

• DAQRI Smart Glasses, including an optional device upgrade benefit during active subscriptions
• DAQRI Worksense Pro software suite with Pro editions of all apps

List price for the DAQRI Worksense Pro Subscription is $549/€469 for a 24-month subscription. A launch promotion is available immediately through May 31, 2018, with a 24-month subscription priced at $389/€329 per month. Contact DAQRI to learn more about the DAQRI Worksense Pro Subscription.

Thank you for helping to make professional-grade AR come to life!

From a user’s perspective, augmented reality may seem easy. Under the hood, however, there are a number of underlying technologies necessary to create integrated wearable AR experiences. In this mini-series, we’re pulling back the curtain with DAQRI’s experts to provide you with a better understanding of what it takes to deliver Professional Grade AR™. We recently discussed the importance of fluid tracking, making the virtual content you see as real as possible, creating a wide field of view, and user-centric models. In this episode, DAQRI Chief Revenue Officer, Chris Kaufield discusses the importance of industry partnerships.

“PROFESSIONAL GRADE AUGMENTED REALITY IS ABOUT MORE THAN JUST ONE WORKER.”

At DAQRI, we believe augmented reality maximizes the efficiency, safety, and effectiveness of both individual workers AND workforces. Every industry has unique tools and requirements, and we know that enterprises need to leverage existing technology investments in order to drive productivity. Industry and technology partnerships bring existing data and tools to life in AR and are a core element of professional-grade augmented reality.

Our partners want to unlock the data, information, and 3D models that reside within computer systems and bring them to life in real world environments where they belong. For example, you can display asset information, such as maintenance data or real-time operating data, in context of the actual asset. You can also compare a 3D building design with what is actually being constructed on the job site.

What types of existing data and tools do you want to bring to life via AR? Let us know!

From a user’s perspective, augmented reality may seem easy. Under the hood, however, there are a number of underlying technologies necessary to create integrated wearable AR experiences. In this mini-series, we’re pulling back the curtain with DAQRI’s experts to provide you with a better understanding of what it takes to deliver Professional Grade AR™. We recently discussed the importance of motion-to-photon latency, fluid tracking, making the virtual content you see as real as possible, and the need for a wide field of view. In this episode, neuroscientist Stef Hutka emphasizes the importance of working with a user-centric model.

A key component in delivering professional-grade AR solutions to enterprise primarily involves:

“Working with a user centric model to build a really well-tailored augmented reality solution.”

When building augmented reality solutions, understanding how end users traditionally complete tasks is critical. Professional-grade AR offers the opportunity to greatly enhance and streamline the tasks that workers do every day; such as training, maintenance, or manufacturing. DAQRI has experience in working directly with end users, and we are seeing the categories of tasks where AR can make a real impact in productivity.

By leveraging AR, the DAQRI AR platform enables organizations to streamline processes that have become overly complicated due to the limitations of traditional methods, dependence on multiple external devices, or simply relying on the memory of workers.

Learn more about DAQRI Smart Glasses at daqri.com.

Wondering how AR can positively impact your team and bottom line?

Here are eight ways AR can help your company today.

1. KNOWLEDGE TRANSFER THROUGH AUGMENTED WORK INSTRUCTIONS

As the “baby-boomer” generation approaches retirement, their decades of experience will soon disappear — forever! Replacing these baby-boomers is the tech-savvy and digitally-aware Millennial workforce. One of the best ways to transfer the “intellectual know-how” of any company preparing to lose its aging workforce is to commit this expertise to digital content and augmented work instructions. Companies can replace weeks of expensive 2D classroom training, manuals, drawings, and work instructions with digital content (3D models and simulations), enabling the new workforce to be functional and efficient while reducing costs and downtime. In the process, the tacit knowledge workers have fortified over decades of operation will be safeguarded and imparted onto the next generation of workers.

2. CONTEXTUALLY RELEVANT INFORMATION

AR delivers on the promise of the Industrial Internet of Things. The IIoT boasts an abundance of information, but lacks the means to properly distill and relay it to team members. Fortunately, AR does just that. An operator can be given small, bite-size pieces of information in a contextually relevant manner, meaning the information is relative to the equipment he is beside and the task he is performing. The operator can then make more well-informed decisions, as they aren’t overloaded with information. More, AR can effectively prevent an “alarm-avalanche” in emergency situations, where one error subsequently triggers a series of errors in other areas of the plant. Contextually relevant alerts and reminders act as a vital tool for operators, improving task performance, efficiency, and safety as a result.

3. REMOTE EXPERT

In the immediate term, baby-boomers can continue fulfilling the “expert” role, acting as real-time, remote mentors for new employees. Using AR’s Remote Expert functionality, experts can assist new hires with tasks, delivering live expert information. Consequently, novice employees no longer need to wait hours or days for feedback due to location or other constraints. Remote Expert allows a user at a computer to connect to a worker in the field with an AR wearable, such as DAQRI Smart Glasses™. The remote worker can speak with the field worker, see exactly what they are looking at, and even draw instructions for them to view. Real-time assistance increases both employee accuracy and cost efficiency, all within a shorter time period.

4. MULTI-SKILLING

The copious amount of time required to achieve expert level produces a shortage of certain industry specialists, creating a bottleneck in some companies. With AR, workers no longer need to be experts to successfully execute a task, just qualified. Augmented work instructions can encapsulate years of experience, empowering someone who is qualified (but not quite an expert) to complete a task they would otherwise be unable to accomplish. Even if they hit a roadblock, an expert can use Remote Expert to talk them through the task, reducing the time to repair a piece of equipment by hours, days, and even weeks.

5. REDUCED LEARNING AND CLASSROOM TIME

Relearning a task because it has not been performed in 6+ months often requires days of classroom work. With augmented work instructions, a worker never loses track of current steps in the process, eliminating the need to revisit the classroom. Additionally, field technicians can visualize what is most important, resulting in greater learning retention (80% compared to 20%) and increased operational efficiency. Through this immersive education, novices are also able to learn faster with the ability to simultaneously put their knowledge to work as they learn on-site. Aside from dramatically reducing the time required to learn and implement a new task, tasks are flawlessly executed by an experienced operator.

6. ERROR REDUCTION

By outlining and monitoring required steps, augmented work instructions can effectively reduce errors to zero, even if a worker is performing the task for the very first time. With all operators performing an identical sequence of steps for a specific task, a certain standard of quality is established. Even experts are reminded of steps they might have forgotten or skipped, ensuring consistency across the board.

7. STREAMLINED DATA COLLECTION

With the ability to actively or passively collect data from the varied steps and tasks of a job, less documentation is required post-task, allowing the worker to complete more assignments and perform less data-entry within a work day.

8. INCREASED SAFETY AND CONFIDENCE

Overall, contextually relevant knowledge inevitably increases worker safety and confidence. Considering highly technical roles in high-stress environments, workers are often confronted by increased uncertainty in their tasks. Access to correct information, a simplified job process, and successful task completion are critical. AR addresses these needs while providing indications of dangers in the area (such as heat sources, radiation, gases) with navigation instructions away from such hazards. AR delivers hands-free work instructions, freeing the operator to complete their task without the need to divert their attention by consulting hand-held manuals. The operator is also given the confidence that they have completed all necessary pre-check safety procedures in the correct sequence (such as LOTO — Lock-Out, Tag-Out) before executing an inspection or maintenance on a potentially dangerous piece of equipment.

THE AR JOURNEY BEGINS TODAY

Accessing the right information when and where it’s needed empowers teams to do their best work. With shared knowledge delivered by AR increasing efficiency, consistency, and safety, the power of augmented reality is palpable.

Guest Post Written by Michael Pentony, Director of Enterprise Sales (EMEA)

From a user’s perspective, augmented reality may seem easy. Under the hood, however, there are a number of underlying technologies necessary to create an integrated wearable AR experiences. In this mini-series, we’re pulling back the curtain with DAQRI’s experts to provide you with a better understanding of what it takes to deliver Professional Grade AR™. We recently discussed the challenges of fluid tracking and of making the virtual content you see as real as possible. In this episode, we focus on the importance of motion-to-photon latency.

Motion-to-photon latency is a very important measurement in determining the quality of a computer vision system. It is the measurement of the time it takes for light to enter a camera, get processed by a computer vision algorithm, generate a pose, render digital content, and display it as augmented reality so that it can be seen by the eye. The time it takes from light entering the camera to AR hitting your eye is the motion-to-photon latency, and,

“Reducing this latency is one of the most important, yet difficult problems in delivering Professional Grade AR.”

If the latency is too high, the content won’t match up very well…things will look “off” and your body might perceive a motion that isn’t matching up with what you’re seeing. This phenomenon happened in the early days of Virtual Reality when many early adopters reported experiences of motion sickness and nausea.

Because we’ve designed the DAQRI Smart Glasses from the bottom up, we’ve achieved industry-leading motion-to-photon latency. As a result, the negative feeling of motion sickness now seems like yesterday’s news, and you will experience the very positive feeling when your digital content appears in the real world, in the right place. No matter where you look, your digital content just stays there, right where you need it…allowing you to wear the glasses for your whole shift so you can achieve more and do your work better.

An augmented reality system really does two things…

“IT FIGURES OUT WHAT YOU’RE LOOKING AT, AND THEN OVERLAYS THE DISPLAY ON TOP OF IT.”

But that’s not as easy as it sounds. If your computer vision isn’t solid, you will get jitter. If your displays don’t update fast enough to render the necessary content, you will get lag or overshoot. These are huge problems in augmented reality and can only be solved when the entire system is optimized for these kinds of challenges.

“EVERY LAST MILLISECOND, WE SWEAT.”

When you put on a pair of DAQRI Smart Glasses, you’re getting an entire AR platform — hardware, hardware buses, firmware, computer vision algorithms, and a display pipeline that has been developed and optimized to function seamlessly and to enable fluid tracking, all so you can achieve more and do your work better.

Los Angeles, CA — November 7, 2017 –­ DAQRI began shipping its DAQRI Smart Glasses™ today to customers worldwide. The professional-grade augmented reality (AR) platform is available for direct purchase through DAQRI’s website as well as through select channel partners. Customers across manufacturing, field services, maintenance and repair, inspections, construction, and more can now deploy the DAQRI AR platform in their operations.

Building on its suite of core technologies, DAQRI is now bringing professional-grade AR to scale through manufacturing and reseller partnerships, making AR a viable option for businesses across the globe to improve efficiency and productivity. “DAQRI has developed a comprehensive strategy to bring professional-grade AR to scale, including hardware, software, developer-friendly tools, and a robust partner ecosystem,” said Roy Ashok, CEO of DAQRI.

To enable its AR platform to deliver value for customers today, DAQRI is partnering with manufacturers and distributors experienced in delivering enterprise technology at scale. DAQRI’s partnership with Flex, the Sketch-to-Scale™ solutions provider that designs and builds Intelligent Products for a Connected World™ will provide global reach and allow DAQRI to meet the needs of each customer’s workforce.

“AR can deliver a transformative experience for both the consumer and enterprise markets,” said Mike Dennison, President, Consumer Technologies Group at Flex. “As the market gains momentum, we’re very excited to be partnered with DAQRI to help this leader in AR innovation redefine what is humanly possible.”

On the distribution side, DAQRI has partnered with Dell, a technology leader invested in AR solutions that can accelerate and improve communication, collaboration, design — and then help drive product lifecycles through build, maintenance, and operations in the field. DAQRI and Dell are collaborating throughout the supply chain, alongside customers and other partners, to build scalable solutions that can optimize performance and impact results.

“The work DAQRI is doing to advance the tangible benefits of AR is why we chose to collaborate with them as part of the Dell VR/AR Technology Partner Program,” said Neil Hand, Vice President of Product Strategy and Innovation at Dell. “The speed of immersive technology innovation is a central force driving business transformation and evolution; DAQRI has developed one of the strongest solutions for professional use that, when paired with Dell, can supercharge business productivity.”

Alliance Partnerships

DAQRI is partnering with other technology leaders to deliver professional-grade AR solutions, including connecting IoT workflows and displaying IoT data to workers, which is a key use case for enterprises.

“Oracle works with DAQRI because it is clearly focused on developing professional-grade AR solutions,” said Bhagat Nainani, Group Vice President of IoT Applications, Oracle. “AR adds a whole new dimension to asset management, and applied the right way, can increase real-time situational awareness and speed up response times, ultimately improving business outcomes and return on assets.”

By integrating with enterprise software offerings from Autodesk, IBM, Oracle, Siemens, Emerson, and others, DAQRI helps customers leverage existing workflows, tools, and data in AR solutions.

DAQRI Customers Solving Problems Today

Customers across manufacturing, field services, maintenance and repair, inspections, construction, and more are deploying the DAQRI AR platform. In the case of Siemens, training and manufacturing assembly operations were condensed into a single seamless process that resulted in major improvements in efficiency and quality.

“In one of our projects leveraging DAQRI’s professional-grade AR platform, gas burners, which typically take two hours to build and a day to train, were assembled with 100% accuracy in under an hour by complete novices,” said Frank Vossnacker, a Senior R&D Project Leader at Siemens, who has been working with the DAQRI team for two years. “It’s clear to us that integrating AR into work training, manufacturing, inspections, and maintenance results in major increases in speed, efficiency, and quality of work.”

Royal BAM Group is looking to scale AR across their operations through a partnership with DAQRI.

Menno de Jonge, Director of Digital Construction at Royal BAM Group, said: “Recognizing AR as the quintessential technology for improving productivity and minimizing failure in facility management, BAM’s partnership with DAQRI represents taking important steps toward implementing AR on a large scale, starting with the first lean hospital in the Netherlands to implement lean operating methodologies, Zaans Medical Centre.”

Additionally, Parker Hannifin, Aprilia, Mortenson Construction, and other enterprises work with DAQRI to help them to achieve more and to solve core business challenges.

DAQRI Smart Glasses: Purpose built with ergonomics and safety

· DAQRI Smart Glasses have a modular design approach, with weight distributed between a lightweight set of glasses and a discrete body-worn compute pack for comfort and flexibility. They feature a hands-free user interface that is easy-to-learn and easy-to-use.

· Comfortably interact with 3D content in a wide field of view (44 degrees diagonal).

· Indoor and outdoor use is enabled by bright, transparent stereoscopic displays. Use DAQRI Smart Glasses in the office and out in the field.

· ANSI Certified.

Mobile and powerful

· A powerful mobile processor drives low-latency rendering and displays complex workloads, such as models and data from PLM systems and professional design tools.

· DAQRI Smart Glasses can become a wireless IoT device that connects with Wi-Fi and Bluetooth. High performance optical sensors include a wide-angle tracking camera, a depth-sensing camera, and an HD color camera for taking photos and videos.

· Advanced, hardware-accelerated computer-vision technology combines simultaneous localization and mapping (SLAM) and sparse mapping technology to deliver persistent and reliable position tracking across large spaces. Users can “pin” an AR model to any point and it will remain in the right spatial position and orientation regardless of user movement over large distances, such as an entire manufacturing facility or construction site.

DAQRI Smart Glasses are the Professional Grade AR™ platform and uniquely deliver powerful, mobile AR experiences with a focus on ergonomics and safety. DAQRI Smart Glasses™ are available for purchase now from the DAQRI website for $4,995, and through select distributors and resellers.

What does it take to deliver Professional Grade AR™ to enterprise? DAQRI Smart Glasses™ lead the way, bringing AR to scale with capabilities purpose-built to deliver contextually-relevant data in the field and in the office. Catch a glimpse of this exciting new product and meet some of the team behind its creation as well as those bringing it to market.

Enterprise workers can now perform tasks faster, more accurately, the first time and every time with DAQRI Smart Glasses.