<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sally Kellaway on Medium]]></title>
        <description><![CDATA[Stories by Sally Kellaway on Medium]]></description>
        <link>https://medium.com/@s.a.kellaway?source=rss-98282075c0b4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*r12fyftkHSVYgTBaox9ZUA.jpeg</url>
            <title>Stories by Sally Kellaway on Medium</title>
            <link>https://medium.com/@s.a.kellaway?source=rss-98282075c0b4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 07:27:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@s.a.kellaway/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Augmented Reality: The New Frontier for Audio (Part 1)]]></title>
            <link>https://medium.com/@s.a.kellaway/augmented-reality-the-new-frontier-for-audio-part-1-8cde2a352712?source=rss-98282075c0b4------2</link>
            <guid isPermaLink="false">https://medium.com/p/8cde2a352712</guid>
            <category><![CDATA[spatial-audio]]></category>
            <category><![CDATA[3d-audio]]></category>
            <category><![CDATA[audio]]></category>
            <category><![CDATA[augmented-reality]]></category>
            <category><![CDATA[audio-technology]]></category>
            <dc:creator><![CDATA[Sally Kellaway]]></dc:creator>
            <pubDate>Wed, 13 Dec 2017 01:25:48 GMT</pubDate>
            <atom:updated>2017-12-13T01:25:48.616Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O5BCnOT3HRQPBUOoXm7eBQ.jpeg" /></figure><p>Depending on who you ask, Augmented Reality is either entering or fully in its hype cycle. 2017 saw both Apple and Google come to market with AR app development kits. Slivers of the public have been enjoying apps that can help you measure objects, place furniture, learn things, do treasure hunts and Fashion/Beauty apps and games are trickling into the market as well.</p><p>Engaging with this kind of content is as simple as picking up your phone and holding it at eye level. But when we inevitably put the phone down after a few minutes wondering why the content wasn’t very engaging, it isn’t always obvious what is missing from the experience. In world where most mobile gamers don’t play with headphones or mute out the audio from the experience, players who engage with content in this way miss out on a lot of the immersive quality that audio brings to an experience.</p><blockquote>The answer to unlocking the next phase of immersion and presence within AR is actually something that we’ve trained ourselves to ignore for many decades.</blockquote><p>This two-part series will delve into initial considerations and learnings that I’ve have come across in my time prototyping and researching Audio for Augmented Reality during my time with, and around OSSIC in 2017. This pair of articles will form a good primer for you regarding the challenges we may see and features we should demand as we see more and more Apps and Games being developed for Mobile AR and headset-based AR.</p><p>This first article will focus on defining the technology itself, what types of AR devices exist, why AR is so challenging and the things that I think will be most difficult to overcome for audio in this new medium.</p><p>The second article will delve into some of the deeper research into what cognitive benefits audio offers AR, where our tools and technologies are at in terms of supporting AR Audio, and what creative practise we need to develop to deliver the ‘ultimate’ AR audio experience.</p><p><strong>Why care about audio in AR?</strong></p><p>Placing an object in space is about way more than just making sure it’s sitting on the &lt;insert random surface of your choosing here&gt; or lit correctly. Visually, these are excellent starting points. Considering lighting alone, shadows and the luminosity of an object are important for that object visually belonging to the space. However, these elements together can still give the effect of a 3D hologram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*58AgIjaLvmkkrcdCZK45QQ.jpeg" /><figcaption>Is the universe a hologram? Not the question of the day, but an interesting one nonetheless. <a href="https://3c1703fe8d.site.internapcdn.net/newman/gfx/news/hires/2015/5-istheunivers.jpg">Source</a>.</figcaption></figure><p>Culturally, we have a vast number of examples and blueprints for holograms. From the earliest research into holography done by Dennis Gabor in the 1940s to Star Wars and the latest Blade Runner, Holography has given us enough touch points to easily round AR down to the simple expressions we’ve seen in early Sci-fi movies. For AR to stick, the impact of the object being projected needs to be so significant that it suspends disbelief immediately.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gKq9Ey_i-S36gtSHbVdGSw.jpeg" /><figcaption>Leia Appears as a Hologram appealing for help in Star Wars IV. <a href="https://i.stack.imgur.com/tGQjG.jpg">Source</a>.</figcaption></figure><p>Our sense of hearing is something that is processed subconsciously and it is widely accepted that it is processed faster than the visual stimulus. The brain is also incredibly fast at determining whether content “suits” the “tone” of what is happening in media that we consume (there has been a vast amount of exploration into this area done in the area of cognitive neuroscience). Having no audio in AR can break immersion (by defying the participant’s expectations of the object interacting with the space) and lead to a faster ‘put down’ time.</p><p><strong>Delving Deeper</strong></p><p>Beyond the initial benefit to the general idea of ‘immersion’, the ‘uses’ of audio within AR technology are many and varied. There are Cognitive, User Experience and User Interface benefits to utilising audio more deeply in AR.</p><p>Generally, use of a greater variety of modalities (sight, haptic, sound etc) for presenting information more generally means that more deliberate choices can be made about how the information is presented, as opposed to defaulting to a visual representation. Avoiding cognitive overload of a user can be as simple as presenting information in a non-visual modality like sound.</p><p>We can consider specific cognitive benefits that have been proved with research (and even evidenced across other entertainment technologies such as games and VR). Including audio in moving cues (eg, skill based or shooting games) can reduce reaction time up to 50% (Barde, Ward, Helton &amp; Billinghurst (2016)). This is why games like Raw Data and Space Pirate Trainer design enemies to emit sounds as they travel through space — whether it’s a whirring engine, a shuffling footstep or smooth servo sound, these cues help you understand the world around you before you have the time to turn your head in the direction the sound is coming from.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*mKQuZrrcPg5Bn3JRiGM1BQ.png" /><figcaption>Space Pirate Trainer: each of these bots emit a sound to update the listener with their location. <a href="http://www.develop-online.net/cimages/f24ede30862a2d6234a134e18b093ea3.png">Source</a>.</figcaption></figure><p>For spoken words, properly positioning (spatialising) sounds increases comprehension and perceived quality of experience in settings that include more than one voice (or sound). Experiences that do not spatialise sound (ie, just use stereo or even mono sound) actually ‘compact’ all the sound into a condensed field, which can make it extremely difficult to hear separate sounds, and the locations of those sounds (particularly in a 3D field).</p><p><strong>AR Hardware ‘Flavours’</strong></p><p>We’ve mostly discussed Mobile AR — AR that you can view with a smart phone. There are a few other types of AR available today, and each have subtle differences when it comes to Audio technology and design.</p><p><strong>Mobile AR </strong>is the ‘smartphone as a window’ execution of AR. This is the style of AR that we have seen developing over the last few years with Google’s Project Tango, Vuforia and Pokemon Go as key early examples. This style of AR has the lowest barrier to entry, but has interesting implications on the perceived location of sounds in the AR space because the device only tracks the device’s position, and not the position of your ears (which is where you expect to hear the sound).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ac3aiSBunSjX3W4kC09azA.jpeg" /><figcaption>Mobile AR using a Tablet. <a href="https://cdn-images-1.medium.com/max/2000/1*HTFLrOCm82wpuNwl6qqsig.jpeg">Source</a>.</figcaption></figure><p><strong>Device-Holstered AR</strong> like the Mira AR platform requires you to insert your mobile device (Apple iPhone only in the case of Mira) into a head-mounted holster that allows for images from the phone to be reflected against a clear display. This style of AR allows for the use of a controller to interact with the objects in the AR scene, and could potentially benefit much more deeply from audio contextualisation as images are more transparent in this style of display.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0OuJXGmKmLSF8C3tEW4lFg.jpeg" /><figcaption>The Prism is the first headset from Mira, allowing users to holster their mobile phone for a shared AR experience. <a href="https://www.ireviews.com/wp-content/uploads/2017/07/iReviews-mira-prism-augmented-reality_2.jpg">Source</a>.</figcaption></figure><p><strong>Head Mounted AR</strong> displays like the Microsoft Hololens, VRVana Totem (which has just been acquired by Apple), Meta and highly anticipated Magic Leap headset are in various stages of existence, and hence hold the most promise for both AR and Audio for AR as well. Designed from the ground up to be devices of the future, these devices often have room-detecting sensors that can feed critical information about the space the device is in for acoustical processing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*kFU0S1u7yADASWzHT8EpuQ.jpeg" /><figcaption>Hololens, with a generated Windows MR ‘Hologram’. <a href="https://www.microsoft.com/en-us/hololens">Source</a>.</figcaption></figure><h3><strong>Hardware Platform Idiosyncrasies</strong></h3><p>Generally, across this spectrum of devices, there are a few critical differences that impact the role of audio. These can be explored in a few categories — Field of View (FOV, Visual), Interaction Styles, Head/Ear position, Processing Limitations, Room Awareness and Potential for Peripherals.</p><p><em>FOV</em></p><p>Field of View is a major limitation of most AR technologies at the moment. Using a mobile device, the field of view is limited to the extent of the screen you are using. For some iPhones, that can be as small as a 4” window into a 360 degree world. Even for the Hololens, the field of view offered is only 35 degrees of the total 360 degrees available. The Totem (unreleased) has a FOV of unto 120 degrees, but it uses passthrough cameras (which results in a different experience for the user).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*2uUMN4sPGqJdyr08Q9JC1Q.gif" /><figcaption>Meta vs Hololens FOV demonstrations: Meta boasts 90 degrees, whereas Hololens only has 35 degrees. <a href="http://www.giganti.co/meta">Source</a>.</figcaption></figure><p>Why is the FOV important for a user? Humans are extremely biased by the visual world they see around them. With a FOV that is reduced from the normal perceptual FOV, it is even less likely that a user will look around just on the off chance that there might be more of the experience happening behind them. Audio is the body’s only 360 degree sense, and hence can be used to help cue the player to turn around to consume more of the total experience (as we discussed earlier).</p><p><em>Interactions across AR hardware</em></p><p>This array of hardware platforms all offer the user a different way that they can interact with the AR scene around them. Mobile AR offers the most rudimentary method of interaction (so far — this field is still nascent) where, if you can see it on the screen, you can also use that screen to interact with it.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fgc5tL6ASlww%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dgc5tL6ASlww&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fgc5tL6ASlww%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/478755cdef8a781668b425090679a9b4/href">https://medium.com/media/478755cdef8a781668b425090679a9b4/href</a></iframe><p>This disconnected method of interaction actually relies very strongly on audio to communicate several elements of the interaction — whether the interaction happened, how ‘intense’ it was, whether it achieved an objective (or not). Complex interactions (as opposed to taps, clicks, turns, etc) need complex modelling of the audio to create the relationship between the user and the object (answering subconscious questions like ‘How big is the object?’, ‘How dense is it?’, ‘How big am I?’, ‘Is this a desired interaction?’).</p><p>In some ways, even controller based interactions (like the Mira controller, which is similar to other VR controllers like the Daydream, Gear and incoming HTC Focus) still have a significant level of abstraction between the player’s input and the interaction with the AR object and hence require a similar level of attention to detail in crafting the interaction sounds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ejWnjKNM8sYfMi9jPuKx7Q.jpeg" /><figcaption>The Mira Prism controller — very similar to the Daydream controller, and many others. <a href="https://www.mirareality.com/">Source</a>.</figcaption></figure><p>The advantage that the Hololens offers is the device’s forward facing sensors that allow the user to directly interact with objects with their own hands. For anyone that tried the Hololens briefly, they probably found learning the gestures to be quite the mental leap (granted, I still can’t use an Xbox controller), which presents an opportunity for audio to help in the education and feedback processes when a user is adapting to new gesture-based interaction schemes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*Vun-zxkGuU9mGM-M0Vzi3g.jpeg" /><figcaption>Hololens gesture — the ‘Air Tap’ for selection. <a href="https://developer.microsoft.com/en-us/windows/mixed-reality/gestures">Source</a>.</figcaption></figure><h4><em>Audio Related Idiosyncrasies</em></h4><p>AR technologies also impose interesting effects on audio processing and consumption in general. I alluded to one effect previously — the location of the ‘window’ into the AR scene is different to the actual location of your head/ears. We also come across interesting cross-platform and platform specific effects. Different devices will have different capacities for space detection, overall processing limits will be different per device, and the presence and position of various peripherals (headphones, speakers) also impact on the audio processing pipeline.</p><p><em>Ear vs Device Position</em></p><p>One of the most prevalent challenges of mobile AR is that the mobile device that is your ‘window’ into the AR world will be held at a different location than the user’s ears are located. This may seem like a simple “yeah so what people won’t notice” if you assume that different in distance and angle is small. However, the average human arm is 25 inches (63.5 cm) long (we understand that humans don’t hold their phones at full extension when they are using them), which is enough distance to have a significant impact on the quality of the sound within the range of the near field Head Related Transfer Function (HRTF) (this HRTF is ‘used’ when the stimulus is within 1 meter/39 inches).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*xWyZ1mx3bCGlkuVYcmudIQ.png" /><figcaption>Demonstrated holding position of a mobile AR device. <a href="https://t3.ftcdn.net/jpg/01/06/36/28/240_F_106362856_WLTY3Eju8n9pbpFWfcYLTknZyXhXweAI.jpg">Edited Source</a>.</figcaption></figure><p>Untrained listeners might not exactly understand why sound ‘sounds weird’, but they are likely to notice that a sound source positioned below them is suddenly in line with their ear-level. Working around this is more complicated than you might assume. You can estimate the distance and angle that the device would be from the ear position and offset for this, but you will be making assumptions about the anatomy of the listener, which will skew their perception of the scene you are trying to create, and you might miss changes in these angles if the user moved at all.</p><p><em>Processing Limitations</em></p><p>Current AR devices have three categories of hardware recruitment — mobile device, all-in-one and tethered devices. Despite ongoing improvements on all range of hardware products, it is worth highlighting that AR experiences that recruit a mobile device to power their experiences will experience processing limitations. A quick comparison of the top of the line iPhone X and Pixel 2 XL with the Microsoft Hololens against a the VR-ready minimum spec defined by HTC Vive demonstrates the gap in technology.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_tUQej-gQIJmm_jTWWz40A.png" /><figcaption><a href="https://store.google.com/gb/product/pixel_2_specs">Pixel Stats Source</a>, <a href="https://ark.intel.com/products/series/88393/6th-Generation-Intel-Core-i5-Processors#@desktop">Intel Processors Stats</a> Source, <a href="https://www.vive.com/us/ready/">HTC Vive Spec Source</a>, <a href="https://www.gsmarena.com/apple_iphone_x-8858.php">iPhone X Stats Source</a>, <a href="http://www.samsung.com/global/galaxy/galaxy-s8/specs/">Samsung Galaxy Stats Source</a>, <a href="https://developer.microsoft.com/en-us/windows/mixed-reality/hololens_hardware_details">Hololens Specs</a></figcaption></figure><p>The difference between the Desktop minimum spec and the specs of the (current) top of the line mobile devices are significant (although it’s worth noting that the current top of the line MacBook Pros don’t meet this minimum spec either). The difference between the Hololens processing specs and the Vive minimum spec are drastic as well, which is entirely ok for a first generation technology that offers potential that is unique as an untethered headworn AR device.</p><p>This processing power gap effectively means that processing capabilities of these devices are limited (which we are used to considering between PC and mobile game development). This means less audio objects and less potential for Room Awareness and rendering on these devices. (Room Awareness is the ability of the device to be ‘aware’ of the space it’s ‘seeing’ — whether that’s by capture or scanning through one or a number of sensors).</p><p><em>Room Awareness and Rendering</em></p><p>I consider the Holy Grail of Audio for Augmented Reality the ability to acoustically conform the sound in the Experience to the environment that the device is placing it in. This particular element will be likely to have different potential and restrictions between the different AR hardware types, as well as timelines and pipelines for achieving this ‘Holy Grail’.</p><p>All AR technologies currently available have restrictions on the FOV that they have with their ability to be aware of the room that the device is in. Whether this is through inside-out tracking or cameras with a double-camera array capable of depth perception. Mobile AR is able to detect surfaces and allow movement around AR objects with the double-camera array that is included on iPhone X and AR-ready iPads (2017). Hololens, Meta and other AR headsets are able to detect the extent and depth of a room. Lower end mobile devices with only one camera are still able to detect surfaces, but do not have the same level of depth detection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VKcgEQzYy1dJxLl6nHrPeQ.png" /><figcaption>ARCore Surface detection. <a href="https://i.kinja-img.com/gawker-media/image/upload/s--PmD13JTg--/c_scale,f_auto,fl_progressive,q_80,w_800/fjnz3jubvyblxdnuf8vs.gif">Source</a>.</figcaption></figure><p>vs</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*3T08fqp1jSX2r9KXbwC6EQ.png" /><figcaption>Hololens Room Scanning. <a href="https://tctechcrunch2011.files.wordpress.com/2016/03/1_windowsholographic-1024x602.png">Source</a>.</figcaption></figure><p>To unlock deeper acoustical rendering on Mobile devices, there may need to be some capture or calibration of the space to use this data for advanced acoustical rendering (similar to how a user must calibrate a VR setup to ensure the external cameras ‘know’ where the headset is in space). Whether this type of calibration can be stored on a mobile device to used for processing, and then whether a real-time geometry based rendering algorithm can be executed on non-tethered devices is a question that we may be able to answer in the coming few years. Currently, physical rendering is only possible with a small number of software spatialisers (such as SteamAudio and NVIDIA).</p><p><em>Potential for Peripherals</em></p><p>Each category of device also offers a different range of audio experience for the end user. This is due to both the in-built hardware offered for playback, and the potential for different peripherals to be connected to the devices. Users trying ARkit or ARcore games or apps might not connect headphones and end up with a mono or stereo down mix of the audio from your experience. The same is true for AR hardware that holsters a mobile device (like Mira).</p><p>Other devices like the Meta and Meta 2 have speakers on the device but <a href="http://devcenter.metavision.com/develop/unity-sdk-features/audio-features/meta-audio">limit the audio output through its device to ‘Quadraphonic’</a>. The Hololens also has speakers on the device, but is capable of full Object based audio. These devices also have an analog jack connection for headphones, but the form of the device may limit the style of headphones that can be used with the device.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ePIifsr4SOQHs0qkRU-Twg.jpeg" /><figcaption>Headphone-related real estate is tight around the Hololens (and other hard-band headmounted devices). Image Credit: Becca Little.</figcaption></figure><h3>The Wrap</h3><p>This ‘technology review’ article is quite literally just the beginning. ARkit and ARcore (the Apple and Google DKs for creating Mobile AR experiences) have been in the wild for only 2-3 months at this stage, with a limited number of AR-capable mobile devices available to experience those apps and games on. Understanding the difference between each technology and what audio considerations exist surrounding each format is the first step to being able to create for this new format.</p><p>In the next article, we will delve deeper into how the technological and research-related details surrounding AR feed into potential creative practise. We will also look more deeply into what technologies are available, or may need to be developed to design audio for AR. We’ll also look into the potential Audio of for AR and what applications of these techniques and technologies may precipitate in the future.</p><p>Sally Kellaway is an industry leading voice for Audio in Virtual, Augmented and Mixed Reality. Her mission is to create, push boundaries and help educate her industries. You can reach her on <a href="https://twitter.com/soundsbysal?lang=en">Twitter</a> or <a href="https://www.linkedin.com/in/sally-anne-kellaway-32705010/">Linkedin</a>. Supporting Sal through the <a href="http://ko-fi.com/soundsbysal">Kofi</a> platform enables her to create more content!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8cde2a352712" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>