<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mark Nguyen on Medium]]></title>
        <description><![CDATA[Stories by Mark Nguyen on Medium]]></description>
        <link>https://medium.com/@the.design.engineer?source=rss-184a1f9fefcb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*lQZQfx3_kfQOpxce2Z1r-A.jpeg</url>
            <title>Stories by Mark Nguyen on Medium</title>
            <link>https://medium.com/@the.design.engineer?source=rss-184a1f9fefcb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 18:30:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@the.design.engineer/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How Do Stencil Shaders Work?]]></title>
            <link>https://medium.com/@the.design.engineer/how-do-stencil-shaders-work-e2d46b27b737?source=rss-184a1f9fefcb------2</link>
            <guid isPermaLink="false">https://medium.com/p/e2d46b27b737</guid>
            <category><![CDATA[graphics]]></category>
            <category><![CDATA[unity]]></category>
            <category><![CDATA[shader-in-unity]]></category>
            <category><![CDATA[shaders]]></category>
            <category><![CDATA[game-development]]></category>
            <dc:creator><![CDATA[Mark Nguyen]]></dc:creator>
            <pubDate>Mon, 19 Sep 2022 15:56:48 GMT</pubDate>
            <atom:updated>2022-09-19T15:56:48.197Z</atom:updated>
            <content:encoded><![CDATA[<p>A stencil shader is used to mask parts of 3D geometry. With 3D geometry masked to a certain area, we can achieve some pretty interesting visual effects, like the one below.</p><p>In this article I’ll explain how they work using Unity as a vehicle for explaining. The concepts work for other game engines and graphics libraries too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6UDLEkgDQV1EETxirCTgSQ.gif" /><figcaption>Fig 1. Circle Mesh acts as the stencil mask and the cube as the stenciled object</figcaption></figure><p>It’s usage ranges from video games to VFX in movies, typically for portals in to different worlds. It’s no surprise that the game, Portal, makes use of stencil shaders for its main game mechanic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*2briq6rjcZBTBCX5X0wKBQ.png" /><figcaption>Portal by Valve</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Hn9ZXQRq5PjT8-X4" /><figcaption>Doctor Strange by Marvel</figcaption></figure><p>The stencil effect requires two shaders: one for <em>writing </em>to the stencil buffer and the other for <em>reading </em>from it. We’ll label these <em>StencilMask</em> and <em>StenciledObject </em>respectively.</p><h3>1. The StencilMask shader</h3><p>This shader can be applied to the geometry that you wish to act as the mask — in fig1, the mask in the circular disc shape.</p><p>Let’s freeze the example to a single frame:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*o06o000F6UpaRtwW" /></figure><p>In the above scene we have one circle mesh and a cube mesh. We want our circle mesh to write to the stencil buffer with the pixels that it covers on the screen. Sounds complex, but it’s actually 2 lines of code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zGOkvf5W-eTzlgezpwLLbA.png" /></figure><p>Whilst we can’t inspect the stencil buffer, I can draw an artist’ impression of what is happening:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KmsLvkZ3Lg8qcWGapkihnw.png" /></figure><p>The stencil buffer looks awfully like a mask texture… and that’s because it is! This buffer will update every frame depending on the position/orientation of the circle and the camera.</p><h3>2. The Stencil Object shader</h3><p>This shader is applied to the <strong>cube object </strong>(or any objects you wish to render inside the afore created mask.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4H59kEGC5aY0A_rM2TQgVA.png" /></figure><p>In the above image, the cube is asking the stencil buffer, which pixels have been denoted by the circle mesh mask? If they contain the reference value or 1, then let’s draw, otherwise don’t draw. This behaviour is defined in the Comp (compare) parameter. Again, sounds difficult to implement but it’s just 2 lines of code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SbAlnlYstLtKOl3ZoN8rag.png" /></figure><p>We’ve defined Comp Equal meaning “render pixels when their reference value is equal to the current value in the stencil buffer”. We could have set it to Comp NEqual to flip the effect:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o-mHsw44qd310JdvkbrSUg.png" /></figure><p>More Comp values can be viewed here: <a href="https://docs.unity3d.com/Manual/SL-Stencil.html#comparison-operation-values">https://docs.unity3d.com/Manual/SL-Stencil.html#comparison-operation-values</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e2d46b27b737" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unity Stencil Shaders are perfect for making out-of-this world AR experiences]]></title>
            <link>https://medium.com/@the.design.engineer/unity-stencil-shaders-are-perfect-for-making-out-of-this-world-ar-experiences-cfd9679404ca?source=rss-184a1f9fefcb------2</link>
            <guid isPermaLink="false">https://medium.com/p/cfd9679404ca</guid>
            <category><![CDATA[shaders]]></category>
            <category><![CDATA[unity]]></category>
            <category><![CDATA[unity-game-development]]></category>
            <category><![CDATA[graphics]]></category>
            <category><![CDATA[tutorial]]></category>
            <dc:creator><![CDATA[Mark Nguyen]]></dc:creator>
            <pubDate>Mon, 19 Sep 2022 14:34:04 GMT</pubDate>
            <atom:updated>2022-09-19T14:34:04.368Z</atom:updated>
            <content:encoded><![CDATA[<p>In this tutorial I’ll show you how to set up stencil shaders and ARFoundation to create a visual effect akin to a portal in to parallel dimension.</p><p>With Stencil Shaders + AR you can make some pretty wild scenes:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplayer.vimeo.com%2Fvideo%2F751246169%3Fh%3D0c503d8a97%26app_id%3D122963&amp;dntp=1&amp;display_name=Vimeo&amp;url=https%3A%2F%2Fvimeo.com%2F751246169&amp;image=https%3A%2F%2Fi.vimeocdn.com%2Fvideo%2F1510371207-b077e8f891b3ca7e0a5d95b7881d7cb297d8ed566bbc429e5e15fbaf1e21d792-d_1280&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=vimeo" width="1920" height="1080" frameborder="0" scrolling="no"><a href="https://medium.com/media/e188ad2685c7336ac3ff2248ba5ae4f6/href">https://medium.com/media/e188ad2685c7336ac3ff2248ba5ae4f6/href</a></iframe><h4>What’s a <em>mask?</em></h4><p>In the context of computer graphics, a mask is used to show specific areas of an image and to hide others. The mask itself is usually not visible — instead the contents that is is <em>masking</em> can be shown and unmasked areas not shown, and vice-versa</p><h3>0. Prerequisites</h3><ol><li>Download and install Unity 2020.3.* via the Unity Hub. Where * = latest version available</li><li>Create a new project using the AR template in Unity Hub</li><li>Edit &gt; Project Settings &gt; XR Plug-in Management &gt; Enable ARCore or ARKit (depending on target platform being Android or iOS)</li><li>Create a new scene and add an ARSession and ARSessionOriginvia the GameObject &gt; XR menu</li><li>Reset the MainCamera’s transform so that its position is 0, 0, 0</li></ol><h3>1. Create the <em>StencilledObject </em>shader</h3><p>This is the shader that will only render if it is behind a <em>stencil mask</em> (we’ll write this shader in step 2).</p><ol><li>In your project, right click &gt; Create &gt; Shader &gt; Standard Surface Shader. Name this file “StencilledObject”</li><li>Modify this shader with the following (see figure below)<br><em>Lines 10 and 11: </em>Define a property containing the compare function — The compare function controls how we perform the test on the stencil shader. Typically we want this to be “equal”, meaning this shader will <em>pass</em> (i.e render pixels) when there is a corresponding value defined by <em>Ref</em> in the stencil buffer. (described below)<br> <br><em>Lines 17 - 21: </em>Here, we define our stencil operation. <em>Ref</em> is the reference value and we can view this as the channel that we are reading from/writing to. This value can range from 0 to 255.<br>That’s it for this shader!</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/0*nTPcxQwBy5FfyKBi" /></figure><p>3. Create a new material (Asset &gt; Material) and assign the Custom/StencilledObject shader that we just created. Name it “StencilledObject”</p><p>4. Create a Cube with the following properties:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*nLQTexs6bZV51HlH7uUw1g.png" /></figure><p>Change the Stencil Comp value to Equal(meaning only render if the stencil buffer at the current pixel is <em>equal</em> to the <em>Ref.</em></p><p>You’ll notice that the cube has disappeared — we’ve told the renderer to only render this object if there’s a value in the stencil buffer that is <em>equal</em> to the Ref value. However, our stencil buffer is currently empty, so let’s write another simple shader to do just that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/0*iPzeGAYML3rLj9Lf" /></figure><h4>What’s a stencil buffer?</h4><p>Essentially an image that is created every frame, with the same dimensions as the final output image, that can be written to, or read from. Useful for excluding certain parts of your scene behind a mask!</p><h3>2. Create the <em>Stencil Mask </em>shader</h3><p>With this shader we can make <em>any</em> object a mask for 3D content</p><ol><li>Asset &gt; Create &gt; Unlit Shader</li><li>Modify the shader by first adding:<br>1. &quot;Queue&quot;=&quot;Geometry-1&quot; This tells Unity to draw this object before Geometry (so that we can write to the stencil buffer before another object reads from it)<br>2. Blend Zero One. Instructs the graphics card to not draw this object and instead draw the background. More info can be found <a href="https://docs.unity3d.com/Manual/SL-Blend.html">here</a><br>3. ZWrite Off Disable writing to the depth buffer so that objects behind this one won’t fail the depth test i.e will render<br>4.Ref 1 operate on the stencil buffer with reference value 1 (same as our StencilledObject shader)<br>5.Comp Always Always render (because we are the mask, so nothing should be masking it) <br>6.Pass Replace when we pass (which is always), write the reference value (1) in to the stencil buffer</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*33wBCiTXu9uy6puUPmltLg.png" /></figure><p>3. Create a sphere, name it “Sphere Stencil Mask” with the following transform parameters:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*HLmtsoqg6rXqJyS74qm2RA.png" /></figure><p>4. Create a new material named “SphereStencilMask” and assign the “Tutorial/Stencil Mask” shader to it.</p><p>At this point both our cube and sphere won’t be visible in the scene (that’s because our stencil mask doesn’t draw anything and our cube is likely failing the stencil test). In the scene view, modify the render mode from “Shaded” to “Shaded Wireframe”. Our sphere mask and stencilled cube are now visible via their wireframes. Move around the scene to view the stencil working as a mask to occlude certain parts of the cube.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/0*1s7mi1VGcBBsDRmV" /></figure><p>That’s it! To add more content besides the cube to your “parallel dimension”, create a new material and assign the <em>StencilledObject </em>shader to it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cfd9679404ca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Improve your Unity AR Apps with These 4 Techniques]]></title>
            <link>https://medium.com/@the.design.engineer/improve-your-unity-ar-apps-with-these-4-techniques-c0f47a098a94?source=rss-184a1f9fefcb------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0f47a098a94</guid>
            <category><![CDATA[unity]]></category>
            <category><![CDATA[unity3d]]></category>
            <category><![CDATA[augmented-reality]]></category>
            <dc:creator><![CDATA[Mark Nguyen]]></dc:creator>
            <pubDate>Mon, 15 Aug 2022 20:18:33 GMT</pubDate>
            <atom:updated>2022-08-15T20:18:33.574Z</atom:updated>
            <content:encoded><![CDATA[<h3>Overview</h3><p>Let’s discuss how we can leverage some of the awesome built-in capabilities of Unity AR Foundation to improve the look and feel of your AR apps. Best of all is that these techniques can be applied in a matter of minutes.</p><p>We’ll go from this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*dehFM-IVr3_Bt7cqm1cm1A.jpeg" /><figcaption>Default Unity AR Template.. floating in space cat does little for realism</figcaption></figure><p>To this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*fVMAugnpWWCY0O38ORU-jA.png" /><figcaption>Shadows, occlusion, environment lighting and placement on the ground. Much better</figcaption></figure><h3>0. Project Setup</h3><p>Skip this step if you have already setup a Unity AR project.</p><p>The quickest way to get started is to use the Unity Hub:</p><ol><li>Select “New Project” and under templates select “AR”</li><li>Open project settings &gt; XR Plug-in management &gt; enable AR Core or AR Kit, depending on your target device.</li></ol><h3>1. Placement on the ground or table using the Plane Manager</h3><p>Whilst incredibly adorable, our cat floating in space does little to add realism, let’s take his anti-gravity boots off and place him on the floor. Remember, your phone has no concept of where the floor is — all it sees is a flat image made of lots of pixels. Your brain adds in the detail of depth, space, perspective etc.</p><ol><li>Add an ARPlaneManager component to the AR Session Origin game object</li><li>Now we need a custom script where we can r<em>aycast </em>on the ground plane. If a raycast is successful then we obtain the position and rotation at which the raycast hit the ground plane.</li></ol><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%2Fn5IozVnCAOD9EVmM8Tub%3F&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2Fn5IozVnCAOD9EVmM8Tub&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5972e6d052987853593a608dfd7945ef/href">https://medium.com/media/5972e6d052987853593a608dfd7945ef/href</a></iframe><p>3. Download the full script, ARPlanePlacement.cs here: <a href="https://gist.github.com/canxerian/a9ecd20abba2257bb078331d23842128">https://gist.github.com/canxerian/a9ecd20abba2257bb078331d23842128</a></p><h3>2. Shadows</h3><p>Here’s what we’ll achieve:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/499/0*xiZpkdAmnUOb6B_E" /><figcaption>Using a plane to render the cat’s shadow</figcaption></figure><ol><li>Create a plane and place in the same hierarchy as your 3D model</li><li>Create a new material and assign it to the plane</li><li>Update the shadow settings on your directional light</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*pMCAYfSC8WSNnfj2lFg6MQ.gif" /></figure><p>We have shadows! One problem.. the plane is also visible. To render the shadows <em>only, </em>we need a shader that outputs no colour for pixels in the shadow map that aren’t shadows. Luckily Unity has a shader that does exactly this.</p><ol><li>Download <a href="https://github.com/Unity-Technologies/arfoundation-demos/blob/master/Assets/Common/Shaders/MobileARShadow.shader">https://github.com/Unity-Technologies/arfoundation-demos/blob/master/Assets/Common/Shaders/MobileARShadow.shader</a></li><li>Assign this shader to the material used on the plane</li></ol><h3>3. Lighting Estimation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*-amfDCRqb6ES9zjMt3Ej4A.png" /><figcaption>Light estimation enables us to mimic the main light source for shadow direction and colours</figcaption></figure><p>We can modify the directional light colour in the Editor and adjust it to match the lighting conditions of our real world environment.</p><p>However, ARFoundation provides a way to do this automatically, allowing us to use the app in different environments and still have convincing lighting. This is made available via process known as <a href="https://developers.google.com/ar/develop/lighting-estimation">Light Estimation</a>.</p><ol><li>Instruct our AR Camera to gather light estimation data:<br>AR Camera &gt; AR Camera Manager &gt; Light Estimation &gt; Everything</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/505/0*-3v9tF7d9wAwwruQ" /><figcaption>Make light estimation data available via scripting</figcaption></figure><p>2. Create a new script and access the estimated lighting information. You can use my script (link to Github Gist in captions):</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcarbon.now.sh%2Fembed%2FOADmlT24PyZVaCk5BS8S%3F&amp;display_name=Carbon&amp;url=https%3A%2F%2Fcarbon.now.sh%2FOADmlT24PyZVaCk5BS8S&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;scroll=auto&amp;schema=carbon" width="1024" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/7277fe32ca2b229c1618bb838b4d0420/href">https://medium.com/media/7277fe32ca2b229c1618bb838b4d0420/href</a></iframe><p>3. Attach this script to the directional light component and assign the references in the inspector:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/0*L_-jpaxve1LytRmG" /></figure><h3>4. AR Occlusion</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/0*KGYDyycYnANytt6T" /><figcaption>Occlusion lets you mask your content behind real world features such as furniture, adding to the realism of your app</figcaption></figure><p>Occlusion gives us the ability to hide/mask certain areas of a 3D object give the impression of depth. It works by using the <a href="https://developers.google.com/ar/develop/depth">approximated depth map </a>of the real world objects in the scene. Phones with LIDAR will yield better results. The above was taken on a Pixel 4.</p><ol><li>Add an AROcclusionManager component to the AR Camera</li><li>Modify settings:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/0*513d9CShOkV31jUm" /></figure><h3>Conclusion</h3><p>Unity’s ARFoundation contains a fantastic set of tools for quickly creating high quality AR content. As is common with Unity, it can take some time to discover the features that give good results. Experiment and see what is most suitable for your experiences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*j3wnSfQYdpcMR_FA81WSZw.gif" /><figcaption>The finished product</figcaption></figure><p>Join the discussion to shape future articles. I post regularly on all things regarding technology and productivity, with a bias on 3D real-time interactive programming and cutting edge devices.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0f47a098a94" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Discovered a Mental Switch for Entering the Flow State]]></title>
            <link>https://medium.com/@the.design.engineer/i-discovered-a-mental-switch-for-entering-the-flow-state-62493ad56371?source=rss-184a1f9fefcb------2</link>
            <guid isPermaLink="false">https://medium.com/p/62493ad56371</guid>
            <category><![CDATA[work]]></category>
            <category><![CDATA[flow]]></category>
            <category><![CDATA[productivity]]></category>
            <dc:creator><![CDATA[Mark Nguyen]]></dc:creator>
            <pubDate>Tue, 09 Aug 2022 10:17:59 GMT</pubDate>
            <atom:updated>2022-08-09T11:26:37.413Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*ozFEwi1XNeGpME6a4ON20w.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@kellysikkema?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Kelly Sikkema</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Real time</h3><p>Operating in <em>real time</em> is a guaranteed method for me to enter the flow state.</p><p>I operate in <em>real time</em> when:</p><ol><li>I give presentations</li><li>I perform live as a musician</li><li>I’m deep in conversation</li></ol><p>These scenarios force me to focus on the shortest path between idea and execution. Coincidentally that path has no room for dwelling on mistakes or the past. My audience holds me accountable and I am inspired to deliver. Being in an environment where every second matters sounds daunting, but it is paradoxically liberating.</p><p>In this mode I am a passenger of time and I have no control over the play head of life. There’s no pause, rewind or undo button. I cannot perform another take and I cannot edit my email for the 15th time. My options are limited to focusing all my energy on the endeavor at hand. This is my version of the flow state.</p><h3>Deferred time</h3><p>The opposite of real time is what I call <em>deferred time.</em></p><p>In this mode we have the ability to rewind time — the opportunity to take another photo, to edit our writing and to analyze. A very important mode to be in and a mode we are in for the majority of our lives. It’s still entirely possible to enter flow state but it’s much less frequent in my experience.</p><p>Deferred time is denoted by a lack of urgency, low risk, a near-unlimited margin for error and plenty of opportunity to procrastinate.</p><h3>How to operate in real time</h3><ol><li>Apply micro deadlines. For example, if you’re writing an article, time-box to 20 minutes</li><li>Be held accountable. Work alongside a colleague or a friend</li><li>Record your screen. Simulate an audience (even if that is only your future self). Seeing the recording length also serves as a reminder of how long you’re spending on a particular task</li><li>Live stream. Go one step further and get a real audience (if your work permits)!</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62493ad56371" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Aligning Realities: How to create XR Shared Experiences]]></title>
            <link>https://medium.com/@the.design.engineer/aligning-realities-how-to-create-xr-shared-experiences-2ace580b827b?source=rss-184a1f9fefcb------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ace580b827b</guid>
            <category><![CDATA[hololens]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[mixed-reality]]></category>
            <category><![CDATA[augmented-reality]]></category>
            <category><![CDATA[unity]]></category>
            <dc:creator><![CDATA[Mark Nguyen]]></dc:creator>
            <pubDate>Sat, 16 Jul 2022 20:16:27 GMT</pubDate>
            <atom:updated>2022-08-16T07:04:08.156Z</atom:updated>
            <content:encoded><![CDATA[<p>A good AR, VR or MR experience seamlessly integrates the real and virtual. There are several methods for achieving this, on a visual level at least. Some good resources can be found here: <a href="https://developers.google.com/ar/design/content/realism">https://developers.google.com/ar/design/content/realism</a> and <a href="https://developers.google.com/ar/develop/depth">https://developers.google.com/ar/develop/depth</a>.</p><p>Additional techniques are required however, to extend this seamless mixing of realities when we involve multiple co-located users, aka <em>shared experiences</em>. Once achieved, the feeling resulting from a virtual subject animating and responding to input is something I liken to magic.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplayer.vimeo.com%2Fvideo%2F730632236%3Fh%3D8bec700e60%26app_id%3D122963&amp;dntp=1&amp;display_name=Vimeo&amp;url=https%3A%2F%2Fvimeo.com%2F730632236&amp;image=https%3A%2F%2Fi.vimeocdn.com%2Fvideo%2F1470468953-618e79aece5a32dd632fdde9d0eea4701fd0d1c4fe6e8d8e068ce3c1d2e3b9c1-d_1280&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=vimeo" width="1920" height="1080" frameborder="0" scrolling="no"><a href="https://medium.com/media/5737500505be726b71a0d2f299dd551f/href">https://medium.com/media/5737500505be726b71a0d2f299dd551f/href</a></iframe><h3>Synchronising State and Spatial Alignment</h3><p>To achieve this, we as developers can broadly split our concerns in to two areas:</p><ol><li>synchronisation of <em>state</em></li><li><em>spatial alignment</em> of AR content as displayed on each user’s screen</li></ol><p>Synchronisation of state is something we experience when playing a multiplayer game, or collaborate on a cloud document. We can either use a central server or a p2p network for transmitting data and the client agrees on how to update animations, material properties and UI, amongst other things. Our concern in this area is focused around matchmaking, latency and editing collisions (i.e the <a href="https://medium.com/coinmonks/operational-transformations-as-an-algorithm-for-automatic-conflict-resolution-3bf8920ea447">Google Docs problem</a>).</p><p>Spatial alignment is a problem less solved but contributes a great deal to realism. The problem stems from each mobile device having its own “relative” coordinate system, as depicted below. For AR content to appear at the same physical location on multiple devices, we need a point of reference for devices to recognise and use as a common coordinate system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tLHAqm1DxQsFSDco" /><figcaption>Relative coordinate systems: two mobile phones in a room with their own coordinate systems. Photo by <a href="https://unsplash.com/@antenna?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Antenna</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a>, comp by Mark Nguyen</figcaption></figure><p>The establishment of an agreed upon point of reference is required for devices to spawn content in the same physical location.</p><p>Without a stable method of spatial alignment, AR content won’t appear in the same location, reducing the level of immersion. This is further exacerbated when we add movement and change points of views, as world tracking abilities vary greatly among devices.</p><p>I list the approaches I’ve taken in my own projects, and the pros and cons As with all technology, they have their limitations.</p><h3>Technique #1 — Fiducial marker</h3><p>We begin with the most straight-forward approach: place marker on a surface then use the phone’s camera to recognise its position in real-world space.</p><p>Types of markers commonly used:</p><ul><li>ArUco (OpenCV)</li><li>QR Code</li><li>Image target (also known as “markerless”)</li></ul><p>Each type of marker has its unique characteristics. In my experience ArUco tends to scan the fastest and give accurate position/orientation on HoloLens/Android/iOS using OpenCV. Image targets on the other hand, can be a better visual fit, though you’ll need to use image quality check tool, such as <a href="https://developers.google.com/ar/develop/augmented-images/arcoreimg">arcoreimg </a>tool, to get information on tracking quality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*loL1RBTx0rcb67vY" /><figcaption>ArUco markers are among many marker types that can be used to establish a common coordinate system</figcaption></figure><h4>When to use</h4><ul><li>Room scale, “scan and play” scenarios, e.g an art placard</li><li>When you want to visually direct users to a location where AR content is available</li><li>Indoor waypoint navigation</li></ul><p>As markers give you pose information, they are useful for creating a common coordinate system for multiple devices.</p><h4>Pros</h4><ul><li>Most accurate for obtaining a pose (compared with spatial anchors and VPS)</li><li>Quick setup — just place a marker where you want to base the XR experience</li><li>Ability to embed simple data (integer up to 2^n where n = ArUco resolution), these can be used as keys for larger data sets that exist either in the app or online</li><li>Can compensate for poor world tracking — older devices / newer devices have better success at aligning with each other.</li><li>Marker can be moved to move the experience</li></ul><h4>Cons</h4><ul><li>Requires marker dimensions to be known before scanning</li><li>Not visually discreet — the marker might look out of place in certain scenarios. In a museum exhibition, for example.</li></ul><h4>How to use them</h4><ol><li>In your Unity project, import ARFoundation and ARCore or ARKit (depending on your target device0</li><li>Add an AR Session Origin to your scene</li><li>Create an empty game object and add an AR Tracked Image Manager component</li><li>Create a Reference Library</li></ol><p><a href="https://docs.unity3d.com/Packages/com.unity.xr.arfoundation@4.0/manual/tracked-image-manager.html">AR tracked image manager</a></p><h3>Technique #2 — Cloud-based Anchors</h3><p>Anchor implementations differ from SDK to SDK, but are generally built on environmental understanding and SLAM. When you create an anchor, the visual features in the environment are used to position 3D content relatively. This makes anchors the preferred method of fixing content to a physical feature, such as a table or floor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ME9lJQWEgr8aXXzx" /><figcaption>Anchors rely on feature points in an image to create a point of reference in 3D space. Photo by <a href="https://unsplash.com/@kyoshireyes?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Kyoshi Reyes</a> on <a href="https://unsplash.com/?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a>, composition by Mark Nguyen</figcaption></figure><p>I am aware of two cloud anchors services: Google Cloud Anchors and Azure Spatial Anchors.</p><h4>When to use</h4><ul><li>Shared experiences across multiple platforms</li><li>Content that doesn’t require precise position and orientation of 3D content, such as a camera facing avatar and billboarded panels.</li><li>Context-aware overlays, such as looking at a shop store front and seeing additional information.</li><li>For experiences involving user generated content, such as a AR multiplayer games.</li></ul><h4>Pros</h4><ul><li>Easy to share and persist</li><li>Cross-platform (via the use of cloud-based services)</li><li>Dynamic — new anchors can be created and then used without developers</li></ul><h4>Cons</h4><ul><li>Requires anchors to be created before being used</li><li>Anchors can be difficult to retrieve, especially if the room layout of lighting setup changes</li><li>Requires UI to prompt user to look for an anchor</li></ul><h4>How to use them</h4><p><a href="https://docs.microsoft.com/en-gb/azure/spatial-anchors/quickstarts/get-started-unity-hololens?tabs=azure-portal">Quickstart: Create a HoloLens app with Unity - Azure Spatial Anchors</a></p><h3>Technique #3 — Visual Positioning System</h3><p>This technique is built on a combination of geospatial data and anchors. This is how Google Maps’ <a href="https://support.google.com/maps/answer/9332056?hl=en-GB&amp;co=GENIE.Platform%3DAndroid">Live View</a> calibrates itself.</p><p>The most popular providers are Niantic’s Ligthship and Google’s Geospatial API.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FBqO12Nbl_N4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DBqO12Nbl_N4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FBqO12Nbl_N4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/79ac39d3d02abc041c061941c27f8d1e/href">https://medium.com/media/79ac39d3d02abc041c061941c27f8d1e/href</a></iframe><h4>When to use</h4><ul><li>Namely, outdoor locations</li><li>Coarse location — content can appear within a few centimeters radius, for example.</li><li>Coarse latency — anchor discovery can take up to a few seconds</li><li>Pokemon GO type experiences</li></ul><h4>How to use</h4><p><a href="https://lightship.dev/docs/ardk/ardk_fundamentals/importing_ardk.html">https://lightship.dev/docs/ardk/ardk_fundamentals/importing_ardk.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ace580b827b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>