<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Haejin Lee on Medium]]></title>
        <description><![CDATA[Stories by Haejin Lee on Medium]]></description>
        <link>https://medium.com/@haejinlee?source=rss-73f688f078e7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*I5-EGvvuozBmRzfk.jpg</url>
            <title>Stories by Haejin Lee on Medium</title>
            <link>https://medium.com/@haejinlee?source=rss-73f688f078e7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 08:45:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@haejinlee/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Imagined Odyssey: Live mixed reality dance performance with 80 audience]]></title>
            <link>https://medium.com/@haejinlee/imagined-odyssey-live-mixed-reality-dance-performance-with-80-audience-f4fdc17b24ec?source=rss-73f688f078e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/f4fdc17b24ec</guid>
            <category><![CDATA[microsoft-design]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[mixed-reality]]></category>
            <category><![CDATA[shared-realities]]></category>
            <category><![CDATA[augmented-reality]]></category>
            <dc:creator><![CDATA[Haejin Lee]]></dc:creator>
            <pubDate>Fri, 27 Apr 2018 14:55:25 GMT</pubDate>
            <atom:updated>2018-04-28T19:49:24.404Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IxhT1wkCplHHaxZ81mhyJg.gif" /></figure><p>The Mather Dance Center in Case Western University campus built in 1907 is not necessarily the place I expect to encounter the first-of-its-kind experimental dance performance. Yet, I was there, on Nov 16 2017, surrounded by 80 audiences all wearing HoloLens to see a dance performance, “Imagined Odyssey”, fusing Holograms with dancer’s movement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uaq9EWux3ikgXo8_AjIIaQ.jpeg" /><figcaption>Inside of Mather Dance Theater studio / Credit: Case Western University</figcaption></figure><p>In “Imagined Odyssey”, dancers passed through holographic golden gate, their physical body movement generated holographic tornado and fiery trails, and mysterious trees grew from the floor. There was definitely room for improvement when it comes to 3D asset quality and how it complements the dance, yet I was still blown away by all the technical challenges they overcame including wirelessly networking 80 HoloLens for shared experience, developing custom IR based dancer tracking system that broadcasts position data to the apps on HoloLens, and creating custom carts to safely transport 80 HoloLens to dance theater.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9zs2poi7QznhThGScjBv3Q.jpeg" /><figcaption>Special container Interactive Commons made to carry 80 HoloLens</figcaption></figure><p>I also felt electrified to see group of 80 people ranging from young university students to senior citizens not just wearing HoloLens but seeing the same thing at the same time together. In this theater, seeing hologram was no more an isolated experience, but shared reality.</p><p>Lastly, I realized again how much mixing live theater with virtual content could be a powerful means to amplify realism in mixed reality experience.</p><blockquote><em>Because Holograms appeared not just in a random place, but in reaction to dancer’s position and gestures, they fused more tightly with the reality and added fresh light to dancer’s movement.</em></blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FarqO9vgS000%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DarqO9vgS000&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FarqO9vgS000%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/7b0a612ecd1f3581cfe1c725a3697c1b/href">https://medium.com/media/7b0a612ecd1f3581cfe1c725a3697c1b/href</a></iframe><h3>Cross pollination</h3><p>So how this cutting edge performance ended up happening in Case Western University? It all started with Mark Grisworld, a world renowned radiology researcher and professor at Case Western, visiting the basement of Microsoft building 92 to see the very first HoloLens demo on Nov 2014. He said the demo was a world shattering experience to the extent he could not stand properly after seeing it. The demo convinced him MR is the new medium Case Western should adopt, went back to the campus and built <a href="http://interactive-commons.webflow.io/">Interactive Commons</a>, a campus-wide incubator to apply cutting edge visualization technology to advance student learning and academic research.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*bIPJVNqddEPXr7q4HDRnAQ.jpeg" /><figcaption>Mark Grisworld(left) on Build 2016</figcaption></figure><p>Mark, as a radiology professor and a director of Interactive Commons, works at the intersection point between many different disciplines. He believes cross-pollination between different domains will create a leap in innovation and tries to apply the power of HoloLens not only to the field of medicine, but to business and fine arts.</p><p>Since Mark brought HoloLens to anatomy classes in 2015, many people in the campus has been intrigued by this ingenious device. Among those is Gary Galbraith, a professor of dance. Mark said Gary came to the classes many times just to observe people wearing HoloLens and they started to tinker together how HoloLens can bring change to dance performance. Through 2 years of exchanging different expertise and interests, “Imagined Odyssey” came to a life.</p><h3>Potential future</h3><p>On my way back to home in Seattle, I thought about what kind of more progress can be made to the dance performance using HoloLens. Then, I remembered seeing a video of “Riders of the storyboard” by DandyPunk showcased in New Frontier at Sundance 2017. Although it used a projection mapping, not augmented reality, it provided a tantalizing glimpse of what future of storytelling and performance could be like using Mixed Reality. Especially, I liked the illusion of the dancer <strong>interacting with 2D graphic novel character </strong>and <strong>turning virtual objects into real ones </strong>which added more realism to this illusion. Besides these, I think <strong>freeing audience from fixed seats and enabling them to move around the performers </strong>will help people quickly realize that holograms are actually placed in the world, around dancers and actually have dimensions.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FKtDmMt6kZU0%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DKtDmMt6kZU0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FKtDmMt6kZU0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5709cb54da06e85cc55e82a70baa9fae/href">https://medium.com/media/5709cb54da06e85cc55e82a70baa9fae/href</a></iframe><p>Mark Grisworld said, “Imagined Odyssey is the only beginning”. I look forward what kind of unique experiences will be shaped by people like Mark and Gary who crosses the boundaries of their disciplines and explore uncharted terrain.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4fdc17b24ec" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Coming Soon to a Mixed Reality Theatre Near You]]></title>
            <link>https://medium.com/microsoft-design/coming-soon-to-a-mixed-reality-theatre-near-you-b26f58e13c62?source=rss-73f688f078e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/b26f58e13c62</guid>
            <category><![CDATA[immersive]]></category>
            <category><![CDATA[mixed-reality]]></category>
            <category><![CDATA[storytelling]]></category>
            <category><![CDATA[virtual-reality]]></category>
            <category><![CDATA[film]]></category>
            <dc:creator><![CDATA[Haejin Lee]]></dc:creator>
            <pubDate>Tue, 18 Jul 2017 16:06:01 GMT</pubDate>
            <atom:updated>2019-08-27T15:54:48.858Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cYpEZyH0wTVMsjTYMuk5fQ.jpeg" /></figure><h4>Peering into the future of storytelling at Tribeca Immersive 2017</h4><p>It was a bright spring evening in New York, when the long shadows of the city stretched across a longer line of people on Varick street. I stood in the middle of that line, invitation in hand, awaiting the <a href="https://tribecafilm.com/immersive">Tribeca Immersive </a>exhibit that showcased stories made with emerging media. In the queue, I began to anticipate what sort of wonders might await inside this mundane grey brick building.</p><p>At 10 minutes past 7pm, the start-time of my 3 hour session ticket, the line started to move. After getting a wrist badge, taking an elevator to 5th floor and walking through a long white corridor bathed with neon purple ambient light, I finally arrived at the entrance where an erie green glow framed the black edges of an aluminum door.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8HTKAkZuHoij5EZiryiMEQ.jpeg" /></figure><p>“Sign up for the film you want to see. The list fills up quite quickly.” shouted the staff at the entrance. I then realized I had to rush to each venue to put my name on a waiting list, <em>and</em> that those lists were filling up quickly. By the time I got in, some of the experiences I really wanted to check out including “Draw me close” and “Tree” had already reached capacity. My heart went into FOMO overdrive as I scrambled to secure a few key spots among the 30 available offerings. While I was frustrated by the structure of this event, I still managed to experience 4 pieces that made my trip from Seattle to New York worthwhile.</p><h4>Spoiler alert: here’s what I saw</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7spvvb_bEMbbljIbAFHo5g.jpeg" /></figure><h3><strong>Life of Us</strong></h3><p>#Embodying beyond human # Social</p><p><em>Life of Us </em>is a story of evolution on Earth told by embodying a series of different characters. This is one of the pieces I was eager to try after listening to <a href="http://voicesofvr.com/504-life-of-us-an-embodied-social-story-of-human-evolution/">Voices of VR interview</a> with the creator, Aaron Koblin. The experience was a pure joy ride. I giggled, yelled, and laughed during the entire experience—so much so that my sides hurt.</p><blockquote><em>One of the key differences between VR storytelling and conventional flat cinema is VR’s ability to turn the audience into the protagonist through the use of an avatar — not just another human but sometimes another creature.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dkg5ai6S6hCbN7b7fJ1rEg.jpeg" /></figure><p>In “Life of Us”, I started my life as a single-celled organism, then evolved into tadpole, lizard, flying dinosaur, ape, human and finally a cyborg. It was ecstatic to breathe bubbles and fire through my mouth and control movement of massive flying dinosaur wings with my arms. Yet, because the story was fast paced and the user control was limited, I wasn’t able to truly feel ownership of each avatar I embodied. Still, I definitely felt like I was no longer normal me, but a different something, thanks in part to their ingenious voice modulation. My voice sounded like as if I inhaled a helium balloon. The funnier thing was that I tried to tweak my real voice to match the sound I heard as my voice — I was modifying myself to match the character I was enacting. I also spit out lot of giberish just to hear my new voice before the magic of helium balloon disappeared.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FRpyFs6O-328%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DRpyFs6O-328&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FRpyFs6O-328%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/aa82ccf5877beb6252256a25ca417d3a/href">https://medium.com/media/aa82ccf5877beb6252256a25ca417d3a/href</a></iframe><p>Embodying different virtual bodies that are quite different from our own is not a new concept. It was first developed by <a href="https://en.wikipedia.org/wiki/Jaron_Lanier">Jaron Lanier </a>when he was at VPL research in 1980&#39;s where he found out human can learn to control bodies that have extra limbs such as a lobster. He named our ability to do this “Homuncular Flexibility.” Also,<a href="https://vhil.stanford.edu/"> Stanford Virtual Human Interaction Lab </a>led by Jeremy Bailenson has been actively exploring this concept by making humans inhabit nonhuman characters such as a cow, a coral reef, and a bird to see whether it increases empathy for the creatures they are embodying. They also created a humanoid avatar with 3 arms just to see if it would increase efficiency for task performance. Yes. The future of our presence in virtual world may be slightly weird, but it is definately liberating and eye-opening.</p><p>Finally, I experienced “Life of us” along with my husband. We saw each other’s evolving bodies and communicated with each other throughout the story, and this social aspect significantly increased my sense of presence in the virtual space. Because the fictional world was shared with another who perceived and reacted to me, this world felt more real. I almost forgot that I was being watched by any other attendees at the festival.</p><h3><strong>Treehugger: WAWONA</strong></h3><p>#Invisible visible, #Sensory soup</p><p><a href="http://marshmallowlaserfeast.com/">Marshmellow Laser Feast </a>is one of the most inspiring creative agencies in the UK that tries to extend human perception by visualizing perspectives inaccessible to human senses. In 2015, they created <a href="https://vimeo.com/140057053">ITEOA(In the Eyes of Animal)</a>, which was an artistic interpretation of forest seen through 3 creatures, an owl, a dragonfly and a frog. In 2016, they started to showcase their newer piece, “Treehugger”, that visualizes tree and it’s vascular/circulatory system of flowing water, and it was one of Tribeca 2017 VR arcade pieces.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplayer.vimeo.com%2Fvideo%2F195539105%3Fapp_id%3D122963&amp;dntp=1&amp;display_name=Vimeo&amp;url=https%3A%2F%2Fvimeo.com%2F195539105&amp;image=https%3A%2F%2Fi.vimeocdn.com%2Fvideo%2F632511685_1280.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=vimeo" width="1920" height="1080" frameborder="0" scrolling="no"><a href="https://medium.com/media/eecce273a271ce3f22706589b7d5be33/href">https://medium.com/media/eecce273a271ce3f22706589b7d5be33/href</a></iframe><p>What distinguished the Treehugger booth from others was a giant black foam tree placed in the center of the space. The size and placement of this foam tree was aligned with the VR experience, so when I reached out for tree in VR, I received tactile feedback from the foam tree. This alignment of physical surface with the virtual tree tricked my brain further into thinking the virtual world is real. I believe in the near future this kind of setting will be adopted by commercial VR arcades or amusement parks to increase realism and provide more reason to experience things on site instead of home.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wmjqr9_906GL0wZt34DYaw.jpeg" /><figcaption>Touching the foam tree while wearing SubPac in the back</figcaption></figure><p>Besides this physical feedback, MFL used<a href="http://subpac.com/?gclid=CjwKEAjwvr3KBRD_i_Lz6cihrDASJADUkGCagF-dm0mPlc9ukKf6Exu9d6qVkpa-ZnwkA9nmVwx9WhoC-Y7w_wcB"> SubPac</a>, an audio device that transmits bass sound directly to the bones and muscles of the body to create more immersive VR experience. It was interesting to feel haptic feedback on my back, but I am unsure how much this helped me feel more immersed. Actually, when Subpac was very active, it broke my illusion because I was more aware of myself wearing this device in real life. I can see Subpac becoming more powerful for loud music concert simulation or FPS games where experiencing vibration is more natural and anticipated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YD-kCqKeJsMkA_JbK9YBzw.jpeg" /><figcaption>Tiny smell diffusing device under headset</figcaption></figure><p>Another interesting addition was smell. MLF attached a tiny device under headset that diffused the smell of the tree, forest, and rain throughout the experience. This made me feel more like I was in the forest. The first time I experienced the use of smell in VR was in Japan pavilion at <a href="http://vrexpo.com/">SVVR conference in 2017</a>. The small startup called <a href="https://vaqso.com/">VAQSO</a> created a short demo, and I was able to smell coffee when I held a coffee cup, chocolate when I picked up a chocolate bar, and (a bit weirdly) perfume when I got close to Sailor Moon-like manga character. I was surprised how I was able to smell the difference immediately after I picked up different objects and brought them closer to my nose. I think smell is very compelling component to increase immersion and I look forward to what’s next regarding this sense.</p><p>Despite all these additions, what created the strongest sense of presence was still visual feedback — clouds surrounding the Sequoia tree and particles indicating nutrients that changed movements or colors in reaction to my hand position. These particles made me feel like I was looking at the matter of this world on atomic level — the atoms that are in perpetual motion to maintain life, and I was mesmerized by its sheer beauty! Although, this experience is fantastic as a VR piece, the real potential is to visualize this invisible layer of a tree actually on top of a real tree using the same AR technology.</p><h3><strong>To be with Hamlet</strong></h3><p>#Mixed reality #Live Performance</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KRa4_eHfg4ZCIlxm-omfqg.jpeg" /><figcaption>(Left) A scene form VR<a href="http://www.gdcvault.com/play/1024639/-To-Be-With-Hamlet"> source </a>(Right) Live performers enacting Hamlet and the ghost in remote location<a href="http://engineering.nyu.edu/press-releases/2017/04/25/virtual-reality-hamlet-hits-stage-tribeca-film-festival"> source</a></figcaption></figure><p><a href="https://tribecafilm.com/filmguide/to-be-with-hamlet-2017">“To be with Hamlet</a>” was an experimental piece that fused VR and live theater to recreate theatrical intimacy in VR. The piece demonstrated a slice of Hamlet in VR, the scene where Hamlet meets the ghost of his murdered father, and used motion capture data from live performers in Brooklyn to control movement and voice of characters.</p><p>I tried the experience with 3 other people in a tiny dark room, and was able to see other audience members floating headset avatars as well as the actors. This was a bit like <a href="https://mckittrickhotel.com/sleep-no-more/">Sleep No More</a>, an experimental site-specific theater in New York, in terms of being able to roam the theatrical environment with other audience members all wearing masks. Yet, different than Sleep No More, these actors were not able to see the audience because they weren’t in the same location and they were not wearing headsets. This limitation blocked interaction between audience and actors, and made me question what the difference is between recorded and live performance if performers cannot see who they are interacting with. I believe the addition of some minor interactions between audience and actors — like performers grabbing a hand in the audience to guide them to a certain place or a simple eye gaze exchange that creates an intimacy as demonstrated in Sleep No More — would add more liveliness to this experience.</p><p>Despite of this limitation, the idea of mixing VR with live theater and multiple audiences still sounds novel and refreshing. I heard ‘To be with Hamlet” is still in a very early stages of development, and I am excited to see how the piece will evolve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U0aPXIyXkC7jK_0AhYdgJQ.jpeg" /><figcaption>Draw me close <a href="https://uploadvr.com/draw-me-close-childhood/">image source</a></figcaption></figure><p>Besides “To be with Hamlet”, “<a href="https://tribecafilm.com/filmguide/draw-me-close-2017">Draw me close</a>” was another piece that involved live performance at Tribeca. Unfortunately, this piece was at capacity, but it still sounds fascinating. “Draw me close” is a memoir of creator Jordan Tannahill’s relationship with his mother in the wake of her terminal cancer diagnosis. In this piece, the audience transforms into the main actor, 5 year old Jordan, and the mother character is enacted by a live performer who is co-located with audience. The experience starts with mom (live performer) hugging Jordan (the audience), and then they do activities together like opening a window and drawing on the floor. Since the illustrated room in VR is aligned with the staged room, the user was able to get all the physical tactile feedback on top of real human touch. I’ve never heard anything like this before, and I really appreciate creators like Jordan Tannahill pushing boundaries what the new medium could offer.</p><h3>The protectors: walk in the ranger’s shoes</h3><p>#Guided story #Documentary</p><p>“<a href="https://tribecafilm.com/filmguide/protectors-walk-in-the-rangers-shoes-2017">The Protectors</a>” is a story about a group of rangers protecting endangered elephants in the National park of the Congo. Through this film, my awareness of the dangers these African rangers and elephants face got immensely heightened. I learned that poachers are well armed not just for killing elephants, but rangers too, if needed. I found out later that <a href="http://news.nationalgeographic.com/2017/05/wildlife-watch-360-videos-the-protectors-garamba-poaching/">21 rangers were killed </a>during the last decade protecting elephants in the park. It’s estimated the park now has<a href="http://news.nationalgeographic.com/2017/05/wildlife-watch-360-videos-the-protectors-garamba-poaching/"> less than 2,000 elephants</a>, down from 20,000 in the 1960s. Although the resolution of video and text rendering were not the best quality, the story felt true and my heart ached when I virtually witnessed a carcass of an elephant killed by poachers. The full VR film is available <a href="https://www.youtube.com/watch?v=RuGeeGRdYlQ">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/1*hWm2pio_HyfxCszEvhwlHQ.jpeg" /><figcaption>Image from National Geographic / Photo by Ian Doss</figcaption></figure><p>VR affords audiences to form intimate connection with others’ situations by providing a firsthand immersive view of events. Because of this affordance, people have been calling VR an “Empathy Machine.” While I’m not a big fan of the term, I really do appreciate creators capturing a part of the world that is not easily accessible and that deserves people’s attention. I also have no doubt that VR is a powerful tool to transport viewers to the situation that film is depicting. Yet I think it is a bit simplistic to state that more immersive technology will automatically induce more empathy. No matter how immersive technology is, if the story is not well crafted and resonant with people, it will not cultivate deeper emotional engagement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nl8byy7RdewNWORPR3Mv3Q.jpeg" /><figcaption>“The protectors” arcade that set the scene before audience watching the film</figcaption></figure><h3>Back to reality</h3><p>10pm. My 3 hour ticket lost its magic, and I found myself back out on Varick street. Much different from the bustling inside, the street was dark, quiet, and slightly chilly — I was back to reality. Yet, my mind continued to replay the pieces I saw, wonder about the ones I didn’t get to see, and ponder the future of storytelling. Through Tribeca Immersive 2017, I saw a glimpse of storytelling using mixed reality, live performance, embodied social presence and multi-sensory output. What will Tribeca Immersive 2018 bring?</p><p>Perhaps stories using Artificial Intelligence as a character that guides the story and dynamically reacts to users? Or perhaps more interactive stories where the user has full agency beyond choosing a story path based on gaze? I am thrilled to see how artists will continue to push this new medium to reveal novel concepts and to create new ways of experiencing hidden, complex, and believable realities through storytelling.</p><p>I’d love to know what you thought of these projects, or what other stories you’ve seen in the immersive realm.</p><p><em>To stay in-the-know with </em><a href="http://www.microsoft.com/design?source=post_page---------------------------"><em>Microsoft Design</em></a><em>, follow us on </em><a href="https://dribbble.com/microsoft?source=post_page---------------------------"><em>Dribbble</em></a><em>, </em><a href="http://www.twitter.com/microsoftdesign?source=post_page---------------------------"><em>Twitter</em></a><em> and </em><a href="http://www.facebook.com/microsoftdesign?source=post_page---------------------------"><em>Facebook</em></a><em>, or join our </em><a href="https://insider.windows.com/?source=post_page---------------------------"><em>Windows Insider program</em></a><em>. And if you are interested in joining our team, head over to </em><a href="http://aka.ms/designcareers"><em>aka.ms/DesignCareers</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b26f58e13c62" width="1" height="1" alt=""><hr><p><a href="https://medium.com/microsoft-design/coming-soon-to-a-mixed-reality-theatre-near-you-b26f58e13c62">Coming Soon to a Mixed Reality Theatre Near You</a> was originally published in <a href="https://medium.com/microsoft-design">Microsoft Design</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Taking the Leap from 2D to 3D]]></title>
            <link>https://medium.com/microsoft-design/taking-the-leap-from-2d-to-3d-4199ccc777a1?source=rss-73f688f078e7------2</link>
            <guid isPermaLink="false">https://medium.com/p/4199ccc777a1</guid>
            <category><![CDATA[mixed-reality]]></category>
            <category><![CDATA[augmented-reality]]></category>
            <category><![CDATA[hololens]]></category>
            <category><![CDATA[virtual-reality]]></category>
            <category><![CDATA[ux]]></category>
            <dc:creator><![CDATA[Haejin Lee]]></dc:creator>
            <pubDate>Tue, 14 Feb 2017 18:14:09 GMT</pubDate>
            <atom:updated>2021-03-12T00:15:03.172Z</atom:updated>
            <content:encoded><![CDATA[<h4>Notes from my first year on the HoloLens design team</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mo4VpvU4zLTL9-jq6a4UAw.gif" /></figure><p>My journey from a 2D flatland to the 3D world started when I joined the HoloLens design team in January, 2016. Before joining the team, I had very little experience in 3D design. It was like the Chinese proverb about a journey of a thousand miles beginning with a single step, except in my case that first step was a leap!</p><blockquote>I felt as though I had jumped into the driver’s seat without knowing how to drive the car. I was overwhelmed and scared, yet very focused.</blockquote><p>During the past year, I picked up skills and knowledge as fast as I could, but I still have a lot to learn. Here, I’ve written up 4 observations with a video tutorial documenting my transition from a 2D to 3D interaction designer. I hope my experience will inspire other designers to take the leap to 3D.</p><h4>Good-bye frame. Hello spatial / diegetic UI</h4><p>Whenever I designed posters, magazines, websites, or app screens, a defined frame (usually a rectangle) was a constant for every problem. Unless you are reading this post in a HoloLens or other VR device, you are<em> looking at this from the outside</em> through 2D screen safely guarded within a frame. Content is external to you. However, Mixed Reality headset <em>eliminates the frame</em>, so you are within the content space, looking and walking through the content from inside-out.</p><p>I understood this conceptually, but in the beginning I made the mistake of simply transferring 2D thinking into 3D space. That obviously didn’t work well because 3D space has its own unique properties such as a view change (based on user’s head movement) and <a href="https://www.youtube.com/watch?v=-606oZKLa_s">different requirement for user comfort</a> (based on the properties of the devices and the humans using them). For example, in a 2D UI design space, locking UI elements into the corner of a screen is a very common pattern, but this HUD (Head Up Display) style UI does not feel natural in MR/VR experiences; it hinders user’s immersion into the space and causes user discomfort. It’s like having an annoying dust particle on your glasses that you are dying to get rid of. Over time, I learned that it feels more natural to position content in 3D space and add “Tag-along” behavior that makes it follow user at a relatively fixed distance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*oytOo9Y8pudX6gRT5ojOnQ.gif" /><figcaption>Body-Locked / Tag Along</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*95OkO1VGWd-zFRJcpoL0uw.gif" /><figcaption>World-Locked</figcaption></figure><p>Tag along is a good option for an item that requires high accessibility and visibility. However, overuse can prevent user from staying engaged with the experience by reminding them that they are in a simulated world. To maintain the feeling of immersion, I found world-locked UI or diegetic UI could be helpful.</p><blockquote>World-locked positioning places an element in a specific area within a user’s environment. It’s one of the most magical things a user can experience in HoloLens.</blockquote><p>This is possible because HoloLens continually tracks the content, the room, and the user’s position.</p><p>“Diegetic UI” is a UI element that gets fully integrated with the fictional setting. The term is originated from film industry to separate the sound that can be heard by the characters inside the film (diegetic sound) and background sound that can be only heard by the audience (non-diegetic sound). When it’s done correctly, “Diegetic UI” tends to create greater illusion.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*MFuuoylPyXFBPG-nZRVkkA.gif" /><figcaption>Diegetic UI + World locked</figcaption></figure><p><a href="https://www.microsoft.com/microsoft-hololens/en-us/apps/Fragments"><em>Fragments</em></a><em>, a first-person crime thriller developed by </em><a href="http://www.asobostudio.com/"><em>Asobo Studio</em></a><em> for HoloLens demonstrates a great Diegetic UI. In this game, the user becomes a main character, a detective who tries to solve a mystery. The pivotal clues to solve this mystery get sprinkled in the user’s physical room and are often times embedded inside a fictional object rather than existing on their own. This diegetic UI tends to be less discoverable than body-locked UI, so the Asobo team cleverly used many cues including virtual characters’ gaze direction, sound, light, and guides (e.g., arrow pointing the location of the clue) to grab user’s attention.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9ASxX5BU8kJnWjXheikKHw.jpeg" /><figcaption>Fragments — Diegetic UI examples (Images from <a href="https://www.microsoft.com/microsoft-hololens/en-us/apps/Fragments">HoloLens site</a>)</figcaption></figure><p>Spatial UI (both body-locked and world-locked) and diegetic UI have their own strengths and weaknesses. I encourage designers to try out as many MR/VR apps as possible, and to develop their own understanding and sensibility for various UI positioning methods.</p><h4>The return of skeuomorphism and magical interaction</h4><p>Skeuomorphism, a digital interface that mimics the shape of real world objects has been “uncool” for the last 5–7 years in the design industry. When Apple finally gave way to flat design in iOS 7, it seemed like Skeuomorphism was finally dead as an interface design methodology. But then, a new medium, MR/VR headset arrived to the market and it seems like Skeuomorphism returned again. : )</p><p><a href="http://jobsimulatorgame.com/"><em>Job Simulator</em></a><em>, a whimsical game developed by </em><a href="http://owlchemylabs.com/games/"><em>Owlchemy Labs</em></a><em> is one of the most popular example for skeuomorphic VR design. Within this game, players are transported into future where robots replace humans and humans visit a museum to experience what it feels like to perform mundane tasks at one of four different jobs: Auto Mechanic, Gourmet Chef, Store Clerk, or Office Worker.</em></p><p><em>The benefit of Skeuomorphism is clear. Familiar environments and objects within this game help new VR users feel more comfortable and present in virtual space. It also makes them feel like they are in control by associating familiar knowledge and behaviors with objects and their corresponding physical reactions. For example, to drink a cup of coffee, people simply need to walk to the coffee machine, press a button, grab the cup handle and tilt it towards their mouth as they would do in the real world.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*0F91a9WFih2HsFxgVVyCjA.gif" /><figcaption>Job Simulator</figcaption></figure><p>Because MR/VR is still a developing medium, using a certain degree of skeuomorphism is necessary to demystify MR/VR technology and to introduce it to larger audiences around the world. Additionally, using skeuomorphism or realistic representation could be beneficial for specific types of applications like surgery or flight simulation. Since the goal of these apps is to develop and refine specific skills that can be directly applied in the real world, the closer the simulation is to the real world, the more transferable the knowledge is.</p><p>Remember that skeuomorphism is only one approach. The potential of the MR/VR world is far greater than that, and designers should strive to create magical hyper-natural interactions — new affordances that are uniquely possible in MR/VR world. As a start, consider adding magical powers to ordinary objects to enable users to fulfill their fundamental desires—including teleportation and omniscience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UhTCiVdf0JQOWcfM2e08ww.jpeg" /><figcaption>Doraemon’s magical door (left) and Ruby slippers(right)</figcaption></figure><p>From “Anywhere door” in <em>Doraemon</em>, “Ruby Slippers” in <em>The Wizard of Oz</em> to “Maurader’s map” in <em>Harry Potter</em>, examples of ordinary objects with magical power abound in popular fiction. These magical objects help us visualize a connection between the real-world and the fantastic, between what is and what could be. Keep in mind that when designing the magical or surreal object one needs to strike a balance between functionality and entertainment. Beware of the temptation to create something purely magical just for novelty’s sake.</p><h4>Understand different input methods</h4><p>When I designed for the 2D medium, I had to focus on touch, mouse, and keyboard interactions for inputs. In the MR/VR design space, <em>our body becomes the interface</em> and users are able to use a broader selection of input methods: including speech, gaze, gesture,<a href="https://en.wikipedia.org/wiki/Six_degrees_of_freedom"> 6-dof controllers</a>, and gloves that afford more intuitive and direct connection with virtual objects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5rt5GfQbx_AJgdBGPVwpbw.jpeg" /><figcaption>Available inputs in HoloLens (Images from <a href="https://www.microsoft.com/microsoft-hololens/en-us/why-hololens">HoloLens site</a>)</figcaption></figure><blockquote>As <a href="https://www.billbuxton.com/">Bill Buxton</a> often says, “Everything is best for something, and worst for something else”.</blockquote><p>For example, gesture input using bare hand and camera sensors on an HMD device frees users hand from holding controllers or wearing sweaty gloves, but frequent use can cause physical fatigue (a.k.a gorilla arm). Also, users have to keep their hands within the line of sight; if the camera cannot see the hands, the hands cannot be used.</p><p>Speech input is good at traversing complex tasks because it allows users to cut through nested menus with one command (e.g., “Show me the movies made by Laika studio.”) and also very economical when coupled with other modality (e.g., “Face me” command orients the hologram a user is looking at towards the user). However, speech input may not work well in noisy environment or may not appropriate in a very quiet shared space.</p><p>Besides gesture and speech, hand-held tracked controllers (e.g., Oculus touch, Vive, etc.) are very popular input methods because they are easy to use, accurate, leverage people’s<a href="https://en.wikipedia.org/wiki/Proprioception"> proprioception</a>, and provide passive haptic cues. However, these benefits come at the cost of not being able to be bare-hands and use full finger tracking.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nCgvfolElikwyhg0lvMNhA.jpeg" /><figcaption><a href="https://senso.me/">Senso</a> (Left) and <a href="https://manus-vr.com/">Manus VR</a>(Right)</figcaption></figure><p>While not as popular as controllers, gloves are gaining momentum again thanks to the MR/VR wave. Most recently, brain/mind input have started to gain traction as an interface for virtual environments by integrating EEG or EMG sensor to headset (e.g., <a href="http://www.mindmaze.com/">MindMaze VR</a>).</p><p>These are a just a sample of input devices available in the market for MR/VR. They will continue to proliferate until the industry matures and agrees upon best practices. Until then, designers should remain aware of new input devices and be well-versed in the specific input methods for their particular project. Designers need to look for creative solutions inside of limitations, while also playing to a device’s strengths.</p><h4>Sketch the scene and test in the headset</h4><p>When I worked in 2D, I mostly sketched just the content. However, in mixed reality space that wasn’t sufficient. I had to sketch out the entire scene to better imagine the relationships between the user and virtual objects. To help my spatial thinking, I started to sketch scenes in <a href="https://www.maxon.net/en/products/cinema-4d/overview/">Cinema 4D </a>and sometimes create simple assets for prototyping in <a href="http://www.autodesk.com/products/maya/overview">Maya</a>. I had never used either program before joining the HoloLens team and I am still a newbie, but working with these 3D programs definitely helped me get comfortable with new terminology, such as <a href="https://en.wikipedia.org/wiki/Shader">shader</a> and<a href="https://knowledge.autodesk.com/support/maya/learn-explore/caas/CloudHelp/cloudhelp/2016/ENU/Maya/files/GUID-07C3BA47-32BB-477B-B6C5-1090E5C9B81C-htm.html"> IK (inverse kinematics)</a>.</p><blockquote>No matter how closely I sketched out the scene in 3D, the actual experience in headset was almost never the same as the sketch. That’s why it’s important to test out the scene in the target headsets.</blockquote><p>For HoloLens prototyping, I tried out all the tutorials at <a href="https://developer.microsoft.com/en-us/windows/holographic/academy">Holographic Academy</a> to start. Then I began to play with <a href="https://github.com/Microsoft/HoloToolkit-Unity">HoloToolkit.Unity</a> that Microsoft provides to developers to accelerate development of holographic applications. When I got stuck with something, I posted my question to <a href="https://forums.hololens.com/categories/questions-and-answers">HoloLens Question &amp; Answer Forum</a>.</p><p>After acquiring basic understanding of HoloLens prototyping, I wanted to empower other non-coders to prototype on their own. So I made a video tutorial that teaches how to develop a simple projectile using HoloLens. I briefly explain the basic concepts, so even if you have zero experience in HoloLens development, you should be able to follow along.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F58612RT2CT8%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D58612RT2CT8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F58612RT2CT8%2Fhqdefault.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f55ead667f2aa729004cae1525086ad1/href">https://medium.com/media/f55ead667f2aa729004cae1525086ad1/href</a></iframe><h4>Take the leap</h4><p>A year ago, I felt like all of this was a bit overwhelming. Now I can tell you that it was 100% worth the effort. MR/VR is still very young medium and there are so many interesting possibilities waiting to be realized. I feel inspired and fortunate be able to play one small part in designing the future. I hope you will join me on the journey into 3D space!</p><h4>Recommended articles on HoloLens app design &amp; development by designers at Microsoft</h4><ul><li><a href="https://medium.com/microsoft-design/designing-typography-insight-for-hololens-a55fc5fe025">Designing Typography Insight for HoloLens</a></li><li><a href="https://medium.com/microsoft-design/how-to-think-about-designing-3d-space-b88faf609df4">Designing for Mixed Reality</a></li></ul><p><em>To stay in-the-know with </em><a href="http://www.microsoft.com/design?source=post_page---------------------------"><em>Microsoft Design</em></a><em>, follow us on </em><a href="https://dribbble.com/microsoft?source=post_page---------------------------"><em>Dribbble</em></a><em>, </em><a href="http://www.twitter.com/microsoftdesign?source=post_page---------------------------"><em>Twitter</em></a><em> and </em><a href="http://www.facebook.com/microsoftdesign?source=post_page---------------------------"><em>Facebook</em></a><em>, or join our </em><a href="https://insider.windows.com/?source=post_page---------------------------"><em>Windows Insider program</em></a><em>. And if you are interested in joining our team, head over to </em><a href="http://aka.ms/designcareers"><em>aka.ms/DesignCareers</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4199ccc777a1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/microsoft-design/taking-the-leap-from-2d-to-3d-4199ccc777a1">Taking the Leap from 2D to 3D</a> was originally published in <a href="https://medium.com/microsoft-design">Microsoft Design</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>