<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Avaer Kazmer on Medium]]></title>
        <description><![CDATA[Stories by Avaer Kazmer on Medium]]></description>
        <link>https://medium.com/@avaerkazmer?source=rss-b307c51ad921------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*H0k3Zl7WF1eVrOf544GUKQ.png</url>
            <title>Stories by Avaer Kazmer on Medium</title>
            <link>https://medium.com/@avaerkazmer?source=rss-b307c51ad921------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 23:53:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@avaerkazmer/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Hello World]]></title>
            <link>https://medium.com/exokitxr/hello-world-143e17883b4a?source=rss-b307c51ad921------2</link>
            <guid isPermaLink="false">https://medium.com/p/143e17883b4a</guid>
            <dc:creator><![CDATA[Avaer Kazmer]]></dc:creator>
            <pubDate>Sun, 07 Apr 2019 05:29:24 GMT</pubDate>
            <atom:updated>2019-04-07T05:29:24.264Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=143e17883b4a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/exokitxr/hello-world-143e17883b4a">Hello World</a> was originally published in <a href="https://medium.com/exokitxr">exokitxr</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Magic Leap One: Vive forAR]]></title>
            <link>https://medium.com/webmr/magic-leap-one-vive-forar-3a86084e31ff?source=rss-b307c51ad921------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a86084e31ff</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[magic-leap]]></category>
            <category><![CDATA[webxr]]></category>
            <category><![CDATA[review]]></category>
            <category><![CDATA[ar]]></category>
            <dc:creator><![CDATA[Avaer Kazmer]]></dc:creator>
            <pubDate>Sat, 11 Aug 2018 20:17:51 GMT</pubDate>
            <atom:updated>2019-04-07T19:44:37.626Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/977/1*rk2xz0nhBx5Dkevt7Elk9g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xiM6HoPgr5ig72oC9HXaoQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*c4KH-rQeIQmmY-0GfRZS1g.png" /></figure><h3>Video review</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fn0uhkLa5lBg%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dn0uhkLa5lBg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fn0uhkLa5lBg%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5f6cd1f7015094893a0bc584f8e39859/href">https://medium.com/media/5f6cd1f7015094893a0bc584f8e39859/href</a></iframe><h3>How I wrote this review</h3><p>As an interested mixed reality technologist, I’ve been watching Magic Leap for a while.</p><p>I mean that in a few ways.</p><p>The first is that I went into my first Magic Leap dive after a couple hundred hours hacking the SDK (Javascript bindings). I smelled the code before I walked into it.</p><p>The second is that I don’t have eggs in any basket. I’m cheering for everyone in the space. I’m watching the race, but I don’t have a ticket on Rony’s Pony.</p><p>Finally, being an engineer, I don’t believe in magic. I believe in smart people working hard and making things at the intersection of technology and market. I feel read enough that I know what’s reasonable in T time and D dollars.</p><p>Context in reviews is important! Especially for something like Magic Leap.</p><p>That said, I think Magic Leap is on the right track. I feel like this week they shipped the Vive of AR.</p><h3>AR vs VR</h3><h4>The AR landscape</h4><p>I feel like “Augmented Reality” as a concept has until this point been plagued by the convenient specter of mobile devices. That’s most folk’s introduction to AR, and while it works great for filters, phones don’t have either the compute or the lenses to make it more than a gimmick. They must exist, but I don’t think I’ve ever met a <em>user</em> of ARKit.</p><p>On the flip side you have the Hololens, but that’s an enterprise device with an enterprise price tag and enterprise developer relations in an industry that’s built on experimentation and wow factor. Its current iteration is also a few years behind Magic Leap.</p><h4>The VR landscape</h4><p>VR is significantly more mature industry. We’re getting started on the adoption curve, but VR is “a thing” with a market definition and I’ve met my share of people who identify as VR gamers. Even lower-end mobile VR is a thing.</p><p>So what does this have to do with Magic Leap? Because I feel like Magic Leap is basically the AR of VR.</p><h3>The experience</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bjt6xINnMqoKBgNRgxzG8w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cDjlpKez0VDqjRNuB75z5Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_OUiAB0wjw2wkBMR_A8Evg.png" /></figure><p>The best way I can describe Magic Leap is that it’s like someone took VR and injected the real world into it.</p><p>By this I mean:</p><p>- An multitasking operating system<br>- An app store<br>- A head mounted display<br>- 6DOF for everything<br>- A controller<br>- Requirements for a play area<br>- Modern GPU hardware<br>- A microphone<br>- A massive, well-implemented SDK<br>- Engine integrations</p><p>If this sounds like a desktop VR platform, that’s not a coincidence.</p><p>The difference with Magic Leap is that it also packs in inside-out tracking, meshing, object recognition, and hands tracking out of the box. But this all feels like an addendum to the solid rendering APIs we’ve grown to love with VR. That is, there is a play space, your application gets matrices and poses, and delivers a binocular render of whatever.</p><p>Whether you’re a user, using Magic Leap is basically like adding AR to your VR.</p><h3>The magic</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qQfZRXOfKqQRprse3VHTCA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N9VaHnb597AoPBAFn-8nwQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E9N1teeVEfs5D5wBojzgWg.png" /></figure><p>I probably just made it seem like this is a device and stack that adds not much that’s new. But adding the real world to VR changes a few things.</p><p>For one, your application is going to suck if it’s not doing mesh occlusion. AR is meant to interact with the real world and if you ignore this your app will be crossing people’s eyes as objects draw behind real walls, which is a physical impossibility the human visual system isn’t prepared to deal with.</p><p>Thankfully the Magic Leap SDK provides excellent API support for injesting meshes and depth, especially if you’re coming from Unity + Unreal.</p><p>But this is just the start. If your application is only occluding content it still is not going to offer much in the way of new experiences. For that, you want to get friendly with the mesh.</p><h3>The mesh</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eE-h3fkSf4C9RzovlTCUZA.png" /></figure><p>The ML1’s secret sauce is its meshing system. It’s really good. Which is good because so much of the ML1 rides on it.</p><p>As look and walk around your space, the hardware is using AI, SLAM, and sensor fusion to piece together a voxel mesh of the real world. You can turn this on if you want to see it in action, but it&#39;s chugging along even if you don&#39;t.</p><p>What this means for applications is that they get a complete, and pretty accurate view of the real world in their world. If an app wants to let you throw a cube and have it bounce around the room, or shoot a rouchet out the window, or place a trex on the desk, or... that&#39;s as easy as adding a collider.</p><h3>The persistence</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ABOo6ro98I4jeKPPgTrLdw.png" /></figure><p>One other important aspect of AR embraced by Magic Leap is that you&#39;re not teleporting around inside a cage or riding a roller coaster, like you do in VR.</p><p>You don&#39;t teleport at all. Instead your content can be experienced anywhere. You can leave it and come back and it&#39;ll still be there. This can work even when the environment changes slightly, or you power off the device.</p><p>This completely changes the way you design and experience content. AR encourages smaller pieces of content that interact with each other, not monolithic gigabyte games that take over your reality and get discarded when the next cool thing comes along.</p><p>This brings us to...</p><h3>The operating system</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q6Pv1dfeTlV58D5B5HNQTg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sO-XpS-lDvgHYv4-6XiH2A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WNY9qEjm4RCDQlZ8RbUCUg.png" /></figure><p>Lumin OS is the Android-derived operating system that runs on the Magic Leap One hardware. As a user one hint at this is that the charging cables are the same ones that work with your Pixel, but otherwise the hardware feels nothing like Android. The Lightpack compute unit is ventilated, with a fan.</p><p>There are more tastes of Android remaining if you are a developer. If you&#39;re familiar with `apk`s, Magic Leap has its own `mpk` format, and there is a debug bridge called `mldb` based on `adb`.</p><p>But the star of the show is the application runtime, which is very much tailored for mixed reality. Meshing, suspend/resume, recording, and other things come standard for all applications. Some applications even get persistent multitasking. To understand that we&#39;ll have to talk runtimes, because Lumin OS has two.</p><h3>The two runtimes</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vyI_c3f2Vwy4ME0SfugE7g.png" /></figure><p>Magic Leap acutally comes with two separate ways of making content:</p><p>- Lumin SDK immersive apps, and<br>- Lumin Runtime apps</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LD3XPtEkCH3gUpuEM76qEA.png" /></figure><p>Immersive apps are like if you literally took VR and added AR. They take over your view of reality while they are active, and they are intended for things that reasonably fill the entire play area while being too complex to coexist with other apps. One example is Create, one of the first-party sandbox experiences from Magic Leap.</p><p>Lumin Runtime on the other hand is based around a scene graph, which is managed, blended, and user interfaced by the operating system. Lumin runtime allows you to for example run The Helio web browser alongside media players, mutiplayer avatars, and such.</p><p>I think both have their place, and this was the right design choice. AR is different in that it makes perfect sense to have streamlined micro-applications as a first class citizen in the operating system. Lumin OS provides a rich set of services to base your applications on -- much richer than OpenVR or OculusVR, which makes multitasking applications on a runtime actually make sense.</p><p>There&#39;s a couple of other things that just make sense with Magic Leap one, and they were a bit of a pleasant surprise to me.</p><h3>Surprise one: audio</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GTBv95-UMgl3L6i2uHrlRg.png" /></figure><p>You might think that this is a graphics headset and audio is a solved problem, but the spatial audio on the Magic Leap One is not only great (without requiring headphones), but also plays a critically important role in the experience.</p><p>Since you are immersed in the messy, noisy real world while wearing a Magic Leap One (as opposed to contained in a carefully constructed reality), it can be hard to find things in the world. It&#39;s literally possible you&#39;ll lose something behind the fridge, or your trex walks to the other room while you&#39;re not looking (or cooking).</p><h3>Surprise two: avatars</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*rJJLQZR3QMsMDBX1XRzRPw.png" /></figure><p>I was pleasantly surprised to find that there is a full social avatar system in place too, and it does eye tracking, voice tracking, samples hand gestures, and more.</p><p>I didn&#39;t get a chance to try out the social functionality yet, but it looks like the ingredients are there for a solid API layer for apps.</p><h3>The lenses: FOV, resolution, opacity, near clip</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u_Nlk4r3b1e6r-8nGj5xHQ.jpeg" /></figure><p>Ah, the thing that everyone was talking about before, during, and after the launch of Magic Leap.</p><p>Let&#39;s get the easy one out of the way: the opacity of the display is a non-issue. The projectino is opaque enough that it&#39;s prefectly capable of fooling you there is a hole to another dimension in your wall. The only case where opacity was noticeable was when looking out through a brightly lit window.</p><p>Resolution is a bit more complicated. First off, there is no screen door effect like you see in VR. Secondly, you don&#39;t see pixels. It&#39;s very much a hologram, and hard to describe otherwise. I suspect the total resolution is less than that of VR devices, but it&#39;s concentrated in a smaller field of view. In short, I did not notice the resolution of the device to be able to tell you what it is -- which is not something I can say for any VR headset.</p><p>Finally, the field of view. You&#39;re not getting VR FOV, but it&#39;s better than the Hololens. It&#39;s a hard-clipped recangle straight ahead. And, you mostly don&#39;t notice, because in AR things tend to follow the laws of physics you&#39;re used to. Toss something over your shoulder, hear it land behind you, turn around, and it&#39;s there. This is partly why audio plays an important role here -- it does a lot to increase the immersion regardless of where you&#39;re looking.</p><p>One thing that played a much more important role in the experience for me was the somewhat far near plane clipping on every one of the default apps. I believe this is a technical store requirement as well. It feels like an aesthetic decision with regards to hand tracking and occluding the meshing rather than a graphical limitation. Which makes sense, but for the creator edition I might have relaxed this requirement so that you could really look at objects up close.</p><h3>The grafix</h3><p>The graphics are another non-issue. Geometry on the default apps is dense, animated, and fluid, and from the technical side you have full OpenGL and Vulkan.</p><p>You sometimes <em>can</em> tell that the CPU is loading and isn&#39;t feeding frames to the GPU -- something you see in VR too -- but this seems like a software optimization opportunity than anything in the hardware.</p><p>I also did notice some slowdown when filling the space up with multiple Lumin Runtime apps, which compete for the machne&#39;s cores. This was a pretty rare ocurrence, and did not happen at all in any of the full-scale immersive applications, which own their render loop.</p><h3>The comfort</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WQBsSZUmCg183fwa8Ld3BA.jpeg" /></figure><p>A question that I was surprised to get was how comfortable the unit is. The answer is that it&#39;s more comfortable than a VR headset, mostly because:</p><ul><li>The fit is looser, since light bleed is not an issue</li><li>There are no cables to step around compared to wired VR</li><li>Magic Leap clearly spent a lot of time thinking about ergonomics</li></ul><p>The unit can run tethered or untethered on a battery pack. The cables are USB-C, but a little on the short side for a mixed reality headset. I suspect you could.</p><p>The battery lasts about as long as you’d expect, and it depends on usage. You’ll get a couple hours out of it when doing heavy things like running a ton of apps and recording review video ;) The whole thing can, of course, run indefinitely when plugged in.</p><p>The thing I did notice is that when coming out of AR, reality had a screen door effect, as if reality had turned to VR. This wasn&#39;t too different from what you get when coming out of a full day of VR, but it was definitely curious. Not at all unpleasant, but interesting biological minutiae, like the afterlights you see when you come home on a sunny day. More science is needed to get to the bottom of this!</p><h3>The unfinished</h3><p>What excited me was that there are several unfinished things sprinkled throughout Lumin OS. Yes, excited -- because that means there&#39;s more in the works, and there&#39;s plenty of room to experiment with the platform. That&#39;s what the creator edition is for!</p><p>One part that feels unfinished is the social system. It seems to be fully fleshed out but hasn&#39;t been interwoven into existing experiences yet.</p><p>Another is the content on Magic Leap World (the ML1 app store). On launch day, there&#39;s not much — mostly just first-party applications. For a creator edition, this is expected; the idea is that the explorers buying the gear today are the ones that will fill up the store. That’s us.</p><p>The Helio web browser (which runs on Lumin Runtime) also feels like it&#39;s touching on something great, but right now the only thing that separates it from a regular web browser is the ability to load and pull out 3D models from the page. That&#39;s kind of neat, but the existing web is a gateway to everything.</p><p>Speaking of, I just got the go-ahead from Magic Leap to ship V8 to Magic Leap World, so I’m getting antsy to ship Emukit and other Javascript/WebXR experiences…</p><h3>Should I get one</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q4asK-C4-TbmjYtIfXasTw.jpeg" /></figure><p>This question answered by a question: are you a creator? Then get yourself a Magic Leap One: Creator Edition. Otherwise, wait.</p><p>(Under creators I include professional Youtubers/streamers/user generated contenteers, modelers, and of course developers)</p><p>If the name didn&#39;t give it away, this is a product made for makers. The technology is here, but you will be starved for content if you&#39;re looking for two thousand dollars worth of enrichment and entertainment today.</p><h3>Links to what I do</h3><p>Check out Exokit: <a href="https://exokit.org/">https://exokit.org</a></p><p>Sign up to our mailing list for updates: <a href="http://eepurl.com/dFiLMz">http://eepurl.com/dFiLMz</a></p><p>Homepage: <a href="https://exokit.org/">https://exokit.org</a><br>GitHub: <a href="https://github.com/exokitXR/exokit">https://github.com/exokitXR/exokit</a><br>Twitter: <a href="https://twitter.com/exokitXR">https://twitter.com/exokitXR</a><br>Instagram: <a href="https://instagram.com/exokitXR">https://instagram.com/exokitXR</a><br>Discord: <a href="https://discord.gg/cf5tfTV">https://discord.gg/cf5tfTV</a><br>Slack: <a href="https://exokit.slack.com/join/shared_invite/enQtNDI3NjcxNzYwMDIxLWU2NmFmOTEzMzk4NWNiYjRhMjVkYzcyNjg5YjUyMzZkYWM1ZGI4M2IwYWZiMjNlMTJjMDlkM2U3Y2JiNTc2M2Q">https://exokit.slack.com/join/shared_invite/enQtNDI3NjcxNzYwMDIxLWU2NmFmOTEzMzk4NWNiYjRhMjVkYzcyNjg5YjUyMzZkYWM1ZGI4M2IwYWZiMjNlMTJjMDlkM2U3Y2JiNTc2M2Q</a><br>Twitch: <a href="https://www.twitch.tv/exokitXR">https://www.twitch.tv/exokitXR</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a86084e31ff" width="1" height="1" alt=""><hr><p><a href="https://medium.com/webmr/magic-leap-one-vive-forar-3a86084e31ff">Magic Leap One: Vive forAR</a> was originally published in <a href="https://medium.com/webmr">Exokit</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DOM to canvas in just 7 years]]></title>
            <link>https://medium.com/webmr/dom-to-canvas-for-webxr-in-just-7-years-e032060437fb?source=rss-b307c51ad921------2</link>
            <guid isPermaLink="false">https://medium.com/p/e032060437fb</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[web]]></category>
            <category><![CDATA[webxr]]></category>
            <category><![CDATA[browsers]]></category>
            <dc:creator><![CDATA[Avaer Kazmer]]></dc:creator>
            <pubDate>Tue, 07 Aug 2018 12:45:10 GMT</pubDate>
            <atom:updated>2019-04-07T19:45:04.850Z</atom:updated>
            <content:encoded><![CDATA[<p>You know what’s harder than creating an alternate reality with Javascript? Drawing a &lt;div&gt;.</p><p>If you don’t know what DOM-to-canvas is, I’d recommend reviewing <a href="https://robert.ocallahan.org/2011/11/drawing-dom-content-to-canvas.html"><em>the classic by Robert O’Callahan</em></a>, and possibly <a href="https://robert.ocallahan.org/2011/09/risks-of-exposing-web-page-pixel-data.html">one in which he talks about security of the scheme</a>.</p><p>But here’s my TLDR:</p><blockquote>Browsers — the thing designed to draw web pages — don’t let developers draw web pages. For reasons.</blockquote><p>You’d be 👌 right to suspect this is one of the most requested features in a browser. Especially in a world where Rob’s predictions about 3D objects being drawn with HTML are coming true with the whole WebXR thing.</p><p>Yet it’s 7 years later and not much has changed with what the major vendors have done with DOM to canvas. It’s still not happening, for reasons.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*EFrp63UsKPG7dDlS1FYpug.jpeg" /><figcaption>WebXR dev see the truth</figcaption></figure><p>Of course, the reason given by browsers for not implementing such a feature is security. The browser uses state such as cookies when rendering a web page, which is exposed when you draw it.</p><p>It’s the same story as cross-origin policies, <em>iframe </em>issues, and CORS. You might recognize this as another list of web developer banes.</p><blockquote>To say we can’t DOM-to-canvas because of the cookies is like saying we can’t take the car because of the elephant.</blockquote><h3>I just want a browser in my browser</h3><p>The impetus for this post was that I wanted to run <a href="https://github.com/shaneharris/aframe-in-app-browser">Shane Harris’s A-Frame in-app Browser in WebXR</a>.</p><p>And all I could think was:</p><ul><li>Cool, I remember haxing this before!</li><li>I want to use this!</li><li>I wonder which hack and caveat this requires to work…</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*izMvu1iOfO-Yd7Q65eYWXw.gif" /><figcaption><a href="https://github.com/shaneharris/aframe-in-app-browser"><strong>shaneharris/aframe-in-app-browser</strong></a></figcaption></figure><p>That’s why I decided to write this post to round up the techniques I’d used and seen used.</p><h3>27 ways to kinda sorta DOM-to-canvas</h3><p>Since browsers didn’t implement this much-asked for feature, developers took it into their own hands with varying degrees of success. Here are some hacks you’ll see in the wild:</p><h3>The Mozilla &lt;svg&gt;</h3><p><a href="https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Drawing_DOM_objects_into_a_canvas">Drawing DOM objects into a canvas</a></p><p>This technique involves packing your DOM into an <strong>XMLSerializer </strong>to get an XML representation which you can wrap in a <strong>&lt;foreignObject&gt;</strong> that you put into an <strong>SVG</strong>, which you load into an <strong>HTMLImageElement</strong>, wait for it to load, and then finally draw to a <strong>&lt;canvas&gt;</strong>.</p><p>Also if you’re using images or certain styles you’ll have to compile them to data URLs and maybe insert a delay to make sure they’ve loaded before you try to draw your SVG.</p><blockquote>This is the recommended API by the premier proponent of the web.</blockquote><h3>The capture tab</h3><p><a href="https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/tabs/captureTab">https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/tabs/captureTab</a></p><p>Extension authors would flip if they couldn’t capture the contents of the browser, so browser vendors acquiesced here.</p><p>You <em>can </em>capture a tab screenshot if you are an extension. Presumably as an extension you are more trustworthy than a site so they are blessed with the privilege.</p><p>The idea is you call a browser-internal API and get back a data URL for a png or jpeg, which you then waste time decoding and drawing to a canvas. You’ll get double-digit FPS out of it too!</p><p>The catch is that you need to be an extension, or write one.</p><p>Unsurprisingly, I have seen sites in the wild that require you to install an extension to unlock this functionality on the site with a hack.</p><h3>The fat companion</h3><p><a href="https://github.com/shaneharris/aframe-in-app-browser/blob/57c3fb4367381725735193b750692b353f94d701/server.js#L9">https://github.com/shaneharris/aframe-in-app-browser/blob/57c3fb4367381725735193b750692b353f94d701/server.js#L9</a></p><p>This is the technique used by <a href="https://github.com/shaneharris/aframe-in-app-browser"><strong>shaneharris/aframe-in-app-browser</strong></a><strong>.</strong></p><p>The idea is that you start an <strong>extra browser</strong> somewhere. And that browser does the drawing for you, and sends you back the results.</p><p>Generally this browser will be running on some server, so this limits usefulness to the cases where you are running a local app or don’t mind the server/bandwidth bills for your app.</p><p>Again this has been used in production.</p><h3><strong>The webkit.js</strong></h3><p><a href="http://trevorlinton.github.io/">http://trevorlinton.github.io/</a></p><p>This one is ingenious. Especially for its time in a pre-WebAssembly world.</p><p>The idea is that you literally take WebKit (the rendering engine for Safari and formerly Chrome), and you compile it to Javascript with emscripten. And then you run it in the browser, so you can finally draw things freely.</p><p>The most shocking thing is that it works, if you don’t mind 0.1 FPS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zCgm6JXDY7bvo9NIDlSFBA.png" /><figcaption>webkit.js in Webkit</figcaption></figure><h3>The time capsule</h3><p>How did we get here?</p><p>How did we get to a point where instead of browser vendors solving a problem so many developers want solved, end developers are inventing new modes of computation inception to compensate?</p><p>In <a href="https://www.paulirish.com/2011/what-feature-would-improve-the-web/"><em>a gem from 2011 Paul Irish</em></a>, we get a time capsule into what web developers were thinking back in the pre-ES5-toric times.</p><p>Here’s some of the cream of the crop.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g4dsBacuNti9bgh9ZDLJnA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IVf0_TjbHJVkOs8ESksslg.png" /></figure><p>Our friend DOM to canvas makes an appearance. As does rethinking the DOM, performance, and straight up throwing away the browser.</p><h3>A step back</h3><p>I can empathize with browser security, and I wouldn’t advocate breaking it. By and large browser vendors have accomplished the impossible with an arbitrary code execution platform that works (tm).</p><p>But the browser has also dug itself an immense hole: it has to support so many APIs and paradigms and interactions that sometimes things just fundamentally don’t mesh. And so this is why we can’t do a fast DOM to canvas. Because reasons.</p><p>And as a WebXR developer I’m caught in the middle. I just want a browser engine that gives me APIs and gets of the way. Maybe I’m running in a container. Maybe I’m shipping to an app store. Maybe all of the DOMs I’m drawing are local files. But the browser doesn’t know.</p><p>So that’s why we have open source.</p><h3>I wanted an escape hatch so I made one</h3><p>I added an Electron API that efficiently streams pixel Buffer diffs at 200 FPS. (<a href="https://github.com/webmixedreality/exokit/pull/326">PR #326</a>)</p><p>I was busy bugfixing by day so I did it overnight. But as they say, an overnight success takes 7 years.</p><p>Especially if you’re a browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*THVHlNKjyQMO_ee_rA2FaQ.png" /></figure><p>Check out Exokit: <a href="https://exokit.org/">https://exokit.org</a></p><p>Sign up to our mailing list for updates: <a href="http://eepurl.com/dFiLMz">http://eepurl.com/dFiLMz</a></p><p>Homepage: <a href="https://exokit.org/">https://exokit.org</a><br>GitHub: <a href="https://github.com/exokitXR/exokit">https://github.com/exokitXR/exokit</a><br>Twitter: <a href="https://twitter.com/exokitXR">https://twitter.com/exokitXR</a><br>Instagram: <a href="https://instagram.com/exokitXR">https://instagram.com/exokitXR</a><br>Discord: <a href="https://discord.gg/cf5tfTV">https://discord.gg/cf5tfTV</a><br>Slack: <a href="https://exokit.slack.com/join/shared_invite/enQtNDI3NjcxNzYwMDIxLWU2NmFmOTEzMzk4NWNiYjRhMjVkYzcyNjg5YjUyMzZkYWM1ZGI4M2IwYWZiMjNlMTJjMDlkM2U3Y2JiNTc2M2Q">https://exokit.slack.com/join/shared_invite/enQtNDI3NjcxNzYwMDIxLWU2NmFmOTEzMzk4NWNiYjRhMjVkYzcyNjg5YjUyMzZkYWM1ZGI4M2IwYWZiMjNlMTJjMDlkM2U3Y2JiNTc2M2Q</a><br>Twitch: <a href="https://www.twitch.tv/exokitXR">https://www.twitch.tv/exokitXR</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e032060437fb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/webmr/dom-to-canvas-for-webxr-in-just-7-years-e032060437fb">DOM to canvas in just 7 years</a> was originally published in <a href="https://medium.com/webmr">Exokit</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exopod 4: Magic Leap Y U NO strem]]></title>
            <link>https://medium.com/webmr/exopod-4-magic-leap-y-u-no-strem-d5dfd4f4b5a9?source=rss-b307c51ad921------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5dfd4f4b5a9</guid>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[webxr]]></category>
            <category><![CDATA[augmented-reality]]></category>
            <category><![CDATA[magic-leap]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Avaer Kazmer]]></dc:creator>
            <pubDate>Fri, 03 Aug 2018 18:28:20 GMT</pubDate>
            <atom:updated>2019-04-07T19:45:18.896Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X7vMn2lryLvyH3-y5GVb4g.jpeg" /></figure><blockquote>No ML stream this month. So says Rony.</blockquote><p>It’s ok, Exopod has you covered! Magic Leap is close to shipping, so enjoy what we hope is the last edition before we’re playing with our new AR headsets!</p><p>Appendages crossed.</p><p>Ronies prayed to.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FxL1wOmdunq0%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DxL1wOmdunq0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FxL1wOmdunq0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/2143cf9591c5737b2e6bdbd0dd34cea1/href">https://medium.com/media/2143cf9591c5737b2e6bdbd0dd34cea1/href</a></iframe><p>In this edition, <strong>Avaer Kazmer</strong>, <strong>Nick Loomis</strong>, <strong>Noah Schiffman</strong>, and <strong>Chris La Torres</strong> dissect:</p><ul><li>Magic Leap’s radio silence mode (<a href="https://twitter.com/rabovitz/status/1024791379959013376">https://twitter.com/rabovitz/status/1024791379959013376</a>)</li><li>FOV on the ML1, and why it does not matter… to discuss why FOV does not matter</li><li>Tech specs!</li><li>Lumin Runtime versus Lumin SDK — who will win?</li><li>Prisms and Prismatic</li><li>Multiplayer world streaming with Prisms</li><li>Helio browser and what you can do with it</li><li>Mobile Companion App — time to throw away your controller</li><li>Magic Leap in public spaces</li><li>How to 5G on a Magic Leap One</li><li>Magic Leap ID</li><li>In App Purchases in AR</li><li>Security considerations for when your reality might not be real</li><li>App Icons that animate</li><li>Dioramas as preview of apps</li><li>Technical Requirements Checklist for submitting to the Magic Leap World app store</li><li>Payment tiers — how to get rich by telling people how to get rich</li><li>Object recognition APIs — AI in your headset</li><li>Exokit browser’s resurrection as Exokit engine</li><li>Haxing Skia libs to work on Arm64 for webby Magic Leapage</li><li>Compiling WebGL and WebXR sites to Magic Leap</li><li>And we introduce ourselves at the end!</li></ul><h4>Join the discussion!</h4><p>Check out Exokit: <a href="https://exokit.org/">https://exokit.org</a></p><p>Sign up to our mailing list for updates: <a href="http://eepurl.com/dFiLMz">http://eepurl.com/dFiLMz</a></p><p>Homepage: <a href="https://exokit.org/">https://exokit.org</a><br>GitHub: <a href="https://github.com/exokitXR/exokit">https://github.com/exokitXR/exokit</a><br>Twitter: <a href="https://twitter.com/exokitXR">https://twitter.com/exokitXR</a><br>Instagram: <a href="https://instagram.com/exokitXR">https://instagram.com/exokitXR</a><br>Discord: <a href="https://discord.gg/cf5tfTV">https://discord.gg/cf5tfTV</a><br>Slack: <a href="https://exokit.slack.com/join/shared_invite/enQtNDI3NjcxNzYwMDIxLWU2NmFmOTEzMzk4NWNiYjRhMjVkYzcyNjg5YjUyMzZkYWM1ZGI4M2IwYWZiMjNlMTJjMDlkM2U3Y2JiNTc2M2Q">https://exokit.slack.com/join/shared_invite/enQtNDI3NjcxNzYwMDIxLWU2NmFmOTEzMzk4NWNiYjRhMjVkYzcyNjg5YjUyMzZkYWM1ZGI4M2IwYWZiMjNlMTJjMDlkM2U3Y2JiNTc2M2Q</a><br>Twitch: <a href="https://www.twitch.tv/exokitXR">https://www.twitch.tv/exokitXR</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5dfd4f4b5a9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/webmr/exopod-4-magic-leap-y-u-no-strem-d5dfd4f4b5a9">Exopod 4: Magic Leap Y U NO strem</a> was originally published in <a href="https://medium.com/webmr">Exokit</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Emukit 2: immersive PSX+N64 VR on the web]]></title>
            <link>https://medium.com/webmr/emukit-2-immersive-psx-n64-vr-on-the-web-7c1ec110a592?source=rss-b307c51ad921------2</link>
            <guid isPermaLink="false">https://medium.com/p/7c1ec110a592</guid>
            <category><![CDATA[webxr]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[vr]]></category>
            <category><![CDATA[webgl]]></category>
            <dc:creator><![CDATA[Avaer Kazmer]]></dc:creator>
            <pubDate>Fri, 27 Jul 2018 21:16:04 GMT</pubDate>
            <atom:updated>2019-04-07T19:47:39.311Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong><em>TLDR: </em></strong><em>We got immersive PS1 working in Javascript + WebXR + WebAssembly with Exokit. More N64VR games too. Video and code at end of post.</em></blockquote><h3>So… with Emukit 2… PlayStation 1 VR is a thing</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qXJ5jjSWKDSc9FecXpDkEA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pqBKJkDlJNdxO4pwaSLolg.png" /></figure><p>There’s also a truckload of N64 VR performance and compatibility improvements in the <strong><em>Emukit 2</em></strong> refresh, so a lot more of your favorite games work in mixed reality:</p><h4>Perfect Dark 64 VR</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*89mhRjMI-p9ABGL-afrpsg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mnRSd7Pf6wC88625L7-J1Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fl00fSnfGoux92JRz555mQ.png" /></figure><h4>Star Fox 64 VR</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rRp3dsjjm5UHWg5PQisvwQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PE8BDaAJeADmeIOYlVYTQA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TCB8VkZ_SkREQs1kAyGAtw.png" /></figure><p>This post is the how and why of why myself and the Exokit dev team are porting yesterday’s games to web mixed reality. But… if you just want the code or the video, it’s at the end of the post.</p><h3>Why Emukit?</h3><blockquote>The point of Exokit (the “post-screen web browser”) is to write a mixed reality web browser in Javascript.</blockquote><blockquote>We want to plug the best language/framework in the world — Javascript — into a future where there are no 2D screens.</blockquote><h4>In 10 years</h4><p>If we suspend disbelief for a second, imagine a world in which every developer uses their favorite language to code their app. Hopefully that’s Javascript but maybe it’s not — in which case she builds to WebAssembly.</p><p>Her code runs automagically on any VR, AR, or neural implant device.</p><p>I have no idea how the tech will evolve — which is why Exokit doesn’t place any bets. Exokit is written in Javascript and integrates with whatever mixed reality API you want:</p><ul><li>Magic Leap</li><li>OpenVR (Steam VR)</li><li>Oculus VR</li><li>Leap Motion Orion</li><li>Desktop</li><li>ARKit+ARCore (in future)</li></ul><h4>An emulator for the web</h4><p>What’s the piece of software that wires everything together? It’ll certainly need to be doing a lot of <em>emulation</em>.</p><p>The piece of software gluing all of this to Javascript would have to take in data from disparate APIs:</p><ul><li>outside-in VR (sensors shoot to the device)</li><li>inside-out AR (device shoots to the world)</li><li>Magic Leap/AR SLAM (localizing in a space without prior info)</li><li>Leap Motion hands detection machine learning (computing geometry of a thing from a raster image)</li><li>and so much more</li></ul><p>This Javascript layer would also need to work the other way, and convert JS things before presenting it to the hardware. For example:</p><ul><li>take an ArrayBuffer and submit it to the GPU</li><li>map a &lt;canvas&gt; to a heaf-mounted display</li><li>negotiate and configure the HMD</li><li>merge multiple sites/apps into one cohrerent reality</li><li>coordinate frame-perfect timing with <strong><em>requestAnimationFrame</em></strong></li><li>simulate input events for testing/botting</li><li>network and multiplayer</li><li>and again so much more</li></ul><p>And this middle layer has to wrap everything under a common API (WebXR) that applications can use to construct a layer of reality and present it to the user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F4aCtzTIJZffz9YkyKxJpw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cyms5tG8COJnkFCM8ha4rA.png" /></figure><blockquote>If this sounds more like an emulator than a browser, I don’t blame you. We call it a browser, but Exokit is actually a 3D emulator for the web.</blockquote><p>When I started Emukit (a mixed reality console emulator that runs on Exokit), it was just another step towards taking everything and putting it into web mixed reality.</p><p>Also, I suck at making games. Maybe I suck a bit less at plugging the plethora of awesome games that defined my childhood into the matrix (<a href="https://threejs.org/docs/#api/math/Matrix4"><strong>THREE.Matrix4</strong></a>).</p><h3>How I did PS1 VR</h3><p>Making the PlayStation (<em>one, not four!</em>) work in VR was not without its challenges. Not the least of which is that the PlayStation is actually a 2D machine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UhDgIIwe6Zwus7vTeRkJKw.png" /></figure><p><em>Yes, the PlayStation is a 2D machine.</em></p><p>You could probably find the place in the <a href="https://twitch.tv/avaer"><strong><em>Twitch livestreams</em></strong></a> where after spending a couple of weeks on the problem I finally cracked open the documentation and realized what I was doing might be impossible. Oops!</p><blockquote>Obviously we cracked the impossible nut.</blockquote><blockquote>Walk with me, down history street and graphics avenue.</blockquote><h3>At the turn of the dimensions</h3><p>As I said, the Playstation GPU is actually completely 2D. It can draw some sweet polygons, gradients, Gouraud shading, and even crappy integer-aligned textures. But… it does it all in X and Y. The PlayStation GPU is literally just a megabyte of 2D pixels and nothing else.</p><p>This is why you hear that the PSX had no Z buffer. The N64 did, and it’s <em>much</em> closer to what we think of 3D graphics today.</p><p>But the PSX had one foot in the SNES and one foot in the Xbox. It couldn’t decide what it was, so it was both.</p><p>So how did we get “3D graphics” on the PSX, and what sorcery is powering Emukit? The answer is an epic hack: the Geometry Transform Engine, or GTE.</p><p>Since we’ve established that the PSX GPU did not do 3D math, the GTE was a coprocessor on the PSX that did the job. It couldn’t draw anything, but it could do 3D vector and matrix math way faster — and in parallel with — the PSX CPU.</p><p>By the way, the PSX CPU <em>really</em> sucked — it was a third as fast as the N64 one, at about 33 Mhz.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*543uzXc30xYw4nCM07XZIA.png" /></figure><h3>PGXP your PSX for WASM WebXR in HMD</h3><p>Thankfully the problem of adding a third dimension to the PlayStation graphics stack is not a new one. The PSX’s suckage in handing perspective and texture mapping is and old meme. Which means that smarter haxors than I have already haxed it.</p><p>The solution is a technique called PGXP, or the Parallel Geometry Transform Pipeline. Essentially this adds another layer of hax to the epic hack that is the GTE.</p><p>So imagine we are a PlayStation game. We load our geometry from CD (excruciating load time included at no charge). We submit our geometry vec3’s to the GTE for transforming to our camera Model/View/Projection matrix. We get back 2D points, which we submit as commands to our crippled 2D GPU.</p><blockquote>Can you see how to get 3D rendering out of this?</blockquote><p>That’s right: we slap in an extra cache for the vertex data coming into the GTE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vsEJBXKazkn9ZADGoEc9oA.png" /></figure><p>When the GTE takes in a point we don’t just output the 2D result for the GPU; we also cache the 3D vertices we were using, based on the X/Y coordinate of the output.</p><p>With this mutated monster of a cache — which did not exist on the real PlayStation — we can reach out from the GPU to peek at the original 3D vertices when it’s time to draw. And just like that, we can kinda-sorta transform the 2D GPU into _actual modern GPU_ by taking this now-3D data and feeding it into a vertex/fragment shader pipeline.</p><p>Once I got 3D points coming into the GPU it was pretty straightforward to add the standard MR-ification hacks from Emukit’s N64VR implementation.</p><p>That is, we use the fact that Exokit is just Javascript to intercept the GL draw calls and double them up, using the two different eye matrices and viewports coming in from WebXR.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gL93i-_aOX4CHzNW50BzMA.png" /></figure><h3>Literally rewriting OpenGL</h3><p>The above isn’t what took me so long with Emukit 2. It took so long because getting things working involved rewriting the Playstation GPU three times, and OpenGL once.</p><p>First, a lesson in WebGL.</p><p>WebGL is just a Javscript binding for OpenGL (ES). In fact that is _literally_ how Exokit implements it.</p><p>And WebGL2 is very close to the full blown OpenGL most applications — inluding games — use. The main omissions are the things that would be rough for Javascript, like memory mapped buffers.</p><p>What this means is that you can use Emscripten to compile 3D applications to WebAssembly + WebGLL2 + Javascript and it often just works. That’s the essence of VRifying the N64 — and now — the PSX,</p><p>The bumps I ran into were twofold, with the emulators and Exokit pointing in opposite directions of history:</p><ul><li>The PlayStation emulators were using legacy OpenGL as old as the PlayStation</li><li>Exokit didn’t support the new WebGL2 endpoints used by Emscripten</li></ul><p>To fix the former I ended up rewriting the legacy OpenGL implementation to use a modern matrix, shader, buffer, and drawcall pipeline. Thank Zeus (<em>the god</em>) for <strong>Khronos </strong>(<em>the consortium</em>)!</p><p>The Playstation OpenGL plugins are often complained about as being outdated, so I’ll look into upstreaming this work if it can be done easily.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XdsMp25hIwODAA2O2Ef1TQ.png" /></figure><p>To fix the latter problem of Exokit not supporting the WebGL2 APIs that Emscripten was using, I simply cracked open MDN for a day (props to Mozilla for fighting the good fight documenting the Web APIs!). And this I implemented WebGL2 in Exokit. This wasn’t hard or complicated at all, since Exokit is written in Javascript.</p><p>The cool thing about the new WebGL2 APIs is that they’re “garbage-free”, meaning they take extra offset arguments so you don’t need to construct new typed array views when calling WebGL.</p><p>Since calling WebGL is more than likely the hotspot of your 90FPS Javascript mixed reality code, this means using WebGL2 APIs can take a meaningful load off of the garbage collector.</p><p>That’s precisely why Emscripten wants to use these new APIs, and why I’m thankful for the stack traces it threw at me. Now that we have WebGL2 in Exokit, everyone wins. And Emukit got marginally faster, even for N64VR.</p><p>Speaking of…</p><h3>Back to past of bringing the past to the future</h3><p>For Emukit 2 I wanted to do a little extra as thanks to everyone supporting the project. That meant getting some of the N64 fan favorites booting and running faster in Emukit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FH9pD12SGSN2R2fk2klkVg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R3hQOctjapU4pPk84SinoA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PE8BDaAJeADmeIOYlVYTQA.png" /></figure><p>To that end, I rewrote the N64VR core to use the new unified MRification pipeline I wrote for PSXVR. If you recall from the last post, we were previously hijacking the GL calls, shaders, and matrices coming out of the emulator, but the emulator itself was basically the stock one from RetroArch, inheriting its compatibility problems.</p><p>So I updated and recompiled the N64 emulator from source, backporting some much coprocessor fixes that were getting in the way of the more complicated games booting.</p><p>More specifically, In addition to favorites like OOT/Majora and Star Fox 64, <strong><em>Perfect Dark works now.</em></strong> Multiplayer/Combat simulator too!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8U4ktpk_OjdRbmsTahA14A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cyms5tG8COJnkFCM8ha4rA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fl00fSnfGoux92JRz555mQ.png" /></figure><p>I also added the Model-View-Projection matrix pipeline to the emulator itself, and baked in the perspective unfolding code to the shader so it runs on the GPU. What this means is that everything should be running smoother with less Javascript garbage generation.</p><p>Some games I tested:</p><h4>Nintendo 64</h4><ul><li>Super Mario 64</li><li>Star Fox 64</li><li>Perfect Dark</li><li>Ocarina of Time</li><li>Majora’s Mask</li></ul><h4>PlayStation</h4><ul><li>Metal Gear Solid</li><li>Crash Bandicoot</li><li>Final Fantasy 7</li><li>Spyro the Dragon</li></ul><p>Amazingly, I didn’t have to change the matrices for any of these. This is using the scaling constants that we established with Ocarina of Time and it all just works.</p><p>I also updated the N64VR controls to work better for games that use the full N64 trident, especially shooters.</p><h3>What’s next?</h3><blockquote>With immersive PlayStation and Nintendo 64 in JavaScript, there’s a lot we could do!</blockquote><blockquote>Emukit is our way of exploring what we can do with a mixed reality web browser engine written in JavaScript.</blockquote><p>Some ideas we’ve been hearing:</p><ul><li>Add a multiplayer layer with WebSockets, WebRTC, or invent something better like <a href="https://github.com/webmixedreality/exokit/issues/221"><strong>WebUDP</strong></a></li><li>Integrate natively with <a href="https://unity3d.com/"><strong><em>Unity</em></strong></a><strong><em> </em></strong>and other engines</li><li>Componentize <a href="http://emukit.exokit.org/"><strong><em>Emukit </em></strong></a>into an entity-component system like <a href="https://aframe.io/"><strong><em>A-Frame</em></strong></a></li><li>Run <strong><em>Emukit </em></strong>games as immersive AR portals in <a href="https://www.magicleap.com/"><strong><em>Magic Leap One</em></strong></a></li><li>Hack the ROMs with a tad of JS to integrate new control schemes, like <a href="https://www.leapmotion.com/"><strong><em>Leap Motion</em></strong></a></li></ul><p>Some of the above may or may not be in progress. I hope to share more soon. ;)</p><h3>Show me the VR</h3><p>I made a video walkthrough and of <strong>Emukit 2</strong>, including how to get started. Also talking about hax.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F-350Vu1H5Vk%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D-350Vu1H5Vk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F-350Vu1H5Vk%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/27a61e6a350699addae41aac28026bf2/href">https://medium.com/media/27a61e6a350699addae41aac28026bf2/href</a></iframe><h3>Show me the code</h3><p>Both <a href="http://github.com/exokitxr/exokit"><strong><em>Exokit </em></strong>(the browser engine)</a>and <a href="http://github.com/exokitxr/emukit"><strong><em>Emukit </em></strong>(the emulator site</a>) are open source on Github.</p><p>Check out Exokit: <a href="https://exokit.org/">https://exokit.org</a></p><p>Sign up to our mailing list for updates: <a href="http://eepurl.com/dFiLMz">http://eepurl.com/dFiLMz</a></p><p>Homepage: <a href="https://exokit.org/">https://exokit.org</a><br>GitHub: <a href="https://github.com/exokitXR/exokit">https://github.com/exokitXR/exokit</a><br>Twitter: <a href="https://twitter.com/exokitXR">https://twitter.com/exokitXR</a><br>Instagram: <a href="https://instagram.com/exokitXR">https://instagram.com/exokitXR</a><br>Discord: <a href="https://discord.gg/cf5tfTV">https://discord.gg/cf5tfTV</a><br>Slack: <a href="https://exokit.slack.com/join/shared_invite/enQtNDI3NjcxNzYwMDIxLWU2NmFmOTEzMzk4NWNiYjRhMjVkYzcyNjg5YjUyMzZkYWM1ZGI4M2IwYWZiMjNlMTJjMDlkM2U3Y2JiNTc2M2Q">https://exokit.slack.com/join/shared_invite/enQtNDI3NjcxNzYwMDIxLWU2NmFmOTEzMzk4NWNiYjRhMjVkYzcyNjg5YjUyMzZkYWM1ZGI4M2IwYWZiMjNlMTJjMDlkM2U3Y2JiNTc2M2Q</a><br>Twitch: <a href="https://www.twitch.tv/exokitXR">https://www.twitch.tv/exokitXR</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c1ec110a592" width="1" height="1" alt=""><hr><p><a href="https://medium.com/webmr/emukit-2-immersive-psx-n64-vr-on-the-web-7c1ec110a592">Emukit 2: immersive PSX+N64 VR on the web</a> was originally published in <a href="https://medium.com/webmr">Exokit</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[N64 VR with Javascript]]></title>
            <link>https://medium.com/webmr/n64-vr-with-javascript-e188de42ced5?source=rss-b307c51ad921------2</link>
            <guid isPermaLink="false">https://medium.com/p/e188de42ced5</guid>
            <category><![CDATA[vr]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[nintendo]]></category>
            <category><![CDATA[web]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Avaer Kazmer]]></dc:creator>
            <pubDate>Sat, 30 Jun 2018 20:11:56 GMT</pubDate>
            <atom:updated>2019-04-07T19:50:46.708Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>TLDR</strong>: I made an N64 emulator work in VR with Javascript and WebXR. <a href="https://github.com/webmixedreality/emukit">Emukit is open source</a>.</p><h3>Avaer Kazmer on Twitter</h3><p>Steps for Emukit N64 VR on Windows: 1) Get exokit https://t.co/II3fA2oZUz 2) Run &quot;C:\exokit\exokit.cmd&quot; https://t.co/Lw8NN5xYiQ 3) When the window pops up, drop your ROM to run it. 4) Click inside the window to enter VR. 5) Controls are on the left hand!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ApvhEOYxIUokyZJALfqqFA.png" /><figcaption>Emukit running Ocarina of Time</figcaption></figure><p>I’ve always been interested in emulation. One of my first serious introductions to programming was hacking Japanese Pokemon Gold into English before the Western release. Finishing Final Fantasy 7 in an emulator at 4x speed is what got me into Final Fantasy. And I have twitch reflexes for the classic Zeldas — on keyboard and mouse.</p><p>So I took the last week off of <a href="https://github.com/webmixedreality/exokit"><strong>Exokit browser</strong></a> work to hack on VR-ifying the N64 in Javascript.</p><p>…And fixed several bugs in <strong>Exokit </strong>in the process!</p><h3>Part 1: Emulating</h3><p>I knew VR-ifying N64 with Javascript would have to be theoretically possible.</p><p>There is a project called <a href="http://www.retroarch.com/"><strong>Retroarch</strong></a> that provides an emulator frontend to various emulator back ends. One of those back ends is <a href="https://github.com/libretro/parallel-n64"><strong>Mupen64/P</strong></a>If you have other ideas, the dev team and I are on on <a href="https://twitch.tv/avaer"><strong>Twitch</strong></a>, <a href="https://twitter.com/webmixedreality"><strong>Twitter</strong></a>, and <a href="https://discord.gg/Apk6cZN"><strong>Discord</strong></a>. Come say hi!<a href="https://github.com/libretro/parallel-n64"><strong>arallel N64</strong></a>. And retroarch and many of its cores are already compiled to <a href="https://webassembly.org/"><strong>WebAssembly</strong></a>, with a layer that binds to WebGL.</p><p>When you’ve got that, anything can be intercepted and hacked with Javascript to draw into a headset with WebXR/WebVR. It’s just a matter of hax!</p><p>So the first step was to boot <a href="https://web.libretro.com/"><strong>Retroarch Web Player</strong></a> locally for hacking. This was more complicated than it seems:</p><blockquote>The web player emulates operating system services in the browser.</blockquote><blockquote>And the browser APIs have some impedance mismatch with what you need for implementing an operating system.</blockquote><h4>Corner cases</h4><p>One of those corner cases is <a href="https://www.npmjs.com/package/browserfs"><em>browserfs</em></a>, which is a file system implementation in Javascript with several backing drivers, such as XMLHttpRequest. File systems often support is blocking I/O — but this is somewhat antithetical to Javascript, expecially Javascript emulation, and double-especially Javascript XR.</p><p>The XMLHttpRequest FS layer was relying on the blocking request mode to implement the blocking I/O calls. But this has long been deprecated in web browsers, and there is straight up no good solution for doing this kind of thing in Node, short of binary modules or `child_process.execSync`.</p><p>I wanted to get this working in Exokit for the performance/hackability benefits (<a href="https://www.npmjs.com/package/exokit">Exokit is a JS npm module</a>, and faster than chrome), and I didn’t want to resort to supporting more backwards-thinking APIs. So I just rewrote the XMLHttpRequest usage to use <em>async </em>calls. Luckily there was no real good reason for them being synchronous in the first place and this was an easy switch.</p><p>One other technicality was that it seems by default Retroarch would try to synchronize audio with video and effectively hang at the Genesis refresh rate (60 Hz) to play audio samples. That is insufficient for good Desktop VR (which generally runs at 90FPS), so I fixed that technicality by simply returning from the audio processing loop early. Which killed audio, but made everything else buttery.</p><p><a href="http://twitch.tv/avaer"><em>I DJ on the Twitch streams anyway</em></a>, so it was an acceptable loss — though the Sonic 3 soundtrack is definitely amazing and worth checking out. Some of the tracks were supposedly done by <strong>Michael Jackson</strong>! <em>&lt;/tangent&gt;</em></p><p>So at this point I could boot Retroarch web player into Exokit and play Sonic 3 (<em>&amp; Knuckles!</em>). It worked first try, too! This was both shocking and exciting because at this point we were running an N64 emulator on top of an assembly emulator, on top of an N64 emulator backed by a WebGL emulated with OpenGL.</p><p>Anyway, we were just a couple of small hacks away from a decent UX for booting relatively arbitrary ROMs.</p><h4>We just had to:</h4><ol><li>Hook up a file `drop` listener to grab files from the user’s operating system and inject them into the emulated filesystem in a well-known place</li><li>Load the appropriate WASM/JS bootstrap core for the target emulator for the ROM type (detected by file name), and</li><li>Call the Retroarch WASM “executable” with the right arguments to load the ROM file from its emulated location</li></ol><p>For good measure at this point I booted the Genesis emulator in an Exokit reality tab as a plane whose material texutre is bound to the iframe framebuffer.</p><p>Hooking up basic gamepad events for a D-pad, A, and Start, we could successfully play emulated Genesis in VR.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G8crYGryibO4WPa76l9C8A.png" /><figcaption>Genesis emulated inside WebXR view WebAssembly RetroArch</figcaption></figure><h3>Part 2: Haxing The Matrix</h3><p>So, we had Retroarch booting to a screen in VR. I figured if Genesis worked then pretty much any Retroarch-supported console would work — including N64 — because abstracting ROM emulation into GL calls is precisely the domain of Retroarch, and the main hard part would be supporting the APIs in the first place.</p><p>But making the emulator run on a screen was boring. It could only work as a (potentially multiplayer) 2D screen, but I wanted to challenge myself.</p><blockquote>So I set out to run N64 games in Javascript mixed reality.</blockquote><p>The first step was to read up on how the N64 does 3D rendering. If I understood that I could probably get a good sense of what the emulator would be doing and the code would be much clearer when I looked at it.</p><h4>N64 rendering pipeline</h4><p>It turns out that the N64 had a pretty basic rendering pipeline, without the modern concept of programmabiity or even shaders. It had GPU opcodes that different games used, but at its core the pipeline was fixed in how it pushed triangles to the screen.</p><p>I also learned that the N64 rendering used the right-hand rule, which was a relief since that would be compatible with <a href="https://threejs./org"><em>THREE.js</em></a> without extra hax!</p><p>From this I figured there would be some standard shader I could hook into to hack in a new model-view/projection matrix to reproject triangles into a VR headset with <a href="https://github.com/ValveSoftware/openvr"><em>OpenVR</em></a>. This turned out to be true.</p><p>Since <a href="https://github.com/webmixedreality/exokit">Exokit </a>implements WebGL in Javascript, I could trap and manifest the GL calls list with <strong>console.log</strong>. I just logged all of the <strong>shaderSource</strong> calls to see the source of the shaders were being put into the pipeline.</p><h4>Vertex shaders source</h4><p>There seemed to be only two vertex shaders in Ocarina of Time, one of which was an obvious quad texture map, and one looked like textured scene geometry, judging from the uniforms. I confirmed this by hacking the <strong>gl_Position</strong> with a <strong>String.replace</strong> to offset Link, Epona, and the Sun programs individually.</p><blockquote>It was all just guesswork; I hacked the program based on its I and looked at what happened when I ran the ROM.</blockquote><p>From this I deduced how the vertex shader projection model worked: it takes triangles generated by the CPU, does multiplication to normalize an integer <strong>position.z</strong> to a <strong>float</strong>, negates it, and throws it through another shader which does a standard projection to the screen. There the z is re-negated to follow the right-hand rule — I have no idea why it’s like this.</p><p>Anyway, since I could control the shaders that the emulator was generating, I added my own uniforms for <strong>viewModelMatrix </strong>and <strong>projectionMatrix </strong>with another <strong>String.replace</strong>, and populated them by hacking in a <strong>getUniformLocation</strong> further in the pipeline.</p><p>Initially I set the matrices to identity (no-op) to make sure the multiplication was not exploding. Then I added a translation in the viewModel, which also seemed to work.</p><p>Then I hooked up the HMD matrices coming from Exokit’s OpenVR binding, and the goggle-faced hacking started.</p><p>The first thing I noticed in VR is that Link wasn’t as _thicc_ as he should be, as pointed out in the Twitch chat. That is, the whole game was running in a flat plane. It was the screen we wanted to get rid of!</p><p>But on closer inspection, it wasn’t <em>actually</em> a screen…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pmRCyLEaOZ3Awg-TwJTYAA.png" /></figure><p>It was actually Hyrule field, except superflat!</p><h4>The curious case of Z</h4><p>The different geometries were separated in the Z dimension. One probably couldn’t tell from a 2D render, but you viscerally see those millimeters with the beauty of depth perception ;).</p><p>So I figured the right fix was to multiply the Z by some factor. After all, it was already being converted from int to float with a division, so perhaps the range got borked in the math.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b_EhbP6lNX11SaZqPvaHtA.png" /></figure><p>The problem is that scaling the Z made it clear this is a logarithmic Z, because the nearest points were stretched out to infinity, even while the furthest points were pretty compact. So we can’t fix this by multiplying by a constant.</p><p>I tried guessing at a couple of constant factors for the logarithm with <em>kind of</em> worked: <em>10, 2, 16, 32, 64, and their inverses</em>. But these fixed the scale of some parts of the scene while stretching others. So I knew that this probably wasn’t a pure logarithm.</p><p>That’s when I realized I’d been resetting the `gl_Position.w` factor to 1 instead of keeping its value as provided by the game. By keeping it the projection matrix math ended up being corrected.</p><blockquote>We had Ocarina of Time’s main menu VRified!</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6NwQS6JJo1wpbq2k1lQQig.png" /></figure><p>You notice some interesting things when you do this:</p><ol><li>The N64 clips everything to a literal frustum.</li><li>The menu is technically super tiny in world space! It’s just really close to the camera.</li><li>The skybox is actually _in front_ of the rest of the geometry, but it’s got the depth test disabled.</li><li>The N64 z-culls its geometry with great efficiency.</li></ol><p>It lagged enough to boot us back to the Vive tracking environment. But… it was working!</p><h3>Part 3: Layering</h3><p>At this point we had the N64 emulated scene running but we had no sense of presence in the world. No hands, no other objects, just the game render.</p><p>I figured we should at least have VR controllers, a virtual console, and a controls tutorial in the world to ground us. I had the models lying around from <a href="https://github.com/webmixedreality/zeo"><strong>Zeo</strong></a>!</p><p>But the question beomces: how do you draw models — or anything — into a scene when the emulator owns and trashes the GL context?</p><p>We can of course set up a scene graph with <em>THREE.js</em> to draw our models from the correct HMD perspective, but we need t consider <em>when</em> we do that draw in the render loop, so that it blends with the emulator without any screen wipes or overwrites.</p><p>In the end I decided to go with letting the emulator do its thing, setting <em>THREE.js</em> to preserve the emulator’s context (avoid <em>gl.clear()</em>), and then drawing the console and controllers afterwards. One gotcha was making sure to inform <em>THREE.js</em> that the states of the gl context like buffers, shaders, and textures are unstable via <a href="https://threejs.org/docs/#api/renderers/WebGLRenderer.state"><strong>renderer.state.reset()</strong></a> before rendering.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*Z8U461nIzcJlbD28IpTLoQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GP96opaNhqsmczl8I7pIxA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tAYlLxaLY229Y-CZLsMDqg.png" /></figure><h3>Part 4: Controls</h3><p>The next step was adding controls to actually control the game.</p><blockquote>The first thing I did was sit down, close my eyes, and figure out a mapping for an N64 controller to the Vive controller. Surprisingly, this is doable for all of the keys that matter.</blockquote><p>The second thing I did was to comment out the weird code that Retroarch Web Player had for handling gamepads via the Gamepad API. It didn’t support all of the N64 buttons, it had the analog axes wrong for the Vive touchpad, and I couldn’t be bothered to debug that.</p><p>Instead I chose to do thing I know, which is both easy to implement, and more importantly, debug: `KeyboardEvent` keys!</p><p>I wrote a loop polling the Vive gamepads via the Gamepad API, and mapped that to keyboard events dispatched on the `window.document`.</p><p>The one remaining problem was that Retroarch actually does not have key bindings for all of the Nintendo 64 controller buttons, so we had no actual key to emit.</p><p>I found out that this was controlled by a file called <em>retroarch.cfg</em>. That file didn’t exist, but with some virtual filesystem hacking I managed to inject it into the right place in the boot loop.</p><p>I went into a VR to “confirm that everything was working”. I actually ended up playing for quite a while ogling the chance to literally walk through the scenes that made up my childhood.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KEofxm0Vecdb0n_Uq_e17Q.png" /></figure><h3>Part 5: Optimizing</h3><p>So at this point everything was working, but it was a bit slow.</p><p>It was time to hook up Exokit (which is just a Node module) to the Node profiler via Chrome Devtools and capture a profile. This just worked and immediately showed us the prime suspect for the slowdown.</p><p>It was the matrix hack; Exhibit A was <strong>glGetUniformLocation</strong>.</p><p>Basically, we were querying the shader for the uniform location to use for our <strong>modelViewMatrix </strong>and <strong>projectionMatrix </strong>every single draw call. This multiplied the GL overhead by 3 or so.</p><p>Luckily, it was an easy fix. Uniform locations don’t change once the shader is set up, so we could just cache the location on the <strong>WebGLShader</strong>, which made the overhead once per shader instead of once per frame.</p><p>There was an immediate difference in performance — it was now pretty consistent regardless of scene complexity (number of shaders per frame). But it still wasn’t 90 FPS. It felt more like 60.</p><p>I profiled again and saw that all of the time was taken up by the WASM. That was disappointing, because it meant that there was probably little that could be done to speed up the rendering besides optimizing the WebAssembly implementation of V8 (which I am not at all qualified to do).</p><blockquote>Or, hacking in a pure binary, threaded, non-WASM version of Retroarch as a node module — which we can do in Exokit, but it would cease to be <em>web! No go!</em></blockquote><p>But then it hit me that what felt like 60 FPS might not be a coincidence — it could be that the WASM blob was trying to do its own timing and introduce a delay to hit a frame rate target, as an emulator would is wont to do.</p><p>This turned out to be right and <strong>retroarch.cfg</strong> was capped at 60 FPS! I bumped it up to 90 and bam! We started hitting the frame target most of the time, and there this completely eliminated any instances of us being booted back to the tracking environment.</p><p>That’s it for the functionality. Now it was time for polish.</p><p>Namely, the whole thing needed a controls explainer and drag-and-drop prompt to tell the user to drop in a ROM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0DqIWV5cKL3_k5gDVNFSFA.png" /></figure><p>These were slapped together in Photoshop as a PNG and drawn as textures in the world. And… that’s it!</p><h3>Future directions (possibly, let me know!)</h3><ul><li>More consoles: Snes, Gameboy, Playstation, Dreamcast, Genesis.</li><li>Multiplayer w/avatars.</li><li>Multiple consoles at once.</li><li>Save states.</li></ul><h3>Video</h3><p>Here’s a longer video of the result in action:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F_yQCjKuRRl0%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D_yQCjKuRRl0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F_yQCjKuRRl0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/764fb3e9e1457810bd7b877830a16412/href">https://medium.com/media/764fb3e9e1457810bd7b877830a16412/href</a></iframe><h3>Try it out!</h3><p>If you have an N64 ROM and a headset, you can load it in VR on the live site at <a href="http://emukit.exokit.org/">https://emukit.exokit.org/.</a> <a href="https://github.com/exokitxr/emukit">Emukit is open source on Github !</a></p><p>Firefox might have performance issues; it runs best on Exokit.</p><h3>Keep in touchCheck out Exokit: <a href="https://exokit.org/">https://exokit.org</a></h3><p>Sign up to our mailing list for updates: <a href="http://eepurl.com/dFiLMz">http://eepurl.com/dFiLMz</a></p><p>Homepage: <a href="https://exokit.org/">https://exokit.org</a><br>GitHub: <a href="https://github.com/exokitXR/exokit">https://github.com/exokitXR/exokit</a><br>Twitter: <a href="https://twitter.com/exokitXR">https://twitter.com/exokitXR</a><br>Instagram: <a href="https://instagram.com/exokitXR">https://instagram.com/exokitXR</a><br>Discord: <a href="https://discord.gg/cf5tfTV">https://discord.gg/cf5tfTV</a><br>Slack: <a href="https://exokit.slack.com/join/shared_invite/enQtNDI3NjcxNzYwMDIxLWU2NmFmOTEzMzk4NWNiYjRhMjVkYzcyNjg5YjUyMzZkYWM1ZGI4M2IwYWZiMjNlMTJjMDlkM2U3Y2JiNTc2M2Q">https://exokit.slack.com/join/shared_invite/enQtNDI3NjcxNzYwMDIxLWU2NmFmOTEzMzk4NWNiYjRhMjVkYzcyNjg5YjUyMzZkYWM1ZGI4M2IwYWZiMjNlMTJjMDlkM2U3Y2JiNTc2M2Q</a><br>Twitch: <a href="https://www.twitch.tv/exokitXR">https://www.twitch.tv/exokitXR</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e188de42ced5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/webmr/n64-vr-with-javascript-e188de42ced5">N64 VR with Javascript</a> was originally published in <a href="https://medium.com/webmr">Exokit</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exopod Three: Learn to code XR]]></title>
            <link>https://medium.com/webmr/exopod-three-learn-to-code-xr-3ca7f3450ba6?source=rss-b307c51ad921------2</link>
            <guid isPermaLink="false">https://medium.com/p/3ca7f3450ba6</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[virtual-reality]]></category>
            <category><![CDATA[augmented-reality]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[future]]></category>
            <dc:creator><![CDATA[Avaer Kazmer]]></dc:creator>
            <pubDate>Wed, 06 Jun 2018 16:12:15 GMT</pubDate>
            <atom:updated>2019-04-07T19:51:22.901Z</atom:updated>
            <content:encoded><![CDATA[<p>In this edition <em>Avaer Kazmer</em>, <em>Chris Eddy, </em>and <em>Noah Schiffman</em> sit down and discuss learning to code, getting a programming job, hacker names, DJing while coding and streaming, Exokit web reality tabs, entrepreneurship, and tons more.</p><p>Plus an exclusive podcast remix by Avaer. <em>Stick around for a surprise!</em> 🎧</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1ojP6hLYVHJ4WaaR8L4s3A.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*SCusrsdFwsOxS8kIv7KW6Q.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*Y8k4uIOaitSOmco_w4-Qig.jpeg" /></figure><h4>Inside:</h4><ul><li>Learning to code</li><li>Perspectives from an up and coming coder</li><li>Perspectives from an old programmer</li><li>Javascript framework overload for the modern dev</li><li>Code schools, #worthit?</li><li>Internships</li><li>What a programmer gets paid for</li><li>Frontend, backend, middleware, full-stack, what does it mean?</li><li>The modern coder resume</li><li>Hiring by socials</li><li>Respecting your coders</li><li>The point of open source</li><li>Adding a byte to Google’s home page</li><li>AWS in plain English</li><li>PHP 5 -&gt; 6</li><li>Github acquired by Microsoft</li><li>How #Exokit started with the Magic Leap SDK</li><li>Inverse kinematics for multiplayer avatars</li><li>#Websites become #webspaces</li><li>Reality tabs from browser tabs</li><li>WebVR, WebXR, implementing them in Javascript</li><li>Questions from the Twitch chat!</li><li>Fractal programming on LSD</li><li>Chris’s “My neighborhood” app idea</li><li>(#google #neighbourly ?)</li><li>Avaer demonstrates his FL studio loop skills</li></ul><p>Check out Exokit: <a href="https://exokit.org/">https://exokit.org</a></p><p>Sign up to our mailing list for updates: <a href="http://eepurl.com/dFiLMz">http://eepurl.com/dFiLMz</a></p><p>Homepage: <a href="https://exokit.org/">https://exokit.org</a><br>GitHub: <a href="https://github.com/exokitXR/exokit">https://github.com/exokitXR/exokit</a><br>Twitter: <a href="https://twitter.com/exokitXR">https://twitter.com/exokitXR</a><br>Instagram: <a href="https://instagram.com/exokitXR">https://instagram.com/exokitXR</a><br>Discord: <a href="https://discord.gg/cf5tfTV">https://discord.gg/cf5tfTV</a><br>Slack: <a href="https://exokit.slack.com/join/shared_invite/enQtNDI3NjcxNzYwMDIxLWU2NmFmOTEzMzk4NWNiYjRhMjVkYzcyNjg5YjUyMzZkYWM1ZGI4M2IwYWZiMjNlMTJjMDlkM2U3Y2JiNTc2M2Q">https://exokit.slack.com/join/shared_invite/enQtNDI3NjcxNzYwMDIxLWU2NmFmOTEzMzk4NWNiYjRhMjVkYzcyNjg5YjUyMzZkYWM1ZGI4M2IwYWZiMjNlMTJjMDlkM2U3Y2JiNTc2M2Q</a><br>Twitch: <a href="https://www.twitch.tv/exokitXR">https://www.twitch.tv/exokitXR</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ca7f3450ba6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/webmr/exopod-three-learn-to-code-xr-3ca7f3450ba6">Exopod Three: Learn to code XR</a> was originally published in <a href="https://medium.com/webmr">Exokit</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ExoPod Two: The Magic Leap C API Society]]></title>
            <link>https://medium.com/webmr/exopod-two-the-magic-leap-c-api-society-ad596f3dccb4?source=rss-b307c51ad921------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad596f3dccb4</guid>
            <category><![CDATA[google]]></category>
            <category><![CDATA[magic-leap]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[virtual-reality]]></category>
            <category><![CDATA[augmented-reality]]></category>
            <dc:creator><![CDATA[Avaer Kazmer]]></dc:creator>
            <pubDate>Sun, 27 May 2018 17:32:16 GMT</pubDate>
            <atom:updated>2019-04-07T19:52:08.970Z</atom:updated>
            <content:encoded><![CDATA[<p>In this edition <em>Avaer Kazmer</em>, <em>Joe McCullough, Noah Schiffman, </em>and <em>Nick Loomis </em>slice through the Magic Leap SDK C API that will slice through the very fabric of our being as a society.</p><p>But also, important things!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*6zktPeqEfPWYbVTdMu4EfA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LG1buigcqUI6uCYXDYB-MQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*Y8k4uIOaitSOmco_w4-Qig.jpeg" /></figure><h4>Tune in for goodies:</h4><ul><li>MLSDK C API dissection</li><li>In-app in-world purchases</li><li>Fake news</li><li>Fake soda</li><li>HyperReality reality</li><li>Google’s existential threat</li><li>How to support any game engine</li><li>Elephant bottle refraction theory</li><li>ARKit, ARCore, and Magic Leap</li><li>Media DRM overload</li><li>AR identity platforms</li><li>Judge Dredd hand login</li><li>Jacked-in clickjacking</li><li>Data security is scary</li><li>Antivirus in mixed reality</li><li>Virtual files</li><li>Virtual keyboards</li><li>Multiplayer all the things</li><li>Alzheimer’s memory drive therapy</li><li>Virtual Autism</li><li>#DeleteFacebook</li><li>Technology that uses us</li><li>Pyramid schemes of convenience</li><li>Location becoming irrelevant</li><li>Order a life online</li><li>Order a wife online</li><li>Ready Player One mothering</li><li>The transitional generation</li><li>4 year olds with cell phones</li><li>Nick’s 3rd grade Halo laptop</li><li>Discord seems kindaok but is it ok?</li><li>AI-generated PR</li><li>AI-generated phone sex</li><li>Pointing fingers, blowing whistles</li><li>Buy dat merch</li><li>Jake Paul in the body</li><li>I’m 12 and what is an exopod?</li></ul><p>Check out Exokit: <a href="https://exokit.org/">https://exokit.org</a></p><p>Sign up to our mailing list for updates: <a href="http://eepurl.com/dFiLMz">http://eepurl.com/dFiLMz</a></p><p>Homepage: <a href="https://exokit.org/">https://exokit.org</a><br>GitHub: <a href="https://github.com/exokitXR/exokit">https://github.com/exokitXR/exokit</a><br>Twitter: <a href="https://twitter.com/exokitXR">https://twitter.com/exokitXR</a><br>Instagram: <a href="https://instagram.com/exokitXR">https://instagram.com/exokitXR</a><br>Discord: <a href="https://discord.gg/cf5tfTV">https://discord.gg/cf5tfTV</a><br>Slack: <a href="https://exokit.slack.com/join/shared_invite/enQtNDI3NjcxNzYwMDIxLWU2NmFmOTEzMzk4NWNiYjRhMjVkYzcyNjg5YjUyMzZkYWM1ZGI4M2IwYWZiMjNlMTJjMDlkM2U3Y2JiNTc2M2Q">https://exokit.slack.com/join/shared_invite/enQtNDI3NjcxNzYwMDIxLWU2NmFmOTEzMzk4NWNiYjRhMjVkYzcyNjg5YjUyMzZkYWM1ZGI4M2IwYWZiMjNlMTJjMDlkM2U3Y2JiNTc2M2Q</a><br>Twitch: <a href="https://www.twitch.tv/exokitXR">https://www.twitch.tv/exokitXR</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad596f3dccb4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/webmr/exopod-two-the-magic-leap-c-api-society-ad596f3dccb4">ExoPod Two: The Magic Leap C API Society</a> was originally published in <a href="https://medium.com/webmr">Exokit</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ExoPod One: Magic Leap Secret Pitchdeck]]></title>
            <link>https://medium.com/webmr/exopod-one-magic-leap-secret-pitchdeck-7fa62beadf91?source=rss-b307c51ad921------2</link>
            <guid isPermaLink="false">https://medium.com/p/7fa62beadf91</guid>
            <category><![CDATA[augmented-reality]]></category>
            <category><![CDATA[ready-player-one]]></category>
            <category><![CDATA[virtual-reality]]></category>
            <category><![CDATA[podcast]]></category>
            <category><![CDATA[magic-leap]]></category>
            <dc:creator><![CDATA[Avaer Kazmer]]></dc:creator>
            <pubDate>Sat, 19 May 2018 14:27:42 GMT</pubDate>
            <atom:updated>2019-04-07T19:52:28.400Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1ojP6hLYVHJ4WaaR8L4s3A.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LG1buigcqUI6uCYXDYB-MQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*Y8k4uIOaitSOmco_w4-Qig.jpeg" /></figure><blockquote>You are cordially invited to the inaugural <em>Exopod </em>with <strong>Avaer Kazmer</strong>, <strong>Noah Schiffman</strong>, and <strong>Nicholas Loomis</strong> as they dissect the Magic Leap secret pitchdeck.</blockquote><p>May contain traces of:</p><ul><li>hex editing</li><li>adulting</li><li>science fiction</li><li>science nonfiction</li><li>heaps of money</li><li>no VR feelsbadman</li><li>full dive syndrome</li><li>neural nets</li><li>splitting the atom</li><li>hyper reality</li><li>eldritch demons selling to kids</li><li>RL artifacts</li><li>mysterious audio quality</li></ul><p>Check out Exokit: <a href="https://exokit.org/">https://exokit.org</a></p><p>Sign up to our mailing list for updates: <a href="http://eepurl.com/dFiLMz">http://eepurl.com/dFiLMz</a></p><p>Homepage: <a href="https://exokit.org/">https://exokit.org</a><br>GitHub: <a href="https://github.com/exokitXR/exokit">https://github.com/exokitXR/exokit</a><br>Twitter: <a href="https://twitter.com/exokitXR">https://twitter.com/exokitXR</a><br>Instagram: <a href="https://instagram.com/exokitXR">https://instagram.com/exokitXR</a><br>Discord: <a href="https://discord.gg/cf5tfTV">https://discord.gg/cf5tfTV</a><br>Slack: <a href="https://exokit.slack.com/join/shared_invite/enQtNDI3NjcxNzYwMDIxLWU2NmFmOTEzMzk4NWNiYjRhMjVkYzcyNjg5YjUyMzZkYWM1ZGI4M2IwYWZiMjNlMTJjMDlkM2U3Y2JiNTc2M2Q">https://exokit.slack.com/join/shared_invite/enQtNDI3NjcxNzYwMDIxLWU2NmFmOTEzMzk4NWNiYjRhMjVkYzcyNjg5YjUyMzZkYWM1ZGI4M2IwYWZiMjNlMTJjMDlkM2U3Y2JiNTc2M2Q</a><br>Twitch: <a href="https://www.twitch.tv/exokitXR">https://www.twitch.tv/exokitXR</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7fa62beadf91" width="1" height="1" alt=""><hr><p><a href="https://medium.com/webmr/exopod-one-magic-leap-secret-pitchdeck-7fa62beadf91">ExoPod One: Magic Leap Secret Pitchdeck</a> was originally published in <a href="https://medium.com/webmr">Exokit</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why mix reality?]]></title>
            <link>https://medium.com/webmr/why-mix-reality-3d2ff6e9a03e?source=rss-b307c51ad921------2</link>
            <guid isPermaLink="false">https://medium.com/p/3d2ff6e9a03e</guid>
            <category><![CDATA[web]]></category>
            <category><![CDATA[mixed-reality]]></category>
            <category><![CDATA[virtual-reality]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[sonic-the-hedgehog]]></category>
            <dc:creator><![CDATA[Avaer Kazmer]]></dc:creator>
            <pubDate>Sun, 13 May 2018 02:59:58 GMT</pubDate>
            <atom:updated>2018-05-13T02:59:58.362Z</atom:updated>
            <content:encoded><![CDATA[<p>I’ve kind of always lived in the future, though it took a long time to realize it.</p><h3>The Curious Case of Sonic 2</h3><p>My life was set when I got a Sega Genesis for my birthday one Christmas (yes they are the same thing).</p><p>This was a complete accident, but it truly set the stage of what I will (hopefully) do for the rest of my life, which is make things with code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*KcDWYO5wNlRetAcvIE6gcg.jpeg" /><figcaption>Sonic <em>The Hedgehog </em>2 for Sega Genesis</figcaption></figure><p>My childhood was spent 30% playing <em>Sonic The Hedgehog 2</em>, 30% hacking the game to cheat my friends and win, and 30% running around the house like a maniac pretending I was in the game.</p><p>The grown-ups thought my time spent running around the house making mouthnoises was me roleplaying Sonic. Sonic’s nemesis, the evil Dr. Robotnik, would capture animals and construct elaborate traps for them. It was up to Sonic to break the machines and save the day.</p><p>But the truth is my childhood fantasy was to be Dr. Robotnik. I wanted to make these impossible worlds of crazy machines that seemed to span the globe. Sonic wasn’t my hero; he was an explorer of worlds built by my hero.</p><p>Nobody really understood what I was trying to accomplish with my ROM hacks on the family computer. But I was making levels and objects and enemies and (embarassing) AIs for the game.</p><p>The rendering technology really sucked and half of the things I made couldn’t be seen in anything but code, but I made up for it with a vivid imagination. I don’t think anyone else saw what I saw when I looked at a blank slate; I saw robots.</p><p>But I think everyone knew from the beginning that my life’s work would be as a programmer.</p><h3>Interesting times</h3><p>For a long time, computing was a thing accessible only to the people with vivid imaginations — it’s no coincidence that geekdom, social ostracism, and dungeons and dragons were par for the course among the early computer crowd.</p><p>Of course, when Drake plays Fortnite with Ninja and it gets covered in mainstream news, the tables have turned. Being a games programmer transferring one’s imagination into the player’s brain for fun and profit is one of the most sought-after careers in the modern world.</p><p>You no longer need to be a really messed-up kid with crazy daydreams to “get” computers. You can see the thing. You can participate in it, be it Instagram, Youtube, or the game du jour on Steam.</p><p>It used to be a controversial idea, or even one considered disordered, for a person spent their lives living in and experiencing an imagined reality through a computer.</p><p>I don’t think that’s a controversial idea anymore. But there is an equally controversial idea today:</p><blockquote>In the future there are no screens, and no computers.</blockquote><h3>The reality we don’t want</h3><p>Imagine you’re bored.</p><p>You’re probably going to crack open a tab with Twitter, Facebook, or Instagram. Maybe Twitch. Steam or console if you’re younger. Wikipedia, if you’re old. Or maybe it’s a game on your phone.</p><p>You’ll probably get flack from someone that you should be working, or doing chores instead — unless the other person is doing the same case, in which you get a free pass.</p><blockquote>We spend our free time living in computers. The real life forced on us is annoying.</blockquote><p>Doesn’t this seem crazy? I don’t mean that it’s crazy that humans have lost touch with their humanity — that’s silly.</p><p>I mean that it’s crazy that we’re all half-living the lives we want to live.</p><ul><li>Why can’t we do the thing we want to be doing, all of the time?</li><li>Why do we need to afk to rl?</li><li>Why is work a place we go?</li><li>Why <em>is </em>location?</li></ul><blockquote>Contrarian thought of the day: these are all technology problems</blockquote><p>Rather, they are technology symptoms under a single technology problem. And that problem is mixed reality.</p><h3>A magicless leap</h3><p>The elders speak of a forgotten original Magic Leap pitch deck. I’m kidding of course; <a href="https://www.docdroid.net/o8vbuRZ/61845907-1-0.pdf">it’s here.</a></p><p>Three things strike me as prescient, real, and <em>non-magical</em> about it:</p><ul><li>It’s billed as a web browser; the thing we already spend all day in</li><li>It aims to turn life into a game; the thing we all want to be doing</li><li>It doesn’t shy away from the problem of virtual work; a thing we’ll still need to do to</li></ul><p>This isn’t a post about Magic Leap. I don’t know if what Magic Leap ships will meet any particular expectation.</p><p>What I do know is that the key ideas in this deck <em>will</em> happen — because we all want them to happen, and if there’s one thing you can count on human will to make things happen. And when mixed reality comes of age, we’ll have a whole slew of new questions to ask.</p><ul><li>What is the meaning of work when work is a game? What is money?</li><li>If I can be with anyone anytime, what is a friend?</li><li>If location doesn’t matter, why a car? Why roads? Why cities?</li><li>What the heck do I need all these old screens for?</li></ul><p>These are all fascinating to ponder and I hope to answer some of them in my lifetime.</p><p>But the reason I’m working on mixed reality is much simpler than any of these. It’s a dumb thought experiment I found myself pondering in bed one night.</p><h3>Avaer’s Basilisk</h3><p>You know <a href="https://rationalwiki.org/wiki/Roko&#39;s_basilisk">Roko’s Basilisk</a>?</p><p>It’s this silly idea that you will be retroactively punished by AI in a future simulation for not building the AI today — and this might be a simulation, so it’s time to get to work.</p><p>I toned down the idea and named it after myself because I’m lazy. Here’s <em>Avaer’s Basilisk</em>:</p><blockquote>If you believe in a possible reality you deeply desire, and you’re not working on making it happen, then today is a punishment.</blockquote><p>I truly believe that Mixed Reality (tm) is a possible future.</p><p>It’s one I’ve wanted ever since I spent my days roleplaying Robotnik, constructing robots with sucky computers made up for with imagination. I can only hope to one day point to the robots strewn around the living room.</p><p>Today I’m working on <a href="https://github.com/webmixedreality/exokit">Exokit</a>. It’s tough, but it’s not a punishment.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3d2ff6e9a03e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/webmr/why-mix-reality-3d2ff6e9a03e">Why mix reality?</a> was originally published in <a href="https://medium.com/webmr">Exokit</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>