<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Resn - Medium]]></title>
        <description><![CDATA[Infecting your screen for your enjoyment. - Medium]]></description>
        <link>https://medium.com/resn?source=rss----1e1081262853---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Resn - Medium</title>
            <link>https://medium.com/resn?source=rss----1e1081262853---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 21:25:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/resn" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Making a connected flip-dot installation]]></title>
            <link>https://medium.com/resn/making-a-connected-flip-dot-installation-9116df42f71f?source=rss----1e1081262853---4</link>
            <guid isPermaLink="false">https://medium.com/p/9116df42f71f</guid>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[internet-of-things]]></category>
            <category><![CDATA[installation]]></category>
            <category><![CDATA[flip-dot]]></category>
            <category><![CDATA[slack]]></category>
            <dc:creator><![CDATA[Resn]]></dc:creator>
            <pubDate>Mon, 22 Jul 2019 10:03:19 GMT</pubDate>
            <atom:updated>2019-07-22T10:24:45.624Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VTWodmTb0x438MFYIKKEEA.png" /></figure><p>Sourced from the <a href="https://resn.co.nz/#!/work/speedfactory-activation/">Speedfactory</a> installation, adidas was kind enough to leave us with two custom made flip-dot panels. These had sparked our attention since the moment they were installed in the project and for a while they had been collecting dust in a backroom of our office. Now that we had some time to play with these panels again, we built ourselves some connected communication boards for internal use in our office displaying birthdays, lunch time, Twitter mentions and other special company updates and messages.</p><h4><strong>Flip-Dot</strong></h4><p>“What is a flip-dot panel?” you might think. Well, simply said it is an electromechanical display with a very small pixel density and a minimal colour range. Our 100×40cm displays have a resolution of 70×28 dots which can either be black or white, nothing else. Each dot can be controlled individually to flip physically between its black and white side with a refresh rate of about 10 fps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fzybw3FetRHveShEw0l_Jg.png" /></figure><p>These flip-dots are produced by Alfazeta and come in panels of 7×28 dots. Our displays consist of 10 of these forming a 70×28 grid.</p><h4><strong>Slack Bot</strong></h4><p>The idea was to use the panels in our office to display internal updates and messages. Since Slack is our common communication software we thought it would be ideal if we could send messages to the flip-dot panels from within this software.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IRODxnc3Z9BCTZr2ByWAdw.png" /></figure><p>We quickly set up a Slack app with a /flipdot slash command in our workspace and wrote a node.js server and client to catch any payload sent to it. This client painted the message to a canvas using an optimised pixel font and sent the image data to the flip-dot as a TypedArray.</p><p>This worked perfectly for text messages but we also wanted the ability to send images or even videos to the flip-dots.</p><h4><strong>Slack Bot v2</strong></h4><p>Instead of slash commands this turn we opted for a real Slack Bot that could be targeted for direct messages. Now we could write messages and send files to @Flip Dot directly from our Slack workspace. The server and client scripts had to be slightly adjusted to handle the new bot and files types.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*xFb02BLg_0VmGOK4HqEKMg.gif" /></figure><p>The video system required a bit of special attention because of the flip-dots low frame rate. We couldn’t simply stream a video in real-time, thus we worked out a promise based system that listens to completed writes by the flip-dot panel’s controller before painting the next frame.</p><h4><strong>The Client</strong></h4><p>Each flip-dot panel is controlled by its own Intel NUC (mainly because we had these on hand, but this could very well be substituted with a Raspberry PI). <a href="http://pm2.keymetrics.io/">PM2</a> — a node.js process manager — is programmed to run on Windows startup. The manager starts our node.js client script and <a href="http://www.yoctopuce.com/EN/virtualhub.php">VirtualHub</a>. This is a small software that handles communication between Javascript and the Yoctopuse — a micro controller that enabled us to connect the flip-dot panel to the NUC via USB.</p><p>PM2 also makes sure that scripts reboot when they crash and has some other handy features like a logging system.</p><p>Since we are not 24/7 at the office the NUC’s are programmed in their BIOS to power on every morning at 8:55h and a Windows task in scheduled at 19:00h every day to power off the machine. The BIOS is also programmed to power on the NUC automatically after a power failure. Together with the PM2 setup the installation handles itself completely without any human input.</p><h4><strong>USB Switch</strong></h4><p>Actually up till now that last line is not completely true. The flip-dot panels itself don’t have any power switch and need to be unplugged manually at the end of the day. To synchronise the flip-dots’ power states with the NUC’s schedule we attached 5v/220v relay modules to the power circuits and connected the input via USB to the NUCs. Now when the computer turns on it will send a 5 volt signal to the relay closing the power circuit of the flip-dot panel. When the computer shuts down no more power will be sent to the relay and the switch will cut the panel’s power supply.</p><h4><strong>More Fun</strong></h4><p>Now that we have a stable setup we can extend the functionalities of our office’s new message boards. The things we already made include:</p><ul><li>Anniversary Bot: scans our HR software for birthdays and Resn anniversaries of employees every day.</li><li>Lunch Bot: listens to a sign from our office chef in the lunch channel.</li><li>Beer Bot: scheduled to print its message on Friday’s end of day.</li><li>Twitter Bot: printing mentions of @resn_has_no_i.</li><li>Clock: a simple digital clock.</li><li>Countdown: a countdown to a set date and time.</li><li>Image and Video upload: displays the file and stores it for easy access later on.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*ilCuJ_jvTxl8a0HrUOemlg.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*vj6k-cDzv5J4u8c7eU3Uvg.gif" /></figure><p>Let’s see what else we can connect to our new toy!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9116df42f71f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/resn/making-a-connected-flip-dot-installation-9116df42f71f">Making a connected flip-dot installation</a> was originally published in <a href="https://medium.com/resn">Resn</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Banditø immersive experience]]></title>
            <link>https://medium.com/resn/bandit%C3%B8-immersive-experience-f258361dead3?source=rss----1e1081262853---4</link>
            <guid isPermaLink="false">https://medium.com/p/f258361dead3</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[twenty-one-pilots]]></category>
            <category><![CDATA[design]]></category>
            <category><![CDATA[development]]></category>
            <category><![CDATA[webgl]]></category>
            <dc:creator><![CDATA[Resn]]></dc:creator>
            <pubDate>Thu, 17 Jan 2019 08:41:59 GMT</pubDate>
            <atom:updated>2019-06-28T15:59:51.593Z</atom:updated>
            <content:encoded><![CDATA[<p><em>An article written by </em><a href="https://twitter.com/jesper_vos"><em>Jesper Vos</em></a><em> for Resn.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xGZLCse1S4T9O43rJDNpFA.jpeg" /></figure><p>Twenty Øne Piløts released its latest album, Trench, accompanied by a convoluted lore about a homonymous fictional world wherein nine ruling bishops keep its population suppressed in the dark and dystopian city, Dema. This generates a contrarian organisation called Banditø, which is determined to escape the bishops’ suppression and seek a place to create a life of their own.</p><p>The band teamed up with Spotify In-House and Resn to recreate this world and give the fans the possibility to experience the journey from Dema to Trench for themselves.</p><p>In this article we will take you through our process of creating the immersive <a href="https://www.imabandi.to/">Banditø experience</a>, from concept to technical execution.</p><blockquote>Only interested in the technical details? Scroll down to “<strong>Technical”.</strong></blockquote><h3>The Concept</h3><p>The experience’s main focus is the journey from Dema to the Trench. We wanted the user to enter an experience that fully embodies this odyssey — travelling through a world built out of moody environments, cryptic graphics and hidden messages.</p><p>Although the Banditø experience was preceded by three music videos — setting some of the scenes we needed to integrate — we created a rough storyboard to define the timeline and the mood of the five chapters we would build.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ML4LGUuDL_HfEKIAYAhGgQ.png" /><figcaption>Timeline and mood of the five chapters</figcaption></figure><p>Each of the chapters connects to one of the five sections in the song’s structure, while a continuous transition from dark to light and from realistic to stylised creates a cohesive narrative. This transition symbolises the journey from conforming to being free.</p><p>We have given the user control over the camera and created multiple paths to the destination. Each path is connected with the other and the user can move between them at certain moments in the journey. This allowed us to enrich the experience but it’s also a metaphor to the mental journey from anxious to liberated. Everyone’s process is different.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RywF2rP9cBeGE4GgVO1Tig.png" /><figcaption>Different paths can be taken to reach the destination</figcaption></figure><h3>Environments</h3><p>Relative to the chapters, we designed five environments. Each with their own look and feel based on their position on the timeline.</p><h4>Dema</h4><p>Upon landing in the experience the user is introduced to Dema — a dark dream sequence where the audience escapes the conformist city.</p><p>A succession of five scenes, from the top of the tower to the hidden tunnel, illustrates the user’s escape. Each of these settings was modelled after surroundings seen in the <a href="https://www.youtube.com/watch?v=hMAPyGoqQVw"><em>Nico and the Niners</em></a><em> </em>music<em> </em>video. Recreating the scenes from the music video was very important since the dedicated fanbase was already familiar with the environments they had seen there.</p><p>In the Banditø experience however, we completely desaturated the colors and applied some heavy fog and hard shadows to achieve the dark and cold feel we were looking for in the first chapter to conform to our timeline concept — dark to light.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g71VZ07uiL592Wtod5vJzA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X7vqeGg-9aP4kwSlClrYTw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aj4XJYXSyZHFos1Eq3_N_w.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y-FP2aqGj4Wcy8TlHPf0fw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yTibuYOH9NFRtshXVW5KuQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FrKzfEPCCA4OtkVwwammLg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*283nnOFGpO_XjqPadIUvXw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9nITqsgL5aeJKuBBswbK8A.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nEIgH1_2-9lMM4pOZhZhCQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rci2XaK77YZbeWp0W2BTsw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IrhvUYzmJAhshRQ8zC38OQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pTOJubwEPv3H5llSYF-z4Q.jpeg" /><figcaption>Left: Stills from <a href="https://www.youtube.com/watch?v=hMAPyGoqQVw">Nico and the Niners</a> music video. | Right: Stills from Banditø experience</figcaption></figure><p>As an additional layer of content and interaction we blended in some short image sequences from the music videos. These “flashbacks” could be triggered by clicking or tapping the scene. We gave every scene its own specific set of images to match the environment.</p><h4>Planes</h4><p>Leaving Dema’s low lit tunnel overexposes the user’s “eyes”. Upon adjusting to the brighter daylight they find themselves in a vast open landscape. Progressing in the scene, the user might get a feeling of anxiety in the endless space. Even more so when five shadows appear to be chasing them.</p><blockquote>🎼 “When in between two places, Where we used to bleed”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NHdgOAzd6TvJvEWe2juzJA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BNzUd91G3bc9vqBNWaDHXg.jpeg" /></figure><p>Within the scene there are a series of “portals” that are connected to similar wide open landscapes. This creates a maze where the user feels lost. At the same time, since different paths can be chosen, this scene will give them a feeling of control.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fEhV4eim1wFOlavMkqPliw.jpeg" /><figcaption>Portals connect the landscapes</figcaption></figure><p>Similar to Dema, we added the “flashback” interaction in this scene. Again we curated the image sequences to match the environment.</p><h4>Set piece</h4><p>The shadows from the previous scene have now caught up with the user. Are they friend or foe? The camera slowly rotates revealing that the user is surrounded by shadowy figures. Gradually they reveal themselves as Banditøs.</p><p>Again, the scene includes recognisable objects. This time less obvious than before, but the users find themselves standing next the burning car seen in the <a href="https://www.youtube.com/watch?v=UOUBW8bkjQ4"><em>Jumpsuit</em></a> music video.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qmxHbyLO3SMGB_Fr6a9pnw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DC69ScN0wNOwV7VSKuXeDg.jpeg" /><figcaption>Left: Still from <a href="https://www.youtube.com/watch?v=UOUBW8bkjQ4">Jumpsuit</a> music video. | Right: Still from Banditø experience</figcaption></figure><p>The set piece accompanies the song’s bridge. It’s the section that connects the dark and realistic settings to the light and stylised scenes. Like the bridge contrasts with the verse in the song, the set piece contrasts with the other scenes visuals. Hence the bright yellow background and duotone details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*LDTJbOPxIrpV20hFXoYYSA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*sOARcnM5tla5bosUiDvcNA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*G9F2JAWSTwHyi6slROobMg.png" /></figure><p>We gave the user the power to uncover the silhouettes by clicking on any of them. This would reveal the floral patterns on the uniforms and identify them as Banditøs.</p><h4>Hills</h4><p>After the set piece the experience continues in a more figurative and stylised world. The user enters a vast open landscape with hills and plateaus where they can decide to create or destroy. Similar to the Planes chapter the scene offers a series of “portals” that are connected to similar landscapes.</p><blockquote>🎼 “I created this world. To feel some control. Destroy it if I want”</blockquote><p>Not only can the user choose his own path in this scene, they also control the landform by clicking anywhere to create or destroy hills.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mX1rnhB2r7RCdteqW1830Q.jpeg" /></figure><p>As the user progresses through these landscapes they will start to notice that the world is morphing. As the song progresses in intensity the deformations will increase until the climax.</p><h4>Gorge</h4><p>This is the last part of the journey. The user is progressing through a gorge, highly stylised to the Banditø colors and patterns. With a nod to the <a href="https://www.youtube.com/watch?v=uv_1AKKKJnk"><em>Levitate</em></a><em> </em>music video more and more Banditøs can be seen on each side of the gorge, above the user, throwing down yellow flower petals.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*19Cr7-L5ac0sOFGOgbOwAA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_fKy9ILn0lhoDpeBWw0OpQ.png" /><figcaption>Left: Still from <a href="https://www.youtube.com/watch?v=UOUBW8bkjQ4">Jumpsuit</a> music video. | Right: Still from Banditø experience</figcaption></figure><p>At the end of the gorge the user finds the Banditø camp. At this point full 360 degrees control of the camera is available, representing the free mind.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yfjLg9dg56bWnDD44Cj5Cw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1SvLvp8rV4N_OpcTEd6BJw.jpeg" /><figcaption>Left: Still from <a href="https://www.youtube.com/watch?v=uv_1AKKKJnk">Levitate</a> music video. | Right: Still from Banditø experience</figcaption></figure><p>The user continues forward into an open ending of the story. Completely free to create their own.</p><h3>Hidden Content</h3><p>Lots of metaphors and cryptic graphics were incorporated into the experience. To satisfy the curiosity of TØP’s very dedicated fanbase we placed five hidden symbols throughout the experience. Since the release of the album the meanings of these five symbols have been a mystery to the fans.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*lINb_LDNz-yf_TplVLZhGQ.png" /></figure><p>Upon finding the symbols the user gathers “secret” documents that explain the meaning of each.</p><h3>Technical</h3><p>Now that you know why we made the experience as it is, let’s have a look at how we made it work.</p><h4>Models</h4><p>Since we were making a 3D experience and we wanted to match the settings from the band’s music videos, we had to recreate what we saw as 3D models. Keeping the vertex count low on these was by all means necessary. Since we knew the camera freedom was restricted (always looking forward) we didn’t have to build geometry for the backside of the models, and any other side that could not be seen during the experience. This saved quite a lot of vertices and thus increased overall performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VJVaq8wFHOLRnV1DQDDymw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EYX0RRI9nw40Iy0O47JA8A.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hq4rxSGvUtmZMG4AHAPfcw.jpeg" /><figcaption>Some of the models created for the Dema scenes</figcaption></figure><p>To further optimise the rendering, we opted for unlit materials in the whole experience. This meant we had to bake lighting and shadows in all textures. Since many models shared the same albedo map, we exported shadow maps separately for every material. In these textures we used different channels for shadows and light sources — multiplying the albedo with the red channel and replacing it with full white based on the green channel of the shadows map.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*vM_VqoyyrqaWFbJIkOyALg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ytSeG3QR_wZfn0ZDfu6GqA.png" /><figcaption>Left: Albedo map | Right: Shadows map</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V99yXGUnMRggmIiZhXKaCA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CS8BSOfuErk24euYiZAJTQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D9DXI7epyTT-1VbG0JGFdw.jpeg" /><figcaption>Left: Raw model | Middle: Model with albedo map | Right: Model with shadows map</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8KWI2AbfBB_odfKdk9UhjA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2aXW9AbJkiq0K7fN7XN8Rw.jpeg" /><figcaption>Left: processed albedo and shadow map | Right: final scene with fog and post processing</figcaption></figure><p>To speed up initial load time we decided to progressively lazy load scenes. At the same time this meant that we would have to upload textures to the GPU while the experience is playing. Because of the high CPU load of this process we noticed huge frame drops during initialisation of new scenes.</p><p>To get back to a smooth experience we converted all images to GPU-ready textures using the <a href="https://www.khronos.org/opengles/sdk/tools/KTX/file_format_spec/">ktx file format</a>. This format can be read by the GPU directly so no decompression by the CPU is needed, removing the frame drops completely.</p><h4>Landscapes</h4><p>To create the smoothly animated terrain in the Planes and the Hills we had to start from scratch to overcome some technical hurdles. We needed a terrain that:</p><ul><li>Could display a complex deformation</li><li>Had a diverse texture that doesn’t look tiled</li><li>Update at a fast rate without causing any frame drops</li></ul><p>We ended up using a custom generated mesh that was deformed by a mix of displacement maps and was procedurally textured in the fragment shader. This mesh had a high poly count near the camera and transitioned to a low poly count in the distance as a Level of Detail implementation (LOD). We then exported this mesh to a GLTF file to avoid having to generate the mesh at runtime.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7vXQ7Yy_e5bevclN2-VSKw.png" /><figcaption>Landscape displacement maps</figcaption></figure><p>In the shader, we mixed multiple textures at different scales to create variety in the look. We applied the displacement texture to a plane in Cinema4D to render out a Global Illumination pass. In turn we multiplied that texture on top of our custom mesh to add high quality lighting that matched up perfectly with our deformation. Finally, to add to the atmosphere, we drew an extra layer of fog on top that was deformed over time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ahNjqRT6Ujhp2FajxTOTBg.jpeg" /><figcaption>Terrain geometry wireframe</figcaption></figure><p>Now we still had to create the illusion of traveling through the landscapes. To prevent artefacts in the moving terrain, we avoided sliding the displacement map on top of the mesh. This would cause each vertex to animate up and down, accentuating the resolution of the mesh. Instead, we decided to animate the mesh’s position. This only meant that, after a little while, we would see the low-vertex geometry close to the camera and even run out of geometry. So we had to find a middle ground between these two approaches. We ended up animating the mesh to the second level of detail, then reset its position and update the displacement map accordingly to achieve a smooth, seamless and continuous motion.</p><h4>Portals</h4><p>The Planes and Hills scenes feature a series of “portals” through which the user can travel to different landscapes. In its essence a portal is simply a mask, and we knew that masking in GL can be done by using the stencil buffer.</p><p>Render order is particularly important when working with the stencil buffer. These are the render calls we made to achieve the effect:</p><ol><li>Render portal geometry to stencil buffer.</li><li>Render portal landscape to frame buffer.</li><li>Render main landscape.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vmE0jJp7DPpUWdfhUpcHvQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VOtm340RyARzpVEIadx-0g.png" /><figcaption>Left: Landscape rendered without stencil — Right: Only the stencilled landscape</figcaption></figure><p>We used Threejs layers to split the portal geometry from the landscape in order to render it separately to the the stencil buffer.</p><p>Finally we rendered the portal landscape texture from the frame buffer object to screen coordinates and onto the main landscape. At the moment the camera is at the position of the portal a “swap” is executed, switching the portal landscape to be the main landscape.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uh1pfuX90Beqcuoi2yLuaw.png" /><figcaption>Final portal</figcaption></figure><p>We tried a method wherein we additionally inverse-stencilled the main landscape to reduce load on the fragment shader, but this didn’t seem to boost performance since an extra draw call would be necessary.</p><h4>Petals</h4><p>We spent a lot of time creating a custom particle system that would fit our needs for the flower petal rain in the final scene of the experience. We wanted the system to support:</p><ul><li>Thousands of flower petals.</li><li>A depth of field simulation.</li><li>Spiral motion around the camera.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_fKy9ILn0lhoDpeBWw0OpQ.png" /></figure><p>In order to fluidly render the amount of petals we needed we used Instanced Geometry for which we copied the mandatory attributes from a simple, custom made, petal model we imported from a gltf file. We generated additional attributes that would determine the position of each petal in 3D space. Based on the z-position we could sample the correct texture from the petal atlas that we made. This way we achieved the Depth of Field (DoF) simulation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*ukzG3Ki2YegOBucmBUS6TA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qOzNCUCzE8q2KqQMDtngSA.png" /><figcaption>Left: petal vortex sketch — Right: petal texture atlas for DoF</figcaption></figure><p>Since all petals now had a set z-position we could animate them in a radial motion around the camera while “falling” from the sky. Since each petal rotated around the centre with its own radius (its z-position), the distance to the camera was constant — keeping its DoF texture sample stable and precise. To keep the distances stable this also meant that we had to move the entire vortex to be centred around the camera at all times, but since it consists of only a single instanced geometry this was no big deal.</p><p>So that’s a behind the scenes look into the creation of Banditø — an immersive experience that generated over 450 thousand sessions on the first day. With an average duration of 05:19 minutes and 60 thousand sessions at peak hour we are very pleased with the results.</p><p>Thanks to Spotify In-House and Twenty Øne Piløts for this amazing collaboration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*0Cwjrv1_lhZNvDaDuZCaRA.png" /><figcaption>View the experience on <a href="https://www.imabandi.to/"><strong>imabandi.to</strong></a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f258361dead3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/resn/bandit%C3%B8-immersive-experience-f258361dead3">Banditø immersive experience</a> was originally published in <a href="https://medium.com/resn">Resn</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Resn’s Little HelpAR]]></title>
            <link>https://medium.com/resn/resns-little-helpar-48618976c336?source=rss----1e1081262853---4</link>
            <guid isPermaLink="false">https://medium.com/p/48618976c336</guid>
            <category><![CDATA[ar]]></category>
            <category><![CDATA[augmented-reality]]></category>
            <category><![CDATA[webgl]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[christmas]]></category>
            <dc:creator><![CDATA[Resn]]></dc:creator>
            <pubDate>Mon, 07 Jan 2019 16:19:13 GMT</pubDate>
            <atom:updated>2019-01-07T16:19:16.782Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8IA_r1p6skQuxa2e.jpg" /></figure><p>Resn’s Little HelpAR is an update of a very Resn yuletide tradition. This year in trembling anticipation for AR on the Web we forged a new experience that was built on the last, one which tips a bell-festooned hat to the emerging world of augmented browser experiences.</p><h3>AR Research</h3><p>First things first, we asked ourselves what Augmented Reality actually means and how it works.</p><p>Simply put, AR can project digital environments and information over the real, physical world. Currently it is most common to use a smartphone to use this technology. Sure, glasses are out there but no one owns a pair.</p><p>The two current major AR platforms, ARKit and ARCore, are already quite well documented. These documentations and other information sources confirm that augmented reality is most often based on the Simultaneous Localisation And Mapping (​<a href="https://www.andreasjakl.com/basics-of-ar-slam-simultaneous-localization-and-mapping/">SLAM</a>​) technique. The core principle of SLAM is to align the digital world with the physical one. ARKit, for example, achieves this by combining movement estimations (from accelerometers) and feature point tracking through computer vision. This specific combination of input and data processing is called ​<a href="https://developer.apple.com/videos/play/wwdc2018/610/">Visual Inertial Odometry</a>​.</p><h3>FAR (Faux Augmented Reality)</h3><p>Resn is an internet company — We love the Web for its versatility and accessibility. So it follows that if we were to make an AR experience then it would need to work in browser.</p><p>We explored the possibilities of the WebXR spec in experimental browsers, but wanted the Little HelpAR to be accessible for a broad, festive audience. In response to that internal brief, we started researching the possibilities and restrictions of what we coined: Faux Augmented Reality (FAR).</p><p>After prototyping multiple odometry and tracking systems we learned that we wouldn’t have the time (Christmas was fast approaching!) and capacity to implement any proper position tracking system in a mobile browser. Luckily with the gyroscope we still had access to accurate orientation data; that’s 3 out of the 6 degrees needed for a full AR experience, the other 3 we figured we’d handle with some good ol’ fashioned creativity.</p><h3>FAR creative solutions</h3><p>We realised that when we placed objects in a FAR scene they would not stick to any position, or scale to fit the ‘real’ world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*9aHiEEPSWfzdwNUK1YzOeg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*0ua5euNa5VlCyQhflewMsA.jpeg" /><figcaption>The same model looks big on the floor and small on the desk</figcaption></figure><p>In AR, casting shadows underneath models boosts the illusion of alignment and place. However in our 3-degrees-of-freedom FAR experience shadows did the opposite, since it’s impossible to anchor a model to a specific real-world position.</p><p>But Jethro is a magical elf with magical powers and we got around this issue by making him levitate in space. With this approach we didn’t have to anchor Jethro to any surface and the resulting illusion of him being in the real world is much more convincing.</p><p>Jethro’s integration felt very believable… up until the point that we would try to walk towards him at which point he kept the same distance at all times.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*ZtUJpzCMX2ftJh2W.jpg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*jZ-jtaFtutPNEkGF.jpg" /><figcaption>Left: without acceleration physics — Right: with acceleration physics</figcaption></figure><p>To counter this we used the accelerometer data to update Jethro’s position and apply forces to his limbs and head, giving the illusion of ‘swimming’ away from, or towards annoying human beings.</p><h3>Content and interactions</h3><p>Little HelpAR was a sequel of sorts to ​<a href="http://littlehelper.resn.global/">last years​</a> wild experience. With that in mind it was necessary for us to bring back the same visual style and absurd elements used to create the myth of Jethro the elf with the massive addition of another dimension, taking things up from 2D to 3D.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VM-LvFanQemISYKp.png" /></figure><p>We used ​<a href="https://www.mixamo.com/">Mixamo</a>​ to automate the skeleton construction with all the textures painted by hand. His shiny new Gnome-Cler jacket for instance, had a matcap material applied to give it that quintessential reflective property of certain other high profile brands.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*lBjYQMEPWLfebM3I.jpg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*p-ZhmfdhwVcx0DXY.jpg" /></figure><p>To separate the ABOUT and SHARE sections, but still integrate them in the FAR experience we created a different world for each which could be entered by summoning a portal and entering therein…</p><p>Inside, users can browse a gallery simply by looking around with their video feed. To give the audience a helping hand we made the interactive elements snap to the camera’s rotation when a set distance threshold is passed. This gives a pleasing interaction and makes it easier to focus on a particular element; we coined this the ‘sloppy lock’.</p><h3>Conclusion</h3><p>We had a lot of fun making this little project and we learned a lot about AR and its current possibilities and restrictions on the web. We also learned how we can create accessible and creative FAR experiences in mobile browsers for the ‘right now’ educating and preparing and eager audience hungry for fun augmented experiences grounded in creative and story.</p><p>We are extremely excited about Augmented Reality and we can’t wait for full WebXR integration in all major browsers. In the meantime, Resn’s Little HelpAR is our Christmas cARd to the world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w_kBph20iB79LZzCXsDeYQ.png" /><figcaption>View on mobile — <a href="https://littlehelpar.resn.global/">littlehelpAR.resn.global</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48618976c336" width="1" height="1" alt=""><hr><p><a href="https://medium.com/resn/resns-little-helpar-48618976c336">Resn’s Little HelpAR</a> was originally published in <a href="https://medium.com/resn">Resn</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kekubian Assassin Technical Insight]]></title>
            <link>https://medium.com/resn/kekubian-assassin-technical-insight-a401717550d?source=rss----1e1081262853---4</link>
            <guid isPermaLink="false">https://medium.com/p/a401717550d</guid>
            <category><![CDATA[games]]></category>
            <category><![CDATA[mobile-games]]></category>
            <category><![CDATA[threejs]]></category>
            <category><![CDATA[mobile]]></category>
            <category><![CDATA[webgl]]></category>
            <dc:creator><![CDATA[Resn]]></dc:creator>
            <pubDate>Tue, 11 Dec 2018 10:19:11 GMT</pubDate>
            <atom:updated>2018-12-11T10:19:11.088Z</atom:updated>
            <content:encoded><![CDATA[<p><em>An article written by </em><a href="https://twitter.com/jesper_vos"><em>Jesper Vos</em></a><em> and </em><a href="https://twitter.com/KMkota0"><em>Jack KMkota0</em></a><em> for Resn.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z-8D3ko5iMuRikgw1izePQ.jpeg" /></figure><h3><a href="https://kekubianassassin.resn.co.nz/">Kekubian Assassin</a> Technical Insight</h3><p>On August 3rd, 2018, HBO aired the first episode of <em>Random Acts of Flyness</em> — a new late-night series from artist Terence Nance. HBO describes the show as, “a fluid, stream-of-consciousness examination of contemporary American life”.</p><p>Episode 2 of the series depicts the show’s main characters playing an arcade game called <em>Kekubian Assassin</em>. To promote the series, HBO commissioned Resn to create a version of <a href="https://kekubianassassin.resn.co.nz/"><em>Kekubian Assassin</em></a> that could be played by anyone on a smartphone.</p><p>In this article, we will take you through some of the challenges we have overcome while making this mini-game.</p><h3>Creating a World</h3><p>Each episode of the series features interconnected vignettes that make it a unique mix of vérité documentary, musical performances, surrealist melodrama, and humorous animation. We wanted our game to feature a mix of styles too, so we blended photographic textures on buildings and low-poly objects for benches, yellow cabs, and other props. We encapsulated this all in a psychedelic, afrofuturist theme.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*bgv-HqiqVFNo2hntgGG3bA.jpeg" /><figcaption>Street — Bar — Park — Shop</figcaption></figure><p>Our game is an infinite runner taking place in four environments in Brooklyn: Street, Bar, Park, and Store. All of the objects were sculpted in Cinema4D and put together in Blender. In order to create an infinite path with turns, we created different blocks for each environment — straight, left turn and right turn.</p><p>We opted for an infinite runner because this enabled us to make an endless world with a minimal amount of assets, reducing loading time and increasing render performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*CtPURkpWRrpRuQC8PqQALg.png" /><figcaption>Environment blocks in Blender</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*nvFMbROs815ah8Lwurj2ig.png" /><figcaption>Street Block — Straight</figcaption></figure><p>By adding metadata and using a specific scene hierarchy in Blender, we exported our environment blocks as GLTF files and imported them into Three.js. We then replaced materials and created instanced geometries of matching models to reduce draw calls. When the game is running, the blocks are positioned and rotated in order to be chained — creating an infinite path. Simultaneously a smoothed path, using <a href="https://en.wikipedia.org/wiki/Centripetal_Catmull%E2%80%93Rom_spline">Catmull-Rom spline</a>, is drawn for the camera to follow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*6oiIX0pV8yDkkvAE6izvZg.png" /></figure><p>The spherical ground, on top of contributing to the visual style, is a great way of giving the illusion of an endless world while reducing the amount of objects in the scene. Since objects that are far from the camera get covered by the ground, we never need to render more than two blocks at the same time. In order to create this type of setup, we need to project the blocks over the world sphere.</p><p>The ground — that includes turning roads, park paths, and indoor, tile floors — is rendered to an off canvas target using a top view orthographic camera. It is updated every frame to match the world blocks. The render target texture is used as the map of the sphere — which has its UVs strategically positioned on top.</p><p>The instanced world objects — buildings, trees, lamps, benches, etc — are projected over the sphere in real time. First, the position is calculated using a <a href="https://math.stackexchange.com/questions/1390198/what-amplification-can-i-apply-to-y-sin-x-for-it-to-be-a-perfect-oscillating">semicircle function</a>. Then, the rotation is adjusted to match the direction of a vector going from the center of the sphere to the repositioned object (multiplying that direction quaternion to the original rotation).</p><p>Finally, to magnify the visual perception of depth, we add a falloff effect to some object materials. It works as a color overlay blended on the object texture with more intensity the further the object is from the camera.</p><h3>Adding animated characters</h3><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*d8n7upc90GawvQSJndC5pg.jpeg" /></figure><p>In <a href="https://kekubianassassin.resn.co.nz/">Kekubian Assassin</a> you play as Najja, a young afro-american woman who has to make her way through Brooklyn. But it seems like she can’t do that without getting bothered by strangers passing by. The players&#39; goal is to stay strong as long as possible by throwing shade at anyone trying to steal her joy.</p><p>People of different cultures will appear along the way and might shout unfriendly phrases at Najja. In order to fill our world with animated characters, we first had to sketch and model them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*BHdOGzRDr0eLMvr0AM2p_w.jpeg" /><figcaption>tiny textures include pre-backed lighting</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*ewItNXb93QSpy6oiD9iUug.jpeg" /><figcaption>We added skeletons to the models with Mixamo</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*cBr0onF5PHZNYBDwR9XSng.jpeg" /><figcaption>Animations combined as NLA tracks in Blender</figcaption></figure><p>Our workflow starts with creating the low poly models in Cinema4D and exporting them with tiny textures that include pre-backed lighting. Thanks to Mixamo we add skeletons to the models and using Blender we export one GLTF per character. Using Mixamo we were also able to add as many as 30 animations to one of the characters. Unfortunately, Mixamo only exports one animation at a time, so they need to be combined as NLA tracks in Blender and exported as a single GLTF [<a href="https://www.donmccurdy.com/2017/11/06/creating-animated-gltf-characters-with-mixamo-and-blender/">this tutorial</a> was a life saver, and thanks to <a href="https://github.com/KhronosGroup/glTF-Blender-Exporter/pull/166">this fork</a> — now merged — we could export all the animations into the same file].</p><p>As a result, we have:</p><ul><li>9 characters exported as skinned meshes without animations [~350kb each].</li><li>1 character exported as skinned mesh and the 30 animations [8mb].</li></ul><p>In order to share the animations between characters, we need to rename all the imported bones to match the ones defined in AnimationTracks, otherwise THREE.AnimationMixer won’t find the right nodes to animate. We also remove all the bone position tracks, since we only need the bone rotations — otherwise, the kiddos stretch would be as long as the adults!</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*kGXzQsM3nzKhqZ72ckDomw.png" /><figcaption>Character animation data</figcaption></figure><p>At this point, we can trigger and mix animations in Three.js. In order to make the characters come to live, we need to trigger them at specific moments. Collaboration with game designers in this area was really important, so we created a simple declarative syntax to define what we called behaviours and patterns.</p><p>We define behaviour as sets of combinations of actions to be triggered a specific events — distance from camera or reaction to interaction. For example (see image for syntax):</p><ul><li>when the character is really far from camera → it is hidden</li><li>when the character is at distance 0.2 from the camera → it falls to the path, then loops through an idle animation</li><li>when the character is at distance 0.1 from the camera → it attacks with an insult animation</li><li>when the player taps on the character → it does a hit animation and then looks down</li></ul><p>In order to mix and match characters with behaviours and put them in the path, we defined patterns. A pattern defines the characters in a section of the path of the length of one block and is tagged with a difficulty (we have up to 5 difficulties). This is an example using the declarative syntax we specified for it:</p><h3>Camera movement</h3><p>The game is played in a first-person view. At first, we just had the camera move through the world in a straight forward fashion — following the smoothed path we generated. The position and perspective felt right, but it didn’t give a good impression that our hero was walking. We needed to add some subtle movement to the camera. Some x- and y-axis rotation mapped to a sinus, and an absolute sinus value moving the y position of the camera did the trick.</p><p>Because the game goes faster and faster over time we multiplied the step interval of the walk movement by the game speed. But we found that at a very high speed the interval became so small the camera didn’t actually move anymore, so we actually slowed down the step interval with an exponential curve. This means the walk movement will increase quickly at first and turn into a running movement later in the game.</p><h3>Making them speak</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*ECzBe5mI_U1l87OGg--EbA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*TFq3JtBw2o4OqNSHJlLNKw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*BhSQgzpqz_EXesoJezXDMw.jpeg" /></figure><p>Now that we could walk through an infinite world with all kinds of characters we needed to emphasize who was being annoying and who should be left alone. We visualised the concept of micro-aggression with speech bubbles. A simple JSON file stored 22 insults connected to specific character and animations. For example “Can you twerk?” will only be said by the white girl while she is twerking, but the same character might say “Come thru kween!” while she is performing another gesture.</p><p>The show and hide methods of the speech bubbles were then connected to the state of the characters. On friendly characters, the speech bubble would never show and on other characters it would only show once their ‘insult’ animation is triggered. Tapping a character without a speech bubble above its head, or not tapping an aggressive character will lose the player a life.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*YkxAD-jCw13eIWg44yVz5Q.png" /></figure><h3>Scowls</h3><p><a href="https://kekubianassassin.resn.co.nz/">Kekubian Assassin</a> is not about violence, it is merely about micro-aggression. In the series, we see Najja shooting scowl’s with a bright green toy gun. We wanted to stay true to this concept and made a sprite animation of this same visual. To show the scowls we positioned a quad in front of every character’s mesh and played the animation when the character is being tapped by the player. To emphasize the character being struck we also always trigger the ‘hurt’ and ‘sad’ animations at this moment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*7-fBlE26yylMI7YDVYJtYg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*OJQh4zt2PAoWHZtSQ9WgmQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*FqyHCNPRoHBc4_ng6kfoTg.jpeg" /></figure><h3>Boss level</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*7-fBlE26yylMI7YDVYJtYg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*OJQh4zt2PAoWHZtSQ9WgmQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*FqyHCNPRoHBc4_ng6kfoTg.jpeg" /></figure><p>The boss level in <a href="https://kekubianassassin.resn.co.nz/">Kekubian Assassin</a> features 3 different characters asking tricky questions that tie in with the series view on contemporary American life. The goal is to send the boss to the 4th dimension by answering 3 questions correctly. The first right answer opens a black hole behind the boss character that starts to pull on him/her and the following questions increase the power of it. Answering the third question correct gives the black hole enough power to completely devour the boss character and the whole galaxy around him.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*uY_K_GM930wRSFNtBqRXKQ.jpeg" /><figcaption>flowmap UV distortion</figcaption></figure><p>We started working on a circular displacement for the background first. After researching some different approaches we settled on using a flowmap. This is simply a texture containing distorted UV coordinates which we can use in a fragment shader. To animate between the different states of the black hole we just had to interpolate between the original UVs and the distorted UVs from the flowmap.</p><p>To create the illusion of an endless spinning universe we duplicated the background texture with a 180° offset and constantly crossfade between the two while interpolating between the original and flowmap UVs every sequence.</p><p>To make the boss level more dynamic we made the character an interactive model. To be able to rotate the head individually from the body we needed these parts to be separate meshes, but because the characters we made before all consist of a single mesh we had to make separate exports for the boss characters. We also optimised these models by cutting geometry from the lower legs since these would never be visible anyway… and we gave him a hat.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*ySGMAl5_YoYTf1Jw7NxVAQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*FIr1W6Hifnd8A1uqWZXqOg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*x1HapS8WqQTwOYIxScYm-g.jpeg" /></figure><p>Now that we have an interactive 3d model in our game we can’t just apply the same UV distortion as we did to the background — we need a single texture for that. Luckily we can convert any scene to a texture by using a framebuffer (RenderTarget in three.js). We applied the same UV distortion to this texture as we did to the background, only instead of endlessly looping a rotation we just added some simple sinus displacement.</p><h3>UI and transitions</h3><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*NpTITfoKW1MilOFBS9AtPw.jpeg" /><figcaption>UI layer system</figcaption></figure><p>On top of all this, an interface was designed in an afro-futuristic theme. We wanted to transition between the world and the UI by masking the views with a spinning pyramid. For this reason, we were forced to build all UI views in WebGL. Because of our highly ornamental design, we used textures for every UI element which we combined into an atlas of 2048x2048px so WebGL could process it.</p><p>For animated UI elements, we generated sprite sheets. To save data we chopped some symmetrical ornaments in half to duplicate and mirror them in GL on runtime, forming the full shape again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*5Ojbi4p_j9OiRaNGCz46SA.png" /></figure><p>Now that we had everything available in WebGL we rendered the UI view we want to transition to a render target. In a shader, we merged this texture output with the texture of a spinning pyramid — rendered in another render target. We could now mix the colors and alpha of the UI texture with the pyramids colors, creating a mask.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*o6dzb2b_1gVcfkQMecdTEQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*J4vkTR4aMRVMB7cCsYkUUg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*I-zTcEWfMN5K7CMafB5axg.jpeg" /><figcaption>using 3D geometry as a mask</figcaption></figure><p>Most views required the pyramid to translate vertically. Very quickly we noticed that the perspective camera didn’t allow us to move the geometry without losing the front-view on it. To get around this issue we animated the UVs of the render target texture instead.</p><h3>Audio</h3><p>The last thing to complete the experience was to add sound to the game. We layed out the audio in 3 layers: background music, environment audio, and character effects. For the latter, we used a similar system as we did for the character animations — one JSON file to connect audio files to characters and their gestures. To keep audio in sync with the game we also connected this to the global game speed.</p><p>To make it a little more dynamic we also connected the volume of the sound effects for every character to their distance from the camera. Characters close to the camera now make a louder noise than characters far away, just like in real life.</p><h3>Conclusion</h3><p>As our first 3D game with animated characters, making <a href="https://kekubianassassin.resn.co.nz/">Kekubian Assassin</a> came with a big learning curve for us. The team embraced these challenges and got motivated to get everything working in time. With everything we have learned from this project we are ready and excited to make more games and 3D experiences.</p><p><em>An article written by </em><a href="https://twitter.com/jesper_vos"><em>Jesper Vos</em></a><em> and </em><a href="https://twitter.com/KMkota0"><em>Jack KMkota0</em></a><em> for Resn.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a401717550d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/resn/kekubian-assassin-technical-insight-a401717550d">Kekubian Assassin Technical Insight</a> was originally published in <a href="https://medium.com/resn">Resn</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sound in Color]]></title>
            <link>https://medium.com/resn/sound-in-color-7d54aabec5f6?source=rss----1e1081262853---4</link>
            <guid isPermaLink="false">https://medium.com/p/7d54aabec5f6</guid>
            <category><![CDATA[design]]></category>
            <dc:creator><![CDATA[Resn]]></dc:creator>
            <pubDate>Mon, 19 Nov 2018 15:36:12 GMT</pubDate>
            <atom:updated>2018-11-19T15:36:00.582Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*V6oiAO8Mk2HQBIFLrxR32w.png" /></figure><p>The challenge for this HP Spectre campaign was to create a unique piece of artwork from each unique sound source. An artwork that did not look computer generated, something you would deem wall-worthy.</p><p>Breaking this challenge down on it’s highest level, we identified two very important steps to tackle:</p><ul><li>SOUND — It needed an easy and intuitive art creation ow, from which a user could input some sound and…</li><li>COLOUR — create an artwork that not only represented the message but looked like a beautiful unique painting.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*ChMdJ6_KykYpiTRZuE3nzQ.png" /></figure><p>We looked at art. Lots of it. From most of the masters we observed that human touch, the sense of organicness. In many you could trace the brushstrokes, see the ourishes and sense the rhythm in which it was carefully placed.</p><p>How could we combine these nuances with all the complexities observed in audio too? When recorded, a sound is represented by succession of frequencies, the peaks and troughs of a waveform. What if we mapped each frequency to a colour?</p><p>That method sounded ideal but it would had only have allowed us to create an artwork by looking at the sound sample piece by piece… each individual peak and trough represented. That wasn’t good enough for us…we wanted to literally picture the whole sound sample as an entre piece too.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*h88VPElTRV9Y3oRZycvAFQ.png" /></figure><p>Stuck on that problem we went back to square one.</p><p>Back to the painters. How does a painter make/manipulate their colours?</p><p>From the original tube of paint to the nal colour applied to the canvas a painter will use a palette of many to adjust the colour to the right tone.</p><p>We replicated that process. The colour tubes would be our input colour frequencies then we combine them in order to represent the sound as a whole.</p><p>We ended up with great diversity of colours not only representing the sound but the whole campaign.</p><p>The nal step was again created by observing the painting process. Using brushes, the painter will move the colours and mix them on the canvas.</p><p>In that manner, we wanted the sound to in uence the way the brush is going to be used. We quickly realised that the composition, direction and balance of the brush is key to get a good aesthetic. We tried a few options.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*DW-fIK2Z9k-GnHmlGZjYMg.png" /></figure><p>We came up with a two step process to creating the final artworks</p><p>S<strong>tep 1 (during recording) — Painting the Background</strong>: By analysing the recorded audio, colours and brushes were assigned to different frequency brackets. If a particular frequency was detected, those corresponding colours were applied to the background.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*d6afS_oALaK2scGYeUgjAw.png" /></figure><p>To get the colours right and harmonic we used the HSB colour space. HSB stands for hue, saturation and brightness and it is more intuitive than RGB. It helps us to get the right spectrum of colours based on the frequencies and makes it easier to keep the contrast and the luminosity consistent by only offsetting the hue. Tints that are close to blue represented low frequencies whilst tints that go towards yellow were for higher frequencies. Those colours are then used as a tint base for the brushes that are being displayed.</p><p>The size of the paintbrush was affected by the volume of the sound giving additional variation.</p><p>Each individual brush was used as a texture on a plane that gets the motion and the transformation in relation to the frequency of the sound. We ended up with a maximum of 20 of those planes at a time to not overkill the performance.</p><p>Show an example of a video in the article that we could use.</p><p>To get the artwork / canvas lled with a maximum 20 planes oating we needed to make sure anything that gets drawn throughout the process remained and dissolved on the canvas as if it were real paint. The trick to achieve this was by keeping the previous frame of what is being drawn and then print it again alongside the new frame, therefore we are always one frame ahead. This was achieved with</p><p>a FBO (Frame buffer object) system. At the end of the background creation, the canvas was blurred to create a consistent colour eld.</p><p><strong>Step 2 (during playback) applying brush strokes / colour mixing:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*H80brjGPzV0yA8ye5ZvesA.png" /></figure><p>Adding the human touch…now that the background was complete, we needed to add a sense of brush strokes and movement to the nal artwork. This was done by displacing the background, using a set of ten unique brushes.</p><p>To get this effect we used the output image / background made throughout the rst step and processed it with another image used for displacing pixels. This second texture is made out of another set of brushes. They are drawn, scaled and moved in relation to the audio frequencies. The result is used as a displacement map to move across the pixels of the rst output image.</p><p>By digitally pushing and pulling the colour of the background in a diagonal direction, we were able to mix colours and achieve the sense of brush strokes.</p><p><strong>Artwork</strong></p><p>It was highly important for all artworks to have a unique visual output where each differed from one another based on the sound recorded. Yet at the same time, having a sense of art direction and overarching style meant creating a set of colours which complemented each other and a designing a group of brushes to be used.</p><p>Each frequency bracket was assigned colour values that gave enough range of difference but that would also work well with each other. Colour selection required constant testing throughout the experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*KErr8gxUl82FHO9Sp9xooA.png" /></figure><p>The algorithm creating the artworks picks a series of brushes to use at the moment of applying the paint strokes. Rather than using a code based paint brush, each brush was designed to look appealing on the canvas. By having a limited set of brushes this gave a more human touch to the experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*pDIR00nZsbZMb2LD_s_nXA.png" /></figure><p>We wanted to create an audio waveform that served two purposes during different moments of the experience. The rst use was to show direct, one to one, feedback of you recording audio whereby the waveform would stretch and bend to the audio input it received. The second use was as a playback device where the user listened back to their audio as their artwork was being brushed.<br>To turn the input waveform into the playback version, we simply compressed its width which in turn created a waveform…</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*7pDwDK4AX_bh_w-XnchzhA.png" /></figure><blockquote>The Sound in Colour experience was created for the HP Reinvent Giving campaign, centered around sharing special moments with friends and family.</blockquote><p><strong>How did we create a print ready le from a website?</strong><br>The rendering of the artwork on the website is only created at a standard web resolution which would not be high enough for a user to print on paper.<br>To create the high resolution version we built a back end system on a server using a high end graphics card to replicate the front end visuals. When a user recorded audio and created an artwork, all of the data of the colours, brushes, positional information and brush strokes was stored in a JSON le. This le was then sent to the back end and recreated at a much high resolution.</p><p>This notion of sharing played an important part in the nal goal of the website</p><p>Sharing digitally and physically your nal art piece.<br>At the end of the website, the user was provided multiple options to engage in this notion of giving. Firstly the user could share their artwork on Twitter, Facebook or download and image to post on Instagram. Yet more importantly, the user could download a high resolution printable PDF version to print themselves as well as print their artwork on canvas.<br>By utilizing a third party printing provider, we were able to generate an artwork at a quality ready for canvas print</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*ChMdJ6_KykYpiTRZuE3nzQ.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d54aabec5f6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/resn/sound-in-color-7d54aabec5f6">Sound in Color</a> was originally published in <a href="https://medium.com/resn">Resn</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tap and Hold]]></title>
            <link>https://medium.com/resn/tap-and-hold-f23b8a741cc8?source=rss----1e1081262853---4</link>
            <guid isPermaLink="false">https://medium.com/p/f23b8a741cc8</guid>
            <category><![CDATA[tap-and-hold]]></category>
            <category><![CDATA[marketing]]></category>
            <category><![CDATA[resn]]></category>
            <dc:creator><![CDATA[Resn]]></dc:creator>
            <pubDate>Mon, 19 Nov 2018 15:35:30 GMT</pubDate>
            <atom:updated>2018-11-19T15:32:57.422Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*5aFMUL74A-Jv4nP_BXbNeQ.jpeg" /></figure><p>We love the internet, we love the creative playfield that is digital, we love what’s possible online, but we’re also keenly aware of the need to sell our clients products in order to make what we love.</p><p>Unbridled, balls-to-the-wall creativity and analytics driven e-commerce are unlikely bedfellows, they’re more like the premise to some 90’s cop buddy movie — It may be hilarious, but they just don’t get on with each other.</p><p>To put it simply: Some people wanna scroll and buy. Other people wanna to play and experience.</p><p>So then, how do we merge sales with marketing? What is the balance of story to make users aware and excited with the practicality of selling products efficiently? Practicality also in this mobile, handheld world we all now live in.</p><blockquote>The answer: We don’t merge, we layer.</blockquote><p>We eliminate any competition by hiding one, below the other — We then access this hidden content via a simple intuitive gestural mechanic.</p><blockquote>Tap &amp; Hold, or tap-to-view-more.</blockquote><p>We have successfully applied this approach to a number of experiences, from interactive music videos to straight up e-comm sites with great success.</p><p>Accessing a deeper layer of ‘culture’ content also means the smaller visual real estate on handhelds is dealt with in a novel way.</p><p>Erik Högfeldt Co-Founder of social and mobile centric creative agency Highfield has his own perspective:</p><blockquote>“Tap &amp; Hold is such a natural behavior now and I think we’re just scratching the surface of what it can become. It feels naturally rewarding, especially in an ecom environment.</blockquote><blockquote>We believe there’s a growing opportunity to increase conversion by closing the behavioral gap between the social platforms brands advertise on and the e-commerce platforms brands want consumers to check out on.</blockquote><blockquote>Now it’s like meeting someone really interesting but once you’ve moved past introductions and they invite you over to their place, they’re less fun than you first thought. Tap and hold to play is definitely one of the techniques that can help shape more fun and consistent consumer journeys on mobile. ”</blockquote><p>It is important to stress here, particularly when speaking of e-commerce, that the hidden culture or ‘story’ content does not have to be accessed. If a user does not engage they still have total access to point of sale and retail focus (the ultimate intention of any branded experience). Engaging with the hidden layer simply enriches the retail experience, it does not penalise those who don’t dive deeper in it rewards them.</p><p>Audio, or talking mobile-first, lack thereof is another challenge to be met with. It’s no mistake that Resn’s uses contextualised and positional visual feedback when users access the second layer — They need to feel responsible, cause and effect.</p><p>Resn is currently exploring iconography, one such way we are changing people’s online consuming habits whilst also creating a new visual dialogue between brands and their consumers.</p><p>The following mobile-centric examples show our evolution and expansion of the content layering concept with each experience exploring and building on the previous.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*wQ34dFeZGVj4BZMao24fYg.gif" /></figure><p>Our very own portfolio site was also the proving ground for this approach.</p><p>The challenge was simple: How do we show the world our personality, our humour, and our interactive chops while not scaring potential clients away?</p><p>Our solution was to ‘hide’ a number of playful, unpredictable, psychedelic and down right weird interactive toys behind a calm, ‘premium’ surface.</p><p>Access to the crazy world was via a simple gestural mechanic, you guessed it — Click &amp; Hold.</p><p>Removing your finger and your back on the surface, calm, premium aka ‘Resn Nights’.</p><p><a href="http://www.resn.co.nz/">view project</a> <strong>«</strong></p><p>A handheld interactive music video which fused the Korean Hip-Hop of Keith Ape to the sales mission of the new water resistant Chuck II’s.</p><p>Users passively watching the video could mix things up at any time by accessing 1 of 23 randomly contextualised animated gifs, creating their own unique experience every time.</p><p>The gifs were a combination of equal parts, product focussed content and neon soaked culture content.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*I8C4Gk0MxF3iiJsROna3fw.gif" /></figure><p><a href="http://counterclimate.converse.com/">view project</a> <strong>«</strong></p><p>Housed on adidas.com the key creative concept of the Originals ZX Flux 2016 campaign was to create a sense of ‘Limitless Possibilities’.</p><p>A CTA to Click &amp; Hold* via a button unleashed a dazzling display of graphic shapes, urban textures, colours and animations reflecting ‘Limitless Possibilities’, releasing at any time brings the user back to the product focused ecomm experience.</p><p>*Copy was intended for the desktop version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*UelDlLS49_5cfpvmdZFcig.gif" /></figure><p><a href="http://www.adidas.co.uk/zx_flux">view project</a> <strong>«</strong></p><p>Our second Tap &amp; Hold enabled interactive music video was created as part of the “Did you mean MailChimp” campaign.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*a4rD78jXHtfZtbaw7fhFJA.gif" /></figure><p>In what could be a world’s first interactive metaphor, users could “lift the veil” of the traditional music video format to reveal a behind-the-scenes perspective that runs in parallel to the main video.</p><p>The Tap and Hold transition utilizes a veil-like shader effect that manipulates the videos in real-time.</p><p><a href="http://www.veilhymn.com/">view project</a> <strong>«</strong></p><p>Our latest release and another collaboration with adidas is also our most exciting and engaging e-comm experience to date.</p><p>It is the culmination of all the above elements with the exciting addition of the contextualisation of the content being delivered to the user.</p><p>The Climazone features 3 video vignettes which depending on where a user is in the experience will play a specific video anchored in a specific part of the site.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*5Lag3CE0eFkgH4xMxUmdCA.gif" /></figure><p><a href="http://www.adidas.com/us/climazone">view project</a> <strong>«</strong></p><p>This is the next logical, and very exciting step for us enriching the experience based on a single customer’s wants, interests and needs based on machine learning… Wow!</p><p>We are entering the Renaissance of the dot com site, technology and the unprecedented flexibility of brand online platforms has empowered us to bring the ‘experience’ to said platforms.</p><p>No longer do we need to rely on microsites to scratch the creative itch and tell the story of the brand — we get to do it all in the same place.</p><p>Resn is a creative agency with a digital obsession. Our singular vision is to infect minds with gooey interactive experiences that amaze and stupefy.</p><p>Since 2004, Resn has worked at the frontier of interactive development and design. Our highly-skilled team has the expertise to take on complex technical challenges and fulfill creative dreams. From early insights and strategy to premium design and development, we collaborate with our clients at every stage, to deliver valuable engagement with their brand. We call this the Resn Effect.</p><p>Resn was recently awarded the prestigious title of Agency of the Year for 2016 by both Awwwards and CSS Design Awards.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f23b8a741cc8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/resn/tap-and-hold-f23b8a741cc8">Tap and Hold</a> was originally published in <a href="https://medium.com/resn">Resn</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>