<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alberto Taiuti on Medium]]></title>
        <description><![CDATA[Stories by Alberto Taiuti on Medium]]></description>
        <link>https://medium.com/@albtaiuti?source=rss-502684413830------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*IR-tY0NJJwCwmVVA1ZDOgQ.jpeg</url>
            <title>Stories by Alberto Taiuti on Medium</title>
            <link>https://medium.com/@albtaiuti?source=rss-502684413830------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 00:37:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@albtaiuti/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Designing Large-Scale AR Apps With ScapeKit]]></title>
            <link>https://medium.com/scape-technologies/designing-large-scale-ar-apps-with-scapekit-a3bce1be1fbc?source=rss-502684413830------2</link>
            <guid isPermaLink="false">https://medium.com/p/a3bce1be1fbc</guid>
            <category><![CDATA[augmented-reality]]></category>
            <category><![CDATA[scapekit]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ios]]></category>
            <category><![CDATA[vision]]></category>
            <dc:creator><![CDATA[Alberto Taiuti]]></dc:creator>
            <pubDate>Thu, 27 Jun 2019 14:01:01 GMT</pubDate>
            <atom:updated>2019-06-27T19:47:42.301Z</atom:updated>
            <content:encoded><![CDATA[<p>When you mention “Augmented Reality” to anybody outside the AR industry, often they reply with “ah, like Pokemon Go!”. Undoubtedly Pokemon Go is, so far, the product that has introduced the largest number of members of the general public to the concept of Augmented Reality, albeit in its first iteration it didn’t use advanced AR.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_mv6qSqqZhRucqvRNQjm4A.jpeg" /><figcaption>Pokemon Go gave us a first glimpse into how much AR can captivate audiences, even if it was using a more primitive form of AR.</figcaption></figure><p>One of the reasons the original iteration of Pokemon Go didn’t fully capitalise on the promise of large-scale AR was that it used the GPS to loosely position content, and the GPS signal has as a maximum precision radius of around 4 metres (when you’re lucky).</p><p>In practice, however, the accuracy of GPS is much worse, especially in city settings: buildings, trees, tunnels, clothing, and humans can prevent GPS signals from the satellites reaching the receiver making accuracy way lower!</p><p>For AR, such precision is not enough, so we need to look elsewhere for our precise positioning needs.</p><h4>Now it’s our turn 💪</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lvI5TRjI2HKtA3NHGi1xYA.jpeg" /></figure><p>Like me, many developers have since wanted to create something similar but which gets closer to delivering the full promise of large-scale AR: accurately positioned content which consistently appears anchored to the right spot in the real world. This can only be achieved if the positioning system used provides centimetre-accurate geolocation measurements.</p><p>One such positioning product is an SDK from the company Scape Technologies called ‘ScapeKit’, which provides reliable, &amp; accurate geolocation measurements using Computer Vision.</p><p>Having worked with Scape’s SDK in the past, I was very excited to hear they had released a new version of ScapeKit which abstracts away much of the geolocation to 3D space calculations I previously had to do manually and introduced many performance improvements to the localisation routines. As you can imagine, I immediately wanted to try it out in a real-world scenario!</p><h4>But what to build? 🧐</h4><p>In order to experiment with the new SDK &amp; large-scale AR more generally, I decided to build a simple but effective AR Gardening app. The application allows users to plant sunflowers anywhere in the world and, thanks to persistence, to find them again exactly where they had planted. I also added some playful mechanics to demonstrate real-time, multi-user interaction at scale.</p><p>For example, if a sunflower is not watered for 24h it will shrivel up until it is watered again. In addition, every 24h the sunflowers will grow by one step, up to the final, fully-blossomed stage. All users can see these changes and are able to interact with the sunflowers in real-time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*RU2WWknVAweYwOvjPV7vxQ.jpeg" /></figure><p>Of course, I have demo videos of the app in action, but I show them towards the end of the article: that way we will have built up knowledge around why we should make certain design decisions when building for AR at scale and by then the videos will make much more sense.</p><p>In this article, I will talk about how you can use ScapeKit to build large-scale, accurately-positioned AR apps as I did for my sample AR Gardening app.</p><p>The goal is to pass on as much knowledge and best practices as possible, which I accumulated empirically (no textbooks or how-tos here, until now!), so that you can get started more quickly and build on top of those findings.</p><p>Without further ado, let’s get going!</p><p><strong>Note: Throughout the whole article, “<em>AR</em>” stands for <em>Augmented Reality</em></strong></p><h3>Requirements for large-scale AR apps</h3><p>First, we must start our journey by planning ahead: what features do we want to include in our large-scale AR experience? To replicate the interactive, multiplayer and real-time AR Gardening app the list is:</p><ul><li>The content is displayed in AR</li><li>The content is persisted</li><li>Users of the app can place content in the real world themselves</li><li>Nearby users receive updates in real-time</li><li>Multiple users can interact in real-time</li></ul><p>Let’s look at these requirements in more detail.</p><h4>Display our content in AR</h4><p>Obviously, we want to display our 3D content in Augmented Reality. To achieve this I used<strong> SceneKit + ARKit</strong>, but you can go with ARCore + SceneForm if you’re on Android, or Unity and Unreal Engine for multiplatform support.</p><p>Using these frameworks makes it very easy to cover our requirement that users should be able to place the 3D content themselves since through these APIs it’s very easy to implement such functionality.</p><h4><strong>Content is persisted</strong></h4><p>“Persistent content” means that the same 3D AR content can be found exactly where it was placed, at two different time points. i.e. if we place an AR interactive Ice-Cream stand at a specific road corner, we want to find it on the exact same spot where we’ve left it when we come back to that area to launch our AR app again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/222/1*0_Ki5g9SiuV7XZEnHsgvOA.gif" /><figcaption>Scape Technologies’ Visual Positioning system in action. It even works at night! ©Scape Technologies Ltd</figcaption></figure><p>To achieve persistence we will use <a href="https://scape.io/">ScapeKit</a> for iOS, which gives us extremely accurate geopositioning measurements (i.e. accurate Latitude, Longitude and orientation) and handles conversion from geocoordinates to Euclidian space coordinates for us to display our content in the 3D scene.</p><h4>Real-time updates</h4><p>When users interact with AR content, any changes made should be reflected in real-time for anybody observing that scene.</p><p>For example, in the AR Gardening app I developed, it’s possible to add 1 artificial day of growth to sunflowers you planted. This has the effect of playing an animation of the flower growing up, and such change is reflected both in the shared data structures and the 3D content for all the other users, in real-time.</p><p>We will cover in a later section what to use to achieve data consistency, real-time updates and interactions.</p><h3>How do we get there? 🛣</h3><p>Now that we have a clear idea of what we want to achieve, we need to get there step by step so that we actually understand what’s going on under the hood.</p><h4>From Latitude and Longitude to Euclidean space</h4><p>All large-scale AR applications have to deal with geocoordinates (e.g. Latitude and Longitude), so this is a good topic to start from. If you’re serious about building large-scale AR apps it’s fundamental to have a grasp of what the positioning system of your choice does under the hood so that you can debug possible issues with confidence.</p><p>When we want to represent the position of something in the real world, outdoors, we assign it a Latitude and Longitude, e.g.: 51.524015, -0.082064.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/949/1*z9HasIO7YzUv7irjzy7zdg.png" /><figcaption>Map of London showing the location of the geocoordinates 51.524015, -0.082064</figcaption></figure><p>However, typically 3D scene graph systems (like SceneKit, SceneForm, Unity etc.) don’t natively work with longitude or latitude. Instead, they work in Euclidian space, so we need to convert from geocoordinates to Euclidian coordinates in order to show the content.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t1i7IOmwtBsMGguB8S7DKg.png" /><figcaption>We know where our object (in this case a plant) should be placed in the world, but what is its coordinate in Euclidian space?</figcaption></figure><p>One common way to do this is to use a point in Euclidian space as the “scene anchor” and to relate to it the geocoordinates measured via our geopositioning system, in this case, ScapeKit. This means that once we have started up the AR and scenegraph system of your choice, we take a geolocation measurement, save it and assign it to a node in the scene. We call this node the “anchor” since all the content will be attached to it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pCr7aYDRXBKjSElWQ4Qk_Q.png" /><figcaption>We assign the measured geocoordinate to a point in Euclidian space, in this case, the scene world-space origin.</figcaption></figure><p>This way we can now display a 3D model at a particular location by interpolating the Euclidian position of the anchor with respect to the displacement of the model to the anchor node in geocoordinates:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C617vwAmFecDmzhx-IYg_g.png" /><figcaption>The Euclidian space position of the plant will be some displacement from the anchor node, directly proportional to the displacement of the plant to the anchor in geocoordinates.</figcaption></figure><p>Easy, right? Not really…</p><h4>The North matters</h4><p>Obtaining an accurate heading is crucial. Because mobile device compasses are typically accurate to plus or minus 30 degrees, the whole 3D scene could be rotated incorrectly around the scene anchor, resulting in content being misplaced, even though our geocoordinates to Euclidian space calculations are correct. This is why it’s imperative that any positioning system provides a highly accurate reading of where ‘North’ really is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cTEwsr-B_U0euEY5Qffmdw.png" /><figcaption>An inaccurate North reading can result in the content being rotated around the origin of the scene, hence appearing in the wrong place. The dotted line represents where we expected the plant to appear.</figcaption></figure><h4>Accurate geolocation matters even more!</h4><p>In the same way that accurate heading plays a crucial role in the correct functioning of large-scale AR apps, another even more important factor is accurate geopositioning.</p><p>If your positioning system doesn’t give you accurate, consistent latitude and longitude readings, the whole of your 3D scene is displaced by some amount each time the user launches the app. Combine that with a maybe inaccurate heading reading, and suddenly your content is all over the place instead of where it should be!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j-QVQbBiSb-E2TPEajjIGg.png" /><figcaption>A combination of poor geolocation and heading readings can lead to the whole scene being badly misplaced. Your users won’t be happy…</figcaption></figure><p>The GPS can give us readings only with a confidence radius of a minimum of a couple of metres (when we’re lucky), which is way too imprecise for AR content persistence. Couple that with an inaccurate heading reading from the magnetometer (which goes crazy in an urban environment) and you can see why having an accurate positioning system is fundamental to develop good large-scale AR apps.</p><p>To drive the point home, imagine we’re using GPS + Magnetometer combo to get readings for our scene anchor, and that our conversion calculations are sound. Now say we want to display an advertisement for Starbucks in front of a specific one of its stores, so we set our 3D content to have a specific geolocation and all is well so far. The issues arise when our users use our app and try and see the content where it should be: however due to the imprecision of our positioning system the content will appear in different places every time a user uses our app! What if the content appeared in front of a competitor cafe nearby instead? 👀</p><p>But worry not, it’s our lucky day! 🎉</p><h4>Enter ScapeKit</h4><p>We now know all the positioning challenges that come from developing large-scale AR apps so you’re ready to learn that actually you don’t have to worry about them that much.</p><p>ScapeKit is an SDK for large-scale geo-based AR which handles all the challenges discussed above:</p><ul><li>It gives accurate, consistent geolocation and heading readings</li><li>It handles the conversions from latitude and longitude to Euclidian space</li><li>It integrates with your scenegraph system of choice, i.e. SceneKit</li></ul><p>That means that all we have to do is pass ScapeKit a geolocation and it will take care of giving back to you an object we can use to place content in your 3D scene at that location.</p><p>You can grab ScapeKit here:</p><p><a href="https://scape.io/sdk/">ScapeKit - The only SDK for Large-Scale AR - Scape Technologies</a></p><h3>Now for some fun</h3><p>Armed with the knowledge of what’s happening under the hood of ScapeKit, let’s move on to actually build the content for our AR app.</p><h4>Storing the data</h4><p>One of the first things we’ll want to do is show multiple objects in the 3D scene, and to make them persistent. This means we will have a bunch of geolocations as such:</p><pre>51.524, -0.0820<br>51.524, -0.0820<br>51.524, -0.0820<br>51.524, -0.0820<br>...</pre><p>To do so we’ll need a way to store their geolocation (and other information) and to make such data accessible by multiple users at the same time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EZa3S-KVp4eqBk0PcxxZzg.png" /><figcaption>We save the position of our objects to Firebase.</figcaption></figure><p>Remember that we also want changes made to the scene to be reflected in real-time to all nearby users!</p><p>Out of the various realtime database solutions out there, I find Firebase to be the most accessible and performant one, so Firebase is the database system I suggest you use and the one I used in the AR Gardening app. It’s well documented, it has SDKs for all the most popular programming platforms/languages and, most of all, it fits all of our requirements:</p><ul><li>It’s fast</li><li>It’s real-time</li><li>Changes made by one user are immediately reflected across all users</li></ul><p>When our users make changes to the app all you have to do is update the relative database document (to use Firebase’s terminology) and all users will receive the update in real-time provided they’re connected to the internet.</p><p>I’m not going to explain how to setup Firebase here since it’s very well documented on their website:</p><p><a href="https://firebase.google.com/docs">Firebase Documentation</a></p><p>Another important thing to keep in mind when using Firebase is to make it the source of truth for our data: we don’t want to have to worry about which of our users have the right data at any given time, so making Firebase the authority on what the data is like at any given time solves this problem.</p><h4>What should we store in the database?</h4><p>That’s very much up to us and what you want our application to do.</p><p>However, there are some guidelines that we should follow which apply well to all large-scale AR applications. At the very least we should store:</p><ul><li>The geolocation of the object (I.e. its latitude and longitude)</li><li>The creation timestamp</li><li>The timestamp for the when the object was last modified</li><li>The ID of the owner/creator</li></ul><p>Even if we won’t need all of these fields in the first version of our app, chances are we will in version 2 and higher so it’s a good idea to set up the database to handle them from the get-go. This way you won’t have to reset the database for our existing users when you update the application and want to support more / different fields.</p><p>As an example database structure, here’s a screenshot captured from the Firebase console of my AR Gardening project:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*Klb7vBP0lcUbN3Tq5cx7Hw.png" /><figcaption>A screenshot of the data in the database for my AR Gardening app.</figcaption></figure><p>It includes the fields I talked about above (although in my case the “last modified timestamp” is represented by the field “last_watered_date”) and additional ones I need to support the functionalities of my application.</p><h4>Important considerations when using Firebase</h4><p>There are two important considerations we should keep in mind when using Firebase in our projects: structuring our data and eliminating lag.</p><p><strong>Planning the structure of the data in advance</strong> allows us to avoid situations in which we have to reset the database for all of our users. Obviously, there will be times when we will need to bite the bullet and update/add fields of our database documents, so the second best thing you can do is make sure to structure the data in a way that allows for easy expansion and migration.</p><p><strong>Lag compensation</strong> means that when the user performs an interaction, the visual feedback for it should happen immediately instead of waiting for the data authority (in our case, Firebase) to reply with the new data before showing the visual feedback. Let’s explore an example to better understand this concept:</p><p>Say we allow our users to tap on the screen to place something. When that happens, we need to do two things: show the new object being placed for the user who performed the tap, and show it to everybody else who’s nearby. A quick solution would be to show a new object only when the data authority (i.e. Firebase) tells the app there’s new data to be displayed. This means that the logic to display a new object in the scene would be unified for both the user performing the interaction and the nearby users. However this represents an issue for the user performing the interaction: she expects the visual feedback to happen immediately but instead, she’ll have to wait for the app to first send the data to the server and for the server to reply “all good, here’s new data”.</p><p>The solution is lag compensation: we will perform the interaction immediately for the user performing it, as if there weren’t an authoritative server to communicate with. We then send the data to the server for storage and when the server replies with the new data, we check whether it’s different from what we expect. If not, we ignore it; if yes, we correct what we had already displayed on screen. For our simple app, most of the time the data authority will reply with the data we expect, hence allowing us to achieve lag compensation very nicely.</p><p>If you want to learn more about it, the link below is a good starting point. Google is also your friend for this.</p><p><a href="https://developer.valvesoftware.com/wiki/Lag_compensation">Lag compensation - Valve Developer Community</a></p><p>Bottom line: make sure you implement some form of lag compensation!</p><h4>Filtering objects</h4><p>At this point, we have a way to store the data for the objects in our scene, most importantly their geo-coordinates. Having combined that data with ScapeKit and your scene graph engine of choice we display them at their correct location in our AR scenes and all is well.</p><p>However what happens when someone places an object in, say, New York? Should our users in London be able to see such an object? Even better, should their app even be notified that such an object exists? Most of the time the answer is no.</p><p>What we want to do is hence to filter the objects visible to a given user based on her geocoordinates. This way we also avoid potentially having every instance of our app download the whole dataset, which could become very large quickly if our app goes viral!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CBdJZ7y91iRJ7Gelw0tKqQ.png" /><figcaption>This diagram shows a top-down view of the 3D scene. The camera symbol represents the position of the user, with the filtering area around it.</figcaption></figure><p>Another important reason to perform location-based filtering is that since we don’t (yet) have occlusion from every building, displaying objects which are too far from the user would break the illusion of AR since such objects would appear in front of buildings whereas they should be behind them, invisible.</p><p>If you’re on iOS, I recommend you use GeoFirestore-iOS to perform location-based filtering. It integrates well with Firebase’s Firestore database and it’s easy to use. You can find out more about it here:</p><p><a href="https://github.com/imperiumlabs/GeoFirestore-iOS/">imperiumlabs/GeoFirestore-iOS</a></p><h4>Setting the ground level</h4><p>Our scene content should be placed on the street level, i.e. on the ground. Therefore we need to use our AR system of choice’s plane estimation feature to “read” the ground level and use that measurement to place content on top of it correctly. This is because currently, ScapeKit doesn’t provide an altitude reading which is accurate enough to set the height of objects with respect to that measurement.</p><p>Failing to set the ground level would result in our content “floating” above the ground in an unnatural way.</p><p>An important lesson I learnt, which I cannot emphasize enough, is that you should make setting this up as easy and as intuitive as you can for the user. AR is, after all, still a relatively unknown technology to the general public and many people will not be familiar with the concept of “scanning the ground”, let alone AR itself. I initially had many instances of users reporting that the content was floating, because I hadn’t made this as easy as it should have been.</p><p>For example a good animation goes much further than a line of text in explaining what to do, but what I found most effective to lead the user to scan the ground correctly was to use the <a href="https://github.com/maxxfrazer/ARKit-FocusNode/">FocusNode repository</a> by <a href="https://mobile.twitter.com/MaxxFrazer">Max Fraser Cobb</a>. The focus node makes it very easy for the user to understand what’s happening and when the content has been scanned sufficiently. For reference, here’s a video I took of my AR Gardening app of such onboarding flow in action:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/1*SK9S9fQVoAauE3_xhe5SkA.gif" /><figcaption>Make sure you engage the user when letting them set the ground level.</figcaption></figure><p>If there’s one thing you should take away from this section is that <strong>you should spend a lot of time getting this part of the onboarding as smooth and as frictionless as possible. </strong>I can speak from experience when advising you to do so, so please do! If you don’t the user will find it boring or think that your application is broken and never open the app again.</p><h4>Give visual feedback</h4><p>This ties in with what we’ve just talked about in terms of making the onboarding as accessible as we can.</p><p>Due to the nature of any localisation system and of the way any AR system first starts up, there will be times when the user is left to wait for either the localisation to happen, for network data to be delivered, or for the AR system to start up (even all three of them at the same time if you’re unlucky).</p><p>Because of this, we must make sure that we display meaningful and well-thought animations to communicate to the user that work is being done in the background and that the app cannot proceed to the next stage unless the background task is complete. For example, most vision-based localization systems will have some delay between sending and receiving a measurement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/1*UhEOhl3KGCSH_yVVhm6qbA.gif" /><figcaption>Showing a loading animation can be useful to hide any network lag.</figcaption></figure><p>In the video above I am using a very simple animation, but it seems to be effective according to user feedback (you can find this and more 2D UI animations for iOS in the repository <a href="https://github.com/ninjaprox/NVActivityIndicatorView">NVActivityIndicatorView</a>).</p><p>It’s very important that we listen to user feedback when implementing these onboarding features: it’s easy to make assumptions about what works best but your users will definitely not like some of your solutions, so we should keep an open mind and follow their suggestions!</p><h4>Allow users to restart the experience in-app</h4><p>Sometimes the measurements are not as accurate as expected, and it can lead to the issues we discussed in the first part of the article. This can often be due to poor internet connection or other external factors.</p><p>Other times ARKit will not stabilise until late and will make your content “jump around” by moving the 3D scene world origin until it stabilises.</p><p>For both of these reasons, you should allow your users to restart the experience with ease. Two buttons at the top of the screen, one to restart the positioning system and the other to restart ARKit and take a new ground measurement, seemed to work fine for me. I experimented with menus and more advanced UI but users preferred the simplicity of two, always-visible buttons. This makes it also extremely easy for them to learn what to do in case the experience is not stable and needs to be restarted.</p><p>Here’s what the UI in my AR Gardening app looks like at the top of the screen:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*jS34sDXOMPHz-2XwJJxIGw.png" /><figcaption>Left button: restart ARKit and take a new ground measurement. Right: Restart the positioning system.</figcaption></figure><p>I recommend you have these buttons and that you plan how to teach your users about their functionality and importance. There will be times when they are needed, sometimes more than you think, due to ARKit not working as well as you’d expect when outdoors. So it’s important your users know how to use them, or they will just quit your app!</p><h3>📺 Finally the demos! 📺</h3><p>I’ve teased you long enough, so here are two videos which show the complete flow of the AR Gardening app with localisation, placement of content, and finally relocalisation and the ability to find the content in the same spot again:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/1*N6Xq9M_cKy_1lJHVxmHhuw.gif" /><figcaption>Localisation + Placement of content.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/1*wBSNIDKBJt_q0Nbv2d8kWw.gif" /><figcaption>Finding the content again after relaunching the app.</figcaption></figure><p>To prove that the app is interactive and multiuser, I have also filmed the same scene with two different phones; note that the phones are simply connected to the internet and not to each other directly. The same effect is replicated if they are not close to each other like in the following video:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/277/1*e0_sNYUNNRYw5Otw5qU4JA.gif" /><figcaption>Content updated on one device is immediately updated on all nearby devices.</figcaption></figure><p>Now that you know I’m not a charlatan and have actually built what I talked about at length, let’s wrap this up!</p><h3>The 1–2–3 of large-scale AR apps</h3><p>Before we sum this up, the steps I recommend you should take when developing a large-scale AR application are:</p><ol><li>Quickly integrate database + geolocation system (e.g. ScapeKit), use basic geometry to create the first implementation</li><li>Spend <strong>a lot of time</strong> getting the onboarding flow for your users right</li><li>Polish the visuals, the interactions, the UI, etc</li></ol><p>By far, step #2 is the most important one: if you don’t spend enough time on it, your users will not use your app a second time and will think that it’s broken.</p><p>I believe that at this stage one of the biggest barriers we have in the AR consumer market is educating users on how to use AR applications and making sure their experiences go as smoothly as possible.</p><p>In fact, Apple recently made available for developers a pre-built UI to onboard users in AR apps:</p><p><a href="https://developer.apple.com/documentation/arkit/arcoachingoverlayview">ARCoachingOverlayView - ARKit | Apple Developer Documentation</a></p><p>They must have recognised the widespread struggle amongst developers in creating consistent onboarding experiences with AR and thought to come to our rescue.</p><h3>Conclusions</h3><p>I hope that armed with this information you’ll be able to get started with large-scale AR apps quickly and that you can avoid going through some of the (sometimes very long) process of trial and error I went through.</p><p>In summary, you should:</p><ul><li>Account for hiccups in ARKit</li><li>Make it dead easy to restart the experience in-app</li><li>Store assets on the device, not the cloud</li><li>Store data in the cloud</li><li>Make the user onboarding your priority</li></ul><h3>Bonus 🎁</h3><p>If you’ve made this far through the article, congrats! Here’s a present for you.</p><p>When you think about it, our Firebase database doesn’t really care where the data we write in it comes from. Until now we’ve talked about integrating Firebase with a mobile app, but what if the data came from somewhere else?</p><p>Say, a web app with a map view of where we want to place content remotely?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rcFlLikvyAsd6yfEoi2EHg.png" /><figcaption>We could use a map view to place content remotely. Due to how we structured the mobile app, these would receive the updates in real-time just like if someone were placing the objects manually via the app.</figcaption></figure><p>I, together with <a href="http://twitter.com/theDart76">Danilo Pasquariello</a>, <a href="https://albertotaiuti.com/xrad-remote-positioning.html">experimented with this at the Scape Hackathon 2019,</a> where we went as far as using WebVR in combination with a 3D map of London for even more precise positioning.</p><p>However, to keep things simple one could use a 2D map provider such as Mapbox or Google Maps and embed a map view in an interactive web app. The user would then be able to tap anywhere on the map, and our web app would save the geo-coordinates of the tap to the Firebase database just like our AR app would. The users of our AR app near the location of the tap could even see the object appearing and being changed in real time! This could enable a first, very cool version of large-scale remote content placement: you could place objects in London from New York 🧙‍♂️.</p><p>I think this idea has lots of potential and that it should be explored further by more developers!</p><h4>Credits</h4><p>Many thanks to the amazing <a href="http://nataliasionek.uk/">Natalia Sionek</a> for having animated the sunflower 3D model in record time! Make sure to check her out.</p><p>The 3D models used in the demo app are:</p><ul><li><a href="https://poly.google.com/view/a9qW8wSfaaX">https://poly.google.com/view/a9qW8wSfaaX</a></li><li><a href="https://poly.google.com/view/6ktZgxSVVn1">https://poly.google.com/view/6ktZgxSVVn1</a></li><li><a href="https://poly.google.com/view/3yzoWzrCgoS">https://poly.google.com/view/3yzoWzrCgoS</a></li><li><a href="https://poly.google.com/view/87L4pG0oFMi">https://poly.google.com/view/87L4pG0oFMi</a></li><li><a href="https://poly.google.com/view/dNUS4xC0D6C">https://poly.google.com/view/dNUS4xC0D6C</a></li></ul><p>Many thanks to the creators of these assets!</p><p><em>If you enjoyed this post, make sure to leave some claps. 👏👏👏</em></p><p><strong><em>You can clap up to 50 times</em></strong><em>, so get clicking/tapping!</em></p><p><em>Please share the post with your iOS designer / iOS developer friends on your social media outlet of choice.</em></p><p><em>Follow me on Twitter: </em><a href="http://twitter.com/albtaiuti"><em>twitter.com/albtaiuti</em></a></p><p><em>Want to hire me for iOS and ARKit work? Click here: </em><a href="https://albertotaiuti.com/"><em>albertotaiuti.com</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a3bce1be1fbc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/scape-technologies/designing-large-scale-ar-apps-with-scapekit-a3bce1be1fbc">Designing Large-Scale AR Apps With ScapeKit</a> was originally published in <a href="https://medium.com/scape-technologies">Scape Technologies</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[WWDC19: Why Apple is still at the top of the AR food chain]]></title>
            <link>https://medium.com/inborn-experience/wwdc19-why-apple-is-still-at-the-top-of-the-ar-food-chain-a8dcc5421b72?source=rss-502684413830------2</link>
            <guid isPermaLink="false">https://medium.com/p/a8dcc5421b72</guid>
            <category><![CDATA[augmented-reality]]></category>
            <category><![CDATA[arkit]]></category>
            <category><![CDATA[apple]]></category>
            <category><![CDATA[ios]]></category>
            <category><![CDATA[wwdc-2019]]></category>
            <dc:creator><![CDATA[Alberto Taiuti]]></dc:creator>
            <pubDate>Fri, 07 Jun 2019 16:22:03 GMT</pubDate>
            <atom:updated>2019-06-26T22:46:32.493Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kXIJacrOHMv4CpWbEknNtQ.png" /><figcaption>I was definitely mind-blown about this year’s updates for Augmented Reality.</figcaption></figure><h3>WWDC19: Why Apple is still at the top of the AR foodchain</h3><p>WWDC19 just finished and Apple didn’t go easy on the competition in the Augmented Reality space.</p><p>Updates to SceneKit, ARKit, CoreML, iOS 13… And these are only the ones relative to Augmented Reality. So many announcements were made that it can be daunting to process them all, especially if your field of expertise is not software engineering.</p><p>In this post I aim to get you up to speed with all that was announced during the opening keynote and avoid any fillers. I’ll give you meaningful summaries of the announcements and most importantly, fill you in with what each means in the grand scheme of things, gathered from my experience in the space: how each fits in Apple’s vision for Augmented Reality and what each means for the industry as a whole.</p><p>Without further ado, let’s get going.</p><h3>Motion capture</h3><p>ARKit3 introduces the ability to track the pose of a human body in real-time with just the back-facing camera on iPhones XR/XS/XSMax.</p><h3>Alberto Taiuti on Twitter</h3><p>Testing out #ARKit3 body pose tracking https://t.co/wyWne19WSe</p><p>ARKit gives you what, in videogames and 3D programming, is called a “skeleton”, both in a 3D (what you see in the video) and in a screen-space (i.e. 2D) version.</p><p>The 3D version of the skeleton can be used with a 3D character model to animate it: this is very much the same technique used in videogames, with the (huge) difference that instead of it being controlled by pre-designed animations, it is driven by the movements of a person.</p><p>This enables a huge amount of new scenarios: from personalised full-body avatars/filters to games which use the different poses made by someone as input. How about jump rope?</p><h3>People occlusion</h3><p>Another huge addition is the ability for ARKit to now generate human body segmentation and estimate the depth of such people in screen-space. What this can be mainly used for is to correctly show AR 3D objects as being hidden by a person who is logically in front of the object in the 3D world.</p><p>This means that whereas in previous versions of ARKit we had this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*t7gOSmySsWFmJsyE-inSBw.png" /><figcaption>What a 3D object would look like if a person stood in front of it in previous ARKit versions</figcaption></figure><p>we now have:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*SkIfAGNmblwQ76CyC9hLBg.png" /><figcaption>Thanks to the new Machine Learning approach, content is rendered correctly as occluded</figcaption></figure><p>This addition works mostly towards improving the realism of the rendered AR scene, but the depth maps generated for this purpose can be used for other situations and applications.</p><p>For example one could use the segmentation map to apply screen-space effects only where people are not, effectively having a green screen effect ready to use anywhere at any time.</p><p><a href="https://twitter.com/nobbis">Tim Field</a> has posted a great video showcasing the depth map generation for people occlusion and segmentation in real time, which I think does a great job at showing what’s going on in the background to achieve the occlusion effect:</p><h3>Tim Field on Twitter</h3><p>People segmentation &amp;amp; depth using #ARKit3 https://t.co/xLzr7ty1bZ</p><h3>Better image &amp; object tracking</h3><p>Apple has made various performance improvements to the existing image and 3D object tracking systems in ARKit.</p><p>For 3D objects the system is now more robust and should be able to detect them faster and with more accuracy. In previous ARKit versions accuracy was indeed an issue, to the point that we didn’t see many apps using this feature at all. Hopefully this will prove to be a good enough improvement that apps using this really interesting feature will start emerging.</p><p>For image tracking, ARKit can now track up to 100 different images at the same time, compared to just a few from the previous version. This is such a large number that probably you will run out of space in your camera feed before you can fit 100 images in one frame.</p><p>At the moment Apple are the only ones offering object and image recognition with such performance and accuracy.</p><h3>Improved surface detection</h3><p>Moreover, ARKit also introduces better and more precise surface detection and classification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*k5FCDA09bzd3S9f5fr5gbQ.png" /><figcaption>Surface detection on featureless surface has been improved via Machine Learning-based techniques</figcaption></figure><p>ARKit can now detect walls and other surfaces with more accuracy thanks to the introduction of Machine Learning-based surface detection. This means that, for example, the system does a much better job at detecting walls and surfaces which have few features, such as white walls inside a house.</p><p>This is particularly relevant for all those ARKit apps out there using AR for interior design, house decoration, surveying, and more. For example, positioning of home furniture can now be made with more precision and purpose.</p><p>I remember seeing a couple of startup working on letting users see what their walls would look like with a new paint colour through AR and I’m sure they’re very happy about this update.</p><h3>Front &amp; back cameras can now run ARKit at the same time</h3><p>Whereas before you had to choose between running either a face tracking session or a world tracking session in ARKit, in ARKit3 you can now run them at the same time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*33cxz6uFMBdI2SYWXvdA2w.png" /><figcaption>Apple showcasing both cameras running ARKit at the same time</figcaption></figure><p>This means that you can, for example, control what content is displayed and how it is displayed in the 3D world view using your face expressions as input. It’s a feature that many people had been waiting for and it’s great to see it being finally made available to developers.</p><h3>RealityKit &amp; Reality Composer</h3><p>Apple also introduced a whole new set of APIs and Apps aimed at making Augmented Reality content creation faster and more accessible to non-technical users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UxAiAhDc8WmiWe0xBu02Mw.png" /></figure><h4>RealityKit</h4><p>The first part of this offering is comprised of a new API for developer called RealityKit. RealityKit is a completely new system and API which is in many ways similar to Apple’s other 3D scene management API, SceneKit, but which doesn’t have anything to do with it. This might seem odd to many developers like me who have been working with SceneKit and were expecting major overhauls to it, but the main takeaway here is that Apple is trying to make it dead easy for anyone to start building AR apps.</p><p>Another thing to note is that RealityKit seems to be missing more advanced features such as particle effects and the ability to generate a mesh from a set of vertices programatically which are instead available in SceneKit, let alone custom shaders.</p><p>Although very promising on the surface, RealityKit seems to be missing some features that many core users will be disappointed to not find, making it unlikely that it will be adopted in more advanced apps.</p><h4>Reality Composer</h4><p>A standalone app, Reality Composer is an editor built on top of RealityKit which lets anyone with an iOS 13 or MacOS 10.15 create AR experiences and scenes. Think of Spark Studio or Lens Studio but from Apple.</p><p>In general it offers the basic functionalities you would expect in any modern 3D scene editor with the addition of being able to anchor your content to different types of AR anchors: face, surfaces etc.</p><p>The main implication of Reality Composer, which might go unnoticed, is its new file format’s integration with QuickLook.</p><p><strong>QuickLook is now able to open RealityKit project files the same way it opens USDZ files and shows them to anyone with an iOS device without the need to install any apps!</strong></p><p>I cannot overstate how huge this is: this way anyone with an iOS 13 (or MacOS 10.15) device can create complex, interactive AR 3D scenes and share them with anybody else without them having to install any additional app. This includes animations and interactions!</p><p>As support and features for RealityKit will surely improve in the next couple of years, expect this paradigm to become more and more prominent.</p><h3>Better support for USD(Z) content creation</h3><p>Although not introduced as part of the ARKit3 offerings, Apple did improve the toolset available for people working with USD content.</p><p>One of the main additions is the new ability to convert from glTF to USDZ models via the pre-built USDPython tools. Now it’s possible to convert the many, already available glTF models found on platforms such as Sketchfab to USDZ locally.</p><p>Another very requested feature was for developers to be able to export SceneKit<em> .scn </em>scenes to USDZ programatically from inside their apps, and this has been addressed with the release of iOS 13. In previous releases, although possible to do so, the exported USDZ file would be incomplete, whereas now the scene is correctly and fully exported. This enables a variety of content creation applications to take advantage of the improved feature.</p><p>I’ve written extensively about USDZ here if you’re interested in finding out more about the nitty-gritty details of the file format: <a href="https://medium.com/@alberto.taiuti/usd-z-deep-dive-for-arkit-part-one-11bcf24a3deb">https://medium.com/@alberto.taiuti/usd-z-deep-dive-for-arkit-part-one-11bcf24a3deb</a></p><h3>Smaller apps &amp; Faster app install times</h3><p>At first glance you might think this section is unrelated to AR, but in my opinion it very much is.</p><p>AR experiences are ephemeral by nature, as they can be often tied to and triggered by a specific real-world location, and the friction of having to download a different app every time a new location is visited can create a lot of friction for users to actually buy and use your app.</p><p>For this reason, Apple having made apps smaller and faster to download will increase the chance users will want to download your location-based apps when they visit a new, specific location.</p><p>Although not the final solution we all want and need, which is a continuous-delivery of AR content platform, this will help with the adoption of AR amongst iOS users.</p><h3>Indoor positioning via WiFi fingerprinting</h3><p>Apple also announced that they are opening a new service for large-scale indoor mapping which makes use of WiFi fingerprinting to localise users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*goMGSwvXbXrPLPH43hOXYg.png" /><figcaption>The process to enable your business to use the Indoor Mapping system by Apple</figcaption></figure><p>WiFi fingerprinting navigation means that in order to find the location of your device in a room, the service triangulates the signals from various WiFi antennas to give you a location accurate up to 3/4 metres. If you’re interested in the usage details, you can find out more about the program in the relative WWDC19 video:</p><p><a href="https://developer.apple.com/videos/play/wwdc2019/245">Introducing the Indoor Maps Program - WWDC 2019 - Videos - Apple Developer</a></p><p>Needless to say, such accuracy is not enough to create persistent AR apps, but it can be used for coarse-grained filtering of some fine-grained relocalisation system, such as Image Anchors or ARWorldMaps: one could create pockets of ARWorldMaps (or Image Anchors) all over the place and then select which one to relocalise against using the WiFi positioning system (although I would argue that by this point you might as well just use Azure Spatial Anchors, which basically do all this for you and more).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o-PufbQWj7itXt4D_8RKlQ.jpeg" /><figcaption>The lighter blue radius of the blue dot represent the confidence of the location measurement</figcaption></figure><p>The main benefit of this localisation solution is that you don’t have to integrate 3rd party libraries, but not even Apple is marketing this as a solution for Indoor AR since they know that the precision is not (and will never be) enough for that with the bare WiFi positioning system.</p><h3>Conclusions</h3><p>These upgrades should convince you that Apple is currently at the forefront for public-facing AR offerings, and that it leaves the other tech giants in the dust (for now). Surely some of these techniques are not bleeding edge, but they are all relatively stable and a solid foundation upon which Apple will surely build in the upcoming years. We should expect to see object occlusion (either by screen-space segmentation or by meshing or by some other technique) at some point too in the future, and obviously enhancements to the other existing offerings, starting from RealityKit. WWDC20 couldn’t happen sooner!</p><p>References</p><ul><li><a href="https://medium.com/@nathangitter">Nathan Gitter</a> : I got inspiration for some of the structure of this post from his excellent posts on iOS and ARKit; make sure to check his Medium articles out.</li><li><a href="https://developer.apple.com/videos/all-videos/">https://developer.apple.com/videos/all-videos/</a></li></ul><p>If you enjoyed this post, make sure to leave some claps. 👏👏👏</p><p><strong>You can clap up to 50 times</strong>, so get clicking/tapping!</p><p>Please share the post with your iOS designer / iOS developer friends on your social media outlet of choice.</p><p>Follow me on Twitter: <a href="http://twitter.com/albtaiuti">twitter.com/albtaiuti</a></p><p>Want to hire me for iOS and ARKit work? Click here: <a href="https://albertotaiuti.com/">albertotaiuti.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8dcc5421b72" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inborn-experience/wwdc19-why-apple-is-still-at-the-top-of-the-ar-food-chain-a8dcc5421b72">WWDC19: Why Apple is still at the top of the AR food chain</a> was originally published in <a href="https://medium.com/inborn-experience">Inborn Experience (UX in AR/VR)</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pre-Signed AWS S3 URLs with Firebase Functions]]></title>
            <link>https://medium.com/@albtaiuti/pre-signed-aws-s3-urls-with-firebase-functions-2f54183e4107?source=rss-502684413830------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f54183e4107</guid>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[firebase]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Alberto Taiuti]]></dc:creator>
            <pubDate>Tue, 15 Jan 2019 10:17:01 GMT</pubDate>
            <atom:updated>2019-01-15T10:17:01.941Z</atom:updated>
            <content:encoded><![CDATA[<p>Say you want to let the users of your app upload or download from an S3 bucket without having to create complex IAM roles and permissions on AWS S3.</p><p>Normally the solution is to use an endpoint in your backend to obtain the pre-signed URL and then send it back to the frontend for the frontend app to use:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/371/1*8GnT_npchHLAgbxCY--IVQ.png" /><figcaption>Obtaining a pre-signed URL from S3 with a custom backend</figcaption></figure><p>We do this because the backend has permissions to contact the S3 services and obtain the URL when instead the frontend doesn’t. It marshals access to the resources for the frontend so that we avoid having to specify rules per-user ourselves.</p><p>But how do you achieve the same result if you don’t have your custom backend?</p><p>This can be the case if you’re only using Firebase to implement a data store for your frontend app.</p><h3>Google Cloud Functions to the rescue</h3><p>Google Cloud Functions (a.k.a. Firebase Functions) are an addition to the services provided by Firebase which allows you to write Javascript functions to be executed on the cloud by calling them from your frontend (or backend) app.</p><p>They can be thought of as something similar to AWS Lambda functions: small-ish functions which can be executed arbitrarily and which require almost no backend setup from your side. More about them here:</p><p><a href="https://firebase.google.com/docs/functions/">Cloud Functions for Firebase | Firebase</a></p><p>What we want to achieve is this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/371/1*-4UVgyxlbjkatM4cRaL1lA.png" /><figcaption>Obtaining a pre-signed URL from S3 with Google Cloud Functions</figcaption></figure><h4>Example setup</h4><p>In this article I assume that:</p><ul><li>You have already setup a frontend application</li><li>You have created a Firebase project</li><li>You have an AWS account and have created an S3 bucket with default permissions (you are the only owner and the only one with access)</li></ul><p>If you haven’t there’s a myriad of articles on the internet explaining how to do so. Here are some:</p><ul><li><a href="https://www.codementor.io/yurio/all-you-need-is-react-firebase-4v7g9p4kf">https://www.codementor.io/yurio/all-you-need-is-react-firebase-4v7g9p4kf</a></li><li><a href="https://medium.com/@julienrioux/deploy-react-js-app-on-firebase-c7fa6ddc153f">https://medium.com/@julienrioux/deploy-react-js-app-on-firebase-c7fa6ddc153f</a></li><li><a href="https://firebase.google.com/docs/web/setup">https://firebase.google.com/docs/web/setup</a></li><li><a href="https://docs.aws.amazon.com/AmazonS3/latest/gsg/SigningUpforS3.html">https://docs.aws.amazon.com/AmazonS3/latest/gsg/SigningUpforS3.html</a></li></ul><p>Next we need to integrate Google Cloud Functions into our project. This article explains well how to do it, so I’m not going to repeat it here:</p><p><a href="https://firebase.google.com/docs/functions/get-started">Get started: write, test, and deploy your first functions | Cloud Functions for Firebase</a></p><h4>CORS setup for S3</h4><p>In order to send an HTTP request to the pre-signed URL from the frontend, we need to setup our S3 bucket with the correct CORS policy. If we fail to do so, our browser or app will fail to send the request.</p><p>This article explains well how to do so:</p><p><a href="https://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html">Cross-origin resource sharing (CORS)</a></p><p>Here is an example CORS policy which you can use:</p><pre>&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?&gt;<br>&lt;CORSConfiguration xmlns=&quot;<a href="http://s3.amazonaws.com/doc/2006-03-01/">http://s3.amazonaws.com/doc/2006-03-01/</a>&quot;&gt;<br>&lt;CORSRule&gt;<br>    &lt;AllowedOrigin&gt;*&lt;/AllowedOrigin&gt;<br>    &lt;AllowedMethod&gt;GET&lt;/AllowedMethod&gt;<br>    &lt;AllowedMethod&gt;POST&lt;/AllowedMethod&gt;<br>    &lt;AllowedMethod&gt;PUT&lt;/AllowedMethod&gt;<br>    &lt;AllowedMethod&gt;HEAD&lt;/AllowedMethod&gt;<br>    &lt;AllowedMethod&gt;DELETE&lt;/AllowedMethod&gt;<br>    &lt;MaxAgeSeconds&gt;3000&lt;/MaxAgeSeconds&gt;<br>    &lt;AllowedHeader&gt;*&lt;/AllowedHeader&gt;<br>&lt;/CORSRule&gt;<br>&lt;/CORSConfiguration&gt;</pre><p>These allow all the possible HTTP actions on our bucket from any origin. You might want to create stricter rules, but that’s up to you.</p><h4>Writing our Cloud Functions</h4><p>If you have setup the above, you should now have a folder named <strong>functions</strong> in your project, with this structure:</p><pre>functions/<br>  index.js<br>  node-modules/<br>  package-lock.json<br>  package.json</pre><p>index.js is where we’re going to write our function, but you could just as easily write it in a separate Javascript file.</p><p>In order to allow the backend to successfully make calls to the AWS services endpoints, in our case S3, we need to create an IAM for it and then obtain the relative API keys.</p><p>Again I’m referring you to an external article which explains how to do so:</p><p><a href="https://realpython.com/python-boto3-aws-s3/#installation">Python, Boto3, and AWS S3: Demystified - Real Python</a></p><p>Essentially, <strong>make sure that your IAM account uses a policy which grants it full access over the S3 resource</strong>. You can use the <em>AmazonS3FullAccess</em> policy for ease.</p><p>If you already have an IAM account, simply make sure that it has access to S3 resources.</p><p>Obtain access keys for your IAM account, so that you can use them to login from the Cloud Functions: these will be given to you as a file either when you first create an IAM account or if you manually obtain them from the IAM management panel (in case you already have an account).</p><p>These look like this:</p><pre>aws_access_key_id = YOUR_ACCESS_KEY_ID<br>aws_secret_access_key = YOUR_SECRET_ACCESS_KEY</pre><p>We can now install the AWS S3 SDK to our cloud functions installation, so that we can make requests to S3 from the cloud functions. Navigate inside of the functions folder generated by the firebase cli tools and run:</p><pre>npm add aws-sdk</pre><h4>Example upload function</h4><p>Let’s write an example Cloud Function which allows the frontend app to upload a file to our S3 bucket.</p><p>Open index.js and delete the existing code. Then write our function:</p><pre>const functions = require(&#39;firebase-functions&#39;);<br>const admin = require(&#39;firebase-admin&#39;);<br>admin.initializeApp();<br>const AWS = require(&#39;aws-sdk&#39;);</pre><pre>exports.getS3SignedUrlUpload = functions.https.onCall((data, context) =&gt; {<br>  AWS.config.update({<br>    accessKeyId: &quot;YOUR_ACCESS_KEY_ID&quot;,<br>    secretAccessKey: &quot;YOUR_SECRET_ACCESS_KEY&quot;,<br>    region: &quot;YOUR_AWS_REGION&quot; // (e.g. :)&quot;eu-west-1&quot;<br>  });<br>  var s3 = new AWS.S3();</pre><pre>const s3Params = {<br>    Bucket: data.S3BucketName,<br>    Key: data.key,<br>    Expires: 600, // Expires in 10 minutes<br>    ContentType: data.contentType,<br>    ACL: &#39;public-read&#39;, // Could be something else<br>    ServerSideEncryption: &#39;AES256&#39;<br>  };</pre><pre>return s3.getSignedUrl(&#39;putObject&#39;, s3Params);<br>});</pre><p>Let’s dissect what’s happening here.</p><p>First we’re importing the required Node.js modules: we need both the Firebase Admin modules and the AWS SDK module so that we can communicate to the S3 endpoint.</p><pre>exports.getS3SignedUrlUpload = functions.https.onCall((data, context) =&gt;</pre><p>specifies the name of the function and what type it is. You can define both HTTP Cloud Functions or “App Functions”, which require less setup and are easier to use. In this case we’re using App Functions. You can read more about them here:</p><p><a href="https://firebase.google.com/docs/functions/callable">Call functions from your app | Firebase</a></p><p>data is a JSON structure containing the following fields:</p><pre>data {<br>  S3BucketName: &quot;my-bucket-name&quot;,<br>  key: &quot;file-to-upload-name&quot;,<br>  contentType: &quot;mime-content-type&quot;, // E.g.: &quot;application/zip&quot;,<br>}</pre><p>which are used to configure the pre-signed URL. It’s constructed in the frontend app and sent along with the Google Cloud Function request. This way we can have a form in the frontend specify name of the file to upload, which file to upload, etc.</p><pre>  AWS.config.update({<br>    accessKeyId: &quot;YOUR_ACCESS_KEY_ID&quot;,<br>    secretAccessKey: &quot;YOUR_SECRET_ACCESS_KEY&quot;,<br>    region: &quot;YOUR_AWS_REGION&quot; // (e.g. :)&quot;eu-west-1&quot;<br>  });<br>  var s3 = new AWS.S3();</pre><p>Here we configure the AWS SDK with the credentials we obtained from the IAM role for the backend and then create an object which allows us to interact with the S3 APIs.</p><pre>  const s3Params = {<br>    Bucket: data.S3BucketName,<br>    Key: data.key,<br>    Expires: 600, // Expires in 10 minutes<br>    ContentType: data.contentType,<br>    ACL: &#39;public-read&#39;, // Could be something else<br>    ServerSideEncryption: &#39;AES256&#39;<br>  };</pre><p>Then we create a JSON object with the parameters necessary to request a pre-signed URL from S3. Some of these are optional, but help to be more precise about what type of file we’re uploading and what encryption level to use. This page explains the various available parameters:</p><p><a href="https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#getSignedUrl-property">Class: AWS.S3</a></p><p>Feel free to specify more parameters and to change the contents of the JSON data structure.</p><pre>return s3.getSignedUrl(&#39;putObject&#39;, s3Params);</pre><p>Finally we send a request to the S3 API to get a pre-signed URL, and return the URL to the client.</p><h4>Example Frontend usage</h4><p>Let’s now use our pre-signed URL in the frontend to upload a file to S3:</p><pre>...</pre><pre>// Retrieve the Cloud Function<br>var getPost = functions.httpsCallable(&#39;getS3SignedUrlUpload&#39;);</pre><pre>// Call the cloud function<br>getPost(s3GetUrlParams).then(result =&gt; {</pre><pre>  // S3 pre-signed URL<br>  const options = {<br>          headers: {<br>            &#39;Content-Type&#39;: &#39;application/zip&#39;,<br>            &#39;x-amz-acl&#39;: &#39;public-read&#39;,<br>            &#39;x-amz-server-side-encryption&#39;: &#39;AES256&#39;<br>          }<br>   };</pre><pre>   // Run a HTTP PUT request to the pre-signed URL<br>   return axios.put(result.data, this.state.zipFile, options)<br>          .then(result =&gt; {...});<br>}</pre><p>In this example I am using ReactJS and Axios, but this flow could be used in any other setting:</p><ol><li>Call the Cloud Function</li><li>Use the returned pre-signed URL</li></ol><h4>What if I want to download a file from an S3 bucket?</h4><p>Most of the code would remain the same.</p><p>In the Cloud Function, instead of asking for a ‘putObject’ URL, you’d ask for a ‘getObject’ one:</p><pre>return s3.getSignedUrl(&#39;getObject&#39;, s3Params);</pre><p>and on the frontend side, you’d send an HTTP GET request rather than a PUT one.</p><h3>Conclusion</h3><p>We’ve seen how it’s possible to perform complex backend-related tasks without having to roll our own backend by just using Google Cloud Functions. In many cases Cloud Functions suffice to run backend-related tasks and they give you the flexibility of not having to spend time writing boilerplate backend code, while at the same time probably being cheaper than having your own backend always up and running.</p><p>Follow me on Twitter: <a href="https://twitter.com/nathangitter">twitter.com/albtaiuti</a></p><h4>If you enjoyed this article, please click the 👏 button and share to help others find it! Feel free to leave a comment below.</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f54183e4107" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How USDZ Enables New Experiences for ARKit: Part One]]></title>
            <link>https://medium.com/@albtaiuti/usd-z-deep-dive-for-arkit-part-one-11bcf24a3deb?source=rss-502684413830------2</link>
            <guid isPermaLink="false">https://medium.com/p/11bcf24a3deb</guid>
            <category><![CDATA[ar]]></category>
            <category><![CDATA[ios-app-development]]></category>
            <category><![CDATA[swift-programming]]></category>
            <category><![CDATA[augmented-reality]]></category>
            <category><![CDATA[ios]]></category>
            <dc:creator><![CDATA[Alberto Taiuti]]></dc:creator>
            <pubDate>Thu, 10 Jan 2019 11:49:39 GMT</pubDate>
            <atom:updated>2019-01-22T03:43:46.236Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ud7JwDc-grF4baKsrEHb0g.jpeg" /></figure><p>At WWDC 2018, Apple announced various improvements to its support for Augmented Reality experiences and one of those was the support of a new file format, USDZ. USDZ enables new experiences and adds new ways in which users can interact with Augmented Reality content.</p><p>I’ve written more about WWDC 2018 here:</p><p><a href="https://medium.com/inborn-experience/apple-steals-the-ar-scene-at-wwdc18-8069e13be267">Suspended - Medium</a></p><p>In this article, I will dispel some myths about the USD(Z) format, talk about how it works with respect to ARKit, and point out some differences which I found while working with it and writing conversion tools for it. We will slowly dissect one of the USDZ examples provided by Apple to understand the format when applied to ARKit and QuickLook.</p><p>There’s a lot to cover, so in the next articles of this series I will explain the nuts and bolts of the USD specification and file format from a more generic point of view. Thus, you can work with it and create USD scenes programmatically via either Python or C++ from any platform, not necessarily a Mac.</p><p><em>Want me to implement custom USDZ projects for you? Click </em><a href="/service/mailto:alberto@albertotaiuti.com"><em>here</em></a><em> to drop me an email.</em></p><h3>What is USD(Z)?</h3><p><strong>U</strong>niversal <strong>S</strong>cene<strong> D</strong>escriptor (a.k.a. <strong><em>USD</em></strong>) is:</p><blockquote>an efficient, scalable system for authoring, reading, and streaming time-sampled scene description for interchange between graphics applications.</blockquote><p>Let’s translate that to English.</p><p>Very simply put, USD is what you would call a 3D asset file format with some special additions. One of these is the ability to package a USDC file with its textures into a USDZ file for consumption by ARKit and QuickLook.</p><h3>What does all this mean for the AR industry and ARKit?</h3><p>USDZ files can, for example, be shared via iMessage or you can open them directly from a webpage with Safari:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*zRZz1kut72EaDrRPs-r6yw.gif" /><figcaption>Visit <a href="https://developer.apple.com/arkit/gallery/">https://developer.apple.com/arkit/gallery/</a> from Safari on iOS to try this out yourself</figcaption></figure><p>The coolest thing about these two examples is that <strong>they show how QuickLook allows users to experience AR content without having to install any new app</strong>.</p><p>This is huge: imagine being able to see any product in 3D, in the real world, before you buy it from your favourite online shop, without having to install any new app! This would not be possible without USDZ’s support from Apple.</p><p>Another use I am experimenting with is to showcase/share medical 3D models easily: send them to your friends with an iOS device and they can immediately check out your work, no apps needed as well. There are a lot more use cases for USDZ and QuickLook.</p><p>USD is developed by Pixar and is used in their software, from Hydra to Presto. Pixar made its implementation open source some years ago on Github, so now anyone can use it to implement conversion tools or to integrate it in their engines and software:</p><p><a href="https://github.com/PixarAnimationStudios/USD">GitHub - PixarAnimationStudios/USD: Universal Scene Description</a></p><p>A single USD file can be thought of as representing a <em>scene</em>, and a USD scene is a hierarchy of what Pixar calls <em>Prims </em>(which intuitively can be thought of as <em>nodes).</em></p><p>USD performs almost the same function that, say, the gLTF or COLLADA file formats do, but compared to these two other example formats, it has both more and fewer features.</p><p>For example, one of the main differences is that gLTF is starting to become widely supported, whereas USDZ is currently only supported officially on iOS’ QuickLook.</p><h3>Getting the sample USDZ asset</h3><p>For the purpose of this article, I will assume that you are on MacOS and you have setup your USD Python tools to be used from the command line in your terminal as explained <a href="https://graphics.pixar.com/usd/docs/USD-Tutorials.html">here</a> after downloading them from <a href="https://s3-eu-west-1.amazonaws.com/convertitore-di-cose-public/USDPython.zip">here</a>.</p><p>Visit <a href="https://developer.apple.com/arkit/gallery/">this website</a> from your browser, click on the guitar 3D model and save it somewhere sensible.</p><h4>Extracting a USDZ asset without renaming it</h4><p>I’ve read online that most people when they want to access the files contained in a .usdz file, they first rename it to have a .zip extension and then extract it from Finder.</p><p>Instead, we can use the command line tool unzip to extract an USDZ file directly as such:</p><pre>unzip stratocaster.usdz -d stratocaster</pre><pre>Archive:  stratocaster.usdz<br> extracting: stratocaster/Stratocaster.usdc<br> extracting: stratocaster/StratocasterGuitar_NeckWood_Albedo.jpg<br> extracting: stratocaster/StratocasterGuitar_ButtonVolume_Albedo.png<br> extracting: stratocaster/StratocasterGuitar_ButtonTone_Albedo.png<br> extracting: stratocaster/StratocasterGuitar_BodyWood_Albedo.jpg<br> extracting: stratocaster/StratocasterStand_Albedo.jpg<br> extracting: stratocaster/StratocasterStand_Metallic.jpg<br> extracting: stratocaster/StratocasterStand_Normal.jpg<br> extracting: stratocaster/StratocasterStand_Roughness.jpg</pre><p>You can now navigate to your extracted folder and directly open the .usdc file:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*Br0YEQV9oIEVUNCANUYvGQ.png" /><figcaption>Opening the extracted <strong>.usdc</strong> file</figcaption></figure><p>Before we continue, let’s understand what all these different file formats mean.</p><h3>What’s the difference between USD, USDC, USDA and USDZ?</h3><p>Files with .usd, .usdc and .usda extensions all represent a USD scene file, but the way they store this data is different.</p><p>Recall that a USD file can be thought of as representing a <em>scene</em>, and a USD scene is a hierarchy of nodes. Most of the time you will have a 1-to-1 correspondence between a USD file and a 3D model, instead of a scene; i.e. you will have one single mesh in the file.</p><h4>.usda</h4><p>.usda files are USD scenes stored on file in ASCII format. Hence, these files are text files containing the scene description in a manner that is human-readable. For example, the following is the .usda file for the representation of a USD scene containing a simple cube:</p><pre>#usda 1.0</pre><pre>def Mesh &quot;cube&quot;<br>{<br>    float3[] extent = [(-0.5, -0.5, -0.5), (0.5, 0.5, 0.5)]<br>    int[] faceVertexCounts = [4, 4, 4, 4, 4, 4]<br>    int[] faceVertexIndices = [0, 1, 3, 2, 2, 3, 5, 4, 4, 5, 7, 6, 6, 7, 1, 0, 1, 7, 5, 3, 6, 0, 2, 4]<br>    normal3f[] normals = [(-0.15186, 0.988312, 0.013374), (-0.206543, 0.977774, -0.036018), (0.026661, 0.999633, 0.004783), (0, -1, 0), (1, 0, 0), (-1, 0, 0)] (<br>        interpolation = &quot;uniform&quot;<br>    )<br>    point3f[] points = [(-0.5, -0.5, 0.5), (0.5, -0.5, 0.5), (-0.5, 0.5, 0.5), (0.5, 0.5, 0.5), (-0.5, 0.5, -0.5), (0.5, 0.5, -0.5), (-0.5, -0.5, -0.5), (0.5, -0.5, -0.5)]<br>    uniform token subdivisionScheme = &quot;none&quot;<br>}</pre><p>As you can see the format is quite readable and makes it easy to understand how the scene is organised.</p><p>However, this format is not efficient as it takes a long time for large files to be parsed into a text file. For this reason, Pixar also specifies the .usdc file format, which allows software to read USD scenes much more quickly.</p><h4>.usdc</h4><p>.usdc files are the binary representation of USD scenes. Hence, they are stored in binary form, and opening them with a text editor is not of much help. Here is the cube scene example again, this time after opening its .usdc equivalent with a text editor:</p><pre>PXR-USDC�������ú��������������������������������������������������������������������������������������������� ���<br>��� ��� �����������������¿���¿���¿���?���?���?����������������������������������������Ô���EEEUU�ÿ�úÿþþú����������������������?������?��������������¿������¿������?����������¿������������������¿���¿���?���?���¿���?���¿���?���?���?���?���?���¿���?���¿���?���?���¿���¿���¿���¿���?���¿���¿�������Ú�������Ö��������ó�primChildren�cube�specifier�typeName�Mesh�properties�subdivisionScheme�points�extent�normals�faceVertexCounts�faceVertexIndices�float3[]�default�int[]B�ò 3f[]�interpolation�uniformq� �ðtoken�variability�none������������������������ð<br>���UEQþö�öóöùk�������� X��)�a���*@�1 <a href="http://twitter.com/d">@d</a>�1)�<br>�1 @�1�1 @¨�2È�! �1 @ø�1�3 @� H �0�1� @�°�,@����� @���������������ð<br>���EUP�ÿüúø÷<br>ô<br>òï�������������� ��������p�����<br>��������°���QA�õþÿ ������������@ÿþ������� ��������p����� �������� ���� �������������ÿû�������TOKENS����������°������î�������STRINGS����������������������FIELDS����������¦�������������FIELDSETS�������B������/�������PATHS�����������q������I�������SPECS�����������º������@�������</pre><p>However, these files can be read by softwares much more quickly than their .usda equivalents. In fact, USDZ packages for QuickLook must use .usdc for the scenes to be loaded quickly on mobile devices.</p><h4>.usd</h4><p>.usd files can be both ASCII-based or binary files. Their extension doesn’t give away what format they are from the outside, but systems reading them can determine their format from the first bytes of the file. Hence, .usd files can be both text-based or binary based, i.e. you can rename either a .usdc or .usda file to .usd and they will work like how they were working before you renamed them.</p><h4>.usdz</h4><p>This format is used by QuickLook to render scenes and can be imported and used in iOS projects developed in Xcode.</p><p>In its current form, it’s more or less a zipped folder containing <strong>at most (and at least) one </strong><strong>.usdc file </strong>and any textures which that scene uses.</p><p>You can learn how to create USDZ files from OBJs here:</p><p><a href="https://developer.apple.com/videos/play/wwdc2018/603/">Integrating Apps and Content with AR Quick Look - WWDC 2018 - Videos - Apple Developer</a></p><h3>Exploring the Stratocaster USDZ asset</h3><p>Now that you have extracted the Strato, you can convert its .usdc file to .usda as such:</p><pre>cd stratocaster/<br>usdcat Stratocaster.usdc -o Stratocaster.usda</pre><p>Open the Stratocaster.usda file with your favourite text editor, and boom:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qhRJyHkHiI4Usei-AKxZnw.png" /><figcaption>The beginning of the <strong>Stratocaster.usda</strong> file</figcaption></figure><p>We can now read the contents of the file to understand how USDZ assets are organised for consumption by QuickLook.</p><p>Open the newly created Stratocaster.usda by double clicking on it so that the changes we made to it are shown in real-time.</p><h4>Materials</h4><p>Just at the top you can see the definition of the various materials for the asset. Some of these use constant properties, whereas the others, such as this one, use a texture map:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/985/1*pNFwn3LnoXhHspuumPaTvw.png" /><figcaption>The material for the body of the Stratocaster, which uses a texture</figcaption></figure><p>For usage with QuickLook, the names of the attributes (a.k.a. the material properties) in the various pbrMat1 need to be exactly the same as shown in the example above. Changing the name of diffuseColor to diffuse in the material “StratocasterGuitar_SynthString_Material” (not the one shown above) results in the loss of albedo colour for the components using that material:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*PGwcT4peRGr1iWONdo2bmg.png" /><figcaption>Notice the strings on the left: they are missing albedo information</figcaption></figure><p>For every material attribute that uses a texture instead of a constant value, QuickLook expects a material Prim like this one:</p><pre>def Shader &quot;color_map&quot;<br>{<br>    uniform token info:id = &quot;UsdUVTexture&quot;<br>    float4 inputs:default = (0, 0, 0, 1)<br>    asset inputs:file = <a href="http://twitter.com/StratocasterGuitar_N">@StratocasterGuitar_N</a>eckWood_Albedo.jpg@<br>    float2 inputs:st.connect = &lt;/Materials/StratocasterGuitar_NeckWood_Material/Primvar.outputs:result&gt;<br>    token inputs:wrapS = &quot;repeat&quot;<br>    token inputs:wrapT = &quot;repeat&quot;<br>    float3 outputs:rgb<br>}</pre><p>and you also need to provide a texture sampler:</p><pre>def Shader &quot;Primvar&quot;<br>{<br>    uniform token info:id = &quot;UsdPrimvarReader_float2&quot;<br>    float2 inputs:default = (0, 0)<br>    token inputs:varname = &quot;Texture_uv&quot;<br>    float2 outputs:result<br>}</pre><p>This is what the material definition looks like when taken as a whole:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*JBj4efa5BX2fl_HSdRlRBw.png" /><figcaption>Another material making use of a texture map for one of its properties.</figcaption></figure><h4>Scene Graph and Meshes</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*6-3wFEa4jMWdrGaSc1DhYA.png" /><figcaption>Expanded view of the scene graph.</figcaption></figure><p>Looking at the expanded Scene Graph for the Stratocaster model, one thing is immediately noticeable: seems that elements are repeating.</p><p>What’s the difference between the “splitContainer” section and the elements ending with “Material”?</p><blockquote><strong><em>tl;dr: </em></strong>They are essentially the same thing. And the nodes without anything attached, the ones ending with “Material”, are simply a by-product of how USD assets are parsed by the viewer and SceneKit.</blockquote><p>If we go back to our .usda file, we can look for the text “StratocasterGuitar_ButtonTone_Material”, and here it is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gn-5_JE6Amt0jYBHWVsEtQ.png" /><figcaption>The ButtonTone submesh</figcaption></figure><p>Submeshes are represented by <strong>GeomSubset </strong>Prims in USD. They are used to specify subsets of the whole model to which to apply a material exclusively.</p><p>If we try to look for any of the “splitContainer” elements, we wouldn’t find them in the file. It is because they are auto-generated from the indices of a given submesh, and they use the material they are assigned via the rel material:binding relationship.</p><p>The indices listed above for the <strong>ButtonTone </strong>element are indices <strong>into faces, instead of vertices</strong>.<strong> </strong>They refer to the global face indices for the whole guitar mesh, which can be found further up in the file:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LrcM_CenCz0Peaw6Hilo1g.png" /><figcaption>Each element in the array describes how many vertices that face uses.</figcaption></figure><p>The “StratocasterGuitar” Prim also defines the elements composing a vertex: positions, normals, texture coordinates, and etc, which are then indexed via vertex indices.</p><h3>Materials in USD</h3><p>The USD specification doesn’t explicitly forbid the usage of non-PBR materials, but <strong>iOS’ QuickLook doesn’t work with non-PBR materials.</strong> So, <strong>make sure that your USDZ file uses PBR materials</strong> or QuickLook will not render them properly.</p><h4>Why is PBR important?</h4><p>PBR stands for “Physically Based Rendering”. PBR enables a variety of enhancements on how realistically objects can be rendered. One of these is the ability to use image-based lighting, which provides more accurate ambient diffuse and specular reflections.</p><p>The differences between “traditional” shading and PBR shading can be stark:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*33KWtM2uyiXavNBOPPZOiQ.jpeg" /><figcaption><a href="https://marmoset.co/posts/pbr-texture-conversion/">https://marmoset.co/posts/pbr-texture-conversion/</a></figcaption></figure><p>You can learn more about PBR from <a href="https://academy.allegorithmic.com/courses/the-pbr-guide-part-1">here</a> and <a href="https://marmoset.co/posts/physically-based-rendering-and-you-can-too/">here</a>.</p><p>In Augmented Reality, PBR is especially relevant as it allows the 3D models being rendered to be shaded in a way which is more aware of its surroundings. For example, if you were to render a USDZ model of a shiny metallic cube, with PBR materials and QuickLook the cube would reflect the environment around it as if it were a real one:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*KjE5HCRmuYoXZWD85kTlQw.gif" /><figcaption>Realistic reflections on a metallic cube with ARKit</figcaption></figure><h3>Current Issues and Limitations with USDZ</h3><h4>Toolset</h4><p>One of the biggest pains by far is the lack of good tools for artists to export their creations from their asset creation software to USDZ directly and without having to use the command line tools. Command line tools are not for everyone, and most of all the provided ones work with only a handful and outdated intermediate asset formats, making it hard to really exploit the functionalities of the USD format.</p><h4>Interactions</h4><p>In their current form, neither the USD or USDZ specifications talk about the ability for the user to interact with the 3D scene described by a USDZ file. Hence, all that’s possible to do at the moment when opening a USDZ file in QuickLook is to look at it, and nothing else. For example, tapping on it won’t trigger any actions and such actions has no way to be coded into the format anyway.</p><p>You can bake animations into the asset, but those are pre-baked and not interactive.</p><h4>PBR only</h4><p>You can’t use non-PBR materials when combining USDZ with QuickLook. Note that nothing stops you from declaring Attributes (in USD-speak) on your Prims which have names resembling the material properties you’d find on, say, a Lambert material, but they won’t work with Pixar’s renderer or Apple’s.</p><p>This is a limitation especially when you want to display widgets/abstract 3D objects rather than 3D models of realistic objects.</p><h4>Apple products only</h4><p>So far USDZ hasn’t been adopted by Android or other platforms, and hence it can only be consumed by users with iOS and MacOS devices. On other platforms, gLTF and gLB seem to be the main candidate to become the USDZ equivalent.</p><p>Moreover, QuickLook is only available on iOS. Although third-party solutions exist, there isn’t an equivalent native solution on Android. But you can be sure that if and when it arrives, it will use the gLTF format and will display the content slightly different from QuickLook.</p><h3>My predictions</h3><p>AR is inherently averse to the concept of apps: you don’t want to switch to a different app every time you want to access a different AR layer or service, let alone download and install an app which you’re going to use only when at a specific location.</p><p>For this reason, the fact that USDZ allows users to interact with AR content without having to install an app is a step forward in that direction, albeit a small one. To really move in the direction of an AR cloud, we need to be able to, at least, add interactions to experiences such as the ones offered by USDZ.</p><p>This calls for a solution closer to a <strong>web browser</strong> than an app store; Mozilla and others are working towards that solution with WebXR.</p><p>Given what the USD format is intended as a format to represent 3D data, <strong>but not interactions</strong>, I don’t think that Apple or Pixar will modify the specifications to introduce interactions and scripting.</p><p>Instead, Apple will either have to bite the bullet and allow users to work with something similar to the WebXR/AR or extend QuickLook/Safari to integrate scripting of the USDZ scenes available.</p><p>Apple will have to gradually introduce better tools to author USD assets if it doesn’t want to officially support gLTF. So, we might see some announcements with respect to that at WWDC 2019.</p><h4>References</h4><p><a href="https://graphics.pixar.com/usd/docs/Introduction-to-USD.html">https://graphics.pixar.com/usd/docs/</a><br><a href="https://graphics.pixar.com/usd/docs/USD-Glossary.html">https://graphics.pixar.com/usd/docs/USD-Glossary.html</a><br><a href="https://graphics.pixar.com/usd/docs/Usdz-File-Format-Specification.html">https://graphics.pixar.com/usd/docs/Usdz-File-Format-Specification.html</a></p><h3>Conclusions</h3><p>USDZ, when combined with ARKit and QuickLook allows you to:</p><ul><li>let users try out 3D models of realistic objects without having to install apps (which is great for e-commerce applications);</li><li>send 3D models to friends via iMessage;</li><li>package USD scenes into a portable format; and</li><li>render objects in 3D realistically thanks to PBR;</li></ul><p>By far the most promising usage for USDZ files is the first one on the list: users can try out products in real-life scale and with a lighting environment that makes them look the same as the final product as much as possible:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0T1YMBxmUNfGKP9jKOn_Cw.png" /><figcaption>What e-commerce websites could show in the near future. From <a href="https://developer.apple.com/arkit/gallery/">https://developer.apple.com/arkit/gallery/</a></figcaption></figure><p><em>Want me to implement custom USDZ projects for you? Click </em><a href="/service/mailto:alberto@albertotaiuti.com"><em>here</em></a><em> to drop me an email.</em></p><p>Follow me on Twitter: <a href="http://twitter.com/albtaiuti">twitter.com/albtaiuti</a></p><p><strong>Follow us on social media platforms:</strong><br>Facebook: <a href="http://facebook.com/AppCodamobile/">facebook.com/AppCodamobile/</a><br>Twitter: <a href="https://twitter.com/AppCodaMobile">twitter.com/AppCodaMobile</a><br>Instagram: <a href="http://instagram.com/AppCodadotcom">instagram.com/AppCodadotcom</a></p><h4>If you enjoyed this article, please click the 👏 button and share to help others find it! Feel free to leave a comment below.</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11bcf24a3deb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating WhatsApp-like animations in iOS]]></title>
            <link>https://medium.com/appcoda-tutorials/creating-whatsapp-like-animations-in-ios-88a7d2d2ca90?source=rss-502684413830------2</link>
            <guid isPermaLink="false">https://medium.com/p/88a7d2d2ca90</guid>
            <category><![CDATA[uikit]]></category>
            <category><![CDATA[ios]]></category>
            <category><![CDATA[apple]]></category>
            <category><![CDATA[swift]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Alberto Taiuti]]></dc:creator>
            <pubDate>Tue, 30 Oct 2018 02:09:11 GMT</pubDate>
            <atom:updated>2018-11-14T05:14:03.418Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to create a fluid and consistent UITabBar and UIToolBar animation like in WhatsApp for your app.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*DyUxGXiq9XirUVHtWs1FWw.gif" /><figcaption>WhatsApp’s transition from a UITabBar to a UIToolBar.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s6upV4tjZqDP0fLikEef6g.gif" /><figcaption>Sample implementation.</figcaption></figure><p>Since I started developing iOS apps, I have been paying more attention to how other extremely successful apps implement UI transitions so that I can implement them in my own ones.</p><p>Recently I was developing an app which had a UICollectionView embedded in a UIViewController, which in turn was embedded in a UICollectionViewController, and I wanted to be able to hide the tab bar at the bottom of the screen to show a tool bar when in edit mode.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*1ol1_iyPWgjhnVxcLzYcEw.png" /><figcaption>The view hierarchy in question.</figcaption></figure><p>One of the apps I use the most is WhatsApp, but until now I had not tried to dissect how it manages its UI transitions. I noticed that the way it handles the transition and position of the UIToolBar at the bottom of the screen when in edit mode fit perfectly what I wanted to implement.</p><p>In this article, I will show you how I implemented it in my app and explain my design decisions. It turns out that doing the above is not trivial and took me a while to figure out. I had to write custom code and too much trial and error. So I hope that this article and the sample project makes your life easier if you want to implement the same.</p><h3>Analysis</h3><p>First, we should check how WhatsApp does things:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*SQ2sSaqNj9EfO0vnEPT5tg.gif" /><figcaption>WhatsApp’s animation slowed down to 10% its original speed.</figcaption></figure><p>From what I can tell, WhatsApp fades the UITabBar in or out and at the same time moves the UIToolBar up or down depending on the editing mode.</p><p>Hence our plan of action is to do exactly the above.</p><h3>Planning</h3><p>These are our requirements:</p><ul><li>We should allow the UINavigationBar to use iOS 11’s large titles</li><li>We should hide the UIToolBar when it’s not being used</li><li>We should do the same for the UITabBar</li></ul><h3>Attack</h3><p>This is where I was completely wrong. I thought that implementing what we discussed above would have been a piece of cake, but it turned out to be extremely convoluted.</p><p>Why?</p><p><strong>It turns out that using the default animation and system for large titles screws up any simple positioning code your might write for your </strong>UIToolBar<strong>.</strong></p><h4>The naive approach</h4><p>The first thing I tried was to use the UINavigationController’s default UIToolBar and change its position according to the editing state of the app.</p><p>However, as I said above, that doesn’t really work if you are using large titles.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*VtuBEMvOb0x6-cQtPpkgAg.gif" /><figcaption>Notice how the UIToolBar gets moved when the large title is animated.</figcaption></figure><p>Moreover, I discovered that I needed to manually resize the <strong>bounds </strong>property of the collection view if I wanted it to cover the whole space between the moved UIToolBar and the UITabBar.</p><h4>The solution</h4><p>In order to circumvent that, we need to add a UIToolBar manually rather than using the default one provided by the UINavigationController object associated with our view controller.</p><p>First we create it lazily:</p><pre>private lazy var toolBar: UIToolbar = {<br>  return UIToolbar()<br>}()</pre><p>and then set up its constraints in <strong>viewDidLoad</strong>:</p><pre>private var toolBarConstraints = [NSLayoutConstraint]()</pre><pre>...</pre><pre><br>override func viewDidLoad() {<br>  super.viewDidLoad()<br>    <br>  ...<br>    <br>  navBar.prefersLargeTitles = true<br>    <br>  navigationController!.isToolbarHidden = true<br>    <br>  view.addSubview(toolBar)<br>  toolBar.translatesAutoresizingMaskIntoConstraints = false<br>  toolBarConstraints.append(contentsOf: [<br>    toolBar.leftAnchor.constraint(equalTo: self.view.leftAnchor),<br>    toolBar.rightAnchor.constraint(equalTo: self.view.rightAnchor),<br>    toolBar.bottomAnchor.constraint(equalTo: self.view.bottomAnchor),<br>    toolBar.topAnchor.constraint(equalTo: tabBar.topAnchor)<br>    ])<br>}</pre><p>We then create some computed properties which we use throughout the sample view controller to setup the custom UIToolBar correctly:</p><pre>private var screenHeight: CGFloat {<br>  return UIScreen.main.bounds.height<br>}<br>  <br>private var toolBarYPos: CGFloat {<br>  if currentState == .normal {<br>    return screenHeight<br>  }<br>  // Ideally check for other states here, but since we only have two, keep it<br>  // simple; would do something like: else if currentState == .edit {<br>  else {<br>    return screenHeight - toolBar.frame.height<br>  }<br>}</pre><p>Finally, we use these computed properties to position the UIToolBar and/or the UITabBar correctly in cases. So when the user taps the Edit or Done buttons:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*BH-cbrn1Dr9uiAK5djclew.gif" /><figcaption>The animation we wanted.</figcaption></figure><p>and this is achieved (partially, but mostly) with this code:</p><pre>@IBAction func onEditBtnTouchUp(_ sender: Any) {<br>    // Switch to the editing mode<br>    if currentState == .normal {<br>      currentState = .edit<br>      <br>      fade(tabBar, toAlpha: 0, withDuration: 0.2, andHide: true)<br>      UIView.animate(withDuration: 0.2, animations: {<br>        // Set edit to done<br>        self.navigationItem.leftBarButtonItem = UIBarButtonItem(barButtonSystemItem: .done, target: self,<br>                                                                action: #selector(self.onDoneBtnTouchUp))<br>        // Fade away + btn<br>        self.plusBtn.isEnabled = false<br>        self.plusBtn.tintColor = UIColor.clear<br>        <br>        // Position the toolbar<br>        self.toolBar.frame.origin.y = self.toolBarYPos<br>      })<br>    }<br>  }<br>  <br>  @objc func onDoneBtnTouchUp(_ sender: Any) {<br>    // Switch to normal state<br>    if currentState == .edit {<br>      currentState = .normal<br>      <br>      fade(tabBar, toAlpha: 1, withDuration: 0.2, andHide: false)<br>      UIView.animate(withDuration: 0.2, animations: {<br>        // Set edit to done<br>        self.navigationItem.leftBarButtonItem = UIBarButtonItem(barButtonSystemItem: .edit, target: self,<br>                                                                action: #selector(self.onEditBtnTouchUp))<br>        <br>        // Fade in + btn<br>        self.plusBtn.isEnabled = true<br>        self.plusBtn.tintColor = nil<br>        <br>        // Position the toolbar<br>        self.toolBar.frame.origin.y = self.toolBarYPos<br>      })<br>    }</pre><h3>Conclusions</h3><p>The final result should look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*vYcPawNKcah8jeC-dXIc7w.gif" /></figure><p>I have setup a repository with the sample project so that you can go and read the full code:</p><p><a href="https://github.com/snowzurfer/whatsapp-bars">snowzurfer/whatsapp-bars</a></p><p>I recommend checking it out, since I deliberately missed some details in the article for the sake of brevity.</p><p>It took me a long time to figure this out, because the interplay between the different UIKit elements did not work as I expected.</p><p>The main takeaways when developing more advanced UI interactions in Swift on iOS are:</p><ol><li>Try the easiest solution first; if that works, cool!</li><li>If not, keep on trying new ones and experiment.</li><li>It can be very frustrating, but don’t give up!</li></ol><p>If you like this kind of stuff, you should follow me on Twitter: <a href="https://twitter.com/nathangitter">twitter.com/albtaiuti</a></p><p>The style of this post was partially inspired by <a href="https://twitter.com/nathangitter">Nathan Gitter</a>’s Medium posts, and I also used <a href="https://github.com/nathangitter/fluid-interfaces">some of his code</a>. Thanks for sharing it with us Nathan!</p><p><strong>Follow us on social media platforms:</strong><br>Facebook: <a href="http://facebook.com/AppCodamobile/">facebook.com/AppCodamobile/</a><br>Twitter: <a href="https://twitter.com/AppCodaMobile">twitter.com/AppCodaMobile</a><br>Instagram: <a href="http://instagram.com/AppCodadotcom">instagram.com/AppCodadotcom</a></p><h4>If you enjoyed this article, please click the 👏 button and share to help others find it! Feel free to leave a comment below.</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88a7d2d2ca90" width="1" height="1" alt=""><hr><p><a href="https://medium.com/appcoda-tutorials/creating-whatsapp-like-animations-in-ios-88a7d2d2ca90">Creating WhatsApp-like animations in iOS</a> was originally published in <a href="https://medium.com/appcoda-tutorials">AppCoda Tutorials</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apple steals the AR scene at WWDC18]]></title>
            <link>https://medium.com/inborn-experience/apple-steals-the-ar-scene-at-wwdc18-8069e13be267?source=rss-502684413830------2</link>
            <guid isPermaLink="false">https://medium.com/p/8069e13be267</guid>
            <category><![CDATA[wwdc]]></category>
            <category><![CDATA[augmented-reality]]></category>
            <category><![CDATA[arkit]]></category>
            <category><![CDATA[ios]]></category>
            <category><![CDATA[apple]]></category>
            <dc:creator><![CDATA[Alberto Taiuti]]></dc:creator>
            <pubDate>Mon, 04 Jun 2018 19:15:59 GMT</pubDate>
            <atom:updated>2018-06-05T00:18:36.326Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9rottEySnatcc8wAZsB5Kw.png" /><figcaption>Craig Federighi at WWDC18</figcaption></figure><p>This year’s WWDC was packed with AR-related content. Apple singlehandedly pulled the rug from under some of the main AR startups working in persistence and content sharing, and more.</p><p>During the short interlude of the keynote dedicated to AR, Craig Federighi introduced <strong>ARKit 2</strong> and showed off a list of new features which will be available with iOS 12.</p><p>By far, the one that got my attention the most was <strong>persistence</strong>. That’s right: <strong>Apple announced it is working on persistence</strong>. Presumably, they are starting with <strong>small-scale persistence</strong>.</p><p>What does that mean and why is this important? If you are familiar with the <strong>AR cloud</strong> development scene, that means that Apple is entering the same market in which companies like Placenote, 6D.ai and Ubiquity6 are also competing. Since Apple owns the ARKit platform, the experience will probably work very smoothly and will be easy to develop with.</p><p>With this addition it will be possible to develop native applications which persistently place AR content in real-world scenes, and the possibilities enabled by this feature are endless. A possible use case mentioned during the keynote was placing persisent content for an interactive, AR-based classroom teaching app. Kids could use their ClassKit-enabled devices to then interact with content directly and exactly where it is supposed to appear in the class.</p><p>From the documentation, it seems like it is possible to save the map and achieve full persistence. <a href="https://developer.apple.com/documentation/arkit/arworldmap">From Apple’s docs:</a></p><blockquote><strong>Persistent AR experiences.</strong> Save a world map when your app becomes inactive, then restore it the next time your app launches in the same physical environment. You can use anchors from the resumed world map to place the same virtual content at the same positions from the saved session.</blockquote><p>The second, very exciting feature announced was <strong>3D object recognition</strong>. That means that you will be able to use 3D, real-world object as markers, similarly to what is already possible to do with images (aka <strong>ARImageAnchors</strong>)<strong> </strong>in iOS 11.4. <a href="https://library.vuforia.com/articles/Training/Object-Recognition">Vuforia already offered this feature</a>, but this was not an iOS-native feature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EgbTH4N2513ZOnqZFg1yfQ.jpeg" /><figcaption>Lego showcasing 3D object recognition at WWDC18. <a href="https://twitter.com/OsFalmer/status/1003693556605837312">Pic by Oscar Falmer</a></figcaption></figure><p>Lego showed an impressive demo which introduced the 3D object recognition feature by using a real Lego house as the anchor for additional AR content. Whit this addition, a myriad of new apps which augment real objects are possible and I look forward to finding out what the process is to load a 3D model to use as an anchor. Possibly, one will be able to add the 3D model to their XCode project similarly to what happens for ARImageAnchors.</p><p>Third, Apple also announced the ability to have shared AR experiences between phones similarly to what offered by Google Cloud Anchors. It showed off a video demo of this feature and announced that an XCode template project will be available soon to learn how to use this new ARKit API.</p><p>All of this is topped up with a new file format, <strong>USDZ</strong>. This file format allows to easily package an AR scene/experience in one file. Think of it as a SCNScene on steroids. It not only encodes geometry, but transforms and animations too. This goes hand in hand with the newly added persistence so that one can deliver packaged AR experiences to a specific location.</p><p>Overall, ARKit 2 looks really exciting and the development community will come up with amazing use cases for the features introduced by it. Apple showed that it has at least an understanding of what is necessary to achieve immersive AR experiences and to fully grasp the potential of Augmented Reality, and is going all in.</p><p>I am genuinely excited about what is now possible to be created using iOS 12 and ARKit 2 and look forward to getting my hands on the new tools.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8069e13be267" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inborn-experience/apple-steals-the-ar-scene-at-wwdc18-8069e13be267">Apple steals the AR scene at WWDC18</a> was originally published in <a href="https://medium.com/inborn-experience">Inborn Experience (UX in AR/VR)</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Google Cloud Anchors doesn't deliver on the hype]]></title>
            <link>https://medium.com/inborn-experience/why-google-cloud-anchors-doesnt-deliver-on-the-hype-e5d0a3fe704d?source=rss-502684413830------2</link>
            <guid isPermaLink="false">https://medium.com/p/e5d0a3fe704d</guid>
            <category><![CDATA[arcore]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[augmented-reality]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Alberto Taiuti]]></dc:creator>
            <pubDate>Thu, 10 May 2018 00:03:45 GMT</pubDate>
            <atom:updated>2018-05-10T09:38:59.171Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*BaUF14vNo2Ss9gDDPMWzRw.png" /></figure><h3>Why Google Cloud Anchors doesn’t deliver on the hype</h3><p>At the Google I/O 2018 conference, Google announced the availability of its Cloud Anchors library.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*jsSOBjtZt3_Uh1oitI0psQ.gif" /><figcaption>Google Anchor enables local multiplayer experiences, but nothing more.</figcaption></figure><p>This library allows sharing of an AR session between two AR-enabled smartphones (for now, until we also have glasses) so that two users can interact with the same AR content, placed at the same location, effectively sharing the same scene.</p><p>This is achieved by Google running the repositioning and persistence feature of ORB-SLAM 2 on their cloud servers. The apps send raw data captured by the devices, and Google’s servers calculate a shared transform for the two (or more) devices which is then used as an origin by both phones to place content with respect to.</p><p><strong>One would think that this is huge</strong>; the AR cloud, AKA persistence, is one of the most requested, talked and anticipated features of AR, and many companies such as Placenote, 6D.ai and BlueVision are trying to achieve this by bringing to the market their implementation of such cloud. Some of these companies run ORB-SLAM 2 on the device rather than on the cloud like Google, but the underlying principle is the same: because by default ARKit and ARCore don’t give you access to the algorithm’s feature points, you need to run your own custom version and then integrate it on top of the existing AR libraries provided by the vendors.</p><p>However, these libraries are still very much in beta, so the fact that Google released their implementation, and that its Cloud Anchors library is available also on iOS is, on the surface, just amazing.</p><p>Google has also released a sample application for iOS <a href="https://github.com/google-ar/arcore-ios-sdk">here</a>. However, the sample application is written in Objective-C (so it would be great if there were a Swift version too).</p><p>Now, why isn’t all this great?</p><p>First of all a bit of context:</p><p><strong>Persistence</strong>, in the AR meaning of the word, is the ability for you to place content somewhere in an AR scene like you already do in many apps, with the addition of being able to find the content in the same place where you’ve left it if you come back to the same physical spot some time after. It’s analogous to you putting a poster up on a wall in real life, and then going away: when you come back to the same wall, you’d expect the poster to still be there, right?</p><p>Well, this feature is not available by default on ARKit and ARCore, so people have come up with custom solutions as I explained above.</p><p>However, at the same time, those custom solutions which I talk about above, fully deliver on persistence, whereas Google’s Cloud Anchors don’t, for the following reasons:</p><h4>The Anchor data can only be accessed within one day of creation</h4><p>So if you create an anchor for you or others to use, when you come back to it one day after, it will be gone.</p><p>Say you place a 3D model of a chair somewhere through AR in your house and anchor it; after one day, it won’t appear in the spot where you want it to be any longer.</p><h4>The raw data sent to the servers is deleted after 7 days</h4><p>Hence not only your anchor transforms will disappear, you won’t be able to retrieve even the raw data consistently.</p><p>At first, I was very excited: I have been waiting for a long time for someone to implement persistence on smartphones properly and had even started to implement my own version.</p><p>However, in order to fully achieve persistence, we need to able to consistently store and share feature points data and for it to be retrieved whenever needed. Without that, the only thing that Google Anchors enables is local multiplayer, which could already be achieved by using a marker or image as a starting anchor, without the need to run ORB-SLAM 2 on the servers and do fancy feature recognition.</p><p>I really hope that this is just the first step for Google Anchors, and that in the near future they will enable the storage of the anchors for an indefinite time and make it easy to share them. Until then, you can achieve the same with image anchors.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5d0a3fe704d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/inborn-experience/why-google-cloud-anchors-doesnt-deliver-on-the-hype-e5d0a3fe704d">Why Google Cloud Anchors doesn&#39;t deliver on the hype</a> was originally published in <a href="https://medium.com/inborn-experience">Inborn Experience (UX in AR/VR)</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[One powerful rule to improve your knowledge exponentially in only one year]]></title>
            <link>https://medium.com/@albtaiuti/one-powerful-rule-to-improve-your-knowledge-exponentially-in-only-one-year-1cb572378f6d?source=rss-502684413830------2</link>
            <guid isPermaLink="false">https://medium.com/p/1cb572378f6d</guid>
            <category><![CDATA[hacks]]></category>
            <category><![CDATA[challenge]]></category>
            <category><![CDATA[books]]></category>
            <category><![CDATA[reading]]></category>
            <category><![CDATA[self-improvement]]></category>
            <dc:creator><![CDATA[Alberto Taiuti]]></dc:creator>
            <pubDate>Thu, 15 Mar 2018 23:35:09 GMT</pubDate>
            <atom:updated>2018-03-15T23:35:09.322Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*L4blgPqcWXAtsGf9." /><figcaption>Photo by <a href="https://unsplash.com/@amadorloureiroblanco?utm_source=medium&amp;utm_medium=referral">Amador Loureiro</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>How to be like the top performers in the world</h4><p>I have always been fascinated by great entrepreneurs and scientists who have been or were hugely successful.</p><p>If you also want to be successful in life, you will ask yourself if there is a trait these people share; some notion that is not visible from on the surface but which, hidden from superficial analysis, connects them.</p><p>What all these people have in common, is that <strong>they read a lot</strong>.</p><p>Take <strong>Elon Musk</strong> for example. <br><a href="https://www.goodreads.com/book/show/25541028-elon-musk">According to Ashlee Vance’s biography of him,</a> as a kid Elon <em>read all the books he could get his hands on</em> and would often disconnect from the world in order to spend as much time as possible doing this.</p><p>The famous saying goes that once he had finished all the books available, he started reading the Encyclopedia Britannica.</p><p>Another great example is Warren Buffet, who said he spends 80% of his time reading.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6J185s2oDZhMRLdpUstdfw.jpeg" /><figcaption>You want to read as much as this guy here does.</figcaption></figure><p>The same goes for other famous and successful people, such as Mark Zuckerberg and Jeff Bezos.</p><p>Bill Gates even has a very popular reading list he periodically publishes so that anyone can read more of what a mastermind like his reads.</p><p>Last year, in May 2017, I set myself to read as much as I could, and bought an Amazon Kindle. I started carrying it with me anywhere I went and would open it to read during any idle time: while on the train or plane, while having lunch, when waiting for my lunch, etc.</p><p>That allowed me to read 33 books in around 7 months, at least according to Goodreads. Hell, one of those was Lord Of The Rings…</p><p><strong>One thing you should absolutely avoid is “just reading”.</strong> Unless you are reading a book for pleasure, I recommend you read with method and using the hacks explained in <a href="https://www.goodreads.com/book/show/18693655-a-mind-for-numbers?ac=1&amp;from_search=true">A Mind For Numbers</a>. Using those reading and learning hacks, coupled with Anki, has allowed me to retain most of what I read in those books.</p><p>I can’t believe how much I did not know, and the more I read, the more I knew I needed to read more. So I look forward to keeping up this count in 2018.</p><p>So start reading more today!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1cb572378f6d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing the GTD system in a Moleskine notebook]]></title>
            <link>https://medium.com/@albtaiuti/implementing-the-gtd-system-in-a-moleskine-notebook-b4082a5c07c6?source=rss-502684413830------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4082a5c07c6</guid>
            <category><![CDATA[getting-things-done]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[leadership]]></category>
            <category><![CDATA[students]]></category>
            <category><![CDATA[self-improvement]]></category>
            <dc:creator><![CDATA[Alberto Taiuti]]></dc:creator>
            <pubDate>Thu, 15 Mar 2018 23:24:50 GMT</pubDate>
            <atom:updated>2018-03-15T23:24:50.497Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JIeN0qT7jXswgv_bpIBo_g.jpeg" /></figure><p>We all want to improve ourselves, waste less time, and lead more efficient lives.</p><p>It can often be hard, in such a busy world filled with so many distractions, to be able to organise, prioritise and be self disciplined so that we can achieve our goals and career aspirations.</p><p>The <a href="https://en.wikipedia.org/wiki/Getting_Things_Done"><strong>Getting Things Done</strong></a><strong> </strong>(known as GTD for short) framework is a great way to do just that and improve your productivity.</p><p>There are many good resources on the internet which explain the underlying principles of the GTD method. Some of these are:</p><ul><li><a href="https://hamberg.no/gtd/">https://hamberg.no/gtd/</a></li><li><a href="http://mikeshea.net/An_Introduction_to_Gettin.html">http://mikeshea.net/An_Introduction_to_Gettin.html</a></li></ul><h4><strong>At its core</strong>, the GTD method consists of:</h4><ul><li>A medium to note down anything which comes to mind</li><li>An agenda to keep a list of appointments</li><li>A place to store longer-term projects</li><li>A list of actionable items which are context-dependent</li></ul><p>My implementation does not use a calendar section because appointment entries can change very quickly (which does not work so well on a paper medium) and because being able to synchronise calendar appointments with other people is way too useful to do away with.</p><p>Here is a simple step-by-step guide based on my tried and tested experience of what works and what doesn’t, which you can use to setup the GTD framework on a pocket-size notebook.</p><p>I will explain in detail why a paper notebook is a great solution to implement GTD and how to adapt it to suit your needs.</p><h3>Step 1</h3><h4>Learn the basics of the GTD method</h4><p>In order to use GTD, you first have to understand it.</p><p><em>The basic idea is that GTD is a framework to enable you to free up your brain power from having to remember all the tasks you have to do so that you can use your brain more for thinking and planning, than as a storage box.</em></p><p>As <a href="https://hamberg.no/erlend/">Erlend Hamberg</a> puts it in his excellent <a href="https://hamberg.no/gtd/">GTD in 15 minutes — A Pragmatic Guide to Getting Things Done</a>:</p><blockquote>Its aim is to make you have 100% trust in a system for collecting tasks, ideas, and projects — both vague things like “invent greatest thing ever” and concrete things like “call Ada 25 August to discuss cheesecake recipe”. <em>Everything!</em></blockquote><p>I highly recommend that website. It’s where I first learnt about GTD and it was the main inspiration for my adaptation of it.</p><blockquote><strong>Pro tip:</strong><br>Don’t be too stringent on the rules of GTD and instead modify it to suit your needs</blockquote><h3>Step 2</h3><h4>Buy a pocket-sized notebook</h4><p>That doesn’t mean a laptop. It means a paper, old-school notebook which can fit in your pockets and you can carry with you <em>all-the-time</em>.</p><p>I personally use <strong>pocket-size Moleskines, squared.</strong></p><p>You can find them very cheaply on Amazon, or in shops if you’re ready to pay a bit more for a wider choice of colours. However, any other similarly-sized notebook will do too.</p><blockquote><strong>Pro tip:</strong><br>Buy soft cover ones so that you don’t have to worry about bending them.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*YyMdKAYjcY5oPaXd6nz9lA.jpeg" /><figcaption>Moleskine Squared Notebook with Soft Cover</figcaption></figure><p>Using a paper notebook is important for 3 reasons:</p><ol><li>It’s extremely easy to hack and personalise</li><li>It doesn’t run out of battery and lasts forever</li><li>You can take notes as complex as you want</li></ol><p>The obvious downside is that it’s not connected to the internet. However, it offers a reliability and freedom which is very difficult to find on electronic devices.</p><p>The notebook will become your new best friend and the cause of much amusement in your friends and people who meet you. You will quickly become “the guy with the notebook” and people will ask you where you’ve left it if they don’t see it on you.</p><h3>Step 3</h3><h4>Buy sticky tabs</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*r49OvDPiZXAFEhtySAQyIQ.jpeg" /><figcaption>Efficiency calls for tabs</figcaption></figure><p>These are an essential tool as they will help you quickly navigate your GTD-Moleskine system.</p><p>I’m a big fan of the Post-It coloured tabs, which are extremely cheap and work very well.</p><blockquote><strong>Pro tip:</strong><br>Buy the packs with tabs of multiple shapes. Having a variety of shapes other than only colours makes it easier to identify sections.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/1*InLexXAeDBDsUhfNqK6FmA.jpeg" /><figcaption>You can find these pretty much at any copy shop or on Amazon</figcaption></figure><h3>Step 4</h3><h4>Buy a pen loop</h4><p>Having a pen or pencil always handy is paramount to being able to rely on your paper GTD system.</p><p>However, let’s be honest: we <em>constantly</em> lose pens and pencils.</p><p>For this reason it is handy to have a <strong>pen loop</strong> attached to your GTD notebook, where you can easily keep a pen and rely on the fact that 90% of the time you won’t lose it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*3kMj_pu_sN68cpk11fVDrQ.jpeg" /><figcaption>Leuchtturm pen loops are excellent</figcaption></figure><h3>Step 5</h3><h4>Set up the notebook</h4><p>We now get to the core of the article, where we will map physical sections of the notebook with the core sections of GTD we introduced before. Before we start, let’s remind ourselves what these are:</p><ol><li>An medium to note down anything which comes to mind</li><li>A list of actionable items which are context-dependent</li><li>A place to store longer-term projects</li></ol><h4>The “@inbox” section</h4><p>Reserve a large chunk of the pages in the notebook for the <strong>inbox </strong>section. It maps directly to the “inbox” section of the GTD system.</p><p>To mark it, write “@inbox” on the first line of the first page, and add the date when you started it. This will be helpful when reviewing items.</p><p>This section is where you note down anything which is thrown at you during the day, be it a request from your boss or the idea for your next project. Any time you encounter something which you might act on, note it down in the inbox section of your notebook and add a square box besides it so that you can tick it off once you have reviewed it.</p><blockquote><strong>Pro tip:</strong><br>It’s also a good place to sketch or draw mind maps.</blockquote><p><strong>Use the notebook’s string bookmark to mark where your latest inbox entry is</strong>. This way you can navigate to the empty section of the inbox and write a new entry extremely quickly.</p><p>Every time you go through the weekly review, mark the location up to which you processed inbox items with a sticky tab. This makes it easy to find where the first item since the last review was entered in case you mess up with the string bookmark.</p><p>The best place in the notebook for the inbox section is from the start of the notebook up until more than half of it. The distribution between the sections is something you will master with time as you use the system <em>so don’t overthink the size of the section</em>s.</p><h4>The “@next actions” section</h4><p>You can reserve a smaller amount of space for this section compared to the inbox section. I recommend anywhere between 20 to 40 pages.</p><p>Every time you start a new one, mark it with a tab of a different colour from the one you used for the @inbox section and remove the tab from the old @next actions section.</p><p>Use two adjacent pages in the notebook so that you can read the list of actions at a glance and start working on them depending on your context.</p><p>The contexts you create depend on your needs and should change with time. For example, for a while I had a @paul (my thesis supervisor) context since I often had actionables related to the weekly meetups I had with him.</p><p>Give more space to contexts which you know will have more actionables so that you don’t have to mess up your nice layout by running out of space.</p><blockquote><strong>Pro tip:</strong><br>Use circles instead of boxes to indicate status. Semicircle: action is active. Full circle: action completed.</blockquote><h4>The “@projects” section</h4><p>This can either be a list which you grow with time or something more similar to the @next actions section.</p><p>I like to let it expand with time as I add new projects, striking off the projects which I have either completed or have become irrelevant.</p><p>As usual mark it with a tab of a different colour from the other sections so that you can quickly find it.</p><h3>Step 6</h3><h4>Running reviews</h4><p>As per the GTD method, you should periodically review the items in your @inbox list.</p><p>As you go through the items in the inbox and either add them as a project, add them as a next action or discard them, tick them off. Once you are done, mark the start of the next @inbox section with a date of the current day.</p><p>If like me you are not integrating a calendar, I suggest not changing the @next actions list until full. This way you maximise the lifespan of your notebook by saving space.</p><h3>Bonus tips</h3><h4>Page numbering</h4><p>When you start a fresh notebook, number the pages from the start of the notebook until its end. This makes it extremely easy to cross-reference pages within the notebook.</p><h4>Intro section</h4><p>Keep a few pages at the start for things such as:</p><ul><li>Phone numbers</li><li>An index</li><li>Information about yourself in case you lose our notebook</li></ul><p>Having this information can be life-saving in cases where you don’t have access to a phone.</p><p>The <strong>index</strong> section is especially useful to organise sections and keep track of entries which you do not need to constantly reference but which you would like to know their location, e.g.: directions from the airport to your AirBnb. Use the page numbers to refer to these sections as a way not to clutter your notebook with tabs.</p><h4>Tabs at end of mole</h4><p>Keep some spare tabs at the end of your notebook so that you can always have some to mark important entries in the @inbox section.</p><h3>Now go and experiment</h3><p>Ultimately, in <em>your</em> GTD system, you should strive for simplicity and avoid to get too worked up on the details. <strong>See what works for you and experiment</strong>.</p><p>Although this might seem like a lot of information and that instead of simplifying your life it clutters it with unnecessary complexity, once you get the hang of it you will not look back.</p><p>Thanks to your notebook you will be able to jot down anything at any time with total freedom of expression. No more of those moments where you have a great idea when having pizza with your friends but can’t recall it after.</p><p>Yes, you will look like a nerd when taking out your Moleskine during dinner, but you should rejoice in knowing you’re way ahead of others when it comes to freeing your mind from clutter and getting things done.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4082a5c07c6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using the Blockchain for scientific research accountability]]></title>
            <link>https://medium.com/@albtaiuti/using-the-blockchain-for-scientific-research-accountability-7a6f187aeba6?source=rss-502684413830------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a6f187aeba6</guid>
            <category><![CDATA[scientific-research]]></category>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[research]]></category>
            <dc:creator><![CDATA[Alberto Taiuti]]></dc:creator>
            <pubDate>Sat, 13 Jan 2018 16:39:18 GMT</pubDate>
            <atom:updated>2018-01-13T16:39:31.469Z</atom:updated>
            <content:encoded><![CDATA[<p>In this post, I will present an idea for a possible application of the Blockchain technology to avoid manipulation of important scientific reports by third parties when the topics treated by these works is controversial, e.g. Climate Change.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EQdMR-XZJtSc1plckJANMA.jpeg" /><figcaption>Polar bears aren’t doing well these days. Picture by Altoday.com</figcaption></figure><p><em>I am going to assume that the reader has a basic knowledge of the blockchain </em>and its possible applications. Check out these resources to learn more about it:</p><ul><li><a href="https://www.youtube.com/watch?v=_160oMzblY8&amp;t=906s">Blockchain 101</a> (video)</li><li><a href="https://www.youtube.com/watch?v=xIDL_akeras">Blockchain 101 — Part 2</a> (video follow up to above)</li><li><a href="https://www.youtube.com/watch?v=Lx9zgZCMqXE">How Bitcoin Works Under the Hood</a> (video)</li><li><a href="https://github.com/ethereum/wiki/wiki/White-Paper">Ethereum Whitepaper</a> (paper)</li></ul><h3>The issue</h3><p>I am currently reading <a href="https://www.goodreads.com/book/show/19069803-merchants-of-doubt"><strong>Merchants of Doubt: How a Handful of Scientists Obscured the Truth on Issues from Tobacco Smoke to Global Warming</strong></a><strong> </strong>by <a href="https://www.goodreads.com/author/show/142427.Naomi_Oreskes">Naomi Oreskes</a> and <a href="https://www.goodreads.com/author/show/677596.Erik_M_Conway">Erik M. Conway</a>.</p><p>The book highlights how interested <strong>3rd parties or government intervened, multiple times, to twist the reports redacted by independent scientists and scientific governing bodies as to create “controversy”</strong> and maintain a level of uncertainty in the population and the government officials and obtain favourable Congress decisions. As Oreskes and Conway put it,</p><blockquote>“The industry had realized you could create the impression of controversy simply by asking questions”</blockquote><p>One such example is the Acid Rain case. After the panel instructed to conduct a detailed investigation of the implications of acid rain on the environment had redacted its report, this was manipulated before being presented to Congress:</p><blockquote>“two parts — the Executive Summary, and Singer’s appendix — had <em>not</em> been approved by all panel members. <strong>Most of the members didn’t even know that the summary had been changed</strong>. … changes in order, in adjectives, and in tone had changed the tenor of the report, <strong>so the reader was left with a very different impression</strong>.”</blockquote><p>Hence the decision to reduce harmful emissions was delayed and the modified report used as a proof that no action was yet necessary.</p><p>After reading this, I realised that things could have gone much differently if the drafts of the report and the version initially approved by the panel <strong>had been saved somewhere which makes it impossible to tamper with the data and which is publicly accessible for other to check the history of the data saved in it.</strong></p><p>What medium allows for traceability, consistency and makes it virtually impossible to modify the history contained in it?</p><p>The answer is <strong>the blockchain.</strong></p><h3>How could the Blockchain be used?</h3><p>If the scientists had committed their paper as it was undergoing external review to the blockchain, they could have presented their case more strongly both to Congress and to the public, proving without any doubt both <strong>when</strong> and <strong>how</strong> they had produced their original work.</p><p>It would be great to create a platform where researchers are able to share their work freely to all the other researchers and only display it publicly after the final revision is complete. The reports could cross-reference each other with the ability to easily retrieve them by their hash id.</p><p>The documents could be represented as digital assets which, once committed to the blockchain, cannot be altered. The researchers could have the ability to commit drafts and keep them invisible, and only make them available if necessary, such as in the case when there is a dispute on whether a given important report has been tampered with or not.</p><p>I would be very interested to hear people’s opinion on this, so do let me know what you think!</p><h4>Further reading:</h4><ul><li><a href="https://www.goodreads.com/book/show/19069803-merchants-of-doubt"><strong>Merchants of Doubt: How a Handful of Scientists Obscured the Truth on Issues from Tobacco Smoke to Global Warming</strong></a><strong> </strong>by <a href="https://www.goodreads.com/author/show/142427.Naomi_Oreskes">Naomi Oreskes</a> and <a href="https://www.goodreads.com/author/show/677596.Erik_M_Conway">Erik M. Conway</a></li><li><a href="https://smt.steem.io/">SMT</a> tokens</li><li><a href="https://www.reddit.com/r/ethereum/comments/7pf84l/blockchain_technology_for_scientific_papers/">https://www.reddit.com/r/ethereum/comments/7pf84l/blockchain_technology_for_scientific_papers/</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a6f187aeba6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>