<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Arseny Kapoulkine on Medium]]></title>
        <description><![CDATA[Stories by Arseny Kapoulkine on Medium]]></description>
        <link>https://medium.com/@zeuxcg?source=rss-8819be2d098a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*lKXeliFOElaUTSnnBIm8Wg.jpeg</url>
            <title>Stories by Arseny Kapoulkine on Medium</title>
            <link>https://medium.com/@zeuxcg?source=rss-8819be2d098a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 13:02:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@zeuxcg/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Creating a Robust Pipeline Cache with Vulkan]]></title>
            <link>https://medium.com/@zeuxcg/creating-a-robust-pipeline-cache-with-vulkan-961d09416cda?source=rss-8819be2d098a------2</link>
            <guid isPermaLink="false">https://medium.com/p/961d09416cda</guid>
            <category><![CDATA[roblox]]></category>
            <category><![CDATA[cache]]></category>
            <category><![CDATA[vulkan]]></category>
            <category><![CDATA[tecnology]]></category>
            <dc:creator><![CDATA[Arseny Kapoulkine]]></dc:creator>
            <pubDate>Thu, 25 Feb 2021 17:58:11 GMT</pubDate>
            <atom:updated>2021-05-12T21:17:00.664Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WbWGJKLNg--6vDLX0KJJuw.jpeg" /></figure><p>There are a lot of new concepts to learn when building a Vulkan renderer. Some of them are easier to deal with than others, and one of the more straightforward additions is the pipeline cache. To make sure pipeline creation is as efficient as possible, you need to create a pipeline cache and use it whenever you need to create a new pipeline. To make sure subsequent runs of your application don’t have to spend the time repeatedly compiling the shader microcode, you need to save the pipeline cache data to a file. Then load it next time your application starts. How hard can it be?</p><p>Pretty hard, as it turns out.</p><h3>What’s in a pipeline cache?</h3><p>Pipeline cache data is a (mostly) opaque blob; you create a VkPipelineCache object, possibly giving it the initial blob to start with, and then at some point you can retrieve the data blob from this object.</p><p>While we don’t know much about the contents of the blob short of reading graphics driver source code,1 the pipeline cache data is guaranteed to start with a structure that identifies the device and looks something like this:</p><pre>struct VkPipelineCacheHeaderOne</pre><pre>{</pre><pre>uint32_t length; // == sizeof(VkPipelineCacheHeaderOne)</pre><pre>uint32_t version; // == VK_PIPELINE_CACHE_HEADER_VERSION_ONE</pre><pre>uint32_t vendorID;</pre><pre>uint32_t deviceID;</pre><pre>uint8_t uuid[VK_UUID_SIZE];</pre><pre>};</pre><p>The header is followed by driver-specific information that typically contains bits of shader microcode (the format of which depends on the GPU) and auxiliary data that may contain arbitrary driver defined structures. Some drivers treat this blob as a structured file stream and read data from it, some drivers store raw structures defined in the driver source in that blob and use memcpy or pointer casts to navigate the data; needless to say, a driver update may invalidate the way the data is stored.</p><p>In theory, the application just needs to use vkGetPipelineCacheData to retrieve a data blob after the application reaches a steady state (for example before the application exits…), save the blob to a file, and then pass this blob using VkPipelineCacheCreateInfo::pInitialData when creating the pipeline cache on the next run. If the contents of the blob doesn’t work for the current version of the driver — maybe the driver was updated, or maybe the user switched to a different GPU — the driver is supposed to ignore the initial data and create an empty pipeline cache.</p><p>But theory and practice are a bit different. The rule of thumb in practice is that a driver will only be able to correctly handle the <em>exact</em> blob that the <em>exact</em> same driver gave your application previously, which is where the problems begin.2</p><h3>Is the driver the same?</h3><p>The specification assumes that the cache isn’t compatible between different devices (which is why vendorID and deviceID are present in the header), and relies on the driver to establish a pipeline UUID (which is a 16-byte GUID) that accurately identifies the full set of factors that lead to being able to interpret a pipeline cache blob — you can think of this as a version number of the pipeline cache format. During a driver upgrade, for example, it may be the case that the pipeline cache format is <em>not</em> updated, in which case the UUID typically shouldn’t change and the application won’t need to recompile the shaders from scratch.</p><p>However, drivers in the wild tend to exhibit two types of problems.</p><p>Some (older) drivers neglect to verify the UUID correctly. As a result, during a driver update the application may try to give the blob with a stale UUID to the driver, the driver will try to interpret this as recent data and as a result, vkCreatePipelineCache may crash. Note that in general vkCreatePipelineCache doesn’t provide a guarantee that it accepts <em>arbitrary</em> data and can handle it cleanly.</p><p>Some drivers, including pretty recent ones, may neglect to update UUID in a driver update that actually breaks compatibility of the shader pipeline binary. This can happen during a driver version update (although this is rare), or (something that happens trivially on current drivers of at least one major vendor) between driver binaries that are built from the same version for different ABI. If a 32-bit driver and a 64-bit driver that ship on the same system have the same pipeline UUID, then <em>saving</em> the cache from a 32-bit version of the application and <em>loading</em> it from a 64-bit version may cause the driver to crash — which is <em>exactly</em> what happens when you ship a 32-bit version of your application and then update it to 64-bit following Google’s guidelines.</p><h3>Is the data the same?</h3><p>Now that we know what awaits us when it comes to header validation, what’s next is validating the data. After calling vkGetPipelineCacheData, the application saves the blob, and loads the exact same blob on the next run.</p><p>It turns out that saving data to a file is <a href="https://danluu.com/deconstruct-files/">basically impossible to do well</a>. Filesystem issues as well as process stability issues may in some cases lead to files that are partially written, have chunks filled with zeroes at the end (or even with garbage), or (as a special case) are created but stay zero-size. On mobile, this can be complicated by the fact that the application is likely to be terminated abruptly at an arbitrary point in time by the user or the OS, something that happens less frequently on desktop. On Android it’s also common to use multi-process (multi-activity) applications and, if your pipeline cache code runs in both processes and shares the same output file, these challenges become even harder to solve.</p><p>The reason why zero-size files are particularly interesting is that there is at least one driver version that we’ve run into where passing a non-NULL pInitialData and initialDataSize == 0 returns an error during pipeline cache creation. Which brings us to the final caveat.</p><h3>Error handling is hard</h3><p>While the spec says that vkCreatePipelineCache should basically always succeed short of running out of memory, such statements in the spec are rarely accurate. When creating the pipeline cache, the driver is supposed to ignore initial data if it’s incompatible. This may occur if it’s zero-sized, if the stored UUID didn’t match the expected UUID, or if deserialization failed for any other reason). Some drivers instead fail to create the pipeline cache.</p><p>The user definitely isn’t at fault here, so aborting the application would not be polite. While it’s generally possible to proceed without a pipeline cache, that’s usually a terrible idea because that means that each pipeline has to be recompiled from scratch. That is, pipeline caches have utility even if they are not serialized to disk because they allow the driver to cache the results of compilation across pipeline objects in memory.</p><p>All of this naturally leads to…</p><h3>It’s not paranoia if they are really out to get you</h3><p>… the solution. When serializing pipeline cache data to the file, we use a header that is filled with enough information to be able to validate the data, with the pipeline cache data following immediately afterwards:</p><pre>struct PipelineCachePrefixHeader</pre><pre>{</pre><pre>uint32_t magic; // an arbitrary magic header to make sure this is actually our file</pre><pre>uint32_t dataSize; // equal to *pDataSize returned by vkGetPipelineCacheData</pre><pre>uint64_t dataHash; // a hash of pipeline cache data, including the header</pre><pre>uint32_t vendorID; // equal to VkPhysicalDeviceProperties::vendorID</pre><pre>uint32_t deviceID; // equal to VkPhysicalDeviceProperties::deviceID</pre><pre>uint32_t driverVersion; // equal to VkPhysicalDeviceProperties::driverVersion</pre><pre>uint32_t driverABI; // equal to sizeof(void*)</pre><pre>uint8_t uuid[VK_UUID_SIZE]; // equal to VkPhysicalDeviceProperties::pipelineCacheUUID</pre><pre>};</pre><p>The hash of the pipeline cache data will allow us to validate the integrity of the data. To reduce the chance of an I/O error <em>actually</em> causing an integrity issue, we create a temporary file and write this header to the file followed by the pipeline cache data, then move the file to the target location using rename.3</p><p>When loading the pipeline cache, we read the header, read the data, validate the data read using dataSize and dataHash, then validate that the data can be safely passed to the driver by comparing the remaining fields with the properties of the device.<a href="https://zeux.io/2019/07/17/serializing-pipeline-cache/#fn:4">4</a></p><p>If the data is valid, vkCreatePipelineCache is called with the correct initial data. Crucially, if this call fails, it suggests that the driver implements additional checks that our logic didn’t detect on its own. So instead of proceeding without the pipeline cache, we create an empty pipeline cache in this case by calling vkCreatePipelineCache again with no initial data.</p><p>We also create the empty pipeline cache if the pipeline cache file was not found or our validation logic classified the data as unusable.</p><p>Note: because we incorporate driverVersion into the header, any driver update will cause pipeline cache to be rebuilt; we include this check because this completely eliminates issues where pipeline cache UUID doesn’t update even if it should — typically driverVersion is updated as part of build process, whereas UUID update is more manual. For applications that target desktop exclusively this can be too aggressive — in general desktop drivers are likely to be more well behaved with respect to handling pipeline cache validity so not all of this advice applies.</p><h3>Conclusion</h3><p>Unfortunately, Vulkan drivers are not always correct and don’t always follow the specification to the letter. Pipeline cache data is an especially fragile part of the Vulkan renderer because I/O is challenging to get right, and there’s often minimal-to-no integrity checks in the driver. However, with enough application-side validation, you <em>can</em> eliminate stability issues coming from the pipeline cache handling in practice — it just takes work.</p><ol><li>Which you can absolutely do these days! For example, <a href="https://github.com/mesa3d/mesa/blob/1d5ee315536d4563714b35004d9efc1bd6621f53/src/amd/vulkan/radv_pipeline_cache.c#L525">here’s an implementation of vkGetPipelineCacheData for radv</a>. ↩</li><li>The remainder of this article is based on the experience of continuously shipping <a href="https://corp.roblox.com/">Roblox</a> client on Android with Vulkan support and surviving through various Android OS updates, driver updates and in general dealing with both early and current Vulkan drivers from all major vendors. ↩</li><li>In theory rename is supposed to be atomic, but in practice the exact semantics and guarantees vary with the file system; hash is useful as a way to perform a robust comparison. ↩</li><li>Depending on the application you may want to also use different file names based on, for example, vendorID or driverABI; this is more interesting on desktop and less interesting on mobile. ↩</li></ol><p>Originally Published on: <a href="https://zeux.io/2019/07/17/serializing-pipeline-cache/">https://zeux.io/2019/07/17/serializing-pipeline-cache/</a></p><p>Arseny Kapoulkine has worked on game technology for the past decade. Having worked on rendering, physics simulation, language runtimes, multithreading and many other areas, he is still discovering exciting problems in game development that require low-level thinking. After helping ship many titles on PS3 including several FIFA games, he joined Roblox in 2012 and has been working on the in-house engine ever since, helping young game developers achieve their dreams.</p><p>Neither Roblox Corporation nor this blog endorses or supports any company or service. Also, no guarantees or promises are made regarding the accuracy, reliability or completeness of the information contained in this blog.</p><p>©2021 Roblox Corporation. Roblox, the Roblox logo and Powering Imagination are among our registered and unregistered trademarks in the U.S. and other countries.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=961d09416cda" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Future Is Bright: Updating Lighting Systems in Roblox]]></title>
            <link>https://medium.com/@zeuxcg/the-future-is-bright-updating-lighting-systems-in-roblox-27dcc85cc391?source=rss-8819be2d098a------2</link>
            <guid isPermaLink="false">https://medium.com/p/27dcc85cc391</guid>
            <category><![CDATA[voxel]]></category>
            <category><![CDATA[graphics]]></category>
            <category><![CDATA[game-development]]></category>
            <category><![CDATA[lighting]]></category>
            <dc:creator><![CDATA[Arseny Kapoulkine]]></dc:creator>
            <pubDate>Fri, 06 Nov 2020 21:27:26 GMT</pubDate>
            <atom:updated>2020-11-13T23:05:39.052Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rtUygbiLrKI1vGujeqObcA.jpeg" /></figure><p>In 2012, we implemented a new lighting system that we’ve used since across all of Roblox — it uses voxels and is updated incrementally (and lazily) by the CPU. It has served us well but as we started to explore ways for Roblox graphics to look much better, we realized the system needs an update. Lighting is a complicated system, so we needed to be careful when deciding on a course for our next-generation technology and consider all the possible tradeoffs and factors. Two alternative systems have been prototyped in 2017 to help guide our decision (which will be discussed further in this document), we’ll call them “voxels” and “shadow maps.” It’s important to understand how both of them work to understand the limitations. A <a href="https://roblox.github.io/future-is-bright/compare#summary">summary table</a> is included at the end of this article.</p><p>A note on this article’s screenshots: the layout of these is always such that the screenshot on the left is using the voxels system, and the screenshot on the right is using the shadow maps system.</p><h3>Implementation — Voxels</h3><p>This would be a significant evolution of our original voxel system. First, the world data is voxelized into a set of voxel grids — each grid is centered around the character and has increasing voxel sizes all the way from 1 stud to 16 studs (with 5 grids in total). Each voxel contains occupancy information (0–100% full). After this, we compute lighting data for each voxel in each grid, based on the occupancy in each voxel, and on the light sources / sunlight direction. All of the above happens on the GPU, as the CPU is not fast enough to update so many voxels at such a high density.</p><p>The system stores all data in voxels, so specifically for each voxel we have:</p><ul><li>Occupancy (multiple values describing how full each voxel is)</li><li>Skylight (how much of the sky is visible from the voxel)</li><li>Sun shadow (how much of the sun is occluded from the voxel)</li><li>Light object color/cone (the approximation of the color/cone of impact of local light sources on the voxel)</li></ul><p>This information is later used to compute the color of each pixel at a given resolution; screen resolution and voxel resolution can be controlled independently. Parts of voxel grid(s) can be updated each frame as lights/objects move.</p><h3>Implementation — Shadow Maps</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/0*qnBIN3LuqfH1G4DU" /></figure><p>The Shadow Maps method uses rasterization to compute most of the shadow effects and executes in three phases. First, for each shadow casting light we update a shadow map by rendering triangles of objects into a texture from the viewpoint of the light (you can think of this as casting a lot of rays from the light source into the scene and remembering the intersection results). Second, we build a spatial acceleration structure that we insert each visible light object into, that is essentially a frustum-shaped voxel grid (a.k.a. a froxel grid).</p><p>The grid covers the entire camera frustum, and within each froxel we record the list of all light objects that intersect it. Finally, when rendering the scene, for each pixel to compute the impact of all lights we look up the froxel our pixel is contained in, go over all lights, and for each light compute the impact of this light using the shadow maps built in step one to determine visibility.</p><p>The system stores all data in two structures:</p><ul><li>Shadow atlas (all shadow maps from visible lights, packed into one big texture)</li><li>Light grid (froxel grid that can efficiently resolve a point in the camera into a list of lights)</li></ul><p>The color of each pixel is computed dynamically and is not stored explicitly. Parts of the shadow atlas can be updated each frame as lights/objects move.</p><h3>Performance — Voxels</h3><p>The voxel technique is fundamentally more scalable — as long as we’re willing to degrade the output quality, we can reduce the number of voxel grids, and update fewer voxels each frame (leading to “light lag” as the light impact of the objects will update more slowly than the objects themselves) Voxels decouple all three sources of complexity: geometrical complexity, light complexity, and pixel count. Geometrical complexity only impacts voxelization cost, so putting more objects doesn’t make anything else slower. Light complexity only impacts light computation cost which does not depend on geometrical complexity <em>or</em> the number of pixels. Finally, computation of the final pixel color is constant-time in the number of voxels/lights/parts, so we can scale the resolution independently without increasing the lighting performance impact.</p><p>You can think of voxel worst-case performance as O(G)+O(L)+O(P), where G is # of triangles (geometry complexity), L is # of lights, P is # of pixels.</p><p>Unfortunately, the peak performance of voxels is suboptimal because the number of voxels scales as N3, and GPUs aren’t ideally suited for the update techniques we have to use to keep performance manageable. Given enough research into GPU computing we should be able to offset the performance loss, but the current baseline cost can be pretty high.</p><h3>Performance — Shadow Maps</h3><p>For shadow maps, the technique is more GPU-friendly as it’s designed around rasterization which is what GPUs excel at. The shadow atlas update cost can be partially mitigated by caching/delaying updates (leading to “shadow lag”). For the shadow maps that we do update, optimizing the efficiency of geometry submission (including mesh level of detail) reduces the cost.</p><p>However, the baseline cost of shadow update in complex scenes is significant since it depends on both the amount of geometric detail and the amount of shadow-casting lights. For a moving light with a large radius inside the building, we have to re-render the entire building each frame to update the shadow information for this light; many moving shadow casting lights in a building result in poor performance (we can choose to update only some lights each frame, resulting in visual artifacts).</p><p>Additionally, the technique does not provide a way to separate resolution from the light count — for each pixel we have to recompute the influence of all lights that cover this pixel. This step also can not be cached, leading to performance issues on high resolutions in densely lit scenes — 20 overlapping lights in a room at 4K resolution might require 160M light evaluations).</p><p>You can think of shadow map worst-case performance as O(GL)+O(LP), where G is # of triangles (geometry complexity), L is # of lights, P is # of pixels.</p><h3>Performance — Evaluation</h3><p>To put the theoretical results above in a more practical environment, here are results from some of the levels we have tested. These results use <em>existing</em> implementations — that have not been necessarily tuned for performance — and also assume <em>no</em> caching whatsoever, as if every light source/part in the world were moving.</p><h4>Paris (sun shadows, very few non-shadow-casting lights)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/0*RT0aI5pj1qvxj_gy" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/0*hJ4z1Oo4D3lrvsng" /></figure><ul><li>Voxels: 6 ms shadow update, 1.5 ms scene render</li><li>Shadow maps: 1 ms shadow update, 2.4 ms scene render</li><li>Baseline voxel shadow computation cost is larger since it’s not GPU-friendly</li></ul><h4>Caves (many shadow-casting lights)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/0*SqQEqP2p2N-h4lGB" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/0*5aZulr5dpF_Vljxf" /></figure><ul><li>Voxels: 7 ms shadow update, 0.9 ms scene render</li><li>Shadow maps: 10 ms shadow update, 2.1 ms scene render</li><li>A lot of geometry and moving lights make shadow map update expensive</li></ul><h4>Western (many shadow-casting lights)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/0*E01BDQpYeox4D6KC" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/0*_2beASWMeZTyaiTc" /></figure><ul><li>Voxels: 8 ms shadow update, 1 ms scene render</li><li>Shadow maps: 15 ms shadow update, 2.5 ms scene render</li><li>With moving light sources and large # of triangles, shadow map update becomes expensive</li></ul><h4>Lights (1,000 non-shadow-casting lights)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/0*9wHkaVLKQ8Ok-M8G" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/0*UwPQAwqd4oPdYVfT" /></figure><ul><li>Voxels: 20 ms light update, 0.5 ms scene render</li><li>Shadow maps: 0.5 ms light update, 5 ms scene render</li><li>Cumulative volume of light-voxel overlap in this case makes voxel light update slow. Additionally, you can see that the approximation for “one light within each voxel” completely breaks down on this level for voxels that are in the near cascade.</li></ul><h3>Performance — conclusion</h3><p>In general shadow maps seem to scale nicely to the workload, however two big concerning areas are:</p><ol><li>Per-pixel cost grows as the resolution keeps bigger, making this solution only practical on average (1080p) resolutions; going beyond 1080p requires a <em>very</em> good GPU.</li><li>Shadow cost grows very quickly when many shadow-casting dynamic lights are close to complex geometry. This can be offset by better culling but remains a fundamental problem.</li></ol><p>In contrast, voxel performance depends on the level contents much less, but has a much higher baseline. This can be offset by better GPU algorithms and by reducing the # of voxels.</p><h3>Memory requirements</h3><p>Memory requirements for shadow maps and voxels depend on the target quality.</p><p>For voxels, we store several textures for each voxel cascade; so the cumulative size depends on the number of cascades and the size of each cascade. We currently use 4 cascades (with voxel sizes 1..8) of 128x64x128 voxels in each, which adds up to 128 MB of VRAM used; we might want to have 2 more cascades (0.5 voxels and 16 voxels) or reconfigure the cascades, which would inflate the requirements up to 192 MB. We can reduce the number of cascades (removing some close cascades) on systems that are memory constrained, so realistically the minimum memory impact could be around 64 MB with two cascades (4..8) and around 96 MB with three cascades (4..16) which would allow for ~1k stud range with light/shadow data.</p><p>For shadow maps, we store the shadow map atlas and the froxel grid for light lookup; the latter is somewhat resolution-dependent, and the shadow map atlas can be reduced in size if we want to reduce the shadow quality to improve memory/performance; the current setup uses 73 MB VRAM, the bulk of which (64 MB) is the shadow map atlas; we can reduce this and limit the number of shadowed lights or shadow quality. We are also considering some variants of the shadow maps that would require more storage to support translucency, which might mean that we’d need more memory for shadow maps (e.g. up to 130 MB or more). The minimum memory impact of the system can probably be reached if we reduce the shadow map atlas size and use a simpler variant, which would cost us around 25 MB total.</p><p>For comparison, the current voxel lighting system has two modes — high quality (PC) and low quality (Mobile); PC variant takes ~40 MB (24 MB RAM, 16 MB VRAM); mobile variant takes ~11 MB (6 MB RAM, 5 MB VRAM).</p><p>In essence, both techniques are reasonably close in terms of their memory impact, but shadow maps are somewhat more scalable for the same light/shadow range.</p><h3>Mobile Compatibility</h3><p>Our audience is getting increasingly mobile; this means that we have to think of mobile when comparing these implementations. On a reasonably high end mobile device we should have enough API capabilities to implement both techniques, but of course they may or may not be practical from the memory and performance perspective.</p><p>Our existing voxel lighting system fits mobile really well — it supports many complex lighting features (light shadows, skylight, etc.) and runs most of the complex computations on CPU, thus having minimal requirements for the GPU performance and feature set. Since we still will have to maintain this system for the low end mobile and desktop market for the foreseeable future, we have several options in terms of mobile support:</p><ul><li>Keep the existing system on mobile, and the new system will be desktop/console only. This means that a large segment of the user base will not have access to the new system.</li><li>Improve the existing system in a modest way for high-end mobile (slightly reduced voxel size, enhanced voxel representation), and use the new system for desktop/console.</li><li>Keep the existing system on low-end mobile, find a way to scale down the new system so that it can work on high-end mobile.</li></ul><p>In all of these cases we have to answer the question of content compatibility — one of the fundamental promises of our platform is “build content once and run it anywhere”, which means that we have to understand how well can content built for the new system look when using the old system, or some enhanced variant thereof.</p><p>This requires further investigation; on the outset it seems likely that new voxel solution is better in that it has a continuum of quality/behavior all the way from the existing voxel system to the highest quality of the new voxel system, whereas shadow maps represent a more abrupt quality transition but have a higher chance of working in a limited capacity on high-end mobile devices.</p><h3>Quality — Lights</h3><p>Shadow map solution provides the ground truth in terms of simulating the lights — on the screenshot with 1000 lights above that is made with shadow maps you can see that the specular highlights — modeled with a BRDF that is getting us basically light reflections — are perfectly preserved.</p><p>Voxel solution is fundamentally worse in that it approximates light influence at each voxel as if it just comes from one light, so in general you see the specular quality suffer; for example, here’s a screenshot with two lights (green &amp; red) over a highly reflective surface:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/0*awS88JG9zv6tusGe" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/0*K2pfZH3UedvfBV8J" /></figure><p>As you can see, while in voxels away from the center of the camera you see distinct colors, in the area with specular highlights the colors merge in 1:1 ratio, producing yellow specular highlights even though there are no yellow lights in the scene. The shadow maps solution, on the other hand, models the light transport precisely.</p><p>In some cases the approximation we use produces results that are very unconvincing, although we could improve this to some degree:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/0*kpp8GJzgmcMGDySx" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/0*lTKfRlA1VsIku0vB" /></figure><p>You can see curved, elongated, and miscolored specular highlights, and a few voxels below one of the parts are missing light information. The same screenshot with shadow maps provides a much better result.</p><h3>Quality — Shadows</h3><p>In general a defining quality of shadow maps is fidelity, and a defining quality of voxel shadows is softness. Shadow maps produce crisp shadows, with minimum representable detail sufficiently high to render out a convincing character shadow. Our voxel shadow algorithm, on the other hand, is very good at producing really soft shadows, but since the smallest voxel size is 1 voxel, the shadows from small parts either don’t register at all, or have vastly incorrect shapes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*vajrX9DPo0gGTaoPjgtw5Q.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*TWgEUgCJmhdMVCn0UWKHMg.jpeg" /></figure><p>For this reason we currently (in the production lighting system) use a shadow map variant to render shadows from the characters — this, however, is a duct tape solution in the sense that it only applies to the sun casting shadows from character, other light sources and/or objects aren’t affected.</p><p>Additionally, a key technique that makes voxels work reasonably fast is using voxel cascades; however, this means that occupancy data gets coarser the further away you are from the point of the scene interest; this means that shadow quality also gets worse as the distance between the shadow caster and the shadow receiver increases:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*goJtFk5yQFU_WpxDB65fqQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*y5q_CENMOeJ8maFrZ79wQg.jpeg" /></figure><p>In this case the size of the smallest voxel would have been enough to render a high quality shadow from the bridge, but the bridge is far away from the water surface so the bridge voxels are too coarse even if the voxels near the water surface are reasonably fine.</p><h3>Quality — Skylight</h3><p>An important feature that we support in the voxel pipeline is computing the skylight factor: how much of the sky is visible from the current voxel? This is used to blend between outdoor and indoor lighting conditions and is very effective at enhancing the lighting quality. In this level, it should be much brighter outside the house than it is inside the house, even in the areas that are in the shadow of the house. Our voxel solution computes the skylight factor and reproduces this well, however the shadow map version lacks the skylight factor which makes the picture dull.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*FDwtjSGyScyovCeqqiy6eg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*YpBdqac-ePg-nMEbpJQzLQ.jpeg" /></figure><h3>Quality — Geometrical Fidelity</h3><p>It’s worth noting the fundamental differences in geometry representation between voxels and shadow maps.</p><p>Voxels assume that all objects that the lighting engine supports can be “voxelized” — that is, for each voxel in the world there’s a fast way to compute the volume of intersection between that object and the voxel. This is analytically computable for primitive shapes, but complex objects like CSGs and MeshParts present a significant challenge in this area. Currently we rely on a crude convex decomposition and a set of hacks to voxelize these efficiently, that often result in visible artifacts. Shadow maps, on the other hand, use the same polygonal representation we use for rendering, thus they are able to represent the shapes of all objects perfectly:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*RvtIf_sWoOWowM5gbH1hjA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*_jInSn3UDlP3ep_TceXT8A.jpeg" /></figure><h3>Quality — Light Leaks</h3><p>While the shape of the shadow is important, what perhaps is even more important is that pixels that should be completely invisible from the light source point of view are treated as such. When various approximations violate this, you get what’s called light leaking — visible strips of light, that are particularly problematic in high-contrast environments, such as being inside a building with bright sun outside. Here’s an example of a light leak:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*m2VI73sKy1KmvL8XFB0FNw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*X5odoxiKlmHsYmij3ggklQ.jpeg" /></figure><p>The rough shape of the light cast through the window isn’t too objectionable here, but what <em>is</em> objectionable is the thin lit part of the floor right next to the wall. Shadow maps preserve the light occlusion much better.</p><p>For voxels there are multiple sources of leaking; we try to mitigate some of that by storing anisotropic occupancy — we store 3 values per voxel, signifying “how much matter is in the voxel projection along the axis A” for axes X/Y/Z; unfortunately, while this helps thin parts cast shadows regardless of their thickness, it can’t eliminate all sources of leaking. The only way for a part to guarantee that the light is completely blocked off is to make it twice as thick as a voxel is, so 2 studs. Additionally, leaking increases with the voxel size, which means that on lower quality levels and/or far away leaking is more pronounced.</p><p>Shadow maps aren’t completely leak-proof but leaking is significantly less of a problem. With our shadow map solution, a part that’s 0.4 studs thick will have no visible light leaking (a part that’s 0.2 studs thick can still leak a bit of light, which might be something we can improve in the future).</p><h3>Quality — Conclusion</h3><p>Overall, shadow maps excel at most quality aspects; the only significant area of improvement is in figuring out a method of computing the skylight factor (it’s possible that this requires a hybrid technique that uses voxelization for skylight — which introduces some problems of the voxel pipeline — or maybe there are alternative solutions to this problem). It would also be nice to be able to support very soft shadows with the same algorithm if the developer so desires, which might be achievable with some extensions to the shadow map algorithm.</p><p>Voxels have reasonable quality, but compared to shadow maps they face many issues, particularly with shadow fidelity and specular highlights; we’ll have to address both issues in some way to be able to ship voxel lighting because the current form of voxel lighting can’t deliver good looking player shadows, and using our current shadow solution means we still don’t get player shadows from light sources other than the sun, which seems incompatible with the future vision.</p><h3>Vision — Translucency</h3><p>Rendering shadows is a reasonably well understood problem for opaque objects; however, translucent objects provide a new challenge — now light impact has to be diminished gradually as the light rays travel through transparent objects. Since our voxel solution works by propagating light through voxel cells using their occupancy value, it’s not too hard to support translucent shadow casters, which can give us low frequency (soft) shadows from particle effects and other translucent objects in the scene, including self-shadowing for the particle effects themselves:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*D_HjAZYsEi6zK-FJQL5YXA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*vJtoggdqzOCKI9mjtBqwHg.jpeg" /></figure><p>Here’s a video of this effect in motion:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FgJbhTBubWxw%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DgJbhTBubWxw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FgJbhTBubWxw%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5280cb43509345410d3e30f1e6376c35/href">https://medium.com/media/5280cb43509345410d3e30f1e6376c35/href</a></iframe><p>For shadow maps this is not currently supported; this means that if we want to support particles or other transparent objects casting shadows, we may not be able to do this with shadow maps. There is some research for alternative shadow map representations that might serve this use case, but it remains to be seen how effective that is.</p><h3>Vision — Vegetation</h3><p>While shadow maps can’t represent translucency very well, what they can do is represent small features of objects (such as vegetation) regardless of whether they are modeled with geometry or with textures. Voxels aren’t small enough to serve this use case (and also it’s not easy to access the texture information since it requires precisely modeling the surface of the mesh instead of the volume). It seems unlikely that we can ever get good looking shadows from vegetation with voxels, whereas shadow maps can support this use case even with existing content, as shown on this screenshot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*e3Wy-VGkkHNnSQUN_dr15Q.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*V1XGTjQu807N5ttm-FRMhg.jpeg" /></figure><h3><strong>Vision — Self-illumination</strong></h3><p>Due to how voxels are implemented, it’s relatively simple to inject light from light sources of arbitrary shape and quantity into the grid without affecting other parts of the pipeline performance-wise; while you can add many light sources to the shadow maps, making lights with custom shapes has architectural and performance penalties. Specifically, what is much easier to do with voxels compared to shadow maps is true self-illumination: currently we support Neon material that “emits light”, but it doesn’t actually inject light onto other objects near it (which can be thought of as a form of global illumination).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/0*3D2xyGCPWELYLsOg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/0*wHDaqJrQtuLNfIHl" /></figure><p>As a developer you can simulate this by adding light sources, but it would be nice to support this automatically; shadow maps make it challenging to model this for all types of shapes that we have, but voxels by necessity support voxelization of all shapes and thus supporting light emission from self-illuminating parts is pretty straightforward.</p><h3>Vision — Global Illumination</h3><p>One big area that we see real-time engines increasingly moving towards solving is global illumination (or GI), which means computing secondary light effects, such as light from a lamp bouncing two times off the walls to provide extra illumination for areas that aren’t directly visible to the light.</p><p>GI in Roblox is unprecedentedly complicated — most solutions to GI sacrifice at least one of things: dynamic lights, dynamic geometry, real-time performance, large scenes, or good results without tuning. But we need all of these. Interestingly, skylight factor and self-illumination, mentioned above, are all parts of a general GI solution.</p><p>It’s not clear which GI solutions will be practical within the extremely lax content constraints we have; so far voxel-based GI seems more promising than other approaches when we need to consider all of the above.</p><p>Of course, having voxel-based GI doesn’t really enforce that direct lighting is computed using voxels. Most research around voxel-based GI today implies using shadow maps to compute direct light visibility, and enhance the results using voxels.</p><p>We have done early experiments in this area but it’s too early to conclude whether or when this can be possible in Roblox.</p><h3>Alternative Approaches</h3><p>There are some other approaches that are potentially interesting and are worth covering in this document:</p><ul><li>Deferred shading: A natural question is whether we should use deferred shading, but it’s important to note that this question is not very interesting for the considerations listed in this document. If we were to use deferred shading, we would still need the same shadow mapping technology, and due to the abundance of transparent objects at Roblox we would still need to support a forward lit pipeline for them, so it’s really a question of “do we also need deferred shading to try to accelerate light processing for unshadowed lights?” The only major advantage deferred shading has over our setup is better utilization of GPU during shading pass, which is worth looking into but does not seem a deal-breaker.</li><li>Distance field tracing (aka voxel cone tracing): One significant issue with voxels as we implement them is that they give you a coarse volume-based scene representation <em>and</em> serve as an atomic unit of computation for light/shadow information. If we only keep the scene representation part, and convert it from volume-based to distance-based — meaning, if in every voxel we record the minimum distance from the voxel center to the scene — then we can compute shadows by tracing through this voxel grid for each pixel we want to compute data for. This may be a balance between shadow maps and voxels in terms of computational efficiency and quality, but it remains to be seen as to how efficient this can be. This approach is starting to get used in some games using Unreal Engine 4 (as they have a built-in distance field shadow solution), and there are several games that use this technology for parts of their lighting pipeline.</li></ul><p>Finally, there is definitely room for some hybrid approaches. Effectively, we have many small mostly independent parts of the system — shadow map atlas, froxel grid for light lookup, cascaded voxel grids with occupancy, cascaded voxel grids with shadow data, cascaded voxel grids with light data, and potentially cascaded distance fields — it’s reasonable to assume that there is some hybrid solution that can use different techniques for solving different problems.</p><p>For example, the main quality concern for shadow maps is their inability to implement skylight; since skylight can be implemented both by using a voxel grid with occupancy &amp; shadow data, and using distance fields, it’s reasonable to try to augment shadow maps with one of these solution to solve the skylight issue. This idea is interesting, but presents several challenges:</p><ul><li>By using a solution that has quality concerns as part of the hybrid, we’d inherit these issues in the resulting solution. For example, voxels are prone to light leaking and inaccurate geometrical representation, and both will be an issue even if we compute just skylight data from the voxel representation.</li><li>By using two solutions with two different source representations, we pay extra memory and performance cost. For example, maintaining the voxelization pipeline in parallel with shadow maps implies we’ll have to pay the cost of both voxelization and rasterization of all shadow casting geometry.</li><li>Supporting each object in multiple techniques requires extra engineering/maintenance work — a hybrid solution will take more time to develop, and more time to maintain in the future.</li></ul><h3>Summary</h3><p>Based on the analysis above, we can summarize the tradeoffs that these two solutions provide. Cells using italic suggest that it might be possible to improve this area with further research. For the purpose of this table, Terrible &lt; Poor &lt; Okay &lt; Good &lt; Excellent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*CN7VJfcwFvswyFBFHxBUOA.png" /></figure><h3>Outcome</h3><p>After evaluating the tradeoffs, it was becoming clear that while voxels are fantastic at modeling indirect lighting, they aren’t optimal for modeling direct lighting. Some advantages of the voxel based solutions for direct lighting looked compelling but it was hard to offset a very high baseline quality and performance cost — especially given the wide range of platforms and hardware we target.</p><p>Because of the performance and quality disparity we’ve decided to start by building a system around shadow maps / forward+ shading to solve the direct lighting problem; this project became known as “Future Is Bright” and was released in production in 2019 for sunlight, and is on track to release for all other types of light sources in 2020.</p><p>However, as we look forward and consider how we’re going to upgrade our current skylight solution (that still uses the original voxel lighting system) and how we’re going to solve global illumination in general given a fully dynamic environment, it’s becoming increasingly likely that a hybrid solution, where rasterization / shadow maps are used for high quality direct lighting, and voxels are used for lower frequency effects such as skylight, global illumination and localized ambient lighting, is going to be where we will land in the future. Time will tell!</p><p>Arseny Kapoulkine has worked on game technology for the past decade. Having worked on rendering, physics simulation, language runtimes, multithreading and many other areas, he is still discovering exciting problems in game development that require low-level thinking. After helping ship many titles on PS3 including several FIFA games, he joined Roblox in 2012 and has been working on the in-house engine ever since, helping young game developers achieve their dreams.</p><p>This article was originally published <a href="https://roblox.github.io/future-is-bright/compare">here</a>.</p><p>Neither Roblox Corporation nor this blog endorses or supports any company or service. Also, no guarantees or promises are made regarding the accuracy, reliability or completeness of the information contained in this blog.</p><p>©2020 Roblox Corporation. Roblox, the Roblox logo and Powering Imagination are among our registered and unregistered trademarks in the U.S. and other countries.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27dcc85cc391" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Faster and Leaner on Mobile: Optimizing Roblox with Vulkan]]></title>
            <link>https://medium.com/@zeuxcg/getting-faster-and-leaner-on-mobile-optimizing-roblox-with-vulkan-7c79950fef69?source=rss-8819be2d098a------2</link>
            <guid isPermaLink="false">https://medium.com/p/7c79950fef69</guid>
            <category><![CDATA[vulkan]]></category>
            <category><![CDATA[mobile]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[roblox]]></category>
            <dc:creator><![CDATA[Arseny Kapoulkine]]></dc:creator>
            <pubDate>Wed, 10 Jun 2020 17:08:16 GMT</pubDate>
            <atom:updated>2020-06-10T17:19:25.219Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d9kE0eKO6orE_JIx08juyA.jpeg" /></figure><p>Roblox runs on millions of Android devices, and it’s important for us to be able to achieve good performance where possible. Rendering performance in particular is challenging — we target many different hardware architectures, many different types of devices on the power/performance curve, and of course many versions of wildly different drivers.</p><p>In 2018, we’ve launched our Vulkan renderer and since then it has been steadily growing in market share compared to our OpenGL renderer, with ~40% of our Android userbase using Vulkan today.</p><p>After the initial launch, with the growing popularity of the Vulkan backend we had to invest a lot of time in analyzing performance and improving it. Doing this required a lot of investigation and experimentation — with minimal performance-focused optimization or code available at the time, we had to try different approaches to each particular problem and find what works best for us.</p><p>At Reboot Develop Red 2019, we gave a combined talk with ARM to cover the performance optimization challenge from two different angles:</p><ul><li>There’s now a growing collection of performance-focused sample Vulkan code that was developed by ARM and donated to Khronos, which you can read about here: <a href="https://www.khronos.org/blog/vulkan-releases-unified-samples-repository">https://www.khronos.org/blog/vulkan-releases-unified-samples-repository</a>. Joe Rozek from ARM talked about the general structure of the samples and the framework used.</li><li>To help other developers optimize Vulkan code and give some insights into the problems and solutions we encountered at Roblox, I talked about several concrete problems we encountered while optimizing our renderer, tradeoffs we had to examine and solutions we implemented.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FhPW5ckkqiqA%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DhPW5ckkqiqA&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FhPW5ckkqiqA%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/7772654968460e8f0b91beb234139c4e/href">https://medium.com/media/7772654968460e8f0b91beb234139c4e/href</a></iframe><p>At GDC 2020, we collaborated with ARM again to cover the aspects of the subject that were omitted from Reboot talk, focusing on the GPU performance optimization for ARM and covering multithreaded command generation in more detail; this talk can be viewed at <a href="https://www.gdcvault.com/play/1026767/Optimizing-Roblox-Vulkan-Best-Practices">GDC Vault</a></p><p><a href="https://www.gdcvault.com/play/1026767/Optimizing-Roblox-Vulkan-Best-Practices">Optimizing Roblox: Vulkan Best Practices for Mobile Developers (Presented by Arm &amp; Roblox)</a></p><p>The video for the talk is posted on <a href="https://www.youtube.com/watch?v=hPW5ckkqiqA">YouTube</a> , and you can download the slides here (<a href="https://zeux.io/data/reboot2019.pdf">https://zeux.io/data/reboot2019.pdf</a>).</p><p>Arseny Kapoulkine has worked on game technology for the past decade. Having worked on rendering, physics simulation, language runtimes, multithreading and many other areas, he is still discovering exciting problems in game development that require low-level thinking. After helping ship many titles on PS3 including several FIFA games, he joined Roblox in 2012 and has been working on the in-house engine ever since, helping young game developers achieve their dreams.</p><p>Neither Roblox Corporation nor this blog endorses or supports any company or service. Also, no guarantees or promises are made regarding the accuracy, reliability or completeness of the information contained in this blog.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7c79950fef69" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[3 Years of Metal]]></title>
            <link>https://medium.com/@zeuxcg/3-years-of-metal-22d74969a21?source=rss-8819be2d098a------2</link>
            <guid isPermaLink="false">https://medium.com/p/22d74969a21</guid>
            <category><![CDATA[graphics]]></category>
            <category><![CDATA[roblox]]></category>
            <category><![CDATA[rendering]]></category>
            <category><![CDATA[mobile]]></category>
            <dc:creator><![CDATA[Arseny Kapoulkine]]></dc:creator>
            <pubDate>Wed, 20 May 2020 17:52:45 GMT</pubDate>
            <atom:updated>2020-05-20T17:52:45.199Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s0c-KZBIq0y9j-AgbcPBpw.jpeg" /></figure><p>Three years ago, we ported our renderer to Metal. It didn’t take much time, it was a blast and it worked really well on iOS. So we wrote an article going through how we made our decision and how it ended up (spoilers: really well!). Most of that original retrospective still applies, but today Metal is in better shape than ever — so we’ve decided to republish it with our three-year update.</p><p>So let’s wind the clock back, pretend it’s December 2016 and we have just shipped a version of our Metal renderer on iOS.</p><h3>Why Metal?</h3><p>When Apple announced Metal at WWDC in 2014, my initial reaction was to ignore it. It was only available on the newest hardware which most of our users didn’t have, and while Apple said it solved CPU performance issues, optimizing for the smallest market would mean the gap between the fastest and slowest devices would grow even more. At the time we were running OpenGL ES 2 only on Apple, and also starting to port to Android.</p><p>Fast forward two and a half years, here’s how the Metal market share looks for our users:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/333/0*RvS2tpX62paMvrfk" /></figure><p>This is much more appealing than it used to be. It is still the case that implementing Metal does not help the oldest devices, but the GL market on iOS keeps shrinking, and the content that we run on these old devices is frequently different from the content that runs on newest devices, so it makes sense to dedicate some effort to make it faster. Given that your iOS Metal code will run on Mac with very few changes, it could make sense to use it on Mac as well even if you are mobile-focused (we currently only ship Metal builds on iOS).</p><p>I think it is worthwhile to analyze the market share in a bit more detail. On iOS, we support Metal for iOS 8.3+; while there are some users who can’t run Metal because of OS version restrictions, most of the 25% who still run GL are simply using older devices that have SGX hardware. They also don’t have any OpenGL ES 3 features, and we’re content with running a lower-end rendering path there (although we’d love all devices to go Metal — fortunately the GL/Metal split will only improve). On Mac, Metal API is newer and the OS plays a pretty significant part — you have to use OSX 10.11+ to use Metal and half of our users simply have a more dated OS — it’s less about the hardware and more about the software (95% of our Mac users run OpenGL 3.2+).</p><p>So given the market share, we still have options that do not involve porting to Metal. One of them is to just use <a href="https://moltengl.com/moltengl/">MoltenGL</a>, which would use the OpenGL code we already have, but supposedly be faster; another is to port to Vulkan (to get better performance on PC, and eventually Android) and use <a href="https://moltengl.com/moltenvk/">MoltenVK</a>. I have briefly evaluated MoltenGL and was not too thrilled with the results — it took some effort to make our code run at all, and while performance was a bit better compared to stock OpenGL I was hoping for more. As for MoltenVK, I think it is misguided to try to implement one low-level API as a layer above another one — you’re bound to get impedance mismatch that will result in suboptimal performance — maybe it will be better than the high-level API you used to use, but it’s unlikely to be as fast as possible, which supposedly is why you’re choosing a low-level API to begin with! One other important aspect is Metal implementation is much simpler than a Vulkan one — more on that later — so in some sense I’d prefer a Metal -&gt; Vulkan wrapper instead of a Vulkan -&gt; Metal.</p><p>It is also worth noting that apparently on iOS 10 on newest iPhones there is no GL driver — GL is implemented on top of Metal. Which means using OpenGL is only really saving you a bit of development effort — not that much, considering that the promise of “write once, run anywhere” that OpenGL has does not really work out on mobile.</p><h3>Porting</h3><p>Overall, porting to Metal was a breeze. We have a lot of experience working with different graphics APIs, ranging from high level APIs like Direct3D 9/11 to low level APIs like PS4 GNM. This gives a unique advantage of being able to comfortably use an API like Metal that is simultaneously reasonably high level but also leaves some tasks like CPU-GPU synchronization for the app developer to do.</p><p>The only hurdle really was getting our shaders to compile — once that was done and it was time to write the code it became apparent that the API is so simple and self-explanatory that the code practically wrote itself. I got the port that rendered most things in a suboptimal fashion running in about 10 hours in a single day, and spent two more weeks cleaning up the code, fixing validation issues, profiling and optimizing and doing general polish. To get an API implementation in this time frame speaks volumes of the quality of the API and the toolset. I believe there are several aspects that contribute:</p><ul><li>You can develop the code incrementally, with good feedback at every stage. Our code started by ignoring all CPU-GPU synchronization, being really suboptimal about certain parts of state setup, using built-in reference tracking for resources and never running CPU and GPU in parallel to avoid running into issues; the optimization/polish phase then converted this into something we could ship, never losing the ability to render in the process.</li><li>The tools are there for you, they work and they work well. This is not as much of a surprise for people who are used to Direct3D 11 — but this is the first time on mobile where I had a CPU profiler, a GPU profiler, a GPU debugger and a GPU API validation layer that all worked well in tandem, catching most issues during development and helping optimize the code.</li><li>While the API is somewhat lower level than Direct3D 11, and it leaves some key low-level decisions to the developer (such as the render pass configuration or the synchronization), it still uses a traditional resource model where each resource has certain “usage flags” it has been created with but does not require pipeline barriers or layout transitions, and a traditional binding model where each shader stage has several slots you can freely assign resources to. Both of these are familiar, easy to understand and require very limited amount of code to get going fast.</li></ul><p>One other thing that helped is that our API interface was ready for Metal-like APIs — it is very lean but it exposes enough detail (such as render passes) to be able to easily write a performant implementation. At no point in our implementation did I need to save/restore state (many API interfaces suffer from this, particularly due to treating render target setup as state changes and resources/state binding persisting through that) or make complicated decisions about resource lifetime/synchronization. About the only “complicated” piece of code needed to render is one that creates the render pipeline state by hashing bits that are needed to create one — pipeline state objects are not part of our API abstraction. Even that is pretty straightforward and fast. I will write more about our API interface in a separate post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/0*qjuG5dJIYPfh_rjb" /></figure><p>So, a week to get the shaders compiling, two weeks to get a polished optimized implementation<a href="https://zeux.io/2016/12/01/metal-retrospective/#fn:2">1</a> — what are the results? The results are great — Metal absolutely delivers on the performance promise. For one, the single threaded dispatch performance is noticeably better than with OpenGL (shrinking the draw dispatch part of our render frame by 2–3x depending on the workload), and this is given that our OpenGL implementation is pretty well tuned in terms of reducing redundant state setup and playing nice with the driver by using fast paths. But it does not stop there — multithreading in Metal is trivial to utilize provided that your rendering code is ready for it. We haven’t switched to threaded draw dispatch yet but are already converting some other parts that prepare resources to happen off the render thread, which, unlike with OpenGL, is pretty much effortless.</p><p>Beyond that, Metal allows us to fix some other performance issues by giving easily accessible and reliable tools. One of the central parts of our rendering code is the system that computes lighting data on the CPU in world space and uploads it to regions of a 3D texture (which we have to emulate on OpenGL ES 2 hardware). The updates are partial so we can’t duplicate the entire texture and have to rely on however the driver implements glTexSubImage3D. At one point we tried to use PBO to improve update performance but faced significant stability issues across the board, both on Android and iOS. On Metal there are two builtin ways to upload a region — MTLTexture.replaceRegion that you can use if GPU is not currently reading the texture, or MTLBlitCommandEncoder (copyFromBufferToTexture or copyFromTextureToTexture) that can upload the region asynchronously just in time for GPU to start using the texture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/355/0*gTXTWcF8IlMatlR8" /></figure><p>Both of these methods were slower than I’d like — the first one wasn’t really available since we had to support efficient partial updates, and it worked purely on CPU using what looked like a very slow address translation implementation. The second one worked but seemed to use a series of 2D blits to fill the 3D texture which were both pretty expensive to set up commands for on the CPU side and also had a very high GPU overhead for whatever reason. If this were OpenGL it would be over — in fact, the performance of these two methods roughly matched the observed cost of a similar update in OpenGL. Fortunately, this being Metal, it has easy access to compute shaders — and a super simple compute shader gave us the capability to do a buffer -&gt; 3D texture upload that was very fast on CPU and GPU and basically solved our performance problems in this part of the code for good<a href="https://zeux.io/2016/12/01/metal-retrospective/#fn:1">2</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zj4qFGUkJo8WO9GvNYpb-w.png" /></figure><p>As a final general comment, maintaining Metal code is pretty much effortless as well — all extra features we had to add so far were easier to add there than on any other API we support, and I expect this trend to continue. There was a bit of a concern that adding one more API would require constant maintenance, but compared to OpenGL this does not really require much work; in fact, since we won’t have to support OpenGL ES 3 on iOS any more, this means we can simplify some OpenGL code we have as well.</p><h3>Stability</h3><p>Today on iOS Metal feels very stable. I am not sure what the situation was like at launch in 2014, or what it is like on Mac today, but both the drivers and the tools for iOS feel pretty solid.</p><p>We had one driver issue on iOS 10 that had to do with loading shaders compiled with Xcode 7 (which we fixed by switching to Xcode 8), and one driver crash on iOS 9 that turned out to be a result of misusing nextDrawable API. Other than that we haven’t seen any behavioral bugs or any crashes — for a relatively new API Metal has been very solid across the board.</p><p>Additionally, the tools you get with Metal are varied and rich; specifically, you can use:</p><ul><li>A pretty comprehensive validation layer that will identify common issues in using the API. It’s basically like Direct3D debug — which is familiar for Direct3D but pretty much unheard of in OpenGL land (in theory ARB_debug_callback is supposed to solve this, in practice it’s mostly unavailable and when it is, not terribly helpful)</li><li>A working GPU debugger which shows all commands you have dispatched along with their state, the render target contents, the texture contents, etc. I don’t know if it has a functioning shader debugger because I never needed that, and the buffer inspection could be a bit easier, but it mostly does the job.</li><li>A working GPU profiler which shows per-pass performance stats (time, bandwidth) and also per-shader execution time. Since the GPU is a tiler you can’t really expect per-drawcall timings, of course. Having this level of visibility — especially considering the complete lack of any GPU timing information in graphics APIs on iOS — is great.</li><li>A working CPU/GPU timeline trace (Metal System Trace) which shows the scheduling of CPU and GPU rendering workload, similar to GPUView but actually easy to use, modulo some UI idiosyncrasies.</li><li>An offline shader compiler that validates your shader syntax, occasionally gives you useful warnings, converts your shader into a binary blob that’s pretty fast to load at runtime and additionally reasonably well optimized beforehand, reducing the load times since driver compiler can be faster.</li></ul><p>If you come from Direct3D or console world, you may take every single one of these for granted — trust me, in OpenGL every single one of these is unusual and is met with excitement, especially on mobile where you are used to dealing with occasionally broken drivers, no validation, no GPU debugger, no GPU profiler that is helpful, no ability to gather GPU scheduling data and being forced to work with a text-based shader language that each vendor has a slightly different parser for.</p><p>Metal is a great API to both write code for, and ship applications with. It’s easy to use, it has predictable performance, it has robust drivers and solid toolset. It beats OpenGL in every single aspect except for portability, but the reality with OpenGL is that you really only should have used it on three platforms (iOS, Android and Mac), and two of those now support Metal; additionally, the portability promise of OpenGL is largely not fulfilled as the code that you write on one platform very frequently ends up not working on another for different reasons.</p><p>If you are using a third-party engine like Unity or UE4, Metal is already supported there; if you aren’t and you enjoy graphics programming or care deeply about performance and take iOS or Mac seriously, I strongly urge you to give Metal a try. You will not be disappointed.</p><h3>Metal Now</h3><p>The biggest changes that happened to Metal from our point of view in the last three years are about adoption at a massive scale.</p><p>Three years ago, a quarter of devices had to use OpenGL. Today, for our audience, this number is ~2% — which means our OpenGL backend barely matters anymore. We still maintain it but this will not continue for long.</p><p>The drivers are also better than ever — generally speaking we don’t see driver issues on iOS, and when we do they often happen on early prototypes, and by the time the prototypes make their way to production, the issues are usually fixed.</p><p>We’ve also spent some time improving our Metal backend, focusing on three areas:</p><h3>Reworking the shader compilation toolchain</h3><p>One other thing that happened in the last three years is the release and development of Vulkan. While it would seem that the APIs are completely different (and they are), Vulkan ecosystem gave the rendering community a fantastic set of open-source tools that, when combined, result in an easy-to-use production quality compilation toolset.</p><p>We used the libraries to build a compilation toolchain that can take HLSL source code (using various DX11 features including compute shaders), compile it to SPIRV, optimize the said SPIRV, and convert the resulting SPIRV to MSL (Metal Shading Language). It replaces our previous toolchain that could only use DX9 HLSL source as an input and had various correctness issues for complicated shaders.</p><p>It is somewhat ironic that Apple didn’t have anything to do with this, but here we are. Huge thanks to the contributors and maintainers of glslang (<a href="https://github.com/KhronosGroup/glslang">https://github.com/KhronosGroup/glslang</a>), spirv-opt (<a href="https://github.com/KhronosGroup/SPIRV-Tools">https://github.com/KhronosGroup/SPIRV-Tools</a>) and SPIRV-Cross (<a href="https://github.com/KhronosGroup/SPIRV-Cross">https://github.com/KhronosGroup/SPIRV-Cross</a>). We have contributed a set of patches to these libraries to help us ship the new toolchain as well, and use it to retarget our shaders to Vulkan, Metal and OpenGL APIs.</p><h3>macOS Support</h3><p>A macOS port was always a possibility but wasn’t a big focus for us until we started missing some features. So we decided that we should invest into Metal on macOS to get faster rendering and unlock some possibilities for the future.</p><p>From the implementation perspective, this wasn’t very hard at all. Most of the API is exactly the same; other than window management, the only area that required substantial tweaks was memory allocation. On mobile, there’s a shared memory space for buffers and textures whereas on desktop, the API assumes a dedicated GPU with its own video memory.</p><p>It’s possible to quickly work around that by using managed resources, where the Metal runtime takes care of copying the data for you. This is how we shipped our first version, but we later reworked the implementation to more explicitly copy resource data using scratch buffers so that we could minimize the system memory overhead.</p><p>The biggest difference between macOS and iOS was stability. On iOS we were dealing with just one driver vendor on one architecture, whereas on macOS we had to support all three vendors (Intel, AMD, NVidia). Additionally, on iOS we — luckily! — skipped the *first* version of iOS where Metal was available, iOS 8, and on macOS this was not practical because we would get too few users to use Metal at the time. Because of the combination of these issues, we have hit many more driver issues in both relatively innocuous and relatively obscure areas of the API on macOS.</p><p>We still support all versions of macOS Metal (10.11+), although we started removing support and switching to legacy OpenGL backend for some versions with known shader compiler bugs that are hard for us to work around, e.g. on 10.11 we now require macOS 10.11.6 for Metal to work.</p><p>The performance benefits were inline with our expectations; in terms of market share, today we are at ~25% OpenGL and ~75% Metal users on macOS platform, which is a pretty healthy split. This means that at some point in the future it may be practical for us to stop supporting desktop OpenGL at all, as no other platforms we support use it, which is great in terms of being able to focus on APIs that are easier to support and get good performance with.</p><h3>Iterating on Performance and Memory Consumption</h3><p>We are historically pretty conservative with the graphics API features that we use, and Metal is no exception. There are several big feature updates that Metal has acquired over the years, including improved resource allocation APIs with explicit heaps, tile shaders with Metal 2, argument buffers and GPU-side command generation, etc.</p><p>We mostly don’t use any of the newer features. So far, the performance has been reasonable, and we’d like to focus on improvements that apply across the board, so something like tile shaders, that requires us to implement very special support for it throughout the renderer and is only accessible on newer hardware, is less interesting.</p><p>Having said that, we spend some amount of time tuning various parts of the backend to just run *faster* — using completely asynchronous texture uploads to reduce stuttering during level loads, which was completely painless, doing the aforementioned memory optimizations on macOS, optimizing CPU dispatch in various places of the backend by reducing cache misses etc., and — one of the only newer features we have explicit support for — using memoryless texture storage when available to significantly reduce the memory required for our new shadow system.</p><h3>The Future</h3><p>Overall, the fact that we didn’t have to spend too much time on Metal improvements is actually a good thing — the code that was written 3 years ago, largely speaking, works and is fast and stable, which is a great sign of a mature API. Porting to Metal was a great investment, given the amount of time it took and the continuous benefits it gives us and our users.</p><p>We constantly reevaluate the balance between the amount of work we do for different APIs — it is very likely that we will need to dive deeper into more modern parts of Metal API for some of the future rendering projects; if it does happen, we will make sure to write another post about this!</p><ol><li>Yeah, okay, and maybe a week to fix a few bugs discovered during testing <a href="https://zeux.io/2016/12/01/metal-retrospective/#fnref:2">↩</a></li><li>The numbers are for 128 KB worth of data updated per frame (two 32x16x32 RGBA8 regions) on A10 <a href="https://zeux.io/2016/12/01/metal-retrospective/#fnref:1">↩</a></li></ol><p>Arseny Kapoulkine has worked on game technology for the past decade. Having worked on rendering, physics simulation, language runtimes, multithreading and many other areas, he is still discovering exciting problems in game development that require low-level thinking. After helping ship many titles on PS3 including several FIFA games, he joined Roblox in 2012 and has been working on the in-house engine ever since, helping young game developers achieve their dreams.</p><p>Neither Roblox Corporation nor this blog endorses or supports any company or service. Also, no guarantees or promises are made regarding the accuracy, reliability or completeness of the information contained in this blog.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22d74969a21" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>