<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Bandinopla on Medium]]></title>
        <description><![CDATA[Stories by Bandinopla on Medium]]></description>
        <link>https://medium.com/@pablobandinopla?source=rss-52ab515439d2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*L0uas1PmQXOm4Md22rU9PA.png</url>
            <title>Stories by Bandinopla on Medium</title>
            <link>https://medium.com/@pablobandinopla?source=rss-52ab515439d2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 10:21:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pablobandinopla/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Simple Cloth Simulation with Three.Js and Compute Shaders on skeletal animated meshes]]></title>
            <link>https://medium.com/@pablobandinopla/simple-cloth-simulation-with-three-js-and-compute-shaders-on-skeletal-animated-meshes-acb679a70d9f?source=rss-52ab515439d2------2</link>
            <guid isPermaLink="false">https://medium.com/p/acb679a70d9f</guid>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[threejs]]></category>
            <category><![CDATA[shaders]]></category>
            <category><![CDATA[gpu-computing]]></category>
            <dc:creator><![CDATA[Bandinopla]]></dc:creator>
            <pubDate>Sun, 15 Feb 2026 13:30:26 GMT</pubDate>
            <atom:updated>2026-02-15T13:30:26.912Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cKK5UM2BnSKkDvGDqXq90g.png" /><figcaption>Cloth demo : <a href="https://bandinopla.github.io/three-simplecloth/">https://bandinopla.github.io/three-simplecloth/</a></figcaption></figure><blockquote><strong>Disclaimer: </strong>this article explains how this particular implementation was developed. Room for improvement is there and anyone with more math skills is invited to suggest improvements by comments or doing a <a href="https://github.com/bandinopla/three-simplecloth/tree/main">pull request at the repo</a>.</blockquote><p>Recently I found about the <a href="https://github.com/mrdoob/three.js/blob/a58e9ecf225b50e4a28a934442e854878bc2a959/examples/webgpu_compute_cloth.html">cloth simulation</a> in the Three.Js <a href="https://threejs.org/examples/">official examples</a> and thought how cool would it be to make something simple that one can just <a href="https://x.com/bandinopla/status/2022595248992719212">paint in blender and move on, no coding complexity</a>. So decided to work on an <a href="https://www.npmjs.com/package/three-simplecloth">NPM Package</a> to do that and posted the <a href="https://github.com/bandinopla/three-simplecloth">open source code too</a>.</p><p>But here I’ll talk about the process that led me to this. <strong>I’ll explain it as if to someone who has never used compute shaders before and has no background in it, physics or math. </strong>This is how I digested the information to learn, I’m no expert on this subject.</p><h3>What is a “compute” shader?</h3><p>When using WebGPU we work on the GPU, the GPU is basically another computer inside of your computer. Logically, when you want to run a program in a computer you need to write/code the program, compile it, and upload it to that computer. <strong>But </strong>the GPU also needs to know <strong>how many times</strong> the program should run ( to plan the execution ) So that is also an argument you will need to provide, the number of executions so the GPU can optimize itself.</p><pre>import { Fn } from &quot;three/tsl&quot;;<br><br>// STEP 1 : define the program<br>const someTSLProgramNode = Fn(()=&gt;{ /*...code here...*/ })()<br>const compiledProgram = someTSLProgramNode.compute( howManyTimesToRun )<br><br>// STEP 2: telling the WebGPURenderer to run it on the GPU<br>renderer.compute( compiledProgram );</pre><p><strong>But wait! There’s more… </strong>the entire point of using the GPU is because it does things in parallel, very fast. It kind of splits the work to be done into tiny little pieces and make them run all at the same time. So when writing data to a variable you no longer feel as safe as when writing a program normally in the CPU, because you have to understand, others may or may not be trying to write their own data at the same time ( <em>since they are indeed running in parallel </em>)</p><h3>WebGPU and Three.js</h3><p>You may be asking: “<em>Why tf do I need compute shaders for this cloth thing?</em>” Well, because simulating cloth requires to do a lot of math on abstract objects ( <em>forces</em> ) that represent parts of the cloth very quickly on every frame multiple times. That’s why <a href="https://openmetal.io/docs/product-guides/private-cloud/gpu-parallel-computing/">GPU parallelization</a> comes in handy.</p><p>To work with the GPU in threejs we use the <strong>THREE.</strong><a href="https://threejs.org/docs/?q=WebGPURenderer#WebGPURenderer"><strong>WebGPURenderer</strong></a> and <a href="https://github.com/mrdoob/three.js/wiki/Three.js-Shading-Language">TSL</a> to write shaders. TSL may sound scary but it is actually a happy friendly wrapper for an even scarier bigger monster: <a href="https://en.wikipedia.org/wiki/WebGPU_Shading_Language">WGSL</a>. If you don’t believe me, go check the docs and you will be looking at TSL with tears in your eyes and a smile…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*8TC1VTvo6MgmfXNB1Wk54w.png" /></figure><p>Anyway, so the familiar way to write shaders for most js devs is to code things related to a material. These modify either the vertices or the colors of a mesh. So why do we need “compute shaders” ? Because we need to define a set of <strong>arbitrary data </strong>structures and manipulate them on every frame, keep them updated, etc, very fast, on every frame, multiple times per second, in parallel. Everyting needs to run on the GPU and the way we run programs there is vía these “compute shaders”. We write them, then compile them, and upload them ( threejs does this for us )</p><h3><strong>GPU Data structures</strong></h3><p>The normal way you may know about defining variables, in the GPU world, needs a little tweak. Because here we are living in a parallel dimension where everything is running at the same time. In the GPU your program is not just 1, it is many at the same time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/0*ZMTdNukXa2snYSYt" /></figure><p>When writing shaders for WebGPU, you don’t follow the usual CPU model of declaring variables or arrays and accessing them freely. Instead, data lives in buffers, <strong><em>which are blocks of GPU memory</em></strong> explicitly created, filled, and bound before execution. <strong>Shaders read and write through these buffers</strong> using indices and structured layouts, making memory access explicit and parallel-friendly rather than implicit and sequential like on the CPU.</p><p>The GPU is built for massive parallel work. It needs to know the exact data layout, sizes, and types beforehand so it can plan memory access, schedule thousands of threads, and avoid chaos like race conditions or misaligned reads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F9a1n_IeVHJP4ZyC" /></figure><blockquote>Remember that the program will be copied and run multiple times in a row, like a massive clone army. But the data won’t, that’s why each tiny little soldier needs to know where the data is.</blockquote><h3>instancedArray( array|count, type )</h3><p>This is the <strong>TSL node</strong> used by your program to store data for elements like the points of the cloth, springs, etc. When the code runs in parallel, in chunks, this nodes give us the way to connect to the data, store and read values from a particular index in the data block in a way that is GPU friendly. When the program runs it is cloned and it is assigned an index that we access within our program using the TSL node <strong>instanceIndex. </strong>Most of time times we will pair these nodes together ( <em>Meaning, we will be accessing the data in the instancedArray at the index provided by the instanceIndex node</em> ) because we will design the code in a way that they coincide in a meaningful way.</p><pre>// creating an array of vector4 objects<br>const positionsStore = instancedArray(vPos, &quot;vec4&quot;) ;<br><br>// STEP 1 : define the program<br>const someTSLProgramNode = Fn(()=&gt;{ <br>    // instanceIndex coincides<br>    const pos = positionsStore.element( instanceIndex );<br>    // do stuff...<br>})()<br><br>// let&#39;s say we know we have 12 position objects... we run the program 12 times<br>const compiledProgram = someTSLProgramNode.compute( 12 )<br><br></pre><h3><strong>The Caveat</strong></h3><p>When you dispatch a compute shader, the work is internally organized into batches called workgroups. ( *<em>Haven’t jumped in the workgroupSize rabbit hole yet </em>)</p><p>The hardware <strong>rounds up</strong> to full workgroups, so <strong>a few extra executions often happen beyond the logical data size</strong>. Those extra threads still run your code and will happily read or write out of bounds if you let them. The standard fix is an early escape.</p><pre>// STEP 1 : define the program<br>const someTSLProgramNode = Fn(()=&gt;{ <br><br>    //<br>    // Escape out of bound index...<br>    //<br>    If( instanceIndex.greaterThanEqual( 12 ), ()=&gt;Return() );<br><br>    // instanceIndex coincides<br>    const pos = positionsStore.element( instanceIndex );<br>    // do stuff...<br>})()</pre><p>enough background, back to the cloth…</p><h3>* * * THE CLOTH * * *</h3><p>We need to represent a cloth piece in data structures to be able to do math operations and simulate it’s behavior. A cloth, if you look close enough is a set of strings entangled in different patters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*LQVoE6RoCY6oonTPAhwiLA.png" /><figcaption>a point per intersection. Each point is connected by strings…</figcaption></figure><p>When the strings entangle, they become physically bound at that intersection point. We represent that as a point. A vector 3 point in space that is the place where the fabric twists and basically connects. This pattern is repeated over and over, generating the cloth. A set of long strings intermingled.</p><p>So the way in which we represent the data structure of the cloth is a serie of dots ( XYZ ), connected to each other by strings ( we will call them springs )</p><p><em>So we have </em><strong><em>dots XYZ</em></strong><em> and </em><strong><em>springs</em></strong><em> ( 2 dots connected )</em></p><p>But it doesn’t end there! How strong are these springs? Are they rigid or rubber like? ( <em>think of cotton vs nylon</em> ) There we have another variable we need to store per spring: the spring’s force ( <em>vector 3</em> ). A spring will try to keep the 2 dots connected by trying to make them move towards each other (<em> that’s why it is a vector 3</em> ). But how close?? HA!!! You found a secret! We need a new variable: the distance at which they should stay connected. Because when you see at the fabric of a cloth, you will see a grid patterns, and they have particular distances/gaps, this is the length at which each point should stay at rest. The goal distance to try to aim for on every run.</p><p><em>So we added </em><strong><em>spring force</em></strong><em> and </em><strong><em>spring length at rest </em></strong><em>( goal length )</em></p><p>Cool, so we have dots connected by springs. So that means, a dot may be pulled by many strings in different directions ( <em>will try to stay at the goal length with it’s brothers on the right, top, left, bottom, etc…</em> )</p><p>We need a place to store how many springs each point has. And also, if a spring pulls to the left, and another to the right and so on, we need a place to store that resulting overall force so by the end of the calculation the point will have a single force vector telling it where to move.</p><p><em>We add s</em><strong><em>prings per dot/point</em></strong><em> and </em><strong><em>point’s force</em></strong><em>.</em></p><p>For the “springs per dot”, the way we store this data is using a pointer array the same size as the number of points, so the index coincides with the point to which it belongs. We store one unsigned int per point that stores an index in another array of unsigned ints that contain the IDs of the springs the point has. And to know how many items to consume, from this big array of numbers that contains all the IDs of many other non related points, is by storing the count at the start of the pointer’s position. So we end up with a big array of numbers like: count, ID1, ID2, ID3, … count, ID1, ID2… count, ID1, ID2, etc…</p><p>We do that because that way, we can run a compute shader per point and <a href="https://github.com/bandinopla/three-simplecloth/blob/d9f982a508cc1e7922109d2f154a0c5a3975218b/src/SimpleCloth.ts#L659">obtain the springs quickly</a>, we just have to loop from the index at the <a href="https://github.com/bandinopla/three-simplecloth/blob/d9f982a508cc1e7922109d2f154a0c5a3975218b/src/SimpleCloth.ts#L663">pointer’s index value + 1</a> ( because the element at the value is the count ) and move as much as the count says. We share one big array with all the points. All that <a href="https://github.com/bandinopla/three-simplecloth/blob/d9f982a508cc1e7922109d2f154a0c5a3975218b/src/SimpleCloth.ts#L224">happens here</a>.</p><pre>  const springPointer = vSpringsPointer.element(instanceIndex);<br>  const springCount = vSpringsPerVertexArray.element(springPointer);<br><br>  const ptrStart = springPointer.add(1).toVar(&quot;ptrStart&quot;);<br>  const ptrEnd = ptrStart.add(springCount).toVar(&quot;ptrEnd&quot;);<br><br><br>   Loop( { start: ptrStart, end: ptrEnd, type: &quot;uint&quot;, condition: &quot;&lt;&quot; },<br>         ({ i }) =&gt; {<br>             const springIndex = vSpringsPerVertexArray.element(i).toVar(&quot;springId&quot;); <br>             const spring = springsStore.element(springIndex).toVar();<br>             const springForce = springForceStore.element(springIndex) ; <br><br>             //....<br> <br>         },<br>     );</pre><p>And those are the data structures we’ll need to represent the cloth in a program.</p><h3>* * * THE MESH * * *</h3><p>When receiving the mesh, it is scanned and data structures are built to setup the nodes we will use...</p><p>For this implementation, instead of building a programmatic grid (<em> like in the official examples</em> ) , I used the mesh as the grid itself, since, every mesh is basically points connected by edges, so I just converted each edge to a spring and the vertices are the cloth’s dots:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iFCrIeCgs2GoSkVZvxxfWw.png" /></figure><h3>The vertex</h3><p>A “vertex” is the full <strong>bundle of attributes</strong>: position, normal, UVs, color, skin weights, etc. If two faces meet at the same point but need different normals for sharp shading, or different UV coordinates for texturing, they cannot share the same vertex entry. So the exporter duplicates the position to store different attribute combinations, and the index buffer references those separate vertices. Same place in space, different data, therefore different vertex.</p><pre>// this will contain vertices at the exact same spatial position<br>geometry.attributes.position</pre><p>So I had to scan the vertices and identify the unique spatial points they were actually representing to generate my “points” array. Did this by creating a <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Set">Set</a> and a “hash” using XYZ.</p><h3>The edges / springs</h3><p>To get the edges, I scanned:</p><pre>geometry.index</pre><p>which is an array of triplets representing a triangle. Each triangle has 3 vertices, each has an ID, the index array is a series of IDs, each 3 is a triangle. Each Triangle has 3 edges, so I created a string using the same logic of creating a hash for it ( <em>to avoid duplicates, since the mesh shares edges all over the place</em> )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*cRMWw8BmHdGNtq-vDifCtA.png" /></figure><p>As you see each edge is also part of another triangle, that’s why a hash is needed to avoid duplicates and create each edge/spring object only once.</p><h3>The cloth paint</h3><p>In blender I vertex painted the mesh. This will add a color attribute to each vertex, that we can later use to know how much to blend/mix the position to the skinned mesh and how much to make it act as cloth:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O950LXk0ftGAI8JHMXNCkQ.png" /><figcaption>each vertex has an assosiated color</figcaption></figure><p>In this case, I derived a number from this paint attribute that goes from 0 to 1. And stored that value as a 4th component in the position of each point : vector 3 + a float = vector 4.</p><p>That factor is used to treat the vertex as cloth or a static skinned vertex, or anything in between.</p><h3>Blending between the skeletal animation and the cloth simulation</h3><p>So this is the core of this issue. The logic behind this was to let three animate the <a href="https://threejs.org/docs/?q=SkinnedMesh#SkinnedMesh">skinned mesh </a>using the <a href="https://threejs.org/docs/?q=computeSkinning#TSL.computeSkinning">computeSkinning</a> TSL node, but only as much as the vertex paint factor allowed, and the same for the cloth. Run the cloth simulation only as much as the factor allows.</p><p>Internally I have an array of points in <strong>world space </strong>( <em>important </em>)<strong> </strong>that, on each run, update with a compute shader to follow the skinned mesh and leave the other points alone. Then, I do the same for the cloth sim and ignore the skinned vertices if they are masked. It looks like this:</p><pre>wPos.xyz.assign(mix(wPos.xyz, skinningWorldPosition, factor)); </pre><p><strong>Running a compute shader…</strong></p><pre>renderer.compute( yourComputedShaderNode )</pre><p>The order of execution of the compute shaders is:</p><p>At startup ( each point is a compute shader run ) :</p><ol><li>initialize the position of the points to match the skinned mesh</li><li>calculate the initial ( goal ) lengths of each spring</li></ol><p>Then, <a href="https://github.com/bandinopla/three-simplecloth/blob/d9f982a508cc1e7922109d2f154a0c5a3975218b/src/SimpleCloth.ts#L851"><strong>on every frame</strong></a> / render:</p><ol><li>make the vertices follow the skinned mesh animation based on factor</li><li>calculate the direction and strength in which each spring is pulling</li><li>for each point, move it in the direction of the combined forces of all the springs assigned to that vertice only as much as the vertex painted factor allows.</li></ol><h3>Initializing skinning position</h3><p>At the start we position all the points where the skinned mesh says so, this is the starting position. This is the compute shader we use for that:</p><pre>    /** <br>     * Initial setup<br>     */<br>    const initializeSkinningPosition = Fn(() =&gt; {<br>        If(instanceIndex.greaterThanEqual(countOfPoints), () =&gt; Return());<br><br>        const uIndex = indicesStore.element(instanceIndex);<br>        const wPos = vPosStore.element(uIndex); <br>     <br>        const skinningWorldPosition = worldMatrix.mul(skinningPosition) ;<br>        wPos.xyz.assign(skinningWorldPosition) ; <br><br>    })()<br>    .compute(countOfPoints)<br>    .setName(&quot;Initialize skinning points&quot;); </pre><p>It runs for each point in the mesh‘s position attribute. Previously during setup an array of indices was created where each mesh vertex points to the ID of the unique cloth’s point ( stored in vPosStore )</p><p>We then let the computeSkinning do it’s magic, we will get a positionin world space that follows the skeletal animation. We set all the cloth’s points to that position.</p><h3>Calculate Rest Lengths</h3><p>Now that every point is in it’s initial position, we can calculate the distances between them as use that as the “goal” rest length of the cloth. The ideal distance at which the cloth will / should aim to conserve.</p><pre> const calculateRestLengths = Fn(()=&gt;{<br>  If(instanceIndex.lessThan(totalSprings), () =&gt; {<br>   const vertexIds = springsStore.element( instanceIndex );<br>   const restLength = restLengthsStore.element( instanceIndex ) ;<br><br>   const Ai = vertexIds.x;<br>   const Bi = vertexIds.y; <br><br>   const posA = vPosStore.element(Ai).xyz;<br>   const posB = vPosStore.element(Bi).xyz;<br><br>   const delta = posB.sub(posA);<br>   const dist = delta.length().max(0.000001);<br>   restLength.assign(dist);<br><br>  });<br> })().compute( totalSprings ).setName(&quot;calculate Rest Lengths&quot;); </pre><p>We just measure the distance between both points of each spring and set that as the rest length in the “restLengthsStore” float array. Notice how the index matches the index in springsStore so they are related by the same index.</p><h3>Update Skinning Positions</h3><p>On each render, we run a compute shader to update the positions in our “vPosStore” ( array of vector 4 points ) to match the skinned rig:</p><pre>    const updateSkinningPoints = Fn(() =&gt; {<br>        If(instanceIndex.greaterThanEqual(countOfPoints), () =&gt; Return());<br><br>        const uIndex = indicesStore.element(instanceIndex);<br>        const wPos = vPosStore.element(uIndex); <br>        const factor = wPos.w.pow(2); // 1 = skinned (Pinned), 0 = cloth (Simulated)<br><br>        const skinningWorldPosition = worldMatrix.mul(skinningPosition) ;<br><br>        wPos.xyz.assign(mix(wPos.xyz, skinningWorldPosition, factor));  <br><br>    })()<br>    .compute(countOfPoints)<br>    .setName(&quot;Update skinning points&quot;); </pre><p>We use the vertex paint factor we stored in the 4th channel to mix between the current position of that point or the one that the skeleton wants.</p><h3>Run the cloth simulation</h3><p>At this point the cloth’s points masked will either be following the skinned mesh or static in world space doing nothing.</p><p>We run 2 compute shaders:</p><ol><li><a href="https://github.com/bandinopla/three-simplecloth/blob/d9f982a508cc1e7922109d2f154a0c5a3975218b/src/SimpleCloth.ts#L614"><strong>computeSpringForces </strong></a><strong>: </strong>for each spring, based on position of point A and B, calculate how much it should pull them in to stay at rest length from each other and store that as the spring’s force.</li><li><a href="https://github.com/bandinopla/three-simplecloth/blob/d9f982a508cc1e7922109d2f154a0c5a3975218b/src/SimpleCloth.ts#L648"><strong>computeVertexForces</strong></a><strong>:</strong> for each point in the cloth, calculate the total force applied to it by each of it’s springs and update it’s position XYZ based on that also using the factor ( to avoid moving static / skinned ones ) This is also the place where wind and gravity are applied and colliders. Anything that moves/affects the position XYZof the cloth’s points.</li></ol><h3>The material</h3><p>All that so far happened in the compute shader world, now, to see it on screen we modify the mesh material to use a custom vertexNode and a normalNode.</p><ol><li><a href="https://github.com/bandinopla/three-simplecloth/blob/d9f982a508cc1e7922109d2f154a0c5a3975218b/src/SimpleCloth.ts#L773">vertexNode</a> : here we read the position of each vertex from the unique cloth’s position array ( since it runs per vertex, we can use the instancedIndex to access all the data related to the vertex and they will match ) . This is how the connection between the data that the compute shaders manipulated now are accessed by the material and visualized.</li><li><a href="https://github.com/bandinopla/three-simplecloth/blob/d9f982a508cc1e7922109d2f154a0c5a3975218b/src/SimpleCloth.ts#L783">normalNode</a>: Since the positions of the mesh’s vertices changed, the normals also need to be calculated, this is done by calculating the new direction in which each face is now facing. If you didn’t do that, the shading would look all weird.</li></ol><h3>That’s it</h3><p>You can now read the details of the <a href="https://github.com/bandinopla/three-simplecloth/blob/main/src/SimpleCloth.ts">source code</a> and fill in the gaps.</p><p>Live demo: <a href="https://bandinopla.github.io/three-simplecloth/">https://bandinopla.github.io/three-simplecloth/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=acb679a70d9f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Grass & Debris in Three.js]]></title>
            <link>https://medium.com/@pablobandinopla/grass-debri-in-three-js-6da6b3d599c3?source=rss-52ab515439d2------2</link>
            <guid isPermaLink="false">https://medium.com/p/6da6b3d599c3</guid>
            <dc:creator><![CDATA[Bandinopla]]></dc:creator>
            <pubDate>Sun, 05 Oct 2025 18:15:59 GMT</pubDate>
            <atom:updated>2025-10-06T12:05:36.407Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article I will explain how I added grass and debris to this scene in <a href="https://threejs.org/">three.js</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i7l96kKet6nABXkapoIM8g.png" /></figure><p><a href="https://youtu.be/JXp8GLp8Mqw">https://youtu.be/JXp8GLp8Mqw</a></p><h3>Overview</h3><p>In broad terms, basically you grab a few geometries and spread them randomly over the surface of the ground. Different types of grass, flowers, stones, maybe dead branches, etc… anything you can think of. The end result is a surface covered in stuff that hopefully feel and look organic.</p><h3>Setup</h3><p>You will need to install the following modules:</p><pre>npm install @three.ez/batched-mesh-extensions @three.ez/simplify-geometry</pre><p>And you will need to call this in your js/ts file to “activate” the module:</p><pre>import { extendBatchedMeshPrototype } from &quot;@three.ez/batched-mesh-extensions&quot;;<br><br>extendBatchedMeshPrototype();</pre><h3>Assets setup</h3><p>I exported to a .glb file all the individual “things” I wanted to spread on my floor’s surface with their origin point at the bottom ( <em>or where I would expect the ground to be at, since they will be placed with their origin at the surface </em>) These are the individual objects ( <em>Meshes </em>) used for the scene ( <em>downloaded from </em><a href="https://sketchfab.com/"><em>sketchfab</em></a> )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7Lpc6KNJMdOislflbv2RAA.png" /></figure><blockquote>An <a href="https://threejs.org/docs/?q=InstancedMesh#api/en/objects/InstancedMesh">InstancedMesh</a> would work too, but the BatchedMesh lets us use LODs, so each object switches geometry based on its screen size, cutting down the workload. Since we’re covering a large area, that’s essential.</blockquote><h3>Load &amp; collect the assets</h3><p>The glb file is loaded as usually using the <a href="https://threejs.org/docs/?q=GLTFLoader#examples/en/loaders/GLTFLoader">GLTFLoader</a> and the <em>gltf.scene</em> is scanned looking for our assets. In my case, I used a flag in their <em>userData ( custom property ) </em>called “<strong>spread</strong>” and set it as a string of some constant values since not all assets should be spread the same way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*0Py5AkVWuCbgkko1jDCWsg.png" /><figcaption>this workflow is designer friendly, since it allows you add / remove items from the editor no code required</figcaption></figure><p><strong>For example </strong>grass will have more instances than trees or rocks, so in my case I defined: “default”, “rare” and “very-rare” to map those values to a particular instance count depending on how many of them I want to instantiate.</p><pre>//<br>// collect the objects we want to use to spread onto the ground<br>//<br>let objectsToSpread: Mesh[] = [];<br>gltf.scene.traverse( o =&gt; {<br>  if (o instanceof Mesh &amp;&amp; o.userData.spread ) {<br>      objectsToSpread.push(o);<br>  }<br>});</pre><h3>For each object to spread…</h3><p>Now that we have the <em>“objectsToSpread”</em> we can create the BatchMeshes. In my case, each has a different material, so one BatchMesh per asset will be created, but ideally you may want all of them to share a common atlas material so you can create just one batch mesh, in any case, the process is the same:</p><pre>spread.forEach( debriMesh =&gt; {<br>    let count = 10000; //default count you want...<br>    let castShadow = false;<br>    let scale = sprite.scale.x; //scale the geometry based on how we have it scaled in our 3d Editor...<br><br>     switch (debriMesh.userData.spread) {<br>      case &quot;rare&quot;:<br>       count = 110;<br>       castShadow = true;<br>       break;<br>    <br>      case &quot;very-rare&quot;:<br>       count = 5;<br>       castShadow = true;<br>       break;<br>     }<br><br>     const debriGeometry = debriMesh.geometry.clone().scale(scale, scale, scale);<br><br>     //<br>     // it is async because the LOD creation is done in a web worker.<br>     //<br>     createDebriBatchMesh( debriGeometry, debriMesh.materialm, count)<br><br>        .then( batchedMesh =&gt; {<br><br>            // this function will be defined below...<br>            spreadOverSurface( batchedMesh, myFloorMesh );<br><br>            yourScene.add( batchedMesh );<br><br>        } );<br>});</pre><h3>Create the BatchMesh</h3><p>Now we will create the batch mesh, the LOD geometries and the instances:</p><pre>import { createRadixSort, getBatchedMeshLODCount } from &quot;@three.ez/batched-mesh-extensions&quot;;<br>import { performanceRangeLOD, simplifyGeometriesByErrorLOD } from &quot;@three.ez/simplify-geometry&quot;;<br>import * as THREE from &quot;three&quot;;<br><br>//...<br><br><br>async function createDebriBatchMesh( geometry:THREE.BufferGeometry, material:THREE.Material, count:number ) {<br><br>  // create the LODs...<br>  const geometriesLODArray = await simplifyGeometriesByErrorLOD( [geometry], 4, performanceRangeLOD );<br>  const { vertexCount, indexCount, LODIndexCount } = getBatchedMeshLODCount( geometriesLODArray );<br><br>  const batchedMesh = new THREE.BatchedMesh( count, vertexCount, indexCount, material );<br>  batchedMesh.customSort = createRadixSort( batchedMesh as THREE.BatchedMesh );<br><br>    <br>  //<br>  // add geometries to the batch + their LODs<br>  //<br>  for ( let i = 0; i &lt; geometriesLODArray.length; i ++ ) {<br><br>   const geometryLOD = geometriesLODArray[ i ];<br>   const geometryId = batchedMesh.addGeometry( geometryLOD[ 0 ], - 1, LODIndexCount[ i ] );<br><br>   batchedMesh.addGeometryLOD( geometryId, geometryLOD[ 1 ], .1 );<br>   batchedMesh.addGeometryLOD( geometryId, geometryLOD[ 2 ], .05 );<br>   batchedMesh.addGeometryLOD( geometryId, geometryLOD[ 3 ], .01 );<br>   batchedMesh.addGeometryLOD( geometryId, geometryLOD[ 4 ], .001 );<br><br>  }<br><br>  //<br>  // create instances...<br>  //<br>  for ( let j = 0; j &lt; count; j ++ ) {<br>    batchedMesh.addInstance( 0 )<br>  }<br><br>  return  batchedMesh;<br><br>}</pre><h3>Spread on the ground</h3><p>Now that we have a batched mesh with a bunch of instances, we must spread those instances on the ground…</p><pre>const m = new THREE.Matrix4(); // reusable...<br><br>/**<br>* Will spread each instance of &quot;batch&quot; on top of the &quot;surface&quot; (asuming it is a floor)<br>*/<br>function spreadOverSurface( batch:THREE.BatchMesh, surface:Mesh ) {<br> <br>  for (let i = 0; i &lt; batch.instanceCount; i++) {<br>    <br>     batch.getMatrixAt(i, m); <br>  <br>     // random Y rotation....<br>     m.makeRotationY(Math.PI*2*Math.random());<br>  <br>     // position on the surface...<br>     const point = randomPointInGeometry(surface.geometry);<br>     m.setPosition(point);  <br>  <br>     // random scale...<br>     let s =.5+Math.random()*2 ;<br>     m.scale(new THREE.Vector3(s, s, s))<br>  <br>     batch.setMatrixAt(i, m);<br>  } <br><br>  this.computeBoundsTree();<br>  this.computeBVH( THREE.WebGLCoordinateSystem ); <br><br>}<br><br>//<br>// could be optimized to reuse vectors instead of recreating them on each call...<br>//<br>function randomPointInGeometry(geom: THREE.BufferGeometry): THREE.Vector3 {<br>   const posAttr = geom.attributes.position as THREE.BufferAttribute<br>   const index = geom.index ? geom.index.array : undefined<br>  <br>   // pick random triangle<br>   const triCount = index ? index.length / 3 : posAttr.count / 3<br>   const triIndex = Math.floor(Math.random() * triCount)<br>  <br>   const getVertex = (i: number, target: THREE.Vector3) =&gt; {<br>    const idx = index ? index[i] : i<br>    target.fromBufferAttribute(posAttr, idx)<br>    return target<br>   }<br>  <br>   const a = new THREE.Vector3()<br>   const b = new THREE.Vector3()<br>   const c = new THREE.Vector3()<br>   getVertex(triIndex * 3 + 0, a)<br>   getVertex(triIndex * 3 + 1, b)<br>   getVertex(triIndex * 3 + 2, c)<br>  <br>   // random barycentric coords<br>   let u = Math.random()<br>   let v = Math.random()<br>   if (u + v &gt; 1) {<br>    u = 1 - u<br>    v = 1 - v<br>   }<br>   const w = 1 - u - v<br>  <br>   // interpolate point<br>   return new THREE.Vector3(<br>    a.x * u + b.x * v + c.x * w,<br>    a.y * u + b.y * v + c.y * w,<br>    a.z * u + b.z * v + c.z * w<br>   )<br>}</pre><h3>That’s it!</h3><p>Things to improve as always I’ll let you find them and leave a comment if you find them and want to help others too!</p><p>As far as animating the grass, that is something done in the material of the grass with shaders. But this is the base most simple phase, then you can build on top of this and get more fancy!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JLCyfhUXEoxfNS8kUnehag.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6da6b3d599c3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Frame Callbacks in Three.js: Recreating Flash’s addFrameScript with AnimationClips]]></title>
            <link>https://medium.com/@pablobandinopla/frame-callbacks-in-three-js-recreating-flashs-addframescript-with-animationclips-5ccc713bbb6e?source=rss-52ab515439d2------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ccc713bbb6e</guid>
            <category><![CDATA[3d-animation]]></category>
            <category><![CDATA[webgl]]></category>
            <category><![CDATA[animation]]></category>
            <category><![CDATA[threejs]]></category>
            <category><![CDATA[gamedev]]></category>
            <dc:creator><![CDATA[Bandinopla]]></dc:creator>
            <pubDate>Wed, 03 Sep 2025 20:14:35 GMT</pubDate>
            <atom:updated>2025-09-03T22:09:48.965Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>addFrameScript was a <a href="https://en.wikipedia.org/wiki/Adobe_Flash">Flash</a> feature that let you run custom code automatically when the playhead reached a specific frame in a timeline animation. It basically linked a piece of logic to a moment in the animation.</blockquote><p>With the release of <a href="https://github.com/mrdoob/three.js/releases/tag/r180">Three.js r180</a> vía the <a href="https://github.com/mrdoob/three.js/pull/31586">PR 31586</a> by <a href="https://github.com/repalash">Palash Bansal</a> we can now <strong>add custom properties to animation clips</strong>, while we author them in <a href="https://blender.org/">Blender</a>, and then when exporting them to GLB we can read them in threejs. This gives us an oportunity to recreate the good’ol addFrameScript in blender like I did in <a href="https://x.com/bandinopla/status/1963299694089503160">my work in progress game</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*kfBQ3m3ftSkBg3mJ2uyw-A.png" /></figure><h3>1. Create the custom property in the action clip</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*40zmU6NxA0t_ESzdZnqquA.png" /><figcaption>adding the property</figcaption></figure><p>In blender, while editing the animation in which you want to add a callback, on the right side panel, create a new <strong>custom integer property</strong><em> ( in “Type” set it to “integer”, and set max to 1000 or something because I think it defaults to 1 so it wont let you type any value greater than 1 otherwise).</em> Name this property something significant ( <em>it will be the value your callback will recieve </em>) and then as a value, type the frame you see in the timeline.</p><p>In the picture above, for the animation representing a melee hit, I will need to know when is the point where the weapon is at the “hit” position ( <em>when I expect to check to see if the character hit something </em>)</p><h3>2. Update Three.js to r180+</h3><p>At the time of this writing, <strong>r180 </strong>is the newest version where this new feature was added (<em> remember to check you got the r180 </em>):</p><pre>npm install three@latest</pre><h3><strong>3. Load the GLB in Three.js and scan for the properties…</strong></h3><p>Load your glb and scan like this:</p><pre>//<br>// load your model ( it has to be a GLB )<br>//<br>new GLTFLoader().load(&quot;myModel.glb&quot;, file =&gt; {<br>        //<br>        // for each animation, let&#39;s see if it has custom properties...<br>        //<br>        file.animations.forEach( clip =&gt; {<br>            //<br>            // for each custom property in the clip...<br>            // for this example, we&#39;ll asume if it has one, it is meant<br>            // to be used as a script event.<br>            //<br>            for (const [key, value] of Object.entries(clip.userData))<br>            {  <br>                // since we put the frame number ( easier to remember, you could use time... )<br>                const frame = value as number; <br><br>                // here we setup the callback<br>                addFrameScript( clip, frame, key ); <br>            } <br><br>        });<br>});</pre><h3>4. Save the callbacks</h3><p>So we can later in the code check when to call a script to handle the events…</p><pre><br>const $clipEvents = new Map&lt;THREE.AnimationClip, [number, string, boolean][]&gt;()<br><br>function addFrameScript( clip:THREE.AnimationClip, frame:number, payload:string ) {<br>     if(!$clipEvents.has(clip))<br>     {<br>        $clipEvents.set(clip,[]);<br>     }<br><br>     //<br>     // I think it is easier to work with frames rater than time codes. But then<br>     // we must convert it to time, because we will be checking the playhead time in seconds.<br>     //<br>     const time = frame / 24; // In blender usually 24 is the default FPS... check your case.<br>     <br>     $clipEvents.get(clip)?.push([ time, payload, false ]);<br>     // the last boolean is a handy flag to have a way to tell if we have already called this callback.<br>}</pre><p>The script above will keep track of our frame scripts in the <em>$clipEvents</em> map, so we now have the information of the time at which each clip should call a script with that payload. This is how we do it:</p><h3>5. Check the time!</h3><p>There are many ways to do this, but basically, at it’s core, you want to check on every frame the current time of each active action clip and see if it has events or not…</p><pre>//<br>// I&#39;ll use forEach for brevity, you can use a vanilla for loop or implement a better method ( let me know if you find a faster method vía Twitter )<br>// Let&#39;s say this is your event loop, where you can do things every frame...<br>//<br>function update( delta:number )<br>{ <br>    //<br>    // And that you kept track of all your active actions: const action = mixer.clipAction(...); and you saved that actionin $myActiveActions array or someting...<br>    //<br>    $myActiveActions.forEach( action =&gt; {<br>        const clip = action.getClip();<br>        if( $clipEvents.has(clip) )<br>        {<br>            const scripts = $clipEvents.get(clip);<br>            const localTime = action.time % clip.duration;<br><br>            scripts.forEach( scriptEvent =&gt; {<br>                  if( localTime&gt;=scriptEvent[0] ) // &gt;= because it will most likely never be ==<br>                  {<br>                      if( !scriptEvent[2] )<br>                      {<br>                          scriptEvent[2] = true;<br>                          frameScriptCallback( clip, scriptEvent[1] );<br>                      }<br>                  }<br>            });<br>            <br>        }<br>    });<br>}</pre><h3>6. React to the event!</h3><p>Now we just do whatever we need to do in response to the event! At this point the clip’s time is where we set it in blender and we are getting the payload back.</p><pre>function frameScriptCallback( clip:THREE.AnimationClip, payload:string )<br>{<br>    // here we can check for colisions, instantiate effects, sounds, etc...<br>}</pre><h3>7. Reset the flags</h3><p>In the demo code we had a flag to avoid triggering the callbacks multiple times (<em> if you don’t use a flag, the code will call the event every frame since it only checks if time &gt; event time </em>), we just want it once per play action… but if it loops or we want to play the animation again we should reset the flags…</p><pre>function resetFlagsOf( clip:THREE.AnimationClip )<br>{<br>    // just call a function like this to reset...<br>    $clipEvents.get(clip)?.forEach( event=&gt;event[2]=false );<br>}</pre><p>Usually you will want to call that in 2 places:</p><ol><li>When you play the animation, righ after mixer.clipAction() or action.reset().play()</li><li>When an animation loops:</li></ol><pre>myAnimationMixer.addEventListener(&#39;loop&#39;, e =&gt; {<br>    const clip = e.action.getClip();<br>    resetFlagsOf(clip)<br>});</pre><h3>And that’s it!</h3><p>You can now easily call javascript code once some frame is reached in your animations. And the cool thing about it is that you can set the event frames in the context of the animation you are working on!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ccc713bbb6e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Use Hand Gestures to control Three.js]]></title>
            <link>https://medium.com/@pablobandinopla/use-hand-gestures-to-control-three-js-99842d66e8ad?source=rss-52ab515439d2------2</link>
            <guid isPermaLink="false">https://medium.com/p/99842d66e8ad</guid>
            <category><![CDATA[mediapipe]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[3d]]></category>
            <category><![CDATA[vr]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Bandinopla]]></dc:creator>
            <pubDate>Wed, 27 Aug 2025 16:38:52 GMT</pubDate>
            <atom:updated>2025-10-28T17:09:51.179Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why hand tracking matters?</h3><p>Imagine controlling a 3D scene without touching a mouse or keyboard — just your hands.</p><p>Wire Google’s <a href="https://ai.google.dev/edge/mediapipe/solutions/vision/hand_landmarker">MediaPipe ( hand module ) </a>into <a href="https://threejs.org/">Three.js</a> and you’ll have Minority-Report-level interactions — today, no headset required. It slashes hardware costs, boosts accessibility, and positions you at the bleeding edge of spatial UX.</p><p>Here are a few ways you could apply it:</p><ul><li><strong>E-commerce:</strong> Shoppers can spin a sneaker, pop it open, and change colors with finger flicks — boosting engagement and conversions.</li><li><strong>Data-viz:</strong> Analysts can “grab” a data point from a 3-D scatter-plot and pull it toward them to reveal hidden dimensions.</li><li><strong>Accessibility:</strong> Users who can’t grip a mouse — think motor disabilities or RSI — gain a controller-free gateway to immersive content .</li><li><strong>Game studios:</strong> Designers can block out melee combos or dance emotes during lunch and play-test them that afternoon.</li><li><strong>Fitness apps:</strong> Instructors record a workout once; AI retargets the motion to a 3-D avatar that users can follow in real time</li><li><strong>Sculpting:</strong> users sculpt virtual clay with their fingers.</li></ul><p>In this article, we’ll explore how to integrate MediaPipe’s hand tracking into a Three.js project, and why this combination is not only exciting but also practical for building intuitive, hands-free 3D interactions.</p><h3>Discovering MediaPipe</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*h1CsN696fkOG5oX41XbPfw.png" /><figcaption>first blind run of the demo implementing both parts</figcaption></figure><p>About 3 months ago I stumbled upon what felt like mystical yet fun looking demos posted by <a href="https://x.com/measure_plan">@measure_plan</a> and was presented with this weird name: <strong>“</strong><a href="https://ai.google.dev/edge/mediapipe/solutions/guide"><strong>MediaPipe</strong></a><strong>”</strong> ( <em>by Google</em> )</p><p>Decided to reach out to him and proposed to do a collab demo project where we could work together on <strong>a </strong><a href="https://en.wikipedia.org/wiki/First-person_shooter"><strong>FPS</strong></a><strong> concept demo driven purely by hand gestures</strong>. We ended up making a <a href="https://x.com/measure_plan/status/1916962443252650097">fun collab demo</a> that sparked my interest to dive into <a href="https://ai.google.dev/edge/mediapipe/solutions/vision/hand_landmarker">the official docs</a>.</p><h3>How did we do this?</h3><p>We organized the work to be done in two areas.</p><ol><li>I would focus on the FPS game itself ( <em>implementing a </em><strong><em>KeyboardAndMouseInput</em></strong><em> Control</em> )</li><li>he would work on the input handling ( <em>implementing a </em><strong><em>HandGesturesInput</em></strong><em> Control</em> ).</li></ol><p>We agreed on a simple common interface to do this:</p><pre>class Control {<br>    /**<br>     * In the Y axis. This would rotate camera side by side...<br>     */<br>    get cameraPan() {<br>        return 0;<br>    };<br><br>     /**<br>     * In the X axis. This would rotate camera side by side...<br>     */<br>    get cameraPitch() {<br>        return 0;<br>    };<br><br>    /**<br>     * Move forward and backwards. Y Axis here for now it won&#39;t be used...<br>     */<br>    get direction() {<br>        return new Vector3(0, 0, 0);<br>    }<br>}</pre><p>That would allow me to focus on the FPS using the keyboard for developing, while he could focus on polishing and implementing the integration with MediaPipe and reading the hand gestures in a way that could feed our interface as expected.</p><h4>Two developers, One Interface</h4><p>This type of methodology is what works best when integrating new input systems. Always using an interface in common. This allows the creation of temporary or new implementations in a trivial manner. Think of how easy would it be, and was, for me to implement keyboard control.</p><h3>So… What is MediaPipe ?</h3><blockquote><a href="https://ai.google.dev/edge/mediapipe/solutions/guide">MediaPipe</a> is a <strong>suite of libraries</strong> and tools for you to quickly apply artificial intelligence (AI) and machine learning (ML) techniques in your applications. You can plug these solutions into your applications immediately, customize them to your needs, and use them across multiple development platforms.</blockquote><p>The spooky word “AI” at first was daunting, but, the magic of abstraction made the scary parts disappear… You just pick the module that has the AI skills you want to feed from and install it as a normal js module.</p><h3>Exploring camera control</h3><p>The first demo I did flying solo, after reading <a href="https://ai.google.dev/edge/mediapipe/solutions/vision/hand_landmarker">all the documentation</a>, was a <a href="https://x.com/bandinopla/status/1917927590834155788">hand controlled flying camera</a> app in Three.js. Since I wanted to use the hand control I had to install the <a href="https://ai.google.dev/edge/mediapipe/solutions/vision/hand_landmarker">Hand Landmarker</a> module ( <em>detects key landmarks of one or more hands in real time, providing their positions in a estimated 3D space</em> )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*2RBul7Pm3_r-AadaBbmqCA.png" /><figcaption>controlling a flying drone type camera just with hand gestures</figcaption></figure><h4>Hands as input</h4><p><strong>The </strong><a href="https://ai.google.dev/edge/mediapipe/solutions/vision/hand_landmarker/web_js"><strong>hand module</strong></a> abstracts the mechanic of inferring the position of the hands and fingers via AI based on an image ( <em>or a video feed, such as the webcam</em> ). It provides information of how the hands are positioned, in the form of points in a “3D” space. ( <strong><em>You never really work with AI</em></strong><em> yourself, this is abstracted from you, a.k.a. made easy !</em> )</p><p>The camera’s feed ( <em>or a single image</em> ) is fed into this module resulting in we getting an array of pre-set 3D points ( <em>called “land marks”</em> ) <strong>Each landmark has always these known IDs, so we know, index 8, for example, is the tip of the index finger…</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-AJWHkfA-x_NiNeraFXWsw.png" /><figcaption>cheat-sheet of landmark’s indexes</figcaption></figure><p>You can check out <a href="https://mediapipe-studio.webapps.google.com/demo/hand_landmarker">the official live demo here</a>. And you can check this <a href="https://codepen.io/mediapipe-preview/pen/gOKBGPN">working code example here</a> to see some code in action.</p><h4>3D Points from the Webcam?</h4><p>Well, it is not a real 3D as you understand it in a 3D scene. The module does use AI to infer the position of each landmark. It will give you a value in screen space and a Z value. Each landmark has this “Z” component, but this value is an approximation ( <strong><em>value is noisy</em></strong><em>; clamp or smooth it with a </em><a href="https://jaantollander.com/post/noise-filtering-using-one-euro-filter/"><em>1-Euro filter</em></a><em>.</em> ) of how far from the camera it is. Based on my tests, we kind-of can use this value to get a pseudo 3D positioning of the hand in space. It is not exact but it works for gesture analysis if you tweak it enough.</p><h4>How did you controlled the camera then?</h4><p>I understood that, after all the setup, all I had to work with, was an array of 3D points ( <em>Vectro3Like points, {x,y,z}</em> ) so after that point on, I was back on my element: vanilla <a href="https://threejs.org/">Three.js</a>.</p><p>Started to <strong>look at my hand and think how could I control a camera with it</strong> having <strong>3D points as input data</strong>? So came up with the following interpretation:</p><ul><li>Pitch → up/down tilt (rotate around X-axis) I used the Y delta between landmark 12 and 9.</li><li>Yaw → left/right turn (rotate around Y-axis) I usd the X delta between landmark 12 and 9</li><li>Roll → tilt sideways (rotate around Z-axis) I used the Math.atan2 between landmark 8 and 12.</li></ul><p>As you can see, nothing is set in stone. You can find many other ways to do the same.</p><p>Now I’ll explain how you can start using and experimenting with MediaPipe for hand gestures:</p><h3>The setup</h3><p>Before we start we need to install this library:</p><pre>npm install @mediapipe/tasks-vision</pre><p>And, since this uses AI, and every AI comes in a model, we’ll need to download the model too.</p><pre>import { FilesetResolver, HandLandmarker } from &#39;@mediapipe/tasks-vision&#39;;<br><br>//<br>// this will return our magical hand gesture recognizer...<br>//<br>const createHandLandmarker = async () =&gt; {<br>  const vision = await FilesetResolver.forVisionTasks(<br>    `https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@0.10.0/wasm`<br>  );<br>  return await HandLandmarker.createFromOptions(vision, {<br>    baseOptions: {<br>      modelAssetPath: `https://storage.googleapis.com/mediapipe-models/hand_landmarker/hand_landmarker/float16/1/hand_landmarker.task`,<br>      delegate: &quot;GPU&quot; //[!] GPU delegate is Chrome-only; Safari falls back to WASM (2–3× slower)<br>    },<br>    runningMode: &quot;VIDEO&quot;, // can be IMAGE | VIDEO | LIVE_STREAM<br>    numHands: 1 // for a simple demo where we only want to scan for one hand only<br>  }); <br>};</pre><h4>Check availability…</h4><p>Remember to check first if the user has granted webcam access…</p><pre>//<br>// in your code, use this to show a message if it is false and only<br>// start your app if this returns true.<br>//<br>const webcamIsAvailable = () =&gt; !!navigator.mediaDevices?.getUserMedia;</pre><h4>The Webcam</h4><p>To get the webcam feed, to be able to read the hand positions we make on the camera, we need to setup the camera first:</p><pre>//<br>// HTML Video element: &lt;video id=&quot;webcam&quot; autoplay playsinline&gt;&lt;/video&gt;<br>//<br>const video = document.getElementById(&quot;webcam&quot;) as HTMLVideoElement;<br><br>//<br>// CANVAS: used to draw the landmarks for debugging <br>// (so you see what the model sees)<br>// &lt;canvas class=&quot;output_canvas&quot; id=&quot;output_canvas&quot;&gt;&lt;/canvas&gt;<br>//<br>const canvasElement = document.getElementById( &quot;output_canvas&quot; ) as HTMLCanvasElement;<br>const canvasCtx = canvasElement.getContext(&quot;2d&quot;);</pre><h4>Flipping the webcam’s image</h4><p>You may need to flip the video coming from the webcam, you can do it via CSS doing:</p><pre>video#webcam, <br>canvas#output_canvas {<br>  transform: rotateY(180deg);<br>  -webkit-transform: rotateY(180deg);<br>  -moz-transform: rotateY(180deg);<br>}</pre><h4>Video &amp; Canvas positioning</h4><p>This is optional if you want to render the debug lines in the canvas. To have the canvas stack on top of the video, you may want to style the canvas with a position: absolute so it renders on top ( <em>make sure both ar wrapped inside a </em><em>position:relative parent</em> )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/1*wZipOcDqXIqONi_h31vnvg.png" /><figcaption>canvas positioned on top of the video will allow to visually see what the AI is seeing</figcaption></figure><h4>Start the feed</h4><p>For the webcam to start, it must be initiated by a click of the user on some button. You can’t start it automatically due to security constrains. So add a button with a click listener…</p><pre>document.getElementById(&quot;webcamButton&quot;).addEventListener(&quot;click&quot;, startWebcam );<br><br>let handLandmarker = null;<br><br>function startWebcam() <br>{<br>    // lazy init the handler...<br>    if( !handLandmarker )<br>    {<br>        handLandmarker = createHandLandmarker()<br>    }<br><br>    handLandmarker.then( handReader =&gt; {<br><br>        //<br>        // ask for the webcam feed<br>        //<br>        return navigator.mediaDevices.getUserMedia({ video:true }).then((stream) =&gt; {<br><br>            // make the video object display the webcam feed...<br>            video.srcObject = stream;<br><br>            // when the video has data to show... the action begins.<br>            video.addEventListener(&quot;loadeddata&quot;, predictWebcam);<br>        });<br><br>    });<br>}</pre><h3>MediaPipe in action!</h3><p>This is the part where we bridge the webcam with MediaPipe to scan the image for a hand…</p><pre>function predictWebcam() {<br><br>    //<br>    // [!] I recommend throttling inference to 30 fps if the scene is 60 fps to avoid thermal throttling on laptops.<br>    // you can use a &quot;timeSinceLastPrediction&quot; to skip this to avoid<br>    // being to intense...<br>    //<br><br>    //<br>    // *** WARNING: Spooky AI function call ***<br>    // in this call the AI magic is done... this is where the AI is run<br>    // behind the scenes... the expensive calculation.<br>    //<br>    const results = handLandmarker.detectForVideo(video, performance.now());<br><br>    //<br>    // here are the landmarks positions... that&#39;s it! Was it hard?<br>    //<br>    const landmarks = results.landmarks;<br>}</pre><h3>Debug the hand ( Optional )</h3><p>To see the debug lines, for the drawing helpers ( <em>that look flashy in a demo</em> ), you will need to install these modules:</p><pre>npm install @mediapipe/drawing_utils @mediapipe/hands</pre><p>And then import:</p><pre>import { drawConnectors, drawLandmarks } from &quot;@mediapipe/drawing_utils&quot;;<br>import { HAND_CONNECTIONS } from &quot;@mediapipe/hands&quot;; // constants</pre><p>And in your predictWebcam function you will call:</p><pre>// draw the segments ( lines from joint to joint )<br>drawConnectors(canvasCtx, landmarks, HAND_CONNECTIONS, { color: &#39;#00FF00&#39;, lineWidth: 2 });<br><br>// draw dots on each joint<br>drawLandmarks(canvasCtx, landmarks, { color: &#39;#FF0000&#39;, lineWidth: 1 });</pre><h3>Integration with Three.js</h3><p>Now, you can do many things with these landmarks, the predictWebcam is the place where you will redirect the landmarks to the place in your app where you do something with them.</p><p><strong>The possibilities are only limited by your imagination.</strong> Each landmark gives you coordinates of key points on the hand, allowing you to track fingers, gestures, and overall hand orientation in real time. You can use this data to manipulate 3D objects, trigger animations, control cameras, or even create entirely new interaction methods in VR or AR. Beyond practical applications, these landmarks invite playful experimentation: design gesture-based games, interactive art, or immersive installations where users’ hands become a natural input device. This is where creativity truly reigns, turning raw data points into dynamic, engaging experiences.</p><h3>Example #1: Control the camera</h3><p>In this sample code, you can pan the camera horizontally with the movement of your hand side to side, and zoom in or out by making your index finger closer or farther from the camera.</p><pre>// Assuming you already have handLandmarker set up and a Three.js scene<br>function updateCameraFromLandmarks(landmarks, camera) { <br>    const wrist = landmarks[0];       // Landmark 0<br>    const indexTip = landmarks[8];    // Landmark 8<br>    // Convert normalized coordinates to screen/world values<br>    const panX = (wrist.x - 0.5) * 10; // adjust scale to your scene<br>    const panY = -(wrist.y - 0.5) * 10;<br>    const zoomZ = 10 + (0.5 - indexTip.y) * 20; // move camera along z<br>    // Apply to camera<br>    camera.position.x = panX;<br>    camera.position.y = panY;<br>    camera.position.z = zoomZ;<br>    camera.lookAt(0, 0, 0);  <br>}</pre><h3>Example #2: Render the hand in the scene</h3><p>You might want to interact with objects in your scene, in which case, you may want to add an object per landmark of the hand in it, as to use them possibly as colliders?</p><pre>// in your scene setup...<br>//<br>const landmarkSpheres = [];<br>const sphereGeometry = new THREE.SphereGeometry(0.02, 8, 8);<br>const sphereMaterial = new THREE.MeshBasicMaterial({ color: 0xff0000 });<br><br>// create a sphere for each 21 known landmarks...<br>for (let i = 0; i &lt; 21; i++) {<br>  const sphere = new THREE.Mesh(sphereGeometry, sphereMaterial);<br>  scene.add(sphere);<br>  landmarkSpheres.push(sphere);<br>}<br><br>function convertLandmarkToThreeJS(lm, scale = 1, offset = { x: 0, y: 0, z: 0 }) {<br>  // lm: {x, y, z} from MediaPipe world coordinates<br>  return {<br>    x: lm.x * scale + offset.x,<br>    y: lm.y * scale + offset.y,<br>    z: lm.z * scale + offset.z<br>  };<br>}<br><br>//<br>// Update sphere positions per frame<br>//<br>function updateLandmarks(landmarks) {<br>  if (!landmarks || landmarks.length !== 21) return;<br>  for (let i = 0; i &lt; 21; i++) {<br>    const lm = landmarks[i];<br>    const lm = convertLandmarkToThreeJS(landmarks[i], 1, undefined);<br>    landmarkSpheres[i].position.set(lm.x, lm.y, lm.z);<br>  }<br>}</pre><h3>Example #3: Detect a pinch gesture</h3><p>You might want to detect if the user is pinching and act upon that gesture:</p><pre>// landmark indices<br>const THUMB_TIP = 4;<br>const INDEX_TIP = 8;<br><br>// threshold distance for pinch (in world coordinates)<br>const PINCH_THRESHOLD = 0.03; <br>let isPinching = false;<br><br>function checkPinch(landmarks) {<br>  if (!landmarks || landmarks.length &lt; 9) return false;<br>  const thumb = landmarks[THUMB_TIP];<br>  const index = landmarks[INDEX_TIP];<br>  // Euclidean distance<br>  const dx = thumb.x - index.x;<br>  const dy = thumb.y - index.y;<br>  const dz = thumb.z - index.z;<br>  const distance = Math.sqrt(dx*dx + dy*dy + dz*dz);<br>  return distance &lt; PINCH_THRESHOLD; // true = the hand is pinching<br>}</pre><h3>The most basic use</h3><p>Once you know the landmarks, which are just “3D” points, you can, for example, drive values out of:</p><ul><li><strong>Distance between some landmarks</strong>: volume, zoom, grab if less than some threshold, release if greater than some threshold… etc.</li><li><strong>Angles between segments</strong>: rotate something based on how where the vector formed by some landmarks point…</li><li><strong>Scale</strong>: using either Z values or the relative distance between landmarks to infer change in size. Example: if yu move the hand closer to the camera, all the relative distances will increase, use this to create a delta value representing how much something has scaled… etc…</li><li><strong>Draw</strong>: you can use the index finger to drive the 2D position of some brush on a canvas or surface and use it to draw. Maybe using the z component as the brush’s pressure.</li></ul><h3>Remember, landmarks are just “3D” points</h3><p><strong>MediaPipe abstracts all the hard AI stuff, leaving you with a nice array of 3D points.</strong>. Nothing to fear about. Once you have the landmarks, you say bye bye to MediaPipe and your journey with Three.js continues !</p><h3>Read more</h3><ul><li><a href="https://x.com/measure_plan">@measure_plan</a> has a <a href="https://www.funwithcomputervision.com/">serie of tutorials</a> that you may find interesting.</li><li><a href="https://ai.google.dev/edge/mediapipe/solutions/vision/hand_landmarker/web_js">Official docs</a> on MediaPipe’s site.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99842d66e8ad" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Three.js and the fall of Constantinople]]></title>
            <link>https://medium.com/@pablobandinopla/threejs-and-the-fall-of-constantinople-ddabc10add56?source=rss-52ab515439d2------2</link>
            <guid isPermaLink="false">https://medium.com/p/ddabc10add56</guid>
            <category><![CDATA[threejs]]></category>
            <category><![CDATA[constantinople]]></category>
            <category><![CDATA[webdev]]></category>
            <category><![CDATA[webgl]]></category>
            <dc:creator><![CDATA[Bandinopla]]></dc:creator>
            <pubDate>Tue, 26 Aug 2025 22:41:34 GMT</pubDate>
            <atom:updated>2025-08-28T12:05:22.173Z</atom:updated>
            <content:encoded><![CDATA[<p>For over a millennium, the <a href="https://en.wikipedia.org/wiki/Walls_of_Constantinople">Theodosian Walls</a> defined the physical and psychological reality of the West. Presiding over the <a href="https://en.wikipedia.org/wiki/Golden_Horn">Golden Horn</a>, they were the ultimate argument in geopolitics, a statement of power so profound that countless emperors considered their city the eternal and impregnable heart of Christendom. To approach from the land was to witness a masterwork of military architecture: a trinity of ramparts, legions of towers, and a defensive moat that shone with the cold light of steel. To speak of <a href="https://en.wikipedia.org/wiki/Constantinople">Constantinople</a> was to speak of a city that had transcended the vulnerabilities of history.</p><h3>The 2D wall on the web</h3><p>An analogous wall, one of logic rather than limestone, once segregated the two-dimensional web from the spatial possibilities of 3D. A decade ago, three-dimensional graphics on the internet were the exclusive domain of fragile, proprietary systems. It was a world of arcane plug-ins, black-box Java applets, and the brittle hope invested in a <a href="https://en.wikipedia.org/wiki/VRML">VRML</a> ( <em>pioneer web 3-D in spirit; it just lacked GPU access</em> ) file — a hope that a simple spinning cube might render without a fatal exception. The term “immersive” was used by developers not as a technical description, but as a hopeful prophecy, much as Byzantine diplomats spoke of their city’s invincibility — a rhetorical shield against an unstable reality.</p><p>In the sections to come, we will examine the siege of both fortifications. The parallels are structural, not just poetic, for they reveal a recurring pattern in human innovation: no system, whether built of stone or of code, is immune to the forces of change. <strong>Every wall is a temporary solution, destined, eventually, to become a gateway.</strong></p><h3>Let’s go back in time…</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DiRIsDjFCHfPLYxBDN6Wfg.png" /></figure><p><a href="https://en.wikipedia.org/wiki/Constantinople">Constantinople</a>, the heart of the <a href="https://en.wikipedia.org/wiki/Byzantine_Empire">Byzantine Empire</a>, had stood for a thousand years as an immovable beacon. Traders, armies, and pilgrims all passed its gates, but its defenses held and the world assumed they always would. Life was defined by that stability, by walls that seemed eternal — until one day they weren’t.</p><h4>The web once had its own “walls.”</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/839/1*G2gGfelG0dlxTGhMinjhjw.png" /></figure><p>Before <a href="https://threejs.org/">Three.js</a> and <a href="https://en.wikipedia.org/wiki/WebGL">WebGL</a>, 3D on the browser meant plugins, clunky Java applets, or experimental hacks that relied on <a href="https://en.wikipedia.org/wiki/Adobe_Flash_Player">Flash</a> with libraries like <a href="https://en.wikipedia.org/wiki/Papervision3D">Papervision3D</a>, <a href="https://en.wikipedia.org/wiki/Away3D">Away3D</a> or <a href="https://github.com/AlternativaPlatform/Alternativa3D">Alternativa3D</a>. It was a bit awkward, relying mostly on the client machine having a plugin installed, somewhat obscure, fragile, and limited to those willing to wrestle with these new tools. For most developers and users, outside of a few amazing demos, the idea of rich, interactive 3D online felt as unreachable as breaching the Theodosian Walls.</p><p>Both Constantinople and the early web shared this illusion of permanence: a world where the boundaries felt fixed and unshakable. But history — and technology — are never still.</p><h3>The Beginning of the End</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g3CoJs63mV1hUOEA1gOoNQ.png" /><figcaption>a new wave of developers…</figcaption></figure><p><strong>The year is 1453</strong>. Constantinople has been surrounded and under siege for weeks by the forces of the Ottoman Sultan <a href="https://en.wikipedia.org/wiki/Mehmed_II">Mehmed II</a>, supported by a massive army and artillery. What had held for centuries suddenly looked fragile. The empire wasn’t finished yet, but the cracks were forming, and everyone could sense the tide was turning.</p><h4>The web had its own Mehmed moment…</h4><p><strong>The year is 2006</strong>. <a href="https://en.wikipedia.org/wiki/Vladimir_Vuki%C4%87evi%C4%87">Vladimir Vukićević</a>, then an engineer at <a href="https://www.mozilla.org/">Mozilla</a>, began experimenting with Canvas 3D, he had an idea that sounded almost absurd: bring real-time 3D graphics into the browser, <strong>without plugins</strong>, using only JavaScript. Vukićević worked on a prototype that showed something radical: the browser itself could be a 3D canvas, speaking directly to the GPU ( <em>via OpenGL initially, which later inspired WebGL</em> ).</p><p><strong>This experiment planted the seed</strong> for what, three years later, became WebGL. In March 2009, the <a href="https://en.wikipedia.org/wiki/Khronos_Group">Khronos Group</a> ( <em>the same consortium behind OpenGL</em> ) announced the first <a href="https://arstechnica.com/information-technology/2009/12/webgl-draft-published-khronos-seeks-community-involvement/">draft of the WebGL specification</a>. It wasn’t polished, it wasn’t friendly — developers had to wrestle with verbose, low-level APIs, shaders, and state management — but it was powerful. For the first time, the walls of the plugin era cracked open. You could write JavaScript, open a browser, and get raw, hardware-accelerated 3D rendering, cross-platform and plugin-free.</p><h3>Paradigm Shifts</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AoZq1kZM4ci5sCFsFTPeXw.png" /><figcaption>new tools break walls</figcaption></figure><p><strong>At dawn on May 29, 1453, Constantinople awoke to its final day</strong>. Mehmed II’s giant bombards, some over 8 meters long ( <em>designed by </em><a href="https://en.wikipedia.org/wiki/Orban"><em>Orban</em></a> ), had been pounding the walls for weeks. That morning, they fired again with relentless rhythm, shaking the ground and tearing open the ancient defenses. A weapon unlike anything the city had faced before had arrived, and with it, an empire that had lasted a thousand years was about to collapse.</p><h4>The web saw a similar paradigm shift centuries later…</h4><p>The eye of what would become the Three.js revolution began to flicker open in the <strong>early 2000s</strong>, when Ricardo Cabello — known as <a href="https://mrdoob.com/">Mr.doob</a> — first stepped onto the stage ( <a href="https://github.com/mrdoob/three.js/issues/1960#issuecomment-5850890"><em>read his recollection</em></a><em> </em>). He emerged as a digital artist, conjuring worlds and illusions with friends at <a href="https://www.pouet.net/lists.php?which=318">demo parties</a>. <strong>By 2006</strong>, the chrysalis split: the artist was transforming into a programmer, summoning his first hand-coded 3D cube in Flash and <a href="https://en.wikipedia.org/wiki/ActionScript">ActionScript</a>. From that spark arose the creature’s first timid form — <strong><em>Three.as </em></strong><em>( “.as” as in Actionscript )</em>, a fledgling Flash 3D library. Yet the beast was incomplete, restless. Its true awakening came <strong>in 2008</strong>, with <a href="https://googleblog.blogspot.com/2008/09/fresh-take-on-browser.html">the dawn of Google Chrome</a>: the spark that ignited the fire. From that moment on, JavaScript and 3D on the web became his singular obsession.</p><p><strong>On April 24, 2010 </strong>the very <a href="https://github.com/mrdoob/three.js/commit/a90c4e107ff6e3b148458c96965e876f9441b147">first version of Three.js</a> was released: a thin, elegant wrapper around raw WebGL. A lone cannonball against the walls of the old web, it announced a new era. Within weeks spinning cubes and flickering lights were running live in browsers; by 2011 GitHub and JSFiddle brimmed with experiments that proved the revolution had arrived.</p><p>Just as Mehmed’s cannons made the unthinkable — breaching Constantinople’s walls — suddenly possible, Three.js made the impossible — real-time 3D on the web — accessible to anyone willing to experiment.</p><h3>The Siege: Struggles &amp; Resistance</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UArjxkeQR6hTtKzheQ0k0g.png" /><figcaption>the resistance…</figcaption></figure><p>The siege of Constantinople dragged on for nearly two months. <a href="https://en.wikipedia.org/wiki/Byzantine_Greeks">The Byzantines</a>, though outnumbered and outgunned, patched their walls at night, built barricades behind breaches, and fought desperately in the streets. Every day was a battle of endurance, a clash between an old order fighting to survive and a new power determined to reshape the world.</p><h4>The web’s 3D landscape faced its own siege…</h4><p>Resistance to 3-D on the web was never a single battle line: security researchers warned that raw GPU access could crash machines or leak data, Microsoft and IT departments kept WebGL switched off, and designers clung to stable Flash or Java pipelines rather than rebuild in fledgling WebGL. The objections — safety, performance, legacy investment — were scattered, but together they propped up the flat web until hardware and standards matured and the wall started to crack.</p><h3>Shifting Worlds</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_HMXeYTa9G_6LxiE9Xo0zA.png" /><figcaption>Flash is dead</figcaption></figure><p><strong>When May 29, 1453 ended</strong>, the unthinkable had happened: <strong>Constantinople was no more</strong>. With its fall, the Ottoman Empire gained control of the <a href="https://en.wikipedia.org/wiki/Bosporus">Bosphorus</a>, cutting off Europe’s old land routes to Asia. Traders and explorers were forced to look for alternatives, a shift that eventually pushed Europeans toward the seas — and into <a href="https://en.wikipedia.org/wiki/Age_of_Discovery">the Age of Exploration</a>. One city’s fall redirected the flow of global power, wealth, and culture.</p><h4>The final trumpet</h4><p><strong>On 25 July 2017 </strong><a href="https://en.wikipedia.org/wiki/Adobe_Flash_Player#End_of_life">Adobe’s announcement that Flash would reach end-of-life</a> on 31 December 2020 served as the final trumpet: <strong>the era of browser plug-ins was over</strong>. With the gates of the old 2-D web closing behind them, creators streamed onto the open WebGL road. Three.js — already battle-hardened by seven years of monthly releases, skeletal animation, OBJ/GLTF loaders, and a thriving ecosystem of shaders and post-processing — became the obvious wagon train. By the end of 2016, flagship projects like Google’s “A Spacecraft for All,” NASA’s “Experience Curiosity,” and National Geographic’s immersive stories had proven that a single JavaScript import could deliver AAA-level 3-D across desktops, tablets, and phones without a single plug-in. What began as an April 2010 experiment had quietly become the production standard: the web was finally, irrevocably three-dimensional.</p><p>Like Constantinople’s fall forcing merchants onto new paths, the collapse of old 3D tools forced creators, educators, and developers to adopt new pipelines. Three.js didn’t just open a technical door — it reshaped the routes of creativity and innovation on the internet.</p><h3>A New Era</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d1oBpO2XaXO3x_tF8z1Qjw.png" /></figure><p>The fall of Constantinople in 1453 didn’t just end the Byzantine Empire — it rippled outward. Scholars fled west carrying ancient manuscripts, fueling the <a href="https://en.wikipedia.org/wiki/Renaissance">European Renaissance</a>. Trade routes shifted, explorers like <a href="https://en.wikipedia.org/wiki/Christopher_Columbus">Columbus</a> set sail in 1492, and the world map itself was redrawn. <strong>Out of collapse came renewal</strong>, a cultural and technological leap that reshaped centuries.</p><h4>Three.js had its own Renaissance moment…</h4><ul><li><strong>2008</strong>: <a href="https://googleblog.blogspot.com/2008/09/fresh-take-on-browser.html">the arrival of Google Chrome</a> pulls <em>Mr.doob </em>from working on his 3D engine ( at the time Three.as ) in Flash/Actionscript into HTML5 and Javascript.</li><li><strong>2010:</strong> Three.js is released to the world</li><li><strong>by 2012</strong>: Three.js 0.x line ( <em>Mr.doob’s first public commits </em>) is already attracting early adopters; by late 2012 the r50 (<em>late 2012</em>) tagged release lands.</li><li><strong>2013</strong>: Chrome Experiments start featuring dedicated Three.js demos (<em>e.g., “Lights”</em> ), not just generic WebGL.</li><li><strong>2014–2015</strong>: High-profile launches — Google’s “A Spacecraft for All” (<em>2014</em>), NASA debuted new browser-based visualisations such as ‘Experience Curiosity’ (<em>2015</em>), built with Three.js, National Geographic’s “Serengeti Lion” (<em>2015</em>) — are all built with Three.js, signalling production readiness.</li><li><strong>2016–2017</strong>: Adoption broadens to data-viz toolkits (<em>e.g., deck.gl, three-geo</em>) and immersive journalism. Release cadence crosses 150 tagged builds around r86 (<em>late 2017</em>).</li><li><strong>December 2020</strong>: Adobe ends Flash support, but its practical demise for 3-D happened years earlier ( <em>the last Flash-based 3-D project of note disappeared around 2014–2015</em> ); the milestone mainly underscores that Three.js/WebGL had long since filled the vacuum.</li></ul><p>Just as the Renaissance redefined art, science, and culture, Three.js redefined what was possible on the web. The fall of old walls — whether stone or software — was not an end but a rebirth, opening the path to richer, more immersive worlds.</p><h3>Conclusion</h3><p>History teaches that no wall, no empire, and no technology is eternal. Constantinople’s fall in 1453 was not just the end of a city, but the birth of a new world. The same lesson echoes in our digital age: the fall of Flash, the rise of WebGL, and the arrival of Three.js in 2010 reshaped the landscape of the web.</p><p><strong>Today, the developers who wield Three.js are more than just coders</strong> They are the web’s new Renaissance artists: with shaders instead of pigment and cameras instead of chisels, they redraw the boundaries of storytelling, education, and design. And they are also the vanguard of the next frontier — VR and AR — already fluent in the language of spatial computing, ready to step through the screen and sculpt worlds we can walk inside.</p><p><strong>The siege never ends.</strong> Technology is always shifting, always marching forward. But those who embrace it — those who dare to pick up the tools of tomorrow — are the ones who shape the future.</p><p><a href="https://threejs.org/">Threejs</a> isn’t just a library; <strong>it’s a revolution still unfolding</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vXhmFfsBshTLO1sINKIpGA.png" /><figcaption>Join the renaissance of 3D on the web</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ddabc10add56" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scroll Driven presentation in ThreeJs with GSAP]]></title>
            <link>https://medium.com/@pablobandinopla/scroll-driven-presentation-in-threejs-with-gsap-a2be523e430a?source=rss-52ab515439d2------2</link>
            <guid isPermaLink="false">https://medium.com/p/a2be523e430a</guid>
            <category><![CDATA[gsap]]></category>
            <category><![CDATA[webdev]]></category>
            <category><![CDATA[scroll]]></category>
            <category><![CDATA[animation-and-design]]></category>
            <category><![CDATA[threejs]]></category>
            <dc:creator><![CDATA[Bandinopla]]></dc:creator>
            <pubDate>Tue, 26 Aug 2025 11:46:37 GMT</pubDate>
            <atom:updated>2025-08-27T18:31:25.654Z</atom:updated>
            <content:encoded><![CDATA[<h3>Scroll Driven presentation in Three.js with GSAP</h3><p>Have you ever landed on a website where the scroll itself feels alive — every movement unfolding 3D scenes, transitions, or playful effects perfectly in sync with your progress down the page? That kind of magic often comes from pairing <a href="https://threejs.org/">Three.js</a> with <a href="https://gsap.com/docs/v3/Plugins/ScrollTrigger/">GSAP’s ScrollTrigger</a>. In this article, I’ll break down how these two tools work together to turn simple scrolling into an immersive animation experience as I did for <a href="https://bandinopla-unitree.vercel.app/">my demo</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rp9_IHsJjKpKymYsJyR7PQ.png" /><figcaption>meet: Mira (screen capture)</figcaption></figure><blockquote>In <a href="https://bandinopla-unitree.vercel.app/"><strong>Meet: Mira</strong></a>, I built a cinematic, scroll-driven WebGL experience that introduces Mira — a robot whose identity shifts through customizable silicone envelopes that drape over her chassis like second skins. The entire story unfolds in a single-page application powered by <a href="https://threejs.org/">Three.js</a> paired with <a href="https://gsap.com/scroll/">GSAP</a>, where scroll gestures choreograph camera moves, lighting, and morph-target animations in real time. Every envelope wraps procedurally around her form, transforming cold metal into something unsettlingly human. It’s part product demo, part speculative fiction — an experiment in how far browser-based 3D can go when narrative, interaction, and visual effects collapse into one seamless performance.</blockquote><h3>How did you paired the scroll with ThreeJs ??</h3><p>Haaa I see you are hungry for knowledge! Time to spill the beans.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u47Acn4w5Al9rkLmGgxEZw.png" /></figure><h3>Layers</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wrJLRbQFTVB9b2j3S7uFhQ.png" /><figcaption>layers, think in: layers.</figcaption></figure><p>First, you have to think in layers. Start simple and slowly add complexity. In broad terms, for my demo I did the following:</p><ol><li><strong>ANIMATION CLIPS</strong> create the animations in your scene in a way that are easy to update with the data provided by GSAP. I spread each section’s shot as a keyframe in a single action so I could mathematically find the right keyframe based on a section’s index.</li><li><strong>HTML SETUP</strong> created the contents/sections in html ( <em>I’ve made each section be a min-height of 100vh</em> ) so far this is basic html. Each section occupies the height of the window, this is so when you align the section in the window, it will be centered and the only thing visible.</li><li><strong>GSAP</strong> Then, I used javascript to iterate over all the sections, and for each section I created a <a href="https://gsap.com/docs/v3/Plugins/ScrollTrigger/">GSAP’s ScrollTrigger</a> This object will inform us on a onUpdate callback the &quot;progress&quot; of the ScrollTrigger instance where 0 is at the start, 0.5 is in the middle, and 1 is at the end. This information will be used later to pair it with Three.</li><li><strong>THREE</strong> Add/Initialize Three in a way that it’s canvas is fixed and behind all the html content. (Always on screen)</li><li><strong>SYNC</strong> Update the threejs scene based on the progress of each section ( <em>I do this on update loop</em> )</li></ol><p>Now let’s dive deeper:</p><h3>1. ANIMATION CLIPS</h3><p>I use <a href="https://www.blender.org/">Blender</a>, but the concepts are IDE agnostic. I knew the number of sections and I knew that when the user scrolled the page I would want to transition from one shot to the next. So the structure that felt fit that better was a single camera action ( <em>animation clip</em> ) with one keyframe per shot separated by an equally amount of keyframes. That way I could map the section’s index in the array with the time in the clip. Easy math…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S2XtzAeGQKvsSTy0Csz9Pg.png" /><figcaption>one action, ech keyframe is a section</figcaption></figure><h3>2. HTML SETUP</h3><p>In my case, I created a “features” array ( <em>my sections</em> ) so I could iterate over them and create the HTML easily ( <em>it is also cleaner to separate the data from the presentation</em> ).</p><pre>const features: { title: string; desc: string }[] = [<br>    {<br>        title: &quot;Meet Mira&quot;,<br>        desc: &quot;Your fully customizable Assistant(*)&quot;, <br>    },<br>    // etc... the rest of the sections...<br>]</pre><p>Then, using <a href="https://react.dev/learn/writing-markup-with-jsx">React’s JSX</a> ( <em>but you can do this in plain javascript if you are not using react</em> ) I created the html sections like so ( <em>notice, I have a reference to the DOM of each section in sectionRefs which is an array of HTMLDivElement</em> ) So far this is basic stuff… a normal page that you have to scroll to see the bottom.</p><pre>.feature {<br>    height: 100vh; <br>    /* ... etc... */<br>}</pre><pre>&lt;main&gt;<br>{<br>    features.map((e, i) =&gt; &lt;div key={i} ref={(el) =&gt; { sectionRefs.current[i] = el }} className=&#39;feature&#39;&gt;<br>        &lt;h1&gt;{e.title}&lt;/h1&gt;<br>        &lt;h2&gt;{e.desc}&lt;/h2&gt; <br>        { i==0 &amp;&amp; &lt;div className=&#39;scrollDown&#39;&gt; ↓ SCROLL  &lt;/div&gt;} <br>    &lt;/div&gt;)<br>}<br>&lt;/main&gt;</pre><h3>3. GSAP</h3><p>To hook <a href="https://gsap.com/docs/v3/Installation">GSAP</a>, on my react’s component’s <a href="https://react.dev/reference/react/useLayoutEffect">useLayoutEffect</a> on mount, I did the following:</p><pre>// in this array i&#39;ll store the GSAP progress of each section<br>    const sectionProgress = useRef(features.map(() =&gt; 0)); //array of 0&#39;s <br>    const sectionRefs = useRef&lt;(HTMLDivElement | null)[]&gt;([])<br><br>// **** I assume you already did -&gt; npm install gsap ****<br>    useLayoutEffect(() =&gt; {  <br>        // this context is used to do a nice cleanup on unmount<br>        const ctx = gsap.context(() =&gt; {<br>            // For each section...<br>            features.forEach((_, i) =&gt; {<br>                const triggerEl = sectionRefs.current[i]<br>                if (!triggerEl) return;<br><br>                //<br>                // GSAP enters the chat...<br>                //<br>                ScrollTrigger.create({<br>                    trigger: triggerEl,<br>                    start: &quot;top bottom&quot;,<br>                    end: &quot;bottom top&quot;,<br>                    //onEnter: () =&gt; console.log(&quot;ENTERED SECTION:&quot;, i),<br>                    //onEnterBack: () =&gt; console.log(&quot;EXITED SECTION:&quot;, i),<br>                    onUpdate: e =&gt; {<br>                        // I save the progress in an array of numbers, this is<br>                        // what I will pass to threejs to sync the animations!<br>                        sectionProgress.current[i] = e.progress;<br>                    }<br>                })<br>            }); <br>        });<br>        // ...<br>        return () =&gt; {<br>            ctx.revert() // cleanup on unmount<br>        }<br>    }, []); //&lt;-- on mount!!</pre><p>The code above will keep the “sectionProgress” array updated, each element will have the “progress” of each section in it (<em> a number from 0 to 1</em> ), being updated. This is the data we will feed to threejs to appear to be in-sync with the scroll!</p><h3>4. THREE</h3><p>In this case I had three fixed in the screen and transparent ( <em>because I wanted to set the background in css</em> ), ignoring the scroll, because we will animate the scene to respond to it vía code.</p><pre>.canvas {<br>    position: fixed;<br>    top: 0px;<br>    left: 0px;<br>    pointer-events: none; <br>    width: 100vw;<br>    height: 100vh; <br>}</pre><p>On my demo I used <a href="https://r3f.docs.pmnd.rs/getting-started/introduction">react-three-fiber</a> and <a href="https://r3f.docs.pmnd.rs/tutorials/loading-models">loaded a glb</a> scene that contained the robot and the shots, camera, etc, but you can use vanilla js, the important thing is to read the contents of sectionProgress.current which is just an number[] So on your main loop, in your three app:</p><pre>const { set } = useThree() // method to modify three&#39;s stuff since im using react three fiber...<br><br>//...etc... declarations...<br><br>// on component mount (or in your initialization method in vanilla js )<br>useEffect(() =&gt; {<br><br>    // ... scene setup<br><br>    // set the camera we used to establish the shots in our 3D ide...<br>    const newCamera = cameras[0] as PerspectiveCamera; <br>    newCamera.aspect = window.innerWidth / window.innerHeight;<br>    newCamera.updateProjectionMatrix();<br>    set({<br>        camera: newCamera //set the main camera to the one I animated in the glb scene<br>    });<br><br>    //<br>    // I had one animation clip, every X frames with a keyframe per section with the shot I wanted<br>    // for that section (important detail)<br>    //<br>    cameraMixer.current = new AnimationMixer( newCamera ); <br>    cameraShotsClip.current = animations.find(c=&gt;c.name==&quot;camera-mobile-shots&quot;)!;<br>    cameraShotAction.current = cameraMixer.current.clipAction(cameraShotsClip.current).play() ;<br>    // pause the animation (because we will change the time via code...)<br>    cameraShotAction.current.paused = true;<br><br>    //<br>    // Same mechanism for the Android... It has one animation clip, with each keyframe in the same time as the camera&#39;s clip, in the pose I want for that section/shot.<br>    //<br>    mixer.current = new AnimationMixer(scene.getObjectByName(&quot;Armature&quot;)!);<br>    androidPosesAction.current = mixer.current.clipAction(animations.find(c=&gt;c.name==&quot;shots&quot;)!).play();<br>    androidPosesAction.current.paused = true;<br><br>// ... rest of scene setup<br><br>}, []); //&lt;-- On mount ( initialization )</pre><h3>5. SYNC</h3><p>So now we have our html with sections, plain vanilla sections, and a threejs app running fixed on the screen under the html ( <em>the html content appears on top of the three js canvas</em> ) They are disconnected, time connect them.</p><p>This is where the bridge happens GSAP -&gt; progress array -&gt; THREE:</p><pre>// my main loop... ( I use this hook since I used react three fiber, but this is just a loop on every frame )<br>useFrame((_state, delta) =&gt; { <br>    // For each section&#39;s GSAP progres...<br>    for (let i = 0; i &lt; progress.length; i++) {<br>        const val = progress[i]; // GSAP progress of this section.<br>        if( val&gt;0 )<br>        { <br>            //<br>            // In my example, I had one camera animation clip, that has one keyframe pero feature/section spread uniformly <br>            // so the math is easy to map index with the time of each section inside of the clip.<br>            // ( did this in Blender ) <br>            //<br>            const step = ((cameraShotsClip.current!.duration ) / (features.length-1));<br>            if( i==0 )<br>            {<br>                continue;<br>            } <br><br>            //<br>            // generate a &quot;cursor&quot; inside of the camera&#39;s animation clip (that holds all the section&#39;s shots)<br>            //<br>            const cursor = step * (i-1);<br><br>            //<br>            // In my case I wanted to start the clip sooner to the first 30% are spent entering the shot.<br>            //<br>            const t = cursor + ( val&lt;0.3? (val/0.3)*step : step );<br><br>            //<br>            // *** MAGIC ALERT ***<br>            // here we scrub the animations!!<br>            // Since I did the same for the android, I just pass them both the same time.,<br>            //<br>            cameraShotAction.current!.time = t<br>            androidPosesAction.current!.time = t<br>        }<br>        <br>    } <br>});</pre><h3>That’s it!</h3><p>Now when you scroll, GSAP updates the number array that has the same length as the “features” / sections in your app. Then, THREE on every frame, reads that array and updates it’s animations accordingly ( <em>by setting the time of the animation vía code while the clip is paused</em> )</p><p>They are both now <a href="https://www.youtube.com/watch?v=Eo-KmOd3i7s">IN SYNC</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5lRPzGH9w_TWFCkwujMTsQ.png" /></figure><h3>Questions?</h3><p>Shoot them up <a href="https://x.com/bandinopla">@bandinopla</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a2be523e430a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Collision detection in ThreeJs made easy using BVH]]></title>
            <link>https://medium.com/@pablobandinopla/collision-detection-in-threejs-made-easy-using-bvh-1ce6012199e8?source=rss-52ab515439d2------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ce6012199e8</guid>
            <category><![CDATA[bvh]]></category>
            <category><![CDATA[threejs]]></category>
            <category><![CDATA[raycasting]]></category>
            <category><![CDATA[game-development]]></category>
            <dc:creator><![CDATA[Bandinopla]]></dc:creator>
            <pubDate>Mon, 25 Aug 2025 19:35:04 GMT</pubDate>
            <atom:updated>2025-09-15T11:05:40.397Z</atom:updated>
            <content:encoded><![CDATA[<p>So you just downloaded a very cool level model that you want to test with your character and walk on it like in a game without feeling like a ghost that can pass trough geometry. Is there an easy way to do this without having to spend hours reading documentation? Yes there is!</p><p>And that’s exactly what I did when I coded my <a href="https://x.com/bandinopla/status/1926428457883312490">“Silent Hill Demo”</a> . I found the school level in <a href="https://sketchfab.com/">Sketchfab</a> and Sherry’s model from Resident evil and wanted to combine the two. For the collision detection ( <em>with the floor, walls, tables, doors, etc…</em> ) I went the <a href="https://github.com/gkjohnson/three-mesh-bvh/">three-mesh-bvh</a> route.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*l4XVK5qGJDJeqbOYS0RlPA.png" /></figure><h3>What is BHV?</h3><blockquote>I always think that real world analogies are the best way to understand things, realistic scenarios, so, with that goal in mind, imagine <a href="https://www.youtube.com/watch?v=WRuxhO354W4">Ana de Armas </a>is celebrating her birthday, and since you are dating her, she obviously invites you over, but somehow <a href="https://www.youtube.com/watch?v=a9ZhleGV4Qc">Sydney Sweeney</a> finds her way into the party, and she is angry with Ana because she has a crush on you and can’t have you since you are already taken, so she decides to get close to the birthday cake and spits on it. Now, how can you know where the spit is on the cake so you avoid eating the entire thing just to find the portion with her spit? Eating the entire cake would be ludacris, so you visually see the grid formed by the slices and you only eat the slice with Sydney Sweeney’s spit on it. This optimization allows you to skip the entire cake and just focus on the important part. That’s what BVH does. It slices and picks one chunk only.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/298/0*81b9o0MuTzjJQenc.gif" /></figure><h3>Colliding by shooting rays</h3><blockquote><em>Tipically to get your character to “hit the floor” you would </em><a href="https://threejs.org/docs/#api/en/core/Raycaster"><em>“shoot a ray”</em></a><em> from the character’s origin in the downward direction, and detect if the floor mesh ( </em>or any other solid<em> ) was hit, in which case you will position the character at that position, giving the illusion of a solid object.</em></blockquote><p>When you shoot a ray to detect collisions, every ray cast into the scene would need to be tested against every single polygon. This brute-force approach quickly becomes impractical because the number of intersection tests grows linearly with the number of polygons.</p><h3>Ok, but… What the hell is BVH?</h3><p>It is a technique, called <strong>Bounding Volume Hierarchy</strong> ( <em>BVH</em> ), a way to sort and group faces in a spatial data structure, used in computer graphics, physics engines, and ray tracing to <em>accelerate intersection tests between rays and complex geometry</em>. “accelerate” as in: not wasting time.</p><p>Instead of checking every polygon in a scene individually (<em>which would be very slow</em>), BVH organizes geometry into a tree of nested bounding volumes ( <em>boxes…</em> ) that enclose groups of polygons.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PB15gRc8ikFPa6p7sajNnQ.png" /></figure><p>In the image above, for example, sherry’s raycast hit going downward will only analize the polygons under her and will ignore the objects and walls and any other area outside of that room.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6sX2WXTckVOVakPSgAOAfA.png" /></figure><p>The image above is from the <a href="https://gkjohnson.github.io/three-mesh-bvh/example/bundle/characterMovement.html">official demo</a>, you can easily see how this BVH mechanism subdivides a level.</p><h3>Ok fine, how do I implement BVH in my game/app ?</h3><p>First you have to understand that this technique requires sorting and grouping polygons that are meant to exist in a common mesh. So we will need to <strong>create a new mesh</strong> that will combine all the polygons in it to allow for this method to work. It will only be used for collision detection, it won’t be visible ( <em>unless you are debugging in which case you may chose to make this collider visible</em> )</p><h3>The setup</h3><pre>npm install three-mesh-bvh</pre><h3>Create the Level Collider</h3><h4>Collect all static meshes</h4><p>During your level/world initialization is usually where you will do this. We need to create a mesh that represents all the static objects in the scene, everything that should be considered a collidable thing. Scan your scene looking for objects that are meant to be static / collidables.</p><pre>const staticMeshes: THREE.Mesh[] = []; // we collect all &quot;static&quot; meshes<br>myLevel.traverse( child =&gt;{ <br>    if (child instanceof THREE.Mesh) {<br>        // check if this mesh should be considered static... <br>        // this is up to you then...<br>        if( child.name.includes(&quot;static&quot;) ) // let&#39;s say we test this...<br>        {<br>            staticMeshes.push( child ); // ok, it is a static collider...<br>        }<br>    }<br>})</pre><h4>Create the level mesh collider</h4><p>Once we have our static objects we are ready to create the “collider” ( <em>the mesh that will be used to implement BVH</em> ) It will combine all the static meshes into one big mesh.</p><p>This is done in 4 steps:</p><ol><li>Extract the geometries from the static meshes we collected earlier</li><li>merge them into one single geometry</li><li>prepare the geometry ( <em>optimize that geometry for bvh to work efficiently</em> )</li><li>Generate the BVH geometry ( <em>the object we will use to test collisions</em> )</li></ol><pre>import * as THREE from &#39;three&#39;;<br>import { MeshBVH, MeshBVHHelper, StaticGeometryGenerator } from &quot;three-mesh-bvh&quot;<br>import * as BufferGeometryUtils from &#39;three/examples/jsm/utils/BufferGeometryUtils.js&#39;;<br>//...<br><br>const environment = new THREE.Group();<br>const visualGeometries: THREE.BufferGeometry[] = [];<br><br>//<br>// 1) extract the geometries from all the static meshes<br>//<br>staticMeshes.forEach(mesh =&gt; {<br>    const geom = mesh.geometry.clone();<br><br>    //<br>    // Because each mesh’s geometry is defined in its local space, <br>    // but when you want to merge them into a single geometry <br>    // you need all their vertices in the same world space.<br>    //<br>    geom.applyMatrix4(mesh.matrixWorld);<br>    visualGeometries.push(geom);<br>});<br><br>//<br>// 2) Merge the geometries into one<br>//<br>const colliderGeometry = BufferGeometryUtils.mergeGeometries(visualGeometries);<br>const colliderMesh = new THREE.Mesh( colliderGeometry ); <br><br>environment.add( colliderMesh );<br><br>//<br>// 3) Prepare the data to be generated ( optimized version )<br>//<br>const staticGenerator = new StaticGeometryGenerator(environment);<br><br>// Tell it to only keep vertex positions (no normals, uvs, etc.) since those aren’t needed for collision/raycast checks.<br>staticGenerator.attributes = [&#39;position&#39;];<br><br><br>//<br>// 4) Generate the BVH geometry<br>//  <br>const mergedGeometry = staticGenerator.generate();<br>mergedGeometry.boundsTree = new MeshBVH(mergedGeometry);<br><br>//<br>// that&#39;s it!<br>//<br>const collider = new THREE.Mesh( mergedGeometry, <br><br>    // this material is only added as a way for you to debug. In case you want to see the mesh that was generated that will be used for testing collisions.<br>    new THREE.MeshBasicMaterial({ wireframe: true, opacity: .1, transparent: true })<br>);</pre><p>things you may be wondering:</p><ul><li>staticGenerator.generate() → produces a plain merged BufferGeometry ( <em>just vertices/indices in world space</em> ). At this point it’s just data, nothing “smart.”</li><li>new MeshBVH(mergedGeometry) → analyzes that geometry and builds the acceleration structure. Assigning it to mergedGeometry.boundsTree is how three-mesh-bvh hooks into Three.js’s raycasting system.</li></ul><h3>Create the player collider</h3><p>Now, the player and every entity you want to move and collide in this world will need also to have a setup like this.</p><p>Usually, to detect collision, the math is done using simple shapes. For characters, a Capsule is usually used because:</p><ul><li><strong>Smooth Movement Over Uneven Surfaces</strong>: The rounded bottom of the capsule allows the character to slide smoothly over small bumps, cracks, and up gentle slopes without getting stuck. A box collider, by contrast, has sharp corners that would catch on these small geometric details, leading to jerky, unnatural movement.</li><li><strong>Prevents Snagging on Corners</strong>: When moving past a corner or through a doorway, the rounded sides of a capsule allow it to slide along the wall smoothly. A box collider would often snag its corner on the wall’s corner, abruptly stopping the player’s movement.</li><li><strong>Good Approximation of a Character</strong>: For most upright characters ( <em>like people</em> ), a capsule is a much better and more realistic approximation of their overall shape than a simple sphere or a box. It’s tall and rounded, just like a person’s general silhouette.</li></ul><p>So for that reason, we first need to attach the information of the player capsule shape to the player:</p><pre>// you have to adjust this depending on your model&#39;s dimensions...  <br>myPlayer.userData.capsuleInfo = {<br>		radius: 0.5,<br>		segment: new THREE.Line3( <br>            new THREE.Vector3(), <br>            new THREE.Vector3( 0, - 1.0, 0.0 ) <br>        )<br>}</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*iCY6zb0pkW7R5Js_LbuD_Q.png" /></figure><h4>Atention!</h4><p>The segment start/end vector 3 positions <em>depends on how your character is setup</em>, and they are expressed in the <strong>object’s local space</strong>. For example, if you had your characters with their pivot point in their feet ( <em>at the bottom </em>) then the start.y position of the capsule would have to be radius + your desired height, and the end.y would be radius. So when we check for how close we are to a polygon, using the radius as minimal distance allowed, to position our character we would have to subtract radius from the end.y to get the final position.</p><h3>Colliding</h3><p>We have a mesh collider and our players have each their capsule info set ( <em>their collision shape</em> ) To start checking for collisions we have to do a few things.</p><p>This will usually happen in your update loop ( <em>if you have constant forces like gravity always pulling entities down</em> ) or anytime you want to move an entity.</p><p>But let’s say is that time in your code where you want to test for collisions. You will have to do this:</p><h4>1) Initialization and Space Transformation</h4><p>The first step is to set up the necessary variables and transform the player’s collision capsule into the <strong>local space of the collider object</strong>.</p><pre>// tempXXXX are top-level or global variables used to be re-used...<br><br>const capsuleInfo = myPlayer.userData.capsuleInfo; // or the current entity being tested... <br><br>// tempMat = THREE.Matrix4()<br>// `collider.matrixWorld` represents the collider&#39;s position and orientation in the WORLD. By inverting it we get a matrix that can transform coordinates from world space into the collider&#39;s LOCAL space<br>tempMat.copy( collider.matrixWorld ).invert();<br><br>tempSegment.copy( capsuleInfo.segment );<br><br>// tempSegment = THREE.Line3()<br>// get the position of the capsule in the local space of the collider<br>tempSegment.start.applyMatrix4( myPlayer.matrixWorld ).applyMatrix4( tempMat );<br>tempSegment.end.applyMatrix4( myPlayer.matrixWorld ).applyMatrix4( tempMat );</pre><h4>2) Creating a Bounding Box</h4><p>Next, an <a href="https://developer.mozilla.org/en-US/docs/Games/Techniques/3D_collision_detection">Axis-Aligned Bounding Box (AABB)</a> is created that fully encloses the player’s capsule. This box serves as a quick, initial check; if this box doesn’t intersect with a part of the environment, then the more complex capsule shape can’t either.</p><pre>// tempBox = new THREE.Box3();<br>tempBox.makeEmpty();<br><br>// get the axis aligned bounding box of the capsule<br>tempBox.expandByPoint( tempSegment.start );<br>tempBox.expandByPoint( tempSegment.end );<br><br>tempBox.min.addScalar( - capsuleInfo.radius );<br>tempBox.max.addScalar( capsuleInfo.radius );</pre><p>The box is first expanded to contain the capsule’s central line segment. Then, its minimum and maximum corners are pushed outwards by the capsule’s radius, ensuring the entire volume of the capsule is contained within the box.</p><h4>3) The BVH Shapecast</h4><p>This is the core of the collision detection process. The shapecast function from three-mesh-bvh efficiently checks the capsule&#39;s bounding box against the collider&#39;s BVH tree. It consists of 2 phases:</p><ol><li>the <strong>“broad phase”</strong> : It performs a fast box-to-box intersection test. If the player’s bounding box doesn’t intersect the BVH node’s box, the function returns false, and the entire branch of the geometry tree is skipped, saving immense computation.</li><li>the <strong>“narrow phase.”</strong> : If intersectsBounds returns true for a leaf node, this function is called for each triangle within that node. It performs the precise, more expensive check to see if the player’s capsule actually touches the triangle.</li></ol><pre>collider.geometry.boundsTree.shapecast( { <br>    intersectsBounds: box =&gt; box.intersectsBox( tempBox ), //&lt;-- tempBox is the bounding box containing our player/entity <br> <br>    intersectsTriangle: tri =&gt; { <br>        // (*)<br>    }} );</pre><h4>4) Collision Resolution Inside intersectsTriangle</h4><p>When a triangle is found to be intersecting the capsule, the code must calculate how to move the player to resolve the collision.</p><pre>// (*)<br>const triPoint = tempVector; // re-using vector: this will be filled by closestPointToSegment<br>const capsulePoint = tempVector2; // re-using vector: this will be filled by closestPointToSegment<br>// distance = shortest distance between the triangle (tri) and the capsule&#39;s central line<br>const distance = tri.closestPointToSegment( tempSegment, triPoint, capsulePoint );<br>if ( distance &lt; capsuleInfo.radius ) {<br><br>    // depth = how far the capsule has penetrated the triangle?<br>    const depth = capsuleInfo.radius - distance;<br>    // direction = the direction the player needs to be pushed to resolve the collision<br>    const direction = capsulePoint.sub( triPoint ).normalize();<br>    //<br>    // push the player/entity away from the collision point...<br>    //<br>    tempSegment.start.addScaledVector( direction, depth );<br>    tempSegment.end.addScaledVector( direction, depth );<br>}</pre><h4>5) Applying the Correction and Updating Player State</h4><p>After the shapecast has checked all potential triangles and adjusted the tempSegment, the final corrected position must be applied back to the actual player object in <strong>world space</strong>.</p><pre>// tempVector &amp; tempVector2 = new THREE.Vector3() <br>const newPosition = tempVector;<br>newPosition.copy( tempSegment.start ).applyMatrix4( collider.matrixWorld ); // local to world...<br><br>// The segmen has not be adjusted for pollision, we can now either use all<br>// the components (xyz) or just adjust Y... but REMEMBER: this is the SEGMENT of the capsule<br>// you have to add or subtract the radius depending on what end are you using and <br>// where is the pivot in your character or object.<br><br>// the total displacement from the player&#39;s original position to the new, collision-free position.<br>// const deltaVector = tempVector2;<br>// deltaVector.subVectors( newPosition, player.position );<br><br>// apply the push... <br>// player.position.add( deltaVector ); // here we are assuming player is in world&#39;s space.<br></pre><h4>6) Optional: Ground Check and Velocity Adjustment</h4><p>if the player was primarily adjusted vertically we assume it’s on something we should consider ground…</p><pre>//how much the character’s vertical coordinate has actually changed compared to how much it should have changed if the character were still in free-fall for the same length of time?<br>// The 0.25 is just a small fudge factor so we don’t demand an exact match—floating-point noise, terrain bumps, or physics solver tolerances make a perfect 0 unrealistic.<br>playerIsOnGround = deltaVector.y &gt; Math.abs( delta * playerVelocity.y * 0.25 );<br><br>// “zero-threshold re-normalization” trick.<br>// the code says: “Keep the intended direction, but throw away any sub-millimetre leftovers that could cause numerical trouble.”<br>const offset = Math.max( 0.0, deltaVector.length() - 1e-5 );<br>deltaVector.normalize().multiplyScalar( offset );<br><br>// adjust the player model<br>player.position.add( deltaVector );<br><br>if ( ! playerIsOnGround ) <br>{<br>        // using the direction of the corrected displacement (deltaVector) as the contact-normal:<br>	deltaVector.normalize();<br><br>        // Whatever kept me from moving further during this frame—floor, wall, or slope—take the velocity that was trying to carry me through it and strip that component away.<br>	playerVelocity.addScaledVector( deltaVector, - deltaVector.dot( playerVelocity ) );<br><br>} <br>else <br>{<br>	playerVelocity.set( 0, 0, 0 );<br>}</pre><p>So that’s it! We corrected the position and the velocity of the player/entity to respond to the level collider.</p><h3>Now go on and start using BVH!</h3><p>Check the <a href="https://github.com/gkjohnson/three-mesh-bvh">official documentation </a>because it has more uses than the one presented here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/176/0*xjBAyPrlSErboLTP.gif" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ce6012199e8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Effortless Serverless Multiplayer in Three.js with Trystero]]></title>
            <link>https://medium.com/@pablobandinopla/effortless-serverless-multiplayer-in-three-js-with-trystero-f025f31150c6?source=rss-52ab515439d2------2</link>
            <guid isPermaLink="false">https://medium.com/p/f025f31150c6</guid>
            <category><![CDATA[threejs]]></category>
            <category><![CDATA[multiplayer]]></category>
            <category><![CDATA[p2p]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[trystero]]></category>
            <dc:creator><![CDATA[Bandinopla]]></dc:creator>
            <pubDate>Sun, 24 Aug 2025 17:16:50 GMT</pubDate>
            <atom:updated>2025-08-24T17:38:28.927Z</atom:updated>
            <content:encoded><![CDATA[<p>Let’s say you want to do a quick prototype in <a href="https://threejs.org/">ThreeJs</a> and play with your friends, but you quickly realize how hard mutliplayer programming is :( Did you know that the multiplayer <strong>doesn’t have to be a headache</strong>? There’s a magical tool available to add multiplayer easy and free: <a href="https://github.com/dmotz/trystero">Trystero</a> to the rescue!</p><blockquote><a href="https://github.com/dmotz/trystero"><strong>Trystero</strong></a> is a minimalist, open-source JavaScript library (MIT-licensed) that empowers you to build multiplayer web apps without needing your own server infrastructure. It handles peer matching using WebRTC and various decentralized signaling backends — like BitTorrent, Nostr, MQTT, IPFS, Supabase, and Firebase — so all your app data flows directly between clients, fully encrypted end-to-end. Trystero also offers developer-friendly abstractions like rooms, event broadcasting, data chunking, progress tracking, and even React hooks — making it a smooth fit for creating serverless multiplayer experiences with <a href="https://threejs.org/">Three.js</a></blockquote><p>And that’s exactly what I used for <a href="https://bandinopla-trystero.web.app/">my multiplayer tech demo</a>! which I was able to code in just a day thanks to how Trystero makes it!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rZ2TqAQQenuAk85x.jpg" /><figcaption>multiplayer in the brower, P2P, no Server, free.</figcaption></figure><h3>A real world example</h3><p>Instead of explaining what <a href="https://oxism.com/trystero/">trystero is</a>, I’ll show you how I used it to make a multiplayer demo, a real use-case. You can refer to <a href="https://github.com/dmotz/trystero">their documentation</a> for the details.</p><p>The demo I used is <a href="https://bandinopla-trystero.web.app/">here.</a> For this demo I automatically put a new player in a queue looking for a free room, let him/her interact and log them off after a few seconds and repeat the process. This is how I did it:</p><h4>Work in layers!</h4><p>So, before doing multiplayer stuff, you may first want to <em>start with single player mindset</em>, but in a way that the player’s input is not tied to the entity being moved. This means, always code first separating the logic for input from the logic for manipulating the avatar/entity. <strong>What does it means? </strong>It means, your entity should have an interface with the methods you want to be “controllable” clearly exposed and public. Think of your players or entities as a control board with buttons. You don’t care who press the buttons, a finger, a cat or a stick, you just care about what you do once a button is pressed.</p><pre>interface IEntity {<br>   walkTo(targetPosition: Vector3, from?: Vector3):void<br>   sayHi():void<br>   wakeup(playerId: string):void;<br>   // etc...<br>}</pre><p>That way, your game entities hace no idea of what input mechanic is used, they will just respond to their interface. And internally, you may use a <a href="https://en.wikipedia.org/wiki/Finite-state_machine">simple state machine</a> to handle the states, transitions, etc. The entity is input agnostic. And that’s what will make hooking trystero super easy.</p><h4>The setup</h4><p>At this point, we assume you have a “single player application”, and controllable entities implement some interface of your choosing. You did all the testing to make sure they work ( <em>like for example, you can do a setTimeout and call entity.walkTo and see if it moves well and stops at destination, etc </em>) . Now you want to “hook” multiplayer capabilities!</p><p>So you install trystero:</p><pre>npm install trystero</pre><blockquote>Players connect directly through P2P with no server in between. But before that connection can happen, they need a way to discover each other. That’s where Trystero comes in, offering <a href="https://github.com/dmotz/trystero?tab=readme-ov-file#get-started">multiple signaling options.</a> I chose the <a href="https://firebase.google.com/">Firebase</a> strategy since it felt simpler and more reliable for a quick demo.</blockquote><h4>The room</h4><p>In Trystero, everything happens inside a “room.” You can think of it like a shared stage where all the players gather , every move, action, or event takes place within that space, and everyone present instantly sees what the others are doing.</p><pre>import { joinRoom } from &#39;trystero/firebase&#39; // because I chose Firebase...<br>//...<br>const serverConfig = { appId: &quot;https://XXXXXXX.firebaseio.com&quot; }; // because I chose Firebase strategy<br>const room = joinRoom( serverConfig, &quot;mySuperFunRoom&quot;);</pre><p><strong>How easy was that</strong>? “room” now is our interface to “the outside world”, it is what we will be using to “talk to the others” in the room.</p><h4>Actions and Listeners</h4><p>So we have a reference to a room. Now what? Well, logically, one would ask “what should I do now?” and that’s exactly what the room is expecting for you to do! Tell the room what can be done, and, if something is done, what to do in response? We do that like this:</p><p>These are the real hooks I used in the demo, a copy paste of them:</p><pre>// these are extracted from my demo app to give a real life example:<br>const [askStatus, onSomeoneAskStatus] = this.room.makeAction&lt;PeerId&gt;(&quot;askstatus&quot;);<br>const [sendStatusOf, onStatusOf] = this.room.makeAction&lt;PlayerStatus&gt;(&quot;statusof&quot;); <br>const [sendPlayerMoved, onSomeoneMoved] = this.room.makeAction&lt;MoveCommand&gt;(&quot;move&quot;);<br>const [sendChat, onChat] = this.room.makeAction&lt;string&gt;(&quot;chat&quot;);<br>const [sendHi, onPlayerHi] = this.room.makeAction&lt;null&gt;(&quot;hi&quot;);</pre><p>Actions and response to actions are handled similar to <a href="https://legacy.reactjs.org/docs/hooks-intro.html">react hooks</a>. You call “<em>room.makeAction</em>” and it will return a <a href="https://en.wikipedia.org/wiki/Tuple">tuple</a> where the first element is the function that will trigger the action, and the second one is a function you will use to <em>subscribe a callback to</em> in case of that action being triggered by someone ( <em>trystero will execute that callback when someone else, not you, triggers that action, because you already know when you do it, obviously</em>)</p><h4>A player has entered the room!</h4><p>So we have an instance of our room, we are connected, and defined what can be done and how to respond to actions… how do we know when someone enters the room?</p><p>Meet: <em>room.onPeerJoin(…)</em></p><pre>//<br>// This will be called when someone joins the room<br>//<br>this.room.onPeerJoin(peerId =&gt; {   <br><br>    // instantiate a player and put it in &quot;limbo&quot;<br>    this.game.spawnPlayer( peerId, false );<br><br>    // get data of that player.... so we call the action &quot;askStatus&quot;<br>    askStatus( peerId, peerId ); <br>});  </pre><h4>Player joined, let’s setup!</h4><p>In my demo, I did 2 things in that callback. I spawn the peer’s avatar (<em>his player mesh</em>) and then immediately after, I call a room function “askStatus” because someone else might know the player ( <em>think of a player disconnecting due to his cat jumping on the keyboard and turning the PC off, he will lose connection, but other players will hold information of where he was, his items, etc… so just in case, we ask everyone in the room “hey, someone knows me?”</em>)</p><pre>  onSomeoneAskStatus( async (target, asker) =&gt; {<br><br>      if( target!=selfId ) return; // only the target can give the status, since each player is it&#39;s own source of truth<br><br>      // do I know the status of this target??<br>      const player = this.game.getPlayer( target );<br><br>      // I used &quot;inLimbo&quot; flag as a way to know if  player is still not active.<br>      if( player &amp;&amp; !player.inLimbo )<br>      { <br>          sendStatusOf( this.packPlayerStatus( player ) , asker);<br>      } <br><br>      // else, it means we are not ready... ignore. <br>  });</pre><p>So a player enters the room, I call askStatus and then on the listener for that action I check if we have info of that player, and if we do, we call the function “sendStatusOf” that will send the status of it…</p><pre>//<br>// we got the status of a player<br>//<br>onStatusOf( (status, by)=&gt; <br>{<br>    let player = this.game.getPlayer( status[0] );<br><br>    if( !player ) return;<br><br>    if( player.inLimbo )<br>    {<br>        this.game.writeChat(`${status[4]} joined the room.`);<br>    }<br>    // here we bridge the trystero world with our world...<br>    this.game.updatePlayerStatus( status[0], {<br>        position: new Vector3( status[1], status[2], status[3] ),<br>        username: status[4]<br>    })<br>} ); </pre><h4>The “schizophrenic” mindset</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UinpS8VWH-tCioy7kHWLxA.png" /></figure><p>Working with Trystero often requires a kind of “schizophrenic” mindset as a developer. In the same block of code, you’ll both <strong>emit actions</strong> and immediately need to <strong>handle those same actions</strong> as if they came from another client. It’s a strange dual perspective — you’re simultaneously the <strong>sender</strong> and the <strong>receiver</strong>, pretending you’re two different people using the app. This way of thinking is essential, because in a <a href="https://en.wikipedia.org/wiki/Peer-to-peer">peer-to-peer</a> system there’s no central authority to process events for you. Every action you fire off has to be treated as though it were broadcast by someone else, ensuring your local state stays in sync with the networked state.</p><h4>The bridge</h4><p>When building with Trystero, you quickly realize it doesn’t magically understand your app’s world — it only moves raw data between peers. That means you, as the developer, have to write the <strong>bridge</strong>: the layer that listens to Trystero events and translates them into meaningful changes in your scene. If a packet says a player moved or fired, your bridge code is what takes that data and updates your Three.js objects accordingly. In other words, Trystero gives you the network plumbing, but you’re responsible for mapping those signals into actions that make sense inside your 3D world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/1*kzAOOj4lnyPcRUJmRX4NHA.png" /><figcaption>triggering the “hi” AnimationClip when .hi() is called</figcaption></figure><pre><br>// bridge trystero&#39;s room action call to our ThreeJs scene<br>onPlayerHi( (ev, who)=&gt;{<br><br>    // Trystero told us that a player is saying hi, so we look that player<br>    // in our scene, and trigger it&#39;s .hi() method!<br>    let sender = this.game.getPlayer(who);<br>    if( sender )<br>    { <br>        sender.hi();<br>    } <br>});</pre><p>It’s a dance between Trystero’s calls and calling our interface. Since we initially coded our game in a way that is input agnostic, it becomes almost trivial to hook trystero into it.</p><h4>Player left the room</h4><p>When a player leaves a room, trystero will call <em>room.onPeerLeave</em></p><pre>  this.room.onPeerLeave( peerId =&gt; {<br><br>      // look for the player with that ID on our ThreeJs scene...<br>      let player = this.game.getPlayer(peerId);<br>      if(!player) return;<br><br>      // show the classic text in the chat&#39;s ui<br>      this.game.writeChat(`${player.username} left the room.`);<br><br>      // remove the player from our scene...<br>      this.game.removePlayer( peerId ); <br><br>  });</pre><p>That’s where you will want to cleanup, return entities to pools, or dispose of them. Any entity representing that player should be removed.</p><h3>Conclusion</h3><p><a href="https://oxism.com/trystero/">Trystero</a> makes multiplayer in the browser feel almost effortless. By handling the heavy lifting of peer-to-peer networking, it removes the need for servers, accounts, or complex backend setups — freeing you to focus entirely on your 3D experience. Combined with <a href="https://threejs.org/">Three.js</a>, it becomes an ideal toolkit for rapid prototyping: you can spin up real-time, collaborative scenes in just a few lines of code, without worrying about technical hurdles or ongoing server costs. Whether you’re experimenting with new gameplay ideas, building interactive art, or testing multiplayer mechanics, Trystero gives you the simplest path to bring your concepts to life.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f025f31150c6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generating Texture Atlases for Optimized Assets]]></title>
            <link>https://medium.com/@pablobandinopla/generating-texture-atlases-for-optimized-assets-63fd7a04021f?source=rss-52ab515439d2------2</link>
            <guid isPermaLink="false">https://medium.com/p/63fd7a04021f</guid>
            <category><![CDATA[games]]></category>
            <category><![CDATA[game-development]]></category>
            <category><![CDATA[threejs]]></category>
            <category><![CDATA[performance-optimization]]></category>
            <category><![CDATA[optimization]]></category>
            <dc:creator><![CDATA[Bandinopla]]></dc:creator>
            <pubDate>Sat, 23 Aug 2025 16:47:33 GMT</pubDate>
            <atom:updated>2025-08-23T23:07:17.515Z</atom:updated>
            <content:encoded><![CDATA[<p>When working with real-time graphics, reducing texture swaps is a huge performance win. One way to achieve this is through <strong>texture atlases</strong>: combining many small textures into a single larger one. Instead of binding dozens of individual textures during rendering, the GPU only needs to work with one atlas, <strong>with UVs mapped to the correct tile</strong>.</p><p>This is something I did when developing my <a href="https://threejs.org/">ThreeJs</a> browser game <a href="https://bandinopla.github.io/pip-skull-demo/">PIP:Skull Demo</a> and will use it as example to showcase this technique.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p8HQL6lGQHxYECLV0MIlKw.png" /><figcaption>in-game capture</figcaption></figure><h3>Pre-Build process</h3><p>When developing we can work with non optimized assets and allow each model to have it’s own texture. But once we are ready for production, we may want to optimize. This involves a pre-build step ( <em>a script you will run before running your actual build script </em>) that will read the original files and write new optimized ones ( <em>you may chose to replace the original glb files with the new one or just tell your build script to read the optimized ones and ignore the development ones</em> ). For this article we will focus on GLB files, a very flexible format. We’ll assume all your game/application assets are in glb format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/1*cImWxFCcL_d0uLj2NhFuFA.png" /><figcaption>1 image atlas with 2 textures ( Boss + Weapon )</figcaption></figure><h4>Reading and writing new optimized glb files</h4><p>For this, we will install the following package that will provide us with many useful tools for optimization:</p><pre>npm install --save @gltf-transform/core @gltf-transform/extensions @gltf-transform/functions sharp</pre><blockquote><a href="https://gltf-transform.dev/">gltf-transform</a> is a powerful toolkit for processing and optimizing glTF files, making it ideal for a pre-build pipeline. Instead of manually tweaking assets, you can script transformations like texture compression, mesh simplification, deduplication, and pruning unused data. By integrating it into a pre-build step, <em>the original glTF remains untouched</em> while the script generates a new, optimized version ready for runtime. This ensures a clean workflow where source assets stay editable, and the final exported models are lightweight, faster to load, and tailored for the target platform.</blockquote><p>I wrote a small pre-build script to automate this process (<em>and you will have to write one for your own project too, each project is different and needs different operations </em>) and a `createAtlas` function to allow me to create texture atlases for different groups of textures ( <em>This depends on your personal organization choice and it is arbitrary </em>) .</p><p>The function takes a list of textures, resizes them to a defined tile size, and packs them into a single atlas according to a grid layout. Using <a href="https://www.npmjs.com/package/sharp">sharp</a>, it composites everything efficiently and can also save the result as a .png for inspection. Alongside the atlas image, it returns metadata about tile positions, making it easy to remap assets later in shaders or materials.</p><pre>/**<br> * Utility class to create texture atlases. You can use this to tell it to <br> * pack a bunch of textures into a single atlas.<br> * <br> * @param {string} atlasName <br> * @param {Texture[]} textures <br> * @param {[number, number]} atlasSize Numbers representing how many tileSize on the width and height. Examlple: 2 = 2 times the tileSize in width.<br> * @param {[number, number]} tileSize Each tile in the atlas will have this size.<br> * @param {{ name:string, top:number, left:number }[]} tiles <br> * @param {boolean} savePngToDisk<br> */<br>async function createAtlas( atlasName, textures, atlasSize, tileSize, tiles, savePngToDisk )<br>{ <br>    const atlasTextures = tiles.map( tile=&gt;textures.find(t=&gt;t.getName()==tile.name )  );<br><br>    const resizedImages = await Promise.all( <br>        <br>        //filer only the textures we are going to pack <br>        atlasTextures<br><br>        .map(async tex =&gt; await sharp(Buffer.from(tex.getImage()))<br>                .resize({ width: tileSize[0], height: tileSize[1], fit: &#39;contain&#39;, background: { r: 0, g: 0, b: 0, alpha: 0 } })<br>                .toFormat(&#39;png&#39;) //&lt;-- you may change this to webp...<br>                .toBuffer()<br>            )<br>    );<br><br>    // Create atlas<br>    const atlas = await sharp({<br>        create: { <br>            width: atlasSize[0]*tileSize[0], <br>            height: atlasSize[1]*tileSize[1], <br>            channels: 4, <br>            background: { r: 0, g: 0, b: 0, alpha: 0 } <br>        }<br>    }).composite(<br>        resizedImages.map((img, index) =&gt; ({<br>            input: img,<br>            left: tiles[index].left * tileSize[0],<br>            top: tiles[index].top * tileSize[1]<br>        }))<br>    ).png().toBuffer();<br><br>    if( savePngToDisk ) // for debugging. It will be packed in the GLB otherwise<br>        await fs.writeFile(`./${atlasName}.png`, atlas); <br><br>    return {<br>        imageData: atlas,<br>        tiles,<br>        atlasSize,<br>        removeUnusedTextures() {<br>            atlasTextures.forEach( textureToRemove =&gt; { <br><br>                textureToRemove.detach();       // Unlink from graph<br>                textureToRemove.setImage(null); // Clear binary data<br>                textureToRemove.dispose();      // Mark for removal<br>                <br>            });<br>        }<br>    };<br>} </pre><p>A nice extra is the removeUnusedTextures() helper. Once packed, the source textures can be safely detached ( <em>and should</em> ) and disposed of, keeping memory usage clean and ensuring only the optimized atlas is used in runtime (<em> so the new glb file doesn’t hold the old textures in it </em>)</p><h4>+ Add the atlas to the document</h4><p>This is how we now insert the atlas texture into the GLB file:</p><pre>const targetTextures = myDoc.getRoot().listTextures().filter(...);<br>const myAtlas = createAtlas(...); // call the utility function to create your atlas...<br>// This adds the atlas image into the glb<br>const myAtlasTexture = myDocument.createTexture(&#39;MyAtlas&#39;).setImage(myAtlas.imageData).setMimeType(&#39;image/png&#39;);</pre><p>By running this step during the build process, I can ship <strong>fewer files, smaller GPU state changes, and more predictable asset management</strong> — all of which translate into smoother rendering and faster load times.</p><h3>But what about the UVs? Doesn’t that break the models?</h3><p>Yes. Bye. The article is done...</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*De0a5lshfgKNyQq2K4_Z3w.png" /><figcaption>boss and drill both have their own textures while developing</figcaption></figure><p>Joking, we will update the UVs obviously! And this can be done automatically too! Check this out: The above’s utility function was used several times to pack different textures into one ( <em>this is arbitrary, depends on your use case, you may want to group them by level, zone, etc.. it’s on you </em>). So we have now groups of textures packed into single images. The next thing we must do is scan the glb file looking for any material that may be using a texture that we have packed, and, if so, we should update the material and UVs to point to the atlas. This is how that is done…</p><h4>Scanning for materials</h4><pre>// search for all the materials in the glb file...<br>myAssetDocument.getRoot().listMaterials().forEach((material) =&gt; {<br>    //... code here<br>})</pre><p>With a reference to the <a href="https://gltf-transform.dev/modules/core/classes/Document">document</a> of our glb file, we will loop each of it’s materials and check if that material exists in our atlas, like so:</p><pre>// in this example we use the color texture, but you may want to scan for <br>// the material.getMetallicRoughnessTexture() and/or <br>// the material.getNormalTexture()<br>const colorTexture = material.getBaseColorTexture()<br>const name = colorTexture.getName();<br>const atlasUvTile = atlas.tiles.find( tile=&gt;tile.name==name);<br>if( atlasUvTile ) {<br>  // This material must be updated!! <br>  material.setBaseColorTexture( myAtlasTexture ); //&lt;-- here myAtlasTexture is a reference to the Texture created by myDoc.createTexture(...)<br>  // and here you can call a function to scan the doc looking for meshes using this material...<br>  // (*)<br>}</pre><p>In the above’s code `atlas` is the object returned by our `createAtlas` utility function. If any of the tiles in that atlas has the same name of the current texture being scanned, then that’s our sign to start doing the replacement!</p><h4>Updating the UVs</h4><p>When multiple textures are combined into a single atlas, the geometry using those textures must also be updated. Each mesh originally referenced its own texture coordinates (UVs) aligned to an individual image ( <em>0 to 1 related to the single image they were using, now 0 to 1 would refer to the total width and height of the atlas containing a bunch of textures in it </em>). After packing textures into a single large atlas, the UVs need to be <strong>scaled and offset</strong> so that they now correctly point to their assigned tile within the atlas.</p><p>We have the `atlasUvTile` which holds the necessary information to do the correct adjustments. Now we have to find all the meshes that are using this texture, like so, scanning our<a href="https://gltf-transform.dev/modules/core/classes/Document"> document</a> for meshes:</p><pre>// (*)<br><br>const w = atlas.atlasSize[0];<br>const h = atlas.atlasSize[1];<br>const tx = atlasUvTile.left;<br>const ty = atlasUvTile.top;<br><br>myAssetDocument.getRoot().listMeshes().forEach(mesh =&gt; {<br>    mesh.listPrimitives().forEach(prim =&gt; {<br>        if (prim.getMaterial() === material) <br>        {<br>            const uvs = prim.getAttribute(&#39;TEXCOORD_0&#39;);<br>            if (uvs) <br>            {<br>                const accessor = uvs.getArray();<br>                for (let i = 0; i &lt; accessor.length; i += 2) <br>                {<br>                    accessor[i] = accessor[i] / w + tx / w; // scale then offset...<br>                    accessor[i + 1] = accessor[i+1] / h + ty / h;<br>                }<br>                uvs.setArray(accessor);<br>            }<br>        }<br>    });<br>});</pre><p>Once a material is confirmed to use the atlas, the code finds every mesh and primitive using that material. For each primitive, it accesses the TEXCOORD_0 attribute (the UV data). The UVs are then modified in place by applying a <strong>scale and offset: </strong>scale shrinks the UVs to fit the size of a single tile inside the larger atlas and offset shifts them to the correct position (tx, ty) inside the grid layout of the atlas.</p><p>In practice, this step ensures that every mesh in the optimized glTF correctly maps its surfaces to the right part of the atlas image. Without it, textures would appear scrambled because the original UVs would no longer match the combined atlas layout.</p><p>The result is an optimized glTF where <strong>multiple textures are merged into fewer images</strong>, reducing texture bindings at runtime while preserving the original visual fidelity of the model. This is an essential optimization step for real-time rendering, especially in games and web applications.</p><p>After all that you may want to run this to remove the now unused textures from the new final optimized glb:</p><pre>myAtlas.removeUnusedTextures() </pre><h3>Caveat: Non-Tiled Textures Only</h3><p>This UV remapping approach works correctly only for <strong>non-tiled textures</strong>. Since the UVs are scaled down and offset to fit within a single cell of the atlas, any material that relies on <strong>texture tiling or repeating patterns</strong> will break — the repetition will be clipped to the bounds of the assigned tile. For assets that depend on seamless tiling (like bricks, floors, or fabrics), this method isn’t suitable without additional handling. In those cases, it’s often better to keep the textures separate or explore more advanced atlas techniques.</p><h3>That’s it!</h3><p>By integrating texture atlases into your pre-build process, you not only streamline asset management but also significantly reduce the workload on the client’s computer. Fewer texture bindings mean less GPU overhead, faster load times, and smoother rendering. In the end, this optimization helps you deliver leaner, more efficient applications that provide a better experience for your users.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=63fd7a04021f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Raymarching: A Better Way to Explore NASA’s X-Ray Scans with Three.js]]></title>
            <link>https://medium.com/@pablobandinopla/raymarching-a-better-way-to-explore-nasas-x-ray-scans-with-threejs-a114a6bee9e7?source=rss-52ab515439d2------2</link>
            <guid isPermaLink="false">https://medium.com/p/a114a6bee9e7</guid>
            <category><![CDATA[nasa]]></category>
            <category><![CDATA[raymarching]]></category>
            <category><![CDATA[threejs]]></category>
            <category><![CDATA[3d]]></category>
            <category><![CDATA[shaders]]></category>
            <dc:creator><![CDATA[Bandinopla]]></dc:creator>
            <pubDate>Fri, 22 Aug 2025 23:00:37 GMT</pubDate>
            <atom:updated>2025-08-27T18:31:47.420Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0ZfSBK--4755gb34b5Od2A.png" /><figcaption><a href="https://bandinopla-astromaterials-3d.vercel.app/">https://bandinopla-astromaterials-3d.vercel.app/</a></figcaption></figure><p>Recently I experimented with NASA’s <a href="https://ares.jsc.nasa.gov/astromaterials3d/"><strong>Astromaterials 3D virtual library</strong></a>, a collection that provides access to digital scans of Apollo samples, meteorites, and Antarctic rocks. Instead of just viewing static models, I wanted to explore their inner structures in real time, not just the way a geologist might examine a thin slice under a microscope.</p><p>To achieve this, I built a <a href="https://bandinopla-astromaterials-3d.vercel.app/"><strong>real-time volumetric data visualization tool</strong></a> using <strong>raymarching</strong>. This technique allowed me to render NASA’s <strong>X-ray computed tomography (CT) data</strong> directly into interactive 3D volumes. Instead of pre-baked meshes, the rocks are reconstructed on the fly, voxel by voxel, giving a much richer sense of depth and material composition.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FPz_lYFEMw0Y%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DPz_lYFEMw0Y&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FPz_lYFEMw0Y%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2c66f0aba49bacea2feae4aadecf90d/href">https://medium.com/media/e2c66f0aba49bacea2feae4aadecf90d/href</a></iframe><p>One key feature I implemented was an <strong>intensity-based filtering system</strong>. With it, you can interactively isolate layers or internal structures based on density values. For example, you can peel away “less dense” material from a meteorite and reveal inclusions or harder structures hidden inside. This makes the exploration process feel less like looking at a 3D model and more like conducting an actual virtual dissection of the sample.</p><p>The result is a tool that not only visualizes NASA’s CT data but also encourages a more intuitive and exploratory way of studying planetary materials. While my project is just a personal take on their library, I think it highlights how modern real-time rendering techniques can make complex scientific datasets more accessible and engaging.</p><p>Now let’s talk about raymarching:</p><h3>What is Ray Marching?</h3><p><a href="https://en.wikipedia.org/wiki/Ray_marching">Raymarching</a> is a way of drawing 3D scenes on a computer without using traditional 3D models (no polygons or vertices involved)</p><p>Instead of having meshes made of triangles, you imagine shooting a ray (like a straight line of sight) from the camera into the scene. The ray moves forward step by step (“marching”) and at each step it asks: <em>am I inside an object yet? how close am I to the nearest surface?</em></p><p>This is usually done with <a href="https://en.wikipedia.org/wiki/Signed_distance_function"><strong>distance functions</strong></a>, which can tell you how far you are from a shape (like a sphere or cube). The ray keeps marching until it gets close enough to a surface, and then you know that’s where the object is.</p><p>It’s like sonar or echolocation, but in reverse: instead of sound bouncing back, you walk forward in little steps until you bump into something. That’s how the computer figures out what’s visible in that direction.</p><h3>But wait, How to combine X Ray slices with Ray Marching?</h3><p>The process started with a simple setup in <a href="https://threejs.org/"><strong>Three.js</strong></a>: a 1x1x1 cube mesh with a custom shader material. The cube serves as the container for the volumetric data, while the fragment shader handles the heavy lifting of the raymarching process.</p><p>Here’s how it works:</p><ol><li><strong>Camera direction &amp; contact point</strong> — Using the camera’s direction and the UV coordinates provided when the fragment shader is triggered, I calculated the local position inside the cube where the ray enters.</li><li><strong>Mapping slices to planes</strong> — NASA provides CT scan data as image slices across three planes: <strong>XZ, XY, and YZ</strong>. By projecting the local position of the ray step onto each plane, I could determine the corresponding UV for each slice atlas.</li><li><strong>Advancing the ray</strong> — As the ray marched through the volume, I sampled these slice images. Black areas were ignored (empty space), while grey or white pixels indicated material density. When the ray hit a non-empty voxel, it stopped — effectively reconstructing the surface or internal feature of the rock.</li><li>This method transforms stacks of 2D CT slices into a <strong>real-time volumetric visualization</strong>. Instead of static meshes, the rock’s structure emerges dynamically, with the raymarching process guided by the X-ray data itself.</li></ol><p>The end result is an interactive way to <strong>peer inside NASA’s astromaterials</strong>, layer by layer, revealing veins, inclusions, and hidden textures that would otherwise remain locked within the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ha9Zamdypx_3KCj67DosGw.png" /></figure><h3>How is the X Ray data packed &amp; read?</h3><p>NASA’s provides an atlas for each face of the bounding box cube that wraps the sample rocks: XY, XZ and YZ like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zodfPRoCpZNAm_TvjiKAiw.png" /></figure><p>And to read each frame in that atlas, you would write a function like this in <a href="https://developer.mozilla.org/en-US/docs/Games/Techniques/3D_on_the_web/GLSL_Shaders">GLSL shader code</a> ( using a ShaderMaterial in theejs) Each frame in the atlas represents a layer. Think of it as cutting many tiny slices of a sweet apple pie…</p><pre>vec3 sampleAtlas(sampler2D atlas, vec4 info, vec2 uv, float slicePercent) <br>{ <br>    // Ensure we&#39;re in [0, 1)<br>    float t = clamp(slicePercent, 0.0, 1.0);<br><br>    float totalFrames = info.w;<br>    float framesOnWidth = info.z;<br><br>    // Compute frame index - avoid int operations when possible<br>    float sliceIndexFloat = floor(t * totalFrames);<br>    sliceIndexFloat = min(sliceIndexFloat, totalFrames - 1.0);<br><br>    // Use mod for column calculation (GPU optimized)<br>    float column = mod(sliceIndexFloat, framesOnWidth);<br>    float row = floor(sliceIndexFloat / framesOnWidth);<br> <br>    // Compute UV in atlas - vectorize operations<br>    vec2 frameOffset = vec2(column, row);<br>    vec2 sampleUV = vec2(<br>        info.x * (frameOffset.x + uv.x),<br>        1.0 - info.y * (frameOffset.y + uv.y)<br>    );<br><br>    return texture(atlas, sampleUV).rgb;<br>}</pre><p>Since the base mesh is a cube, the ray position at any given time, being a vector 3, can be used to obtain the UV in each face of the cube since we intentionally kept the cube at 1x1x1 units. So the position itself is the UV on that face.</p><ul><li>Atlas XY would have each frame belonging to the Z axis.</li><li>Atlas XZ would have each frame belonging to the Y axis.</li><li>Atlas YZ would have each frame belonging to the X axis.</li></ul><p>this way we can easily know what polor to use at any given point of the ray.</p><h3>Optimizing Raymarching with Early Exit Based on Dominant Axis</h3><p>While building my real-time volumetric renderer for NASA’s CT scan data, one of the biggest challenges was <strong>performance</strong>. Raymarching is expensive by nature, especially when sampling multiple 2D atlases per step (XY, XZ, YZ). To keep things interactive in the browser, I experimented with ways to cut down unnecessary texture lookups.</p><h4>The Problem</h4><p>At every ray step, I needed to sample <strong>three different slice atlases</strong> (XY, XZ, YZ) to reconstruct the density at that position. This was accurate, but costly , especially when rays were aligned close to one axis. In those cases, many of the extra samples contributed little to the final result.</p><h4>The Optimization</h4><p>The trick was to look at the <strong>dominant axis of the ray direction</strong>. If the ray is traveling almost entirely along one axis, then the majority of meaningful data will come from the corresponding atlas.</p><p>So instead of always sampling all three, I added an <strong>early exit</strong>:</p><ul><li>Check which axis has the highest weight (the strongest contribution).</li><li>If one axis dominates (greater than 0.8 in my case), only sample that atlas.</li><li>Otherwise, fall back to sampling all three.</li></ul><p>This way, rays aligned with X, Y, or Z take a big shortcut, while oblique angles still get the full accuracy.</p><h4>The Result</h4><p>By cutting down the number of texture lookups in these dominant cases, the shader runs significantly faster without noticeable visual loss. It’s a simple heuristic, but it has a big payoff: many rays in typical viewing angles align with a major axis, so this optimization kicks in often.</p><p>In other words, <strong>don’t do the work if the math tells you it won’t matter much.</strong></p><pre>vec3 sampleVolumeOptimized(vec3 uvs, vec3 weight) {<br>    // Find dominant axis to reduce texture samples<br>    float maxWeight = max(weight.x, max(weight.y, weight.z));<br>    <br>    // If one axis dominates heavily, only sample that texture<br>    if (maxWeight &gt; 0.8) {<br>        if (weight.z &gt; 0.8) {<br>            return sampleAtlas(xyAtlas, xyInfo, uvs.xy, uvs.z);<br>        } else if (weight.y &gt; 0.8) {<br>            return sampleAtlas(xzAtlas, xzInfo, vec2(uvs.x, 1.0-uvs.z), uvs.y);<br>        } else if (weight.x &gt; 0.8) {<br>            return sampleAtlas(yzAtlas, yzInfo, vec2(uvs.y, 1.0-uvs.z), uvs.x);<br>        }<br>    }<br>    <br>    // For oblique angles, sample all three (fallback)<br>    vec3 xyPixel = sampleAtlas(xyAtlas, xyInfo, uvs.xy, uvs.z);<br>    vec3 xzPixel = sampleAtlas(xzAtlas, xzInfo, vec2(uvs.x, 1.0-uvs.z), uvs.y);<br>    vec3 yzPixel = sampleAtlas(yzAtlas, yzInfo, vec2(uvs.y, 1.0-uvs.z), uvs.x);<br> <br>    <br>    return xyPixel + xzPixel * weight.y + yzPixel * weight.x;<br>}</pre><p>This optimization is just one of many small tricks that make real-time raymarching practical in the browser. By letting the math guide where to spend GPU effort , and where to cut corners, it’s possible to turn heavy scientific datasets like NASA’s CT scans into smooth, interactive visualizations. It’s a reminder that performance often comes not from doing <em>more</em>, but from knowing when you can safely do <em>less</em>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a114a6bee9e7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>