TL;DR: Worklets 0.8.0 introduces the Shareable — a runtime-bound shared memory primitive that powers Reanimated’s Shared Values under the hood. Now it’s available as a lean standalone building block for your own use cases.

Sharing JavaScript memory in React Native

Sharing memory between JavaScript Runtimes in React Native is difficult. Not only do we have to overcome the limitations of the single-threaded paradigm of JavaScript, but also cover the API needs of typical React Native apps.

Today we present to you the Shareable, the last missing shared memory data type in React Native Worklets. But before we dive into the Shareable itself, let’s take a look at what exactly the other data types are.

3 types of shared memory

Since every JavaScript Runtime has its own memory space, which by design can’t be accessed from another Runtime’s memory space, we must add intermediate steps, with C++ and JSI — JavaScript Native Interface — that will create a bridge connecting these memory spaces. That way we can create an illusion that even though the memory spaces are totally disjoint, there’s some kind of link between them

We can distinguish three different types of shared memory, based on their use case:

Copy-only memory

The simplest of them all, a way to move any value from Runtime A to Runtime B as-is: a number, an object, a string, basically everything that makes sense. However, changes to such a value on any Runtime aren’t reflected in the other Runtimes. Each Runtime is unaware that it has a copy and simply treats its value as the original. This is the Serializable. Depending on the definition of shared memory, this type might not even be considered one.

Unbound memory

The first type of memory that is actually shared — meaning that changes to it on Runtime A will be seen by Runtime B and all the other Runtimes. This is achieved by keeping the actual “real” value outside of any Runtime, in C++, guarded by mutexes and other thread-safe mechanisms. Each JavaScript Runtime holds just a specific kind of reference to the C++ value. Each read or write means crossing the boundary between JavaScript and C++. Because the memory is thread-safe, each Runtime can safely access it at its own discretion. This is the Synchronizable

Runtime-bound memory

Type of asymmetric shared memory that overcomes some issues of unbound memory — operations on unbound memory are much more expensive than ones on plain JavaScript values — due to the need to cross the boundary between JavaScript and C++ with JSI on each access. Runtime-bound memory optimizes access time on a single designated JavaScript Runtime, called the Host Runtime, by simply holding the value there as a plain JavaScript value. Other Runtimes only have a specific kind of reference that “knows” that the actual value lives on another Runtime — if you want to access it with that reference, you must go to C++, obtain the Host Runtime exclusively, read the actual value, and return it all the way back to the accessor Runtime. This is the Shareable, the spotlight of today’s blogpost.

Why Shared Value wasn’t enough

The Shareable is by far the most complicated type of memory available in Worklets. You might wonder what purpose such a complicated concept could possibly serve. It turns out it has been there, in React Native, for years — this is exactly how Reanimated’s Shared Value (also called the Mutable) works. A Shared Value is a runtime-bound type of shared memory, specifically tailored for reactivity and driving animations. It lives on the UI Runtime where it’s updated and read sometimes even hundreds of times per frame in apps with a lot of animations. When you assign a new animation to it in your component, you actually schedule a call to the UI Runtime with your specific reference, where you update the value — and your animation is also copied to the UI Runtime, as a Serializable.

We wanted to go a step further with the Shared Value, expanding the possibilities of what you can do with shared memory in your apps. However, Shared Value’s close relation to animations and the UI Runtime proved to be a serious obstacle when trying to use it in a non-animation use case. This is why we had to take a step back, separate the shared memory concept from the animation logic, and expose a building block which Reanimated can take and turn into a Shared Value by adding animations to it. The same building block that you can take and do whatever you want with. This building block, the shared memory “primitive”, is the Shareable.

With React Native Worklets 0.8.0, you can freely use the Shareable in your apps, simplify some of your Shared Value use cases, and even create your own custom shared memory types by building on top of the Shareable. It’s all in your hands now, so go ahead and experiment with it!

We’re Software Mansion: multimedia experts, AI explorers, React Native core contributors, community builders, and software development consultants.

Worklets 0.8: The Shareable — Shared Memory’s Missing Piece was originally published in Software Mansion on Medium, where people are continuing the conversation by highlighting and responding to this story.

Popcorn is a library we develop that lets you run Elixir in the browser (zip-ties and glue included!). As you might guess, it has a bit more convoluted setup than a typical Elixir or JS library.

For a long while, using Popcorn meant copying two JS files from our template into your project as well as installing the popcorn Hex package for Elixir. The only advantage of copying was simplicity of development (for us).
A lot of projects use Phoenix which already includes a way to install a JS package. Other projects, like our Elixir language tour are React apps. Integrating via npm install is way easier than manually tracking JS files.

And frankly, it’s weird we had an official Elixir package and hacks for JS parts.

We decided to just extract those files into an npm library and make setup simpler for users. It didn’t come easy. You may even say it came with unrelenting perversity of inanimate objects.

The Architecture™

The main problem is that our architecture choices force us to place JS and Wasm files in carefully chosen places, with carefully chosen filenames. It mainly comes from our use of Emscripten.

A few words on the tech stack — we use AtomVM, which is a BEAM runtime written in C.
It compiles to Wasm via Emscripten, generating a .wasm file and .mjs JS glue code. We take them and throw them into an iframe (for isolation). Then it spawns the WebWorkers it needs to emulate threads.
End-user code communicates with it by exchanging messages back and forth with the iframe.

Here’s how it looks:

We have one file with the main script that has access to the DOM, another file in an iframe that loads and isolates AtomVM (and has access to the DOM via parent context), and many WebWorkers that load AtomVM directly (but incidentally use the same file that is loaded in the iframe).

It works beautifully until you start poking it:
- the iframe has a different view of the base URL than the main JS context.
- bundlers don’t have very good support for Wasm.
- WebWorkers with shared memory need CORS/COEP headers set by the server.
- Emscripten by default is configured to use one instance of the Wasm module and isn’t afraid of global variables.
- it also generates the JS in a way that hardcodes the JS glue filename.

Given all that, we have two requirements for the library:
- we want iframe.js to be a separate entrypoint that resolves AtomVM.mjs correctly.
- we want AtomVM.mjs to find its Wasm file (both in iframe and WebWorker contexts).

Stubbing Your Toe on Imports

I mentioned bundlers. For non-frontend developers, these are tools that take your Javascript project and output a single file containing all the JS code. Two popular choices are Esbuild and Rollup. To my complete astonishment, they don’t really care about Wasm and treat it as any other (binary) asset.
This is great, since you can have the unique experience of reviewing how different bundlers handle totally nonstandard ways of importing and using Wasm files.

I wanted to mark generated files as external modules (which means that they aren’t included in the main JS bundle (which means that they keep their filenames)).
This made sense to me at the time but had unfortunate implications: iframe.ts entrypoint which imported the external AtomVM module wasn’t resolving the import correctly because entrypoints are controlled by end-user’s code and we need them to be specifically next to AtomVM.mjs script. This would impact how users structure their projects.

The logical way of fixing that is to either change AtomVM.mjs to be bundler-friendly (removing the need to handle this file specially) or provide a plugin for users that configures it correctly. I really wanted to avoid the second one since it meant more setup for end-users and possibly more things that can break when you configure your bundler in a non-standard way.

Here are two fun excerpts from Emscripten-generated JS code.

Exhibit One:

function findWasmBinary() {
  if (Module["locateFile"]) {
    return locateFile("AtomVM.wasm");
  }
  // Use bundler-friendly `new URL(..., import.meta.url)` pattern; works in browsers too.
  return new URL("AtomVM.wasm", import.meta.url).href;
}

Exhibit Two:

worker = new Worker(new URL("AtomVM.mjs", import.meta.url), {...});

Long story short, Emscripten really wants you to keep the generated filenames unchanged and place them in the same directory.
Bundlers, on the other hand, really want to keep everything in the same bundle (and change the filenames to include content hash).
This was the second signal to try to improve AtomVM.mjs code to make it bundle-able.

Small note: WebWorkers

There are two main ways to use your code in WebWorkers. You can either provide a link to a script, similar to how HTML imports scripts or you can pass the JS string with script on creation (this approach is deprecated to my knowledge). I want to emphasize that both iframe and WebWorkers create new JS contexts — you don’t have access to variables or loaded scripts across them. If you need anything (a function, a variable, an entire library), it needs to be included in the JS code passed to the WebWorker.

Emscripten has an option to remove the need for a separate .wasm file by including the binary blob in the script, in the form of a base64 encoded string (with a mere 33% size increase due to encoding).
WebWorkers need a separate script to load and bundlers have an option for that. This wouldn’t be a problem if we didn’t have the same script that is loaded in both the main browser context and WebWorkers, behaving differently in each.

We can remove the hardcoded .wasm name by using base64.
Now we need to remove the import for the script itself when inside WebWorker context. Well, what if we just pack the entire script into base64 string (which includes base64-packed VM code)?

Some thoughts shouldn’t be thought.

Plugins

At this point, I was three weeks into research, mostly reading generated code, and the Emscripten and bundler docs. Trying to eliminate Wasm files or working around Emscripten looked like a dead-end.
I came back to the original idea of wrapping the the necessary logic into a plugin that will integrate with the user’s project.

This has numerous advantages compared to the previous attempt such as: not losing your sanity.

I also realized after many plugin drafts that we don’t need any sophisticated bundler features, it’s enough to put our files in the output directory and make sure bundlers don’t touch them.
In other words, we came back to cp template.js actual-file.js but with one additional (and useful!) abstraction. This allows us to have extremely simple plugins for both and to integrate with Vite, too (which uses both under the hood).

Full workflow: we develop Popcorn locally with TypeScript and other tools, we then ship both our custom code (API and plugins) and AtomVM compiled via Emscripten, and with some strategic file placement and hooking into the user’s build pipeline we force all of that to work.

This solves both of our previously stated constraints:

We want iframe.js to be a separate entrypoint that resolves AtomVM.mjs correctly.

Popcorn ships source files, which we then copy. iframe.js is a sibling file of AtomVM.mjs, no special handling needed to resolve imports.

We want AtomVM.mjs to find its Wasm file (both in iframe and WebWorker contexts).

Same as above. We also don’t modify the filenames so the generated Emscripten code always refers to files that exist (and not, for example, have a content hash appended to its name).

We’ve been using this approach in the Elixir language tour for a while now, and it works nicely — PR replacing the old implementation was satisfyingly red.

Best part of the diff:

export default defineConfig({
  base: "/",
  plugins: [
    react(),
 // ...
    svgr(),
+   popcorn({ bundlePath: "./public/wasm/bundle.avm" }),
    cookOnChange()
  ],

It simplifies the developer experience a lot, with both JS-first projects and Elixir-first projects.

Packing It All Up

It’s available to use now. You’ll need the @swmansion/popcorn npm package and popcorn Hex package. We have examples to show how it works (especially hello-popcorn with an Elixir-first approach and hello-react with a JS-first approach).

Since the last release (which was ages ago), we’ve also worked a lot on improving AtomVM and our use of it. You can find a changelog in Github releases.

We also saw some really cool demos: Steffen’s unmodified LiveView working in the browser and Peter’s collection of small demos (building on Steffen’s site, I especially like the sqlite one).

We’re also cooking something in the shadows — float on until that time. It may be slightly interesting.

Take care,
Kuba!

We are Software Mansion — Elixir experts, AI explorers, React Native core contributors, community builders, and software development consultants. Need help with building your next product? You can hire us.

The Three-Lang Problem: shipping Elixir runtime, JS, and WebAssembly as one npm package was originally published in Software Mansion on Medium, where people are continuing the conversation by highlighting and responding to this story.

After the Jelly Slider implementation gained a lot of attention, I decided to write this post to explain how it works in detail and show you how to recreate it yourself, or even build your own visualization inspired by it.

We’ll explore how the simulation functions and how it’s rendered to the screen using only TypeScript with the help of TypeGPU. I’ll walk through the reasoning behind my decisions and share some insights into the trade-offs made along the way. Of course, there’s always room for further improvements and optimizations — it all comes down to the balance between time and effort.

Disclaimer:
This is not a step by step implementation tutorial. I will focus primarily on explaining the methodology and will occasionally include snippets from the original code, which you can find in its entirety on the TypeGPU examples page or directly in the GitHub repository. Some snippets may be simplified or modified to avoid overwhelming the reader with code. You are free to reference the original implementation at any time, as the function names are kept intact.

The Simulation

The actual physical simulation happens in two-dimensional space, which greatly simplifies the amount of work we have to do. The implementation supports an arbitrary number of points but is empirically tuned to work with 17, which provides a good balance between performance and visual complexity.

The overall structure of the simulation is quite well represented by the code itself:

update(dt: number) {
  for (let s = 0; s < this.substeps; s++) {
    this.#integrate(h, damp, compression);
    this.#projectConstraints();
  }

  this.#computeNormals();
  this.#computeControlPoints();
  this.#updateGPUBuffer();
  this.#computeBezierPipeline.dispatchThreads(...BEZIER_TEXTURE_SIZE);
}

Where:

• h is the duration of a sub-step
• damp is a constant that represents additional resistance to movement
• compression is a normalized value that represents how far apart the ends are

Let’s go over the process step by step.

Integration

The main method used for calculating point positions is Verlet integration, which — despite its intimidating name — is actually very simple. We can think of it this way: if a point was previously at position A and is now at position B, its velocity corresponds to the vector from A to B.

Verlet integration updates each point’s position using its current and previous states, without explicitly storing velocity. The next position is estimated from the previous two, with some adjustments applied over time. In this case, there’s no gravity — instead, the middle section of the slider is slightly lifted to fit a sine-like curve when compressed.

Constraint projection

As mentioned earlier, the integration relies on the current position, previous position, and small adjustments over time. Apart from applying a varying arching force, we also need to enforce certain constraints. If we only applied the force, the points would drift away, and the result wouldn’t look stable.

To keep the simulation in shape, after each iteration we project constraints. In our case, those constraints are:

• the first point should stay fixed at its initial position
• the last point (handle) should stay fixed where the cursor is
• the distance between neighboring points should remain constant
• points near the ends should resist bending

They are enforced in two ways. The first two are hard constraints — we simply set the positions directly. The last two use distance projection. For the distance constraint, we measure the actual distance between two points and move each halfway toward their ideal separation. After several iterations, all segments are nearly at their correct lengths.

For bending resistance, the principle is similar, but instead of connecting two actual points equally, we compare a point to where a corresponding position along the y-axis would be, applying a smaller correction. In practice, this means we pull the endpoints back toward a straight line, but only partially.

These operations are enough for us to know where the slider is, but we still need to compute some additional data useful for rendering — specifically, the normals and control points.

Computing Normals

You can think of normals as the relative “up” directions at each point. They’re useful in many contexts — most commonly for calculating how light reflects off a surface during rendering. In our case, we won’t use them directly for the simulation (for reasons we’ll cover later), but they’re still important when we approximate the curvature of the slider.

Calculating normals using three points is straightforward: we take the line defined by connecting the previous and next points and then compute a vector that’s perpendicular to it. For the edge cases — the first and last points — we can simply assume the normal points straight up.

Computing Control Points

The last piece of information we need before rendering relates to how we want to represent the slider. There are several options, but the approach I chose is to model it as a series of quadratic Bézier segments. This choice works well for two main reasons:

1. A quadratic Bézier curve is defined by just three points.
2. They can be easily represented as SDFs (signed distance fields — we’ll cover what those are in the rendering section).

We already have a list of points for the slider, but we still need a third point for each segment to form the curves. This is where normals help. By using them, we can derive tangent lines, and the intersection of the tangents between two consecutive points gives us the control point for that segment.

There’s one edge case we need to handle for this method to work reliably. When the tangents are parallel or nearly parallel, the control point either doesn’t exist (the lines never intersect) or ends up far from where it should be. Fortunately, this is easy to fix — in those cases, we simply place the control point at the midpoint between the two points.

Updating the GPU buffers

Now that we have all the necessary information, the final step is to prepare the data for use inside the shaders. The key idea here is that, for the data to be accessible on the GPU, we need to upload it into a buffer.

TypeGPU simplifies this process by providing schemas, so we don’t have to manually compute buffer sizes or memory layouts. Thanks to that, the entire GPU resource management for the slider’s data comes down to just a few straightforward steps:

class Slider {
  // Type declarations
  pointsReadonly: TgpuReadonly<d.WgslArray<d.Vec2f>>;
  controlPointsReadonly: TgpuReadonly<d.WgslArray<d.Vec2f>>;
  #normals: d.v2f[];
  #controlPoints: d.v2f[];

  constructor(...) {
    // Buffer creation
    this.#root.createReadonly(
      // Create a schema for n 2-component vectors
      d.arrayOf(d.vec2f, this.n),
      // The initial data
      this.#pos,
    );
    this.#root.createReadonly(
      d.arrayOf(d.vec2f, this.n - 1),
      this.#controlPoints,
    );
  }

  #updateGPUBuffer() {
    // Updating the buffers
    this.pointsReadonly.write(this.#pos);
    this.controlPointsReadonly.write(this.#controlPoints);
  }
}

After this step, it’s really up to us how we interpret this data in the rendering, so we can basically treat the entire engine as a black box.

The Rendering

To recap the simulation part, let’s quickly review the data we have and outline how we’ll use it.

We have:

• an ordered list of 2D points that define the slider’s path,
• an ordered list of control points computed from those points.

Looking at the reference render, the slider clearly doesn’t look strictly 2D — yet it only ever deforms along two axes. This means we can render it as a 2D line and simply extrude it along the third axis, placing the slider in 3D space even though the simulation itself is purely 2D.

We could build this with a mesh and deform it, and it would probably work — but looking at the reference, it’s clear we’ll also need advanced lighting, soft shadows, refraction, and some level of GI (global illumination) or AO (ambient occlusion). It’s all doable, but for a one‑off example the code would get complex quickly; we’d essentially be building a mini 3D engine from scratch. Fun, but not practical here.

Ray marching ends up being a much better fit:

• The slider needs to be perfectly smooth, which would be hard with a mesh unless we simulate a large number of points. With an SDF for the Bézier curve, we get a smooth surface “for free.”
• AO (ambient occlusion) is trivial with ray marching — because of SDFs, a single ray marched a few short steps is enough for good results.
• The background is easy to express as SDFs and extremely cheap to compute.

How does ray marching work?

There are many great resources that explain what ray marching is and how it works, so I will not go into much detail here. If you want an excellent reference, I recommend the articles by Inigo Quilez. For now, all we need to know is:

• we set a camera position (the camera will stay stationary, which is very convenient for us)
• we define a perspective projection that determines the direction of our rays
• for each pixel on the screen, we send a ray according to that direction — that ray determines what color we assign to the pixel

The ray checks its distance to the scene (the nearest object) and keeps moving forward until that distance is close enough to zero. At that point, it calculates the lighting for that position in space.

The background

For the ray to know how far it is from each element in the scene, we need to describe everything using signed distance fields. We’ll start by building the background. First, we’ll add a simple plane. We’ll define a function using TypeGPU, and by adding the `use gpu` directive at the top, the function becomes available inside a shader. We’ll also use the @typegpu/sdf helper package, which gives us SDF functions for common shapes (though you can implement them yourself from any reference — it’s just math).

const getMainSceneDist = (position: d.v3f) => {
  'use gpu';
  return sdf.sdPlane(position, d.vec3f(0, 1, 0), 0.06);
};

We defined a plane that spans the xz axis at y=0.06. It appears to cover the entire screen because the camera is tilted downward. That’s just an arbitrary choice — we could just as well keep the camera fixed and rotate the scene instead.

Now let’s create the cutout where the slider will sit. We’ll use a 2D rounded box and then extrude it into the third dimension.

const rectangleCutoutDist = (position: d.v2f) => {
  'use gpu';
  return sdf.sdRoundedBox2d(
    position,
    d.vec2f(1, 0.2), // width and height
    0.2, // the roundness factor
  );
};

const getMainSceneDist = (position: d.v3f) => {
  'use gpu';
  return sdf.opExtrudeY(
    position,
    rectangleCutoutDist(position.xz),
    0.01,
  );
}; 

This will produce a slim rounded rectangle floating in space — but that’s not what we need, we actually need the opposite since it’s supposed to be a cutout. Thanks to the nature of SDFs, it’s as simple as flipping the sign on the rounded‑rectangle function.

Our rectangle shape was already rounded, but this roundedness only applies to the 2D shape. The extrusion that brings it into the third dimension creates sharp edges, which do not match what we want. A solution for that is to use the inflation operation, which is very simple in nature — we basically subtract a flat amount from the distance. This means the shape will grow a bit, but all the sharp edges will get smoothed out.

After applying those operations, all that’s left is to patch the hole in the middle. We will do that using the union operation, which is basically taking the minimum of two distance fields. We add a plane at a small offset and make our scene the min of the two.

The final code for the background

const rectangleCutoutDist = (position: d.v2f) => {
  'use gpu';
  return sdf.sdRoundedBox2d(
    position,
    d.vec2f(1, 0.2),
    0.2,
  );
};

const getMainSceneDist = (position: d.v3f) => {
  'use gpu';
  return sdf.opUnion( // basically std.min
    sdf.sdPlane(position, d.vec3f(0, 1, 0), 0.06),
    sdf.opExtrudeY( // make it 3d
      position,
      -rectangleCutoutDist(position.xz), // sign flip
      0.01,
    ) - 0.02, // inflation
  );
};

You may have noticed that the background in the images is not just a uniform color but appears “lit.” There are many approaches to simulating lighting, but we will settle for one of the more popular and simple ones: the Phong lighting model. It is very well documented, and TypeGPU has a dedicated example for it.

Basically, all we need is a light direction vector and a surface normal. The normal is a vector pointing away from the surface at a given point. With these, we can approximate lighting using three components:

• Ambient — the light that just is. It represents light that has bounced around the environment multiple times, without actually simulating those bounces.
• Diffuse — the light scattered by the material when it is hit by the light source. It depends on how directly the surface faces the light.
• Specular — the light that is directly reflected toward the observer, producing visible highlights on shiny surfaces.

If we look at the example code — cleaned up and reduced to just the core logic — it looks like this:

const calculateLighting = (
  hitPosition: d.v3f,
  normal: d.v3f,
  rayOrigin: d.v3f,
) => {
  'use gpu';
  // NOTE: This code operates under the assumption that light direction
  // is already normalized

  // the uniform contains the direction from the light, but for
  // the calculations we need the direction towards the light
  const lightDir = std.neg(lightUniform.$.direction);
  // we calculate the direction of the "eye" by taking the a vector from
  // the position the ray hit to the ray origin
  const viewDir = std.normalize(rayOrigin.sub(hitPosition));
  const reflectDir = std.reflect(std.neg(lightDir), normal);

  // ambient
  const ambientLight = AMBIENT_COLOR.mul(AMBIENT_INTENSITY);

  // diffuse
  const diffuseTerm = std.max(std.dot(normal, lightDir), 0.0);
  const diffuseLight = lightUniform.$.color
    .mul(diffuseTerm);

  // specular
  const specularTerm = std.max(std.dot(viewDir, reflectDir), 0) **
    SPECULAR_POWER;
  const specularLight = lightUniform.$.color.mul(specularTerm);

  // saturate(x) is basically max(0, min(1, x)))
  return std.saturate(diffuseLight.add(ambientLight).add(specularLight));
};

One important detail we’ve conveniently ignored so far is that we’ve been using normals as if we already had them — but we don’t.

In typical 3D models, normals are provided per vertex, along with position data and other attributes such as texture coordinates. The rendering pipeline simply interpolates them across the surface.

Here, however, we’re not working with a traditional mesh but rendering a signed distance field, which means there are no predefined vertices and no stored normals, so we need to compute the normal ourselves — fortunately, this is conceptually pretty easy, even if it can be computationally expensive.

The example above computes a normal for an arbitrary point in 2D space, with greatly exaggerated distances for clarity. This process generalizes easily to 3D — you simply sample additional points and move the vectors into three-dimensional space. The important detail is that these sample points should be arranged symmetrically and contribute equally along each axis to produce a correct and unbiased normal. In the Jelly Slider, for performance reasons, we did not use the typical six-point normal approximation with two samples per axis. Instead, we used four points corresponding to the vertices of a regular tetrahedron. The underlying idea remains the same, and in practice, the results are not noticeably worse.

A key performance insight becomes obvious when we look at the main scene with normal visualization, where the absolute values of xyz components are mapped to rgb colors.

From the image above, it’s clear that a constant normal of [0, 1, 0] would already be a reasonable approximation for the entire scene — and for most of it, even a perfect one. The surface is largely flat and aligned in a single direction, so computing a full SDF-based normal at every point would be unnecessary overhead. That’s why, in the code, the normal calculation for the main scene is simplified to this:

const getNormalMain = (position: d.v3f) => {
  'use gpu';
  // check if we are outside the cutout area
  if (std.abs(position.z) > 0.22 || std.abs(position.x) > 1.02) {
    return d.vec3f(0, 1, 0);
  }
  // the full tetrahedral normal approximation
  return getNormalMainSdf(position, 0.0001);
};

The getNormalMainSdf function uses one of my favorite TypeGPU features: slots. Since there are multiple SDFs in the scene for which we may want to compute normals, we avoid duplicating code by implementing the tetrahedral normal approximation once and defining the actual SDF function as a slot. The slot represents any function that takes a position and returns a distance. This allows us to reuse the same normal computation logic across different parts of the scene by simply plugging in a different SDF implementation.

const getNormalFromSdf = tgpu.fn([d.vec3f, d.f32], d.vec3f)(
  (position, epsilon) => {
    'use gpu';
    // create the offset vectors (scaled by some small value)
    const offset1 = k.xyy * epsilon;
    const offset2 = k.yyx * epsilon;
    const offset3 = k.yxy * epsilon;
    const offset4 = k.xxx * epsilon;

    // scale the vectors by distance
    // sdfSlot.$ will be substituted with a function of choice later
    const sample1 = offset1 * sdfSlot.$(position + offset1);
    const sample2 = offset2 * sdfSlot.$(position + offset2);
    const sample3 = offset3 * sdfSlot.$(position + offset3);
    const sample4 = offset4 * sdfSlot.$(position + offset4);

    // create and normalize our normal from samples
    const gradient = sample1 + sample2 + sample3 + sample4;
    return std.normalize(gradient);
  },
);

// fill the slot with our main scene sdf (which we created earlier)
const getNormalMainSdf = getNormalFromSdf.with(sdfSlot, getMainSceneDist);

You may have noticed that the main scene — aside from currently lacking any jelly — looks somewhat flat. That’s because there are no shadows. While we could add them, shadow calculations in an SDF setup tend to be quite expensive, and in this case, we’ll rely on a couple of other tricks instead.

Ambient occlusion is a pretty common technique in computer graphics, but in the context of SDFs, I would argue it’s even more important due to the inherent nature of distance fields — specifically the fact that at any point in space, we already know the distance to the closest element of the scene, with no additional tricks required. Combined with the fact that we know how to calculate the normal at any point, we can fairly accurately estimate how occluded that point is, meaning how exposed it should be to ambient light.

The diagram above illustrates how the occlusion calculation works over three steps, assuming each sample is treated equally, which in practice is usually not the case. In the Jelly Slider, we also use three steps, but each consecutive step contributes only half as much as the previous one.

const calculateAO = (position: d.v3f, normal: d.v3f) => {
  'use gpu';
  let totalOcclusion = d.f32(0);
  let totalWeight = d.f32(0);
  let sampleWeight = d.f32(1);

  for (let i = 1; i <= 3; i++) {
    const sampleHeight = (AO_RADIUS / 3) * d.f32(i);
    // get the point for AO sample
    const samplePosition = position + normal * sampleHeight;
    const distanceToSurface = getSceneDistForAO(samplePosition);

    // std.max is here to safeguard from numerical errors
    // distanceToSurface should never be > sampleHeight
    const occlusionContribution = std.max(0, sampleHeight - distanceToSurface);

    totalOcclusion += occlusionContribution * sampleWeight;
    totalWeight += sampleHeight * sampleWeight;
    sampleWeight *= 0.5;
  }

  const rawAO = 1.0 - totalOcclusion / totalWeight;
  return std.saturate(rawAO);
};

Since we already have the scene distance function and normal computation in place, the ambient occlusion implementation essentially comes down to performing three fixed ray marching steps along the normal and measuring the returned distances. While three steps are not free, they are still significantly cheaper than computing proper shadows.

It is important to stress that ambient occlusion is not a replacement for shadows. They model fundamentally different phenomena and only happen to produce a similar sense of depth in our very flat scene. The slider does use shadows, but they are not calculated using ray marching.

Jelly time

Now that we’ve established a solid foundation for rendering the scene, prepared the cutout for the jelly to sit in, and set up the simulation infrastructure, we’re ready to tackle arguably the most important part — actually rendering the slider itself.

There are many feasible ways to get it on the screen — and quite a few were explored before settling on the final approach. Here are a couple of screenshots from different phases of experimentation:

There were two main challenges to solve, and for a long time, they felt mutually exclusive:

1. Appearance — we want the jelly to have a smooth surface and be as close to the reference in visual quality as possible.
2. Performance — the goal was to have it run smoothly even on mid to low-end phones.

The final approach took some iteration, but thankfully, a few things were working in our favor. The key idea was to pick the right primitive to construct the SDF from.

The initial attempt was to build it from a high number of points and line segments, since distances to those are very cheap to compute. This failed mainly because the physics simulation became the bottleneck at higher point counts. To achieve a convincing curve — especially when heavily bent — the number of required points was simply too high.

The second idea was to use Bézier curves (as you already know, this is the approach I ultimately landed on — just not immediately). Bézier curves give you very smooth visuals with a fraction of the control points, which is a huge advantage. The downside is that computing the distance to a Bézier curve is not trivial, both computationally and conceptually.

Calculating the distance to the Jelly Slider — which consists of 16 Bézier segments — for every ray is technically possible, but computationally far too expensive, even for mid-range desktops, let alone mobile devices.

To put this into perspective: in the best-case scenario for a 2K output at 2560 by 1440 pixels, we would need to solve 16 quadratic equations per ray, roughly 8 to 64 times per pixel per frame. Factoring that out gives approximately 1,887,436,800 quadratic equations to solve every single frame — clearly not a viable approach.

That said, this is a worst-case, brute-force estimate. There is, of course, a lot of room for optimization.

There are three main avenues we can explore in search of optimizations, though the first two are inherently connected.

The key observation is that the slider is fundamentally two-dimensional. We only expand it into three dimensions using an SDF extrusion along the third axis, but the core information — the curvature — lives strictly in 2D.

This means we can precompute the expensive Bézier distance calculations once within a limited 2D region around the slider. Instead of solving quadratic equations at render time, we evaluate them ahead of time and store the results in a texture.

At runtime, this reduces the problem to a simple texture lookup. Not only is that dramatically cheaper than solving equations per pixel, but it also gives us interpolation for free*, allowing us to reduce the resolution while still maintaining smooth results.

* Of course, not completely free, but still cheaper than doing it by hand.

Since we’re already precomputing the data, we also store some useful auxiliary information to further reduce runtime work. Because the Jelly Slider is fundamentally 2D, we can pack quite a lot of data into a single pixel.

Each vec4f in the texture is laid out as follows:

• x — the raw distance to the nearest Bézier segment
• y — the progress value, meaning how far along the slider we are, normalized to the 0–1 range
• z and w — the precomputed normal at that point

By storing the normal directly, we avoid additional texture samples to reconstruct it, and we also retain the full effective resolution of the data. Reconstructing normals from quantized distance values would introduce additional error and visibly degrade the result. Precomputing and storing them explicitly keeps both the shader logic simpler and the visual quality higher.

Great, so we now have a 2D texture containing all the information we need — but how do we actually use it?

One of the great properties of SDFs is that you can evaluate them anywhere in space by simply passing in a position and the primitive parameters. That convenience no longer fully applies here. The texture has limited bounds, so we need to know when to sample it and how to map world space positions to texture space.

This is where the third optimization helps — and conveniently solves this problem as well. We define a bounding volume in world space that fully contains the slider.

This helps in two ways.

First, ray–box intersection tests are simple and computationally cheap, so we can quickly determine whether a ray even has a chance of hitting the slider.

Second, once we transform a point into the box’s local space — effectively redefining the coordinate system with one corner of the box as the origin — we regain the convenient SDF-like behavior. A normalized position inside this box space directly maps to texture coordinates, which tells us exactly where to sample the precomputed distance field to retrieve the jelly slider data.

The code that converts our encoded data into a meaningful shader variable looks like this:

const sdInflatedPolyline2D = (p: d.v2f) => {
  'use gpu';
  const bbox = getSliderBbox();

  // checking where to sample the texture as shown above
  const uv = d.vec2f(
    (p.x - bbox.left) / (bbox.right - bbox.left),
    (bbox.top - p.y) / (bbox.top - bbox.bottom),
  );
  const clampedUV = std.saturate(uv);

  // decoding the slider info
  const sampledColor = std.textureSampleLevel(bezierTexture.$, filteringSampler.$, clampedUV, 0);
  const segUnsigned = sampledColor.x;
  const progress = sampledColor.y;
  const normal = sampledColor.zw;

  return LineInfo({
    t: progress,
    distance: segUnsigned,
    normal: normal,
  });
};

This function computes the distance to the slider for a given 2D point. On its own, that would make the slider infinite along the third dimension, so we add a helper that extrudes the shape into 3D. This both bounds the volume and adds an end cap, implemented as a half-pie SDF computed from the last two points of the slider.

const sliderSdf3D = (position: d.v3f) => {
  'use gpu';
  const poly2D = sdInflatedPolyline2D(position.xy);

  let finalDist = d.f32(0);
  if (poly2D.t > 0.94) {
    finalDist = cap3D(position);
  } else {
    const body = sdf.opExtrudeZ(
      position,
      poly2D.distance,
      LINE_HALF_THICK
    ) - LINE_RADIUS;
    finalDist = body;
  }

  return LineInfo({
    t: poly2D.t,
    distance: finalDist,
    normal: poly2D.normal,
  });
};

Actually, the fully rendered preview in the image above is a bit premature. At this stage, we only have the raw information about the slider, so it actually looks more like this:

Since our current lighting logic only knows how to render the ground, it treats rays that hit the slider the same way. Another issue that becomes immediately obvious near the cap area is that linear interpolation is not enough to produce believable lighting for the slider. We will need some additional smoothing logic along the edges.

Our slider is fixed along the x-axis, so creating rounded normals for the edges to better simulate the smooth shape is fairly straightforward. We check the z position of the hit and create a vector pointing in the z direction, vec3f(0, 0, 1)with the same sign as the z position. Then, using the slider width, which is conveniently constant, we blend between the interpolated normal read from the texture and this z vector.

This requires some trial and error with the blending ratios to get the smoothing right, but it ultimately allows us to match the curvature of the cap almost perfectly.

Now the only core thing left is to make the slider transparent. That might sound scary and complicated, but we actually already have most of the pieces in place. The main missing part is the refraction logic so the light bends realistically inside the slider. Once we determine where the refracted ray should travel, we can simply march along that direction while ignoring the slider itself.

In code, this becomes a simple conditional branch in the ray marching loop. rayMarchNoJelly is a small ray marching routine that runs for six steps, which is more than enough to produce convincing refractions. If nothing is hit after six steps, we simply take the closest point reached so far, which turns out to be a good enough approximation.

if (hitInfo.distance < SURF_DIST && hitInfo.objectType === ObjectType.SLIDER) {
  const hitPosition = rayOrigin + rayDirection * distanceFromOrigin;

  const N = getNormal(hitPosition, hitInfo);

  const eta = 1 / JELLY_IOR;
  const refrDir = std.refract(rayDirection, N, eta);

  const env = rayMarchNoJelly(hitPosition, refrDir);
  const progress = hitInfo.t;
  const jellyColor = jellyColorUniform.$;

  const scatterTint = jellyColor.rgb * 1.5;
  const absorb = (1 - jellyColor.rgb) * 20;

  const T = beerLambert(absorb * (progress ** 2), 0.08);

  const lightDir = std.neg(lightUniform.$.direction);

  const forward = std.max(0, std.dot(lightDir, refrDir));
  const scatter = scatterTint * (JELLY_SCATTER * forward * progress ** 3);
  const refractedColor = env * T + scatter;

  return d.vec4f(refractedColor, 1);
}

The code is simplified by assuming a constant thickness, which turns out to be good enough. We could ray march inside the volume or derive a formula to calculate how much jelly the light travels through, but that would either be expensive or time-consuming for a detail that is barely noticeable.

Putting everything we discussed in this article together, we get:

Not quite a 1:1 match with the original example, but close enough without diving into too much additional detail. Everything that is missing can be derived from what we have covered so far. The shadow is simply the slider texture projected onto the plane, the subtle glow is derived from the slider distance, the highlight in the middle is entirely hard-coded, and the text rendered on the ground is just an image.

The code is public, and I encourage you to explore and experiment with it. With what you know now, you will probably be able to quickly identify the remaining tricks used in the implementation.

Thank you for reading to the end :) I hope this article was detailed enough to convey some useful techniques and spark interest in graphics programming while still staying engaging. If you have any questions, feel free to reach out on the TypeGPU Discord server, where I am often around, or contact me directly on X.

Breaking Down the Jelly Slider was originally published in Software Mansion on Medium, where people are continuing the conversation by highlighting and responding to this story.

The release of Expo SDK 55 marks a turning point for media handling in React Native. Following its previous deprecation, expo-av has now been fully removed, with its functionality officially split into two specialized, modern packages: expo-video and expo-audio.

Having recently led the migration to expo-video at Expensify (as part of the Software Mansion team), we’ll walk you through the key differences between the libraries and share what we learned along the way, so you can migrate with confidence.

Decoupling logic from UI

The most important architectural change is the separation of concerns. We are moving from the monolithic <Video/> component to a dual-system: VideoPlayer for logic control and <VideoView/> for the UI.

This split closely mirrors native media handling, where playback logic and visual rendering have always been distinct entities. By aligning with Android’s ExoPlayer/PlayerView and iOS’s AVPlayer/AVPlayerViewController, expo-video gives developers the same granular control and architectural clarity found in native development.

expo-av: initialization

const videoRef = useRef<Video>(null);

<Video
  ref={videoRef}
  source={{ uri: VIDEO_URL }}
/>

expo-video: initialization

const player: VideoPlayer = useVideoPlayer(VIDEO_URL);

<VideoView player={player} />

Plus, you can now define the player’s initial state directly during initialization — not just via props on the View, which is much more intuitive:

const player = useVideoPlayer(VIDEO_URL, (player) => {
  player.loop = true;
  player.muted = true;
  player.playbackRate = 1.75;
});

VideoPlayer vs. VideoView: method mapping

Since playback logic has moved from the Video ref to the VideoPlayer object, most methods (play/pause, volume/mute, playback rate, etc.) are now accessed directly through the player. Only a few presentation-related methods remain on the VideoView. This ensures that the native playback logic and the visual presentation layer remain decoupled.

expo-av: monolithic methods

videoRef.current.playAsync();
videoRef.current.pauseAsync();
videoRef.current.setRateAsync(newRate, true);
videoRef.current.setIsMutedAsync(!isMuted);
videoRef.current?.presentFullscreenPlayer();
videoRef.current?.dismissFullscreenPlayer();

expo-video: player vs. view methods

// Playback logic (Player)
player.play();
player.pause();
player.playbackRate = newRate;
player.muted = !muted;

// Presentation logic (View)
videoViewRef.current?.enterFullscreen();
videoViewRef.current?.exitFullscreen();

Synchronous vs. asynchronous calls

Another game-changing update in expo-video is the introduction of synchronous function calls. This preserves the execution order of your code without the need for async/await boilerplate.

expo-av: toggle playback (async)

const togglePlayback = useCallback(async () => {
  if (!videoRef.current) return;
  if (isPlaying) {
    await videoRef.current.pauseAsync();
  } else {
    await videoRef.current.playAsync();
  }
}, [isPlaying]);

expo-video: toggle playback (sync)

const togglePlayback = useCallback(() => {
  player.playing ? player.pause() : player.play();
}, [player]);

Modern state management

Finally, expo-video addresses one of the biggest pain points of expo-av: state updates via the onPlaybackStatusUpdate callback. In the old API, this single, massive function was triggered by every state change (including time updates), forcing developers to parse a huge AVPlaybackStatus object and trigger multiple setState calls.

expo-video solves this with a subscription-based approach, splitting updates into atomic, manageable events.

useEvent

Use this hook when you need a value (like isPlaying) that stays in sync with a specific event (playingChange). You can also provide a default value:

const { isPlaying } = useEvent(player, "playingChange", {
  isPlaying: player.playing,
});

useEventListener

You can use this hook to react to events immediately. For optimal performance, you should avoid using useEffect for event-driven logic and instead handle reactions directly within this listener:

useEventListener(player, "playToEnd", () => {
  console.log("Video finished playing");
});

Performance impact

In expo-av, a typical status update handler looked like this:

const handlePlaybackStatusUpdate = useCallback((status: AVPlaybackStatus) => {
  if (status.isLoaded) {
    setIsPlaying(status.isPlaying);
    setPosition(status.positionMillis);
    if(status.rate > 2){
      setRateText("It's really fast");
    }
  }
}, []);

In expo-video, we achieve the same result with significantly better performance and less boilerplate:

const { isPlaying } = useEvent(player, "playingChange", {
  isPlaying: player.playing,
});

const { currentTime } = useEvent(player, 'timeUpdate', {
  currentTime: 0
});

useEventListener(player, "playbackRateChange", (payload) => {
  if(payload.playbackRate > 2){
    setRateText("It's really fast");
  }
});

Why is this better? In this model, isPlaying only updates when the playingChange event occurs, and the playbackRateChange logic only executes when the rate actually changes. In expo-av, both would be re-evaluated every time currentTime is updated (which happens multiple times per second), leading to unnecessary overhead.

Just the tip of the iceberg

If you want to see these differences in action, you can check out this Expo Snack demo with our side-by-side comparison.

The differences highlighted above are just the beginning. expo-video offers a wealth of new possibilities: from custom buffering and reusing the same VideoPlayer across different screens to native thumbnail generation and robust Picture-in-Picture support. React Native and Expo developers now have more power than ever to tailor the video experience to their needs.

You can explore the full potential through expo-video docs at docs.expo.dev/versions/latest/sdk/video

Need support with your migration?

Migrating from expo-av to expo-video in large-scale projects is more than just a syntax swap — it gives you a chance to refresh your workflow and boost performance.

If your team has encountered a unique edge case, needs a performance audit, or simply wants professional help navigating this transition, reach out to us at Software Mansion. As creators of some of the key tools within the React Native ecosystem (such as Reanimated or Gesture Handler), we’re ready to help you make the most of expo-video and the new SDK 55.

We’re Software Mansion: multimedia experts, AI explorers, React Native core contributors, community builders, and software development consultants.

The Future of Video in React Native: Moving from expo-av to expo-video was originally published in Software Mansion on Medium, where people are continuing the conversation by highlighting and responding to this story.

Meet the first beta release of Gesture Handler 3. This release brings your favorite gesture handling library up to speed in the new world, where the New Architecture is the Architecture, manual memoization is quickly becoming a relic of the past, thanks to the React Compiler, and interactions are more beautiful and more performant than ever, thanks to Reanimated 4.

New API based on hooks

To adapt to today’s environment, we had to make some changes, the most notable of which is the new hook-based API, which is compatible with React Compiler. This introduces a conceptual shift compared to previous implementations, where the gesture lifecycle was fully managed by the component it was attached to. In the new API, gestures are managed entirely using hooks, and components are responsible only for attaching them to the correct views.

If you have been using Gesture Handler so far, the new API will look familiar, but with some significant changes. The most noticeable ones are the renamed callbacks — onStart becomes onActivate, and onEnd becomes onDeactivate. The new names align more closely with the gesture state flow.

You can see the most important changes in the updated migration guide in our documentation.

Saying farewell to the legacy Architecture

Gesture Handler has supported the New Architecture since version 2.3.0. Keeping support for both the new and the legacy architectures proved difficult at times, and prevented us from utilizing the new features. Since the New Architecture has become the only architecture, Gesture Handler 3 will no longer support the legacy architecture.

This shift enabled us to make significant internal changes, unlocking previously unavailable capabilities:

• You can now effortlessly attach gestures to Text components and inline fragments.
• The StateManager, previously only available in select callbacks, is now generally available on the JavaScript and main threads.
• The button implementation has been completely revamped.
• The new API comes with support for React Native’s built-in Animated API.

…and there are many more enhancements on the horizon.

Updated Reanimated integration

We have also overhauled our integration points with Reanimated, making them even more performant. In combination with the new API, this enabled us to implement a long-requested feature — using SharedValues directly in the gesture config. This, finally, allows for changing the gesture properties with no re-renders.

The improved integration will also benefit projects like RNRepo — a repository for React Native pre-built artifacts. Now, it will be possible to build the projects independently of each other.

Everything is still there

We are aware that adopting new things isn’t easy. In order to make the switch as simple as possible, we decided to keep all APIs available in the previous versions. This means that upgrading to Gesture Handler 3 should be as simple as installing the new version and checking the migration guide to adapt to a few name changes.

Adopting the new API itself can be done gradually — component by component or screen by screen. The only caveat is that the hooks API is not cross-compatible with the previous ones. To set up relations, like simultaneous, between the gestures, all of them will need to use the hooks API.

If you can use LLM Agents in your codebase, you can also try our migration skill, which should make the adoption even easier.

Thanks to Shopify

As always, we at Software Mansion would like to thank Shopify for their continued support of our open-source projects and trust throughout the years.

We’re Software Mansion: multimedia experts, AI explorers, React Native core contributors, community builders, and software development consultants.

Introducing Gesture Handler 3.0: Hook-Based API, Deeper Reanimated Integration & More was originally published in Software Mansion on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you’ve been working with React Native for a while, you’ve probably hit a wall where animations, gestures, or app performance should have been smooth, but weren’t.

For a long time, React Native developers had to choose between developer comfort and performance. JavaScript was easy and expressive, but pushing it too far was challenging. As apps grew more ambitious, adding real‑time camera features, advanced graphics, or audio processing, it became clear that we needed better tools for concurrency.

In this article, we tell the story of how the React Native Worklets library was born, what it does, what it brings to the ecosystem, and how it powers tools like the new VisionCamera v5. Let’s dive in!

How React Native Worklets was born

At first React Native Worklets wasn’t a separate project. The library was originally developed as an internal component for tools like React Native Reanimated or React Native Gesture Handler, allowing developers to create smooth animations and interactions that felt truly native.

In the early days of React Native, your animation logic would exist solely in JavaScript, running on the same thread, the JS thread, as the React part of your application. This meant that any expensive operation, such as data processing, network activity, or even unrelated JavaScript work, could negatively impact animation performance. Reanimated addressed this by introducing a way to execute animation logic outside of React’s render cycle, safe from JS thread bottlenecks.

The technical mechanism behind this approach became known as worklets — small, isolated pieces of JavaScript code designed to run on the UI thread, on a separate JavaScript runtime. They had a restricted execution environment by design, which made their behavior predictable and safe to run concurrently.

An example of how worklets can be used to react to gestures synchronously on the UI thread to trigger animations and layout changes:

const pan = Gesture.Pan()
    .onChange((event) => {
      'worklet';
      offset.value = event.translationX;
    })
    .onFinalize(() => {
      'worklet';
      offset.value = withSpring(0);
    });

As Reanimated evolved, it became clear that this execution model was useful far beyond animations. The same characteristics that made worklets good for animations (low latency, independence from React, and concurrent execution) also applied to gesture handling, graphics rendering, camera frame processing, audio processing, and other performance-critical tasks.

For example, react-native-vision-camera introduced “Frame Processors” — a mechanism to run a JS function on every Frame the Camera “sees”:

```ts
const frameProcessor = useFrameProcessor((frame) => {
  'worklet'
  const objects = detectObjects(frame)
  const label = objects[0].name
  console.log(`You're looking at a ${label}.`)
}, [])
return <Camera frameProcessor={frameProcessor} />
```

At that point, worklets started to outgrow their original context. Developers and library authors needed access to multithreading capabilities even when animations weren’t involved. So, we realized that keeping worklets tightly coupled to Reanimated equaled unnecessary dependencies and limited adoption.

Extracting worklets into a standalone library was a practical decision that allowed us to focus more on multithreading and provide a general-purpose solution for concurrency in React Native.

React Native Worklets as a multithreading engine

Today, React Native Worklets enables full-scale multithreading within the React Native framework. You can think of it as an engine that lets you run JavaScript logic concurrently, without writing native code.

The key idea is simple: purely native apps can utilize multiple CPU cores to increase performance — this should be easily done in React Native too. Heavy calculations, continuous processing, or real-time data handling don’t have to execute within your components. With React Native Worklets, you define what pieces of your JavaScript code should run independently, keeping your UI responsive and predictable.

Thanks to it, instead of jumping into native modules or complex bridges, developers can stay in JavaScript and use familiar APIs. This is exactly why worklets are such a good fit for many specialized libraries.

Integrating React Native Worklets and VisionCamera

react-native-vision-camera (often referred to simply as “VisionCamera”) is a powerful, high-performance Camera library for React Native. Next to photo, video, or snapshot capture, it also supports real-time “Frame Processing” via useFrameProcessor(…).

Since the Camera runs on a separate Camera Thread, VisionCamera conventionally would have to perform a Thread-hop to call the JS Frame Processor, which can cause stalling, or even worse — requires serialization for a Frame — at 4k, that is ~33MB per Frame, or ~1GB/s.

Thanks to worklets, VisionCamera instead synchronously calls your JS Frame Processor on the Camera Thread without a Thread-hop nor serialization! Without worklets, VisionCamera Frame Processors would’ve never existed.

As a user, you don’t need to configure anything — if react-native-worklets is installed, VisionCamera V5 can use it automatically thanks to Nitro’s worklets integration.

What’s next?

The react-native-worklets library is still young, and there’s plenty of room to grow. We’re actively working on making it easier to adopt, as multithreading shouldn’t feel like an advanced trick; it should be something devs reach for naturally when they want to improve app performance.

We’re also really excited to see what the community will build on this foundation. Rendering, audio, automation, and real-time processing: these are all areas where react-native-worklets can simplify work and make limitations disappear. So, take a look at our docs and get started!

We’re Software Mansion: multimedia experts, AI explorers, React Native core contributors, community builders, and software development consultants.

Behind the Scenes of React Native Multithreading: Vision Camera V5 x React Native Worklets was originally published in Software Mansion on Medium, where people are continuing the conversation by highlighting and responding to this story.

Picture a lively dinner party: glasses clinking, half-finished sentences, and three people laughing at the same time. To a human, navigating this is instinctual. To an AI, it is a nightmare. We have effectively mastered the predictable flow of a one-on-one chat. But handling a group conversation, where people interrupt and talk over each other, is much more difficult.

Together with the Fishjam team, we set out to showcase our Selective Forwarding Unit solution by building a unique demo app that solves this problem. That’s how the Deep Sea Stories game came to life.

The premise is simple: a group of detectives enters a conference room to solve a mystery. The twist? The “Riddle Master”, the entity that knows the secret solution and answers questions is actually a Gemini Voice AI Agent. This required the agent to listen, understand, and respond to a group of users in real-time.

The anatomy of a voice agent

First, let’s look at how an AI Voice Agent typically processes data. It typically operates through a modular pipeline that includes the following steps:

• Speech-to-text (S2T): The system converts the user’s spoken input into text using models like Google Speech-to-Text, OpenAI Whisper or ElevenLabs’ transcription service.
• Large Language Model (LLM): The transcribed text is processed by an LLM (e.g. Gemini, GPT-4, Claude) to understand the context and generate an appropriate text response.
• Text-to-speech (TTS): The text response is converted back into natural-sounding speech using services like Google Cloud TTS, ElevenLabs or Azure TTS.
• Real-time audio streaming: The audio is delivered back to the user with minimal latency.

A second architecture gaining popularity, and notably used in the newest Gemini Live API models is Speech-to-Speech. Unlike traditional pipelines that convert speech to text and back again, this architecture feeds raw audio directly into the model and generates audio output in a single step. This unified approach not only reduces latency but also preserves non-verbal features, enabling the model to recognize and replicate subtle human emotions, tone, and pacing with high fidelity.

One-to-one vs. group contexts

Most standard SDKs make setting up a one-on-one conversation relatively simple. For example, using the Gemini Live API SDK:

const { GoogleGenAI } = require("@google/genai");

// 1. Setup
const ai = new GoogleGenAI({ apiKey: "YOUR_API_KEY" });

async function startAgent() {
  // 2. Connect
  const session = await ai.live.connect({
    model: 'gemini-2.5-flash-native-audio-preview-12-2025',
    config: { responseModalities: ["AUDIO"] },
  });

  console.log("Agent Connected!");

  // 3. Listen for the Agent's Voice
  session.receive(async (msg) => {
    // This loop runs every time the AI sends an audio chunk
    if (msg.serverContent?.modelTurn?.parts) {
      const audioData = msg.serverContent.modelTurn.parts[0].inlineData.data;
      console.log(`Received Audio Chunk (${audioData.length} bytes)`);
      // In a real app, you would send 'audioData' to your audio output device
    }
  });

  // 4. Send Your Voice (Simulated)
  // Real apps pipe microphone data here continuously
  console.log("Sending audio...");
  await session.sendRealtimeInput([{
    mimeType: "audio/pcm;rate=16000",
    data: "BASE64_ENCODED_PCM_AUDIO_STRING_GOES_HERE" 
  }]);
}

startAgent();

However, these SDKs assume a single audio input stream. In a conference room, audio streams are distinct, asynchronous, and overlapping. We had to determine how to aggregate these inputs for the Riddle Master without losing context or introducing unacceptable latency.

We evaluated three specific architectural strategies to handle the multi-speaker environment:

• Server-side aggregation: This method involves mixing all player audio streams into a single channel before sending it to the AI Agent. While simple to implement, mixing audio makes it incredibly difficult for the Speech-to-Text (S2T) model to transcribe accurately, especially when users talk over one another. This results in “hallucinations” or missed queries.
• Agent per client: This approach assigns a separate Voice AI agent to every single player in the room. This creates a chaotic user experience (all agents speaking at once) and prevents a shared game state. It is also cost-prohibitive, as every user stream consumes separate processing tokens.
• Server-side filtering using VAD: In this approach, we implemented a centralized gatekeeper using Voice Activity Detection (VAD). We wait for a player to speak, lock the “input slot” and forward only that specific player’s audio to the AI agent. Once they stop speaking, the lock is released, allowing another player to ask questions. This is the solution we finally went with.

Beyond one-on-one: A “Deep Sea Stories” game web app

Key technologies

• Fishjam: A real-time communication platform handling peer-to-peer audio streaming via WebRTC (SFU). (Not familiar with WebRTC/SFUs? Check out our guide)
• Gemini GenAI Voice Agent: Provides an easy SDK that makes creating voice agents and initializing audio conversations simple.

Architecture overview

The game logic is handled on the backend, which manages the conferencing room and peer connections.

• Player connection: When players join the game using frontend client, they connect audio/video via the Fishjam Web SDK. (See: Fishjam React Quick Start).
• The bridge: When the game starts, the backend creates a Fishjam Agent. This agent acts like a “ghost peer” in the audio-video room; its sole purpose is to capture audio of the players and forward it to the AI, and vice versa.
• The brain: The backend initiates a WebSocket connection with the Gemini agent and forwards the audio stream from players to Gemini and vice versa.

Implementation details

1. Initalizing Clients and game room

import { FishjamClient } from '@fishjam-cloud/js-server-sdk';
import GeminiIntegration from '@fishjam-cloud/js-server-sdk/gemini';

const fishjamClient = new FishjamClient({
  fishjamId: process.env.FISHJAM_ID!,
  managementToken: process.env.FISHJAM_TOKEN!,
});

const genAi = GeminiIntegration.createClient({
  apiKey: process.env.GOOGLE_API_KEY!,
});

const gameRoom = await fishjamClient.createRoom();

2. Creating the Fishjam agent

When the first player joins the game room, we create the Fishjam agent to capture players’ audio on the backend.

import GeminiIntegration from '@fishjam-cloud/js-server-sdk/gemini';

const { agent } = await fishjamClient.createAgent(gameRoom.id, {
  subscribeMode: 'auto',
  // Use our preset to match the required audio format (16kHz)
  output: GeminiIntegration.geminiInputAudioSettings,
});
// agentTrack enables to send audio back to players 
const agentTrack = agent.createTrack(GeminiIntegration.geminiOutputAudioSettings);

3. Configuring and initializing the AI riddle master

When users select a story scenario, we configure the Gemini agent with the specific context (the riddle solution and the “Game Master” persona).

const session = await genAi.live.connect({
  model: GEMINI_MODEL,
  config: { 
      responseModalities: [Modality.AUDIO],
      systemInstruction: "here's the story: ..., and its solution: ... you should answer only yes or no questions about this story",
  },
  callbacks: {
    // Gemini -> Fishjam
    onmessage: (msg) => {
      if (msg.data) {
        // send Riddle Master's sudio responses back to players
        const pcmData = Buffer.from(msg.data, 'base64');
        agent.sendData(agentTrack.id, pcmData);
      }

      if (msg.serverContent?.interrupted) {
        console.log('Agent was interrupted by user.');
        // Clears the buffer on the Fishjam media server
        agent.interruptTrack(agentTrack.id);
      }
    }
  }
});

4. Bridging audio (the glue)

The final piece of the puzzle is the bridge between the SFU and the AI. We capture audio streams from the Fishjam agent (what the players are saying) and pass them through a custom VAD (Voice Activity Detection) filter. This filter implements a “mutex” lock mechanism: it identifies the first active speaker, locks the channel to their ID, and forwards only their audio to Gemini. All other simultaneous audio is ignored until the active speaker finishes their turn.

Below is the simplified code of this logic:

// State to track who currently "holds the floor"
let activeSpeakerId: string | null = null;

// We capture audio chunks from ALL players in the room
agent.on("audioTrack", (userId, pcmChunk) => {
    vadService.process(userId, pcmChunk);
});

// VAD Processor Logic
vadService.on("activity", (userId, isSpeaking, audioData) => {
    if (activeSpeakerId === null && isSpeaking) {
        activeSpeakerId = userId; // Lock the floor
    }

    // We only forward audio if it comes from the person holding the lock
    if (userId === activeSpeakerId) {
        voiceAgentSession.sendAudio(audioData);

        // If the active speaker stops speaking (silence detected), release the lock
        if (!isSpeaking) {
             // (Optional: Add a debounce delay here to prevent cutting off pauses)
             activeSpeakerId = null; 
        }
    }
});

Challenges in group AI

Building a multi-user voice interface introduces unique challenges compared to 1-on-1 chats:

• Floor control: Standard Speech-to-Text models can struggle when multiple players speak simultaneously. Determining which player the AI should respond to or if it should simply listen requires careful handling.
• Latency: Real-time responsiveness is critical for immersion. The entire pipeline (Audio → Text → LLM → Audio) must execute in milliseconds.
• Audio quality: Maintaining clear audio through transcoding and streaming across different networks is essential.

Fortunately, Fishjam’s WebRTC implementation largely solves the latency and audio quality issues. The challenges of Floor Control needed carefully structured implementation on the backend, but it was not really that hard!

Try the game yourself!

We have implemented the functionality described above in a live demo. Gather your friends and try to solve a mystery with our AI Riddle Master!

• Play the Demo: Deep Sea Stories
• View the Code: GitHub Repository

If you are working on AI-based features with real-time video or audio and need assistance, feel free to reach out to us on Discord.

We are Software Mansion — software development consultants, a team of React Native core contributors, multimedia and AI experts. Drop us a line at projects@swmansion.com and let’s find out how we can help you with your project.

Voice AI: How We Built a Multi-Speaker AI Agent using Gemini was originally published in Software Mansion on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the previous parts of our series, we’ve explored the key concepts behind building AI agents with LangChain and LangGraph, and we’ve constructed a fully-functional AI chatbot along the way. Today, we’re diving into a case study from the development of our very own marketing agent.

At Software Mansion, we specialize in building AI tools. Finding places where people might benefit from them is time-consuming, so we wanted to automate the process with an AI agent identifying opportunities to showcase our solutions. In this article, we’ll tell you how we built the agent that uses a search engine to find websites where our products could be promoted. The code is available here.

Our approach relies on a system consisting of three agents: Critic, Selector, and Search. Assigning non-overlapping tasks to different agents improves token usage efficiency, simplifies history management, and enhances results by keeping each context smaller. Different foundation models can also be used per agent based on task complexity.

The Critic agent

First, the Critic takes specific website content as input and evaluates if it’s suitable for advertising our products. We use structured output, so the model lists the pros and cons of selecting the website. The process starts by loading a description of the product to advertise, then introducing the Critic’s role via a system prompt, and finally passing the website content for evaluation.

Since multiple websites are passed to the Critic, we use batch invocation to save time. Our agentic workflows are I/O-bound, spending most of their time waiting for responses from the foundation model API, so running several invocations concurrently helps speed up the process.

To use batch invocation, we simply need to pass a list of initial states instead of a single one, and we receive a list of results in return:

initial_states = [{...}, {...}, ...]
responses: list[BaseMessage] = model.batch(initial_states)

The Selector agent

Another agent, very similar in structure, is the Selector. At this point, we have a list of candidate websites along with evaluations of their suitability for advertising. The Selector uses these evaluations to decide which website to return.

First, the product description is loaded, then we define the Selector’s role through a system prompt, and finally, the selection is made. Structured output is used again to ensure the response follows the correct format.

The Search agent

The Search agent is the core of the system, orchestrating the two remaining agents. In our case, it uses a search engine based on DuckDuckGo as a tool.

Like the other agents, it begins by loading the product description. Then, the foundation model uses a tool to search the web and selects which websites to load based on their titles and summaries. Once loaded, the content is passed to the Critic. Based on the Critic’s feedback and the iteration limit, the model either continues searching (goes back to search) or summarizes the results by calling the Selector.

The marketing agent in action

The main goal of our project was to automate the process of finding marketing opportunities for React Native ExecuTorch. The prototype was initially built with smolagents, but we later migrated to the LangChain environment, which gave us greater control, reduced token usage, and produced more precise results.

As the project matured, we expanded it to cover other products and built a web UI to give everyone at our company — not just AI developers — easy access to the agentic system.

Now, users of our marketing agent can conveniently store configurations for multiple products they want to promote. As an example, we’ll show you the setup for our React Native ExecuTorch library:

First, the user needs to describe the product, the use cases, the audience, and other important information. Next, there is a field where agentic prompts can be edited. Each comes with a description of its role in the system, so that the user knows exactly what to edit without having to understand the workflow of our subagents.

Configurations are stored in the browser memory, but they can also be imported or exported. Once the configuration is ready, users just need to click “Start Search” and wait for the results. When the search is complete, they get a list of websites for advertisement, along with justifications for choosing them.

That was a quick glimpse into our real-world project, which has saved us a lot of time and is now used in our products. We hope that after reading this series, you’re ready to build efficient agentic systems with LangGraph. Good luck!

We are Software Mansion — software development consultants, a team of React Native core contributors, multimedia and AI experts. Drop us a line at projects@swmansion.com and let’s find out how we can help you with your project.

Building Agents With LangGraph Part 4/4: How We Built Our Marketing Agent was originally published in Software Mansion on Medium, where people are continuing the conversation by highlighting and responding to this story.

Several years ago, Huawei, together with industry partners, started working on a brand new operating system. This is how OpenHarmony was born, an open-source competitor to the well-established iOS and Android markets. Similarly to how Google builds Android on top of AOSP, Huawei develops its proprietary HarmonyOS NEXT distribution from the OpenHarmony codebase, which powers Huawei devices.

We’re happy to share that Software Mansion has been cooperating with Huawei from the start on this project, and we’ve been taking an active part in developing React Native OpenHarmony (RNOH) — a React Native library letting you run apps directly on OpenHarmony devices.

Understanding HarmonyOS Next

Since HarmonyOS Next is a completely separate operating system, native apps from other platforms aren’t directly compatible with it. However, while it differs from Android and iOS, it still feels familiar and shares many concepts when it comes to interactions and overall user experience.

The Developer Experience is also quite familiar. You can use ArkTS and C++ to write HarmonyOS Next-native applications, similar to using Swift on iOS and Kotlin on Android.

ArkTS is based on TypeScript, so this means developers don’t need to learn a new technology from scratch to build apps for HarmonyOS Next. To make the development process more comfortable, Huawei has prepared a dedicated IDE for Harmony called DevEco Studio. It’s similar to Android Studio, so if you have prior experience with Android native development, the onboarding process should be pretty seamless.

Developing React Native Open Harmony (RNOH)

As already mentioned, React Native Open Harmony (RNOH) is a library that makes it possible to run React Native apps on OpenHarmony devices. You can use it by either integrating it into an existing app or starting a new project from scratch. RNOH is built using React Native’s New Architecture to ensure seamless performance and reliability. By the way, if you haven’t migrated your app yet, it’s the best time to do so, because from React Native 0.82 onwards, only the New Architecture is supported. This means the migration is crucial if you want to benefit from the latest features — you can visit our website to learn more about the benefits of migrating and get support.

For now, RNOH supports React Native 0.77 and 0.72, and we’re currently working on adding support for 0.82 to ensure developers can take advantage of the latest features. However, for legacy use cases, we’ve also created a Fabric-based adaptation for version 0.61 to maintain compatibility with older applications.

While React Native Open Harmony is still a young project, it has already established its role in the HarmonyOS ecosystem. Many well-known applications have adopted it, including — but not limited to — major social media platforms like YY and Xiaohongshu (Little Red Book), as well as CTrip (the Chinese domestic version of Trip.com).

Today, more than 27 million devices on the Chinese market are running on HarmonyOS Next. Already, more than 300 000 apps and meta-services are available, and there are more than 10 million registered developers. It’s exciting for us to be part of this ecosystem!

How we helped create React Native Open Harmony

Our cooperation with Huawei began as a series of proof-of-concept implementations, eventually growing into a long-term collaboration, with our team developing a significant portion of RNOH’s core code.

Although Fabric is mostly written in C++, integrating it with HarmonyOS Next still posed significant challenges. We had to put considerable effort into bridging the gap between ArkTS and React Native’s codebase, as well as mapping React Native’s core components to their native equivalents on HarmonyOS Next.

In addition, we’ve built the native OpenHarmony implementation of the TurboModule system from scratch, which is used to access native APIs for functionality like networking and vibrations. Through this work, we’ve become familiar with HarmonyOS’s runtime environment, component system, UI rendering pipeline, and native development workflow using ArkTS. We’ve learned how to bridge OpenHarmony’s platform-specific conventions with React Native’s cross-platform architecture — knowledge that we continue to apply as we expand RNOH’s capabilities and performance.

As modern React Native apps rely heavily on a vibrant ecosystem of open source libraries, we’ve also made sure that key libraries are supported on OpenHarmony. Without them, it wouldn’t be a practical and usable platform for React Native apps.

Our Software Mansion team took on the task of porting several widely used libraries, helping us to further increase our OpenHarmony expertise. Some of the libraries ported by our team include:

• react-native-reanimated
• react-native-gesture-handler
• react-native-screens
• react-native-webview
• react-native-svg
• react-native-safe-area-context

At the same time, Huawei worked on porting other important libraries, significantly expanding OpenHarmony’s compatibilities within the broader React Native ecosystem.

This project has been a rewarding journey for our team, allowing us to deepen our expertise in React Native and its modern Fabric architecture even more.

Beyond the React Native side, it also allowed us to gain a much deeper understanding of HarmonyOS NEXT itself, strengthening our ability to integrate React Native with diverse native ecosystems.

What’s in store for React Native OpenHarmony?

Our partnership with Huawei continues as we work on introducing new features, resolving bugs, improving performance, and updating React Native OpenHarmony to stay compatible with the latest versions of React Native.

If you need help migrating your existing application to HarmonyOS or want to develop a new application that’s compatible with Huawei’s platform, you can always reach out to us at projects@swmansion.com. We’ll be happy to use our first-hand experience to help you build your HarmonyOS app!

We’re Software Mansion: multimedia experts, AI explorers, React Native core contributors, community builders, and software development consultants.

Huawei x Software Mansion: Bringing React Native Support to HarmonyOS NEXT was originally published in Software Mansion on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the previous part of our series, we’ve added web search and intelligent shutdown to our chatbot to explore tools and structured output.

In this tutorial, we’re going to enrich our chatbot with two functionalities — Retrieval Augmented Generation and long-term memory. You can find scripts from this part here.

If you’re not familiar with how Retrieval Augmented Generation works, make sure to read our in-depth article on RAG.

Adding Retrieval Augmented Generation to our LangGraph chatbot

Let’s start by loading the doc(s) — as an example, we’re going to use the React Native ExecuTorch documentation. The library lets you run AI models natively on edge devices:

from langchain_community.document_loaders import WebBaseLoader

loader = WebBaseLoader("https://docs.swmansion.com/react-native-executorch/")
docs = loader.load()

Then, we need to split the loaded docs into chunks:

from langchain_text_splitters import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
all_splits = text_splitter.split_documents(docs)

We also need to create a vector store for chunks, which is based on an embedding model. Embeddings don’t have to match those used by the foundational model, since they’re only needed for retrieving text from the vector store. A popular option here is all-MiniLM-L6-v2.

from langchain_core.vectorstores import InMemoryVectorStore
from langchain_huggingface import HuggingFaceEmbeddings

embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
vector_store = InMemoryVectorStore(embeddings)

Next, we add the chunks to the vector store:

_ = vector_store.add_documents(documents=all_splits)

The only thing left is to retrieve the relevant pieces for the user’s prompt. To do that, we update ask_llm. In this example, we just pass the retrieved text directly as context. Alternatively, you could call the LLM separately to generate a summary of that context — feel free to try that approach as an exercise.

def ask_llm(state: State) -> State:
    user_query = input("query: ")

    retrieved_docs = vector_store.similarity_search(user_query)
    context = "\n\n".join([doc.page_content for doc in retrieved_docs])
    user_message = HumanMessage(f"Context:\n{context}\n\nUser question:\n{user_query}")

    # homework: try to add an intermediate step, where a prompt is created by the LLM based on the query and context, before passing it to the LLM below.

    answer_message: AIMessage = model_with_search.invoke(
        state["messages"] + [user_message]
    )

    return {
        "messages": [user_message, answer_message],
    }

)

Check our GitHub repository to see what the final code should look like. The parts added in this tutorial are highlighted below:

from langchain_core.vectorstores import InMemoryVectorStore
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_community.document_loaders import WebBaseLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter

embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-MiniLM-L6-v2")
vector_store = InMemoryVectorStore(embeddings)

loader = WebBaseLoader("https://docs.swmansion.com/react-native-executorch/")
docs = loader.load()

text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
all_splits = text_splitter.split_documents(docs)

_ = vector_store.add_documents(documents=all_splits)


def ask_llm(state: State) -> State:
    user_query = input("query: ")

    retrieved_docs = vector_store.similarity_search(user_query)
    context = "\n\n".join([doc.page_content for doc in retrieved_docs])
    user_message = HumanMessage(f"Context:\n{context}\n\nUser question:\n{user_query}")

    # homework: try to add an intermediate step, where a prompt is created by the LLM based on the query and context, before passing it to the LLM below.

    answer_message: AIMessage = model_with_search.invoke(
        state["messages"] + [user_message]
    )

    return {
        "messages": [user_message, answer_message],
    }

)

query: Hi
answer:  Hello! How can I assist you today?
query: what's react native executorch?
answer:  React Native ExecuTorch is an on-device AI and large language model (LLM) toolkit tailored for the React Native ecosystem. It allows developers to run AI models and LLMs locally on mobile devices, leveraging Meta’s ExecuTorch AI framework. Here are some key aspects of React Native ExecuTorch:

1. **On-Device Model Execution**: It enables AI models and LLMs to operate directly on the device, ensuring that user data remains private and eliminating the need for external API calls.

2. **Cost-Effective**: By performing computations on-device, it reduces dependency on cloud infrastructure, which can lower server costs and minimize latency.

3. **Privacy-Focused**: By executing models locally, it ensures that data stays on the device, offering maximum privacy for users.

4. **Developer-Friendly**: With a declarative API for on-device inference, developers do not need extensive AI expertise to implement these models in their React Native applications.

5. **Part of PyTorch Edge**: It is built on the ExecuTorch foundation, which is part of the PyTorch Edge ecosystem, enabling efficient AI deployment for cross-platform applications in React Native.

This toolkit is ideal for developers looking to integrate AI capabilities within their React Native apps without relying on external cloud services.
query: Can I see it in action?
answer:  To see React Native ExecuTorch in action, you can download and use "Private Mind," which is an on-device AI chatbot. This application operates entirely offline and showcases the capabilities of React Native ExecuTorch. Here’s how you can interact with it:

1. **Download and Install**: You can download Private Mind from the App Store or relevant app distribution platforms.

2. **Features to Explore**:
   - **Chat Freely**: Engage in unrestricted conversations with the AI chatbot.
   - **Privacy Assurance**: Experience the privacy benefits as all data processing occurs locally on your device.
   - **Model Testing and Benchmarking**: Browse, test, and evaluate local language models supported by React Native ExecuTorch.
   - **Customization**: Customize AI assistants to fit your workflow and personal style.

This application serves as a practical demonstration of how React Native ExecuTorch enables the deployment of AI models on mobile devices. It illustrates the privacy, cost-effectiveness, and variety of models you can leverage through this toolkit.

Adding long-term memory

So far, you’ve seen how to add short-term memory — the message history within a single conversation. There’s also long-term memory, which persists across workflow invocations. In this article, we’ll focus on a simple example of resuming a conversation, though the topic is much broader. You can read more about it here.

First, we need to set up a checkpointer and a store, which we’ll pass as parameters when compiling the graph:

from langgraph.checkpoint.memory import InMemorySaver
from langgraph.store.memory import InMemoryStore

checkpointer = InMemorySaver()  # used to store individual states
store = InMemoryStore()  # used to store states acrross threads
workflow = graph.compile(checkpointer=checkpointer, store=store)

To be able to resume the conversation, you need to identify it with thread_id:

config = {"recursion_limit": 100, "configurable": {"thread_id": "default-session"}}

Now, you pass the config when invoking the workflow, and you can resume thanks to get_state:

# we start a thread
workflow.invoke(
    {"iteration": 0},
    config=config,
)

# now we resume the conversation (thread) in another invocation
workflow.invoke(
    workflow.get_state(config),  # last state of the agent
    config=config,
)

# homework: modify the agent so that it stores the state externally and loads the summarization or truncated history of your previous discussions when you run the script and resumes from there.

Your script should look like this, with the part added in this tutorial shown below:

from langgraph.checkpoint.memory import InMemorySaver
from langgraph.store.memory import InMemoryStore

checkpointer = InMemorySaver()
store = InMemoryStore()
workflow = graph.compile(checkpointer=checkpointer, store=store)

config = {"recursion_limit": 100, "configurable": {"thread_id": "default-session"}}

workflow.invoke(
    {"iteration": 0},
    config=config,
)

workflow.invoke(
    workflow.get_state(config),
    config=config,
)

Congrats! You’ve finished building your agentic AI chatbot in LangGraph.

What’s next in the series?

In part 4, we’re diving into a real-world agentic system we built to identify marketing opportunities. This case study offers a hands-on look at how a LangGraph-based system can help your team in practice. So, be sure to check it out here.

We are Software Mansion — software development consultants, a team of React Native core contributors, multimedia and AI experts. Drop us a line at projects@swmansion.com and let’s find out how we can help you with your project.

Building Agents With LangGraph Part 3/4: RAG and Long-Term Memory was originally published in Software Mansion on Medium, where people are continuing the conversation by highlighting and responding to this story.