<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kirsten Lindsmith on Medium]]></title>
        <description><![CDATA[Stories by Kirsten Lindsmith on Medium]]></description>
        <link>https://medium.com/@kirstenlindsmith?source=rss-ea893de087f8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*0cvVkvKc9H8xvPrZ.jpg</url>
            <title>Stories by Kirsten Lindsmith on Medium</title>
            <link>https://medium.com/@kirstenlindsmith?source=rss-ea893de087f8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 03:56:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kirstenlindsmith/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Translating PoseNet into React.js]]></title>
            <link>https://medium.com/@kirstenlindsmith/translating-posenet-into-react-js-58f438c8605d?source=rss-ea893de087f8------2</link>
            <guid isPermaLink="false">https://medium.com/p/58f438c8605d</guid>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[react]]></category>
            <category><![CDATA[posenet]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[reactjs]]></category>
            <dc:creator><![CDATA[Kirsten Lindsmith]]></dc:creator>
            <pubDate>Sat, 16 Mar 2019 15:17:52 GMT</pubDate>
            <atom:updated>2019-03-16T16:18:50.783Z</atom:updated>
            <content:encoded><![CDATA[<p>PoseNet is a supremely useful real-time machine learning model, built to track body position. Part of the TensorFlow.js family, this pretrained model is accessible to seasoned programmers and computer-shy artists alike, thanks to the flexibility of Javascript*.</p><p>While PoseNet is easy to use at the basic level, its out-of-the-box tutorials from TensorFlow.js are designed to manipulate the DOM directly, an often tedious task when designing more complex software. In this article, we will go over the basics of adapting PoseNet for use with React.js, a library that makes the process of manipulating the DOM both streamlined and scalable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*hUD9dpf-U9U5nAUk3wgk0A.gif" /><figcaption>from the TensorFlow.js <a href="https://github.com/tensorflow/tfjs-models/tree/master/posenet/demos">official PoseNet demo</a></figcaption></figure><p>First, let’s take a look at a few ways in which PoseNet’s official github demo loads the model and manipulates the DOM, and their counterparts in React.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/587/1*jXd2av27vT1UFivmPF-p6w.png" /><figcaption>A simplified version of the PoseNet demo’s setUpCamera function, including global video sizing variables and PoseNet model import</figcaption></figure><p>The most important thing to note here is the direct call to <em>document</em> on line 37. How might we do this in React?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*d9yUqhGJKN9RcfrbTaSY9g.png" /><figcaption>setUpCamera as seen in a React component</figcaption></figure><p>At first glance this looks nearly identical. But note line 67. Where are we getting the video from, if not a direct access method like <em>document.getElementById</em> (or similar)? What is <em>this.video </em>and where did it come from? Line 34 in the code below is the first step:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/1*--v7pMvPwfsAo-iEA_OD1g.png" /><figcaption>example of using static defaultProps for config, and DOM-grabbing helper functions</figcaption></figure><p>Note: The first aspect of setting up PoseNet as a React-compatible model isn’t strictly necessary, but I personally would highly recommend it. React.js allows the static serving of default props, as seen on line 7 above. Then, if those static props are referenced in the constructor’s call to super (see line 27), they will become available in a way that is functionally analogous to global variables. This is how we can set (and easily alter) config variables like the <em>videoWidth</em> and <em>videoHeight</em>.</p><p>Now let’s get back to accessing the webcam and manipulating the DOM canvas. Lines 30 and 34 are essential in this process. Instead of grabbing a DOM element with <em>document.getElementById</em>, we will directly assign the class component’s <em>this.video </em>variable to a specified DOM element. These two functions won’t be called until the React component’s render function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/1*ELj8oH7zqk3ZCViNxsuf1Q.png" /><figcaption>using this.getVideo and this.getCanvas</figcaption></figure><p>If we attach these two helper functions directly to the JSX returned by <em>render, </em>then <em>this.video </em>will automatically refer to the video element, and <em>this.canvas </em>will refer to the canvas element. (The ID on line 201 will be explained later).</p><p>Running the <em>setUpCamera</em> function in vanilla PoseNet is done as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*D55dK11Uk6EcGV2cbeEd2g.png" /><figcaption>loadVideo calls setUpCamera and then .play</figcaption></figure><p>The<em> loadVideo</em> function is called at the end of the PoseNet demo, in a master setup function, <em>bindPage</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*xholRMpX0LxF3cxHd8Sjcw.png" /></figure><p>The React version of this uses the built-in function, <em>componentDidMount</em>, which, as the name implies, runs automatically upon the initial mounting of the component to the page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/1*Jx8OF9RAOO8-geqdkzrhkA.png" /></figure><p>The call to <em>setUpCamera</em> and loading the PoseNet model are in two seperate try/catch blocks to give more accurate error reporting. This way, if an error does occur, it will be clear exactly which of the two steps failed.</p><p>You’ll notice that on line 57 we’re making a call to a function called <em>detectPose</em>. This is analogous to the call to <em>detectPoseInRealTime</em> on line 296 of the previously seen <em>bindPage</em> function. Let’s explore the differences between how vanilla PoseNet and the React adaptation are actually running TensorFlow.js’ PoseNet model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*Jcnm3UOtGd8RlP975GLUWg.png" /><figcaption>PoseNet’s out-of-box demo is complicated by their GUI that lets the user adjust config in real-time</figcaption></figure><p>The above code block may look complex and scary, but a large portion of that is the demo’s GUI cluttering up the view. This is removable for anyone wanting to run vanilla PoseNet in their own app, but I’m leaving it in because this article is about adapting the out-of-box demo code.</p><p>A few of the key points to note are lines 187 (pulling the canvas from the DOM directly, the same way the webcam set up, i.e. <em>video </em>was handled), the <em>switch</em> block starting on line 207, and the <em>forEach</em> on line 238 that makes calls to various draw functions. The <em>requestAnimationFrame</em> function on line 254 is built into PoseNet.</p><p>Let’s compare this to the React version, which I’m calling <em>detectPose. </em>Its helper function,<em> poseDetectionFrame </em>has been set aside as its own standalone function, though you could still nest it inside <em>detectPose</em> if you prefer that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*InZ7NspA0EZUjhwSBYSU4Q.png" /><figcaption>Collapsed lines 103, 113, and 117 in this article for vertical space at the unfortunate expense of readability</figcaption></figure><p>Another long and scary looking function. In many ways, this <em>detectPose</em> function is extremely similar to its PoseNet counterpart, so let’s step through it to figure out how exactly we’re using the data PoseNet produces.</p><p>The <em>detectPose</em> function grabs its size variables from the statically served <em>defaultProps</em>, and then pulls the canvas context using the same method as vanilla PoseNet. It then makes a call to <em>poseDetectionFrame</em>, the real mover and shaker, so let’s dive right into that.</p><p>First, on line 102 we are pulling a whole lot of global variables from those same statically served <em>defaultProps</em>. Because this isn’t meant to be a tutorial on PoseNet itself, I won’t go into too much detail explaining what they all mean. The most important ones, all relating to “pose confidence” are essentially the threshold at which we the developers can choose to tell PoseNet to trust its guesses about a pose location. A small minimum confidence threshold tells PoseNet “even if you’re not all that sure, give us the data anyway,” whereas a large minimum confidence threshold would tell PoseNet “don’t give us the location data unless you’re really certain about its accuracy!”</p><p>Next, we dive into a nested asynchronous helper function. The async aspect is essential, as we are going to need to wait on the PoseNet model itself to use its built in methods to estimate poses, a fairly hefty task.</p><p>Inside the <em>switch</em> case, we are either going to use the <em>multi-pose</em> algorithm available in PoseNet, or <em>single-pose</em>. If we’re recording multiple people, we’re going to just set that <em>poses</em> array to hold everything PoseNet can find. Alternatively, the single-person algorithm produces only one pose at a time, thus the need to <em>push</em> on line 118.</p><p>Again, this isn’t meant to be an in-depth PoseNet tutorial, so we’re going to skim over a lot of the meat of this function. The biggest thing to note is how we use <em>forEach </em>on each pose in the resulting <em>poses</em> array to draw the key body points and skeleton.</p><p>But wait, we have one final step!</p><p>If you were to render this React component as-is, you would see <em>two </em>videos, one on top of the other. The first, oddly reversed, is the raw webcam footage. The second is the properly mirrored canvas with the key points and skeleton PoseNet shows us based on our minimum confidence thresholds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*VmnDRKhNBHcsh-EfoBmzuA.png" /><figcaption>PoseNet recognizes my painting as a human figure, the highest compliment an artist can hope for</figcaption></figure><p>What’s going on? This is where that unexplained video ID on line 163 of the React component’s render function comes into play. The <em>video</em> element is essential to getting the webcam up and running, but the <em>canvas</em> is where the PoseNet is working its magic (and listening to our config, like <em>flipHorizontal</em>, which does the mirroring).</p><p>In the CSS stylesheet, the ID <em>videoNoShow</em> is enabled to hide that video element. But your likely first instinct, <em>display: none,</em> isn’t enough in this case. We also need to transform the element:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/286/1*uWueEKThwOMWB6ZOxTJ6LQ.png" /><figcaption>video element CSS</figcaption></figure><p>Congratulations! After that final tweak, you’re now ready to render your PoseNet React component. It’s up to you to build on top of PoseNet to create a unique application, but now you can use React.js to make your life easier during the process. Enjoy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VcwdXSNjA4CpzDBM0rRMCg.png" /></figure><p>P.S. You can download the boilerplate code to run this React adaptation of PoseNet <a href="https://github.com/kirstenlindsmith/PoseNet_React">here on github</a>.</p><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p>*Note: One of the things that makes PoseNet so accessible to non-programmers is the creation of tools like ML5 and P5. If you aren’t a Javascript expert but still want to start playing around with PoseNet, I recommend checking out <a href="https://www.youtube.com/watch?v=EA3-k9mnLHs">this basic tutorial</a> that uses the aforementioned libraries.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=58f438c8605d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>