<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sha Qian on Medium]]></title>
        <description><![CDATA[Stories by Sha Qian on Medium]]></description>
        <link>https://medium.com/@shaqian629?source=rss-aa546742d090------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*jO0sITmqWJC4O1GT.jpg</url>
            <title>Stories by Sha Qian on Medium</title>
            <link>https://medium.com/@shaqian629?source=rss-aa546742d090------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 23:46:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shaqian629/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Using PiHelper to monitor and manage Raspberry Pis]]></title>
            <link>https://medium.com/swlh/using-pihelper-to-monitor-and-manage-raspberry-pis-9a24065febed?source=rss-aa546742d090------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a24065febed</guid>
            <category><![CDATA[raspberry-pi]]></category>
            <category><![CDATA[sftp]]></category>
            <category><![CDATA[ios-apps]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[ssh]]></category>
            <dc:creator><![CDATA[Sha Qian]]></dc:creator>
            <pubDate>Wed, 18 Mar 2020 16:13:19 GMT</pubDate>
            <atom:updated>2020-11-08T11:18:55.023Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://apps.apple.com/us/app/pihelper/id1369930932"><strong>PiHelper</strong></a> is an iOS app that can monitor and manage Raspberry Pi or similar Linux machines.</p><p>This guide will help you get started and get the most out of PiHelper.</p><p><strong>Why the app reads clipboard:<br></strong>In TERM, above the keyboard and under ESC button, there’s a clipboard-like (paste) button. When user taps on the button, the app reads the data in clipboard and paste it to the shell. It is useful when you need to create or edit a file. The app does NOT use the data anywhere else.</p><h3><strong>Overview</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K3EVPRv3Aa-rLi9FQ7UygQ.jpeg" /><figcaption>The 5 main screens of the app</figcaption></figure><p>From left to right, the screens are</p><ul><li><strong>Main </strong>screen<strong>: </strong>where you can have<strong> </strong>an overview of all machines, add/delete a machine, and change settings of the app.</li><li><strong>Dashboard</strong> screen: where you can check the real-time status of a machine, run custom commands, and go to GPIO, Terminal and SFTP screens.</li><li><strong>GPIO</strong> screen: shows GPIO status and lets you manage the pins.</li><li><strong>Terminal</strong> screen: a simple SSH client.</li><li><strong>SFTP</strong> screen: a simple SFTP client.</li></ul><p>Now let’s take a closer look at each screen and their functionalities.</p><h3><strong>Main screen</strong></h3><h4>Add a connection</h4><p>When you open the app for the first time, there’s nothing on the main screen but an Add connection button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*Q7iahZNoRXBIiSRQcCd5RA.png" /><figcaption>Tap the button to create a new connection</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*CXdUlbJqAGNcktf-mE9Vvg.png" /></figure><p>To create a connection to a machine, the following information is needed:</p><ul><li><strong>Connection name</strong>: Any name that can help you identify this machine.</li><li><strong>Host</strong>: An IP address (i.e. <em>10.0.0.1</em>) or a host name (i.e. <em>testmypi.com</em>).</li><li><strong>Port</strong>: The port on which SSH is listening. By default, it’s <em>22</em>.</li><li><strong>Password</strong> or <strong>Private key</strong>: The authentication method. You can select either one of them.</li><li><strong>Username</strong>: The username of the SSH user. Used in both Password and Private key authentication.</li><li><strong>Password</strong>: The password of the SSH user. Used only in Password authentication.</li><li><strong>Private key</strong>: The private key used by the SSH user. Used only in Private key authentication.</li><li><strong>Passphrase</strong>: Optional. Used only when the private key is protected by a passphrase.</li><li><strong>Sudo password</strong>: Optional. Used only by <em>Reboot</em> and <em>Halt</em> button.</li></ul><p>For private key authentication, here’s an example of how the key should look like. Note the app is tested with RSA keys only.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*BPyJSgO2wP8ufwumyQzTNA.png" /></figure><p>There are 2 buttons at the bottom of the screen.</p><ul><li><strong>Test connection</strong>: Test the connection to the machine. The result will pop up from the bottom.</li></ul><p>Note that connection error usually indicates an issue with the machine or network, while authentication error means when connection is established but the credentials (<em>username</em>, <em>password</em> or <em>key</em>) are wrong.</p><ul><li><strong>Save</strong>: Save the connection and go back to main screen.</li></ul><p>Once you setup the connections, the app will automatically connect to all machines and show their status (CPU, memory and disk usage) on the main screen. The status is refreshed <strong>every 10 seconds</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*2D3aagfs_whbuBCs9bQjZw.png" /></figure><h4>Edit or d<strong>elete a connection</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/33/1*kkWlaQlBAQjeTMdbeo6w-A.png" /><figcaption>Tap the button to delete a connection</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/33/1*OAcOw6aFG1Q8BlUUdMzpJA.png" /><figcaption>Tap the button to edit a connection</figcaption></figure><h4>Reboot or halt a machine</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/33/1*UHELf52vf93amievkXe9WQ.png" /><figcaption>Tap the button then select Reboot or Halt</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/324/1*7MohST9_mZyBGghQM64hOg.png" /></figure><h4>Reorder the connections</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/33/1*hFlpQI7ug-JIACIh5QTybg.png" /><figcaption>Tap the right button on header bar</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/258/1*Umpq1CUmClofx49woBzTtA.png" /></figure><p>You can sort the connection by the following property in <strong>ascending</strong> or <strong>descending</strong> order:</p><ul><li><strong>Creation</strong> time</li><li>Connection <strong>Name</strong></li><li><strong>Host</strong></li></ul><h4>Change settings of the app</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/33/1*p3RVyL2wjRV-qmAeZuFfXw.png" /><figcaption>Tap the left button on header bar</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/266/1*Q6tW67AcTuifcIOBmXT6Fg.png" /></figure><ul><li><strong>Language</strong>: English, Chinese, Japanese or Russian</li><li><strong>Temperature scale</strong>: Celsius or Fahrenheit</li><li><strong>Remove Ads:</strong> Pay $1 to support the app</li><li><strong>Restore purchase</strong>: Restore your previous purchase</li><li><strong>Report issue</strong>: Email to <em>pihelper@outlook.com</em> to report issue</li><li><strong>Write review</strong>: Review the app on App Store</li></ul><h3>Dashboard screen</h3><p>The charts and graphs on Dashboard screen are updated <strong>every 5 seconds</strong>. Refer to the last section of this article for the commands used by the app to get the data.</p><h4>CPU and Memory chart</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*uNtHleQpCfD2EjkBxhMTgg.png" /><figcaption>Shows the overall CPU and memory usage</figcaption></figure><h4><strong>Top CPU process</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*tiJ6dZLrkDLdOAaxRakjpg.png" /><figcaption>Shows the 5 processes with top CPU usage</figcaption></figure><h4>Storage usage</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/158/1*CqFDFmtiBWrfloYrgtkyCA.png" /><figcaption>Shows used and free disk space</figcaption></figure><h4>Temperature</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/159/1*45vV7rO-R5QSvS-84uPiAA.png" /><figcaption>CPU and GPU temperature</figcaption></figure><p>Note the scale (°C or °F) can be changed in settings on main screen.</p><h4>Custom commands</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/33/1*0KKeooAF6X2KFK1Kh5rcWg.png" /><figcaption>Tap the left button on header bar</figcaption></figure><p>In the commands screen, you can setup up to 15 custom commands.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*I3MWJ6ki4u5TtRo7DDCdqA.png" /></figure><p>The saved commands are listed in the dropdown menu.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*wuozqbdsulZ2GoIgh4dn5Q.png" /></figure><p>When you tap a custom commands, wait for a few seconds and let it complete. The app will show the result of the command: the output if there’s any, success or failure if there’s no output.</p><h4>Go to other screens</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/314/1*vwLMZfj1uQ4mIZhUaSiKtA.png" /><figcaption>These buttons navigate you to other screens.</figcaption></figure><h3>GPIO screen</h3><p>This screen shows the GPIO status. You can tap the <em>OUT/IN</em> or <em>0/1</em> button under <em>MODE</em> or <em>V </em>column<em> </em>to change the input/output of a pin.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*Nr7prqJ2s2JtMofpPdNFVw.png" /></figure><p>In protected mode, you will be prompted to confirm each change you make.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/248/1*p7eOdDbKG571tZGAvsvopA.png" /></figure><h4>Disable protected mode</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/33/1*aly6NP2QUJdPsIJoetuicQ.png" /><figcaption>Tap the left button on header bar</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/251/1*lZal72KKT8KyRoOtH7dbiw.png" /><figcaption>Confirm if you want to make changes without confirmation</figcaption></figure><h3>Terminal screen</h3><p>The terminal is a simple SSH client. You can run commands and see the output in real-time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*Bun_BH2OFReY2Gi_1lc1cg.png" /></figure><p>One caveat is that the terminal is not stable (causes crashes) when network is slow or blips. If you use SSH extensively, I recommend to get a specialized app like <a href="https://apps.apple.com/us/app/termius-ssh-shell-console-terminal/id549039908">Termius</a>.</p><h3>SFTP screen</h3><p>The SFTP client by default goes to the home directory of the SSH user. You can navigate through the file system, and perform basic operations like delete, upload or download.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*OrUHpJf-R-398UM1QgvqcA.png" /></figure><h4>Go to parent directory</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/33/1*B1KgK7c1DBPxoK3d4ctdlg.png" /><figcaption>Tap the button to go to parent directory</figcaption></figure><h4>Refresh the directory</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/33/1*NHzz3JlvYI57aoC7ppb3XQ.png" /><figcaption>Tap the button to refresh the current directory</figcaption></figure><h4>Filter the files</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/33/1*69GC4heuQbXe_aRb9C7FJA.png" /><figcaption>Tap this button to open the filters</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*BEQKvZeS497MIHfCPFNpGQ.png" /></figure><p>You can sort the files and directories by the following property in <strong>ascending</strong> or <strong>descending</strong> order:</p><ul><li><strong>Name</strong></li><li><strong>Size</strong></li><li><strong>Last modified</strong> time</li></ul><p>Also you can choose to list all/files only/directories only, toggle hidden files, or search for a file/directory name.</p><h4>Create folder or upload file</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/33/1*Z-uKmCyUxmxQnHKQp-pxPQ.png" /><figcaption>Tap the button then select Create folder or Upload file</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*pkPUdGqEDHfBxviQ7g4Zjw.png" /></figure><h4>Delete or rename directory</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*qfzm-nXVHLPn1elLxNs1Ww.png" /><figcaption>Swipe left on the directory then tap delete or edit button</figcaption></figure><h4>Delete, rename or download file</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*yc9ZyYu_5joTfBdMUgeb3g.png" /><figcaption>Swipe left on the file then tap delete, edit or download button</figcaption></figure><h3>Underlying commands</h3><p>Here are the commands used by the app to get data from the machine.</p><p>If any of the chart or graph shows no data, you can run the corresponding command manually to troubleshoot the issue.</p><h4>CPU chart</h4><pre>CPU=`top -bn2 -d 1 | grep &#39;Cpu(s)&#39; | awk &#39;{print $2+$4}&#39; | tail -n1`;echo &#39;cpu:&#39;$CPU</pre><p>Expected output:</p><pre>cpu:0.9</pre><h4>Memory chart</h4><pre>TOT=`cat /proc/meminfo | grep MemTotal: | awk ‘{print $2}’`;USED=`cat /proc/meminfo | grep Active: | awk ‘{print $2}’`;USEDPERC=$[$USED * 100 / $TOT];echo ‘mem:’$USEDPERC</pre><p>Expected output:</p><pre>mem:13</pre><h4>Disk usage</h4><pre>STORUSED=`df --output=used / | awk &#39;END {print $1}&#39;`;echo &#39;storUsed:&#39;$STORUSED;STORTOTAL=`df --output=size / | awk &#39;END {print $1}&#39;`;echo &#39;storTotal:&#39;$STORTOTAL</pre><p>Expected output:</p><pre>storUsed:4663464<br>storTotal:7613000</pre><h4>Top CPU Process</h4><pre>PROC=`ps -Ao comm,pcpu --sort=-pcpu | head -n 6 | sed 1d | tr &#39;\n&#39; &#39;\#&#39;`;echo &#39;proc:&#39;$PROC</pre><p>Expected output:</p><pre>proc:systemd 0.5#Xorg 0.4#lxpanel 0.4#pcmanfm 0.3#bash 0.3#</pre><h4>CPU Temperature</h4><pre>cpuTemp0=$(cat /sys/class/thermal/thermal_zone0/temp 2&gt;/dev/null) || cpuTemp0=0;cpuTemp1=$(($cpuTemp0/1000));echo cputemp=$cpuTemp1</pre><p>Expected output:</p><pre>cputemp=39</pre><h4>GPU Temperature</h4><pre>echo gpu$(/opt/vc/bin/vcgencmd measure_temp 2&gt;/dev/null)</pre><p>Expected output:</p><pre>gputemp=40.2&#39;C</pre><h3>Credits</h3><p>The app uses <strong>NMSSH</strong> to establish SSH connection:<br><a href="https://github.com/NMSSH/NMSSH">https://github.com/NMSSH/NMSSH</a></p><p>The app uses <strong>wiringPi</strong> to access GPIO:<br><a href="http://wiringpi.com/">http://wiringpi.com/</a></p><p>Thank you for reading till the end. Hope the app is useful. :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a24065febed" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/using-pihelper-to-monitor-and-manage-raspberry-pis-9a24065febed">Using PiHelper to monitor and manage Raspberry Pis</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PoseNet for iOS, Android and Flutter using TensorFlow Lite]]></title>
            <link>https://medium.com/flutter-community/posenet-for-ios-android-and-flutter-using-tensorflow-lite-836788a110c7?source=rss-aa546742d090------2</link>
            <guid isPermaLink="false">https://medium.com/p/836788a110c7</guid>
            <category><![CDATA[flutter]]></category>
            <category><![CDATA[android]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[ios]]></category>
            <dc:creator><![CDATA[Sha Qian]]></dc:creator>
            <pubDate>Wed, 05 Jun 2019 22:01:02 GMT</pubDate>
            <atom:updated>2019-06-05T22:01:02.365Z</atom:updated>
            <content:encoded><![CDATA[<p>PoseNet is a well-known pose estimation model in TensorFlow.js. There’s a tflite version (<a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/gpu/multi_person_mobilenet_v1_075_float.tflite">multi_person_mobilenet_v1_075_float.tflite</a>) of the model provided <a href="https://www.tensorflow.org/lite/models/pose_estimation/overview">here</a> on TensorFlow’s webpage but no official tutorial yet on how to use it.</p><p>I ported the code of <a href="https://github.com/tensorflow/tfjs-models/tree/master/posenet">PoseNet for TensorFlow.js</a> to Android and iOS in the <a href="https://pub.dev/packages/tflite">Flutter tflite</a> plugin. In this post I will share the native code used to run the model, and the Flutter code to use the plugin.</p><p>Here’s a short demo of how PoseNet performs in a Flutter App on iOS.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FuJtZWdbsP34%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DuJtZWdbsP34&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FuJtZWdbsP34%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/1d70e98f4a832ad6da01363304c926ca/href">https://medium.com/media/1d70e98f4a832ad6da01363304c926ca/href</a></iframe><h3><strong>Understanding the Outputs of Multi-Pose Estimation</strong></h3><h4><strong>Shape of the Outputs</strong></h4><p>I started with the <a href="https://medium.com/tensorflow/real-time-human-pose-estimation-in-the-browser-with-tensorflow-js-7dd0bc881cd5">blog post</a> then explored the PoseNet tfjs source code. As I understand it, the outputs of multi-pose estimation should be 4 arrays of 4 dimensions like below:</p><ul><li>Scores : [1] [height] [width] [<strong>Number of keypoints</strong>]</li><li>Offsets: [1] [height] [width] [<strong>Number of keypoints * 2</strong>]</li><li>Displacements(Forward): [1] [height] [width] [<strong>Number of edges * 2</strong>]</li><li>Displacements(Backward): [1] [height] [width] [<strong>Number of edges * 2</strong>]</li></ul><p>The last dimension of the offsets and displacements arrays is the number of keypoints/edges multiplies 2. This is because half of the array are the <strong>x</strong> of the vectors and the other half are the <strong>y</strong>.</p><p>Scores array is the <strong>keypoint heatmap</strong> and offsets array is the <strong>offset vectors </strong>in the picture. The <a href="https://medium.com/tensorflow/real-time-human-pose-estimation-in-the-browser-with-tensorflow-js-7dd0bc881cd5">blog post</a> explained them in details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*ey139jykjnBzUqcknAjHGQ.png" /></figure><p>Displacements array (or displacement vectors) are used when we traverse along a <strong>part-based graph </strong>(edges)<strong> </strong>to locate a target keypoint from a known source keypoint. We start from finding a <strong>root keypoint</strong> which has the highest score in a local window on the heatmap then the root keypoint becomes our first known source keypoint.</p><h4><strong>Part-based Graph</strong></h4><p>PoseNet outputs 17 body parts and the parts are chained in a graph. When the source keypoint is a child node, we use the backward displacements array to locate the parent. When the source keypoint is a parent node, we use the forward displacements to locate the child.</p><p>With <strong>17 parts</strong>, there should be <strong>16 edges</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2Ni6GPPYPI6YGxvKegO2Cg.png" /></figure><h4>Calculation</h4><ul><li>To locate a target keypoint on heatmap from a source keypoint:</li></ul><pre>targetKeypointHeatmapPositions = sourceKeypointHeatmapPositions + displacementVectors</pre><ul><li>To find the actual position of a keypoint on an image:</li></ul><pre>keypointPositions = heatmapPositions * outputStride + offsetVectors</pre><h4><strong>Problem with </strong>multi_person_mobilenet_v1_075_float.tflite</h4><p>The outputs of <a href="https://storage.googleapis.com/download.tensorflow.org/models/tflite/gpu/multi_person_mobilenet_v1_075_float.tflite">multi_person_mobilenet_v1_075_float.tflite</a> from TensorFlow’s webpage are:</p><ul><li>Scores :[1][23][17][17]</li><li>Offsets: [1][23][17][34]</li><li>Displacements(Forward):[1][23][17][<strong>64</strong>]</li><li>Displacements(Backward):[1][23][17][<strong>1</strong>]</li></ul><p>The shape of the two displacements arrays are different. It doesn’t seem right because with 16 edges, forward and backward displacements arrays should both have <strong>32</strong> values on the last dimension: x and y of 16 vectors . When I first experimented the model, I got <strong>index out of range</strong> exceptions when traversing backward from the root keypoint.</p><p>I found a <a href="https://stackoverflow.com/questions/55136861/how-to-use-outputs-of-posenet-model-in-tflite">StackOverflow</a> thread discussing about this model file and others are having issues with it too. <strong>Big thanks</strong> to the answerer. His version of the converted <a href="https://drive.google.com/file/d/1DDD4wDi6vfenK2lF6W4UxPtq7dnHHJJX/view?usp=sharing">tflite file</a> generates expected output.</p><h3>Code for <strong>Running PoseNet and Estimating Poses</strong></h3><blockquote><em>The code is extracted from my Flutter project so you may need to make some adjustments before running it in a native Android/iOS project.</em></blockquote><h4>Loading Model in TensorFlow Lite:</h4><ul><li>Android:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2e24d23d0d13fc9ebcbb6e9a6375948b/href">https://medium.com/media/2e24d23d0d13fc9ebcbb6e9a6375948b/href</a></iframe><ul><li>iOS:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/15780f4e4e6266eaca5c4583dbbaa16b/href">https://medium.com/media/15780f4e4e6266eaca5c4583dbbaa16b/href</a></iframe><h4>Feeding an Image to Input Tensor:</h4><ul><li>Android</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8fe3a0d7c5d9b7e999807fdc4b832b4a/href">https://medium.com/media/8fe3a0d7c5d9b7e999807fdc4b832b4a/href</a></iframe><ul><li>iOS</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/37aac28349215d3290f0238f5be63546/href">https://medium.com/media/37aac28349215d3290f0238f5be63546/href</a></iframe><h4><strong>Running Inference:</strong></h4><ul><li>Android</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1227e847ba689a8f11808c2fe2cf7eb4/href">https://medium.com/media/1227e847ba689a8f11808c2fe2cf7eb4/href</a></iframe><ul><li>iOS</li></ul><pre>TfLiteStatus status = interpreter-&gt;Invoke();</pre><pre>if (status != kTfLiteOk) {<br>  NSLog(@&quot;Failed to invoke!&quot;);<br>  return result(empty);<br>}</pre><h4><strong>Decoding Poses:</strong></h4><ul><li>Android</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d8e6fcc53c684fe459ceb1c902e4a119/href">https://medium.com/media/d8e6fcc53c684fe459ceb1c902e4a119/href</a></iframe><ul><li>iOS</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5b2b7565df7b98571a5fb0d3eadbf995/href">https://medium.com/media/5b2b7565df7b98571a5fb0d3eadbf995/href</a></iframe><h4>Output format:</h4><p>x, y are between [0, 1]. You can scale x by the width and y by the height of the image.</p><pre>[ // array of poses/persons<br>  { // pose #1<br>    score: 0.6324902,<br>    keypoints: {<br>      0: {<br>        x: 0.250,<br>        y: 0.125,<br>        part: nose,<br>        score: 0.9971070<br>      },<br>      1: {<br>        x: 0.230,<br>        y: 0.105,<br>        part: leftEye,<br>        score: 0.9978438<br>      }<br>      ......<br>    }<br>  },<br>  { // pose #2<br>    score: 0.32534285,<br>    keypoints: {<br>      0: {<br>        x: 0.402,<br>        y: 0.538,<br>        part: nose,<br>        score: 0.8798978<br>      },<br>      1: {<br>        x: 0.380,<br>        y: 0.513,<br>        part: leftEye,<br>        score: 0.7090239<br>      }<br>      ......<br>    }<br>  },<br>  ......<br>]</pre><h3>Using PoseNet in Flutter Tflite Plugin</h3><blockquote><em>Refer to </em><a href="https://github.com/shaqian/flutter_tflite#Installation"><em>the installation instructions</em></a><em> if you are new to this plugin.</em></blockquote><h4>Loading the Model:</h4><pre>await Tflite.loadModel(model: “assets/posenet_mv1_075_float_from_checkpoints.tflite”);</pre><h4>Running Inference:</h4><pre>var result = await runPoseNetOnImage(<br>  path: filepath,     // required<br>  imageMean: 125.0,   // defaults to 125.0<br>  imageStd: 125.0,    // defaults to 125.0<br>  numResults: 2,      // defaults to 5<br>  threshold: 0.7,     // defaults to 0.5<br>  nmsRadius: 10,      // defaults to 20<br>  asynch: true        // defaults to true<br>);</pre><h4>Rendering the Keypoints:</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1c8d8e065f202f8b53446e9b16f9c011/href">https://medium.com/media/1c8d8e065f202f8b53446e9b16f9c011/href</a></iframe><h3>Source Code</h3><h4>Flutter Tflite Plugin</h4><p>The Android and iOS native code are used in this plugin:<br><a href="https://github.com/shaqian/flutter_tflite">https://github.com/shaqian/flutter_tflite</a></p><h4>Real-time Pose Estimation</h4><p>The app in the video demo is available here: <a href="https://github.com/shaqian/flutter_realtime_detection">https://github.com/shaqian/flutter_realtime_detection</a></p><p><strong>Thank you for reading till the end. :)</strong></p><p><a href="https://twitter.com/FlutterComm">JavaScript is not available.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=836788a110c7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/flutter-community/posenet-for-ios-android-and-flutter-using-tensorflow-lite-836788a110c7">PoseNet for iOS, Android and Flutter using TensorFlow Lite</a> was originally published in <a href="https://medium.com/flutter-community">Flutter Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Object Detection and ARFoundation in Unity]]></title>
            <link>https://medium.com/@shaqian629/object-detection-and-arfoundation-in-unity-8782b1ee6ea3?source=rss-aa546742d090------2</link>
            <guid isPermaLink="false">https://medium.com/p/8782b1ee6ea3</guid>
            <category><![CDATA[augmented-reality]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[unity]]></category>
            <dc:creator><![CDATA[Sha Qian]]></dc:creator>
            <pubDate>Tue, 02 Apr 2019 13:25:32 GMT</pubDate>
            <atom:updated>2019-04-03T00:28:32.561Z</atom:updated>
            <content:encoded><![CDATA[<p>This is a simple example of using <strong>TensorFlow</strong> and <strong>ARFoundation</strong> plugin in Unity. The example detects if an apple is in the camera view then place an AR apple object on it.</p><p>Here’s a demo video captured on a iOS device.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FSfB4FHqzam0%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DSfB4FHqzam0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FSfB4FHqzam0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/333b4747eb006395a62426805cabf14d/href">https://medium.com/media/333b4747eb006395a62426805cabf14d/href</a></iframe><h3>Source code</h3><ul><li>TF-Unity-ARFoundation: <a href="https://github.com/shaqian/TF-Unity-ARFoundation">https://github.com/shaqian/TF-Unity-ARFoundation</a></li><li>TF-Unity: <a href="https://github.com/shaqian/TF-Unity">https://github.com/shaqian/TF-Unity</a></li></ul><h3>Capturing image from AR camera</h3><p>The scene in <a href="https://github.com/shaqian/TF-Unity-ARFoundation">TF-Unity-ARFoundation</a> is modified the CameraImage scene from the official Unity <a href="https://github.com/Unity-Technologies/arfoundation-samples">arfoundation-samples</a>. The code that captures image from AR camera and converts it to RGB texture is borrowed from <a href="https://github.com/Unity-Technologies/arfoundation-samples/blob/359daccb9bf84c225a59ebcf8188a0e3aea89772/Assets/Scripts/TestCameraImage.cs#L66-L115">here</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/beb79e3a07a416d09664dce02fd5af4d/href">https://medium.com/media/beb79e3a07a416d09664dce02fd5af4d/href</a></iframe><h3>Detecting object using TensorFlowSharp Plugin</h3><p>I created the scripts in <a href="https://github.com/shaqian/TF-Unity">TF-Unity</a> for running inferences using Unity TensorFlowSharp plugin. The scripts is tested with <strong>MobileNet</strong> model for <strong>image classification</strong>, and <strong>SSD MobileNet</strong> and<strong> Tiny YOLOv2</strong> model for <strong>object detection</strong>.</p><p>To run object detection with SSD MobileNet model, we first need to initialize the detector. The model and label file are passed to the component as TextAsset.</p><pre>[SerializeField]<br>TextAsset model;</pre><pre>[SerializeField]<br>TextAsset labels;</pre><pre>Detector detector = new Detector(model, labels, input: &quot;image_tensor&quot;);</pre><p>Everytime a new camera frame is captured, we pass the texture to the detector and run inference to get the detected objects.</p><pre>var outputs = detector.Detect(m_Texture, angle: 90, threshold: 0.6f);</pre><p>We then loop the outputs and look for a target object with class name “apple”.</p><pre>for (int i = 0; i &lt; outputs.Count; i++)<br>{<br>    var output = outputs[i] as Dictionary&lt;string, object&gt;;<br>    if (output[&quot;detectedClass&quot;].Equals(&quot;apple&quot;))<br>    {<br>        DrawApple(output[&quot;rect&quot;] as Dictionary&lt;string, float&gt;);<br>        break;<br>    }<br>}</pre><p>A single output is in the following format where x, y, w, h is the <strong>left</strong>, <strong>top</strong>, <strong>width</strong> and <strong>height</strong> of a box area containing the object. The values are ratio instead of actual size.</p><pre>{<br>  &quot;detectedClass&quot;: &quot;apple&quot;,<br>  &quot;confidenceInClass&quot;: 0.123,<br>    &quot;rect&quot;: {<br>    &quot;x&quot;: 0.15,<br>    &quot;y&quot;: 0.33,<br>    &quot;w&quot;: 0.80,<br>    &quot;h&quot;: 0.27,<br>  }<br>}</pre><h3>Placing the AR Object</h3><p>Now we have the position of the object on the 2D screen, how to locate it in 3D space? I’ve seen people talk about solvePnP in OpenCV but that seems too complicated for newbie like me.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2XbdFhjgGbeTvzySdSswuw.png" /></figure><p>For simplicity, I use <strong>Raycast</strong> to do a hit test against the center point of the detected box, then place the AR object at the hit position. (Note the the y-axis of the box from the inference output is at the top of the screen but that of the AR camera is at the bottom.)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/358efc5b859e5e481da6b36b8770cdd5/href">https://medium.com/media/358efc5b859e5e481da6b36b8770cdd5/href</a></iframe><p>That’s it!</p><p>One problem with this example is that because the hit position is on the surface of the object, the AR object is not exactly at the same position of the real object, and sometimes it looks like the real apple and AR apple are sticked together 😅. Suggestions for improvement are welcome.</p><p><strong>Thank you for reading till the end. :)</strong></p><figure><a href="https://usejournal.com/?utm_source=medium.com&amp;utm_medium=noteworthy_blog&amp;utm_campaign=guest_post_image"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f2IVAl0TbsfES9cFGYr40g.png" /></a></figure><p>📝 Read this story later in <a href="https://usejournal.com/?utm_source=medium.com&amp;utm_medium=noteworthy_blog&amp;utm_campaign=guest_post_read_later_text">Journal</a>.</p><p>🗞 Wake up every Sunday morning to the week’s most noteworthy Tech stories, opinions, and news waiting in your inbox: <a href="https://usejournal.com/newsletter/?utm_source=medium.com&amp;utm_medium=noteworthy_blog&amp;utm_campaign=guest_post_text">Get the noteworthy newsletter &gt;</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8782b1ee6ea3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Loading Local Assets in WebView in Flutter]]></title>
            <link>https://medium.com/flutter-community/loading-local-assets-in-webview-in-flutter-b95aebd6edad?source=rss-aa546742d090------2</link>
            <guid isPermaLink="false">https://medium.com/p/b95aebd6edad</guid>
            <category><![CDATA[android]]></category>
            <category><![CDATA[flutter]]></category>
            <category><![CDATA[ios]]></category>
            <dc:creator><![CDATA[Sha Qian]]></dc:creator>
            <pubDate>Fri, 22 Feb 2019 13:07:55 GMT</pubDate>
            <atom:updated>2019-02-22T13:25:16.507Z</atom:updated>
            <content:encoded><![CDATA[<p>The Flutter <a href="https://pub.dartlang.org/packages/webview_flutter">WebView</a> plugin is in preview and at the moment it only loads URIs. It’d be nice if we can load local html (along with Javascript and CSS) files from local assets.</p><p>I made minimal changes to the plugin to provide a way to load asset files and have submitted a pull request: <a href="https://github.com/flutter/plugins/pull/1247">https://github.com/flutter/plugins/pull/1247</a></p><p>In case the pull request got rejected, you can refer to the <strong>File changed</strong> to patch your copy of webview package.</p><h3>Patching WebView plugin</h3><p>Allow me to briefly explain what the pull request does to patch the webview plugin.</p><h4>iOS:</h4><p>The plugin uses the <strong>WKWebView</strong> class in native code. It calls the <a href="https://developer.apple.com/documentation/webkit/wkwebview/1414954-loadrequest?language=objc">loadRequest</a> method to invoke a request to the URI passed in from Flutter.</p><p>The WKWebView class provides another method <a href="https://developer.apple.com/documentation/webkit/wkwebview/1414973-loadfileurl?language=objc">loadFileURL</a> that can load a file URL. The definition is:</p><pre>- (<a href="https://developer.apple.com/documentation/webkit/wknavigation?language=objc">WKNavigation</a> *)loadFileURL:(<a href="https://developer.apple.com/documentation/foundation/nsurl?language=objc">NSURL</a> *)URL <br>      allowingReadAccessToURL:(<a href="https://developer.apple.com/documentation/foundation/nsurl?language=objc">NSURL</a> *)readAccessURL;</pre><p>In order to use this method, we need to know the URL of our asset file. We can find some hints in the Flutter online doc <a href="https://flutter.io/docs/development/ui/assets-and-images#ios">Sharing assets with the underlying platform</a>. The example returns a path but we can use another method <a href="https://developer.apple.com/documentation/foundation/nsbundle/1411540-urlforresource?language=objc">URLForResource</a> to get the URL.</p><pre>NSString* key = [registrar lookupKeyForAsset:@&quot;icons/heart.png&quot;];<br>NSString* path = [[NSBundle mainBundle] pathForResource:key ofType:nil];</pre><p>With the current implementation, the <strong>FLTWebViewController</strong> and <strong>FLTWebViewFactory</strong> class in webview plugin don’t have access to registrar, so I changed their <strong>initWithMessenger</strong> method to <strong>initWithRegistrar</strong>. Now we have the answer to loading asset file in iOS webview:</p><pre>NSString* key = [_registrar lookupKeyForAsset:url];<br>NSURL* nsUrl = [[NSBundle mainBundle] URLForResource:key withExtension:nil];<br>[_webView loadFileURL:nsUrl allowingReadAccessToURL:[NSURL URLWithString:@&quot;file:///&quot;]];</pre><h4>Android:</h4><p>The webview plugin uses the <a href="https://developer.android.com/reference/android/webkit/WebView">WebView</a> class on Android side and it calls the <a href="https://developer.android.com/reference/android/webkit/WebView.html#loadUrl(java.lang.String)">loadUrl</a> method. This method already supports asset file URL in the following format:</p><pre>file:///android_asset/{RELATIVE-PATH}</pre><p>My first attempt was using “file:///android_asset/{MY-FOLDER}/{MY-FILE}.html” but it failed with File Not Found error.</p><p>I did some research and it turns out Flutter stores the user defined assets in a <strong>flutter_assets</strong> directory. So the answer to loading assets in Android webview is:</p><pre>webView.loadUrl(&quot;file:///android_asset/flutter_assets/&quot; + url);</pre><h3>Prepare the html and js file</h3><p>To verify the functionality of the patched package, I decided to test if the webview can render a Three.js scene.</p><p>The html below is slightly altered from the sample code in the <a href="https://threejs.org/docs/index.html#manual/en/introduction/Creating-a-scene">Three.js introduction</a>. It generates a white cube with rotating animation.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6358496f0bec6f4a3e00cc2840ebd272/href">https://medium.com/media/6358496f0bec6f4a3e00cc2840ebd272/href</a></iframe><p>We also need a copy of three.js: <a href="https://threejs.org/build/three.js">https://threejs.org/build/three.js</a></p><p>Then create a <strong>assets</strong> folder in the root of the Flutter project and place <strong>three.html</strong> and <strong>three.js</strong> in it. Finally in <strong>pubspec.yaml,</strong> add:</p><pre>assets:<br>  - assets/three.js<br>  - assets/threejs.html</pre><h3>Create the WebView</h3><p>The dart code below creates the WebView and renders the local <strong>assets/three.html</strong> file.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4ef7627c111c24d2204f403a10b4561c/href">https://medium.com/media/4ef7627c111c24d2204f403a10b4561c/href</a></iframe><h3>Inject Javascript code</h3><p>The <strong>WebViewController</strong> class has an <strong>evaluateJavascript</strong> method. Using this method we can inject Javascript code to the webview.</p><p>The code in this example sets a random Hex color value to the material of the <strong>cube</strong> object defined in <strong>three.html</strong>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1aeeb66ae36806b0fbdc94e0d435bc0f/href">https://medium.com/media/1aeeb66ae36806b0fbdc94e0d435bc0f/href</a></iframe><h3>The result</h3><p>Here’s the demonstration of the example. A rotating cube is rendered in the scene and every time I tap the refresh button, the cube changes its color.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*wdcisODftoKG3f_62ylHVg.gif" /></figure><h4>Thank you for reading till the end. :)</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b95aebd6edad" width="1" height="1" alt=""><hr><p><a href="https://medium.com/flutter-community/loading-local-assets-in-webview-in-flutter-b95aebd6edad">Loading Local Assets in WebView in Flutter</a> was originally published in <a href="https://medium.com/flutter-community">Flutter Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real-time Object Detection in Flutter]]></title>
            <link>https://medium.com/@shaqian629/real-time-object-detection-in-flutter-b31c7ff9ef96?source=rss-aa546742d090------2</link>
            <guid isPermaLink="false">https://medium.com/p/b31c7ff9ef96</guid>
            <category><![CDATA[android]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[ios]]></category>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[flutter]]></category>
            <dc:creator><![CDATA[Sha Qian]]></dc:creator>
            <pubDate>Mon, 18 Feb 2019 15:27:30 GMT</pubDate>
            <atom:updated>2019-02-20T21:14:30.576Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently Flutter team added image streaming capability in the camera plugin. This allows you to capture the frame in a live camera preview.</p><p>I created a demo app that uses image streaming with tflite (TensorFlow Lite) plugin to achieve real-time object detection in Flutter.</p><p>Here’s a short video captured on my iPad demonstrating the app.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fr8mjuCImfcI%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dr8mjuCImfcI&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fr8mjuCImfcI%2Fhqdefault.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/69f63d64abe5907640e0b63271189e83/href">https://medium.com/media/69f63d64abe5907640e0b63271189e83/href</a></iframe><h3>Image streaming in camera plugin</h3><p>Refer to <a href="https://pub.dartlang.org/packages/camera">the link</a> to add the camera plugin to the Flutter project.</p><p>To start image streaming, call <strong>startImageStream</strong> in the camera controller. The method is triggered every time a new frame arrives.</p><pre>controller.startImageStream((CameraImage img) { &lt;YOUR CODE&gt; });</pre><p>The output CameraImage class has 4 members: <strong>image format</strong>, <strong>height</strong>, <strong>width</strong> and finally <strong>planes</strong> which consists of the bytes of the image.</p><pre>class CameraImage {<br>  final ImageFormat format;<br>  final int height;<br>  final int width;<br>  final List&lt;Plane&gt; planes;<br>}</pre><p>The format of the image varies with the platforms:</p><ul><li>Android: <a href="https://developer.android.com/reference/android/graphics/ImageFormat.html#YUV_420_888">android.graphics.ImageFormat.YUV_420_888</a></li><li>iOS: <a href="https://developer.apple.com/documentation/corevideo/1563591-pixel_format_identifiers/kcvpixelformattype_32bgra?language=objc">kCVPixelFormatType_32BGRA</a> (Note that the format was <a href="https://developer.apple.com/documentation/corevideo/1563591-pixel_format_identifiers/kcvpixelformattype_420ypcbcr8biplanarvideorange?language=objc">kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange</a> in v2.8.0. Later in v4.0.0 it was changed back to 32BGRA.)</li></ul><p>As a result of the different format, the output CameraImage on iOS and Android are different:</p><ul><li>Android: <strong>planes</strong> is a list of bytes arrays of <strong>Y, U </strong>and<strong> V planes</strong> of the image.</li><li>iOS: <strong>planes</strong> contains a single array containing the <strong>RGBA</strong> bytes of the image.</li></ul><p>Knowing the format is important for properly decoding the image and feeding it to TensorFlow Lite.</p><h3>Decoding CameraImage</h3><p>It’s possible to decode the image in Dart code but unfortunately the <a href="https://pub.dartlang.org/packages/image">image</a> library works slowly on iOS. Decoding the frames from native code is faster and more efficient.</p><ul><li>iOS:</li></ul><p>Since the format is already RGBA, we only need to extract the Red, Green, Blue values from the bytes and feed them to the input tensor of a TensorFlow Interpreter.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c6cbdb05128d7f9d8586116e9269f3f9/href">https://medium.com/media/c6cbdb05128d7f9d8586116e9269f3f9/href</a></iframe><ul><li>Android:</li></ul><p>We need to first create a bitmap in RGBA format from the bytes of YUV planes. An easy way is to use a render script to achieve the conversion.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/84a031848fbd543ea7ff67cbe7388d4e/href">https://medium.com/media/84a031848fbd543ea7ff67cbe7388d4e/href</a></iframe><p>The code that converts NV21 to RGBA is as below. I got it from <a href="https://stackoverflow.com/a/36409748">https://stackoverflow.com/a/36409748</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cef5fd1312d74551b774a502cf00acf7/href">https://medium.com/media/cef5fd1312d74551b774a502cf00acf7/href</a></iframe><p>Once we have the raw bitmap, we can resize it to fit the input size required and feed the RGB values to the input tensor.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/49dc138f32b056a421f42823b8fe31e7/href">https://medium.com/media/49dc138f32b056a421f42823b8fe31e7/href</a></iframe><h3>Detect objects using tflite plugin</h3><p>The <a href="https://pub.dartlang.org/packages/tflite">tflite</a> plugin wraps TensorFlow Lite API for iOS and Android. It implemented native code for feeding input and extracting output of popular models. For object detection, it supports <strong>SSD MobileNet</strong> and <strong>YOLOv2</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*715GU9k6CGqkjXkWSg8-lg.jpeg" /></figure><p>The plugin provides a <strong>detectObjectOnFrame</strong> method which can decode image stream from camera plugin (under the hood it uses the code described above), run inference and return the recognitions.</p><p>We can simply pass the planes bytes of CameraImage to the method and get the detected objects.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/32bed297b32cacc3edf18e88bcd8827e/href">https://medium.com/media/32bed297b32cacc3edf18e88bcd8827e/href</a></iframe><p>The output is a list of objects in the following format:</p><pre>{<br>  detectedClass: “hot dog”,<br>  confidenceInClass: 0.123,<br>  rect: {<br>    x: 0.15,<br>    y: 0.33,<br>    w: 0.80,<br>    h: 0.27<br>  }<br>}</pre><p>x, y, w, h define the <strong>left</strong>, <strong>top</strong>, <strong>width</strong> and <strong>height</strong> of the box that contains the object. The values are between [0, 1]. We can scale x, w by the width and y, h by the height of the image.</p><h3>Display the outputs</h3><p>A small problem with camera plugin is the preview size does not always fit the screen size. It’s recommended to put the preview in an AspectRatio widget to avoid stretching it but that will leave margins.</p><p>I personally prefer the camera preview to take the whole screen so I find a workaround by putting the preview in an <strong>OverflowBox </strong>widget: first compare the height/width ratio of the preview and the screen, then scale the preview to cover the screen by fitting either the screen width or screen height.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/72eb37534a7f1414c8685eaf69dd4c95/href">https://medium.com/media/72eb37534a7f1414c8685eaf69dd4c95/href</a></iframe><p>Accordingly, when drawing the boxes, scale the x, y, w, h by the scaled width and height. Note that x or y has to be subtracted by the difference between scaled height(width) and screen height(width), since a part of the preview is hidden behind OverflowBox.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dd5e7660938335cde8a10a65ab1d293a/href">https://medium.com/media/dd5e7660938335cde8a10a65ab1d293a/href</a></iframe><h3>Inference time per frame</h3><p>I tested the app on my iPad and Android phone. SSD MobileNet works well on both platforms but Tiny YOLOv2 has quite a lag on Android.</p><ul><li>iOS (A9)<br>SSD MobileNet: ~100 ms<br>Tiny YOLOv2: 200~300ms</li><li>Android (Snapdragon 652):<br>SSD MobileNet: 200~300ms<br>Tiny YOLOv2: ~1000ms</li></ul><h3>Source Code</h3><p>The souce code of the demo app is available at <a href="https://github.com/shaqian/flutter_realtime_detection">https://github.com/shaqian/flutter_realtime_detection</a></p><p><strong>Thank you for reading till the end. :)</strong></p><figure><a href="https://usejournal.com/?utm_source=medium.com&amp;utm_medium=noteworthy_blog&amp;utm_campaign=guest_post_image"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f2IVAl0TbsfES9cFGYr40g.png" /></a></figure><p>📝 Read this story later in <a href="https://usejournal.com/?utm_source=medium.com&amp;utm_medium=noteworthy_blog&amp;utm_campaign=guest_post_read_later_text">Journal</a>.</p><p>🗞 Wake up every Sunday morning to the week’s most noteworthy Tech stories, opinions, and news waiting in your inbox: <a href="https://usejournal.com/newsletter/?utm_source=medium.com&amp;utm_medium=noteworthy_blog&amp;utm_campaign=guest_post_text">Get the noteworthy newsletter &gt;</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b31c7ff9ef96" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>