<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[AI2 Labs - Medium]]></title>
        <description><![CDATA[Everyone talks about it but not many actually realize that the development and deployment of AI is really hard work! We would like to share all the little things that we do as we explore the uncharted waters of real world AI. - Medium]]></description>
        <link>https://medium.com/ai-innovation?source=rss----86b4b44aa53e---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>AI2 Labs - Medium</title>
            <link>https://medium.com/ai-innovation?source=rss----86b4b44aa53e---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 22:09:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/ai-innovation" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Guide for JavaScript Developers to build C++ Add-ons with node-addon-api]]></title>
            <link>https://medium.com/ai-innovation/a-guide-for-javascript-developers-to-build-c-add-ons-with-node-addon-api-28c84a0c0cb1?source=rss----86b4b44aa53e---4</link>
            <guid isPermaLink="false">https://medium.com/p/28c84a0c0cb1</guid>
            <category><![CDATA[cpp]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[addons]]></category>
            <dc:creator><![CDATA[Bianca Cao]]></dc:creator>
            <pubDate>Fri, 08 Jan 2021 05:57:10 GMT</pubDate>
            <atom:updated>2021-01-08T05:57:10.512Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Do you expect to reuse a third party C++ library in your Node.js project? <br>Are you jealous about the outstanding C++ performance while coding something computationally heavy in JavaScript? </strong><br>It is not something impossible to access C++ from JavaScript. Here you will see how I make it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ay2gJp-3Esu9CmnXrSx9Ow.png" /></figure><h3>Background Introduction</h3><p>JavaScript, as the programming language for the Web, is quite popular and easy to learn, while C++ is static typed and compiled language that quite hard to pick up by anyone who only stick with dynamic and interpreted languages. To be specific, JavaScript allows you to change variables dynamically into any data type but in C++ you need to declare it first. In addition, C++ need a compiler to transfer your codes into executable ones, yet JavaScript interpreter will just read and run them line by line. That is why developers usually would not feel pleasantly transforming from JavaScript to C++.</p><p>Nowadays, JavaScript can not only be used for websites, but also be strong enough to support some server-side applications. More and more people are using web frameworks with great performance and lower develop cost including <a href="http://expressjs.com/">Express</a>, <a href="https://github.com/koajs/koa#readme">Koa</a>, <a href="https://www.npmjs.com/package/fastify">fastify</a>, <a href="https://github.com/socketio/socket.io#readme">socket.io</a> and so on.</p><p>However, some underlying libraries have not been extended into Node.js yet. So, what are the options if you want to call C++ functions or libraries from JavaScript without an available Node package?</p><p>This blog will explain<strong> several alternatives</strong> and dig into <a href="https://github.com/nodejs/node-addon-api"><strong>node-addon-api</strong></a><strong>, </strong>talking about how I <strong><em>build my first JavaScript bindings to OpenMesh C++ library</em></strong>. You can also find my source code for my at <a href="https://github.com/BiancaZYCao/TriMesh4Js">https://github.com/BiancaZYCao/TriMesh4Js</a> to have a quick read.</p><h3>Option 1 FFI: <a href="https://github.com/node-ffi/node-ffi">node-ffi</a> OR <a href="https://github.com/node-ffi-napi/node-ffi-napi">node-ffi-napi</a></h3><blockquote>A foreign function interface (FFI) is a mechanism by which a program written in one programming language can call routines or make use of services written in another.</blockquote><p>You can call C/C++ codes by loading DLL (<em>Dynamic Link Library</em>) to use those functions <strong>without writing any C/C++ code</strong>. In <a href="https://docs.google.com/document/d/1T5DkDNLZipR9RaBLJrAwdcvk_2Xu2TKkPN_28u-FTp0/edit#">Foreign Function Interface for Node.js</a>, you can find this as a summary for its mechanism:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*lNdf59nXNjwYSnNgVY4G4w.png" /><figcaption>synchronous translation pipeline</figcaption></figure><p>To give you an idea about what it looks like, here is <strong>a simple example</strong> to bind C++ <em>ceil() </em>function:</p><pre>var ffi = require(&#39;ffi-napi&#39;);<br>var libm = ffi.Library(&#39;libm&#39;, { <br>      &#39;ceil&#39;: [ &#39;double&#39;, [ &#39;double&#39; ] ] <br>    }); <br>libm.ceil(1.5); // 2</pre><p><strong>👍 Advantages:</strong></p><ul><li>You can do it <strong>without</strong> source code file</li><li>There is <strong>no need to write any C/C++ code</strong></li><li>There is<strong> no need to recompile </strong>after any changes</li></ul><p><strong>✋ Limitations:</strong></p><ul><li><strong>acts like a black box sometimes</strong>: It will be <strong>hard</strong> to understand<em> what is going on inside</em>, unless you have <strong>strong debugging skills</strong> to make sure that <em>all the arguments passed into a certain function correctly for C++ execution</em> and <em>whether the results returned back is right</em>.</li><li><strong>a certain performance loss</strong>: As you can see in the pipeline image It basically <strong><em>convert</em></strong><em> the interface</em> defined in JavaScript into an interface in C, and<strong> </strong><em>share NodeJS’s Buffer memory</em> with <strong>the loaded DLL</strong>. That is why it works with any DLL yet may sacrifice some performance.</li><li><strong>accepts DLL only</strong>: As DLL is<strong> the only thing</strong> it calls, you have to pack <strong>your own written codes</strong> into DLL before using it.</li></ul><p>In short, this will be a good choice when you would like to <strong>call the shared library directly</strong> just for <strong>some functions</strong> that you are familiar with by <em>passing the parameters in and get value back</em>. Unfortunately, I gave it up after getting lost in the functions for some reasons which I will explain later.</p><p><strong><em>looking for an easy way to use it?</em></strong> You might want to try <a href="https://github.com/tjfontaine/node-ffi-generate">node-ffi-generate</a>.</p><h3>Option 2 Node.js Extensions</h3><p>Actually, Node.js offers <strong>three official ways</strong> for implementing C/C++ addons: <em>N-API, nan, or direct use of internal V8, libuv and Node.js libraries.</em> As you do not need to understand all those underlying components/APIs, <strong>N-API is recommended</strong> among these choices <strong>unless you need those functionalities have not exposed</strong>.</p><h4>A Native Tool : NAN (Native Abstractions for Node.js)</h4><p>Before N-API released, <a href="https://github.com/nodejs/nan">NAN</a> provided <strong><em>a V8 abstraction layer</em></strong> to support add-on development easier working <strong>with various version</strong>. As the older way to implement C++ bindings, the <strong>main disadvantage</strong> is that all the codes needs to <strong>be updated</strong> along with every upgrade of V8 Engine. Thus, you are not likely to choose it especially building the new Add-ons.<br><strong>Here is an example</strong> about NAN binding functions (You can find <a href="https://github.com/justadudewhohacks/opencv4nodejs/blob/master/cc/core/Mat.cc">this</a> in <a href="https://github.com/justadudewhohacks/opencv4nodejs">opencv4nodejs</a>. This function is to <em>get raw pixel values from a image mat</em> in Node.js: const [b, g, r] = matBGR.atRaw(200, 100); )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*QJ0y5Be1LOnuV2TVUWD81Q.png" /><figcaption>NAN extension example</figcaption></figure><h4>The Modern Choice: <a href="https://nodejs.org/api/addons.html#addons_n_api">N-API</a></h4><blockquote>N-API provides an <a href="https://nodejs.org/en/docs/guides/abi-stability/"><strong>ABI-stable</strong> </a>API that can be used to develop <strong>native add-ons</strong> for Node.js, simplifying the task of building and supporting such add-ons across Node.js versions (stable since Node.js v10).</blockquote><p><em>Application Binary Interface</em> (ABI) plays a key role on the interaction between software programs and system binaries. <em>ABI stability</em><strong> </strong>guarantees different version compilers will generate compatible binary code following some certain rules.</p><p><em>Similar to NAN</em>, <a href="https://nodejs.org/api/addons.html#addons_n_api">N-API</a> allows you to pass data between C/C++ and JavaScript and to do any operations you want to on both sides. Besides, it also <strong>abstracts out</strong> its API from the underlying JavaScript engine to make your addons <strong>separate from </strong>the underlying V8 engine. Below illustrate how N-API works and the “helloworld” example code will be shown afterwards in my personal journey.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*rSLCBazNVu1TKMMUAQm3gw.png" /><figcaption>How N-API works</figcaption></figure><h4>A Simpler Choice for C++ Add-ons: <a href="https://github.com/nodejs/node-addon-api">node-addon-api</a></h4><blockquote>It contains <strong>header-only C++ wrapper classes</strong> which simplify the use of the <strong>C based </strong><a href="https://nodejs.org/dist/latest/docs/api/n-api.html"><strong>N-API</strong></a> provided by Node.js when using C++.<br>The purpose of this module is to raise the N-API API from the level of “C” up to “C++“.</blockquote><p>Even though it is not counted as a Node.js official module, it at least provides <strong>an easier option for beginners</strong> who expect to</p><ul><li>make <strong>bindings</strong> on an existing C++ library</li><li>migrate <strong>computationally intensive tasks</strong> in C++ for better performance</li></ul><p>without much effort and fully understanding underlying Node Libraries.</p><p>The journey to create customized modules with <em>node-addon-api</em> is <strong>less painful</strong> provided with the <em>C++ object model</em> in napi.h. The example codes were also <strong>more readable and clearer</strong> than <em>N-API </em>so it finally became <strong>my choice</strong>. After building your Add-ons, you are able to called it via require(). I will compare the codes with <em>N-API</em> in details in next chapter. Here is <strong>my understanding about the overall architecture</strong> that maybe helpful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s0gQDwG-N1bl1Y9bjqQOww.png" /><figcaption>Big Picture about C++ Add-ons</figcaption></figure><h3>My Experience with these options</h3><p>The node-ffi caught my eyes firstly but it was no longer campatible, so I quickly moved on n<a href="https://github.com/node-ffi-napi/node-ffi-napi">ode-ffi-napi</a> instead. It was confusing because the dll files were just black boxes to me and it was not easy to use with sepcific data structure. For example, to create a 5-number array and fill it with 9 as [9,9,9,9,9] ( This should be included in basic libc++ dll), I felt hard to bind with FFI even I know how to code it in C++ like<br>std::array&lt;int,5&gt; myarray;<br> myarray.fill(9);<br>It kept telling me symbol/image not found if not correct. And yes, I did export symbols from dll, but it was also not much helpful to match those functions. I had no idea about how to diagnose and debug such an simple function, not to mention, using it for a C++ library holding complicated data structures.</p><p>After a while, I started to looking at the extension option. NAN was kind of outdated and N-API was still hard to pick up for me especially because I never learned or wrote any code in C/C++ . The “helloworld” example is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*v5J1_UQ8LPE89jTlcq2Taw.png" /><figcaption>helloworld example N-API</figcaption></figure><p>Compared with that, The “helloworld” example in node-addon-api looks much more friendly and clear.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*KY6XijncoBHA_FxDjaquRw.png" /><figcaption>helloworld example using node-addon-api</figcaption></figure><p>Therefore, I started my journey with node-addon-api and finally came up with something can be called in my Node.js project indeed.</p><h3>How to start the first real binding?</h3><h4>Environment Prerequisites</h4><p>-C++ ( I used CMake/make to compile C++ code) <br>- Python (for node-gyp)<br>- Node and npm<br>- Command line tools &amp; editor<br>- C++ library (<em>OpenMesh</em> <a href="https://www.graphics.rwth-aachen.de/software/openmesh/download/">download</a> or simply brew install open-mesh)</p><h4>Dependencies need to be installed in your Node.js project</h4><p><a href="https://github.com/nodejs/node-gyp">node-gyp</a> — Node.js native addon build tool (installed globally)<br><em>other build tool alternatives are </em><a href="https://github.com/cmake-js/cmake-js"><em>CMake.js</em></a><em> </em><a href="https://nodejs.github.io/node-addon-examples/build-tools/node-pre-gyp"><em>node-pre-gyp</em></a><em> and </em><a href="https://github.com/prebuild/prebuild#prebuild"><em>prebuild</em></a><em> </em><br><a href="https://github.com/nodejs/node-addon-api">node-addon-api</a> : follow <a href="https://github.com/nodejs/node-addon-api/blob/master/doc/setup.md">setup</a> <br><a href="https://github.com/TooTallNate/node-bindings">bindings</a> : npm install --save bindings</p><h4>Reminder before we start</h4><p>For people like me who <strong>never touched C/C++ before</strong>, it would be very helpful to prepare yourself with basic concepts including compiler, linker, pointer, reference, handle, etc.</p><p>You can find sample codes in <a href="https://github.com/nodejs/node-addon-examples">node-addon-examples</a> and <a href="https://github.com/mapbox/node-cpp-skel">node-cpp-skel</a> to play around fundamental bindings. However, <em>one or more practical repositories are highly recommended</em> when it comes to build something more complicated than simple functions, classes and objects. <a href="https://github.com/nodriver-ai/MAVJS">MAVJS</a> brought me inspirations personally, but you can pick any other projects to refer to (not only about how to code add-ons, but also how to write binding configuration, test in .js and even deploy it as a package)</p><blockquote>I will focus mainly on my experience about writing binding C++ codes for OpenMesh.</blockquote><h3>Write the binding codes in C++</h3><p>Since OpenMesh is used as a data structure which is a mesh object wrapper, I edit on <a href="https://github.com/nodejs/node-addon-examples/tree/master/6_object_wrap">node-addon-examples/6_object_warp</a> templates. I am going to highlight only a few points. Please check my code <a href="https://github.com/BiancaZYCao/TriMesh4Js/blob/master/myRealMesh.cc">myRealMesh.cc</a> for details.</p><h4>How to write an Object Wrapper Class</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q3miCw_fkzOTwOoHPRAINg.png" /><figcaption>a typical Object Wrap</figcaption></figure><h4>Things should pay attention to or may be confusing</h4><ol><li>Check <a href="https://github.com/nodejs/node-addon-api/blob/master/doc/object_lifetime_management.md">object_lifetime_management</a> if Napi::HandleScope or Napi::EscapableHandleScope is necessary (usually not)</li><li>Build Class Constructor properly by creating a peristent reference to allow functions to be called on a specific instance. Do not forget to add the SuppressDestruct method if you need to prevent its destructor from attempting to reset the reference when the environment is no longer valid.</li><li>Check if your object is supporting multiple instances (can be created and distinguished)</li></ol><h4>How to implement a function</h4><p>A typical Napi function will include:<br>- receiving arguments passing in as Napi::CallbackInfo&amp; <br>- transferring into C++ type like double &amp; doing some operations in C++<br>- returning result in any Napi data type</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*Y0RG1RHcV0UXC2GhQhnEIQ.png" /><figcaption>A typical function</figcaption></figure><h4>Basic Napi Data types</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t_Qpsk8jgtGln94u5x-g5Q.png" /><figcaption>Basic Napi Data types</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-HCS78WJOpsYfVKd0RNM7A.png" /><figcaption>Inheritance relationship</figcaption></figure><h4>Something I did creatively</h4><p>Here I wanna share something creative (though may not be normal): how I defined a const pointer to pass the handle reference around.<br> In OpenMesh, the VertexHandle represents every vertex on the mesh and is required when adding a new face (a triangle with 3 vertices). When I looked up in <a href="https://www.graphics.rwth-aachen.de:9000/OpenMesh/openmesh-python/-/blob/master/src/Mesh.hh">OpenMesh Binding for Python</a>, the mechanism is just exposing everything directly to Python output like &lt;openmesh.VertexHandle object at 0x126471378&gt;. Unlike that in pybind11, I tried to return the reference back in JavaScript with Napi::External but it became invalid when passing back to add a face with error Segmentation fault: 11. After printing everything out, I realized it is dynamically changes every time manipulating the mesh. <br>The easiest solution is to use vertexID tracking each point instead, which means you are only sending and receiving integer value for all operations. Or, if sticking on the reference, I have to manually claim a constant pointer in my object wrapper (myMesh) to make it into a static value that can be matched on both C++ and Js sides.</p><pre>std::vector&lt;TriMesh::VertexHandle&gt;  vHStorage_; <br>std::vector&lt;TriMesh::VertexHandle&gt; * const ptrVHS = &amp;vHStorage_; //vHStorage_ is dynamic while ptrVHS remains as a constant</pre><h4>Following Steps</h4><p><strong>prepare your inding.gyp file</strong><br>Before building your add-on code into .node, you need to declare all the information in binding.gyp, including target_name, source code files, directories, libraries and compile conditions. For details, you can take a look at the <a href="https://gyp.gsrc.io/docs/UserDocumentation.md">gyp user documentation</a>.</p><p><strong>build executable .node files by node-gyp or other tools</strong><br>Basically, just run node-gyp configure then node-gyp build. If you meet any errors or warnings at this stage, you need to fix them before continue. Sometimes, testing in pure C++ or by debuggers like <a href="https://www.gnu.org/software/gdb/">gdb</a> will be useful to diagnose your bugs. In addition, every time you changes your C++ code, you need to rebuild it again as update.</p><p><strong>try the add-ons in your .js file &amp; write test.js<br></strong>To call it, just one more line on top of your .js file:<br>let addon = require(&quot;bindings&quot;)(&quot;&lt;target_name&gt;&quot;) <br>and call it just like any other libraries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/1*vpGYfeR-VuJRTaP8qfdJIQ.png" /><figcaption>call bindings to build a tri-Mesh</figcaption></figure><p>For test files, I used <a href="https://mochajs.org/"><strong><em>mocha</em></strong></a><em> </em>and <strong><em>assert</em></strong> module to check all the features in my extension code work well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*PDDXc_9uYMrOpCDIbG3SJw.png" /><figcaption>test sample with mocha for the get mesh points function</figcaption></figure><p>Congratulations! You have just crossed the finish line. 🎉 🎉🎉</p><p>I cannot believe I made this happen within 2 weeks, starting from all the concepts and learning C/C++. After exploring all the documentations, code samples and blog articles, I feel like it is worthwhile to record this as a supplementation to limited resources especially for JavaScript developers. Hope my journey be useful to choose a suitable tool for your case and to create C++ addons for NodeJS.</p><h4>Reference Links</h4><p><a href="https://stackoverflow.com/questions/16783023/node-ffi-vs-node-extension-for-accessing-existing-c-functionality?rq=1">https://stackoverflow.com/questions/16783023/node-ffi-vs-node-extension-for-accessing-existing-c-functionality?rq=1</a><br><a href="https://github.com/node-ffi-napi/node-ffi-napi">https://github.com/node-ffi-napi/node-ffi-napi</a><br><a href="https://github.com/nodejs/node-addon-api">https://github.com/nodejs/node-addon-api</a><br><a href="https://github.com/nodejs/node-addon-examples">https://github.com/nodejs/node-addon-examples</a> <a href="https://github.com/mapbox/node-cpp-skel">https://github.com/mapbox/node-cpp-skel</a><br><a href="https://nodejs.github.io/node-addon-examples/">https://nodejs.github.io/node-addon-examples/</a><br><a href="https://medium.com/@a7ul/beginners-guide-to-writing-nodejs-addons-using-c-and-n-api-node-addon-api-9b3b718a9a7f">https://medium.com/@a7ul/beginners-guide-to-writing-nodejs-addons-using-c-and-n-api-node-addon-api-9b3b718a9a7f</a><br><a href="https://github.com/nodriver-ai/MAVJS">https://github.com/nodriver-ai/MAVJS</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28c84a0c0cb1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-innovation/a-guide-for-javascript-developers-to-build-c-add-ons-with-node-addon-api-28c84a0c0cb1">A Guide for JavaScript Developers to build C++ Add-ons with node-addon-api</a> was originally published in <a href="https://medium.com/ai-innovation">AI2 Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[In-Domain GAN Inversion for Anime Character]]></title>
            <link>https://medium.com/ai-innovation/in-domain-gan-inversion-for-anime-character-f6341d72e835?source=rss----86b4b44aa53e---4</link>
            <guid isPermaLink="false">https://medium.com/p/f6341d72e835</guid>
            <category><![CDATA[generative-adversarial]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[anime]]></category>
            <dc:creator><![CDATA[Jiasheng Tang]]></dc:creator>
            <pubDate>Fri, 08 Jan 2021 05:56:52 GMT</pubDate>
            <atom:updated>2021-01-08T05:56:52.226Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently I came across this interesting paper <a href="https://genforce.github.io/idinvert/">In-Domain GAN Inversion for Real Image Editing</a> (a.k.a IDInvert), and I was amazed by the potential of this technique and the large number of potential applications of it. Below is a list of applications this paper has envisioned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*Zj6ODywsfBg8F2NCW0mPQA.png" /><figcaption>List of applications by <a href="https://genforce.github.io/idinvert/">In-Domain GAN Inversion For Real Image Editing</a> (IDInvert)</figcaption></figure><p>To me, the applications should not stop at “Real Image Editing”, and should be extended to the art creation world. Just imagine that, after an artist has drawn his character, he can edit the character’s pose and expression without redrawing it. He can even pick a certain feature from another character, and blend into his own character within seconds. It certainly opens the door to new ways of creation. With this excitement, I decided to first experiment it out on anime characters.</p><p>Before I jump into my experiment, allow me to give an overview of IDInvert so that you can better appreciate the power of it, and understand the limitation of it.</p><h3>Let’s first talk about StyleGAN</h3><p><a href="https://github.com/NVlabs/stylegan">StyleGAN </a>is the state of the art in image generation. It was introduced by Nvidia, and it is the technology behind <a href="https://thispersondoesnotexist.com/">ThisPersonDoesNotExist</a> and <a href="https://www.thiswaifudoesnotexist.net/">ThisWaifuDoesNotExist</a>. The images generated by StyleGAN are near-perfect, most people cannot tell from them apart from the real photo. Moreover, it enables users to control how the image looks through different levels of styles:</p><ul><li>Coarse styles — corresponding to pose, hair, face shape</li><li>Middles styles — Facial features, eyes</li><li>Fine styles — color scheme</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*RN5c3XJ62oKkJzB80T4FRw.png" /><figcaption>Illustration of <a href="https://github.com/NVlabs/stylegan">StyleGAN</a> image editing</figcaption></figure><p>However, there is a great limitation for StyleGAN image editing — it is powerless when the image is a photo taken in the real world. The StyleGAN’s image editing is through the manipulation of the latent space vector, but a photo taken in the real world (same for an image drawn in the real world) does not have this latent space vector as it is not generated by StyleGAN.</p><p>In other words, StyleGAN is like Superman in the DC universe. It can do marvelous things in its own universe, but can’t help anyone in our universe.</p><p>Nonetheless, the power of StyleGAN is so tempting, that many are trying to break the barrier between the real and the generated, and here comes the technique called <strong><em>Inversion.</em></strong></p><h3>Invert a real image, to make it part of the generated world</h3><p>StyleGAN is trained to convert the latent space vector into an image, how about we just train another network to convert the image back into its latent space vector?</p><p>This is exactly what most of the people have tried, and the steps are:</p><ol><li>Use StyleGAN to produce thousands of images, so we have the “latent space vector — image” pairs as the training dataset</li><li>Train an encoder network to convert the image back into the latent space vector. <em>Note that if feed this latent space vector back to StyleGAN, the generated image will slightly differ from the original image. An additional optimization process is needed to fine-tune the vector. I omit this process for simplicity. You can refer to the </em><a href="https://arxiv.org/pdf/2004.00049.pdf"><em>paper</em></a><em> for more information.</em></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*DPVLK0haTGmqzguUVhcjcA.png" /><figcaption>An illustration of the inversion training process. G here is the StyleGAN network, it is black indicating that it is not updated in the training process. Zsam is the latent space vector that generates the image through StyleGAN. E here is the encoder network to convert the image back to the latent space vector Zenc, it is blue indicating it is updated in the training process. Reference: <a href="https://arxiv.org/pdf/2004.00049.pdf">https://arxiv.org/pdf/2004.00049.pdf</a></figcaption></figure><p>This method can indeed find a latent space vector that reconstructs a real-world image, but the problem is that this vector may not fall in the domain where the StyleGAN was trained with. As a result, this latent space vector may not support the image editing capability well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*nrcdtyOxQbXnBsvg4iOxDg.png" /><figcaption>Adopting the above-mentioned inversion technique, the interpolation of the two images gives a bad result.</figcaption></figure><h3>IDInvert makes sure the inverted latent space vector is In-Domain</h3><p>The trick in IDInvert is a “domain-guided encoder”. Instead of just looking at the similarity between the original vector fed into StyleGAN and the encoded vector from the image, IDInvert feeds the encoded vector back to StyleGAN, and check the generated image. The training process is very similar to the StyleGAN training process, just that instead of updating the StyleGAN generator, it updates the encoder network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*iFD4Tb6BhiSRRUt-w5lI0g.png" /><figcaption>An illustration of the IDInvert training process. E here is the encoder network to convert the image back to the latent space vector Zenc. Zenc is then fed into G (StyleGAN) to generate an image, and a discriminator D is used to differentiate the real image from the generated image. Reference: <a href="https://arxiv.org/pdf/2004.00049.pdf">https://arxiv.org/pdf/2004.00049.pdf</a></figcaption></figure><p>In this way, the encoder network learns to output vectors that are in-domain, and this ensures that the inverted latent space vector can well support image editing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/1*FQ8PKlE8RLVU7O_89n-AKw.png" /><figcaption>IDInvert gives good interpolation result</figcaption></figure><h3>Experiment IDInvert on Anime Character</h3><p>IDInvert provides source code for both <a href="https://github.com/genforce/idinvert">TensorFlow implementation</a> and <a href="https://github.com/genforce/idinvert_pytorch">PyTorch implementation</a>. A big thanks to the author for making it so convenient to try out. I used the PyTorch implementation source code.</p><p>For StyleGAN anime portrait pre-trained model, I get it from <a href="https://www.gwern.net/index#">gwern</a>. It also provides the anime portrait face dataset for download, which consist of 300k of anime face images, and the image size is 512*512px.</p><p>I used two Tesla P100 GPUs to train, used batch size 8, and trained 2000k iterations. It took about 2 weeks to finish the training.</p><p>However, the results turn out to be not very good. When feeding the encoded vector back to the StyleGAN (with the optimization process included), I found that the inverted images only have a rough resemblance with the input image, the details are very different, and there are artifacts present.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/1*zvh5XRdbuAngjZ1W0RuMkg.png" /><figcaption>The upper row is the input and inversion anime portrait using IDInvert. The bottom row is the input and inversion real-person portrait using IDInvert</figcaption></figure><p>Compare to the real-person portrait example in the IDInvert paper, the anime results are much worse.</p><h3>Conclusion</h3><p>Through the experiment, I have identified a few possible reasons for the bad performance of anime portrait results, and what can be improved:</p><ol><li>The anime portrait image is 512x512px which is a bigger size compared to the examples in the IDInvert paper. A bigger image size usually makes the task more challenging. A potential solution is to train a smaller size version first, then use it as a pre-train model for the bigger size version.</li><li>The training hyper-parameters I used are the default ones provided by IDInvert source code. This set of parameters were optimized for real-person portraits, may need to fine-tune for anime character images</li><li>The anime portraits dataset has a mix of the artist’s styles, which makes it challenging to learn. A curated dataset with a more standard style will give better performance.</li></ol><p>Although I did not obtain good results for my experiments, I still believe in the potential of the GAN-Inversion technology, and dream of the day that we can fully utilize GAN and make art creation more enjoyable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/1*49tjZ2SvqJ0sM4DArysX0w.gif" /><figcaption><a href="https://tenor.com/search/cat-dreaming-gifs">https://tenor.com/search/cat-dreaming-gifs</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6341d72e835" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-innovation/in-domain-gan-inversion-for-anime-character-f6341d72e835">In-Domain GAN Inversion for Anime Character</a> was originally published in <a href="https://medium.com/ai-innovation">AI2 Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[TraceTogether UI/UX Study — 
 or, why I gave my personal details without a fuss]]></title>
            <link>https://medium.com/ai-innovation/tracetogether-ui-ux-study-or-why-i-gave-my-personal-details-without-a-fuss-ba9b51647b3d?source=rss----86b4b44aa53e---4</link>
            <guid isPermaLink="false">https://medium.com/p/ba9b51647b3d</guid>
            <category><![CDATA[ux-research]]></category>
            <dc:creator><![CDATA[Joel Tan]]></dc:creator>
            <pubDate>Sat, 28 Nov 2020 13:28:19 GMT</pubDate>
            <atom:updated>2020-11-28T13:28:19.776Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>TraceTogether UI/UX Study — <br> or, why I gave my personal details without a fuss</strong></h3><p>TL;DR: TraceTogether has an excellent user onboarding experience</p><p><strong>My problem</strong></p><p>I am very paranoid about sharing my personal data. I have dropped out of many sign-up processes that require me to submit my personal details (e.g. signing up to crypto platforms, KYC).</p><p><strong>What is TraceTogether</strong></p><p>TraceTogether is a mobile application which the Singaporean Government had implemented to allow for better contact tracing in response to the COVID-19 pandemic in Singapore.</p><p>It works by exchanging short-distance Bluetooth signals between phones to detect other participating TraceTogether users in close proximity. Records of such encounters are stored locally on each user’s phone.</p><p><strong>No escape</strong></p><p>To prepare for Phase 3, I finally downloaded TraceTogether app today after resisting for 7 months. I was not looking forward to it.</p><p><strong>Turnaround</strong></p><p>To my surprise, every step of the sign up process was delightful. Screenshots and delightful touch points below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ANeG0DUsUaCMzPXKzGG0wQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o3v8wko9JHHWwGJ_dFRAag.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OsbgHRgHbCBqX4pgBEeyKA.jpeg" /></figure><p><strong>Aftermath</strong></p><p>While I am still uneasy with sharing my personal data, the thoughtful UX design and excellent copywriting made the sign-up process painless. Excellent work by the TraceTogether team.</p><p><strong>Found this post useful? Smash the *clap* button below to help others find this post.</strong></p><p><strong>Have you experienced an AWESOME onboarding experience lately? Let me know in the comments below!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba9b51647b3d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-innovation/tracetogether-ui-ux-study-or-why-i-gave-my-personal-details-without-a-fuss-ba9b51647b3d">TraceTogether UI/UX Study — 
 or, why I gave my personal details without a fuss</a> was originally published in <a href="https://medium.com/ai-innovation">AI2 Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Lessons Learnt From Shipping An AI Application]]></title>
            <link>https://medium.com/ai-innovation/lessons-learnt-from-shipping-an-ai-application-ed7cdccdfcf8?source=rss----86b4b44aa53e---4</link>
            <guid isPermaLink="false">https://medium.com/p/ed7cdccdfcf8</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[learning-and-development]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[Daniel Tan]]></dc:creator>
            <pubDate>Sat, 28 Nov 2020 13:27:57 GMT</pubDate>
            <atom:updated>2020-11-28T13:27:57.118Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n22oOEyMFRdeIoYxvzfFKg.jpeg" /></figure><p>This is part of the MLOps series.</p><p>We finally launched a limited public beta for our Autopaint tool, <a href="https://creart.innovrepublic.com/">https://creart.innovrepublic.com/</a> , and these are lessons learnt during the past few months of development. Some of them might seems weird, but bear with me.</p><p>TLDR:</p><ol><li>Don’t skip design sprints.</li><li>Treat AI deployments as independent deployments</li><li>Enable non-developers to edit static configuration of your application.</li><li>Mirror the deployment setup as close as production</li></ol><h3>Don’t Skip Design Sprints</h3><p>Typically developers like to take a bunch of requirements and begin hacking away. This is the advice to stop and think more about the application before beginning, especially for an AI application. Most products probably don’t need “machine learning” or “artificial intelligence” as their core technology, because all they really need is some good statistics algorithm (discounting how AI/ML can bring some good marketing, but that’s a business decision). However, if your application uses AI/ML as the core, then you probably want to take a step back and understand what does your users want from AI/ML, and how would they be using it.</p><p>An unused product feature is good as dead.</p><p>Users typically want to use AI/ML for any of the following reasons (<a href="https://www.ciosummits.com/what-consumers-really-think-about-ai.pdf">check out this report</a>):</p><ol><li>The product can learn. (57% of people surveyed thinks AI can do this)</li><li>The product can replicate human interaction and interpret speech (35% and 37% respectively)</li></ol><p>So, if your product claims to have AI, and it misses the baseline of what people think AI can do, you’re going to disappoint your users. By then, you might as well say your product has a proprietary algorithm and users will be none the wiser (Looking at you Uber).</p><p>As such, you probably want to reflect that in your product design, including UI/UX. You probably want to provide and highlight some of the following in your application:</p><ol><li>The ability to replicate human results (approximated results).</li><li>The ability to learn (error correction).</li></ol><p>Non-AI applications typically don’t provide this because their results typically carry a low risk factor. When was the last time you expected a calculator to fail? And yet AI applications will fail, because they only provide a good approximation of the results the users want. As such AI applications should be designed as a <strong>loop</strong>, instead of a straight line.</p><p>This is where the design sprints come in, to provide some buffer for the development (including designers, artists and techies) team to think about how their program works.</p><h3>Treat AI Deployments As Independent Deployments</h3><p>Now the typical SDLC goes like this: planning→analysis→design→implementation→maintenance.</p><p>Even though it’s supposedly a cycle, maintenance doesn’t always go to planning. Once a program hits maintenance, it more or less stays there, and a new version of the program restarts the process as a new process.</p><p>As for AI deployments, that’s usually not the case. During the development of AI applications I drew from my experience in game development where there’s some stuff that’s different from the usual SDLC:</p><ol><li>Artists and game designers have their own development cycle, just like AI scientists, and it is separate from the tech cycle. This means that you actually have to plan your time accordingly to stagger between artists and game designers. This also means there are constant mini-cycles of iteration and communication going on instead of one big cycle with clear, designed stages.</li><li>The maintenance step is a misnomer because you have to constantly monitor for user behaviour to ensure that your application is doing what you think it should. Just like how you can’t expect players to play according to what you think the game is capable of, you should never expect the AI to produce results the users expect. This means your deployment must be capable of being patched outside of the tech cycle, and at arbitrary short times (Remember the racist AI bot?). For games, it’s stuff like asset bundles to update game mechanics and art, and for AI it is splitting model deployment and code deployment.</li></ol><h3>Enable Others To Edit Static Configuration Of The Application</h3><p>AI engineers are usually focused on making sure their models run well, and AI development is never-ending. They are always in development mode, and the line between production and development is pretty thin. In addition to that, most AI engineers are Python based, so you can’t expect them to understand Golang code or whatever.</p><p>This is fine and dandy because most AI applications are micro-service based architectures. So teams don’t usually come into conflict… until they do, because the AI engineer is likely the domain expert, and they will need to tweak some variables of your application. During development of our application, we began to understand just how much needs to be tweaked, especially since AI is the core of our application.</p><p>As such, we abstracted out configuration so that our AI engineers can tweak JSON files to what they needed, without asking the developers for help. This helped us immensely, especially when the developers were busy fixing bugs during a tight sprint while the models are trained for deployment, so the AI engineers can come fix some of the smaller items just by changing the configuration.</p><p>The same happened when it comes to deploying the applications to different environments. The AI engineer abstracted the file paths for models and other micro-services to some JSON file, and the operations person just needed to maintain different configuration for different environments without bothering too much of the core code.</p><h3>Mirror The Deployment Setup As Close As Production</h3><p>We chose to use Kubernetes via <a href="https://microk8s.io/">Microk8s</a> as our core method of deployment as it paid off greatly when we moved from Aliyun to Azure. It took us a total of a week to get everything setup as it should, and by the end of the week everything was working as it should without a hitch, save for a gotcha where Azure doesn’t actually mount your new disk for you (<a href="https://docs.microsoft.com/en-us/azure/virtual-machines/linux/attach-disk-portal">https://docs.microsoft.com/en-us/azure/virtual-machines/linux/attach-disk-portal</a>)</p><p>I created my own templating tool for managing environments (<a href="https://github.com/danieltanfh95/clojube">https://github.com/danieltanfh95/clojube</a>) but you’re also welcome to use Kustomize or Helm. In general this is what we did to achieve this:</p><ol><li>Containerization. This allowed us to prevent “it works on my machine” sort of problems. You especially don’t want to deal with issues like this during a tight deployment schedule.</li><li>Decentralized, versioned configuration for K8s. Kubernetes allows you to save your deployment as YAML files and it’s very powerful, because now you can tie your deployment with your code version! This means given some kubernetes cluster you can ensure that for a particular version of your application the cluster will be setup exactly as you intended for that version.</li><li>Decentralized CI/CD configuration. We used Gitlab for this. This allowed to build and maintain separate docker containers of our application, specific to some environment. This is especially important because you may have multiple production environments and they might need some change in configuration in some way. Doing this at the CI/CD level allows your developers to focus on getting features out instead of trying to fit their code to the environment. Also, gitlab-runners are decentralized, so it means we can setup some configured GPU machine will build our models that the AI engineers can also change.</li></ol><p>So there you go, some unconventional learnings from developing and deploying an AI application for public use.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed7cdccdfcf8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-innovation/lessons-learnt-from-shipping-an-ai-application-ed7cdccdfcf8">Lessons Learnt From Shipping An AI Application</a> was originally published in <a href="https://medium.com/ai-innovation">AI2 Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beginner’s Guide to Retrain GPT-2 (117M) to Generate Custom Text Content]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/ai-innovation/beginners-guide-to-retrain-gpt-2-117m-to-generate-custom-text-content-8bb5363d8b7f?source=rss----86b4b44aa53e---4"><img src="https://cdn-images-1.medium.com/max/1276/1*ceUS920Pza9WHJaWd-iebA.jpeg" width="1276"></a></p><p class="medium-feed-snippet">In this article, we will be exploring the steps required to retrain GPT-2 (117M) using custom text dataset on Windows. For start, GPT-2 is&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/ai-innovation/beginners-guide-to-retrain-gpt-2-117m-to-generate-custom-text-content-8bb5363d8b7f?source=rss----86b4b44aa53e---4">Continue reading on AI2 Labs »</a></p></div>]]></description>
            <link>https://medium.com/ai-innovation/beginners-guide-to-retrain-gpt-2-117m-to-generate-custom-text-content-8bb5363d8b7f?source=rss----86b4b44aa53e---4</link>
            <guid isPermaLink="false">https://medium.com/p/8bb5363d8b7f</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[gpt-2]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Ng Wai Foong]]></dc:creator>
            <pubDate>Mon, 02 Nov 2020 06:31:29 GMT</pubDate>
            <atom:updated>2020-11-19T00:37:24.703Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Beginner’s Guide to OpenAI Five at Dota2]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/ai-innovation/beginners-guide-to-openai-five-at-dota2-3b49ee5169b8?source=rss----86b4b44aa53e---4"><img src="https://cdn-images-1.medium.com/max/1083/1*5PPYvv00Tob8A-w3u8ogYQ.png" width="1083"></a></p><p class="medium-feed-snippet">This article is about OpenAI Five, a machine learning experiment aimed at creating a team of bots to play against human players at Dota2&#x2026;</p><p class="medium-feed-link"><a href="https://medium.com/ai-innovation/beginners-guide-to-openai-five-at-dota2-3b49ee5169b8?source=rss----86b4b44aa53e---4">Continue reading on AI2 Labs »</a></p></div>]]></description>
            <link>https://medium.com/ai-innovation/beginners-guide-to-openai-five-at-dota2-3b49ee5169b8?source=rss----86b4b44aa53e---4</link>
            <guid isPermaLink="false">https://medium.com/p/3b49ee5169b8</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[dota-2]]></category>
            <category><![CDATA[openai]]></category>
            <dc:creator><![CDATA[Ng Wai Foong]]></dc:creator>
            <pubDate>Mon, 02 Nov 2020 06:31:10 GMT</pubDate>
            <atom:updated>2020-11-09T05:30:50.808Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Create Your Own Customized GIF or Video]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/ai-innovation/create-your-own-customized-gif-or-video-d285b3396aaa?source=rss----86b4b44aa53e---4"><img src="https://cdn-images-1.medium.com/max/2600/1*9_t6zpC9apqk4a8OAZkaeQ.jpeg" width="4081"></a></p><p class="medium-feed-snippet">A beginner&#x2019;s guide to create animated gif or video using ScreenToGif</p><p class="medium-feed-link"><a href="https://medium.com/ai-innovation/create-your-own-customized-gif-or-video-d285b3396aaa?source=rss----86b4b44aa53e---4">Continue reading on AI2 Labs »</a></p></div>]]></description>
            <link>https://medium.com/ai-innovation/create-your-own-customized-gif-or-video-d285b3396aaa?source=rss----86b4b44aa53e---4</link>
            <guid isPermaLink="false">https://medium.com/p/d285b3396aaa</guid>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[tips]]></category>
            <category><![CDATA[guides-and-tutorials]]></category>
            <category><![CDATA[gif]]></category>
            <category><![CDATA[videos]]></category>
            <dc:creator><![CDATA[Ng Wai Foong]]></dc:creator>
            <pubDate>Mon, 02 Nov 2020 06:30:55 GMT</pubDate>
            <atom:updated>2020-11-09T05:07:54.435Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[How to Create C/C++ Dynamic-link Libraries in Windows]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/ai-innovation/how-to-create-c-c-dynamic-link-libraries-in-windows-28abefc988c9?source=rss----86b4b44aa53e---4"><img src="https://cdn-images-1.medium.com/max/2600/1*ZQQDGp9NtsCj9ofVaQ7abQ.jpeg" width="6000"></a></p><p class="medium-feed-snippet">Modularized your program into separate components to ease deployment and installation</p><p class="medium-feed-link"><a href="https://medium.com/ai-innovation/how-to-create-c-c-dynamic-link-libraries-in-windows-28abefc988c9?source=rss----86b4b44aa53e---4">Continue reading on AI2 Labs »</a></p></div>]]></description>
            <link>https://medium.com/ai-innovation/how-to-create-c-c-dynamic-link-libraries-in-windows-28abefc988c9?source=rss----86b4b44aa53e---4</link>
            <guid isPermaLink="false">https://medium.com/p/28abefc988c9</guid>
            <category><![CDATA[c]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[shared-libraries]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[windows-10]]></category>
            <dc:creator><![CDATA[Ng Wai Foong]]></dc:creator>
            <pubDate>Mon, 02 Nov 2020 06:30:42 GMT</pubDate>
            <atom:updated>2020-12-09T08:09:56.620Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[8-bit Quantization On Pytorch]]></title>
            <link>https://medium.com/ai-innovation/8-bit-quantization-on-pytorch-b46da8c19eaa?source=rss----86b4b44aa53e---4</link>
            <guid isPermaLink="false">https://medium.com/p/b46da8c19eaa</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Hongze]]></dc:creator>
            <pubDate>Mon, 02 Nov 2020 06:30:32 GMT</pubDate>
            <atom:updated>2020-11-02T06:30:32.109Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*FvdXOjtyfokJZ1sb5_Aqww.jpeg" /></figure><h3>What is Quantization?</h3><p>Quantization refers to a technique that uses fewer bits than floating-point precision for calculation and storage. A quantized model uses integer tensor instead of floating-point tensor to perform some or all of the operations. This is a more compact model representation and can take advantage of high-performance vector operations on many hardware platforms. PyTorch supports INT8 quantization. Compared to FP32, the model size is reduced by 4x, and the memory bandwidth requirement is also reduced by 4x. Hardware support for INT8 operation makes its calculations generally 2–4 times faster than FP32. Quantization is mainly a technique to accelerate inference, and the operation after quantization only supports forward calculation.</p><p>PyTorch supports multiple quantization methods for deep learning models. In most cases, the model is trained using FP32 and then converted to the INT8 model. In addition, PyTorch also supports the training of quantization perception, which can model the errors that occur in the quantization process and perform forward and reverse calculations through the fake-quantization module. It should be noted that all calculations are performed on floating-point numbers. At the end of quantitative perception training, PyTorch provides a conversion tool to convert the trained model to lower accuracy.</p><p>At a lower level, PyTorch provides a way to represent quantized tensors and use them for calculations. These tensors can be used to build models directly and perform all calculations with low accuracy. It also provides a high-level API that includes a typical workflow for converting from FP32 to a low-precision model with minimal loss of precision.</p><h3>How to Quantize Tensors?</h3><p>PyTorch provides both per-tensor and per-channel asymmetric linear quantization. Per-tensor means that all values ​​in the tensor are scaled in the same way. Per-channel means that for a given dimension, it is usually the channel dimension of the tensor, and each slice of the tensor in this dimension will use different scaling and offset (so that scaling and offset can be represented by vectors), this ensures that there are fewer errors in the quantization process.</p><p>The conversion process from floating point to fixed point uses the following mapping equation:</p><p><em>Q</em>(<em>x</em>, <em>scale</em>, <em>zero</em>_<em>point</em>)=<em>round</em>(<em>x/scale </em>+ <em>zero</em>_<em>point</em>)</p><p>It is worth noting that there is no loss in the representation of the floating-point zero before and after quantization. It means that the zero before quantization corresponds to a fixed-point value after quantization, which ensures that it will not be introduced quantization errors in some operations like Zero Padding.</p><p>In order to quantize in PyTorch, we need to be able to represent the quantized data with tensor. A quantized tensor can store quantized data (represented by int8/uint8/int32) and quantization parameters, such as scaling and quantization zeros. The quantized tensor can use multiple operations, so that the quantized tensor can easily perform operations, and can also be serialized and parallelized.</p><h3>How to Quantize Models?</h3><p>Pytorch provides three approaches to quantize models.</p><ol><li><strong>Dynamic Quantization</strong>: This is the simplest to apply form of quantization where the weights are quantized ahead of time but the activations are dynamically quantized during inference. This is used for situations where the model execution time is dominated by loading weights from memory rather than computing the matrix multiplications. This is true for LSTM and Transformer type models with small batch size. Applying dynamic quantization to a whole model can be done with a single call to torch.quantization.quantize_dynamic().</li><li><strong>Post-Training Static Quantization:</strong> This is the most commonly used form of quantization where the weights are quantized ahead of time and the scale factor and bias for the activation tensors are pre-computed based on observing the behavior of the model during a calibration process. Post Training Quantization is typically when both memory bandwidth and compute savings are important with CNNs being a typical use case.</li><li><strong>Quantization Aware Training:</strong> In rare cases where post-training quantization does not provide adequate accuracy training can be done with simulated quantization using the torch.quantization.FakeQuantize. Computations will take place in FP32 but with values clamped and rounded to simulate the effects of INT8 quantization. The sequence of steps is very similar.</li></ol><p>In the following articles, I will take some examples to explain how to do quantization in a real-world project. See you next time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b46da8c19eaa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-innovation/8-bit-quantization-on-pytorch-b46da8c19eaa">8-bit Quantization On Pytorch</a> was originally published in <a href="https://medium.com/ai-innovation">AI2 Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Customize Your In-Game Faces]]></title>
            <link>https://medium.com/ai-innovation/customize-your-in-game-faces-728c1096e24d?source=rss----86b4b44aa53e---4</link>
            <guid isPermaLink="false">https://medium.com/p/728c1096e24d</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[gaming]]></category>
            <dc:creator><![CDATA[Hongze]]></dc:creator>
            <pubDate>Mon, 02 Nov 2020 06:30:07 GMT</pubDate>
            <atom:updated>2020-11-02T06:30:07.885Z</atom:updated>
            <content:encoded><![CDATA[<p>In Role-Playing Games(RPGs), the character customization system is an important part. Players are allowed to edit the facial parameters of in-game characters. Netease Fuxi AI Lab has released a paper named Face-to-Parameter Translation for Game Character Auto-Creation which proposed an end-to-end approach for face-to-parameter translation and game character auto-creation method. Today, let’s focus on how to reimplement this method.</p><h3>Overview</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l7wwpfPCYEpxUerCUn0YQw.png" /><figcaption>Fig. 1</figcaption></figure><p>The whole processing pipeline is shown above(Fig.1). The Imitator aims to imitate the behavior of a game engine by taking in user-customized facial parameters and producing a facial image. The Feature Extractor focuses on extracting two kinds of features, 256-d facial embeddings and the facial semantic features on the real-world images and the rendered game characters. The final part is the optimization part. The gradient descent method is used to solve the optimization problem. The following parts are the details on each component’s implement.</p><h3>pre-requisites</h3><blockquote>Unity-3d (≥ 2018.3.14 required)</blockquote><blockquote>Tensorflow (≥ 1.12 required)</blockquote><blockquote>Keras (≥ 2.2.4 required)</blockquote><h3>Imitator</h3><p>Before training the Imitator, we used Unity-3d to design a male character that has 216 facial parameters. Some rendered faces are shown in Fig. 2. We randomly generated 20,000 individual faces with their corresponding facial customization parameters for training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2JTFyupwThqL9USitEmBJQ.png" /><figcaption>Fig. 2</figcaption></figure><p>The Imitator is based on DC-GAN’s generator. The whole structure of the network is shown below. Unlike the training process of the GANs, the imitator is fully supervised. The input is the facial parameters and the output is the front view of the game character. So the imitator is similar to the decoder of an Auto-encoder network(Fig.4).</p><pre>model = Sequential()</pre><pre>model.add(Dense(256 * 4 * 4, activation=&quot;relu&quot;, input_dim=self.latent_dim))<br>model.add(Reshape((4, 4, 256)))<br>model.add(UpSampling2D())<br>model.add(Conv2D(256, kernel_size=4, padding=&quot;same&quot;))<br>model.add(BatchNormalization(momentum=0.8))<br>model.add(Activation(&quot;relu&quot;))<br>model.add(UpSampling2D())<br>model.add(Conv2D(256, kernel_size=4, padding=&quot;same&quot;))<br>model.add(BatchNormalization(momentum=0.8))<br>model.add(Activation(&quot;relu&quot;))<br>model.add(UpSampling2D())<br>model.add(Conv2D(128, kernel_size=4, padding=&quot;same&quot;))<br>model.add(BatchNormalization(momentum=0.8))<br>model.add(Activation(&quot;relu&quot;))<br>model.add(UpSampling2D())<br>model.add(Conv2D(64, kernel_size=4, padding=&quot;same&quot;))<br>model.add(BatchNormalization(momentum=0.8))<br>model.add(Activation(&quot;relu&quot;))<br>model.add(UpSampling2D())<br>model.add(Conv2D(32, kernel_size=4, padding=&quot;same&quot;))<br>model.add(BatchNormalization(momentum=0.8))<br>model.add(Activation(&quot;relu&quot;))<br>model.add(UpSampling2D())<br>model.add(Conv2D(self.channels, kernel_size=4, padding=&quot;same&quot;))<br>        model.add(Activation(&quot;tanh&quot;))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/1*JOkfva-B2_D3vJV3AYPOCQ.png" /><figcaption>Fig. 4</figcaption></figure><p>We used the SGD optimizer with the batch_size = 16 and momentum = 0.9. The learning rate is set to 0.01 and the loss function is Mean_Absolute_Error.</p><pre>optimizer = SGD(0.01, 0.9)<br>self.generator.compile(loss=&#39;mae&#39;, optimizer=optimizer)</pre><p>As the paper says, the training stops after 500 training epochs and the learning rate decay is set to 10% per 50 epochs. After training, the performance of the Imitator is very good. Some details of the generated images(Fig. 5) are very similar to images that the game engine produced.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ClU3caOdVQ2CY0lS7Bi5jw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TcXQBt2QBMFq0Cw299fWEw.png" /><figcaption>Fig. 5</figcaption></figure><h3>Feature Extractor</h3><p>Feature extractor is mainly used to measure the facial similarity between real-world images and the game engine produced images corresponding to the Discriminative Loss and Facial Content Loss. The final loss function can be written as a linear combination of the two objectives <em>L1</em> and <em>L2</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*ISBme6Adny3sZwej8g6QeA.png" /><figcaption>Final Loss function</figcaption></figure><p>Next, I will talk about how to implement these parts.</p><p><strong>Discriminative Loss</strong></p><p>This part is a face recognition problem. As the paper says, I used the Light-CNN v29 model to extract the 256-d embedding code. Following the <a href="https://gist.github.com/kairess/950cb3db37d1236bf46a89d381f73996">implementation of Light-CNN v29</a>, I added the training code with the Adam optimizer and Categorical Cross-Entropy Loss.</p><pre>lcnn = build()<br>optimizer = Adam(0.00001)<br>lcnn.compile(loss=&#39;categorical_crossentropy&#39;,<br>                         optimizer=optimizer,<br>                         metrics=[&#39;accuracy&#39;])</pre><p>The training data I used is the <a href="https://www.researchgate.net/publication/308278553_MS-Celeb-1M_A_Dataset_and_Benchmark_for_Large-Scale_Face_Recognition">MS-Celeb-1m Aligned Face dataset</a>. The whole training time took about 2 weeks. Finally, the model reached 97.5% accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/281/1*EaUnWYSEryBGApXNPhyppA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/257/1*b8HanioHbjOw4t2MbS1-2Q.png" /><figcaption>Accuracy and Loss of Light CNN training</figcaption></figure><p><strong>Facial Content Loss</strong></p><p>As for Facial Content Loss, the paper describes a condensed version of the segmentation network. They use Resnet-50 as the backbone of the segmentation network by removing the fully connected layers and adding a 1*1 convolution layer at its top. To increase the output resolution, they change the stride from 2 to 1 at Conv_3 and Conv_4. To avoid extra effort on re-training the model on the ImageNet, I used a <a href="https://github.com/divamgupta/image-segmentation-keras">pre-trained segmentation model</a> which based on U-net architecture and Resnet-50.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ftdwrix6SKdyQcYYBlTcJQ.png" /><figcaption>U-net Architecture</figcaption></figure><p>The training dataset I used is the Helen Face dataset. Instead of using 11 labels, I just selected 5 labels which are eyebrows, eyes, nose, lips, and background.</p><p>I used Categorical Cross-Entropy as the loss function and Adam as the optimizer. Some results are shown in Fig. 6.</p><pre>optimizer = Adam(0.00001)<br>self.resnet.compile(loss=&#39;categorical_crossentropy&#39;,<br>                         optimizer=optimizer,<br>                        metrics=[&#39;accuracy&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J2MuRyS3kZJuz7vPwELXRA.png" /><figcaption>Fig. 6</figcaption></figure><h3><strong>Optimization</strong></h3><p>As the paper says, I solved the following optimization problem with the gradient descent method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/205/1*RbmsTGSB04k57e5Lvs4nJg.png" /></figure><p>Fix the Imitator, the face recognition model, and the segmentation model, initialize and update the facial parameters <strong>x</strong> until reaching the max number of iterations. We set alpha as 0.01, the max number of iterations as 50, the learning rate as 10 and its decay rate as 20% per 5 iterations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/1*-qrQdaB27AK-pJjMa1Mzlw.png" /></figure><p>In Keras, we can easily extract the gradient between the network’s output and input using K.gradients function. Some results were generated by our optimization model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*ZQzo5L-BETPNN4NQdtwNEA.png" /><figcaption>Generated results</figcaption></figure><h3>Some thoughts</h3><p>This paper originated a new approach to customize in-game characters. It is very detailed about each of the components. However, it should be noted that the face alignment is very imported to replicated the results. Another part we need to be careful about is the optimization part. I have tried several methods to calculate the gradients between the output and the input such as <a href="https://arxiv.org/abs/1803.02924">the Newton CG algorithm</a>, <a href="https://en.wikipedia.org/wiki/Sequential_quadratic_programming">the Sequential quadratic programming</a> and the Newton conjugate gradient <a href="https://en.wikipedia.org/wiki/Trust_region">trust-region</a> algorithm. In terms of the performance and time, to calculate gradients from Keras is the most efficient way. Further, I will release the code as soon as possible.</p><p>See you next time!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=728c1096e24d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-innovation/customize-your-in-game-faces-728c1096e24d">Customize Your In-Game Faces</a> was originally published in <a href="https://medium.com/ai-innovation">AI2 Labs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>