<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Aadhithya Sankar on Medium]]></title>
        <description><![CDATA[Stories by Aadhithya Sankar on Medium]]></description>
        <link>https://medium.com/@asankar96?source=rss-82053676fe58------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*1eXhxB-aO_ONAW9jtHIakA.jpeg</url>
            <title>Stories by Aadhithya Sankar on Medium</title>
            <link>https://medium.com/@asankar96?source=rss-82053676fe58------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 05:29:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@asankar96/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Create your own programming in 10 minutes!]]></title>
            <link>https://asankar96.medium.com/create-your-own-programming-in-10-minutes-8cc4cadfebd7?source=rss-82053676fe58------2</link>
            <guid isPermaLink="false">https://medium.com/p/8cc4cadfebd7</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Aadhithya Sankar]]></dc:creator>
            <pubDate>Fri, 15 Jul 2022 21:03:15 GMT</pubDate>
            <atom:updated>2022-07-15T21:03:15.285Z</atom:updated>
            <content:encoded><![CDATA[<h3>Create your own programming in 10 minutes</h3><h4>Ever wanted to create your own esolang? With the rajiniPP project you can create one in less than 10 minutes</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*dltW5Rj9xQ9BqNQEYxTjkw.png" /><figcaption>Tokens</figcaption></figure><p>In my previous post, I introduced to you, rajini++, an esoteric programming language with syntax made of Rajinikanth dialogues. In case if you haven’t read it here, you can find info about rajini++ here:</p><p><a href="https://towardsdatascience.com/rajini-the-superstar-programming-language-db5187f2cc71">rajini++: The Superstar Programming Language</a></p><p>One feature / design choice that wasn’t discussed in the previous post was that the project was designed with flexibility in mind. That allows one to create their own esoteric programming language with all the features supported by rajini++ fairly easily and quickly! All you needs to do is edit one file, edit a couple of filenames, and you are good to go!</p><h3>How does an interpreter / compiler work?</h3><p>First things first, we first need to know what the building blocks of an interpreter / compiler are. Broadly speaking, a compiler has 3 major components:</p><ol><li><strong>The Lexer:</strong> The lexer’s role is to take the program as input and divide it into <em>Tokens</em>.</li><li><strong>The Parser: </strong>The parser’s role is to do a syntax check of the program. It takes the list of tokens as input and create an <a href="https://en.wikipedia.org/wiki/Abstract_syntax_tree">AST</a> as output.</li><li><strong>The Code Generator: </strong>The code generator’s role is to transform the AST created from the parser into machine language.</li></ol><p>rajini++ uses the rajinipp interpreter and has just the lexer and parser, it does not convert rajini++ code into machine language, itrather transpiles and executes rajini++ code in python.</p><p>More information about compiler construction can be found in this medium article:</p><p><a href="https://medium.com/@marcelogdeandrade/writing-your-own-programming-language-and-compiler-with-python-a468970ae6df">Writing your own programming language and compiler with Python</a></p><h3>Enough talk. Time to build your own programming language.</h3><p>Now, we will see how you can create “your own” programming language in less than 10 minutes by just making some changes to the rajini++ code. In this example I will be creating a language called arnoldLang based on dialogues by Arnold Schwarzenegger.</p><h4>Step 0: Prerequisites</h4><ul><li>In this tutorial, we will modify some parts of the code written for rajini++. You need a text editor to do this.</li><li>rajinipp and this tutorial are based on python. Some knowledge of python is required.</li></ul><h4>Step 1: Clone/Download the rajiniPP github repository</h4><p>First things first we need to get a copy of the rajiniPP code. This can be done by either cloning the <a href="https://github.com/aadhithya/rajiniPP.git">repository</a> or downloading the code.</p><p><strong>Note:</strong> For this tutorial, you need python 3.8.x installed on your machine.</p><p><strong>Clone the repository</strong></p><p>If you have git installed on your computer then clone the repository using the following command:</p><p>git clone https://github.com/aadhithya/rajiniPP.git</p><p><strong>Download the repository</strong></p><p>If you don’t have git or don’t know how to use git, you can download the code at this page: <a href="https://github.com/aadhithya/rajiniPP.git">rajiniPP repository</a></p><p>At the page click on code and then on Download ZIP to download the code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HIpBvDdfj5BrGBndim46mA.png" /><figcaption>Fig 1: Download rajiniPP code.</figcaption></figure><p>Now you can unzip the code to a folder using your favourite archiver.</p><p><strong>Note:</strong></p><ul><li>We use poetry to manage environments, requirements and builds. You need to install poetry and set up the project:<br>pip install poetry <br>in the project directory:<br>poetry install <br>get environment path: poetry env info . copy the path to the virtual environment.<br>activate environment: poetry env use path/to/env <br>Or you can use your own virtual environment.</li><li>Install project dependencies: poetry install</li></ul><h4>Step 2: Editing the token.yml file</h4><p>Now, this is the most important part of creating the new arnoldLang programming language. rajiniPP is set up in such a way that the key words for the language are stored and loaded from the token.yml file. Therefore, changing the keywords in the token.yml file will de facto lead to a “new” esoteric language with a different set of keywords. Keep in mind that the syntax of the program still remains the same as rajini++ and all we are doing here is changing the keywords. Changing the syntax means changing the production rules and that is beyond the scope of this tutorial.</p><p>The token.yml file is present in rajiniPP/token.yml</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/268/1*z1txPfajLjUyeEen3UFAUQ.png" /><figcaption>Fig 2: The token.yml file is located in rajiniPP/token.yml</figcaption></figure><p>This is how the token.yml file for rajini++ looks:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6f24f9f248071e86c08513b81de5df7c/href">https://medium.com/media/6f24f9f248071e86c08513b81de5df7c/href</a></iframe><p>Here, we are just interested in the PGM_START , PGM_END , PRINT , rajini++ and flow control commands. Now, in order to create your own language, all you need to do is change the values of these tokens to the ones you want. Your creativity is the limit here!</p><p>For my arnoldLang, inspired by <a href="https://github.com/lhartikk/ArnoldC">ArnoldC</a>, I have come up with the following keywords for the tokens:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f302d2a6d85d2faa5ea1e475f4cb0d05/href">https://medium.com/media/f302d2a6d85d2faa5ea1e475f4cb0d05/href</a></iframe><p>And that is the most important part done! All we need to do now is to write programs in arnoldLang and run it!</p><p>The math ops, logical ops and other fields make use of regular expressions and it is too easy to mess up. So, unless you are experienced with regex, I suggest you leave them untouched!</p><h4>Step 3: Change references to rajinipp and rajini++</h4><p>The next step is to change all the references to rajini++ in the project to arnoldLang. The following changes need to be made:</p><ol><li>rajinipp/__init__.py : In __version_str__ rajini++ with your language’s name. In our example’s case, it is arnoldLang.</li><li>rajinipp/__main__.py : change all occurrences of rajini++ and rajinipp with your language’s name. Here we replace it with arnoldLang.</li><li>rajinipp : change the name of the rajinipp folder to the name of your language(&lt;yourLang&gt;) . Make sure that the name contains no spaces. In this case we rename the rajinipp folder to arnoldLang.</li></ol><p>And the rajinipp folder should have been renamed to arnoldLang like so:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/275/1*nLg1PSa-gTDrZ_SZAqeeCw.png" /></figure><p>Congratulations we are 99% done with creating our new arnoldLang. We can already start writing arnoldLang programs and testing the interpreter. Here’s the hello world program:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/58a8978fdcab5972a82ed5c45b9933e7/href">https://medium.com/media/58a8978fdcab5972a82ed5c45b9933e7/href</a></iframe><p>and will output the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xAZYSUhQayFaCt1Ncvx1fw.png" /><figcaption>arnoldLang hello world program</figcaption></figure><p>Assuming that you have renamed the folder properly, you should be able to run arnoldLang programs from the project root directory using the following command:</p><p>python -m arnoldLang run path/to/program.arnold</p><h4>Step 4: Install your language as a package</h4><p>At this point we have created arnoldLang and it can be invoked using the python -m command. If you are satisfied with this, you can call it a success and stop here, but if you want to make this as an installable package just like rajini++ and also potentially publish the package to the pypi registry so that others can try your language out, you are just a few steps away, continue reading!</p><p>In order to package the project and make it installable, we need to make the following changes to the pyproject.toml file:</p><ol><li>pyproject.toml : change the name field to your language’s name and set a discription of your choice.</li><li>pyproject.toml : set the line underneath the [tool.poetry.scripts] section to the following: &lt;yourLanguage&gt; = &lt;yourLanguage&gt;.__main__:app<br>For our example this will be: arnoldLang = arnoldLang.__main__:app . Make sure that &lt;yourLang&gt; matches the name of the folder you set in step 3 of this section. What is happening is we are asking poetry to set the app function in the __main__.py file as the entrypoint when one calls yourLang in the terminal/command prompt.</li><li>under the [tool.semantic_release] change replace rajinipp in rajinipp.__init__:__version__ with &lt;yourLang&gt; . For our example this becomes: arnoldLang.__init__:__version__ . This is required by the semantic release tool to find set the version number automatically.</li></ol><p>In the end for the arnoldLang, the pyproject.toml file should look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e8c2f473f727ef5d88fdd583f7c10254/href">https://medium.com/media/e8c2f473f727ef5d88fdd583f7c10254/href</a></iframe><p>now to create the installable package, all you need to do is run the following command:</p><p>poetry build</p><p>If everything worked fine, you’ll see such an output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*8W_szeE7zYZXswFYmmkckA.png" /></figure><p>The package(whl file) can be found in the dist folder. You can now easily install the package using the following command:</p><p>pip install path/to/&lt;whl-file&gt;</p><p>in the case of arnoldLang it is pip install dist/arnoldLang-0.3.1-py3-none-any.whl</p><p>Test installation with arnoldLand version .</p><p>If you see the version information printed, congratulations, you have successfully installed your own language! You can now run programs by simply typing arnoldLang run path/to/pgm.arnold . You can also start the interactive shell by running the arnoldLang shell command! Enjoy!</p><p>Here’s an example program from rajini++ and its equivalent in our newely created arnoldLang:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JZD0teiMujAVRvhNnbzKbA.png" /><figcaption>Left: If-else conditional in rajini++, Right: if-else contitional in arnoldLang</figcaption></figure><p>as expected, the output of running the arnoldLang or rajini++ programs is the same.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B8tU4gNw-EL_P574qy0Nmw.png" /></figure><h4>Step 5: Congratulations!</h4><p>Congratulations, you have now created your own programming language based on the flexible foundation provided by the rajini++ project! This tutorial has hopefully made you more interested about how programming languages are written. As a next step, dive into the code of rajini++ and try to understand how the BNF productions relate to the syntax of the program and how you can change the production rules to modify the syntax of the language. If this seems too complicated for you, have fun creating programming languages for your favourite actors, movies or tv shows! Enjoy!</p><h3>Useful Resources</h3><ul><li><a href="https://towardsdatascience.com/rajini-the-superstar-programming-language-db5187f2cc71">rajini++: The Superstar Programming Language</a></li><li><a href="https://github.com/aadhithya/rajiniPP">GitHub - aadhithya/rajiniPP: Programming language based on Iconic dialogues of Rajinikanth. 🤘🏽</a></li><li><a href="https://medium.com/@marcelogdeandrade/writing-your-own-programming-language-and-compiler-with-python-a468970ae6df">Writing your own programming language and compiler with Python</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8cc4cadfebd7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[rajini++: The Superstar Programming Language]]></title>
            <link>https://medium.com/data-science/rajini-the-superstar-programming-language-db5187f2cc71?source=rss-82053676fe58------2</link>
            <guid isPermaLink="false">https://medium.com/p/db5187f2cc71</guid>
            <category><![CDATA[interpreters]]></category>
            <category><![CDATA[tech]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Aadhithya Sankar]]></dc:creator>
            <pubDate>Tue, 17 May 2022 16:20:19 GMT</pubDate>
            <atom:updated>2022-05-17T16:20:19.102Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9YHy81Ghl15z5ZeFixsbcA.png" /><figcaption>Introducing rajini++, the superstar programming language! (<a href="https://github.com/aadhithya/rajiniPP">Source</a>)</figcaption></figure><p>Introducing rajini++, an esoteric programming language based on the dialogues of superstar Rajinikanth. The syntax and keywords used in rajini++ are based on the dialogues of Superstar Rajinikanth.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5dda90ce9b263b3f753d37d69a8e8d4e/href">https://medium.com/media/5dda90ce9b263b3f753d37d69a8e8d4e/href</a></iframe><h3>Who is Rajinikanth?</h3><p>Before we start, you might wonder who Superstar Rajinikanth is. Well, for starters, he is one of India’s biggest and most popular actors and has been in the cinema industry for over 45 years! To know more about the legend follow the links below:</p><ul><li><a href="https://www.youtube.com/watch?v=YDUQZwMHMoo">Who is Rajinikanth</a>?</li><li><a href="https://en.wikipedia.org/wiki/Rajinikanth">https://en.wikipedia.org/wiki/Rajinikanth</a></li></ul><h3>Origins of rajini++</h3><p>Well, the origins of rajini++ goes back some 6 years when I first came across this <a href="https://en.wikipedia.org/wiki/Esoteric_programming_language">esolang</a> called <a href="http://lhartikk.github.io/ArnoldC/">ArnoldC</a>. Right from that moment I had wanted to create such an esolang for the superstar. 6 years and a lot of learnings later, here we are, finally with a usable version of the rajini++ language.</p><h3>Get rajini++</h3><p>The rajinipp package interprets and executes rajini++ programs. It is completely written in python and can be easily installed using pip with the following command:</p><ul><li>pip install rajinipp</li></ul><p><strong>Note: </strong>The rajinipp interpreter requires python ≥ 3.8. You can get python <a href="https://www.python.org/downloads/">here</a>. pip is also required to install the rajinipp package. You can get pip <a href="https://pip.pypa.io/en/stable/installation/">here</a>.</p><ul><li>Test rajinipp installation: rajinipp version</li></ul><h3>The rajini++ Language</h3><p>rajini++ is based on the most iconic dialogues of Rajinikanth. The syntax and keywords used in rajini++ are based on the dialogues of Superstar Rajinikanth.</p><p>rajini++ is fairly feature rich and supports the following features:</p><ul><li>Math Ops (SUM, SUB, MUL, DIV, MOD)</li><li>Unary Ops (SUM, SUB)</li><li>Printing</li><li>Variable declaration</li><li>Variable access</li><li>Variable manipulation and assignment</li><li>Datatypes(Bool, String, Number)</li><li>Logical ops (≥, &gt;, ≤, &lt;, ==, !=)</li><li>If statement</li><li>If-Else statement</li><li>For loop</li><li>While loop</li><li>Functions</li><li>Functions with return</li></ul><h3>Example Programs</h3><ol><li><strong>If-Else Program</strong></li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/492adb34479b54adb8e3910e17636065/href">https://medium.com/media/492adb34479b54adb8e3910e17636065/href</a></iframe><p>Output:</p><pre>Example: If-Else Statement<br>x ( 5.0 ) is less than 10!</pre><p><strong>2. FizzBuzz program</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1c3dc14e53ce8951cd3beb54bec600f6/href">https://medium.com/media/1c3dc14e53ce8951cd3beb54bec600f6/href</a></iframe><p>Output:</p><pre>1.0<br>2.0<br>Fizz<br>4.0<br>Buzz<br>Fizz<br>7.0<br>8.0<br>Fizz<br>Buzz<br>11.0<br>Fizz<br>13.0<br>14.0<br>FizzBuzz</pre><p><strong>3. Function with Return</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/34104534ff49a6e6788ebd58d2667d77/href">https://medium.com/media/34104534ff49a6e6788ebd58d2667d77/href</a></iframe><p>Output:</p><pre>Hi from main!<br>Hello from myfunc_one!<br>returning ix = 100.0 to main<br>Value returned from myfunc_one: 100.0</pre><h3>Running rajini++ programs</h3><p>The rajini++ programs are stored in .rpp files. To run a rajini++ program one simply needs to run the following command:</p><p>rajinipp run path/to/my_program.rpp</p><p><strong>Note: </strong>be mindful of the fact that paths use backward slashes in windows!</p><h3>The rajinipp Interactive Shell</h3><p>Apart from being able to run rajini++ programs, the rajinipp python package also provides an interactive shell(<em>experimental</em>) to try out rajini++ commands. One can enter the rajinipp shell using the following command:</p><p>rajinipp shell</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xVjx1EePbcz_qnjL3Mp5_g.png" /><figcaption>The rajinipp shell.</figcaption></figure><p>Exit shell using ctrl+D .</p><p><strong>Note: </strong>The rajinipp shell at its current stage only supports a limited subset of the features of the rajini++ language. Control Statements(Loops, if-else) and functions are not supported.</p><h3><strong>Embedding rajini++ code in python scripts</strong></h3><p>Because the rajinipp interpreter is written completely in python, it is possible to run rajini++ code inside python scripts (<em>experimental</em>)!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/37886819d4f5ffc14b61ef6cf7e18a05/href">https://medium.com/media/37886819d4f5ffc14b61ef6cf7e18a05/href</a></iframe><p>Output:</p><pre>Hello, World from python!<br>Hello world from rajini++!<br>Executing 5+5 in rajini++<br>5 + 5 = 10.0</pre><p><strong>Note: </strong>The rajinipp.runner.RppRunner.eval() method at its current stage only supports a limited subset of the features of the rajini++ language. Control Statements(Loops, if-else) and functions are not supported.</p><h3>Learn more about rajini++</h3><ul><li><strong>Learn the rajini++ Language: </strong>The rajini++ language documentation with syntax and examples can be found at the <a href="https://github.com/aadhithya/rajiniPP/wiki/">rajiniPP Wiki</a>.</li><li><strong>Example programs</strong> can be found here: <a href="https://github.com/aadhithya/rajiniPP/tree/master/examples">Example Programs</a>.</li><li><strong>The rajini++ Language Spec:</strong> The rajini++ commands and its equivalent in python3 can be found <a href="https://github.com/aadhithya/rajiniPP/wiki/1.-rajiniPP:-Language-Specification">here</a>. The language spec is not yet complete and is a work in progress.</li><li><strong>The rajinipp Interpreter Documentation:</strong> The documentation for the rajinipp interpreter can be found <a href="https://github.com/aadhithya/rajiniPP/wiki/2.-rajinipp:-The-interpreter">here</a>.</li></ul><p><strong>Note: </strong>As new features are continuously added to the rajinipp repository, the language syntax and documentation are subject to change. Please refer to the <a href="https://github.com/aadhithya/rajiniPP/wiki/">official wiki</a> to stay updated on the latest changes.</p><h3>Audit the code and Contribute to rajini++</h3><p>If the project has caught your interest and if you want to audit or contribute to the project head to the github repo:</p><p><a href="https://github.com/aadhithya/rajiniPP">GitHub - aadhithya/rajiniPP: Programming language based on Iconic dialogues of Rajinikanth. 🤘🏽</a></p><h3>rajinipp on PyPI</h3><p>rajinipp is on PyPI and can be found here:</p><p><a href="https://pypi.org/project/rajinipp/">rajinipp</a></p><p>Stay tuned for the next post where I will be writing about how you can fork the <a href="https://github.com/aadhithya/rajiniPP">rajiniPP repo</a> and create your own esolang in a matter of few minutes!</p><h3>Some Iconic Rajinikanth Dialogues</h3><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FpI6lkreanbg%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DpI6lkreanbg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FpI6lkreanbg%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5e239ae01327815e4f20c5547d0f9464/href">https://medium.com/media/5e239ae01327815e4f20c5547d0f9464/href</a></iframe><p><strong>Stay Rajinified! Katham Katham!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db5187f2cc71" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/rajini-the-superstar-programming-language-db5187f2cc71">rajini++: The Superstar Programming Language</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystified: Wasserstein GAN with Gradient Penalty]]></title>
            <link>https://medium.com/data-science/demystified-wasserstein-gan-with-gradient-penalty-ba5e9b905ead?source=rss-82053676fe58------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba5e9b905ead</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Aadhithya Sankar]]></dc:creator>
            <pubDate>Sat, 02 Oct 2021 12:49:25 GMT</pubDate>
            <atom:updated>2021-10-04T20:15:20.569Z</atom:updated>
            <content:encoded><![CDATA[<h3>Demystified: Wasserstein GAN with Gradient Penalty(WGAN-GP)</h3><h4>What is Gradient Penalty? Why is it better than gradient clipping? How to implement Gradient Penalty?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*ieyAKSxgJGqX9lktL_ujnA.png" /><figcaption>Fig. 1 (Left) Gradient norm when using weight clipping either explodes or vanishes, not using GP. (Right) Weight Clipping pushes weights towards two values, unlike GP. Image Source: [1]</figcaption></figure><p>In this post we will look into Wasserstein GANs with Gradient Penalty. While the original Wasserstein GAN[2] improves training stability, there still are cases where it generate poor samples or fails to converge. Recap that the cost function of WGAN is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/1*OScRv3BjfaCrd-o6z3IcUg.png" /><figcaption>Eq. 1: WGAN value function.</figcaption></figure><p>where <em>f </em>is<strong> </strong>1-Lipschitz continuous. The issues with WGAN arises mainly because of the weight clipping method used to enforce Lipschitz continuity on the critic. WGAN-GP replaces weight clipping with a constraint on the gradient norm of the critic to enforce Lipschitz continuity. This allows for more stable training of the network than WGAN and requires very little hyper-parameter tuning. WGAN-GP and this post builds on top of Wasserstein GANs, which has already been discussed in a previous post in the <a href="https://asankar96.medium.com/list/demystified-deep-learning-1273a227565c"><em>Demystified Series</em></a>. Check the post below to learn about WGAN.</p><p><a href="https://towardsdatascience.com/demystified-wasserstein-gans-wgan-f835324899f4">Demystified: Wasserstein GANs (WGAN)</a></p><h4>Statement 1</h4><blockquote>The optimal 1-Lipschitz function that is differentiable, <strong><em>f* </em></strong>that minimises Eq. 1 has unit gradient norm almost everywhere under ℙr and ℙg.</blockquote><p>ℙr and ℙg are the real and fake distributions respectively. Proof for statement 1 can be found in [1].</p><h3>Issues with Gradient Clipping</h3><h4>Capacity Underuse</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/1*4hWCkkakgFiq3NU7g9WDbQ.png" /><figcaption>Fig. 2: value surfaces learnt by WGAN critic (top) with gradient clipping, (bottom) with gradient penalty. Image Source: [1]</figcaption></figure><blockquote>Using Weight clipping to enforce the k-Lipschitz constraint leads to the critic learning very simple functions.</blockquote><p>From Statement 1, we know that the gradient norm of the optimal critic is 1 almost everywhere in both ℙr and ℙg. In the weight clipping set up, the critic tries to attain its maximum gradient norm <em>k</em><strong><em>, </em></strong>and ends up learning simple functions.</p><p>Fig. 2 shows this effect. The critics are trained to convergence with fixed generated distribution(ℙg) as real distribution(ℙr)+ unit gaussian noise. We can clearly see that Critic trained with weight clipping ends up learning simple functions and fails to capture higher moments whereas the critic trained with Gradient Penalty does not suffer from this issue.</p><h4>Exploding and Vanishing Gradients</h4><blockquote>The interaction between the weight constraint and the loss function makes training of WGAN difficult and leads to exploding or vanishing gradients.</blockquote><p>This can be clearly seen in Fig.1 (left) where the weights of the critic explodes or vanishes for different clipping values. Fig. 1 (right) also shows that the gradient clipping pushes weights of the critic to the two extreme clipping values. On the other hand, the critic trained with Gradient Penalty does not suffer from such issues.</p><h3>Gradient Penalty</h3><blockquote>The idea of Gradient Penalty is to enforce a constraint such that the gradients of the critic’s output w.r.t the inputs to have unit norm (Statement 1).</blockquote><p>The authors propose a soft version of this constraint with penalty on the gradient norm on the samples <strong>x̂</strong> ∈ ℙ<strong>x̂</strong>. The new objective is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*OMOCwiwLF4WRXmQ174hUcQ.png" /><figcaption>Eq. 2: Critic loss function</figcaption></figure><p>In Eq. 2 the terms to the left of the sum is the original critic loss and the term to the right of the sum is the gradient penalty.</p><p>ℙ<strong>x̂ </strong>is the distribution obtained by uniformly sampling along a straight line between the real and generated distributions ℙr and ℙg. This is done because the optimal critic has straight lines with unit gradient norm between the samples coupled from ℙr and ℙg.</p><p>λ, the penalty coefficient is used to weight the gradient penalty term. In the paper, the authors set λ=10 for all their experiments.</p><p><strong>Batch Normalisation is not used in the critic anymore </strong>because batch norm maps a batch of inputs to a batch of outputs. In our case <strong>we want to be able to find gradients of each output w.r.t their respective inputs.</strong></p><h3>Code</h3><h4>Gradient Penalty</h4><p>The implementation of gradient penalty is shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5adf01b038aa73fe42052cfd198eee27/href">https://medium.com/media/5adf01b038aa73fe42052cfd198eee27/href</a></iframe><h4>WGAN-GP</h4><p>The code to train the WGAN-GP model can be found here:</p><p><a href="https://github.com/aadhithya/gan-zoo-pytorch">GitHub - aadhithya/gan-zoo-pytorch: A zoo of GAN implementations</a></p><h3>Outputs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/1*sagQ0eASN_ODqJzEKQs6jw.png" /><figcaption>Fig. 3: Images generated by WGAN-GP model. Note that the results are early results, the trainnig was stopped as soon as it was confirmed that the model was training as expected. Image Source: [3]</figcaption></figure><p>Fig.3 shows some early results from training the WGAN-GP. <strong>Please note that the images in Fig. 3 are early results, the training was stopped as soon as it was confirmed that the model was training as expected. The model was not trained to convergence.</strong></p><h3>Conclusion</h3><p>Wasserstein GANs offer much needed stability in training Generative Adversarial Networks. However, the use of gradient clipping leads of various issues such as exploding and vanishing gradients, etc. The Gradient Penalty constraint does not suffer from these issues and therefore allows for easier optimisation and convergence compared to the original WGAN. This post looked at these issues, introduced the Gradient Penalty constraint and also showed how to implement Gradient Penalty using PyTorch. Finally the code to train WGAN-GP model along with some early stage outputs were provided.</p><p>If you liked this post, consider following the author, <a href="https://medium.com/u/82053676fe58">Aadhithya Sankar</a>.</p><h3>References</h3><p>[1] Gulrajani, Ishaan, et al. “Improved training of wasserstein gans.” <em>arXiv preprint arXiv:1704.00028</em> (2017).</p><p>[2] Arjovsky, Martin, Soumith Chintala, and Léon Bottou. “Wasserstein generative adversarial networks.” <em>International conference on machine learning</em>. PMLR, 2017.</p><p>[3] <a href="https://github.com/aadhithya/gan-zoo-pytorch">https://github.com/aadhithya/gan-zoo-pytorch</a></p><h3>More Posts by the Author</h3><p>If you enjoyed this post, you may also like:</p><ul><li><a href="https://towardsdatascience.com/principal-component-analysis-part-1-the-different-formulations-6508f63a5553">Principal Component Analysis Part 1: The Different Formulations.</a></li><li><a href="https://towardsdatascience.com/a-primer-on-atrous-convolutions-and-depth-wise-separable-convolutions-443b106919f5">A Primer on Atrous Convolutions and Depth-wise Separable Convolutions</a></li><li><a href="https://towardsdatascience.com/real-time-artwork-generation-using-deep-learning-a33a2084ae98">Real-time Artwork Generation using Deep Learning</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba5e9b905ead" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/demystified-wasserstein-gan-with-gradient-penalty-ba5e9b905ead">Demystified: Wasserstein GAN with Gradient Penalty</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Principal Component Analysis Part 1: The Different Formulations.]]></title>
            <link>https://medium.com/data-science/principal-component-analysis-part-1-the-different-formulations-6508f63a5553?source=rss-82053676fe58------2</link>
            <guid isPermaLink="false">https://medium.com/p/6508f63a5553</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[principal-component]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[dimensionality-reduction]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Aadhithya Sankar]]></dc:creator>
            <pubDate>Wed, 29 Sep 2021 00:24:21 GMT</pubDate>
            <atom:updated>2021-09-29T00:24:21.973Z</atom:updated>
            <content:encoded><![CDATA[<h4>What is Principal Component Analysis? What are the Maximum Variance and Minimum Error formulations of PCA? How do we reduce dimensionality using PCA?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*ewfs7RSRgVRPhu6zGdAkbw.png" /><figcaption>Fig.1: Principal components of a multivariate gaussian centered at (1,3). Image Source: [3].</figcaption></figure><p>We all know that Principal Component Analysis is one of the standard methods used in dimensionality reduction. There are multiple posts detailing the code and implementation of PCA; in this post, however, we will look into how PCA is formulated and how we arrive at the PCA algorithm. While it is widely known that the Principal components are the eigenvectors corresponding to the largest eigenvalues of the covariance matrix; this post will explore why that is the case and how that solution is arrived at. The content of this post will be based on the material provided in Chapter 12 of [1] and [2].</p><p>Before we jump into PCA, it is important to know what eigenvalues and eigenvectors are. Let <strong>A</strong> ∈ <strong>R</strong>^{n×n} be an<strong> n×n</strong> matrix. Then, a vector <strong>x</strong> ∈ <strong>C</strong>^n is called the eigenvector of <strong>A </strong>if</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/189/1*5v6N2ZGCTX8uq-7e1cV7aA.png" /><figcaption>Eq. 1: Eigenvalues and Eigenvectors</figcaption></figure><p>The λ ∈ <strong>C </strong>is called the Eigenvalue of <strong>A.</strong></p><p>The Principal Component Analysis(PCA) problem can be formulated in two ways: <strong>Maximum Variance Formulation</strong> and <strong>Minimum Error Formulation</strong>. In the Maximum Variance Formulation, the goal is to find the orthogonal projection of the data into a lower dimensional linear space such that the variance of the projected data is maximised . In the Minimum Error Formulation, PCA is defined as the linear projection that minimises the average projection cost (mean squared error) between the data points and their projection . We will see in the following sections that, both these formulations lead to the same solution.</p><h3>PCA: Maximum Variance Formulation</h3><p>Given the set of observations {<strong>x</strong>_n}, n = 1, 2, …, N and <strong>x</strong>_n ∈ <strong>R</strong>^D, our goal according to the maximum variance formulation is to find the orthogonal projection of <strong>x</strong>_n onto a space with dimensions <strong><em>M&lt;D</em></strong>.</p><p>Now let us consider the simplest case where <strong>M</strong>=1. We define a vector <strong>w1</strong>∈ <strong>R</strong>^D<strong> </strong>as the direction of the lower dimensional space. Since we are only interested in the direction of the space, we set <strong>w1 </strong>to be of unit length. i.e.,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/125/1*AN4yU0amfCu-2a1slWgLOQ.png" /><figcaption>Eq.2: <strong>w1 </strong>is a unit vector.</figcaption></figure><p>Then the data observations <strong>x</strong>_n can be projected onto this new space as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/139/1*p5q4E5p0ZGsmIBJVHIiuvw.png" /><figcaption>Eq.3: projection of <strong>x</strong>_n onto new space defined by <strong>w1</strong>.</figcaption></figure><p>If <strong>x̄ </strong>is the mean of the data observations in the original space, then the mean of the samples in the projected space is given by</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/111/1*j_7foOjWZCBbaYiKruUJ6g.png" /><figcaption>Eq. 3: projection of <strong>x̄</strong> onto new space defined by <strong>w1</strong>.</figcaption></figure><p>Now, we can write the variance of the projected data as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*3K5GuaWjrHDYRoEjSubDLw.png" /><figcaption>Eq. 4: Variance of <strong>x̂</strong>.</figcaption></figure><p>where <strong>S</strong> is the covariance matrix of the observed data in the original high dimensional space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/297/1*l4RD51u2PYqppcBPT7gTpQ.png" /><figcaption>Eq. 6: Covariance of <strong>x</strong>.</figcaption></figure><p>Now, according to the maximum variance formulation definition, we need to maximise the variance of <strong>x̂</strong>. This can be done by maximising Eq. 4. Eq. 4 has a trivial solution when ||<strong>w1</strong>|| → ∞. To prevent this, we make use of the unit norm constraint we set earlier in Eq.2. To address this, we introduce a Lagrange multiplier λ1 and and formulate our optimisation objective as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/1*zmf3-Xru6yD3LSCtha2WyQ.png" /><figcaption>Eq. 7: Maximum Variance Optimisation Problem.</figcaption></figure><p>Setting the derivative of Eq. 7 w.r.t <strong>w1 </strong>to 0, we get a stationary point when,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*623zJO52EXPX4wlOlqHVsQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/166/1*5_ZE2LR83bpRg01Kk2zK9A.png" /><figcaption>Eq. 8: Setting derivative of Eq. 7 w.r.t <strong>w1 </strong>to zero.</figcaption></figure><p>This shows that at the stationary point, <strong>w1 </strong>must be an eigenvector of <strong>S </strong>and λ1 the eigenvalue. corresponding to the eigenvector <strong>w1</strong>. Left-multypling Eq.7 with <strong>w1</strong>^T, we ca see that the maximum variance is equal to the eigenvalue λ1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/1*NW0R2YVA4Ss6Vb60IoNFXA.png" /><figcaption>Eq. 9: The maximum variance in the lower dimensional space is equal to the eigenvalue corresponding to eigenvector <strong>w1</strong>.</figcaption></figure><p>We can identify additional principal components by choosing directions that maximise variance while being orthogonal to the existing ones. For the general case of a lower dimensional space with <strong>M </strong>dimensions with <strong>M&lt;D, </strong>the principal components are the eigenvectors <strong>w</strong>1, <strong>w</strong>2, … <strong>w</strong>m corresponding to the M largest eigenvalues λ1, λ2, …, λm.</p><h3>PCA: Minimum Error Formulation</h3><p>Let {<strong>x</strong>_n}, n = 1, 2, …, N and <strong>x</strong>_n ∈ <strong>R</strong>^D be the set of data observations, then our goal according to the minimum error formulation of PCA is to find transformation that minimises the reconstruction error:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/381/1*P1-xozurlAAGZuGa30QoSQ.png" /><figcaption>Eq. 10: Minimum Error Objective</figcaption></figure><p>where <strong>x̃ </strong>is the reconstruction generated from a lower dimensional latent variable. Here, we have a complete D dimensional orthonormal(orthogonal and unit length) basis <strong>w</strong>_i, where i= 1, 2, … D. Then we have</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/262/1*i9UX0xVF56Rt0GB1ys4zeA.png" /><figcaption>Eq. 11: Orthonormal Basis.</figcaption></figure><p>where δᵢⱼ is the kronecker delta. Since the basis is complete, we can represent any vector as a linear combination of the basis vectors</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/152/1*SqlRwp32k5sX8_xK_CPMPw.png" /><figcaption>Eq. 12</figcaption></figure><p>Since we have orthonormal basis, we have the solution for α_ni as the dot product of the <strong>x</strong>_n and <strong>w</strong>_i. Then, we can write Eq. 12 as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/1*Gvjn388QF2a6VtNVcto-sQ.png" /><figcaption>Eq. 13 Proof for this can be found in [4]</figcaption></figure><p>With dimensionality reduction, our goal is to find a M dimensional representation for our D dimensional data (with <strong><em>M&lt;D</em></strong>) by projecting it onto the lower dimensional space. We can represent this M dimensional space using the first M basis vectors</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/1*GteRydGumfzwkN0eROviPQ.png" /><figcaption>Eq. 14</figcaption></figure><p>and the remaining (D-M) basis are shared by all data points(shared offsets). In Eq. 14, <em>z_ni </em>depends on the individual data points while b_i are constants that are shared by all the data points.</p><p>From Eq. 13 and Eq. 14 we can compute the difference between <strong>x</strong>_n and <strong>x̃</strong>_n as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/1*f0C0BCtHPhfLHa-L3vtDbw.png" /><figcaption>Eq. 15: Difference between <strong>x</strong>_n and <strong>x̃</strong>_n.</figcaption></figure><p>Now, we can substitute Eq. 15 in Eq. 10 to get the objective function as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/1*xAIElnelJsu8vGW9ZEWQ3w.png" /><figcaption>Eq. 16</figcaption></figure><p>Taking derivative w.r.t <em>z_nj </em>and setting to zero, we get</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/153/1*OKMkV82lgDDW7CvXKhSOAA.png" /><figcaption>Eq. 17</figcaption></figure><p>Taking derivative w.r.t <strong>b</strong>_j and setting to zero, we get</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/138/1*R91383fCRbwxEObh74UFIw.png" /><figcaption>Eq. 18</figcaption></figure><p>Now, we can substitute Eq. 17 and Eq. 18 in Eq. 16 and get</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*MzgXNUXw5Ez2rEcgBeHN4Q.png" /><figcaption>Eq. 18</figcaption></figure><p>Our goal is to minimise J(<strong>w</strong>), but we observe that a trivial solution exists to this problem when <strong>w</strong> = 0. To overcome this, we again make use of the property of orthonormal bases and set the normalisation constraint ||<strong>w</strong>||<strong>² = </strong>1.</p><p>Now, we look at a simple example where D=2 and M=1. We have to choose a direction <strong>w</strong>2 such that we can minimise the following objective</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/293/1*LdSfBPKjOcaTWSX50Ycb0w.png" /><figcaption>Eq. 19</figcaption></figure><p>As before, taking derivative w.r.t <strong>w2 </strong>and setting to 0, we get</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/125/1*w3gvk19QldrbRdDdea9kyQ.png" /><figcaption>Eq. 20</figcaption></figure><p>where <strong>w2 </strong>is the eigenvector corresponding to eigenvalue λ2. Substituting Eq. 20 in Eq. 19, we get <strong><em>J = λ2</em></strong><em>, </em>i.e., <strong>J </strong>is minimised when we choose the eigenvector with the smallest eigenvalue. In the general case, when we have <strong><em>M&lt;D</em></strong>, the Minimum Error for reconstruction <strong>J</strong> is obtained by choosing the eigenvectors <strong>w</strong>_i corresponding to the <strong><em>(D-M) </em></strong>smallest eigenvalues obtained from the covariance matrix given by</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/117/1*DcVch42alEE7fDVjv6NNNA.png" /><figcaption>Eq. 21</figcaption></figure><p>The minimum reconstruction error(distortion measure) is given by</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/131/1*e8g2VO1agsQph7HXUn0BmA.png" /><figcaption>Eq. 22</figcaption></figure><p>the sum of <strong><em>(D-M) </em></strong>eigenvalues. Therefore, our goal is to use the M largest eigenvalues so that the distortion measure <strong>J, </strong>now constituting of the <strong><em>(D-M)</em></strong> smallest eigenvalues is minimised. Thus we can conclude that minimising the reconstruction error maximises the variance of the projection.</p><h3><strong>Dimensionality Reduction with PCA</strong></h3><p>Now that we’ve seen that the M principal components are the M eigenvectors corresponding to the M largest eigenvalues of the covariance matrix, we move on to see how PCA is applied to reduce dimensionality.</p><p>Dimensionality reduction using PCA consists of 4 steps:</p><p><strong>1. Center the data<br></strong>The first step is to compute and subtract the mean from the data points so that the data is centred around 0 and therefore has zero mean.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/121/1*EH0i5lp6i0Ue_GauPfep_w.png" /><figcaption>Eq. 23: Centered Data <strong>X̂</strong></figcaption></figure><p><strong>2. Compute Covariance Matrix</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/125/1*YY2ffs_-UM24ajHEm9Ds0Q.png" /><figcaption>Eq. 24: Covariance Matrix</figcaption></figure><p><strong>3. Compute Eigen Values and vectors using Eigen Value Decomposition<br></strong>The goal of PCA is the transformation of the coordinate system such that the covariance between the new axes are 0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*9-2QLVeHnOlKtJP50-SQ-Q.png" /><figcaption>Fig. 2: Goal of PCA is to find space W such that the covariance between new axes is 0.</figcaption></figure><p>So we perform Eigen Value Decomposition of the Covariance matrix <strong>S</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/104/1*yy6HWo8ZpJNb7OzGOZbRdQ.png" /><figcaption>Eq. 25: EVD of the covariance matrix.</figcaption></figure><p>Here, Γ∈ℝ^(DxD) is the matrix of eigenvectors and Λ∈ℝ^(DxD) is a diagonal matrix containing the eigenvalues.<br><strong>4. Dimensionality Reduction<br></strong>Now that we have the eigenvectors Γ , to reduce the dimensionality, we can truncate Γ by keeping only the columns(eigenvectors) corresponding to the largest M eigenvalues. We call the truncated Γ matrix as Γ’. Then the representation in the reduced space is obtained by</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/87/1*wZ-F7zQZUkutc83iiLEQjg.png" /><figcaption>Eq. 26: Dimensionality Reduction</figcaption></figure><h4>Code: Dimensionality Reduction using PCA</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4dc0444f20111ee94fde7eff3dc56ad5/href">https://medium.com/media/4dc0444f20111ee94fde7eff3dc56ad5/href</a></iframe><h4><strong>Performance of PCA with EVD</strong></h4><p>PCA using Eigen Value Decomposition(EVD) is very expensive with a complexity of O(D³) where D is the dimensionality of the input data. EVD computes all the eigenvalue and eigenvector pairs, where as usually we need only the eigenvectors corresponding to the M largest eigenvalues. Therefore in practice, much efficient iterative approaches such as Power Iteration method is used to compute the eigenvectors.</p><h4><strong>PCA and Data Standardisation[2]</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aOPir1Iu38GZbkVxaAeG0g.png" /><figcaption>Fig.3 PCA can get mislead by unstandardised data. (a) Principal Component is skewed because PCA is misled by the unstandardised data. (b) PCA when the scales are standardised. Image generated from code at [5].</figcaption></figure><p>The principal directions from PCA are the ones along which the variance is the most. So, PCA can be mislead by directions along which the variance appears high just because of the measurement scale. We can see this in Fig. 3(a) where the Principal Component is not aligned properly because it is mislead by the unstandardised scale. Fig. 3(b) shows the correct principal component when the scales are standardised. Therefore, care needs to be taken to standardise the data to avoid such issues.</p><h3>Conclusion</h3><p>In this rather long post, we dove deep and looked at the two formulations of PCA: the Maximum Variance and Minimum Error Formulation. We saw that both the formulations had the same solution/algorithm — select as the new basis, the eigenvectors corresponding to the M largest eigenvalues of the Covariance matrix of the data. We saw how PCA can be use for dimensionality reduction and how it can be implemented in python. Finally we briefly looked into the importance of standardising the data and how it affects the algorithm. This brings to the end of this post, which is merely Part 1 of this series on PCA. The next parts will cover Probabilistic PCA, Singular Value Decomposition, Auto encoders and the relationship between Autoencoders, PCA and SVD.</p><p>Follow <a href="https://medium.com/u/82053676fe58">Aadhithya Sankar</a> to get notified when the next parts are made available!</p><p>If you find any mistakes, please leave a comment, I will fix them!🙏🏽 ✌🏽</p><h3>References</h3><p>[1]<a href="https://www.microsoft.com/en-us/research/uploads/prod/2006/01/Bishop-Pattern-Recognition-and-Machine-Learning-2006.pdf"> Bishop, Christopher M. Pattern Recognition and Machine Learning. New York :Springer, 2006.</a></p><p>[2] <a href="http://noiselab.ucsd.edu/ECE228/Murphy_Machine_Learning.pdf">Murphy, Kevin P. <em>Machine learning: a probabilistic perspective</em>. MIT press, 2012.</a></p><p>[3]<a href="https://commons.wikimedia.org/wiki/File:GaussianScatterPCA.svg#/media/File:GaussianScatterPCA.svg">https://commons.wikimedia.org/wiki/File:GaussianScatterPCA.svg#/media/File:GaussianScatterPCA.svg</a></p><p>[4] <a href="https://www.math.ucdavis.edu/~linear/old/notes21.pdf">https://www.math.ucdavis.edu/~linear/old/notes21.pdf</a></p><p>[5] <a href="https://github.com/probml/pyprobml">Murphy, K., Soliman, M., Duran-Martin, G., Kara, A., Liang Ang, M., Reddy, S., &amp; Patel, D. (2021). PyProbML library for Probabilistic Machine Learning [Computer software].</a></p><h3><strong>Resources</strong></h3><p>Here are some resources that can help understand the topic better</p><ol><li><strong>PCA: Maximum Variance Formulation (University of Amsterdam)</strong></li></ol><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F-A64n7Ss35g%3Flist%3DPL8FnQMH2k7jzhtVYbKmvrMyXDYMmgjj_n&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D-A64n7Ss35g&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F-A64n7Ss35g%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/bc3e18508ac1f85ec2e621bf91a5a82e/href">https://medium.com/media/bc3e18508ac1f85ec2e621bf91a5a82e/href</a></iframe><p><strong>2. PCA Minimum Error Formulation (University of Amasterdam)</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F9EgXm_CtfBg%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D9EgXm_CtfBg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F9EgXm_CtfBg%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/b04fbcee5b626c656cd8fe99a7f721d4/href">https://medium.com/media/b04fbcee5b626c656cd8fe99a7f721d4/href</a></iframe><h3>More Works by the Author</h3><p>If you liked this post, you might also enjoy the following posts:</p><ul><li><a href="https://towardsdatascience.com/real-time-artwork-generation-using-deep-learning-a33a2084ae98">Real-time Artwork Generation using Deep Learning</a></li><li><a href="https://towardsdatascience.com/a-primer-on-atrous-convolutions-and-depth-wise-separable-convolutions-443b106919f5">A Primer on Atrous Convolutions and Depth-wise Separable Convolutions</a></li><li><a href="https://towardsdatascience.com/demystified-wasserstein-gans-wgan-f835324899f4">Demystified: Wasserstein GANs (WGAN)</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6508f63a5553" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/principal-component-analysis-part-1-the-different-formulations-6508f63a5553">Principal Component Analysis Part 1: The Different Formulations.</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real-time Artwork Generation using Deep Learning]]></title>
            <link>https://medium.com/data-science/real-time-artwork-generation-using-deep-learning-a33a2084ae98?source=rss-82053676fe58------2</link>
            <guid isPermaLink="false">https://medium.com/p/a33a2084ae98</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Aadhithya Sankar]]></dc:creator>
            <pubDate>Sat, 25 Sep 2021 15:13:42 GMT</pubDate>
            <atom:updated>2021-09-25T20:44:52.149Z</atom:updated>
            <content:encoded><![CDATA[<h4>Adaptive Instance Normalisation(AdaIN) for Style Transfer between any arbitrary content-style image pair.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dn3YllcfXH-oSDdz90Ozww.png" /><figcaption>Artwork generated by AI. Image Source: [6]</figcaption></figure><p>In this post we will be looking into the paper “Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization”(AdaIN) by Huang et. al. We are looking at this paper because it had some key advantages over the other state-of-the-art methods at the time or release.</p><p>Most important of all, this method, once trained, can be used to transfer style between any arbitrary content-style image pair, even ones not seen during training. While the method proposed by Gatys et. al. can also transfer styles between any content-style image pair, it is drastically slow compared to this method as it iteratively optimises for the stylised image during inference. The AdaIN method is also flexible, it allows for control over the strength of the transferred style in the stylised image and also allows for extensions such as style interpolation and spatial controls.</p><p>Going forward, we will first familiarise ourselves with Instance normalisation, adaptive instance normalisation and then dive deep into the AdaIN paper’s working. Finally we will see some outputs and look at the code to implement Adaptive Instance Normalisation and to train the style transfer network.</p><h3>Instance Normalisation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N0uMSJp3_0X-De3qjvfdEg.png" /><figcaption>Fig. 1: Different Nmalisation Techniques. Image Source: [3].</figcaption></figure><p>Batch Normalisation normalises the feature map across the entire batch of fixed size using the mean and variance of the entire batch. Instance normalisation(a.k.a. Contrast Normalisation) on the other hand normalises each channel for each sample.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/338/1*6fgg6-Sg-QoBj6-tWorLdQ.png" /><figcaption>Eq. 1: Instance Normalisation</figcaption></figure><p>From Eq. 1, we can clearly see that each channel for each sample is normalised separately. Another difference to batch norm is that instance norm is also applied during inference unlike batch norm.</p><p>Instance norm normalises the contrast of the input image making the stylised image independent of the input image contrast, thus improving the output quality[2]. The authors of [1] also state that instance normalisation acts as a form of style normalisation by matching the feature statistics(mean and variance). Here the, style to which the images are normalised is defined by the learnable affine parameters γ and β[1].</p><p>To learn more about instance norm and other normalisation techniques, checkout <a href="https://medium.com/u/e1cd4f0da28f">Aakash Bindal</a>’s post <a href="https://medium.com/techspace-usict/normalization-techniques-in-deep-neural-networks-9121bf100d8">here</a>.</p><h4>Conditional Instance Normalisation</h4><p>Conditional Instance Normalisation was introduced in [4], where the authors proposed learning different γs and βs for each style image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/345/1*UWcV3ASGUyPQlV1SR9Dfeg.png" /><figcaption>Eq. 2: Conditional Instance Normalisation.</figcaption></figure><p>While this method works well it requires 2xCxS extra parameters (C: # channels, S: # styles), thus increasing the size of the network. Since Instance norm performs a form of style transfer[1], having a set of parameters for each style allows us to normalise images to each of these stlyes.</p><h3>Adaptive Instance Normalisation</h3><p>The authors extend the idea that affine parameters γ and β define the normalisation style in instance norm to propose Adaptive Instance Normalisation. In the proposal, Instance Normalisation is modified to adapt to different content-style image pairs.</p><p>AdaIN takes content features and style features <strong>x</strong> and <strong>y</strong> as input and simply matches the statistics(mean and variance) of content features to that of the style features <strong>y</strong>. There are no learnable parameters here and the transformation is obtained as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*f467KqXa9zl4kHIyY9xxng.png" /><figcaption>Eq. 3: Adaptive Instance Normalisation.</figcaption></figure><p>Where σ(⋅) is the standard deviation, σ²(⋅) the variance and μ(⋅), the mean, all computed along spatial dimension as in Instance Norm.</p><h3>AdaIN Style Transfer Network</h3><h4>Architecture</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*u0dYdsjzoFAMuj3QKDeitQ.png" /><figcaption>Fig.2: Proposed Style Transfer Network Architecture/ Image Source: [1]</figcaption></figure><p>The AdaIn StyleNet follows an Encoder-Decoder architecture(Fig. 2). The encoder <strong>f(⋅)</strong> is the first few pre-trained layers(up to relu4_1)of the VGG19 network. The encoder is fixed and not trained.</p><p>Adaptive Instance Normalisation is performed on the outputs of the encoder as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/397/1*2usNftwNEfYQTKa9dK4eCQ.png" /><figcaption>Eq. 4: AdaIN mixing.</figcaption></figure><p>where f(c) and f(s) are the feature maps produced by the encoder for the content and style images respectively. The mixing coefficient α∈[0,1] controls the strength of the style in the stylised image. α is set to 1 during training.</p><p>The decoder <strong>g(⋅)</strong> is the reverse of the encoder with the pooling replaced by 2x nearest neighbour upscaling. The decoder is initialised with random weights and its weights are learned. The stylised image is obtained by passing the transformed feature map <strong>t</strong> into the generator.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/179/1*q0d_mHqOBnSqpOquk8MNLQ.png" /><figcaption>Eq. 5: Stylised Image generation</figcaption></figure><h4>Loss Function</h4><p>The loss objective is a combination of style and content loss.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/180/1*ZJqClzUknwLkqLeiDYiUIQ.png" /><figcaption>Eq. 6: Total Loss</figcaption></figure><p>The content loss is the euclidean distance between the feature map of stylised image obtained from the VGG encoder and the AdaIN output <strong><em>t</em></strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/221/1*QjAcCDVVP2GLHbKJ9EBnqQ.png" /><figcaption>Eq. 7: Content Loss</figcaption></figure><p>AdaIN output <strong><em>t</em></strong> is used as the content target as it helps the model converge faster. The style loss is obtained as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*x1rcpRG3P1_QpQwVtMtB4w.png" /><figcaption>Eq. 8: Style Loss</figcaption></figure><p>Each <strong>ϕ</strong>i is a layer of the VGG 19 network. Here relu1_1, relu2_1, relu3_1 and relu4_1 are used.</p><h3>Output Samples</h3><p>Here are some artwork generated by the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pazQE1LoR9Jb7Cf0AgOdxw.png" /><figcaption>Style Transfer in action. Image Sources: (a)[7], (b)[8], (c )[6]</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kYkgaa4mrXjYkoBzhgxicg.png" /><figcaption>Style Transfer in action. Image Sources: (a)[9], (b)[10], (c )[6]</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UzjWhUzJjiT9oIPP54O69A.png" /><figcaption>Controlling the strength of stye on the Stylised image during inference using α. Image Source: [1]</figcaption></figure><h3>Code and Pre-trained Models</h3><p>Code Snippet to implement Adaptive Instance Normalisation:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7059fad3e400385b071508f2bfd74eca/href">https://medium.com/media/7059fad3e400385b071508f2bfd74eca/href</a></iframe><p>The code for training the AdaIN style transfer network can be found here:</p><p><a href="https://github.com/aadhithya/AdaIN-pytorch">GitHub - aadhithya/AdaIN-pytorch: PyTorch implementation of &quot;Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization&quot; by Xun Huang, Serge Belongie.</a></p><p>The weights for pre-trained models(both pytorch and onnx formats) can be found here:</p><p><a href="https://github.com/aadhithya/AdaIN-pytorch/releases/tag/1.0">Release AdaIN StyleNet Weights · aadhithya/AdaIN-pytorch</a></p><h3>Conclusion</h3><p>This post introduced Instance normalisation and how Adaptive instance normalisation can be used for style transfer between arbitrary content-style image pairs. By training a generator network and using Adaptive instance normalisation we are able to transfer style between any content-style image pair. We are also able to control the strength of the style on the generated image during runtime. Another advantage the method provides is the inference speed over the other models at the time. Finally, some outputs were shown and the code and pre-trained weights for inference made available for free, non-commercial use.</p><p>If you find any mistakes in the post, please leave a comment, I will fix them!</p><p>If you liked the post, please consider following the author, <a href="https://medium.com/u/82053676fe58">Aadhithya Sankar</a>.</p><h3>References</h3><p>[1] Huang, Xun, and Serge Belongie. “Arbitrary style transfer in real-time with adaptive instance normalization.” <em>Proceedings of the IEEE International Conference on Computer Vision</em>. 2017.</p><p>[2] D.Ulyanov, A.Vedaldi, and V.Lempitsky. Improved texture networks: Maximizing quality and diversity in feed-forward stylization and texture synthesis. In <em>CVPR</em>, 2017.</p><p>[3]Wu, Yuxin, and Kaiming He. “Group normalization.” <em>Proceedings of the European conference on computer vision (ECCV)</em>. 2018.</p><p>[4] V. Dumoulin, J. Shlens, and M. Kudlur. A learned representation for artistic style. In <em>ICLR</em>, 2017.</p><p>[5] Y. Li, N. Wang, J. Liu, and X. Hou. Demystifying neural style transfer. <em>arXiv preprint arXiv:1701.01036</em>, 2017.</p><p>[6] <a href="https://github.com/aadhithya/AdaIN-pytorch">https://github.com/aadhithya/AdaIN-pytorch</a></p><p>[7] <a href="https://en.wikipedia.org/wiki/File:Side_profile,_Brihadeeswara.jpg">https://en.wikipedia.org/wiki/File:Side_profile,_Brihadeeswara.jpg</a></p><p>[8]<a href="https://en.wikipedia.org/wiki/Olive_Trees_(Van_Gogh_series)#/media/File:Van_Gogh_The_Olive_Trees..jpg">https://en.wikipedia.org/wiki/Olive_Trees_(Van_Gogh_series)#/media/File:Van_Gogh_The_Olive_Trees..jpg</a></p><p>[9]<a href="https://en.wikipedia.org/wiki/Taj_Mahal#/media/File:Taj_Mahal_in_India_-_Kristian_Bertel.jpg">https://en.wikipedia.org/wiki/Taj_Mahal#/media/File:Taj_Mahal_in_India_-_Kristian_Bertel.jpg</a></p><p>[10]<a href="https://en.wikipedia.org/wiki/The_Starry_Night#/media/File:Van_Gogh_-_Starry_Night_-_Google_Art_Project.jpg">https://en.wikipedia.org/wiki/The_Starry_Night#/media/File:Van_Gogh_-_Starry_Night_-_Google_Art_Project.jpg</a></p><h3>Other Works by the Author</h3><p>If you liked this post, here are some other posts you might enjoy:</p><ul><li><a href="https://towardsdatascience.com/learning-disentangled-representations-with-invertible-flow-based-interpretation-networks-9954554a28d2">Learning Disentangled Representations with Invertible(Flow-based) Interpretation Networks</a></li><li><a href="https://towardsdatascience.com/demystified-wasserstein-gans-wgan-f835324899f4">Demystified: Wasserstein GANs (WGAN)</a></li><li><a href="https://towardsdatascience.com/a-primer-on-atrous-convolutions-and-depth-wise-separable-convolutions-443b106919f5">A Primer on Atrous Convolutions and Depth-wise Separable Convolutions</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a33a2084ae98" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/real-time-artwork-generation-using-deep-learning-a33a2084ae98">Real-time Artwork Generation using Deep Learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learning Disentangled Representations with Invertible(Flow-based) Interpretation Networks]]></title>
            <link>https://medium.com/data-science/learning-disentangled-representations-with-invertible-flow-based-interpretation-networks-9954554a28d2?source=rss-82053676fe58------2</link>
            <guid isPermaLink="false">https://medium.com/p/9954554a28d2</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Aadhithya Sankar]]></dc:creator>
            <pubDate>Sat, 18 Sep 2021 19:30:48 GMT</pubDate>
            <atom:updated>2021-09-25T21:16:00.455Z</atom:updated>
            <content:encoded><![CDATA[<h4>What are disentangled representations? How can we learn disentangled representations for any arbitrary model using flow-based generative models?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D9oLFmNzmqJHK4VZ5EOgcw.png" /><figcaption>Fig. 1: The IIN network can be applied to arbitrary existing models. IIN takes the representation z, learned by the arbitrary model and factorised it into smaller factors such that each factor learns to represent one generative concept. Image Source:[1].</figcaption></figure><p>Disentangled representations can be useful in tackling many downstream tasks and help improve robustness and generalisability of models. In this post, we will look into how we can learn disentangled representations from the representations learned by arbitrary pre-trained models using flow-based generative models. Specifically, we will be looking into the Invertible Interpretation Network(IIN) proposed in the paper “A Disentangling Invertible Interpretation Network for Explaining Latent Representations” by Esser et. al.[1]. We will see the idea behind IIN, how they work and what their uses are. We will also take a brief look into the results achieved by the paper.</p><h3>Disentangled Representations</h3><p>A disentangled representation is one where change in single latent units are sensitive to change in one generative factor while remaining invariant to changes in other factors[2]. In other words, given a disentangled representation, a change in a latent unit will result in a change in one generative factor and vice-versa.</p><p>Disentangled representations are useful in a variety of tasks like novel sample generation, novelty detection, learning compressed representations, etc. Good representations can help the robustness and generalisability of models. In addition to that, disentangled representations can also help in tasks that require knowledge transfer, in scenarios where learned representations can help models converge fast, like transfer learning or domain adaptation.</p><h3>Flow-based Generative Models</h3><p>When we look at generative models like VAEs and GAN, neither of them explicitly learn the real data distribution p(<strong>x</strong>) because p(<strong>x</strong>)=∫p(<strong>x</strong>|<strong>z</strong>)p(<strong>z</strong>)d<strong>z </strong>is often intractable as we cannot possibly integrate over all values of <strong>z ~ p(z)</strong>. Flow-based generative models, on the other hand, are able to overcome this issue by using normalising flows.</p><p><strong>Normalising Flow</strong>: A normalising flow transforms a simple distribution to a more complex one by applying a series of bijective transforms, one on top of another. The bijective functions also have an inverse, meaning we can compute both in the forward and reverse directions.</p><p>A flow-based generative model is just a series of normalising flows, one stacked on top of another. Since the transformation functions are reversible, a flow-based model is also reversible(<strong>x → z</strong> and <strong>z →x)</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*3GRq6GCnItJiAVzIIsHZyg.png" /><figcaption><strong>Eq. 1: A flow</strong></figcaption></figure><p>Here, <strong>z_0 </strong>is the initial distribution, <strong>x = z_k </strong>is the final distribution we want to learn and f_i(.) are a series of bijective transformation functions.</p><p>The Flow-based model allows us to directly Optimise for the log-likelihood of the data as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E7FVKJtTmX4wprpClJfZzA.png" /><figcaption><strong>Eq. 2: Log-likelihood calculation.</strong></figcaption></figure><p>Here, π_0(z_0) is the initial distribution. Generally, the transformations are chosen such that the log-determinant is easy to compute.</p><p>To learn more about flow-based generative models and how they work, check this amazing blog <a href="https://lilianweng.github.io/lil-log/2018/10/13/flow-based-deep-generative-models.html">post</a> out!</p><h3>Invertible Interpretation Networks For Learning Disentangled Representations</h3><p>The main idea behind the paper[1] is to learn a non-linear mapping that transforms the learned representations from any arbitrary pre-trained model into a space where the new representations can be factorised into factors such that the representation of each factor corresponds to one human-comprehensible semantic concept. To that end the authors propose a flow-based invertible network <strong>T </strong>that learns this mapping from the learned larent space to the new, disentangled space(<strong>Fig.1</strong>), i.e:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/184/1*6xUVJYeSeVCjtEzrkFFKxg.png" /></figure><p>And since <strong>T </strong>is invertible, the inverse is also possible:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/229/1*wdL4T7qsofXfyKshj_JOnQ.png" /></figure><p>To achieve a disentangled and interpretable space, the authors propose to factorise <strong>Z~(read Z tilde) </strong>to <strong>K+1 </strong>factors where each of the factor represent one interpretable concept.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*bqzk_4T1D9X8z5UDEj_JAQ.png" /><figcaption><strong>Factorisation of z~. z~ is factorised to k factors where each z~_k is of N_k dimensions. N_k for all k sums up to N.</strong></figcaption></figure><p>Using Flow-based invertible network makes sense here because we are able to learn the mapping from T:<strong>z →z~</strong>, but also <strong>z~ → z</strong>. This means, the invertible model <strong>T</strong> should also allow us to meaningfully modify the original latent representation <strong>z</strong> by modifying individual factors <strong>z~_k</strong> in the disentangled space.</p><p>Achieving disentanglement of the factors <strong>z~_k </strong>means the distributions of each <strong>z~_k</strong> is independent of each other for all k and the joint distribution over all <strong>z~_k </strong>can be written as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*6qm5BcqQkz8OtF6zqiDbQQ.png" /><figcaption>Eq. 2: Factorisation of p(z~).</figcaption></figure><p>Each factor z~_k is modelled as a unit gaussian and therefore the joint distribution becomes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/366/1*gcugZSaux-PHbGrOOU92Hg.png" /><figcaption>Eq. 3: Joint distribution p(z~)</figcaption></figure><p>In order to have disentangled factors, we need to satisfy 2 conditions:<br><strong>i)</strong> each z~_k should be sensitive to only one interpretable concept<br><strong>ii)</strong> z~_k should be invariant to all other concepts that it doesn’t represent.</p><p>This is achieved by looking at pairs of images (<strong>x</strong>_a, <strong>x_b</strong>). Each semantic concept <strong>F ∈ {1, …, K} </strong>is represented by <strong>z~_F</strong>. Each <strong>z~_F </strong>is trained using image pairs (<strong>x</strong>_a, <strong>x</strong>_b) drawn from p(<strong>x</strong>_a, <strong>x</strong>_b|F). The representation of all other semantic concepts apart from <strong>F ∈ {1, …, K}, </strong>are learned by the residual factor <strong>z~_0.</strong></p><p>Given a pair of images sharing the same concept <strong>F</strong> and their representations <strong>z~^a</strong> and <strong>z~^b </strong>(representations of the images from the Invertible model), we need ensure <br><strong>i) </strong>the representations for the F-th factor for both the images are similar. To do this the F-th factor of image b as a gaussian centered around the F-th factor of image a:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/1*J9p_rbStJ2rNvIwvQyemdg.png" /><figcaption><strong>Eq. 4</strong></figcaption></figure><p>Positive correlation factor σ_ab ∈ (0, 1) controls how close/similar <strong>z~^a_F </strong>and <strong>z~^b_F </strong>are.</p><p><strong>ii) </strong>the representations of all other factors are invariant(no correlation) and are modelled as unit gaussians:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*CKa5Fm0RI5Soko5jUiyPWw.png" /><figcaption><strong>Eq. 5</strong></figcaption></figure><p>To train the Flow model, we maximise the likelihood of the original representations of the image pairs (<strong>z</strong>_a, <strong>z</strong>_b). The likelihood for the pair is computed as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*AmaY7RMcFzjssWSPPI5oPQ.png" /><figcaption><strong>Eq. 6</strong></figcaption></figure><p>Taking log and applying Eqs. 2, 4, and 5 to Eq.6, we get the loss function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nrnxuFlY1MPZOFECXFCDsg.png" /><figcaption><strong>Eq. 7: Loss for one image pair</strong></figcaption></figure><p>where,</p><p>the terms in the red box correspond to the log likelihood for all factors of p(z^a),<br>the terms in the blue box correspond to the log likelihood for all factors except for p(z^b),<br>the terms in the green box correspond to the log likelihood of the factor F for image b and,</p><p>T(<strong>z</strong>^j_k) = z~_k</p><p>The total loss is the loss for image pairs over all the factors F:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/1*Z19h4jZ3tmX59COJuznGbg.png" /><figcaption><strong>Eq. 8: Total Loss Function</strong></figcaption></figure><h4>Estimating The Dimensions For Each Factor</h4><p>In the IIN model we factorise the latent <strong>z~ </strong>to K+1 factors(residual + K factors) where each of the K factors can have different dimensionalities subject to the condition that the sum of all dimensionalities equals that of <strong>z~</strong>.</p><p>The dimensionality of each factor is estimated by computing the scores for each factor. The score foe a factor F is calculated as as the sum of correlations of original representations between all pairs of images (<strong>z</strong>^a, <strong>z</strong>^b) ~ p(<strong>z</strong>^a, <strong>z</strong>^b|F), i.e.,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/1*Sd_bEFNy_sV9408RrROLEA.png" /><figcaption><strong>Eq. 9: Score for each factor.</strong></figcaption></figure><p>The residual factor is given the score <strong>N</strong>(dimensions of <strong>z</strong>). Then the number of dimensions for each factor is estimated as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/288/1*mMWGHbI81SAbPrZgCx3ljw.png" /><figcaption><strong>Eq. 10: Estimated dimension of each factor.</strong></figcaption></figure><h3>Results</h3><h4>Swapping Disentangled Factors</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/1*Nv2jzgKdXY-fwZR4bI1GLg.png" /><figcaption>Fig. 2: combining residual(<strong>z~</strong>_0) of target image(left most coulmn) with animal class factor(<strong>z~</strong>_1) of target image(top most row). Image Source:[1].</figcaption></figure><p>In Fig. 2, combining the residual(<strong>z~</strong>_0) of target image(left most coulmn) with animal class factor(<strong>z~</strong>_1) of target image(top most row) results in transfer of animal type in the resulting image while mainting the pose of the source image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/469/1*Kw0Bv5Qs0MreChN-h1Q8SA.png" /><figcaption>Fig. 3: interpolating along the smile factor along the semantic axis F → F~. Image Source:[1]</figcaption></figure><p>On interpolating along the semantic axis for smiling on the CelebA dataset, it can be seen the model is able to control the amount of smiling while keeping the other factors relatively constant. More information on how the interpolation is performed can be found in [1].</p><h3>Code</h3><p>The code for the paper can be found <a href="https://github.com/CompVis/iin">here</a>.</p><h3>Conclusion</h3><p>In this article, we discussed what disentangled representations are and their uses, then briefly looked into how flow-based generative models worked. We also dove deep into the Invertible Interpretation Network. The IIN is a powerful network can be used on any pretrained network to learn a mapping from the original representations to disentangled representations. Taking advantage of the invertibility of flow-based models, the model can also be used to arbitrarily modify disentangled factors <strong>z~</strong>_k to that effect semantically meaningful changes in the original representation.</p><p>P.S: Please leave a comment if you find any mistakes/issues, I will be happy to fix them! :)</p><h3>References</h3><p>[1] Esser, Patrick, Robin Rombach, and Bjorn Ommer. “A disentangling invertible interpretation network for explaining latent representations.” <em>Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition</em>. 2020.</p><p>[2] Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. In <em>IEEE Transactions on Pattern Analysis &amp; Machine Intelligence</em>, 2013.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9954554a28d2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/learning-disentangled-representations-with-invertible-flow-based-interpretation-networks-9954554a28d2">Learning Disentangled Representations with Invertible(Flow-based) Interpretation Networks</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Demystified: Wasserstein GANs (WGAN)]]></title>
            <link>https://medium.com/data-science/demystified-wasserstein-gans-wgan-f835324899f4?source=rss-82053676fe58------2</link>
            <guid isPermaLink="false">https://medium.com/p/f835324899f4</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[generative-adversarial]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Aadhithya Sankar]]></dc:creator>
            <pubDate>Fri, 17 Sep 2021 13:17:08 GMT</pubDate>
            <atom:updated>2021-10-03T13:54:34.455Z</atom:updated>
            <content:encoded><![CDATA[<h4>What is the Wasserstein distance? What is the intuition behind using Wasserstein distance to train GANs? How is it implemented?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E7eio04uWQyuP_IHnK-qpA.png" /><figcaption><strong>Fig. 1: Optimal discriminator and critic when learning to differentiate two Gaussians[1].</strong></figcaption></figure><p>In this article we will read about Wasserstein GANs. Specifically we will focus on the following: i) What is Wasserstein distance?, ii) Why use it? iii) How do we use it to train GANs?</p><h3>Wasserstein Distance</h3><p>The Wasserstein distance (Earth Mover’s distance) is a distance metric between two probability distributions on a given metric space. Intuitively, it can be seen as the minimum work needed to transform one distribution to another, where work is defined as the product of mass of the distribution that has to be moved and the distance to be moved. Mathematically, it is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bo6JYKzi6h2vNW3FxN56Rw.png" /><figcaption><strong>Eq. 1: Wasserstein Distance between distributions P_r and P_g.</strong></figcaption></figure><p>In Eq. 1, Π(P_r ,P_g ) is the set of all joint distributions over <strong>x</strong> and <strong>y </strong>such that the marginal distributions are equal to P_r and P_g. γ(x, y) can be seen as the amount of mass that must be moved from x to y to transform P_r to P_g[1]. The Wasserstein distance is then the cost of the optimal transport plan.</p><h3>Wasserstein Distance vs. Jensen-Shannon Divergence</h3><p>The original GAN objective is shown to be the minimisation of the Jensen-Shannon Divergence[2]. The JS Divergence is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IyGiXiOGC78Djdwvn_YpNA.png" /><figcaption><strong>Eq. 2: JS Divergence between P_r and P_g. P_m = (P_r + P_g)/2</strong></figcaption></figure><p>Compared to JS, Wasserstein distance has the following advantages:</p><ul><li>Wasserstein Distance is continuous and almost differentiable everywhere, which allows us to train the model to optimality.</li><li>JS Divergence locally saturates as the discriminator gets better, thus the gradients becomes zero and vanishes.</li><li>Wasserstein distance is a meaningful metric, i.e, it converges to 0 as the distributions get close to each other and diverges as they get farther away.</li><li>Wasserstein Distance as objective function is more stable than using JS divergence. The mode collapse problem is also mitigated when using Wasserstein distance as the objective function.</li></ul><p>From Fig. 1 we see clearly that the the optimal GAN discriminator saturates and results in vanishing gradients while the WGAN critic optimising the Wasserstein distance has stable gradients throughout.</p><p>For mathematical proof and a more detailed look into this topic, check the paper out <a href="https://arxiv.org/pdf/1701.07875.pdf">here</a>!</p><h3>Wasserstein GAN</h3><p>Now that it can be clearly seen that optimising for Wasserstein Distance makes more sense than optimising JS Divergence, it is also to be noted that the Wasserstein Distance defined in Eq.1 is highly intractable[3] as we cannot possibly compute the infimum(greatest lower bound) over all γ ∈Π(Pr ,Pg ). However, from the Kantorovich-Rubinstein duality we have,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-ERuZ0kLek8Q4VMgyLFohw.png" /><figcaption><strong>Eq. 3: Wasserstein Distance under 1-Lipschitz condition.</strong></figcaption></figure><p>here we have W(P_r, P_g) as the supremum(Lowest upper bound) over all 1-Lipschitz functions f: X → R.</p><p><strong>K-Lipschitz continuity: </strong>given 2 metric spaces (X, d_X) and (Y, d_Y), the transformation function f: X → Y is K-Lipschitz Continuous if</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wNKZyfomrjejuhKSpS9iAQ.png" /><figcaption><strong>Eq. 3: K-Lipschitz Continuity.</strong></figcaption></figure><p>where d_X and d_Y are distance functions in their respective metric spaces. When a function is K-Lipschitz, from Eq. 2, we end up with K ∙ W(P_r, P_g).</p><p>Now, if we have a family of parameterised functions {f_w} where w∈W that are K-Lipschitz continuous, we can have</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*td_Cl8eQWE6CuFGeUirTGA.png" /><figcaption><strong>Eq. 4</strong></figcaption></figure><p>i.e., the w∈W maximises Eq. 4 gives the Wasserstein distance multiplied by a constant.</p><h4>WGAN Critic</h4><p>To this effect the WGAN introduces a critic instead of the discriminator we’ve come to know with GANs. The critic network is similar in design to a discriminator network, but predicts the Wasserstein distance by optimising to find <strong>w* </strong>that will maximise Eq 4. To that end, the objective function of the critic is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H4BHrbDUL0USZ6giQ1yvWQ.png" /><figcaption><strong>Eq. 5: Critic Objective Function.</strong></figcaption></figure><p>Here, to enforce Lipschitz continuity on the function f, the authors resort to restricting the weights w to a compact space. This is done by clamping the weights to a small range([-1e-2, 1e-2] in the paper[1]).</p><p>The difference between the discriminator and the critic is that the discriminator is trained to correctly identify the samples from P_r from samples from P_g, the critic estimates the Wasserstein distance between P_r and P_g.</p><p>Here is the python code to train the Critic.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5e87aa33f3bb3819b63542ec21476e9b/href">https://medium.com/media/5e87aa33f3bb3819b63542ec21476e9b/href</a></iframe><h4><strong>WGAN Generator Objective</strong></h4><p>Naturally, the objective of the generator is to minimise the Wasserstein distance between P_r and P_g. The generator tries to find θ<strong>* </strong>that minimizes the Wasserstein distance between P_g and P_r. To that end, the objective function of the generator is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ii6g9bJyd6EPGTk-uY4_qA.png" /><figcaption><strong>Eq. 6: Generator Objective Function.</strong></figcaption></figure><p>Here, again the main difference between a WGAN generator and the standard generator is that the WGAN generator tries to minimise the Wasserstein distance between P_r and P_g while the standard generator tries to fool the discriminator with the generated images.</p><p>Here is the python code to train the generator:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f73c719f49292054de44c9bdce347050/href">https://medium.com/media/f73c719f49292054de44c9bdce347050/href</a></iframe><h3>Training Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*LfS8fAE_s6XYl4DCtvDTaQ.png" /><figcaption>Fig. 2: Early results from WGAN training[3].</figcaption></figure><p>Fig.2 shows some early results from training the WGAN. <strong>Please note that the images in Fig. 2 are early results, the training was stopped as soon as it was confirmed that the model was training as expected.</strong></p><h3>Code</h3><p>The complete implementation of the Wasserstein GAN can be found <a href="https://github.com/aadhithya/gan-zoo-pytorch"><strong>here</strong></a><strong>[3].</strong></p><h3>Conclusion</h3><p>WGANs offer much stable training and a meaningful training objective. This article introduced and gave an intuitive explanation about what Wasserstein Distance is, the advantages Wasserstein Distance has over Jensen-Shannon divergence used by the standard GAN and how Wasserstein distance is used to train the WGAN. We also saw code snippets to train the Critic and generator and ample outputs from early satges of training the model. Although WGANs have many advantages over the standard GAN, the authors of the WGAN paper clearly acknowledge that weight clipping is not the optimal way to enforce Lipschitz Continuity[1]. To fix this they propose the Wasserstein GAN with Gradient Penalty[4], which we will discuss in a later post.</p><p>If you liked this check the <a href="https://asankar96.medium.com/demystified-wasserstein-gan-with-gradient-penalty-ba5e9b905ead">next post</a> in this series which talks about <a href="https://asankar96.medium.com/demystified-wasserstein-gan-with-gradient-penalty-ba5e9b905ead">WGAN-GP</a>!</p><h3>References</h3><p>[1] Arjovsky, Martin, Soumith Chintala, and Léon Bottou. “Wasserstein generative adversarial networks.” <em>International conference on machine learning</em>. PMLR, 2017.</p><p>[2] Goodfellow, Ian, et al. “Generative adversarial networks.” <em>Communications of the ACM</em> 63.11 (2020): 139–144.</p><p>[3] gan-zoo-pytorch (<a href="https://github.com/aadhithya/gan-zoo-pytorch">https://github.com/aadhithya/gan-zoo-pytorch</a>).</p><p>[4] Gulrajani, Ishaan, et al. “Improved training of wasserstein gans.” <em>arXiv preprint arXiv:1704.00028</em> (2017).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f835324899f4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/demystified-wasserstein-gans-wgan-f835324899f4">Demystified: Wasserstein GANs (WGAN)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Primer on Atrous Convolutions and Depth-wise Separable Convolutions]]></title>
            <link>https://medium.com/data-science/a-primer-on-atrous-convolutions-and-depth-wise-separable-convolutions-443b106919f5?source=rss-82053676fe58------2</link>
            <guid isPermaLink="false">https://medium.com/p/443b106919f5</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[thoughts-and-theory]]></category>
            <dc:creator><![CDATA[Aadhithya Sankar]]></dc:creator>
            <pubDate>Wed, 15 Sep 2021 14:35:10 GMT</pubDate>
            <atom:updated>2021-09-20T16:57:53.266Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/thoughts-and-theory">Thoughts and Theory</a></h4><h3>A Primer on Atrous(Dilated) and Depth-wise Separable Convolutions</h3><h4>What are atrous/dilated and depth-wise separable convolutions? How are the different from standard convolutions? What are their uses?</h4><p>With properties such as weight sharing and translation invariance, Convolutional layers and CNNs have become ubiquitous in Computer Vision and Image Processing tasks using deep learning methods. With that in mind, this article aims at discussing some of the developments we’ve seen in convolutional networks. Specifically we focus on <strong>two </strong>developments: Atrous(Dilated) convolutions and Depth-wise Spearable convolutions. We will see how these two types of Convolutions work, how they are different from normal convolutions and why we may want to use them.</p><h3>Convolutional Layer</h3><p>Before we get into the topic, let’s quickly remind ourselves how convolutional layer works. At its core, convolutional filters are simply feature extractors. What were hand crafted feature filters before are now learned through the “magic” of back-propagation. We have a kernel(weights of the conv layer) that is slid over the input feature map and at each location, element-wise multiplication followed by a summation of theproducts is performed to obtain a scalar value. The same operation is performed at each location. Fig. 1 shows this in action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/244/1*5ks4cemhWo6bO4ATrxkRyg.gif" /><figcaption>Fig. 1: A 3x3 Convolution filter in action[1].</figcaption></figure><p>The convolutional filter detects a particular feature by <strong>sliding</strong> over the input feature map, i.e, it looks for that feature at each location. This intuitively explains the translation invariance property of Convolutions.</p><h3>Atrous(Dilated) Convolution</h3><p>To understand how atrous convolution differs from the standard convolution, we firs need to know what <strong>receptive field</strong> is. Receptive Field is defined as the size of the region of the input feature map that produces each output element. In the case of Fig.1, the receptive field is 3x3 as each element in the output feature map sees(uses) 3x3 input elements.</p><p>Deep CNNs use a combination of Convolutions and max-pooling. This has the disadvantage that, at each step, the spatial resolution of the feature map is halved. Implanting the resultant feature map onto the original image results in sparse feature extraction. This effect can be seen in Fig. 2. The conv. filter downsamples the input image by a factor of two. Upsampling and imposing the feature map on the image shows that the responses correspond to only 1/4th of the image locations(Sparse feature extraction).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*dxK0C3WBBqk_eF0k8KRScQ.png" /><figcaption>Fig. 2: Sparse Feature Extraction in DCNN[2].</figcaption></figure><p>Atrous(Dilated) convolution fixes this problem and allows for dense feature extraction. This is achieved a new parameter called <strong>rate(r)</strong>. Put simply, atrous convolution is akin to the standard convolution except that the weights of an atrous convolution kernel are spaced <strong>r </strong>locations apart, i.e., the kernel of dilated convolution layers are sparse.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*xBz2R6qoArKjkthYzKLZMQ.png" /><figcaption>Fig. 3: Standard vs Dilated Kernel.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*SVkgHoFoiMZkjy54zM_SUw.gif" /><figcaption>Fig. 4: 3x3 Atrous(dilated) Convolution in action[1].</figcaption></figure><p>Fig. 3(a) shows a standard kernel and Fig. 3(b) a Dilated 3x3 kernel with a <strong>rate r = 2</strong>. By controlling the rate parameter, we can arbitrarily control the receptive fields of the conv. layer. This allows the conv. filter to look at larger areas of the input(receptive field) without a decrease in the spatial resolution or increase in the kernel size. Fig. 4 shows a dilated convolutional filter in action.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/1*zKvtCFhcHpMCQhhjanq12w.png" /><figcaption>Fig. 5. Dense Feature extraction using dilated convolutions[2].</figcaption></figure><p>Compared to standard convolution used in Fig. 2, it can be seen in Fig. 5 that dense features are extracted by using a dilated kernel with rate r=2. Dilated convolutions can be trivially implemented by just setting the <strong>dilation </strong>parameter to the required dilation rate.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7138530f77462cc757693e89c8051ad2/href">https://medium.com/media/7138530f77462cc757693e89c8051ad2/href</a></iframe><h3>Depth-wise Separable Convolution</h3><p>Depth-wise separable convolution was introduced in Xception net[3]. Fig.6 shows a standard convolution operation where the convolution acts on all channels. For the configuration shown in Fig. 6, we have 256 5x5x3 kernels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/912/1*XHQj7F9r6iynLb7QWSreYw.png" /><figcaption>Fig. 6: Standard 5x5 convolution applied to a 12x12x3 input.</figcaption></figure><p>Fig. 7(a) shows depth-wise convolution where the filters are applied to each channel. This is what differentiates a Depth-wise separable convolution from a standard convolution. The output of the depth-wise convolution has the same channels as the input. For the configuration shown in Fig. 7(a), we have 3 5x5x1 kernels, one for each channel. Inter-channel mixing is achieved by convolving the output of depth-wise convolution with a 1x1 kernel of required number of output channels (Fig. 7(b)).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*UXHwtvGpSS34CK6I1Zu3Hg.png" /><figcaption>Fig. 7: 5x5 Depth-wise Separable convolution followed by 1x1 conv.</figcaption></figure><h4>Why choose Depth-wise Separable Convolution?</h4><p>To answer this we take a look at the number of multiplications required to perform a standard convolution and a depth-wise separable convolution.</p><p><strong>Standard Convolution<br></strong>For the configuration specified in Fig. 6, we have 256 kernels of size 5x5x3. The total multiplications required to compute the convolution:<br><strong>256*5*5*3*(8*8 locations) = 1228800</strong></p><p><strong>Depth-wise Separable Convolution<br></strong>For the configuration specified in Fig. 7, we have 2 convolution operations:<br>1) 3 kernels of size 5x5x1. Here, the number of multiplications required is: <strong>5*5*3*(8*8 locations) = 4800<br></strong>2) 256 kernels of size 1x1x3 for the 1x1 convolution. The number of multiplications required: <strong>256*1*1*3*(8*8 locations) = 49152<br></strong>Total multiplications required for Depth-wise separable convolutions:<strong> 4800 + 49512 = 54312.</strong></p><p>We can quite clearly see that the <strong>depth-wise convolutions require much less computations than the standard convolution</strong>.</p><p>In pytorch, depth-wise separable convolutions can be implemented by setting the group parameter to the number of input channels.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b1859e22c99e9da5c9dca05987db4b7f/href">https://medium.com/media/b1859e22c99e9da5c9dca05987db4b7f/href</a></iframe><p><strong>Note: </strong>The <strong>groups</strong> parameter in pytorch has to be a multiple of the <strong>in_channels </strong>parameter. This is because in pytorch, the depth-wise convolution is applied by dividing the input features into <strong>groups=g </strong>groups. More info <a href="https://pytorch.org/docs/master/generated/torch.nn.Conv2d.html#torch.nn.Conv2d">here</a>.</p><h3>Conclusion</h3><p>This post delved into two popular types of convolution: atrous(dilated) convolution and depth-wise separable convolutions. We saw what they were, how they were different from the standard convolution operation and also saw the advantages they posed over the standard convolution operation. Finally we also saw how the atrous(dilated) and depth-wise separable convolution can be implemented using pyTorch.</p><h3><strong>References</strong></h3><p>[1] Convolution arithmetic (<a href="https://github.com/vdumoulin/conv_arithmetic">https://github.com/vdumoulin/conv_arithmetic</a>)</p><p>[2] Chen, Liang-Chieh, et al. “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.” <em>IEEE transactions on pattern analysis and machine intelligence</em> 40.4 (2017): 834–848.</p><p>[3] Chollet, François. “Xception: Deep learning with depthwise separable convolutions.” <em>Proceedings of the IEEE conference on computer vision and pattern recognition</em>. 2017.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=443b106919f5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-primer-on-atrous-convolutions-and-depth-wise-separable-convolutions-443b106919f5">A Primer on Atrous Convolutions and Depth-wise Separable Convolutions</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>