<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nolan Kent on Medium]]></title>
        <description><![CDATA[Stories by Nolan Kent on Medium]]></description>
        <link>https://medium.com/@nkent?source=rss-352fdc0a3e4c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*h05QQ7H5q0dYwBVSxIWh9g.png</url>
            <title>Stories by Nolan Kent on Medium</title>
            <link>https://medium.com/@nkent?source=rss-352fdc0a3e4c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 04:28:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nkent/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Malware Analysis with Visual Pattern Recognition]]></title>
            <link>https://medium.com/data-science/malware-analysis-with-visual-pattern-recognition-5a4d087c9d26?source=rss-352fdc0a3e4c------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a4d087c9d26</guid>
            <category><![CDATA[computer-security]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[malware]]></category>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <dc:creator><![CDATA[Nolan Kent]]></dc:creator>
            <pubDate>Wed, 27 May 2020 14:51:42 GMT</pubDate>
            <atom:updated>2020-05-27T17:44:19.388Z</atom:updated>
            <content:encoded><![CDATA[<h4>The secret to quickly reverse-engineering binary files</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GdCEaV0CR3-YJ0dk1wdu3A.png" /><figcaption>32-bit PE file displayed in Hiew text mode</figcaption></figure><h3>Part 1: Introduction and Basic Patterns</h3><p>I originally wrote this article for the benefit of fellow malware analysts when I was on Symantec’s Security Response team, analyzing and classifying 20+ files per day. “Extended ASCII analysis” is a technique for quickly gaining a high-level understanding of a file through pattern recognition. The technique is very beneficial for analysts who can’t spend a lot of time on each file. For many types of malware, it can allow accurate classification in seconds. The technique involves training your visual system to recognize patterns unique to raw binary files. This article is meant to serve as an introduction and reference, but it may require examining hundreds or thousands of files to become comfortable with the technique. It is also meant to serve as an intuitive justification for the use of convolutional neural networks in malware classification, which often perform well on the same tasks as the human visual system. I think this type of domain knowledge is essential for building models that are robust to changes in input distribution (more about that in <a href="https://towardsdatascience.com/covariate-shift-in-malware-classification-77a523dbd701">https://towardsdatascience.com/covariate-shift-in-malware-classification-77a523dbd701</a>). I wrote the original version of this article in 2014, so some of the specific malware screenshots may be out of date, but the basic concepts remain applicable.</p><p>Some of the article assumes familiarity with windows binary reverse-engineering, particularly experience with <a href="https://en.wikipedia.org/wiki/Portable_Executable">Portable Executable</a> (PE) files: the file format of windows .exe and .dll files. The technique can provide insight into other file formats as well, but given the quantity of windows executable malware (and benign files), they are a good starting point. However, even if you aren’t familiar with reverse-engineering, I hope this article can demonstrate the value of training/applying your visual system on other types of raw data. Raw data may only be difficult to interpret because you haven’t looked at enough of it to pick up on the patterns: once you start to recognize patterns, you may notice things that get left out of more processed visualizations.</p><h3>The Problem</h3><p>There are a lot of binary analysis tools that display specific information about parts of a file. For example, <a href="http://www.angusj.com/resourcehacker/">Resource Hacker</a> shows resources in a PE file, string dumpers show strings, <a href="https://www.aldeid.com/wiki/PEiD">PEID</a> shows PE/packer information, etc. Running a large number of tools and combining their outputs to form a conclusion can take quite a bit of time. Because most of these tools provide a processed representation that focuses on particular aspects of a binary file, some bytes may go completely unnoticed. Most tools are also dependent on file type, so relying on them means it’s challenging to analyze unfamiliar formats.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tChZA1iZczJ_gxW-nm_AAg.png" /><figcaption>Different tools give different views into a file, but don’t generally offer an overview that is both comprehensive and quick to digest</figcaption></figure><p>While <a href="https://www.hex-rays.com/products/ida/">IDA</a> — the most popular interactive disassembler/decompiler in the industry — allows for a fairly complete analysis of most executable files, it can take time to identify encrypted/compressed areas of the file. Because IDA focuses on analyzing code, identifying encrypted data often involves identifying the decryption code. Again, this can be time-consuming, and identifying encrypted data is essential for malicious files. Most common malware are packed using low entropy custom packers designed to avoid antivirus software. Tools like PEID that identify packed executables don’t work consistently because many packer checks are based on either known packers or file entropy. PEID also can’t tie a specific packer to a specific threat: if it could, then it would make a great antivirus engine on its own. A faster way to get information about all the bytes in a file would be ideal for forming first impressions.</p><h3>Solution</h3><p>Scrolling through <a href="http://www.hiew.ru/">Hiew’s</a> TEXT view (or opening the file in a hex editor, or even just notepad.exe) shows all the data within a binary file, with each byte represented by an extended ASCII symbol. Different patterns of data result in different patterns of symbols. Because the human brain is very good at identifying visual patterns, it can categorize sections of data very quickly. Common types of data include:</p><ul><li>native code (x86, x64, ARM, etc.)</li><li>Microsoft Intermediate Language (MSIL)</li><li>poorly encrypted data</li><li>high entropy data (compressed/well encrypted/random data)</li><li>images</li><li>relocation sections</li><li>strings (including non-roman alphabets).</li></ul><p>Identifying poorly encrypted data can be very helpful because that usually indicates a custom packer used for avoiding antiviruses rather than a commercial packer that might be used for compression and protection.</p><p>For example, many Trojan.Asprox.B samples (see image below) are encrypted in a way that results in a very consistent pattern of bytes. This means it’s possible to identify them in less than a second.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3nR6w9wlyZ9r7UnEjAElmA.png" /><figcaption>Encrypted section of Trojan.Asprox.B: even though the data is encrypted, many repeating patterns are visible (indicating encryption is weak)</figcaption></figure><p>While this technically gives information about (nearly) every byte in the file, it is not best for all types of analysis.</p><h3>Limitations</h3><p>This method focuses on a broad understanding rather than a deep understanding:</p><ul><li>It can be used to recognize code, but understanding the code still requires a disassembler/decompiler</li><li>It can be used to identify images embedded in a PE file, but seeing the actual image still requires an image viewer program</li><li>It can be used to recognize encrypted/compressed data, but you must decrypt/decompress the data before it can be analyzed further</li></ul><p>Viewing the data as symbols instead of hexadecimal numbers makes human pattern recognition much faster and easier.</p><h3>ASCII Background</h3><p>The key to this method is mapping each byte in a file to one of 256 different symbols, and then using the brain’s pattern recognition abilities to interpret the result. Formal ASCII only uses 7 bits: 0x00 to 0x7F. This includes only half of all possible symbols — most of them “printable” symbols such as alphanumeric characters. To use this technique, we need a way to print a symbol for every byte, however different computer systems handled the 8th bit differently. Windows refers to different character mappings as ‘code pages’, the two common groups of code pages are ANSI and OEM.</p><p>ANSI refers to Windows code pages; the default is called “Windows-1252”. This code page doesn’t map every single byte: 0x81,0x8c,0x8f,0x90,0x9D don’t map to any symbol and are usually displayed as a ‘.’ or ‘?’. They are, therefore, impossible to distinguish, which hides information.</p><p>OEM refers to console/DOS applications. The code page from the IBM PC is called code page 437. It maps all bytes to symbols; however, it maps 0x00,0xFF, and 0x20 to the same symbol — a blank space. This rarely causes problems in practice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*ByfujIUQwl04d4GWQMjrOg.png" /><figcaption>Mapping of every byte to a symbol</figcaption></figure><p>You can use this technique with a standard text editor, such as notepad. However, this creates issues with some control characters and lacks features. I personally use the hex editor “Hiew”.</p><h3>Patterns and High Entropy Data Background</h3><p>As previously stated, quickly identifying patterns is the primary purpose of this technique. The ability to recognize benign patterns to focus on potentially malicious patterns is crucial for malware analysis. One simplified approach to thinking about patterns more quantitatively is counting how often certain characters and sequences of characters show up in a chunk of data. Usually, some characters/sequences are prevalent, while others are rare. The exception is high entropy data, where most characters appear a roughly equal number of times. Note in the security industry, “entropy” is usually calculated as the entropy of a categorical distribution where each byte is its own category, and where the probability of a byte is given by its frequency in the chunk of data we’re considering. As this is a univariate distribution, it does not capture structure related to relative positions of bytes. However, here I’ll use the term ‘high entropy’ to refer to data that appears uniformly random, even when it comes to sequences of bytes. The image below is an example of high entropy data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gUFFxM8ECPgB6fMZS6c1IQ.png" /><figcaption>High entropy data</figcaption></figure><p>High entropy data usually indicates one of four things:</p><ul><li>compressed data</li><li>well-encrypted data</li><li>random data</li><li>constants used in cryptographic calculations.</li></ul><p>In a perfectly high entropy file, each byte will, on average, appear once in every 256 bytes or around .39% of the time. The following image shows the frequency of the most common characters and sequences of 2 characters in a high entropy file (in this case, a chunk of a .zip file). While none of the characters in the top 20 most frequent appear .39% of the time, they are all reasonably close. The difference could be accounted for by sampling error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/85/1*OFm44nUzLxHug_Z_IfY01g.png" /><figcaption>How often the most common unigrams and bigrams appear in a sample of high entropy data</figcaption></figure><p>In the section, I discuss how to recognize patterns produced by standard compiled code. This is obviously a common pattern in executable files.</p><h3>x86 (32-bit) Code</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*EmquD6EGzgIGDac18xTJpg.png" /><figcaption>Example of x86 code</figcaption></figure><p>I generated the following image from around 7kb of x86 code. It shows some of the most common characters/sequences in code, and (very) roughly their corresponding opcodes. This example doesn’t represent all possible x86 code, as different compilers/source code generates different looking compiled code. For example, some compilers/compiler options align functions on 16-byte boundaries, some on 4, some don’t align. Some compilers use nops to pad, and some use int3. Obviously different source files will result in different compiled files. However, after training your visual system, it’s usually easy to recognize all types of x86 code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*qAVRgTn3w33oD58GpFg-mw.png" /><figcaption>How the most common sequences in x86 code map to operations</figcaption></figure><p>A few things to note: 0x00 and 0xFF are by far the most common, and usually present as offsets (for example, an offset used in a jmp instruction). These bytes map to blank spaces and produce visible gaps in x86 code. In this code, the compiler used the int3 (CC) instruction as padding between functions. Sequences involved in function beginning/ends are prevalent (padding, push ebp, mov ebp esp for beginning, ret, padding for end). After some practice locating sequences like these, it becomes much easier to recognize x86 data instantly.</p><h3>Conclusion</h3><p>Unfortunately, becoming comfortable with this technique may require examining hundreds or thousands of malicious and benign samples. If you regularly reverse engineer files, I would highly recommend trying this out, even if initially it takes a while to gain useful information with it. Malicious packers that use weak encryption have been one of the main tools used by malware authors to hide their attacks for a long time, but modern techniques such as emulation can help uncover these attacks by unpacking the payload (which is much easier to identify than the packed file). Understanding file structure can also help inspire and guide malware detection models such as neural networks <a href="https://medium.com/ai-ml-at-symantec/deep-learning-for-malware-classification-dc9d7712528f">https://medium.com/ai-ml-at-symantec/deep-learning-for-malware-classification-dc9d7712528f</a> for malware detection.</p><h3>Part 2: Pattern Atlas</h3><p>In this part, I go over a large number of patterns that I’ve encountered frequently. This can help serve as a reference if you want to learn the technique.</p><h3>x64 Code</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*ynOSZf20jmKDlgxzQQDSug.png" /><figcaption>Example of x64 code</figcaption></figure><p>64-bit code has many similarities with 32-bit code: offsets and padding are still present, and may operations map to the same bytes. The most obvious differentiating factor is how frequent the ‘H’ character is in 64-bit code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/189/1*v2tA1QMoWZVU92Y_9Z2Nbw.png" /><figcaption>Most common 1–4grams in a sample of x64 code</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*vyP6ZeaVaefGex2AMe0ghw.png" /><figcaption>Direct contrast of a chunk of x86 vs x64 code. Note in this case the x64 code does not having padding between functions while the x86 code does, but that’s not always the case. If you count the number of ‘H’ characters in each section, it’s clear the bottom as more</figcaption></figure><h3>MSIL Code</h3><p>MSIL bytecode is very different from x86. Almost 25% of the bytes are 0x00, so there will be many significant gaps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*ZTWd6exT_Xz0F_l-TqWtWQ.png" /><figcaption>A sample of MSIL code</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/216/1*njlmKf9Ugh7wlbcMf6Bscw.png" /><figcaption>Most common 1–4grams in MSIL code</figcaption></figure><h3>Portable Executable (PE) Files</h3><p>In windows, the previously discussed code almost always shows up in the context of PE files (often seen as files with the .exe and .dll extensions), which have several recognizable patterns</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nVrKfOp6Y2RlCwaBv-s3yw.png" /><figcaption>Sections of a PE file</figcaption></figure><p>Because PE files have more than just code, it’s good to get familiar with what patterns appear in different sections. A normal file might have:</p><ul><li>Code in a .text section</li><li>Imports/exports/strings in a .rdata section</li><li>Strings in a .data section</li><li>Images/version info in a .rsrc section</li><li>Relocation data in a .reloc section</li><li>A signature appended on the end</li></ul><p>It’s important to be able to distinguish between common benign data (such as relocation data) and potentially malicious data (poorly encrypted, unknown, etc) to analyze files quickly.</p><p>The following is an example of a large amount of relocation data</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*dZ6Wr6O5ZlxBE8-2wkk9gQ.png" /><figcaption>Relocation data. In the disassembly view, Hiew highlights the bytes that correspond to global addresses which need modification when the file is loaded into memory</figcaption></figure><p>The following is an example of an icon file, commonly found in the resource section and displayed in explorer</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*GNchy4o14D9Lb14TKJFu-w.png" /><figcaption>.ico file</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*QQK5ja2oBhKbci62_2YkfA.png" /><figcaption>Icons that might appear in a PE file. Obviously a PE file with a PDF icon is extremely suspicious because it can trick a user into thinking it’s a document instead of an executable</figcaption></figure><p>The following is an example of a signature, appended to the end of the file</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bzhep1RQi_4IM39OX_1Qcg.png" /><figcaption>PE file digital signature</figcaption></figure><p>These types of signatures can be viewed and validated in explorer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/839/1*MAel7bpsOcuSUY55wX546w.png" /><figcaption>Signature details in explorer</figcaption></figure><p>When examining a clean or malicious PE file, it is normal to come across areas of high entropy data. The next section discusses some approaches to quickly getting insight into the purpose of this data, which often appears for legitimate reasons.</p><h3>High Entropy Data</h3><p>Many image formats contain high entropy data due to compression:</p><p>PNG:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YbCFqHeEEshumjKoU5x2wQ.png" /><figcaption>PNG File: most image with a .png extension will look like this</figcaption></figure><p>GIF</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/1*hGAVE53CLDtRbe8kNOLAYQ.png" /><figcaption>GIF</figcaption></figure><p>JPEG</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*yXDsRuTFG6cn5D1SZOWvfQ.png" /><figcaption>Jpeg</figcaption></figure><p>Compressed archives use compression (clearly), here is an example of a .zip file:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*kpm4NStPND0qy6ffA8WiEw.png" /><figcaption>zip file</figcaption></figure><p>In information theory, entropy can be used as a measure of randomness. Beyond simply identifying it, it’s impractical for a human to extract information from high entropy data because it looks uniformly random (no patterns such as repeating sequences, all bytes appear a similar number of times). However, there are sometimes easy ways to learn more about it. Most high entropy data is compressed and comes with header bytes at the top that indicates its purpose. Here are some standard formats that use compressed data</p><p>Images</p><ul><li>jpeg</li><li>gif</li><li>png (uses zlib)</li></ul><p>Archives</p><ul><li>zip</li><li>rar</li><li>.tar.gz</li></ul><p>Installers</p><ul><li>Nullsoft</li><li>Inno</li><li>Self-extracting archives</li></ul><p>Autoit scripts</p><p>zlib library</p><p>The following images show compressed data appended to the end of a PE file. In each case, there’s a header that indicates what the data is used for (installer packages, autoit script)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*30jWhNwDoh3BR_3SaU6IBA.png" /><figcaption>Nullsoft installer (starts with NullsoftInst)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*zv_UNX_VlkyMc8bH-2sXKw.png" /><figcaption>Inno installer (starts with zlb)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*A2GlExUyzL2vyedh9WIiiQ.png" /><figcaption>Autoit script (has AU3!)</figcaption></figure><p>The following is an example of a compressed flash file (CWS). It uses zlib, which almost always starts with 0x78 (‘x’).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*E_MzVhDh5xEff-lv7OYA1g.png" /><figcaption>CWS file that uses zlib</figcaption></figure><p>PNGs and PDFs may also use zlib.</p><p>High entropy data can also be crypto/compression constants. PEID’s “Krypto Analyzer” (KANAL) plugin locates the offsets/virtual address of common crypto constants, and IDA can then be used to investigate further. Once you have identified data as a crypto constant, its purpose is known, and you don’t need to investigate it further.</p><p>Below are some individually encrypted strings, which look high entropy (plus gaps between strings) because there isn’t much data. You might notice a chunk at the bottom that starts with “<strong>û</strong>0”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/1*hh0wCFiNY9Gjyun63dk0Ug.png" /><figcaption>Encrypted strings followed by a CRC constant</figcaption></figure><p>The KANAL extension of PEID can be used to investigate that chunk, which turns out to be used in a CRC calculation (image cross references with raw offset instead of virtual address). With practice, KANAL is no longer necessary to find crypto constants like CRC constants or a Rijndael S-box</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*yg97FHqu-9AjDDYpsyK-Qw.png" /><figcaption>KANAL displays offset of CRC constant</figcaption></figure><p>Meanwhile, IDA can be used to investigate the strings to find the decryption function</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*6k4G3olgaZ8xsacJIQrj0g.png" /><figcaption>IDA can be used to find the code that references encrypted strings (sub_1400EF84)</figcaption></figure><p>The location of data within a PE file can give additional hints. If it’s in the resource section, I always check for a PNG header because files sometimes store their higher resolution icons as PNGs. If it’s appended to the PE image, the file might be an installer, autoit script, or signed (there are discernible strings inside a signature). High entropy data outside of a PE file with no apparent headers might be some custom format. Sometimes there is a length indicator or magic number at the beginning, which might give some hint about the layout.</p><p>Sometimes malware hides malicious data inside of high entropy data with a standard header. Below is an example of a PNG file that has an encrypted MZ file, which is evident after decompressing the PNG’s data. It’s usually a good idea to view the PNGs inside a file to make sure they don’t contain malicious data (in my experience, this is rare though).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*P9fplrz5vHuWUF8Nhej3Uw.png" /><figcaption>PNG file that contains malicious data</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-rzXh6ZTgQRrbz3PiIIW4g.png" /><figcaption>UPX packed file. PE header followed by high entropy data followed by x86 decryption stub.</figcaption></figure><p>Secure encryption is high entropy and indistinguishable from random/compressed data. However, much of the encryption used by malicious files is weak and contains easily recognizable patterns. The next section discusses data patterns that emerge from weak encryption.</p><h3>Weak Encryption</h3><p>Weak encryption is any encryption that is easy to break with modern cryptanalysis, which in practice translates to pretty much any encryption scheme that is not well known and tested. Outside of sophisticated attacks, where the key might be stored on a remote server or dynamically generated in a machine-specific way, the encryption key to decrypt a packer’s payload must be in the binary, so the protection against cryptanalysis that secure encryption offers doesn’t help. Therefore the goal of most malware packers is not secure encryption or even protection from analysis but is instead to hide from antivirus software. Poorly encrypted data is, therefore, an indicator that the data might be used for malicious purposes. One example of a weak encryption scheme is a short xor key — in this case, it’s sometimes possible to recognize the algorithm and the key by briefly looking at the encrypted data. Note that there are legitimate reasons to use encryption to protect a file from analysis: for example, intellectual property protection. However, there really any legitimate reasons to hide from antivirus products (as long as those products properly reflect a user’s interests).</p><p>Here is an example of 1-byte xor encryption at the beginning of a PE file. Because 0x00 XORed with any byte is that same byte, and we know PE headers have 0x00 padding between the header and the code, it’s pretty clear what the key is: 0xFE (the box symbol).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*FKusqI8_irkPnFsSKTr8yg.png" /><figcaption>1-byte xor encryption of PE file</figcaption></figure><p>After this PE file has been encrypted, any sequence of bytes that could be used to detect the original PE file will no longer work. Modern antivirus features like emulators and dynamic analysis were invented to maintain protection against this type of evasion. 1-byte xor keys are rare in practice because packers change encryption keys every time they pack a new executable to avoid detection on the encrypted data. 1-byte xor keys only allow 255 variations (XORing with the 0 byte does not encrypt).</p><p>Here is the same file XORed with a 4-byte xor key, it’s still easy to recognize the key</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*52YshwdwwmSDC9p6E3XM0g.png" /><figcaption>4-byte xor encryption of PE file. Key boxed in red</figcaption></figure><p>Here is the same file XORed with a 16-byte xor key</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*i6BAc2bVCi-xsfl8hkL7lA.png" /><figcaption>16-byte xor encryption of PE file</figcaption></figure><p>Finally, here is the same file XORed with a 64-byte xor key. While it’s still possible to see patterns with some effort, it’s not trivial to see the key anymore (though it would probably still be pretty easy using standard cryptanalysis). If the file was encrypted with a random xor key equal to its length, then it would be perfectly encrypted and perfectly high entropy (this ‘perfect’ type of encryption is called a one time pad)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*G8_b0gi8PsMWwSlhLxNw1w.png" /><figcaption>64-byte xor encryption of PE file</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/649/1*vyFtHiMd3wRiQ9IJkfD_5g.png" /><figcaption>Some repeating sequences are still visible in the 64-byte version</figcaption></figure><p>Here’s a practical example where a 1-byte xor key is used. The following image shows a collection of strings used in a .jar file that has been obfuscated. It isn’t super easy to see the xor key, but simple cryptanalysis would break this easily (think of what characters appear the most in strings).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/1*llOPTa4Lj71mqrn_aiu0ew.png" /><figcaption>Obfuscated with JShrink</figcaption></figure><p>Here is the unencrypted version</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*oE5Zc2crIhAJh6taWRoVLw.png" /><figcaption>Unencrypted JShrink data</figcaption></figure><p>The next section shows some examples of data that might look like weak encryption at first glance but is a different type of data. When practicing this technique, it is useful to look up in IDA how the code uses data you don’t recognize.</p><h3>Not Weak Encryption</h3><p>It takes some practice to distinguish suspicious data from benign data such as .ico files, relocation data, and strings in languages with non-roman alphabets (such as the GB 2312 character set for simplified Chinese characters).</p><p>Here is an icon. Even with weak encryption, it’s rare for the encrypted data to be lower entropy than the unencrypted data. Therefore I wouldn’t find this data too suspicious because most potentially malicious data (code, strings) has higher entropy than this. In other words, the fact that there is so much predictable repetition means that this chunk of data likely cannot encode as much information as normal x86 code in the same number of bytes. There are exceptions depending on how crazy the encryption algorithm is.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*JZHudOBSEOB1-45RJOaOAg.png" /><figcaption>.ico file</figcaption></figure><p>Here are some Chinese strings. At first glance, they look like they might be poorly encrypted ASCII strings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gY0TDk1fai9eQh2aNzr-9w.png" /><figcaption>Chinese strings viewed in Hiew and IDA</figcaption></figure><p>Here are strings in various languages/encodings. Some are easier to differentiate than others.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/1*gvaTc9D1rJMEOZD7z87Nmg.png" /><figcaption>Various ways to encode different languages</figcaption></figure><p>The best way to become familiar with what is and isn’t poorly encrypted data is to look at many examples of clean and malicious files and examine how suspicious-looking data is used in the code. The next section shows examples of actual malware that use weak encryption to hide from antivirus software.</p><h3>Malware Examples</h3><p>Common x86 malware tends to have a decryption code stub combined with one or more packed layers. Familiarity with x86 patterns means recognizing the code stub is usually straightforward, and sometimes it’s even possible to distinguish between different layers of encryption. If there are large regions of the PE file with strange-looking data, it’s usually a good idea to check memory dumps for known malicious strings. This process can allow you to quickly determine what threat a file is without even opening IDA (sometimes in less time than it would take IDA to load the file). In all of the following examples, if you were to take any file packed with the same packer, it would look about the same, though the specific bytes would be different.</p><p>Below is an image of Trojan.Asprox.B which includes the unpacking stub on the top and the packed data on the bottom</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/1*ipX0_HoOIUcV6L0yvG75CA.png" /><figcaption>Trojan.Asprox.B. x86 unpacker stub followed by poorly encrypted data</figcaption></figure><p>The following is an image of Trojan.Tracur. Three different sections are visible: outlined in red is the x86 unpacking code, outlined in green is the first layer, and outlined in orange is the top of the final layer (which is an encrypted MZ file).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N7ay2UbiAb1iEYiCG8KXOA.png" /><figcaption>Trojan.Tracur. x86 unpacker stub followed by poorly encrypted data followed by a different type of poorly encrypted data</figcaption></figure><p>Here are some encrypted sections in Trojan.Zbot samples:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*2FDeO2uOaaWsnwVGYZYKjg.png" /><figcaption>Trojan.Zbot poorly encrypted data</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*vCDSRynOJQLGndnvyUVueA.png" /><figcaption>Trojan.Zbot poorly encrypted data with PE header</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*g7bbG4VFkjO4EuWOQQUNmQ.png" /><figcaption>Trojan.Zbot poorly encrypted data with PE header</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*ZsA0ZUN9xUR4qhSvOQD7wQ.png" /><figcaption>Trojan.Zbot poorly encrypted data</figcaption></figure><p>There are a vast number of patterns that can be produced by weak encryption, so it takes a lot of practice to be able to recognize it generically. The next section has some miscellaneous tips for using this technique.</p><h3>Additional Notes</h3><ul><li>Familiarity with code makes finding shellcode easier, see Hangul Word Processor exploit below (bytes for “push ebp mov ebp, esp” easy to see at start)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*JyxBpe5y3bxQrQeqPCKrvg.png" /><figcaption>Hangul word processor exploit. One green box has the bytes that correspond to the beginning of a function, and the other contains data for loading/referencing kernel32.dll. These are highly suspicious in the context of a document file.</figcaption></figure><ul><li>This technique can be applied to any binary file and is especially useful for dumps</li><li>It can also be used to browse memory in a debugger like <a href="http://www.ollydbg.de/">ollydbg </a>when using the full ASCII view. Unfortunately, many debuggers will map characters above the normal ASCII range to “.”</li><li>Sometimes different IDEs/compilers create different file structures, and this technique can help recognize/analyze those files. The example below is Delphi. One quick way to get a lot of information quickly on Delphi files is to go to the entry point (f8 f5 in non-text mode), then scroll up to see the custom strings. In Delphi the strings are mixed in with the code, and the library strings are closer to the top.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*880WzImO1eqh4AcW2uHRZg.png" /><figcaption>Delphi file</figcaption></figure><h3>Hiew Background</h3><p>Hiew is a full-featured hex editor designed with PE analysis in mind and is well suited for this technique. It has three view modes: text, hex, and disassembly. This technique primarily uses the text view. While it’s good to be familiar with all of Hiew’s features/hotkeys, the most important here are ‘Enter’ to switch view, ‘Pageup/Pagedown’ to scroll, and F2 to toggle Wrap/Unwrap (make sure “Unwrap” is displayed in the bottom left next to “2”). F8 in text mode can be used to switch between character encoding; if set up correctly, the table should be left on “As Is” (if you see a lot of ‘?’, it may mean it opened in Unicode mode).</p><p>Thanks to Mircea Ciubotariu for introducing me to this method of analyzing files, and for creating the font used in this article (available <a href="https://drive.google.com/file/d/1TdS03Yb-jpwlUdFVBnV974ZyEk0iNjkA/view?usp=sharing">here</a>). Thanks to Geoffrey So and Andrew Gardner for providing suggestions and feedback on the article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a4d087c9d26" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/malware-analysis-with-visual-pattern-recognition-5a4d087c9d26">Malware Analysis with Visual Pattern Recognition</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Covariate Shift in Malware Classification]]></title>
            <link>https://medium.com/data-science/covariate-shift-in-malware-classification-77a523dbd701?source=rss-352fdc0a3e4c------2</link>
            <guid isPermaLink="false">https://medium.com/p/77a523dbd701</guid>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[malware]]></category>
            <dc:creator><![CDATA[Nolan Kent]]></dc:creator>
            <pubDate>Tue, 26 May 2020 18:24:37 GMT</pubDate>
            <atom:updated>2020-07-31T00:47:15.452Z</atom:updated>
            <content:encoded><![CDATA[<h4>Classifiers in rapidly changing environments need to go beyond generalization</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*CUnBfU5kLNU-AFmPs2hTkA.png" /><figcaption>Ignoring large parts of a PE file in malware classification is like ignoring major facial features in face recognition: both can result in performance problems even if the training error is low for a model. However, for PE files, the performance gap may be between testing and real-world datasets instead of training and testing datasets.</figcaption></figure><h3>Introduction</h3><p><a href="https://rtg.cis.upenn.edu/cis700-2019/papers/dataset-shift/dataset-shift-terminology.pdf">Covariate shift</a> in the data-generating distribution lowers the usefulness of generalization error as a measure of model performance. By unpacking the definitions, the previous sentence translates to “high accuracy on the distribution of files we sampled our training set from (excluding the training set itself) does not always mean high accuracy on other important distributions (even if the true relationship between a sample and its label does not change).” A fruit classifier that had a single example of a banana in its training set might perform poorly on a sample set consisting solely of 10,000 variations of bananas, even if it did very well on a held-out test set sampled from the same distribution as its training set (where presumably bananas are scarce). This performance gap can be a problem because approximating generalization error with a held-out test set is the canonical way to measure a model’s performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lyl8ke-xG7Xoganl6lm7ug.png" /><figcaption>A rapidly changing input distribution is a common challenge in practical machine learning</figcaption></figure><p>This concept is essential for malware classification, and many <a href="https://ieeexplore.ieee.org/document/8681127">papers</a> discuss it in the context of sampling bias, where the samples used for training are not randomly sampled from the same distribution as the samples used for testing. Anecdotally, the intuition that models with good generalization may have poor performance once deployed — due to the difference between the files the model was trained on and the files it scans — is held by many malware analysts I’ve worked with, including those with little ML experience. However, it’s not always clear how to identify and fix issues related to covariate shift: conventional regularization techniques that improve generalization may not help when good generalization isn’t enough. After spending 5 years reverse engineering and classifying malware (using traditional, non-ML based techniques), I developed lots of intuitions about malware classification that I wasn’t able to express until I grew a passion for machine learning. My goal with this blog is to explain these opinions in the more formal context of machine learning. As some of these attempts at formal explanations are imprecise (and possibly incorrect), another goal of this blog is to solicit discussion, feedback, and corrections. I pack in a lot of different concepts and background because I feel relating concepts to each other helps improve understanding, especially if they can be tied back to practical goals like malware classification.</p><p>I hope that anyone interested in covariate shift can benefit from this blog, but the target audience is a person with some experience in machine learning who wants to apply it to computer security-related tasks, particularly “static” malware classification (where we want to classify a file given its bytes on disk, without running it). One of the key conclusions that I hope this blog supports is the importance of incorporating prior knowledge into a malware classification model. I believe prior knowledge is particularly relevant when discussing the ways a model might represent its input before the classification step. For example, I detail some domain knowledge that I believe is relevant to CNN malware classification models in this article: <a href="https://towardsdatascience.com/malware-analysis-with-visual-pattern-recognition-5a4d087c9d26#35f7-d78fef5d7d34">https://towardsdatascience.com/malware-analysis-with-visual-pattern-recognition-5a4d087c9d26#35f7-d78fef5d7d34</a></p><p>I’ve included an appendix that provides additional background by going over some core concepts used in the main section (such as inductive bias, generalization, and covariate shift)</p><h3>Covariate Shift in Malware Classification</h3><p>I do not think models that demonstrate good generalization on large sample sets are useless. If a model does not perform well on the distribution it was trained on, that’s a problem. There is plenty of empirical evidence in the security industry that a model that generalizes well without being super robust to changes in the input distribution can still provide value, especially if retrained frequently and targeted at specific customer environments. However, at this point, many models used in the industry demonstrate high accuracy on held-out test sets. I believe improving performance on samples from important new distributions, such as files created a month after the model was trained, has a more practical impact than making tiny marginal improvements in test set accuracy. The next few sections go over why these new distributions are essential and why they are an example of “covariate shift.” If you’re new to the concept of covariate shift,<em> </em>I’d recommend going through the background section in the appendix.</p><h4>Why p(x) changes</h4><p>In the case of malware classification, <em>p(x) </em>is the distribution of binary files analyzed by the model. Practical malware classifiers are never applied to the distribution they are trained on; they need to perform well on sets of samples in production environments that are continually evolving. For many antivirus vendors, that means performing well on files present on every machine with that antivirus, as well as samples submitted to other services like VirusTotal and antivirus review tests. There are two main reasons why generalization performance may not be a good predictor of how well a model performs on these samples:</p><p><strong>Sample selection bias:</strong> It is rare in the security industry to train models on a random selection of files that the model encounters when deployed. This is because accurate labeling is expensive: it often requires some level of manual human analysis, and it requires having access to all scanned files. It is normal to include files that analysts have labeled for other reasons, such as for the direct benefit of customers or files that have been supplied and labeled by third parties. If human analysis isn’t involved, there is a danger of feedback loops between multiple automated analysis systems. Because we can’t feasibly train the model on a truly random sample of the target distribution, the sample selection process is biased. Technically we can get around this by randomly sampling files our scanner encounters and labeling them with human analysts, but this would result in a small sample set of mostly clean files. I refer to the distribution of files that are encountered by the scanner after the model has been deployed as the “in-production distribution.”</p><p><strong>Environment shift:</strong> Every time you copy, edit, update, download, or delete a file, you modify the distribution your malware scanner encounters. When summed across every machine, the shift in distribution can be significant enough to mean a classifier that performed well last week may degrade significantly this week. Malware classifiers also exist in an adversarial setting, where attackers make deliberate modifications to the input to avoid the positive classification. A typical strategy to mitigate this issue is constant retraining, but it is impossible to ever train on the actual in-production distribution (even if we did have labels for them) unless we never scan files created after we gathered our training set. This scenario produces a type of sample selection bias that is impractical to overcome: we cannot train on samples that will be created in the future.</p><h4>Why p(y|x) does not change</h4><p>Covariate shift requires that <em>p(y=malicious|x=file)</em> does not change when <em>p(x) </em>does (see appendix). In the case of malware classification, <em>p(y = malicious|x)</em> is the probability a given file <em>x</em> is malicious. If <em>p(y|x) </em>and <em>p(x)</em> both change, it may be impossible to maintain accuracy. I can’t prove <em>p(y|x) </em>won’t change, but I argue that changes are generally, for certain classes of models, independent of <em>p(x)</em>. Here <em>p</em> refers to the true data-generating distributions rather than a probabilistic classifier’s attempt to approximate <em>p(y|x).</em></p><p>There are two main reasons why <em>p(y=malicious|x) </em>would be anything other than 1 or 0:</p><p>A. Uncertainty in the input: <em>y</em> is a deterministic function of some representation <em>x’</em>, but the input <em>x</em> the model uses does not contain all the relevant parts of <em>x’. </em>This concept is touched upon in the “deterministic vs stochastic classification” section in the appendix, but it’s a critical case, so I’ll expand on it in the next section. I use <em>x’ </em>to refer to inputs that can be mapped deterministically to a label and <em>x</em> to refer inputs that may or may not be deterministically mappable to a label.</p><p>B. Uncertainty in the label: The definition of <em>y</em> does not fully account for all inputs (grayware is an example in malware classification). I’ll leave a full explanation to the appendix section “Why p(y|x) does not change: Uncertainty in the label.”</p><h4>Uncertainty in the input</h4><p>In this part, I argue that the bytes in a file correspond to <em>x’.</em></p><p>Here’s a simple example. I precisely define ‘eight’ to be a specific shape shown in Figure 1</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/21/1*EnUB0FecO6sJgCMl2Whkdw.png" /><figcaption>Figure 1: The definition of ‘eight’</figcaption></figure><p>and ‘three’ to be the shape shown in Figure 2</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/19/1*_3oU9umjbAIGeygNkDAa4g.png" /><figcaption>Figure 2: The definition of ‘three’</figcaption></figure><p>If we have access to the entire input (<em>x’</em> = Figure 1 or Figure 2), we know with perfect certainty what the label should be. However, we may not have access to the full input as illustrated below, where some of the input is hidden with a red box:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/59/1*9caiM194iFn6Ys8KhzXlvg.png" /><figcaption>Figure 3: Data can be hidden, in this case by a red box</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/315/1*YbgDIUUUZsLmYMX_-WmaLQ.png" /><figcaption>Figure 4: Uncertainty due to hidden data</figcaption></figure><p>In this case, our input <em>x</em> (Figure 3) maps stochastically (Figure 4) to two different x’<em> </em>(Figure 1,2), which map deterministically to the <em>y</em> values ‘eight’ and ‘three’. I’ll refer to the mapping function as <em>g</em>, in this case, <em>g</em> adds a red box that hides the difference between an eight and three. I refer to the function that maps <em>x’</em> to <em>y</em> as <em>f_true</em>, which is effectively a perfect classifier that avoids randomness by having access to all relevant information. See the ‘deterministic vs stochastic classification’ section in the appendix for more details.</p><p>If the training set contains 90 copies of Figure 3 labeled as “3” and 10 copies labeled as “8” and we have a uniform prior, then <em>p(y=8|Figure 3) = 0.1 </em>and <em>p(y=3|Figure 3) = 0.9</em>. If the test set is sampled from the same distribution, it is likely to have a similar ratio of 3’s to 8’s, so our model will generalize well. However, a new-distribution test set might have 8’s appear 9 times as much as 3’s on average. If we tested on a sample from that distribution, we would probably get abysmal performance. Unless we can un-hide some of the data, it may not be feasible to have good generalization error and new-distribution error. In this blog, I use new-distribution error to refer to the equivalent of generalization error when the model is applied to a distribution different than the one its training set was sampled from (more on this in the covariate shift section of the appendix). Practically, attempting to find the new-distribution error for the in-production distribution is likely a better measure of performance for a malware classifier than the generalization error. Figure 5 illustrates this example. A key takeaway is that when we transition from population 1 to population 2, we undergo covariate shift when considering our raw sample (figure 2 in both cases) and <em>p(y|x) </em>defined by <em>f_true</em>. However, the type of shift we encounter when we consider an intermediate representation that hides relevant information <em>(</em>figure 3) isn’t covariate shift: it’s worse than covariate shift. This is because <em>p(y=label “three”|x=figure 3) </em>goes from 90% in population 1 to 10% in population 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zMS1hlOnt4WXqLTjE9Y3Pg.png" /><figcaption>Figure 5: <em>p(y|x)</em> is constant for f_true, but changes when transformed by<em> g</em>. The population box indicates the population we are sampling from (9x[image] means there are 9 of that image in the population), the sample box indicates the sample we draw from that population (‘3’ in both cases), the function box indicates what function we apply to the sample, and the p(y|x) boxes indicate the conditional distribution of possible labels given the function output for the population.</figcaption></figure><p>In summary: If information is hidden (input is <em>g(x’)</em>) then <em>p(y|x) </em>may change with <em>p(x) </em>because the change to <em>p(x)</em> is caused by the underlying distribution <em>p(x’)</em> changing. This change in <em>p(y|x) </em>can occur when there’s sampling bias or environment shift, which are both common in malware classification.</p><p>As an example, say your dataset consisted of many recent malicious samples, along with clean samples from a wide range of dates. Given that malicious samples change more frequently than clean samples, this is a common situation. Say the model uses the (<a href="https://docs.microsoft.com/en-us/windows/win32/debug/pe-format#file-headers">PE</a>) file’s timestamp value to classify the files (<em>g</em> hides all information except the timestamp). This feature works equally well for the training set and test sets sampled from the same distribution, as in both cases, malware tends to be more recent than clean files. The feature technically generalizes well, but any classifier that is based entirely on timestamp will not perform well when a dataset contains recent clean files, such as on any machine that just had a software update. To relate this to the deterministic vs stochastic classification section of the appendix, <em>y=f_ideal(x = PE Timestamp)</em> is very different from <em>y=f_true(x’ = all bytes in a file), </em>where <em>x = g(x’)</em> and <em>g </em>is a feature extractor that extracts the PE file’s timestamp. Note this assumes the malicious samples do not make deliberate modifications to the PE timestamp, which can happen.</p><h4>Required assumption to keep p(y|x) constant</h4><p>The definition of malware is precise, unchanging, and is dependent solely on the bytes of a file (<em>x’</em>), so that <em>y</em> is a deterministic function of <em>x</em>: <em>p(y = malicious|x’) = 1 or 0 = f_true(x’ = all bytes in a file).</em> Here’s another way to think about that: given that the space of files that can be practically represented is finite (though absurdly large), we can give a label to every point in that space. This assumption requires that we consider a file in the context of its dependencies, as some types of malware have multiple components.</p><p>In the next part, I discuss ways to try and determine if an approach is more or less vulnerable to changes in input distribution. There is always some vulnerability, as our classifiers <a href="https://en.wikipedia.org/wiki/Curse_of_dimensionality">need</a> dimensionality reduction, and we don’t have dimensionality reduction techniques that won’t lose at least some information relevant to the deterministic mapping from <em>x’</em> to <em>y (f_true)</em>.</p><h3>Mitigating Issues Caused By Covariate Shift</h3><p>After making some assumptions, the previous section established that covariate shift is an ok way to describe what happens when we consider all the bytes in a file and the distribution of files changes. Unlike scenarios where both <em>p(x)</em> and <em>p(y|x) </em>change, we have a path to creating a classifier that works: find a function as close as possible to <em>f_true</em>. Here’s another way to look at that: probabilistic classifiers attempt to find <em>p(y=label|x’=bytes in a file). </em>The data generating distribution has a <em>p(y|x’) = f_true </em>which does not change when <em>p</em>(x’) changes. If we find <em>f_true, </em>then our model’s <em>p(y|x’</em>) will not change with changes in <em>p(x’).</em></p><p>An alternative might be trying to predict precisely where the input distribution will shift, but I’m skeptical of how well that would work in a highly dynamic, adversarial setting. In the fruit classifier example from the introduction<em>, </em>trying to model <em>f_true</em> would mean creating a model that understood the definition of each fruit and was able to compare any input to that definition: this approach would work when the dataset shifts to mostly bananas. Predicting where the input distribution will shift might means putting a higher emphasis on a specific class during training. If we picked a class other than bananas, our model would still perform poorly after the input distribution shifts to mostly bananas.</p><h4>What makes a model vulnerable to covariate shift, and how can that be fixed?</h4><p>Hypothetically a new-distribution dataset sampled randomly from the files scanned by a deployed model could be used to measure relevant new-distribution error. Receiving more false-positive reports than expected given the test set error may also hint at high new-distribution error. However, as perfect generalization does not guarantee robustness to changes in the input distribution (see the PE timestamp example in the previous part), the strategies that can be used to increase generalization such as common regularization techniques will not help past a certain point (though if our model doesn’t generalize that’s bad news for new-distribution performance). We need to find a new knob to turn to improve the new-distribution error once we have a low generalization error. I think that one of the best ways to do that is to find ways to incorporate prior knowledge to get the best inductive bias. There are several ways to restrict the hypothesis space of our learner to exclude models that generalize well but perform poorly on the new-distribution test set. In the example where the timestamp is used as a feature to produce a model that generalizes but is not robust to changes in the input distribution, excluding hypotheses that rely on the PE timestamp may help reduce new-distribution error. This exclusion is based on the prior knowledge that clean and malicious files can both be new or old. For PE files, there are a huge number of features that may need to get excluded, and features we want to keep may be difficult to compute (though it may be possible to use neural networks as feature extractors for more complicated functions).</p><p>In the extreme, if we have prior knowledge of the function <em>y = f_true(x’)</em> that maps <em>x’</em> to <em>y </em>across the entire domain of <em>x’</em>, then we can exclude all hypotheses that are not <em>f_true(x’)</em>. While this function is generally not practical to find, it is required for malware analysts to be able to mentally approximate <em>f_true</em> to be able to classify all files they see, even if they haven’t seen similar files before. It helps that in practice, we don’t need to consider the whole domain of possible files, just all files created by modern humans and our tools.</p><p>We can (and probably should) incorporate inductive bias at every step: what data to use, how to preprocess it, what model to use, how to pick architecture, features, hyperparameters, training method, etc. An example of each of these is beyond the scope of this blog; instead, I’m going to focus on feature extraction, which is one of the most popular ways to introduce inductive bias.</p><h4>Achieving robustness to changes in the input distribution</h4><p>Feature engineering is a crucial step for many types of models as features contribute directly to the inductive bias of the model: if features exclude parts of the input data, the model cannot use that data as part of the learned hypothesis. Transforming the data into a lower-dimensional set of features is also very important, as one of the main challenges in finding <em>f_true(x’)</em> is that <em>x’</em> is very high dimensional: a <a href="https://en.wikipedia.org/wiki/Curse_of_dimensionality">major problem</a> for many types of classifiers. Hand engineered features are common for some models like random forests, while other models like convolutional neural networks automatically extract features. For both hand-engineered and automatically extracted features, models tend to follow the architecture shown in Figure 6:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*4rg7QKLwn9ih5uYUxGFh6g.png" /><figcaption>Figure 6: Common architecture in many machine learning models</figcaption></figure><p>Each box depends entirely on the previous box. The first three boxes contain groups of features or a ‘representation.’ Boxes toward the right cannot contain more information about the input than boxes to their left. Typically, the goal of an architecture like this is to throw away information that is irrelevant to the classification. Less irrelevant information allows the classifier to deal with a lower-dimensional representation than <em>x’ </em>while maintaining performance. Specifics depend on the model, with neural networks having many more layers of representation than decision tree-based models, for example. The critical detail is that each representation tends to have less information about the input than the previous representation, and if the information lost is relevant to <em>f_true(x’)</em>, then all downstream portions of the model have to deal with the ‘Uncertain input’ case, where I argued that <em>p(y|x)</em> could change with <em>p(x)</em>. Therefore, we want to ensure the lower-dimensional representation still contains enough information to compute <em>f_true(x’) (</em>see Figure 7), even though it contains less information about the original input overall. Unfortunately, we usually can’t achieve this in practice, so when considering a model’s intermediate representation of the input, we’re no longer dealing with covariate shift. That said, we can still work to mitigate the amount <em>p(y|x=intermediate representation) </em>changes with <em>p(x)</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/399/1*TXFR17RtdpYiQRFC7-VMvA.png" /><figcaption>Figure 7: prior knowledge tells us if we have to hide information, it’s better to hide the bottom half than left half when classifying 3 vs 8: the left two images can still be classified perfectly while a classifier for the right two images depends on the proportion of 3s and 8s staying constant</figcaption></figure><p>As discussed in the paper <a href="https://arxiv.org/abs/1608.08225">“Why does deep and cheap learning work well”</a>, this concept is related to the data processing inequality:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/167/1*ktk0DlZIfw0wa0dnRVYyGQ.png" /></figure><p>&gt;= is replaced by = if <em>g(x)</em> is a sufficient statistic for <em>y. </em>Mutual information (<em>I)</em> is distribution-specific, so even if <em>g(x)</em> is a sufficient statistic for <em>y </em>given <em>p(x), </em>it may not be when the distribution shifts to <em>p’(x)</em>. To be perfectly robust to change in the input distribution, we need <em>y </em>to be a deterministic function of <em>g(x). </em>If that’s the case, <em>g(x)</em> is a sufficient statistic for <em>y</em> for all distributions over x. We already established that this is the case for the data-generating distribution: covariate shift implies that <em>p(y|x’) </em>does not change with <em>p(x’).</em></p><p>Classification is discussed in ‘deep and cheap learning’ paper as the reverse of a Markov chain-based generation procedure, which makes intuitive sense for malware classification: the author(s) of a file will know if the functionality they programmed is malicious or not as an early step in the creation process. An example of reversing the process for malware is shown in Figure 8:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/1*uBYGsZs8TF3pMX2LMf3yqQ.png" /><figcaption>Figure 8: Markov chains representing generation vs classification of malware</figcaption></figure><p>There are shortcuts analysts use to speed up classification, but this diagram is not too far off from the engineering and reverse engineering that goes into malware creation and human analysis (if the analyst wants the most accurate result and has a lot of time). Unfortunately, each of these steps is complicated, and creating a robust function to transition between them is probably impractical today — malware packers are still the default way to hide from antivirus products. We need an alternative, but it helps to keep this roughly idealized Markov chain in mind: note that at no step in the classification process is information relevant to malicious functionality lost. In a later blog, I may discuss alternatives inspired by the shortcuts malware analysts use to speed up classification, with a focus on malware packers.</p><p>The key takeaway is that if a representation the model uses at any point does not contain information directly related to the file being malicious or not, the model may be more vulnerable to changes in the input distribution. For example, some models in the security industry primarily use values extracted from the PE header as features. However, this representation excludes a lot of information related to <em>f_true(x’)</em>, as prior knowledge tells us PE header values do not give much indication about a file’s behavior, which is required to determine if the file is malicious. This is compounded by the fact that the default way to evade antivirus signatures — custom malware packers — can set the PE header to pretty much whatever they want. A model that depends entirely on PE header values may do a good job detecting files packed with malware packers present in its training set, but will likely do worse when presented with a new malware packer. A representation that includes a file’s possible behaviors, however, would not throw away information related to <em>f_true(x’)</em>. This is difficult to do in the context of static detection (detection without executing the file); however, we can still use prior knowledge to try and extract features that are more closely related to <em>f_true(x’)</em> than PE header data. An example of this is a feature that detects obfuscated data, as obfuscation is often applied to make antivirus detection more difficult: something only malware needs to do.</p><p>The problem of throwing away information relevant to the data-generating distribution’s <em>p(y|x’)</em> was also illustrated back in figure 5: by hiding important data with a function <em>g</em>, a model that’s built on top of <em>g’s </em>representation will often have the relation between <em>g(x’)</em> and the label change when the raw input distribution <em>p(x’)</em> changes.</p><h3>Conclusion/Future Work</h3><p>Given that there are a lot of tools focused on improving generalization, it may be helpful to think about how we can tie generalization error to improving robustness to changes in the input distribution. If our samples are represented in a way that makes malicious files from a new distribution appear the same as malicious files from our original distribution, lowering generalization error may be all that’s necessary. I plan to discuss some ways to approach robustness to changes in input distribution in another blog, with a focus on representation. One example is using convolutional neural networks to specifically detect visual features in a file that human analysts would immediately recognize as malicious across any distribution of files, but for which it would be difficult to manually program heuristics. I would also like to find good methods for measuring new-distribution error for better model comparison. Some distributions are clearly better than others for measuring new-distribution error, and the practical problem of finding the true label of all in-production samples still prevents that distribution from being used in practice. For example, an alternative could be finding the true labels for a small subset (&lt;1000) of samples from the in-production distribution and calculating accuracy (or precision) along with a confidence interval that accounts for the limited sample set. Precision is often easier to work within the security industry due to difficulty in finding in-production false negatives in an unbiased way (due to the number of true negatives), and the fact that false positives on average negatively impact more people than false negatives (clean files are much more likely to appear on more than one machine). I would also like to consider adversarial machine learning in the context of covariate shift.</p><p>In summary: In any situation where our data generating distribution undergoes covariate shift, we can potentially find a model that is robust to changes in the input distribution <em>p(x’). </em>One approach to this is to use domain knowledge to ensure that each intermediate representation utilized by our model does not throw away information relevant to the true relationship between <em>p(y|x)=f_true.</em></p><p>*****************************************************************</p><h3>Appendix</h3><h3>Background</h3><p>In this part, I go over some relevant background concepts. Here’s a quick summary:</p><p>Training error measures how poorly a model performs on its training data, and out-of-sample (generalization) error measures how poorly a model performs on samples from the same distribution as the training data. New-distribution error measures how poorly a model performs on samples taken from a different distribution than the training data.</p><h4>Deterministic vs Stochastic Classification</h4><p>Given input space <em>X</em> and label space <em>Y</em>, the most desirable outcome of classification is often to find a function <em>f_ideal</em> that perfectly maps inputs to labels: <em>y = f_ideal(x)</em> (where <em>x ∈ X, y ∈ Y</em>) for our input distribution. This function would correspond to a Bayes optimal classifier with a Bayes error of 0. Unless we can somehow prove that we found <em>f_ideal</em>, we can only use a hypothesis function <em>h</em> that approximates it. I’ll discuss this more later in the context of measuring errors.</p><p>In many cases, the relation between <em>Y</em> and <em>X</em> isn’t deterministic: for the same <em>x</em>, there may be different possible <em>y</em> values. In that case, instead of <em>y = f_ideal(x)</em> we have a probability distribution where <em>y</em>～<em>p(y|x)</em>. If this conditional distribution is Bernoulli — such as malicious (<em>y</em>=1) vs clean (<em>y</em>=0) for <em>x</em>=input file — we can have our classifier output 1 if <em>p(y=1|x) &gt; 0.5 </em>and 0 otherwise. This produces the Bayes optimal classifier, but the Bayes error is not 0 due to irreducible uncertainty. A common explanation for why the same <em>x</em> has different <em>y</em> values is because we don’t have enough information. That is, <em>x = g(x’) </em>where <em>x’</em> is the data we would need to have to compute y deterministically. <em>g </em>is a function that potentially hides information about x’: a lossy compressor, for example. This scenario implies there is a function <em>f_true</em> such that <em>y = f_true(x’)</em>. The key difference between <em>f_true</em> and a <em>f_ideal</em> is that <em>f_true</em> is perfectly accurate across the entire input domain rather than a single distribution of inputs. In this case, we have an underlying distribution <em>p(x’)</em> where we sample <em>x’</em> from and then convert to <em>x</em> using the deterministic, many-to-one function <em>g</em>. The reason the same <em>x </em>may have different <em>y</em> values is because a single sample of <em>x</em> represents multiple samples of <em>x’</em> which have different <em>y</em> values.</p><p>A trivial example is if we want to determine the average color (<em>y</em>) of an image (<em>x’</em>), but we’re only given (with <em>g</em>) a subset (<em>x</em>) of the pixels. In some cases, we have no access to <em>x’</em>, just <em>x</em>, for example, recommender systems may need to simulate a person’s brain (<em>x’</em>) to be 100% certain what to recommend, so they use a collection of behavioral features instead (<em>x</em>). However, in other cases, we do have access to <em>x’</em>, but we create <em>g</em> as part of our model because <em>x’</em> has <a href="https://en.wikipedia.org/wiki/Curse_of_dimensionality">too many dimensions</a> for our classifier to handle. This scenario is central to my understanding of what can make malware classifiers vulnerable to changes in the input distribution, and much of this blog expands on it. I use <em>x’</em> in cases where I believe the input deterministically determines the output; otherwise, I’ll use <em>x</em> for situations where the relation between <em>x</em> and <em>y</em> may or may not be deterministic. Note we can only learn the Bayes optimal classifier if our inductive biases do not exclude it from our hypothesis space; otherwise, we want to learn a function that approximates it as well as possible given the restrictions.</p><h4>A review of inductive bias</h4><p>As one of the most important topics in machine learning, inductive bias is relevant to most machine learning related discussions, but it is essential for this blog. <a href="http://A review of generalization">Inductive bias</a> refers to the set of assumptions a learner makes about the relationship between its inputs and outputs, which serve to prevent the learner from forming some (hopefully undesirable) hypotheses. All models we use today have some level of inductive bias, which is required to be useful. However, if the assumptions do not adequately match the problem, the model may not work well. Some views of conventional regularization methods frame them as an inductive bias for simple functions (Occam’s razor).</p><p>For example:</p><ul><li>If a learner assumes a simple relationship between input and output when the actual relationship is complex, it may lead to underfitting (poor performance on training data)</li><li>If a learner assumes a complex relationship when the actual relationship is simple, it may lead to overfitting (poor performance on held-out test data)</li><li>If a learner assumes a relationship holds across changes in distribution when it doesn’t, it may make the model vulnerable to changes in the input distribution (poor performance on new-distribution test set)</li></ul><p>In the next section, I’ll review generalization: the final prerequisite for a discussion of covariate shift.</p><h4>A review of generalization</h4><p>Generalization is a fundamental concept in machine learning, but to understand a model’s performance under covariate shift it helps to know how covariate shift is different from performance on an out-of-sample test set (a common way to estimate generalization).</p><p>Classification models are usually created by training a model on a set of labeled samples. Given this data, it is trivial to create a (non-parametric) model that performs flawlessly on its training data: for each input that is in the dataset, return the label for that input (memorization). For each input that is not in the dataset, return some arbitrary label. While this model performs perfectly well on the training data, it probably won’t do better than random guessing on any other data. Given we usually want to label other data, we need a way to distinguish between two models that have perfect training accuracy, but different accuracy on unseen data.</p><p>Informally, generalization usually refers to the ability of a model to perform well on unseen inputs <em>from the same distribution as the training data</em>. For example, if we train two models on photographs of cats and dogs, and both have excellent performance on the training data, we will usually want to use the one that performs best on photographs of cats and dogs that weren’t in the training set. We can (probably) find which model this is by comparing performance on an “out-of-sample” test set we randomly selected to be held out from the training process. Using a random held-out test set ensures that both the test set and the training set were sampled from the same distribution.</p><p>However, just like how two models with the same training accuracy might not be equal, two models with the same accuracy on an out-of-sample test set may not be equal. For example, if one of the models can correctly classify images of cats and dogs drawn by humans in addition to photographs, even though we only trained it on photographs, it may be preferable to a model that can only classify photographs. This evaluation criterion also applies to samples designed to produce an incorrect classification (adversarial examples): if the model is not trained on any adversarial examples, good generalization does not imply that it will properly classify new examples that have been adversarially modified.</p><h4>Malware packers and generalization</h4><p>Bypassing malware classifiers with adversarial machine learning is in some cases overkill: creating a new malware packer (that produces samples with extremely low probability in the model’s training distribution) often achieves the same result (a FN) as adversarial machine learning for many types of malware classifiers that generalize well. Attackers have been creating new malware packers for years to avoid algorithmic detections employed by traditional antivirus software. Malware packers hide a malicious payload behind one or more layers of encryption/obfuscation, leaving just an executable decryption stub available for static analysis. The stub, along with other file attributes, can be made to have very little correlation with previous malware packers. As a result, any payload hidden by that packer will, by most standard measures, be dissimilar to files in any train or test set existing models have utilized. As a consequence, malware classifiers that generalize well do not require attackers to change their high-level strategy for avoiding detection: they can keep pumping out new malware packers to hide from both traditional antivirus techniques and ML-based static detection. A malware classifier that is somewhat robust to changes in the input distribution (in this case caused by the creation of new packers) could be much more difficult to bypass. Note malware packers are different from commercial packers such as UPX, which aren’t designed to avoid detection by antivirus software.</p><h4>Covariate shift</h4><p>In the case of generalization, we take both the training and out-of-sample sets from the same data-generating distribution <em>p(x, y)=p(y|x)p(x) </em>over inputs and labels. Covariate shift is typically defined as a scenario where the input distribution <em>p(x) </em>changes, but <em>p(y|x)</em> does not. Cases where <em>p(y|x) </em>changes, but <em>p(x) </em>does not are referred to as concept shift. If the data-generating <em>p(y|x) </em>changes, statistical models that approximated the original <em>p(y|x) </em>will not be as effective.</p><p>Here’s an example: Say we have a training set that consists of a single example, <em>x</em> = 5.02, with a label <em>y</em> = 10.04 for a regression problem. If our inductive bias is that there is a linear relationship between <em>x</em> and <em>y</em>, we might get the model <em>y = 2x</em>. If that is the actual relationship, this model works for the entire domain of <em>x</em>. If the distribution of <em>x</em> is a Gaussian with unit variance and 5.0 mean, our test set might contain the example <em>x</em> = 4.4, <em>y </em>= 8.8, and we’d get perfect test accuracy. However, the model still works even if we change the distribution of <em>x</em> to be a uniform distribution between 1 billion and 2 billion. The model is robust to changes in the input distribution, achieved by finding <em>y = f_true(x)</em>. However, we’re out luck if the relation between <em>y</em> and <em>x</em> (<em>p(y|x) </em>in the stochastic case) changes in addition to <em>p(x): </em>say in one data set <em>x</em>=5.02 produces <em>y</em>=10.04 consistently, and in another <em>x</em>=5.02 produces <em>y</em>=20 consistently. Therefore it’s important that <em>p(y|x) </em>does not change.</p><p>In some cases, it can work to think of ‘robustness to changes in the input distribution’ as an extension of generalization. Take, for example, 3 different datasets. The first is a training set, the second is a test set from the same distribution as the training set (used to test generalization), and the third is a new-distribution test set, where the members are from the same domain as the other two datasets but sampled using a different distribution (used to test robustness to changes in the input distribution).</p><ul><li>If a learner’s inductive bias causes it to underfit the training data, it will (probably) perform poorly on all 3 sets.</li><li>If a learner’s inductive bias causes it to overfit the training data, it will (probably) perform poorly on the test set and new-distribution test set.</li><li>If a learner’s inductive bias allows it to perform well on the distribution its training data was sampled from and no other distributions, it will (probably) perform poorly on the new-distribution test set.</li></ul><p>I’ll try and relate generalization and robustness to changes in the input distribution a bit more mathematically here. I loosely took the formulation of generalization used here from <a href="https://en.wikipedia.org/wiki/Generalization_error">https://en.wikipedia.org/wiki/Generalization_error</a></p><p><em>V</em> is defined as our loss function, <em>h_n</em> is our classifier trained on <em>n</em> training samples. <em>x</em> is the input, for example, binary files, and <em>y</em> is the label, for example, clean/malicious.</p><p>Training error can be calculated by averaging the loss function over the training samples:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/255/1*U2_1rENHzPRP3YgzVVV1gw.png" /></figure><p>Generalization error is calculated by taking the expectation of the loss function over the entire input distribution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/1*EPu_SXGx9v9lXMjgcR8kkw.png" /></figure><p>Generalization gap or sampling error is calculated as the difference between training and generalization error:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/219/1*Otu3jI7VoF_0YsuWfTJthg.png" /></figure><p>Given that for covariate shift <em>p(y|x)</em> is constant and <em>p(x)</em> changes to a new distribution <em>p’(x)</em>, we can rewrite <em>I_G[h_n]</em> to create a “new-distribution error”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/376/1*LqjLEt-5NqmUJ67woLmtMA.png" /></figure><p>Generalization gap is formulated as the difference between training and generalization error, so it may also be helpful to express a distribution gap as the difference between <em>I_R[h_n]</em> and generalization error. This difference can be calculated with simple subtraction like in <a href="https://arxiv.org/abs/1902.10811">https://arxiv.org/abs/1902.10811</a> (but reversed for consistency): <em>I_R[h_n]-I_G[h_n]</em>. If the new distribution is harder to classify, then the gap is positive; if it is easier, the gap is negative, and if it’s the same difficulty, the gap is 0. A significant difference between the distribution gap and generalization gap is that the generalization gap is always positive, but with a new distribution, it may be that all the samples are in ‘easy’ regions for the classifier. Because training error is optimistically biased, it is very likely for training data to be in the ‘easy’ region for a classifier trained on it, keeping the generalization gap greater than or equal to 0.</p><p>If the distributions aren’t significantly different, it may be informative to examine samples that are likely in the new distribution but unlikely in the original to provide a better test of robustness to changes in the input distribution. This next part is very informal, but one approach could be to change <em>p’(x)</em> to remove probability mass from areas that are likely under <em>p(x)</em> so that we are only evaluating samples that are more likely under <em>p’(x)</em> than <em>p(x)</em>. This approach gives an error that is more focused on the samples that have become more likely in the new distribution. Here’s a pdf <em>p’’</em> that subtracts <em>p(x) </em>from <em>p’(x)</em> (max prevents negative probability density, and the denominator keeps the total area at 1)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/378/1*2syLQM7T5INSXNH4zVLvHQ.png" /></figure><p>This equation may not be intractable, but my assumption would be using a new distribution test set sampled from <em>p’(x)</em> could help just like using an out-of-sample test set helps measure generalization error. It may take prior knowledge to confirm if a sample from <em>p’(x)</em> was also likely in <em>p(x)</em>. For example, to better approximate <em>R[h_n]</em> rather than <em>I_R[h_n]</em> it may be worth focusing on the error of samples that are confirmed to be new malware families that had 0 probability in <em>p(x)</em>. In the photo-vs-hand drawn case (from the ‘brief review of generalization’ section), it should be safe to assume that natural images of cats and dogs are rare when sampling hand-drawn images (with the exception of photorealistic rendering), so samples from <em>p’’(x)</em> (images of hand-drawn cats and dogs excluding those similar natural images of cats and dogs) should be similar to samples from <em>p’(x) </em>(images of hand-drawn cats and dogs)<em>.</em></p><h3>Why p(y|x) does not change: Uncertainty in the label</h3><p>Figure 9 shows a simple example of case B from the ‘Why p(y|x) does not change’ section. In this case, we have access to the entire input, but we define the label ‘eight’ to apply to anything that looks like an eight, and the label ‘three’ to apply to anything that looks like a three.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/279/1*dIvq31MQmSpdDBzKAjUKDQ.png" /><figcaption>Figure 9: Uncertainty due to definition</figcaption></figure><p>Uncertainty arises because the input in the top left of figure 9 looks like both an eight and a three.</p><p>Often there is a spectrum of certainty, where examples near the middle may be more vulnerable to changing definitions, as shown in Figure 10:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/1*JLjcXjAhFvMMadhU5XA7QQ.png" /><figcaption>Figure 10: Spectrum of ‘looks like 8’ to ‘looks like 3’</figcaption></figure><p>Because the definition of malware is not precise, the same type of uncertainty can arise. A parallel example to Figure 10 for malware would look something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*Cwt0uLrh_pcPQnz9tY56Og.png" /><figcaption>Figure 11: Spectrum of maliciousness</figcaption></figure><p>Due to uncertainty in definitions, files with behavior closer to the middle of the spectrum (sometimes called Potential Unwanted Applications or Grayware) can be more challenging to classify. Utilizing ‘soft labels’ — labels above 0 and below 1 — is one way to deal with uncertainty in the label definition.</p><p>The definition of what is malicious may change, which implies a change in <em>p(y|x)</em>. However, there are a few reasons why I think we can disregard this in practice:</p><p>1. Relating to the ‘why p(x) changes’ section, the two main causes of <em>p(x) </em>changing — sample selection bias and environment shift — should not change our definitions.</p><p>2. Practically, we care most about the cases on the extreme ends of the spectrum where it is unlikely for changing definitions to have an impact. A file specifically designed to exfiltrate a user’s credit card info without their knowledge or permission will always be considered malicious, and an essential operating system file will always be considered clean. In the industry, changes in what we consider worth detecting is generally caused by debates around whether a certain behavior is clean or “potentially unwanted”</p><p>Obviously some future programs could become very difficult to classify, possibly requiring ethical philosophy to solve: for example <em>p(y=malicious|x=code for an AGI).</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=77a523dbd701" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/covariate-shift-in-malware-classification-77a523dbd701">Covariate Shift in Malware Classification</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Reversible Generative Models]]></title>
            <link>https://medium.com/ai-ml-at-symantec/introduction-to-reversible-generative-models-4f47e566a73?source=rss-352fdc0a3e4c------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f47e566a73</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[generative-model]]></category>
            <dc:creator><![CDATA[Nolan Kent]]></dc:creator>
            <pubDate>Wed, 12 Feb 2020 17:08:26 GMT</pubDate>
            <atom:updated>2020-02-12T17:08:26.788Z</atom:updated>
            <content:encoded><![CDATA[<p>How “Glow” produces high-quality images with change of variables</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/1*XUoN3mPwl0ulGlMi-jVnyQ.gif" /><figcaption>Animation of the coupling layer and its inverse, fundamental to recent advances in flow-based models.</figcaption></figure><p><strong>Preface</strong></p><p>I adapted this blog on flow-based models from a technical presentation I gave after reimplementing the ‘Glow: Generative Flow with Invertible 1x1 Convolutions’ paper from OpenAI as a personal project. The blog consists of two parts:</p><ol><li>Introduce a type of generative model that uses change of variables. These can also be described as flow-based models, reversible generative models, or as performing nonlinear independent component estimation.</li><li>Explain the ‘coupling layer’ introduced in “NICE: Non-linear independent component estimation,” which is fundamental to recent advances in these models.</li></ol><p>My goal is to help a reader familiar with machine learning brush up on the background necessary to understand the <a href="https://arxiv.org/abs/1807.03039">Glow paper</a>, which has shown amazing results. To get an idea of what Glow can do, check out this blog post and play around with the tool:</p><p><a href="https://blog.openai.com/glow/">Glow: Better Reversible Generative Models</a></p><p>This model has shown remarkable success on high-resolution images, and the following two papers introduced many of the concepts it uses:</p><ul><li><a href="https://arxiv.org/abs/1410.8516">NICE: Non-linear Independent Components Estimation</a></li><li><a href="https://arxiv.org/abs/1605.08803">Density estimation using Real NVP</a></li></ul><p>Here’s my quick and dirty reimplementation of Glow using tensorflow:</p><p><a href="https://github.com/nolan-dev/glow_reimplementation">nolan-dev/glow_reimplementation</a></p><h3>Part 1: Introduction</h3><p>For context, here are some common goals of generative models:</p><ul><li>Model the data distribution so that we can take samples and evaluate densities</li><li>Encode data as latent variables and decode the latent variables to exactly reproduce the original data</li><li>Use latent variables that encode meaningful information useful for downstream tasks</li><li>Produce subjectively high-quality data</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*yFG6mwNMcry4ncquUwbY9g.png" /><figcaption>Figure 1: Image to latent to image example. Face from <a href="https://openai.com/blog/glow/">https://openai.com/blog/glow/</a></figcaption></figure><p>Most generative models do not support all of these goals. However, flow-based models like Glow do, with some caveats. Unlike several other popular approaches, they attempt to make finding the density function tractable.</p><p><strong>Making the density tractable</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/197/1*3H8XSEqZVv4EpwOJQGq81Q.png" /><figcaption>Figure 2: Image modified from <a href="https://openai.com/blog/glow/">https://openai.com/blog/glow/</a></figcaption></figure><p>The exact definition is fuzzy, but usually, generative models attempt to model the distribution they are trained on in a way that allows sampling from it (generation). For <a href="https://arxiv.org/abs/1406.2661">GANs</a>, this just means learning to sample from that distribution. <a href="https://arxiv.org/abs/1312.6114">Variational autoencoders</a> allow sampling but also attempt to approximate the probability density function by maximizing the evidence lower bound. Autoregressive models and change of variables models, on the other hand, attempt to make finding the density function (or rather, the best density function in their hypothesis space) tractable by maximizing the likelihood of the training samples.</p><p>The following image from Ian Goodfellow’s NIPS 2016 tutorial illustrates how conventional generative models relate to each other in the context of maximizing the likelihood of a density function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/1*vfsp65R_q6M3xdSRJUmsSQ.png" /><figcaption>Figure 3: Image Credit: Ian Goodfellow <a href="https://arxiv.org/abs/1701.00160">https://arxiv.org/abs/1701.00160</a>. This blog focuses on change of variables models</figcaption></figure><p>Here’re some examples of how we might calculate the density function:</p><p>Say we have a multivariate distribution over a space <em>H</em>, with samples from that distribution <em>h </em>consisting of independent components <em>h_d. </em>We can calculate the density at <em>h</em> by multiplying the density of each component together:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/276/1*3BQilO8EKeGlXfeGC6YYaA.png" /></figure><p>For example, <em>H </em>might consist of two specific attributes of the face which we assume are independent: <em>h_1 </em>indicates if the face is smiling, <em>h_2 </em>indicates if the face is wearing glasses. To get the probability of a specific face that is wearing glasses and not smiling, we can multiply <em>p(h_1=not smiling)*p(h_2=wearing glasses). </em>The independence assumption reduces the problem to finding the density functions of the components.</p><p>Unfortunately, the components we have access to often aren’t independent, so (to avoid <a href="https://en.wikipedia.org/wiki/Naive_Bayes_classifier">naivety</a>) we have to incorporate the dependencies using the chain rule of probability:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/1*Vbp0BJbp5Lyberzw9dMWGQ.png" /></figure><p>Continuing with the previous example, if our dataset consists of before-and-after images of people trying on glasses for the first time, it may be that a person is more likely to be smiling if they are experiencing better vision for the first time (wearing glasses). Therefore, to get the probability of a specific face that is wearing glasses and not smiling, we multiply <em>p(h_1=not smiling|h_2=wearing glasses)*p(h_2=wearning glasses). </em>This calculation incorporates the dependency between the two components.</p><p>In the case of images, the probability distribution is over all images <em>X</em>, and we want to calculate the density at a specific image <em>x</em>. Calculating the density at an image requires accounting for the dependencies of the image’s components: the pixels. For example, pixel 2 in an image is usually very dependent on pixel 1. Except for borders, pixels that are close together tend to be part of the same object and are therefore highly correlated. Borders make up a small portion of a natural image, as they are nearly 1-dimensional features in a 2-dimensional space. On the other hand, a latent variable like ‘wearing glasses’ is often independent of a latent variable like ‘smiling.’</p><p>Nonlinear independent component estimation, when applied to images, aims to <strong>transform the dependent distribution over pixels <em>x</em> to an independent distribution over latent variables <em>h</em> by applying a function <em>f(x). </em></strong>The Glow paper, its predecessors, and this blog are mostly about finding the right function.</p><p><em>f(x) </em>must be invertible so we can go back to <em>X </em>from <em>H </em>(otherwise, we can’t generate samples). The invertibility requirement forbids lossy dimensionality reduction: the latent space must have as many components as the image space (usually height*width*3 color channels). Figure 4 roughly shows the concept of going from a complex distribution to a simpler one and vice versa.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*F0o1LqBd2KprDpXXt_BB0Q.png" /><figcaption>Figure 4: Loose, informal illustration of converting from a complex, dependent distribution to a simpler, independent distribution. Note that most distributions don’t look like snakes.</figcaption></figure><p>Once we have a function <em>f(x)</em> to map to from samples of <em>X</em> to an independent space <em>H </em>with a known probability density function<em>, </em>we could try to calculate the density of a specific image <em>x </em>using the following naive approach:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/1*XAh0mFy18-7xsDAabKnxpQ.png" /></figure><p>If you’re as rusty with change of variables as I was when I first read the Glow paper, step 4 may seem intuitive. However, it’s not entirely correct due to how continuous distributions work. I’ll go over why in the next part. If you are familiar with change of variables for integrals, it will be mostly review.</p><h4>Change of variables</h4><p>As a simple example, say <em>X</em> is a univariate distribution over beard size instead of pixels. In the illustration below (Figure 5), a light beard is most probable, with larger beards or no beards having lower probabilities. Following step 2, we can apply a function to convert <em>X</em> to <em>H: h = f(x).</em> Say <em>f(x)=x/10.</em> It doesn’t make sense to talk about independent variables when there is only one; this example is meant to illustrate a basic change of variable that can then be extended to include multivariate distributions. Figure 5 and 6 show the before and after of this transformation, with a brown curve indicating the probability density function displayed above the images:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ssza3x2l6ipx_izy2PzM2Q.png" /><figcaption>Figure 5: Modified images of Dr. Geoffrey Hinton generated at <a href="https://openai.com/blog/glow/">https://openai.com/blog/glow/</a> using a pre-trained Glow model</figcaption></figure><p>Apply <em>h = x/10.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/1*24oyS3ZJOyEEAY5tBJsiWw.png" /><figcaption>Figure 6: After applying h=x/10, probability density at h=0 is greater than the probability density at x=0. Images are clearly ‘denser.’ The height of the curve increases with the density of images.</figcaption></figure><p>The image that was previously at 10 is now at 1, and the image that was at -10 is now at -1. The image that was at 0 is still at 0, but it is clear that the probability density is higher at -1, 0, and 1 in the new image than it was at -10, 0, and 10 in the old image, just as the density of images has increased.</p><p>The area under the curve of a probability density function must always be 1, so lowering the width must increase the height and vice versa. This concept can be expressed more mathematically using the cumulative density functions <em>F, </em>which are calculated as the integral of the probability density function (pdf) from negative infinity to the point under consideration. The fundamental theorem of calculus indicates that the pdf is, therefore, the derivative of the cumulative density function. After we change from <em>x</em> to <em>h = f(x)</em>, we need to use the chain rule to calculate the new pdf. The following equation gives the actual pdf for <em>x</em> in terms of the pdf for <em>h:</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/1*Q8p8iOQ6uT09rvD7LeXlQg.png" /></figure><p>Therefore the process should be the:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*IhpcGq-4Tpv2-vW8R8ixaQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*emF7LSFQ_Gu8JJ3n280NbA.png" /></figure><p>Therefore the density at <em>x </em>can be calculated as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*dJXIIZ_e_iTLG56VGqK04g.png" /></figure><p><a href="https://math.stackexchange.com/questions/856654/why-absolute-values-of-jacobians-in-change-of-variables-for-multiple-integrals-b">This StackExchange post</a> includes answers with a good explanation for why we use the absolute value of the determinant.</p><p>For me, knowing that each step in an equation is mathematically valid does not always allow me to understand that equation intuitively. The following MSPaint diagram roughly demonstrates how the height of the new pdf needs to change to match the change in the ‘width’ of the space. My favorite youtube channel, 3Blue1Brown, has a video that superbly animates derivatives as stretching or squishing of space <a href="https://www.youtube.com/watch?v=CfW845LNObM&amp;t=3m7s">https://www.youtube.com/watch?v=CfW845LNObM&amp;t=3m7s</a>, which is an essential intuition for understanding change of variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OzTaleVBJtZVpCPZ3MdtCA.png" /><figcaption>Figure 7: Demonstration of change of variables. The original distribution (black curve, above) is mapped to a new distribution (red curve, below) using f(x). The lines that show the mapping also show the derivative of f(x) around the points that the function maps: if two points are farther apart in the new space, then the slope between them is greater than 1, if they’re closer together, then the slope is less than 1.</figcaption></figure><h4>Finding the function</h4><p>I haven’t mentioned yet why we need the exact density function to generate samples (clearly, we need the density function if we want to calculate densities). GANs, for example, only use it <a href="https://www.youtube.com/watch?v=HGYYEUSm-0Q&amp;t=44m22s">implicitly</a>. Up to this point, I’ve assumed the function that maps pixels to independent components is known, but this won’t be the case for a dataset with non-trivial images. Now, machine learning comes in: we can have a computer learn the function instead of writing it ourselves. With the density function, learning the parameters of <em>f(x)</em> can be done via maximum likelihood estimation. Specifically, we can maximize the log-likelihood of <em>f(x)’s </em>parameters θ to produce a distribution with independent components <em>p_H</em> for input <em>x</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*qwBQY0ltibqPVwI6J6RVRA.png" /></figure><p>If we choose the independent distribution to be a Gaussian with mean 0 and variance σ², the equation becomes</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*jFrOGBpfafjvyT9ryA7iuw.png" /></figure><p>Without the log determinant term, this is equivalent to minimizing the mean squared error between the output of the function and 0. In this case, the best θ is the one that causes <em>f </em>to map every value to the mode of the distribution (0), as illustrated in Figure 8. This mapping produces a Dirac delta distribution rather than a Gaussian:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*85DgIRF87Vml-eLsIwm6ZQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*wspFCWSzloL_lMrUkrq8vQ.png" /><figcaption>Figure 8: All points mapped to the mode of the target distribution due to the small slope, producing a Dirac delta distribution when slope converges to 0.</figcaption></figure><p>Here’s another way to understand how the log determinant term is required to find the right function:</p><p>If we want to map to a 0 centered Gaussian, the log-likelihood loss we minimize is <em>f(x)</em>²/2σ² (log of the Gaussian density equation with 0 mean). Without the log determinant, this means a function that maps all training data to 0 would have 0 loss (bad: Gaussians should have some variance). The function cannot do this if we maximize its slope near the training data: nonzero slope means not all points are mapped to the same location. Given this, it seems intuitive that the log determinant term should be related to variance in some way. Often when we maximize the MLE of a Gaussian, we can disregard constant factors such as the Gaussian’s variance; however, in this case, we cannot because the log determinant term is not multiplied by the same constant. Because the log-likelihood of a Gaussian is the mean squared error divided by 2σ², variance is effectively a weight term, with higher variance increasing the influence of the log determinant term by lowering the influence of the log density term. For the log determinant term to disappear completely, the variance would have to be 0, giving infinite weight to the log density term, which would produce the Dirac delta type pdf shown in Figure 8.</p><p>The next section on the “coupling layer” explains how we might create and invertible function with a tractable Jacobian determinant that is powerful enough to map natural images to their independent components.</p><h3>Part 2: Coupling Layer</h3><p>Hopefully, the previous section made it clear that we can create a generative model by using an invertible function with tractable <a href="https://www.khanacademy.org/math/multivariable-calculus/multivariable-derivatives/jacobian/v/jacobian-prerequisite-knowledge">Jacobian</a> determinant to map the input data to an independent representation. Given the complexity of some types of input data (images), it would be challenging to create that function manually. If the function is differentiable and parameterized by a sufficient number of variables, we may be able to use an optimization process to learn it. Neural networks have shown success at learning complicated differentiable functions, but they are generally not invertible with a tractable Jacobian determinant. The coupling layer fulfills all of these requirements: the layer is invertible, has a tractable Jacobian determinant, is differentiable, and incorporates a neural network powerful enough to handle complex input. It achieves this with a ‘triangular’ function structure, where we split the inputs and outputs into two parts. The first layer in the network needs to perform the initial split into <em>x_1</em> and <em>x_2</em>, which is usually done across channels or with a spatial checkerboard pattern, but to simplify the illustration, Figure 9 shows an example of splitting along the height</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/1*4ky_blHLQabB88a7NZZVxw.png" /><figcaption>Figure 9: Splitting an image to get x_1 and x_2</figcaption></figure><p>The inverse of this specific split is to combine the top and bottom halves into a single image. At the start of a coupling layer, “Key” data is passed through a neural network and combined with “Message” data. The “Key” is also passed along unmodified, which means we do not need to invert the neural network, as the next layer has a version of the network’s input. Coupling layers can be stacked to build more powerful functions that are still invertible with tractable Jacobian determinants.</p><p>Figure 10 shows a diagram of a coupling layer, the input is <em>x_1</em> and <em>x_2</em>, and the output is the ‘new key data’ and ‘new message data.’ The neural network, m, has influenced half that data. By stacking these layers, we can get a ‘deep’ representation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/1*XUoN3mPwl0ulGlMi-jVnyQ.gif" /><figcaption>Figure 10: Coupling layer (plus mix) used to build powerful invertible functions with tractable jacobian determinants. One frame shows the coupling layer, and the other shows its inverse.</figcaption></figure><p>g is an invertible function combining <em>x_2</em> and the output of <em>m(x_1) (m</em> can be a neural net). The diagram is equivalent to the following equations (without the mix function) which also demonstrate the invertibility of the layer</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*QuIJznCsPssBEb5clxLFMQ.png" /></figure><p>The triangular nature of the Jacobian for these equations means only the partial derivative of <em>y_2</em> with respect to <em>x_2</em> is needed, which does not require differentiating the neural network <em>m </em>(keeping the calculation simple)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/215/1*xXtPa4xks_bzLBViZBczsg.png" /></figure><p>Therefore, by picking the right <em>g </em>and mix functions, we can get an invertible function with a tractable Jacobian determinant even if <em>m</em> is highly complex.</p><p>g is often defined as the following simple function, to produce an ‘affine coupling layer’:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*Ah07GomE5tjCP0QfH32Slw.png" /></figure><p>The circle with a dot in the middle used to combine <em>x_2</em> and <em>m_2(x_1)</em> is the <a href="https://en.wikipedia.org/wiki/Hadamard_product_(matrices)">Hadamard</a> product, which is an element-wise multiplication. <a href="http://neuralnetworksanddeeplearning.com/chap2.html#the_hadamard_product_$s_\odot_t$">Backpropagation</a> also uses the Hardmard product.</p><p><em>y_1 </em>and <em>y_2</em> are ‘mixed’ with an invertible transformation. Without mixing, the “Message” data will never be passed through a neural net, and the network won’t learn a deep representation. The diagram below shows two options for this transformation</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*5iluZZEK7zRd-wsToipdKg.png" /><figcaption>Figure 11: NICE (top) vs. Glow (bottom) mixing layers</figcaption></figure><p>The first example is what the <a href="https://arxiv.org/abs/1410.8516">original</a> paper did: the ‘key’ and ‘message’ data swap position. The inverse is trivial, swap them back.</p><p>The second example is closer to how the ‘Glow’ paper implements the mixing operation. The paper points out that the ‘swap’ operation is a specific instance of changing the permutation of channels. It uses a more general permutation implemented as a square matrix transform with learnable weights, initialized as a rotation matrix. This transform is where “Invertible 1x1 Convolutions” from the title of the paper comes in: one way to do a channel-wise ‘permutation’ is 1x1 convolutions. 1x1 convolutions are convolutions with filter height and width of 1 and have been used in architectures such as the <a href="https://arxiv.org/pdf/1409.4842.pdf">inception network</a> to change the number of channels. In Glow, the number of channels must stay the same for the architecture to be invertible (unless we also alter the spatial dimensions to keep the total number of components constant).</p><h3>Conclusion</h3><p>I think an important takeaway from this type of model is that we shouldn’t assume the restrictions imposed by a particular approach are insurmountable. I can see myself assuming that the requirement for an invertible function with a tractable Jacobian determinant excludes the use of neural networks. This assumption turns out to be incorrect, and the solution is an effective approach to model the independent components of high dimensional data. The architecture is still restricted, however, which means models like GANs that have more flexibility have been able to <a href="https://arxiv.org/abs/1812.04948">produce high-quality results</a> with much less training time and memory usage. I’m also interested in gaining a better understanding of how generative models behave and how they might learn useful concepts in an unsupervised way. I discussed approaches to this in previous blogs <a href="https://towardsdatascience.com/animating-ganime-with-stylegan-part-1-4cf764578e">https://towardsdatascience.com/animating-ganime-with-stylegan-part-1-4cf764578e</a> and <a href="https://towardsdatascience.com/animating-ganime-with-stylegan-the-tool-c5a2c31379d">https://towardsdatascience.com/animating-ganime-with-stylegan-the-tool-c5a2c31379d</a> in the context of GANs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f47e566a73" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-ml-at-symantec/introduction-to-reversible-generative-models-4f47e566a73">Introduction to Reversible Generative Models</a> was originally published in <a href="https://medium.com/ai-ml-at-symantec">AI/ML at Symantec</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Animating gAnime with StyleGAN: The Tool]]></title>
            <link>https://medium.com/data-science/animating-ganime-with-stylegan-the-tool-c5a2c31379d?source=rss-352fdc0a3e4c------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5a2c31379d</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[generative-art]]></category>
            <category><![CDATA[towards-data-science]]></category>
            <dc:creator><![CDATA[Nolan Kent]]></dc:creator>
            <pubDate>Thu, 07 Nov 2019 16:35:18 GMT</pubDate>
            <atom:updated>2019-11-07T16:50:43.272Z</atom:updated>
            <content:encoded><![CDATA[<h4>In-depth tutorial for an open-source GAN research tool</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*tPCCnqPMA0h5rRkZ7cEnHw.png" /><figcaption>Visualization of feature map 158 at a layer with resolution 64x64</figcaption></figure><h3>0. Preface</h3><p>This is a tutorial/technical blog for a research tool I’ve been working on as a personal project. While a significant portion of the blog assumes you have access to the tool while reading it, I attempted to include enough screenshots that it should be clear how it works even if you do not have time to try it out yourself.</p><p>In the tutorial we’ll interact with a trained StyleGAN model to create (the frames for) animations such as this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*K2CIdiRe-197lqFoLojn3w.gif" /><figcaption>Spatially isolated animation of hair, mouth, and eyes</figcaption></figure><p>In the animation above, the transformations that change the mouth, eyes, and hair are mostly separate. This is often preferable to other methods (that I’m aware of) for creating talking animations with GANs, which may cause side effects such as hair loss:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*FIkVRGWkonMFiPdsecxeBw.gif" /><figcaption>Another animation we’ll create to demonstrate how changes in the ‘mouth’ attribute can influence other parts of an image. Note the hair thickening and thinning along the edges.</figcaption></figure><p>We’ll also build simple heuristic facial feature detectors by using feature maps at various layers in the network:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bgwkhHezq2ocany708eoKA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pSMVTR14gb7TwtX0NRXUuA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Du3rV7jSp_IlXPm0qEU-Yw.png" /><figcaption>Using feature maps at various layers to detect mouths, eyes, and cheeks</figcaption></figure><p>These detectors can then be used to automate meaningful modifications in order to isolated portions of images:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VQLpkfjhcqRi9crtVUubpQ.png" /><figcaption>These images generated were in a single batch without human intervention or a labeled dataset</figcaption></figure><p>None of these steps require labels for the training set, but there is a bit of manual work involved.</p><h3>1. Introduction</h3><p>You can download a compiled version of the tool from one of the following links:</p><ul><li><a href="https://drive.google.com/file/d/1cv2SiWQKtlC-XCAeAiGHh2C8XCAZe9xd/view?usp=sharing">GanStudio_x64_v1.zip</a></li><li><a href="https://mega.nz/#!VCIRyIRI!t_g2OQYkuqtPAdd5wgsSHUEYhYI47ip84jydGZMI-bg">MEGA</a></li></ul><p>sha256: ec2a11185290c031b57c51edb08bf786298201eb36f801b26552684c43bd69c4</p><p>It comes with a model trained on an <a href="https://www.gwern.net/Danbooru2018">anime dataset</a> (which a <a href="https://www.gwern.net/Faces">number of projects</a> are based on). Unfortunately, due to the nature of the dataset, there is a lack of gender diversity. I’m currently trying to train a model to produce higher quality masculine images, but it will take a while to complete. The dataset also contains NSFW images, and while I’ve generated thousands of images and never encountered anything NSFW, I didn’t vet every image in the training set (the risk may increase with large modifications to feature maps in early layers). If you run into problems, you can create an issue in <a href="https://github.com/nolan-dev/GANInterface">https://github.com/nolan-dev/GANInterface</a> and I’ll try to respond.</p><p>This blog has two parts: a basic tutorial, and an advanced tutorial. The basic tutorial demonstrates how to use the tool and shouldn’t require much technical knowledge to complete (though it does include some technical explanations for those interested). The advanced tutorial demonstrates how to customize the tool and is more technical — you should be familiar with <a href="https://towardsdatascience.com/applied-deep-learning-part-4-convolutional-neural-networks-584bc134c1e2">feature maps</a> in convolutional neural networks, for example.</p><p>I introduced the tool in a previous blog and shared the source code, but getting it working from there is complicated and requires a model trained with my custom StyleGAN implementation. I hope supplying a compiled version of the tool and a pre-trained model makes it easier to try out. The tool is part of a larger project (including a reimplementation of StyleGAN) which the previous blog discusses, but reading it is not a prerequisite for this blog. Here’s a link if you’re interested though:</p><p><a href="https://towardsdatascience.com/animating-ganime-with-stylegan-part-1-4cf764578e">Animating gAnime with StyleGAN: Part 1</a></p><p>As this is a research tool, I’ve been adding and subtracting features regularly to get a better idea of how the model works and the best way to interact with it. There are a lot of minor features, but major ones include:</p><ul><li>Modify the latent vector that produces an image in order to interpolate between images, express certain features, and trade off between quality and variation (truncation trick)</li><li>Modify feature maps to change specific locations in an image: this can be used for animation</li><li>Read and process feature maps to automatically detect meaningful features</li><li>Automate all of the above by creating batch jobs</li></ul><p>Like the previous blog, my goal for this blog is to get others’ perspectives on the topic, detail my experience working on the project, and receive constructive criticism/corrections. The tutorial format of this blog is meant to mitigate the underdeveloped UI of the tool, and make it possible to use without dealing with the messy source code. Unfortunately, it’s Windows only, however it has been tested on a free tier Windows AWS instance (Microsoft Windows Server 2019 Base, image generation will be slow with this however).</p><h3>2. Basic Tutorial</h3><p>A quick note before we get started: I modified the tool while writing this, so some screenshots are slightly different from the current version, but everything is in roughly the same place.</p><p>Once you’ve downloaded and opened the zip file linked above, you’ll be presented with several files/folders:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/1*GPdiOCfvnSKMShNML77bmA.png" /></figure><p>I’ll explain some of these in more detail in the advanced section (3.3), but at this point the only important file is GanStudio.exe. When you run it, click ok to the disclaimers (and hopefully don’t get any errors), you’ll be presented with something like the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0T3OM1t1TggXnImIYCmuOg.png" /></figure><p>Due to the complexity of the UI, the first time I reference a part of the tool in this tutorial I’ll have a nearby screenshot with the relevant part outlined in red. Many of the UI elements have tooltips.</p><p><strong>Setup:</strong></p><p>Using this tool involves interacting with windows explorer, and it’s easiest to view generated files with “Large” or “Extra large” icons. Select one of these by right clicking on an explorer window and selecting ‘View’:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*kPt3820Y8OLJ1pF4B_6tJw.png" /></figure><p>In many cases it is also helpful to sort images by date modified, also achieved through the right click menu:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*oXvrSl4yseY5fMfOEjIKZQ.png" /></figure><p><strong>Generate New Image</strong></p><p>To test out image generation, click Generate New (3). This will produce a new latent code, and display the image for it. Note generating the first image usually takes longer than subsequent images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kkc9pYeFHKSHKj5H-jbppQ.png" /></figure><p>The image was created randomly, but was interpolated to be close to the ‘average’ image. This results in higher quality images, but reduces variation. If you have the quality slider (1) in the same place as the above image, your image will likely be similar: a brownish haired girl with purple and/or blue eyes.</p><p><strong>Load Image</strong></p><p>To keep images in this tutorial consistent with what the tool will produce, I’ve provided a sample image. Click ‘Import Image’ (above, 2). This will create an open file dialog box in the ‘portraits’ directory. Navigate to the directory above ‘portraits’ and select ‘tutorial_files’:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/1*i_MxovtR6ZQ-S_OSKopv0Q.png" /></figure><p>Double click on sample_01.png to load it. All images you generate are saved to the ‘portraits’ folder and you can load them again using this method.</p><p>GANs cannot normally load an arbitrary image, however this tool will append the latent code that generated the image to every PNG file it writes to the disk. The Import Image button reads the latent code that was written to the image you select. As long as the tool has loaded the model that generated the image it will be able to recreate it.</p><p><strong>Modify Attribute</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sx57Pg8_ebOdknS4W3_a1Q.png" /><figcaption>Hover over attribute label that have been cut off to see full name</figcaption></figure><p>To start modifying attributes, select the ‘Attributes_0’ tab (above). Attributes include hair/eye color, intensity of background, accessories, and mouth state (smile/open). Moving a slider that corresponds to an attribute to the right will increase the influence of that attribute on the image, moving it to the left will decrease said influence. Some of them work better than others. After selecting a location, press ‘Update This Image’ (above). Here are some examples:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jDq-fvrsCJtXLj8nkZXb5g.png" /><figcaption>Left to right/top to bottom: Positive open_mouth, positive skin_tone, negative background and negative black_hair, positive blonde_hair and negative smile</figcaption></figure><p>One downside of modifying attributes in this way is that they are not always spatially isolated; modifying an attribute that should only influence one part of an image will also influence other parts. This is particularly problematic if we want to create animations. To see the problem in action perform the following steps (screenshot below for reference):</p><ol><li>Scroll down to the ‘open_mouth’ slider</li><li>Move it to to the right</li><li>Press ‘Update This Image’. The mouth should now be open slightly</li><li>Select <em>Batch-&gt;Attribute-&gt;Spectrum</em></li><li>Select ‘OK’ to generate 5 images, and ‘No’ to the ‘Slide Past 0?’ prompt.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5i_eWae-G3ofXRYUjvZ4Cw.png" /></figure><p>This will produce 5 images with the ‘open mouth’ attribute shifting from 0 to the selected location on the slider:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*xJ87_0p1MexWg228pG4Etg.png" /></figure><p>Throwing these into a gif generator produces the following animation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*FIkVRGWkonMFiPdsecxeBw.gif" /><figcaption>Frame order based on the first number in the filename: 0, 1, 2, 3, 4, 3, 2, 1</figcaption></figure><p>As you can see, features all over the image change even though we only selected an attribute related to the mouth.</p><p><strong>Modifying Specific Locations</strong></p><p>In this section we’ll make a modification to just the mouth, without altering other features. Unfortunately, at this point images imported with the Import Image button will not reflect the changes made here.</p><p>Repeat the instructions in the <em>Load Image</em> section to get back to the base image (or reset the attribute sliders and update). Well use the ‘Spatial’ tab (below) to modify isolated parts of an image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SeBxjL0dxU_Oud1dTQOFIw.png" /></figure><p>The UI is complicated, however for this section we’ll only be using a couple parts. The first thing we need to do is indicate which part of the image we want to change:</p><ol><li>Select the ‘mouth_open’ tab.</li><li>Click, hold, and drag the cursor across the mouth before letting go</li></ol><p>This makes a selection around the mouth, and ensures that our changes to the image will only influence the area selected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/208/1*iiwcIWMhLJBA1PV8LbWNFg.png" /><figcaption>Click on locations within these squares to create them on your image, or click and drag across this portion of the image</figcaption></figure><p>This will produce a light green square, unless “Swap green for blue in visualization” is selected. I’ll have that option selected for this tutorial to improve visibility, and hopefully be more colorblind friendly when we start dealing with the red boxes that indicate negative influence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/1*z421eggV5a8eWr0zDsxYXw.png" /></figure><p>If you selected a location in error, you can remove squares by holding ‘control’ when you make a selection:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/268/1*C-xb4VtdZykeC4pKdRR3_w.png" /><figcaption>Remove an undesired selection. Click and drag with ctrl held to remove multiple selections.</figcaption></figure><p>The following are all of the ways you can ‘draw’ on the image. Some of these aren’t needed yet, but will be useful later:</p><ol><li>Left click to produce a box with size depending on the selected resolution (more on that later). This box indicates a positive influence at that location and will either be green or blue depending on your settings.</li><li>Right click to produce a red box with negative influence at that location</li><li>Ctrl+click to erase a box at a location</li><li>Click and drag to draw a large rectangular area when performing any of the above</li><li>If ‘No Selection Overlap’ is not checked, you can select the same location multiple times to increase the (positive or negative) magnitude of the selection. This is visualized as higher color intensity and thicker boxes.</li></ol><p>With the mouth selected, move the slider below the ‘mouth_open’ tab to the right until the number on the bottom left is around 100. This slider is the “Feature Map Multiplier Slider”, which influences the active tab exponentially as it is moved to the right (positive influence) or left (negative influence). With the number in the bottom left of the slider set to around 100, select ‘Update This Image’:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UskSJnrMLBX7a6Ji1Ji_Ow.png" /></figure><p>This should produce the following image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/1*D4ziBGBrN4Y78fIJelkzKA.png" /></figure><p>As the name of the tab would imply, this opened the mouth. Let’s try and animate with this method. Select <em>Batch-&gt;Fmap-&gt;Combinatoric </em>(I’ll expand on why it’s called that in the advanced tutorial):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BlGsKgxR38UQTgzdtvsNFg.png" /></figure><p>Select 5 for images to generate:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/521/1*eqK_EYHEA0q2dDKD3yM4Ow.png" /></figure><p>Select 0 for the start point. The batch will consist of 5 images with the slider regularly spaced between the start and end point (0, 20, 40, 80, 100). Because the mouth is closed by default in this image, a start point of 0 (no influence) means closed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*knJbArVQIPktK8n0ndBUTQ.png" /></figure><p>This will produce 5 images with less spatial entanglement than the attribute method:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*GjX_RRPr5b41lj28UdQ4Sw.png" /></figure><p>A gif generator produces the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*1zz6gbQxLakUPXP4xwYlNg.gif" /></figure><p>This same process can be used with the other tabs, and different tabs can be combined. If you’re using the <em>Combinatoric </em>batch generator, you’ll need to keep the multiplier bar at zero for all tabs except the active one to avoid producing combinations of multiple tabs. This can be done by pressing ‘Set All to Zero’ before changing the active tab’s multiplier:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/336/1*EBjFrONnnpaZ_1Dma6sV1g.png" /></figure><p>Here’s a list of some of the possible changes. Note that using large multipliers has a good chance of producing strange artifacts:</p><ol><li>red_or_blue_eyes</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tKZX9Ilo9c0YyjubferMgA.png" /></figure><p>With the following start and end points:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/443/1*KgLeRiT60_77sSaXCoTaBQ.png" /><figcaption>The dialog actually says Start Point, this is an old screenshot. The start point corresponds to the first image that gets generated in the batch, and the end point corresponds to the last.</figcaption></figure><p>Produces:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*laRgCCutG6IudWvzMM0BcQ.gif" /></figure><p>2. blush</p><p>Settings:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*vwf0_mu6lz3PcSZ0IqGpXA.png" /><figcaption>If the screenshot and the tool disagree, believe the tool. This is an old screenshot (should say Start Point instead of End Point)</figcaption></figure><p>Animation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*yLrVWopYYi1yI5PkZgk8MQ.gif" /></figure><p>3. hairband</p><p>Settings:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ml9P56rM9313W-xEp_2E6A.png" /><figcaption>I actually had the variable for this prompt called ‘startValue’ originally, don’t know why I made the prompt say End Point. You may have guessed this is an old screenshot</figcaption></figure><p>Animation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*7LvWLKixhq5JkKf-hWdnWw.gif" /></figure><p>4. hair</p><p>Settings:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WsxyXYeruTviT-9XRHAg7w.png" /><figcaption>You’d think it would make sense to start at the low number and end at the high number, but actually this is an old screenshot and End Point should be Start Point. It will start at 100 and shift down to -257</figcaption></figure><p>Animation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*oDjPoLIyamV0UFCrGlTsyw.gif" /><figcaption>These settings slightly influence the mouth because the effective receptive field of convolutions late in the network cover a lot of spatial locations in early layers.</figcaption></figure><p>5. ice_is_nice (???)</p><p>Settings:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e5Jnnc1BNDUXG3bxv-vsaA.png" /><figcaption>We’re selecting the entire image here. For scenarios like this, I’d advocate for the click and drag method over clicking 4096 times with extreme precision. Also: it should say Start Point instead of End Point</figcaption></figure><p>Animation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*0H-1_lvqVg2g53wk4G5iiA.gif" /><figcaption>Example of a modification late in the network: details are influenced, but overall structure stays the same</figcaption></figure><p><strong>Other things to try:</strong></p><ul><li>Batch generate new images with the quality and attribute bars at various locations:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/222/1*BLC3G93gqPy-tc9xmveC2Q.png" /></figure><ul><li>Use ‘Set As Base Image’ (below) to make the quality bar interpolate between the current image (instead of the average image) and new images. This can be useful for finding new images that are slightly different from the current image when combined with <em>Batch-&gt;New latents </em>(above).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/365/1*3Kjy-BIkJnYyO203uXPWKw.png" /></figure><ul><li>Use ‘Toggle Prev’ to quickly switch between the current image and the previous one to inspect changes.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/506/1*OfhoPIyYa5e0ytmH2hV2ZA.png" /><figcaption>Focus on the red box, not the blue box around ‘Toggle Show Selection’ which I clicked earlier. At the moment toggle show selection doesn’t even work.</figcaption></figure><ul><li>Use <em>Misc-&gt;Interpolate Between Two Images</em> to generate a spectrum of images in between two existing images.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*hrkQdHBcuAn6MSu0oO4Nwg.png" /></figure><h3>3. Advanced Tutorial</h3><p>This section assumes some familiarity with convolutional neural networks and StyleGAN.</p><h4>3.1 Finding New Features</h4><p>Tabs in the ‘spatial’ section (mouth_open, hairband, etc) correspond to values added to specific feature maps at a specific layer. Try selecting the mouth_open tab. In the combo box above the tabs, it should show a resolution: 16x16. Early layers in StyleGAN have low resolution feature maps, while later layers have high resolution feature maps (resolution regularly doubles). As the images we generate are 256x256 pixels, the layer that corresponds to 16x16 is early in the network. To view which feature maps are modified by the mouth_open tab, press ‘View All Fmap Mults’ with ‘Filter Zero’ checked and select the ‘Feature Map Input’ tab:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gSMEy4IkQHhqw_gahV_-ww.png" /></figure><p>This means spatial locations selected in the image by clicking on it are multiplied by -2, then multiplied by the feature map multiplier slider, and the result is added to feature map 33 on the layer that has a 16x16 resolution.</p><p>Some tabs influence multiple feature maps:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*xxnb27C0XpLeafEfFo_urg.png" /><figcaption>The feature maps influenced by the ‘hairband’ tab, viewed by clicking on ‘View All Fmap Mults’</figcaption></figure><p>I found these multipliers manually by playing around with the tool. I used two methods:</p><ol><li>To modify an existing attribute in an image (for example, mouth), I use trial and error by modifying different feature maps and seeing which one produces the desired result.</li><li>To add an attribute (hairband), I see which feature maps are active when that attribute is present.</li></ol><p>In the next two sections I’ll walk through examples for these methods.</p><p><strong>Method 1: Modify the mouth</strong></p><p>Given that the network can generate images with open and closed mouths (required to fool the discriminator) and that feature maps at each layer are representations of the final image, it makes sense that modifying the feature maps could be used to open or close a mouth. However, there’s no guarantee that modifying a single feature map will result in a meaningful change — it may be that we need to modify a combination of many feature maps to get the desired result. That said, single feature maps are easier to work with (at least with the current tool), so seeing how each one influences an image can serve as a starting point.</p><p>Here’s how I found feature map 33 to open/close mouths. First, I added a 16x16 tab with the ‘Add Tab’ button (below). I chose this resolution because it produces boxes that are reasonably mouth-sized. Smaller resolutions would change areas beyond the mouth, while larger resolutions often result in changes with finer granularity than opening or closing a mouth (choice of resolution is heuristic at this point). By clicking ‘View All Fmap Mults’ again we see that no feature maps are set for the new tab. Then I moved the slider to around 190, again a heuristic decision based on past experience with the model. Finally, as we did before, I selected the two boxes that contain the mouth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QGo1rj1-gCs0HwG-pMPUAA.png" /><figcaption>Adding a new tab, which initially does not influence any feature maps</figcaption></figure><p>Then, I selected <em>Batch -&gt; Fmap -&gt; Axis-aligned</em>, and selected 512 images to get generated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*sKbb4UA5xMNNsnoLt94elA.png" /></figure><p>This will actually produce 1024 images, as for each feature map it both adds and subtracts the value specified in the multiplier bar (190 in this case) to the spatial location marked in the image (the mouth). Batch generation pops up a window that shows how many images have been generated and allows you to interrupt the process. Clicking on the counter next to ‘Generating image’ opens the directory to which they are being written:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/292/1*-WXFproR54o7fW8ZSvCOmw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Eub_iEmFcbqsYnQ3KYZ43Q.png" /><figcaption>The number prepended to the file names is the feature map that was modified</figcaption></figure><p>The sample that starts with ‘33_n_sample’ (n stands for negative) clearly has an open mouth, while ‘33_p_sample’ does not. This means that when 190 was subtracted from feature map 33 around the mouth, the mouth opened.</p><p>I set feature map 33 to -1 using the ‘Set Fmap’ box (below). This makes it so moving the slider to the right will open the mouth (which feels more intuitive than setting feature map 33 to 1 and having the tab named ‘mouth_close’), and I renamed the tab to mouth_open using the ‘Rename Tab’ button. The Save Tabs button next to Rename Tab can be used to save the tabs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q-I2L4Z7Ha8a410MPBaNYQ.png" /></figure><p><strong>Method 2: Add a hairband</strong></p><p>This method relies on finding existing images with the desired attribute. This requires a base of images to work off of, which can be generated with <em>Batch-&gt;New latents</em>. In these cases I’ll usually move the Quality bar a bit past the middle to ensure there’s a reasonable amount of variation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/278/1*yqMytbPqOtpRHj6Cd7VHKQ.png" /></figure><p>It may take a few hundred samples to get several samples with hairbands. I added one to the tutorial_files directory which I’ll load for this tutorial (sample_02.png). Before loading it, create a 16x16 tab and make sure ‘Fmaps To Get’ is set to ‘All’ or ‘Current Tab’ (below). These options get and store extra output from the network when a new image is created: the feature maps for the current tab, or for all tabs. This can slow things down a bit, so it’s not the default option (also as of this writing it has something like a 0.5% chance of causing a crash, relevant for large batches).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B0qMUELhjA6Om6xblB1ofQ.png" /></figure><p>Then, do the following:</p><ol><li>Select boxes around the hairband</li><li>Under ‘Add View Feature Map Buttons’, select ‘Sort by Similarity To Selections’</li><li>Press ‘Update This Image’</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4p-QWCgVj5_wbIpFzmc_Mw.png" /></figure><p>This adds a bunch of buttons to the ‘Feature Map Output’ tab:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*gO_W51bf4jDL8Wys7iXeGw.png" /></figure><p>These buttons correspond to feature maps. They are sorted by magnitude of the dot product of the feature map and the selection on the image (after they’re flattened). This makes it so feature maps with large magnitudes around the hairband will show up earlier in the list of buttons.</p><p>Here’s an example of the 310th feature map. Blue corresponds to positive values, red to negative. The larger absolute value at a location, the thicker and more saturated the box drawn there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/1*Ki2LOjDpR8KoiL7OhxILTw.png" /></figure><p>This feature map seems to have large positive values around the hairband, but also around the mouth. While it’s clearly used for more than just hairbands, we can try modifying it to see what the result is. Set feature map 310 to 1 for that tab, erase the selections around the hairband (ctrl+click and drag), and load sample_01.png again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/278/1*doKsxZNH8tUOFL12spWl1g.png" /></figure><p>Try selecting hair, increasing the weight to around 100, and updating the image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Zi7R344X6jjOlYVZd6le2w.png" /></figure><p>…not has much changed. However, as we didn’t produce any weird artifacts, it may not hurt to increase the magnitude beyond 100.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FuzNH_aTkEkyh-CVXQBWxw.png" /></figure><p>Around 600 we do get what looks like a hairband. My initial hypothesis is that one reason we need to use a large magnitude is because hairbands are uncommon.</p><p>For the hairband tab included with the tool I set several other feature maps, which were active around the hairband in example images, to 3. Setting them to a number larger than 1 helps normalize the expected range of the tabs so that setting the multiplier to around 100 should express the wanted attribute.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/1*dthfqJmkPBmvkOvvaGiLyg.png" /></figure><h4>3.2 Automatic Feature Detection</h4><p>One problem with the way we’ve been modifying attributes is that it requires manually selecting the squares we want to change. This is definitely a downside when compared to modifying the latent vector in the ‘mouth_open’ direction, which did not require us to know the mouth’s location (even though it also modifies non-mouth features). This prevents the method from scaling well; while easier than drawing, every modification still requires human intervention. However, as we saw in the previous section, some feature maps correlate with the location of attributes: maybe feature map 310 could be used to generically detect hairbands, for example. Let’s see if we can find a way to detect the mouth in an image using just a linear combination of feature maps.</p><p>First, repeat the process used to get feature maps active around a hairband, only this time select the mouth:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lr7uFUwr7CKEePHuCIo80w.png" /></figure><p>Like before, we can click a button to show a feature map:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*-GryiNlikBlnntmdUKLGGA.png" /></figure><p>This method is a bit slow, however, when it comes to viewing and comparing a large number of feature maps. Instead, type 20 into the text box below the ‘Add From Output’ button and press the button. Then, press ‘View Multiple Fmaps’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*0KtHyuEK8iiHekPhEFyJzg.png" /></figure><p>‘Add From Output’ with 20 adds the feature maps from the first 20 buttons to the ‘Fmaps’ text box, and ‘View Multiple Fmaps’ displays them all side by side.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OIEdY3FbH1SpQ1OVvXNTdA.png" /></figure><p>The first 4 (along with several others), which correspond to feature maps 234, 34, 370, and 498 all look like they could be mouth detectors. However, we don’t know if they consistently detect the mouth for new images. To test this, we can generate several new images with the Quality bar to the right of the center for decent variance. First make sure ‘Record Feature Maps’ is checked. Use ‘Reset History’ to clear existing records. Also, make sure ‘Fmaps To Get’ is set to ‘Current Tab’ (the tool does not record history for all resolutions, just the resolution that corresponds to the current tab). Then, we can generate a number of new images by using <em>Batch-&gt; New latents </em>and the tool will record their feature maps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lR5fNxhm-VB7Bj2RnRZydw.png" /></figure><p>In this case I’ll generate 10 new images, which will be different from any of yours. To view the same feature map for all the images, type the feature map into the Fmaps box and select ‘View History’. I’ll do that for each of 234, 34, and 370.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*fRxMb1PIfiOdbgrPUXdhDQ.png" /></figure><p>234:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6L3vSplEWLMUsJLqINnc5w.png" /></figure><p>34:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0nzOgfnJnYiFXS4X5kApBA.png" /></figure><p>370:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bgwkhHezq2ocany708eoKA.png" /></figure><p>It doesn’t hurt that the mouth doesn’t change position much, but these feature maps do seem to track it reliably.</p><p>The same process can be applied to other attributes at other layers. Here are some examples:</p><p>Eyes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pSMVTR14gb7TwtX0NRXUuA.png" /></figure><p>Cheeks:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Du3rV7jSp_IlXPm0qEU-Yw.png" /></figure><p>Background:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nJE4ZGG9JM0oOO-IPIgs6Q.png" /></figure><p>In many cases I get the most consistent results by combining multiple feature maps. To do this, I made python scripts callable from the tool (which could be useful for future features as well), as I’d rather do multidimensional data processing with numpy. The tool uses the PATH environmental variable to find pythonw.exe, so that will need to be set before running the tool. The scripting functionality is the newest feature in the tool and even less developed than the rest. Here’s an example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*RKMDDRCHL_EocMsw-vmqMA.png" /></figure><p>spatial_map.py is stored in the ‘scripts’ directory, and you’ll need to install its dependencies to use it. The tool passes the path to a directory where the feature maps specified in ‘Script Feature Maps’ are written. It then combines those feature maps and outputs the result, which is read by the tool and used to make selections in the image. Here’s an example that uses some of the feature maps we found earlier which correlated with mouth location:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a6aj8FTgu8aQb8-LHumR0g.png" /></figure><p>Moving the mouth slider to ~100 and updating the image opens the mouth like normal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/277/1*kqR_VLjwLYFvQbCU9lBGvg.png" /></figure><p>This lets us automate image creation with certain attributes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WALB3yC0_HSqrXKNL0t6sQ.png" /></figure><p>By selecting ‘Run and Modify’ under ‘Run Script When Generating’, setting mouth_open to around 100, and generating new latents, we can ensure new images have an open mouth. By setting mouth_open to -100 and applying the settings to the directory with open mouth images we can generate the same images with closed mouths.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*stYkX7xcPSeV0S4Z-S8SXw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KAzzFM08cpx6zgby25ZBVg.png" /></figure><p>Applying settings to a directory can be done with <em>Misc-&gt;Apply Current Sliders To Directory </em>and selecting the directory with generated images:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/437/1*mTzcFGnAMvUbudzG-Ty8VQ.png" /></figure><p>Alternatively, this can be achieved in one go using the <em>Batch-&gt;Fmap-&gt;New Latent Combinatoric</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xt_g6Oe-6vUSQXrZCZyNLA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VQLpkfjhcqRi9crtVUubpQ.png" /></figure><p>Note that this option will do a combinatoric modification based on all tabs with non-zero multipliers. For example, if the ‘hair’ tab has a non-zero multiplier and we select 3 ‘hair’ images to be generated and 2 ‘mouth_open’, it will create 6 variations of each image with the following attributes:</p><ul><li>short hair, closed mouth</li><li>short hair, open mouth</li><li>medium length hair, closed mouth</li><li>medium length hair, open mouth</li><li>long hair, closed mouth</li><li>long hair, open mouth</li></ul><p>A couple more points on scripts:</p><ul><li>You can add a multiplier in front of feature maps to change the influence of various maps when they’re combined.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/242/1*zgd3fsA6ZZbQg_w2wuCkCA.png" /></figure><ul><li>The resolution of feature maps used to detect facial features does not need to match the resolution to which we input data. The script should resize as needed:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sj2sTHS9CiBZ2thITICOjg.png" /></figure><ul><li>There are other potential uses for scripts, such as recording a representation of the feature maps for analysis after the tool is closed.</li></ul><p>This covers most of the tool’s functionality. In the next section I discuss the architecture in more detail.</p><h4>3.3 Tool Details</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GsM_Nr3q0lbcMIl0XON87Q.png" /><figcaption>From TensorFlow python implementation to generating images with the tool</figcaption></figure><p>There are 4 main components of the tool:</p><ol><li>TensorflowInterface.dll: loads the model using the tensorflow C api</li><li>GanTools.dll: loads TensorflowInterface.dll into a C# environment</li><li>GanStudio.exe: loads GanTools.dll in order to interact with the model</li><li>data files: graph.pb (the model) and the files in data_[MD5 of graph.pb] which contain model-specific information and are loaded to aid interaction.</li></ol><p>These are all visible in the zip file:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/1*GPdiOCfvnSKMShNML77bmA.png" /></figure><p>Images are written to the ‘portraits’ directory, images marked as favorites are written to the ‘favorites’ directory, and the ‘scripts’ directory contains python scripts the tool can load. The rest of the files are libraries used by other components of the tool.</p><p>The data directory contains several files:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iFF0i9cqwvpz2SvlIZ9N2w.png" /></figure><ul><li>saved_fmaps includes tab (mouth_open, hairband, blush, etc) data that gets loaded when the tool starts. Tab data includes which feature maps get modified, script information, and multipliers.</li><li>attributes.csv contains vectors used by the ‘Attributes_0’ tab to modify the latent vector in meaningful ways.</li><li>average_latent.csv contains the average (intermediate) latent value generated by averaging around 10,000 latent values generated by the mapping network. It is used by the Quality/Uniqueness bar to apply the truncation trick.</li><li>latents.csv includes latent values for all generated images. It was more important before I started appending latent values to images, but it can still serve as a backup.</li></ul><p>All of these are model-specific, which is why I tied the directory to the model by appending the graph.pb file’s hash to the directory name.</p><p>TensorflowInterface.dll is the component that loads graph.pb, and it uses <a href="https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-windows-x86_64-1.14.0.zip">tensorflow.dll</a> to interact with it. It does this through 3 main functions which it exports:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*1rlE1DxBDo1TXdECx3u7hw.png" /></figure><p>Much of this interaction is done using the names of tensors in the model. Though the source code is available, I think one of the best ways to get these names (and a better understanding of the model) is through a tool like <a href="https://github.com/lutzroeder/netron">netron</a>. In the following image, netron is used to examine the inputs to an add operation that incorporates the feature map modifications this tool makes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*3G0kqBm21nj1UzKc-r_LyQ.png" /><figcaption>Operations to allow adding/subtracting values in feature maps (screenshot of <a href="https://github.com/lutzroeder/netron">Netron</a>)</figcaption></figure><h3>4. Conclusion</h3><p>Combined with the previous blog in this series, I’ve written a fairly comprehensive record of the work I’ve done for the project so far. Two exceptions that these blogs do not cover in detail are the ability to generate 512x256 (height x width) images and the model I trained with more male images.</p><p>The model used in this blog can actually produce 512x256 images, however I sliced off the bottom early on in the network:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/769/1*VFP2R87y86FHlInlZzYbYQ.png" /><figcaption>Using netron to view graph.pb</figcaption></figure><p>This improved image creation speed and reduced the size of the UI, but most importantly it kept the images suitable for all audiences.</p><p>I originally thought the model with male images was worse due to the increased variance and lower average quality in images, as to include a significant number of male images I had to reduce the ‘favorites’ threshold (see original blog). However, when I was updating the code for my StyleGAN reimplementation I noticed I had introduced a bug (<a href="https://github.com/nolan-dev/stylegan_reimplementation/commit/da7c6abd039f1547f3d60c38be356b160e1dd120">here’s the correction</a>) when I was implementing style mixing (not yet included in the tool). The bug meant that the weights used to convert from intermediate latent to style were shared between two adaptive instance normalization layers. This reduced the capacity of the network, and could have been a reason that the new model was worse than the old, less diverse model (used in this blog). This is one of the most annoying types of bugs to deal with: it doesn’t prevent the model from working, but it can reduce performance in a way that isn’t really noticeable until the model is done training, and the reduction in performance may be attributed to other factors. Andrej Karpathy put it very well in this series of tweets:</p><h3>Andrej Karpathy on Twitter</h3><p>To get neural nets to work one must be super-OCD about details. With bugs nets will train (they &quot;want&quot; to work), but work silently worse.</p><h3>Andrej Karpathy on Twitter</h3><p>The &quot;move fast &amp;amp; fix stuff until it compiles then it&#39;s probably fine&quot; approach is inadequate when each bug silently subtracts 2% accuracy.</p><p>For personal projects, including the tool discussed in this blog, I prefer using the “move fast &amp; fix stuff” approach (malware analysis made me almost enjoy debugging). However this does not work for implementing deep learning models, and while I tend to move much more slowly to avoid mistakes like this when using TensorFlow, sometimes they still pop up if I’ve spent too much time on a different part of a project. This is something I’m actively trying to improve on, and one strategy I’ve found helpful is viewing the model’s graph in TensorBoard or netron to get a different perspective.</p><p>As this tool is mostly based on viewing and modifying feature maps, I’m interested in adapting it to work on generative models other than GANs. It would be great if models that can take arbitrary images as input could demonstrate the same type of unsupervised facial feature detection in their internal representations as StyleGAN does. I also want to compare this approach to animation with <a href="https://nvlabs.github.io/few-shot-vid2vid/">video-to-video synthesis</a>, and do more research into other work on using generative models for animation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5a2c31379d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/animating-ganime-with-stylegan-the-tool-c5a2c31379d">Animating gAnime with StyleGAN: The Tool</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Animating gAnime with StyleGAN: Part 1]]></title>
            <link>https://medium.com/data-science/animating-ganime-with-stylegan-part-1-4cf764578e?source=rss-352fdc0a3e4c------2</link>
            <guid isPermaLink="false">https://medium.com/p/4cf764578e</guid>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[anime]]></category>
            <category><![CDATA[generative-art]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Nolan Kent]]></dc:creator>
            <pubDate>Sun, 27 Oct 2019 18:59:17 GMT</pubDate>
            <atom:updated>2019-11-07T23:10:07.466Z</atom:updated>
            <content:encoded><![CDATA[<h4>Introducing a tool for interacting with generative models</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*uiBC50ZJD4UPNNzfVmWWMQ.png" /><figcaption>Figure 1: Modifying spatial map(s) at a single location to produce an animation</figcaption></figure><h3>1.1: Preface</h3><p>This is a technical blog about a project I worked on using <a href="https://en.wikipedia.org/wiki/Generative_adversarial_network">Generative Adversarial Networks</a>. As this was a personal project, I played around with an anime dataset that I wouldn’t normally use in a professional environment. Here’s a link to the dataset along with a detailed write-up about models that use the dataset:</p><ul><li><a href="https://www.gwern.net/Danbooru2018">Danbooru2018: A Large-Scale Crowdsourced and Tagged Anime Illustration Dataset</a></li><li><a href="https://www.gwern.net/Faces">Making Anime Faces With StyleGAN</a></li></ul><p>Most of the work I did was purely for learning purposes, but some of the more interesting results I ended up with were mouth animations for rectangular portraits:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*NwD2m-RyUlk5k3GRhcJYCQ.gif" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*jeAQ8L6-QzSboWIcj_2-Ig.gif" /><figcaption>Figure 2: Quick/Smooth mouth animation. Due to lack of data, male portraits tend to be lower quality</figcaption></figure><p>I’ll go into the technical details of what I did and some of the lessons I learned. Part of the project was a tool to make interacting with and learning about generative adversarial networks easier, but at this point it is not user friendly. If I continue with this project one of my goals will be to publish a version of that tool that anyone can immediately start using to create animations like Figure 2, but right now it is primarily a tool for research:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NyZbGX3wB8KlXTAAXYDQ3w.png" /><figcaption>Figure 3: A screenshot of part of the tool. The UI definitely needs work, but at this point it’s still a prototype that I’m adding and removing features from on a regular basis</figcaption></figure><p>I’ve found that incorporating experimental code into a tool like this instead of working with a jupyter notebook makes it much easier to repeat experiments with different settings. Some concepts only really stick through repetition, so without the tool I feel like I would have missed out on several of the insights mentioned in the blog. If you’re just interested example animations and not the technical details, you can skip to the <em>Results: Animation </em>section.</p><p>One of the main problems with personal projects is that they only involve a single perspective. My goal for this blog is to get other’s perspective on the topic, detail my experience working on the project and receive constructive criticism/corrections.</p><h3>1.2: Introduction and Results Summary</h3><p>I’ve been in the habit of regularly reimplementing papers on <a href="https://en.wikipedia.org/wiki/Generative_model">generative models</a> for a couple years, so I started this project around the time the <a href="https://arxiv.org/abs/1812.04948">StyleGAN</a> paper was published and have been working on it on and off since then. It includes 3 main parts:</p><p>1. A reimplementation of StyleGAN with some tweaks</p><p>2. Models trained using that implementation</p><p>3. A tool used to visualize and interact with the models</p><p>I started out reimplementing StyleGAN as a learning exercise, and because at the time the <a href="https://github.com/NVlabs/stylegan">official code</a> was not available. The results were so much better than the other models I worked with that I wanted to go more in depth. One application of generative models that I’m excited about is automatic asset creation for video games. StyleGAN is the first model I’ve implemented that had results that would acceptable to me in a video game, so my initial step was to try and make a game engine such as Unity load the model. To do this I made a .NET DLL which could interact with the model and theoretically be imported into Unity. To test the DLL I created a harness to interact with it. I ended up adding more and more features to the harness as I thought of them, until it became one of the biggest parts of the project. Here’s the overall architecture:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GsM_Nr3q0lbcMIl0XON87Q.png" /><figcaption>Figure 4: From TensorFlow python implementation to generating images with the tool</figcaption></figure><p>I’m a fan of using tools to visualize and interact with digital objects that might otherwise be opaque (such as malware and deep learning models), so one feature I added was visualization of <a href="https://towardsdatascience.com/applied-deep-learning-part-4-convolutional-neural-networks-584bc134c1e2">feature maps</a> (Figure 5) and the ability to modify them. Observing where feature maps at various layers are most active across different images helped me understand what the model was doing, and made it simple to automatically locate some facial features. When it came to modification, I noticed that adding or subtracting values from feature maps in specific locations could be used to make meaningful changes — for example opening and closing a mouth (Figure 2). Combined with automatic facial feature detection, this can be used to make consistent, meaningful modifications in all images generated without needing labels (Figure 9,10).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2mP85Sej0NDdMrYV0w0ZCQ.png" /><figcaption>Figure 5: Feature maps can be used to identify meaningful locations (top: eyes, bottom: faces). Red squares indicate negative activations, green squares indicate positive activations, white squares indicate activations close to 0</figcaption></figure><p>To summarize: a feature uses feature maps to modify facial features.</p><p>Here’s how the rest of the blog is structured:</p><ul><li>Applications of generative models</li><li>Animations generated by the tool</li><li>Brief discussion of the code</li><li>Reimplementation details</li><li>Discussion of the data</li><li>Discussion of the training procedure</li></ul><h4>Applications</h4><p>Overall, there are several applications that make me interested in generative models:</p><ul><li>Better procedural generation for game assets</li><li>Making the creation of art easier and faster</li><li>The fundamental potential of unsupervised learning combined with disentanglement of generative factors</li></ul><p><em>Better procedural generation for game assets</em></p><p>I am excited about the possibility of generative models bringing procedural generation to the next level. As far as I’m aware, modern rules based procedural generation techniques cannot randomly create samples from highly complex distributions. For example, a subsection of a procedurally generated level can be mostly independent from the rest of the level and still be acceptable to a player. Randomly generating graphics such as character portraits is much more difficult, as good-looking images tend to reflect the real world which has so many inter-dependencies that each pixel needs to be considered in the context of every other pixel. Even stylized/drawn images need consistent lighting, anatomy, textures, perspective, style, etc. in order to look good. This also applies to automatically generating audio, dialog, animations, and story-lines. Generative models are currently the best way I’m aware of to reliably create samples from such complicated distributions.</p><p><em>Making the creation of art easier and faster</em></p><p>In the context of generative models, interactive tools also have the potential to allow laypersons to create images that would otherwise require an experienced artist, or allow artists to speed up the more routine parts of their work. I don’t think generative models are going to remove the need for creative artists any time soon, as generative models (along with most machine learning models) focus on modeling a specific distribution. This makes creating high quality images that are different from any samples in the training distribution (creative images) difficult. Tools like the one used in this blog that allow humans to add custom alterations can help to produce more unique images however, especially if the model has learned some fundamental graphical concepts such as lighting and perspective.</p><p><em>The fundamental potential of unsupervised learning combined with disentanglement of generative factors</em></p><p>As unlabeled data vastly outnumbers labeled data and deep learning is extremely data hungry, I think unsupervised/semi-supervised learning has the potential gradually replace supervised methods in the coming years. Some approaches to deep generative models in particular have been made to <a href="https://arxiv.org/pdf/1206.5538.pdf">disentangle</a> the factors of variation in a dataset: see Figure 7, 8 for examples of how StyleGAN can do this (at least partially). There isn’t actually an agreed upon <a href="https://arxiv.org/abs/1812.02230">formal definition of disentangment</a>, and my understanding is that there is <a href="https://arxiv.org/abs/1905.12506">limited experimental evidence that it is useful for downstream tasks</a>. However, working with GANs has made me optimistic that it will be useful. While the generator can not classically produce internal representations for images it did not generate, there are <a href="https://arxiv.org/abs/1906.00446">several</a> <a href="https://arxiv.org/abs/1807.03039">other</a> types of generative models that can compete with GANs in visual quality and may be better suited for downstream tasks.</p><h4>Results: Animations</h4><p>The best model I was able to train was on an anime dataset (<a href="https://www.gwern.net/Danbooru2018">https://www.gwern.net/Danbooru2018</a>). I’ll go into the advantages and disadvantages of that in the data section, one of the main disadvantages being a lack of diversity: it is harder to produce male images. All of the examples below were generated with the tool. I originally created images like these in a jupyter notebook, but having a dedicated tool sped things up significantly and gave me a different perspective into how the model works. The images below are roughly ordered by how complicated it is to make them: Figures 7/8 require more work than Figure 6, and without a tool Figures 9/10 would have been harder to create than Figures 7/8.</p><p>Figure 6 is an example of interpolation between the intermediate latent variables of several images. A cool result that GANs can achieve is ensuring the interpolated images have quality similar to the end points.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/253/1*b6NLHb5eLZ8nVnwtnJWkPQ.gif" /><figcaption>Figure 6: Interpolating between random latent vectors</figcaption></figure><p>Figure 7 is an example of modifying an image by locating a vector in the intermediate latent space with a certain meaning (in this case hair or eye color) and moving in that direction. For example, the vector for black hair can be found by taking many images of faces with black hair and averaging their latent values, and subtracting the result from the average of all other images. I’ll discuss this more in the Training/Postprocessing section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/253/1*_7wCtcU0Z79gt6fcUMzXIw.gif" /><figcaption>Figure 7: Changing a latent vector in a meaningful direction</figcaption></figure><p>Figure 8 shows the same concept used to generate Figure 7 when applied to the attribute ‘mouth open’. This works to some extent, but the attribute is not perfectly disentangled: changes can be seen in every part of the image, not just at the mouth. Common sense tells us that a person can move their mouth without significantly changing much else. A simple hack could be to paste the animated mouth on an otherwise static image, but this won’t work in cases where changing the ‘mouth open’ vector also changes skin tone or image style.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/252/1*NQES6-KQFpBkI-jZuk8Eug.gif" /><figcaption>Figure 8: Vector to change mouth also changes other attributes</figcaption></figure><p>Figure 9 shows an example of modifying a specific feature map at a spatial location close to the character’s mouth to produce a talking (or chewing) animation that does not cause global changes. With a little bit of manual work, the feature map(s) that need to be modified to produce this change can be found without needing more than a couple examples of labeled data. All of this can be done through the tool. The process is inspired by how the <a href="https://arxiv.org/abs/1511.06434">DCGAN</a> paper demonstrated window deletion by modifying feature maps. The modifications are simple additions or subtractions to local areas of specific feature maps. I’ll show how this can be done with the tool in a future blog.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*oAddg8CmbKiAbc6JNFgedw.gif" /><figcaption>Figure 9: Spatially local modification to produce talking animation. Once feature map(s) that correspond to a meaningful change are found, they can be applied to most images, regardless of quality or style.</figcaption></figure><p>Figure 10 shows the same thing as Figure 9, except applied to the presence/absence of a hairband. This technique can be applied to many different attributes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*Y4dhvyprFN2y6a79rKpQ8Q.gif" /><figcaption>Figure 10: Adding/subtracting hairband using local modifications</figcaption></figure><h4>Results: Code</h4><p>Given my limited free time and that my primary goal was learning, I deprioritized a few aspects that are important to most projects:</p><ul><li>Planning/design: I started with as little planning as possible and added features/changes as I thought of them</li><li>UI design: I used a greedy approach to adding features as I thought of them.</li><li>Code style: I did not go into the project with the intent for others to read the code, instead my goal was to get results as fast as possible. With a malware reverse engineering background, I’m not particularly bothered by needing to debug low quality code myself. Moving fast worked well for the tool, but I go much slower for implementing the deep learning models as mistakes can be hard to catch and debug. The code quality is far from what would be acceptable for professional projects.</li></ul><p>With these disclaimers out of the way, here’re the gitub links. They are under active development so there may be breaking bugs in some commits:</p><p>StyleGAN reimplementation:</p><p><a href="https://github.com/nolan-dev/stylegan_reimplementation">nolan-dev/stylegan_reimplementation</a></p><p>Tool:</p><p><a href="https://github.com/nolan-dev/GANInterface">nolan-dev/GANInterface</a></p><h3>1.3: Implementation Details</h3><p>In this section I’ll go into the technical details of what I did to reimplement StyleGAN and train a model with it</p><h4>StyleGAN reimplementation</h4><p>I started reimplementing StyleGAN shortly after the <a href="https://arxiv.org/abs/1812.04948">paper</a> came out, which was a few months before the official code was released. I’ll discuss some of the challenges I encountered and approaches I took in this section, which assumes familiarity with the StyleGAN paper and TensorFlow.</p><p>StyleGAN is based on <a href="https://arxiv.org/abs/1710.10196">PGGAN</a>, which I had already reimplemented. These models use ‘progressive growing’, where the discriminator and generator grow during training to handle higher and higher resolutions. Growing a model is somewhat uncommon — all other models I’ve implemented have never needed to change their architecture during training. Fortunately, TensorFlow has some convenient functions, such as the ability to load saved weights for only parts of a model and randomly initialize the rest. This is also used in transfer learning.</p><p>I really like the TensorFlow 2.0 style of creating models as classes that inherit from tf.keras.Model. I created most of the layers, the generator, the discriminator, and the mapping network this way. I also tried to make it an option to switch between eager execution and traditional graph-based execution. Eager execution allows for much easier debugging, which I view as a way to better understand a program (a common technique in malware analysis). Unfortunately, at the time eager execution seemed to be significantly slower than running in graph mode, so eventually I stopped keeping that functionality up to date. One of the nice things about using tf.keras.Model is that it works for both eager and graph mode, so theoretically it should not be too difficult to get eager working again. In the mean time I’ve just used the tfdebug command line interface and TensorBoard which I’m fairly comfortable with at this point.</p><p>StyleGAN has a few important differences from PGGAN. There’s the titular method of supplying the image-specific latent data as a style (non-spatial attribute) to the feature maps while using an “adaptive instance normalization” operation. Conceptually this is straight forward to implement as detailed in the paper, but my choice to use tf.nn.moments to calculate mean and variance did not work as well as the official implementation’s version which calculated these values with lower level operations. My first guess was that this was due to numerical issues, which was not something I felt like debugging at the time so I did not end up looking into it more. I’m usually happy to look deeper into issues like this as they’re obvious chances to learn more, but as this project is a hobby I have to prioritize to make the best use of my time.</p><p>StyleGAN also uses an intermediate latent space which hypothetically (with some empirical evidence presented in the paper) promotes disentanglement by adding flexibility and dependencies to the range of latent values. For example, if we assume a population where men never have long hair, a latent vector that corresponds to hair length should never be in the ‘long hair’ region when another latent that corresponds to gender is in the ‘male’ region. If the latent vectors are independent, which is what happens without the mapping network, and we end up sampling ‘long hair’ alongside ‘male’, the generator will have to create an image that is either not male or has short hair in order to fool the discriminator. This means even if the hair length latent is in the ‘long hair’ region, we may end up creating an image with short hair when other latent values are in normal parts of their range. Note that some definitions of disentanglement require axis-alignment (modifying a single latent value results in a meaningful change), while my understanding is that the mapping network of StyleGAN encourages the intermediate latent space to be a rotation of axis-aligned disentanglement (modifying a vector of latent values results in a meaningful change).</p><p>In my opinion the use of intermediate latent values is just as interesting as incorporating information into the network via styles. It is also simple to implement. When I refer to latent value throughout this blog series I will be referring to the intermediate latent value unless otherwise specified.</p><p>It is annoyingly common for what initially looks like a minor detail in a paper to turn out to be the most difficult part to implement. This was the case for StyleGAN — one difference between StyleGAN and PGGAN is the use of bilinear upsampling (and downsampling) and of <a href="https://arxiv.org/abs/1801.04406">R1 regularization</a> (a gradient penalty on the discriminator). Both of these are simple to implement individually, however when I tried to combine them it turned out that TensorFlow did not have a way to compute the second derivative of the tf.nn.depthwise_conv2d operation. Depth-wise convolutions are used to apply a filter individually to each channel. This is not normally needed in convolutional neural networks because (outside some CNNs meant for mobile devices) each filter connects to all channels in the previous layer. The blur filter that is commonly used to implement bilinear interpolation operates on one feature map at a time, so it requires a depthwise convolution. Without a second derivative implemented, I couldn’t compute the R1 penalty which requires taking the gradient of a gradient. At the time I didn’t know enough about automatic differentiation to easily implement the second derivative myself. I spent some time trying to get a better understanding, however at that point I had everything finished except for this part, and a short time later the official code was released. The team at Nvidia solved the problem nicely using two tf.custom_gradient functions in the blur filter.</p><p>I made several experimental tweaks to StyleGAN with varying degrees of success.</p><p>1. I tested out rectangular images by changing the initial resolution to 8x4 and growing from there</p><p>2. I tried conditioning StyleGAN using <a href="https://arxiv.org/abs/1610.09585">ACGAN</a> and <a href="https://arxiv.org/abs/1802.05637">cGan with Projection Discriminator</a></p><p>Rectangular images worked well. Conditioning on labels with ACGAN and cGan did not. This may have been due to my hyperparameter selections, but generally the results were worse than training without conditioning and then finding vectors in latent space that corresponded to meaningful features (discussed in Training/Postprocessing section).</p><h4>Data</h4><p>The FFHQ dataset — a collection of 70,000 high resolution images — was released alongside the StyleGAN paper. To get closer to the goal of generating entire people, I tried modifying the data extraction script supplied by Nvidia to extract images with an 8:4 (h:w) aspect ratio. In addition to extracting the face, this extracted the same amount of data from below the face. Doubling both the height and the width would require accounting for 4x the input dimensions, but just doubling the height requires 2x the dimensions. Data below a person’s face should also have less variance (variance mostly comes from different clothing) than background data, and not capturing background data means the network does not need to dedicate capacity to data that isn’t part of the person.</p><p>Unfortunately, only 20,000 of the 70,000 images in the FFHQ dataset had enough data below the person’s face to create images with the desired aspect ratio. As can be seen in Figure 11, I was not able to get particularly high quality results with this small dataset, but there may be ways to improve the quality (such as expanding the criteria for including an image).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/254/1*lWxXuudrkELfkAllkec5KQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*0LJdzSLSvYFUw-nLJGuwYA.png" /><figcaption>Figure 11: Rectangular FFHQ images tended to be low quality</figcaption></figure><p>I am also interested in the ability of GANs to perform on stylized drawings, and I saw that <a href="https://www.gwern.net/Danbooru2018">https://www.gwern.net/Danbooru2018</a> had recently been released. This dataset has a large number of high resolution images with very rich tag data. It does have some potential downsides, such as a disproportionately small number of male images which significantly reduces the quality of images in that class (Figure 12). The male images in this figure were the highest quality images cherry-picked out of around 1000 total images generated. I do think there is a lot of potential for improvement here, particularly using the truncation trick around the average male image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*WqQ07OgamjArDsSFKR8ROQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*AvDBJmj2bg_YfD4p-yqDeg.png" /><figcaption>Figure 12: Most male portraits (top) are low quality due to dataset limitations. Even when minimally cherry-picked for quality, female portraits (bottom) tend to be higher quality. None of the images here use the truncation trick for higher quality</figcaption></figure><p>The dataset also contains a significant portion of NSFW images, though I think one potential use of generative models is automatically modifying media to make it more suitable for different audiences.</p><p>To select candidate images to include, I was able to take advantage of metadata including ‘favorites’ (how many people favorited an image), resolution, tags, and creation date. To reduce variance and increase quality, I excluded images with few favorites and images that were created more than 6 years ago. I also only kept images with a resolution of at least 512x256 (HxW), my target resolution for the model. Finally, I filtered out images with tags that would increase variance in a portrait-style dataset, such as ‘lying_down’ and ‘from_side’, or that implied very NSFW images.</p><p>To generate the data I used a combination of the following two tools which I modified to extract the images of the desired aspect ratio:</p><ul><li><a href="https://github.com/qhgz2013/anime-face-detector">qhgz2013/anime-face-detector</a></li><li><a href="https://github.com/nagadomi/lbpcascade_animeface">nagadomi/lbpcascade_animeface</a></li></ul><p>These tools did not always extract images correctly, so I used illustration2vec to filter the result, as images where a person could not be detected were likely bad.</p><p><a href="https://github.com/rezoo/illustration2vec">rezoo/illustration2vec</a></p><p>I also created a tool that would display a large grid of images from which I could manually delete bad images quickly, but this became too time consuming for datasets above 30,000 images. I ended up with several different datasets by including images of various qualities and tags which ranged from 40,000 to 160,000 images. All of these resulted in much better models than the 20,000 image FFHQ dataset I built.</p><h4>Training/Postprocessing</h4><p>I trained the model using as many as 80 epochs per resolution, which breaks down to 40 epochs for the PGGAN transition phase and 40 epochs for the stabilization phase. For the 160,000 image dataset this took over a month, and was probably overkill. I used <a href="https://github.com/horovod/horovod">Horovod</a> to distribute training across two Nvidia Titan RTX graphics cards, which allowed large batch sizes in the early steps, and kept me from needing to ever go below 16 samples per batch.</p><p>To gather attributes I generated hundreds of thousands of images and scanned them with illustration2vec to get attribute estimates. For each attribute with more than 1000 corresponding images, I averaged the latent values together and subtracted them from the overall average image. This creates a vector that, when added to a latent, should promote the expression of the desired attribute. While this worked pretty well (which I interpret as further evidence the mapping network disentangles attributes), I’m interested in improving the process, potentially by using techniques described in <a href="https://arxiv.org/abs/1907.10786">https://arxiv.org/abs/1907.10786</a>. In some cases I tried to manually disentangle correlated attributes: for example if blonde hair is correlated with blue eyes, subtracting a vector that corresponds to blue eyes from the blonde hair vector may help a bit. I then exported these attributes vectors into a csv file that could be loaded by the tool.</p><h4>Conclusion/Misc Notes</h4><p>I’ve gained a lot of experience by trying to reimplement papers that rely on fundamentals I don’t understand and then learning those fundamentals with a clear context for how they are used. The process feels a lot like backpropagation, and means I’ve spent 30+ hours trying to fully understand a single paper early on. This is the first time I’ve tried creating a tool to enhance that understanding, and I think it may become a standard strategy for me going forward.</p><p>I believe visual tools are a great way to improve understanding of a topic. Coming from a malware analyst background, tools like <a href="http://www.hiew.ru/">Hiew</a> were essential in shaping how I understand malware, even though it was not initially intuitive how visual analysis can be helpful for windows executables. Given the bandwidth and processing power available to the human visual system, my hypothesis is that we can quickly gain a lot of insight from most data with spatially local structures when represented as an image. This is also the type of data that convolutional neural networks seem to work well with (not too surprising given their biological inspiration). The concept also applies to convolutional networks themselves, which is one part of why I worked on this project: hopefully visualizing the feature maps of convolutional neural networks might help me understand them better.</p><p>In the next part of this blog series I go into more detail about the tool and how it can be used to create animations. I also share a compiled version of the tool and a trained model to interact with, as the process to do that with just the source code is complicated at the moment.</p><p><a href="https://towardsdatascience.com/animating-ganime-with-stylegan-the-tool-c5a2c31379d">Animating gAnime with StyleGAN: The Tool</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4cf764578e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/animating-ganime-with-stylegan-part-1-4cf764578e">Animating gAnime with StyleGAN: Part 1</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>