Want to learn applied Artificial Intelligence from top professionals in Silicon Valley or New York? Learn more about the Artificial Intelligence program at Insight.

Are you a company working in AI and would like to get involved in the Insight AI Fellows Program? Feel free to get in touch.

All the code and online demo are available at the project page.

Teaching computers to draw photos according to descriptions

Describing an image is easy for humans, and we are able to do it from a very young age. In machine learning, this task is a discriminative classification/regression problem, i.e. predicting feature labels from input images. Recent advancements in ML/AI techniques, especially deep learning models, are beginning to excel in these tasks, sometimes reaching or exceeding human performance, as is demonstrated in scenarios like visual object recognition (e.g. from AlexNet to ResNet on ImageNet classification) and object detection/segmentation (e.g. from RCNN to YOLO on COCO dataset), etc.

However, the other way around, generating realistic images based on descriptions, is much harder, and takes years of graphic design training. In machine learning this is a generative task, which is also much more challenging than discriminative tasks, as a generative model has to produce much richer information (like a full image at some level of detail and variation) based on a smaller seed input.

Despite the difficulty in creating such types of applications, generative models (with some control) can be extremely useful in many cases:

• Content creation: Imagine if an advertisement company could automatically generate attractive product images that match the content and style of the webpage where these images are inserted; a fashion designer could get inspiration by asking an algorithm to produce 20 examples of shoe designs that are related to “leisure”, “canvas”, “summer” and “passionate”; and a new game could allow players to create realistic avatars based simple descriptions.
• Content-aware smart editing: We could allow a photographer to change the facial expression, amount of wrinkles and hair style of a profile photo with several clicks; and a Hollywood studio artist could transform the footage shot on a cloudy evening to look like it was shot on a sunny morning, with sunshine shedding light from the left side of the screen.
• Data augmentation: An autonomous driving car company could synthesize realistic videos of a particular type of accident scenario to augment the training dataset; and a credit card company could synthesize data of a particular type of fraud data that is underrepresented in the dataset to improve their fraud detection system.

In this post, we will describe our recent work called Transparent Latent-space GAN (TL-GAN), which extends current cutting edge models to provide a new interface. We are currently working on a paper, that will have more technical details.

Overview of generative models

The deep learning community is making rapid progress on generative models. Among them are three promising types of models: autoregressive models, variational autoencoders (VAE) and generative adversarial networks (GAN), illustrated as the figure below. If you are interested in the details, please check out this awesome OpenAI blog post.

So far, GANs produce images of the highest quality (photo-realistic and diverse, with convincing details in high resolution). Look at the stunning images generated by Nvidia’s recent work with pg-GAN (progressively-growing GAN). For this reason, this blog post will focus on GAN models.

Controlling the output of GAN models

The original version of GAN and many popular successors (like DC-GAN and pg-GAN) are unsupervised learning models. After training, the generator network takes random noise as input and produces a photo-realistic image that is barely distinguishable from the training dataset. However, we cannot further control the features of the generated images. In most applications (such as the scenarios described in the first section), users would like to generate samples with custom features (like age, hair color, facial expression, etc), and ideally, tuning each feature continuously.

To achieve controlled synthesis, numerous variants of GAN have been created. They can be roughly divided into two types: style-transfer networks and conditional generators.

Style-transfer networks

Style-transfer networks, represented by CycleGAN and pix2pix, are models trained to translate image from one domain to another (e.g. from horse to zebra, from sketch to colored images). As a result, we cannot continuously tune one feature gradually between two discrete states (eg. add slightly more beard on the face). Also, one network is dedicated to one type of transfer, so it requires ten different neural networks to tune 10 features.

Conditional generators

Conditional generators, represented by conditional GAN, AC-GAN, and Stack-GAN, are models that jointly learn images with feature labels during training time, enabling the image generation to be conditioned on custom features. Therefore, when you want to add new tunable features to the generation process, you have to retrain the whole GAN model, which takes an enormous amount of computing resources and time (e.g. days to weeks on a single K80 GPU with the perfect set of hyper-parameters). In addition, you have to rely on a single dataset that contains all the custom feature labels to perform the training, instead of leveraging different labels from multiple datasets.

Our model, which we call Transparent Latent-space GAN (TL-GAN), addresses these problems of existing methods by approaching controlled generation task from a novel angle. It offers users the ability to gradually tune one or multiple features using a single network. Besides, adding new tunable features can be done very efficiently in less than one hour.

TL-GAN: a novel and efficient approach for controlled synthesis and editing

Making the mysterious latent space transparent

We will leverage NVIDIA’s pg-GAN, the model that generates the photo-realistic high resolution face images as shown in the the previous section. All the features of a generated 1024px*1024px image are determined solely by a 512-dimentional noise vector in the latent space (as a low-dimensional representation of the image content). Therefore, if we could understand what the latent space represents (i.e., making it transparent), we could completely control the generation process.

By experimenting with the pre-trained pg-GAN, I found that the latent space actually has two good properties:

• It is well populated, meaning that most points in the space will generate reasonable images
• It is quite continuous, meaning the interpolation between two points in the latent space usually leads to a smooth transition of corresponding images.

With this in mind, my intuition was that it is possible to find directions in the latent space that are predictive of features that we care about (eg. male-female). If so, we can use unit vectors of these directions as the feature axes for controlling the generation process (more male-like or more female-like).

Approach: uncovering the feature axes

To find these feature axes in the latent space, we will build a link between a latent vector z and the feature labels y through supervised learning methods trained on paired (z,y) data. Now the problem becomes how to get such paired data, since existing datasets only contain images x and their corresponding feature labels y.

Potential approaches:

One potential approach is to compute the corresponding latent vectors z of images x_real from an existing dataset labeled with features of interest y_real. However, the GAN network does not offer an easy way to compute z_encode=G^(−1)(x_real), making this idea difficult to implement.

A second potential approach is to generate synthetic images x_gen using GAN from a random latent vector z as x_gen=G(z). The problem here is that the synthetic images are unlabeled, and we can not easily leverage the available labeled dataset.

To solve this problem, the key innovation of our TL-GAN model is to train a separate feature extractor (a classifier for discrete label or regressor for continuous label) model y=F(x) using an existing labelled image dataset (x_real, y_real), and then couple the trained GAN generator G with the feature extractor network F. Once this is done, we can predict the feature labels y_pred of the synthetic images x_gen using the trained feature extractor network, and thus establish the link between z and y through synthetic images as x_gen=G(z) and y_pred=F(x_gen).

Now that we have the paired latent vector and features, we can train a regressor model y=A(z) to uncover all feature axes for controlling the image generation process.

The above figure shows the architecture of the TL-GAN model, which contains five steps:

1. Learning the distribution: Choose a well-trained GAN model and take the generator network. I chose the well-trained pg-GAN (provided by Nvidia), which offers the best face generation quality.
2. Classification: Choose a pre-trained feature extractor model (could be a convolutional neural network or other computer vision models), or train your own feature extractor network using a labelled dataset. I trained a simple convolutional neural network on the CelebA dataset (which contains 30,000+ face images with 40 labels each).
3. Generation: Generate a number of random latent vectors, pass through the trained GAN generator to produce synthetic images, then use a trained feature extractor to produce features for every image.
4. Correlation: Use a Generalized Linear Model (GLM) to perform regression between latent vectors and features. The regression slope becomes the feature axes.
5. Exploration: Start from one latent vector, move it along one or more feature axes, and examine how this affects the generated images.

I made this process very efficient; once we have a pre-trained GAN model, identifying feature axes only takes one hour on a single-GPU machine. This is achieved by several engineering tricks including transfer learning, downsampling image size, pre-cache synthetic images, etc.

Results

Let us see how this simple idea works.

Moving latent vector along feature axes

First, I tested whether the discovered feature axes can be used to control the corresponding feature of the generated image. To do this, I generated a random vector z_0 in the latent space of GAN, and produced a synthetic image x_0 by passing it through the generator network x_0=G(z_0). Next, I moved the latent vector along one feature axis u (a unit vector in the latent space, say, corresponding to the gender of the face) by distance λ, to a new location x_1=x_0+λu, and generated a new image x1=G(z1). Ideally, the corresponding feature of the new image would be modified toward the expected direction.

The results of moving the latent space vector along several example feature axes (gender, age, etc) are shown below. This works surprisingly well! We can smoothly morph the image between male ←→ female, young←→old, etc.

Disentangling correlated feature axes

The examples above shows a shortcoming of the initial method, i.e., the entangled feature axes. For instance, when I intended to reduce the amount of beard, the generated faces became more female-like, which is not what a user would expect. This problem is due to the fact that the gender feature and the beard feature are by nature correlated, modifying one leading to the corollary change of the other. Similar things happened to other features like hairline and wavy hair. As is illustrated in the figure below, the original beard axis is not perpendicular to the gender axis in the latent space.

To solve this problem, I used straight-forward linear algebra tricks. Specifically, I projected the beard axis to a new direction that is orthogonal to the gender axis, which effectively removes their correlation, and thus could potentially disentangle these two features of generated face images.

I applied this feature disentangling method to the same example face. This time I used gender axis and age axis as reference features, projected all other feature axes so that they became orthogonal to gender and age, and examined the generated images when the latent vector moved along the newly generated feature axes (shown in the figure below). As we expected, the features including hairline, wavy hair and beard now modify the gender of face anymore, which behaves as expected.

Flexible interactive editing

To see how flexibly our TL-GAN model can control the image generation process, I built an interactive GUI to explore the effect of gradually tuning feature values along different feature axes, as shown below:

Again, the model works surprisingly well when I use the features axes to control the generated images!

Recap

This project provides a novel method to control the generation process of an unsupervised generative model like GAN (generative adversarial network). Using an already well-trained GAN generator (Nvidia’s pg-GAN), I made its latent space transparent by discovering the meaningful feature axes in it. When a vector moves along a feature axis in the latent space, the corresponding image morphs over this feature, which enables controllable synthesis and edit.

This method has clear advantages:

1. Efficiency: To add a new feature tuner for the generator, you do not need to re-train the GAN model, thus it only takes <1h to add 40 feature tuners with our method.
2. Flexibility: You can use any feature extractor trained on any dataset to add more feature tuners to the well-trained GAN.

A word about ethics

This work allows us to have fine grained control over image generation, but it still relies heavily on the features of our dataset. Training on photos of celebrities of Hollywood means that our model will be very good at generating photos of a predominantly white and attractive demographic. In turn, this would lead to users only being able to generate these specific kind of faces which are only representative of a small subset of people. If we were to deploy this as an application, we would want to make sure that we have augmented our initial dataset to take into account the diversity of our users.

In addition, while this tool could be a huge creative help, we should ask ourselves how this model could be used for nefarious purposes. If we can generate realistic looking faces of any type, what are the implications for our ability to trust in what we see. These sort of issues are important to tackle today. As we have seen with the recent applications of deepfakes, the capability of AI methods is increasing at a fast pace, so it is vital for us to start conversations about how to best deploy them.

Online demo and code

All the code and online demo of this work are available at this Github project page.

If you want to play with the model in your web-browser

You do not have to download any code, model or data. Just follow the instructions on this section of the GitHub README page. You can tweak faces in your web-browser as is demonstrated in the the above Video.

If you want to try my code

Just follow the README page of the GitHub repository. The code is built on Anaconda Python 3.6 with Tensorflow and Keras.

If you want to contribute

You are very welcome! Feel free to submit a Pull Request or file an issue on Github

About me

I recently finished PhD in computational and cognitive neuroscience from Brown University, with a concurrent master degree in computer science focusing on machine learning. In the past, I studied how neurons in the brain collectively process information to achieve high-level functions like visual perception. I am enthusiastic about the algorithmic approach to understand, mimic, and implement intelligence, as well as apply them to solve challenging real-world problems. I am actively looking for ML/AI researcher positions in the tech industry.

Acknowledgements

This work was done as the project for the Insight AI fellow program over three weeks. I give my thanks to the program director Emmanuel Ameisen and Matt Rubashkin for their general guidance, especially Emmanuel Ameisen for his suggestions and editing work on this blog post. I also give my thanks to all Insight staff for providing a great learning environment, and to Insight AI fellows from whom I learned a lot. Special thanks to Ruobing Xia for providing numerous inspirations as I decided on project directions, and for the enormous help structuring and editing this blog post.

This passage was originally posted on my personal blog.

Want to learn applied Artificial Intelligence from top professionals in Silicon Valley or New York? Learn more about the Artificial Intelligence program at Insight.

Are you a company working in AI and would like to get involved in the Insight AI Fellows Program? Feel free to get in touch.

Generating custom photo-realistic faces using AI was originally published in Insight on Medium, where people are continuing the conversation by highlighting and responding to this story.

Our lab has set up super powerful Linux server for data analysis (with multiple Xeon CPUs, multiple NVIDIA Tesla GPUs for cuda, and a lot of storage space. Being used to work on my own Macbook Pro, it did take me some time to figure out the best practice to work remotely.

Below I listed possible methods to connect to the server and the their pros and cons based on my experience.

Summery:

NoMachine works best, for NoMachine connection, jumps to section 4. For file sharing jump to 1.1

1. Terminal ssh

It requires the server to enable ssh, which most servers do.

1.1 Mac/Linux

pure terminal, no GUI

Open terminal, type

ssh username@hostname

hostname is the ip address or domain name and username is your account name on the server.

with GUI through XWindow forwarding

if you want to open GUI application from the remote server, just do:

ssh -X username@hostname

or

ssh -Y username@hostname

The -X or -Y option allows the XWindows of GUI apps to be forwarded to the local desktop

eg. bash xterm will open a term with gui, similarly, other applications like PyCharm, MatLab can pop-up a window and you can work the the interface as you are working locally.

Note: if the server is in an internal network (like university), you may need to first connect through VPN to ssh in.

file transfer

To transfer files between server and client, you could:

• ftp use tools like FileZilla (connect server through the sftp protocol, i.e. ftp via ssh), or
• mount the remote directory in the naive file manager with the help of Mac Fusion

1. install Fuse for Mac
2. in terminal: sshfs username@hostname:/remote/directory/path /local/mount/point
3. the remote directory can be accessed using Finder

1.2 MS Windows:

Windows does not come with a linux-like terminal, but there is a great application “MobaXterm”, which enables ssh connection and file transfer between server and client. Isn’t it awesome?

official website: https://mobaxterm.mobatek.net

Similar with Mac/Linux, but more convenient, see simple demo at https://mobaxterm.mobatek.net/demo.html

1.3 Terminal Pros/Cons

• Pros: convenient, minimal need to install software on server and client sides
• Cons: LOST WORKING PROGRESS when connection is lost!, eg., 1) your session will be terminated and all the work will be lost when the network is not stable; 2) you can not continue to work on the same session by using another computer at home

good for short-term connection for simple tasks

2. Terminal ssh with screen

2.1 basics

To keep the session alive despite lost connection, screen command is very helpful, but it only supports pure terminal applications, not GUI applications.

Basically, screen is a container, within in which other programs can run. the screen container persists even if the current terminal session gets interrupted. You can later attach to the "screen" from another computer to proceed your work

see:

1. official documentation: https://www.gnu.org/software/screen/
2. quick guide https://www.rackaid.com/blog/linux-screen-tutorial-and-how-to/

2.2 screen Pros/Cons

• Pros: current work survives over interrupted connection
• Cons: terminal only, does not support GUI

3. GUI session via Xpra

3.1. Xpra is screen for X11

We want a program like screen but supports GUI windows. After some search, I found xpra

xpra is, according to official site, a -

multi-platform screen and application forwarding system “screen for X11”

Another description of xpra, from Ubuntu documentation (https://help.ubuntu.com/community/Xpra):

Xpra is a tool which allows you to run X programs — usually on a remote host — and then direct their display to your local machine, disconnect from these programs, and reconnect from the same or another machine, all without losing any state. It differs from standard X forwarding in that it allows disconnection and reconnection without disrupting the forwarded application; it differs from VNC and similar remote display technologies in that xpra is rootless: i.e., applications forwarded by xpra appear on your desktop as normal windows managed by your window manager, rather than being all “trapped in a box together”. Xpra also uses a custom protocol that is self-tuning and relatively latency- insensitive, and thus is usable over network connections that are too slow or unreliable for standard X forwarding.

To use xpra, we have to install it on both the remote server and the local client.

3.2 Xpra installation

To use Xpra, we have to install it on both the remote server and the local client.

You can install through package manager like apt-get (eg. on Unbuntu, sudo apt-get install xpra) or by downloading and running installation files which can be found on the official site: https://xpra.org/trac/wiki/Download

3.3 Basic Xpra usage

I found the best way to use is:

3.3.1. start a session

from local client terminal:

local_client> xpra start ssh/username@hostname/100 --start=xterm

• here 100 is a identifier of the session name, different user should use different identifiers and the same user an start multiple sessions by specifying distinct idendifiers, like 101, 5000, etc.
• xterm is the GUI program to run on start, you can start any other GUI program from xterm

3.3.2 disconnect from a session:

from local client terminal:

simply close the terminal in which the xpra session was running, or interrupt the current session by pressing ctrl+c. Or any unstable network will disconnect the local session

3.3.3 attach/return to a previous session

on local machine:

local_client> xpra attach ssh/username@hostname/100

You can resume the previous session

3.3.4 check xpra sessions

if you want to list all the running sessions, just log into the remote server:

remote_server> xpra list

if you want to terminate all sessions:

remote_server> xpra stop

Note that this will terminate all your (but not other users’) xpra sessions and your work will be lost

if the font in your session is very small/large, this could be because mismatched DPI between server and client, you could try specifying --dpi when starting or attaching a session, like

local_client> xpra attach ssh/username@hostname/100 --dip=96

3.3.5 more details of xpra usage

• see https://xpra.org for details
• see http://manpages.ubuntu.com/manpages/xenial/man1/xpra.1.html for usage

3.5 Xpra Pros/Cons

• Pros: enables GUI application to persists over disconnect/reconnect
• Cons: not very stable, the session could crash (but you can reconnect and you will not lose your work), and the DPI problem and key-mapping problem can be annoying.

4. NoMachine connection

4.1 A better alternative to Xpra

NoMachine, from my experience, is a better alternative of Xpra.

Like Xpra, it allows disconnecting and reconnecting without losing the work in the forwarded GUI application window or the full desktop, but it seems more stable and does not suffer from DPI mismatch or key-mapping problems. Also it provides a nice interface and you do not have to type in any commands for connections

see official site: https://www.nomachine.com

NoMachine uses X11 mechanism to display the remote GUI. On Mac, it is supported through the program XQuartz. You have to install it if it is not available on your machine.

XQuartz

4.2 NoMachine installation

You have to install NoMachine on both remote server and local client. Simply follows the instructions

NoMachine - Getting started with NoMachine

If you have to install in command line: here is the instrctions:

NoMachine - How to install NoMachine from the command line on Linux

4.3 full desktop forwarding

In local client machine, open NoMachine. Just follows the steps, connect using either NX or SSH protocols. start a desktop session

4.4 application window forwarding

4.4.1 start a session

In the local client machine, open NoMachine. Just follows the steps to establish a connection, (connect using either NX or SSH protocols). Then start a custom session

use floating window, and run xterm at start

Now you will see a GUI xterm popping up, and you can type in command and start any GUI program like matlab, charm, jupyter notebook, .etc. If you want to keep using the terminal after starting the application, add a "&" after command, like matlab &, the terminal will be still alive while you are using matlab

4.4.2 disconnect from a session

To disconnect, simply close the NoMachine window (not the application window), or in the NoMachine window -> Connection -> Disconnect

4.4.2 attach to a existing session

Open NoMachine, open the connection, and the previous session is shown there, double connect to connect back and resume the session.

To terminate the session, right click the session -> terminate session (ongoing work in the session will be lost)

4.5 Keyboard shortcuts for Mac

When you are working on Mac, sometimes you want to map the command key as control key for the remote Linux machine, so that your familiar shortcuts on Mac would work on the linux system in most times. eg. cmd+C on your Mac is mapped as ctrl+C on the remote Linux. The easiest way to do that is to set the Xquatz default mapping file.

In terminal, create a file named .Xmodmap under your Mac home folder ~/, and put the commands in this file as below. This script will run automatically when the XQuartz starts, and applies only within XQuartz (with which NoMachine creates GUI).

reference:

10.3: Get xmodmap to remap alt / option in X11 - Mac OS X Hints

4.6 NoMachine Pros/Cons

• Pros: GUI application persists over interrupted connection; more stable than Xpra, no DPI or keyboard mapping problem; convenient interface and no need to type in command
• Cons: have not been found yet!

Desktop forwarding vs application window forwarding:

• desktop forwarding: feel like working locally on the server, but more sluggist
• application window forwarding: less sluggish, like running a local application. The Only option if the remote server is in terminal only mode (did not startx)

5. Conclusion

After comparison, the NoMachine application window forwarding mode via NX protocol works best from my experience. You only have to install NoMachine on the server and client, and followed by several clicks.