Recognizing Human Emotion from Audio Recording

Recognizing human emotion has always been a fascinating task for data scientists. Lately, I am working on an experimental Speech Emotion Recognition (SER) project to explore its potential. I selected the most starred SER repository from GitHub to be the backbone of my project.

Before we walk through the project, it is good to know the major bottleneck of Speech Emotion Recognition.

Major Obstacles:

• Emotions are subjective, people would interpret it differently. It is hard to define the notion of emotions.
• Annotating an audio recording is challenging. Should we label a single word, sentence or a whole conversation? How many emotions should we define to recognize?
• Collecting data is complex. There are lots of audio data can be achieved from films or news. However, both of them are biased since news reporting has to be neutral and actors’ emotions are imitated. It is hard to look for neutral audio recording without any bias.
• Labeling data require high human and time cost. Unlike drawing a bounding box on an image, it requires trained personnel to listen to the whole audio recording, analysis it and give an annotation. The annotation result has to be evaluated by multiple individuals due to its subjectivity.

Project Description:

Using Convolutional Neural Network to recognize emotion from the audio recording. And the repository owner does not provide any paper reference.

Data Description:

These are two datasets originally made use in the repository RAVDESS and SAVEE, and I only adopted RAVDESS in my model. In the RAVDESS, there are two types of data: speech and song.

Data Set: The Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS)

• 12 Actors & 12 Actresses recorded speech and song version respectively.
• Actor no.18 does not have song version data.
• Emotion Disgust, Neutral and Surprised are not included in the song version data.

Total Class:

Here is the emotion class distribution bar chart.

Feature Extraction:

When we do Speech Recognition tasks, MFCCs is the state-of-the-art feature since it was invented in the 1980s.

This shape determines what sound comes out. If we can determine the shape accurately, this should give us an accurate representation of the phoneme being produced. The shape of the vocal tract manifests itself in the envelope of the short time power spectrum, and the job of MFCCs is to accurately represent this envelope. — Noted from: MFCC tutorial

We would use MFCCs to be our input feature. If you want a thorough understanding of MFCCs, here is a great tutorial for you. Loading audio data and converting it to MFCCs format can be easily done by the Python package librosa.

Default Model Architecture:

The author developed the CNN model with Keras and constructed with 7 layers — 6 Conv1D layers followed by a Dense layer.

model = Sequential()
model.add(Conv1D(256, 5,padding='same', input_shape=(216,1))) #1
model.add(Activation('relu'))
model.add(Conv1D(128, 5,padding='same')) #2
model.add(Activation('relu'))
model.add(Dropout(0.1))
model.add(MaxPooling1D(pool_size=(8)))
model.add(Conv1D(128, 5,padding='same')) #3
model.add(Activation('relu'))
#model.add(Conv1D(128, 5,padding='same')) #4
#model.add(Activation('relu'))
#model.add(Conv1D(128, 5,padding='same')) #5
#model.add(Activation('relu'))
#model.add(Dropout(0.2))
model.add(Conv1D(128, 5,padding='same')) #6
model.add(Activation('relu'))
model.add(Flatten())
model.add(Dense(10)) #7
model.add(Activation('softmax'))
opt = keras.optimizers.rmsprop(lr=0.00001, decay=1e-6)

The author commented layer #4 and #5 in the latest notebook (18 Sep 2018 Update) and the model weight file does not fit the network provided, thus, I cannot load the weight provide and replicate its result 72% Testing Accuracy.

The model only simply trained with batch_size=16 and 700 epochs without any learning rate schedule, etc.

# Compile Model
model.compile(loss='categorical_crossentropy', optimizer=opt,metrics=['accuracy'])

# Fit Model
cnnhistory=model.fit(x_traincnn, y_train, batch_size=16, epochs=700, validation_data=(x_testcnn, y_test))

Its loss function is categorical_crossentropy and the evaluation metric is accuracy.

My Experiment

Exploratory Data Analysis:

In the RADVESS dataset, each actor has to perform 8 emotions by saying and singing two sentences and two times for each. As a result, each actor would induce 4 samples for each emotion except neutral, disgust and surprised since there is no singing data for these emotions. Each audio wave is around 4 second, the first and last second are most likely silenced.

The standard sentences are:

1. Kids are talking by the door.
2. Dogs are sitting by the door.

Observation:

After I selected 1 actor and 1 actress’s dataset and listened to all of them. I found out male and female are expressing their emotions in a different way. Here are some findings:

• Male’s Angry is simply increased in volume.​
• Male’s Happy and Sad significant features were laughing and crying tone in the silenced period in the audio.
• Female’s Happy, Angry and Sad are increased in volume.​
• Female’s Disgust would add vomiting sound inside.

Replicating Result:

The author excluded the class neutral, disgust and surprised to do a 10 class recognition for the RAVDESS dataset.

I tried to replicate his result with the model provided, I can achieve a result of

However, I found out there is a data leakage problem where the validation set used in the training phase is identical to the test set. So, I re-do the data splitting part by isolating two actors and two actresses data into the test set which make sure it is unseen in the training phase.

• Actor no. 1–20 are used for Train / Valid sets with 8:2 splitting ratio.
• Actor no. 21–24 are isolated for testing usage.
• Train Set Shape: (1248, 216, 1)
• Valid Set Shape: (312, 216, 1)
• Test Set Shape: (320, 216, 1) — (Isolated)

I re-trained the model with the new data-splitting setting and here is the result:

Benchmark:

From the train valid loss graph, we can see the model cannot even converge well with 10 target classes. Thus, I decided to reduce the complexity of my model by recognizing male emotions only. I isolated the two actors to be the test set, and the rest would be the train/valid set with 8:2 Stratified Shuffle Split which ensures there is no class imbalance in the dataset. Afterward, I trained both male and female data separately to explore the benchmark.

Male Dataset

• Train Set = 640 samples from actor 1- 10.
• Valid Set = 160 samples from actor 1- 10.
• Test Set = 160 samples from actor 11- 12.

Male Baseline

Female Dataset

• Train Set = 608 samples from actress 1- 10.
• Valid Set = 152 samples from actress 1- 10.
• Test Set = 160 samples from actress 11- 12.

Female Baseline

As you can see, the confusion matrix of the male and female model is different.

- Male: Angryand Happy are the dominant predicted classes in the male model but they are unlikely to mix up.​

- Female: Sad and Happy are the dominant predicted classes in the female model and Angry and Happy are very likely to mix up.

Referring to the observation form the EDA section, I suspect the reason for female Angry and Happy are very likely to mix up is because their expression method is simply increasing the volume of the speech.

On top of it, I wonder what if I further simplify the model by reducing the target class to Positive, Neutral and Negative or even Positive and Negative only. So, I grouped the emotions into 2 class and 3 class.

2 Class:

• Positive: happy, calm.
• Negative: angry, fearful, sad.

3 Class:

• Positive: happy.
• Neutral: calm, neutral.
• Negative: angry, fearful, sad.

(Added neutral to the 3 class to explore the result.)

Before I do the training experiment, I tune the model architecture with the male data by doing 5 class recognition.

# Set the target class number
target_class = 5

# Model 
model = Sequential()
model.add(Conv1D(256, 8, padding='same',input_shape=(X_train.shape[1],1))) #1
model.add(Activation('relu'))
model.add(Conv1D(256, 8, padding='same')) #2
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Dropout(0.25))
model.add(MaxPooling1D(pool_size=(8)))
model.add(Conv1D(128, 8, padding='same')) #3
model.add(Activation('relu')) 
model.add(Conv1D(128, 8, padding='same')) #4
model.add(Activation('relu'))
model.add(Conv1D(128, 8, padding='same')) #5
model.add(Activation('relu'))
model.add(Conv1D(128, 8, padding='same')) #6
model.add(BatchNormalization())
model.add(Activation('relu'))
model.add(Dropout(0.25))
model.add(MaxPooling1D(pool_size=(8)))
model.add(Conv1D(64, 8, padding='same')) #7
model.add(Activation('relu'))
model.add(Conv1D(64, 8, padding='same')) #8
model.add(Activation('relu'))
model.add(Flatten())
model.add(Dense(target_class)) #9
model.add(Activation('softmax'))
opt = keras.optimizers.SGD(lr=0.0001, momentum=0.0, decay=0.0, nesterov=False)

I added 2 Conv1D layers, 1 MaxPooling1D layer and 2 BarchNormalization layers, moreover, I changed the dropout value to 0.25. Lastly, I changed the optimizer to SGD with 0.0001 learning rate.

lr_reduce = ReduceLROnPlateau(monitor=’val_loss’, factor=0.9, patience=20, min_lr=0.000001)

mcp_save = ModelCheckpoint(‘model/baseline_2class_np.h5’, save_best_only=True, monitor=’val_loss’, mode=’min’)

cnnhistory=model.fit(x_traincnn, y_train, batch_size=16, epochs=700, validation_data=(x_testcnn, y_test), callbacks=[mcp_save, lr_reduce])

For the model training, I adopted Reduce Learning On Plateau and save the best model with the min val_loss only. And here are the model performance of different target class setups.

New Model Performance

Male 5 Class

Female 5 Class

Male 2 Class

Male 3 Class

Augmentation

After I tuned the model architecture, optimizer and learning rate schedule, I found out the model still cannot converge in the training period. I assumed it is the data size problem since we have 800 samples for train valid set only. Thus, I decided to explore the audio augmentation methods. Let’s take a look at some augmentation method with code. I simply augmented all of the datasets once to double the train / valid set size.

Male 5 Class:

Dynamic Value Change

def dyn_change(data):
    """
    Random Value Change.
    """
    dyn_change = np.random.uniform(low=1.5,high=3)
    return (data * dyn_change)

Pitch Tuning

def pitch(data, sample_rate):
    """
    Pitch Tuning.
    """
    bins_per_octave = 12
    pitch_pm = 2
    pitch_change =  pitch_pm * 2*(np.random.uniform())   
    data = librosa.effects.pitch_shift(data.astype('float64'), 
                                      sample_rate, n_steps=pitch_change, 
                                      bins_per_octave=bins_per_octave)

Shifting

def shift(data):
    """
    Random Shifting.
    """
    s_range = int(np.random.uniform(low=-5, high = 5)*500)
    return np.roll(data, s_range)

White Noise Adding

def noise(data):
    """
    Adding White Noise.
    """
    # you can take any distribution from https://docs.scipy.org/doc/numpy-1.13.0/reference/routines.random.html
    noise_amp = 0.005*np.random.uniform()*np.amax(data)
    data = data.astype('float64') + noise_amp * np.random.normal(size=data.shape[0])
    return data

We can see that the augmentation can jack up the Validation Accuracy a lot, 70+% in general. Especially that adding white noise can achieve 87.19% Validation Accuracy, however, the Testing Accuracy and Testing F1-score dropped more than 5% respectively. Then, I wonder if I mixing different augmentation methods would bring a good result.

Mixing Multiple Methods

Noise Adding + Shifting

Testing Augmentation on Male 2 Class Data

Male 2 Class:

Noise Adding + Shifting
For all sample

Noise Adding + Shifting
For positive sample only since the 2 class set is imbalance (skewed toward negative).

Pitch Tuning + Noise Adding
For all sample

Pitch Tuning + Noise Adding
For positive sample only

Conclusion

In the end, I only have time to experiment with the male data set. I re-split the data with stratified shuffle split to make sure there is no data imbalance nor data leakage problem. I tuned the model by experimenting the male dataset since I want to simplified the model at the beginning. I also tested the by with different target label setups and augmentation method. I found out Noise Adding and Shifting for the imbalanced data could help in achieving a better result.

Key Take Away

• Emotions are subjective and it is hard to notate them.
• We should define the emotions that suitable for our own project objective.
• Do not always trust the content from GitHub even it has lots of stars.
• Be aware of the data splitting.
• Exploratory Data Analysis always grant us good insight, and you have to be patient when you work on audio data!
• Deciding the input for your model: a sentence, a recording or an utterance?
• Lack of data is a crucial factor to achieve success in SER, however, it is complex and very expensive to build a good speech emotion dataset.
• Simplified your model when you lack data.

Further Improvement

• I only selected the first 3 seconds to be the input data since it would reduce the dimension, the original notebook used 2.5 sec only. I would like to use the full length of the audio to do the experiment.
• Preprocess the data like cropping silence voice, normalize the length by zero padding, etc.
• Experiment the Recurrent Neural Network approach on this topic.

About Me

• GitHub: rezachu/emotion_recognition_cnn
• Linkedin: Kai Cheong, Reza Chu

Remarks: Some papers related to this topic will be noted in the GitHub soon.

Speech Emotion Recognition with Convolution Neural Network was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this guide, I have chosen an expired Kaggle competition — Plant Seedlings Classification to be the template project. And the guide is constructed by five parts.

a. Google Cloud Storage Utilization
b. Initialize GPU Compute Engine
c. Install CUDA, cuDNN & Tensorflow-GPU
d. Install Jupyter Notebook
e. Running Keras Transfer Learning model with GPU

Part A: Google Cloud Storage Utilization

Step: 1
In the beginning, we need to apply a Google Cloud Platform (GCP) account and the new client would have $300 credit to spend in the first 12 months.

Once we logged in to the GCP console, we need to create a new project in order to use the GCP services. Simply click the New Project button and type the project name. At the same time, it would assign a unique project ID for your project which we need to use to access the GCP in terminal later.

Step: 2
Now, we would create a bucket in the Storage. There are three ways to create a bucket in the Storage.

1. Web UI
2. Command Line
3. Programmatically (Not going to talk about it)

Web UI:
We simply need to click the CREATE BUCKET button and configure the bucket setting.

Command Line:
This one is a bit tricky. First, we need to create an environment with Python 2.7. Then, we would install the Google Cloud SDK. Since I use MacOS, I would use it as an example. To install and initiate the Google Cloud SDK, we only need to type three lines of code in the terminal.

• Enter the following at a command prompt:

curl https://sdk.cloud.google.com | bash

• Restart the Shell:

exec -l $SHELL

• Run gcloud init to initialize the gcloud environment:

gcloud init

After running “gcloud init”, we simply need to select the target project created and log in with our Gmail. And here is an Installation Quickstarts Guide where you can choose the appropriate method according to your machine setting.

Now, we can start to work with the gsutil tool. Here are a couple of major functions.

#Create a bucket:
gsutil mb gs://<bucket_name>

#Copy some files to the bucket:
gsutil cp <file_name_*> gs://<bucket_name>

#List the files in the bucket:
gsutil ls -l gs://<bucket_name>

#Copy a file from our bucket back to the target directory
gsutil cp gs://<bucket_name>/<file_name_*> /<target_dir>

#Delete the files:
gsutil rm gs://<bucket_name>/<file_name_*>

#Remove the bucket:
gsutil rb gs://<bucket_name>

Here are few things we need to upload to the bucket:
- Training & Testing data
- pre-trained model

I created a folder name “plant-seedling” which has two sub-folders “Train” & “Test” to store the data separately. Also, we can download the pre-trained model weight of VGG16 & VGG19 here. Once it finished uploading these files to the bucket, we can move on to the next part.

Part B: Initialize GPU Compute Engine

Step 1:
First, we need to click into the compute engine page of the GCP console.

Then, we need to click the “CREATE INSTANCE ” button at the top bar and we will get directed to this page.

Step 2:
In this phase, we need to configure the instance setting before creating it.

In the figure above, I underlined some items that we should change. First, we need to assign a unique name for this instance. Then, we have to choose the regions and zone for the instance. Please be reminded that only some of the regions would provide GPU instance service. We can check the information through this link: https://cloud.google.com/compute/docs/gpus/

After that, we need to decide the machine type and no worry since we can change the machine type at any time we want even the instance is created. In this case, I choose a high memory 4 cores CPU machine with 1 NVIDIA Tesla K80 GPU. You could adjust the machine type according to your demand and you can check the price with the GCP Price Calculator here.

In the lower part of the configuration page, we would check both HTTP & HTTPS traffic boxes at the Firewall. And then, we click the “Change” button at the Boot Disk section. (The default is Debian GNU/Linux 9 (stretch) with 10 GB memory)

Once we clicked in, we could select the Ubuntu 16.04 LTS disk and change the size to 100 GB since we have a thousands of image to train and need to reserve memory space for pre-trained-weight and the best trained models.

Step 3:
The last step before launching our instance, we need to request Google to increase our GPU quota since the default would be zero and normally it only takes few hours for approval.

Normally, if we start our GPU instance now, it will pop up this notification on the right top corner. What we need to do is to click the request increase button.

To edit the quota, we need to click the Compute Engine API with the same GPU (NVIDIA K80) we have chosen in the machine-type setting and the location has to be as same as the region of the instance. KINDLY REMINDED that do not choose the Preemptible NVIDIA K80 one since it is for the Preemptible instance which would be automatically removed after 24 hours. Then we can click the EDIT QUOTAS button at the top to increase the GPUs quota and wait for Google’s approval.

Congratulation, we finally can start our first GPU instance by clicking the Start button!

Alert: The instance would start to charge once you turned it on, therefore, please be reminded to stop it if you are not using it.

Step 4:
In this part, we would start the instance and create a Python 3 virtual environment.

Once we clicked the SSH button, it would pop up a new window that has a terminal view. Then we only need to run these few lines of code step by step in order to set up our python environment.

# Download the Anaconda3
wget http://repo.continuum.io/archive/Anaconda3-4.0.0-Linux-x86_64.sh

# Install Anaconda3
bash Anaconda3-4.0.0-Linux-x86_64.sh

# It will ask you this question and we would enter yes
Do you wish the installer to prepend the 
Anaconda3 install location to PATH 
in your /home/haroldsoh/.bashrc ? 
[yes|no][no] >>> yes

# Use it right away
source ~/.bashrc

# Create a Python 3 virtual environment
conda create -n <env_name> python=3.5

# Activate our environment
source activate <env_name>

Part C: Install CUDA, cuDNN & Tensorflow-GPU

Step 1:
To use the GPU function in our instance, we need to download couples of a package. The first one is the CUDA tool-kit and we can install it by running these codes.

# Activate your environment
source activate <env_name>

# download the CUDA 9.2
curl -O https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_9.2.88-1_amd64.deb

# install the CUDA
sudo dpkg -i cuda-repo-ubuntu1604_9.2.88-1_amd64.deb

# Downgrade it to 9.0 since tensorflow-gpu can only use v9.0
sudo apt-get update
sudo apt-get install cuda-9-0

# check if CUDA is installed
nvidia-smi

If CUDA successfully installed, it would show something like this:

There is another alternative to install the CUDA because sometimes the curl might not work well. We could download the CUDA package here and upload it to the bucket and then run this two commands to copy the file to the instance.

# Initialize GCP in the instance
gcloud init

# Transfer the file to your instance
gsutil cp -r gs://<bucket_name>/<file_path> /home/<username>

# install the CUDA
<Follow the upper commands>

Step 2:
Before we install the cuDNN, we need to register a Nvidia Developer Account first. Then, we need to download two packages for the Ubuntu instance.

We have to download:
- cuDNN v7.1.4 Runtime Library for Ubuntu16.04 (Deb)
- cuDNN v7.1.4 Developer Library for Ubuntu16.04 (Deb)

Next, we can use the bucket upload & file transfer method I mentioned above to put these two files into the instance home path.

Lastly, we would run these commands to install the packages. Please be remembered to install the Runtime Library first, otherwise, it would return an error.

# install the cuDNN Runtime Library
sudo dpkg -i libcudnn7_7.1.4.18-1+cuda9.0_amd64.deb

# install the cuDNN Developer Library
sudo dpkg -i libcudnn7-dev_7.1.4.18-1+cuda9.0_amd64.deb

Step 3:

Installing tensorflow-gpu package is the easiest part of this guide. We simply need to run two codes.

# install the tensorflow-gpu
pip3 install tensorflow-gpu

# Make sure the package is up to date
pip3 install --upgrade tensorflow-gpu

Step 4:
To verify if the GPU is ready to use, we would have to run these few lines of code.

# enter python 
python

# check tensorflow
>>>import tensorflow as tf
>>>from tensorflow.python.client import device_lib
>>>print(device_lib.list_local_devices())

It should return something like this if everything was installed correctly:

Part D: Install Jupyter Notebook

Before we install the Jupyter notebook, we need to configure the instance network setting.

Step 1:
Since the initial IP is dynamic, so we would need to go to VPC network > External IP address page to configure the IP type from Ephemeral to Static.

Step 2:

After we clicked into the Firewall rules page below the External IP addresses tab. We need to create a new firewall rule.

We would insert two items into the configuration setting.

We would enter “0.0.0.0/0” for the Source IP ranges and specify the protocols and ports to “tcp:<port_number>” and I selected tcp:4000 for my rule.

Before we move on, please ensure the external IP address is In use by our target instance. We can click the change button on the right-hand side to do the setting.

Step 3:
Now, we can start the SSH terminal to install our Jupyter Notebook.

# Activate your environment
source activate <env_name>

# download the jupyter notebook to your environment
pip3 install Jupyter Notebook

Once the Jupyter Notebook package is installed, we need to configure the Jupyter Notebook setting.

# Generate Configuration file
jupyter notebook --generate-config

# Check if the configure file exist
ls ~/.jupyter/jupyter_notebook_config.py

# change directory to target path
cd /home/<uesr_name>/.jupyter/

# open jupyter_notebook_config.py
nano jupyter_notebook_config.py

And we need to add these few lines into the configuration file.

c = get_config()
c.NotebookApp.ip = '*'
c.NotebookApp.open_browser = False
c.NotebookApp.port = <Port Number>

The jupyter_notebook_config.py should look something like this and I use green underline to highlight where should we insert the code.

Save and quit the file and we are good to launch the Jupyter Notebook now!

Step 4:
Launching Jupyter Notebook in the Google Cloud instance is a bit different from what we do on our local machine. The command line is variated and we need to open the notebook by ourselves.

# Launch the Notebook
jupyter-notebook --no-browser --port=<PORT-NUMBER>

# Insert this url in the browser url section to open notebook
http://<External Static IP Address>:<Port Number>

Now we should see something like this:

Sometimes, it might ask you to insert password or token by showing this page:

Don’t worry, it is a normal security precaution of Jupyter Notebook to prevent random website you visit from executing code on your machine. If we look up the SSH terminal, you will find out an URL in this format.

# First time visit token
http://<bucket_name>:<port_number>/?token=<token>

All we need to do is to copy the <token> part and paste it to the Token column and set a new password in order to log in the notebook smoothly.

Part E: Running Keras Transfer Learning model with GPU

The Kaggle Plant Seedlings Classification project provided 4750 training images divided into 12 classes and 794 testing images and all of them have different resolution and size. As a result, we need to crop the image and downsize them before feeding into our Keras model. On top of it, the data is imbalance since some of the classes have significantly more samples than other classes. Here is a brief glance at the data.

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The number of category is 12
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Here are the training data intel:
- Black-grass 263 images
- Charlock 390 images
- Cleavers 287 images
- Common Chickweed 611 images
- Common wheat 221 images
- Fat Hen 475 images
- Loose Silky-bent 654 images
- Maize 221 images
- Scentless Mayweed 516 images
- Shepherds Purse 231 images
- Small-flowered Cranesbill 496 images
- Sugar beet 385 images

In this transfer learning model, I adopted VGG 16 with pre-trained “imagenet” weights. The input shape of the model is (224, 224, 3) and the output shape is (12). And I simply removed the last dense layer with output shape 1000 and added a dense layer with output shape 12, “softmax” activation and the weight initializer was “glorot_norma”l ( aka Xavier Normal). And here is the model highlight.

# VGG_16 model above with the last layer
model.add(Dense(1000, activation='softmax'))    
    
# Load VGG_16 pretrained weight
model.load_weights('./pre_trained_model/vgg16_weights.h5', by_name=True)

# Remove the last softmax Dense layer
model.pop()

# Add a new Dense layer to output 12 classes
# Initialize the weight with Xavier Normal Initialization
model.add(Dense(12, activation='softmax', kernel_initializer='glorot_normal'))
    
# Using SGD as an optimizer with learning rate=0.001 and decay = 0.000001
sgd = SGD(lr=1e-3, decay=1e-6, momentum=0.9, nesterov=True)

# Compile model with custom fscore metric 
model.compile(optimizer=sgd, loss='categorical_crossentropy', metrics=['accuracy', fscore])

Basically, with all the setup we made before, the Jupyter Notebook would use GPU to run automatically. And, I didn’t do much data preprocessing and hyperparameter tuning since it is a walk through guide project. I simply ran 20 epochs with a batch size = 32, each epoch took around 130 seconds where the CPU required at least 6000 seconds per epoch. Therefore, the accuracy is not that high but the training fscore was still higher than 0.89.

Epoch 1/20
3800/3800 [==============================] - 144s 38ms/step - loss: 2.4417 - acc: 0.1300 - fscore: 0.0000e+00 - val_loss: 2.4202 - val_acc: 0.1221 - val_fscore: 0.0000e+00

.
.
.

Epoch 20/20
3800/3800 [==============================] - 126s 33ms/step - loss: 0.2892 - acc: 0.8887 - fscore: 0.8912 - val_loss: 0.4537 - val_acc: 0.8495 - val_fscore: 0.8551

The whole Jupyter Notebook can be cloned on my Github and my next blog post will talk about how to increase the accuracy of this project. Hopefully, it could help anyone who has no idea how to train their model on cloud and stay tuned!

Complete Step by Step Guide of Keras Transfer Learning with GPU on Google Cloud Platform was originally published in DataDrivenInvestor on Medium, where people are continuing the conversation by highlighting and responding to this story.

Lately, I had accomplished Andrew Ng’s Deep Learning Specialization course series in Coursera. I hope this review would be insightful for those whom might want to enter this field or simply consolidate your deep learning knowledge.

Course Overview

1. Deep Learning and Neural Network:
   In course 1, it taught what is Neural Network, Forward & Backward Propagation and guide you to build a shallow network then stack it to be a deep network. Also, you will learn about the mathematics (Logistics Regression, Gradient Descent and etc.) related to it step by step. Andrew explained the maths in a very simple way that you would understand it without prior knowledge in linear algebra nor calculus. Last but not least, it discussed the activation functions and their Pros & Cons respectively. — Favorite topic: how to get your matrix dimension right
2. Improving DNN: Hyperparameter Tuning, Regularization, and Optimization:
   In course 2, we would proceed to learn different regularization techniques and why they could reduce overfitting. On top of it, Andrew would teach you how to evaluate your deep learning model and note down his hyperparameter tuning technique in different situations (It is a gold for DL Enthusiast) — favourite topic: Minibatch application and how it affects the model.
3. Structuring Machine Learning Projects:
   In course 3, Andrew would talk about how to set up evaluation metric since the highest accuracy model might not be the best in daily practice. He also discusses how to select and split the train/test/validation set and the methods we could use while lacking of data — favourite topic: error analysis table which drill down the prediction results to look for model tuning insights.
4. Convolutional Neural Network:
   In course 4, Andrew would teach what is computer vision and how does a Convolutional Neural Network work by explaining the theory and maths of convolutional filter, Maxpooling filter, etc. On top of it, he would go through the structure of the famous pre-trained network: LeNet5, AlexNet’ VGG-16, and ResNet, then guide you to utilize them with transfer learning. — favourite topic: Image localisation and detection.
5. Sequence Models:
   In the last course, it first talks about RNNs and LSTM then dive in to NLP and teach the major concepts such as Word Embedding, Word2Vec, Bleu Score, Beam Search, Skip Gram, etc. Lastly, you could practice how to build a machine translation model .— favourite topic: Trigger Detection with the self-made audio data.

Why you should take it

Course Design:
The course was designed to educate Deep Learning in a simple way in order to lower the entry barrier of this industry and boost up the development of Artificial Intelligence. These five courses are a step by step series to cover all fundamental aspect of deep learning although you could only take those you are interested. It is absolutely suitable for Deep Learning beginner with fundamental Python programming skill.

Programming Assignment:
The program assignments after each week’s lecture are very practical, especially it requires you to code every function you have learned in the class instead of simply calling the framework. For example, it requires you to code the forward & backward propagation, gradient descent, splitting minibatch and etc. in Python. Although it is not a difficult task that could transform you to be a coding veteran, this process could really help the beginner understanding the mechanism behind it. And for the DL partitioner, it is a good chance to consolidate your knowledge of the Neural Network. On top of it, all of the assignments were graded automatically, so you can get the result right away and proceed asap. Lastly, the classroom forum would provide all you need to solve the assignment. And the honour of code prevented students from posting the actual code on the forum. In the other words, you have to pass all of the assignment by yourself. It might be challenging for a beginner but you would get the reward at the end of the course.

Heroes of Deep Learning Interview:
Despite the great course content that enables us to build and train Deep Learning model. We all know the wide variation of DL application requires us to iterate countless experiments to bring the model out from the lab to daily practice. Andrew and the guests including Geoffrey Hinton, Pieter Abbeel, Ian Goodfellow, etc. shared a lot of experiences they learned from their works and their career path in the industry which is gold to us.

Course Certification:
After you accomplished the courses it would issue 5 course certifications plus one deep learning specialization certification which could directly attach to your Linkedin profile. For those who want to switch the career path, I could say this course could really grant you the knowledge you expected and the validation from an authority.

Course Workload and Pricing:
Each course would have four-week syllabus in average which requires to devote you 2 to 4 hour a week. The workload is not big at all for people who have a full-time job. And the course fee is only $49 per month with 7 days free trial which is arguably one of the cheapest MOOC course I have ever taken.

Conclusion

After taking this course, I can foresee more and more DL talent would pop out since the DL knowledge enables us to drill down to the topic we are interested in and connecting us to the entry of this industry. I would recommend this course to everyone who is interested in Deep Learning and not only for beginner but for those who has knowledge in this field as well. Knowledge consolidation is always good and teaches you new stuff. If you don’t want to miss the A.I. trend in this decade, dive in the course now.

Related Posts from DDI:

• Deep Learning Explained in 7 Steps - Updated | Data Driven Investor
• Which is More Promising: Data Science or Software Engineering? | Data Driven Investor

Andrew Ng’s Deep Learning Specialization Course Review was originally published in DataDrivenInvestor on Medium, where people are continuing the conversation by highlighting and responding to this story.

In the past two years, Artificial Intelligence has become one of the hottest topics. Thanks to AlphaGo, the general public believes A.I. is a monster that had beaten human and sooner it is going to take their jobs away. As a machine learning engineer, I see the necessity to explain what is A.I. to the layman which could clear the illustration forged by the media. Here we go.

“AI is all about figuring what to do when you don’t know what to do.”, Peter Norvig.

Artificial Intelligence is not a magical program that can think like a human. The mechanism behind it is to utilise different mathematic algorithms to learn from a tremendous amount of structured data to solve dedicated cognitive problems that originally can be only solved by human intelligence. And the robust part of A.I. is it can acquire a pattern from the provided data to solve a problem in high accuracy. Meanwhile, we don’t even know what is actually happening inside the program but the algorithm would take care of it. In a simple word, A.I. learns from the data feed by human to solve dedicated problem. Let’s take a look to these six major subfields in A.I.

1. Computer Vision:
e.g. Recognizing object from real world images or video, just like the iPhone X’s Face ID.

2. Planning:
e.g. Searching for the shortest path between your home and the nearest shopping mall.

3. Natural Language Processing:
e.g. Interpreting what are you saying, writing, etc.

4. Knowledge Representation and Reasoning:
e.g. Chatbot that answer customer questions.

5. Robotics:
e.g. The self driving car that take care of your travel.

6. Machine Learning:
e.g Predicting the housing price by learning from the historical data.

How does Artificial Intelligence Agent solve problems?

So, how does A.I. learn? Typically, there are three learning methods, Supervised Learning, Unsupervised Learning and Reinforcement Learning. In supervised learning, we would label the correct answer inside the data and the agent would learn the pattern by comparing the result with the target label in order to achieve accurate prediction.; In unsupervised learning, we would not tell what we are looking for but let the learning algorithm to help us divide the data into category as many as we desired.; Lastly, reinforcement learning is more like learning by doing. At the beginning, we do not have any labeled data, however, we would set up different policies with corresponding reward or penalty. The A.I. would learn the optimal by trying different actions in each condition to achieve the highest reward under the specified environment and policies.

Can Artificial Intelligence Agent replace human?

To develop an A.I. agent to solve a task, there are four basic criteria.

• 1. We need to clearly define the problem domain and the learning objective.
• 2. We have to provide structured data for the agent or we are able to design a good environment, policies and reward system for reinforcement learning.
• 3. The data provided must contain useful information for the agent to learn from it. If the data fed in have no value, the agent would never be able to solve the task.
• 4. Decide an evaluation method to measure the performance of the agent. Otherwise, we won’t know the model is accurate or not.

Rule no.1: Garbage In, Garbage Out.

From my experience, the hardest part of building a prediction model is to decide what sample data should I use and how to gather those data. Luckily, there are many open-sourced data and algorithm library that could let us to train our model much easier than few year before. However, you could also gather the desired data by yourselves.

Conclusion ~ Will it take our job?

Undoubtedly, A.I. would disrupt our world in the soon future, however, we should firstly understand it and think of the opportunities created by A.I. instead of blindly worrying the fourth industrial revolution would replace all of the jobs. A.I. technology required a large amount of training data to look for a specific pattern in order to resolve the problem. And it has its own bottleneck such as: Do we have the data? Does the specific problem has a pattern? Can we structure the information into computer interpretable data? Is the computer powerful enough to handle a huge amount of data. Moreover, A.I. would soon replaces jobs that are highly repeatable, however, it would also create many job posts, for instance: data scientist, data security engineer, etc. On top of it, applying the new technology to the real world needs to solve many political, legislative and humane problems which might take years. Let’s say the self driving car application, lots of researches and experiments proved that self driving car is safer than human driving. However, once it killed people in an accident (which happened lately in U.S.), who is going to take the responsibility? In addition, do we need a driver sitting in the driver seat in case of any malfunction, so he can switch it to manual mode to save lives? If a driver is compulsory for self driving car after legislation, it ain’t gonna bring a big impact on the driver job.

It is very likely a large amount of traditional labor force, especially the grassroots class would be replaced by Robots and AI, however, the replacing process might take years and the society would have time to digest it gradually. The most important thing is we should learn about what is AI and think about how is it going to change our job instead of blindly panic. Hopefully this read could give you a brief glance of the A.I. world and motivated you to learn more about A.I.

What is AI? A Brief Explanation for layman was originally published in DataDrivenInvestor on Medium, where people are continuing the conversation by highlighting and responding to this story.