Image Dataset Download and Setup

Here is the data used. We will use a google colab for this task. Place the data in your google drive and mount it to your colab notebook.

We will be classifying eyes_data. so we created 2 folders corresponding to 2 different types of eye classes. We have normal and those with keratoconus, as shown below.

Note: we have very low amounts of data. There are 48 training images and 12 validation images. This is considered low amount of data for any machine learning task.

Approach

We will be using a Convolution Neural Network(CNN) with a ResNet34 architecture.

First, what’s CNN again? We can think of a Convolutional Neural Network (CNN or ConvNet) as a list of layers that transform the image volume into an output volume, which can be a class score as it is the case in this tutorial.

Transfer Learning

Transfer learning is a technique that addresses the two problems posed by deep learning:

1. Training a CNN requires a large number of images due to the high number of parameters it contains.
2. Training a CNN requires large computational power.

Transfer learning relies on the fact that the lower layers of the CNNs that were trained on natural images contain generic, low-level features like edges, colors, etc regardless of the specific images in the data set. Using transfer learning makes it possible to fine-tune a CNN on a small data set in a relatively short time.

What is special about ResNet architecture is how it tackles the degradation problem most common in deep networks, where the model accuracy gets saturated and then degrades rapidly.

Resnet architecture introduces an “identity shortcut connection” or often referred to as a “skip connection”, which skips one or more layers. The shortcut connections simply perform identity mappings, and their outputs are added to the outputs of the stacked layers, as shown in the figure below. The skip function creates what is known as a residual block, F(x) in our figure, and that’s where the name Residual Nets (ResNets) came from. connection” or often referred to as a “skip connection”, which skips one or more layers. The shortcut connections simply perform identity mappings, and their outputs are added to the outputs of the stacked layers, as shown in the figure below. The skip function creates what is known as a residual block, F(x) in our figure, and that’s where the name Residual Nets (ResNets) came from.

Comprehensive empirical evidence has shown that the addition of these identity mappings allows the model to go deeper without degradation in performance and such networks are easier to optimize than plain stacked layers. There are several variants of ResNets, such as ResNet50, ResNet101, ResNet152; the number represents the number of layers (depth) of the ResNet.

We split the data into 80% training and 20% test data using imageDataBunch then proceed to create a learner.

Picking a learning rate

Use the above plot to pick adequate learning rates for your model. We need two learning rates since we are using cyclic learning rates:

• The first learning rate is just before the loss starts to increase, preferably 10x smaller than the rate at which the loss starts to increase.
• The second learning rate is 10x smaller than the first learning rate, so 1e-03 in our example.

Result Visualization

We use a confusion matrix and look at how sample images were classified.

What can we learn from this matrix?

• There are two possible predicted classes: “kt_eyes” and “normal_eyes”. The classifier made a total of 12 predictions.
• Out of those 12 cases, the classifier predicted “kt_eyes” 8 times, and “normal_eyes” 4 times.
• In reality, 5 patients in the sample have normal_eyes, and 7 patients had kt_eyes.

Let’s now define the most basic terms, which are whole numbers (not rates):

• true positives (TP): These are cases in which we predicted kt_eyes (they have the disease), and they do have the disease.
• true negatives (TN): We predicted normal eyes, and they don’t have the disease.
• false positives (FP): We predicted kt_eyes, but they don’t actually have the disease. (Also known as a “Type I error.”)
• false negatives (FN): We predicted normal_eyes, but they actually do have the disease. (Also known as a “Type II error.”)

Our accuracy was 91.6% (TP+TN)/total

We can also plot images with top losses; in other words, the images that the model was most confused about. A high loss implies high confidence about the wrong answer.

Areas of further Research/ Improvement

1. Dataset; More images are required to build a robust model.
2. Further improvements/work: Build the CNN with Keras.

References

KeratoDetect: Keratoconus Detection Algorithm Using Convolutional Neural Networks

Conclusion

The full code can be found here.

Thank you for reading, follow @itsmuriuki on Twitter.

Back to code and more learning.

Thanks kanjuru and kenneth for going through the draft.

Problem

There is large consent that successful training of deep networks requires many thousand annotated training samples. The paper presents a network and training strategy that relies on the strong use of data augmentation to use the available annotated samples more efficiently.

The typical use of convolutional networks is on classification tasks, where the output to an image is a single class label. However, in many visual tasks, especially in biomedical image processing, the desired output should include localization i.e. a class label is supposed to be assigned to each pixel. Moreover, thousands of training images are usually beyond reach in biomedical tasks.

Solution

The U-Net architecture is built upon the Fully Convolutional Network and modified in a way that it yields better segmentation in medical imaging. The paper use’s excessive data augmentation by applying elastic deformations to the available training images. This allows the network to learn invariance to such deformations, without the need to see these transformations in the annotated image corpus.

Architecture

The u-net comprises of two parts an encoder/contraction path(left side) and a decoder/expansion path(right side).

Contraction path consists of a repeated application of a 3x3 convolutions(unpadded) each followed by a ReLU and a 2x2 max pooling operation with stride 2 for downsampling. At each downsampling step, we double the number of feature channels. This captures context via a compact feature map.

The expansion path consists of upsampling of the feature map followed by a 2x2 convolution(“up-convolution”) that halves the number of feature channels a concatenation with the cropped feature map from the contracting path, and a 3x3 convolutions, followed by a ReLU. The upsampling of the feature dimension is done to meet the same size as the block to be concatenated on the left.

The expansion increases the “what” which helps in getting more features but losses the localization, localization information is concatenated from the contraction path.

The cropping is necessary due to the loss of border pixels in every convolution. At the final layer, a 1x1 convolution is used to map each 64-components feature vector to the desired number of classes. In this case, it is 2 as the output feature map has 2 classes; cells and membrane.

The main contribution of this paper

a. Overlap- tile strategy

Prediction of the segmentation in the yellow area requires image data within the blue area as input. missing input data is extrapolated by mirroring, this is used to predict pixels in the border region of the image.

b. Data augmentation by applying elastic deformations to training images.

This allows the network to learn invariance to such deformations, without the need to see these transformations in the annotated image corpus. This is important in biomedical segmentation since deformation is the most common variation in tissue and realistic deformations can be simulated efficiently.

c. Separation of touching objects of the same class.

This is done using a weighted loss, where the separating background labels between touching cells obtain a large weight in the loss function. This force the network to learn the small separation borders between touching cells.

Real world application of this contribution

Thousands of training images are beyond reach in biomedical tasks and require experts and take lots of time to annotate. This could automate the process thus lowering the cost and time it takes to annotate.

This can also be applied in other areas such as quality control and inspection and manufacturing.

In this paper, the techniques were applied in the segmentation of neuronal structures in electron microscopic recordings, cell segmentation task in light microscopic images and HeLa cells on a flat glass recorded by differential interference contrast microscopy.

References

[1] Ronneberger, O., Fischer, P., & Brox, T. (2015, October). U-net: Convolutional networks for biomedical image segmentation.

Thanks for reading! follow @itsmuriuki.

Back to learning!

Paper Summary: U-Net: Convolutional Networks for Biomedical Image Segmentation was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Problem

Adding more layers leads to the model’s accuracy saturating, then rapidly decaying, and higher training errors — the degradation problem.

Driven by the significance of network depth, a question arose: Is learning better networks as easy as stacking more layers? An obstacle to this question was the vanishing/exploding gradients problem.

The degradation of the training accuracy indicates that not all systems are similarly easy to optimize. Considering a shallower architecture and its deeper counterpart that adds more layers onto it: the added layers are identity mapping and the other layers are copied from the learned shallower model. therefore, a deeper model should produce no higher training error than its shallower counterpart.

Solution: Deep residual learning framework

Instead of hoping every few stacked layers directly fit a desired underlying mapping, we explicitly let these layers fit a residual mapping.

If H(x) is the underlying mapping of data to be fit by a few layers, then normally these layers try to learn a function F(x) that equals H(x), with x denoting the input to the network.

In ResNets, layers learn a residual function, F(x) = H(x) — x. Then x is added to the result afterwards, this is known as a shortcut connection. If the dimensions of x and F(x) are not equal e.g when changing the input/output channels, then a linear projection can be added to match their dimensions.

The identity shortcut can be directly used when the input and output are of the same dimension. when dimensions increase: (A) The shortcut still performs identity mapping, with extra zero entries padded for increasing dimension; (B) The projection shortcut is used to match dimensions (done by 1x1 convolutions).

The author’s hypothesis is that it is easy to optimize the residual mapping function F(x) than to optimize the original, unreferenced mapping H(x).

x = number of inputs

H(x) = mapping function from input to output, normally H(x) would be learned)

F(x) = residual function

proposed: F(x) = H(x) — x

F(x ) is learned and H(x) is recovered by adding x

Evaluation

plain and residual net CNN of 18 and 34 layers were trained.

In plain CNN the error increases for plain networks when depth is increased from 18 to 34 layer while it reduces in ResNet with an increase in depth and the error was lower compared to plain nets.

The main contribution of this paper

1. Residual learning eases optimization.
2. Solved the degradation problem.
3. Faster training of deep neural networks.
4. Decreases the error rate for deeper networks.

Real-world application of this contribution

It led to major improvements in Image recognition, image detection, image localization, and segmentation tasks. It contributed heavily to industrial applications such as medical image analysis, satellite image processing, and robot vision.

This architecture has more layers but lesser parameters than previous models hence making it fast.

Here is a pytorch implementation of ResNet.

References

[1] K. He, X. Zhang, S. Ren, and J. Sun, “Deep Residual Learning for Image Recognition,” 2015.

Thanks for reading! follow @itsmuriuki.

Back to learning!

This blog post series will be updated as I have a second take on the fast ai lessons. These are my personal notes; a strive to understand things clearly and explain them well. Nothing new, only living up this blog.

Fast.ai takes the approach of “here is how to use the software to do something then looks behind the scenes by looking at the details.”

Dropout

learn = ConvLearner.pretrained(arch, data, ps=0.5, precompute=True)

precompute=True precomputes the activations that come out of the last convolutional layer. An activation is a number calculated based on some weights also known as parameters that make up a kernel or filters that are applied to previous layers activation which could otherwise be the inputs or results of other calculations.

By typing the name of your learner object you can actually see the layers in it:

learn

Sequential(
  (0): BatchNorm1d(1024, eps=1e-05, momentum=0.1, affine=True)
  (1): Dropout(p=0.5)
  (2): Linear(in_features=1024, out_features=512)
  (3): ReLU()
  (4): BatchNorm1d(512, eps=1e-05, momentum=0.1, affine=True)
  (5): Dropout(p=0.5)
  (6): Linear(in_features=512, out_features=120)
  (7): LogSoftmax()
)

Linear(in_features=1024, out_features=512) a linear layer means a matrix multiplier. In this case we have a matrix with 1024 rows and 512 columns. It takes in 1024 activations and spits out 512 activations.

ReLu. Replaces the negatives with zeros.

Linear(in_features=512, out_features=120) the second linear layer takes the first linear layer activations 512 activations and put them through a new matrix multiply 512 by 120 and outputs 120 activations.

Softmax — An activation function that returns numbers that adds up to 1 and each of them is between 0 and 1. For minor numerical precision reasons, it turns out to be better to take the log of the softmax than softmax directly. That is why when we get predictions out of our models, we have to do np.exp(log_preds).

What does Dropout(p=0.5) do?

Applying dropout means picking activation in a layer and deleting them. p is the probability of deleting an activation. The output activation do not change much.

Randomly throwing away half of the activations in a layer has an interesting effect, it forces it not to overfit. If a particular activation activation had learned just the exact dog or cat it forces the model to find a representation without a random half of the activations.

If you have done all the data augmetation or provided the model with enough data you could, you are left with very few things you could do but to a large degree you are stuck. Geofrey Hinton et al came up with dropout which is lossely inspired by how the brain works.

p=0.01 you are throwing away 1% of activations it will not change things much at all, therefore it is not going to prevent overfitting i.e will not generalize.

p=0.99 you will throw away 99% of the activations. It will also not overfit thus great for generalisation but will kill your accuracty.

High p values generalize well but decrease you training accuracy, low p values will generalize less well but will give you a better accuracy.

Early in training why is the validation losses better than the training losses given that the dataset it has not seen you don’t expect the losses to be much better? this is because when we look at the validation set we trun off dropout. When doing inference i.e predicting we want to be using the best model we can. Dropout happens during training.

Do you have to do anything to accommodate for the fact that you are throwing away activations? We do not(fast.ai) When we say p=0.5 behind the scene pytorch throws away half the activations and doubles the activation already there therefore the average activation does not change.

In Fast.ai, you can pass in ps which is the p value for all of the added layers. It will not change the dropout in the pre-trained network since it should have been already trained with some appropriate level of dropout:

learn = ConvLearner.pretrained(arch, data, ps=0.5, precompute=True)

We can remove dropout by settingps=0.But after a couple of epochs we start to massively overfit. The training loss is less than the validation loss. With ps=0 dropout is not added to the model at all.

You may have noticed, it has been adding two Linear layers. We do not have to do that. There is xtra_fc (extra fully connected layers) parameter you can set. Which you can pass a list of how big you want each of the extra fully connected layer to be. You do need at least one which takes the output of the convolutional layer (1096 in this example) and turns it into the number of classes 120 dog breeds i.e defined by your problem:

If the xtra_fc is empty it means don’t add any additional linear layers just the one we have to have, thus giving us a minimum model.

Is there a particular p value we should be using by default? For the first layer we have p=0.25 and for the second layer we have p=0.5 that seems to work for most things.If you find it overfiting keep increasing it, say to 0.2. If underfitting decrease it.

ResNet34 has less parameters so it does not overfit as much, but for bigger architecture like ResNet50, you often need to increase dropout.

Is there a particular way in which you can determine if it is overfitted? Yes, you can see the training loss is much lower than the validation loss.You cannot tell if it is too overfitted. Zero overfitting is not generally optimal. The only thing you are trying to do is to get the validation loss low, so you need to play around with a few things and see what makes the validation loss low.

Why does average activation matter? If we delete half of the activations, the next activation which takes them as input will also get halved, and everything after that. For example, fluffy ears are fluffy if activation is greater than 0.6, and now it is only fluffy if it is greater than 0.3, which is changs the meaning. The goal here is delete activations without changing the meaning.

Can we have different level of dropout by layer? Yes, that is why it is called ps and we could pass an array: ps =[0.1,0.2]

There is no rule of thumb for when earlier or later layers should have different amounts of dropout yet. If in doubt, use the same dropout for every fully connected layer. Often people only put dropout on the very last linear layer.

Why monitor loss and not accuracy? Loss is the only thing that we can see for both the validation set and the training set and we are able to compare them. As we will learn later, the loss is the thing that we are actually optimizing so it is easier to monitor and understand what that means.

By adding dropout we seem to be adding some random noise and that means we don’t do much learning, do we need to adjust learning rate? It does not seem to impact the learning rate enough to notice. In theory, it might but not enough to affect us.

Structured and Time Series Data

The are two types of columns in the data:

Categorical — it has a number of “level” e.g. StoreType, Assortment

Continuous — it has a number where the difference or ratio of that number has some kind of meaning. e.g CompetitionDistance

We will not cover data cleaning and feature engineering we will assume that has been done. We need to convert to input compatible with a neural network.

This includes converting categorical variables into contiguous integers or one-hot encodings, normalizing continuous features to standard normal, etc…

Numbers like year, month and day though we could treat them as continuous, we don’t have to. when we decide to make some thing categorical we are telling the neural net to treat every level differently. But, when say it is continuous we are telling it to come up with a smooth function to fit them.

So often things that actually are continuous but do not have many distinct levels (e.g. Year, DayOfWeek), it often works better to treat them as categorical. As each day may behave qualitatively differently.

We get to say which variables are categorical and which ones are continuous, this is a modeling decision you have to make.

If something is coded in your data as “a, b, c” you have to call it as a categorical, if it starts out as continuous you have to choose whether to treat that as categorical or continuous.

Note that the continuous variables are actual floating point numbers. Floating numbers have many levels i.e it has high cardinality e.g. the cardinality of the DayOfWeek is 7.

Do you ever bin continuous variables?Not at the moment but one thing we could do with, say Max_Temperature, is to group into 0–10, 10–20, 20–30, and call that categorical. A group of researchers found that sometimes binning can be helpful.

If you are using year as a category, what happens when a model encounters a year it has never seen before? It will be treated as an unknown category. Pandas has a special category called unknown and if it sees a category it has not seen before, it gets treated as unknown.

If our training data set does not have categories but out test have unknow what will the model do, will it predict? It will be part of the unknown category, it will still predict it will process the value 0 behind the seen, if there are unknown in the training set it will have learnt to predict with them if it hasn’t it will have a random vector.

n = len(joined); n
844338

We have 844338 rows that is the dates at each store.

Loop through cat_vars and turn applicable data frame columns into categorical columns.

Loop through contin_vars and set them as float32 (32 bit floating point) because that is what PyTorch expects. for example Promo, SchoolHoliday

Start with a sample

We tend to start with small sample of the data set. For images that would mean resizing the image to 64 x 64 or 128 x 128.But with structured data we start out with a random sample of rows.

Thus giving us 150000 rows of data to start with.

Looking at the data:

Even though we set some of the columns as “category” e.g. ‘StoreType’, ‘Year’. That was saved internally, Pandas still display as string in the notebook.

proc_df (process data frame) is a fast ai function that:

Takes a dataframe tell it what your dependent variable is, puts it into a separate variable, and deletes it from the original data frame. In other words, df do not have Sales column, and y contains theSales column.

do_scale Neural nets really like to have the input data to all be somewhere around zero with a standard deviation of somewhere around 1. To make this happen we take our data, subtract the mean, and divide it by the standard deviation. It returns a special object mapper which keeps track of what mean and standard deviation it used for that normalization so you can do the same to the test set later.

nas It also handles missing values, for categorical variable, it becomes ID: 0 and other categories become 1, 2, 3, and so on. For continuous variable, it replaces the missing value with the median and create a new column that is boolean that says whether it was missing or not.

Year 2014 for example becomes 2 since categorical variables have been replaced with contiguous integers starting at zero. The reason for that is, we are going to put them into a matrix later, and we would not want the matrix to be 2014 rows long when it could just be two rows.

we now have a data frame which does not contain the dependent variable and where everything is a number. That is where we need to get to to do deep learning.

In this case, we need to predict the next two weeks of sales therefore we should create a validation set which is the last two weeks of our training set.

In time series data, cross-validation is not random. Instead, our holdout data is generally the most recent data, as it would be in real application. This issue is discussed in detail here. One approach is to take the last 25% of rows (sorted by date) as our validation set.

Putting together our model

it is important that you have a strong understanding of your metric i.e how you are going to be judged. In this competition, we are going to be judged on Root Mean Square Percentage Error (RMSPE).

This mean that we take the actual sale minus the predicted value of a single prediction find its percentage then find the average.

When you take the log of the data, getting the root mean squared error will actually get you the root mean square percentage error:

We can create a ModelData object directly from out data frame:

md = ColumnarModelData.from_data_frame(PATH, val_idx, df, 
         yl.astype(np.float32), cat_flds=cat_vars, bs=128, 
         test_df=df_test)

We will start by creating model data object md which has a validation set, training set, and optional test set built into it. From that, we will get a learner, we will then optionally call lr_find, then call learn.fit and so forth.

The difference here is that we are not using ImageClassifierData.from_csv or from_paths, we need a different kind of a model data called ColumnarModelData and we call from_data_frame.

Path specifies where to store model file e.t.c

val_idx list of indexes of rows we will put in the validation set.

df the dataframe the independent variables.

yl dependent variable which is the log of y returned by proc_df i.e yl = np.log(y)

cat_flds specifies which things you want treated as categorical. Everything was turned into a number unless we specify, it will treat them all as continuous. pass the list of names cat_vars

bs batch size

Now we have a standard model data object that contains train_dl, val_dl , train_ds , val_ds , etc.

Creating a learner suitable for our model data:

m = md.get_learner(emb_szs, len(df.columns)-len(cat_vars),
                   0.04, 1, [1000,500], [0.001,0.01], 
                   y_range=y_range)

• 0.04 : how much dropout to use at the very start
• [1000,500] : how many activations to have in each layer
• [0.001,0.01] : how many dropout to use at later layers
• emb_szs embeddings

Embeddings

let’s keep away categorical variables for a moment and look at the continuous variables:

You never want to put ReLU in the last layer because softmax needs negatives to create low probabilities.

Simple view of a fully connected layer:

Takes in an input as a rank one tensor put it through a linear layer (matrix product), through an activation(ReLu), linear layer again then softmax and finally an output. We can add more linear layers or dropout.

Skip the softmax for regression problems.

Categorical variables

We create a new matrix of 7 rows and as many columns as we choose 4, for example and fill it with floating numbers. To add “Sunday” to our rank 1 tensor with continuous variables, we do a look up to this matrix, which will return 4 floating numbers, and we use them as “Sunday”.

Initially, these numbers are random. But we can put them through a neural net and update them using gradient descent in a way that reduces the loss. This matrix is just another bunch of weights in our neural net called an embedding matrix

An embedding matrix is something where we start out with an integer between zero and maximum number of levels of that category. We index into the matrix to find a particular row, and we append it to all of our continuous variables, and everything after that is just the same as before (linear → ReLU → linear etc).

What do those 4 numbers represent? They are just parameters that we are learning that happen to end up giving us a good loss. We will discover later that these particular parameters often are human interpretable and quite interesting but that a side effect. They are a set of 4 random numbers we are learning.

Do you have good heuristics for the dimensionality of the embedding matrix? We use the cardinality of each variable (that is, its number of unique values) to decide how large to make its embeddings.

Above is a list of every categorical variable and its cardinality.

We have 7 days of the week but on looking at DayOfWeek cardinality is 8 the extra cardinality is added just incase there is an unknown in the test set, divide that by 2, thus the 4 random numbers. Even if there were no missing values in the original data, you should still set aside one for unknown just in case. Years are also 3 but we add one for the unknown e.t.c

The rule of thumb for determining the embedding size is the cardinality size divided by 2, but no bigger than 50.

Looking at the store we will have 1116 stores for look up that returns a rank one tensor of length 50. We have to build an embedding matrix for every categorical variable.

We then pass then embedding sizes emb_szs to get_learner which tells the learner that for every categorical variable which embedding to use for that variable.

m = md.get_learner(emb_szs, len(df.columns)-len(cat_vars),
 0.04, 1, [1000,500], [0.001,0.01], y_range=y_range)
m.summary()

Is there a way to initialize embedding matrices besides random? The basic idea is if somebody else at Rossmann had already trained a neural network to predict cheese sales, you may as well start with their embedding matrix of stores to predict liquor sales. This is what happens, for example, at Pinterest and Instacart. Instacart uses this technique for routing their shoppers, and Pinterest uses it for deciding what to display on a webpage. They have embedding matrices of products/stores that get shared in the organization so people do not have to train new ones.

What is the advantage of using embedding matrices over one-hot-encoding?For the day of week example above, instead of the 4 numbers, we could have easily passed 7 numbers e.g. [0, 1, 0, 0, 0, 0, 0] for Sunday. That also is a list of floats and would totally work and that is how, generally speaking, categorical variables have been used in statistics for many years called “dummy variables”. The problem with that is, the concept of Sunday could only ever be associated with a single floating point number. So it gets this kind of linear behavior, it says Sunday is more or less of a single thing. With embeddings, Sunday is a concept in a four dimensional space. What happens is that these embedding vectors tend to get these rich semantic concepts. For example, if it turns out that weekends have a different behavior you tend to see that Saturday and Sunday will have some particular number higher or for example certain days of the week tend to be associated with higher sales for example buying of liquor on Friday.

By having higher dimensionality vector rather than just a single number, it gives the deep learning network a chance to learn these rich representations.

The idea of an embedding is what is called a “distributed representation” — the most fundamental concept of neural networks. This is the idea that a concept in neural network has a high dimensional representation which can be hard to interpret. These numbers in this vector does not even have to have just one meaning. It could mean one thing if this is low and that one is high, and something else if that one is high and that one is low because it is going through this rich nonlinear function. It is this rich representation that allows it to learn such interesting relationships.

Are embeddings suitable for certain types of variables? Embedding is suitable for any categorical variables. The only thing it cannot work well for would be something with too high cardinality. If you had 600,000 rows and a variable had 600,000 levels, that is just not a useful categorical variable. But in general, the third winner in this competition really decided that everything that was not too high cardinality, they put them all as categorical. The good rule of thumb is, if you can make a categorical variable, you may as well because that way it can learn this rich distributed representation; where else if you leave it as continuous, the most it can do is to try and find a single functional form that fits it well.

How matrix algebra works behind the scene:

Doing a lookup is identical to doing a matrix product between a one-hot encoded vector and the embedding matrix.

After multiplying you are left with only one vector. Modern libraries implement this as taking an integer and doing a look up into an array.

Could you touch on using dates and times as categorical and how that affects seasonality? The following extracts particular date fields from a complete datetime for the purpose of constructing categoricals. You should always consider this feature extraction step when working with date-time. Without expanding your date-time into these additional fields, you can’t capture any trend/cyclical behavior as a function of time at any of these granularities. We’ll add to every table with a date field.

add_datepart function takes a data frame and a column name. It optionally removes the column from the data frame and replaces it with lots of column representing all of the useful information about that date such as day of week, day of month, month of year, is it start of quater, end of quater etc. we end up with a list of our features:

For example, DayOfWeek now becomes eight rows by four columns embedding matrix. Conceptually this allows our model to create some interesting time series models. If there is something that has a seven day period cycle that goes up on Mondays and down on Wednesdays but only for daily and only in Berlin, it can totally do that, it has all the information it needs.

This is a fantastic way to deal with time series. You just need to make sure that the cycle indicator in your time series exists as a column. If you did not have a column called day of week, it would be very difficult for the neural network to learn to do divide mod seven and look up in an embedding matrix. It is not impossible but really hard.

For example if you are predicting sales of beverages in San Francisco, you probably want a list of when the ball game is on at AT&T park because that is going to to impact how many people are drinking beer in SoMa. So you need to make sure that the basic indicators or periodicity is in your data, and as long as they are there, neural net is going to learn to use them.

Learner

m = md.get_learner(emb_szs, len(df.columns)-len(cat_vars),
                   0.04, 1, [1000,500], [0.001,0.01], y_range=y_range)
m.summary()

emb_szs embedding size

len(df.columns)-len(cat_vars) number of continuous variables in the data frame.

0.04 embedding matrix has its own dropout and this is the dropout rate.

1 how many outputs we want to create i.e output of the last linear layer which is sales.

[1000, 500] number of activations in the first linear layer, and the second linear layer.

[0.001,0.01] dropout in the first linear layer, and the second linear layer.

We have a leaner, let’s find the learning rate(lr = 1e-3):

Fit

start with sample data:

metrics this is a custom metric which specifies a function exp_rmspe that is called at the end of every epoch and prints out a result.

fit all the data:

m.fit(lr, 1, metrics=[exp_rmspe], cycle_len=1)

[ 0.       0.00676  0.01041  0.09711]

By using all of the training data, we get a RMSPE of 0.09711. This would have gotten us at the top% of the leaderboard.

So this is a technique for dealing with time series and structured data. Interestingly, compared to the group that used this technique Entity Embeddings of Categorical Variables, the second place winner did way more feature engineering. The winners of this competition were actually subject matter experts in logistics sales forecasting so they had their own code to create lots and lots of features.

Folks at Pinterest who build a very similar model for recommendations also said that when they switched from gradient boosting machines to deep learning, they did way less feature engineering and it was much simpler model which requires less maintenance. So this is one of the big benefits of using this approach to deep learning, you can get state of the art results but with a lot less work.

Are we using any time series in any of these? Indirectly, yes. As we just saw, we have a DayOfWeek, MonthOfYear, etc in our columns and most of them are being treated as categories, so we are building a distributed representation of January, Sunday, and so on. We are not using any classic time series techniques, all we are doing is two fully connected layers in a neural net. The embedding matrix is able to deal with things like day of week periodicity in a richer way than than any standard time series techniques.

What is the difference between image models and this model? There is a difference in a way we are calling get_learner. In imaging we just did Learner.trained and pass the data:

learn = ConvLearner.pretrained(arch, data, ps=0., precompute=True)

For these kinds of models, in fact for a lot of the models, the model we build depends on the data. In this case, we need to know what embedding matrices we have. So in this case, the data objects creates the learner:

m = md.get_learner(emb_szs, len(df.columns)-len(cat_vars), 0.04, 1,
                   [1000,500], [0.001,0.01], y_range=y_range)

Steps

Step 1. List categorical variable names, and list continuous variable names, and put them in a Pandas dataframe:

cat_vars = ['Store', 'DayOfWeek', 'Year', 'Month', 'Day', 'StateHoliday', 'CompetitionMonthsOpen',
    'Promo2Weeks', 'StoreType', 'Assortment', 'PromoInterval', 'CompetitionOpenSinceYear', 'Promo2SinceYear',
    'State', 'Week', 'Events', 'Promo_fw', 'Promo_bw', 'StateHoliday_fw', 'StateHoliday_bw',
    'SchoolHoliday_fw', 'SchoolHoliday_bw']

contin_vars = ['CompetitionDistance', 'Max_TemperatureC', 'Mean_TemperatureC', 'Min_TemperatureC',
   'Max_Humidity', 'Mean_Humidity', 'Min_Humidity', 'Max_Wind_SpeedKm_h', 
   'Mean_Wind_SpeedKm_h', 'CloudCover', 'trend', 'trend_DE',
   'AfterStateHoliday', 'BeforeStateHoliday', 'Promo', 'SchoolHoliday']

Step 2. Create a list of which row indexes you want in your validation set:

val_idx = np.flatnonzero(
    (df.index<=datetime.datetime(2014,9,17)) & (df.index>=datetime.datetime(2014,8,1)))

Step 3. Call this exact line of code:

md = ColumnarModelData.from_data_frame(PATH, val_idx, df, 
         yl.astype(np.float32), cat_flds=cat_vars, bs=128, 
         test_df=df_test)

Step 4. Create a list of how big you want each embedding matrix to be:

emb_szs = [(c, min(50, (c+1)//2)) for _,c in cat_sz]
emb_szs
[(1116, 50),
 (8, 4),
 (4, 2),
 (13, 7),
 (32, 16),
 (3, 2),
 (26, 13),
 (27, 14),
 (5, 3),
 (4, 2),
 (4, 2),
 (24, 12),
 (9, 5),
 (13, 7),
 (53, 27),
 (22, 11),
 (7, 4),
 (7, 4),
 (4, 2), ...

Step 5. Call get_learner — you can use these exact parameters to start with, it it overfits you can play around with them.

m = md.get_learner(emb_szs, len(df.columns)-len(cat_vars), 0.04, 1,
                   [1000,500], [0.001,0.01], y_range=y_range)

Step 6. Call m.fit

m.fit(lr, 3, metrics=[exp_rmspe])

How to use data augmentation for this type of data, and how does dropout work? No idea. May be it has to be domain-specific, but he(Jeremy) has never seen any paper or anybody in industry doing data augmentation with structured data and deep learning. He thinks it can be done but has not seen it done. What dropout is doing is exactly is throwing half of the activation of a rank one tensor. Which also applies to the embedding matrix.

What is the downside? Almost no one is using this. Why not? Basically the answer is , no one in academia is working on this because it is not something that people publish on. As a result, there have not been really great examples people could look at and say “oh here is a technique that works well so let’s have our company implement it”. But perhaps equally importantly, until now with this Fast.ai library, there has not been any way to do it conveniently. If you wanted to implement one of these models, you had to write all the custom code yourself. Now it is 6 step process. There are a lot of big commercial and scientific opportunity to use this and solve problems that previously haven’t been solved very well.

Natural Language Processing

This is the most most up-and-coming area of deep learning and it is two or three years behind computer vision. The state of software and some of the concepts is much less mature than it is for computer vision.

One of the things you find in NLP is there are particular problems you can solve and they have particular names. There is a particular kind of problem in NLP called “language modeling” — it means building a model given a few words of a sentence, to predict what the next word is going to be.

For example when typing and you press space and the next word is suggested that is a language model.

For this exercise we downloaded 18 months papers from arXiv.org

Data:

<cat> — category of the paper. CSNI is Computer Science and Networking.

<summ> — abstract of the paper.

Let’s look at the output of a trained language model we pass the category in this case csni (computer science and networking) and some priming text:

The model learned by reading arXiv papers that somebody who is writing about computer networking would talk like this. Remember, it started out not knowing English at all. It started out with an embedding matrix for every word in English that was random. By reading lots of arXiv papers, it learned what kind of words followed others.

The model not only learned how to write English pretty well, but also after you say something like “convolutional neural network” it use parenthesis to specify an acronym “(CNN)”.

We will try to create a pre-trained model which is used to do some other tasks. For example, given IMDB movie reviews, we will figure out whether they are positive or negative. Which is a classification problem.

We would like to use a pre-trained network which at least knows how to read English. So we will train a model that predicts the next word of a sentence i.e. language model, and just like in computer vision, stick some new layers on the end and ask it to predict whether something is positive or negative.

Basically, create a language model and make it the pre-trained model for a classification model.

Why would doing directly what you want to do doesn’t work better? It just turns out it doesn’t perform well empirically. There are several reasons. First of all, we know fine-tuning a pre-trained network is really powerful. So if we can get it to learn some related tasks first, then we can use all that information to try and help it on the second task. The other reason is IMDB movie reviews are up to a 1000 words long. So after reading a 1000 words(integers) knowing nothing about how English is structured or the concept of a word or punctuation is, all you get is a 1 or a 0 (positive or negative). Trying to learn the entire structure of English and then how it expresses positive and negative sentiments from a single number is just too much to expect. By building a language model first we build a neural net that understand the English language of movie review then we hope that the things it learns will be helpful in deciding if a review is positive or negative.

Is this similar to Char-RNN by Karpathy? This is somewhat similar to Char-RNN which predicts the next letter given a number of previous letters. Language model generally work at a word level but they do not have to, and we will focus on word level modeling.

To what extent are these generated words and sentences actual copies of what it found in the training dataset? The words are definitely words it has seen before because it is not at character level so it can only give us the words it has seen before. The Sentences, there are several rigorous ways of doing it but the easiest way would be by looking at examples like above, Where it creates two categories and looking at them it creates similar concepts.

Most importantly, when we train the language model, we will have a validation set so that we are trying to predict the next word of something that has never seen before.

Examples of text classification:

1. Hedge fund, might identify things in articles or Twitter that caused massive market drops in the past.
2. Recognising customer service queries which tend to be associated with people who cancel their subscriptions in the next month.
3. Classifying if documents into whether they are part of legal discovery or not.

IMDB

Imports

torchtext PyTorch’s NLP library.

Data

The large movie view dataset contains a collection of 50,000 reviews from IMDB. The dataset contains an even number of positive and negative reviews. The authors considered only highly polarized reviews. A negative review has a score ≤ 4 out of 10, and a positive review has a score ≥ 7 out of 10. Neutral reviews are not included in the dataset. The dataset is divided into training and test sets. The training set is the same 25,000 labeled reviews.

The sentiment classification task consists of predicting the polarity (positive or negative) of a given text.

However, before we try to classify sentiment, we will simply try to create a language model; that is, a model that can predict the next word in a sentence. Why? Because our model first needs to understand the structure of English, before we can expect it to recognize positive vs negative sentiment.

So our plan of attack is the same as we used for Dogs v Cats: pretrain a model to do one thing (predict the next word), and fine tune it to do something else (classify sentiment).

Unfortunately, there are no good pre-trained language models available to download, so we need to create our own.

We do not have separate test and validation in this case. The training directory has bunch of files in it representing movie reviews:

Checking out an example of a movie review:

let’s check how many words are in the training dataset and test dataset respectively:

Before we can analyze text, we must first tokenize it. This refers to the process of splitting a sentence into an array of words or more generally, into an array of tokens.

A good tokenizer will do a good job of recognizing pieces in your sentence. Each separated piece of punctuation will be separated, and each part of multi-part word will be separated as appropriate below is an example of a tokenization:

We use Pytorch’s torchtext library to preprocess our data, telling it to use the spacy library to handle tokenization.

Create a field

First, we create a torchtext field, which describes how to preprocess a piece of text — in this case, we tell torchtext to make everything lowercase, and tokenize it with spacy.

TEXT = data.Field(lower=True, tokenize="spacy")

lower=True — lowercase the text

tokenize=spacy_tok — tokenize with spacy_tok

We create a ModelData object for language modeling by taking advantage of LanguageModelData, passing it our torchtext field object, and the paths to our training, test, and validation sets. In this case, we don't have a separate test set, so we'll just use VAL_PATH for that too.

bs=64; bptt=70

FILES = dict(train=TRN_PATH, validation=VAL_PATH, test=VAL_PATH)
md = LanguageModelData.from_text_files(PATH, TEXT, **FILES, bs=bs, bptt=bptt, min_freq=10)

PATH where the data is, where to save models e.t.c

TEXT torchtext’s Field definition of how to preprocess text.

**FILES list of all of the files we have: training, validation, and test

bs batch size

bptt Back Prop Through Time. It means how long a sentence we will stick on the GPU at once

min_freq=10 : In a moment, we are going to be replacing words with integers i.e a unique index for every word. If there are any words that occur less than 10 times, just call them unknown.

After building our ModelData object, it automatically fills the TEXT object with a very important attribute: TEXT.vocab. This is a vocabulary, which stores which words (or tokens) have been seen in the text, and how each word will be mapped to a unique integer id. We'll need to use this information again later, so we save it.

(Technical note: python’s standard Pickle library can't handle this correctly, so at the top of this notebook we used the dill library instead and imported it as pickle).

This is the start of the mapping from integer IDs to unique tokens:

vocab let’s us take a word and match it to an integer and an integer to a word. We will work using integers.

Is it common to do any stemming or lemma-tizing? Not really, no. Generally tokenization is what we want. To keep it as general as possible, we want to know what is coming next so whether it is future tense or past tense or plural or singular, we don’t really know which things are going to be interesting and which are not, so it seems that it is generally best to leave it alone as much as possible.

When dealing with natural language, isn’t context important? Why are we tokenizing and looking at individual word? No, we are not looking at individual word — they are still in order. Just because we replaced I with a number 12, they are still in that order.

There is a different way of dealing with natural language called “bag of words” and they do throw away the order and context. In the Machine Learning course, we will be learning about working with bag of words representations but they are no longer useful or in the verge of becoming no longer useful. We are starting to learn how to use deep learning to use context properly.

Batch size and Back Prop Through Time (BPTT)

What happens in a language model is even though we have lots of movie reviews, they all get concatenated together into one big block of text. So we predict the next word in this huge long thing which is all of the IMDB movie reviews concatenated together.

We start with a big block of text with for example 64 million words. We split up the concatenated reviews into batches. In this case, we will split it to 64 sections. We then move each section underneath the previous one, and transpose it. We end up with a matrix which is 1 million by 64. We then grab a little chunk at time and those chunk lengths are approximately equal to BPTT(this changes with every epoch). We grab a little 70 long section and that is the first thing we chuck into our GPU i.e. the batch.

Our LanguageModelData object will create batches with 64 columns (that's our batch size), and varying sequence lengths of around 80 tokens (that's our bptt parameter - backprop through time).

Each batch also contains the exact same data as labels, but one word later in the text — since we’re trying to always predict the next word. The labels are flattened into a 1d array.

We grab our first training batch by wrapping the data loader(md.trn_dl) with iter then calling next

We get back a 77 by 64 tensor which is approximately 70 rows but not exactly.

Torchtext randomly change the bptt number every time so each epoch it gets slightly different bits of text — similar to shuffling images in computer vision. We cannot randomly shuffle the words because they need to be in the right order, so instead, we randomly move their breakpoints a little bit.

77 represents the first 77 words of the first movie review.

Why not split by a sentence? Not really. Remember, we are using columns. So each of our column is of length about 1 million, so although it is true that those columns are not always exactly finishing on a full stop, they are so long we do not care. Each column contains multiple sentences.

Creating a model

Now that we have a model data object that can feed us batches, we can create a model. First, we are going to create an embedding matrix.

4602 number of batches

34945 number of unique tokens in the vocab (unique words have to appear at least 10 times min_freq=10 otherwise they will be replased with Unk)

1 length of the dataset i.e the whole corpus

20621966 number of words in the corpus .

34945 is used to create an embedding matrix:

Each word gets an embedding vector. This is a categorical variable it is just a very high cardinality categorical variable. It is 34945 categorical variable that is why we create an embedding matrix for it.

We have a number of parameters to set:

em_sz = 200  # size of each embedding vector
nh = 500     # number of hidden activations per layer
nl = 3       # number of layers

The embedding size is 200 which is much bigger than our previous embedding vectors. Not surprising because a word has a lot more nuance to it than the concept of Sunday.

Generally, an embedding size for a word will be somewhere between 50 and 600.

Researchers have found that large amounts of momentum don’t work well with these kinds of RNN (Recurrent neural networks) models, so we create a version of the Adam optimizer with less momentum than it’s default of 0.9.

opt_fn = partial(optim.Adam, betas=(0.7, 0.99))

Fastai uses a variant of the state of the art AWD LSTM Language Model developed by Stephen Merity. A key feature of this model is that it provides excellent regularization through Dropout. There is no simple way known (yet!) to find the best values of the dropout parameters below — you just have to experiment…

However, the other parameters (alpha, beta, and clip) shouldn't generally need tuning

If you try to build an NLP model and you are under-fitting, then decrease all these dropouts, if overfitting, then increase all these dropouts in roughly this ratio.

There are other kind of ways we can avoid overfitting. For now, learner.reg_fn = partial(seq2seq_reg, alpha=2, beta=1) works reliably so all of your NLP models probably want this particular line.

learner.clip=0.3 when you look at your gradients and you multiply them by the learning rate and decide how much to update your weights by, this will not let them be more than 0.3. Prevents us from taking too big steps (high learning rate).

There are word embedding out there such as Word2vec or GloVe. How are they different from this? And why not initialize the weights with those initially? People have pre-trained these embedding matrices before to do various other tasks. They are not called pre-trained models; they are just a pre-trained embedding matrix and you can download them. There is no reason we could not download them. Building a whole pre-trained model in this way did not seem to benefit much if at all from using pre-trained word vectors; where else using a whole pre-trained language model made a much bigger difference. Maybe we can combine both to make them a little better still.

What is the architecture of the model? It is a recurrent neural network using something called LSTM (Long Short Term Memory).

Fitting the model

learner.fit(3e-3, 4, wds=1e-6, cycle_len=1, cycle_mult=2)
learner.save_encoder('adam1_enc')
learner.load_encoder('adam1_enc')

learner.fit(3e-3, 1, wds=1e-6, cycle_len=10)

In the sentiment analysis section, we’ll just need half of the language model — the encoder, so we save that part:

learner.save_encoder('adam3_10_enc')
learner.load_encoder('adam3_10_enc')

Language modeling accuracy is generally measured using the metric perplexity, which is simply exp() of the loss function we used.

math.exp(4.165)

64.3926824434624

pickle.dump(TEXT, open(f'{PATH}models/TEXT.pkl','wb'))

Testing

We can play around with our language model a bit to check it seems to be working OK. First, let’s create a short bit of text to ‘prime’ a set of predictions. We’ll use our torchtext field to numericalize it so we can feed it to our language model.

We haven’t yet added methods to make it easy to test a language model, so we’ll need to manually go through the steps:

Let’s see what the top 10 predictions were for the next word after our short text:

Let’s see if our model can generate a bit more text all by itself:

Sentiment Analysis

We have a pre-trained language model and now we want to fine-tune it to do sentiment classification.We’ll need the saved vocab from the language model, since we need to ensure the same words map to the same IDs.

TEXT = pickle.load(open(f'{PATH}models/TEXT.pkl','rb'))

sequential=False tells torchtext that a text field should be tokenized (in this case, we just want to store the 'positive' or 'negative' single label).

We need to not treat the whole thing as one big piece of text but every review is separate because each one has a different sentiment attached to it.

splits is a torchtext method that creates train, test, and validation sets. The IMDB dataset is built into torchtext, so we can take advantage of that. Take a look at lang_model-arxiv.ipynb to see how to define your own fastai/torchtext datasets.

Splits allows us to look at a single label t.label and some of the text t.text

fastai can create a ModelData object directly from torchtext splits that we can train on md2:

get_model gets us our learner, then we can load into it the pre-trained language model m3.load_encoder(f’adam3_10_enc’).

Because we’re fine-tuning a pre-trained model, we’ll use differential learning rates, and also increase the max gradient for clipping, to allow the SGDR to work better.

We make sure all except the last layer is frozen. Then we train a bit, unfreeze it, train it a bit. The nice thing is once you have got a pre-trained language model, it actually trains really fast.

A recent paper from Bradbury et al, Learned in translation: contextualized word vectors, has a handy summary of the latest academic research in solving this IMDB sentiment analysis problem. Many of the latest algorithms shown are tuned for this specific problem.

As you see, we just got a new state of the art result in sentiment analysis, decreasing the error from 5.9% to 5.5%! You should be able to get similarly world-class results on other NLP classification problems using the same basic steps.

There are many opportunities to further improve this:

For example we could start training language models that look at lots of medical journals and then make a downloadable medical language model that then anybody could use to fine-tune on a prostate cancer subset of medical literature.

We could also combine this with pre-trained word vectors.

We could have pre-trained a Wikipedia corpus language model and then fine-tuned it into an IMDB language model, and then fine-tune that into an IMDB sentiment analysis model and we would have gotten something better than this.

Collaborative Filtering

Data:

Movie lens data

http://files.grouplens.org/datasets/movielens/ml-latest-small.zip

path='data/ml-latest-small/'

It contains a userId the movie movieId rating and timestamp

Our goal will be for some user-movie combination we have not seen before, we have to predict if they will like it. This is how recommendation engines are built.

Let’s read the movie names too:

We will create(in the next lesson) a kind of cross tab of users by movies:

Thanks for reading! follow @itsmuriuki.

Back to learning!

Deep Learning; Personal Notes Part 1 Lesson 4: Structured learning, Natural language Processing… was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

This blog post series will be updated as I have a second take on the fast ai lessons. These are my personal notes; a strive to understand things clearly and explain them well. Nothing new, only living up this blog.

Quick Dogs Vs Cats

Here is an end to end process to get a state of the art result for dogs vs. cats:

PATH = "data/dogscats/"

We assume that your data is in the data folder. But you may want to put them somewhere else. In that case you use symbolic link or symlink for short.

Note: We did not set pre_compue=True. It is a shortcut which caches some of the intermediate steps that do not have to be recalculated each time and can be left out. When pre_compute=True, data augmentation does not work.

learn.unfreeze() 
learn.bn_freeze(True) 
%time learn.fit([1e-5, 1e-4,1e-2], 1, cycle_len=1)

bn_freeze — If you are using a bigger deeper model like ResNet50 or ResNext101 i.e anything with number bigger than 34 on a dataset that is very similar to ImageNet i.e. side-on photos of standard object whose size is similar to ImageNet between 200–500 pixels, you should add this line. It causes the batch normalization moving averages not to be updated.

Using Other Libraries — Keras

Just like fast ai sits on top of pytorch, keras sits on top of TensorFlow, MXNet, CNTK, etc.

You need to install Keras or tensorflow as a backend:

pip install tensorflow-gpu keras

Imports:

import numpy as np
from keras.preprocessing.image import ImageDataGenerator
from keras.preprocessing import image
from keras.layers import Dropout, Flatten, Dense
from keras.applications import ResNet50
from keras.models import Model, Sequential
from keras.layers import Dense, GlobalAveragePooling2D
from keras import backend as K
from keras.applications.resnet50 import preprocess_input

Data path:

PATH = "data/dogscats/"
sz=224
batch_size=64

train_data_dir = f'{PATH}train'
validation_data_dir = f'{PATH}valid'

Keras uses the idea of train folder and validation folder with subfolders with the label names.

Keras requires much more code and many more parameters to be set.

train_datagen = ImageDataGenerator(preprocessing_function=preprocess_input,
    shear_range=0.2, zoom_range=0.2, horizontal_flip=True)

test_datagen = ImageDataGenerator(preprocessing_function=preprocess_input)

train_generator = train_datagen.flow_from_directory(train_data_dir,
    target_size=(sz, sz),
    batch_size=batch_size, class_mode='binary')

validation_generator = test_datagen.flow_from_directory(validation_data_dir,
    shuffle=False,
    target_size=(sz, sz),
    batch_size=batch_size, class_mode='binary')

Rather than creating a single data object, in Keras you define DataGenerator which specifies how to generate the data also specify what kind of data augmentation we want it to do (shear_range=0.2, zoom_range=0.2, horizontal_flip=True) and also what kind of normalization (preprocessing_function=preprocess_input)to do. In other words, in Fast.ai, we can just say “whatever ResNet50 requires, just do that for me please” but in Keras, you need to know what is expected. There is no standard set of augmentations.

train_generator — generates images by looking from a directory, setting the size of images and the size of the mini batches and class. When training you randomly reorder the images show that they are shown in different order to make them random by shuffling.

class_mode= ‘binary’ — if you have 2 possible out comes use binary if multiple say ‘categorical’

train_generator — generates images by looking from a directory, setting the size of images and the size of the mini batches.

Then create a validation data generator validation_generator where the generator does not have data augmentation. And also tell it not to shuffle the dataset for validation because otherwise you cannot keep track of how well you are doing.

Creating a model

base_model = ResNet50(weights='imagenet', include_top=False)
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(1024, activation='relu')(x)
predictions = Dense(1, activation='sigmoid')(x)

We use ResNet50 as keras does not have ResNet34. For keras you cannot tell it to create a model suitable for a particular dataset. You have to do it by hand.

First create a base_model then construct layers you want to add to it i.e x in this case we add 3 layers.

Freezing Layers and Compiling a model

model = Model(inputs=base_model.input, outputs=predictions)

for layer in base_model.layers: layer.trainable = False

model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics=['accuracy'])

layer.trainable=False — loop through the layers and freeze using .trainable = False

Compiling a model by Passing the optimizer you want to use, the loss to look for and the metric to use.

Fitting

%%time
model.fit_generator(train_generator, train_generator.n // batch_size, epochs=3, workers=4,
        validation_data=validation_generator, validation_steps=validation_generator.n // batch_size)

Call fit_generator and pass the train_generator and the validation _generator

For keras expects you to tell it how many batches there are per epoch , so the number of bathes = size is the generator divide by batch size, tell it how many epochs, and also how many workers i.e processors.

Fine Tunning

Unfreeze some layers, compile, then fit again.

There is no concept of layer groups or differential learning rates or partial un freezing you have to print out all the layers and decide how many you want to fine tune. We fine tune from 140 onwards.

split_at = 140

for layer in model.layers[:split_at]: layer.trainable = False
for layer in model.layers[split_at:]: layer.trainable = True

model.compile(optimizer='rmsprop', loss='binary_crossentropy',
    metrics=['accuracy'])

%%time
model.fit_generator(train_generator, train_generator.n // batch_size, epochs=1, workers=3,
        validation_data=validation_generator, validation_steps=validation_generator.n // batch_size)

After fine tuning you have to re compile the model then fit.

Submitting results to Kaggle

In a kaggle competition the is a part called evaluation that describes how the competition will be evaluated:

For datasets where the lables are in different folders use ImageClassifierData.from_paths. If you have aCSV file with labels you use ImageClassifierData.from_csv.

To create a submission you need to use:

data.classes : contains all the different classes

data.test_ds.fnames : contains test file names

It is a good idea to use Test time augmentation (TTA). By using is_test=True it will give you predictions on the test set rather than the validation set.

log_preds, y = learn.TTA(is_test=True) 
probs = np.exp(log_preds)

Most PyTorch models will give you back the log of the predictions, so you need to do np.exp(log_preds) to get the probability.

probs.shape #(n_images, n_classes(breeds))
(10357, 120)

Converting the matrix to the kaggle format we use a pandas dataframe:

df = pd.DataFrame(probs)
df.columns = data.classes

Create a pandas dataframe and pass the matrix(probs). Set the names of the columns to be data.classes

df.insert(0, 'id', [o[5:-4] for o in data.test_ds.fnames])

Insert a new column at position zero named id that contains the file names. This is how the file name looks like. test/ab2520c527e61f197be228208af48191.jpg’ . Remove the first 5 and last 4 letters to get the IDs.

Looking at the dataframe:

SUBM = f'{PATH}/subm/'
os.makedirs(SUBM, exist_ok=True)
df.to_csv(f'{SUBM}subm.gz', compression='gzip', index=False)

Call df.to_csv to create a CSV file and compress it using compression=’gzip’. This saves the file to the serve. You can use kaggle CLI to submit by using this command$ kg submissions or download it to your computer and upload it to kaggle to do that use FileLink(f’{SUBM}subm.gz’). Which will give you a link to download the file from the server to your computer.

Individual Prediction

Running a single image through a model to get a prediction.

opening the first file in the validation set:

fn = data.val_ds.fnames[0]
fn
'train/000bec180eb18c7604dcecc8fe0dba07.jpg'
Image.open(PATH + fn).resize((150, 150))

Running the prediction:

trn_tfms, val_tfms = tfms_from_model(arch, sz)
im = val_tfms(open_image(PATH + fn)) # open_image() returns numpy.ndarray
preds = learn.predict_array(im[None])
np.argmax(preds)

The image must be transformed. tfms_from_model returns training transforms and validation transforms. In this case, we will use validation transform.

Everything that gets passed to or returned from a model is generally assumed to be in a mini-batch. Here we only have one image, but we have to turn that into a mini-batch of a single image. In other words, we need to create a tensor that is not just [rows, columns, channels] , but [number of images, rows, columns, channels]. By indexing into an array with im[None]that adds an additional unit axis to the start thus turning it from an image into a mini batch of images.

Theory On Convolutional Neural Network

Here is a Convolution neural network visualisation video by Otavio Good creator of Word Lens.

If you input an image the computer recognises them as numbers(pixels).

Input Layer

This number 7 image data is from the MNIST database and we are assuming you are using a pre-train model for the classification.

Hidden Layer 1

A hidden layer is what transforms the inputs to discern more complex features from the data for the output layer to make a better assessment.

We apply a filter/kernel that detects horizontal edges mostly 3X3. Note, the 1’s on top and 0’s in the middle and -1’s at the bottom of kernel A:

 1,   1,   1
 0,   0,   0
-1,  -1,  -1

If the filter is multiplied with the input we allocate high numbers for the high numbers as the are multiplied by 1 and almost nothing for the low numbers as they are multiplied by 0 or -1. Thus giving us the output of the first convolution called an activation which is basically a number that is calculated by taking some number from the input then applying some kind of linear operation i.e a convolutional kernel to calculate an output(the activation).

Conv1 shows the activations of both after taking the 3x3 section of the input and multiplying it by the convolutional kernel.

we assume the network is trained and at the end of the training it created a convolutional filter with A which has 9 numbers in it.

Convolution is something where we have a little matrix nearly always 3x3 in deep learning and multiply every element of that matrix by every element of 3x3 section of an image and add them all together to get the result of that convolution at one point.

let’s apply a second convolutional filter B which also has 9 numbers. Filter B finds the vertical edge and outputs it as a hidden layer.

1,   0,  -1
1,   0,  -1
1,   0,  -1

Pytorch does not store them as 2 separate 9 digit arrays, it stores it as tensors. which are multidimensional arrays. The additional axis allows us to stack additional filters together. Filter and kernel mean the same it refers to the 3x3 matrix.

The hidden layer (conv1) is of size two as it has 2 filters/kernels.

Next we will apply another filter i.e filter C in hidden layer 2 which is two 2x3x3 kernel then apply rectified linear unit(ReLu) to eliminate the negatives. Thus creating a second hidden layer conv2.

Rectified Linear Unit (ReLU) -is a non-linearity that throw away the negatives.

Architecture

An architecture means how big is your kernel at layer one, how many filters are in your kernel at layer one e.t.c. We have a 3x3 in layer one and a 3x3 in layer 2, this architecture starts out with two 3x3 convolutional kernel. The second layer has two 2x3x3 kernels.

Max pooling

It means replacing the highest value in an output matrix with the maximum. e.g. 2x2 max pooling. Note: it is non overlapping thus reducing the resolution.

Below is the max pooled layer:

The result of the max pool get fit onto dense weights a fully connected layer.

Fully connected layer picks every single max pooled activations and gives them a weight and picks the sum product of the max pooled activations and the weights of the 2 levels of the 3 dimensional tensor. to out put a dense activation

This is different from a convolution, in a convolution we go through activation a few at a time by using 3x3 kernels . But in a fully connected layer we create a big weight matrix equal to the entire input.

Architectures that make heavy use of fully connected layers can have a lot of weights therefore can have trouble with overfitting and being slow. For Example VGG has upto 19 layers, contains a fully connected layers with 4096 weights connected to 4096 hidden layer activations thus (4096 x 4096 x number of kernels) which is almost 300Million weights.

what if we had 3 channels of the input what would be the shape of the filter? if we had 3 channels of the input it would look exactly like conv1 which has 2 channels therefore 2 filters thus 2 filters per channels.

Rather than starting out with carefully trained filters when we are training from scratch we start out with random numbers and use stochastic gradient descent to improve those numbers and make them less random.

In practice we have to calculate more than one number for the ten digits. we would have ten dense activations.

Softmax

Predicting if an image is a cat, dog, plane, fish or building.

Out of the fully connected layer we would have 5 numbers. Note that in the last layer there is no Relu so we can have negatives. You want to turn the 5 numbers into probabilities from 0 to 1 and match it if it is a cat, dog, plane, fish or building. The probabilities of the 5 classes should be between 0 and 1 and all of them should sum up to 1. To do that we use an activation function this is a function applied to activations. it takes in one number and spits out another number.

We need to stack linear and non-linear function together to do deep learning. The non- linearity we are using after each hidden layer is a ReLu. An activation function is a non-linear function.

Softmax activation occurs in the final layer. softmax spits out numbers between 0 and 1 that also sum up to 1.

To make softmax work we need to get rid of all the negatives. we will use logarithms and exponents. They appear a lot of machine and deep learning.

We do e^-0.36 to get the exp then add all of them up to get 51.31. To get the softmax we divide by 51.31 and get 0.01. The sum of the softmax should add up to 1.

A characteristic of the exp is that if a number(output) is slightly bigger than the other it tends to make the exp even much more bigger. The softmax tends to pick one thing strongly i.e it allocates to one class a high probability.

What kind of activation function do we use if we want to classify the picture as cat and dog? Softmax does not like to predicting multiple things. It wants to pick one thing. One reason we might want to do that is to do multi-label classification.

Planet Competition: Understanding the Amazon from Space

For the cat vs dog competition single-label classification the image is either a cat or a a dog not neither or both. For the Satellite competition the images are classified by weather (haze and clear), agriculture, primary rain forest and water(river). In this case we need to predict multiple things and softmax would not be great as it wants to pick one thing.

anthropomorphizing your activation functions (give personalities) softmax tends to pick one particular thing.

Fast.ai library will automatically switch into multi-label mode if there is more than one label. So you do not have to do anything. But here is what happens behind the scene:

from planet import f2 

metrics=[f2] 
f_model = resnet34

label_csv = f'{PATH}train_v2.csv'
n = len(list(open(label_csv)))-1
val_idxs = get_cv_idxs(n)

def get_data(sz):
    tfms = tfms_from_model(f_model, sz,aug_tfms=transforms_top_down,
max_zoom=1.05)
    return ImageClassifierData.from_csv(PATH, 'train-jpg',label_csv, tfms=tfms,suffix='.jpg', val_idxs=val_idxs, test_name='test-jpg')

data = get_data(256) #gets images of 256x256

we use from_csv since multi-label classification cannot be done with Keras style approach where subfolder is the name of the label.

transform_top_down : it does more than just a vertical flip. There are 8 possible symmetries for a square. It can be rotated through 0, 90, 180, 270 degrees and for each of those, it can be flipped (dihedral group of eight)

x,y = next(iter(data.val_dl))

Using ds (dataset) we can get an individual image.

dl is a data loader which will give you a mini-batch, specifically transformed mini-batch. With a data loader, you cannot ask for a particular mini-batch, you can only get back the next mini-batch. In Python, it is called “generator” or “iterator”.

PyTorch really leverages modern Python methodologies. if you know python well, PyTorch comes very naturally. If you don’t know python well, PyTorch is a good reason to learn python well.

x is a mini-batch of images, y is a mini-batch of labels. Printing y:

This gives you a batch size of 64 by default and 17 which are the possible classes.

lets look at the 0 image labels:

list(zip(data.classes, y[0]))

[('agriculture', 1.0),
 ('artisinal_mine', 0.0),
 ('bare_ground', 0.0),
 ('blooming', 0.0),
 ('blow_down', 0.0),
 ('clear', 1.0),
 ('cloudy', 0.0),
 ('conventional_mine', 0.0),
 ('cultivation', 0.0),
 ('habitation', 0.0),
 ('haze', 0.0),
 ('partly_cloudy', 0.0),
 ('primary', 1.0),
 ('road', 0.0),
 ('selective_logging', 0.0),
 ('slash_burn', 1.0),
 ('water', 1.0)]

It’s agriculture, clear, primary slash_burn and water.

PyTorch and fast.ai are turn labels into one-hot-encoded labels. If the actual label is dog, it will look like this:

we take the softmax and compare with the actuals to actually predict. The difference between actuals and softmax summed up together give us the error i.e. loss function.

One-hot-encoding is terribly inefficient for sorting, so we will store an index value (single integer) rather than 0’s and 1’s for the target value (y).If you look at the y values for the dog breeds competition, you won’t actually see a big lists of 1’s and 0's, but you will see a single integer.

PyTorch internally converts the index to one-hot-encoded vector (even though you will literally never see it). PyTorch has different loss functions for ones that are one hot encoded and others that are not — but these details are hidden by the fast.ai library so you do not have to worry about it. But the cool thing to realize is that we are doing exactly the same thing for both single label classification and multi label classification.

Does it make sense to change the base of log for softmax? No, changing the base is just a linear scaling which neural net can learn easily:

Note: images are just matrices of numbers. The image was washed out, so making it more visible (“brightening it up a bit”) we multiply by *1.4.

It is good to experiment images like this because these images are not at all like ImageNet. The vast majority of things you do involving convolutional neural net will not actually be anything like ImageNet they will be medical imaging, classifying different kinds of steel tube, satellite images, etc

sz=64

data = get_data(sz)

data = data.resize(int(sz*1.3), 'tmp')

We will not use sz=64 for cats and dogs competition because we started with pre-trained ImageNet network which starts off nearly perfect. If we re-trained the whole set with 64 by 64 images, we would destroy the weights that are already very good. Remember, most of ImageNet models are trained with 224 by 224 or 299 by 299 images.

There are no images in image net that look like the satellite image above. The only useful bit of the imageNet are the first layer that finds edges and gradients, the second layer that finds texture and repeating patterns.

Starting out by first training with small images works well when using satellite images.

learn = ConvLearner.pretrained(f_model, data, metrics=metrics)

lrf=learn.lr_find()
learn.sched.plot()

fitting

[lr/9, lr/3, lr]is used because the images are unlike ImageNet image and earlier layers are probably not as close to what they need to be.

plotting the loss:

lets double the size of our image size to 128 first then fit, then double again to 256 it gets you to 93.5% accuracy:

Test time augmentation get us to 93.6%:

data = data.resize(int(sz*1.3), 'tmp')

What does this do?

when we specify what transforms to apply, we pass a size:

tfms = tfms_from_model(f_model, sz,
        aug_tfms=transforms_top_down, max_zoom=1.05)

One thing the data loader does is to resize the images on-demand. This has nothing to do with data.resize . If the initial input image is 1000 by 1000, reading that JPEG and resizing it to 64 by 64 takes more time than training the convolutional net.

data.resize tells it that we will not use images bigger than sz*1.3 so go through once and create new JPEGs of this size, which are rectangular, so new JPEGs whose smallest edge is of sizesz*1.3 center-cropped. This works like the batch scripts that loop through to resize images. It will save you a lot of time. data.resize is a speed up convenience function.

metrics=[f2]

Instead of accuacy, we used F-beta, it is a way of weighing false negatives and false positives. The reason we are using it is because this particular Kaggle competition wants to use it. Take a look at planet.py to see how you can create your own metrics function using Skitlearn. This is what gets printed out at the end . After every epoch[ 0. 0.08932 0.08218 0.9324 ]

Activation function for multi-label classification

Fast.ai checks the labels to see whether an image has more than one label to it and picks the activation function automatically.

Activation function for multi-label classification is called sigmoid.

To find the sigmoid pick the exp divide it by 1+exp i.e:

0.01/1+0.01 = 0.01
0.08/1+0.08 = 0.07 

This enable multiple things to be predicted at once.If something is less than 0 it’s sigmoid is less than 0.5 if more than 0 it is greater than 0.5.

Why don’t we start training with differential learning rate rather than training the last layers alone? You can skip training just the last layer and go straight to differential learning rates, but you probably do not want to. Convolutional layers all contain pre-trained weights, so they are not random — for things that are close to ImageNet, they are really good; for things that are not close to ImageNet, they are better than nothing. All of our fully connected layers, however, are totally random. Therefore, you would always want to make the fully connected weights better than random by training them a bit first. Otherwise if you go straight to unfreeze, then you are actually going to be fiddling around with those early layer weights when the later ones are still random — which is probably not what you want.

When you unfreeze what are you trying to change? The kernel(filter) and weights. what training means is setting the filters and the dense weights. the dense weights are for the fully connected layer and the filters/kernel weights for the convolution. Activations on the other hand are calculated from the weights and previous layers activations or inputs.

How do you get the 64 by 64 image size? This depends on the transforms by default our transform takes the smallest edges, zooms it out and pics a centre crop for it, but when using data augmentation it picks it randomly.

Accuracy is not what the model tries to optimise. It optimises the loss function e.g cross entropy the metric is what is printed out for us to see what is going on.

When you use the differential learning rates, do those three learning rates spread evenly across the layers? fast.ai library has a concept of “layer groups”. In something like ResNet50, there are hundreds of layers and you probably do not want to write hundreds of learning rates, so the library decids for you how to split them and the last one always refers to just the fully connected layers that we have randomly initialized and added the rest learning rates are split halfway through the layers.

Visualizing the layers

learn.summary()

[('Conv2d-1',
  OrderedDict([('input_shape', [-1, 3, 64, 64]),
               ('output_shape', [-1, 64, 32, 32]),
               ('trainable', False),
               ('nb_params', 9408)])),
 ('BatchNorm2d-2',
  OrderedDict([('input_shape', [-1, 64, 32, 32]),
               ('output_shape', [-1, 64, 32, 32]),
               ('trainable', False),
               ('nb_params', 128)])),
 ('ReLU-3',
  OrderedDict([('input_shape', [-1, 64, 32, 32]),
               ('output_shape', [-1, 64, 32, 32]),
               ('nb_params', 0)])),
 ('MaxPool2d-4',
  OrderedDict([('input_shape', [-1, 64, 32, 32]),
               ('output_shape', [-1, 64, 16, 16]),
               ('nb_params', 0)])),
 ('Conv2d-5',
  OrderedDict([('input_shape', [-1, 64, 16, 16]),
               ('output_shape', [-1, 64, 16, 16]),
               ('trainable', False),
               ('nb_params', 36864)]))
 ...

conv2d-1 is the name of the layer.

‘input_shape’, [-1, 3, 64, 64] — PyTorch lists channel 3 before the image size 64, 64. Some of GPUs computations run faster when it is in that order. This is a 4 dimensional mini batch.

-1 means however big the batch size is, which can be changed. Keras uses None

‘output_shape’ , [-1, 64, 32, 32] 64 is the number of kernels, 32 by 32 is the stride. works like maxpooling it changes the size.

Learning rate finder for a very small dataset returned strange numbers and the plot was empty. The learning rate finder will go through a mini-batch at a time. If you have a tiny dataset, there is just not enough mini-batches. So the trick is to make your batch size very small.

Structured and Time Series Data

There are two types of dataset.

Unstructured — Audio, images, natural language text where all of the things inside an object are all the same kind of things — pixels, amplitude of waveform, or words.

Structured — Profit and loss statement, information about a Facebook user where each column is structurally different. “Structured” refers to columnar data as you might find in a database or a spreadsheet where different columns represent different kinds of things, and each row represents an observation.

Structured data is often ignored in academics because it is pretty hard to get published in fancy conference proceedings if you have a better logistics model. But it makes the world go round, makes everybody money and efficiency. We will not ignore it because we are doing practical deep learning, and Kaggle does not either because people put prize money up on Kaggle to solve real-world problems.

The motivation behind exploring this architecture is it’s relevance to real-world application. Most data used for decision making day-to-day in industry is structured or time series data. We will explore the end-to-end process of using neural networks with practical structured data problems.

Example :

predict sales for a large grocery chain. here

Forecast sales using store, promotions and competitor data. here

Rossmann Store Sale

Imports:

from fastai.structured import *
from fastai.column_data import *
np.set_printoptions(threshold=50, edgeitems=20)

PATH='data/rossmann/'

structured — not PyTorch specific and also used in machine learning course doing random forests with no PyTorch at all. It can used on its own without any of the other parts of Fast.ai library.

fastai.column_data — allows us to do Fast.ai and PyTorch stuff with columnar structured data.

For structured data we need to use Pandas a lot. Not familiar with Pandas, here (Python for Data Analysis)is a good book by the pandas author.

Create datasets

In addition to the provided data, we will be using external datasets put together by participants in the Kaggle competition for example google trends and weather . You can download all of them here.

There is a lot of data pre-processing, this notebook contains the entire pipeline from the third place winner (Entity Embeddings of Categorical Variables). Data processing is not covered in this course, but is covered in machine learning course in some detail because feature engineering is very important.

Looking at CSV files:

StoreType — you often get datasets where some columns contain a “code”. It really does not matter what the code means. Stay away from learning too much about it and see what the data says first. Then later look at the data dictionary often provided with the data.

Joining tables

This is a relational dataset, and you have join quite a few tables together — which is easy to do with Pandas using merge:

fast.ai also provides the following:

add_datepart(train, "Date", drop=False)

Takes a date and pulls out a bunch of columns such as “day of week”, “start of a quarter”, “month of year” and so on and adds them all to the dataset.

Duration section will calculate things like how long until the next holiday, how long it has been since the last holiday, etc.

Saving the whole structured file that contains the data:

joined.to_feather(f'{PATH}joined')

to_feather : Saves a Pandas’ data frame into a “feather” format which takes it as it sits in RAM and dumps it to the disk. So it is really really fast. Ecuadorian grocery competition has 350 million records, so you will care about how long it takes to save using feather it takes 6 sec.

The data contains how many of a particular item was sold on a particular date in a store. The goal is to predict how many of those items will be sold on a particular store at a future date.

Next Lesson

We split columns into two types: categorical and continuous.

Categorical variable: is that store_id 1 and store_id 2 are not numerically related to each other. they are categories. Days of the week; Monday (day 0) and Tuesday (day 1) e.t.c. Categorical data will be one hot encoded, and

Continuous variable: Things like distance in kilometers to the nearest competitor is a number we treat numerically. Continuous data will get fed into fully connected layer as it is.

steps:

1. Create a validation set.
2. ColumnarModelData.from_data_frame is how we load column data. the basic API concept are the same as of image recognition.
3. .get_learner
4. .lr_find() to find our best learning rate and plot it.
5. .fit with a metric
6. .fit with a cycle_len

Thanks for reading! follow @itsmuriuki.

Back to learning!

Deep Learning; Personal Notes Part 1 Lesson 3: CNN theory; Convolutional filters, Max pooling… was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

This blog post series will be updated as I have a second take on the fast ai lessons. These are my personal notesm a strive to understand things clearly and explain them well. Nothing new, only living up this blog.

Lesson 1 review

We used three lines of code to build an image classifier:

The organisation of data under PATH involves a train folder and valid folder. Under each of these folders are classification labels i.e. cats and dogs with corresponding images in them.

The training output: [epoch number , training loss, validation loss,accuracy]

0      0.157528   0.228553   0.927593

Choosing a good learning rate

The learning rate decide how quickly we zoom/hone in on a solution. This involves finding the minimum of a function that might have many parameters.

We start at a random point then find the gradient to determine which way is up or down. The distance we travel to the minima is proportional to the gradient; If it is steeper we are further away. We pick a gradient at a point and multiply it by a number. (learning rate/step).

If the learning rate is too small, it will take very long time to get to the bottom.If the learning rate is too big, it could get oscillate away from the bottom. If training a neural net and you find that the loss or accuracy is speeding to infinity the learning rate is too high.

We use a learning rate finder (learn.lr_find) to find an appropriate learning rate. With each mini-batch (i.e how many images we look at each time as we use parallel processing power of the GPU effectively, generally 64 or 128 images at a time). We gradually increase the learning rate multiplicatively, eventually the learning rate will be too big that the loss will start getting worse.

We plot the learning rate against loss to determine the lowest point. We then backtrack with one magnitude and pick that as our learning rate as that is the place the loss is decreasing 0.01.

Math notations in python

Learning rate is the key number to set. Fast ai picks the rest of the hyper parameters for you. There are some more things we can tweak to get slightly better results.

This learning rate finder technique sits on top of Adam optimiser. Momentum and Adam are other ways of improving gradient descent.

The most important thing you can do to make the model better is to give it more data. Since these models have millions of parameters, if you train them for a while, they start to “overfit”.

Overfitting — Is whereby the model starts to see the specific details of the images in the training set rather than learning something general that can be transferred to the validation set.

We can either collect more data or use Data augmentation.

Data Augmentation

This refers to randomly changing the images in ways that shouldn’t impact their interpretation. Such as horizontal flipping, zooming, and rotating.

We can do this by passing aug_tfms (augmentation transforms) to tfms_from_model with a list of functions to apply that randomly change to the image however we wish. For photos that are largely taken from the side (e.g. most photos of dogs and cats, as opposed to photos taken from the top down, such as satellite imagery) we can use the pre-defined list of functions such astransforms_side_on. We can also specify random zooming of images up to specified scale by adding the max_zoom parameter.

You build a data class 6 times and each time you plot the same cat. Let’s look at some cat pictures of data augmentation.

Scenarios:

You want to use different types of data augmentation for different types of images (flip horizontally, vertically, zoom in, zoom out, vary contrast and brightness, and many more). for example, you want to recognise letters and digits you don’t want to flip horizontally as they will have different meaning. You don’t want to flip vertically for cats and dogs as the images are mostly upright. For icebergs in satellite images you may want to flip them upside down as it doesn’t matter which side the satellite was when taking the image.

transfrom side_on — is used for images taken on from the side ,lightly varies the photos zoom, rotates them slightly and varies contrast and brightness,

It is not exactly creating new data, but allows the convolutional neural net to learn how to recognize cats or dogs from different angles.

tsf — contains the data augmentation.

The data object includes augmentation. Initially the augmentation doesn’t do anything because of precompute=True

In the above picture each different layer has this activations that look for anything like the middle of flowers or eye balls of birds (circled in red) etc. The latter layers of this convolutional neural networks have activation (numbers) that for instance in the above picture specify the location and level of confidence (probability) of the eyeball of a bird.

We have pre-trained networks that has learnt to recognise features (certain kind of things e.g gradient, edges circles e.t.c). We take the second last layer that has all the necessary information to recognise these certain kind of things for example the level of “eyeballness”, “fluffy earness” etc. We save for every image this activations and call them pre-computed activations. We can then create a new classifier that takes advantage of this pre-computed activations. We can quickly train a simple linear model based on the pre-computed activations. That is what precompute=True means.

This is why when you train your model for the first time, it takes longer — it is pre-computing these activations.

Although we are trying to show a different version of the cat each time, we had already pre-computed the activations for a particular version of the cat i.e. we are not re-calculating the activations with the altered version. When precompute=True data augmentation does not work. We have to set it to learn.precompute=False for data augmentation to work.

Bad news is that accuracy is not improving, the good news is that the training loss (trn_loss), a way of measuring if the error of this model is getting better, is decreasing. The validation error (val_loss) is not decreasing, but we are not overfitting. Overfitting would mean that the training loss is much lower than the validation loss. In other words, when your model is doing much better job on the training set than it is on the validation set, that means your model is not generalizing.

Cycle_len parameter. What is it?

cycle_len=1 This enables Stochastic Gradient Descent with Restarts (SGDR). The basic idea is as you get closer and closer to the spot with the minimal loss, you may want to start decreasing the learning rate (taking smaller steps) in order to get to exactly the right spot. The idea of decreasing your learning rate as you train is called learning rate annealing. This is helpful because as we get closer to the optimal weights, we want to take smaller steps.

Stepwise annealing — you train a model with a certain learning rate for a while, and when it stops improving, manually drop down the learning rate. pick another learning rate and repeat the process very manually.

Cosine annealing — this turns out to be a better approach simply pick some kind of functional form — turns out the really good functional form is one half of the cosign curve which starts with a high learning rate at the beginning, then drop quickly when you get closer.

During training it is possible for gradient descent to get stuck at local minima rather than the global minimum.

At local minima the loss is worse and with a slightly different data set it won’t generalize. At global minimum the model will generalize better.

Note that annealing is not necessarily the same as restarts

We are not starting from scratch each time, but we are ‘jumping’ a bit to ensure we are in the best minima.

However, we may find ourselves in a part of the weight space that isn’t very resilient i.e small changes to the weights may result in big changes to the loss. We want to encourage our model to find parts of the weight space that are both accurate and stable. Therefore, from time to time we increase the learning rate (this is the ‘restarts’ in ‘SGDR’), which will force the model to jump to a different part of the weight space if the current area is “spikey”. Here’s a picture of how that might look if we reset the learning rates 3 times (in this paper they call it a “cyclic LR schedule”):

By increasing the learning rate suddenly, gradient descent may “hop” out of the local minima and find its way toward the global minimum. Doing this is called stochastic gradient descent with restarts (SGDR), an idea shown to be highly effective in this paper.

The number of epochs between resetting the learning rate is set by cycle_len, and the number of times this happens is referred to as the number of cycles, and is what we're actually passing as the 2nd parameter to fit(). So here's what our actual learning rates looks like:

The learning rate is reset at the start of each epoch to the original value you entered as a parameter, then decreases again over the epoch as described above in cosine annealing.

Each time the learning rate drops to it’s minimum point (every 100 iterations in the figure above), we call it a cycle.

Can we get the same effect by using random starting point? Before SGDR was created, people used to create “ensembles” where they would relearn a whole new model ten times in the hope that one of them would end up being better. In SGDR, once we get close enough to the optimal and stable area, resetting will not actually “reset” but the weights keeps getting better. So SGDR will give you better results than just randomly try a few different starting points.

We pick the highest learning rate that is 1e-2 (0.01) for the SGD to use. We change the learning rate every single mini batch. The number of times we reset it is defined by the cycle_len=1 parameter. 1 means reset it after every epoch.

Our main goal is to generalize and not end up in the narrow optima. In this method, are we keeping track of the minima and averaging them and ensembling them? We are not currently doing that but if you wanted it to generalize even better, you can save the weights right before the resets and take the average. But for now, we are just going to pick the last one. (at the 1000 iteration)

There is a parameter called cycle_save_namewhich you can add as well as cycle_len, which will save a set of weights at the end of every learning rate cycle and then you can ensemble them.

Our validation loss isn’t improving much. So there’s probably no point further training the last layer on its own.

Saving and loading the model

From time to time save your weights call learn.save and pass the filename 224_lastlayer

Pre-computed activations and resized images are saved in the data folder in tmp files. Deleting the tmp folder is fast ai equivalent of turning on and off

Models are saved in the models folder when learn.save is called

What if you wanted to retrain a model from scratch? There is generally no reason to delete the pre-computed activations, because the precomputed activations are without any training.

Fine-tuning and differential learning rate annealing

So far anything we have done has not change the pre-trained filters. We have used a pre-trained model that knows how to find edges and gradients(layer1), corners and curves(layer2), then repeating partners, texts (layer3) and eventually eyeballs(layer4 and 5). We have not retrained any of those activation more specifically weights in the convolutional kernel. All we have done is we added some new layers on top and learned how to mix and match pre-trained features.

Images like satellite images, CT scans, etc have totally different kinds of features all together compared to ImageNet images. So you need to re-train many layers. For dogs and cats, images are similar to what the model was pre-trained with, but we still may still find it helpful to slightly tune some of the later layers.

Now that we have a good final layer trained, we can try fine-tuning the other layers. To tell the learner that we want to unfreeze the remaining layers, just call unfreeze(). This tells the learner we want to start changing the convolutional filters.

learn.unfreeze()

A frozen layer is a layer that is not trained i.e it is not updated.

unfreeze() unfreezes all the layers.

Layer one which detects edge and gradient and layer two which detects curves and corners don’t need much learning, they don’t need to change much. while the much later layers need to change. This is universally true when training for other image recognition.

What we do is create an array of learning rate.

lr=np.array([1e-4,1e-3,1e-2])

The earlier layers (as we’ve seen) have more general-purpose features. Therefore we would expect them to need less fine-tuning for new datasets. For this reason we will use different learning rates for different layers: the first few layers will be at 1e-4 for basic geometric features and layers closest to the pixels, the middle layers at 1e-3 for the middle sophisticated convolutional layers, and 1e-2 as before for the layers we add on top (fully connected layers). We refer to this as differential learning rates, although there’s no standard name for this technique in the literature that we’re aware of.

Why 3? Actually they are 3 ResNet blocks but for now, think of it as a group of layers.

How is differential learning rate different from grid search? There is no similarity to grid search. Grid search is where you are trying to find the best hyperparameters. For differential learning rate it tries a lot of learning rate, it tries to find which is best. For the entire training it uses a different learning rate for each layer.

What if I have a bigger images than the model is trained with? With this library and modern architectures we are using, we can use any size we like.

Can we unfreeze just specific layers? We are not doing it yet, but if you wanted, you can do learn.unfreeze_to(n) which will unfreeze layers from layer n onwards . It almost never helps because, using differential learning rates the optimizer can learn just as much as it needs to. The one place it is helpful is if you are using a really big memory intensive model and if you running out of GPU, the less layers you unfreeze, the less memory and time it takes.

Note; you can’t unfreeze one specific layer.

Earlier we said 3 is the number of epochs, but it is actually cycles. In this case learn is doing 3 cycles of 1 epoch.(cycle_len=1)

If cycle_len=2 , It will do 3 cycles where each cycle is 2 epochs (i.e. 6 epochs).

Then why did it do 7 epochs? It is because of cycle_mult this doubles the length of each cycle.(1 epoch + 2 epochs + 4 epochs = 7 epochs).

Using differential learning rate we have a model that is 99.05% accurate.

If the cycle length is too short, it starts going down to find a good spot, then pops out, and goes down trying to find a good spot and pops out and so on such that it never actually get to find a good spot. Earlier on, you want it to do that because it is trying to find a spot that is smoother, but later on, you want it to do more exploring. That is why cycle_mult=2 seems to be a good approach.

We are introducing more and more hyper parameters having told you that there are not many. You can get away with just choosing a good learning rate, but then adding these extra tweaks helps get that extra level-up without any effort. In general, good starting points are:

• n_cycle=3, cycle_len=1, cycle_mult=2
• n_cycle=3, cycle_len=2 (no cycle_mult)

Why do smoother surfaces correlate to more generalized networks?

X-axis is showing how good this is at recognizing dogs vs. cats as you change this particular parameter. To be generalizable means that we want it to work when we give it a slightly different dataset. Slightly different dataset may have a slightly different relationship between this parameter and how cat-like vs. dog-like it is. It may, instead look like the red line. In other words, if we end up at X or Z, then it will not going to do a good job on this slightly different dataset. Or else, if we end up at Y, it will still do a good job on the red dataset.

Let’s take a look at pictures we predicted incorrectly

When we do the validation set, all of our inputs to our model must be square. The GPU does not go very quickly if you have different dimensions for different images. It needs to be consistent so that every part of the GPU can do the same thing.

To make it square, we just pick out the square in the middle, as you can see below, it is understandable why this picture was classified incorrectly.

We will use Test Time Augmentation(TTA) or inference time or test time it makes predictions not just on the images in your validation set, but also makes predictions on a number of randomly augmented versions of them too (by default, it uses the original image along with 4 randomly augmented versions given that they move around). It then takes the average prediction from these images, and uses that as our final prediction. To use TTA on the validation set, we can use the learner’s TTA() method.

The accuracy improved to 99.25%. The Neural net gets multiple argumentations of the same picture making the accuracy go up.

NOTE; TTA is for validation/ test set. when training we are not doing TTA.

Why not add a border or padding to make it square? It does not help much with neural net as the image of the cat does not change. Zooming would work. Reflection padding where by you add borders on the outside to reflect the image making the image bigger, works well with satellite imagery.Generally speaking, using TTA plus data augmentation, the best thing to do is try to use as large images as possible. If you crop you tend to lose for example the dogs face.

Data augmentation for non-image dataset? No one seems to know. It seems like it would be helpful, but there are very few number of examples. In natural language processing, people tried replacing synonyms for instance, but on the whole the area is under researched and under developed.

Can we use a sliding window to generate other images for example generate 3 image parts from one picture of a dog? For training that would not be better because we would not get much better variations, because you have like three standard ways you are giving it to look at the data. You want to give it as many ways to look at the data. Having fixed crop locations plus random contrast, brightness, rotation changes might be better for TTA.

Analyzing results

Confusion Matrix

A quick way to evaluate classification algorithm is using a confusion matrix. It helps with identifying which group of classification you are having trouble with.

preds = np.argmax(probs, axis=1)
probs = probs[:,1]

from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y, preds)

plot_confusion_matrix(cm, data.classes)

We have 987 cats that we predicted right and 13 we predicted wrongly. 993 dogs that we predicted were right and 7 that we got wrong.

Steps to train a world-class image classifier

1. Enable data augmentation(side_on or top_down depending on what you doing), and precompute=True
2. Use lr_find() to find highest learning rate where loss is still clearly improving.
3. Train last layer from precomputed activations for 1–2 epochs.
4. Turn off precompute ( precompute=False)which allows us to use data augmentation for 2–3 epochs with cycle_len=1
5. Unfreeze all layers.
6. Set earlier layers to 3x-10x lower learning rate than next higher layer. Rule of thumb: for pre-trained 10x for ImageNet like images, 3x for satellite or medical imaging.
7. Use lr_find() again (Note: if you call lr_find having set differential learning rates, it prints out the learning rate of the last layers.)
8. Train full network with cycle_mult=2 until over-fitting.

Let’s do it again

This challenge is to determine the breed of a dog in an image.

Use the kaggle CLI to download data. It is an unofficial kaggle command line tool. Useful for downloading the data when using cloud VM instances such as AWS or paperspace. Make sure you accept the competition rules before using the CLI by clicking the download button first. If you have you account connected with another account for login you have to forget your password and choose the third option to set up a new password and link your two accounts.

$ kg download -u <username> -p <password> -c dog-breed-identification -f <name of file>

Where dog-breed-identification is name of the competition, you can find the name of competition at end of URL of competition after /c/ part, https://www.kaggle.com/c/dog-breed-identification .

Here is the actual command;

$ kg download -u gerald -p mypassword -c dog-breed-identification

Once the file download is complete, we can extract the files using following commands.

#To extract .7z files
7z x -so <file_name>.7z | tar xf -

#To extract.zip files
unzip <file_name>.zip

structure of the dogbreeds folder

This is different to our previous dataset. Instead of train folder which has a separate folder for each breed of dog, it has a CSV file with the correct labels.

The imports

from fastai.imports import *
from fastai.torch_imports import *
from fastai.transforms import *
from fastai.conv_learner import *
from fastai.model import *
from fastai.dataset import *
from fastai.sgdr import *
from fastai.plots import *

PATH = "data/dogbreeds/"
sz = 224
arch = resnext101_64
bs = 58

We will read CSV file with Pandas. Which is used to do structured data analysis.

label_csv = f'{PATH}labels.csv'
n = len(list(open(label_csv))) - 1 # header is not counted (-1)
val_idxs = get_cv_idxs(n) # random 20% data for validation set

n 
10222

val_idxs
array([2882, 4514, 7717, ..., 8922, 6774,   37])

len(val_idxs) #20% of 10222
2044

n = len(list(open(label_csv)))-1 : Open the CSV file, create a list of rows, then take the length. -1 because the first row is a header. Hence n is the number of images/rows we have.

val_idxs = get_cv_idxs(n) : “get cross validation indexes” — this will return, by default, random 20% of the rows (indexes) to use as a validation set. You can also send val_pct to get a specific percentage e.g val_idxs = get_cv_idxs(n, val_pct=1.0) gets 100%, but 20% is the default.

This consists of image name or id and the label.

Below is a pandas frame to group how many dogs are of the different breeds.

There is 120 rows representing 120 breeds.

Going through the steps;

tfms = tfms_from_model(arch, sz, aug_tfms=transforms_side_on, max_zoom=1.1)
data = ImageClassifierData.from_csv(PATH, 'train', f'{PATH}labels.csv', test_name='test', # we need to specify where the test set is if you want to submit to Kaggle competitions
                                   val_idxs=val_idxs, suffix='.jpg', tfms=tfms, bs=bs)

Enabling data augmentation;

tfms = tfms_from_model(arch, sz, aug_tfms=transforms_side_on, max_zoom=1.1)

call tfms_from_model and pass aug_tfms=transforms_side_on there are probably side on photos.

max_zoom — we will zoom into the image with up to 1.1 times

data = ImageClassifierData.from_csv(PATH, 'train', f'{PATH}labels.csv', test_name='test', val_idxs=val_idxs, suffix='.jpg', tfms=tfms, bs=bs)

ImageClassifierData.from_csv — last time, we used from_paths (which says the name of the folder are the name of the labels) but since the labels are in CSV file, we will call from_csv instead and call f’{PATH}labels.csv csv file that contains the labels. PATH is the contains all the data, train folder contain the training data. test_name specifies where the test set is if you will submit to Kaggle later.

val_idx — there is no validation folder but we still want to track how good our performance is locally. Separates out images and puts them in a validation set.

suffix=’.jpg’ — File names have .jpg at the end, but CSV file does not. So we will set suffix so it knows the full file names.

Get the training data set in the data object using trn_ds which contains the file names. Below is an example of a filename (fnames).

img = PIL.Image.open(fn); img

We need to check the size of the image. Then we need to know how to deal with them depending on whether they are too large or too small. Most of ImageNet models are trained on either 224 by 224 or 299 by 299 images.

Create a dictionary comprehension;

size_d = {k: PIL.Image.open(PATH + k).size for k in data.trn_ds.fnames}

Go through all the files and create a dictionary that maps the name of the file to the size of that file.

row_sz, col_sz = list(zip(*size_d.values()))

Takes the dictionary and turns it to rows and columns. Then turn them into numpy arrays as shown below:

row_sz = np.array(row_sz); col_sz = np.array(col_sz)

Here are the first five row sizes:

row_sz[:5]
array([500, 500, 500, 500, 500])

ploting with matplotlib. Images and the number of pixels:

from the histogram most images are around 500 pixels.

Plotting those less than 1000pixels to zoom in on the diagram:

4599 images lie within 451 pixels.

How many images should be in the validation set? The size of the validation set depends on the size of your dataset. It should not always be 20%. If you train the same model multiple times and you are getting very different validation set results, then your validation set is too small.

The image of the dog seems to be at the centre and taking the largest part of the frame. Therefore we don’t need cropping, this would be different for medical imaging as sometimes the tumor might be on one side of the frame thus requiring zooming.

Initial Model

Here is the regular two lines of code. When we start working with new dataset, we want everything to go super fast. So we made it possible to specify the size and start with something like 64 which will run fast. Later, we will use bigger images and bigger architectures at which point, you may run out of GPU memory. If you see CUDA out of memory error, the first thing you need to do is to restart kernel, then make the batch size smaller.

Precompute

we will use a pre-computed classifier.

We get 91% accuracy for the 120 classes. With no data augmentation or unfreezing.

Let’s turn pre-compute off and a few more epochs.

The accuracy improved to 92%.

Anepoch is one pass through the data, a cycle is how many epochs you said is in a cycle. It is the learning rate going from what you asked for down to zero. In this case since the cycle_len=1 the epochs and cycles are the same.

let’s save

learn.save(‘224_pre’)
learn.load(‘224_pre’)

Increase image size

If you trained a model on smaller size images, you can then call learn.set_data and pass in a larger size dataset. That is going to take your model, however it has been trained so far, and it is going to let you continue to train on larger images.

learn.set_data(get_data(299, bs))
learn.freeze()

Starting training on small images for a few epochs, then switching to bigger images, and continuing training is an amazingly effective way to avoid overfitting.

set_data doesn’t change the model at all. It just gives it new data to train with.

Validation loss (0.239) is much lower than training loss (0.297). This is a sign of underfitting. Cycle_len=1 may be too short. The learning rate is getting reset before it had a chance to zoom in properly.

let’s add cycle_mult=2 (i.e. 1st cycle is 1 epoch, 2nd cycle is 2 epochs, and 3rd cycle is 4 epochs = 7 epochs)

The validation loss and training loss are getting closer and smaller and almost the same. Implying we are on the right track.

Test Time Augmentation(TTA)

Other things to try:

• Running one more cycle of 2 epochs
• Unfreezing; in this case, training convolutional layers did not help in the slightest, since the images actually came from ImageNet.
• Remove validation set and just re-run the same steps, and submit that. This lets us use 100% of the data.

How do we deal with unbalanced dataset? This dataset is not totally balanced it is between 60 and 100 images for each class i.e breed, but it is not unbalanced enough to give it a second thought. A recent paper says the best way to deal with very unbalanced dataset is to make copies of the rare cases.

Difference between precompute=True and unfreeze?

We started with a pre-trained network which was finding activation with rich features. We added a couple of layers on the end of it which start out random. With everything frozen and precompute=True, all we are learning is the layers we have added. With precompute=True, we actually pre-calculate how much the image look likes the activations, therefore data augmentation does not do anything because we are showing exactly the same activations each time.

We then set precompute=False, it means we are still only training the last layers we added because it is frozen but data augmentation is now working because it is actually going through and recalculating all of the activations from scratch.Then finally, we unfreeze which is saying “okay, now you can go ahead and change all of these earlier convolutional filters”.

Why not just set precompute=False from the beginning? The only reason to have precompute=True is it is much faster 10 or more times. If you are working with quite a large dataset, it can save quite a bit of time. There is no accuracy reason ever to use precompute=True

Minimum version that would get you good results;

1. use lr_find() to find highest learning rate where loss is still clearly improving.
2. Train last layer with data augmentation (i.e. precompute=False) for 2–3 epochs with cycle_len=1 (By default everything is frozen from the start)
3. Unfreeze all layers.
4. Set earlier layers to 3x-10x lower learning rate than next higher layer. (Use differential learning rates)
5. Train full network with cycle_mult=2 until over-fitting.

Does reducing the batch size only affect the speed of training? Yes, If you are showing it less images each time, then it is calculating the gradient with less images hence, less accurate. In other words, knowing which direction to go and how far to go in that direction is less accurate. So as you make the batch size smaller, you are making it more volatile. It impacts the optimal learning rate that you would need to use, but in practice, dividing the batch size by 2 vs. 4 does not seem to change things very much. If you change the batch size by much, you can re-run learning rate finder to check if it has changed.

What would you have done if the dog was off to the corner or tiny ? This will be covered in Part 2, but there is a technique that allows you to figure out roughly which parts of an image most likely have the interesting things in them. Then you can crop out that part.

Further Improvements

1. Assuming the size of images you were using is smaller than the average size of images you have been given, you can increase the size. As we have seen before, you can increase it during training.
2. Using better architecture. There are different ways of putting together what size convolutional filters and how they are connected to each other. Different architectures have different number of layers, size of kernels, number of filters, etc.

We used ResNet34, It does not have too many parameters and works well with small dataset. ResNext50 takes twice as long and 2–4 times more memory than ResNet34.

I ran into RuntimeError: cuda runtime error (2) : out of memory. Try restarting your kernel and using a smaller batch size, I used 10

Using ResNext50 achieved 99.65% accuracy.

Thanks for reading! follow @itsmuriuki.

Back to learning!

Deep Learning; Personal Notes Part 1 Lesson 2, Learning rate, Data Augmentation, Annealing, Test… was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

I will be renting Paperspace GPUs (graphical processing units) to train neural networks as the cost seem quite affordable. Specifically, we will be using Nvidia GPU as they support CUDA.

fast.ai requires python 3

A brief recap on Fast ai

Fast ai is built on top of pytorch which is a deep learning library built by Facebook. The library takes the best practices and approaches in the industry and research. Each time an interesting paper is published they test it on a variety of datasets, tune it and implement it on the library.

Fast ai flexibility allows you to use this curated best practices as much or as little as you want. It makes it easy to write your own data augmentation, loss function and network architecture. The philosophy of fast ai is to learn things on as needed- basis.

Fast ai is code-driven and uses a top-down approach or the whole-game approach by David Perkins. Whereby it gets students using neural nets right away and getting results ASAP .

It then gradually peels back the layers to look at what is under the hood rather than the bottom-up approach where you learn the building blocks you need, and finally putting everything together.

Let’s take on the Cats vs Dogs challenge.

In this competition, you’ll write an algorithm to classify whether images contain either a dog or a cat. It is easy for humans to identify dogs and cats, your computer will find it a little more challenging.

We are going to use convolutional neural networks to classify the images

The imports

The data path

This folder structure is the standard way to share image classification files. Each folder tells you the label e.g. cats and dogs.

Looking at a cat’s image

f’{PATH}valid/cats/{files[4]}’ — This is a Python 3.6. format string which is a convenient to format a string.

plotting the image

let’s look at how the data looks like…

The image is a 3 rank tensor or a 3 dimensional array. (374, 500, 3)

(66,38,24) are the red, blue and green pixel values between 0 and 255. The numbers represent a subset of what an image looks inside a computer. We will take these numbers and use them to predict if the image is a cat or a dog by looking at a lot of pictures.

Let’s jump to training a model.

We’re going to use a pre-trained model that is a model created by someone else to solve a different problem.

Instead of building a model from scratch to solve a similar problem. We’ll use a model trained trained on ImageNet (1.2 million images and 1000 classes). The model is a Convolutional Neural Network(CNN).

What is a pre-trained model?

A pre-trained model is a model that has been previously trained on a dataset and contains the weights and biases that represent the features of whichever dataset it was trained on.

Learned features are often transferable to different data. For example, a model trained on a large dataset of bird images will contain learned features like edges or horizontal lines that would be transferable to your dataset.

Why use a pre-trained model?

Pre-trained models are beneficial to us for many reasons. By using a pre-trained model you are saving time. Someone else has already spent the time and compute resources to learn a lot of features and your model will likely benefit from it.

We will be using the resnet34 model.

Resnet34 is a version of the model that won the 2015 ImageNet competition. Other architectures in computer vision include; AlexNet, VGG Net, GoogleNet, ResNeXt etc

archdefines the architecture we will be using (resnet34)

data object contains the training and validation data.

ImageClassifierData.from_paths reads in images and their labels given as sub-folder names:

• path: a root path of the data (used for storing trained models, precomputed values, etc)
• bs: batch size. Default 64.
• tfms: transformations (for data augmentations). e.g. output of tfms_from_model. Default 'None'
• trn_name: a name of the folder that contains training images. Default 'train'
• val_name: a name of the folder that contains validation images. Default 'valid'
• test_name: a name of the folder that contains test images. Default 'None'
• num_workers: number of workers. Default '8'

tfms= tfms_from_modeltransformations — this gets the data ready to pass to our model. Normalization and resizing for example.

learn contains the model

ConvLearner.pretrained:

• f: arch. E.g., resnet34
• data: previously defined data object
• precompute: include/exclude precomputed activations. Default 'False' (precomputing- using pre trained weights)

learn.fit trains/fits the model through a given learning rate and epochs. In this instance, it is going to do 3 epochs with a 0.01 learning rate, meaning it is going to look at each image three times in total.

trn_loss and val_loss are the values of the cross-entropy loss function.

How good is this model? our model got an accuracy of 99%. The state of the art was 80%, but the competition resulted to a huge jump of 98.9% accuracy. Less than 4 years later that result can be beaten in seconds.

So as at 2013 you could get a Kaggle winning image classifier at the end in 17 sec and 3 lines of code. But, first it downloads a pre-trained model from the internet; this takes 2 min. It then pre-computes some caches; this takes 1 min and a half the first time you run the model. Subsequent training takes 20sec.

Hence it is not factual to say that deep learning takes a lot of time, resources and data.

Analysing Results

Looking at the validation set, it is a bunch of 0's and 1’s, where 0’s represent cats and 1’s represent dogs.

Let’s look at the first ten predictions

The fist column is cats the second is dogs. The model returns the log of the prediction instead of probabilities. To get the probabilities get the exp of the log.

Printing some random correct images

A dog is 1 so anything greater than 0.5 is a dog and anything less is a cat.

Printing incorrect images

Most correct cats

Most correct dogs

Most incorrect cats (classified as dogs yet they are cats)

Most incorrect dogs (dogs that the machine actually thought were cats)

Most uncertain prediction which are closes to 0.5. The classifier does not really know what to do with these images.

The reason for looking at your data is to try and visualise what the model has built so that you can take advantage of the things it is doing well and fix the thing it is doing badly. In this case we see that there are images that are not supposed to be here. We can also improve the model using data argumentation.

Assignment: grab some data of two or more different things put them in different folders and pass it through the tree lines of code (model). The best photos would be the normal day today photos people take pictures of. Note: satellite images, CT scans, microscope and pathology picture won’t work for this code. (10 examples should do the trick)

Image Classification Examples

Check out this image classifier that uses mouse movement images to detect fraudulent transactions.

Let us backtrack a little, what is Deep Learning?

Deep Learning is a kind of Machine Learning. It is however not equal to Machine Learning. In Machine Learning we have to do most of the feature engineering ourselves, in deep learning we leave that to the neural network.

C-Path (Computational Pathologist) is an example of a traditional Machine Learning approach. Pathology slides of breast cancer biopsies were taken for the study. Then many pathologists consulted on ideas about what kinds of patterns or features might be associated with long-term survival. They then wrote specialist algorithms to calculate these features, run through logistic regression and predict the survival rate. The algorithms outperformed pathologists. It however took domain experts and computer experts many years of work to build.

Below are the properties that make deep learning class of algorithm better:

Instead of building domain specific functions with lots of feature engineering, you build an infinitely flexible function that can solve any problem, if you set the parameter of that function correctly. Then establish a fast and scalable all-purpose way of setting the parameters of that function.

The underlying function that deep learning use is called a neural network

A neural network contains a number of linear layers interspaced with a number of non-linear layers. Thus giving us the universal approximation theorem that says these kind of functions can solve any given problem to arbitrary close accuracy — as long as you add enough parameters. This proves it is an infinitely flexible function.

To do an all purpose parameter fitting, we use gradient descent. For the different parameters that we have we need to ask ourselves how good they are at solving the problem at hand.

Let’s figure out a better set of parameters which happen to follow the loss function downwards to find the minimum thus ending up at local minimum.

The finding of parameters have to be done in a reasonable amount of time thus, the use of GPUs.

From the diagram below GPUs are 10X faster than the CPUs.

To make a neural net scalable and fast we need multiple hidden layers hence the word “deep learning”.

link to another example;

Diagnosing lung cancer

Things you could use deep learning for;

A key piece of the Convolutional neural network is the convolution, this website has a visual explanation of the convolution. Also check out this neural network interactive book.

How to set the parameters

Let’s take a quadratic function and try to find the minimum.

You start by randomly picking a point Xn and calculating the value of your quadratic at that point on the quadratic curve by finding the derivative. This gives you the gradient that tells you which way is down and which way is up.

Take a small step in the left direction to create a new point Xn+1 and keep repeating the process by taking small steps, each time it takes you closer to minimum.

The small step and number denoted by ℓ is called the learning rate.

With too high step size you end up with divergence rather than convergence. If the learning rate is too small it will take longer to converge.

Combining linearity, non-linearity, convolution and gradient decent gives us a Convolutional Neural Network

Choosing a learning rate

learn.fit(0.01, 3)

0.01 is the learning rate. This is something you have to set.

Learning rate is how much you multiply your gradient when taking each step in your gradient decent. Setting the learning rate well is important so that you don’t diverge or take too long to get to convergence. Learning rate affects model performance.

The method learn.lr_find() helps you find an optimal learning rate. It uses the technique developed in the 2015 paper Cyclical Learning Rates for Training Neural Networks, where we simply keep increasing the learning rate from a very small value, until the loss stops decreasing. We can plot the learning rate across batches to see what this looks like.

We first create a new learner, since we want to know how to set the learning rate for a new untrained model.

Our learn object contains an attribute sched that contains our learning rate scheduler, and has some convenient plotting functionality including this one:

Note that in the previous plot iteration is one iteration or minibatch of SGD(stochastic gradient descent). In one epoch there are (num_train_samples/num_iterations) of SGD.

Stochastic Gradient Descent — samples are selected randomly (or shuffled) instead of as a single group (as in standard gradient descent) or in the order they appear in the training set.

We can see the plot of loss versus learning rate to see where our loss stops decreasing:

Go back and look at what point did we see the best improvement, then use that learning rate. The loss is still clearly improving at lr=1e-2 (0.01), so that’s what we use.

Note that the optimal learning rate can change as we train the model, so you may want to re-run this function from time to time.

Loss is how accurate the model is, how far is the prediction from the goal. we pick the point the curve drops quickly and not the minimum point as this might be the point it jumped too far.

The training of the model does not get to 100% as it notices the loss is getting worse and stops. It gets to 75%

We then pick the learning rate where the loss is still clearly improving, In this case. highest learning rate possible but still improving.

Choosing the number of epochs

learn.fit(0.01, 3)

3 this is the number of epochs.

An epoch is one complete presentation of the data set to be learned to the model.

On each epoch we print training loss, validation and accuracy.

You can run as many epochs as you like but the accuracy might start getting worse due to overfitting.

A few other considerations are the size of the model and the amount of data. If both are too large, it will require a lot of time to train the model. Therefore you only choose your epoch based on the amount of time you have.

Tips and Tricks on jupyter notebook

1.Checking out a function name you can’t remember type the first letter and hit tab once.

2. If you can’t remember what the arguments are to a method hit shit+tab

3. If you can’t remember what the function does hit shift+ tab + tab. It bring up the documentation.

4. Typing ?? before a function bring up the source code.

5. pressingH bring out the keyboard shortcut. Try learning 4 to 5 of them a day and practice.

Always stop your paperspace, crestle or AWS after finishing your work. otherwise you will be charged.

Take your time and hangout in the forums.

Thanks for reading, I appreciate! follow @itsmuriuki.

Back to learning!

Deep Learning; Personal Notes Part 1 Lesson 1, Image Classification was originally published in Coinmonks on Medium, where people are continuing the conversation by highlighting and responding to this story.

TLDR: writing as proof of work

This blog post won’t be long. I don’t intend to waste your time by telling you why I do things the way I do them as my circumstances are definitely different from yours.

I started on Data Science as a hobby then found myself spending most of my time on it. This goes without saying that most of the skills I’ve since acquired are self-taught. The approach I took was breadth first then depth.

The plan was to first get basic working knowledge of the field then dive straight to doing projects this and this. So far so good. One thing still bothers me though…how to keep track of my progress as there are no grades or levels. As such writing I believe will get me to the bottom of concepts I learn and the projects I build will demonstrate my skill. What I intend to write will not be new. I plan to just use this blog as my notebook to document what I learn and courses I pursue. I will strive to understand things clearly and explain them well.

Just to make sure I’m doing world-class stuff I will try to enter a few Kaggle competitions and try to achieve a good ranking. This will all be used as proof of work.

Thanks for reading, I appreciate! follow @itsmuriuki.

Back to learning!

If you’re not just a soccer fan but also a techie, I guess you have realized that Machine Learning and Artificial Intelligence are presently buzzwords too. Let us combine these two to predict which country will win the FIFA World Cup.

Disclaimer: This should in no way be used for betting or any financial decision. Should you choose to, who am I to stop you.(just don’t forget me if you hit a jackpot 😃).

A lot of factors are involved in the game of football and as such all of them cannot be scoped out in a machine learning model. This is just a hacker trying some cool shit with data…

Goal

1. The goal is to use Machine Learning to predict who is going to win the FIFA World Cup 2018.
2. Predict the outcome of individual matches for the entire competition.
3. Run simulation of the next matches i.e quarter finals, semi finals and finals.

These goals present a unique real-world Machine Learning prediction problem and involve solving various Machine Learning tasks: data integration, feature modelling and outcome prediction.

Data

I used two data sets from Kaggle. You can find them here. We will use results of historical matches since the beginning of the championship (1930) for all participating teams.

Limitation: FIFA ranking was created in the 90’s thus a huge portion of the dataset is lacking. So let’s stick to historical match records.

Environment and tools: jupyter notebook, numpy, pandas, seaborn, matplotlib and scikit-learn.

We are first going to do some exploratory analysis on the two datasets, do some feature engineering to select most relevant feature for prediction, do some data manipulation, choose a Machine Learning model and finally deploy it on the dataset.

Let the rubber hit the road!

First things first, import the necessary libraries and load the datasets into a

Dataframe.

Importing libraries

Loading the datasets…

Ensure the datasets are loaded in the dataframes by calling world_cup.head() and results.head() for both datasets as shown below:

Exploratory Analysis

After analyzing both datasets, the resulting dataset has data on past matches. The new (resulting) dataset will be useful for analysis of and predicticting future matches.

Exploratory analysis and feature engineering: which involve establishing which features are relevant for the Machine Learning model is the most time consuming part of any Data science project.

Let’s now add goal difference and outcome column to the results dataset.

Check out the new results dataframe.

Then we’ll work with a subset of the data. One that includes games played only by Nigeria. This will help us focus on what features are interesting for one country and later expand to countries participating in the world cup.

The first World Cup was played in 1930. Create a column for year and pick all the games played after 1930.

We can now visualize the most common match outcome for Nigeria throughout the years.

Getting the winning rate for every country that will participate in the world cup is a useful metric and we could use it to predict the most likely outcome of each match in the tournament.

Venue of the matches won’t matter that much.

Narrowing to the teams participating in the World Cup

Create a dataframe with all the participating teams.

We then further filter the results dataframe to show only teams in this years world cup from 1930 onwards as well as drop duplicates.

Create a year column and drop games before 1930 as well as columns that won’t affect match outcome for example date, home_score, away_score, tournament, city, country, goal_difference and match_year.

Modify the “Y” (prediction label) in order to simplify our model’s processing.

The winning_team column will show “2” if the home team has won, “1” if it was a tie, and “0” if the away team has won.

Convert home_team and away _team from categorical variables to continuous inputs, by setting dummy variables.

Using pandas, get_dummies() function. It replaces categorical columns with their one-hot (numbers ‘1’ and ‘0’) representations enabling them to be loaded into Scikit-learn model.

We then separate the X and Y set and split the data into 70 percent training and 30 percent test.

We will use logistic regression, a classifier algorithm. How does this algorithm work? It measures the relationship between the categorical dependent variable and one or more independent variables by estimating probabilities using a logistic function. Specifically the cumulative logistic distribution.

In other words logistic regression attempts to predict an outcome (a win or a loss) given a set of data points (stats) that likely influence that outcome.

The way this works in practice is you feed the algorithm one game at a time, with both the aforementioned “set of data” and the actual outcome of the game. The model then learns how each piece of data you feed it influences the outcome of the game positively, negatively and to what extent.

Give it enough (good) data and you have a model that you can use to predict future outcomes.

A model is as good as the data you give it.

Let’s have a look at our final dataframe:

Looks great. We are now ready to pass this to our algorithm:

Our model got a 57% accuracy on the training set and 55% accuracy on the test set. This doesn’t look great but let’s move on.

At this point we will create a dataframe that we will deploy our model.

We will start by loading the FIFA ranking as of April 2018 dataset and a dataset containing the fixture of the group stages of the tournament obtained from here. The team which is positioned higher on the FIFA Ranking will be considered “favourite” for the match and therefore, will be positioned under the “home_teams” column since there are no “home” or “away” teams in World Cup games. We then add teams to the new prediction dataset based on ranking position of each team. The next step will be to create dummy variables and and deploy the machine learning model.

Match Prediction

By now you are wondering will we ever get to the predictions? That has been too much code and talk, when will you show us the prediction? Just hold on tight we are almost there…

Deploying the model to the dataset

We start with deploying the model to the group matches.

Here are the results of group stages.

The model predicted three draws it also bet safely between Portugal and Spain but awarded Spain a high probability of winning. I used this website for simulation of the group stage matches.

Below is the round of 16 knockout simulation.

The model predicted a quarter final between: Portugal vs France, Brazil vs England, Spain vs Argentina and Germany vs Belgium.

Here are the predictions;

The semis

Portugal vs Brazil and Germany vs Argentina

prediction;

And finally the big match

Brazil vs Germany

prediction;

According to this model Brazil is likely to win this World Cup.

Areas of further Research/ Improvement

1. Dataset; to improve dataset you could use FIFA, the game not the organisation, to assess the quality of each team player.
2. A confusion matrix would be great to analyse which games the model got wrong.
3. We could ensemble that is, we could try stacking more models together to improve the accuracy.

Conclusion

There is so much that can be done to improve this work. For now let us see whether we get lucky. Full disclosure: I’m not a huge fan of football 😀. So predict at your own risk.

The full code can be found here.

Thank you for reading. I appreciate. Follow @itsmuriuki on Twitter.

Back to code and more learning.

Thanks Oluwole and Hazel for going through the draft.

Predicting FIFA World Cup 2018 using Machine Learning. was originally published in Good Audience on Medium, where people are continuing the conversation by highlighting and responding to this story.

First what is the problem?

The problem is to build a model that will predict house prices with a high degree of predictive accuracy given the available data. More about it here. “With 79 explanatory variables describing (almost) every aspect of residential homes in Ames, Iowa, this competition challenges you to predict the final price of each home.”

Where I got the data

The dataset is the prices and features of residential houses sold from 2006 to 2010 in Ames, Iowa. Obtained from here.

Environment and tools.

jupyter notebook, numpy, pandas, seaborn, matplotlib, scipy and scikit-learn.

First things first, Import the python libraries and dataset

pearsonr (Pearson correlation coefficient) is a measure of the linear correlation between two variables X and Y. It has a value between +1 and −1, where 1 is total positive linear correlation, 0 is no linear correlation, and −1 is total negative linear correlation.

Load the train and test file

Lets drop the ‘Id’ column in the data as it is not necessary for prediction

Let’s see what the dataset looks like

The training data has 1460 observations and 80 explanatory variables and the test file has 1459 observations and 79 explanatory variables. The test data doesn’t have the target variable which is the ‘SalePrice’.

Here is a data description file. (data_description.txt)

• SalePrice — the property’s sale price in dollars. This is the target variable that you’re trying to predict.
• LotArea: Lot size in square feet
• Neighborhood: Physical locations within Ames city limits
• OverallQual: Overall material and finish quality
• OverallCond: Overall condition rating
• YearBuilt: Original construction date
• TotalBsmtSF: Total square feet of basement area
• GrLivArea: Above grade (ground) living area square feet

e.t.c

I dived into exploring the data and I started with a descriptive statistic summary of the target variable (SalePrice) .

This gives us the count, mean, standard deviation, percentile 25%, 50% and 75%, min and max.

I went ahead plotted the distribution of the ‘SalePrice’ and normal probability graph which is used to identify substantive departures from normality. This includes identifying outliers, skewness and kurtosis. I used the QQ-plot

The SalePrice deviates from normal distribution and is positively biased

The SalePrice also does not align with the diagonal line which represent normal distribution in normal probability graph. A normal probability plot of a normal distribution should look fairly straight, at least when the few large and small values are ignored.

Went on to use logarithmic transformation to make highly skewed distributions less skewed.

Correlation Analysis

Here I explored the correlation between the features and the target by plotting a correlation matrix

from the figure above I could only tell that OverallQual, TotalBsmtSF, GarageCars, GarageArea have a high positive correlation with SalePrice. Unfortunately, the colors are not very clear. Below is a better representation with values.

From this we can tell which features (OverallQual, GrLivArea, GarageCars, GarageArea, and TotalBsmtSF)are highly positively correlated with the SalePrice. Here is a correlation guide.

GarageCars and GarageArea have strongly correlation, TotalBsmtSF and 1stFlrSF have similarly high correlation. I assumed that the number of cars stored in a garage strongly depends on the area of the garage. On looking at the data homes with 0 GarageCars had 0 GarageArea indicating they don’t have a garage.

Here are some scatter plots to visualize some of the features relationship against the SalePrice.

For more scatter plots here is the notebook.

Dealing with outliers

I deleted the GrLivArea as it was recommended by the author of the data . “I would recommend removing any houses with more than 4000 square feet from the data set (which eliminates these five unusual observations) before assigning it to students.”

Inputting Missing Values

Let’s have a look at the training data (train.csv)

To try and understand what the missing values are I looked at the data documentation this helped me transform other features to reflect the assumptions I made. for Example, GarageArea is zero, indicates we don’t have a garage and cars should be transformed to 0 as well. I also set mode value for missing entries and transformed numerical features to categorical features.One other key feature that I added was the Total surface area as TotalSF by adding together TotalBsmtSF, 1stFlrSF and 2ndFlrSF.

I also log transformed highly skewed features using box cox transformation which is a way to transform non-normal dependent variables into a normal shape. This were 59 skewed features. Normality is an important assumption for many statistical techniques; In this case the data isn’t normal, applying a Box-Cox also increases the ability to run a broader number of tests.Went on to adding of dummy variables for categorical features

Linear regression modelling

I used LASSO (least absolute shrinkage and selection operator) and Gradient boosting regression models to train the dataset and make predictions separately.

LASSO is a regression model that does variable selection and regularization. The LASSO model uses a parameter that penalizes fitting too many variables. It allows the shrinkage of variable coefficients to 0, which essentially results in those variables having no effect in the model, thereby reducing dimensionality. Since there are quite a few explanatory variables, reducing the number of variables may increase interpretability and prediction accuracy.

Gradient boosting models are one of the most popular algorithms on Kaggle. A variant of GBMs known as the XGBoost has been a clear favorite for many recent competitions. The algorithm works well right out of the box. It is a type of ensemble model, like the random forest, where multiple decision trees are used and optimized over some cost function. The popularity and ability to score well in competition are reasons enough to use this type of model for house price prediction problem.

I used cross validation and RMSLE(Root mean squared logarithmic error ) to see how well each model performs.

Here are the scores I received from my two models

Went ahead and stacked the two models together it has been proved by several research papers and here, that stacking two models together what is called ensembling can improve the accuracy.

Finally, I did training and prediction of the stacked regressor

Conclusion

After submission to the Kaggle competition, I was placed top 12%. Not bad! Here is my notebook and the leaderboard. More can be definitely done.

Thanks for reading! I appreciate you.follow @itsmuriuki.

Back to code, Back to more learning.

Thanks Oluwole for going through the draft.

Predicting House Prices Using Linear Regression was originally published in Africa Creates on Medium, where people are continuing the conversation by highlighting and responding to this story.