Reading papers can be uncomfortable and scary at first. And one way to overcome this is by reading a well-written article in an easier tone. If a paper you picked is challenging to digest, get some intuition from a demystifying video or a blog (like this one 😉). After you get the core concept of the network, then try again with the original one. It’ll become more comfortable and more fun to read than just jumping right into the hard one.

This is the third story of “Deep Dive into the Computer Vision world,” and the complete set of the series is as follows:

1. Starting from VGG, ResNet, Inception Network and MobileNet
2. Regions with CNN, let’s start object detection
3. YOLO, SSD, FPN and RetinaNet, more unified ones
4. From object detection to instance segmentation (TBU)

In the previous post, we discussed object detection networks- R-CNN, Fast R-CNN, and Faster R-CNN. These are based on two-stage architecture. Generating the candidates for object locations at the first stage and localizing the objects and classifying the detected objects at the second stage.

Now the next networks to be explored are SSD, YOLO, FPN and RetinaNet. These networks are based on one-stage architecture. The biggest change is that these no longer need regional proposals, hence forming a more unified and fully convolutional network, which results in brings faster performance. Now let’s see how this can be possible one by one.

SSD

SSD stands for Single Shot Detector, and there are three main points in its architecture. First, this network can detect in multiple scales. With the layers coming after the base network, it yields several feature maps with different resolutions, which enables the network to work at multi-scales.

Second, these predictions are made in a convolutional way. A set of convolutional filters make the predictions for the location of the objects and the class score. This is where the name of “Single Shot” comes from. Instead of having additional classifiers and regressors, now the detections are made in a single shot!

And lastly, there are a fixed number of default boxes associated with these feature maps. Similar to the anchors of Faster R-CNN, the default boxes are applied at each feature map cell.

The entire architecture of the network is as shown above. It uses VGG-16 as its base network, and there are additional feature layers which decrease in size progressively. This can be considered to be quite complicated at first glance, so let’s split it into two parts.

In the picture on the left, the feature layer from the base network produces a detection by the convolutional filters. When the dimensional size of the feature layer is N x N with P channels, the filter size here is 3x3xP. Now, as I said above, there are a set of default bounding boxes at each cell. So if we have K number of the boxes and C number of classes (including the “background” class), we compute the four offset values against the original default box, and the class scores for each box. Therefore the total number of filters becomes (NxN)K(4+C).

SSD makes all the predictions for the bounding box location and the class scores at once with the convolutional filters. It doesn’t require a region proposal network anymore. And the same process is done at the extra feature layers in different scales, which improves accuracy significantly. Recall that the feature maps in different resolutions, see the different sizes of figures or objects in an image (I explained this with Inception Network). Therefore, the prediction across scales becomes critical in detection tasks.

After we get the outcome from all these layers, we match them with the ground truth box and choose the best one. As most of the boxes are going to be overlapped heavily, we can use Non-Maximum Suppression to choose one. But still, there is quite a challenging problem, the class imbalance. When we use the regional proposals, they are reasonable candidates for having an object. But here, the network starts from a set of default boxes at each location with multiple scales and shapes. As a result, most of the boxes are going to be negative in the outcome. This creates an imbalance in classes. For example, the “Background” class becomes an easy example and the real “object” classes become hard examples. The easy negative examples can overwhelm the distribution of the classes and degenerate the model. (Note that this issue is going to be dealt with again in RetinaNet)

To remedy this, SSD takes only some of the negative matches so that the ratio between positive and negative matches could be 3:1. This is called hard negative mining. Hard negative mining is a classic problem in object detection, which means a negative class is hard to tell as negative. A model should classify it as negative, but it gives high confidence as the negative cases become way more than the positive cases. We input only the positive matches into the objective function so that it calculates the loss between the ground truth and the predicted bounding box.

One interesting thing is that about 80% of the forward time is spent on the base network, which indicates that a faster base network could even further improve the speed. And data augmentation also plays an important role for higher accuracy. Compared to Faster R-CNN using only an original image and horizontally flipping, variations in input sizes, shapes, and random sampling improve the performance. Also, it is observed that various default box shapes are better for accuracy as well.

YOLO

YOLO is an acronym of You Only Look Once. Similar to SSD, YOLO doesn’t use regional proposals and works in a “single-shot” way. There are 3 main variations of YOLO. Rather than covering all the transformation process, I’m going to wrap all and explore the final version. If you’d like to know all the detailed process, check out this excellent article by Jonathan Hui. (Literally, this is the best YOLO tutorial I’ve ever seen).

The whole network architecture is as shown above. The numbers in the circles are the layer numbers. It uses Darknet-53 as its base network, and there are additional layers for detecting objects. Let’s first put our attention at the tail where the predictions are made. YOLO is also a fully convolutional network, so it predicts the detection by convolution. After passing through several layers, the detection is made by 1x1 convolution from the feature maps in three different scales.

When the size of the feature map is N x N with P channels, 1x1xP size of filters is applied. YOLO also has the default bounding boxes (They are called anchors in the original paper). For each box at each cell, the 4 bounding box offset values, the objectness score and the class scores are predicted. The objectness score reflects how confident it is that the box contains an object. So now the number of predictions for each grid becomes 5+C. So if the number of boxes is K, the total number of filters is (NxN)K(5+C) for each scale.

As I said, the resolution of the feature maps gets smaller as it goes deeper, and this gives an impact on the scales when the feature maps see the objects. So to achieve multiple scales, YOLO brings higher resolution features from a previous layer simply by a passthrough layer. This is the same with the shortcut connection of ResNet.

So at the 84th layer, we take the feature map from the two previous layers and upsample it by 2. And then we bring a feature map from the 36th layer and concatenate it with the upsampled features. The same process is repeated one more time at the 96th layer for the final scale. By doing so, we can add finer and more meaningful features into the prediction.

The base network was AlexNet at ver.1 and Darknet was first used at ver.2 with 19 convolutional layers. This became 53 layers at ver.3 as you can see above. Another important thing worth mentioning is that the default boxes are extracted from K-means clustering, rather than being selected by hand. The network learns to adjust the boxes during training but having better priors from the start helps the network for sure. So YOLO ver.3 extracts the 9 default boxes by clustering. By concatenating the higher resolution features from the previous layer and a passthrough layer, YOLO gets better detection with small objects compared to SSD.

FPN

FPN stands for Feature Pyramid Network. A network with a pyramid structure has been studied and used before this. And SSD is also a pyramid-like structure (try rotating it vertically). But it can’t be said it’s a pyramidal feature hierarchy strictly as shown in figure ©. Because it starts to form a pyramid from a higher level, not from the low level. (Remember VGG Conv5_3 was the first layer for detection)

What’s the point having this kind of architecture? The information for the precise location of an object can be detected in the deeper level of the network. On the contrary, the semantics gets stronger in the lower level. (we’re going to talk about this in the next series). When a network has one way of just going more in-depth, there aren’t many semantic data we can use to classify objects at a pixel level. Therefore by implementing a feature pyramid network, we can produce multi-scale feature maps from all levels, and the features from all these levels are semantically strong.

So the architecture of FPN is as shown above. There are three directions, bottom-up, top-down and lateral directions. The bottom-up pathway is the feedforward computation of the backbone ConvNet and ResNet as used in the original paper. There are 5 stages and the size of the feature map at each stage has a scaling step of 2. They are differentiated by their dimensions, and we call them as C1, C2, C3, C4 and C5. The higher the layer goes, the smaller the size of the feature maps gets along the bottom-up pathway.

Now at the level C5, we move to the top-down pathway. Upsampling is applied to the output maps at each level. And the corresponding map that has the same spatial size from the bottom-up pathway, is merged via a lateral connection. A 1x1 convolution is applied before the addition to match the depth size. Then, we apply a 3x3 convolution on the merged map to reduce aliasing effect of upsampling. By doing so, we can combine the low resolution and semantically strong features (from top-down pathway) with high resolution and semantically weak features (from bottom-up pathway). This implies we can have rich semantics at all scales. And as shown in the picture, The same process is iterated and produces P2, P3 P4 and P5.

Now the last part of the network is an actual classification. Actually, FPN isn’t an object detector in itself. A detector isn’t built-in so we need to plug in a detector from this stage. RPN and Fast R-CNN are used in the original paper, and only FPN with RPN case is depicted in the above picture. In the RPN, there are two sibling layers for an object classifier and bounding box regressor. So in FPN, this part is attached to the tail of each level of P2, P3, P4 and P5. I won’t cover much about this here, because the reason I picked FPN was to explain RetinaNet.

RetinaNet

RetinaNet is more about proposing a new loss function to handle class imbalance, rather than publishing a novel new network. Let’s bring the class imbalance back again. One-stage detectors such as SSD and YOLO are apparently faster but still fall behind in accuracy compared to two-stage detectors. And the class imbalance issue is one of the reasons for this drawback.

So the authors proposed a new loss function called focal loss by putting a weight on easy examples. The mathematical expression is as shown above. It’s a cross-entropy loss multiplied with a modulating factor. The modulating factor reduces the impact of easy examples on the loss.

For example, compare the loss when Pₜ= 0.9 and γ = 2. If we say the cross-entropy loss as CE, then the focal loss becomes -0.01CE. The loss becomes 100 times lower. And if the Pₜ gets bigger, say 0.968, with the constant γ, the focal loss becomes -0.001CE (as (1–0.968)² = (0.032)² ≈ 0.001). So it puts down the easy examples even harder and adjusting the imbalance in return. And when γ increases, as you can see the graph on the right, the loss gets smaller as the weight increases when Pₜ is constant.

The architecture of RetinaNet is as shown above. As we now know ResNet, RPN of Faster R-CNN and FPN, there is nothing new here. The network can be divided into two parts, a backbone with (a) and (b), and two subnets for classification and box regression. The backbone network is composed of ResNet and FPN and the pyramid has levels P3 to P7.

As with RPN, there are prior anchor boxes as well. The size of anchors changes according to its level. At the level P3, the anchor area is 32*32 and at P7, it’s 512*512. The anchors have three different ratios and three different sizes, so the number of anchors A for each level is 9. Therefore, the dimension of output from a box regression subnet is (N x N) x 4A. And when the number of classes in the dataset is K, the output from a classification subnet becomes (N x N) x KA.

The result was quite eye-catching. RetinaNet outperformed all the previous networks in both accuracy and speed. RetinaNet-101–500 indicates the network with ResNet-101 and a 500-pixel image scale. Using larger scales gives higher accuracy than all two-stage approaches, while still being fast enough as well.

Take away

There are two kinds of object detection networks, one-stage and two-stage. Each has its own pros and cons. Two-stage networks, such as Faster R-CNN, are accurate but impractical for real application. One-stage networks, such as SSD and YOLO, are fast enough but the accuracy is relatively low. This was the motivation of RetinaNet, developing a one-stage network achieving the two rabbit.

Object detection should work on multi-scales because the scales of objects aren’t constant. We saw what strategies the networks use for scale-invariant detection. SSD had multiple layers, downsizing the resolutions in a way. YOLO had 3 layers in multi-scales with upsampling and passthrough layers. FPN and RetinaNet made a pyramid working both in top-down and lateral directions.

Class imbalance is another pain in an object detection task. There have been several methods to remedy this. SSD used hard negative mining but there is a major drawback that we have to use only some of the examples collected. Another solution RetinaNet proposed was a focal loss. By adding the weight parameter, we can suppress the contribution of the easy examples and also, use all the examples efficiently.

References

• Wei Liu et al. [SSD: Single Shot MultiBox Detector], 2016
• Joseph Redmon et al. [You Only Look Once: Unified, Real-Time Object Detection], 2016
• Joseph Redmon et al. [YOLO9000: Better, Faster, Stronger], 2016
• Tsung-Yi Lin et al. [Feature Pyramid Networks for Object Detection], 2017
• Joseph Redmon et al. [YOLOv3: An Incremental Improvement], 2018
• Tsung-Yi Lin et al. [Focal Loss for Dense Object Detection], 2018

Did this story resonate with you? Please share your insight with us. I’m always open to talk, so feel free to leave comments below and share your thoughts. I also share interesting and useful resources on LinkedIn so feel free to follow and reach out to me. I’ll be back with another interesting story, next time. As always, stay tuned!

Deep Dive into the Computer Vision World: Part 3 was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

This is the second story of “Deep Dive into the Computer Vision world,” and the complete set of the series is as follows:

1. Starting from VGG, ResNet, Inception Network and MobileNet
2. Regions with CNN, let’s start object detection
3. YOLO, SSD and RetinaNet, more unified ones
4. From object detection to instance segmentation (TBU)

In the previous post, we discussed five popular networks, VGG, ResNet, Inception Network, Xception and MobileNet. These networks are now a major component in advanced networks, hence it’s necessary to know their architectures.

Now, we’re going to the next page. From image classification to object detection. So far, what we’ve discussed was one large convolutional neural network. Although they have many layers, they are one network at the end. But R-CNN and its variants that will be our focus, have this network as a part. A CNN is now a part of an entire “system” processing more complex tasks such as object detection and image segmentation. Starting with R-CNN, we’re going to see how these networks had been transformed and the underlying idea for the changes.

From image classification to object detection

Before we jump right into the R-CNN “family,” let’s briefly check the basic idea of image classification and image detection. What’s the difference between them? What additional work do we need to do for detecting objects in an image?

First, whatever the number of classes we have, there is going to be an additional class- the background. An object detector is required to answer the question, “is there an object?”, which is not the case for image classification. Second, when there is an object, it’s still not sufficient with “yes, there is!” (Quite ridiculous to imagine..😅) It should also tell us, “where is the object located?” We need the position of the detected objects. This may sound simple but the implementation isn’t. And when we take speed and efficiency challenges into account, things get even more complicated.

Therefore object detection would be like.. looking for the regions of an object, localizing it and classifying what it is. Having this basic concept in mind, we’re now ready to start the second topic from R-CNN.

R-CNN

R-CNN answers the question, “to what extent can the CNN classification results on ImageNet generalize to object detection results?” So this network can be said to be the beginning of object detection family tree, which has great importance in the application of a neural network. The basic structure is composed of three steps-extracting region proposals, computing CNN, and classification.

First, we extract some regions that seem promising to have an object from an input image. R-CNN used selective search for getting those regions of interest (ROI). Selective search is a region proposal algorithm that segments an image based on the intensity of the pixels. If you’d like to get an in-depth understanding of Selective Search, here is the original paper. The basic idea is shown below. It first starts with the overly segmented picture and draws a bounding box around each segment. And based on their similarities in color, texture, size and shape compatibility, it keeps grouping adjacent ones and forming larger segments. R-CNN extracts around 2000 region proposals by this method and feeds them to the CNN. And this is the reason it is named R-CNN, Regions with CNN features.

After getting the candidates, the second step is entering a large convolutional neural network. Each proposal is resized into a fixed size and is input into the CNN separately. R-CNN used AlexNet (It was 2014, there weren’t ResNet or InceptionNet at that time) and we get 4096-dimensional feature vector from each proposal.

And at the last step, the output extracted from the CNN is fed into a set of class-specific linear SVM models. We optimize one linear SVM per each class, and we get the output image with the bounding boxes whose score is higher than a threshold value. In case of having overlapping boxes, it only takes one by applying non-maximum suppression.

One interesting part worth mentioning is how it overcame the data scarcity issue. The researchers had the challenge to train such a large network with only a small quantity of labeled data. And the solution was pre-training the CNN on a different data with labels (which is supervised learning), and then do fine-tuning with the original dataset.

You may have noticed some inefficient parts that could be enhanced. The selective search is computation intensive. And processing the CNN for each ROI was repetitive work, which requires a huge amount of computation cost again. The need for the pre-training process and the separate classifiers were unattractive. And the storage for these models is too big. Although R-CNN was a milestone achievement, it had several drawbacks to be improved.

Fast R-CNN

Fast R-CNN is the next version of the previous work. What changes had been made here? Processing convolutional mapping repetitively was improved. The first change occurred at the repetitive convolution layers.

Let’s assume it takes N seconds to compute one single convolutional network. As R-CNN input 2000 RoI to the network separately, the total processing time will be 2000*N seconds. Instead of processing the CNN individually, now we do it only once by sharing the convolution with the proposals altogether.

As you can see above, this network takes two data input, an original image and a set of the region proposals as input. The whole image feeds forward through the network to produce a feature map. With this feature map, each region proposal passes a pooling layer and fully connected layers to create a feature vector. Therefore, the convolutional computation is done once, not for each proposal.

And the multiple classifiers are replaced with two sibling layers. One is a softmax function for classifying the object with the possibility estimates for each class, and the other is a bounding-box regressor returning the coordinates of the detected object. So the resulting feature vector is fed into these two layers and we get the outcome from the two layers.

Now, this model made changes to share convolutions and to detach the additional classifiers. By merging “multiple bodies” into one and taking off the “heavy tail,” we could achieve less computation and storage. This architecture allows us to train all the weights together, including even that of the softmax classifier and the multiple regressors. And this means the propagation will flow back and forth updating all the weights. Awesome progress! However, we still have a chance to get better performance.

Faster R-CNN

Still, Fast R-CNN was hard to be used in real time detection. And the major reason for such time delay stemmed from selective search. It had computation bottleneck and it needed to be replaced with a more efficient way. But how? How could we get region proposals without Selective Search? What about maximizing the usage of the ConvNet? Researchers found that the feature maps in Fast R-CNN also can be used for generating region proposals. Therefore by being free from Selective Search, much efficient network could be developed.

Faster R-CNN is composed of two modules, Region Proposal Network (RPN) and Fast R-CNN. We first input an image to a “mini-network” and it will output the feature maps. By sliding a small window over the feature maps, we extract region proposals. And each proposal is fed into two sibling layers, a softmax classifier and a bounding-box regressor.

These two layers may seem similar to the last layers of Fast R-CNN, but they are derived for a different purpose. For each proposal, the classifier estimates the probability of the presence of an object in an image. And the regressor returns the coordinates of the bounding box. So they are for generating candidates, not predicting the actual objects.

We only take the proposals whose scores are higher than a threshold and input them with the feature map together to Fast R-CNN. The following steps are the same. They are input into a convolutional network and RoI pooling layers. And then the last layers will be a classifier and a regressor that finally predicts the real objects in images.

One important property of this network is translation invariant, which is achieved by anchors and the way it computes proposals relative to the anchors. What is translation invariant? Simply put, it means we can detect an object regardless of its rotation, relocation, size change and what not. An object in an image can be at the center or the upper left. The same object can be at wide or long scale depending on the perspective. To prevent a model from failing to locate an object because of translation, we make anchor boxes of multiple scales and aspect ratios as in the picture shown above. These boxes are put at the center of the sliding window. So if there’s K number of the boxes at a certain position, we get 2K scores and 4K coordinates of K boxes.

In conclusion, at the Region Proposal Network, we slide a window with multiple anchor boxes over the feature map and evaluate each box by the classifier and the regressor. The proposals below the threshold are rejected, hence, feeding only the promising one to the next steps.

More than that, this model optimized in 4 steps of alternative training by fine-tuning the layers unique to RPN and Fast R-CNN separately while fixing the shared layers. This allows the model to share weights forming a unified network and brought higher performance both in efficiency and accuracy. To compare the performance, The time for proposals of Faster R-CNN was 10 milliseconds per image (5 frames per second for the whole process), while 2 seconds for Selective Search using CPU.

Take away

From R-CNN to Faster R-CNN, the networks got transformed removing dependence on other components. R-CNN could be enhanced by dropping linear SVMs and sharing convolution computations. And Fast R-CNN also changed to share the convolution to remove Selective Search. By integrating the whole process into one network, we could achieve higher accuracy as well as faster speed.

References

• J.R.R. Uijlings et al, Selective Search for Object Recognition, 2012
• Ross Girshick et al, Rich feature hierarchies for accurate object detection and semantic segmentation, 2014
• Kaiming He et al, Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition, 2015
• Ross Girshick, Fast R-CNN, 2015
• Shaoqing Ren et al, Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, 2015
• Kaiming He et al. Mask R-CNN, 2018

Did this story resonate with you? Please share your insight with us. I’m always open to talk, so feel free to leave comments below and share your thoughts. I also share interesting and useful resources on LinkedIn so feel free to follow and reach out to me. I’ll be back with another interesting story, next time. As always, stay tuned!

Deep Dive into the Computer Vision World: Part 2 was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

After finishing the basics of neural networks, the next step would be learning some of the “famous rock star models” in this field. ResNet, Inception Net, Faster R-CNN, YOLO and so on. Studying these models can be divided into three parts: The applications, the implementations, and intuition behind these architectures. There are already plenty of resources on how to use pre-trained models and how to build them. But grasping the real intuition behind the model is overlooked sometimes. What’s the researchers’ intention for building a model with such structures? What motivated them to take such an approach? And what can we infer from the outcome?

This article is the next series to “The Most Intuitive and Easiest Guide” tutorials and the complete set of the series is as follows:

1. Starting from VGG, ResNet, Inception Network and MobileNet
2. Regions with CNN, let’s start object detection!
3. YOLO, SSD and RetinaNet, more unified ones
4. From object detection to instance segmentation (TBU)

This article assumes that you’re already familiar with the basic concept of convolutional neural network. On top of that, we’re going to dive deeper and see various convolutional transformation and modification with VGG, ResNet, Inception Network and MobileNet. Walking through them one by one, we’ll answer the questions mentioned above so that we truly understand the underlying meaning of these architectures.

VGG

ImageNet is a large dataset used for the research in the computer vision field. Since 2010, a competition has been held annually under the name of ImageNet Large Scale Visual Recognition Challenge (ILSVRC). The VGGNet was the winner of the ILSVRC competition in 2014. Although it has a simple architecture compared to the other fancy networks, it is still one of the most popular networks.

One interesting part of this network is that it used only 3x3 filters even in the first layers. If we compare it to the previous ones, AlexNet, the winner in 2012, used 11x11 and 5x5 filters at the first two layers. And ZFnet, the winner in 2013, used 7x7 filters. Instead of using a relatively large size of a filter in the first convolution layer, the size of the filters VGG network used is 3x3 only.

So what’s the intention for trying this? The answer is to reduce the number of parameters. Let’s do some calculation here. How many parameters does a single 7x7 convolution layer with the channel size C require? It’s 49C². Now, what’s the number with 3 layers of 3x3 convolution layers with the same channel size? (9C²)x3 = 27C². See how many parameters decreases. Reducing the number of weights can handle overfitting by reducing the complexity of the network. Moreover, as we can have an activation layer for each convolution layer, this encourages the model’s learning as well.

Along with the VGG networks or AlexNet, most of the Convolutional networks followed the basic structure: a series of convolutional layers, a pooling layer and an activation layer with some of the normalization. But going deeper and deeper, people came across some serious problems and started to redesign this approach.

ResNet

Intuitively we’d like to expect “deeper is better.” However, researchers found was it wasn’t the case. As a network gets extremely deeper and deeper, the performance gets worse. One of the reasons is a vanishing/ exploding gradient problem. Although we already have several ways to address this issue such as gradient clipping, there is a degradation problem that isn’t due to overfitting. The birth of ResNet started from here. Why is that?

Let’s assume that we have a good network with desirable accuracy. Now we’re going to copy this network and add just 1 layer but having no additional function. This is called an identity mapping, which means getting the output exactly the same with the input. Compared to other complex layers with many parameters, this will be a piece of cake to our “smart” model. So there should be no harm to the performance of the network, right?

Then what will be the outcome if we attach more and more identity layers? As identity layers give no impact to the output, there should be no degradation in the performance. It seems a reasonable guess, but the outcome of this experiment was different from our expectations. The performance of this model was worse than the shallower model without identity layers.

This outcome indicates “deeper is better” is not applicable to neural networks even with the identity mapping. Just attaching identity layers directly to a network makes it hard to train. Okay, then what about putting identity layers as an additional layer?

Now, this is where ResNet comes into play. Let’s say H(x) as the outcome of the network. And if we put an additional path, the original function becomes F(x) + x where F(x) is the function of the main branch as shown above. The identity layer is also called a shortcut connection or a skip connection. This brings the concept of “residual learning” to the network. Now the main branch is trained to approximate H(x) — x and make F(x) go closer to zero. This means giving the layers a certain direction for learning, which makes the training easier. Moreover, the network can skip some of the layers through the identity layer. This allows us to have deeper layers but still with high performance.

You may ask how it’s possible to add F(x) and x when the two layers have different sizes. We can handle this problem by the zero padding or a linear projection at the shortcut connections. So there are two kinds of shortcut connections in ResNet, one without weights and the other with weights.

The picture above shows the convolutional block of ResNet 50-layers. It has a main branch with 3 convolution layers and a shortcut connection with a 1x1 convolution. Here we may need to ask, why 1x1–3x3–1x1 convolution?

To answer this question, it is necessary to understand how the dimension of data changes according to filter size. Please have a look at the picture on the right. If we say the shape of the input is (32, 32, 256), the dimension decreases when it passes the 3x3 filter compared to the previous step. And when it passes the second 1x1 filter, it increases again. This is making a bottleneck. By forcing the data to fit into a smaller dimension, we can get a more efficient feature representation.

Optimizing the deep neural network easily with much fewer filters, much less computation, and higher accuracy. This simple yet novel reformulation brought such a beautiful achievement is how ResNet blew people’s minds in 2015. With this fundamental intuition on ResNet, I’d like you to explore the network from the original paper. And here is the second part of the research on experimenting the variants of the residual block. Or you can find an easy explanation on ResNet here.

GoogLeNet

Going deeper with high performance is good. We need deep neural networks to implement complex and challenging tasks. But there is another factor to consider when we evaluate the networks. Think about when we have to embed a model in a mobile application. It’ll be not desirable to have a “too heavy” model for sure. But more than that, there is a computational cost problem that is quite critical in real practice. So now the question becomes, how can we go deeper with higher efficiency?

One place where the researchers had found inefficient computation occurred was a fully connected architecture. In a convolutional approach, a fully connected architecture indicates that a layer l feeds from a layer l-1 and is connected to the next layer l+1 as shown below on the left. The researchers found any uniform increase in the number of the filters doubles the computation. So they suggest moving from densely connected architectures to sparsely connected architectures.

Instead of implementing one type of convolutions, the sparse architecture does multiple convolutions with multiple filter sizes as shown above. And the dimension of output from these layers can be matched up with ‘SAME’ padding.

There is an important meaning in having several filters with different sizes here. During the training, the scale of features a network detects are small and local at first, and it gets bigger as it goes deeper. For example, let’s say the model detects a cat. The model sees a small scale of features such as the fur pattern of the cat, and then it moves to a bigger scale such as the shape of its ear, and then the number of legs.

So even with the pictures where the cats are at different scales like above, the model still processes the same scale of feature mapping. This is inefficient and it hurts the performance. Therefore having multiple filters indicates now it knows how to “zoom in and out” according to the input image. It enables the model to detect the image in different scales at a time. And by doing so, we can also handle off the work of choosing parameters as well as reducing the inefficient computation. This is the first kick of Inception network.

So the Inception Module looked like the one on the left of the picture above. But there was a problem. But what about the computation cost? Sparely connected architectures still require the same or more amount of computation anyways! The solution of the researchers to handle this problem was implementing 1x1 convolutions as shown above on the right. The purpose of the 1x1 convolutions here is dimension reduction just as what we discussed with ResNet. See the computation times are reduced given the (N, N, C) shape of the input image. If the size of the image is 32 x 32 with the 129 channels, we can reduce the number of parameters 9 times less!

The final architecture of GoogleNet is shown above. The modules are concatenated, forming 22 layers in total. This is quite a deep network and it also couldn’t avoid the notorious vanishing gradient problem. So the way the researchers took to handle this issue was making side branches.

What do we mean by “gradient vanish?” It indicates that the value of the gradient gets smaller and smaller as it goes deeper, which means the gradient has little information to learn. So we can say that the gradient from the early stages carries more information. In other words, we can have additional information by adding helper classifiers in the intermediate layers. Moreover, as we discussed above, the features detected are different along with the layers. Therefore the “intermediate products” are also beneficial for classifying the labels. We add the loss from these branches to the total loss of the network with weighted.

MobileNet

Inspired by the networks we discussed above, “going deeper and more complicated” was the high trend in CNN research. Although those networks brought higher accuracy, they were not efficient or ‘light’ enough for a real-world application. To be carried out in a computationally limited platform with a timely manner, they need to be smaller and faster.

One solution the researchers proposed was forming a factorized convolutions. Factorizing convolutions indicates factorizing a standard convolution into a depthwise convolution and a pointwise convolution. This is called depthwise separable convolution.

In depthwise separable convolution, we first split each channel of the input and the filter, and convolve the input with the corresponding filter, or in a depthwise manner. And then concatenate the output together. This is depthwise convolution. After that, we do pointwise convolution, which is the same as 1x1 convolution. Now see the final outcome. Isn’t it the same with that of a standard 3x3 convolution? If so, what’s the point for doing this?

The answer is computation. Let’s compare the computational cost of the two. When the shape of input is (N, N, C1) and the output channel is C2 with the filter size F, the total computation of the two should be as shown on the left. So as you can see, the computational cost is reduced with depthwise separable convolution. According to the original paper, MobileNet requires 8 to 9 times less computation with only a small reduction in accuracy compared to the same architecture but with standard convolutions.

Depthwise separable convolution is also used to Xception, and its paper was released earlier than that of MobileNet. But I found MobileNet paper gives a more straightforward explanation on the concept of depthwise separable convolution.

Take away

Now do the architectures become simpler and tangible to you? Like I said, understanding the thesis behind the structure is essential. So I’d like to go over some kicks of the networks again.

Firstly, the filter size at the convolution layers has more meanings than just feature mapping. It can change the number of parameters like VGG. We can also create bottlenecks to extract meaningful representations of the data. 1x1 convolution can serve more than a linear transformation. It can serve as a useful trick for giving more non-linearity, changing the dimension, and reducing the computation without impacting the input data significantly.

To get higher performance, ResNet utilized the residuals by shortcut connection. This layer promoted the optimization better by conveying weights across the layers efficiently. On the other hand, the strategy of Inception Network was using multiple filters by building sparse architectures. It was for dealing with the change in scales during training. Lastly, MobileNet chose to factorize a convolution into two steps. With depthwise separable convolutions, MobileNet could reduce computation and the model size dramatically.

References

• Karen Simonyan and Andrew Zisserman, Very deep convolutional network for large-scale image recognition, 2015
• Kaiming He, et al. Deep residual learning for image recognition, 2015
• Christian Szegedy, et al. Going deeper with convolutions, 2014
• Christian Szegedy et al., Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning, 2016
• Franc¸ois Chollet, Xception: Deep Learning with Depthwise Separable Convolutions, 2017
• Andrew G. Howard et al., MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications, 2017

Did this story resonate with you? Please share your insight with us. I’m always open to talk, so feel free to leave comments below and share your thoughts. I also share interesting and useful resources on LinkedIn so feel free to follow and reach out to me. I’ll be back with another interesting story, next time. As always, stay tuned!

Deep Dive into the Computer Vision World: Part 1 was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

A hackathon is a competition where various people, including programmers, developers and graphic designers, collaborate to design software projects intensively in a short period of times. The goal of a hackathon is to create a functional product at the end of the event.

Last month, I had an opportunity to participate in the JunctionX Seoul Hackathon held by Junction Ltd. Our team made an iOS application that brings an optimized shopping service experience for Korean beauty product powered by image search engines. This application can help travelers who want to explore Korean beauty products but not knowing what to try due to an obvious language barrier. Users can find useful data about a product just by scanning it and the information is translated into their local languages. There is also a searching history page which could be utilized for further customized service. You can find more about it here as well.

Rather than talking about what I made, however, I’d like to share what I learned in the process. So today, I’m going to talk about the 4 reasons why you should apply for a place in a Hackathon (especially if you’re an aspiring data scientist), how you should prepare and some of my tips. I hope this article can motivate you to go out of your comfort zone and get some profound inspiration just as I did.

1. Why Hackathons

If you haven’t tried a hackathon or a programming competition before, and say you ask me if it’s worth trying, I’ll nod my head without hesitation. Here are the 4 reasons why you should go to a hackathon.

Beyond “modeling”

We, data science learners, tend to work alone or study alone. We tend to invest most of our time in data preprocessing and wangling. And we pour most of our energy into studying machine learning or deep learning algorithms. Our project usually starts with importing data (or sometimes with building data by ourselves) and ends with the evaluation of the predictions.

Almost every real-world project in the world sadly doesn’t work that way. We don’t work alone. We work with other team members. From database to deployment and product management, all the processes should be done through collaboration. Having toolkits beyond the “modeling” territory can be a great plus for aspirants. (Many articles are pointing out this and you can find one here as well) Now the important question, where can we learn and experience this?

“Unlike competitions, nobody gives you two .csv files called train and test and a nicely written evaluation metric. Almost 80% of the efforts go into defining the problem and getting and processing data. Remaining 20% effort goes into pure modeling and deployment.”

A hackathon can be a great chance to collaborate with others and make a real-world project. Although there are some limitations, still you can learn how to communicate with others (who do not have much of the knowledge in machine learning as you do) and understand the overall workflow for releasing an actual product. And if you get a chance of being on the team with someone who knows a lot better than yourself in machine learning, I believe it’ll be such a great time to push your limits as well.

Business thinking is more crucial than you think

The first step of the project will be deciding what to build. And I can tell half of the winning chance is determined here. You should get off on the right foot. Implementation is also a critical point but the business value plays an essential role. If it’s neglected, your work can be nothing but just showing off how technically competent you are.

“Likewise, data scientists are hired to create business value, not just to build models. Ask yourself: How will the outcome of my work influence company decisions? What do I have to do to maximize this effect? With this entrepreneurial spirit, the third wave data scientist does not only produce actionable insights, but also seeks that they bring about real change.”

You’d have seen many people pointing out the importance of business mindset already. Finding a hidden need of users and observing a problem in the market. Suggesting a solution with your technical knowledge. Developing a service that can become profitable. At a hackathon, you are asked to think and discuss deeply with this viewpoint.

I heard of a guy who had won programming competitions more than 15 times. He retired in his ‘prize hunting’ career after he won one of Google’s competitions. Do you know what his secret was? It was just business sense and a little bit of wit. By making a clever and creative product, he made people shout “Brilliant!” which definitely leaves an impression on panels. So if you’re looking for an additional tip for winning, having business values with some wit can be a great bonus.

Great chance to handle extreme stress

The goal of a hackathon is creating a functional product during a short period (commonly 2 or 3 days). You are asked to present a prototype or demo of your product during these days. It’s all about limited time, limited resources and limited energy problems. Sometimes, you could have situations where things go wrong and you’d have to instantly find solutions. Sometimes you have to prioritize your work, which means what to come first and when to stop. You could feel intense pressure to do all your tasks and not to be a burden on the team. And as time passes, becoming completely famished and exhausted is literally inevitable! (More like energy drain or depletion until finally “falling” asleep).

If you’re someone like me, who tend to freeze at an extremely stressful moment, you’d be overwhelmed even by the air at the competition hall. It was my first time in a Hackathon so I experienced my brain being paralyzed several times. I had to get some fresh air and relax to ease off the tension. But this also became a great chance for me to learn how to dealing with stress. Because it’s unavoidable meeting those unexpected problems in the workplace and we have to manage those demanding moments.

The real-time of ”Hello, world”

Meeting new people is also a great pleasure, especially if they are like-minded people. At a hackathon, you can meet various people from different regions and with different specialties but with the same interest. They can be web developers, app developers or designers but they are people who’re willing to invest their weekend in developing something cool.

I met people who came from Russia to Korea just for this event. I also met a Ukrainian lady studying Computer Science in Korea. There were people from Denmark, French, China, and Japan. Meeting new people itself is a great joy. But more than that, I could hear what kind of pain points they have and what they are really passionate about. My horizon could become broadened in programming by chatting with fellow participants. And this is a reason why I think you should try a Hackathon with generous topics rather than heavily focused on the data science fields.

Also, it’d be good to care about what others do as well. Although you’ll be too busy to take extra care of others, this will help you understand the whole process as I mentioned above. We’re here to learn new things, anyway!

2. How to prepare

So now does a Hackathon sound worth the try? If so, then your next question could be how you should prepare. Actually, nothing. Just go for it. Whatever skillsets you have, don’t hesitate to apply for. Hackathons are not a stage only for professionals. But if you’d like to prepare something before the day, then I’d like to list some of the skills I find useful and applicable.

Backend and Deployment

This is not a project that starts with importing the data and ends with some evaluation plots. You need to collect the data and build a database according to your team project. After you finish with data analysis or modeling part, you have to pass your model to frontend. Or you might need to deploy your work on a web by yourself. Therefore having database and Flask in your toolkits can be a huge asset for this event.

If you’re totally new to database, you can start your learning from understanding the difference between the SQL and NoSQL. Here is a compact SQL course for beginners but there are tons of tutorials on databases besides this. So go and pick according to your taste.

Flask is a microframework written in Python which is easy to learn and very useful for the model deployment. Here is an excellent tutorial on how to deploy a Keras deep learning models with Flask by Ben Weber. You can also find a series of video courses for Flask here.

API and Cloud Service

Conditions could be different for each hackathon, but the one I took part in highly depended on API and Cloud Services. There were several sub tracks and the participants were asked to use particular API to participate in a given track. But besides this condition, you’d need to collect data by yourself anyway. Where will you get the data? And what about the computing power for training? You’d have probably left your “super-computer” with GPUs at home.

Therefore it’d be better to have an experience of getting data from API and handling data in a JSON format. If you’re new to API, this article can explain to you what API is all about.

Ability to use a cloud service like AWS or Google Cloud GPU is also necessary. Here is an official guide for AWS EC2 but here is a friendly video by School of AI. You can also find a more detailed tutorial for beginners by Michael Galarnyk.

Fully charged energy of yourself

Get enough sleep. Take control of your condition. This is not a joke. During the competition, I slept for 4 hours in total. Sleeping less can be a good strategy but you need to be in good condition when you enter the hall. Also, a neck pillow and a blanket for a catnap can be as much useful item as “Flask” or “AWS.” 😂😂

3. My small last tips

Lastly, here are some of my small tips.

How to form your team

If you’re a solo applicant, you have to build your team from the bottom. There can be a social network platform like Slack where you can get the information from other applicants and make a team before the event. You might get a chance to join a premade team with one opening.

If you have an idea on what you want to develop, divide it into parts and look for crew members who can take on each part. You need to find someone who has the skill different from us (like data modeling). The common list of members you’d need to have is a backend engineer, a web/app developer, and a UI/UX designer.

Be honest with what you can do

You might think your ability is not enough for doing this. So you might want to hide your level or pretend knowing more than you do. But you really don’t have to. Hackathons are open to everyone. And no matter how much you know about programming, there is a portion that can be filled only by you! Is the person next to you know much more than you do? Awesome! You get more chances to learn. Asking for help is a better way to “help” your team than struggling alone and wasting precious time.

Enjoy, Enjoy and Enjoy!

It’s more than winning. Although you should do your best during the time, don’t lose a smile on your face. Enjoy the time for learning, meeting new people and working with them. Enjoy the time for being creative and developing a new thing that doesn’t exist in the world. Enjoy the time for being stressed and going beyond your limits. It will be extremely demanding. At the end of the event, however, you’ll go back to your home with new thoughts and lessons on your mind even if you don’t win a prize. And you’d find yourself looking forward to the next Hackathon like me. 😉

Did this story resonate with you? Please share your insight with us. I’m always open to talk, so feel free to leave comments below and share your thoughts. I also share interesting and useful resources on LinkedIn so feel free to follow and reach out to me. I’ll be back with another interesting story, next time. As always, stay tuned!

Hackathon! Get Ready with Me 🤓😎 was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

There are lots of ways out there one can adapt to make learning progress efficiently. As for me, combining studies with a little bit of fun is the best strategy. In this series of tutorials, many images have been used to demonstrate image processing concepts. It might interest you to note that people enjoyed the parts of the series where I applied the concept to the images in a funny way. And I recommend you also merge fun with studying.

This is the last part of OpenCV tutorial for beginners and the complete set of the series is like follows:

1. Understanding color models and drawing figures on images
2. The basics of image processing with filtering and gradients
3. From feature detection to face detection
4. Contour detection and having a little bit of fun

From the first series, we saw how to draw a figure on an image. We also talked about how to apply blurring and thresholding with various options. And in part 3, we discussed several detection algorithms including edge detection. There is another detection technique called contour detection which is essential in object detection. And after that, we’re going to round up this series with a little bit of fun playing with masking.

This article assumes you’ve been following the previous steps or you already know those concepts. But if it’s not the case, please check the previous parts of this series. The complete code for this tutorial is available on Github as always.

Contour detection

You may be already familiar with the word ‘contour.’ I’ve used this term several times in previous posts. A contour line indicates a curved line representing the boundary of the same values or the same intensities. A contour map is the most straightforward example we can think of.

But then you may ask this. What’s the difference between edges and contours? The two terms are often used interchangeably so it could be a bit confusing. To put it simply, the concept of edges lies in a local range while the concept of contours is at the overall boundary of a figure. Edges are points whose values change significantly compared to their neighboring points. Contours, on the other hand, are closed curves which are obtained from edges and depicting a boundary of figures. You can find a further explanation here.

# Load the image
img = cv2.imread('images/pine_apple.jpg')
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
plt.imshow(img)

So what we’re going to do is detecting the contour of this pineapple. Before applying the detection algorithm, we need to convert the image into grayscale and apply thresholding as follows. All of these steps are what we’ve discussed in the previous series.

# Blurring for removing the noise 
img_blur = cv2.bilateralFilter(img, d = 7, 
                               sigmaSpace = 75, sigmaColor =75)

# Convert to grayscale 
img_gray = cv2.cvtColor(img_blur, cv2.COLOR_RGB2GRAY)

# Apply the thresholding
a = img_gray.max()  
_, thresh = cv2.threshold(img_gray, a/2+60, a,cv2.THRESH_BINARY_INV)
plt.imshow(thresh, cmap = 'gray')

Contour detection can be implemented by the functioncv2.findContours() in OpenCV and there are two important parameters here. mode is the way of finding contours, and method is the approximation method for the detection. I ask you to find other information from the documentation.

# Find the contour of the figure 
image, contours, hierarchy = cv2.findContours(
                                   image = thresh, 
                                   mode = cv2.RETR_TREE, 
                                   method = cv2.CHAIN_APPROX_SIMPLE)

The mode cv2.RETR_TREE finds all the promising contour lines and reconstructs a full hierarchy of nested contours. The method cv2.CHAIN_APPROX_SIMPLE returns only the endpoints that are necessary for drawing the contour line. And as you can see above, this function gives the image, the detected contours and their hierarchy as its output.

The returned contour is a list of points consisting of the contour lines. To draw the outer line of the figure, we’ll sort the contours by their area. With the selected contour line, cv2.drawContours() will depict the bounding line along with the points as shown below.

# Sort the contours 
contours = sorted(contours, key = cv2.contourArea, reverse = True)

# Draw the contour 
img_copy = img.copy()
final = cv2.drawContours(img_copy, contours, contourIdx = -1, 
                         color = (255, 0, 0), thickness = 2)
plt.imshow(img_copy)

It’s simple, right? Although the shadow is also added at the bottom, the outcome is pretty satisfactory.

More about contours

There are more things we can do with the contour. We can find the centroid of an image or calculate the area of a boundary field with the help of the notion called image moment. What does a moment mean here? The word ‘moment’ is a short period of time in common usage. But in physics terminology, a moment is the product of the distance and another physical quantity meaning how a physical quantity is distributed or located. So in computer vision, Image moment is how image pixel intensities are distributed according to their location. It’s a weighted average of image pixel intensities and we can get the centroid or spatial information from the image moment.

There are three types of moments- spatial moments, central moments, and central normalized moments. We can get the image moment with the function cv2.moments() in OpenCV and it returns 24 different moments. If you print the output M as shown below, it’ll return the 24 moments in a dictionary format.

# The first order of the contours
c_0 = contours[0]

# image moment
M = cv2.moments(c_0)
print(M.keys())

I’d like to focus on the implementation of the image moments here. Additional reading resources can be found at the end of this article if you may have an interest.

To get the area of the contours, we can implement the function cv2.contourArea() . Why don’t we try several contours here? If you input the first, second and third contours, you’ll get the decreasing values as shown below. This shows that the contour detection algorithm forms the hierarchy of the detected boundaries.

# The area of contours 
print("1st Contour Area : ", cv2.contourArea(contours[0])) # 37544.5
print("2nd Contour Area : ", cv2.contourArea(contours[1])) # 75.0
print("3rd Contour Area : ", cv2.contourArea(contours[2])) # 54.0

The arc length of the contour can be obtained by the function cv2.arcLength() . The parameter closed indicates whether the curve should be closed or not.

# The arc length of contours 
print(cv2.arcLength(contours[0], closed = True))      # 2473.3190
print(cv2.arcLength(contours[0], closed = False))     # 2472.3190

Now let’s try plotting the centroid and the extreme points of our pineapple. We can get the centroid point with the formula as follows.

# The centroid point
cx = int(M['m10'] / M['m00'])
cy = int(M['m01'] / M['m00'])

The extrema are the endpoints on the left and right, at the top and bottom. And we can arrange x and y coordinates separately as follows.

# The extreme points
l_m = tuple(c_0[c_0[:, :, 0].argmin()][0])
r_m = tuple(c_0[c_0[:, :, 0].argmax()][0])
t_m = tuple(c_0[c_0[:, :, 1].argmin()][0])
b_m = tuple(c_0[c_0[:, :, 1].argmax()][0])

pst = [l_m, r_m, t_m, b_m]
xcor = [p[0] for p in pst]
ycor = [p[1] for p in pst]

Now let’s plot all these points on the image.

# Plot the points
plt.figure(figsize = (10, 16))
plt.subplot(1, 2, 1)
plt.imshow(image, cmap = 'gray')
plt.scatter([cx], [cy], c = 'b', s = 50)

plt.subplot(1, 2, 2)
plt.imshow(image, cmap = 'gray')
plt.scatter(xcor, ycor, c = 'b', s = 50)

Other shapes of contours

Besides the compact contour, we can also draw a convex contour or rectangular contour line of a figure. Let’s try a straight rectangular shape first. With the outer contour line, we’ll draw the rectangle around the object. The function cv2.boundingRect() returns the 4 points of the bounding box as shown below.

# The first order of the contours
c_0 = contours[0]

# Get the 4 points of the bounding rectangle
x, y, w, h = cv2.boundingRect(c_0)

# Draw a straight rectangle with the points
img_copy = img.copy()
img_box = cv2.rectangle(img_copy, (x, y), (x+w, y+h), color = (255, 0, 0), thickness = 2)

Note that this straight rectangle isn’t the minimum among other possible boundings. We can extract the rectangle with the minimum area with the function cv2.minAreaRect() which finds a rotated rectangle enclosing the input 2D point set. After that, we get the 4 corners of this rectangle and put them at the contour parameter as shown below.

# Get the 4 points of the bounding rectangle with the minimum area
rect = cv2.minAreaRect(c_0)
box = cv2.boxPoints(rect)
box = box.astype('int')

# Draw a contour with the points 
img_copy = img.copy()
img_box_2 = cv2.drawContours(img_copy, contours = [box], 
                             contourIdx = -1, 
                             color = (255, 0, 0), thickness = 2)

Now let’s check the result and compare the two different contour boxes.

plt.figure(figsize = (10, 16))
plt.subplot(1, 2, 1); plt.imshow(img_box)
plt.subplot(1, 2, 2); plt.imshow(img_box_2)

We can also draw a convex shape contour with the function cv2.convexHull(). This takes a set of points and returns the convex hull from the given set. And by inputting this returned points as contours in cv2.drawContours() , we can get the convex contour as follows.

# Detect the convex contour
hull = cv2.convexHull(c_0)

img_copy = img.copy()
img_hull = cv2.drawContours(img_copy, contours = [hull], 
                            contourIdx = 0, 
                            color = (255, 0, 0), thickness = 2)
plt.imshow(img_hull)

There are so many things to cover in contour detections. I encourage you to check the documentation and explore more options by yourself.

A little bit of fun with masking

For the last exercise of this series, I’d like to have some fun time with doing image masking. We can add two different images with arithmetic operations such as image addition or bitwise operation. So our final task is attaching our Mr.pineapple on a man’s left shoulder.

# Import the large image
backpacker = cv2.imread('images/backpacker.jpg')
backpacker = cv2.cvtColor(backpacker, cv2.COLOR_BGR2RGB)
plt.imshow(backpacker)

We’ll cut the region of interest (the shoulder part) from the large image and the small image as shown below. Note that the size of the two images (the height and the width of the image) should be the same here.

# Crop the small image and the roi
roi = backpacker[750:1150, 300:500]
img_2 = img[40:440, 80:280]

plt.figure(figsize = (6, 6))
plt.subplot(1, 3, 1); plt.imshow(roi)
plt.subplot(1, 3, 3); plt.imshow(img_2)

The next step is making the mask for each image. We’re going to implement bitwise operation with the masks which should be a binary image. By thresholding the small image img_2 , we create the mask. And then with cv2.bitwise_not(), we make another mask which is exactly the opposite of the first one.

# Creating the mask for the roi and small image
img_gray = cv2.cvtColor(img_2, cv2.COLOR_RGB2GRAY)
_, mask = cv2.threshold(img_gray, 254/2+100, 255, cv2.THRESH_BINARY)
mask_inv = cv2.bitwise_not(mask)

plt.figure(figsize = (6, 6))
plt.subplot(1, 3, 1); plt.imshow(mask, cmap = 'gray')
plt.subplot(1, 3, 3); plt.imshow(mask_inv, cmap = 'gray')

If we implement the function cv2.bitwise_and() and the masks, they will pass only the white area of the image. Therefore if we apply the first mask to the roi image, we can set the background image from it. In the same way, if we apply the second one to the img_2 image, we can make the foreground image with the fruit.

# Masking 
img_bg = cv2.bitwise_and(roi, roi, mask = mask)
img_fg = cv2.bitwise_and(img_2, img_2, mask = mask_inv)
dst = cv2.add(img_fg, img_bg)

plt.figure(figsize = (10, 6))
plt.subplot(1, 3, 1); plt.imshow(img_bg)
plt.subplot(1, 3, 2); plt.imshow(img_fg)
plt.subplot(1, 3, 3); plt.imshow(dst)

The outcome is a bit shambles considering the shadow at the bottom but let’s keep it. If you’d like to learn how the bitwise operation works, a detailed explanation can be found here. Now we’re ready to attach this combined image to the original one. So we can simply do this by putting the dst image at the roi position as shown below.

# Final output
backpacker[750:1150, 300:500] = dst
display(backpacker)

😄👋 Have a nice time with the backpacker man!

Summary

From part 1 to part 4, we’ve learned quite a lot about the basics of image processing and computer vision. Did you enjoy all these tutorials so far? I hope you now get confident with the skills while following this series.

If you’re a complete starter in image processing, it is normal being unfamiliar with the terms and the usage of the OpenCV functions at first. The best way to practice is by implementing all of these by yourself. Pick any kind of images you like and just have fun with them. Changing color modes, rotating and resizing the image, masking and pasting it on a different image. Or you could also apply the detection technique of edges, corners and contours. While having fun by combining several preprocessing and trying different parameters, you can get used to OpenCV library faster without realizing.

Resources

• Image moments on Wikipedia
• Image moments course on Udacity
• Shape Matching using Hu Moments (C++/Python)
• Find the Center of a Blob (Centroid) using OpenCV (C++/Python)

Are there errors you would love to correct? Please share your insight with us. I’m always open to talk, so feel free to leave comments below and share your thoughts. I also share interesting and useful resources on LinkedIn so feel free to follow and reach out to me. I’ll be back with another interesting story, next time. As always, please stay tuned! 😎

Computer Vision for Beginners: Part 4 was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Detection task is one of the major tasks for computer vision, and there are so many ways we can utilize this technique. Identifying an error that can be critical when it occurs but can’t be recognizable by human’s eyes. Detecting an unsafe and risky motion or moment to save a life. Perceiving spatial information for a self-driving car. The examples of Object detection usage are countless and making those tasks automatic will bring us safety as well as efficiency.

This is the third part of OpenCV tutorial for beginners and the complete set of the series is as follows:

1. Understanding color models and drawing figures on images
2. The basics of image processing with filtering
3. From feature detection to face detection
4. Contour detection and having a little bit of fun

We’ve discussed several color modes and how to draw figures on images such as a rectangle and a circle. And then in part 2, we talked about image processing concepts such as blurring, gradient, erosion, dilation and so on. Today we are going to apply those concepts to detect features of images and face detection at the end.

This article assumes you’ve been following the previous steps or you already know those concepts. But if it’s not the case, please check the previous parts of this series. The complete code for this tutorial is available on Github. Now let’s get it started!

Edge Detection

Edge detection means identifying points in an image where the brightness changes sharply or discontinuously. We can draw line segments with those points, which are called edges. Actually, we already learned one of the edge detection techniques last time. Do you remember? Gradient filtering with Sobel and Laplacian operation. By calculating the derivatives of pixel values in a given direction, gradient filtering can depict the edges of images.

Canny detection is another type of edge detection techniques. It’s one of the most popular algorithms for detecting edges, which is performed in four steps: Noise reduction, Finding gradient and its direction, Non-maximum suppression and hysteresis thresholding.

The algorithm starts with Gaussian blurring and I guess we already know the reason for removing noises in images. And then it finds the intensity gradient of the image with a Sobel kernel. With the derived gradient and direction, every pixel is checked if a certain point is a local maximum in its surrounding points. If it’s not, this point is suppressed to zero (total absence, black). This is called non-maximum suppression.

If the point is considered to be a local maximum, it goes to the next stage. The final stage is the last decision stage whether the edges at the previous steps are really edges or not. This is called Hysteresis Thresholding and we need two threshold values here.

Given the two different threshold values, we get three ranges of values. So if the intensity gradient of a point is higher than the upper threshold, it will be considered as ‘sure-edge.’ If the gradient of a point is lower than the lower threshold, the point will be discarded. And in case of the gradient being in the middle of the two thresholds, we see its connectivity to other ‘sure-edge’ points. If there’s no connection, it will be discarded as well.

img = cv2.imread('images/giraffe.jpg')
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

# Canny detection without blurring
edges = cv2.Canny(image=img, threshold1=127, threshold2=127)

plt.figure(figsize = (20, 20))
plt.subplot(1, 2, 1); plt.imshow(img)
plt.axis('off')
plt.subplot(1, 2, 2); plt.imshow(edges)
plt.axis('off')

I just used the median value for the two thresholds without blurring and the result isn’t quite desirable. Now let’s try different threshold values this time.

# Set the lower and upper threshold
med_val = np.median(img)

lower = int(max(0, .7*med_val))
upper = int(min(255, 1.3*med_val))

To see how blurring can change the outcome, I’m going to apply two different sizes of kernels, (5x5) and (9x9). And I’m going to try changing the upper threshold values by adding 100. So we have 4 types of processed images as follows:

# Blurring with ksize = 5 
img_k5 = cv2.blur(img, ksize = (5, 5))

# Canny detection with different thresholds
edges_k5 = cv2.Canny(img_k5, threshold1 = lower, threshold2 = upper)
edges_k5_2 = cv2.Canny(img_k5, lower, upper+100)

# Blurring with ksize = 9 
img_k9 = cv2.blur(img, ksize = (9, 9))

# Canny detection with different thresholds
edges_k9 = cv2.Canny(img_k9, lower, upper)
edges_k9_2 = cv2.Canny(img_k9, lower, upper+100)

# Plot the images
images = [edges_k5, edges_k5_2, edges_k9, edges_k9_2]
plt.figure(figsize = (20, 15))
for i in range(4):
    plt.subplot(2, 2, i+1)
    plt.imshow(images[i])
    plt.axis('off')
plt.show()

As you can see above, blurring helps to remove noises and we got the better result with (9x9) size of the kernel. Also, we got a better result with the higher upper threshold value.

Corner Detection

Corner detection is another detection algorithm which is widely used in object detection, motion detection, video tracking and so on. What is a corner in image processing? How can we define a corner with pixels? We see a corner as a junction where edges intersect. Then how can we find them? Finding all edges first and then locating the points where they cross each other? Actually we have another way of making things more efficient, which are Harris corner detection and Shi & Tomasi corner detection.

These algorithms work as follows. We detect points where there’s a considerable change in their intensity values in all directions. And then we construct a matrix to extract eigenvalues from it. These eigenvalues are for the sake of scoring points to decide if it’s a corner or not. The mathematical expressions are shown below.

Now let’s see how we can implement these with codes. We first need to convert the image into grayscale. Harris corner detection can be performed with a function cv2.cornerHarris() in OpenCV.

img = cv2.imread('images/desk.jpg')
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
img_gray = cv2.cvtColor(img, cv2.COLOR_RGB2GRAY)

# Apply Harris corner detection
dst = cv2.cornerHarris(img_gray, blockSize = 2, ksize = 3, k = .04)

The parameter blockSize is the size of the window to consider as neighborhood and k is Harris detector free parameter which is shown in the equation above. The result is the score R and we’ll use this to detect corners.

# Spot the detected corners
img_2 = img.copy()
img_2[dst>0.01*dst.max()]=[255,0,0]

# Plot the image
plt.figure(figsize = (20, 20))
plt.subplot(1, 2, 1); plt.imshow(img)
plt.axis('off')
plt.subplot(1, 2, 2); plt.imshow(img_2)
plt.axis('off')

Let’s try Shi-Tomasi corner detection this time. We can use this with a function cv2.goodFeaturesToTrack(). We set the maximum number of corners sorted by the most likelihood (maxCorners). We also assign minimum distance (minDistance) and minimum quality level (qualityLevel) that is required to be considered as corners. After we get the detected corners, we’ll mark those points with circles as follows.

# Apply Shi-Tomasi corner detection
corners = cv2.goodFeaturesToTrack(img_gray, maxCorners = 50, 
                                  qualityLevel = 0.01, 
                                  minDistance = 10)
corners = np.int0(corners)

# Spot the detected corners
img_2 = img.copy()
for i in corners:
    x,y = i.ravel()
    cv2.circle(img_2, center = (x, y), 
               radius = 5, color = 255, thickness = -1)

# Plot the image
plt.figure(figsize = (20, 20))
plt.subplot(1, 2, 1); plt.imshow(img)
plt.axis('off')
plt.subplot(1, 2, 2); plt.imshow(img_2)
plt.axis('off')

Face Detection

The last feature we’re going to see is a face. Face detection is a technology identifying the presence and the position of human faces in digital images. I want you to differentiate face detection from face recognition which indicates detecting the identification of a person by his or her face. So face detection can’t tell us to whom the detected face belongs.

Face detection is basically a classification task so it’s trained to classify whether there is a target object or not. And Haar Feature-based Cascade Classifier is one of the face detection models available in OpenCV. This is a pre-trained model, which means it already completed training with thousands of images. The 4 key points for understanding this algorithm are Haar features extraction, integral image, Adaboost and cascade classifiers.

Haar-like features are image filters or image kernels used in object detection and the examples are shown above. They owe their name to their intuitive similarity with Haar wavelets which is originally proposed by Alfréd Haar. During detection, we pass the window on an image and do the convolutional operation with the filters to see if there’s the feature we’re looking for is in the image. Here is the video which visualizes how the detection works.

So how we decide whether there is a wanted feature or not in a given area? Let’s take a look at the picture above. We have a kernel whose upper half is dark and lower half is light. Then we get the mean of the pixel values for each region and subtract the gap between the two. If the result is higher than a threshold, say 0.5, then we conclude there’s the feature we’re detecting. We repeat this process for each kernel while sliding the window over the image.

Although this isn’t a complex calculation, the total amount of computation becomes huge when we consider it in a whole image. If you’ve seen the video mentioned above, you’d get the intuition into the amount of calculation involved. And this is where an integral image comes into play. The integral image is a way of image representation which is derived to make the feature evaluation faster and more effective.

As you can see below, there are the pixels of an image on the left and an integral image on the right. Starting from the left upper point, it computes the accumulated sum of pixels under a given rectangular area. On the integral image, the sum of the pixels in the box with the dashed line is written at the lower right point of the box on the right.

With this pre-calculated table, we can simply get the summed value for a certain area by the values of sub-rectangles (the red, orange, blue and purple box).

So we solved the computational cost with the integral image. But we’re not done yet. Think about when the detecting window is at the blank background where there’s no object or face. It’ll still be a waste of time if it performs the same process in such a part. And there’s another kick making this detector faster. Implementing a cascade classifier with Adaboost!

A cascade classifier constructs stepwise stages and gives an order among Haar-like features. Basic forms of the features are implemented at the early stages and the more complex ones are applied only for those promising regions. And at each stage, the Adaboost model will be trained by ensembling weak learners. If a subpart, or a sub-window, is classified as ‘not a face-like region’ at the prior stage, it’s rejected to the next step. By doing so, we can only consider the survived ones and achieve much higher speed.

Where’s our hero?

To make this tutorial more interesting, I’d like to implement this Cascade classifier to detect the face of our marvel heroes. (Machine learning should be fun, don’t you think? 😃😃) Let’s bring Captin Marvel as our first guest.

We’re going to use only the part of this image. So let’s get the region of interest around her face first and then convert the image into grayscale. The reason for using only one channel is because we’re only interested in the changes in the light intensity of the features.

cap_mavl = cv2.imread('images/captin_marvel.jpg')

# Find the region of interest
roi = cap_mavl[50:350, 200:550]
roi = cv2.cvtColor(roi, cv2.COLOR_BGR2GRAY)
plt.imshow(roi, cmap = 'gray')

Haar Cascade classifier files basically come along with OpenCV. You can probably find them in the OpenCV folder on your computer. Or you can simply download the file here.

# Load Cascade filter 
face_cascade = cv2.CascadeClassifier('haarcascades/haarcascade_frontalface_default.xml')

Next, we’re going to create a function detecting a face and drawing a rectangle around it. To detect the face, we can use the method .detectMulitiScale() of the classifier face_cascade that we loaded above. It returns the four points of the identified region so we’ll draw a rectangle at that position. scaleFactor is a parameter for how much the image size is reduced at each image scale and minNeighbors for how many neighbors each candidate rectangle should be trained. Now let’s apply this function to the image and see the result.

# Create the face detecting function 
def detect_face(img):
    
    img_2 = img.copy()
    face_rects = face_cascade.detectMultiScale(img_copy, 
                                               scaleFactor = 1.1,
                                               minNeighbors = 3)
    
    for (x, y, w, h) in face_rects:
        cv2.rectangle(img_2, (x, y), (x+w, y+h), (255, 255, 255), 3)
        
    return img_2

# Detect the face
roi_detected = detect_face(roi)
plt.imshow(roi_detected, cmap = 'gray')
plt.axis('off')

Great! I think this is quite satisfactory. Why don’t we call other heroes altogether this time? We can implement this classifier to an image with multiple faces as well.

# Load the image file and convert the color mode
avengers = cv2.imread('images/avengers.jpg')
avengers = cv2.cvtColor(avengers, cv2.COLOR_BGR2GRAY)

# Detect the face and plot the result
detected_avengers = detect_face(avengers)
display(detected_avengers, cmap = 'gray')

And yes, it sometimes makes a failure catching a ‘non-face’ object or missing a ‘real-face.’ It’s funny that it detected Spider-Man successfully while mistook the hands of Captin America and Black Widow as eyes. We usually get better results with a face looking in front and showing its forehead and eyes clearly.

Want to try yours?

Would you like to try yours with a Webcam? We can apply the same process. Save the following code scrip as a file and import it on your terminal. If you’re to use Jupyter notebook, put the codes in one cell and execute. You can shut down the window by pressing ESC.

Did it successfully detect your face? Hope it doesn’t select your nostril for eyes. 😅

What’s next?

Did you enjoy the story? Besides the front face classifier, there are various models for eyes, upper body, lower body and so on. You can also try the model detecting a Russian car number plate. So I’d recommend you to play with it as well. Would you like to test your understanding on image processing? Try this 25 Questions of image processing. And here are such great articles providing detailed explanations for the questions you should check.

• My take on 25 Questions to test a Data Scientist on Image Processing with Interactive Code- Part 1
• My take on 25 Questions to test a Data Scientist on Image Processing with Interactive Code- Part 2

Are there errors you would love to correct? Please share your insight with us. I’m always open to talk, so feel free to leave comments below and share your thoughts. I also share interesting and useful resources on LinkedIn so feel free to follow and reach out to me. I’ll be back with another interesting Computer Vision story. Please stay tuned!

Computer Vision for Beginners: Part 3 was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Preparation refers to the process of getting ready for a task before actually starting it. It could be a preparing step of writing a story, job interview or data modeling. And having a thorough preparation can’t be overemphasized of its importance. If we get off without enough preprocessing, it’s hard to expect to get a desirable result no matter how good data we have.

This is the second part of OpenCV tutorial for beginners and the complete set of the series is as follows:

1. Understanding color models and drawing figures on images
2. The basics of image processing with filtering
3. From feature detection to face detection
4. Contour detection and having a little bit of fun

Today we’re going to talk about how to manipulate images. These are going to be a preprocessing stage. When it comes to detecting edges and contours, noise gives a great impact on the accuracy of detection. Therefore removing noises and controlling the intensity of the pixel values can help the model to focus on the general details and get higher accuracy. Blurring, thresholding, and morphological transformation are the techniques we use for this purpose.

This post assumes you are already familiar with the concept of convolution. But if it’s not the case, I’d like to recommend you check this post first. The complete code for this tutorial is also available on Github. Now let’s walk through how to apply image filtering with OpenCV one by one.

Blurring

The goal of blurring is to perform noise reduction. But we have to pay extra care here. If we apply edge detection algorithms to the images with high resolution, we’ll get too many detected outcomes that we aren’t interested in.

On the contrary, if we blur the images too much, we‘ll lose the data. Therefore we need to find an adequate amount of blurring we’re going to apply without losing desirable edges.

There are several techniques used to achieve blurring effects but we’re going to talk about the four major ones used in OpenCV: Averaging blurring, Gaussian blurring, median blurring and bilateral filtering. All four techniques have a common basic principle, which is applying convolutional operations to the image with a filter (kernel). The values of the applying filters are different between the four blurring methods.

Average blurring is taking the average of all the pixel values under the given kernel area and replace the value at the center. For example, suppose we have a kernel with the size of 5X5. We calculate the average of the convoluted outcome and put that result to the center of the given area.

Then what will it be like if we increase the size of the kernel? As the size of filters gets bigger, the pixel values will be normalized more. Therefore we can expect the image to get blurred the more. Let’s check out the result with the code as follows. (For comparison, I’ll keep attaching the original image to the result)

# Import the image and convert to RGB 
img = cv2.imread('text.jpg')
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

# Plot the image with different kernel sizes
kernels = [5, 11, 17]

fig, axs = plt.subplots(nrows = 1, ncols = 3, figsize = (20, 20))
for ind, s in enumerate(kernels):
    img_blurred = cv2.blur(img, ksize = (s, s))
    ax = axs[ind]
    ax.imshow(img_blurred)
    ax.axis('off')
plt.show()

Medium blurring is the same with average blurring except that it uses the median value instead of the average. Therefore when we have to handle sudden noises in the image such as ‘salt and pepper noise,’ it’ll be better to use medium blurring than average blurring.

Gaussian blurring is nothing but using the kernel whose values have a Gaussian distribution. The values are generated by a Gaussian function so it requires a sigma value for its parameter. As you can see the image above, the values of the kernel go higher near the center and go smaller near the corner. It’s good to apply this method to the noises that have a normal distribution such as white noise.

Bilateral Filtering is an advanced version of Gaussian blurring. Blurring produces not only dissolving noises but also smoothing edges. And bilateral filter can keep edges sharp while removing noises. It uses Gaussian-distributed values but takes both distance and the pixel value differences into account. Therefore it requires sigmaSpace and sigmaColor for the parameters.

# Blur the image 
img_0 = cv2.blur(img, ksize = (7, 7))
img_1 = cv2.GaussianBlur(img, ksize = (7, 7), sigmaX = 0)   
img_2 = cv2.medianBlur(img, 7)
img_3 = cv2.bilateralFilter(img, 7, sigmaSpace = 75, sigmaColor =75)

# Plot the images
images = [img_0, img_1, img_2, img_3]
fig, axs = plt.subplots(nrows = 1, ncols = 4, figsize = (20, 20))

for ind, p in enumerate(images):
    ax = axs[ind]
    ax.imshow(p)
    ax.axis('off')
plt.show()

Thresholding

Thresholding transforms images into binary images. We need to set the threshold value and max values and then we convert the pixel values accordingly. There are five different types of thresholding: Binary, the inverse of Binary, Threshold to zero, the inverse of Threshold to Zero, and Threshold truncation.

img = cv2.imread('gradation.png')

# Thresholding 
_, thresh_0 = cv2.threshold(img, 127, 255, cv2.THRESH_BINARY)
_, thresh_1 = cv2.threshold(img, 127, 255, cv2.THRESH_BINARY_INV)
_, thresh_2 = cv2.threshold(img, 127, 255, cv2.THRESH_TOZERO)
_, thresh_3 = cv2.threshold(img, 127, 255, cv2.THRESH_TOZERO_INV)
_, thresh_4 = cv2.threshold(img, 127, 255, cv2.THRESH_TRUNC)

# Plot the images
images = [img, thresh_0, thresh_1, thresh_2, thresh_3, thresh_4]

fig, axs = plt.subplots(nrows = 2, ncols = 3, figsize = (13, 13))
for ind, p in enumerate(images):
    ax = axs[ind//3, ind%3]
    ax.imshow(p)
plt.show()

You can see how each type of thresholding can be expressed in a mathematical way and I(x, y) is the intensity at the point, or the pixel value at (x, y). But I prefer to understand the concept visually. Take a look at the picture on the right. The images are way better for you to comprehend the difference between the types.

But don’t you think it’s too harsh to take just one value of threshold and apply it to all parts of an image? What if we have a picture with various amount of lighting in different areas? In this case, applying one value to the whole image would be a bad choice. A better approach would be using different thresholds for each part of the image. There is another technique called Adaptive thresholding, which serves this issue. By calculating the threshold within the neighborhood area of the image, we can achieve a better result from images with varying illumination.

# Convert the image to grayscale
img = cv2.imread('text.jpg')
img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

# Adaptive Thresholding
_, thresh_binary = cv2.threshold(img, thresh = 127, maxval = 255, type = cv2.THRESH_BINARY)
adap_mean_2 = cv2.adaptiveThreshold(img, 255, 
                                    cv2.ADAPTIVE_THRESH_MEAN_C, 
                                    cv2.THRESH_BINARY, 7, 2)
adap_mean_2_inv = cv2.adaptiveThreshold(img, 255, 
                                        cv2.ADAPTIVE_THRESH_MEAN_C, 
                                        cv2.THRESH_BINARY_INV, 7, 2)
adap_mean_8 = cv2.adaptiveThreshold(img, 255, 
                                    cv2.ADAPTIVE_THRESH_MEAN_C, 
                                    cv2.THRESH_BINARY, 7, 8)
adap_gaussian_8 = cv2.adaptiveThreshold(img, 255, 
                                    cv2.ADAPTIVE_THRESH_GAUSSIAN_C, 
                                    cv2.THRESH_BINARY, 7, 8)

We need to convert the color mode to grayscale to apply adaptive thresholding. The parameters of adaptive thresholding are maxValue (which I set 255 above), adaptiveMethod , thresholdType , blockSize and C . And the adaptive method here has two kinds: ADAPTIVE_THRESH_MEAN_C , ADAPTIVE_THRESH_GAUSSIAN_C . Let’s just see how images are produced differently.

# Plot the images
images = [img, thresh_binary, adap_mean_2, adap_mean_2_inv, 
          adap_mean_8, adap_gaussian_8]
fig, axs = plt.subplots(nrows = 2, ncols = 3, figsize = (15, 15))

for ind, p in enumerate(images):
    ax = axs[ind%2, ind//2]
    ax.imshow(p, cmap = 'gray')
    ax.axis('off')
plt.show()

We have the original image and the one with binary thresholding on the left line. Compare this with the second and third image on the upper line, which is produced by ADAPTIVE_THRESH_MEAN_C . It shows a more detailed result than that of binary thresholding. We can also see that it gets more explicit when the C value is bigger. C indicates how much we’ll subtract from the mean or weighted mean. With the two images on the right line, we can also compare the effect of ADAPTIVE_THRESH_MEAN_C and ADAPTIVE_THRESH_GAUSSIAN_C with the same C value.

Gradient

I believe we are already familiar with the concept of gradients. In mathematics, the gradient geometrically represents the slope of the graph of a function with multi-variables. As it is a vector-valued function, it takes a direction and a magnitude as its components. Here we can also bring the same concept to the pixel values of images as well. The image gradient represents directional changes in the intensity or color mode and we can use this concept for locating edges.

# Apply gradient filtering
sobel_x = cv2.Sobel(img, cv2.CV_64F, dx = 1, dy = 0, ksize = 5)
sobel_y = cv2.Sobel(img, cv2.CV_64F, dx = 0, dy = 1, ksize = 5)
blended = cv2.addWeighted(src1=sobel_x, alpha=0.5, src2=sobel_y,
                          beta=0.5, gamma=0)
laplacian = cv2.Laplacian(img, cv2.CV_64F)

Sobel operation uses both Gaussian smoothing and differentiation. We apply it by cv2.Sobel() and two different directions are available: vertical (sobel_x) and horizontal (sobel_y). dx and dy indicates the derivatives. When dx = 1 , the operator calculates the derivatives of the pixel values along the horizontal direction to make a filter.

We can also apply in both directions by summing the two filters of sobel_x and sobel_y . With the function cv2.addWeighted() , we can calculate the weighted sum of the filters. As you can see above in the code cell, I gave the same amount of weight to the two filters.

Laplacian operation uses the second derivatives of x and y. The mathematical expression is shown below.

A picture is worth a thousand words. Let’s see how the images are like.

# Plot the images
images = [sobel_x, sobel_y, blended, laplacian]
plt.figure(figsize = (20, 20))
for i in range(4):
    plt.subplot(1, 4, i+1)
    plt.imshow(images[i], cmap = 'gray')
    plt.axis('off')
plt.show()

It’s apparent that the first and second image have a directional pattern. With the first image, we can clearly see the edges in the vertical direction. With the second image, we can see the horizontal edges. And both the third and fourth images, the edges on both directions are shown.

Morphological transformations

It’s also possible to manipulate the figures of images by filtering, which is called as morphological transformation. Let’s talk about erosion and dilation first.

Erosion is the technique for shrinking figures and it’s usually processed in a grayscale. The shape of filters can be a rectangle, an ellipse, and a cross shape. By applying a filter we remove any 0 values under the given area.

Let’s see how these can be implemented in codes.

img = cv2.imread('simpson.jpg')

# Create erosion kernels 
kernel_0 = np.ones((9, 9), np.uint8)
kernel_1 = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (9, 9))
kernel_2 = cv2.getStructuringElement(cv2.MORPH_CROSS, (9, 9))

kernels = [kernel_0, kernel_1, kernel_2]

# Plot the images
plt.figure(figsize = (20, 20))
for i in range(3):
    img_copy = img.copy()
    img_copy = cv2.erode(img_copy, kernels[i], iterations = 3)
    plt.subplot(1, 3, i+1)
    plt.imshow(img_copy)
    plt.axis('off')
plt.show()

Check out how the Simpson’s family shrank with different types of kernels applied. (Sorry to Simpson for losing his hands!) We can see the image with the ellipse filter is eroded in a “round” way while the one with the basic filter with a squared shape is eroded in a “linear” way. The last one with the cross filter shows it shrank in a “diagonal” way.

Dilation is the opposite to erosion. It is making objects expand and the operation will be also opposite to that of erosion. Let’s check out the result with the code as follows.

# Apply dilation
kernel = np.ones((9, 9), np.uint8)
img_dilate = cv2.dilate(img, kernel, iterations = 3)

plt.figure(figsize = (20, 10))
plt.subplot(1, 2, 1); plt.imshow(img, cmap="gray")
plt.subplot(1, 2, 2); plt.imshow(img_dilate, cmap="gray")
plt.show()

Opening and closing operation is the mixed version of erosion and dilation. Opening performs erosion first and then dilation is performed on the result from the erosion while closing performs dilation first and the erosion.

As you can see the picture above, closing is useful to detect the overall contour of a figure and opening is suitable to detect subpatterns. We can implement these operators with the function cv2.morphologyEx() shown below. The parameter op indicates which type of operator we’re going to use.

# Apply the operations
kernel = np.ones((9, 9), np.uint8)

img_open = cv2.morphologyEx(img, op= cv2.MORPH_OPEN, kernel)
img_close = cv2.morphologyEx(img, op= cv2.MORPH_CLOSE, kernel)
img_grad = cv2.morphologyEx(img, op= cv2.MORPH_GRADIENT, kernel)
img_tophat = cv2.morphologyEx(img, op= cv2.MORPH_TOPHAT, kernel)
img_blackhat = cv2.morphologyEx(img, op= cv2.MORPH_BLACKHAT, kernel)

# Plot the images
images = [img, img_open, img_close, img_grad, 
          img_tophat, img_blackhat]

fig, axs = plt.subplots(nrows = 2, ncols = 3, figsize = (15, 15))
for ind, p in enumerate(images):
    ax = axs[ind//3, ind%3]
    ax.imshow(p, cmap = 'gray')
    ax.axis('off')
plt.show()

Note that the Simpson’s hand is depicted differently in the images with the opening filter and closing filter. Gradient filter (MORPH_CGRADIENT) is the subtracted area from dilation to erosion. Top hat filter (MORPH_TOPHAT) is the subtracted area from opening to the original image while black hot filter (MORPH_BLACKHAT) is that from closing. I’d recommend you to visit here to get further explanations on morphological operators.

What’s next?

Did you enjoy the various image manipulation techniques? Besides the things we’ve discussed, there are other things available in OpenCV. So don’t hesitate to take a visit and check out OpenCV documentation. Next time will be about the detection techniques such as contour detection and face detections.

Are there errors you would love to correct? Please share your insight with us. I’m always open to talk, so feel free to leave comments below and share your thoughts. I also share interesting and useful resources on LinkedIn so feel free to follow or reach out to me. I’ll be back with the next series of the story. Stay tuned!

Computer Vision for Beginners: Part 2 was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Computer Vision is one of the hottest topics in artificial intelligence. It is making tremendous advances in self-driving cars, robotics as well as in various photo correction apps. Steady progress in object detection is being made every day. GANs is also a thing researchers are putting their eyes on these days. Vision is showing us the future of technology and we can’t even imagine what will be the end of its possibilities.

So do you want to take your first step in Computer Vision and participate in this latest movement? Welcome you are at the right place. From this article, we’re going to have a series of tutorials on the basics of image processing and object detection. This is the first part of OpenCV tutorial for beginners and the complete set of the series is as follows:

1. Understanding color models and drawing figures on images
2. The basics of image processing with filtering
3. From feature detection to face detection
4. Contour detection and having a little bit of fun

The first story of this series will be about installing OpenCV, explaining color models and drawing figures on images. The complete code for this tutorial is also available on Github. Now let’s get it started.

Introduction to OpenCV

Image processing is performing some operations on images to get an intended manipulation. Think about what we do when we start a new data analysis. We do some data preprocessing and feature engineering. It’s the same with image processing. We do image processing to manipulate the pictures for extracting some useful information from them. We can reduce noises, control the brightness and color contrast. To learn detailed image processing fundamentals, visit this video.

OpenCV stands for Open Source Computer Vision library and it’s invented by Intel in 1999. It’s first written in C/C++ so you may see tutorials more in C languages than Python. But now it’s also getting commonly used in Python for computer vision as well. First things first, let’s set up a proper environment for using OpenCV. The installation can be processed as follows but you can also find the detailed description here.

pip install opencv-python==3.4.2
pip install opencv-contrib-python==3.3.1

After you finish the installation, try importing the package to see if it works well. If you get the return without any errors, then you’re now ready to go!

import cv2
cv2.__version__

The first step we’re going to do with OpenCV is importing an image and it can be done as follows.

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# Import the image
img = cv2.imread('burano.jpg')
plt.imshow(img)

Have you ever been to Burano? It’s one of the most beautiful islands in Italy. If you haven’t been there, you should definitely check this place for your next holidays. But if you already know this island, you’d probably notice there’s something different in this picture. It’s a little bit different from the pictures we usually see from Burano. It should be more delightful than this!

This is because the default setting of the color mode in OpenCV comes in the order of BGR, which is different from that of Matplotlib. Therefore to see the image in RGB mode, we need to convert it from BGR to RGB as follows.

# Convert the image into RGB
img_rgb = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
plt.imshow(img_rgb)

Now, this is Burano! Such a lovely island in Italy!

More than just RGB

Let’s talk about color modes a little bit more. A color model is a system for creating a full range of colors using the primary colors. There are two different color models here: additive color models and subtractive color models. Additive models use light to represent colors in computer screens while subtractive models use inks to print those digital images on papers. The primary colors are red, green and blue (RGB) for the first one and cyan, magenta, yellow and black (CMYK) for the latter one. All the other colors we see on images are made by combining or mixing these primary colors. So the pictures can be depicted a little bit differently when they are represented in RGB and CMYK.

You would be pretty accustomed to these two kinds of models. In the world of color models, however, there are more than two kinds of models. Among them, grayscale, HSV and HLS are the ones you’re going to see quite often in computer vision.

A grayscale is simple. It represents images and morphologies by the intensity of black and white, which means it has only one channel. To see images in grayscale, we need to convert the color mode into gray just as what we did with the BGR image earlier.

# Convert the image into gray scale
img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
plt.imshow(img_gray, cmap = 'gray')

Actually, RGB images are made up by stacking three channels: R, G, and B. So if we take each channel and depict them one by one, we can comprehend how the color channels are structured.

# Plot the three channels of the image
fig, axs = plt.subplots(nrows = 1, ncols = 3, figsize = (20, 20))

for i in range(0, 3):
    ax = axs[i]
    ax.imshow(img_rgb[:, :, i], cmap = 'gray')
plt.show()

Take a look at the images above. The three images show you how each channel is composed of. In the R channel picture, the part with the high saturation of red colors looks white. Why is that? This is because the values in the red color parts will be near 255. And in grayscale mode, the higher the value is, the whiter the color becomes. You can also check this with G or B channels and compare how certain parts differ one from another.

HSV and HLS take a bit different aspect. As you can see above, they have a three-dimensional representation, and it’s more similar to the way of human perception. HSV stands for hue, saturation and value. HSL stands for hue, saturation and lightness. The center axis for HSV is the value of colors while that for HSL is the amount of light. Along the angles from the center axis, there is hue, the actual colors. And the distance from the center axis belongs to saturation. Transforming the color mode can be done as follows.

# Transform the image into HSV and HLS models
img_hsv = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)
img_hls = cv2.cvtColor(img, cv2.COLOR_BGR2HLS)

# Plot the converted images
fig, (ax1, ax2) = plt.subplots(nrows = 1, ncols = 2, figsize = (20, 20))
ax1.imshow(img_hsv)
ax2.imshow(img_hls)
plt.show()

But why do we have to transform the colors? What are these for? One example that can give the answer is lane detection. Please take a look at the picture below. See how the lanes are detected in different color modes. During the computer vision task, we do multiple color mode transformation along with masking. If you’d like to find more about how image processing is applied in the lane detection task, feel free to check out this post by nachiket tanksale.

Now I believe you get the idea. Image processing is ‘data preprocessing.’ It’s reducing noises and extracting useful patterns to make classification and detection tasks easier. Therefore all these techniques including the ones we’ll discuss later, are for helping the model to detect the patterns easier.

Drawing on images

Let’s bring some figures on the image. Now, we’re going to Paris. Have you ever heard of the wall of love? It’s a wall which is filled with the words “I love you” in all kinds of international languages. What we’re going to do is finding the words in our language and marking them with a rectangle. As I’m from South Korea, I’ll look up for ‘I love you’ in Korean. First, I’ll make a copy of the original image and then draw a rectangle with cv2.rectangle() We need to give the coordinates values for the upper left point and the lower right point.

# Copy the image
img_copy = img.copy()

# Draw a rectangle 
cv2.rectangle(img_copy, pt1 = (800, 470), pt2 = (980, 530), 
              color = (255, 0, 0), thickness = 5)
plt.imshow(img_copy)

Great! I think I caught the right position. Let’s try again. I can see one more Korean word from the image so I’ll make a circle this time. With cv2.circle() , we need to specify the point of its center and the length of its radius.

# Draw a circle 
cv2.circle(img_copy, center = (950, 50), radius = 50, 
           color = (0, 0, 255), thickness = 5)
plt.imshow(img_copy)

We can also put text data on the image. Why don’t we write the name of this wall this time? With cv2.putText() , we can designate the position and the font style and size of the text.

# Add text 
cv2.putText(img_copy, text = "the Wall of Love", 
            org = (250, 250),
            fontFace = cv2.FONT_HERSHEY_DUPLEX, 
            fontScale = 2, 
            color = (0, 255, 0), 
            thickness = 2, 
            lineType = cv2.LINE_AA)
plt.imshow(img_copy)

This is really a “lovely” wall, isn’t it? Try this yourself and find “I love you” in your language! 😍

More than images

Now we’ve been to Italy and France. Where would you like to go next? Why don’t we put a map and mark the places? We’re going to create a window and draw figures not by designating the points but by clicking directly on the window. Let’s try a circle first. We first create a function which will draw a circle with the data for the position and clicking of the mouse.

# Step 1. Define callback function
def draw_circle(event, x, y, flags, param):

    if event == cv2.EVENT_LBUTTONDOWN:
            cv2.circle(img, center = (x, y), radius = 5, 
                       color = (87, 184, 237), thickness = -1)

    elif event == cv2.EVENT_RBUTTONDOWN:        
            cv2.circle(img, center = (x, y), radius = 10,  
                       color = (87, 184, 237), thickness = 1)

With cv2.EVENT_LBUTTONDOWN or cv2.EVENT_RBUTTONDOWN , we can bring the data for the position when we press the buttons of the mouse. The position of the mouse will be (x, y) and we’ll draw a circle whose center is at that point.

# Step 2. Call the window
img = cv2.imread('map.png')

cv2.namedWindow(winname = 'my_drawing')
cv2.setMouseCallback('my_drawing', draw_circle)

We’ll set a map as the background of the window and name the window as my_drawing. The name of the window can be anything, but it should be the same because this acts like the id of the window. Using the cv2.setMouseCallback() , we make a connection between the window and the function draw_circle we made at step 1.

# Step 3. Execution
while True:
    cv2.imshow('my_drawing',img)
    if cv2.waitKey(10) & 0xFF == 27:
        break

cv2.destroyAllWindows()

Now we execute the window using while loop. Don’t forget to set the break unless you are making an infinite loop. The condition of the if clause is setting the window to be shut down when we press ESC on the keyboard. Save this as a file and import it on your terminal. If you’re to use jupyter lab, put the codes in one cell and execute. Now, tell me! Where do you want to go?

Let’s try a rectangle. As a rectangle requires two points for pt1 and pt2 in cv2.rectangle() , we need an additional step to set the first click point as pt1 and the last point as pt2. And we’re going to detect the movement of the mouse with cv2.EVENT_MOUSEMOVE and cv2.EVENT_LBUTTONUP .

We first define drawing = False as a default. When the left button is pressed, drawing becomes true and we give that first position as pt1. If drawing is on, it’ll take the current point as pt2 and keep drawing rectangles while we move the mouse. It’s like overlapping the figures. When the left button is up, drawing becomes false and it takes the last position of the mouse as its final point of pt2.

# Initialization
drawing = False
ix = -1
iy = -1

# create a drawing function
def draw_rectangle(event, x, y, flags, params):
    
    global ix, iy, drawing

    if event == cv2.EVENT_LBUTTONDOWN:
        drawing = True
        ix, iy = x, y
        
    elif event == cv2.EVENT_MOUSEMOVE:
        if drawing == True:
            cv2.rectangle(img, pt1=(ix, iy), pt2=(x, y), 
                          color = (87, 184, 237), thickness = -1)
            
    elif event == cv2.EVENT_LBUTTONUP:
        drawing = False
        cv2.rectangle(img, pt1=(ix, iy), pt2=(x, y), 
                     color = (87, 184, 237), thickness = -1)

Replace draw_circle function to draw_rectangle in step 1. Please don’t forget to make a change inside the callback function, cv2.setMouseCallback() as well. So the whole code script will be as follows. Save this script file and run it on the terminal or the jupyter notebook.

What’s next?

Did you enjoy the first time with OpenCV? You can also try other functions such as drawing a line or a polygon. Feel free to check the documentation for it, which can be found here. Next time, we’re going to talk about more advanced technologies such as attaching two different images, image contour and object detection.

⭐️ And you might also find these series interesting as well ⭐️

• The basic concept of convolutional neural networks:

The Most Intuitive and Easiest Guide for CNN

• Advanced networks of convolutional neural networks:

Deep Dive into the Computer Vision World: Part 1

Are there errors you would love to correct? Please share your insight with us. I’m always open to talk, so feel free to leave comments below and share your thoughts. I also share interesting and useful resources on LinkedIn so feel free to follow or reach out to me. I’ll be back again with another interesting story next time!

Computer Vision for Beginners: Part 1 was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Dash is an open source python library which enables us to create web applications with Plotly. It makes it easy to build an interactive visualization with simple reactive decorators like a dropdown, a slider bar, and markdown text data. We can even update the plots according to the input data with callbacks, and all these functions are available without Javascript or HTML directly. Plotly is a very powerful tool that enables us to make an informative and effective plot in a handy way and Dash can be considered as a stage for showcasing awesome visualization.

Today I’m going to show you how to create a dashboard with Plotly. And we’ll make a time series line plot and add a dropdown and a slider bar interacting with the plot. There is nice documentation on how to use this library, but I find it could be hard to read the example codes and utilize them at first. So this could be a kind bridge for you to understand how to use Dash with Plotly.

Installation

To build a dashboard, we need to install some packages as follows.

pip install plotly==2.5.1
pip install dash==0.21.0
pip install dash-core-components==0.22.1
pip install dash-html-components==0.10.0
pip install dash-renderer==0.12.1

As I said above, dash-core-components allows us to build not only graphs but also a dropdown and a text box so that we can update the components accordingly. dash-html-components enables us to use HTML & CSS with Python. It helps us to place HTML components such as Div, H1, and H2 on our dashboard.

If this is your first time using Dash and HTML syntax, it can be a bit complex and hard to read. Therefore I’d recommend you to take the following script as a skeleton guide to Dash. From now on, what we’re going to do is filling this out step by step.

First things first, we import the libraries. You can add other libraries as you need. And then we initialize Dash by calling the Dash class. This is like putting a blank whiteboard on our table and what we are doing is building up additional applications on that board.

# Step 1. Launch the application
app = dash.Dash()

# Step 2. Import the dataset
df = pd.read_csv('finance-charts-apple.csv')

Now let’s bring some ingredients from the CSV file with pandas. The dataset we’re going to use is the Apple stock price data which is available here.

Simple scatter plot on a dashboard

I won’t cover the details on Plotly itself, but you can find a nice tutorial for Plotly from this video if it’s necessary. You can also learn how to make a 3D plot or Choropleth Map in the series of the video. Here we’ll make a line plot which shows fluctuations in the stock price.

# Step 3. Create a plotly figure
trace_1 = go.Scatter(x = st.Date, y = st['AAPL.High'],
                    name = 'AAPL HIGH',
                    line = dict(width = 2,
                                color = 'rgb(229, 151, 50)'))

layout = go.Layout(title = 'Time Series Plot',
                   hovermode = 'closest')

fig = go.Figure(data = [trace_1], layout = layout)

Now, it’s time for dash-html-components to come into play. We first put a division and then bring the graph inside it. id is giving a name to this component so that we can call it by its name. You’ll understand what this is for later. And then we make a server to run in step 6. If we set the debug mode equals true, we can easily change and update the app while the server is running.

# Step 4. Create a Dash layout
app.layout = html.Div([
                dcc.Graph(id = 'plot', figure = fig)
                      ])

# Step 6. Add the server clause
if __name__ == '__main__':
    app.run_server(debug = True)

Let’s save this script with the name of app.py and import it on the terminal (or anaconda prompt). Note that the working directory should be the same with where you saved the file.

C:\Users\jjone\Dash> python app.py

If there is no typo or syntax error, you’ll see the localhost address. You can copy and paste it or you can just type localhost:8050 on a new web tab.

Adding a header and a paragraph

We can also have additional components such as putting text data just like HTML. You can find what kind of elements are available here. Let’s put a simple heading and a paragraph on our page.

# Step 4. Create a Dash layout
app.layout = html.Div([
                # adding a header and a paragraph
                html.Div([
                    html.H1("This is my first dashboard"),
                    html.P("Learning Dash is so interesting!!")
                         ], 
                    style = {'padding' : '50px' , 
                             'backgroundColor' : '#3aaab2'}),

# adding a plot        
                dcc.Graph(id = 'plot', figure = fig)
                      ])

On top of the graph, I’m going to put one more division and add a header and a paragraph inside it. We have two major components, html.Div and dcc.Graph. Inside html.Div, there are two additional components, header ( html.H1 ) and a paragraph ( html.P ). We can change the margin of the components or the background color with the style attribute. It should be designated in a dictionary format supporting CSS properties.

Please pay extra care where the parenthesis starts and ends. It’s important to understand the range of the paragraph for each component. As there are so many parenthesizes and square brackets, it can be confusing at first. And It’s so easy to make a syntax mistake as well.

Put this code at step 4 in our template and see the result. We can simply check the outcome by pressing F5 if the server isn’t shut down.

Now I hope you get the idea of what the dash components and the HTML components are. How to put multiple components and how they go along with together.

Dropdown

Let’s try making a dropdown on our dashboard this time. We’re going to make another plot which changes its y-axis according to the given option. By the documentation of dash components, we can putDropdown as follows.

As you can see, the options should be in the format of a dictionary. In our case, the options will be the columns only with the continuous variables, from the 2nd column to the 10th. With the list comprehension, we can make the option dictionary in just one line.

features = st.columns[1:-1]
opts = [{'label' : i, 'value' : i} for i in features]

Now instead of the HTML components, we’re going to put the dropdown component at the bottom of the plot. Take a look at the bold font first because others are just for decorating the major two applications.value is the default value of the dropdown.

# Step 4. Create a Dash layout
app.layout = html.Div([
                # adding a plot
                dcc.Graph(id = 'plot', figure = fig),

                # dropdown
                html.P([
                    html.Label("Choose a feature"),
                    dcc.Dropdown(id = 'opt', 
                                 options = opts,
                                 value = opts[0])
                        ], style = {'width': '400px',
                                    'fontSize' : '20px',
                                    'padding-left' : '100px',
                                    'display': 'inline-block'})
                      ])

And just as what we did before, we can replace this code at step 4 in our template and check the outcome.

Connecting to graph with callback

To update the graph according to the choice of the dropdown, we need to make a connection between input data (the dropdown) and output data (the graph). This will be done by adding a callback function in step 5.

# Step 5. Add callback functions
@app.callback(Output('plot', 'figure'),
             [Input('opt', 'value')])

Do you remember what id we gave to each component? plot and opt. Therefore we can call them with their name as shown above. We take the input data from the options by the name of opt and give the output to the line plot by the name of plot.

def update_figure(X):
    trace_2 = go.Scatter(x = st.Date, y = st[X],
                        name = X,
                        line = dict(width = 2,
                                    color = 'rgb(106, 181, 135)'))
    fig = go.Figure(data = [trace_1, trace_2], layout = layout)
    return fig

So we take the input data and then return the output as we want by creating an update function. What is our input data here? It’s the name of the feature variable selected from the dropdown, and we are to make it as the y-axis of our second line plot. As there is nothing to be changed with the trace_1, we can simply add trace_2 to the data list. Now if we put this code at step 5 and rerun app.py, the result will be is shown below.

I’d like to briefly talk about a callback before we move on to the next section. Because it’s one of the most frequently used functions while programming any kind of applications. In a nutshell, a callback is a function that is executed later when it gets the real execution command and it’s usually for the purpose of updating. Callback functions don’t work right after they get the command. These people are kind of lazy. They will do their work after the other command lines are executed and we call them back again with the real execution command.

See how our update_figure() works. This function does his work not right after we import this app but when we give the actual input signal to it by “calling back.” This is what a callback function is. It’s totally fine you use this without understanding but I hope you get the rough idea here. You can also find more details from this article.

RangeSlider

Lastly, let’s try a range slider. We’ll add a year range slider to the plot, and it’s very similar to dropdowns. Let’s check the documentation of dash components first.

I think you can easily get the idea by this time. The key point here is how to make a mark dictionary. As the date period starts from (Feb 17 2015) to (Feb 17 2017), I’d like to add 7 marks between the period as follows.

st['Date'] = pd.to_datetime(st.Date)
dates = ['2015-02-17', '2015-05-17', '2015-08-17', '2015-11-17',
         '2016-02-17', '2016-05-17', '2016-08-17', '2016-11-17',  
         '2017-02-17']

date_mark = {i : dates[i] for i in range(0, 9)}

Now we can simply put the range slider at the dropdown place. I’ll name it as slider this time. min and max are the minimum and the maximum of the slider and value is the default setting of the slider. Again, all the other parts are designing the HTML components with CSS style.

# Step 4. Create a Dash layout
app.layout = html.Div([
                # adding a plot
                dcc.Graph(id = 'plot', figure = fig),

                # range slider
                html.P([
                    html.Label("Time Period"),
                    dcc.RangeSlider(id = 'slider',
                                    marks = date_mark,
                                    min = 0,
                                    max = 8,
                                    value = [3, 4]) 
                        ], style = {'width' : '80%',
                                    'fontSize' : '20px',
                                    'padding-left' : '100px',
                                    'display': 'inline-block'})
                      ])

After making a slider, what will be the next step? Connecting it to the plot! Again we’ll take the input data from the slider and return the output to the plot. But this time, there is a small trick here. Did you notice that the default value of the slider is a list with two values? Please check the code box above again. value = [3, 4] Different from the dropdown component, the range slider takes two values in a list. Therefore when we get the input data from the slider, there will be two values of the starting and ending value (X[0] , X[1])

# Step 5. Add callback functions
@app.callback(Output('plot', 'figure'),
             [Input('slider', 'value')])
def update_figure(X):
    st2 = st[(st.Date > dates[X[0]]) & (st.Date < dates[X[1]])]
    trace_1 = go.Scatter(x = st2.Date, y = st2['AAPL.High'],
                        name = 'AAPL HIGH',
                        line = dict(width = 2,
                                    color = 'rgb(229, 151, 50)'))
    fig = go.Figure(data = [trace_1], layout = layout)
    return fig

Now we’ll filter the data and update the plot with the given period. If you reload the web page again, you’ll see the same result with me.

Bring it all together!

I think we covered quite a lot of things so far. From putting a graph to adding a callback. Now why don’t we bring it all together? I’d recommend you to review the script below step by step. You can combine all the components of the graph, the dropdown and the slider at step 4 just like replacing and changing modules.

Are you ready to see how our first dashboard looks like? 😃😃🙌🙌 BOOM!!

Great work! Did you enjoy building a dashboard? I hope now you become comfortable reading and understanding other Dash code examples. Dash provides such a nice guide so feel free to explore the sites on your own. You can also check other stuff available in Dash by Plotly.

🌟 Introducing Dash 🌟

Deployment

So far our application is locally hosted on your machine. To make it available to other people, we need to deploy it on the web. There are two choices you can take, App authorization and deploying it to Heroku. The first one requires us to install the dash-auth package an there are two different types of authentication, HTTP Basic Auth or Plotly OAuth. You can follow this tutorial.

Another way is by using the hosting platforms such as Heroku, AWS, Google Cloud Platform and so on. Heroku is the easiest way for deploying and you can follow this tutorial. The other platforms are just the same with Heroku.

Beside dropdown, we can also add other components such as text area or uploading data. I recommend you to explore what kind of dash components are available. You can also add as many HTML components as you want and design your board in a much fancier way with CSS properties. If you need to learn some basics of HTML & CSS, you can find a great tutorial from this video.

Thank you for reading and I hope you found this post interesting. If there’s anything need to be corrected, please share your insight with us. I’m always open to talk so feel free to leave comments below and share your thoughts. I also share interesting and useful resources on LinkedIn so feel free to follow and reach me out. I’ll be back again with another interesting story next time!

Another Stage Of Visualization: Be Reactive with Dash was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Everything has its past. And sometimes the past defines us. What route we’ve walked through and what choices we made along the way. They are curved as our history and tells us what kind of person we were and where we’re headed. This is also applicable to data. They can have their past, and that history can be used for predicting what’s coming next, the future. As we have a fortune teller, likewise data, which is called the sequence models.

This is the third part of ‘The Most Intuitive and Easiest Guide’ series to neural networks. The complete set of this post is as follows:

1. The Easiest Guide for ANN
2. The Easiest Guide for CNN
3. The Easiest Guide for RNN (this post)

This post assumes that you have pre-knowledge on the basics of neural networks. If you are completely new to this, feel free to check out the first series. Today’s keywords are sequential, vanishing gradients, memory and gates. It can be hard to understand the structures and the mathematical expressions of recurrent neural networks. So this post will focus on the basic intuition with least amount of maths, just like what we did with ANN in the previous post.

What’s new here?

If you’ve ever heard of sequence models, then you’d probably have seen this kind of diagrams shown below. And It’s so natural that you don’t get the concept portrayed in the diagram at one sight. This is because the sequence models are a bit different from the other neural networks discussed previously.

So before jumping deep into this neural network, why don’t we talk about where the sequence models are used in the real world. This is because it can help us to understand the concept by getting a hint from those examples. So where RNN get the spotlight by data scientists?

Speech recognition can be one typical example we can find in the closest distance. Alexa, Siri, and Google home. Those devices response to our call and can do some simple tasks such as setting the alarm or sending a message to someone else. We can also find it with google translation or chatbots as well.

The sequence model is also popular in the field of genetics. If you have a little background knowledge on DNA, it can be much easier to understand the meaning of sequence here. In human molecules, DNA(deoxyribonucleic acid) is composed of a chain of 4 different types of nucleotides: A, C, G, and T. Genetic information exists in the successive combination of the nucleotides. Therefore these orders are essential for all kinds of living things and the sequence model has high usage in predicting the uncovered part of genes.

What about time series analysis? This neural network is also able to predict what value will come next in stock market price or temperature data. For example, we gather 30 days of temperature data and use this series of data to predict what will be the temperature of 31st.

Did you notice the common characteristic of these examples? Speech, DNA chains, and time series. The answer is “sequential.” The data is not static or discontinuous, but forgoing and successive. But what do we mean by sequential or consecutive when it comes to data?

Let’s recall what we’ve been doing with other models like ANN. We had sample data and passed them through layers from left to right. This means that we will input the data at a time and then they will travel toward the output layer. It was feedforward propagation and had only one direction of the flow.

In the case of RNN, however, the data aren’t inputted at the same time. As you can see the picture on the right, we will input X1 first and then input X2 to the result of X1 computation. So in the same way, X3 is computed with the result from X2 computation stage.

Therefore when it comes to data, ‘sequential’ means we have an order in time between the data. When it’s ANN, there isn’t any concept of order in X1, X2 and X3. We just input them at once. In the case of RNN, however, they are inputted at different times. Therefore if we change the order, it becomes significantly different. A sentence will lose its meaning. And when it comes to DNA, this change might create… a mutant.

So the RNN has its appeal in that we can connect the data with the previous data. This means that the model starts to care about the past and what is coming next. As the recurrent units hold the past values, we can refer to this as memory. We are now able to consider a real meaning of ‘context’ in data.

The structure of Recurrent Neural Network

Now with this basic intuition, let’s go deeper into the structure of RNN. This is a simple RNN with one shallow layer. Our model is now going to take two values: the X input value at time t and the output value A from the previous cell (at time t-1).

Please take a look at the equation ①. There are weights and bias as we did with simple ANN. It’s just adding one more input value A0. And there are two different outputs from the cell. The output of A1 which will go to the next unit(②) and the final output Y1 of the unit cell(③). Don’t get stressed by all these subscripts. They are just indicating to what value the weight belongs. Now let’s move to the next units.

Simple and cool, right? Now we can predict the future. If it’s about the stock market, we can predict the stock price of a company. We then guess If there is a big dataset say, the history data from 10 years ago, we’ll get better accuracy. So the longer the data, the better the outcome, Right? But the truth is, this model ain’t as ideal as we’d expect.

The real thing starts from here

The idea of remembering the past is fantastic. But there is one critical problem in back-propagation. Back-propagation is a step for going backward to update the weights of each layer. To update the weights, we get the gradient of the cost function and keep multiplying the gradients at the given layers using chain rule. The actual backward propagation in RNN is a bit complex than this diagram but let’s skip them for simplicity. (For example, The real backward propagation takes not only the final output Yt but also all the other output Y used by the cost function.)

Imagine when the gradients are bigger than 1. The updated values become so big to use it for optimizing. This is called exploding gradients. But this isn’t a severe problem because we can fix the range that gradients can’t go over. The real problem occurs when the gradients are smaller than 1. If we keep multiplying the values lower than 1, the result becomes smaller and smaller. After some steps, there will be no significant difference in outcome, and it can’t make any update in weights. It is called vanishing gradients. It means the back-propagation effect can’t go far enough to reach the early stage of layers.

This is why people say RNN has a bad memory. If the length of the input values gets longer, we can’t expect actual optimization. This is a really critical problem because the power of neural networks comes from updating weights. Would there be other ways to fix this problem?

Let’s just forget about it!

To be honest, it’s very hard to get good memories for people with bad memories. We all know that. We can’t remember too many things. The same goes for our model. Instead of dragging all the past, maybe it’ll be better to remember selectively. This is like choosing only the important information and forgetting the rest. Having all those past values causes the vanishing gradients. Therefore we’ll give an additional step to simple RNN, which is called Gated Recurrent Units.

The diagram on the left shows the computation inside the unit cell of RNN. There’s nothing new here. It’s just showing that it takes two input values and returns two output values after the computation. On the right side, you can see one small change inside the box. The green box. What is this? This is a gate controller. This box determines whether we should remember or not.

We’ll compute the tanh activation (①) as what we did with RNN, but this won’t be used right ahead. It’s like a candidate. Here we’ll also get the new value (②) for the gate. Since it takes the sigmoid function, the output value always goes between 0 to 1. So by multiplying it with the ① value, we’re going to decide whether we’ll use it or not. When it’s 0, “No Use” or “No Update.” (Use the previous value in this case) When it’s 1, “Use” or “Update.” This is like opening and closing a gate.

The gate and the gate

Now we finally arrived at the LSTM, Long Short Term Memory networks. Actually, LSTM was proposed earlier than GRUs. But I brought GRUs first because the structure of LSTM is more complex compared to that of GRUs with two more gates. Additionally, there will be a new concept here which is called ‘cell state.’ But you’re already used to this concept. It’s putting the value containing the memories aside from the hidden state value A. Let’s have a look at what LSTM has in details. The blue box and the green box (the input gate) in the middle is the same with GRUs. There are two additional green boxes here: the forget gate and the output gate.

The computation will also be the same, so we’re going to get the output from the tanh function (①) and the input gate value from the sigmoid function (②). But the multiplying step is a bit different this time. Instead of taking only the result of the input gate, we also consider the forget gate value coming from the left.

Let’s see the equation in line 3 on the right. We have the forget gate, and this will be used for updating the cell state value like what you can see in line 4. Lastly there is one gate left, the output gate. And as you can see in line 6, we get the A value at time t by multiplying the output gate value and the tanh activation value of C at time t.

With these three additional gates, LSTM can have a stronger memory ability. Those are all about controlling which parts or what amount it should remember. It’s so robust and effective that it’s really popular among the sequence models. Then could we say is it better than GRUs for all cases? As we say there’s no silver bullet all the time, GRUs has its own strength as well. Its structure is simpler than LSTM, so it’s sometimes proper to use GRUs to make a big model.

Conclusion

What we’ve discussed so far was the sequence model with one shallow layer. Then what it would be like when we add more layers? If we say adding more layers goes horizontal in ANN or CNN, it goes vertically in RNN. But the sequence model is already a big model with one or two layers, it may end up overfitting if we add a few more layers on top of that. Therefore applying normalization techniques such as dropout or batch normalization is required.

This post was mainly about understanding the structure of each model from simple RNN to LSTM. The mathematical expressions we walked through can be complicated at first sight. After you get familiar with the overall structures of the models, however, it becomes easy to understand. Because maths are just a numeric expression to represent the concept in a clear and effective manner. If you can see the meaning behind all those numbers, then maths will become fun and interesting too.

I also brought other additional resources as always. When you’re ready to take a further step with the sequence models, I highly recommend you to check out these articles as well.

• Understanding LSTM Networks by Christopher Olah: Such a great article on understanding the structure of RNN and LSTM. And Colah’s blog is really popular. You can see many other materials which have this blog as a reference.
• Must-Read Tutorial to Learn Sequence Modeling by Pulkit Sharma: I already posted his work in the previous blog, and this is the series for the sequence models. Highly recommend you to check out other series as well if you haven’t.
• Recurrent Neural Networks by Example in Python by Will Koehrsen: A gentle guide from the top writer of Medium. This will be a great start for building your first RNN in Python.

Thank you for reading and I hope you found this post interesting. If there’s anything need to be corrected, please share your insight with us. I’m always open to talk so feel free to leave comments below and share your thoughts. I’ll come back with another exciting project next time. Until then, happy machine learning!

The Most Intuitive and Easiest Guide for Recurrent Neural Network was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.