How to handle coloured images with convolutions
Originally published at https://ameersaleem.substack.com.
Hello fellow machine learners,
Last week, we explored how convolutions can be used to extract features from greyscale images. In this article, we’ll discuss some additional convolution techniques before turning our attention to coloured images. I will be extending a few examples from the previous piece, so I would strongly recommend that you have a look at that before this one:
All done with the pre-reading? Awesome. In that case…
…let’s get to unpacking!
Padding
In the Prewitt kernel examples from last time, we convolved the MNIST 28x28 images with 3x3 kernels. What I didn’t mention, however, was that the convolution outputs were 26x26 images, the dimensions of which do not match those of the original MNIST images. This problem is exacerbated in the case where we want to apply multiple convolutions in a row to the same input . Is there a way to ensure that the output size matches the input size in spite of convolving the input with a 3x3 kernel?
To do this, we simply add a border of 0s around the original input image, and convolve this with the kernel. This border is called padding, and we can make this border as thick as we like. Padding with zeroes is the standard method of padding, because the inclusion of zero values is best for computational efficiency (any number multipled by zero equals zero). Recall from last week that
Let’s add one row of padding around the input image, as follows:
The convolution is computed in the same way as before, except there are more individual convolution operations to carry out. Starting in the top left corner:
The next result comes from sliding the kernel one unit to the right:
Hopefully you see that we get four columns in total:
And I will leave you to check that I’ve computed the rest of the output correctly:
By padding with one extra row, our output now has two more rows and columns than the original result had. Padding helps us to not only maintain the input dimension size, but it also ensures that values on the input image’s border are worth the same as those closer to the centre of the image. What I mean is that, without padding, the borderline digits are convolved with fewer times than the digits in the middle. Thus, padding can be used to ensure that each digit in the input space is involved in the same number of convolution operations.
Stride
Conversly, there may be instances where you wish to reduce the output size. One reason could be to reduce the computational cost of convolving an input with a kernel. This could be of use for something such as the tiger example from last week, the likes of which took my laptop a bit of time to apply convolutions operations to.
The stride tells us how far our sliding window needs to move between each individual convolution operation. I will illustrate the concept with a 5x5 input and a 3x3 kernel. Under our convolution rules so far, this yields a 3x3 output, in which the kernel is convolved with the input in nine ways in total:
This whole time, we have moved the sliding window across/down by one space. This is an implicit implementation of a stride value of 1. Now, if we were to move the sliding window two spaces each time vertically/horizontally instead of just one, then we get four convolution operations instead of nine:
Let’s compare the output by stride when using a 3x3 Prewitt kernel:
With this depiction, stride can also be thought of as selecting a subset of values from the stride=1 output. Indeed, with stride=2, the output takes the values in the four corners of the original output.
Convolution output size
So padding increases the convolution output, whereas stride decreases it. As such, there is a formula that we can follow that yields the convolution output dimensions given the dimensions of the input object and kernel.
For square inputs and kernels, we have that
By “padding size”, we mean the size of the padding border, i.e. my previous example had a padding size of 1. As mentioned earlier, the default stride value is 1.
If the input or kernel is non-square, then you’d have to apply the formula separately to the height and width of the output:
Indeed, you don’t have to apply the same number of padding rows as padding columns. Similarly, you can take vertical strides of a different size to that of the horizontal strides. In the future, I’ll probably stick to the square case for all the parameters to help keep things simple. Regardless, these formulae can help you predict what output dimensions to expect when tackling convolutions yourself 😎
Get AmeerSaleem’s stories in your inbox
Join Medium for free to get updates from this writer.
Also, I came across this paper that goes through convolution arithmetic in more depth. Do have a look if you’re interested!
Colour channels
Greyscale images are comprised of pixels that take an integer value between 0 and 255, the former denoting a black pixel and the latter representing a white pixel. However, a pixel in a coloured image takes on a slightly different representation.
A coloured pixel can be represented by a tuple of three integers, each between 0 and 255. The first value denotes the presence of the colour red, the second that of blue and the third green. Each integer value represents the intensity of the corresponding colour: a pure red pixel is represented by (255,0,0), a pure green pixel is (0,255,0) and a pure blue pixel is (0,0,255). You’ve probably heard of this as the RGB colour system: it’s used for digital images. In the context of image processing, we think of each of these colours as a separate colour channel.
Pure white is a result of full intensity for each colour, so the tuple value is (255,255,255). Conversely, the colour black arises in the absence of any of these colours, with a tuple value of (0,0,0). This aligns with the intuition presented in our discussion of greyscale images. In fact, last week’s tiger image was originally a fully coloured image- to convert it to greyscale, I took the average value across the colour channels.
Overall, a coloured image is a 2x2 grid, with each entry represented by a 3-dimensional array. When applying a convolution to a coloured image, we can apply a different kernel to each colour channel separately and then add up the results. We stack each of these separate kernels together and consider them to be part of one kernel. This kernel can be thought of as a 3D kernel, but I think it’s best illustrated as three 2D kernels:
The above example showcases a 3x3x3 kernel. The convolution of the RGB image and the 3D kernel is achieved by conducting convolutions across each of the three channels:
This gives us three separate results
which we can then sum together to get the overall convolution output:
As long as the number of input channels matches the number of channels of the kernel, we’ll be good.
The kernels that are part of the same stack do not have to have the same values. Let’s see how this works for a real image. I am reusing the tiger image we used previously. We can construct a 3x3x3 kernel designed to extract the green parts of the image. In Python, such a kernel may be defined as
And the convolution output looks like the following when we apply the ‘coolwarm’ colour map:
According to the colour map, the red areas are the areas where the original image is the most green, while the blue areas are the areas which have the least green. I appreciate that this may be a bit confusing given the talk of RGB colours, so here is the same result showcased with the greyscale colour map instead, which I hope makes things a bit clearer:
The white areas represent the greener areas of the original image, while the darker areas represents the parts of the original image that were less green/had no green at all.
Multiple kernels for multiple channels
I would like to wrap this article up on a note about multiple kernels. Suppose we wanted to extract n different features from a coloured image. This can be achieved by applying n of our 3D kernels. The input image is convolved separately with the n kernels, yielding n separate outputs. Here is a graphic to display this for n=2:
Essentially, the number of outputs is simply equal to the number of kernels we use. I think the difficulty in understanding this lies in the dimension sizes of each object at each step. Or at least, this is what confused me the most when I was learning about image convolutions for the first time.
Packing it all up
The previous section acts as a prelude to where we’re going next: networks comprised of many kernels. I really wanted to take my time going through convolutions, because I think the topic can appear quite daunting without discussing each component one at a time. Thus, I decided to start with the discrete 1D convolution: we then introduced the 2D convolution within the context of image processing; and today we’ve tackled ‘3D kernels’. And of course, the discussion of image processing provides the perfect excuse for us to play around with pictures in code 🐯🐅
Oh, and you can check out today’s code here. I extended the notebook that we built out last time.
Here is today’s roundup:
📦 Padding can be used to increase the convolution output size, and usually consists of adding a border of zeroes around the input image before convolving. This is useful if you want to keep the output the same size as the input, but it also ensures that each pixel is weighted the same (i.e. with padding, every digit is used the same number of times).
📦 Stride is the opposite of padding, where we intentionally skip convolution operations in order to reduce the size of the output. Stride can be implemented for the sake of computational efficiency, which is of immense benefit when chaining together convolutions successively.
📦 Convolutions can be applied to coloured images. For RGB images, we can apply “3D kernels”, which are essentially a stack of three 2D kernels. Each kernel is convolved with a separate colour channel and the three convolutions are summed together to give us the convolution output. These subkernels do not need to be the same; in fact, applying different values for each can help us to extract different information about the presence of different colours in the original image.
Training complete!
I hope you enjoyed reading as much as I enjoyed writing 😁
Do leave a comment if you’re unsure about anything, if you think I’ve made a mistake somewhere, or if you have a suggestion for what we should learn about next 😎
Until next Sunday,
Ameer
Sources
- My GitHub repo where you can find the code for the entire newsletter series: https://github.com/AmeerAliSaleem/machine-learning-algorithms-unpacked and you can find today’s notebook here
- “A guide to convolution arithmetic for deep learning”, by Vincent Dumoulin and Francesco Visin: https://arxiv.org/pdf/1603.07285
- Source for tiger image: https://www.pexels.com/photo/photo-of-tiger-and-cub-lying-down-on-grass-2541239/
Originally published at https://ameersaleem.substack.com.
