<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tomasz Kacmajor on Medium]]></title>
        <description><![CDATA[Stories by Tomasz Kacmajor on Medium]]></description>
        <link>https://medium.com/@tomasz.kacmajor?source=rss-b86619ed92c2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*VuB474F9DqHX6Cn8nEYIUg.jpeg</url>
            <title>Stories by Tomasz Kacmajor on Medium</title>
            <link>https://medium.com/@tomasz.kacmajor?source=rss-b86619ed92c2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 15:01:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tomasz.kacmajor/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Traffic Sign Recognition using Convolutional Neural Network]]></title>
            <link>https://medium.com/hackernoon/traffic-sign-recognition-using-convolutional-neural-network-8a1f90e8fb24?source=rss-b86619ed92c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a1f90e8fb24</guid>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[self-driving-cars]]></category>
            <category><![CDATA[image-recognition]]></category>
            <category><![CDATA[deep-neural-networks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Tomasz Kacmajor]]></dc:creator>
            <pubDate>Sun, 15 Oct 2017 11:26:14 GMT</pubDate>
            <atom:updated>2019-05-20T09:23:57.145Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iBJ_SZKT7k-SQNen.jpg" /></figure><p>I’m going to describe a complete pipeline for <strong>Traffic Sign Recognition</strong> problem posed in <a href="https://www.udacity.com/course/self-driving-car-engineer-nanodegree--nd013">Udacity course</a> “Self-Driving Cars Engineer”. Traffic Sign Recognition is a basic, day by day task for self-driving cars. That’s why it has to be covered in the <a href="http://tomaszkacmajor.pl/index.php/2017/05/07/self-driving-cars-in-python/">series about Self-Driving Cars</a> where I present different projects related to this field. The recognition system processes a traffic sign image extracted from the road scene. Eventually, it should classify that sign into one of 43 categories. In order to make it happen, a <strong>Convolutional Neural Network</strong> is applied, being trained with 50.000 images beforehand.</p><h3>Goals of the project</h3><p>The goal of the Traffic Sign Recognition project is to build a <strong>Deep Neural Network</strong> (DNN) which is used to classify traffic signs. We should train the model so it can decode traffic signs from natural images using the <a href="http://benchmark.ini.rub.de/?section=gtsrb&amp;subsection=dataset">German Traffic Sign Dataset</a>. This data should be firstly preprocessed in order to maximize the model performance. After choosing model architecture, fine tuning and training, the model will be tested on new images of traffic signs found on the web. Because we deal with images classification, a Convolutional Neural Network is chosen as a type of DNN, which is a common choice for this type of problems. The code is written in <strong>Python</strong> with use of <strong>TensorFlow</strong> library. It’s great for making quick, high-level changes in our model architecture. In addition, TensorFlow supports computation on GPU which can really speed up required calculations. Finally, to make all calculations, I launched Amazon Web Services EC2 GPU instance to have bigger advantage over my laptop capabilities.</p><h3>Project pipeline</h3><p>My pipeline consists of 7 steps, which are quite common in classification problems:</p><ol><li>Loading the data</li><li>Dataset exploration and visualization</li><li>Data preprocessing</li><li>Data augmentation</li><li>Designing, training and testing a CNN model</li><li>Using the model on new images</li><li>Analyzing softmax probabilities</li></ol><blockquote><em>You can find a complete code for the project on </em><a href="https://github.com/tomaszkacmajor/CarND-Traffic-Sign-Classifier-P2/blob/master/Traffic_Sign_Classifier.ipynb"><em>github</em></a><em>. It is presented in a convenient way using </em><strong><em>Jupyter Notebook</em></strong><em>where immediate results are showed after each code block.</em></blockquote><h3>Dataset</h3><p>The dataset is divided into training set (34.799 samples), validation set (4.410 samples) and test set (12.630 samples). Each sample represents a traffic sign labeled as one of 43 classes. It can be e.g. a stop sign, yield, 30 km/h speed limit etc. The shape of a traffic sign image is scaled to 32x32 pixels in 3 channel RGB representation (32x32x3). Below, there are a few random samples from the dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/0*H8YQvhJtmYiqxFoe.png" /></figure><p>We should firstly explore the dataset, understand it against the problem to solve. Let’s see how many samples we have here for each traffic sign class. We wouldn’t like the model to be biased towards any of the class. Below, there is a histogram of sample occurrences in the training set for each label.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/0*X-DsQGNCFLy_6ON0.png" /></figure><p>Right now, we can see that some labels are greatly underrepresented while others have quite many representatives in the dataset. Should we dismiss the latter ones to equalize the histogram? Let’s draw firstly the subset of images which are belonging to the same class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/0*pJ022OVEpNln6XJ1.png" /></figure><p>We can observe that images from the same class can be represented quite differently in the dataset. Generally, there can be different <strong>lighting conditions</strong>, image can be <strong>blurred</strong>, <strong>rotated</strong> or <strong>scaled</strong>. Indeed, these are samples which are extracted from real world images. And our model have to handle all of these conditions. So, it’s probably better not to truncate our dataset in order to obtain data balance. Let’s “produce” some new samples instead, mostly for underrepresented signs.</p><h3>Data preprocessing</h3><p>To generate so-called <strong>augmented data</strong> I randomly chose images to copy. To provide additional information to the model, I randomly rotated this copy and changed its brightness. For all these operations <strong>OpenCV</strong> library was used. I performed these operations until each label had 3200 samples. This increased the training set to 139.148 samples. As an illustration, here’s a drawing of a sample traffic sign with generated images (rotated and different brightness).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/561/0*aYsJYWMpJfxJ10Cg.jpg" /></figure><p>Next, I decided to convert the images from RGB to grayscale. As a consequence, we have 3 times less of the data to process which strongly influences on the training time. In addition, in the <a href="http://yann.lecun.com/exdb/publis/pdf/sermanet-ijcnn-11.pdf">paper</a> dealing with Traffic Sign Recognition as well, the authors discover that rejecting color information can even boost the final result. To experiment, at an early stage of my model architecture, I trained the model for 20 epochs using RGB, YUV color spaces and gray-scaled images. Likewise, the latest variation ended up with the best results. Finally, I also normalized the image data so that each pixel lies between -1 and 1. It prevents from numerical instabilities which can occur when the data resides far away from zero value.</p><p>Here is an example of a traffic sign image before and after grayscale and normalization. Histograms of both images are depicted as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/0*gUe_Mt19zmMhCfax.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/615/0*XDKXTQNw3p9Bu_ZR.png" /></figure><h3>Designing a Deep Neural Network model</h3><p>Now, it’s finally time to feed the data to the neural network. Choosing the <a href="https://hackernoon.com/tagged/network">network</a> architecture, tuning different parameters again and again is probably the most demanding task. There are no clear rules for the model optimization. Besides some proven rule of thumbs, our experience often plays a big role. Furthermore, when dealing with deep neural networks you have to wait for the results of each tested model a relatively long time. Of course, it depends on the available processing power. For this project I used <a href="https://www.cloudar.be/awsblog/how-to-use-aws-ec2-gpu-instances/">AWS EC2 instance</a> with Nvidia GPU inside, which increased the speed about 6 times comparing to results obtained on my laptop running on i7 CPU core.</p><p>To give a brief overview of the model complexity, I will name the most important model parameters. As this is a general look into the Traffic Sign Recognition project, I’m not going to explain them in details. These parameters, called sometimes “hyperparameters” are: <a href="https://medium.com/towards-data-science/epoch-vs-iterations-vs-batch-size-4dfb9c7ce9c9">batch size, number of epochs</a>, <a href="https://datascience.stackexchange.com/questions/410/choosing-a-learning-rate">learning rate</a>, <a href="https://chatbotslife.com/regularization-in-deep-learning-f649a45d6e0">loss regularization</a>, <a href="https://medium.com/@amarbudhiraja/https-medium-com-amarbudhiraja-learning-less-to-learn-better-dropout-in-deep-machine-learning-74334da4bfc5">dropout</a> rate or pooling type. They are often discussed and benchmarked by researchers. However, the choice of type and model structure itself is equally significant. Convolutional Neural Network, introduced very nicely <a href="https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets">here</a>, fits very well for our task. But there are many well-established sub-types of CNN like <a href="http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf">LeNet</a> (1990s), <a href="https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf">AlexNet</a> (2012), <a href="https://arxiv.org/abs/1409.4842">GoogLeNet </a>(2014) or <a href="http://www.robots.ox.ac.uk/~vgg/research/very_deep/">VGGNet</a> (2014). They are differing with number of neuron layers (model depth), connections between them, number of operations or parameters which are updated on each iteration. A good comparison of the most popular architectures can be found <a href="https://medium.com/towards-data-science/neural-network-architectures-156e5bad51ba">here</a>.</p><h4>Convolutional Neural Network</h4><p>Let’s discuss briefly the concept of convolutional neural networks. They are very successful in image recognition. The key part to understand, which distinguishes CNN from traditional neural networks, is the <strong>convolution</strong> operation. Having an image at the input, CNN scans it many times to look for certain <strong>features</strong>. This scanning (convolution) can be set with 2 main parameters: stride and padding type. As we see on below picture, process of the first convolution gives us a set of new frames, shown here in the second column (layer). Each frame contains an information about one feature and its presence in scanned image. Resulting frame will have larger values in places where a feature is strongly visible and lower values where there are no or little such features. Afterwards, the process is repeated for each of obtained frames for a chosen number of times. In this project I chose a classic <a href="http://yann.lecun.com/exdb/publis/pdf/lecun-98.pdf">LeNet</a> model which contains only two convolution layers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*eVft9WEk_9LofP74.png" /><figcaption>from <a href="https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner%27s-Guide-To-Understanding-Convolutional-Neural-Networks/">https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner%27s-Guide-To-Understanding-Convolutional-Neural-Networks/</a></figcaption></figure><p>The latter layer we are convolving, the more high-level features are being searched. It works similarly to human perception. To give an example, below is a very descriptive picture with features which are searched on different CNN layers. As you can see, the application of this model is face recognition. You may ask how the model knows which features to seek. If you construct the CNN from the beginning, searched features are random. Then, during training process, weights between neurons are being adjusted and slowly CNN starts to find such features which enable to meet predefined goal, i.e. to recognize successfully images from the training set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*20-cv2j2DBjSgWKM.jpg" /><figcaption>from <a href="https://devblogs.nvidia.com/parallelforall/deep-learning-nutshell-core-concepts/">from: https://devblogs.nvidia.com/parallelforall/deep-learning-nutshell-core-concepts/</a></figcaption></figure><p>Between described layers there are also <strong>pooling</strong> (sub-sampling) operations which reduce dimensions of resulted frames. Furthermore, after each convolution we apply a non-linear function (called <strong>ReLU</strong>) to the resulted frame to introduce non-linearity to the model.</p><p>Eventually, there are also <strong>fully connected layers</strong> at the end of the network. The last set of frames obtained from convolutional operations is flattened to get a one-dimensional vector of neurons. From this point we put a standard, fully-connected neural network. At the very end, for classification problems, there is a <strong>softmax</strong> layer. It transforms results of the model to probabilities of a correct guess of each class, here a traffic sign index.</p><p>Below, there is a summary of the model I chose and fine tuned with marked dimensions for each layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/457/0*AoSTQ4Mr0YJHrxGC.jpg" /></figure><h4>Tuning the model</h4><p>I followed a simple, iterative process of finding the best model architecture. After changing one of the model parameters, I ran only 20 epochs of the training and observed the validation error trying to set it on minimum level. It is very important to consider mainly validation error while tuning the model. Minimizing only the error based on training data can easily lead to unwanted model <a href="https://en.wikipedia.org/wiki/Overfitting">overfitting</a>.</p><p>Below, there are details of intermediate steps that I took and the corresponding validation accuracies after 20 epochs of training. Sometimes the differences between two given approaches seemed to be huge and it was hard to make a choice between them. But note that for each training procedure there is a random weight initialization which influences the final error. Especially when number of epochs is small. That’s why during final model tuning I used more than 20 epochs — about 100.</p><ul><li>Initial LeNet model, choosing input images color representation — 91 %</li><li>Input images normalization — ~91 %</li><li>Training set augmantation — 93 %</li><li>Learn rate optimization, from this stage I tested for 100 epochs — 95 %</li><li>Finding optimum image transformations during training set augmentation — 96 %</li><li>Trying different pool methods, trying dropout, choosing L2 loss, tuning learn rate again — 96.8</li></ul><p>My <strong>final model results</strong> were as follows:</p><ul><li>Training set accuracy of <strong>99.5 %</strong></li><li>Validation set accuracy of <strong>96.8 %</strong></li><li>Test set accuracy of <strong>94.6 %</strong></li></ul><p>I am quite satisfied with these results. Authors of the earlier referenced <a href="http://yann.lecun.com/exdb/publis/pdf/sermanet-ijcnn-11.pdf">paper</a> (Sermanet and LeCun) reach the level of accuracy equal to 99.17%. It is considered as being above human performance which is 98.81% !</p><h3>Testing the model on new images</h3><p>Finally, we would like to test our Traffic Sign Recognition system on completely unseen sign images. Surely, the accuracy obtained on the test set is also a very good indication of the model performance. But let’s find some new images which are not from our German Traffic Sign Dataset. Under images, there are model predictions, indication if the prediction is correct and model certainty.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K2T4_AwNt2GzC6kQ.jpg" /></figure><p>We can see that the results are really good. I also collected images from Polish roads in Gdansk and extracted some traffic signs to test my model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*j5L6IBSjguVrbP6j.jpg" /></figure><p>The results are far worse although I chose signs which are identical to German ones. One exception is a Yield sign which is yellow instead of white. So, here a positive surprise that the model predicted correctly. Unfortunately, there are two signs which are not recognized at all. So far I didn’t make more research why this happened.</p><h3>Conclusions</h3><p>I successfully implemented a Convolutional Neural Network to the Traffic Sign Recognition task. It was done using an open-source Tensorflow library for Python. I chose a popular and simple LeNet CNN architecture. I see the biggest room for improvement here. Many modern Deep <a href="https://hackernoon.com/tagged/learning">Learning</a> systems use more recent and more complicated architectures like GoogLeNet or ResNet. This comes in more computational cost, on the other hand. <a href="https://www.analyticsvidhya.com/blog/2017/08/10-advanced-deep-learning-architectures-data-scientists/">Here</a> you can find a brief, illustrative comparison of the most popular architectures. The most difficult part of the project was to fine tune a CNN model parameters. It was sometimes cumbersome as I was not sure in which direction I should go. But this is the art of Machine Learning. I researched similar projects and tried to bring some ideas into my model. The interesting part was also data augmentation with image rotation and changing brightness which was also advised by many people doing this project.</p><p>More details and the complete code can be found on <a href="https://github.com/tomaszkacmajor/CarND-Traffic-Sign-Classifier-P2">github</a>.</p><p><em>Originally published at </em><a href="http://tomaszkacmajor.pl/index.php/2017/10/15/traffic-sign-recognition-using-cnn/"><em>ProggBlogg</em></a><em>.</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fdde502%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2Fhackernoon%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href">https://medium.com/media/3c851dac986ab6dbb2d1aaa91205a8eb/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a1f90e8fb24" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/traffic-sign-recognition-using-convolutional-neural-network-8a1f90e8fb24">Traffic Sign Recognition using Convolutional Neural Network</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hough Lines Transform Explained]]></title>
            <link>https://medium.com/@tomasz.kacmajor/hough-lines-transform-explained-645feda072ab?source=rss-b86619ed92c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/645feda072ab</guid>
            <category><![CDATA[image-processing]]></category>
            <category><![CDATA[opencv]]></category>
            <category><![CDATA[self-driving-cars]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Tomasz Kacmajor]]></dc:creator>
            <pubDate>Mon, 05 Jun 2017 19:26:25 GMT</pubDate>
            <atom:updated>2020-11-15T08:32:22.330Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*Bu7d2Eg0h_6dbmJz.jpg" /></figure><p>Hough Lines Transform is the key method used in the previous <a href="http://tomaszkacmajor.pl/index.php/2017/05/17/lane-lines-detection/">project</a> where lane lines are detected. It is very helpful in many Computer Vision applications. The original form of Hough Transform aimed to <strong>identify straight lines</strong>. And that’s what I’m going to explain today. Furthermore, this technique was later generalized to detect also other shapes like circles, ellipses etc. <a href="https://en.wikipedia.org/wiki/Generalised_Hough_transform">[1]</a>.</p><h4>The goal of the method, image preparation</h4><p>To use Hough Lines Transform, processed image should be binary. But we would like to search for the straight lines on an original, color image. Therefore, probably the most common solution is to firstly grayscale the image and then to detect edges. Such mask of edges can be then fetched to the Hough Lines method which should output a set of straight lines found on an image.</p><h4>Straight lines representations</h4><p>As we learned from quite early school classes, the straight line can be represented by two parameters. The simplest and most widely used pair of parameters is <em>(a, b)</em> which correspond to slope and intercept. The line is then described as: <em>y = a x + b</em></p><p>Let’s forget for a while about these parameters. We can also unambiguously describe the line using the pair <em>(ρ, θ)</em> in polar system. The first parameter, <em>ρ,</em> is the shortest distance from the origin to the line (approaching the line perpendicularly). The second, <em>θ</em>, is the angle between x-axis and the distance line. One of the benefits of such representation is that we can describe vertical lines by <em>ρ</em> and <em>θ</em> which is impossible by using only <em>(a, b)</em> parameters in Cartesian system.</p><figure><img alt="Line" src="https://cdn-images-1.medium.com/max/412/0*VAXB92TCIKUfTBhR.jpg" /></figure><p>For a given line, we can determine specific <em>ρ</em> and <em>θ</em>. Then, the following equation is satisfied for each <em>x</em>,<em> y</em> point belonging to this line:</p><p><em>ρ</em> = <em>x</em> cos(<em>θ</em>) + <em>y</em> sin(<em>θ</em>)</p><h4>Mapping from Image space to Hough space</h4><p>Let’s draw a line on an image space again. As we already know, it is represented by some <em>ρ</em> and <em>θ</em>. So, we can draw such point in <em>(ρ, θ)</em> coordinates which will be later called a Hough space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/0*M6izxOsRBBKlkZzq.jpg" /></figure><p>Now, in the image space, we are drawing other lines which are intersecting at one common point. Let’s see what points will be produced in Hough space which are corresponding to these lines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/0*JT-hhPkp-Tx4ywtu.jpg" /></figure><p>It turns out that these points in <em>(ρ, θ)</em> space are forming a sinusoid. Drawing infinite number of additional lines intersecting at this one point would result in a continuous sinusoid in Hough space. So, maybe, we can say that a <strong>point</strong> in image space results in a <strong>sinusoid</strong> in Hough space? Let’s recall the equation <em>ρ</em> = <em>x</em> cos(<em>θ</em>) + <em>y</em> sin(<em>θ</em>). Indeed, for fixed (<em>x</em>,<em> y</em>) parameters representing point in image space and sliding through all possible values of <em>θ</em> in some range, we obtain <em>ρ</em> values which form a sinusoid.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/0*pJDI5sW6oEBDQQqd.jpg" /></figure><p>To sum up, we observed following relations between <strong>Image space </strong>and <strong>Hough space:</strong></p><p>Straight line → Point</p><p>Point → Sinusoid</p><h4>Finding Hough Lines</h4><p>Finally, maybe the most interesting effect. If we draw points which form a line in the image space, we will obtain a bunch of sinusoids in the Hough space. But, magically, they are intersecting at exactly one point!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/0*VPVsLApWiEayRGdQ.jpg" /></figure><p>It means that, to identify candidates for being a straight line, we should seek for intersections in Hough space. Below, you can see 2 lines and corresponding image in Hough space. Sure enough, there are 2 main intersections highlighted here for convenience. They are representatives (by <em>ρ</em> and <em>θ</em> parameters) of the straight lines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KoAmeuoTu9Uz0eNW.png" /></figure><p>from <a href="https://en.wikipedia.org/wiki/Hough_transform">en.wikipedia.org/wiki/Hough_transform</a></p><h4>Hough Lines parameters</h4><p>In real application, for example using OpenCV library, the Hough space of an image is divided into uniform clusters. The grid is defined by parameters which can be called <em>rho resolution</em> and <em>theta resolution</em>. They often equals 1 pixel and 1 degree respectively. We sweep through all cells in the grid and count how many votes there are for a certain line. If there are more votes than a given threshold, we claim that the straight line has been found and it’s described by <em>ρ</em> and <em>θ</em> parameters from the considered cluster.</p><h4>Implementation in Python</h4><p>OK, now we are ready to find Hough Lines in real image using OpenCV and Python. After image loading I perform grayscaling, blurring and edge detection. Then, OpenCV <em>cv2.HoughLines</em> function is called to obtain set of detected lines. Next, the results are presented. Note that function <em>cv2.HoughLines</em> requires 3 parameters which were discussed before. They were chosen experimentally for the given image example. It’s good to play with these parameters. E.g. if threshold is lower, more lines will be found and vice versa.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6b0a28efb6c403a99bc33a7135b11a08/href">https://medium.com/media/6b0a28efb6c403a99bc33a7135b11a08/href</a></iframe><p>Below, there are some helper functions. I attach them for a better understanding of the entire code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/005f1e5e612b11f569fcd20b269c2af2/href">https://medium.com/media/005f1e5e612b11f569fcd20b269c2af2/href</a></iframe><h4>Example results</h4><p>Results can be seen below. If there are too much lines in our opinion, or too few of them, we should adjust <em>rho resolution</em>, <em>theta resolution</em> and <em>threshold</em> parameters. But remember also about tuning parameters which are used in blurring and edge detection. Whole pipeline should be revised to fit your needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*sNJApeyG47u88RX5.jpg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*lDKnd01MkMDt7O29J5kTkg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*Nf9YIUaBSpRPP6Vi.jpg" /></figure><p>It’s worth noting that in OpenCV there exists another version of the function to find Hough Lines. It’s named <em>HoughLines</em><strong><em>P</em></strong>. P suffix stands for <em>probabilistic</em> here. It has more efficient implementation and the function outputs extremes of detected lines <em>(x0, y0, x1, y1)</em> which can be very useful. Below, there is an example of the same image with the straight lines found using <em>HoughLinesP</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*xh2St-2WT5OUTBkq.jpg" /></figure><h4>References</h4><p>1. Generalized Hough Transform on <a href="https://en.wikipedia.org/wiki/Generalised_Hough_transform">Wikipedia</a><br> 2. Hough Transform on <a href="https://en.wikipedia.org/wiki/Hough_transform">Wikipedia</a><br> 3. OpenCV 2.4 <a href="http://docs.opencv.org/2.4/doc/tutorials/imgproc/imgtrans/hough_lines/hough_lines.html">documentation</a> — Hough Lines Transform<br> 4. OpenCV 3.1 <a href="http://docs.opencv.org/3.1.0/d6/d10/tutorial_py_houghlines.html&quot;">documentation</a> — Hough Lines Transform</p><p><em>Originally published at </em><a href="http://tomaszkacmajor.pl/index.php/2017/06/05/hough-lines-transform-explained/"><em>ProggBlogg</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=645feda072ab" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[First step to stay on track — lane lines detection]]></title>
            <link>https://medium.com/@tomasz.kacmajor/first-step-to-stay-on-track-lane-lines-detection-86a7fb25e0d9?source=rss-b86619ed92c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/86a7fb25e0d9</guid>
            <category><![CDATA[opencv]]></category>
            <category><![CDATA[self-driving-cars]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[image-processing]]></category>
            <dc:creator><![CDATA[Tomasz Kacmajor]]></dc:creator>
            <pubDate>Wed, 17 May 2017 16:47:14 GMT</pubDate>
            <atom:updated>2017-06-08T21:37:23.136Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fpmuyOx1N8TJAJiF.jpg" /></figure><p>Although Deep Neural Networks play bigger and bigger role in scene recognition, classic Computer Vision methods are still valid and are applied to problems from autonomous cars industry. I’m going to present how to perform lane lines detection using OpenCV library and Python language using image processing techniques. It’s the first project in the <a href="http://tomaszkacmajor.pl/index.php/2017/05/07/self-driving-cars-in-python/">series about Self-Driving Cars cars</a>.</p><p>Today, in the middle of 2017, we are not surprised anymore that most of high-end cars are equipped with systems like <strong>lane keeping assistance</strong> or <strong>lane changing support</strong>. Lane keep assist can save us from getting off the track by providing feedback like steering wheel vibrations. Lane change assist is designed to confirm that the lane change operation is made safely by monitoring the “blind spot” at the same time. These features are achieved by mounting a camera e.g. beside the rear mirror and constant analysis of incoming images. This can be done by well-known image processing techniques like <strong>Canny</strong> method for edge detection or <a href="http://tomaszkacmajor.pl/index.php/2017/06/05/hough-lines-transform-explained/"><strong>Hough Lines transform</strong></a> for deriving straight lines from the image.</p><p>The goal of the first assignment from <a href="https://www.udacity.com/drive">Udacity course</a> “Self-Driving Cars Engineer” is to find lane lines from either single images or video stream using OpenCV library and Python. I’m going to describe whole pipeline of the system, comment pros and cons of the approach and present the final output. The extended version of this post, with bits of code per each processing step, can be found on my <a href="https://github.com/tomaszkacmajor/CarND-LaneLines-P1">github</a>.</p><h4>For those who are eager for the final result — you can see how it works in the video below.</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F-kU6nh0G6jY%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D-kU6nh0G6jY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F-kU6nh0G6jY%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/933eef9cbe22f0c10de912d03e3a7d28/href">https://medium.com/media/933eef9cbe22f0c10de912d03e3a7d28/href</a></iframe><h4>Lane lines detection pipeline</h4><p>My pipeline consists of 10 steps:</p><ol><li><a href="#reading-image-or-video-frame">Reading image or video frame</a></li><li><a href="#filtering-white-and-yellow-colors">Filtering white and yellow colors</a></li><li><a href="#conversion-to-grayscale">Conversion to grayscale</a></li><li><a href="#gaussian-blurring">Gaussian blurring</a></li><li><a href="#edge-detection">Edge detection</a></li><li><a href="#region-of-interest-definition">Region of interest definition</a></li><li><a href="#hough-lines-detection">Hough lines detection</a></li><li><a href="#filtering-hough-lines">Filtering Hough lines</a></li><li><a href="#averaging-line-segments">Averaging line segments</a></li><li><a href="#applying-moving-average-on-final-lines">Applying moving average on final lines</a></li></ol><h4>Reading image or video frame</h4><p>Below, there are 3 examples of loaded images. Later, after each step, intermediate results will be shown for these samples images. The third image is the most demanding for processing as there are shadows and contrasts between yellow line and the road is very small.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oOs2wrWFHJSbiLz7.jpg" /></figure><h4>Filtering white and yellow colors</h4><p>This step wouldn’t be necessary for the first two easier images. In the third example, however, proceeding directly to the next step (gray scale conversion) would produce very similar gray colors for the yellow lane and the bright road. We would like to differentiate these two objects somehow. Thus the idea of initial filtering of 2 key colors which are the main components of the road lanes. Firstly, the image is converted to <a href="https://en.wikipedia.org/wiki/HSL_and_HSV">HSL color space</a>. HSL (Hue, Saturation, Lightness) color space concept is based on human vision color perception. That is why it’s easier to distinguish desired colors (yellow and white) than in RGB space even if there are shadows on the image.</p><p>For extraction of white color I filtered only high lightness from the “L” component of HSL color space. For yellow lanes I chose Hue to equal more or less 30 to select yellow color and Ii filtered Saturation to be quite high. Below, there are test images after such filtering.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GBdltXlVAdlwjAos.jpg" /></figure><h4>Conversion to grayscale</h4><p>As in many computer vision applications, the image is converted to grayscale. It’s mainly for the simplicity and speed of further operations. For instance, edge detectors find big gradients between adjacent pixels. So, it will be easier to compare pixels only in one dimension (grayscale) than in RGB or HSL color spaces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*O4OTt3knkQ5n10Um.jpg" /></figure><h4>Gaussian blurring</h4><p>To supress noise and spurious gradients <a href="https://en.wikipedia.org/wiki/Gaussian_blur">Gaussian smoothing</a> is applied. Here, it’s again preparation for edge detection step. Borders between lane and road can be not so smooth, so we don’t want the edge detector to classify such regions as additional lines. The size of smoothing kernel defines how blurred the output is and how much time such operation takes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*u55yqltGi4uKzMwV.jpg" /></figure><h4>Edge detection</h4><p>To detect edges, let’s use popular <a href="http://docs.opencv.org/2.4/doc/tutorials/imgproc/imgtrans/canny_detector/canny_detector.html">Canny</a> method. It’s called with 2 parameters: low and high thresholds which should be found by trial and error. According to the OpenCV documentation:</p><ul><li>If a pixel gradient is higher than the upper threshold, the pixel is accepted as an edge</li><li>If a pixel gradient value is below the lower threshold, then it is rejected.</li><li>If the pixel gradient is between the two thresholds, then it will be accepted only if it is connected to a pixel that is above the upper threshold.</li></ul><p>Canny recommended a upper:lower ratio between 2:1 and 3:1. I chose values of 80 and 40. Below, there are outputs of this operation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZQ7l_WwoAXycYA1u.jpg" /></figure><h4>Region of interest definition</h4><p>To filter out unnecessary objects in the image, the region of interest is defined. Such mask (here it’s trapezoid) is then applied to the working image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/937/0*fAfBGpPsRy6QXhFc.jpg" /></figure><p>This is the result of such masking:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vRxIXkn6HZMnlw5H.jpg" /></figure><h4>Hough lines detection</h4><p>Now, having edges detected in our interest area, all straight lines need to be identified. This is done by Hough transform explained in another post <a href="http://tomaszkacmajor.pl/index.php/2017/06/05/hough-lines-transform-explained/">here</a>. This operation has quite many parameters which need to be tuned experimentally. Speaking at high level, they define how long or how “straight” the sequence of pixels should be to be classified as one line. There is a nice example in OpenCV <a href="http://docs.opencv.org/2.4.13.2/modules/imgproc/doc/feature_detection.html">document</a> about feature extraction showing the result of Hough Line transform on exemplary image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/867/0*1b6VYvotZqa8sA5c.png" /></figure><p>Below, there are our tested images with found Hough lines plotted in green.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Xx91-dQXCiAzXDcq.jpg" /></figure><h4>Filtering Hough lines</h4><p>As we can see above, some line segments are unwanted. For example, small horizontal lines or some lines appearing on cars which are inside the region of interest. Therefore, for each Hough line we calculate a slope parameter. After some experimentation, only lines with slopes between 17 and 56 degrees of slope were left for further analysis.</p><p>Below, there are only filtered Hough lines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*f3zNMhcv_p51D2nV.jpg" /></figure><h4>Averaging line segments</h4><p>All found Hough lines should be right now averaged/extrapolated to produce only two lines representing lanes. The first task is to divide lines into 2 groups (left and right — deduced from the slope sign). Then, one can use best linear fit for the points representing line segments or take an average from these lines. I decided to apply weighted average to calculate resulting slopes and intercepts. Here, lenghts of line segments serve as weights. The longer segment is, the more influence it has on the results. In addition, to amplify the importance of the segment length, the weight is calculated as square the lenght parameter.</p><p>The final output of the pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RXXHRn63eGKJ-shR.jpg" /></figure><h4>Applying moving average on final lines</h4><p>While running the pipeline on the video stream, we can observe that the lines are flickering. To avoid it, we can apply <a href="https://en.wikipedia.org/wiki/Moving_average">cumulative moving average</a> of the line parameters. For each frame it averages last <em>n</em> results including current one. <em>Cumulative</em> version of this method applies bigger weights for more recent results. By having in memory the last averaged frame we can also use it in case when no line is found in a frame by some mistake.</p><h4>Results and possible improvements</h4><p>The pipeline successfully performs lane lines detection. It is smooth and stable even in complex scenes with shadows.</p><p>However, it detects only straight lines which maybe can be overcome by using some kind of higher order polynomial fit to handle curvy lanes. Also, what would happen when another car appears in our region of interest? It could produce some lines that could be identified as lanes. We should probably detect such cars at the same time to ensure that such object is definitely not a lane. What if we are driving from/up the slope? Then, predefined region of interest could be not valid anymore. I wonder also what happens when some white/yellow flat signs are marked on the road. Such special cases along with driving during the night should be tested and some additional features should be applied to avoid wrong lane lines detection.</p><p>More details and the complete code can be found on <a href="https://github.com/tomaszkacmajor/CarND-LaneLines-P1">github</a>.</p><p><em>Originally published at </em><a href="http://tomaszkacmajor.pl/index.php/2017/05/17/lane-lines-detection/"><em>ProggBlogg</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86a7fb25e0d9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>