<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sofiane Sahir on Medium]]></title>
        <description><![CDATA[Stories by Sofiane Sahir on Medium]]></description>
        <link>https://medium.com/@sahir.sofiane?source=rss-4c917479ce84------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*jFbL8nL8C11uTYsYD5n6Ug.jpeg</url>
            <title>Stories by Sofiane Sahir on Medium</title>
            <link>https://medium.com/@sahir.sofiane?source=rss-4c917479ce84------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 20:20:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sahir.sofiane/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Canny Edge Detection Step by Step in Python — Computer Vision]]></title>
            <link>https://medium.com/data-science/canny-edge-detection-step-by-step-in-python-computer-vision-b49c3a2d8123?source=rss-4c917479ce84------2</link>
            <guid isPermaLink="false">https://medium.com/p/b49c3a2d8123</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[edge-detection]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[image-processing]]></category>
            <dc:creator><![CDATA[Sofiane Sahir]]></dc:creator>
            <pubDate>Fri, 25 Jan 2019 13:52:09 GMT</pubDate>
            <atom:updated>2019-01-27T14:31:33.405Z</atom:updated>
            <content:encoded><![CDATA[<h3>Canny Edge Detection Step by Step in Python — Computer Vision</h3><p>When it comes to image classification, the human eye has the incredible ability to process an image in a couple of milliseconds, and to determine what it is about (label). It is so amazing that it can do it whether it is a drawing or a picture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YTzWlyYyoU0eLFC0gMykUg.png" /><figcaption>Drawing of a car (Left) — Real car picture (Right): The human eye is able to classify both.</figcaption></figure><p>The idea today is to build an algorithm that can sketch the edges of any object present on a picture, using the Canny edge detection algorithm.</p><p>First of all, let’s describe what is the Canny Edge Detector:</p><blockquote>The <strong>Canny edge detector</strong> is an <a href="https://en.wikipedia.org/wiki/Edge_detection">edge detection</a> operator that uses a multi-stage <a href="https://en.wikipedia.org/wiki/Algorithm">algorithm</a> to detect a wide range of edges in images. It was developed by <a href="https://en.wikipedia.org/wiki/John_F._Canny">John F. Canny</a> in 1986. Canny also produced a <em>computational theory of edge detection</em> explaining why the technique works. (Wikipedia)</blockquote><p>The Canny edge detection algorithm is composed of 5 steps:</p><ol><li>Noise reduction;</li><li>Gradient calculation;</li><li>Non-maximum suppression;</li><li>Double threshold;</li><li>Edge Tracking by Hysteresis.</li></ol><p>After applying these steps, you will be able to get the following result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/1*XAgKINgc2c2gNa2nV3zbNQ.png" /><figcaption>Original image on the left — Processed image on the right</figcaption></figure><p>One last important thing to mention, is that the algorithm is based on grayscale pictures. Therefore, the pre-requisite is to convert the image to grayscale before following the above-mentioned steps.</p><h3>Noise Reduction</h3><p>Since the mathematics involved behind the scene are mainly based on derivatives (cf. Step 2: Gradient calculation), edge detection results are highly sensitive to image noise.</p><p>One way to get rid of the noise on the image, is by applying Gaussian blur to smooth it. To do so, image convolution technique is applied with a Gaussian Kernel (3x3, 5x5, 7x7 etc…). The kernel size depends on the expected blurring effect. Basically, the smallest the kernel, the less visible is the blur. In our example, we will use a 5 by 5 Gaussian kernel.</p><p>The equation for a Gaussian filter kernel of size (2<em>k</em>+1)×(2<em>k</em>+1) is given by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/1*YpLYVBomcYNNbwncG5iP9Q.png" /><figcaption>Gaussian filter kernel equation</figcaption></figure><p>Python code to generate the Gaussian 5x5 kernel:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a3393dde6f5d0b12205a2203f5e1d701/href">https://medium.com/media/a3393dde6f5d0b12205a2203f5e1d701/href</a></iframe><p>After applying the Gaussian blur, we get the following result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*-P1q_Io0i1kElOw4q3GAzw.png" /><figcaption>Original image (left) — Blurred image with a Gaussian filter (sigma=1.4 and kernel size of 5x5)</figcaption></figure><h3>Gradient Calculation</h3><p>The Gradient calculation step detects the edge intensity and direction by calculating the gradient of the image using edge detection operators.</p><p>Edges correspond to a change of pixels’ intensity. To detect it, the easiest way is to apply filters that highlight this intensity change in both directions: horizontal (x) and vertical (y)</p><p>When the image is smoothed, the derivatives <strong><em>Ix</em></strong> and <strong><em>Iy</em></strong> w.r.t. <strong><em>x</em></strong> and <strong><em>y</em></strong> are calculated. It can be implemented by convolving <strong><em>I</em></strong> with Sobel kernels<strong><em> Kx</em></strong> and <strong><em>Ky</em></strong>, respectively:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/276/1*r5Y22pLJjDMl19tcqXseZQ.png" /><figcaption>Sobel filters for both direction (horizontal and vertical)</figcaption></figure><p>Then, the magnitude <strong><em>G</em></strong> and the slope <strong><em>θ</em></strong> of the gradient are calculated as follow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/147/1*HmperNZ1AXXNONDbrsJNAg.png" /><figcaption>Gradient intensity and Edge direction</figcaption></figure><p>Below is how the Sobel filters are applied to the image, and how to get both intensity and edge direction matrices:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/40e66c0075296a3aa1b51db9e53a5eb0/href">https://medium.com/media/40e66c0075296a3aa1b51db9e53a5eb0/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*ZCyKWsmDoj6V-dNwKlKxyA.png" /><figcaption>Blurred image (left) — Gradient intensity (right)</figcaption></figure><p>The result is almost the expected one, but we can see that some of the edges are thick and others are thin. Non-Max Suppression step will help us mitigate the thick ones.</p><p>Moreover, the gradient intensity level is between 0 and 255 which is not uniform. The edges on the final result should have the same intensity (i-e. white pixel = 255).</p><h3>Non-Maximum Suppression</h3><p>Ideally, the final image should have thin edges. Thus, we must perform non-maximum suppression to thin out the edges.</p><p>The principle is simple: the algorithm goes through all the points on the gradient intensity matrix and finds the pixels with the maximum value in the edge directions.</p><p>Let’s take an easy example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*CWrXNSbe7s4qSFr5vylyvQ.png" /></figure><p>The upper left corner red box present on the above image, represents an intensity pixel of the Gradient Intensity matrix being processed. The corresponding edge direction is represented by the orange arrow with an angle of -pi radians (+/-180 degrees).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/1*K-gnZg4_VPk57Xs0XflIrg.png" /><figcaption>Focus on the upper left corner red box pixel</figcaption></figure><p>The edge direction is the orange dotted line (horizontal from left to right). The purpose of the algorithm is to check if the pixels on the same direction are more or less intense than the ones being processed. In the example above, the pixel <strong><em>(i, j) </em></strong>is being processed, and the pixels on the same direction are highlighted in blue <strong><em>(i, j-1)</em></strong> and <strong><em>(i, j+1). </em></strong>If one those two pixels are more intense than the one being processed, then only the more intense one is kept. Pixel <strong><em>(i, j-1) </em></strong>seems to be more intense, because it is white (value of 255). Hence, the intensity value of the current pixel <strong><em>(i, j)</em></strong> is set to 0. If there are no pixels in the edge direction having more intense values, then the value of the current pixel is kept.</p><p>Let’s now focus on another example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/373/1*QjoPwEgQ6NOynOwxc4847A.png" /></figure><p>In this case the direction is the orange dotted diagonal line. Therefore, the most intense pixel in this direction is the pixel <strong><em>(i-1, j+1).</em></strong></p><p>Let’s sum this up. Each pixel has 2 main criteria (edge direction in radians, and pixel intensity (between 0–255)). Based on these inputs the non-max-suppression steps are:</p><ul><li>Create a matrix initialized to 0 of the same size of the original gradient intensity matrix;</li><li>Identify the edge direction based on the angle value from the angle matrix;</li><li>Check if the pixel in the same direction has a higher intensity than the pixel that is currently processed;</li><li>Return the image processed with the non-max suppression algorithm.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/30a3a4c777614b3ae9a43afd1632c341/href">https://medium.com/media/30a3a4c777614b3ae9a43afd1632c341/href</a></iframe><p>The result is the same image with thinner edges. We can however still notice some variation regarding the edges’ intensity: some pixels seem to be brighter than others, and we will try to cover this shortcoming with the two final steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*LmBYILsRp_KlVpNweqHh0g.png" /><figcaption>Result of the non-max suppression.</figcaption></figure><h3>Double threshold</h3><p>The double threshold step aims at identifying 3 kinds of pixels: strong, weak, and non-relevant:</p><ul><li>Strong pixels are pixels that have an intensity so high that we are sure they contribute to the final edge.</li><li>Weak pixels are pixels that have an intensity value that is not enough to be considered as strong ones, but yet not small enough to be considered as non-relevant for the edge detection.</li><li>Other pixels are considered as non-relevant for the edge.</li></ul><p>Now you can see what the double thresholds holds for:</p><ul><li>High threshold is used to identify the strong pixels (intensity higher than the high threshold)</li><li>Low threshold is used to identify the non-relevant pixels (intensity lower than the low threshold)</li><li>All pixels having intensity between both thresholds are flagged as weak and the Hysteresis mechanism (next step) will help us identify the ones that could be considered as strong and the ones that are considered as non-relevant.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2844c7a22607e25102f9bef6e29e61fc/href">https://medium.com/media/2844c7a22607e25102f9bef6e29e61fc/href</a></iframe><p>The result of this step is an image with only 2 pixel intensity values (strong and weak):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*FF6b8FJ2oppREoh9T-hdfA.png" /><figcaption>Non-Max Suppression image (left) — Threshold result (right): weak pixels in gray and strong ones in white.</figcaption></figure><h3>Edge Tracking by Hysteresis</h3><p>Based on the threshold results, the hysteresis consists of transforming weak pixels into strong ones, if and only if at least one of the pixels around the one being processed is a strong one, as described below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*jnqS5hbRwAmU-sgK552Mgg.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/078f683e22947dca15385730cf30fdcb/href">https://medium.com/media/078f683e22947dca15385730cf30fdcb/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*pStG0ObTAddm46mdHxY5zA.png" /><figcaption>Results of hysteresis process</figcaption></figure><p>All the code used is available in the following Git Repository</p><p><a href="https://github.com/FienSoP/canny_edge_detector">FienSoP/canny_edge_detector</a></p><h4>Bibliography</h4><ul><li><a href="https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_canny/py_canny.html">https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_imgproc/py_canny/py_canny.html</a></li><li><a href="https://homepages.inf.ed.ac.uk/rbf/HIPR2/canny.htm">https://homepages.inf.ed.ac.uk/rbf/HIPR2/canny.htm</a></li><li><a href="http://justin-liang.com/tutorials/canny/">http://justin-liang.com/tutorials/canny/</a></li><li><a href="https://en.wikipedia.org/wiki/Canny_edge_detector">https://en.wikipedia.org/wiki/Canny_edge_detector</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b49c3a2d8123" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/canny-edge-detection-step-by-step-in-python-computer-vision-b49c3a2d8123">Canny Edge Detection Step by Step in Python — Computer Vision</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data science and Machine learning: Competitions vs Real world projects]]></title>
            <link>https://medium.com/@sahir.sofiane/data-science-and-machine-learning-competitions-vs-real-world-projects-7d97e055f7d1?source=rss-4c917479ce84------2</link>
            <guid isPermaLink="false">https://medium.com/p/7d97e055f7d1</guid>
            <category><![CDATA[information-technology]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[project-management]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Sofiane Sahir]]></dc:creator>
            <pubDate>Wed, 29 Aug 2018 19:46:13 GMT</pubDate>
            <atom:updated>2019-01-27T14:15:36.885Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SzgDOTyj9zLNfo2r1VJ8QQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/RbwoCABWQ9w?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">rawpixel</a> on <a href="https://unsplash.com/search/photos/data?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Who never heard about Kaggle competition website? Anyone interested in Data Science or Machine Learning area knows Kaggle or similar platforms.</p><p>Well, if you have just joined this amazing community and you don’t know what Kaggle is let me summarize it for you: it is a platform that gives to companies an opportunity to let the world’s best statisticians, data scientists and machine learning engineers to work on a huge amount of their data in order to solve a specific data problem definition (e.g. Computer vision algorithms, NLP issues, Text classification etc…).</p><p>The problem needs to be solved as a contest and the participants are all competitors fighting either for the contest reward (up to §1.000.000) or fame and glory.</p><p>These platforms are so popular and famous that the contest final ranking (called the Leaderboard) is proudly raised and mentioned during job interviews. But here is the thing: going through a competition and working as a data scientist or ML engineer have completely two different approaches. And this is what I am going to try to emphasize in this article.</p><h3><strong><em>Steps of a real world machine learning project</em></strong></h3><p>A machine learning project tackles an issue which is by definition very complex to solve. It is divided in several phases and each one of them is important and requires a specific attention.</p><p>Let’s consider a SPAM detection system as an example and let’s focus on each basic steps of its implementation.</p><ol><li><strong>Understand the business needs</strong></li></ol><p>First of all and before going deep into any data science or artificial intelligence considerations, we need to ask ourselves right questions about the business to understand its basis :</p><ul><li>Why developing such a system?</li><li>How could it help our customer?</li><li>What purpose does the business want to reach with such a system?</li><li>Etc…</li></ul><p><strong>2. Formalize the system problem definition</strong></p><p>Then, the system has to be broken down in functional constituents:</p><ul><li>What is the definition of a SPAM?</li><li>What do we really want to forecast thanks to the system to be developed</li><li>Etc…</li></ul><p><strong>3. Data collection</strong></p><p>During this step, we need to figure out what kind of data we need to build up such a system. In our case, emails could be collected or generated. Moreover, we need to know how to label them correctly as a SPAM or NO SPAM.</p><p>Sometimes it is really tricky to get the data collected/generated and correctly labelled and it could ruin the whole performance of the system you try to build. If the inputs are wrong then you can be pretty sure that the output won’t be the one expected as well.</p><p><strong>4. Data pre-processing</strong></p><p>Then, we need to find out a way to pre-process the data if we need to. Most of the time, data needs to be pre-processed before using it as input for algorithms. If we go back to our SPAM system, we may need to “clean” emails by converting all words letters in lowercase or by applying Natural Language Processing (NLP) principles in terms of data pre-processing (such as Lemmatization or Stemming techniques etc…)</p><p><strong>5. Modeling</strong></p><p>During this phase, we build the model(s) and once again the right questions need to be addressed :</p><ul><li>What models family is the most appropriated to perform this kind of task?</li><li>How to measure my model(s) performance?</li><li>How to choose the best model among the ones to be implemented?</li><li>Etc…</li></ul><p><strong>6. Model testing</strong></p><p>Once the best model is selected, it has to be tested in the “real world” to check if it is really efficient when it comes to a real scenario. Doing this helps to validate the expected behavior and to confirm that no bias has been introduced during the model learning phase (cf. <a href="https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff">Bias vs Variance</a>). After that, we should be able to answer the following questions:</p><ul><li>Does the model really block SPAMs?</li><li>How many times the system blocks an email which is not a SPAM?</li><li>Etc…</li></ul><p><strong>7. Production</strong></p><p>If all goes well up to this point then the model can be pushed in production to final users. And then it follows the usual steps of a regular deployment project.</p><p>One may believe that we are all done, but in fact it is just the beginning:</p><ul><li>The system efficiency in production has to be monitored</li><li>The system needs to keep learning based on new data</li><li>From time to time, we will have to make sure that the system implementation still fits the business requirements which can evolve with time</li><li>… And finally go through the whole process again and again.</li></ul><h3><strong><em>Steps of a Data Science competition</em></strong></h3><p>We have to admit that it is much less complex when it comes to a data science competition:</p><ul><li>business requirements are already understood</li><li>problem is already formalized</li><li>data has been already collected and provided to competitors</li></ul><p>Even the target metrics used to evaluate and compare competitors models efficiency are provided. Then, all we need to focus on as a competitor is to:</p><ul><li>pre-process the data</li><li>choose the model(s) to implement</li><li>select the best one(s) (i.e. the one(s) which optimize the target metrics)</li></ul><p>Sometimes, it requires more than that and we need to dig into business principles to get more intuition about the data and problem to solve. Usually, it ends up to the generation of new features or even to magic features (the Holy Graal for competitors).</p><p>Some competitions allow the usage of external data in addition to the one provided. In such cases, the data collection step is crucial to win the competition.</p><p>Bottom line, there are less steps to tackle when it comes to a data science competition compared to a « real world » project because a lot of work has already been done by the competition holders themselves. Plus, those tasks are usually the most complex ones (target metric definition, problem formalization etc…). Moreover, deployment matters are out of scope in a competition. The only thing competitors need to focus on is the model to implement and the target metric value to optimize to get the best score in the Leaderboard.</p><p>Speed computation, model complexity or even memory consumption are definitely not something to care about for a competitor as long as the results are reproducible. Plus, it is not unusual in a competition that the best scores are achieved based on a Data Leakage.</p><h3><strong><em>Let’s summarize</em></strong></h3><ul><li><strong>Real world machine learning projects are by definition very complex to manage and are divided in several steps.</strong></li><li><strong>Competitions mainly focus on the implementation modeling and data exploration phases and do not address business needs formalization, deployment and testing issues.</strong></li><li><strong>Competitions are an extraordinary tool to learn for curious people interested in data science and machine learning. But also a place for most experienced one to compete again others.</strong></li></ul><p><em>This article is based on the Coursera course “</em><a href="https://www.coursera.org/learn/competitive-data-science/home/welcome"><em>How to win a Data Science Competition</em></a><em>” from the </em><a href="https://www.coursera.org/specializations/aml"><em>Advanced Machine Learning</em></a><em> </em>specialization.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d97e055f7d1" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>