<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ömer Faruk Tuna on Medium]]></title>
        <description><![CDATA[Stories by Ömer Faruk Tuna on Medium]]></description>
        <link>https://medium.com/@merfaruktuna?source=rss-a915aefc856e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*V9qaYADdi1xDgoxDgkAZpw.jpeg</url>
            <title>Stories by Ömer Faruk Tuna on Medium</title>
            <link>https://medium.com/@merfaruktuna?source=rss-a915aefc856e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 22:33:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@merfaruktuna/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Güvenilir Yapay Zeka]]></title>
            <link>https://medium.com/@merfaruktuna/g%C3%BCvenilir-yapay-zeka-d8dd59e70a6?source=rss-a915aefc856e------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8dd59e70a6</guid>
            <dc:creator><![CDATA[Ömer Faruk Tuna]]></dc:creator>
            <pubDate>Sun, 09 Apr 2023 18:52:57 GMT</pubDate>
            <atom:updated>2023-04-09T18:52:57.982Z</atom:updated>
            <content:encoded><![CDATA[<p>Yapay zekanın, modern toplum yaşantımıza devrim niteliğinde bir etki getirdiğini söylersek yanılmayız sanırım. Günümüzde, yapay zekanın hayatımızın birçok alanında yeri doldurulamaz bir rol oynadığını görmekteyiz. Yaşamlarımızın yapay zeka uygulamalarıyla etkileşim içinde olduğu alanlar her geçen gün artmakta. Ancak yakın geçmişimizde, yapay zeka sistemlerinin bu hızlı ve etkileyici gelişimlerinin yanı sıra güvenilmez yanlarına da tanık olduk. Örnek vermek gerekirse, güvenlik açısından kritik roller üstlenen yapay zeka tabanlı sistemlerin art niyetli saldırılara karşı pek de dayanıklı olmadığı gösterildi. Otonom araçlardaki derin öğrenme tabanlı görüntü tanıma sistemlerinin, kötü niyetli saldırganlar tarafından değiştirilen yol işaretlerini ayırt edemeyerek yolcu güvenliği için büyük tehdit oluşturabildiği fark edildi. Başka bir örnekte, yapay zeka algoritmalarının önyargılı ve adaletsiz kararlar alabildiği saptandı. Yapay zeka tabanlı sohbet botlarının, kullanıcılarını rahatsız eden uygunsuz, ırkçı ve cinsiyetçi içerik üretebildiğini gördük. Dahası, bu yapay zeka tabanlı sistemlerin kullanıcı mahremiyetini ihlal edebileceğini ya da kurumsal şirketlerin iş sırlarını ifşa etme potansiyeline sahip olduğunu biliyoruz. Gerekli önlemler alınmaz ise, yapay zekanın güvenle kullanılabilmesinin önündeki bu açıklar, mevcut sistemleri kullanılamaz hale getirebilir, ciddi ekonomik ve güvenlik sorunlarına yol açabilir. Bu sebeple, paylaşımımda Güvenilir Yapay Zeka’ya ulaşabilmemiz için dikkat edilmesi gereken 6 temel ilkeyi paylaşmak istedim. Bir sonraki paylaşımımda bu temel ilkeleri biraz daha açıp, örnekler ile daha anlaşılır hale getirmeyi umuyorum.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/1*C0pXVPCgBxR4gNlDWwzTwQ.png" /><figcaption>Güvenilir Yapay Zekanın Temel İlkeleri</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8dd59e70a6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Can we trust AI?]]></title>
            <link>https://medium.com/@merfaruktuna/can-we-trust-ai-2dad885535c3?source=rss-a915aefc856e------2</link>
            <guid isPermaLink="false">https://medium.com/p/2dad885535c3</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[robustness]]></category>
            <category><![CDATA[adversarial-attack]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ömer Faruk Tuna]]></dc:creator>
            <pubDate>Sun, 27 Sep 2020 15:16:38 GMT</pubDate>
            <atom:updated>2020-09-27T19:44:13.337Z</atom:updated>
            <content:encoded><![CDATA[<p>Until recently, the major driving force in machine learning was to produce increasingly more accurate models. Around 2013, machine learning started to reach human-level performance on several benchmark tasks. For example in image domain, ML models are able to recognize images better than human accuracy. ML models are also used for speech recognition, NLP tasks, game playing etc. During that period almost no attention had been paid to the security and robustness of these models.</p><p>However by the end of 2013, researchers have figured out that existing deep neural networks are actually vulnerable to attacks. Szegedy et al. [1] first noticed the existence of adversarial examples in the image classification domain: it is possible to transform an image by a small amount and thereby change how the image is classified. In fact, very small and often imperceptible perturbations of the data samples are sufficient to fool state-of-the-art classifiers and result in incorrect classification. Attacks utilizing the vulnerability of DNN’s can seriously undermine the security of these ML based systems, sometimes with devastating consequences. For example, it may be possible that autonomous vehicles can be crashed, illicit or illegal content can bypass content filters, or biometric authentication systems can be manipulated to allow improper access.</p><p>Since the first discovery of the adversarial examples, huge amount of research work has been done in both attacking the DNN’s and defensing these attacks to produce robust ML models.</p><p>Before dive into the details of the various attack algorithms, I would like to show you a simple example. Assume that we have a simple image from ImageNet database. Using a pre-trained ResNet model, we can correctly recognize the image and predict correctly which class the image belongs to. And then we add a small amount of noise to this image and we wonder how the model will predict the new image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UZHzZnI6NBPln3UG-LlYCQ.png" /></figure><p>As can be seen from above figure, the model prediction has not changed and the model still correctly recognize the image subject to a uniformly distributed random noise.</p><p>Then we have tried to implement one of the attack algorithms (DeepFool Attack) to fool the ML model and here is the results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JjGqN-oiybrPtxkLQrTe7Q.png" /></figure><p>Here is another attack (PGD), this time with different input image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lj_QDKK8mTt5SxyTCGX9Ew.png" /></figure><p>As can be seen from above two figures, we could easily able to fool the image classifier by adding a small but crafted perturbation to input image. In these examples, our concern was to only fool the classifier, we did not aim to manipulate a misclassification to a specific output. But, it is actually possible to apply a targeted attack and force the ML model to produce a specific output. Here is an example of a targeted attack, again from ImageNet dataset samples in which we want the model to recognize cat as elephant.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HXgtToVpUSNeqFwGKpRxQg.png" /></figure><p>These examples show that DNN models are indeed vulnerable to attacks. Thus, defending against various attack types and increasing the robustness of the ML models without compromising clean accuracy is of crucial importance. Assuming that these ML models will serve nation wide critical business and strategical areas in very near future, we should pay outmost attention to not only performance of ML models but also to security concerns of these DNN architectures.</p><p>Before moving on further, let’s give brief information on different attack categories [2] : Black Box Attack, Gray Box Attack and White Box Attack.</p><p>· White Box Attack: Attackers know full knowledge about the ML algorithm, ML model, (i.e ., parameters and hyper-parameters), architecture, etc. Generally attackers use the gradients from the DNN to generate adversarial samples.</p><p>· Black Box Attack: Attackers almost know nothing about the ML system and defense strategy.</p><p>· Gray Box Attack: Attackers know the architecture of the DNN classifier and its defense strategy, but have no knowledge of their parameters. Attackers use a network of the same architecture and the same defense strategy to generate adversarial samples.</p><p>Prior work has shown it is actually possible to train a substitute model given black-box access to a target model, and by attacking the substitute model, we can then transfer these attacks to the target model. This phenomena is known as Attack Transferability.</p><p>Adversarial perturbations are generalizable across different models. And this fact can actually become a real concern form a security point of view!</p><p>We can also classify the attack strategies based on their final aims like untargeted attack and targeted attacks.</p><p>· An untargeted attack is the one where the attacker perturbs the input image such as to make the model predict any class other than the true class.</p><p>· A targeted attack is the one where the attacker perturbs the input image in a way such that the model predicts a specific target class.</p><p>The examples we provide above can be classified in white box attack category , and all but the last was untargeted attack.</p><p>Since the first discovery of adversarial examples in late 2013, different kind of attack strategies have been proposed by research community like:</p><p>· Fast Gradient Sign Method [3] : It uses the gradient of the loss function to determine in which direction the pixel’s intensity should be changed (whether it should be increased or decreased) to minimize the loss function; then, it shifts all pixels simultaneously. It is important to note that the fast gradient sign attack was designed to be fast, rather than optimal. It is not meant to produce the minimal adversarial perturbations. This method provides only a coarse approximation of the optimal perturbation vectors often leads to sub-optimal solutions.</p><p>· Iterative Gradient Sign Method [4] : Kurakin et al. introduce a simple refinement of the fast gradient sign method where instead of taking a single step of size ε in the direction of the gradient sign, multiple smaller steps α are taken, and the result is clipped by the same ε. This method is also known as PGD (Projected Gradient Descent) or I-FGSM which expands for Iterative Fast Gradient Sign Method. It is just FGSM applied to an image iteratively.</p><p>· Jacobian based Saliency Map Attack (JSMA) [5]: Papernot et al. introduced an attack optimized under L_0 norm distance is a greedy algorithm that picks pixels to modify one at a time, increasing the target classification on each iteration. They use the gradient to compute a saliency map, which models the impact each pixel has on the resulting classification. A large value indicates that changing it will significantly increase the likelihood of the model labeling the image as the target class l. Given the saliency map, it picks the most important pixel and modify it to increase the likelihood of class l. This is repeated until either more than a set threshold of pixels are modified which makes the attack detectable, or it succeeds in changing the classification.</p><p>· DeepFool [6]: It is an untargeted attack technique optimized for the L_2 distance metric. At each iteration, it linearizes the network and seeks the smallest perturbation to transform current images towards the linearized decision boundary. When we are thinking of a multi-class problem, we can think that the input x has multiple hyperplanes around it that separate it from other classes. What this attack does is it finds that closest hyperplane (most similar class after the true class) and projects this input x onto the hyperplane and pushes it a little beyond, misclassifying it.</p><p>· Carlini &amp; Wagner Attack [7]: Carlini and Wagner reformulate attacks as optimization instances that can be solved using stochastic gradient descent to generate more sophisticated adversarial examples.</p><p>In my next coming post, I will be providing a solid foundation on formulating an adversarial attack. And we will be going over the details of some of the above mentioned attack types. Then we will discuss about training a robust classifier.</p><p>Stay tuned:)</p><p><strong>Ömer Faruk Tuna</strong></p><p>REFERENCES</p><p>[1] SZEGEDY, C., ZAREMBA, W., SUTSKEVER, I., BRUNA, J., ERHAN, D., GOODFELLOW, I., AND FERGUS, R. Intriguing properties of neural networks. ICLR (2013).</p><p>[2] Zhihao Z , Pengyu H. Robust Detection of Adversarial Attacks by Modeling the Intrinsic Properties of Deep Neural Networks</p><p>[3] GOODFELLOW, I. J., SHLENS, J., AND SZEGEDY, C. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014).</p><p>[4] KURAKIN, A., GOODFELLOW, I., AND BENGIO, S. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533 (2016).</p><p>[5] PAPERNOT, N., MCDANIEL, P., JHA, S., FREDRIKSON, M., CELIK, Z. B., AND SWAMI, A. The limitations of deep learning in adversarial settings. In 2016 IEEE European Symposium on Security and Privacy (EuroS&amp;P) (2016), IEEE, pp. 372–387.</p><p>[6] MOOSAVI-DEZFOOLI, S.-M., FAWZI, A., AND FROSSARD, P. Deep- fool: a simple and accurate method to fool deep neural networks. arXiv preprint arXiv:1511.04599 (2015).</p><p>[7] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy (SP), 2017.</p><p>[8] <a href="https://adversarial-ml-tutorial.org/">https://adversarial-ml-tutorial.org</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2dad885535c3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Backpropagation simplified!]]></title>
            <link>https://medium.com/@merfaruktuna/backpropagation-simplified-1d594e091804?source=rss-a915aefc856e------2</link>
            <guid isPermaLink="false">https://medium.com/p/1d594e091804</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[backpropagation]]></category>
            <category><![CDATA[gradient-descent]]></category>
            <dc:creator><![CDATA[Ömer Faruk Tuna]]></dc:creator>
            <pubDate>Tue, 08 Sep 2020 20:56:55 GMT</pubDate>
            <atom:updated>2020-09-08T21:22:54.923Z</atom:updated>
            <content:encoded><![CDATA[<p>If you are reading this post, then you should already know that backpropagation algorithm is used to train a neural network through a method called chain rule. In simple terms, after each forward pass through a network, backpropagation performs a backward pass and adjusts the model’s parameters (weights and biases).</p><p>Backpropagation is by far the most fundamental building block in a neural network because this is the moment when the magic happens and your model actually learns through the data. I am sure you can see tons of resources online that attempt to explain how backpropagation works, but only few of them can actually simplify the learning process.</p><p>In this post, I will try to explain how the backpropagation algorith works in a simplified and generalized way. So, let’s get started!</p><p>Before we jump into the math part, here is a simple diagram of a fully connected neural network:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/1*OEpk4gDVSq348TqWhQJKeg.png" /></figure><p>This network consists of one input layer, three hidden layers and one output layer. But the formulas that we will be deriving throughout this post will be valid for a neural network with any number of hidden layers.</p><p>We will assume that all the activation functions of our network are Sigmoid and the cost function is Root Mean Square Error.</p><p>So the cost function is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/161/1*OC9zR7_aHN-3sUiaowKuRA.png" /><figcaption>Eq:1</figcaption></figure><p>Note that, in above cost function J, ŷ is predicted output and y is the real output.</p><p>2 other important formulas related with output layer that we will use later are:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/184/1*7GIJ-2-SYz0RQWl1KiFGTQ.png" /><figcaption>Eq: 2, 3</figcaption></figure><p>In equation 2, a is the activation of layer L-1, b is bias term and z is the weighted output of layer L.</p><p>In order to update the weights between the last hidden layer and output layer, by chain rule, we can use this formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/148/1*d5ei_GvhKXMK-eQCmt1C7Q.png" /><figcaption>Eq: 4</figcaption></figure><p>Note that we can easily derive from equation 2 that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/113/1*FgJ2_Bti_1uuocWWbAqhwg.png" /><figcaption>Eq: 5</figcaption></figure><p>So, equation 4 can be written as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/136/1*lu9-ld1ZrU-uer6PF4a-nA.png" /><figcaption>Eq: 6</figcaption></figure><p>Let’s use δ to denote the derivative of cost function with respect to z for each layer. So we can write :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/95/1*pE8o2ImLEIs2NrwhJtT6aA.png" /><figcaption>Eq: 7</figcaption></figure><p>By the way, we can calculate δ(L) using below formulas:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/132/1*Rio2JQyjAXxfHvnmyKn0Jw.png" /><figcaption>Eq: 8</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/255/1*gf3vbN-IcIB3xYR3WQeMmg.png" /><figcaption>Eq: 9</figcaption></figure><p>Of course, in equation 8, we can use below formula which can be derived from equation 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/130/1*_iAlgqmvdv3yGDMse7VQ6w.png" /><figcaption>Eq: 10</figcaption></figure><p>Going a little back, using equation 7, we can rewrite equation 6 as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/143/1*_WS0TWPxzsMYFaTqh9rZKQ.png" /><figcaption>Eq: 11</figcaption></figure><p>Actually, equation 11 is very important for us. Because we will use this as a generalized formula to update the weights of any layer. For example, for layer number L-1, just update the indexes of equation 11, and we can easily find the derivative of cost function w.r.t weights of layer L-1.</p><p>I mean this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/176/1*ICdXZKnBM6gSTApQOcFAiw.png" /><figcaption>Eq: 12</figcaption></figure><p>Of course for this task we first need to find δ for that layer. And now we come to that part:)</p><p>So, where were we? Yes, we need to calculate:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/134/1*qet327NkZL4Id40AXaH1nw.png" /><figcaption>Eq: 13</figcaption></figure><p>We know from chain rule that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/182/1*O4zCiVtWKb_q-byYD94Q0w.png" /><figcaption>Eq: 14</figcaption></figure><p>And again using chain rule, equation 14 can be rewritten further as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*e5keAaCm46xnIrxatTyZsw.png" /><figcaption>Eq: 15</figcaption></figure><p>Then, by the help of equation 2 and 3, we can rewrite equation 15 as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/224/1*Ho_YI2k7ka5rQFms1K5erA.png" /><figcaption>Eq: 16</figcaption></figure><p>So by combining equation 7, 13 and 16, we can write:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/201/1*VN283ULd5V6mWWKXvq7tfQ.png" /><figcaption>Eq: 17</figcaption></figure><p>So, as you can guess, we can use equation 17 in Equation 12 and then we can use the result to update the weights of layer L-1. And please note that you can change the indexes by 1 and this time you can calculate δ for layer L-2 and so on.</p><p>If you have come till here, you noticed we have derived so many formulas. But at the end we could find a generalize formula to use in back propagation.</p><p>The key generalized formulas are: <strong>equation 7, 11 and 17</strong>. And remember, we start from the last layer, and we go backward. We can iteratively find δ for related layer , and then use it to calculate weight gradients.</p><p>And of course one last formula that you all know is the weight update rule:)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/201/1*bu5NHawFspQLzLDhaT-szw.png" /><figcaption>Eq: 18</figcaption></figure><p>where η is the learning rate.</p><p>I hope this post is useful for you to understand backpropagation. See you in next posts!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1d594e091804" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Class Based Ensemble Construction]]></title>
            <link>https://medium.com/@merfaruktuna/class-based-ensemble-construction-e69ec529f200?source=rss-a915aefc856e------2</link>
            <guid isPermaLink="false">https://medium.com/p/e69ec529f200</guid>
            <category><![CDATA[ensemble]]></category>
            <category><![CDATA[classifier]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ensemble-learning]]></category>
            <dc:creator><![CDATA[Ömer Faruk Tuna]]></dc:creator>
            <pubDate>Thu, 06 Feb 2020 17:24:44 GMT</pubDate>
            <atom:updated>2020-02-08T17:43:43.306Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Introduction</strong></p><p>There are many different learning algorithms in literature. Though these are generally successful, no single algorithm is always the most accurate. In this study, we are going to discuss different models composed of multiple learners that complement each other so that by combining them, we attain higher accuracy. And we propose a new approach for constructing an ensemble via class based performance.</p><p><strong>Background Information:</strong></p><p>There is no single learning algorithm that always induces the most accurate learner. The usual approach is to try many and choose the one that performs the best on a separate validation set.</p><p>Each learning algorithm dictates a certain model that comes with a set of assumptions. This inductive bias leads to error if the assumptions do not hold for the data. Learning is an illposed problem and with finite data, each algorithm converges to a different solution and fails under different circumstances. The performance of a learner may be fine-tuned to get the highest possible accuracy on a validation set, but this fine tuning is a complex task and still there are instances on which even the best learner is not accurate enough. The idea is that there may be another learner that is accurate on these. By suitably combining multiple base learners then, accuracy can be improved. Recently with computation and memory getting cheaper, such systems composed of multiple learners have become popular.</p><p>There are basically two questions here:</p><p>1. How do we generate base-learners that complement each other?</p><p>2. How do we combine the outputs of base-learners for maximum accuracy?</p><p>Model combination is not a trick that always increases accuracy;model combination does always increase time and space complexity of training and testing, and unless base-learners are trained carefully and their decisions combined smartly, we will only pay for this extra complexity without any significant gain in accuracy.</p><p><strong>Generating diverse learners:</strong></p><p>Because there is no point in combining learners that always make similar decisions, the aim is to be able to find a set of learners who differ in their decisions so that they complement each other. At the same time, there cannot be a gain in overall success unless the learners are accurate, at least in their domain of expertise. We therefore have this double task of maximizing individual accuracies and the diversity between learners.</p><p>We can use different learning algorithms to train different base-learners. Different algorithms make different assumptions about the data and lead to different classifiers. For example, one base-learner may be parametric and another may be nonparametric. When we decide on a single algorithm, we give emphasis to a single method and ignore all others. Combining multiple learners based on multiple algorithms, we free ourselves from taking a decision and we no longer put all our eggs in one basket.</p><p><strong>Subset Selection Methods in classifier combination:</strong></p><p>Ensemble construction can be viewed as an optimization problem and methods proposed in the literature correspond to different search strategies in optimization: there are greedy ‘‘forward” algorithms that are incremental, and add one model at a time if the addition improves the criterion that is to be optimized. There are ‘‘backward” search methods that prune from a large set if the removal is not harmful. There are also ‘‘floating” methods that do both, as well as ones that use genetic algorithms whose operators allow both addition and deletion.</p><p><strong>Model Combination:</strong></p><p>There are different ways the multiple base-learners are combined to generate the final output.</p><p>Multiexpert combination methods have base-learners that work in parallel. These methods can in turn be divided into two:</p><p>• In the global approach, given an input, all base-learners generate an output and all these outputs are used. Examples are voting and stacking.</p><p>• In the local approach, or learner selection, for example, in mixture of experts, there is a gating model, which looks at the input and chooses one (or very few) of the learners as responsible for generating the output.</p><p>Multistage combination uses a serial approach where the next base-learner is trained with or tested on only the instances where the previous base-learners are not accurate enough. The idea is that the base-learners (or the different representations they use) are sorted in increasing complexity so that a complex base-learner is not used (or its complex representation is not extracted) unless the preceding simpler base-learners are not confident. An example is cascading.</p><p>There are several rules to combine multiple learners, an example of these rules are shown in below table:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/1*oBU0fGTVmPPsabkCLqhMuQ.png" /></figure><p><strong>Proposed Method:</strong></p><p>Based on the above information, my proposal can be classifed as Multiexpert Combination — global approach.</p><p>For subset selection:</p><p>Instead of searching for the candidate algorithms to be used in the ensemble, I have followed a novel strategy and pick the most suitable algorithms based on some smart logic. Using training set, I have calculated performance metrics of each base classifier for each class. The ones performing best scores for each class has been selected and used in the ensembling phase. I have used two different performance metric for classifier evaluations: f1-score and precision.</p><p>For model combination:</p><p>I have used posterior probability outputs of each selected base classifier to calculate sum, median, max, product, weighted sum and weighted median values. I have chosed the class with maximum value.</p><p><strong>Experimental details:</strong></p><p>Datasets: I have used 3 datasets in total. These are Optdigits, Pendigits and Pageblocks. All of them are from UCI.</p><p>Base classifiers: I have used 16 base classifiers which has been chosen to span the wide spectrum of possible machine learning algorithms as much as possible:</p><p>• knn: k-nearest neighbor with k . 1; 3; 5.</p><p>• mlp: Multilayer perceptron was with D inputs and K classes, the number of hidden units is taken as D(ml1), K(ml2),(D+K)/2(ml3), (D + K)(ml4), 2.(D + K)(ml5)</p><p>• lnp: Linear perceptron</p><p>• svm: Support vector machines (SVM) with a linear kernel (sv1), polynomial kernel of degree 2 (sv2), and a radial (Gaussian) kernel (svr).</p><p>• CART: Decision tree algorithm</p><p>• Naive Bayes algorithm</p><p>• LDA (Linear Discriminant Analysis) algorithm</p><ul><li>QDA (Quadratic Discriminant Analysis) algorithm</li></ul><p>Division of training, validation, and test sets: Dataset is first divided into two parts, with 1/3 as the test set, and 2/3 as the training set, train-all, (with stratification). The training set,train-all, is then resampled using 5 fold cross-validation (with stratification). I have calculated accuracy of each algorithm using 5 fold cross validaton on training set and noted these accuracy values. In 5-fold CV, I have obtained 5 different trained models for each base learner, than picked the one that has the best score and used that model on the test data for ensembling.</p><p><strong>Experimental Results:</strong></p><p>Below are the accuracy values of 16 base classifiers in training set of Optdigit dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*m625Xmf2N791CWic2K7vBw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KlAt_QDFLXXJNRcGOvhevw.png" /></figure><p>When we check the performance of each classifier for each class (based on f1 score), we saw that k-nearest neighbor algorithm (with k = 1) , sv2 and sv3 are the best performed classifiers on class level. And when we used these 3 algorithms to construct an enseble, we have observed below values:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/413/1*dFaT4_GzeRzZNxSc2bN5qg.png" /></figure><p>When we check the performance of each classifier for each class (based on precision), we saw that k-nearest neighbor algorithm (with k = 1) , MLP with 2(D+K), sv2, sv3 and naive bayes are the best performed classifiers on class level. And when we used these 5 algorithms to construct an enseble, we have observed below values:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/471/1*LPSoMh4PmwazaTwYAzUvjw.png" /></figure><p>Below are the accuracy values of 16 base classifiers in training set of Pendigit dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*SvW40I9CJc6WT9lHVPzC5A.png" /></figure><p>When we check the performance of each classifier for each class (based on precision), we saw that k-nearest neighbor algorithm (with k = 1) , k-nearest neighbor algorithm (with k = 5), sv1, sv3 and QDA are the best performed classifiers on class level. And when we used these 5 algorithms to construct an enseble, we have observed below values:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/672/1*DMJMtSvEwj6rknpijwo5WA.png" /></figure><p>When we check the performance of each classifier for each class (based on f1-score), we saw that k-nearest neighbor algorithm (with k = 1) , k-nearest neighbor algorithm (with k = 5) and sv3 are the best performed classifiers on class level. And when we used these 3 algorithms to construct an enseble, we have observed below values:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*ksFnw9TiahLC6uviTcEQ3A.png" /></figure><p><strong>Conclusion:</strong></p><p>We see that the resulting ensemble outperforms each of the individual base classifiers. So,using the algorithms which has highest score for each class only can be a good way to construct an ensemble.</p><p>Compared to incrementally constructing an ensemble which has polynomial O(N-squared) time complexity, we can construct an ensemble with above mentioned approach with O(N) time complexity. This way, we have both increased resulting accuracy and decreased time complexity.</p><p><strong>References:</strong></p><p>[1] Ethem Alpaydın : Introduction to Machine Learning, 2nd Edition (2010)</p><p>[2] Aydın Ulas, Murat Semerci, Olcay Taner Yıldız, Ethem Alpaydın Incremental construction of classifier and discriminant ensembles, in: Information Sciences 179 (2009) 1298–1318</p><p>[3] A. Asuncion, D.J. Newman, UCI machine learning repository, 2007. &lt;http://www.ics.uci.edu/~mlearn/MLRepository.html&gt;.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e69ec529f200" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Language Modelling and Deep Neural Network Based Text Generation]]></title>
            <link>https://medium.com/@merfaruktuna/introduction-to-language-modelling-and-deep-neural-network-based-text-generation-2bdfb1ab5088?source=rss-a915aefc856e------2</link>
            <guid isPermaLink="false">https://medium.com/p/2bdfb1ab5088</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[text-generation]]></category>
            <category><![CDATA[language-model]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ömer Faruk Tuna]]></dc:creator>
            <pubDate>Fri, 10 Jan 2020 08:00:46 GMT</pubDate>
            <atom:updated>2020-01-10T08:11:09.795Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Introduction to Language Modelling and Deep Neural Network Based Text Generation</strong></p><p><strong>Introduction</strong></p><p>NLP studies involve a number of important tasks like text classifications, sentiment analysis, machine translation, text summarization etc. One other core tasks of NLP is related with language modelling which involves generating text, conditioned on some input information. Before the recent advancement in deep neural network models, the most commonly used methods for text generation were either based on template or rule-based systems, or probabilistic language models such as n-gram or log-linear models [Chen and Goodman, 1996, Koehn et al., 2003]. Language Model is the task of predicting what word comes next or more generally, a system that assigns probability to a piece of a text sequence.N-gram is the simplest language model and its performance is limited by its lack of complexity. Simplistic models like this one cannot achieve fluency, enough language variation and correct writing style for long texts. For these reasons, neural networks (NN) are explored as the new main standard despite their complexity. And Recurrent Neural Networks (RNN) became a fundamental architecture for sequences of any kind. RNN is nowadays considered as the default architecture for text but RNNs have problems of their own: it cannot remember for long the content of the past and it struggles to create long relevant text sequences because of exploding or vanishing gradient problems. For these reasons, other architectures such as Long Short Term Memory (LSTM) [Alex Graves et all, 2014] and Gated Recurrent Units (GRU) [Kyunghyun Cho et al, 2014] were developed and became the state of the art solution for many language generation tasks. In this post, we will be using LSTM to generate sequences of text.</p><p><strong>Language Model</strong></p><p>Models that assign probabilities to sequences of words are called language models. There are primarily two types of Language Models:</p><p>1) Statistical Language Models: These models use traditional statistical techniques like N-grams, Hidden Markov Models (HMM) and certain linguistic rules to learn the probability distribution of words.</p><p>2) Neural Language Models: They use different kinds of Neural Networks to model language and have surpassed the statistical language models in their effectiveness.</p><p><strong>N-Gram Models</strong></p><p>We have described language models as calculating the probability of next word given a sequence of words. Let’s begin with the task of computing P(w|h), the probability of a word w given some history h. Suppose the history h is “its water is so transparent that” and we want to know the probability that the next word is “the”:</p><p>P(the|its water is so transparent that).</p><p>Instead of computing the probability of a word given its entire history, we can approximate the history by just the last few words. Below is the mathematical representation of different n-gram models:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*mN-KczcW6Ym4hmnIvHhTRg.png" /></figure><p>For example, for bigram conditional probability can be calculated as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1009/1*-OlBEd4_son8V8I6XCiU5A.png" /></figure><p><strong>Limitations of N-gram models:</strong></p><p>N-gram based language models do have a few drawbacks:</p><p>• N-gram is the simplest language model and its performance is limited by its lack of complexity. Simplistic models like this one can not achieve fluency,</p><p>• The higher the N, the better is the model usually. But this leads to lots of computation overhead that requires large computation power in terms of RAM.</p><p>• N-grams are a sparse representation of language. This is because we build the model based on the probability of words co-occurring. It will give zero probability to all the words that are not present in the training corpus.</p><p>Due to these drawbacks, we will be building our character based text generation model based on neural network architecture.</p><p><strong>RNN’s (Recurrent Neural Networks):</strong></p><p>A major characteristic of most neural networks such as densely connected networks and convnets, is that they have no memory. Each input shown to them is processed independently, with no state kept in between inputs. With such networks, in order to process a sequence or a temporal series of data points, you have to show the entire sequence to the network at once: turn it into a single data point.</p><p>A recurrent neural network (RNN) processes sequences by iterating through the sequence elements and maintaining a state containing information relative to what it has seen so far. In effect, an RNN is a type of neural network that has an internal loop.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/340/1*dVx75zPBflSmFfOxZcqG0g.png" /><figcaption>Simple RNN architecture</figcaption></figure><p><strong>Types of RNN</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-b5IzmC4HlxrJHzg6SqBBA.png" /><figcaption>Image from Andrej Karpathy Blog “Unreasonable Effectiveness of Recurrent Neural Networks”</figcaption></figure><p>There are five different types of RNN models:</p><p>(1) Vanilla mode of processing without RNN, from fixed-sized input to fixed-sized output (e.g.image classification).</p><p>(2) Sequence output (e.g. image captioning takes an image and outputs a sentence of words).</p><p>(3) Sequence input (e.g. sentiment analysis where a given sentence is classified as expressing positive or negative sentiment).</p><p>(4) Sequence input and sequence output (e.g. Machine Translation: an RNN reads a sentence in English and then outputs a sentence in French).</p><p>(5) Synced sequence input and output (e.g. video classification where we wish to label each frame of the video).</p><p><strong>Pseudocode for RNN</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*-RzyGJQVbZ_Nf3XkXXZMzw.png" /><figcaption>Simple RNN unrolled over time</figcaption></figure><p><strong>Disadvantages of RNN’s:</strong></p><p>Major disadvantages of RNN’s are:</p><p>• Vanishing Gradients (this problem can be solved if you use LSTM or GRU’s)</p><p>• Exploding Gradients (this problem can be solved if you truncate or squash the gradients)</p><p><strong>Short Term Dependencies:</strong></p><p>Predict the last word in “the clouds are in the <strong><em>sky</em></strong>,”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*pcUNLGBWx0ws3GP6loOBLw.png" /><figcaption>Image from Christopher Olah Blog “Understanding LSTM Networks”</figcaption></figure><p>When the gap between the relevant information and the place that it’s needed is small, RNNs can learn to use the past information.</p><p><strong>Long Term Dependencies:</strong></p><p>Predict the last word in the text “I grew up in France… I speak fluent <strong><em>French</em></strong>.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*KTt6sApVRu_q1eHjrRnnRw.png" /><figcaption>Image from Christopher Olah Blog “Understanding LSTM Networks”</figcaption></figure><p>The gap between the relevant information and the point where it is needed to become very large. Unfortunately, as that gap grows, RNNs become unable to learn to connect the information.</p><p>So, as a summary, RNN’s are not able to keep track of long-term dependencies. It cannot process very long sequences. That’s why we will use an upgraded version of RNN called LSTM model.</p><p><strong>LSTM:</strong></p><p>Long Short Term Memory networks — usually just called “LSTMs” — are a special kind of RNN, capable of learning long-term dependencies. They were introduced by Hochreiter &amp; Schmidhuber (1997), and were refined and popularized by many people in following work.</p><p>They work tremendously well on a large variety of problems, and are now widely used.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/1*tDLLNuDx4JoeiOKjMTVJCg.png" /><figcaption>The repeating module in an LSTM</figcaption></figure><p>LSTM is made up of 3 gates.</p><p>1) Input Gate: We decide to add new stuff from the present input to our present cell state scaled by how much we wish to add them.</p><p>2) Forget Gate: After getting the output of previous state, h(t-1), Forget gate helps us to take decisions about what must be removed from h(t-1) state and thus keeping only relevant stuff.</p><p>3) Output Gate: Finally we’ll decide what to output from our cell state which will be done by our sigmoid function.</p><p>And here are the formulas used for an LSTM networks:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/412/1*ba074nX9sbBisfNb7ZcGSA.png" /></figure><p><strong>Implementing Character Level Text Generation:</strong></p><p>The universal way to generate sequence data in deep learning is to train a network to predict the next token or next few tokens in a sequence, using the previous tokens as input.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*JhnZiMpZ05_iHxedx6Ydfw.png" /><figcaption>Image from François Chollet’s book of Deep Learning with Python</figcaption></figure><p>When generating text, the way you choose the next character is crucially important. A naive approach is greedy sampling, consisting of always choosing the most likely next character. But such an approach results in repetitive, predictable strings that don’t look like coherent language.</p><p>A more interesting approach makes slightly more surprising choices: it introduces randomness in the sampling process, by sampling from the probability distribution for the next character. This is called stochastic sampling.</p><p>Sampling probabilistically from the softmax output of the model is neat: it allows even unlikely characters to be sampled some of the time, generating more interesting looking sentences and sometimes showing creativity by coming up with new, realistic sounding words that didn’t occur in the training data.</p><p>In order to control the amount of stochasticity in the sampling process, I have used a parameter called the softmax temperature that characterizes the entropy of the probability distribution used for sampling. It characterizes how surprising or predictable the choice of the next character will be. Given a temperature value, a new probability distribution is computed from the original one (the softmax output of the model) by reweighting it in the following way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*aqwQz23LLcSLeIxKfZ3_pw.png" /></figure><p><strong>Sampling Strategy:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/818/1*15OFEdW92cdZAQCNncQkhA.png" /><figcaption>Image from François Chollet’s book of Deep Learning with Python</figcaption></figure><p><strong>Training Data</strong></p><p>In this study, I have used some of the writings (İnce Memed 1 and İnce Memed 2) of Yaşar Kemal (modern Turkish author) to train an LSTM network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*9jYv9DEyrYXayAlXb-D_mQ.png" /></figure><p>The language model we’ll learn will be specifically a model of Yaşar Kemal’s writing style and topics of choice, rather than a more generic model of the Turkish language.</p><p>Corpus length: 1420227</p><p>Number of sequences: 473389</p><p>Unique characters: 56</p><p>[‘\n’, ‘ ‘, ‘!’, ‘“‘, “‘“, ‘,’, ‘-’, ‘.’, ‘0’, ‘1’, ‘2’, ‘3’, ‘4’, ‘5’, ‘6’, ‘7’, ‘8’, ‘9’, ‘:’, ‘;’, ‘?’, ‘a’, ‘b’, ‘c’, ‘d’, ‘e’, ‘f’, ‘g’, ‘h’,‘i’, ‘j’, ‘k’, ‘l’, ‘m’, ’n’, ‘o’, ‘p’, ‘r’, ‘s’, ‘t’, ‘u’, ‘v’, ‘x’, ‘y’, ‘z’, ‘â’, ‘ç’, ‘ë’, ‘î’, ‘ö’, ‘ü’, ‘ğ’, ‘ı’, ‘ş’, ‘̇’, ‘̈’]</p><p><strong>Results:</strong></p><p>I have used Keras library to train an LSTM network with the above mentioned training data.</p><p>Here is the training loss after 23 epochs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/1*F5ZZf_Fw_PyiewNLW7tSUA.png" /></figure><p>Here is the training accuracy after 23 epochs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/1*IkV9BZo1gc25osfQ3hu0FQ.png" /></figure><p><strong>Generated Text After 1st Epoch</strong></p><p>— — Generating with seed: “u. halbuki memed onun tam aksi. sevinç içinde. memed de kapı»</p><p>— — — temperature: 0.2</p><p>u. halbuki memed onun tam aksi. sevinç içinde. memed de kapıya da ali safa bey de onun at da bir gelecek bir atları da atın da bir sara bir tarladı. bu da da ali safa bey olursun da bir de bir at başından da atın arak bir beni de bir sesle kara da bir tarla da yanında sara da bir at bir de da bir de beni da bir de da kara saran söylemedi. bu kararak da da bir de da da da başını bir de bir serinden bir de kadar bir içinde bir at sağın bir at başını da da da</p><p>— — — temperature: 0.5</p><p>bir de kadar bir içinde bir at sağın bir at başını da da da size çakırdikenlik bir çuyunların bulutuları benim kim bilir ki sız soradın başını bir sapa baktı. da de karan da kalanlarının üstüne karastır. olmadan kalıyordu. düşünü kara var. onun kadar sustun da yapıların da oturdular kadar bir bularak kalınıyordu. ali safa bey seri da at geliyordu. sesi gibi memed de kimeler bir de köylerinden vardı. da yakıp kapıya da çalına da sonra başını gibi de s</p><p>— — — temperature: 1.0</p><p>n vardı. da yakıp kapıya da çalına da sonra başını gibi de sağsız da attı yukaları seni gibi. artadağı suylarını aldırsın memed atlar oradamiz diye mit daha tartısı ova toprakasın öldürdü. çazar delikleri mi dalartdı. sonra bir korkayıp geçorme, dermiyor, dal vardı. döndü. sarkeni candarman ötlemiyordu. de sesle sarttı, gizsen. bazların battı. geliyor, vurmayı boşu bir iğ tış, esme na da kadar allah sauk olyaradılar, izrisinlediğini ata ya. da onun şah</p><p><strong>Generated Text After 20th Epoch</strong></p><p>— — Generating with seed: “ bacakları üstüne ancak dikilebilen koca osman, atlılar ge»</p><p>— — — temperature: 0.2</p><p>bacakları üstüne ancak dikilebilen koca osman, atlılar geçirdi. memed bir de karanlığa karartı bir karanlık kaldı. sonra bir türlü bir karanlık bir kurşun ali safa bey bu ali safa bey de kalabalık kaldı. kadınların bir karanlık karışının bu sabahları bir de ali safa bey de bir karanlık karanlık duruyordu. bir toprak karanlık bir baban gelirdi. arkasında bir karanlık gibi değildi. bu kadar bir kurşun karanlık bir yanını kaldı. bir anlar kaldı. bir anla</p><p>— — — temperature: 0.5</p><p>kurşun karanlık bir yanını kaldı. bir anlar kaldı. bir anların başını kalmasın başında döndü. bu sevin ana bu ben bu büyük çekiyor. ali safa bey onu sonunu geçmiyordu. bu köylüler çok yaşardı. bana gelir. adam kokusu durduğu kara bir kurşun ağamızı düşündü. ayrıldı. memed bu yanda kırmızı kalmış, insan karanlığın altında senin bir yarasını olmaz. o da bağırarak en yaşanıyordu. o düşünür. karanlık bir konuşuyordu. ne desin işlerinin atların içind</p><p>— — — temperature: 1.0</p><p>. karanlık bir konuşuyordu. ne desin işlerinin atların içinden. ı̇şte: gözündeki verme göne gözlerini banambandı. uyukoysunu turadan kalmış, bu doyucuları hiç gelken “yerlerde çıkardı… dimli kayanın bir diyü geçiyor. birlik seler ne yaptılam da idiyordu. karanlarca durdu. sizili bir kap şeyi gelir tenk içinde insanın altındaki devaşın dinini yüz ağılda… süleyman: a, diye dikte sızanın saza ğeni patı gittiği çizdi.</p><p><strong>Result Analysis:</strong></p><p>As can be seen from the outputs, a low temperature value results in repetitive and predictable text, but local structure is highly realistic: in particular, all words are real Turkish words.</p><p>With higher temperatures, the generated text becomes more interesting, surprising, even creative; it sometimes invents completely new words that sound somewhat plausible (such as banambandı and karanlarca).</p><p>By training a bigger model, longer, on more data, you can achieve generated samples that look much more realistic than this one. But, of course, you should not expect to generate any meaningful text, other than by random chance: all you’re doing is sampling data from a statistical model of which characters come after which characters.</p><p><strong>Conclusion:</strong></p><p>You can generate discrete sequence data by training a model to predict the next tokens(s), given previous tokens. In the case of text, such a model is called a language model. It can be based on either words or characters. Sampling the next token requires balance between adhering to what the model judges likely, and introducing randomness. One way to handle this is the notion of softmax temperature. You should experiment with different temperatures to find the right one.</p><p><strong>Referrences:</strong></p><p>[1] Alex Graves. “Generating Sequences With Recurrent Neural Networks”, 2014.</p><p>[2] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, Yoshua Bengio</p><p>“Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling”, 2014.</p><p>[3] Sepp Hochreiter, Jurgen Schimidhuber “Long Short Term Memory”, Neural Computation: 9(8): 1735–1780,1997.</p><p>[4] Andrej Karpathy Blog “Unreasonable Effectiveness of Recurrent Neural Networks”, May 21 2015.</p><p>[5] Christopher Olah Blog “Understanding LSTM Networks”, Aug 27 2015.</p><p>[6] François Chollet’s book “Deep Learning with Python”, 1st edition</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2bdfb1ab5088" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Detecting number of circles in a binary input image]]></title>
            <link>https://medium.com/@merfaruktuna/detecting-number-of-circles-in-a-binary-input-image-d1163ba6d57f?source=rss-a915aefc856e------2</link>
            <guid isPermaLink="false">https://medium.com/p/d1163ba6d57f</guid>
            <category><![CDATA[image-processing]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[hough-transform]]></category>
            <category><![CDATA[circle-detection]]></category>
            <dc:creator><![CDATA[Ömer Faruk Tuna]]></dc:creator>
            <pubDate>Sun, 20 Jan 2019 11:16:18 GMT</pubDate>
            <atom:updated>2019-01-20T12:07:32.660Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Detecting number of circles in a binary input image</strong></p><p>In this post, I would like to mention about how to detect the circles and also the total number of circles in an input image. My algorithm is taking a binary image as input but it is pretty easy to change any input image to a binary format using existing libraries. So, I simply skip that part. And as a preprocessing step, I convert the binary image to edge image. The edges are the actual necessary data for me to detect the existence of circular shapes. This is why I implement a simple filtering operation in x and y axis to produce the edge image.</p><p>Circle detection has been widely studied in literature as part of image processing domain, and some useful techniques has been proposed so far. One of these techniques called <em>Hough Circle Transform</em> is widely adopted in image processing community and we can see lots of different implementations using availabe image processing libraries.</p><p>Instead using any available tool, I would like the share how the algorithm works under the hood, and a simple python code which can achieve the task for us.</p><p>The logic behind Hough Circle Transform algorithm comes from the mathematical expression of a circle in polar coordinate system as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/287/1*Kd9XvS2-34nCey4Sab2SaA.png" /></figure><p>where,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*4vG_mGZr_9mWgahcQdxcrw.png" /></figure><p>We need to transform our 2 dimensional input edge image E(x,y) to 3 dimensional Accumulator matrix(x0,y0,r).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J8hbqEtHUrMPv_QZCos8bA.png" /><figcaption>Reference image taken from <a href="http://me.umn.edu/courses/me5286/vision/Notes/2015/ME5286-Lecture9.pdf">http://me.umn.edu/courses/me5286/vision/Notes/2015/ME5286-Lecture9.pdf</a></figcaption></figure><p>By analyzing the Accumulator Matrix, we can figure out the coordinates of potential circle centers and their corresponding radius. We use a simple voting technique here. Main idea is to record votes for each possible circle coordinates and look for indices that get many votes.</p><p>In order to achieve this task, we can use below pseudocode:</p><pre><em>Apply filtering operation to detect edges<br><br>PSEUDOCODE for HOUGH CIRCLES METHOD<br><br>Input a binary edge image E(x,y)<br>Initialize accumulator array A(x0,y0,r) to zeros<br>Dimensions of A :<br>x0 --&gt; rows of E matrix<br>y0 --&gt; columns of E matrix<br>r  --&gt; sqrt( power(rows of E matrix,2) + power(columns of E matrix,2) )<br><br><br>for all x:<br>    for all y:<br>        if E(x,y):<br>            for all x0:<br>                for all y0:<br>                    r = sqrt( power((x-x0),2) + power((y-y0),2) )<br>                    increment A at (x0,y0,r)<br>                    end<br>                end<br>            end<br>        end<br>    end<br>end<br><br>Search for the peaks in A(x0,y0,r) - the corresponding indices of A are the parameter of the detected circles.</em></pre><p>In order to test my algorithm, I have used an input image called binary_circles.tif. Here is the input image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/72/1*uVOYjr4JojCSPydOUYv-jw.png" /></figure><p>After the preprocessing step of finding the edges, I get below image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/71/1*ro_L7xHMUorwRl6bR51_-g.png" /></figure><p>When we inspect the input image, we can see that the average radius length of the circles is around 10 pixel.</p><p>So in the accumulation matrix, we only consider the possible circles coordinates (x0,y0) with r = 10</p><p>In the above pseudocode, instead of ‘for all x0:’ and ‘for all y0:’ , I have used a range of ‘x-15,x+15’ and ‘y-15,y+15’, where x and y are edge points. Becaue the target Radius I am looking for is 10, I dont need to search for Radius which are bigger. This way , I have achived great time performance in the code.</p><p>The resulting accumulator can be used to detect the number of circles.</p><p>I have coded a simple function find_number_of_circles(Accumulator_Matrix, ratioparameter) which takes argument as Accumulator matrix and a threshold and outputs the # of circles.</p><p>Finally, here is the output of my code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/270/1*_N9OsIMX8AO8SrEt3mXY_w.png" /></figure><p>As you can see, my code correctly found the number of circles in input image:)</p><p>I hope this post is useful for you to understand the theory behind Hough Circle Transform. The code used in this post can be found under this github repo:</p><p><a href="https://github.com/omerfaruktuna/houghcircledetection.git">omerfaruktuna/houghcircledetection</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d1163ba6d57f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Intelligent Food Transportation System]]></title>
            <link>https://medium.com/%C3%B6mer-faruk-tuna/an-intelligent-food-transportation-system-5a9e37b3a5cb?source=rss-a915aefc856e------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a9e37b3a5cb</guid>
            <category><![CDATA[iot]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[raspberry-pi]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <dc:creator><![CDATA[Ömer Faruk Tuna]]></dc:creator>
            <pubDate>Sat, 29 Dec 2018 10:06:41 GMT</pubDate>
            <atom:updated>2018-12-29T10:18:31.628Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>An Intelligent Food Transportation System</strong></p><p>Transportation of food via vehicle involves several challanges. First of all, foods such as milk or meat products should be stored in a specific condition. It should be transported from one location to another in a predefined temperature and humidity. Otherwise it can be contaminated and may easily threaten human health. Therefore measuring the environmental conditions of the place in which food is kept is vital. Second challange is to improve the safety of the driver throughout his trip in case of a potential accidents. Accidents may result in deaths, severe injuries, and loss of income to the impacted families. Accident detection and prevention is a keystone in improving driver safety. In this paper, we propose a combined system to measure and report environmental conditions plus an accident detection mechanism to give an alert to emergency center for early intervention.</p><p>Our study aims at developing an effective and low cost system for measuring environmental conditions of the container where the food is stored and also an accident detection and efficient emergency notification system to reduce number of fatalities, injured persons, and providing medical treatment fast.</p><p>This study particularly presents the overall proposed system including its key phases, hardware/software components, cloud infrastructure used, any third party libraries utilized, proposed algorithm, and how the overall proposed system works. The study also focuses on discussing how data are acquired using temperature, humidity, accelerometer and gyroscope sensors, how the acquired data are filtered using complementary filter, how the filtered data in terms of roll, pitch are calculated and used to identify vehicle rollover.</p><p><strong>PROPOSED SYSTEM</strong></p><p>Our system aims at solving 2 important problems. Monitoring the environmental conditions by measuring and periodically reporting the temperature and humidity levels, and also detecting any kind of rollover as a result of an accident throughout the trip.</p><p><strong>A.</strong> <strong>Enviromental condition monitoring</strong></p><p>We have used a digital sensor device called DTH11 to measure temperature and humidity levels. Raspberry Pi 3B+ device with Wifi module is utilized and DTH11 is connected to it via I2C protocol. I2C protocol is a simple, powerful, and flexible communication protocol. It supports master-slave operation and enables up to 400 KHz data transfer rate. DTH11 python library is used to measure the sensor values and AWS IOT Device SDK for Python on Raspberry Pi is used to periodically sent the measurement values to cloud. We have chosed to use AWS as our cloud solution because it is very handy and comes with different options to get and store IOT data. Once the sensor values are sent to AWS IOT thing we have created, we could successfully transfer it to AWS Elastic Search service (Used as DB to store telemetric data). After that, We have used Kibana dashboard plugin to visualize the data. It is possible to use this URL to reach the visual data via internet from anywhere so it is very useful.</p><p>Here is the simple diagram showing the overall system architecture and components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*9lZ7ZJ-4XsGD5LAiudW9jw.png" /><figcaption>Overall system architecture</figcaption></figure><p><strong>A.</strong> <strong>Rollover Detection</strong></p><p>In this part, a low cost MPU-6050 motion sensor and again Raspberry Pi device used to acquire and process data. MPU-6050 includes gyroscope/accelerometer sensors to sense rotational motion and changes in orientation.</p><p>Gyroscope is used to measure 3-axis rotation angles and vehicle rollover can be identified by analyzing roll and pitch angles.</p><p>• X-axis rotation angle called roll angle, <br> • Y-axis rotation angle called pitch angle, <br> • Z-axis rotation angle called yaw angle.</p><p>Gyroscope angle calculation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/286/1*_9bwqWgS6rwShi61DEHY6w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*0Nts3tS0gnvbLyn_KLDxvQ.png" /></figure><p>Accelerometer angle calculation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/436/1*c7yWT5b-AEYMC67Y-wS8lw.png" /></figure><p>Vehicle crash will impact acceleration forces measured by accelerometer. Large variation in acceleration forces particularly X-axis and Y-axis within a very short time period means a possible vehicle collision.</p><p>Proposed Algorithm:</p><p>We have used both the accelerometer and gyroscope data for the same purpose: obtaining the angular position of the object. The gyroscope can do this by integrating the angular velocity over time. To obtain the angular position with the accelerometer, we are going to determine the position of the gravity vector (g-force) which is always visible on the accelerometer. This can easily be done by using an <a href="http://en.wikipedia.org/wiki/Atan2">atan2</a> function. In both these cases, there is a big problem, which makes the data very hard to use without filter.</p><p>The problem with accelerometers is that an accelerometer measures all forces that are working on the object, it will also see a lot more than just the gravity vector. Every small force working on the object will disturb our measurement completely. And the problem with gyroscopes is that because of the integration over time, the angular velocity measurement has the tendency to drift, not returning to zero when the system went back to its original position. The gyroscope data is reliable only on the short term, as it starts to drift on the long term.</p><p>When looking for the best way to combine the accelerometer and gyroscope data, some people chose to use the very powerful but complex Kalman filter. Although the Kalman filter is great, there are 2 big problems with it that make it hard to use: very complex to understand, very hard to implement on certain hardware. So we chosed to use the other option which is Complimentary Filter.</p><p>The complementary filter is used to enable the use of the gyroscope data on the short term because it is very precise and not susceptible to external forces. On the long term, the accelerometer data is used because it does not drift. An algorithm for acquiring data using sensors, filtering and analyzing data, detecting collision and rollover is developed.</p><p>The gyroscope data is integrated every timestep with the current angle value. After this it is combined with the low-pass, data from the accelerometer (already processed with atan2).</p><p>Complimentary Filter:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*w83RudzEdehqvw8UxYLkEA.png" /></figure><p>We have chosed as 0.98 in the above equation to calculate the final angle used to detect rollover.</p><p>Vehicle usually starts rollover at 46 degrees or more and at -46 degrees or less. If the final roll or pitch angle exceeds 46 degrees or is less than -46, then a rollover is detected, notification system will be activated, and vehicle rollover will be reported to the emergency services. However, these threshold values can be set to different angle degrees to warn drivers and early notify emergency services.</p><p>Collision can be detected by calculating the difference between two consecutive Ax or Ay readings and compared to a specific threshold value. Collision is detected when the variation in acceleration force exceeds the threshold value.</p><p><strong>EXPERIMENTS AND RESULTS</strong></p><p>Part 1:</p><p>We have read temperature and humidity sensor data from DHT11 which is connected to Raspberry Pi and have it stream to AWS IoT via MQTT protocol. We have setup the Amazon ElasticSearch Service and configure a rule so that we could visualize the data on Kibana plugin.</p><p>Reading sensor data on Raspberri Pi:</p><p>Using Python Library of DTH11 sensor, we could successfully read sensor data. As the next step, we made an MQTT connection to AWS IOT and sent the measurement data to cloud.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*RRLzsbJuJWaBgS6Lo_4eeQ.png" /></figure><p>Testing via MQTT client:</p><p>Once we have read the measurement data and sent to AWS IOT platform, we checked whether the data is successfully published to a topic we created via a 3rd party MQTT client. We used a client called MQTTfx , used necessary certificates, configured related IOT endpoint, subscribed to the topic and could able to see that we could successfully sent the sensor data to the cloud.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eQSsxR-ttBWaejtdKPm66w.png" /></figure><p>Visualizing sensor data:</p><p>ElasticSearch is a full-text search engine. In this study, we used it as a database to store measured sensor data. We setup a rule in AWS IoT so that all incoming messages are published to the ElasticSearch cluster. We then used the Kibana plugin to visualize the data. Below is the output from Kibana dashboard:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yjCkeqjIUinJLCub5hMsqA.png" /></figure><p>Part 2:</p><p>We have used MPU6050 sensor to measure rotational movements in X, Y and Z axis. By using a complimentary filter and a simple thresholding mechanism, we could able to detect any kind of rollover in X or Y axis. We chosed 46 degree as our threshold because vehicle usually starts rollover at 46 degrees or more and at -46 degrees or less.</p><p>Below figure shows how to combine data from the accelerometer and gyroscope using a complementary filter to produce a better readings from the MPU6050 sensor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*swMazzXRHuOGV4ZmA3N63g.png" /></figure><p>The following graph show a simple rotation in Y of roughly 50–60 degrees. The purple line shows the accelerometer data and as we can see from the spikes it’s a noisy data set. The green line show the rotation angle calculated from summing the individual angles read from the gyroscope. While this data is less noisy it is prone to drift over time. The blue line shows the complementary filter at work. It combines the two data sets by merging fast rotations from the gyroscope with the slower trends from the accelerometer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*LYJDSuFlfEqAojSW_ZXZ8Q.png" /></figure><p><strong>CONCLUSION AND FUTURE WORK</strong></p><p>In this study, we have proposed a combined solution to keep track of the safety of vehicle and the environmental conditions of the food which is transported into it. Utilized hardware/software components, system architecture and proposed algorithm’s details have been presented.</p><p>Future research may include:</p><p>· Adding a GPS and GSM module to the system. Using GPS data to more accurately detect the collusions and report the exact location of the incident via GSM module.</p><p>· Testing and evaluating system performance in a realistic environment.</p><p>· Expanding the functionality of the proposed system to avoid the occurrence of an accident and detect dangerous driving.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a9e37b3a5cb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/%C3%B6mer-faruk-tuna/an-intelligent-food-transportation-system-5a9e37b3a5cb">An Intelligent Food Transportation System</a> was originally published in <a href="https://medium.com/%C3%B6mer-faruk-tuna">Ömer Faruk Tuna</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>