<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vladimir Stojoc on Medium]]></title>
        <description><![CDATA[Stories by Vladimir Stojoc on Medium]]></description>
        <link>https://medium.com/@vladimirstojoc?source=rss-f00ca4664fb7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Saa-i2fltDb-iYxCiUnl5A.jpeg</url>
            <title>Stories by Vladimir Stojoc on Medium</title>
            <link>https://medium.com/@vladimirstojoc?source=rss-f00ca4664fb7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 12:56:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vladimirstojoc/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Drawing images with Fourier Series]]></title>
            <link>https://medium.com/@vladimirstojoc/drawing-images-with-fourier-series-e7468ecb47b0?source=rss-f00ca4664fb7------2</link>
            <guid isPermaLink="false">https://medium.com/p/e7468ecb47b0</guid>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[complex-number]]></category>
            <category><![CDATA[drawing]]></category>
            <category><![CDATA[fourier-transform]]></category>
            <category><![CDATA[fourier-series]]></category>
            <dc:creator><![CDATA[Vladimir Stojoc]]></dc:creator>
            <pubDate>Tue, 05 Apr 2022 16:14:33 GMT</pubDate>
            <atom:updated>2022-04-05T16:14:33.442Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*_2h5fmH446-VjSf-J2tyTA.png" /></figure><p>Before I begin explaining what the hell is <strong>Fourier Series</strong> let me show you the<em> </em><strong><em>magic</em></strong><em> </em>that it can do<em> ✨.</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F4oAISIw9e50%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D4oAISIw9e50&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F4oAISIw9e50%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/fb78b7c8c7cff452b0c7ae8bba42082b/href">https://medium.com/media/fb78b7c8c7cff452b0c7ae8bba42082b/href</a></iframe><blockquote>Pretty cool huh?</blockquote><p><a href="https://github.com/tokyo-s/FourierDrawings">GitHub - tokyo-s/FourierDrawings: This is one of most interesting projects that i made so far, and it is about a math concept that is called, Fourier Series, shortly, it is saying that with the help of math and complex space, you can draw different images using rotations</a></p><p>And yeah, this can be done with any image, the only constraint for now, is that it should be an <strong>.svg</strong> file.</p><p>Now, let’s go through some of the basics of <strong>Fourier Series</strong> to understand how this works.</p><h3>How does this work?</h3><p>The principal component of these drawings is <strong>rotation</strong>. What does it mean?</p><p>Imagine that you have an arrow <strong>↑</strong>, and you rotate it clockwise with fixed starting point, what does it do?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/1*5GDODdJu7fEUyxbrdBMmbg.png" /><figcaption>clockwise rotation of an arrow</figcaption></figure><p>Right, a <strong>circle</strong>. Now what if we combine more arrows one other another and rotate them?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J3jGq410nGK8DvuzfwsgGg.gif" /></figure><p>Speicfically in this case, the arrows even have a different direction of rotation, and actually the output depends also on the number of arrows and the speed of rotations for each arrow.</p><p>So, by combining a lot of arrows, and tweak their parameters of rotations like direction, speed and size, we can get something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/0*Z1u_J8Eenet_0rGA.gif" /><figcaption>Fourier dinosaur</figcaption></figure><h4>Deep Dive (almost)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/473/0*F-QTEeTrb2nFMI0Q.png" /><figcaption>Euler’s identity</figcaption></figure><p>The formula that you see is called <strong><em>Euler’s identity</em></strong>, the formula that is considered the most beautiful math formula. Using this formula we can create any rotation that is needed. here the numbers just show direction where positive number is counterclockwise and negative is clockwise.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5WbyewQNAlDr9z9_w_CCCA.gif" /></figure><p>Some things that may confuse you is the letter ‘i’ in the formula, part of the power of ‘e’.<br>The letter ‘i’ means the <strong><em>imaginary number</em></strong>, and if you didn’t hear about the <strong>complex numbers</strong>, don’t worry, you just have to understand the main logic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/0*TkT-qogxxc6DX4_h.png" /><figcaption>Complex Numbers</figcaption></figure><p>One of the features of imaginary numbers is their way how multiplication works. ‘i’ is equal √-1, and the power of i is that i² = -1, if we continue then i³=-i and i⁴ = i²*i² = -1*-1 = 1, and the process repeats in cycles of 4.</p><p>Usually a complex number is written by the formula a+bi where a represents the <strong><em>real part</em></strong> and bi repretsents the <strong><em>imaginary part</em></strong><em>. </em>Any number actually can be represented by a complex number, you just put b=0 as in 64 = 64+0i.</p><p>Why do we use complex numbers? We actually need them to create the <strong>rotations</strong>, and specifically we need to do all our operations in a complex space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/0*NJgXH34kUIfcIdqP.png" /><figcaption>Complex space</figcaption></figure><p>The only difference between a simple space and complex space is that on the x axis you have the real part of the complex number, and on the y axis, you have the imaginary part, the ‘i’ part.</p><p>Still, why do we use complex space, and how does it help to do the rotation?<br>All is pretty simple, we use the power rule that we just describe, if you remember the pattern: 1, i, -1, -i, 1…., well, if you look at the complex space axis, this just describes the all 4 directions of the two axis, and the multiplication by i just makes you a 90 degree counterclockwise rotation, let’s go through an example:</p><p>let’s <em>imagine </em>you have a point in complex space 1+0i which happens to be on the right side of x axis, multiply it by i and you get 1i+0 which already is upper part of y axis, a <strong>rotation</strong>. If you continue like, multiply by i, you will get -1+0i (left x axis) and after another one -1i+0 (bottom y axis).</p><p>Now when we know how the rotations are done, the only thing that is not clear is how the algorithm determins which should be the speed, direction and size of every arrow, to create a very beautiful copy of the image.</p><p>But honestly it is not the easiest thing to understand, because it contains some formulas with intergrals that may not be that easy for everyone. But I will leave the link to the video which explains it very well if you are interested.</p><p>But right now, let us speak about the problem I found.</p><h4>The Problem</h4><p>Though this is cool and hard, this only works on pictures that consist of just one line, meaning you can draw all the picture without the need to raise the hand from the paper, an example would be the dinosaur that you saw earlier, it is a big picture, but it is actually just a one-liner. The pictures that have more than one, won’t work, here we have to invent something different.</p><h4>The Solution</h4><p>Because we are using an .svg file, I already have all the functions that represent every line of the image, so the solution is simple, iterate through lines, and repeat the draw process. (the hardest part of this were to center all the drawings and get a real copy of an image)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*XTCXhD4y_Y083mZDujCohw.gif" /><figcaption>Result example, all 6 elements are just one svg file</figcaption></figure><h4>Future Plans</h4><p>In the future versions I plan to add an automatic img to svg converter in order to automate 100% this process and not make user make it by hand. I plan to use some API from a known converter website.</p><p>If you have any ideas of what I can do in order to inprove this app, please contact me with the links below or write me a letter at <em>vladimir.stojoc@gmail.com</em></p><h4>Additional links</h4><ul><li>3Blue1Brown</li></ul><p>The video from which I took some screens and one of my favorite videos, from my favorite math content maker, this is how I found out about this concept. If you love math I recommend this youtube channel, it’s the best that you can find.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fr6sGWTCMz2k%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dr6sGWTCMz2k&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fr6sGWTCMz2k%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/f8ce9678040b05068909eec2c8a51ec6/href">https://medium.com/media/f8ce9678040b05068909eec2c8a51ec6/href</a></iframe><p><strong>Thank you very much for reading this post!</strong></p><ul><li><a href="https://www.facebook.com/vova.stojoc/">Log in or sign up to view</a></li><li><a href="https://www.linkedin.com/in/vladimir-stojoc-a5789a1ba/">Vladimir Stojoc - Data Scientist - Entangled AI | LinkedIn</a></li></ul><p><a href="https://www.instagram.com/_stoves/">https://www.instagram.com/_stoves/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7468ecb47b0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Whiten your data with imperio ZCATransformer]]></title>
            <link>https://medium.com/softplus-publication/whiten-your-data-with-imperio-zcatransformer-3c7a7f328163?source=rss-f00ca4664fb7------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c7a7f328163</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <dc:creator><![CDATA[Vladimir Stojoc]]></dc:creator>
            <pubDate>Wed, 18 Aug 2021 10:03:00 GMT</pubDate>
            <atom:updated>2021-08-18T10:03:00.293Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QoPnxdrbLOgCztCTjKDOWA.png" /></figure><p>Feature engineering is the process of transforming your input data in such a way that it will be more representative of the Machine Learning Algorithms. However, it is very often forgotten because of the inexistence of an easy-to-use package. That’s why we decided to create the one — <strong>imperio</strong>, the third our unforgivable curse.</p><h3>How ZCATransformer works?</h3><p>Whitening or Sphering is a data pre-processing step. It can be used to remove correlation or dependencies between features in a dataset. This may help to better train a machine learning model.</p><p>Eigenvalues and eigenvectors are first calculated from the covariance of a zero centered data set. These are then used for Whitening the data using ZCA (zero component analysis method).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gVclkP2gcxnwWhKv.png" /></figure><h3>Using imperio ZCATransformer:</h3><p>All transformers from imperio follow the transformers API from sci-kit-learn, which makes them fully compatible with sci-kit learn pipelines. First, if you didn’t installed the library, then you can do it by typing the following command:</p><pre><strong>pip install imperio</strong></pre><p>Now you can import the algorithm, fit it and transform some data.</p><pre><strong>from imperio import ZCATransformer</strong></pre><pre><strong>zca = ZCATransformer()<br>X_transformed = zca.fit_transform(X,y)</strong></pre><p>As we said it can be easily used in a sci-kit learn pipeline.</p><pre><strong>from sklearn.pipeline import Pipeline<br>from imperio import ZCATransformer<br>from sklearn.linear_model import LogisticRegression</strong></pre><pre><strong>pipe = Pipeline(<br>    [ <br>     (&#39;zca&#39;, ZCATransformer()),<br>     (&#39;model&#39;, LogisticRegression())<br>])</strong></pre><p>Besides the sci-kit learn API, Imperio transformers have an additional function that allows the transformer to be applied on a pandas data frame.</p><pre><strong>new_df = zca.apply(df, target = &#39;target&#39;)</strong></pre><p>The <strong>ZCATransformer </strong>constructor has the following arguments:</p><ul><li><strong>column_index </strong>(<em>list</em>, <em>default </em>= None): The list of indexes of the categorical columns to apply the transformer on. If set to None it will be applied to all columns.</li></ul><p>The apply function has the following arguments.</p><ul><li><strong>df </strong>(<em>pd.DataFrame</em>): The pandas DataFrame on which the transformer should be applied.</li><li><strong>target </strong>(<em>str</em>): The name of the target column.</li><li><strong>columns </strong>(<em>list</em>, <em>default </em>= None): The list with the names of columns on which the transformers should be applied.</li></ul><p>Now let’s apply it to a data set. We will use it on the <a href="https://www.kaggle.com/uciml/pima-indians-diabetes-database">Pima diabetes data set</a>. Note, we can apply it only on numeric data, so choose the columns if needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*te5JGhe98MtPCCwl5-Z9Cw.png" /></figure><p>As we can observe from confusion matrices, initially we got 77% accuracy, and after applying ZCATransformer we obtained 83% accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*0oML3GNc6rmb1RKo.gif" /></figure><p>Made with ❤ by Sigmoid.</p><p>Follow us on Facebook, Instagram and LinkedIn:</p><p><a href="https://www.facebook.com/sigmoidAI">https://www.facebook.com/sigmoidAI</a></p><p><a href="https://www.instagram.com/sigmo.ai/">https://www.instagram.com/sigmo.ai/</a></p><p><a href="https://www.linkedin.com/company/sigmoid/?fbclid=IwAR1KOuBgH9JODVF1Zh_Agv0Lc5R_9SnM1DqICHEhUS3AVmOZbWvTfMzsnes">https://www.linkedin.com/company/sigmoid/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c7a7f328163" width="1" height="1" alt=""><hr><p><a href="https://medium.com/softplus-publication/whiten-your-data-with-imperio-zcatransformer-3c7a7f328163">Whiten your data with imperio ZCATransformer</a> was originally published in <a href="https://medium.com/softplus-publication">softplus-publication</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Whiten your data with imperio WhiteningTransformer]]></title>
            <link>https://medium.com/softplus-publication/whiten-your-data-with-imperio-whiteningtransformer-c1c5c39e4012?source=rss-f00ca4664fb7------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1c5c39e4012</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <dc:creator><![CDATA[Vladimir Stojoc]]></dc:creator>
            <pubDate>Mon, 09 Aug 2021 10:04:07 GMT</pubDate>
            <atom:updated>2021-08-09T10:04:30.287Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kvRjDIQIlNGKL1ihR0gtkA.png" /></figure><p>Feature engineering is the process of transforming your input data in such a way that it will be more representative of the Machine Learning Algorithms. However, it is very often forgotten because of the inexistence of an easy-to-use package. That’s why we decided to create the one — <strong>imperio</strong>, the third our unforgivable curse.</p><h3>How WhiteningTransformer works?</h3><p>Whitening or Sphering is a data pre-processing step. It can be used to remove correlation or dependencies between features in a dataset. This may help to better train a machine learning model.</p><p>Eigenvalues and eigenvectors are first calculated from the covariance of a zero centered data set. These are then used for Whitening the data using PCA (principal component analysis).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oD6yoa1SUONSrOB2" /></figure><h3>Using imperio WhiteningTransformer:</h3><p>All transformers from imperio follow the transformers API from sci-kit-learn, which makes them fully compatible with sci-kit learn pipelines. First, if you didn’t installed the library, then you can do it by typing the following command:</p><pre><strong>pip install imperio</strong></pre><p>Now you can import the algorithm, fit it and transform some data.</p><pre><strong>from imperio import WhiteningTransformer</strong></pre><pre><strong>whitening = WhiteningTransformer()<br>X_transformed = whitening.fit_transform(X,y)</strong></pre><p>As we said it can be easily used in a sci-kit learn pipeline.</p><pre><strong>from sklearn.pipeline import Pipeline<br>from imperio import WhiteningTransformer<br>from sklearn.linear_model import LogisticRegression</strong></pre><pre><strong>pipe = Pipeline(<br>    [ <br>     (&#39;whitening&#39;, WhiteningTransformer()),<br>     (&#39;model&#39;, LogisticRegression())<br>])</strong></pre><p>Besides the sci-kit learn API, Imperio transformers have an additional function that allows the transformer to be applied on a pandas data frame.</p><pre><strong>new_df = whitening.apply(df, target = &#39;target&#39;)</strong></pre><p>The <strong>WhiteningTransformer </strong>constructor has the following arguments:</p><ul><li><strong>column_index </strong>(<em>list</em>, <em>default </em>= None): The list of indexes of the categorical columns to apply the transformer on. If set to None it will be applied to all columns.</li></ul><p>The apply function has the following arguments.</p><ul><li><strong>df </strong>(<em>pd.DataFrame</em>): The pandas DataFrame on which the transformer should be applied.</li><li><strong>target </strong>(<em>str</em>): The name of the target column.</li><li><strong>columns </strong>(<em>list</em>, <em>default </em>= None): The list with the names of columns on which the transformers should be applied.</li></ul><p>Now let’s apply it to a data set. We will use it on the <a href="https://www.kaggle.com/uciml/pima-indians-diabetes-database">Pima diabetes data set</a>. Note, we can apply it only on numeric data, so choose the columns if needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*og-FElg8rYefxFZUCF6xgA.png" /></figure><p>As we can observe from confusion matrices, initially we got 74% accuracy, and after applying WhiteningTransformer we obtained 79% accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*0oML3GNc6rmb1RKo.gif" /></figure><p>Made with ❤ by Sigmoid.</p><p>Follow us on Facebook, Instagram and LinkedIn:</p><p><a href="https://www.facebook.com/sigmoidAI">https://www.facebook.com/sigmoidAI</a></p><p><a href="https://www.instagram.com/sigmo.ai/">https://www.instagram.com/sigmo.ai/</a></p><p><a href="https://www.linkedin.com/company/sigmoid/?fbclid=IwAR1KOuBgH9JODVF1Zh_Agv0Lc5R_9SnM1DqICHEhUS3AVmOZbWvTfMzsnes">https://www.linkedin.com/company/sigmoid/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1c5c39e4012" width="1" height="1" alt=""><hr><p><a href="https://medium.com/softplus-publication/whiten-your-data-with-imperio-whiteningtransformer-c1c5c39e4012">Whiten your data with imperio WhiteningTransformer</a> was originally published in <a href="https://medium.com/softplus-publication">softplus-publication</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transform your categorical columns with imperio SmoothingTransformer]]></title>
            <link>https://medium.com/softplus-publication/transform-your-categorical-columns-with-imperio-smoothingtransformer-d6534c283c60?source=rss-f00ca4664fb7------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6534c283c60</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <dc:creator><![CDATA[Vladimir Stojoc]]></dc:creator>
            <pubDate>Mon, 26 Jul 2021 10:04:27 GMT</pubDate>
            <atom:updated>2021-07-26T10:06:15.724Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qNWZC1CEHvsQmWwBGqmGfg.png" /></figure><p>Feature engineering is the process of transforming your input data in such a way that it will be more representative of the Machine Learning Algorithms. However, it is very often forgotten because of the inexistence of an easy-to-use package. That’s why we decided to create the one — <strong>imperio</strong>, the third our unforgivable curse.</p><p>One of the main problems that appears when working with data is the categorical values. There are a lot of techniques that help to transform categorical data in numeric but every one of them is working differently and you never know which will work better for your data. In this article we will present you the Smoothing technique.</p><h3>How SmoothingTransformer works?</h3><p>Smoothing is a regularization technique that is based on the following idea. Usually a categorical column is encoded with the Target Imputation technique, but it doesn’t really care if the category is big or small, it applies same operation to every category witch is not very correct. In Smoothing, if category is big, has a lot of data points, then we can trust the estimated encoding, but if category is rare it’s the opposite. The formula down below uses this idea. It has hyper parameter alpha that controls the amount of regularization. When alpha is zero, we have no regularization, and when alpha approaches infinity everything turns into global mean.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Q5rvvKslNe4x9xzTHHWxg.png" /></figure><p>In some sense alpha is equal to the category size we can trust. It’s also possible to use some other formula, basically anything that punishes encoding software categories can be considered smoothing. Smoothing obviously won’t work on its own but we can combine it with for example, CV loop regularization.</p><h3>Using imperio SmoothingTransformer:</h3><p>All transformers from <strong>imperio</strong> follow the transformers API from sci-kit-learn, which makes them fully compatible with sci-kit learn pipelines. First, if you didn’t installed the library, then you can do it by typing the following command:</p><pre><strong>pip install imperio</strong></pre><p>Now you can import the algorithm, fit it and transform some data.</p><pre><strong>from imperio import SmoothingTransformer</strong></pre><pre><strong>smoothing = SmoothingTransformer()<br>X_transformed = smoothing.fit_transform(X,y)</strong></pre><p>As we said it can be easily used in a sci-kit learn pipeline.</p><pre><strong>from sklearn.pipeline import Pipeline<br>from imperio import SmoothingTransformer<br>from sklearn.linear_model import LogisticRegression</strong></pre><pre><strong>pipe = Pipeline(<br>    [ <br>     (&#39;smooth&#39;, SmoothingTransformer()),<br>     (&#39;model&#39;, LogisticRegression())<br>])</strong></pre><p>Besides the sci-kit learn API, Imperio transformers have an additional function that allows the transformer to be applied on a pandas data frame.</p><pre><strong>new_df = smoothing.apply(df, target = &#39;target&#39;)</strong></pre><p>The <strong>SmoothingTransformer </strong>constructor has the following arguments:</p><ul><li><strong>alpha</strong>(float, <em>default </em>= 100): Number that controls the amount of regularization.</li><li><strong>categorical_index </strong>(<em>list</em>, <em>default </em>= None): The list of indexes of the categorical columns to apply the transformer on. If set to None it will be applied to all columns.</li><li><strong>min_int_freq </strong>(int, default = 10): Number that indicates the number of minimal values in a categorical column for the transformer to be applied.</li></ul><p>The apply function has the following arguments.</p><ul><li><strong>df </strong>(<em>pd.DataFrame</em>): The pandas DataFrame on which the transformer should be applied.</li><li><strong>target </strong>(<em>str</em>): The name of the target column.</li><li><strong>columns </strong>(<em>list</em>, <em>default </em>= None): The list with the names of columns on which the transformers should be applied.</li></ul><p>Now let’s test with <a href="https://www.kaggle.com/ronitf/heart-disease-uci">Heart Disease</a>, a classic Machine Learning dataset. Here we set index=[2,6,10,11,12] in the algorithm constructor to show what are the categorical columns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*XNsWtrF52advboHcgPKHcQ.png" /></figure><p>As we can observe from confusion matrices, initially we got 77% accuracy, and after applying <strong>SmoothingTransformer </strong>we obtained 83% accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*0oML3GNc6rmb1RKo.gif" /></figure><p>Made with ❤ by Sigmoid.</p><p>Follow us on Facebook, Instagram and LinkedIn:</p><p><a href="https://www.facebook.com/sigmoidAI">https://www.facebook.com/sigmoidAI</a></p><p><a href="https://www.instagram.com/sigmo.ai/">https://www.instagram.com/sigmo.ai/</a></p><p><a href="https://www.linkedin.com/company/sigmoid/?fbclid=IwAR1KOuBgH9JODVF1Zh_Agv0Lc5R_9SnM1DqICHEhUS3AVmOZbWvTfMzsnes">https://www.linkedin.com/company/sigmoid/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6534c283c60" width="1" height="1" alt=""><hr><p><a href="https://medium.com/softplus-publication/transform-your-categorical-columns-with-imperio-smoothingtransformer-d6534c283c60">Transform your categorical columns with imperio SmoothingTransformer</a> was originally published in <a href="https://medium.com/softplus-publication">softplus-publication</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Split your data into clusters with imperio ClusterizeTransformer]]></title>
            <link>https://medium.com/softplus-publication/split-your-data-into-clusters-with-imperio-clusterizetransformer-1e5d4166296f?source=rss-f00ca4664fb7------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e5d4166296f</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Vladimir Stojoc]]></dc:creator>
            <pubDate>Thu, 08 Jul 2021 10:03:44 GMT</pubDate>
            <atom:updated>2021-07-08T10:03:44.775Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6TTh_-zHNazPh9RaUnqe2g.png" /></figure><p>Feature engineering is the process of transforming your input data in such a way that it will be more representative of the Machine Learning Algorithms. However, it is very often forgotten because of the inexistence of an easy-to-use package. That’s why we decided to create the one — <strong>imperio</strong>, the third our unforgivable curse.</p><p>The technique discussed in this article doesn’t transform your data actually, it adds a new column to it, which can be very helpful to your model. This technique is called ClusterizeTransformer.</p><h3>How ClusterizeTransformer works?</h3><p>If you ever heard about Unsupervised learning, then this technique won’t be hard for you, unsupervised learning is a type of <a href="https://en.wikipedia.org/wiki/Algorithm">algorithm</a> that learns patterns from data without labels, meaning without the target column. In this circumstances were developed other techniques that can predict something based only on the input data. One of these algorithms clustering algorithms is KMeans.</p><p>I won’t explain how KMeans and many other clustering algorithms works, but shortly it just finds a way to divide your data in different groups, usually the number of groups is specified by the user. Thus, in this example below, the data was divide in three clusters, red, blue and green.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/0*T4cLMRP0yYlgBm2V" /></figure><p>After the clustering algorithm is applied, it returns the labels corresponding to each data point, for example: 1 for green, 2 for red, 3 for blue. In that way we can obtain a new column, which will represent the number of the cluster for every data point.</p><h3>Using imperio ClusterizeTransformer:</h3><p>All transformers from imperio follow the transformers API from sci-kit-learn, which makes them fully compatible with sci-kit learn pipelines. First, if you didn’t installed the library, then you can do it by typing the following command:</p><pre><strong>pip install imperio</strong></pre><p>Now you can import the algorithm, fit it and transform some data.</p><pre><strong>from imperio import ClusterizeTransformer</strong></pre><pre><strong>kmeans = KMeans(n_clusters=2)<br>cluster = ClusterizeTransformer(kmeans)<br>X_transformed = cluster.fit_transform(X)</strong></pre><p>As we said it can be easily used in a sci-kit learn pipeline.</p><pre><strong>from sklearn.pipeline import Pipeline<br>from imperio import ClusterizeTransformer<br>from sklearn.preprocessing import  StandardScaler<br>from sklearn.linear_model import LogisticRegression<br>from sklearn.cluster import KMeans</strong></pre><pre><strong>pipe = Pipeline(<br>    [<br>     (&#39;std&#39;, StandardScaler()),    <br>     (&#39;cluster&#39;, ClusterizeTransformer(KMeans(n_clusters = 5))),<br>     (&#39;model&#39;, LogisticRegression())<br>])</strong></pre><p>Besides the sci-kit learn API, Imperio transformers have an additional function that allows the transformer to be applied on a pandas data frame.</p><pre><strong>new_df = cluster.apply(df, target = &#39;target&#39;)</strong></pre><p>The ClusterizeTransformer constructor has the following arguments:</p><ul><li><strong>algorithm </strong>(Object): The instance of the algorithm that will do the clusterization.</li><li><strong>column_index </strong>(<em>list</em>, <em>default </em>= None): The list of indexes of the columns to apply the transformer on. If set to None it will be applied to all columns.</li></ul><p>The apply function has the following arguments.</p><ul><li><strong>df </strong>(<em>pd.DataFrame</em>): The pandas DataFrame on which the transformer should be applied.</li><li><strong>target </strong>(<em>str</em>): The name of the target column.</li><li><strong>columns </strong>(<em>list</em>, <em>default </em>= None): The list with the names of columns on which the transformers should be applied.</li></ul><p>Now let’s test with <a href="https://www.kaggle.com/ronitf/heart-disease-uci">Heart Disease</a>, a classic Machine Learning dataset. Note, we can apply it only on numeric data, so choose the columns if needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*hykLwHbwIsymxolrZSF0Ug.png" /></figure><p>As we can observe from confusion matrices, initially we got 75% accuracy, and after applying ClusterizeTransformer we obtained 93% accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*0oML3GNc6rmb1RKo.gif" /></figure><p>Made with ❤ by Sigmoid.</p><p>Follow us on Facebook, Instagram and LinkedIn:</p><p><a href="https://www.facebook.com/sigmoidAI">https://www.facebook.com/sigmoidAI</a></p><p><a href="https://www.instagram.com/sigmo.ai/">https://www.instagram.com/sigmo.ai/</a></p><p><a href="https://www.linkedin.com/company/sigmoid/?fbclid=IwAR1KOuBgH9JODVF1Zh_Agv0Lc5R_9SnM1DqICHEhUS3AVmOZbWvTfMzsnes">https://www.linkedin.com/company/sigmoid/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e5d4166296f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/softplus-publication/split-your-data-into-clusters-with-imperio-clusterizetransformer-1e5d4166296f">Split your data into clusters with imperio ClusterizeTransformer</a> was originally published in <a href="https://medium.com/softplus-publication">softplus-publication</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Crucio SMOTENC for unbalanced datasets]]></title>
            <link>https://medium.com/softplus-publication/using-crucio-smotenc-for-unbalanced-datasets-21362841f1c1?source=rss-f00ca4664fb7------2</link>
            <guid isPermaLink="false">https://medium.com/p/21362841f1c1</guid>
            <category><![CDATA[sigmoid]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[unbalanced-data]]></category>
            <category><![CDATA[categorical-data]]></category>
            <dc:creator><![CDATA[Vladimir Stojoc]]></dc:creator>
            <pubDate>Wed, 16 Jun 2021 10:04:29 GMT</pubDate>
            <atom:updated>2021-06-16T10:04:32.634Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*Pa88_BhXCbJMTr60rs6gHw.png" /></figure><h4>Unbalanced data</h4><p>In simple terms, an unbalanced dataset is one in which the target variable has more observations in one specific class than the others. An example can be a dataset for transactions: usually, 99% of this set consists of simple, not fraudulent transactions, and only 1% of them are fraud.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/0*XGdFXgSL9DfDmAoF.png" /><figcaption><strong>Unbalanced dataset</strong></figcaption></figure><h4>How does SMOTENC work?</h4><p>SMOTENC (Synthetic Minority Over-sampling Technique for Nominal and Continuous) is actually very similar to classic SMOTE but, unlike SMOTE, SMOTE-NC is able to work with both numerical and categorical features.</p><p>SMOTE-NC slightly changes the way a new sample is generated by performing something specific for the categorical features. In fact, the categories of a new generated sample are decided by picking the most frequent category of the nearest neighbors present during the generation, and while calculating Euclidian distance when searching for k-nearest-neighbors, the algorithm just adds the square of the median of standard deviations of every column that represent numerical values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*veewvqcWSbNvV6EJ.png" /></figure><h4>Using Crucio SMOTENC</h4><p>In case you didn’t install <strong>Crucio</strong> yet, then use in the terminal the following:</p><pre><strong>pip install crucio</strong></pre><p>Now we can import the algorithm and create the <strong>SMOTENC</strong> balancer.</p><pre><strong>from crucio import SMOTENC<br>smotenc = SMOTENC()<br>balanced_df = smotenc.balance(df, &#39;target&#39;)</strong></pre><p>The <strong>SMOTENC</strong>() initialization constructor can contain the following arguments:</p><ul><li><strong>k </strong>(int &gt; 0, default = 5) : The number of nearest neighbors from which SMOTE will sample data points.</li><li><strong>seed </strong>(int, default = 45) : The number used to initialize the random number generator.</li><li><strong>binary_columns </strong>(list, default = None) : The list of binary columns from data set, so sampled data be approximated to nearest binary value.</li></ul><p>The <strong>balance()</strong> method takes as parameters the panda’s data frame and the name of the target column.</p><h3>Example:</h3><p>So I chose a data set where we have to predict the type of a <strong>Pokemon</strong> (<strong>Legendary </strong>or not), the Legendary class constitute <strong>8%</strong> out of all dataset, so it is definitely an imbalanced dataset.</p><p>We recommend you, before applying any module from <strong>Crucio</strong>, to first split your data into a train and test data sets and balance only the train set. In such a way you will test the performance of the model only on natural data.</p><p>The basic <strong>Random Forest</strong> algorithm gives an accuracy of approximately <strong>97%</strong> by training on imbalanced data, so now it’s time to test out SMOTE algorithm.</p><pre><strong>smotenc = SMOTENC()<br>new_df = smotenc.balance(df,&#39;Legendary&#39;)</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QYHEjJNmcAe_x0cvR1cBhg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/1*7TCaWoFi25SpOp7wZzad7A.png" /></figure><p>After balancing data, we increased accuracy from <strong>97% </strong>to <strong>100%.</strong></p><h3>Conclusion:</h3><p>SMOTENC is a very good technique to balance datasets when you have categorical columns and you don’t wanna lose their predicting power. I encourage you to test it and many other balancing methods from Crucio such as SMOTETOMEK, SCUT, SLS, ADASYN, ICOTE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/0*G7qyJ8JoPnWW3pov.gif" /></figure><p>Made with ❤ by Sigmoid.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21362841f1c1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/softplus-publication/using-crucio-smotenc-for-unbalanced-datasets-21362841f1c1">Using Crucio SMOTENC for unbalanced datasets</a> was originally published in <a href="https://medium.com/softplus-publication">softplus-publication</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Crucio SMOTE and Clustered Undersampling Technique for unbalanced datasets]]></title>
            <link>https://medium.com/softplus-publication/using-crucio-smote-and-clustered-undersampling-technique-for-unbalanced-datasets-ab2391077811?source=rss-f00ca4664fb7------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab2391077811</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[unbalanced-data]]></category>
            <category><![CDATA[sigmoid]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Vladimir Stojoc]]></dc:creator>
            <pubDate>Wed, 09 Jun 2021 10:04:32 GMT</pubDate>
            <atom:updated>2021-06-09T10:06:43.504Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*jzxYnQduIyA_fvZfcfLajQ.png" /></figure><h4>Unbalanced data</h4><p>In simple terms, an unbalanced dataset is one in which the target variable has more observations in one specific class than the others. An example can be a dataset for transactions: usually, 99% of this set consists of simple, not fraudulent transactions, and only 1% of them are fraud.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/0*XGdFXgSL9DfDmAoF.png" /><figcaption><strong>Unbalanced dataset</strong></figcaption></figure><h4>How does SCUT work?</h4><p>SMOTE and Clustered Undersampling Technique (SCUT) uses the Expectation Maximization (EM) algorithm. The EM algorithm replaces the hard clusters with a probability distribution formed by a mixture of Gaussians. Instead of being assigned to a particular cluster, each member has a certain probability to belong to a particular Gaussian distribution of the mixture. The parameters of the mixture, including the number of Gaussians, are determined with the Expectation Maximization algorithm. An advantage of using EM is that the number of clusters does not have to be specified beforehand. EM clustering may be used to find both hard and soft clusters. That is, EM assigns a probability distribution to each instance relative to each particular cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/0*i33hJki13Qo6jMtz.png" /></figure><ol><li>For all classes that have a number of instances less than the mean m, oversampling is performed in order to obtain a number of instances equal to the mean. The sampling percentage used for SMOTE is calculated such that the number of instances in the class after oversampling is equal to m.</li><li>For all classes that have a number of instances greater than the mean m, undersampling is conducted to obtain a number of instances equal to the mean. Recall that the EM technique is used to discover the clusters within each class Subsequently, for each cluster within the current class, instances are randomly selected such that the total number of instances from all the clusters is equal to m. Therefore, instead of fixing the number of instances selected from each cluster, we fix the total number of instances. It follows that a different number of instances may be selected from the various clusters. However, we aim to select the instances as uniformly as possible. The selected instances are combined together in order to obtain m instances (for each class).</li><li>All classes for which the number of instances is equal to the mean m are left untouched.</li></ol><h4>Using Crucio SCUT</h4><p>In case you didn’t install Crucio yet, then use in the terminal the following:</p><pre><strong>pip install crucio</strong></pre><p>Now we can import the algorithm and create the <strong>SCUT</strong> balancer.</p><pre><strong>from crucio import SCUT<br>scut = SCUT()<br>balanced_df = scut.balance(df, &#39;target&#39;)</strong></pre><p>The <strong>SCUT</strong>() initialization constructor can contain the following arguments:</p><ul><li><strong>k </strong>(int &gt; 0, default = 5) : The number of nearest neighbors from which SMOTE will sample data points.</li><li><strong>seed </strong>(int, default = 45) : The number used to initialize the random number generator.</li><li><strong>binary_columns </strong>(list, default = None) : The list of binary columns from the data set, so sampled data is approximated to the nearest binary value.</li></ul><p>The <strong>balance()</strong> method takes as parameters the panda’s data frame and the name of the target column.</p><h3>Example:</h3><p>So I chose a data set where we have to predict the type of a <strong>Pokemon</strong> (<strong>Legendary </strong>or not), the Legendary class constitute <strong>8%</strong> out of all dataset, so it is definitely an imbalanced dataset.</p><p>We recommend you, before applying any module from Crucio, to first split your data into a train and test data sets and balance only the train set. In such a way you will test the performance of the model only on natural data.</p><p>The basic <strong>Random Forest</strong> algorithm gives an accuracy of approximately<strong> 92%</strong> by training on imbalanced data, so now it’s time to test out SMOTE algorithm.</p><pre><strong>scut = SCUT()<br>new_df = scut.balance(df,&#39;Legendary&#39;)</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K0S3RaJWjgBvAuhBC3y09A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/1*ZtoLleZu0CPFcCWY8GGXUQ.png" /></figure><p>After balancing data, we increased accuracy from <strong>92.5% </strong>to <strong>97.5%.</strong></p><h3>Conclusion:</h3><p>SCUT is a very good technique to balance datasets, it uses undersampling based on clusters which can be very powerful sometimes, so I encourage you to test it and many other balancing methods from Crucio such as SMOTETOMEK, SMOTENC, SLS, ADASYN, ICOTE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/0*G7qyJ8JoPnWW3pov.gif" /></figure><p>Made with ❤ by Sigmoid.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab2391077811" width="1" height="1" alt=""><hr><p><a href="https://medium.com/softplus-publication/using-crucio-smote-and-clustered-undersampling-technique-for-unbalanced-datasets-ab2391077811">Using Crucio SMOTE and Clustered Undersampling Technique for unbalanced datasets</a> was originally published in <a href="https://medium.com/softplus-publication">softplus-publication</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Balancing datasets using Crucio Safe-Level-SMOTE]]></title>
            <link>https://medium.com/softplus-publication/balancing-datasets-using-crucio-safe-level-smote-a38d467bfa31?source=rss-f00ca4664fb7------2</link>
            <guid isPermaLink="false">https://medium.com/p/a38d467bfa31</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[smote]]></category>
            <category><![CDATA[unbalanced-data]]></category>
            <category><![CDATA[sigmoid]]></category>
            <dc:creator><![CDATA[Vladimir Stojoc]]></dc:creator>
            <pubDate>Wed, 02 Jun 2021 10:01:18 GMT</pubDate>
            <atom:updated>2021-06-02T10:01:18.278Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*SCs_o8np9R6p845PrcUnrg.png" /></figure><h4>Unbalanced data</h4><p>In simple terms, an unbalanced dataset is where the target variable has more observations in one specific class than the others. An example can be a dataset for transactions: usually, 99% of this set consists of simple, not fraudulent transactions, and only 1% of them are fraud.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/0*XGdFXgSL9DfDmAoF.png" /><figcaption><strong>Unbalanced dataset</strong></figcaption></figure><h4>What is SLS and how it works?</h4><p>Based on SMOTE, Safe-Level-SMOTE assigns each positive instance its safe level before generating synthetic instances. Each synthetic instance is positioned closer to the largest safe level so all synthetic instances are generated only in safe regions. The safe level computes by using nearest neighbor minority instances and is defined as:</p><p><strong><em>safe level (sl) = the number of positive instances in k nearest neighbors.</em></strong></p><p>If the safe level of an instance is close to 0, the instance is nearly noise. If it is close to k, the instance is considered safe. The safe level ratio is defined as the formula:</p><p><strong><em>safe level ratio = sl of a positive instance / sl of the nearest neighbor.</em></strong></p><p>It is used for selecting the safe positions to generate synthetic instances. By synthesizing the minority instances more around the larger safe level, we achieve a better accuracy performance than SMOTE.</p><h4>Using Crucio SLS</h4><p>In case you didn’t install Crucio yet, then use in the terminal the following:</p><pre><strong>pip install crucio</strong></pre><p>Now we can import the algorithm and create the SLS balancer.</p><pre><strong>from crucio import SLS<br>sls = SLS()<br>balanced_df = sls.balance(df, &#39;target&#39;)</strong></pre><p>The <strong>SLS</strong>() initialization constructor can contain the following arguments:</p><ul><li><strong>k </strong>(int &gt; 0, default = 5) : The number of nearest neighbors from which SMOTE will sample data points.</li><li><strong>seed </strong>(int, default = 45) : The number used to initialize the random number generator.</li><li><strong>binary_columns </strong>(list, default = None) : The list of binary columns from the data set, so sampled data be approximated to a nearest binary value.</li></ul><p>The <strong>balance()</strong> method takes as parameters the panda’s data frame and the name of the target column.</p><h3>Example:</h3><p>So I chose a data set where we have to predict the type of a <strong>Pokemon</strong> (<strong>Legendary </strong>or not), the Legendary class constitutes <strong>8%</strong> out of all dataset, so it is definitely an imbalanced dataset.</p><p>We recommend you, before applying any module from Crucio, to first split your data into a train and test data sets and balance only the train set. In such a way you will test the performance of the model only on natural data.</p><p>The basic <strong>Random Forest</strong> algorithm gives an accuracy of approximately <strong>96%</strong> by training on imbalanced data, so now it’s time to test out SMOTE algorithm.</p><pre><strong>sls = SLS()<br>new_df = sls.balance(df,&#39;Legendary&#39;)</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-BPozmztgX1TIDBsb0DdNw.png" /><figcaption><strong>Confussion matrix</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/1*AZegOSDQb84asqtd4Earxw.png" /><figcaption><strong>Data representation, before (above) and after (below)</strong></figcaption></figure><p>After balancing data, we increased accuracy from <strong>96% </strong>to <strong>99%.</strong></p><h3>Conclusion:</h3><p>SLS is an interesting technique that tries to improve the popular SMOTE algorithm, so I encourage you to test it and see the difference with some others balancing methods from Crucio such as SMOTETOMEK, SMOTEENN, ADASYN, ICOTE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/0*G7qyJ8JoPnWW3pov.gif" /></figure><p>Made with ❤ by Sigmoid.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a38d467bfa31" width="1" height="1" alt=""><hr><p><a href="https://medium.com/softplus-publication/balancing-datasets-using-crucio-safe-level-smote-a38d467bfa31">Balancing datasets using Crucio Safe-Level-SMOTE</a> was originally published in <a href="https://medium.com/softplus-publication">softplus-publication</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Crucio SMOTEENN for balancing data]]></title>
            <link>https://medium.com/softplus-publication/using-crucio-smoteenn-for-balancing-data-6bf7a4ab7c61?source=rss-f00ca4664fb7------2</link>
            <guid isPermaLink="false">https://medium.com/p/6bf7a4ab7c61</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[smote]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[sigmoid]]></category>
            <dc:creator><![CDATA[Vladimir Stojoc]]></dc:creator>
            <pubDate>Wed, 19 May 2021 10:02:51 GMT</pubDate>
            <atom:updated>2021-05-19T10:02:51.047Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*vly8-g-He8BIIWA1IuZUmQ.png" /></figure><p>This article will be about a combination of the two most powerful algorithms used for oversampling and undersampling imbalanced datasets, SMOTE and ENN.</p><p>I talked about SMOTE in one of my <a href="https://medium.com/softplus-publication/using-crucio-smote-for-balancing-data-4905b704da4f">previous</a> articles, so you should definitely check it out because I will now rely more on the ENN algorithm.</p><h4>How ENN works?</h4><p>ENN(Edited Nearest Neighbor) is an undersampling technique that looks for noise in data, it computes KNN for every minority class example, and depending on them it decides if among those samples exists an imposter, if yes, then this example will be deleted.</p><p>After the ENN technique, we simply apply SMOTE to oversample minority class examples.</p><h3>Using SMOTEENN from Crucio</h3><p>If you still haven’t installed Crucio just type the following in the following in the command line.</p><pre><strong>pip install crucio</strong></pre><p>Now we have to import and use our algorithm</p><pre><strong>from crucio import SMOTEENN</strong></pre><pre><strong>enn = SMOTEENN()<br>new_df = enn.balance(df,&#39;target&#39;)</strong></pre><p>The <strong>SMOTEENN</strong>() initialization constructor can contain the following arguments:</p><ul><li><strong>k </strong>(int &gt; 0, default = 5) : The number of nearest neighbors from which SMOTE will sample data points.</li><li><strong>seed </strong>(int, default = 45) : The number used to initialize the random number generator.</li><li><strong>binary_columns </strong>(list, default = None) : The list of binary columns from the data set, so sampled data is approximated to the nearest binary value.</li></ul><p>The <strong>balance()</strong> method takes as parameters the panda’s data frame and the name of the target column.</p><h3>Example:</h3><p>I took the same dataset from the previous article, the one about <strong>Legendary</strong> <strong>Pokemons</strong>.</p><p>Classifying our pokemons without any help, got us an accuracy of <strong>95.8%</strong></p><p>which is a pretty good accuracy, but we can try to increase it without <strong>SMOTEENN</strong> algorithm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/1*GorawdqLscHdP5g7sBL30w.png" /></figure><pre><strong>smote = SMOTEENN()</strong></pre><pre><strong>new_df = smote.balance(df,’Legendary’)</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*ca0weNH4MKSqBJvCjEc86A.png" /></figure><p>Now we got an even better accuracy, <strong>100%</strong> percent, yep that’s a small dataset, but it can be really good at a specific situation, so I recommend remembering it and try using it next time you see an imbalanced dataset.</p><p>And here is a little plot demonstrating how new examples were sampled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*5zgfA-nyA7biy1sMC4xOWQ.png" /></figure><h3>Conclusion:</h3><p><strong>SMOTEENN</strong> is an interesting technique that combines both undersampling (using <strong>ENN</strong>) and oversampling (using <strong>SMOTE</strong>), and this combination can bring you great results if used wisely.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*w4KpOub0BMXJzOK03IZz8A.gif" /></figure><p>Made with ❤ by Sigmoid.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6bf7a4ab7c61" width="1" height="1" alt=""><hr><p><a href="https://medium.com/softplus-publication/using-crucio-smoteenn-for-balancing-data-6bf7a4ab7c61">Using Crucio SMOTEENN for balancing data</a> was originally published in <a href="https://medium.com/softplus-publication">softplus-publication</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Crucio SMOTETOMEK for balancing data]]></title>
            <link>https://medium.com/softplus-publication/using-crucio-smotetomek-for-balancing-data-bf8ca2b99a1f?source=rss-f00ca4664fb7------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf8ca2b99a1f</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[sigmoid]]></category>
            <category><![CDATA[smote]]></category>
            <category><![CDATA[imbalanced-data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Vladimir Stojoc]]></dc:creator>
            <pubDate>Sat, 10 Apr 2021 22:20:07 GMT</pubDate>
            <atom:updated>2021-05-08T13:03:21.637Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*nB_0VRPVexajPNs3jcln2g.png" /></figure><p>This article will be about a combination of the two most powerful algorithms used for oversampling and undersampling imbalanced datasets, SMOTE and Tomek Links.</p><p>I talked about SMOTE in my <a href="https://medium.com/softplus-publication/using-crucio-smote-for-balancing-data-4905b704da4f">previous article</a>, so you should definitely check it out because I will now rely more on the Tomek Links algorithm.</p><p>So starting with an explanation, a Tomek link is created when there exists a pair of instances of opposite classes who are their own nearest neighbors. In other words, they are pairs of opposing instances that are very close together.</p><p>We will use the Tomek link in order to exclude examples from the majority class that are too close to examples from the minority class.</p><p>Then we simply apply SMOTE to oversample minority class examples.</p><h3>Using SMOTETOMEK from Crucio</h3><p>If you still haven’t installed Crucio just type the following in the following in the command line.</p><pre><strong>pip install crucio</strong></pre><p>Now we have to import and use our algorithm</p><pre><strong>from crucio import SMOTETOMEK</strong></pre><pre><strong>tomek = SMOTETOMEK()<br>new_df = tomek.balance(df,&#39;target&#39;)</strong></pre><p>The <strong>SMOTETOMEK</strong>() initialization constructor can contain the following arguments:</p><ul><li><strong>k </strong>(int &gt; 0, default = 5) : The number of nearest neighbors from which SMOTE will sample data points.</li><li><strong>seed </strong>(int, default = 45) : The number used to initialize the random number generator.</li><li><strong>binary_columns </strong>(list, default = None) : The list of binary columns from the data set, so sampled data is approximated to the nearest binary value.</li></ul><p>The <strong>balance()</strong> method takes as parameters the panda’s data frame and the name of the target column.</p><h3>Example:</h3><p>I took the same dataset from the previous article, the one about <strong>Legendary</strong> <strong>Pokemons</strong>.</p><p>Classifying our pokemons without any help, got us an accuracy of <strong>95.8%</strong></p><p>which is a pretty good accuracy, but we can try to increase it without <strong>SMOTETOMEK</strong> algorithm</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*qzJnq0M_0on0UqBkakTqKg.png" /></figure><pre><strong>smote = SMOTETOMEK()</strong></pre><pre><strong>new_df = smote.balance(df,’Legendary’)</strong></pre><p>Now we got an even better accuracy, <strong>99%</strong> percent, yep that’s a small dataset, but it can be really good at a specific situation, so I recommend remembering it and try using it next time you see an imbalanced dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*8OOzsmGVn6yIiyqyV8_Fxw.png" /></figure><p>And here is a little plot demonstrating how new examples were sampled</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/736/1*toCVYdZNsXj-kSTMHpy3hQ.png" /></figure><h3>Conclusion:</h3><p>SMOTETOMEK is an interesting technique that combines both undersampling (using Tomek Links) and oversampling (using SMOTE), and this combination can bring you great results if used wisely.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*w4KpOub0BMXJzOK03IZz8A.gif" /></figure><p>Made with ❤ by Sigmoid.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf8ca2b99a1f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/softplus-publication/using-crucio-smotetomek-for-balancing-data-bf8ca2b99a1f">Using Crucio SMOTETOMEK for balancing data</a> was originally published in <a href="https://medium.com/softplus-publication">softplus-publication</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>