<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mehmet Toprak on Medium]]></title>
        <description><![CDATA[Stories by Mehmet Toprak on Medium]]></description>
        <link>https://medium.com/@toprak.mhmt?source=rss-2d2112a2cc66------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*02x_KjUVzGADe7SGxvbCLA.jpeg</url>
            <title>Stories by Mehmet Toprak on Medium</title>
            <link>https://medium.com/@toprak.mhmt?source=rss-2d2112a2cc66------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 02:56:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@toprak.mhmt/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Gradient Boosting and Weak Learners]]></title>
            <link>https://medium.com/@toprak.mhmt/gradient-boosting-and-weak-learners-1f93726b6fbd?source=rss-2d2112a2cc66------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f93726b6fbd</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ensemble]]></category>
            <category><![CDATA[boosting]]></category>
            <category><![CDATA[ensemble-learning]]></category>
            <dc:creator><![CDATA[Mehmet Toprak]]></dc:creator>
            <pubDate>Thu, 24 Sep 2020 17:58:35 GMT</pubDate>
            <atom:updated>2020-09-24T17:58:35.592Z</atom:updated>
            <content:encoded><![CDATA[<p>In this blog, we’ll explore one of the most powerful ensemble methods around — gradient boosting!</p><p>You will be able to:</p><ul><li>Compare and contrast weak and strong learners and explain the role of weak learners in boosting algorithms</li><li>Describe the process of boosting in Adaboost and Gradient Boosting</li><li>Explain the concept of a learning rate and the role it plays in gradient boosting algorithms</li></ul><h3>Weak learners and boosting</h3><p>The first ensemble technique we learned about was <em>Bagging</em>, which refers to training different models independently on different subsets of data by sampling with replacement. The goal of bagging is to create variability in the ensemble of models. The next ensemble technique we’ll learn about is <em>Boosting</em>. This technique is at the heart of some very powerful, top-of-class ensemble methods currently used in machine learning, such as <em>Adaboost</em> and <em>Gradient Boosted Trees</em>.</p><p>In order to understand boosting, let’s first examine the cornerstone of boosting algorithms — <em>Weak Learners</em>.</p><h3>Weak learners</h3><p>All the models we’ve learned so far are <em>Strong Learners</em> — models with the goal of doing as well as possible on the classification or regression task they are given. The term <em>Weak Learner</em> refers to simple models that do only slightly better than random chance. Boosting algorithms start with a single weak learner (tree methods are overwhelmingly used here), but technically, any model will do. Boosting works as follows:</p><ol><li>Train a single weak learner</li><li>Figure out which examples the weak learner got wrong</li><li>Build another weak learner that focuses on the areas the first weak learner got wrong</li><li>Continue this process until a predetermined stopping condition is met, such as until a set number of weak learners have been created, or the model’s performance has plateaued</li></ol><p>In this way, each new weak learner is specifically tuned to focus on the weak points of the previous weak learner(s). The more often an example is missed, the more likely it is that the next weak learner will be the one that can classify that example correctly. In this way, all the weak learners work together to make up a single strong learner.</p><h3>Boosting and random forests</h3><h3>Similarities</h3><p>Boosting algorithms share some similarities with random forests, as well as some notable differences. Like random forests, boosting algorithms are an ensemble of many different models with high inter-group diversity. Boosting algorithms also aggregate the predictions of each constituent model into an overall prediction. Both algorithms also make use of tree models (although this isn’t strictly required, in the case of boosting).</p><h3>Differences</h3><h4>1: Independent vs. iterative</h4><p>The difference is in the approach to training the trees. Whereas a random forest trains each tree independently and at the same time, boosting trains each tree iteratively. In a random forest model, how well or poorly a given tree does has no effect on any of the other trees since they are all trained at the same time. Boosting, on the other hand, trains trees one at a time, identifies the weak points for those trees, and then purposefully creates the next round of trees in such a way as to specialize in those weak points.</p><h4>2: Weak vs. strong</h4><p>Another major difference between random forests and boosting algorithms is the overall size of the trees. In a random forest, each tree is a strong learner — they would do just fine as a decision tree on their own. In boosting algorithms, trees are artificially limited to a very shallow depth (usually only 1 split), to ensure that each model is only slightly better than random chance. For this reason, boosting algorithms are also highly resilient against noisy data and overfitting. Since the individual weak learners are too simple to overfit, it is very hard to combine them in such a way as to overfit the training data as a whole — especially when they focus on different things, due to the iterative nature of the algorithm.</p><h4>3: Aggregate predictions</h4><p>The final major difference we’ll talk about between the two is the way predictions are aggregated. Whereas in a random forest, each tree simply votes for the final result, boosting algorithms usually employ a system of weights to determine how important the input for each tree is. Since we know how well each weak learner performs on the dataset by calculating its performance at each step, we can see which weak learners do better on hard tasks. Think of it like this — harder problems deserve more weight. If there are many learners in the overall ensemble that can get the same questions right, then that tree isn’t super important — other trees already provide the same value that it does. This tree will have its overall weight reduced. As more and more trees get a hard problem wrong, the “reward” for a tree getting that hard problem correct goes higher and higher. This “reward” is actually just a higher weight when calculating the overall vote. Intuitively, this makes sense — trees that can do what few other trees can do are the ones that we should probably listen to more than others, as they are the most likely to get hard examples correct. Since other trees tend to get this wrong, we can expect to see a general split of about 50/50 among the trees that do not “specialize” in the hard problems. Since our “specialized” tree has more weight, its correct vote will carry more weight than the combined votes of the half of the “unspecialized” trees that get it wrong. It is worth noting that the “specialized” trees will often do quite poorly on the examples that are easy to predict. However, since these examples are easier, we can expect a strong majority of the trees in our ensemble to get it right, meaning that the combined, collective weight of their agreement will be enough to overrule the trees with higher weights that get it wrong.</p><h3>Understanding Adaboost and Gradient boosting</h3><p>There are two main algorithms that come to mind when Data Scientists talk about boosting: <em>Adaboost</em> (short for Adaptive Boosting), and <em>Gradient Boosted Trees</em>. Both are generally very effective, but they use different methods to achieve their results.</p><h3>Adaboost</h3><p>Adaboost was the first boosting algorithm invented. Although there have been marked improvements made to this algorithm, Adaboost still tends to be quite an effective algorithm! More importantly, it’s a good starting place for understanding how boosting algorithms actually work.</p><p>In Adaboost, each learner is trained on a subsample of the dataset, much like we saw with <em>Bagging</em>. Initially, the bag is randomly sampled with replacement. However, each data point in the dataset has a weight assigned. As learners correctly classify an example, that example’s weight is reduced. Conversely, when learners get an example wrong, the weight for that sample increases. In each iteration, these weights act as the probability that an item will be sampled into the “bag” which will be used to train the next weak learner. As the number of learners grows, you can imagine that the examples that are easy to get correct will become less and less prevalent in the samples used to train each new learner. This is a good thing — if our ensemble already contains multiple learners that can correctly classify that example, then we don’t need more that can do this. Instead, the “bags” of data will contain multiple instances of the hard examples, thereby increasing the likelihood that the learner will create a split that focuses on getting the hard example correct.</p><p>The following diagram demonstrates how the weights change for each example as classifiers get them right and wrong.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wANOE0U-Pxk9y82w.png" /></figure><p>Pay attention to the colors of the pluses and minuses — pluses are meant to be in the blue section, and minuses are meant to be in the red. The decision boundary of the tree can be interpreted as the line drawn between the red and blue sections. As you can see above, examples that were misclassified are larger in the next iteration, while examples that were classified correctly are smaller. As we combine the decision boundaries of each new classifier, we end up with a classifier that correctly classifies all of the examples!</p><p><em>Key Takeaway:</em> Adaboost creates new classifiers by continually influencing the distribution of the data sampled to train each successive learner.</p><h3>Gradient boosting</h3><p><em>Gradient Boosted Trees</em> are a more advanced boosting algorithm that makes use of <em>Gradient Descent.</em> Much like Adaboost, gradient boosting starts with a weak learner that makes predictions on the dataset. The algorithm then checks this learner’s performance, identifying examples that it got right and wrong. However, this is where the gradient boosting algorithm diverges from Adaboost’s methodology. The model then calculates the <em>Residuals</em> for each data point, to determine how far off the mark each prediction was. The model then combines these residuals with a <em>Loss Function</em> to calculate the overall loss. There are many loss functions that are used — the thing that matters most is that the loss function is <em>differentiable</em> so that we can use calculus to compute the gradient for the loss, given the inputs of the model. We then use the gradients and the loss as predictors to train the next tree against! In this way, we can use <em>Gradient Descent</em> to minimize the overall loss.</p><p>Since the loss is most heavily inflated by examples where the model was wrong, gradient descent will push the algorithm towards creating a new learner that will focus on these harder examples. If the next tree gets these right, then the loss goes down! In this way, gradient descent allows us to continually train and improve on the loss for each model to improve the overall performance of the ensemble as a whole by focusing on the “hard” examples that cause the loss to be high.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*izeNMvTd_USBde6f.png" /></figure><h3>Learning rates</h3><p>Often, we want to artificially limit the “step size” we take in gradient descent. Small, controlled changes in the parameters we’re optimizing with gradient descent will mean that the overall process is slower, but the parameters are more likely to converge to their optimal values. The learning rate for your model is a small scalar meant to artificially reduce the step size in gradient descent. Learning rate is a tunable parameter for your model that you can set — large learning rates get closer to the optimal values more quickly, but have trouble landing exactly at the optimal values because the step size is too big for the small distances it needs to travel when it gets close. Conversely, small learning rates means the model will take a longer time to get to the optimal parameters, but when it does get there, it will be extremely close to the optimal values, thereby providing the best overall performance for the model.</p><p>The sklearn library contains some excellent implementations of Adaboost, as well as several different types of gradient boosting classifiers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f93726b6fbd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Semi-Supervised Learning and Look-Alike Models]]></title>
            <link>https://medium.com/@toprak.mhmt/semi-supervised-learning-and-look-alike-models-96b5f28666fa?source=rss-2d2112a2cc66------2</link>
            <guid isPermaLink="false">https://medium.com/p/96b5f28666fa</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[semi-supervised-learning]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <dc:creator><![CDATA[Mehmet Toprak]]></dc:creator>
            <pubDate>Fri, 18 Sep 2020 17:48:10 GMT</pubDate>
            <atom:updated>2020-09-18T17:48:10.238Z</atom:updated>
            <content:encoded><![CDATA[<p>In this blog, we’ll learn about some unsupervised learning techniques we can use to supplement our supervised learning techniques. You will be able to:</p><ul><li>Identify appropriate use cases for semi-supervised learning</li><li>Identify appropriate use cases for look-alike models</li></ul><h3>Combining Supervised and Unsupervised Learning</h3><p>For the majority of this section, we’ve focused exclusively on popular unsupervised learning techniques and their most common use cases. However, in the real world, there are also plenty of examples where it works to our advantage to bring supervised and unsupervised learning algorithms together to supplement each other. In this blog, we’ll look at two common areas combining supervised and unsupervised learning algorithms that allow us to be more effective than just using them on their own.</p><h3>Use Case 1: Look-Alike Models</h3><p>As we’ve learned when working with clustering algorithms, one of their most common use cases is for market segmentation. A more advanced, but similar use case is to then use these market segments to create <em>look-alike models</em> to help us identify more customers or market segments that we can plausibly assume are equally valuable, due to their similarity with valuable customers or market segments we’ve already identified.</p><p>Take a look at the following infographic that provides a visual representation of look-alike modeling:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7HCAWnTx9TIwJpde.png" /></figure><p>In the example above, the dark blue smiley faces represent customer segments that we already know are valuable. These are customers that we have identified in our data, and know for a fact have been good for us. Under normal circumstances, this would mean that we can divide our customers (or, more often, potential customers) into two groups: the group we know is valuable, and everyone else, who are all unknown to us.</p><p>This is where <em>look-alike modeling</em> comes in. A look-alike model uses a distance metric of our choice to rate the similarity of each customer in our group of unknowns to customers in our known, valuable group. For customers that look extremely similar to customers in own known valuable group, we can assume with a very high likelihood that these customers will also be valuable, and should direct resources at capturing them! We’ll likely also see customers that are only somewhat similar to our valuable group, which tells us that they <em>could possibly be valuable</em>, but we aren’t sure. And finally, customers that look nothing like our known valuable customers segment, should probably be left alone.</p><p>If this sounds suspiciously like clustering to you, you are absolutely correct! Although this could also be framed as a classification or regression problem, it’s quite common to see clustering used to help determine similarity. After all, if we want to build a supervised learning model to predict if an unknown customer looks like our known valuable customers, then we need plenty of labeled examples, and we don’t always have that luxury!</p><p>In the real-world, using look-alike models to find other customers that could potentially be valuable to us is often referred to as <em>prospecting</em>. Viewed in terms of the infographic above, we would choose direct resources to market to the customers that look like our valuable customers to increase our <em>top-of-funnel</em>, meaning that we are trying to increase the number of potential customers that haven’t shown interest in our product or company yet but are likely to, due to their similarity to customers that already have.</p><h3>Use Case 2: Semi-Supervised Learning</h3><p>The second use case we’ll talk about combines supervised and unsupervised learning to allow us access to more (pseudo) labeled data so that we can better train our supervised learning models. This technique is called <em>semi-supervised learning</em>. You may also hear it commonly referred to as <em>weakly supervised learning</em>, but it means the same thing.</p><p>Picture the following scenario:</p><p>We are trying to build a supervised learning model, and we have 100,000 observations in our dataset. However, labels are exceedingly expensive, so only 5,000 of these 100,000 observations are labeled. In traditional supervised learning, this means that in a practical sense, we really only have a dataset of 5,000 observations, because we can’t do anything with the 95,000 unlabeled examples — or can we?</p><p>The main idea behind <em>semi-supervised learning</em> is to generate <em>pseudo-labels</em> that are possibly correct (at least better than random chance). To do this, we don’t usually use clustering algorithms — instead, we use our supervised learning algorithms in an unsupervised way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ql9IMDT479Fk5QR7.png" /></figure><p>Supervised learning typically follows a set pattern:</p><ol><li><em>Train your model on your labeled training data</em>. In the case of our example above, we would build the best model possible with our tiny dataset of 5,000 labeled examples.</li><li><em>Use your trained model to generate pseudo-labels for your unlabeled data</em>. This means having our trained model make predictions on our 95,000 unlabeled examples. Since our trained model does better than random chance, this means that our generated pseudo-labels will be at least somewhat more correct than random chance. We can even put a number to this, by looking at the performance our trained model had on the test set. For example, if our trained model had an accuracy of ~70%, then we can assume that ~70% of the pseudo-labels will be correct, ~30% will be incorrect.</li><li><em>Combine your labeled data and your pseudo-labeled data into a single, new dataset.</em>. This means that we concatenate all our labeled data of 5,000 examples with the 95,000 pseudo-labeled examples.</li><li><em>Retrain your model on the new dataset</em>. Although some of the pseudo-labeled data will certainly be wrong, it’s likely that the amount that is correct will be more useful, and the signal that these correctly pseudo-labeled examples provide will outweigh the incorrectly labeled ones, thereby resulting in better overall model performance.</li></ol><h3>Benefits and Drawbacks of Semi-Supervised Learning</h3><p>If semi-supervised learning sounds a bit risky to you, you’re not wrong. When done correctly, semi-supervised learning can increase overall model performance by opening up access to much more data than we would have access to, and more data almost always results in better performance, but without the exorbitant costs of paying to have humans generate labels for the data needed.</p><p>However, there are definitely some problems that can arise from using a semi-supervised learning approach, if we’re not careful and thoughtful throughout.</p><h4>Feedback Loops and Self-Fulfilling Prophecies</h4><p>Semi-supervised learning tends to work fairly well in many use cases and has become quite a popular technique in the field of Deep Learning, which requires massive amounts of labeled data that is often very expensive to obtain. But what happens when our dataset is extremely noisy to begin with? In that case, our incorrect pseudo-labels may skew the model by introducing more “noise” than “signal”. This is partially because we can end up in a feedback loop of sorts. Think about an example where the model has generated an incorrect pseudo-label. If a model trained only on the real data with no pseudo-labels got this example wrong, then what happens when you train the model on the same example, but this time provide a pseudo-label that “confirms” this incorrect belief? When done correctly, we can hope that the signal provided by all the correctly pseudo-labeled examples will generalize to help the model correct its mistakes on the ones it got wrong. However, if the dataset is noisy, or the original model wasn’t that good to begin with (or both), then it can be quite likely that we are introducing even more incorrect information than correct information, moving the model in the wrong direction.</p><p>So how do we make sure that we’re not making these mistakes when using a semi-supervised approach? <em>Use a holdout set!</em> You should definitely have a test set that the model has never seen before to check the performance of your semi-supervised model. Obviously, make sure that your test set only contains actual, ground-truth labeled examples, no pseudo-labels allowed! Also, the noisier your dataset or more complicated your problem, the more likely you are to run into trouble with semi-supervised learning. When possible, try to structure your tasks as binary classification tasks, rather than multi-categorical, and make sure that your dataset is as clean as possible before attempting semi-supervised learning. Although it seems risky, there’s a reason companies that are heavy into deep learning and AI research such as Google, Microsoft, and Facebook make heavy use of semi-supervised learning — when done correctly, it works wonders, without costing an arm and a leg to pay for labeling!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=96b5f28666fa" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Big Data]]></title>
            <link>https://medium.com/@toprak.mhmt/big-data-7eeeadcf67e1?source=rss-2d2112a2cc66------2</link>
            <guid isPermaLink="false">https://medium.com/p/7eeeadcf67e1</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[big-data-analytics]]></category>
            <dc:creator><![CDATA[Mehmet Toprak]]></dc:creator>
            <pubDate>Fri, 11 Sep 2020 01:10:39 GMT</pubDate>
            <atom:updated>2020-09-11T01:10:39.089Z</atom:updated>
            <content:encoded><![CDATA[<p>In the information age, data in huge quantities has become available to analysts and decision-makers. Due to a vast increase in the amount of such data in recent times, a number of specialized platforms and development paradigms have been developed that can handle big data. Using such specialist approaches allows data scientists to gain valuable insights from complex data, ranging from daily transactions to customer interactions and social network data.</p><p>This blog aims to focus on some of the different analytical approaches and tools data scientists apply to big data in order to gain valuable insights that aid business decision making.</p><h3>What is Big Data</h3><p>The topic of “big data” has received a lot of hype lately, accompanied by a huge amount of interest from big businesses as it can potentially provide them data-driven decision-making abilities. Big data is one of the most discussed topics in business today across industry sectors, although it was barely known a few years ago. This blog will focus on what big data is, why it is important, and the benefits it brings.</p><p>Big data is no different than normal data that we have seen so far; it’s only “bigger.” This changes the analytical landscape that must be used as the huge size increase of the data requires specialist tools, techniques, and platforms. It helps us solve new problems and find improved ways to find answers to old problems.</p><h3>Defining Big Data</h3><p>Despite all the hype around this topic, there is no clear consensus on how to define big data. The term often gets related to business analytics and data mining for identifying relationships and associations present in huge amounts of transaction data.</p><p>In the data science domain, big data usually refers to datasets that grow so large that they become awkward to work with using traditional database management systems and analytical approaches. They are datasets whose size is beyond the ability of commonly used software tools and storage systems to capture, store, manage, as well as process the data within a tolerable elapsed time.</p><h4>How Big is “Big” Data?</h4><p>Big data sizes are constantly increasing, currently ranging from a few terabytes (TB) to many petabytes (PB) of data in a single dataset. Consequently, some of the difficulties related to big data include capturing, storing, searching, sharing, analyzing, and visualizing. Today, enterprises are exploring large volumes of highly detailed data to discover trends and pieces of information considered incapable of being captured before.</p><p>Here are some of the examples of big data:</p><ul><li>Web traffic data: Data points such as number of page views, previous web page, user information, advertisement click-through rate, pages per visit, average visit duration</li><li>Text data: Emails, tweets, news reports, voice recordings, and text gathered from crawling the web can make massive datasets that are valuable to data scientists</li><li>Location and time data: GPS data helps Google determine which roads have higher traffic and which businesses will be busier at certain hours</li><li>Social network data: Using the information of relationships between users on Facebook, LinkedIn, Twitter, Reddit, and countless other websites and apps</li><li>Smart grid and sensor data: With the advent of the Internet of Things (IoT), more and more devices are able to record data at all times, making it possible to gather lots of data instantaneously</li></ul><h3>3 V’s of Big Data</h3><p>Doug Laney published a paper on three defining characteristics of big data. Three main features characterize big data: volume, variety, and velocity, or the three V’s. The volume of the data is its size, and how enormous it is. Velocity refers to the rate with which data is changing, or how often it is created. Finally, variety includes the different formats and types of data, as well as the different kinds of uses and ways of analyzing the data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JXE78nrnsD0s_IMx.png" /></figure><p>Let’s look a bit deeper into what these 3 V’s refer to:</p><h3>VOLUME</h3><p>Volume refers to the amount of data generated through websites, portals, and online applications in a data-driven business. Especially for online retailers, volume encompasses the available data that are out there and need to be assessed for relevance.</p><p>Consider the following:</p><p>As of 2019, Facebook has 2.32 billion users, Youtube: 1.9 billion users, WhatsApp: 1.6 billion users and Instagram: 1 billion users. Every day, these users contribute to billions of images, posts, videos, tweets, etc. You can now imagine the insanely large amount (or volume) of data that is generated every minute around the world. Data volume is the primary attribute of big data. Big data can be quantified by size in Terabytes (TBs) or Petabytes (PBs), as well as even the number of records, transactions, tables, or files. Additionally, one of the things that makes big data really big is that it’s coming from a greater variety of sources than ever before, including logs, clickstreams, and social media as we will see below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3LgC4E77tYFdFeYa.png" /></figure><h3>VELOCITY</h3><p>Velocity refers to the speed with which data is generated, and as internet speeds have increased and the number of users has increased, the velocity has also increased substantially.</p><p>The following image created by Lori Lewis and Chadd Callahan shows what happens on major social media platforms in one minute.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*Uo5Mr5BPCZO-4B9I.jpg" /></figure><p>Velocity is basically the frequency of data generation or the frequency of data delivery. The leading edge of big data is streaming data, which is collected in real-time from the websites.</p><p>Tools within the big data stack help companies hold this explosion in velocity, accept the incoming flow of data, and at the same time process it quickly enough so that it does not create bottlenecks.</p><h3>VARIETY</h3><p>Variety in big data refers to all the structured and unstructured data that has the possibility of getting generated either by humans or by machines. Structured data is whatever data you could store in a spreadsheet. It can easily be cataloged and summary statistics can be calculated for it. Unstructured data are raw things like texts, tweets, pictures, videos, emails, voice mails, hand-written text, ECG readings, and audio recordings. Humans can only make sense of data that is structured, and it is usually up to data scientists to create some organization and structure to unstructured data.</p><p>Variety is all about the ability to classify the incoming data into various categories and turn unstructured data into something with more structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OSOmMUwCUGJyvcWj.png" /></figure><p>This leads us to the most widely used definition in the industry by Gartner:</p><h3>Big data is high-volume, high-velocity and/or high-variety information assets that demand cost-effective, innovative forms of information processing that enable enhanced insight, decision making, and process automation.</h3><p>Any data sources that fall under those 3 Vs are sources of big data, no matter how you define it.</p><p>NOTE: <em>Some researchers have discussed the addition of a fourth V, or Veracity. Veracity focuses on the quality of the data. This characterizes big data quality as good, bad, or undefined due to data inconsistency, incompleteness, ambiguity, latency, deception, and approximations</em></p><p>The important thing to remember from this three-pronged definition of Big Data is that there is not a single component that makes data “big” or not. It is also a futile effort to try and make a definition of what the threshold is to make something “big data” rather than normal data. As technologies evolve and new distributed algorithms are created, what was once big data will no longer be big, and we will raise the bar.</p><h3>Big Data Analytics</h3><p>With the evolution of technology and the increased amounts of data, as discussed above, the need for faster and more efficient ways of analyzing such data has also grown exponentially. Having big data alone is no longer enough to make efficient decisions at the right time. As we mentioned above, Big Data cannot be easily analyzed with traditional data management and analysis techniques and infrastructures. Therefore, there arises a need for new tools and methods specialized for big data analytics, as well as the required architectures for storing and managing such data. Accordingly, the emergence of big data has an effect on everything from the data itself to its collection, analysis, and visualization, as well as the final extracted decisions.</p><p>The image below shows the technology stack, or the key tools and platforms heavily being employed in big data analytics today.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eFJJ2RRXMEex8z93.png" /></figure><p>Explaining each one of these tools/platforms etc. is outside the scope of this blog. You are, however, encouraged to look up these technologies and see their role in big data analytics. Such a stack maps the different big data storage, management, analytics tools/methods, visualization, and evaluation tools to the different phases of the decision-making process.</p><p>The key activities associated with big data analytics are reflected in four main areas:</p><ul><li>Big data warehousing and distribution</li><li>Big data storage</li><li>Big data computational platforms</li><li>Big data analyses, visualization, and evaluation</li></ul><p>Such a framework can be applied for knowledge discovery and informed decision-making in big data-driven organizations.</p><h3>Example Business Applications of Big Data Analytics</h3><p>Along with some of the most common advanced data analytics methods such as regression analysis, association rules, clustering, and classification, some additional analyses have become common with big data.</p><p>For example, social media has recently become important for social networking and content sharing. Yet, the content that is generated from social media websites is enormous and remains largely unexploited. However, social media analytics can be used to analyze such data and extract useful information and predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GjNO4h-G2A1O93UH.png" /></figure><blockquote><em>Social media analytics is based on developing and evaluating informatics frameworks and tools in order to collect, monitor, summarize, analyze, as well as visualize social media data.</em></blockquote><p>Social media analytics facilitates understanding the reactions and conversations between people in online communities, as well as extracting useful patterns and intelligence from their interactions and what they share on social media websites.</p><p>On the other hand, text mining and NLP techniques are used to analyze a document or set of documents in order to understand the content within and the meaning of the information contained. Text mining has become very important nowadays since much of the information stored consists of text — in the form of emails, SMS texts, social media feeds, blogs, etc. While data mining deals with structured data, text presents special characteristics which basically follow a non-relational form and require wisely thought-out schemas to grant it more structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/0*sBvn49ju00b7r3UG.png" /></figure><p>Sentiment analysis/opinion mining is also becoming more and more important as online opinion data, such as blogs, product reviews, forums, and social data from social media sites, like Twitter and Facebook, grow tremendously.</p><blockquote><em>Sentiment Analysis focuses on analyzing and understanding emotions from subjective text patterns and is enabled through text mining. It identifies the opinions and attitudes of individuals towards certain topics, and it is useful in classifying viewpoints as positive or negative.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UJvkizDtsBU8ZrdJ.png" /></figure><p>Sentiment analysis uses NLP and text analytics in order to identify and extract information by finding words that are indicative of certain sentiments, as well as relationships between words so that sentiments can be accurately identified.</p><p>And finally, one of the leading applications in big data analytics is recommendation systems. Powerful recommendation engines can be built for anything from movies and videos to music, books, and products as offered by Netflix, Pandora, or Amazon. As customers of an online retailer browse through products, the Recommendation system offers recommendations of products they might be interested in. In our daily online browsing and shopping routine, most of us often come across messages like the one shown below. This is a recommendation system doing its job.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8NmFX7qVUOHV7QqL.png" /></figure><p>Recommendation systems have been immensely beneficial for both businesses and consumers. Big data is the driving force behind recommendation systems. A typical recommendation system cannot do its job without sufficient data and big data supplies plenty of user data such as past purchases, browsing history, and feedback for the recommendation systems to provide relevant and effective recommendations. In a nutshell, even the most advanced recommendations cannot be effective without big data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7eeeadcf67e1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Comparing Machine Learning Techniques Using Pipelines]]></title>
            <link>https://medium.com/@toprak.mhmt/comparing-machine-learning-techniques-using-pipelines-f4773f72fc1e?source=rss-2d2112a2cc66------2</link>
            <guid isPermaLink="false">https://medium.com/p/f4773f72fc1e</guid>
            <category><![CDATA[gridsearchcv]]></category>
            <category><![CDATA[pipeline]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Mehmet Toprak]]></dc:creator>
            <pubDate>Sat, 05 Sep 2020 16:16:07 GMT</pubDate>
            <atom:updated>2020-09-05T16:16:07.261Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In this blog, I’ll use a Dataset created by Otto group, which was also used in a <a href="https://www.kaggle.com/c/otto-group-product-classification-challenge/data">Kaggle competition</a>.</p><p>The description of the data set is as follows:</p><p>The Otto Group is one of the world’s biggest e-commerce companies, with subsidiaries in more than 20 countries, including Crate &amp; Barrel (USA), Otto.de (Germany) and 3 Suisses (France). They are selling millions of products worldwide every day, with several thousand products being added to our product line.</p><p>A consistent analysis of the performance of our products is crucial. However, due to their global infrastructure, many identical products get classified differently. Therefore, the quality of our product analysis depends heavily on the ability to accurately cluster similar products. The better the classification, the more insights Otto Group can generate about their product range.</p><p>In this lab, you’ll use a data set containing:</p><ul><li>A column id, which is an anonymous id unique to a product</li><li>93 columns feat_1, feat_2, ..., feat_93, which are the various features of a product</li><li>a column target - the class of a product</li></ul><h3>Objectives</h3><p>You will be able to:</p><ul><li>Compare different classification techniques</li><li>Construct pipelines in scikit-learn</li><li>Use pipelines in combination with GridSearchCV</li></ul><h3>The Data Science Workflow</h3><p>You will be following the data science workflow:</p><ol><li>Initial data inspection, exploratory data analysis, and cleaning</li><li>Feature engineering and selection</li><li>create a baseline model</li><li>create a machine learning pipeline and compare results with the baseline model</li><li>Interpret the model and draw conclusions</li></ol><h3>Initial data inspection, exploratory data analysis, and cleaning</h3><p>The data is stored in “otto_group.csv”.</p><p>Things to do here:</p><ul><li>Check for NAs</li><li>Check the distributions</li><li>Check how many inputs there are</li><li>…</li></ul><pre>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt</pre><pre>data = pd.read_csv(&quot;otto_group.csv&quot;)</pre><pre>data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TZSRaTB3IIOGsx9KEGmt0g.png" /></figure><pre>data.info()</pre><pre>&lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;<br>RangeIndex: 61878 entries, 0 to 61877<br>Data columns (total 95 columns):<br>id         61878 non-null int64<br>feat_1     61878 non-null int64<br>feat_2     61878 non-null int64<br>feat_3     61878 non-null int64<br>feat_4     61878 non-null int64<br>feat_5     61878 non-null int64<br>feat_6     61878 non-null int64<br>feat_7     61878 non-null int64<br>feat_8     61878 non-null int64<br>feat_9     61878 non-null int64<br>feat_10    61878 non-null int64<br>feat_11    61878 non-null int64<br>feat_12    61878 non-null int64<br>feat_13    61878 non-null int64<br>feat_14    61878 non-null int64<br>feat_15    61878 non-null int64<br>feat_16    61878 non-null int64<br>feat_17    61878 non-null int64<br>feat_18    61878 non-null int64<br>feat_19    61878 non-null int64<br>feat_20    61878 non-null int64<br>feat_21    61878 non-null int64<br>feat_22    61878 non-null int64<br>feat_23    61878 non-null int64<br>feat_24    61878 non-null int64<br>feat_25    61878 non-null int64<br>feat_26    61878 non-null int64<br>feat_27    61878 non-null int64<br>feat_28    61878 non-null int64<br>feat_29    61878 non-null int64<br>feat_30    61878 non-null int64<br>feat_31    61878 non-null int64<br>feat_32    61878 non-null int64<br>feat_33    61878 non-null int64<br>feat_34    61878 non-null int64<br>feat_35    61878 non-null int64<br>feat_36    61878 non-null int64<br>feat_37    61878 non-null int64<br>feat_38    61878 non-null int64<br>feat_39    61878 non-null int64<br>feat_40    61878 non-null int64<br>feat_41    61878 non-null int64<br>feat_42    61878 non-null int64<br>feat_43    61878 non-null int64<br>feat_44    61878 non-null int64<br>feat_45    61878 non-null int64<br>feat_46    61878 non-null int64<br>feat_47    61878 non-null int64<br>feat_48    61878 non-null int64<br>feat_49    61878 non-null int64<br>feat_50    61878 non-null int64<br>feat_51    61878 non-null int64<br>feat_52    61878 non-null int64<br>feat_53    61878 non-null int64<br>feat_54    61878 non-null int64<br>feat_55    61878 non-null int64<br>feat_56    61878 non-null int64<br>feat_57    61878 non-null int64<br>feat_58    61878 non-null int64<br>feat_59    61878 non-null int64<br>feat_60    61878 non-null int64<br>feat_61    61878 non-null int64<br>feat_62    61878 non-null int64<br>feat_63    61878 non-null int64<br>feat_64    61878 non-null int64<br>feat_65    61878 non-null int64<br>feat_66    61878 non-null int64<br>feat_67    61878 non-null int64<br>feat_68    61878 non-null int64<br>feat_69    61878 non-null int64<br>feat_70    61878 non-null int64<br>feat_71    61878 non-null int64<br>feat_72    61878 non-null int64<br>feat_73    61878 non-null int64<br>feat_74    61878 non-null int64<br>feat_75    61878 non-null int64<br>feat_76    61878 non-null int64<br>feat_77    61878 non-null int64<br>feat_78    61878 non-null int64<br>feat_79    61878 non-null int64<br>feat_80    61878 non-null int64<br>feat_81    61878 non-null int64<br>feat_82    61878 non-null int64<br>feat_83    61878 non-null int64<br>feat_84    61878 non-null int64<br>feat_85    61878 non-null int64<br>feat_86    61878 non-null int64<br>feat_87    61878 non-null int64<br>feat_88    61878 non-null int64<br>feat_89    61878 non-null int64<br>feat_90    61878 non-null int64<br>feat_91    61878 non-null int64<br>feat_92    61878 non-null int64<br>feat_93    61878 non-null int64<br>target     61878 non-null object<br>dtypes: int64(94), object(1)<br>memory usage: 44.8+ MB</pre><pre>data.isna().any()</pre><pre>id         False<br>feat_1     False<br>feat_2     False<br>feat_3     False<br>feat_4     False<br>feat_5     False<br>feat_6     False<br>feat_7     False<br>feat_8     False<br>feat_9     False<br>feat_10    False<br>feat_11    False<br>feat_12    False<br>feat_13    False<br>feat_14    False<br>feat_15    False<br>feat_16    False<br>feat_17    False<br>feat_18    False<br>feat_19    False<br>feat_20    False<br>feat_21    False<br>feat_22    False<br>feat_23    False<br>feat_24    False<br>feat_25    False<br>feat_26    False<br>feat_27    False<br>feat_28    False<br>feat_29    False<br>           ...  <br>feat_65    False<br>feat_66    False<br>feat_67    False<br>feat_68    False<br>feat_69    False<br>feat_70    False<br>feat_71    False<br>feat_72    False<br>feat_73    False<br>feat_74    False<br>feat_75    False<br>feat_76    False<br>feat_77    False<br>feat_78    False<br>feat_79    False<br>feat_80    False<br>feat_81    False<br>feat_82    False<br>feat_83    False<br>feat_84    False<br>feat_85    False<br>feat_86    False<br>feat_87    False<br>feat_88    False<br>feat_89    False<br>feat_90    False<br>feat_91    False<br>feat_92    False<br>feat_93    False<br>target     False<br>Length: 95, dtype: bool</pre><pre># We only looked at some of the features here, make sure to do a quick check for all the features (similar structure)<br>feat = data.loc[:, &#39;feat_1&#39;:&#39;feat_30&#39;]<br>feat.hist(figsize=(30,20));</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uQsQBEhTyHVSMcRv2b06GA.png" /></figure><p>If you look at all the histograms, you can tell that a lot of the data are zero-inflated, so most of the variables contain mostly zeros and then some higher values here and there. No normality, but for most machine learning techniques this is not an issue.</p><pre>feat.boxplot(figsize=(10,10));</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z_6cJGfp1CK6pHB52vqMyQ.png" /></figure><p>Because the data is zero-inflated the boxplots look as shown above. Because there are this many zeroes, most values above zero will seem to be outliers. The safe decision for this data is to not delete any outliers and see what happens. With many 0s, sparse data is available and high values may be super informative. More-over, without having any intuitive meaning for each of the features, we don’t know if a value of ~260 is actually an outlier.</p><pre># is there any missing data?</pre><pre>feat.isna().any().any()</pre><pre>False</pre><h3>Feature engineering and selection with PCA</h3><p>Have a look at the correlation structure of your features using a heatmap.</p><pre>feat = data.loc[:, &#39;feat_1&#39;:&#39;feat_93&#39;]</pre><pre>import seaborn as sns<br>sns.heatmap(feat.corr(), center=0);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*grQh6bxyvAO0qENe0Y4QJg.png" /></figure><p>Use PCA to downscale your features. Use PCA to select a number of features in a way that you still keep 80% of your explained variance.</p><pre>from sklearn.decomposition import PCA<br>pca_1 = PCA(n_components=20)<br>pca_2 = PCA(n_components=40)<br>pca_3 = PCA(n_components=60)</pre><pre>principalComponents = pca_1.fit_transform(feat)<br>principalComponents = pca_2.fit_transform(feat)<br>principalComponents = pca_3.fit_transform(feat)</pre><pre>print(np.sum(pca_1.explained_variance_ratio_))<br>print(np.sum(pca_2.explained_variance_ratio_))<br>print(np.sum(pca_3.explained_variance_ratio_))</pre><pre>pca = PCA(n_components=27)<br>principalComponents = pca.fit_transform(feat)<br>print(np.sum(pca.explained_variance_ratio_))</pre><pre>0.7275643792253975<br>0.8885505230944062<br>0.9598045557819248<br>0.8003287639023293</pre><pre>import seaborn as sns<br>sns.heatmap(pd.DataFrame(principalComponents).corr(), center=0);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*XhYdGfkSJ7HcjjlVkgPz0Q.png" /></figure><h3>Create a train test split with a test size of 40%</h3><p>This is a relatively big training set. Feel free to make it smaller (down to ~20%), but for an initial run you can try smaller training sets so the computation time is more manageable.</p><p>For now, simply use the original data and not the principal components. We looked at the PC’s first to get a sense of our correlation structure, and to see how we can downsize our data without losing too much information. In what’s next, you’ll make PCA part of the pipeline!!</p><pre>y = data[&#39;target&#39;]<br>X = data.loc[:, &#39;feat_1&#39;:&#39;feat_93&#39;]</pre><pre># Load and split the data</pre><pre>from sklearn.model_selection import train_test_split<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42)</pre><h3>Create a baseline model</h3><p>Create your baseline model <em>in a pipeline setting</em>. In the pipeline</p><ul><li>Your first step will be to scale your features down to the number of features that ensure you keep just 80% of your explained variance (which we saw before)</li><li>Your second step will be the building a basic logistic regression model.</li></ul><p>Make sure to fit the model using the training set, and test the result by obtaining the accuracy using the test set.</p><pre>from sklearn.linear_model import LogisticRegression<br>from sklearn.pipeline import Pipeline</pre><pre># Construct some pipelines<br>pipe_lr = Pipeline([(&#39;pca&#39;, PCA(n_components=27, random_state=123)),<br>         (&#39;clf&#39;, LogisticRegression(random_state=123))])</pre><pre># Fit the pipelines<br>pipe_lr.fit(X_train, y_train)</pre><pre>print(pipe_lr.score(X_test, y_test))</pre><pre>0.7171541693600517</pre><pre>np.sum(pipe_lr.predict(X_test) == y_test)/len(y_test)</pre><pre>0.7171541693600517</pre><h3>Create a pipeline consisting of a linear SVM, a simple Decision Tree and a simple Random Forest Classifier</h3><p>Repeat the above, but now create three different pipelines:</p><ul><li>One for a standard linear SCM</li><li>One for a default decision tree</li><li>One for a RandomForestClassifier</li></ul><pre># ⏰ This cell may take several minutes to run<br>from sklearn import svm<br>from sklearn.pipeline import Pipeline<br>from sklearn.ensemble import RandomForestClassifier<br>from sklearn import tree</pre><pre>## KEEP IT FOR NOW<br># Construct some pipelines<br>pipe_svm = Pipeline([(&#39;pca&#39;, PCA(n_components=27)),<br>        (&#39;clf&#39;, svm.SVC(random_state=123))])<br>        <br>pipe_tree = Pipeline([(&#39;pca&#39;, PCA(n_components=27)),<br>        (&#39;clf&#39;, tree.DecisionTreeClassifier(random_state=123))])</pre><pre>pipe_rf = Pipeline([(&#39;pca&#39;, PCA(n_components=27)),<br>        (&#39;clf&#39;, RandomForestClassifier(random_state=123))])</pre><pre># List of pipelines, List of pipeline names<br>pipelines = [pipe_svm, pipe_tree, pipe_rf]<br>pipeline_names = [&#39;Support Vector Machine&#39;,&#39;Decision Tree&#39;,&#39;Random Forest&#39;]</pre><pre># Loop to fit each of the three pipelines<br>for pipe in pipelines:<br>    print(pipe)<br>    pipe.fit(X_train, y_train)</pre><pre># Compare accuracies<br>for index, val in enumerate(pipelines):<br>    print(&#39;%s pipeline test accuracy: %.3f&#39; % (pipeline_names[index], val.score(X_test, y_test)))</pre><pre>Pipeline(memory=None,<br>     steps=[(&#39;pca&#39;, PCA(copy=True, iterated_power=&#39;auto&#39;, n_components=27, random_state=None,<br>  svd_solver=&#39;auto&#39;, tol=0.0, whiten=False)), (&#39;clf&#39;, SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,<br>  decision_function_shape=&#39;ovr&#39;, degree=3, gamma=&#39;auto&#39;, kernel=&#39;rbf&#39;,<br>  max_iter=-1, probability=False, random_state=123, shrinking=True,<br>  tol=0.001, verbose=False))])<br>Pipeline(memory=None,<br>     steps=[(&#39;pca&#39;, PCA(copy=True, iterated_power=&#39;auto&#39;, n_components=27, random_state=None,<br>  svd_solver=&#39;auto&#39;, tol=0.0, whiten=False)), (&#39;clf&#39;, DecisionTreeClassifier(class_weight=None, criterion=&#39;gini&#39;, max_depth=None,<br>            max_features=None, max_leaf_nodes=None,<br>            min_impurity_decrease=0.0, min_impurity_split=None,<br>            min_samples_leaf=1, min_samples_split=2,<br>            min_weight_fraction_leaf=0.0, presort=False, random_state=123,<br>            splitter=&#39;best&#39;))])<br>Pipeline(memory=None,<br>     steps=[(&#39;pca&#39;, PCA(copy=True, iterated_power=&#39;auto&#39;, n_components=27, random_state=None,<br>  svd_solver=&#39;auto&#39;, tol=0.0, whiten=False)), (&#39;clf&#39;, RandomForestClassifier(bootstrap=True, class_weight=None, criterion=&#39;gini&#39;,<br>            max_depth=None, max_features=&#39;auto&#39;, max_leaf_nodes=None,<br>           ...timators=10, n_jobs=1,<br>            oob_score=False, random_state=123, verbose=0, warm_start=False))])<br>Support Vector Machine pipeline test accuracy: 0.749<br>Decision Tree pipeline test accuracy: 0.666<br>Random Forest pipeline test accuracy: 0.745</pre><h3>Pipeline with grid search</h3><p>Construct two pipelines with grid search:</p><ul><li>one for random forests — try to have around 40 different models</li><li>one for the adaboost algorithm</li></ul><p>As extra, level-up work, construct a pipeline with grid search for support vector machines.</p><ul><li>Make sure your grid isn’t too big. You’ll see it takes quite a while to fit SVMs with non-linear kernel functions!</li></ul><h3>Random Forest pipeline with grid search</h3><pre># imports<br>from sklearn import svm<br>from sklearn.model_selection import GridSearchCV<br>from sklearn.pipeline import Pipeline</pre><pre># ⏰ This cell may take a long time to run!<br># Construct pipeline<br>pipe_rf = Pipeline([(&#39;pca&#39;, PCA(n_components=27)),<br>            (&#39;clf&#39;, RandomForestClassifier(random_state = 123))])</pre><pre># Set grid search params<br>param_grid_forest = [ <br>  {&#39;clf__n_estimators&#39;: [120],<br>   &#39;clf__criterion&#39;: [&#39;entropy&#39;, &#39;gini&#39;], <br>   &#39;clf__max_depth&#39;: [4, 5, 6],  <br>   &#39;clf__min_samples_leaf&#39;:[0.05 ,0.1, 0.2],  <br>   &#39;clf__min_samples_split&#39;:[0.05 ,0.1, 0.2]<br>  }<br>]</pre><pre># Construct grid search<br>gs_rf = GridSearchCV(estimator=pipe_rf,<br>            param_grid=param_grid_forest,<br>            scoring=&#39;accuracy&#39;,<br>            cv=3, verbose=2, return_train_score = True)</pre><pre># Fit using grid search<br>gs_rf.fit(X_train, y_train)</pre><pre># Best accuracy<br>print(&#39;Best accuracy: %.3f&#39; % gs_rf.best_score_)</pre><pre># Best params<br>print(&#39;\nBest params:\n&#39;, gs_rf.best_params_)</pre><pre>Fitting 3 folds for each of 54 candidates, totalling 162 fits<br>[CV] clf__criterion=entropy, clf__max_depth=4, clf__min_samples_leaf=0.05, clf__min_samples_split=0.05, clf__n_estimators=120 <br>[CV]  clf__criterion=entropy, clf__max_depth=4, clf__min_samples_leaf=0.05, clf__min_samples_split=0.05, clf__n_estimators=120, total=   7.2s<br>[CV] clf__criterion=entropy, clf__max_depth=4, clf__min_samples_leaf=0.05, clf__min_samples_split=0.05, clf__n_estimators=120</pre><pre>                        .<br>                        .</pre><pre>[CV]  clf__criterion=gini, clf__max_depth=6, clf__min_samples_leaf=0.2, clf__min_samples_split=0.2, clf__n_estimators=120, total=   2.0s<br>[CV] clf__criterion=gini, clf__max_depth=6, clf__min_samples_leaf=0.2, clf__min_samples_split=0.2, clf__n_estimators=120 <br>[CV]  clf__criterion=gini, clf__max_depth=6, clf__min_samples_leaf=0.2, clf__min_samples_split=0.2, clf__n_estimators=120, total=   2.0s<br>[CV] clf__criterion=gini, clf__max_depth=6, clf__min_samples_leaf=0.2, clf__min_samples_split=0.2, clf__n_estimators=120 <br>[CV]  clf__criterion=gini, clf__max_depth=6, clf__min_samples_leaf=0.2, clf__min_samples_split=0.2, clf__n_estimators=120, total=   2.0s<br><br><br>[Parallel(n_jobs=1)]: Done 162 out of 162 | elapsed: 11.1min finished<br><br><br>Best accuracy: 0.623<br><br>Best params:<br> {&#39;clf__criterion&#39;: &#39;entropy&#39;, &#39;clf__max_depth&#39;: 6, &#39;clf__min_samples_leaf&#39;: 0.05, &#39;clf__min_samples_split&#39;: 0.05, &#39;clf__n_estimators&#39;: 120}</pre><p>Use your grid search object along with .cv_results to get the full result overview</p><pre>gs_rf.cv_results_</pre><h3>Adaboost</h3><pre># ⏰ This cell may take several minutes to run<br>from sklearn.ensemble import AdaBoostClassifier<br># Construct pipeline<br>pipe_ab = Pipeline([(&#39;pca&#39;, PCA(n_components=27)),<br>            (&#39;clf&#39;, AdaBoostClassifier(random_state = 123))])</pre><pre># Set grid search params<br>adaboost_param_grid = {<br>    &#39;clf__n_estimators&#39;: [30, 50, 70],<br>    &#39;clf__learning_rate&#39;: [1.0, 0.5, 0.1]<br>}</pre><pre># Construct grid search<br>gs_ab = GridSearchCV(estimator=pipe_ab,<br>            param_grid=adaboost_param_grid,<br>            scoring=&#39;accuracy&#39;,<br>            cv=3, verbose=2, return_train_score = True)</pre><pre># Fit using grid search<br>gs_ab.fit(X_train, y_train)</pre><pre># Best accuracy<br>print(&#39;Best accuracy: %.3f&#39; % gs_ab.best_score_)</pre><pre># Best params<br>print(&#39;\nBest params:\n&#39;, gs_ab.best_params_)</pre><pre>                        .<br>                        .</pre><pre>[CV] ..... clf__learning_rate=0.1, clf__n_estimators=70, total=   8.2s<br>[CV] clf__learning_rate=0.1, clf__n_estimators=70 ....................<br>[CV] ..... clf__learning_rate=0.1, clf__n_estimators=70, total=   8.1s<br><br><br>[Parallel(n_jobs=1)]: Done  27 out of  27 | elapsed:  2.7min finished<br><br><br>Best accuracy: 0.665<br><br>Best params:<br> {&#39;clf__learning_rate&#39;: 0.5, &#39;clf__n_estimators&#39;: 70}</pre><p>Use your grid search object along with .cv_results to get the full result overview</p><pre>gs_ab.cv_results_</pre><h3>Level-up: SVM pipeline with grid search</h3><pre># ⏰ This cell may take a very long time to run!<br># Construct pipeline<br>pipe_svm = Pipeline([(&#39;pca&#39;, PCA(n_components=27)),<br>            (&#39;clf&#39;, svm.SVC(random_state=123))])</pre><pre># Set grid search params<br>param_grid_svm = [<br>  {&#39;clf__C&#39;: [0.1, 1, 10]  , &#39;clf__kernel&#39;: [&#39;linear&#39;]},<br>  {&#39;clf__C&#39;: [1, 10], &#39;clf__gamma&#39;: [0.001, 0.01], &#39;clf__kernel&#39;: [&#39;rbf&#39;]},<br> ]</pre><pre># Construct grid search<br>gs_svm = GridSearchCV(estimator=pipe_svm,<br>            param_grid=param_grid_svm,<br>            scoring=&#39;accuracy&#39;,<br>            cv=3, verbose=2, return_train_score = True)</pre><pre># Fit using grid search<br>gs_svm.fit(X_train, y_train)</pre><pre># Best accuracy<br>print(&#39;Best accuracy: %.3f&#39; % gs_svm.best_score_)</pre><pre># Best params<br>print(&#39;\nBest params:\n&#39;, gs_svm.best_params_)</pre><p>Use your grid search object along with .cv_results to get the full result overview</p><pre>gs_svm.cv_results_</pre><p>Note that this solution is only one of many options. The results in the Random Forest and Adaboost models show that there is a lot of improvement possible tuning the hyperparameters further, so make sure to explore this yourself!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4773f72fc1e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bayes’ Theorem]]></title>
            <link>https://medium.com/@toprak.mhmt/bayes-theorem-89daf9f11769?source=rss-2d2112a2cc66------2</link>
            <guid isPermaLink="false">https://medium.com/p/89daf9f11769</guid>
            <category><![CDATA[bayes-theorem]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Mehmet Toprak]]></dc:creator>
            <pubDate>Sat, 29 Aug 2020 12:39:39 GMT</pubDate>
            <atom:updated>2020-08-29T12:39:39.153Z</atom:updated>
            <content:encoded><![CDATA[<p>Bayes theorem is an indispensable law of probability, allowing you to deductively quantify unknown probabilities. The theory rests upon conditional probability. Let’s take a look at it in practice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*GdSYgE8NxZw1XzK5KZ03lg.png" /></figure><h3>Breaking the formula apart</h3><p>Bayes’ theorem is quite intuitive, decomposing the conditional probability of ‘A given B’ in terms of the probability that both events are true divided by the probability that B is true. Bayes theorem takes this natural idea a step further, expressing the probability that both events are true as a conditional probability multiplied by the condition itself.</p><p>To recap:</p><p>Bayes’ Theorem takes the definition of the conditional likelihood:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*1_Is_gPcOilZ1WZa1sxpvw.png" /></figure><p>and rewrites the 𝑃(𝐴∩𝐵)P(A∩B) as 𝑃(𝐵|𝐴)𝑃(𝐴)P(B|A)P(A), which makes perfect sense; the probability of B given A is true, multiplied by the probability that A is true, gives us the probability that both are true.</p><p>Making this substitution, you have Bayes’ Theorem:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*7tSzvDBV0J4CNzVFPTV1PA.png" /></figure><h3>A simple example</h3><p>Let’s take a simple theoretical example to demonstrate. Imagine there are two fish tanks at the local pet store. The small tank holds 10 Betta fish. The large tank has 200 goldfish and 35 Betta fish. Given that a fish is a Betta fish, what’s the probability it comes from the small tank?</p><p>On the one hand, it seems that if you were to select a fish from the large tank, you’d probably end up with a goldfish. However, because these tanks are of such vastly different sizes, the probability that the fish came from the larger tank is actually more probable.</p><p>Using Bayes’ theorem, you are looking to find the probability that the fish came from the small tank, given that it is a Betta fish:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*SEBJXRhTR6b138d2nSWMoA.png" /></figure><p>While concrete, this example fails to demonstrate the full power of Bayes’ theorem since you had all of the underlying information, so you don’t even need to use Bayes’ theorem. You could have simply looked at the number of Betta fish in the small tank versus the number of Betta fish overall: 10/45 giving you exactly the same result.</p><h3>An NLP example</h3><p>With this simple example out of the way, let’s examine a more practical example from the field of Natural Language Processing.</p><p>A common introductory example to Natural Language Processing or classification is detecting spam. While you may enjoy spam in a can, you probably don’t enjoy getting spam in your inbox. Bayes’ theorem can serve as a natural classification method in these scenarios. Assume that the word “offer” (as in Special Offer, We Have an Offer for You, or Don’t Miss This Offer!) occurs in 73% of the spam messages you receive. In comparison, only 10% of your desired mail contains the word “offer”. If 20% of the messages you receive are spam, and you receive another message with the word “offer”, what is the probability that it is spam?</p><p>As you might have guessed, you can solve this using the Bayes’ theorem!</p><p>First, set up the problem:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0aEnkIMf0YfDUBFT6b5zhw.png" /></figure><p>As you can see, while spam has a much higher occurrence of the word “offer”, the presence of the word alone does not provide strong confidence that the message is spam. To provide more statistical power, you will eventually extend Bayes’ theorem to multiple observations simultaneously using the relative probabilities of multiple words.</p><p>In this blog, you were introduced to the Bayes’ theorem, and saw how it can be used to quantify conditional probabilities.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=89daf9f11769" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Convolutional Neural Networks (CNNs) for Classification]]></title>
            <link>https://medium.com/@toprak.mhmt/convolutional-neural-networks-cnns-for-classification-dc5705f2adce?source=rss-2d2112a2cc66------2</link>
            <guid isPermaLink="false">https://medium.com/p/dc5705f2adce</guid>
            <category><![CDATA[image-recognition]]></category>
            <category><![CDATA[convolutional-network]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Mehmet Toprak]]></dc:creator>
            <pubDate>Wed, 19 Aug 2020 19:53:40 GMT</pubDate>
            <atom:updated>2020-08-19T19:53:40.769Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello! In this blog, I explain how CNN can solve an image classification problem such as digit recognition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aB4IS1o2Tw8JuAnIrRJ5xw.png" /></figure><p>First, let’s see why traditional shallow neural networks do not work as well as CNN. As mentioned before, one of the important steps in the modeling for classification with Shallow Neural Networks, is the feature extraction step. These chosen features could simply be the color, object edges, pixel location, or countless other features that could be extracted from the images. Of course, the better and more effective the feature sets you find, the more accurate and efficient the image classification you can obtain. However, as you can imagine, the process of selecting the best features is tremendously time-consuming, and is often ineffective. Also, extending the features to other types of images becomes an even greater problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2y7mda6ju2h2TYjG603lYg.png" /></figure><p>Convolutional neural networks (or CNNs) try to solve this problem by using more hidden layers, and also with more specific layers. So, when using CNN, instead of you choosing image features, to classify dogs vs. cats, for instance, CNNs can automatically find those features and classify the images for you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AIxwyLGi5e3IDhKu9XRGPQ.png" /></figure><p>Instead of using a dogs and cats dataset, though, let’s use a more practical example here. The MNIST dataset is a “database of handwritten digits that has a training set of 60,000 examples. The great thing about the MNIST dataset is that the digits have been size-normalized and centered in a fixed-size image”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uMzFoGQyj9DdRbT8VBs6yA.png" /></figure><p>Now, consider the digit recognition problem. We would like to classify images of handwritten numbers, where the observations are the intensity of the pixels. And, the target will be a digit, from 0 to 9. So, our objective here is to build a digit recognition system using CNN.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JCjoUxjdI5WTf0sYV--kaw.png" /></figure><p>Now let’s look at it from a higher level. Basically, if we look at the pipeline of our deep learning process, we can see the following phases: First, pre-processing of input data. Second, training the deep learning model. And third, inference and deployment of the model. In the first part, we have to convert the images to a readable and proper format for our network. Then, an untrained network is fed with a big dataset of the images in the Training phase, so as to allow the network to learn. That is, we build a CNN, and train it with many hand-written images in the training set. Finally, we use the trained model in the Inference phase, which classifies a new image by inferring its similarity to the trained model. So, this model can be deployed and used as a digit recognition model for unseen cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QgaOszM_PqyLgZnNu9uwUQ.png" /></figure><p>Now let’s focus on the training process. As mentioned before, a deep neural network not only has multiple hidden layers, the type of layers and their connectivity also is different from a shallow neural network, in that it usually has multiple Convolutional layers, pooling layers, as well as fully connected layers. The convolutional layer applies a convolution operation to the input, and passes the result to the next layer. The pooling layer combines the outputs of a cluster of neurons in the previous layer into a single neuron in the next layer. And then, the fully connected layers connect every neuron in the previous layer to every neuron in the next layer. In the next blog, you will learn more about these layers and their specifications, in detail. Thanks for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dc5705f2adce" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Convolutional Neural Networks (CNNs)]]></title>
            <link>https://medium.com/@toprak.mhmt/introduction-to-convolutional-neural-networks-cnns-f88129da62bb?source=rss-2d2112a2cc66------2</link>
            <guid isPermaLink="false">https://medium.com/p/f88129da62bb</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[image-recognition]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[convolutional-network]]></category>
            <dc:creator><![CDATA[Mehmet Toprak]]></dc:creator>
            <pubDate>Sun, 16 Aug 2020 18:44:48 GMT</pubDate>
            <atom:updated>2020-08-16T18:44:48.945Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello.In this blog, I’ll be reviewing a brief introduction to the convolutional neural network model. Convolutional neural networks or CNNs have gained a lot of attention in the machine learning community over the last few years.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pSqZnXco3rr292giPOyxoQ.png" /></figure><p>This is due to the wide range of applications where CNNs excel, such as object detection and speech recognition. In a later part of this blog, I’ll examine the object recognition problem, which is a key consideration when using CNNs. You’ll also get an idea of how a CNN structure allows it to extract the elements from an image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*CULlonbyWY5URZ0UYtD6kA.png" /></figure><p>For example, you’ll see how a CNN learns to pick out a person or a dog or even chairs from an image, even if the chairs are only partially visible. Indeed, this is not an easy task for computers and it took years of dedicated research to achieve this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*scjJ_gB0V_fZW6btj6Ztsw.png" /></figure><p><em>The original goal of the CNN was to form the best possible representation of our visual world in order to support recognition tasks.</em> The CNN solution needed to have two key features to make it viable. First, It needed to be able to detect the objects in the image and place them into the appropriate category. Second, it also needed to be robust against differences in pose, scale, illumination, confirmation and clutter. The importance of this second feature simply cannot be overstated, since this was historically a huge limitation of hand-coded algorithms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mwiTlOtXBoeWB8R9JzcH-A.png" /></figure><p>Interestingly enough, the solution to the object recognition issue was inspired by examining the way our own visual cortex operates. In short, the CNN starts with an input image. It then extracts a few primitive features and combines those features to form certain parts of the object. Finally, it pulls together all of the various parts to form the object itself. In essence, it is a hierarchical way of seeing objects. That is, in the first layer, very simple features are detected. Then these are combined to shape more complicated features in the second layer and so on to detect things like a cat or a dog or any other object we’re needing to detect. The convolutional neural network was developed from this sequence of steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wlw_z5tqjd1rdfcPYY9W1Q.png" /></figure><p>Let’s consider the building in the lower right-hand corner of this image. How can a trained CNN learn and recognize that it’s an image of a building, rather than some of the other elements that are situated in the photograph, such as a person or even those clouds in the sky?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-dAKfHwB-qRY4FYcSDGS4g.png" /></figure><p>Well, in the training phase, CNNs would receive many building images as input. Then CNNs will automatically find that the best primitive features for a building would be things like horizontal and vertical lines. Much like the neurons firing in a person’s visual cortex, the first layer in a CNN reacts when it receives an image that has horizontal or vertical lines. Once these simpler features such as lines are combined, CNNs learn to form higher abstract components, like the windows of the building and the building’s external shape. It can then use these basic parts to form the complete object and learn how the entire building essentially looks like. Later, if it sees an image of a building that it hasn’t seen before, CNNs can make a decision if the new input image is of a building based on the persistence of the various features it has stored. It is the same process for learning and detecting other objects, such as faces, animals, cars, and so on. So as you can see, the CNN is a set of layers with each of them being responsible to detect a set of feature sets, and these features are going to be more abstract as it goes further into examining the next layer. So at this point, you should have a basic understanding of the principles behind a convolutional neural network, and how CNNs might be used in an application. Thanks for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f88129da62bb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is MSSQL?]]></title>
            <link>https://medium.com/@toprak.mhmt/what-is-mssql-9a152d7d4ed0?source=rss-2d2112a2cc66------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a152d7d4ed0</guid>
            <category><![CDATA[mssql]]></category>
            <category><![CDATA[sql-server]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Mehmet Toprak]]></dc:creator>
            <pubDate>Wed, 05 Aug 2020 22:47:21 GMT</pubDate>
            <atom:updated>2020-08-05T22:47:21.188Z</atom:updated>
            <content:encoded><![CDATA[<p>MSSQL is a suite of database software published by Microsoft and used extensively within our enterprise. Typically, it includes a relational database engine, which stores data in tables, columns and rows, Integration Services (SSIS), which is a data movement tool for importing, exporting and transforming data, Reporting Services (SSRS), which is used to create reports and serve reports to end users, and also Analysis Services (SSAS), which is a multidimensional database used to query data from the main database engine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/0*So1LzZwBYRNgp2i0.jpg" /></figure><p>Microsoft SQL Server (MSSQL) is widely used in enterprise deployments. MSSQL is a scalable data platform which includes several ETL (Extract, Transform and Load) tools and reporting services where data can be added, modified and queried using a standardized structured query language (SQL). MSSQL is an evolving data platform used for mission critical business and data solutions on premise, in the cloud and on hybrid platforms.i</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/120/0*bb3kA4oJfk2C0D_7.png" /></figure><h3>A Brief History of MSSQL</h3><p><a href="https://www.atlantic.net/cloud-hosting/how-to-using-basic-sql-syntax/">SQL syntax</a> and MSSQL are related but distinct. SQL syntax is the language used to query databases, and MSSQL is the Microsoft database product suite which uses SQL syntax. The very first versions of SQL server were developed by Ashton Tate, Sybase and Microsoft between 1988 to 1993 and were built on Unix-based operating systems. As Microsoft started to dominate desktop operating systems in the early 1990’s, focus shifted to develop SQL Server for Windows. In 1993, SQL Server 4.21 was released, and this was the first SQL database platform created for Microsoft Windows to take advantage of the graphical user interface.</p><p>Up to now, Sybase had licensed their Dbase database technology to Microsoft; this technology was widely used in SQL Server, but the terms of the license did not allow Microsoft to change the source code without explicit consent from Sybase. In 1994, the companies parted ways, and Microsoft shifted development focus to release SQL Server 6.0 (SQL95). They quickly followed in 1996 with SQL Server 6.5, which introduced Internet and Data Warehousing support for the fledgling World Wide Web.ii</p><p>In 1998, Microsoft completely rewrote SQL Server 7.0 removing any legacy Sybase functions and adding a vast amount of new features. SQL Server 2000 was released to coincided with Windows Server 2000.iii This started off the SQL Server suite release cycle familiar today, all subsequent versions of SQL have been upgraded with added features and services and generally coincide with an Operating System release date.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/120/0*xH_fr7xV5ISMFGR-.png" /></figure><h3>Why Choose MSSQL</h3><p>MSSQL server is an incredibly popular database solution used today, and one of its strongest advantages is its ease of use. MSSQL comes with many excellent tools which make database development a fast and agile process. SQL Server management studio allows any approved user to manage and maintain the databases, run SQL queries, perform backups and analyze performance charts. MSSQL integrates with Visual Studio to give your DevOps team a powerful, familiar platform to create and manage custom applications which seamlessly integrate with MSSQL Server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/120/0*5guR0kRLJFpkj1uq.png" /></figure><h3>MSSQL Main Features</h3><p>There are many products which make up the SQL Server database platform, but there are 4 key services built into MSSQL which define it and make it a popular choice as a database management system (DBMS). These options are available to install when deploying the MSSQL instance. The latest releases of MSSQL are not just compatible with Windows; more recently, Microsoft has offered SQL for Linux (Red hat and SUSE), as well as Docker container platforms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/120/0*zUsGa2teyyXTjEQk.png" /></figure><h3>Database Engine</h3><p>The SQL Database engine is the core of the MSSQL product suite. This was the original product which is used to store, process and secure data. The data is stored in one or many database instances. Some of the key database engine features include storing data in instance tables, and the ability to do XML data import, Blob data management (Binary Large Objects), DB Triggers, transaction logs, data compression, data search and maintenance plans, to name a few.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/120/0*VG1cn8LqvRMcV1Ny.png" /></figure><h3>Integration Services (SSIS)</h3><p>SSIS is a data movement tool which can import and export data from a database. It is widely used to design ETL processes (Extract, Transform, Load). You can <em>extract</em> data from almost any source (for example, other databases, text files or Excel documents), <em>transform</em> it by merging, filtering, sorting fields or aggregating dataiv, and you can <em>load</em> this data into a destination, often a shared folder or even another database/application.v</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/120/0*gzfuVCBRM9zwhKyY.png" /></figure><h3>Reporting Services (SSRS)</h3><p>SSRS is a comprehensive reporting platform for SQL Server which is used to create and deploy several types of paginated reports. SSRS can report on any data in a database and display it in either simple charts or complex data visualizations. SSRS can report on almost any database source data. Reports can be displayed on the reporting services website or integrated into any application supporting .NET. SSRS statements are commonly used by companies to display KPIs like sales data or number of orders processed per hour and thus are one of the features most highly sought by executive teams.vi</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/120/0*Q1eEHDRdYbSUeJgu.png" /></figure><h3>Analysis Services (SSAS)</h3><p>SSAS is a multidimensional, online analytical processing (OLAP) and data mining server.vii SSAS is a separate database which is fed data via SSIS from any other data source to build cubes of relational data in a data warehouse. This is a highly efficient database allowing deep queries to run in near-instant timing across vast amounts of data. An example of a typical SSAS statements would be the analysis of all sales in the EMEA region for a particular month; SSAS will allow you to do this using MDX expressions.viii</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/120/0*tKW37ITrT6hCsaS5.png" /></figure><h3>Summary</h3><p>MSSQL Server is one of the best database solutions available today in the SQL marketplace. If you are looking for a secure, easy to manage and high-performance database management system, then MSSQL should be a serious option to consider. It enables users to analyze data, forecast sales and even predict customer behaviour using business intelligence analytics.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a152d7d4ed0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MySQL vs MS SQL Server]]></title>
            <link>https://medium.com/@toprak.mhmt/mysql-vs-ms-sql-server-ed6affee6924?source=rss-2d2112a2cc66------2</link>
            <guid isPermaLink="false">https://medium.com/p/ed6affee6924</guid>
            <category><![CDATA[relational-databases]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mssql]]></category>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Mehmet Toprak]]></dc:creator>
            <pubDate>Fri, 31 Jul 2020 15:31:11 GMT</pubDate>
            <atom:updated>2020-07-31T15:31:11.514Z</atom:updated>
            <content:encoded><![CDATA[<p>We’re looking at MySQL vs MS SQL Server — Which Reigns Supreme?Relational database management systems (RDBMS) form the backbone of enterprise IT. Data-driven business intelligence, analytics, and reporting workloads simply couldn’t exist without the support of a high-powered, high-performance relational database. Good databases can bring your enterprise IT to the next level — while bad databases can bring it to its knees.</p><p>The crucial question is then: which RDBMS is right for your business needs and objectives? When comparing relational database solutions, two names seem to come up more than any others: MySQL and Microsoft SQL Server.</p><p>MySQL is an open-source RDBMS solution that was purchased by Oracle in 2008. As a component of the popular LAMP web application development stack, MySQL powers some of the world’s most highly visited websites, including Facebook, Twitter, and YouTube.</p><p>Microsoft SQL Server (MSSQL) is an RDBMS solution developed and maintained by Microsoft. First released in 1989, SQL Server is now available in many different versions with different feature sets, including Enterprise, Standard, and Express versions.</p><p>The question of MySQL vs. MS SQL Server is a tough one, and there’s no right answer for every organization. Below, we’ll go over the most important factors to consider when choosing between SQL Server and MySQL.</p><h3>MySQL vs. Microsoft SQL Server: Operating System</h3><p>When selecting an enterprise-grade database, most companies have already invested a great deal of time, money, and expertise in their IT infrastructure — including their choice of operating system. This means that whether you go with MySQL or SQL Server, your choice of database solution needs to mesh well with your existing workflow.</p><p>Fortunately, both MySQL and SQL Server run on virtually all major operating systems. MySQL is traditionally associated with Linux as part of the famed LAMP stack (Linux, Apache, MySQL, PHP). However, it also offers full-fledged support for Windows and Mac OS X, as well as other Unix-based operating systems.</p><p>SQL Server, meanwhile, is developed by Microsoft and was originally written for Windows. In recent years, however, Microsoft has made strides in embracing the open-source community, which includes providing support for both Linux and Mac OS X. The most recent versions of SQL Server run on Linux natively, and will run on Mac OS X within a Docker container.</p><p>The bottom line: The right call here depends on what operating system your company is already using. Both platforms support Windows and Linux, although there are certain “home court advantages” to each one.</p><p>Using SQL Server makes a little more sense if you’re already a Windows and .NET shop. On the other hand, if you use Linux and Python/Java/PHP, MySQL is probably the better choice here.</p><h3>MySQL vs. Microsoft SQL Server: Performance</h3><p>Database performance is crucial for any software application. If your database doesn’t respond quickly and efficiently, the entire system is slowed down, causing unacceptable delays to the end user. Poor database performance directly harms key business metrics such as user experience and revenue.</p><p>However, database performance is also heavily dependent on an immense number of variables. Slight differences in workloads can dramatically skew results, while minor tweaks can see major improvements. In other words, a well-designed database is worth its weight in gold.</p><p>MySQL and SQL Server both tout extensive performance and scaling capabilities. When it comes down to cold, hard facts, however, SQL Server seems to have the advantage. <a href="https://www.ijarcce.com/upload/2015/march-15/IJARCCE%2039.pdf">According to one study</a>, SQL Server consistently beat MySQL in comparison testing involving high volumes of SELECT, INSERT, UPDATE, and DELETE queries.</p><p>Also note that MySQL is considered to be Oracle’s “entry-level” database. For top-shelf performance needs, Oracle will likely steer you toward their flagship database offering, <a href="https://www.oracle.com/database/technologies/">Oracle Database</a>. On the other hand, SQL Server is Microsoft’s flagship database offering.</p><p>The bottom line: It’s not a slam dunk, but SQL Server’s better numbers and flagship status give it the edge here.</p><h3>MySQL vs. Microsoft SQL Server: Toolset</h3><p>No database is an island — whether you choose MySQL or SQL Server, you’ll need a good toolset to work with it. Both solutions offer front-end clients specifically for this purpose.</p><p>MySQL’s client application is <a href="https://www.mysql.com/products/workbench/">MySQL Workbench</a>, which runs on Windows, Linux, and Mac OS X. MySQL Workbench offers several important database management tools, including:</p><ul><li>Database connection and management</li><li>SQL editing and execution</li><li>Database and schema modeling</li><li>Performance monitoring and query statistics</li></ul><p>SQL Server’s client application is <a href="https://docs.microsoft.com/en-us/sql/ssms/download-sql-server-management-studio-ssms?view=sql-server-ver15">SQL Server Management Studio (SSMS)</a>. Note that SSMS is only available on Windows, unlike SQL Server itself. However, Microsoft provides an extension for the Visual Studio Code editor to execute SQL from a Linux machine.</p><p>SSMS has a more robust feature set than MySQL Workbench, including:</p><ul><li>Extensive database management tools</li><li>A robust set of security, reporting, and analysis services</li><li>A powerful execution plan visualizer for identifying performance bottlenecks</li><li>Integrated source control</li><li>Real-time activity monitoring, including filtering and automatic refresh</li></ul><p>The bottom line: Both toolsets offer the basics: the ability to execute SQL, as well as view and manage databases. However, SSMS for SQL Server comes out on top here, with features that please novice users and seasoned DBAs alike. The execution plan visualizer, in particular, can pay for itself many times over.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ed6affee6924" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is Microsoft Azure and Why Use It?]]></title>
            <link>https://medium.com/@toprak.mhmt/what-is-microsoft-azure-and-why-use-it-d4645645c848?source=rss-2d2112a2cc66------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4645645c848</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[microsoft-azure]]></category>
            <dc:creator><![CDATA[Mehmet Toprak]]></dc:creator>
            <pubDate>Sat, 25 Jul 2020 02:10:36 GMT</pubDate>
            <atom:updated>2020-07-25T02:10:36.304Z</atom:updated>
            <content:encoded><![CDATA[<h3>An In-Depth Look at the Cloud Services Platform from Microsoft</h3><p>Today plenty of businesses still have real concerns about migrating applications to the cloud. Fears about network security, availability, and potential downtime swirl through the heads of chief decision makers, sometimes paralyzing organizations into standing pat on existing tech — even though it’s aging by the minute.</p><p>Enter Microsoft Azure, the industry leader’s solution for going to a partially or totally cloud-based architecture. Below is a detailed look at what Azure is, the power of partnering with Microsoft for a cloud or hybrid cloud solution, and the best way to get full and actionable visibility into your aggregated logs and infrastructure metrics so your organization can react quickly to opportunities.</p><h4>What is Microsoft Azure?</h4><p>Microsoft has leveraged its constantly-expanding worldwide network of data centers to create Azure, a cloud platform for building, deploying, and managing services and applications, anywhere. Azure lets you add cloud capabilities to your existing network through its platform as a service (PaaS) model, or entrust Microsoft with all of your computing and network needs with Infrastructure as a Service (IaaS). Either option provides secure, reliable access to your cloud hosted data — one built on Microsoft’s proven architecture. Azure provides an ever expanding array of products and services designed to meet all your needs through one convenient, easy to manage platform. Below are just some of the capabilities Microsoft offers through Azure and tips for determining if the Microsoft cloud is the right choice for your organization.</p><h3>What can Microsoft Azure Do?</h3><p>Microsoft maintains a growing directory of Azure services, with more being added all the time. All the elements necessary to build a virtual network and deliver services or applications to a global audience are available, including:</p><ul><li>Virtual machines</li></ul><p>Create Microsoft or Linux virtual machines (VMs) in just minutes from a wide selection of marketplace templates or from your own custom machine images. These cloud-based VMs will host your apps and services as if they resided in your own data center.</p><ul><li>SQL databases</li></ul><p>Azure offers managed SQL relational databases, from one to an unlimited number, as a service.This saves you overhead and expenses on hardware, software, and the need for in-house expertise.</p><ul><li>Azure Active Directory Domain services</li></ul><p>Built on the same proven technology as Windows Active Directory, this service for Azure lets you remotely manage group policy, authentication, and everything else. This makes moving and existing security structure partially or totally to the cloud as easy as a few clicks.</p><ul><li>Application services</li></ul><p>With Azure it’s easier than ever to create and globally deploy applications that are compatible on all popular web and portable platforms. Reliable, scalable cloud access lets you respond quickly to your business’s ebb and flow, saving time and money. With the introduction of Azure WebApps to the Azure Marketplace, it’s easier than ever to manage production, testing and deployment of web applications that scale as quickly as your business. Prebuilt APIs for popular cloud services like Office 365, Salesforce and more greatly accelerate development.</p><ul><li>Visual Studio team services</li></ul><p>An add-on service available under Azure, Visual Studio team services offer a complete application lifecycle management (ALM) solution in the Microsoft cloud. Developers can share and track code changes, perform load testing, and deliver applications to production while collaborating in Azure from all over the world. Visual Studio team services simplify development and delivery for large companies or new ones building a service portfolio.</p><ul><li>Storage</li></ul><p>Count on Microsoft’s global infrastructure to provide safe, highly accessible data storage. With massive scalability and an intelligent pricing structure that lets you store infrequently accessed data at a huge savings, building a safe and cost-effective storage plan is simple in Microsoft Azure.</p><h3>Why are people trusting their workloads to Microsoft Azure?</h3><p>It’s been said that the on-premise data center has no future. Like mainframes and dial-up modems before them, self-hosted data centers are becoming obsolete, being replaced by increasingly available and affordable cloud solutions. Several important players have emerged in the cloud service sphere, including Amazon Web Services (AWS), perennial computing giant IBM, and Apple’s ubiquitous iCloud, which holds the picture memories and song preferences of hundreds of millions of smartphone users, among other data. With so many options, why are companies like 3M, BMW, and GE moving workloads to Microsoft Azure? Just some of the reasons:</p><ul><li>Flexibility</li></ul><p>With Microsoft Azure you can spin up new services and geometrically scale your data storage capabilities on the fly. Compare this to a static data center, which would require new hardware and OS purchasing, provisioning, and deployment before additional power could be brought to bear against your IT challenges. This modern flexibility makes Azure a tempting solution for organizations of any size.</p><ul><li>Cost</li></ul><p>Azure solutions don’t just make it faster and easier to add and scale infrastructure, they make it cheaper. Physical services and infrastructure devices like routers, load balancers and more quickly add up to thousands or even hundreds of thousands of dollars. Then there’s the IT expertise required to run this equipment, which amounts to major payroll overhead. By leveraging Microsoft’s massive infrastructure and expertise, Azure can trim our annual IT budget by head-turning percentages.</p><ul><li>Applications</li></ul><p>With a la carte service offerings like Visual Studio Team Services, Visual Studio Application Insights, and Azure’s scalable, on-demand storage for both frequently accessed and ‘cold’ data, Microsoft makes developing and testing mission-critical apps a snap. Move an application from test to production mode on the fly across a globally distributed network. Microsoft also offers substantial licensing discounts for migrating their existing apps to Azure, which represents even more opportunity for savings.</p><ul><li>Disaster recovery</li></ul><p>Sometimes the unthinkable becomes the very immediate reality. Another advantage of Microsoft Azure lay in its high-speed and geographically decentralized infrastructure, which creates limitless options for disaster recovery plans. Ensure that your critical application and data can run from redundant sites during recovery periods that last minutes or hours instead of days. Lost time is lost business, and with Azure you can guarantee continuous service delivery even when disaster strikes.</p><p>The combination of Microsoft’s vast infrastructure, constant application and services development, and powerful presence in the global IT marketplace has made Microsoft Azure solutions the choice of two-thirds of the world’s Fortune 500 companies. But the infinite scalability of Azure can make it just as right for your small personal business.</p><h3>Logging capabilities within Microsoft Azure</h3><p>The secret gold mine of any infrastructure and service solution is ongoing operational and security visibility, and ultimately these comes down to extracting critical log and infrastructure metrics from the application and underlying stack. The lack of this visibility is like flying a plane blind — no one does it. Azure comes with integrated health monitoring and alert capabilities so you can know in an instant if performance issues or outages are impacting your business. Set smart alert levels for events from:</p><ul><li>Azure diagnostic infrastructure logs</li></ul><p>Get current insights into how your cloud network is performing and take action to resolve slow downs, bottlenecks, or service failures.</p><ul><li>Windows IIS logs</li></ul><p>View activity on your virtual web servers and respond to traffic patterns or log-in anomalies with the data Azure gathers on IIS 7.</p><ul><li>Crash dumps</li></ul><p>Even virtual machines can ‘blue screen’ and other virtual equipment crashes can majorly disrupt your operations. With Microsoft Azure you can record crash dump data and troubleshoot to avoid repeat problems.</p><ul><li>Custom error logs</li></ul><p>Set Azure alerts to inform you about defined error events. This is especially helpful when hosting private applications that generate internal intelligence about operations, so you can add these errors to the health checklist Azure maintains about your network.</p><p><em>Microsoft Azure gives you the basic tools you need for error logging and monitoring, diagnostics, and troubleshooting to ensure continuous service delivery in your Azure cloud environment.</em></p><h3>Gain Full Visibility into Azure with the Unified Logs and Metrics</h3><p>Even with Azure’s native logging and analytics tools, the vast amount of data flowing to make your network and applications operate can be overwhelming. The volume, variety and velocity of cloud data should not be underestimated. With the help of Sumo Logic, a trusted Microsoft partner, management of that data is simple.</p><p>The Sumo Logic platform unifies logs and metrics from the structured, semi-structured, and unstructured data across your entire Microsoft environment. Machine learning algorithms process vast amounts of log and metrics data, looking for anomalies and deviations from normal patterns of activity, alerting you when appropriate.</p><p>With Log Reduce, Log Compare and Outlier Detection, extract continuous intelligence from your application stack and proactively respond to operational and security issues.</p><p>The Sumo Logic apps for Microsoft Azure Audit, Microsoft Azure Web Apps, Microsoft Windows Server Active Directory, Microsoft Internet Information Services (IIS), and the popular Windows Performance app, make ingesting machine data in real-time and rendering it into clear, interactive visualizations for a complete picture of your applications and data.</p><p>Before long the on-premise data center — along with its expensive hardware and hordes of local technicians on the payroll — may be lost to technology’s graveyard. But smart, researched investment into cloud capabilities like those provided in Microsoft Azure will make facing tomorrow’s bold technology challenges and possibilities relatively painless.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4645645c848" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>