<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Soumya Ranjan Mishra on Medium]]></title>
        <description><![CDATA[Stories by Soumya Ranjan Mishra on Medium]]></description>
        <link>https://medium.com/@soumya1999rta?source=rss-72aad6e9c52b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*r8xgaLXREN3_xseQJQW11g.jpeg</url>
            <title>Stories by Soumya Ranjan Mishra on Medium</title>
            <link>https://medium.com/@soumya1999rta?source=rss-72aad6e9c52b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 17:23:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@soumya1999rta/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How should you approach designing an Uber Trip System?]]></title>
            <link>https://soumya1999rta.medium.com/how-should-you-approach-designing-an-uber-trip-system-6f3fa7380542?source=rss-72aad6e9c52b------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f3fa7380542</guid>
            <category><![CDATA[trip]]></category>
            <category><![CDATA[uber]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[system-design-assignment]]></category>
            <dc:creator><![CDATA[Soumya Ranjan Mishra]]></dc:creator>
            <pubDate>Sun, 04 Jul 2021 10:34:34 GMT</pubDate>
            <atom:updated>2021-07-04T10:34:34.057Z</atom:updated>
            <content:encoded><![CDATA[<p>Contributors: <a href="https://huda-nur-ed.medium.com/">F M Nurul Huda Pathan</a>, <a href="https://medium.com/@soumya1999rta">Soumya Ranjan Mishra</a></p><p>This article aims to make you understand the System Design of the UBER application. We hope you will enjoy the blog. Let’s Start!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/0*6dAK4S_cgAXKdhfs" /></figure><p>Uber is an American technology company that provides ride-hailing, food delivery (Uber Eats), package delivery, couriers, freight transportation, and, through a partnership with Lime, electric bicycle and motorized scooter rental. Most of its revenues come from ride-hailing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/0*qBxSKOMm8irh8EHM" /></figure><p>Let’s now discuss how the ride-hailing operations are done in Uber.</p><ul><li>Uber provides different service options. It includes a private ride in a car with a driver for up to four passengers. Depending on the location, It also offers other levels of service at different prices including black luxury vehicles, newer or premium level vehicles, cars with leather seats, sport utility vehicles, minivans, vans, hatchbacks, electric cars, hybrid vehicles, motorcycles, auto-rickshaws, actual taxicabs, lower-cost shared transport with other passengers going in the same general direction.</li><li>It determines the fees and terms on which drivers transport riders. It uses a dynamic pricing model. Fares fluctuate depending on the local supply and demand at the time of service. Customers are quoted the fare in advance. Service is generally accessed via a mobile app.</li><li>Users need to set up a personal profile with a name, phone number, other information, and payment preference, which could be a credit card, e-commerce payment system, or, in some cases, cash. After the service is complete, the customer may be given the option to provide a gratuity to the driver, which is also billed to the customer’s payment method.</li><li>Drivers usually provide a vehicle, which could be owned, rented, or leased. Drivers must meet requirements for age, health, car age, and type, have a driver’s license and a smartphone or tablet, and may be required to pass a background check. In many cities, vehicles must pass annual safety inspections and/or must have an emblem posted in the passenger window. Some cities also require drivers to have a business license. There may be accommodations for hearing-impaired drivers. Drivers may be notified before accepting a trip if it will be longer than 45 minutes.</li><li>After each trip completion, drivers and customers may rate each other, and users with low ratings may be deactivated.</li></ul><p>Now, let’s discuss how an Uber Trip system can be designed. The system must have the following standard features.</p><ul><li>Riders will be able to view the available drivers in nearby locations.</li><li>Riders will be able to request rides from source to destination.</li><li>Nearby drivers will get notified of the ride, and one of the drivers will confirm the ride.</li><li>Pickup ETA will be shown to customers when a ride is dispatched to pick up the rider.</li><li>Once the trip is completed, the rider must be shown the trip details such as the trip map, price, etc. in the ride receipt.</li></ul><p>The above features when implemented will give the following lifecycle diagram for a trip.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*53oDwyxoSfiT0-xf" /></figure><p>The Uber Trip System architecture will have the following components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/0*B8QZskPj4ru70DTZ" /></figure><h4>Pre-Trip Component:</h4><p>It has the following subcomponents as</p><ul><li>Driver Location Manager: This component will be responsible for maintaining the changing driver locations. It will ingest the messages emitted by the drivers’ uber app containing their current locations and update the car location index.</li><li>Trip Dispatcher: It will be responsible for handling the trip requests from users and dispatch a driver in response to those requests.</li><li>Arrival ETA Calculator: This component will be responsible for calculating the ETA for the driver to reach the rider once the driver has accepted the ride.</li></ul><h4>On-Trip Component:</h4><p>It has the following sub-components which include</p><ul><li>Trip Recorder: GPS signals transmitted from the ride when the trip is in progress are recorded. These GPS signals will be then recorded in a data store which will be used by subsequent sub-component systems such as Map Matcher and Pricing Calculator.</li><li>Map Matcher: It will be responsible for matching the trip on the actual map using algorithms specialized for this purpose.</li><li>Price Calculator: It will then use the recorded trip information for computing the price which users have to pay for the trip.</li></ul><p><strong>Data Storage:</strong></p><ul><li>Driver Locations: Here, the driver location information such as latitude and longitudes will be stored.</li><li>Trip Records: This database will record trip details. These details can be used to draw insights by the company to draw patterns and business insights.</li></ul><p><strong>How are the pre-trip components designed?</strong></p><p>Pre-trip components perform the following actions.</p><ul><li>Update Driver Location -It involves updating the driver locations as drivers keep changing their locations.</li><li>List nearby drivers-It comprises the steps involved in fetching the list of nearby drivers.</li><li>Requesting for a ride-It includes a series of steps involved in requesting and dispatching a ride.</li></ul><p>These components are designed to show the cars in the nearby locations and dispatch the ride on users’ requests. The location of the cars can be stored in an in-memory car location index, which can support high read and write requests. The driver app will keep on sending the updated car locations, which will be updated in this index. The index will be queried when the rider app requests for nearby car locations or places a ride request.</p><p>Car Location Index will maintain the driver locations in a distributed index, which is spread across multiple nodes. Nodes contain driver information containing their locations, the number of people in the car, and if they are on a trip. These pieces of information are aggregated as objects and are distributed across the nodes using some mechanisms (e.g., city, product, and so forth) along with replication.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/0*8gYjH1JXfERUVhRQ" /></figure><p>Some properties a car index location must contain are-</p><ul><li>It must store the current locations of all drivers and must support fast queries by location and properties such as driver’s trip status.</li><li>It must support a high volume of reads and writes.</li><li>The data stored in these indexes must be ephemeral, and we don’t need long-term durable storage.</li></ul><p>ETA for the car arrival is calculated by considering factors such as route, traffic, and weather account. There are two primary components of estimating an ETA given an origin and destination on a road network such as</p><ul><li>calculating the route from origin to destination with the least cost</li><li>estimating the time taken to traverse the route.</li></ul><p>Every intersection of a road on a map is represented by a node and each road segment by a directed edge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*w_JD4inGuNuQ6mDf" /></figure><p>We can use Dijkstra’s algorithm to find the shortest path between source and destination. This algorithm has a time complexity of O(N*logN) to find the shortest path for N nodes. But ride-hailing apps like Uber require a highly scaled approach to determine the routes. To solve this we can use partitioning the entire graph, pre-computing the best path within partitions, and interacting with just the boundaries of the partitions. It will give a time complexity of O(√N*log√N). Thus the complexity to find the shortest path can be reduced.</p><p>After the optimal route is found, we can find the estimated time to traverse the road segment by taking traffic into account, which is a function of time, weather, and real-life events. This traffic information can be used to populate the edge weights on the graph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*M0hSEuQv__wuSVVY" /></figure><p><strong>How are On-Trip and Post-Trip Components designed?</strong></p><p>These components are responsible for handling the processes which involve from start to completion of the trip. It keeps track of the locations of the ride when the trip is in progress. It is also responsible for generating the route map and computing the trip cost when it gets completed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*v1FqqHATixbSJyRn" /></figure><p>GPS locations sent by the rider/driver app while the trip is in progress are stored in a database. Some of the characteristics of this database are-</p><ul><li>It must support a high volume of writes.</li><li>The storage must be durable.</li><li>It must support time-series-based queries such as the location of the driver in a given time range.</li></ul><p><strong>Map Matcher</strong></p><p>Map Matcher module takes in raw GPS signals as input and outputs the position on the road network. The input GPS signals consist of latitude, longitude, speed, and course. The positions on a road network consist of latitude (on an actual road), longitude (on a real road), road segment id, road name, and direction/heading. The route map generated by Map Matcher is used for-</p><p>i) finding the actual position of the driver on the road network</p><p>ii) calculating the fare prices.</p><h3>Optimization</h3><ul><li>One of the problems with the proposed design is that both the heavy reads and writes are directed to the data store. We can optimize the system by separating the read traffic from the writes by caching the raw driver locations in a distributed cache (<a href="https://redis.io/">Redis</a>). Redis is an open-source, in-memory data structure store, used as a database, cache, and message broker. The raw driver locations from the Redis cache are leveraged by Map Matcher to create the matched map and persist it in the data store.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/0*avJqZJDI2u7MnrCb" /></figure><ul><li>We can also improve the predicted ETAs by applying machine learning on top of the ETA calculation mechanism mentioned above. The first challenge of using machine learning involves <a href="https://en.wikipedia.org/wiki/Feature_engineering">feature engineering</a> to come up with features such as region, time, trip type, and driver behaviour to predict realistic ETAs.</li><li>Another exciting challenge would be to select the machine learning model, which can be used for predicting the ETAs. For this use-case, we would rely on using a non-linear(e.g., ETA doesn’t linearly relate with an hour of the day) and non-parametric (there is no prior information on the interaction between different features and ETA) model such as Random Forest, Neural networks and KNN learning. We can further improve the ETA prediction by monitoring customer issues when the ETA prediction is off.</li></ul><h3>Extended Requirement</h3><ul><li>In the design above, the drivers were being notified of the nearby riders, and they were accepting the rides they wanted. As an extension of the existing implementation, we may extend the system to enable automated matching of drivers to riders.</li><li>The drivers are matched to riders based on factors such as the distance between drivers and riders, the direction in which the driver is moving, traffic, and so forth. We can apply the naïve approach of matching riders to the nearest driver.</li><li>However, this approach doesn’t tackle the problem of something called <em>trip upgrade</em>, which occurs when a rider is matched to a driver and, at the same time, an additional driver, and an additional rider enters the system. The <em>trip upgrade</em> scenario is illustrated in the images below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0ssYDEwDvmb5Rgdu" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GMDXIOOU4TTWDgNQ" /></figure><p><strong>How To Handle Total Data center Failure?</strong></p><ul><li>It doesn’t happen very often, but there could be an unexpected cascading failure or an upstream network provider could fail.</li><li>Uber maintains a backup data center and the switches are in place to route everything over to the backup data center.</li><li>The problem is the data for in-process trips may not be in the backup data center. Rather than replicate data they use driver phones as a source of trip data.</li><li>What happens is the Dispatch system periodically sends an encrypted State Digest down to driver phones. Now let’s say there’s a data center failover. The next time the driver phone sends a location update to the Dispatch system will detect that it doesn’t know about this trip and ask them for the State Digest. The Dispatch system then updates itself from the State Digest and the trip keeps on going like nothing happened.</li></ul><p><strong>Conclusion</strong></p><p>Uber is a company with a complicated history. Still, its founders had made something that was impossible. They survived sabotages, strikes, and discontent with the governments of different countries around the whole world. The necessity to provide people with good services at affordable prices is urgent nowadays. Not all people can use the quality they want. Uber opens new perspectives and possibilities.</p><p>As a final word, We would like to suggest some points that might help to improve are as follows:</p><ol><li>Reduce Driver Downtime</li><li>Own Local Event Recommendations</li><li>Customize the experience</li><li>Create Loyalty Programs</li></ol><p><em>Finally, we had a word on Uber system design. If our work helped you, do clap for us. If you suggest adding something more, Feel free to reach us. Just drop me a letter. See you next time with another blog!!!</em></p><p><strong>Reference:</strong></p><ol><li><a href="https://www.techtakshila.com/system-design-interview/chapter-3/">https://www.techtakshila.com/system-design-interview/chapter-3/</a></li><li><a href="https://www.geeksforgeeks.org/system-design-of-uber-app-uber-system-architecture/">https://www.geeksforgeeks.org/system-design-of-uber-app-uber-system-architecture/</a></li><li><a href="https://www.educative.io/blog/uber-backend-system-design">https://www.educative.io/blog/uber-backend-system-design</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f3fa7380542" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generative Adversarial Networks (GANs): Connects Imagination To Reality.]]></title>
            <link>https://medium.com/almabetter/generative-adversarial-networks-gans-connects-imagination-to-reality-ba7493141f6e?source=rss-72aad6e9c52b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba7493141f6e</guid>
            <category><![CDATA[generative-adversarial]]></category>
            <category><![CDATA[almabetter]]></category>
            <category><![CDATA[convolutional-network]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Soumya Ranjan Mishra]]></dc:creator>
            <pubDate>Sat, 26 Jun 2021 02:15:30 GMT</pubDate>
            <atom:updated>2021-06-26T06:41:34.911Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*DJGVvDYO6-u-WYhk.gif" /><figcaption>Living Mona Lisa from Few-Shot Adversarial Learning of Realistic Neural Talking Head Models</figcaption></figure><p>This article aims to make you understand Generative Adversarial Networks (GANs), their types, working principle, applications, Math, and Statistics behind them. The most common definition for GAN may be it connects the imaginary world to reality. I hope you will enjoy the blog. Let’s Start!</p><h3><strong>INTRODUCTION</strong></h3><p><em>Generative Adversarial Networks (GANs) are nothing but a framework for estimating generative models via an adversarial process.</em></p><p>As I told you before that the most common definition for GAN may be it connects the imaginary world to reality. Before going further let’s know when and how GAN was introduced as a reinforcement Deep Learning technique? A generative adversarial network (GAN) is a class of machine learning frameworks designed by <a href="https://en.wikipedia.org/wiki/Ian_Goodfellow">Ian Goodfellow</a> and his colleagues in 2014. Two neural networks contesting with each other in a game (in the form of a zero-sum game, where one agent’s gain is another agent’s loss).</p><p>This technique basically learns the pattern and statistics behind the training set. Using them the technique generates new data from given random noise with the same statistics as the training set. For example, a GAN trained on photographs can generate new photographs that aren’t completely real but look the same as the given images. These are capable of <em>generating wisdom from garbage data</em>. They can be used for generating images, videos, alter images, and much more! The recent research in GANs is focused on generating <a href="https://www.youtube.com/watch?v=l82PxsKHxYc">Deep Fakes</a> (of humans) and detecting them.</p><h3><strong>STATISTICS BEHIND GAN</strong></h3><p><strong>Probability</strong></p><p>Probability is a measure of how likely an event is to occur. For example- Today there is a 60% chance of rain. There is a 50% chance of getting head in one toss. The odds of winning the lottery are a million to one. If an event is certain to happen, the probability of the event is 1. If an event doesn’t happen, the probability will be 0 and the probability never be negative.</p><p><strong>Dependent and Independent Events</strong></p><p>Two events are independent if the result of the second event is not affected by the result of the first event. Two events are dependent if the result of the first event affects the outcome of the second event so that the probability is changed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*cKYm34YHJ2Cxot3eNvt_dA.png" /></figure><p><strong>Joint Probability</strong></p><p>A joint probability is a statistical measure the calculates the likelihood of two events occ during together and at the same point in time. A joint probability is the probability of event Y occurring at the same time event X occurs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/443/1*kX1opO2awDKDb0klQ6nqJQ.png" /></figure><p><strong>Conditional Probability</strong></p><p>Conditional probability is defined as the likelihood of an event or outcome occurring, based on the occurrence of a previous event or outcome. Conditional probability is calculated by multiplying the probability of the preceding event by the updated probability of the succeeding, or conditional, event.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/445/1*JfmP1_dlJfPFuQLvEyp-Nw.png" /></figure><p><strong>Bayes Theorem</strong></p><p>Baye’s theorem, named after 18th-century British mathematician Thomas Bayes, is a mathematical formula for determining conditional probability. The theorem provides a way to revise existing predictions or theories given new or additional evidence.</p><p>Let A and B be two events with P(B) not equal to zero. The conditional probability of A given B is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/266/1*nU_GNYbvuHmr_IOHdWh49w.png" /></figure><h3><strong>Generative v/s Discriminative Modeling</strong></h3><h4>Generative Models</h4><p>Generative models are models where the focus is the distribution of individual classes in a dataset and the learning algorithms tend to model the underlying patterns/distribution of the data points. These models use the intuition of joint probability in theory, creating instances where a given feature (x)/input and the desired output/label (y) exist at the same time.</p><p>Generative models use probability estimates and likelihood to model data points and distinguish between different class labels in a dataset. These models are capable of generating new data instances. However, they also have a major drawback. The presence of outliers affects these models to a significant extent.</p><p>Examples of generative models :</p><p>* Naive Bayes Classifier</p><p>* Hidden Markov Model</p><ul><li>Linear Discriminant Analysis</li></ul><p>GANs too, are generative models. But unlike others, the novelty lies in its training evaluation measures, i.e. the adversaries.</p><h4>Discriminative Models</h4><p>Discriminative models, also called conditional models, tend to learn the boundary between classes/labels in a dataset. Unlike generative models, the goal here to find the decision boundary separating one class from another. So while a generative model will tend to model the joint probability of data points and is capable of creating new instances using probability estimates and maximum likelihood, discriminative models(just as in the literal meaning) separate classes by rather modeling the conditional probability and do not make any assumptions about the data point. They are also not capable of generating new data instances. Discriminative models have the advantage of being more robust to outliers, unlike generative models.</p><p>Examples of discriminative models :</p><p>* Logistic Regression</p><p>* Support Vector Machine</p><ul><li>Decision Trees</li></ul><p>In GANs, discriminators introduce the ‘adversarial’ nature which we are about to discuss in the next section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*ExDqA5QMDnrg2FVGZr-jmg.png" /></figure><h3><strong>GAN, Architecture, and Working</strong></h3><p>Here, we discuss the actual architecture of GANs and see how it works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*T1CJvRBQnpNFKZw0bU-wVA.png" /></figure><p>Here is the basic architecture of how GAN works and how the training goes on. We will discuss the architecture block by block. Let’s take the example of generating pictures of dogs through GAN.</p><p><strong>Latent Random variables</strong> may be termed as noise. The data is from a random distribution. The data taken here will lead to the final desired pictures. <strong>Real-world images</strong> are those images whose are the ream images and data from the real distribution. This is the actual distribution. The goal of a GAN is to generate data belonging to this distribution from the noise mentioned in the earlier sub-section. In this case, we will take various real dog images. The purpose of the <strong>discriminator</strong> is to classify the image as real or fake. So, the discriminator can accurately be classifying the real images as real or fake images as fake. Again there will occur a backpropagation to reduce the noise and those who are classified as fake will go for a further <strong>generator</strong> process to train again. In our example, the discriminator can be a basic <strong>CNN-based binary image classifier</strong>.</p><p><em>The Discriminator is nothing but a trainable evaluation metric that tells how good/bad is the quality of the generated data.</em></p><p>The<strong> Generator</strong> is the main component of the GAN. In this part, the generative network attempts to learn the pattern, trends, and statistics of the distribution and generate data belonging to it.</p><p>Now I guess you will able to understand why it’s called two nets are competing. We will also discuss why it’s called where one agent’s gain is another agent’s loss in the mathematics part.</p><h3><strong>Working</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A75a2pQgC4oXE_683O2A8Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5g6wTmog1i3lyDOfEQtdwg.png" /></figure><p>Here the training has started from taking noise from the random data to the generator and it trained on that data and now other real images are taken from real-world dog images and both images from real-world and generator will go to discriminator to get identified and classified as dogs images are real or fake.</p><p>In the first image now discriminator identified a dog and no dog as it’s shown above the probability values for being a dog. Thus, those samples will be backpropagated to the discriminator where the weights will get updated again till we get the probability value for being a dog only. Now in the second image, all the images will be calculated their probabilities according to being a dog but we need the probability value must be 1 for a dog. That’s why we backpropagated those samples that are having a probability of being a dog. In the second image, we don’t require samp-le real images because the discriminator has already aware of the trends and patterns to identify a dog.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kEpdCE4EKeidEuqqkblWQQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_gOeQL5yzAcQcHqjfzjY6w.png" /></figure><p>Here the same happens what I have explained above and we can see the random noise is getting a dog-like shape. While running the training process for lots of epochs our predicted and generated images will be getting more meaningful and the epoch will run until the loss is minimized.</p><p>Before going to the loss function I would like to discuss the difference between CNN and DCNN.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3-AZm4dU7UWZbkqu0Rg5Ow.png" /><figcaption>CNN Architecture</figcaption></figure><p>For detailed knowledge of CNN, you may refer to: <a href="https://towardsdatascience.com/a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way-3bd2b1164a53">https://towardsdatascience.com/a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way-3bd2b1164a53</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UPWZz1mVNuPp3lzmqOe9bg.png" /><figcaption>DCNN architecture</figcaption></figure><p>DCNN is also known as Deconvolutional Neural Network. Using simple words, CNN we input the image and it goes through convolution, pooling, and then flattening into a vector form for the training process. DCNN is the opposite of it and it can create an image from the flattened vector. It’s the reverse one.</p><p>For detailed knowledge of DCNN, You may refer to: <a href="https://searchenterpriseai.techtarget.com/definition/deconvolutional-networks-deconvolutional-neural-networks">https://searchenterpriseai.techtarget.com/definition/deconvolutional-networks-deconvolutional-neural-networks</a></p><h3>Training Loss</h3><p>The GANs <a href="https://arxiv.org/abs/1406.2661">paper</a> defines the training loss as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/0*pwYsn7G4fUioIQbC.png" /></figure><p>This is the Minmax loss, the discriminator tries to maximize it and the generator tries to minimize <em>log(1 — D(G(z)))</em> as it cannot touch <em>log(D(x)).</em></p><p>However, the <a href="https://arxiv.org/abs/1406.2661">paper</a> also suggests modifying this loss for the generator to maximize <em>log(D(G(z))).</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7ut69PuFs3fCIxbpjWV-Cg.png" /></figure><p>Here we get the intuition of why it’s called as to where one agent’s gain is another agent’s loss. Because at the same time discriminator tries to maximize the loss function and the generator tries to minimize the loss function.</p><h3><strong><em>Application of GAN</em></strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/1*dr-Yh1IEyiEwX7SeOs4UNQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*_-EuYRVhSB1618YyvNeqTw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ldF15CEH5l_F9OH1Hmx7Og.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JEdTyAcwtBP6ghsKroeMSg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4M08bPYJBMqPnWk8He8EWQ.png" /><figcaption>Enhanced picture resolution</figcaption></figure><p>In this method, the clearest and resolution enhanced photo of a moon can be processed using lots of the moon’s real images.</p><ul><li><a href="https://arxiv.org/abs/1909.11573">DeepFake</a> Generation and Detection is one of the latest research topics in GANs which essentially generates edited or tampered images that look realistic to the naked eye. This can be easily done manually with the help of photo/video editing software and hence, is vulnerable to misuse. DeepFake detection can be derived as a corollary to generation.</li></ul><h3><strong><em>Types of GAN:</em></strong></h3><ol><li><strong><em>Vanilla GAN</em></strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iNiNHDE91c7DoadBdolVRA.png" /></figure><p>2.<strong><em>Deep Convolutional GANs (DCGAN)</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O0yjB72a95My4_LTsqWPUw.png" /></figure><p>3.<strong><em>Conditional GANs (CGAN)</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4u4BDOmmt2Hy7R19oYvAWA.png" /></figure><p>4.<strong><em>Super Resolution GAN</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ar8sMeQn9J09ZI75lbOw-Q.png" /></figure><p>5.<strong><em>info GAN</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JKM_Xb8xBpMKZb26xgNyJg.png" /></figure><h3><strong>CONCLUSION</strong></h3><p>The main purpose of this article is to get the basic knowledge of the adversarial training evaluation of a generative model. We have discussed generative and discriminative modeling, and how they differ from each other. We discussed how GANs work and the idea behind it. We learned the training aspect of the GANs as well as the issues that arise while actually training the network.</p><p><em>Finally, we had a word on some well-known flavors of GANs.I think my work will help you in certain ways. If you suggest adding something more, Feel free to reach me. Just drop me a mail. See you next time with another blog!!!</em></p><h3>References:</h3><ol><li>Original paper: <a href="https://arxiv.org/abs/1406.2661">https://arxiv.org/abs/1406.2661</a></li><li><a href="https://machinelearningmastery.com/what-are-generative-adversarial-networks-gans/"><em>https://machinelearningmastery.com/what-are-generative-adversarial-networks-gans/</em></a></li><li><a href="https://en.wikipedia.org/wiki/Generative_adversarial_network">https://en.wikipedia.org/wiki/Generative_adversarial_network</a></li><li><a href="https://towardsdatascience.com/a-comprehensive-guide-to-generative-adversarial-networks-gans-fcfe65d1cfe4">https://towardsdatascience.com/a-comprehensive-guide-to-generative-adversarial-networks-gans-fcfe65d1cfe4</a></li><li><a href="https://machinelearningmastery.com/impressive-applications-of-generative-adversarial-networks/">https://machinelearningmastery.com/impressive-applications-of-generative-adversarial-networks/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba7493141f6e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/almabetter/generative-adversarial-networks-gans-connects-imagination-to-reality-ba7493141f6e">Generative Adversarial Networks (GANs): Connects Imagination To Reality.</a> was originally published in <a href="https://medium.com/almabetter">AlmaBetter</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DATA PREPROCESSING]]></title>
            <link>https://medium.com/almabetter/data-preprocessing-ea09fac6a7f7?source=rss-72aad6e9c52b------2</link>
            <guid isPermaLink="false">https://medium.com/p/ea09fac6a7f7</guid>
            <category><![CDATA[data-cleaning]]></category>
            <category><![CDATA[data-preprocessing]]></category>
            <category><![CDATA[image-preprocessing]]></category>
            <category><![CDATA[dimensionality-reduction]]></category>
            <category><![CDATA[text-preprocessing]]></category>
            <dc:creator><![CDATA[Soumya Ranjan Mishra]]></dc:creator>
            <pubDate>Wed, 16 Jun 2021 09:23:56 GMT</pubDate>
            <atom:updated>2021-06-24T11:05:58.522Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*e7lYKpF5FJYjNMVPlQgaKg.png" /><figcaption>Data Preprocessing for Better Result</figcaption></figure><h3>INTRODUCTION</h3><p>This article aims to make you understand the basic data preprocessing of various data types before any machine learning process. The process of cleaning data prepares the available data for further machine learning process to reach the client’s requirement.</p><h3>1) What is Data?</h3><p>Data is a collection of facts, such as numbers, words, measurements, observations, or just descriptions of things.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*Oti8QJW3K2Tzmmn94u960A.jpeg" /></figure><p>There are mainly three types of data that you will encounter while doing Data Science. Those are Numerical &amp; Categorical, Textual, and Image Data. Let’s deep dive into the Data Pre-processing factory and let’s know what’s actually happening in the factory.</p><h3>2) What is Data Preprocessing?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/267/1*RZt16UjJ_QSYymaV_OmBkA.jpeg" /><figcaption>Data Pre-processing</figcaption></figure><p>Data preprocessing is a process of preparing the raw data and making it suitable for a machine learning model. It is the first and crucial step while creating a machine learning model. It is a data mining technique that involves the transformation of raw data into an insightful and organized format. A real-world data generally contains noises, missing values, and maybe in an unusable format that cannot be directly used for machine learning models. For resolving such issues it prepares raw data for further processing. Data preprocessing is a required task for cleaning the data and making it suitable for a machine learning model which also increases the accuracy and efficiency of a machine learning model.</p><h3>3) Why do we need data preprocessing?</h3><p>Data preprocessing is an integral step in Machine Learning as the quality of data and the useful information that can be derived from it directly affects the ability of our model to learn; therefore, we must preprocess our data before feeding it into our model. Real-world data is often incomplete, inconsistent, lacking in certain behaviors or trends, and is likely to contain many errors. To better visualize and extract the hidden pattern, Data Preprocessing must be done.</p><h3>4) What are Features?</h3><p>Now the world is full of data. There are billions of data created every day and they are getting stored in databases through a proper channel as well as a proper order. Those are called datasets. A Dataset is nothing but a collection of samples or observations organized with some columns and separated from each other. Those columns that tell the properties and qualities of any object or final output are known as independent columns or features. The Dependent columns are the final output of the features. In other words, features are the traits of the dataset otherwise called Variable and Attribute. For example, RAM, storage, and model could be considered as features of a Smart Phone. However these could belong to different data types, and we might have to deal with them differently.</p><h3>5) Data Preprocessing on various types of Data</h3><p>Now we will deep dive into some data preprocessing techniques for various data types.</p><h3>5.1) Numerical &amp; Categorical Data</h3><p>Numerical data is a data type expressed in numbers, rather than natural language description. Sometimes called quantitative data, numerical data is always collected in number form. Categorical variables represent types of data that may be divided into groups. Examples of categorical variables are race, sex, age group, and educational level.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/0*iTOIEyeEpjm22GMa.png" /></figure><p>5.1.1) <strong>Feature Engineering</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*sCBBdHxu0IUMXdTsv0NBGA.jpeg" /></figure><p>Feature engineering is the process of transforming raw data into features that better represent the underlying problem to the predictive models, resulting in improved model accuracy on unseen data.</p><p>5.1.1.1) <strong>Handling Missing Values</strong></p><p>Handling the missing values is one of the greatest challenges faced by analysts because making the right decision on how to handle it generates robust data models. Let us look at different ways of imputing the missing values.</p><p><strong>5.1.1.1.1) Mean/Median/Mode Replacement</strong></p><p>Mean/Median/Mode imputation has the assumption that the data are missing completely at Random. We solve this by replacing the NaN value with the most frequent occurrence of the variables. In the case of the categorical variables, we can use mode.</p><p><strong>5.1.1.1.2) Random Sample Imputation</strong></p><p>Random Sample imputation consists of taking random observation from the dataset and we use this observation to replace the NaN variable. It assumes the data are missing completely at random.</p><p><strong>5.1.1.1.3) Capturing NaN value with a new feature</strong></p><p>It works well if the data are not missing completely at random. Let there is a feature named Age, and some of the values are missed. Thus, we can create another feature while replacing NaN with 1 and non-NaN with 0.</p><p><strong>5.1.1.1.4) End of Distribution Imputation</strong></p><p>Here, we will find out the extreme value of the distribution and impute that in the place of missing values. It’s easy to implement and captures the importance of missingness if there is one.</p><p><strong>5.1.1.1.5) Arbitrary Imputation</strong></p><p>This technique was derived from the Kaggle competition as it consists of replacing NaN with an arbitrary value.</p><p><strong>5.1.1.1.6) Frequent Categories Imputation</strong></p><p>It is used where a very less number of missing values are present. It’s otherwise known as the mode imputation while the total count of the mode value will be imputed. It’s easy and fast to implement.</p><p><strong>5.1.1.1.7) Creating a Sub-model to predict the missing value.</strong></p><p>Here, we will create a sub-model while taking the non-missing value as a separate dataset(we will split and create a model with this dataset) and the missing vale as another dataset whose values are to be predicted.</p><p><strong>5.1.1.1.8) Deleting Column if missing value &gt; 60%</strong></p><p>If the complete row is having NaN values then it doesn’t make any value out of it. So such rows/columns are to be dropped immediately. Or if the % of row/column is mostly missing say about more than 60% then also one can choose to drop.</p><p><strong>5.1.1.1.9) Using Algorithms Which Support Missing Values</strong></p><p>KNN is a machine learning algorithm that works on the principle of distance measure. This algorithm can be used when there are nulls present in the dataset. While the algorithm is applied, KNN considers the missing values by taking the majority of the K nearest values. In this particular dataset, taking into account the person’s age, sex, class, etc, we will assume that people having the same data for the above-mentioned features will have the same kind of fare.</p><p>Unfortunately, the SciKit Learn library for the K — Nearest Neighbour algorithm in Python does not support the presence of the missing values.</p><p>Another algorithm that can be used here is RandomForest. This model produces a robust result because it works well on non-linear and categorical data. It adapts to the data structure taking into consideration of the high variance or the bias, producing better results on large datasets.</p><p><strong>5.1.1.1.10) Checking for duplicates</strong></p><p>If the same row or column is repeated, you can also drop it by keeping the first instance. So that while running machine learning algorithms, so as not to offer that particular data object an advantage or bias.</p><p>5.1.1.2) <strong>Handling Outliers</strong></p><p>An <em>outlier</em> is an observation that lies an abnormal distance from other values in a random sample from a population. In a sense, this definition leaves it up to the analyst (or a consensus process) to decide what will be considered abnormal. Before abnormal observations can be singled out, it is necessary to characterize normal observations. A point beyond an inner fence on either side is considered a <strong>mild outlier</strong>. A point beyond an outer fence is considered an <strong>extreme outlier</strong>.</p><p>Handling Outliers is a good practice before passing the data to the ML model as we can get a better model with good metrics.</p><p><strong>5.1.1.2.1) Using Standard Deviation in Symmetric Curve</strong></p><p>In a Gaussian distribution while it’s the symmetric curve and outlier are present. Then, we can set the boundary by taking standard deviation into action.</p><p><strong>5.1.1.2.2) Using IQR in skew-symmetric Curve</strong></p><p>The box plot is a useful graphical display for describing the behavior of the data in the middle as well as at the ends of the distributions. The box plot uses the <a href="https://www.itl.nist.gov/div898/handbook/eda/section3/eda351.htm">median</a> and the lower and upper quartiles (defined as the 25th and 75th <a href="https://www.itl.nist.gov/div898/handbook/prc/section2/prc252.htm">percentiles</a>). If the lower quartile is Q1 and the upper quartile is Q3, then the difference (Q3 — Q1) is called the interquartile range or IQ. A box plot is constructed by drawing a box between the upper and lower quartiles with a solid line drawn across the box to locate the median. The following quantities (called <em>fences</em>) are needed for identifying extreme values in the tails of the distribution:</p><ol><li>lower inner fence: Q1–1.5*IQ</li><li>upper inner fence: Q3 + 1.5*IQ</li><li>lower outer fence: Q1–3*IQ</li><li>upper outer fence: Q3 + 3*IQ</li></ol><p><strong>5.1.1.2.3) Using Outlier Insensitive Algorithms.</strong></p><p>Some algorithms that are not sensitive to outliers are Naive Bayes Classifier, Support Vector Machine, Decision Tree, Ensemble Techniques, and K-Nearest Neighbours. We can use these algorithms to get rid of outliers.</p><p><strong>5.1.1.3) Categorical Encoding</strong></p><p>This means that <strong>categorical</strong> data must be <strong>encoded</strong> into numbers before we can use it to fit and evaluate a model. There are many ways to <strong>encode categorical</strong> variables for modeling as follows.</p><p><strong>5.1.1.3.1) Nominal Encoding</strong></p><p>Variable comprises a finite set of discrete values with no relationship between values. Some examples include:</p><ul><li>A “<em>pet</em>” variable with the values: “<em>dog</em>” and “<em>cat</em>“.</li><li>A “<em>color</em>” variable with the values: “<em>red</em>“, “<em>green</em>“, and “<em>blue</em>“.</li><li>A “<em>place</em>” variable with the values: “<em>first</em>“, “<em>second</em>“, and “<em>third</em>“</li></ul><p><strong>5.1.1.3.1.1) One Hot Encoding</strong></p><p>One Hot Encoding means to encode or create additional features for all unique categorical values. For example, ‘Apple’, ‘ Chicken’, and ’Brocoli’ have been encoded as 1 in that place only where it is present.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*RXzhO6vA37-d2LAxGl_TAQ.png" /></figure><p><strong>5.1.1.3.1.2) One Hot Encoding with many categorical</strong></p><p>It is the same as one-hot encoding but the difference is Let you have more than 20 unique categories. Then if you will perform one-hot encoding to all those you will have additional 20 columns. This will lead to the Curse of Dimensionality. So, It has been observed from the KDD Orange Cup challenge that some top n no. of maximum categories are taken and encoded.</p><p><strong>5.1.1.3.1.3) Mean Encoding</strong></p><p>Here, You will calculate the mean of the unique categories to the no. of occurrence in that dataset and encode them while creating a new column.</p><p><strong>5.1.1.3.2) Ordinal Encoding</strong></p><p>Variable comprises a finite set of discrete values with a ranked ordering between values. In ordinal encoding, each unique category value is assigned an integer value.</p><p>For example, “<em>red</em>” is 1, “<em>green</em>” is 2, and “<em>blue</em>” is 3.</p><p><strong>5.1.1.3.2.1) Label Encoding</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/1*v0xwVyr7MyHD6XPBgA82hw.png" /></figure><p>In this type of encoding, You have to encode the unique categories as per their rating as shown in the above image.</p><p><strong>5.1.1.3.2.2) Target guided Ordinal Encoding</strong></p><p>First, you have to calculate the mean as same in mean encoding but here you won’t encode them with their mean values. Here, you will have to perform label encoding based on those mean values. so, in other words, “target guided ordinal encoding = mean encoding +label encoding.”</p><p><strong>5.1.1.3.3) Count (or) Frequency Encoding</strong></p><p>In this type of encoding, you have to encode the unique categories by their total no. of occurrences in that dataset and particularly in that categorical column.</p><p><strong>5.1.1.3.4) Probability ratio Encoding</strong></p><p>This encoding is purely based on probability. If it’s possible to calculate the probability or it’s a probability type problem, You can use this type of encoding by calculating the probabilities. For example, If you are working with some dataset like survive and not- survive in “Titanic Dataset”. There you can use probability encoding.</p><p><strong>5.1.1.4) Data Transformation</strong></p><p>It refers to putting the values in the same range or same scale so that no variable is dominated by the other.</p><p>Most of the time, the collected data set contains features highly varying in magnitudes, units, and ranges. If scaling is not done then the algorithm only takes magnitude into account and not units hence incorrect modeling. To solve this issue, we have to do scaling to bring all the variables to the same level of magnitude.</p><p><strong>5.1.1.4.1) Standardization</strong></p><p>Standardization is another scaling technique where the values are centered around the mean with a unit standard deviation. This means that the mean of the attribute becomes zero and the resultant distribution has a unit standard deviation.</p><p><strong>5.1.1.4.2) Normalization</strong></p><p>Normalization is a scaling technique in which values are shifted and rescaled so that they end up ranging between 0 and 1. It is also known as Min-Max scaling.</p><p><strong>5.1.1.4.3) Robust Scaler</strong></p><p>It is used to scale the feature to median and quantile. Scaling using median and quantiles consists of subtracting the median from all the observations and then dividing by the interquartile difference. The inter quantile difference is the difference between the 75th and 25th.</p><p><strong>5.1.1.4.4) Gaussian Transformation</strong></p><p>Some machine learning algorithms like linear and logistic regression assume that the features are normally distributed. So, we need to transform the features by the following methods.</p><p>Logarithmic Transformation: Transforming using a logarithmic function. (ex: np.log(df[‘’]))</p><p>Inverse Transformation: Transforming using an inverse function. (ex: 1/(df[‘’]))</p><p>Square Root Transformation: Transforming using a square function. (ex: np.sqrt(df[‘’])</p><p>Exponential Transformation: Transforming using an exponential function. (ex: (df[‘’])**(1/1.2))</p><p><strong>Box-Cox Transformation</strong>: A Box-Cox transformation is a transformation of a non-normal dependent variable into a normal shape. Normality is an important assumption for many statistical techniques; if your data isn’t normal, applying a Box-Cox means that you can run a broader number of tests.</p><p>The Box-Cox transformation is named after statisticians George Box and <a href="https://www.britannica.com/biography/David-Cox-British-statistician">Sir </a>David Roxbee Cox who collaborated on a 1964 paper and developed the technique.</p><p>T(y)=(y exp(lambda)-1)/(lambda)</p><p>where y is the response variable and ‘lambda’ is the transformation parameter from -5 to 5. In the transformation, all values of ‘lambda’ are considered and the optimal value for a given variable is selected.</p><p><strong>5.1.1.5) Noisy Data</strong></p><p>Noisy data is just meaningless. This type of data has no interpretation for machines. There are several reasons behind this i.e., poor and faulty data collection, data entry errors, and so on. It can be handled in the following ways:</p><p><strong>5.1.1.5.1) Binning Method</strong></p><p>To use this method we have to sort the data first. Because this method works on sorted data to smooth it. we have to divide the whole data into two segments of equal size, and no. of methods will be performed to complete the task. here, we consider each segment as a separate one and can be replaced by its boundary values or mean to complete the task.</p><p><strong>5.1.1.5.2) Clustering</strong></p><p>This approach involves the similar feature data groups to be a cluster. If any outliers present, they might be undetected or don’t fall inside the clusters.</p><p>For example, We have lots of student data and we want to perform work on those having AI knowledge. Thus we will cluster only those students who know AI.</p><p><strong>5.1.1.5.3) Regression</strong></p><p>We have to assume a regression function before this method. Here data can be made smooth by fitting the data into that function. The regression function we have to assume might be linear one (having one independent variable) or multiple (having multiple independent variables).</p><p><strong>5.1.2) Handling Imbalanced Dataset</strong></p><p>Dealing with imbalanced datasets entails strategies such as improving classification algorithms or balancing classes in the training data (data preprocessing) before providing the data as input to the machine learning algorithm. The latter technique is preferred as it has wider application.</p><p>The main objective of balancing classes is to either increase the frequency of the minority class or decrease the majority class&#39;s frequency. This is done to obtain approximately the same number of instances for both classes. Let us look at a few resampling techniques:</p><p><strong>5.1.2.1) Cross-Validation</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nZ9E2mgvLwT0nlt_.jpg" /></figure><p><strong>Cross</strong>-<strong>validation</strong> is a resampling procedure used to evaluate machine learning models on a limited data sample. The procedure has a single parameter called k that refers to the number of groups that a given data sample is to be split into. As such, the procedure is often called k-fold <strong>cross</strong>-<strong>validation</strong>.</p><p><strong>5.1.2.2) Under Sampling</strong></p><p>Undersampling can be defined as removing some observations of the majority class. This is done until the majority and minority class is balanced out.</p><p>Undersampling can be a good choice when you have a ton of data -think millions of rows. But a drawback to undersampling is that we are removing information that may be valuable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*y7VNxXAEKMW0UYg6e1h7SA.jpeg" /></figure><p><strong>5.1.2.3) Over Sampling</strong></p><p>Oversampling can be defined as adding more copies to the minority class. Oversampling can be a good choice when you don’t have a ton of data to work with.</p><p>When undersampling, a con to consider is that it can cause overfitting and poor generalization to your test set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/1*c5Ptqc2q2as4jdJKUkYaTw.jpeg" /></figure><p><strong>5.1.2.4) Synthetic Minority over Sampling Technique (SMOTE)</strong></p><p>This technique generates synthetic data for the minority class.</p><p>SMOTE (Synthetic Minority Oversampling Technique) works by randomly picking a point from the minority class and computing the k-nearest neighbors for this point. The synthetic points are added between the chosen point and its neighbors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x_HOkRys1lBn_NjJ6KqxgQ.png" /></figure><p><strong>5.1.2.5) Tree-Based Algorithm</strong></p><p>While in every machine learning problem, it’s a good rule of thumb to try a variety of algorithms, it can be especially beneficial with imbalanced datasets.</p><p>Decision trees frequently perform well on imbalanced data. In modern machine learning, tree ensembles (Random Forests, Gradient Boosted Trees, etc.) almost always outperform singular decision trees, so we’ll jump right into those:</p><p>Tree base algorithm work by learning a hierarchy of if/else questions. This can force both classes to be addressed.</p><p><strong>5.1.3) Data Integration</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/521/0*bihjFaOEzcef5ovw.png" /></figure><p>Data Integration is nothing but a data preprocessing technique that combines data from multiple sources while providing users a unified view of these data. For example, In a school, the school authority wants to conduct an interschool competition between three schools. Now school 1 has its own student data but it will collect other school’s student data from different sources and add them up which is called data integration and the outcome data will be a unified view. The condition for data integration is they must have the same entity and similar features. There are mainly 2 major approaches for data integration:-</p><h4>Tight Coupling</h4><p>In tight coupling, data is combined from different sources into a single physical location through the process of ETL — Extraction, Transformation, and Loading.</p><h4>Loose Coupling</h4><p>In loose coupling data only remains in the actual source databases. In this approach, an interface is provided that takes a query from the user and transforms it in a way the source database can understand, and then sends the query directly to the source databases to obtain the result.</p><p><strong>5.1.4) Data Reduction</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/0*nzXdpe4xjrkJM6Ic" /></figure><p>Let us assume we have a dataset of billions of rows of data. Now we will require high computationally cost and timing cost to work on that population. So, Here Data reduction comes in handy. Data reduction is a process that reduced the volume of original data and represents it in a much smaller volume. Data reduction techniques ensure the integrity of data while reducing the data. It is done to avoid the curse of dimensionality. Curse of Dimensionality refers to a set of problems that arise when working with high-dimensional data and working with high dimensionality we will have a chance of overfitting. So, without changing the feature and deleting the attributes</p><p><strong>5.1.4.1) Dimensionality Reduction</strong></p><p>Dimensionality reduction eliminates the attributes from the data set under consideration thereby reducing the volume of original data. In the section below, we will discuss three methods of dimensionality reduction. Those are Wavelet Transform, Principal Component Analysis, and Attribute Subset Selection. It will reduce the data from high dimensionality to low dimensionality.</p><p><strong>5.1.4.2) Numerosity Reduction</strong></p><p>The numerosity reduction reduces the volume of the original data and represents it in a much smaller form. This technique includes two types parametric and non-parametric numerosity reduction. Parametric numerosity reduction incorporates ‘storing only data parameters instead of the original data. One method of parametric numerosity reduction is the ‘regression and log-linear method. Non-parametric numerosity reduction involves Histogram, Sampling Techniques, and Data cube aggregation. Sampling Techniques involve several techniques like Random Sampling, Stratified sampling, Selective sampling, and Cluster sampling.</p><h3><strong>5.2) Textual Data</strong></h3><p>Preprocessing the text data is a very important step while dealing with text data because the text at the end is to be converted into features to feed into the model. The objective of preprocessing text data is that we won’t get rid of characters, words, others that don’t give value to us. We want to get rid of punctuations, stop words, URLs, HTML codes, spelling corrections, etc. We would also like to do Stemming and Lemmatization so that in features duplication of words is not there which convey almost the same meaning.</p><p><strong>5.2.1) Tokenization</strong></p><p>Tokenization is the process of breaking down a piece of text into small units called tokens. A token may be a word, part of a word, or just characters like punctuation. In other words, <em>Tokenization is breaking a text chunk into smaller parts. Whether it is breaking Paragraphs in sentences, sentence into words, or word in characters.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QhFl0rG26UntiFmS.jpeg" /></figure><p><strong>5.2.2) Stemming</strong></p><p>Stemming is the process of producing morphological variants of a root/base word. Stemming programs are commonly referred to as stemming algorithms or stemmers. A stemming algorithm reduces the words “chocolates”, “chocolatey”, “choco” to the root word, “chocolate” and “retrieval”, “retrieved”, “retrieves” reduce to the stem “retrieve”. Stemming is an important part of the pipelining process in Natural language processing. The input to the stemmer is tokenized words.</p><p><strong>5.2.2.1) Porter Stemming</strong></p><p>It is one of the most popular stemming methods proposed in 1980. It is based on the idea that the suffixes in the English language are made up of a combination of smaller and simpler suffixes. This stemmer is known for its speed and simplicity. The main applications of Porter Stemmer include data mining and Information retrieval. However, its applications are only limited to English words. Also, the group of stems is mapped onto the same stem and the output stem is not necessarily a meaningful word. The algorithms are fairly lengthy in nature and are known to be the oldest stemmer.<br><strong>Example: </strong>EED -&gt; EE means “if the word has at least one vowel and consonant plus EED ending, change the ending to EE” as ‘agreed’ becomes ‘agree’.</p><p><strong>5.2.2.2) Snowball Stemming</strong></p><p>When compared to the Porter Stemmer, the Snowball Stemmer can map non-English words too. Since it supports other languages the Snowball Stemmers can be called a multi-lingual stemmer. The Snowball stemmers are also imported from the nltk package. This stemmer is based on a programming language called ‘Snowball’ that processes small strings and is the most widely used stemmer. The Snowball stemmer is way more aggressive than Porter Stemmer and is also referred to as Porter2 Stemmer. Because of the improvements added when compared to the Porter Stemmer, the Snowball stemmer is having greater computational speed.</p><p><strong>5.2.2.3) N-gram Stemming</strong></p><p>An n-gram is a set of n consecutive characters extracted from a word in which similar words will have a high proportion of n-grams in common. <br><strong>Example: </strong>‘INTRODUCTIONS’ for n=2 becomes : *I, IN, NT, TR, RO, OD, DU, UC, CT, TI, IO, ON, NS, S*</p><p><strong>5.2.3) Lemmatization</strong></p><p>Lemmatization technique is like stemming. The output we will get after lemmatization is called ‘lemma’, which is a root word rather than root stem, the output of stemming. After lemmatization, we will be getting a valid word that means the same thing.</p><p>In simpler forms,<strong> <em>a method that switches any kind of a word to its base root mode is called Lemmatization.</em></strong></p><p>In other words, Lemmatization is a method responsible for grouping different inflected forms of words into the root form, having the same meaning. It is similar to stemming, in turn, it gives the stripped word that has some dictionary meaning. The Morphological analysis would require the extraction of the correct lemma of each word.</p><p>For example, Lemmatization clearly identifies the base form of ‘troubled’ to ‘trouble’’ denoting some meaning whereas, Stemming will cut out the ‘ed’ part and convert it into ‘trouble which has the wrong meaning and spelling errors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2K4VxxRtewNw4iP-Kh5Z7Q.png" /></figure><p><strong>5.2.4) Removing Stop Words</strong></p><p>A stop word is a commonly used word (such as “the”, “a”, “an”, “in”) that a search engine has been programmed to ignore, both when indexing entries for searching and when retrieving them as the result of a search query. <br>We would not want these words to take up space in our database, or taking up the valuable processing time. For this, we can remove them easily, by storing a list of words that you consider to stop words. NLTK(Natural Language Toolkit) in python has a list of stopwords stored in 16 different languages. You can find them in the nltk_data directory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/0*o5JLqwvo9__O8ULT.png" /></figure><p><strong>5.2.8) Vectorizer</strong></p><p>Vectorization is the process of converting words into numbers is called Vectorization, Which is a methodology in NLP to map words or phrases from vocabulary to a corresponding vector of real numbers which is used to find word predictions, similarities, etc. The vectorization is used in a use case like Text classification, Compute Similar words, Document Clustering / Grouping, Natural language Processing (NLP), feature extraction in Text Classification.,</p><p>To vectorize a corpus with a bag-of-words (BOW) approach, we represent every document from the corpus as a vector whose length is equal to the vocabulary of the corpus.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oW2_dyNuLLQ4gH4H.png" /></figure><p>We usually use Count Vectorizer, TF-IDF vectorizer, and Hash vectorizer. Word2vec is a method to efficiently create word embeddings</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tYvq8L0kohslIM8g.png" /></figure><h3><strong>5.3) Image Data</strong></h3><p>Image pre-processing is the term for operations on images at the lowest level of abstraction. These operations do not increase image information content but they decrease it if entropy is an information measure. The aim of pre-processing is an improvement of the image data that suppresses undesired distortions or enhances some image features relevant for further processing and analysis tasks.</p><p><strong>5.3.1) Pixel brightness transformations/ Brightness corrections</strong></p><p>Brightness transformations modify pixel brightness and the transformation depends on the properties of a pixel itself. In PBT, the output pixel’s value depends only on the corresponding input pixel value. Examples of such operators include brightness and contrast adjustments as well as color correction and transformations.</p><p>Contrast enhancement is an important area in image processing for both human and computer vision. It is widely used for medical image processing and as a pre-processing step in speech recognition, texture synthesis, and many other images/video processing applications.</p><p>g(x)=αf(x)+β, alpha and beta control contrast and brightness of the image.</p><p><strong>5.3.1.1) Gamma correction</strong> — You can perform a non-linear adjustment to individual pixel values. While performing image normalization we just do linear operations on individual pixels, i.e. scalar multiplication and addition/subtraction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/221/0*ZoWgT2EKgpf2B_Kk" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/273/0*1CXJuaxe_LoNwOn_" /></figure><p><strong>5.3.1.2) Histogram equalization</strong> — You all have done some contrast enhancement while doing photoshop or editing a photo. Thus, Histogram equalization is nothing but known as a contrast enhancement technique. Because it’s the capability to perform on all types of images. It also employs non-linear and non-monotonic transfer functions to map the intensity of pixels in both input and output images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/0*qi7Jb3UrHlwGDtXY.jpg" /></figure><p><strong>5.3.1.3) Sigmoid Stretching</strong> — Before diving into it let’s know what is a sigmoid function. The sigmoid function is a continuous non-linear activation function used in logistic regression and in neural networks when we have 2 classes to predict. The function gives an “S” type curve in the graph.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/148/0*U8xX9RTnmNlEaYlV" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/277/0*tzsiKQCpeaV8VG2j" /></figure><p>where :</p><p>g (x,y) is Enhanced pixel value</p><p>c is the Contrast factor</p><p>th is the Threshold value</p><p>fs(x,y) is original image</p><p>Now, Here the hyperparameter is “c” and “th”. By adjusting them we will able to control the overall contrast enhancement by lightening and darkening.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/301/0*l5eeG_p1EKw-Xih4" /></figure><h3><strong>6) Conclusion</strong></h3><p><em>In this article, I have explained several types of data preprocessing. I think my work will help you in certain ways. If you suggest adding something more, Feel free to reach me. Just drop me a mail. See you next time with another blog!!!</em></p><h3><strong>7) References</strong></h3><ol><li><a href="https://analyticsindiamag.com/5-ways-handle-missing-values-machine-learning-datasets/">https://analyticsindiamag.com/5-ways-handle-missing-values-machine-learning-datasets/</a></li><li><a href="https://www.geeksforgeeks.org/interquartile-range-to-detect-outliers-in-data/">https://www.geeksforgeeks.org/interquartile-range-to-detect-outliers-in-data/</a></li><li><a href="https://www.statisticshowto.com/box-cox-transformation/">https://www.statisticshowto.com/box-cox-transformation/</a></li><li><a href="http://www.lastnightstudy.com/Show?id=39/Data-Integration-In-Data-Mining">http://www.lastnightstudy.com/Show?id=39/Data-Integration-In-Data-Mining</a></li><li><a href="https://towardsdatascience.com/understanding-nlp-word-embeddings-text-vectorization-1a23744f7223">https://towardsdatascience.com/understanding-nlp-word-embeddings-text-vectorization-1a23744f7223</a></li><li><a href="https://www.mygreatlearning.com/blog/introduction-to-image-pre-processing/?amp">https://www.mygreatlearning.com/blog/introduction-to-image-pre-processing/?amp</a></li><li><a href="https://binaryterms.com/data-reduction.html">https://binaryterms.com/data-reduction.html</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ea09fac6a7f7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/almabetter/data-preprocessing-ea09fac6a7f7">DATA PREPROCESSING</a> was originally published in <a href="https://medium.com/almabetter">AlmaBetter</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>