<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yogita Kinha on Medium]]></title>
        <description><![CDATA[Stories by Yogita Kinha on Medium]]></description>
        <link>https://medium.com/@yogita.kinha?source=rss-ccb50745d2fb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*5pXMTlnlZarnu-QQ.</url>
            <title>Stories by Yogita Kinha on Medium</title>
            <link>https://medium.com/@yogita.kinha?source=rss-ccb50745d2fb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 00:59:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yogita.kinha/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[An easy guide to choose the right Machine Learning algorithm for your task]]></title>
            <link>https://medium.com/dataseries/an-easy-guide-to-choose-the-right-machine-learning-algorithm-for-your-task-b0f6d77aab75?source=rss-ccb50745d2fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/b0f6d77aab75</guid>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Yogita Kinha]]></dc:creator>
            <pubDate>Sat, 13 Jul 2019 00:00:00 GMT</pubDate>
            <atom:updated>2019-09-24T17:49:09.148Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/492/0*-tWL2XIgL3uWzHro.jpg" /></figure><p>Well, there is no straightforward and sure-shot answer to this question. The answer depends on many factors like the problem statement and the kind of output you want, type and size of the data, the available computational time, number of features and observations in the data, to name a few.</p><p><strong>Here are some important considerations while choosing an algorithm</strong></p><ol><li><strong>Size of the training data</strong></li></ol><p>It is usually recommended to gather a good amount of data to get reliable predictions. However, many a time the availability of data is a constraint. So, if the training data is smaller or if the dataset has a fewer number of observations and a higher number of features like genetics or textual data, choose algorithms with high bias/low variance like Linear regression, Naïve Bayes, Linear SVM.</p><p>If the training data is sufficiently large and the number of observations is higher as compared to the number of features, one can go for low bias/high variance algorithms like KNN, Decision trees, kernel SVM.</p><p>2. <strong>Accuracy and/or Interpretability of the output</strong></p><p>Accuracy of a model means that the function predicts a response value for a given observation which is close to the true response value for that observation. Highly interpretable algorithm (restrictive models like Linear Regression) means that one can easily understand how any individual predictor is associated with the response while the flexible models give higher accuracy at the cost of low interpretability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/0*t7lcMGhuCpoE0e6O.png" /></figure><p><em>A representation of trade-off between Accuracy and Interpretability, using different statistical learning methods. (</em><a href="https://cdn.oreillystatic.com/en/assets/1/event/105/Overcoming%20the%20Barriers%20to%20Production-Ready%20Machine-Learning%20Workflows%20Presentation%201.pdf"><em>source</em></a><em>)</em></p><p>Some algorithms are called Restrictive because they produce a small range of shapes of the mapping function. For example, linear regression is a restrictive approach because it can only generate linear functions such as the lines.</p><p>Some algorithms are called flexible because they can generate a wider range of possible shapes of the mapping function. For example, KNN with k=1 is highly flexible as it will consider every input data point to generate the mapping output function. The below picture displays the tradeoff between flexible and restrictive algorithms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/0*WGQLiMO2gdN2MkC0.png" /></figure><p><em>A representation of trade-off between Flexibility and Interpretability, using different statistical learning methods. (</em><a href="https://www-bcf.usc.edu/~gareth/ISL/ISLR%20Sixth%20Printing.pdf"><em>source</em></a><em>)</em></p><p>Now, to use which algorithm depends on the objective of the business problem. If inference is the goal, then restrictive models are better as they are much more interpretable. Flexible models are better if higher accuracy is the goal. In general, as the flexibility of a method increases, its interpretability decreases.</p><p>3. <strong>Speed or Training time</strong></p><p>Higher accuracy typically means higher training time. Also, algorithms require more time to train on large training data. In real-world applications, the choice of algorithm is driven by these two factors predominantly.</p><p>Algorithms like Naïve Bayes, Linear and Logistic regression are easy to implement and quick to run. Algorithms like SVM which involve tuning of parameters, Neural networks with high convergence time, random forests need a lot of time to train the data.</p><p>4. <strong>Linearity</strong></p><p>Many algorithms work on the assumption that classes can be separated by a straight line (or its higher-dimensional analogue). Examples include logistic regression and support vector machines. Linear regression algorithms assume that data trends follow a straight line. If the data is linear then these algorithms perform quite good.</p><p>However, not always the data is linear, so we require other algorithms which can handle high dimensional and complex data structures. Examples include kernel SVM, random forest, neural nets.</p><p>Best way to find out the linearity is to either fit a linear line or run a logistic regression or SVM and check for residual errors. Higher error means the data is not linear and would need complex algorithms to fit.</p><p><strong>5. Number of features</strong></p><p>The dataset may have a large number of features which may not all be relevant and significant. For certain type of data such as genetics or textual, the number of features can be very large compared to the number of data points.</p><p>A large number of features can bog down some learning algorithms, making training time unfeasibly long. SVM is better suited in case of data with large feature space and lesser observations. PCA and feature selection techniques should be used to reduce dimensionality and select important features.</p><p>Here is a handy<strong> cheat sheet</strong> that details the algorithms you can use for different types of machine learning problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/0*XitqTqJMytRRsCFL.png" /><figcaption><a href="https://blogs.sas.com/content/subconsciousmusings/2017/04/12/machine-learning-algorithm-use/"><em>source</em></a></figcaption></figure><p>Machine learning algorithms can be divided into supervised, unsupervised and reinforcement learning, as discussed in my previous <a href="https://www.edvancer.in/understanding-artificial-intelligence-machine-learning-and-data-science/">blog</a>. This article walks you through the process of how to use the sheet.</p><p>The cheat sheet is majorly divided into two types of learning:</p><p><strong>Supervised learning algorithms</strong> are employed where the training data has output variables corresponding to the input variables. The algorithm analyses the input data and learns a function to map the relationship between the input and output variables.</p><p>Supervised learning can further be classified into: Regression, Classification, Forecasting and Anomaly Detection.</p><p><strong>Unsupervised Learning</strong> algorithms are used when the training data does not have a response variable. Such algorithms try to find the intrinsic pattern and hidden structures in the data. Clustering and Dimension Reduction algorithms are types of unsupervised learning algorithms.</p><p>The below infographic simply explains Regression, classification, anomaly detection and clustering along with examples where each of these could be applied.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/0*QaFNVjTO_zxM8qIc.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/0*vaT6fMofyQUaBJvW.png" /><figcaption><a href="https://docs.microsoft.com/en-us/azure/machine-learning/studio/algorithm-choice"><em>source</em></a></figcaption></figure><p>The main points to consider when trying to solve a new problem are:</p><ul><li>Define the problem. What is the objective of the problem?</li><li>Explore the data and familiarise yourself with the data.</li><li>Start with basic models to build a baseline model and then try more complicated methods.</li></ul><p>Having said that, always remember that “ <em>better data often beats better algorithms</em>” as discussed in my previous <a href="https://www.edvancer.in/data-cleaning">blog</a>. Equally important is designing good features. Try a bunch of algorithms and compare their performances to choose the best one for your specific task. Also, try ensemble methods as they generally provide much better accuracy.</p><p><em>Originally published at </em><a href="https://www.edvancer.in/an-easy-guide-to-help-you-choose-which-machine-learning-algorithm-to-choose-for-your-business-problem/"><em>https://www.edvancer.in</em></a><em> on July 13, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b0f6d77aab75" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataseries/an-easy-guide-to-choose-the-right-machine-learning-algorithm-for-your-task-b0f6d77aab75">An easy guide to choose the right Machine Learning algorithm for your task</a> was originally published in <a href="https://medium.com/dataseries">DataSeries</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to deal with Missing values in your dataset]]></title>
            <link>https://medium.com/limitedio/how-to-deal-with-missing-values-in-your-dataset-988696b1f450?source=rss-ccb50745d2fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/988696b1f450</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[handling-missing-values]]></category>
            <category><![CDATA[imputation]]></category>
            <dc:creator><![CDATA[Yogita Kinha]]></dc:creator>
            <pubDate>Tue, 02 Jul 2019 00:00:00 GMT</pubDate>
            <atom:updated>2019-07-13T12:02:12.101Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/397/0*qjYMN0Y641bwwPXl.png" /></figure><p>In the last <a href="https://www.edvancer.in/data-cleaning">blog</a>, we discussed the importance of the data cleaning process in a data science project and ways of cleaning the data to convert a raw dataset into a useable form. Here, we are going to talk about how to identify and treat the missing values in the data step by step.</p><p>Real-world data would certainly have missing values. This could be due to many reasons such as data entry errors or data collection problems. Irrespective of the reasons, it is important to handle missing data because any statistical results based on a dataset with non-random missing values could be biased. Also, many ML algorithms do not support data with missing values.</p><p><strong>How to identify missing values?</strong></p><p>We can check for null values in a dataset using pandas function as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/0*G5NfVkXFISLaXu9u.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/0*0kJOpnZwIkexS2QL.png" /></figure><p>But, sometimes, it might not be this simple to identify missing values. One needs to use the domain knowledge and look at the data description to understand the variables. For instance, in the dataset below, isnull() does not show any null values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/0*a1DTLwaEwyd6eZ_V.png" /></figure><p>In this example, there are columns that have a minimum value of zero. On some columns, a value of zero does not make sense and indicates an invalid or missing value.</p><p>On analysing the features carefully, the following columns have an invalid zero minimum value:</p><p>1: Plasma glucose concentration</p><p>2: Diastolic blood pressure</p><p>3: Triceps skinfold thickness</p><p>4: 2-Hour serum insulin</p><p>5: Body mass index</p><p>It becomes clear on checking the number of zeros in these columns that columns 1,2 and 5 have few zero values, whereas columns 3 and 4 have a lot more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/0*pTqGYKRJKu9Zbj_F.png" /></figure><p>Missing values in each of these columns may need different strategies. We could mark these zero values as NaN to highlight missing values so that they could be processed further.</p><p><strong>Quick classification of missing data</strong></p><p>There are three types of missing data:</p><p><strong>MCAR:</strong> Missing Completely At Random. It is the highest level of randomness. This means that the missing values in any features are not dependent on any other features values. This is the desirable scenario in case of missing data.</p><p><strong>MAR</strong>: Missing At Random. This means that the missing values in any feature are dependent on the values of other features.</p><p><strong>MNAR</strong>: Missing Not At Random. Missing not at random data is a more serious issue and in this case, it might be wise to check the data gathering process further and try to understand why the information is missing. For instance, if most of the people in a survey did not answer a certain question, why did they do that? Was the question unclear?</p><p><strong>What to do with the missing values?</strong></p><p>Now that we have identified the missing values in our data, next we should check the extent of the missing values to decide the further course of action.</p><p><strong>Ignore the missing values</strong></p><ul><li>Missing data under 10% for an individual case or observation can generally be ignored, except when the missing data is a MAR or MNAR.</li><li>The number of complete cases i.e. observation with no missing data must be sufficient for the selected analysis technique if the incomplete cases are not considered.</li></ul><p><strong>Drop the missing values</strong></p><p><strong>Dropping a variable</strong></p><ul><li>If the data is MCAR or MAR and the number of missing values in a feature is very high, then that feature should be left out of the analysis. If missing data for a certain feature or sample is more than 5% then you probably should leave that feature or sample out.</li><li>If the cases or observations have missing values for target variables(s), it is advisable to delete the dependent variable(s) to avoid any artificial increase in relationships with independent variables.</li></ul><p><strong>Case Deletion</strong></p><p>In this method, cases which have missing values for one or more features are deleted. If the cases having missing values are small in number, it is better to drop them. Though this is an easy approach, it might lead to a significant decrease in the sample size. Also, the data may not always be missing completely at random. This may lead to biased estimation of parameters.</p><p><strong>Imputation</strong></p><p>Imputation is the process of substituting the missing data by some statistical methods. Imputation is useful in the sense that it preserves all cases by replacing missing data with an estimated value based on other available information. But imputation methods should be used carefully as most of them introduce a large amount of bias and reduce variance in the dataset.</p><p><strong>Imputation by Mean/Mode/Median</strong></p><p>If the missing values in a column or feature are numerical, the values can be imputed by the mean of the complete cases of the variable. Mean can be replaced by median if the feature is suspected to have outliers. For a categorical feature, the missing values could be replaced by the mode of the column. The major drawback of this method is that it reduces the variance of the imputed variables. This method also reduces the correlation between the imputed variables and other variables because the imputed values are just estimates and will not be related to other values inherently.</p><p><strong>Regression Methods</strong></p><p>The variables with missing values are treated as dependent variables and variables with complete cases are taken as predictors or independent variables. The independent variables are used to fit a linear equation for the observed values of the dependent variable. This equation is then used to predict values for the missing data points.</p><p>The disadvantage of this method is that the identified independent variables would have a high correlation with the dependent variable by virtue of selection. This would result in fitting the missing values a little too well and reducing the uncertainty about that value. Also, this assumes that relationship is linear which might not be the case in reality.</p><p><strong>K-Nearest Neighbour Imputation (KNN)</strong></p><p>This method uses k-nearest neighbour algorithms to estimate and replace missing data. The k-neighbours are chosen using some distance measure and their average is used as an imputation estimate. This could be used for estimating both qualitative attributes (the most frequent value among the k nearest neighbours) and quantitative attributes (the mean of the k nearest neighbours).</p><p>One should try different values of k with different distance metrics to find the best match. The distance metric could be chosen based on the properties of the data. For example, Euclidean is a good distance measure to use if the input variables are similar in type (e.g. all measured widths and heights). Manhattan distance is a good measure to use if the input variables are not similar in type (such as age, gender, height, etc.).</p><p>The advantage of using KNN is that it is simple to implement. But it suffers from the curse of dimensionality. It works well for a small number of variables but becomes computationally inefficient when the number of variables is large.</p><p><strong>Multiple Imputation</strong></p><p>Multiple imputations is an iterative method in which multiple values are estimated for the missing data points using the distribution of the observed data. The advantage of this method is that it reflects the uncertainty around the true value and returns unbiased estimates.</p><p>MI involves the following three basic steps:</p><ol><li>Imputation: The missing data are filled in with estimated values and a complete data set is created. This process of imputation is repeated m times and m datasets are created.</li><li>Analysis: Each of the <em>m</em> complete data sets is then analysed using a statistical method of interest (e.g. linear regression).</li><li>Pooling: The parameter estimates (e.g. coefficients and standard errors) obtained from each analysed data set are then averaged to get a single point estimate.</li></ol><p>Python’s Scikit-learn has methods — impute.SimpleImputer for univariate (single variable) imputations and impute.IterativeImputer for multivariate imputations.</p><p>The MICE package in R supports the multiple imputation functionality. Python does not directly support multiple imputations but IterativeImputer can be used for multiple imputations by applying it repeatedly to the same dataset with different random seeds when sample_posterior=True.</p><p>In summation, handling the missing data is crucial for a data science project. However, the data distribution should not be changed while handling missing data. Any missing data treatment method should satisfy the following rules:</p><ol><li>Estimation without bias — Any missing data treatment method should not change the data distribution.</li><li>The relationship among the attributes should be retained.</li></ol><p>Hope you enjoyed reading the blog!!</p><p>Please share your feedback and topics that you would like to know about.</p><p>References:</p><p><a href="https://machinelearningmastery.com/handle-missing-data-python/">https://machinelearningmastery.com/handle-missing-data-python/</a> <a href="https://en.wikipedia.org/wiki/Imputation_(statistics)">https://en.wikipedia.org/wiki/Imputation_(statistics)</a> <a href="https://stats.idre.ucla.edu/stata/seminars/mi_in_stata_pt1_new/">https://stats.idre.ucla.edu/stata/seminars/mi_in_stata_pt1_new/</a></p><p><a href="https://pdfs.semanticscholar.org/e4f8/1aa5b67132ccf875cfb61946892024996413.pdf">https://pdfs.semanticscholar.org/e4f8/1aa5b67132ccf875cfb61946892024996413.pdf</a></p><p><em>Originally published at </em><a href="https://www.edvancer.in/data-cleaning-missing-values-treatment"><em>https://www.edvancer.in</em></a><em> on July 2, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=988696b1f450" width="1" height="1" alt=""><hr><p><a href="https://medium.com/limitedio/how-to-deal-with-missing-values-in-your-dataset-988696b1f450">How to deal with Missing values in your dataset</a> was originally published in <a href="https://medium.com/limitedio">LiMiTeD</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scope and Impact of AI in Agriculture]]></title>
            <link>https://medium.com/limitedio/scope-and-impact-of-ai-in-agriculture-a27a30f7aa2a?source=rss-ccb50745d2fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/a27a30f7aa2a</guid>
            <category><![CDATA[ai-in-agriculture]]></category>
            <category><![CDATA[prédiction]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[agriculture]]></category>
            <dc:creator><![CDATA[Yogita Kinha]]></dc:creator>
            <pubDate>Sun, 30 Jun 2019 00:00:00 GMT</pubDate>
            <atom:updated>2019-07-13T12:02:44.508Z</atom:updated>
            <content:encoded><![CDATA[<p>The Green Revolution during the 1950s and 1960s remarkably drove up the global food production around the world, saving a billion people from starvation. The revolution led to the adoption of new technologies like high-yielding varieties (HYVs) of cereals, chemical fertilizers and agro-chemicals, better irrigation and mechanization of cultivation methods.</p><p>India followed suite and adopted the use of hybrid seeds, machine, fertilisers and pesticides. While these practices solved the food shortage problem, they created some problems too in terms of excessive use of fertilisers and pesticides, depletion of ground-water, soil degradation etc. These problems were exacerbated by lack of training to use modern technology and awareness about the correct usage of chemicals etc.</p><p>According to the UN Food and Agriculture Organization, the global population will increase by 2 billion by 2050. With limited arable land available and exponentially increasing mouths to feed, we’re now in need of a second Green Revolution. A Green Revolution that is smarter, agile &amp; environmentally conscious — a Green Revolution driven by big data, Internet of Things (IoT), artificial intelligence (AI), and machine learning.</p><p>Some of the challenges faced by farmers from seed sowing to harvesting of crops are as follows:</p><ol><li>Crop diseases infestations</li><li>Lack of storage management</li><li>Pesticide control</li><li>Weed management</li><li>Lack of irrigation and drainage facilities.</li></ol><p>The major advantage of focusing on AI-based methods is that they tackle each problem separately and rather than generalising, provide customised solutions to a specific problem.</p><p><strong>Potential AI application and development areas</strong></p><ul><li><strong>Predictive and Recommendation Analytics — </strong>AI and Machine learning can help farmers by recommending the sowing dates for different crops based on weather conditions. ML models can also suggest tweaks in cropping patterns to boost yields. Using the historic production data, weather forecasts, seed information, and demand and supply information, ML can be used to forecast the amount of seed that should be grown to fulfil the growing needs.</li><li><strong>Supply Chain management-</strong> AI can be used to predict the demand by analysing inputs like rate of population growth, historical demand patterns of food commodities, region wise staple food patterns, among other inputs. This information can be used to efficiently modify the cropping patterns and also minimise wastage of crop yields and increase profits to farmers.</li><li><strong>Identifying Plant Diseases </strong>— Crop images are analysed using computer vision technology and segmented into areas like background, healthy part and diseased part. The diseased part is then captured and sent to remote labs for further diagnosis.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/0*nW1mCNNrOwb0GMc7.png" /></figure><ul><li><strong>Detecting pest infestations </strong>— Similarly, the leaf images pre-processing help early detection of pests infestation and allow farmers to act quickly and minimise losses.</li><li><strong>Soil Health Monitoring </strong>— ML and Deep learning applications are used to identify potential defects and nutrient deficiencies in the soil. The algorithms analyse the soil samples and correlate particular foliage patterns with certain soil defects, plant pests and diseases.</li></ul><p><strong>There is considerable progress in this direction globally</strong></p><p>Farmers are deploying robots, ground-based wireless sensors, and drones to assess growing conditions. Many researchers and pilot projects have been conducted to test the implications of the involvement of AI applications in improving agriculture.</p><ul><li><a href="https://www.sciencedirect.com/science/article/pii/S2589721719300182#bb0320">Song and He (2005) </a>carried out a research to develop intelligent and carry-home diagnosis expert system (ES) based on Artificial neural networks to detect crop nutrition disorders in time. The field validation indicated that the prediction errors were less than 8%.!!</li><li><a href="https://www.sciencedirect.com/science/article/pii/S2589721719300182#bb0285">Shahzadi et al. (2016) </a>conducted research to develop an expert system to discriminate weed from crops with the help of image analysis and neural networks. Out of the 100 farmers who participated in the trial of this system, 65 were satisfied.</li><li><a href="https://www.sciencedirect.com/science/article/pii/S2589721719300182#bb0015">Arif et al. (2012) </a>developed two ANN models to estimate soil moisture in Paddy fields with limited meteorological data and estimated soil moisture with R2 values of 0.80 and 0.73 for training and validation processes, respectively, Thus, the ANN model reliably estimates soil moisture with limited meteorological data.</li><li>Researchers <a href="https://www.sciencedirect.com/science/article/pii/S2589721719300182#bb0225">Patil and Thorat (2016) </a>developed a ML-based system which predicted grape disease beforehand. The system also suggests the pesticides to use to avoid the spread of the disease.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/387/0*DQTkwKHdrluOCdHf.png" /></figure><p><em>Grape disease detection system using ML algorithms</em></p><p>Many big organisations and start-ups are working on to develop applications and IOT enables devices to deploy AI applications to help farmers on a large scale.</p><ul><li>Microsoft in collaboration with the International Crop Research Institute for the Semi-Arid Tropics (ICRISAT), has developed an AI-Sowing App. The app sends sowing advisories to farmers on the optimal date to sow. The farmers don’t need to install any sensors in their fields or incur any capital expenditure. All they need is a feature phone capable of receiving text messages. The pilot project uses an AI sowing app to recommend sowing date, land preparation, soil test-based fertilization, farmyard manure application, seed treatment, optimum sowing depth and more to farmers which has resulted in 30% increase in average crop yield per hectare</li><li>Microsoft in yet another initiative has collaborated with United Phosphorus Limited to build a Pest Risk Prediction API that leverages AI and machine learning to indicate in advance, the risk of pest attack. Based on the weather condition and crop growth stage, pest attacks are predicted as High, Medium or Low.</li><li>A tech-based company named NatureFresh, based out of USA, is developing technology to predict the amount of time a crop will take t produce a yield. A robotic lens will take close pictures of the plants. The images will flow into an artificial intelligence algorithm that would predict precisely how long it will take for the crop to ripen and be ready to be picked, packed and delivered to the grocery stores. This technology would be able to determine the amount of yield that will be available to sell in the future.</li></ul><p>From detecting pests to predicting what crops will deliver the best returns, artificial intelligence can help humanity confront one of its biggest challenges: feeding an additional 2 billion people by 2052, even as climate change disrupts growing seasons, turns arable land into deserts, and floods once-fertile deltas with seawater.</p><p>Farmers can use AI to determine the optimal date to sow crops, precisely allocate resources such as water and fertilizer, identify crop diseases for swifter treatment, and detect and destroy weeds. Machine learning makes these activities smarter over time. It can also help farmers forecast the year ahead by using historical production data, long-term weather forecasts, genetically modified seed information, and commodity pricing predictions, among other inputs, to recommend how much seed to sow.</p><p>References:</p><p><a href="https://www.intel.in/content/www/in/en/big-data/article/agriculture-harvests-big-data.html">The Future of AI in Agriculture - Intel</a></p><p><a href="https://www.mindtree.com/sites/default/files/2018-04/Artificial%20Intelligence%20in%20Agriculture.pdf">https://www.mindtree.com/sites/default/files/2018-04/Artificial%20Intelligence%20in%20Agriculture.pdf</a></p><p><a href="https://cloudblogs.microsoft.com/2018/11/29/feeding-the-world-with-ai-driven-agriculture-innovation/">https://cloudblogs.microsoft.com/2018/11/29/feeding-the-world-with-ai-driven-agriculture-innovation/ </a><a href="https://www.sciencedirect.com/science/article/pii/S2589721719300182">https://www.sciencedirect.com/science/article/pii/S2589721719300182</a></p><p><em>Originally published at </em><a href="https://www.edvancer.in/scope-and-impact-of-ai-in-agriculture/"><em>https://www.edvancer.in</em></a><em> on June 30, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a27a30f7aa2a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/limitedio/scope-and-impact-of-ai-in-agriculture-a27a30f7aa2a">Scope and Impact of AI in Agriculture</a> was originally published in <a href="https://medium.com/limitedio">LiMiTeD</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Cleaning — the secret ingredient to the success of any Data Science Project]]></title>
            <link>https://medium.com/limitedio/data-cleaning-the-secret-ingredient-to-the-success-of-any-data-science-project-e42d5e0df05a?source=rss-ccb50745d2fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/e42d5e0df05a</guid>
            <category><![CDATA[data-pre-processing]]></category>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[outlier-detection]]></category>
            <category><![CDATA[data-cleaning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Yogita Kinha]]></dc:creator>
            <pubDate>Wed, 26 Jun 2019 00:00:00 GMT</pubDate>
            <atom:updated>2019-07-02T09:50:27.845Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Cleaning — the secret ingredient to the success of any Data Science Project</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/0*o19WF7HWux6cwTOa.png" /></figure><p>In the last few blogs, we saw how to summarise and analyse the data using statistical methods and visualisations. But the raw data need to be processed to convert it into a usable shape. Data Preparation is the most important and foremost part of Data Science. It involves data pre-processing and data wrangling.</p><p>Raw data is collected from various sources and usually unsuitable for analysis. For example, there might be many entries for a shopper leading to duplicity, or there might a typo in noting down the email id of a customer, or some of the questions might have been left blank by a surveyor. With an uncleaned dataset, no matter what type of algorithm you try, you will never get accurate results. That is why data scientists spend a considerable amount of time on data cleaning.</p><p><strong>Better data beats fancier algorithms.</strong></p><p>Though, the steps and techniques for data cleaning will vary for each dataset. But the following steps could be used as a standard approach for any dataset type:</p><ol><li>Identifying relevant data and removing irrelevant data</li><li>Fix Irregular cardinality and structural errors</li><li>Outliers</li><li>Missing data treatment</li></ol><p>As discussed in the <a href="https://www.edvancer.in/descriptive-statistics-for-data-science/">blog</a>, running a basic descriptive statistics test provides an initial sense check of data in terms of missing values, variation in a feature, cardinality of features.</p><p><em>Figure 1: Summary Statistics</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/0*dN3ZmPOganJdT6LT.png" /></figure><p><em>Figure 2: Check for Null Values</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/0*LdKUIfRJjxtNg_O9.png" /></figure><p><em>Figure 3: Outliers highlighted in red</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/286/0*-_XHnZNxDPhTbrrc.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/285/0*h_-ooCosF6llsxWk.png" /></figure><h3>Identifying relevant data and removing irrelevant data</h3><p>Mainly two checks should be carried out for identifying irrelevant data:</p><ol><li><strong>Duplicate entries</strong></li></ol><p>Data is usually collected from many sources and joined together to form one dataset. This might lead to the duplicity of observations. This might not be problematic if observations (small in number) are repeated a few times but could lead to erroneous behaviour if observation(s) are repeated far too many times. Hence, it is better to drop duplicate observations to have a cleaner dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/0*gZhHUzVQA8b_RST_.png" /></figure><p><strong>2. Irrelevant observations</strong></p><p>The dataset would contain observations which might not be useful for your specific task. For example, if you were analysing the shopping behaviour of women only, you would not require observations for men in your dataset — row-wise.</p><p>Similarly, your data might have a column for id or names of employees which would not be very helpful in making predictions and could be dropped — column-wise.</p><h3>Fix Irregular cardinality and structural errors</h3><ol><li>Drop the columns which have a cardinality of 1 (for categorical features), or zero or very low variance (for continuous features). Such features do not provide much information and are not useful for building predictive models.</li><li>Categorical columns could have a high number of classes due to typos or inconsistent capitalization. For example, the gender column might have many classes like male, female, m, f, M, and F, these represent only two levels — Male and Female. Such classes should be mapped to proper levels and the rest of the levels should be dropped. Bar plots could be used to highlight such issues.</li><li>Ensure that the data is represented by the correct data type i.e. numbers are stored as int/float, date as a date object.</li></ol><p>There could be many issues like whitespaces in the feature values, data in a feature could have mixed data types like a numerical column might have some numbers as numerals and some as strings or objects. Fixing such errors would result in a cleaner and easy to interpret and use dataset.</p><h3>Outliers</h3><p>In the previous <a href="https://www.edvancer.in/data-visualisation-descriptive-statistics">blog</a>, we discussed ways to identify outliers. To summarise, data points could be treated as if the data points lie:</p><ol><li>above <strong>Q3 + 1.5*IQR</strong> or below <strong>Q1–1.5*IQR</strong>, if the data follows a non-Gaussian distribution</li><li>2 or 3 standard deviations away from the feature mean (z-score), if the data follows a Gaussian distribution</li><li>Visualise univariate variables by plotting Box plots, histograms or scatterplot (as shown in figure 3)</li></ol><p>These methods are good for initial analysis of univariate or bivariate (scatterplot) variables but do not hold much value for multivariate data or data with high dimensions. For such cases, advanced methods like clustering, PCA, LOF (Local Outlier Factor) &amp; HiCS: High Contrast Subspaces for Density-Based Outlier Ranking should be leveraged.</p><h4>Ways to deal with Outliers</h4><p><strong>1. To retain or to trim</strong></p><p>We should not drop any observations without careful consideration, as such information can be valuable to understand the unusual behaviour or anomalies in the data unless we are sure that the outlier may be due to measurement error or if the presence of outliers influences the fit of a model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/0*od6K8T-BwdqGTAHg.png" /></figure><p><em>Figure 4: With Outliers (left pic), the regression line seems to be influenced by the extreme points in the top. Without Outliers (right pic). the regression line better fits the data after dropping the extreme points.</em></p><p><strong>2. Winsorising or Clamp Transformation: </strong>Winsorising replaces the outliers with the nearest nonsuspect data. This is also called clamp transformation as we clamp all values above an upper threshold and below a lower threshold to these threshold values, thus capping the values of outliers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/0*G_5g1CKmlr8fkLeZ.png" /></figure><p>where <em>fi </em>is a specific value of feature <em>f</em>, and lower and upper are the lower and upper thresholds., given by either the IQR method or z-score discussed above.</p><p>It is recommended to apply the clamp transformation in cases where it is suspected that a model is performing poorly due to the presence of outliers. The better way to evaluate the impact of winsorising is by comparing the performance of different models trained on datasets where the transformation has been applied and where it has not.</p><p><strong>3. Use algorithms robust to outliers</strong></p><p>Tree-based algorithms and boosting methods are insensitive to outliers due to the intrinsic nature of the recursive binary splitting approach used to partition the feature space. These algorithms are the best bet when there are outliers in the input features.</p><p>If there are outliers in the target variable, tree-based algorithms are good but care must be taken to choose the loss function. Reason being that if we use the mean squared error function, then the difference is squared and would highly influence the next tree since boosting attempts to fit the (gradient of the) loss. However, there are more robust error functions that can be used for boosted tree methods like Huber Loss and Mean Absolute Deviation loss.</p><p>Data Cleaning and Descriptive Statistics and EDA are a very important part of data science life cycle. Understanding the data is crucial to get high-level insights about the population behaviour and it also helps you decide which transformations to apply, if required and which algorithms would give best results for your specific business problem.</p><p>We will talk about how to treat missing values in the next blog.</p><p>References:</p><p><a href="https://www.r-bloggers.com/outlier-detection-and-treatment-with-r/">https://www.r-bloggers.com/outlier-detection-and-treatment-with-r/</a></p><p><em>Originally published at </em><a href="https://www.edvancer.in/data-cleaning"><em>https://www.edvancer.in</em></a><em> on June 26, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e42d5e0df05a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/limitedio/data-cleaning-the-secret-ingredient-to-the-success-of-any-data-science-project-e42d5e0df05a">Data Cleaning — the secret ingredient to the success of any Data Science Project</a> was originally published in <a href="https://medium.com/limitedio">LiMiTeD</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understand the data better by Visualisation]]></title>
            <link>https://medium.com/dataseries/understand-the-data-better-by-visualisation-acd336ecbf81?source=rss-ccb50745d2fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/acd336ecbf81</guid>
            <category><![CDATA[data-visualisation]]></category>
            <category><![CDATA[histograms]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[descriptive-statistics]]></category>
            <category><![CDATA[box-plot]]></category>
            <dc:creator><![CDATA[Yogita Kinha]]></dc:creator>
            <pubDate>Tue, 18 Jun 2019 00:00:00 GMT</pubDate>
            <atom:updated>2019-07-01T16:14:54.823Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/738/0*_B0hVkzYdgKnhI55.jpg" /></figure><p>Continuing with the last series of <a href="https://www.edvancer.in/DESCRIPTIVE+STATISTICS+FOR+DATA+SCIENCE-2">blog</a>s, in this blog, we are going to explore graphical ways of representing descriptive statistics. We have learnt that we can get the measures of central tendency and dispersion to understand the centre and spread of the features (a variable is a feature in a dataset). It is easy to analyse with these measures when the data is small. For larger datasets with large feature space (variables), visualisation provides a quick, easy-to-read-and-interpret pictorial representation of the distribution of data.</p><p><strong>Graph Types for Univariate and Bivariate Data</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/258/0*qhx6SlrD7vUX7y3R.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/0*mQVl2_wXqLMfjLGl.png" /></figure><p><strong>Histograms </strong>are used to <strong>visualise the distribution of a continuous variable</strong>. A histogram displays the frequency of occurrence of values of a variable. The values are converted into bins of values (on the horizontal axis) and on the vertical axis are displays bars corresponding to the count or percentage of observations falling in the bin.</p><p>Histograms help understand:</p><ol><li>the centre of the data i.e. whether the data is symmetrically centred around mean or is asymmetric in nature</li><li>the dispersion of data i.e. whether the data is closely clustered around the mean or spread out more in either direction</li><li>the skew of the data i.e. whether the data is left-skewed or right-skewed.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/0*u1SYqan0I-TVushp.png" /><figcaption><em>Figure 1: Histograms representing various distributions of data</em></figcaption></figure><p>4. the data is bimodal meaning the data might be from two different distributions or processes or from a process that frequently shifts.</p><p>5. the data is multimodal meaning the sample might have two or more different subpopulations which have different characteristics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/0*9BM-N7SrlH17paf_.png" /><figcaption><em>Figure 2: Multimodal Distribution</em></figcaption></figure><p>Bimodal and multimodal distributed data needs in-depth analysis to find out whether the data has different peak values in the distribution by virtue of the characteristics of one population or the population exhibits extreme patterns or behaviours or comes from a number of distinct groups. For example, if we were to measure the weights of a randomly selected group of men and children, we would expect a bi-modal distribution with a peak at around 50kg for children and 70kg for men.</p><p>6. Histograms can also be used to identify outliers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/227/0*75bbeRCdomWF2pVN.png" /><figcaption><em>Figure 3: Histogram depicting Outliers</em></figcaption></figure><p>In this histogram, the bar at value 70 looks like an outlier. An outlier would be an isolated bar in a histogram. This observation should be analysed carefully to see if this is an error in measurement or an observation occurred under unusual conditions or it might be an observation that accurately describes the variability in the study area.</p><p><strong>Bar graph</strong> represents categorical data summarized in a frequency, relative frequency, or percent frequency distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/336/0*gMFAqWz3wAUoqIYV.png" /><figcaption><em>Figure 4</em>:<em> Distribution of total bill on weekdays and weekend</em></figcaption></figure><p>A bar graph shows comparisons among discrete categories. One axis of the chart shows the specific categories being compared, and the other axis represents a measured value.</p><p>Bar graphs are also a useful method of visualising ordinal i.e. ordered categorical data. Bar charts arranged from highest to lowest incidence are called Pareto charts.</p><p>When there is no natural ordering of the categories being compared, bars on the chart may be arranged in any order.</p><p><strong>Pie chart</strong> is commonly used for presenting relative frequency distributions for qualitative data. The circle is divided into sectors corresponding to the relative frequency for each class.</p><p>We can use pie chart when categories have no particular order or are dichotomous.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/452/0*-KjMswAC4vAPdxBe.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/0*3WFHyM17uJ3Tgf3k.png" /></figure><p><strong>Box Plot </strong>displays the distribution of data based on the five-number summary: minimum, first quartile, median, third quartile, and maximum.</p><p>A box plot displays the full range of variation (from min to max), the likely range of variation (the IQR), a typical value (the median), and possible outliers. Since, boxplots break the data into quartiles, they work best when the data has at least 20 data points per group.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/274/0*bTyyMb0puNNGdiFZ.png" /></figure><p>The image on the left shows how boxplots compare to the probability distribution function for a normal distribution. Notice how each whisker contains 24.65% of the distribution rather than an exact 25%. Boxplots consider the observation is beyond the whiskers to be outliers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/0*kP9o50zxx-Gn51i8.png" /></figure><p>John Tukey has provided a precise definition for two types of outliers:</p><p><strong>Outliers</strong>: Data points lying beyond 3×IQR or the third quartile, or 3×IQR or below the first quartile.</p><p><strong>Suspected outliers</strong>: either 1.5×IQR or more above the third quartile or 1.5×IQR or more below the first quartile.</p><p><strong>Violin Plots </strong>are a combination of boxplot and density plot to show the distribution of the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/229/0*Igfuve3Tv2C2jzYH.png" /></figure><p>A violin plot shows the full distribution of the data along with the summary statistics (by box plot) such as mean/median and interquartile ranges.</p><p>Violin plots are particularly useful when the amount of data is huge, and the data distribution is multimodal. In this case, a violin plot shows the presence of different peaks, their position and relative amplitude.</p><p><strong>Line Graphs</strong> are used to display numerical values as a series of data points over a continuous interval or time period.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/316/0*JxlmePnGO7iW_K2-.png" /></figure><p>A Line Graph is similar to scatterplot except that the data in the line graph are chronologically ordered. Line graphs are usually used to show trends in data over intervals of time. The direction of the line indicates an upward or downward trend in the value of a feature over time.</p><p><strong>Scatter plots </strong>are similar to the line graph and depict the relationship between two numerical variables. Scatter plots graphically show how one variable is affected by a change in the values of another variable. Scatter plots capture the linear relationship between variables called Correlations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/274/0*qhpkyDSIj1-csSeA.png" /></figure><p>Correlation could be positive meaning increase in the value of a variable results in an increase in another, could be negative meaning increase in the value of one variable leads to decrease in value of another, or the variables could be not related at all.</p><p>A line of <a href="https://en.wikipedia.org/wiki/Curve_fitting">best fit</a> or a trendline can be drawn in order to study the relationship between the variables. An equation for the correlation between the variables can be determined by established best-fit procedures. For a linear correlation, the best-fit procedure is known as <a href="https://en.wikipedia.org/wiki/Linear_regression">linear regression</a>.</p><p>An interesting point to note about correlation is that correlation does not mean causation i.e. change in one variable does not cause the change in another variable. Correlation merely indicates the association between the variables.</p><p><strong>References:</strong></p><p><a href="http://www.cs.put.poznan.pl/jstefanowski/sed/DM14-visualisation.pdf">http://www.cs.put.poznan.pl/jstefanowski/sed/DM14-visualisation.pdf</a> <a href="https://homes.cs.washington.edu/~suinlee/genome560/lecture1.pdf">https://homes.cs.washington.edu/~suinlee/genome560/lecture1.pdf</a> <a href="http://www.physics.csbsju.edu/stats/box2.html">http://www.physics.csbsju.edu/stats/box2.html</a> <a href="https://datavizcatalogue.com/methods/violin_plot.html">https://datavizcatalogue.com/methods/violin_plot.html</a> <a href="https://statisticsbyjim.com/basics/data-types/">https://statisticsbyjim.com/basics/data-types/</a> <a href="http://www.j-pcs.org/viewimage.asp?img=JPractCardiovascSci_2018_4_2_116_240962_f1.jpg">http://www.j-pcs.org/viewimage.asp?img=JPractCardiovascSci_2018_4_2_116_240962_f1.jpg</a></p><p><em>Originally published at </em><a href="https://www.edvancer.in/data-visualisation-descriptive-statistics"><em>https://www.edvancer.in</em></a><em> on June 18, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=acd336ecbf81" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dataseries/understand-the-data-better-by-visualisation-acd336ecbf81">Understand the data better by Visualisation</a> was originally published in <a href="https://medium.com/dataseries">DataSeries</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A must-read collection of books for Data Science]]></title>
            <link>https://medium.com/limitedio/a-must-read-collection-of-books-for-data-science-403b222fa1a4?source=rss-ccb50745d2fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/403b222fa1a4</guid>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[top-10-books]]></category>
            <category><![CDATA[data-science-book]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning-book]]></category>
            <dc:creator><![CDATA[Yogita Kinha]]></dc:creator>
            <pubDate>Sat, 15 Jun 2019 00:00:00 GMT</pubDate>
            <atom:updated>2019-07-02T09:51:35.651Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/738/0*_q-vGNDY_VDyKXHV.jpg" /></figure><p>“If you don’t like to read, you haven’t found the right book.” — J.K. Rowling</p><p>Well, this is absolutely true in my case. I had never read a book outside of my school curriculum ever before I stumbled upon the Harry Potter series. The books beautifully depict the adventurous journey of little novice magicians to become powerful wizards.</p><p>Following the same chain of thoughts, I present to you a collection of books which one must read to get on the journey to become an expert magician of machine learning and data science.</p><ol><li><strong>Think Stats Paperback by Allen B. Downey</strong></li></ol><p>Statistics lays the foundation of machine learning and data science. This book introduces and details about the rules of probability and probability distributions, data visualization techniques, and statistical methods including hypothesis testing and correlation.</p><p>Every chapter has a case study with python code snippets for better understanding of concepts.</p><p>Download link: <a href="http://greenteapress.com/thinkstats/thinkstats.pdf">http://greenteapress.com/thinkstats/thinkstats.pdf</a></p><p><strong>2. Automate the Boring Stuff with Python by Al Sweigart</strong></p><p>A fun book for quickly learning python programming for total beginners. This book will make your life easy by helping you automate mundane tasks like searching for a string in a pdf or send reminder emails or format data in excel etc.</p><p>Download link: <a href="https://automatetheboringstuff.com/">https://automatetheboringstuff.com/</a></p><p><strong>3.</strong> <strong>Python Machine Learning by Sebastian Raschka</strong></p><p>The book covers the Machine Learning concepts along with simple mathematical details and python code snippets using Scikit-Learn libraries.</p><p>A good read for everyone — if you have already studied machine learning theory in detail, this book will show you how to put your knowledge into practice. If you have used machine learning techniques before and want to gain more insight into how machine learning really works, this book is for you. Don’t worry if you are completely new to the machine learning field; you have even more reason to be excited.</p><p>Download link: <a href="http://file.allitebooks.com/20151017/Python%20Machine%20Learning.pdf">http://file.allitebooks.com/20151017/Python%20Machine%20Learning.pdf</a></p><p><strong>4. Fundamentals of Machine Learning for Predictive Data Analytics</strong> <strong>by John D. Kelleher, Brian Mac Namee and Aoife D’Arcy</strong></p><p>An introductory text on the fundamentals of machine learning, and the applications of machine learning. Chapters in this book begin with a big idea explaining the concept intuitively and building on to the technical details gradually. This book also has many real-time case studies showcasing the whole lifecycle of a data science project.</p><p>Download link:</p><p><a href="http://222.195.93.137/gitlab/winston.wen/bookshelf/raw/82f0d2633bd8385c6a077fbfc1cf05a6e20c178c/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/MIT.Fundamentals.of.Machine.Learning.for.Predictive.Data.Analytics.0262029448.pdf">http://222.195.93.137/gitlab/winston.wen/bookshelf/raw/82f0d2633bd8385c6a077fbfc1cf05a6e20c178c/%E6%9C%BA%E5%99%A8%E5%AD%A6%E4%B9%A0/MIT.Fundamentals.of.Machine.Learning.for.Predictive.Data.Analytics.0262029448.pdf</a></p><p><strong>5. An Introduction to Statistical Learning</strong> <strong>with R by Trevor Hastie and Rob Tibshirani</strong>: A must-read for a broader understanding of ML concepts. This book explains the topics in great detail with simplicity, detailing from assumptions to high-level mathematical details, to interpreting outputs to applications of the concepts, along with a lab section in R with every chapter.</p><p>Download link: <a href="https://www-bcf.usc.edu/~gareth/ISL/ISLR%20Sixth%20Printing.pdf">https://www-bcf.usc.edu/~gareth/ISL/ISLR%20Sixth%20Printing.pdf</a></p><p><strong>6.</strong> <strong>Introduction to Data Science by Rafael A Irizarry</strong></p><p>This book introduces concepts and skills to tackle real-world data analysis challenges. It provides a simple introduction to all the essential concepts ranging from probability, statistical inference, linear regression and machine learning. It covers basics of R programming, packages like dplyr, ggplot2, caret, file organization with UNIX/Linux shell, version control with Git and GitHub, and reproducible document preparation with knitr and R markdown.</p><p>Download link: <a href="https://rafalab.github.io/dsbook/">https://rafalab.github.io/dsbook/</a></p><p><strong>7. Advanced R by Hadley Wickham</strong></p><p>This book is a must to read for anyone wishing to write efficient and faster codes in R. This book is for advanced intermediate R programmers and for programmers transitioning to R from other languages. A good book to read after having read the Introduction to Data Science book which covers the basics of R programming.</p><p>Download link: <a href="https://englianhu.files.wordpress.com/2016/05/advanced-r.pdf">https://englianhu.files.wordpress.com/2016/05/advanced-r.pdf</a></p><p><strong>8. Deep Learning with Python by FRANÇOIS CHOLLET</strong></p><p>This book is for anyone who wishes to explore deep learning from scratch or broaden their understanding of deep learning. This book offers a practical, hands-on exploration of deep learning. It focuses more on to explain the quantitative concepts via code snippets and to build practical intuition about the core ideas of machine learning and deep learning. The code examples use the Python deep-learning framework Keras, with Tensor-Flow as a backend engine. Personally, this is one of the few technical books that I enjoyed reading, just like reading the Harry Potter series as a kid.</p><p>Download link: <a href="https://tanthiamhuat.files.wordpress.com/2018/03/deeplearningwithpython.pdf">https://tanthiamhuat.files.wordpress.com/2018/03/deeplearningwithpython.pdf</a></p><p><strong>9. Machine Learning Yearning by Andrew Ng</strong></p><p>This book focuses on how to make the ML algorithms work efficiently. The book does not talk about the ML concepts in detail, but explains how to structure the machine learning project development, details the best practices and common pitfalls along with case studies. Andrew Ng gives all the important tips on troubleshooting a machine learning system in real life.</p><p>Download link: <a href="https://www.deeplearning.ai/machine-learning-yearning/">https://www.deeplearning.ai/machine-learning-yearning/</a></p><p><strong>10. The Art of Data Science by Roger D. Peng and Elizabeth Matsui</strong></p><p>The Art of Data Science” dives into the practice of exploring and finding discoveries within any lake of data at your fingertips. It focuses on the process of analysing data and filtering it down to find the underlying stories. The authors use their own experiences to coach both beginners and managers through analysing data science.</p><p>Hope you find this list useful.</p><p>Please share your feedback about these books and suggest more books for all of us to read!!</p><p>Happy reading!!</p><p><em>Originally published at </em><a href="https://www.edvancer.in/a-must-read-collection-of-books-for-data-science/"><em>https://www.edvancer.in</em></a><em> on June 15, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=403b222fa1a4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/limitedio/a-must-read-collection-of-books-for-data-science-403b222fa1a4">A must-read collection of books for Data Science</a> was originally published in <a href="https://medium.com/limitedio">LiMiTeD</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to quantify the relationship between categorical and continuous variables]]></title>
            <link>https://medium.com/limitedio/how-to-quantify-the-relationship-between-categorical-and-continuous-variables-caa2c75c4723?source=rss-ccb50745d2fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/caa2c75c4723</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[descriptive-statistics]]></category>
            <category><![CDATA[chi-square]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[correlation]]></category>
            <dc:creator><![CDATA[Yogita Kinha]]></dc:creator>
            <pubDate>Fri, 07 Jun 2019 00:00:00 GMT</pubDate>
            <atom:updated>2019-07-01T16:34:52.465Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/738/0*U5PzA0jUe_xeDdZz.png" /></figure><p><em>In the last </em><a href="https://www.edvancer.in/descriptive-statistics-for-data-science/"><em>blog</em></a><em>, we discussed ways to get descriptive statistics for univariate continuous variables. If there are no outliers and the distribution is symmetric, the mean and standard deviation are excellent measures of central tendency and dispersion, whereas the median and IQR may be more appropriate if outliers or strong skewness is present.</em></p><p>In data analysis and modelling, we are mostly interested in understanding the relationship between two or more variables, and determine which variables are related in a predictive sense to a particular response variable. <strong>Measures of Association are used to quantify the relationship between two or more variables.</strong></p><p>The method used to determine any association between variables would depend on the variable type. Data could be on an interval/ratio scale i.e. continuous, or at an ordinal/rank scale, or a nominal/categorical scale. Based on the type of data, the following methods of association could be used to determine the association and the strength of association between variables.</p><p><strong>Pearson’s correlation coefficient </strong>measures the strength of the linear relationship between two variables on a <strong>continuous</strong> scale. For example, the relationship between height and weight of a person or price of a house to its area.</p><blockquote>The correlation coefficient, r (rho), takes on the values of −1 through +1. Values of −1 or +1 indicate a perfect linear relationship between the two variables, and a value of 0 indicates no linear association. (There could be non-linear association).</blockquote><p>A value between 0 and +1 means the variables are positively correlated i.e. increase in the value of one variable leads to an increase in another. For example, a house with more sqft would have a higher price.</p><p>A value between -1 and 0 means the variables are negative correlated i.e. increase in the value of one variable leads to decrease in another.</p><p>Any value that is not 0 or −1 or +1 indicates a linear relationship, although not a perfect linear relationship.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/295/0*VWvUxzypQsM9mB7I.png" /><figcaption>Figure 1: Positively Correlated: horsepower increases as the weight of the cars increases</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/293/0*aalczOuQZFwVXdBQ.png" /><figcaption>Figure 2: Negatively correlated: miles per gallon decreases as the weight of cars increases</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/293/0*2L0uUJ75hwYvzFAm.png" /><figcaption>Figure 3: Weakly correlated: there seems to be a very weal correlation between mpg and acceleration</figcaption></figure><p><strong>Spearman rank-order correlation coefficient </strong>measures the measure of the strength and direction of association that exists between two variables. The test is used for either <strong>ordinal variables</strong> or for continuous data that has failed the assumptions necessary for conducting the Pearson’s product-moment correlation. For example, you could use a Spearman’s correlation to understand whether there is an association between exam performance and time spent revising; whether there is an association between depression and length of unemployment.</p><p>It is denoted by r<em>s</em>, can take values from +1 to -1. A r<em>s</em> of +1 indicates a perfect association of ranks, a r<em>s</em> of zero indicates no association between ranks and a r<em>s</em> of -1 indicates a perfect negative association of ranks. The closer r<em>s</em> is to zero, the weaker the association between the ranks.</p><p><strong>Kendall’s tau-b (<em>τ</em>b) correlation coefficient</strong> can also be used as an alternative to the Spearman rank-order correlation coefficient, especially when you have a small sample size with many tied ranks.</p><p>Point to note here is that all these coefficients measure of the strength of an association (specifically the linear relationship), they do not measure the significance of the association. The significance of an association is a separate analysis of the sample correlation coefficient, r, using a t-test to measure the difference between the observed r and the expected r under the null hypothesis.</p><p><strong>Chi-Square test</strong> is used to determine the association between two <strong>categorical</strong> variables. The chi-square test, unlike Pearson’s correlation coefficient or Spearman rho, is a measure of the significance of the association rather than a measure of the strength of the association.</p><p>This test should be used to determine whether variables like education, political views and other preferences vary based on demographic factors like gender, race and religion. Or, to verify the influence of gender on purchase decisions and patterns.</p><p>The Chi-square test is based on hypothesis testing, and the hypothesis is defined as:</p><p><strong>Null Hypothesis</strong>: There is no relationship between the two variables of interest.</p><p><strong>Alternate Hypothesis</strong>: There is a significant relationship between the variables of interest.</p><p>The test involves comparing the observed frequencies of the variables to the expected frequencies which are calculated assuming the variables were independent of each other. If the p-value is less than the critical value, the test is significant. This means that there is an association between two variables. Otherwise, the test is nonsignificant and implies that there is not an association.</p><p>We summarised the Measures of Association here in this blog. In the next blog, we will talk about visualisation techniques of descriptive statistics with python code.</p><p>References:</p><p><a href="https://www.britannica.com/topic/measure-of-association">https://www.britannica.com/topic/measure-of-association</a> <a href="https://statistics.laerd.com/spss-tutorials/spearmans-rank-order-correlation-using-spss-statistics.php">https://statistics.laerd.com/spss-tutorials/spearmans-rank-order-correlation-using-spss-statistics.php</a></p><p><em>Originally published at </em><a href="https://www.edvancer.in/DESCRIPTIVE+STATISTICS+FOR+DATA+SCIENCE-2"><em>https://www.edvancer.in</em></a><em> on June 7, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=caa2c75c4723" width="1" height="1" alt=""><hr><p><a href="https://medium.com/limitedio/how-to-quantify-the-relationship-between-categorical-and-continuous-variables-caa2c75c4723">How to quantify the relationship between categorical and continuous variables</a> was originally published in <a href="https://medium.com/limitedio">LiMiTeD</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DESCRIPTIVE STATISTICS FOR DATA SCIENCE? It’s Easy If You Do It Smart]]></title>
            <link>https://medium.com/limitedio/descriptive-statistics-for-data-science-its-easy-if-you-do-it-smart-a970388e1636?source=rss-ccb50745d2fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/a970388e1636</guid>
            <category><![CDATA[descriptive-statistics]]></category>
            <category><![CDATA[variance]]></category>
            <category><![CDATA[categorical-data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Yogita Kinha]]></dc:creator>
            <pubDate>Mon, 27 May 2019 00:00:00 GMT</pubDate>
            <atom:updated>2019-07-01T16:47:48.279Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/738/0*1WDstZ69NCowLWT-.jpg" /></figure><p><strong>So you have the business objective and you have painstakingly collected the data as well, now what? What to do with the data, how to understand the distribution of the target variable and how to analyse the influence or effect of other factors on the target?</strong></p><p><strong>The answer to this is Statistics.</strong></p><p>Statistics is a branch of applied mathematics that provides methods and tools to collect, analyse, present, and interpret data, and to make decisions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/305/0*8r3_9j0cayVRk8Nh.png" /></figure><p>Statistics is about studying the data to get useful insights and making educated estimates about the population.</p><p>The picture on the left quantifies the motivational reasons for betrayals among allies, families, kingdoms, lovers and rivals in the most famous show of our times — GoT. Had Danaerys had this summary at her disposal, she would have known better to keep her allies under strict watch 😉</p><p>Statistics can be divided into two main areas: Descriptive Statistics and Inferential Statistics. Let’s understand the basics of data before diving deeper into descriptive and inferential statistics.</p><p><strong>Variable Types: </strong>A data set is a collection of observations on one or more variables. For example, the height of all students in a class can be represented by the variable height and similar weight as weight variable. Depending on what type of data a variable holds, variables can be classified as in the picture below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/0*AFaOwcxlsjZ9AJ-l.png" /></figure><p><strong>Population and Sample</strong></p><p>A population consists of all elements-individuals, items, or objects-whose characteristics are being studied. The population that is being studied is also called the target population. Since, most of the times, it is impossible or too expensive to get the data for the whole target population, a portion of the population is selected for study referred to as a sample. A sample is used to make inferences about the population behaviour.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/0*JhGyzmk6gMoXuuZL.png" /></figure><p>Now that we have an understanding of data and samples &amp; population, how do we analyse the data and make it useful?</p><p>Here comes into picture the two areas of Statistics: Descriptive Statistics and Inferential Statistics</p><p><strong>Descriptive statistics</strong>, as the name suggests, is used to display and describe data by using tables, graphs and summary measures. <strong>Inferential Statistics</strong>, on the other hand, pertains to studying a sample and use the results to make decisions or predictions about a population.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/414/0*_IfH2epLdGgtN3tP.png" /></figure><p>In this article, we are going to focus on Descriptive Statistics and its various measures.</p><p>Descriptive Statistics and exploratory data analysis should be the first steps while building predictive or inference models. Descriptive statistics help understand large amounts of data by providing methods to summarise the data and retrieve information about the underlying structure of the data.</p><p>There are two ways to Descriptive statistics: Numerical and Graphical.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*DyWeSRz5Vjjr3p0miDpa4g.png" /></figure><p><strong>Measures of Central Tendency: </strong>These are computed to estimate a central value around which the values in the data are distributed<strong>. </strong>Mean is the most commonly used method of describing the central tendency. Types of mean used are the arithmetic mean, weighted mean, geometric and harmonic mean depending on the type of data type. Median is the value at the exact middle of a sorted data series. Mode is the most frequently occurring value in the data set.</p><p>Mean should be used for continuous data having symmetric distributions and no outliers. Median should be used when the data is skewed or has outliers. Mode should be used for qualitative and categorical data, for example, if we want to know the soil type in a locality then mode would be the best measure to use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/361/0*cQEhDvy7_E1J3fK0.png" /><figcaption>Measures of Central Tendency</figcaption></figure><p><strong>Measures of Dispersion: </strong>Measures of central tendency do not give complete information about the distribution of type. They only provide the central value estimates but tell nothing about the variation in the data. The measures of central tendency and dispersion taken together give a better picture of a data set than the measures of central tendency alone.</p><p><strong>Range</strong> is calculated as the difference between the largest value and the smallest value in the dataset. The range is the simplest measure but is quite sensitive to extreme values.</p><p><strong>Variance and Standard Deviation</strong> are the most used measure of dispersion. Variance is calculated by taking the average of the squared differences between each value and the mean and Standard Deviation is equal to the square root of the variance. Standard Deviation is simply the standardised form of variance.</p><p><strong>Standard Deviation</strong> tells about the spread of values around the mean. In general, a lower value of the standard deviation for a data set indicates that the values of that data set are spread over a relatively smaller range around the mean. While a larger value of the standard deviation for a data set indicates that the values of that data set are spread over a relatively larger range around the mean.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/299/0*8GYPap3dUyOr65Rz.png" /><figcaption>Spread of Variance</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/0*LTWa9XnRK4b0fhwv.png" /><figcaption>Clustering of data around mean &amp; Standard Deviation</figcaption></figure><p><strong>Quartiles and Interquartile Range (IQR)</strong></p><p>Quartiles divide a rank-ordered data set into four equal parts. Q1 represents the first quartile and is the value for which 25% of the observations are smaller and 75% are larger. Q2 is the same as the median (50% are smaller, 50% are larger). Q3 is the third quartile and only 25% of the observations are greater than it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/296/0*Z6q5c9PGMFFv-6_R.png" /></figure><p>IQR is equal to the difference between 75th and 25th percentiles, or between upper and lower quartiles, i.e. IQR = Q3 − Q1. IQR is used as a measure of dispersion for skewed data and to identify outliers. Upper outliers: Q3+1.5*IQR and Lower outliers: Q1–1.5*IQR.</p><p><strong>Measures of Association </strong>indicate how two variables are associated with each other, if at all. The two commonly used approaches for testing this are Chi-Square and Correlation.</p><p>We will discuss these approaches of association and graphical methods of descriptive statistics in the upcoming blogs. Keep watching this space for more.</p><p>Hope you like the reads. Please share constructive feedback and topics that you would like to read about.</p><p><strong>References</strong></p><p><a href="https://sciencestruck.com/descriptive-vs-inferential-statistics">https://sciencestruck.com/descriptive-vs-inferential-statistics</a> <a href="https://www.nature.com/articles/6400501/figures/1">https://www.nature.com/articles/6400501/figures/1</a> <a href="https://infograph.venngage.com/templates/infographics/statistical?preview=6dd8cb98-4a89-4a0a-9b65-5ba57eecd0b4">https://infograph.venngage.com/templates/infographics/statistical?preview=6dd8cb98-4a89-4a0a-9b65-5ba57eecd0b4</a></p><p><a href="https://www.gs.washington.edu/academics/courses/akey/56008/lecture/lecture2.pdf">https://www.gs.washington.edu/academics/courses/akey/56008/lecture/lecture2.pdf</a> <a href="https://ia800608.us.archive.org/24/items/IntroductoryStatistics7thEdP.MannWiley2010WW/Introductory%20Statistics%207th%20ed%20-%20P.%20Mann%20%28Wiley%2C%202010%29%20WW.pdf">https://ia800608.us.archive.org/24/items/IntroductoryStatistics7thEdP.MannWiley2010WW/Introductory%20Statistics%207th%20ed%20-%20P.%20Mann%20%28Wiley%2C%202010%29%20WW.pdf</a></p><p><a href="https://en.wikipedia.org/wiki/Normal_distribution">https://en.wikipedia.org/wiki/Normal_distribution</a></p><p><em>Originally published at </em><a href="https://www.edvancer.in/descriptive-statistics-for-data-science/"><em>https://www.edvancer.in</em></a><em> on May 27, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a970388e1636" width="1" height="1" alt=""><hr><p><a href="https://medium.com/limitedio/descriptive-statistics-for-data-science-its-easy-if-you-do-it-smart-a970388e1636">DESCRIPTIVE STATISTICS FOR DATA SCIENCE? It’s Easy If You Do It Smart</a> was originally published in <a href="https://medium.com/limitedio">LiMiTeD</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 common myths about shifting career to Data Science]]></title>
            <link>https://medium.com/limitedio/5-common-myths-about-shifting-career-to-data-science-82f700e20ef6?source=rss-ccb50745d2fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/82f700e20ef6</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[learning]]></category>
            <dc:creator><![CDATA[Yogita Kinha]]></dc:creator>
            <pubDate>Mon, 06 May 2019 00:00:00 GMT</pubDate>
            <atom:updated>2019-06-07T02:43:31.783Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*oKxjJWjeL1EJgg95.png" /></figure><p>There are various myths rampant about data science. Data Science did not just come into being overnight, rather it has evolved over a long period of time to become an independent field. In 1960, Peter Nour used it as a substitute for computer science. In 1974, Naur published a study which freely used the term data science for the contemporary data processing methods used in a wide range of applications. Fast forwarding to 2001 where William S. Cleveland introduced data science as an independent discipline, extending the field of statistics to incorporate advances in computing with data. In 2008, DJ Patil and Jeff Hammerbacher used the word “Data Scientist” for the first time to define their jobs at LinkedIn and Facebook, respectively.</p><p>This brief introduction conveys that data science has been in existence for long. It is gaining more and more popularity due to the availability of huge amounts of data, better computing capabilities and development of scalable machine learning algorithms and libraries. Since it is a fast-paced developing field opening many opportunities and possibilities, more and more people are focusing on building their career in data science. There has been a lot of confusion about the roles &amp; responsibilities of a data scientist and vague myths about how to become one. In this article, I try to address a few of them and shine a light on their realities.</p><p><strong>One needs to have a masters and/or PhD in Mathematics or Statistics or Data Science to pursue a career in Data Science.</strong> We discussed the kinds of roles required in a data science project and team in our last <a href="https://www.edvancer.in/how-to-become-a-data-scientist-ai-expert/">article</a>. To a person with a sound academic background in any of these fields, data science would be a natural choice of career. She/he could choose to work in any of the roles as per interests. Such a person could be involved in developing new algorithms or refining existing ones and so on. Having said that, it does not mean that a person with no such expertise can’t be successful in data science. There are roles like Data Analyst and Data Engineer which do not require expertise in math or stats. One needs to develop an understanding of business &amp; data and pick up on skills like excel and coding in python or R or spark. It is easy to build knowledge of distributed computing through online courses. One can learn stats and machine learning concepts from online or self-paced or classroom-based <a href="https://www.edvancer.in/courses/all/">courses</a>. With a good mix of theoretical concepts and hands-on experience on data science projects offered by these courses, one can easily switch to a career in data science.</p><p><strong>Data Science is all about analytical tools and coding.</strong> Data Science is not equivalent to learning and working on analytical tools like SAS or Kafka. It is also not all about always coding in languages like R and Python. Data Science projects go through similar life cycles as a normal software project. One needs to have business acumen, interpretation skills, problem-solving mindset, ability to be creative with data, presentation skills and communication skills. Working on tools and coding is a part of a data science role. A data scientist should be able to understand the business problem and think of innovative ways to solve it. A data scientist should also be capable of presenting findings and observations to non-experts through simple graphs and plain English.</p><p><strong>Data Scientists always work on developing predictive models. </strong>Due to the romanticism of the word data scientist, we might think that a data scientist’s job is to create complex predictive models only. But its only half the truth of the story. Building models is the least time consuming and the easiest task in the job list of a data scientist. A major chunk of a data scientist’s time goes towards gathering relevant data and cleaning it, processing and feature engineering. Data collection is the first and most challenging task in the data science project. One needs to figure out the data that should be analyzed for the stated problem. The data then needs to be processed to make sense of it. Choosing the right variables for a given problem can be very daunting and frustrating. That’s where domain knowledge helps. Hence, it’s not all about machine learning algorithm only and always.</p><p><strong>Working on Kaggle challenges translates to real-time project experience. </strong>In Kaggle competitions, companies share problem statements with the datasets and invite participants to solve them. These competitions are indeed great resources for learning about machine learning algorithms and feature engineering methods. But these competitions do not map to real-time data science projects. The datasets shared for such competitions are provided by the host companies. The major task of gathering, cleaning and processing of the data is done by the hosts themselves.. While in real time projects, data collection and processing tasks are to be done by the data science teams. These two tasks are very time consuming and complex. Finding the right data is supreme because analysis done on incorrect data would be not useful to anyone.</p><p><strong>You can learn Data Science in a few months. </strong>Data Science is not like working on a tool or fixed format use cases that you can master in a few months. Data science is about identifying patterns and deriving insights from the data; and transforming such knowledge to solve business problems. Each business problem is unique and may require a different thought process and ideas. One becomes a data science pro by working on a multitude of problem scenarios, continued learning and hard work. It is a hap field and it is exciting to be a part of it, but it comes with great efforts and learning.</p><p>It could be difficult to choose/change a career. Nevertheless, do a lot of research and make informed choices. It is important to have the aptitude and attitude to make it big in data science.</p><p>References:</p><p><a href="https://en.wikipedia.org/wiki/Data_science">https://en.wikipedia.org/wiki/Data_science</a></p><p><em>Originally published at </em><a href="https://www.edvancer.in/5-common-myths-about-shifting-career-to-data-science/"><em>https://www.edvancer.in</em></a><em> on May 6, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=82f700e20ef6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/limitedio/5-common-myths-about-shifting-career-to-data-science-82f700e20ef6">5 common myths about shifting career to Data Science</a> was originally published in <a href="https://medium.com/limitedio">LiMiTeD</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Are we really using Artificial Intelligence in our daily lives?]]></title>
            <link>https://medium.com/limitedio/are-we-really-using-artificial-intelligence-in-our-daily-lives-c5b73ac9b22e?source=rss-ccb50745d2fb------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5b73ac9b22e</guid>
            <category><![CDATA[everyday-life]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[virtual-assistant]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[learning]]></category>
            <dc:creator><![CDATA[Yogita Kinha]]></dc:creator>
            <pubDate>Sun, 21 Apr 2019 00:00:00 GMT</pubDate>
            <atom:updated>2019-05-14T04:40:03.299Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/738/0*4NqTjclDPA13_YW0.jpg" /></figure><p>The applications of Artificial Intelligence and Data Science are enormous in our daily lives. AI is making our life simpler each moment of the day without us realizing this. In this article, we will go through some examples of AI applications which we interact with every day.</p><h4>Virtual Assistants powered by Speech Recognition &amp; Natural Language</h4><p>To begin with, only by saying ‘Hello Siri’ or ‘Ok Google’, we are invoking the magic of neural networks. Neural networks used in these virtual assistants enable them to come alive to the sound of ‘Hello Siri’ or ‘Ok Google’ and help us semi-automate mundane tasks like setting the alarm, or finding a restaurant nearby, or checking the weather etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/363/0*qGP0XEQ3x0Ov27Eb.png" /></figure><p>Capability to answer such questions like a list of nearby restaurants or the time to drive to your office or news of the day are examples of natural language processing (NLP) tasks. The algorithms are trained on a large amount of data to understand the query and generate a suitable response.</p><h4>Google Photos and it’s magic by virtue of Image recognition</h4><p>Google photos uses ML and Neural networks to detect faces and backgrounds in the pictures and automatically classifies your images into different groups like people, places, videos etc.</p><p>Google photos keeps on improving itself by asking you to help map pictures to specific person or location etc. Google photos assistant uses ML to gather pictures with some similarities, for instance, pictures clicked at a certain place, with certain people or backgrounds, and stitches them together to make collages or videos for you.</p><h4>Music and Shopping Recommendations</h4><p>When you create a new Spotify account, the app asks you to select your preferred languages and favourite singers/artists. Then it runs ML in the background to fetch songs and albums based on your interests; and voila you are set to go on a road trip or make your usual office ride melodious. Spotify’s ML and NLP based recommendation system creates a personalised Discovered weekly collection every Monday for all its users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/252/0*uLpFZll78RJlghWk.png" /></figure><p>Similarly, Netflix recommends movies and shows to you based on your likes and ratings given by other people with similar interests as yours. E-commerce websites like Alibaba, Amazon create customized pages for each individual user.</p><p>All of these online platforms use machine learning to analyse the content of your music or movie interests or shopping habits and perform identification, personalized recommendation, and analysis.</p><h4>Social media networking</h4><p>Social media sites like Facebook, Instagram use AI and ML algorithms to provide personalized feed stories, friend suggestions, videos etc. Instagram and Facebook display targeted ads on your page by observing your past activities, likes and clicks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/277/0*b7EPedCyco-1Xk06.png" /></figure><p>LinkedIn uses ML algorithms to provide right job recommendations to candidates and hiring managers, to connect with professionals with similar backgrounds, and to provide each user with the right kind of feed. LinkedIn uses AI to suggest replies like “Thank you” or “Welcome” to your LinkedIn messages.</p><h4>Travel and Navigation made easy by AI</h4><p>The mobility-on-demand services like uber and ola use AI to match a vehicle and find the best route to enable the cab to reach you asap. Carpooling features of cabs use AI to align people travelling on the same route. These apps use AI to tell you the time it will take to reach your destination considering traffic conditions at that time. Surge pricing is also a function of AI (wink wink).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/170/0*vsBnwuB7tumgWqZW.png" /></figure><p>Logistics companies use AI to optimize the shortest routes for quick delivery of products. Google maps use AI (and GPS) to find the best route for you and to deliver the live traffic updates to you.</p><h4>Search Engines use AI</h4><p>Search engines like google, bing, yahoo use AI to optimise the search results and give faster responses. Search engines use Natural Language processing and image analysis to understand search queries and find the most relevant information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/359/0*u7b-rohbgqV4HP-b.png" /></figure><p>Google has started using RankBrain which is machine learning-based search engine algorithm from 2015 onwards. RankBrain helps to rank web pages to provide more relevant search results for users.</p><h4>Smart features in Gmail</h4><p>Gmail uses AI to classify the incoming emails into primary, social, promotions and spams based on their text content. Gmail suggests smart replies like “Thanks” or “Let’s do It” as per the context of the email and your writing style. Gmail also provides a nudging feature which reminds you to follow up on emails that you have not got any reply for and have forgotten about.</p><h4>Surprised!!😎</h4><p>This is the extent to which our lives are simplified by AI (Machine Learning + Natural Language Processing + Ranking Algorithms + Recommendation systems). And we have not even talked about self-driving cars, chatbots, and robots.</p><p><em>Originally published at </em><a href="https://www.edvancer.in/artificial-intelligence-in-our-daily-lives/"><em>https://www.edvancer.in</em></a><em> on April 21, 2019.</em></p><p>References:</p><p><a href="https://appletoolbox.com/2016/10/hello-hey-siri-tricks-tips/">https://appletoolbox.com/2016/10/hello-hey-siri-tricks-tips/</a> <a href="https://www.theverge.com/2015/9/30/9416579/spotify-discover-weekly-online-music-curation-interview">https://www.theverge.com/2015/9/30/9416579/spotify-discover-weekly-online-music-curation-intervie</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5b73ac9b22e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/limitedio/are-we-really-using-artificial-intelligence-in-our-daily-lives-c5b73ac9b22e">Are we really using Artificial Intelligence in our daily lives?</a> was originally published in <a href="https://medium.com/limitedio">LiMiTeD</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>