<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dr. Virendra Kumar Shrivastava on Medium]]></title>
        <description><![CDATA[Stories by Dr. Virendra Kumar Shrivastava on Medium]]></description>
        <link>https://medium.com/@profvkshrivastava?source=rss-11777614c8cc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*l362wuEi_hXVFeF9qNq5gA.jpeg</url>
            <title>Stories by Dr. Virendra Kumar Shrivastava on Medium</title>
            <link>https://medium.com/@profvkshrivastava?source=rss-11777614c8cc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 03:34:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@profvkshrivastava/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Unleashing the Power of Big Data Analytics and Machine Learning: Transforming Social Media Content…]]></title>
            <link>https://profvkshrivastava.medium.com/unleashing-the-power-of-big-data-analytics-and-machine-learning-transforming-social-media-content-7e0ac367ccbc?source=rss-11777614c8cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/7e0ac367ccbc</guid>
            <dc:creator><![CDATA[Dr. Virendra Kumar Shrivastava]]></dc:creator>
            <pubDate>Fri, 01 Sep 2023 11:34:00 GMT</pubDate>
            <atom:updated>2023-09-01T11:34:00.064Z</atom:updated>
            <content:encoded><![CDATA[<h3>Unleashing the Power of Big Data Analytics and Machine Learning: Transforming Social Media Content Moderation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*wn32XUPYnne0aiqcFh_n4g.jpeg" /><figcaption>Courtesy by: Courtesy by: Paul Lachine</figcaption></figure><p><strong>Introduction:</strong></p><p>In the age of disruptive technologies, the explosion of social media and online platforms has revolutionized communication and information-sharing. However, this transformation has also given rise to significant challenges, such as the spread of hate speech and misinformation. To counter these threats, the convergence of Big Data and Machine Learning has emerged as a powerful toolset. By harnessing the vast amounts of data generated online and leveraging sophisticated algorithms, researchers, tech companies, and social media organizations are making steps in identifying and mitigating hate speech and misinformation. This blog explores the role of Big Data and Machine Learning in combatting hate speech and misinformation, highlighting key advancements, challenges, and the way forward.</p><p><strong>Understanding Hate Speech and Misinformation:</strong></p><p>Hate speech encompasses various forms of offensive, discriminatory, or harmful language targeted towards individuals or groups based on attributes such as race, ethnicity, religion, gender, or sexual orientation. Misinformation, on the other hand, involves the dissemination of false or misleading information with the potential to misguide or deceive the audience. Both hate speech and misinformation pose serious threats to social cohesion, individual well-being, and democratic discourse in the civil society.</p><p><strong>The Role of Big Data:</strong></p><p>Big Data refers to the gigantic volume of structured and unstructured data generated at an unprecedented pace. Online interactions, social media posts, and digital footprints contribute to this data deluge. The sheer magnitude of data available provides an opportunity to uncover patterns, trends, and insights that were previously inaccessible.</p><p><strong>Machine Learning’s Contribution</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/301/1*_ZfCLie0Jw6K2wwTBz08Bw.jpeg" /></figure><p>Machine Learning, a subset of Artificial Intelligence, equips computers with the ability to learn from data and improve their performance over time without explicit programming. This technology has proven invaluable in identifying hate speech and misinformation through pattern recognition and predictive analysis.</p><p>a) <strong>Automated Content Moderation:</strong> Machine Learning algorithms can be trained to analyze text, images, and videos for hate speech indicators. Platforms like Facebook and X (previously known as Twitter), Telegram etc. employ these algorithms to automatically detect and remove offensive content. By learning from a vast dataset of labeled content, these algorithms become increasingly adept at identifying new instances of hate speech.</p><p>b) <strong>Sentiment Analysis:</strong> Machine Learning models can gauge the sentiment expressed in a piece of text, enabling the identification of potentially harmful content. Advanced sentiment analysis techniques consider not only the words used but also the context and underlying emotions.</p><p>c) <strong>Network Analysis:</strong> Machine Learning algorithms can analyze the connections and interactions within social networks. This analysis helps identify influential users who may be contributing to the spread of hate speech or misinformation. By targeting these users, platforms can disrupt the amplification of harmful content.</p><p>d) <strong>Detecting Deepfakes:</strong> Deepfakes are highly convincing manipulated media, such as videos or audio recordings, that can spread misinformation. Machine Learning algorithms can be trained to identify anomalies in media files, helping to detect deepfakes and flag potentially manipulated content.</p><p><strong>Challenges and Ethical Considerations:</strong></p><p>While Big Data and Machine Learning offer promising solutions, several challenges and ethical considerations must be addressed:</p><p>a) <strong>Contextual Understanding:</strong> Recognizing the nuances of language and context is complex. Algorithms may struggle to differentiate between genuine hate speech and discussions about hate speech, potentially leading to over-censorship.</p><p>b) <strong>Bias in Algorithms:</strong> Machine Learning models can inadvertently perpetuate bias present in training data, leading to uneven enforcement of content policies. Ensuring fairness and minimizing bias in algorithmic decision-making remains a significant challenge.</p><p>c) <strong>Freedom of Expression:</strong> Striking the balance between curtailing hate speech and safeguarding freedom of expression is challenging. Overly aggressive content moderation may hinder open discourse.</p><p>d) <strong>Privacy Concerns:</strong> The use of Big Data for monitoring online behavior raises privacy concerns. Striking a balance between effective monitoring and protecting user privacy is essential.</p><p>e) <strong>Cat-and-Mouse Game:</strong> As algorithms improve, those generating hate speech and misinformation also adapt their tactics. This creates a constant challenge of staying ahead in the detection game.</p><p><strong>The Way Forward</strong></p><p>To leverage the potential of Big Data and Machine Learning in combating hate speech and misinformation, several steps should be taken:</p><p>a) <strong>Human Oversight:</strong> While automation is crucial, human moderators and reviewers play a vital role in refining algorithms and making nuanced decisions.</p><p>b) <strong>Interdisciplinary Approach:</strong> Tackling hate speech and misinformation requires collaboration between experts in linguistics, sociology, psychology, computer science, and ethics.</p><p>c) <strong>Data Collaboration: </strong>Public-private partnerships that bring together tech companies, researchers, policymakers, and civil society can foster data sharing for training robust algorithms.</p><p>d) <strong>Explainable AI:</strong> Developing algorithms that provide transparent explanations for their decisions can aid in addressing bias concerns and promoting accountability.</p><p><strong>Conclusion:</strong></p><p>Big Data and Machine Learning offer a potent combination to tackle the pervasive issues of hate speech and misinformation online. The evolution of disrupted technologies in this field is a witness to human innovation and adaptability. While challenges remain, the strides made in automating content moderation, sentiment analysis, network analysis, and deepfake detection are promising. By embracing interdisciplinary collaboration, transparency, and a commitment to ethical use, we can look forward to a safer and more inclusive digital landscape.</p><p><strong>References:</strong></p><p>1. Jahan, M. S., &amp; Oussalah, M. (2023). A systematic review of Hate Speech automatic detection using Natural Language Processing. Neurocomputing, 126232.</p><p>2. Saleem, H., Kashif, M., &amp; Shah, M. A. (2021). A survey of hate speech detection using natural language processing. Human-centric Computing and Information Sciences, 11(1), 1–24.</p><p>3. Zeerak Waseem, Dirga Kumar Lamichhane. (2020). “Understanding Abusive Language Detection: A Survey.” ACM Computing Surveys, 53(6), 1–30.</p><p>4. Zannettou, S., Caulfield, T., &amp; Blackburn, J. (2019). The web of false information: Rumors, fake news, hoaxes, clickbait, and various other shenanigans. Journal of Data and Information Quality (JDIQ), 11(4), 1–37.</p><p>5. Davidson, T., Warmsley, D., Macy, M., &amp; Weber, I. (2017). Automated hate speech detection and the problem of offensive language. arXiv preprint arXiv:1703.04009.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e0ac367ccbc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Pig: Illustrate Basic Operators with Example]]></title>
            <link>https://aws.plainenglish.io/apache-pig-illustrate-basic-operators-with-example-9d485f6415d9?source=rss-11777614c8cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d485f6415d9</guid>
            <category><![CDATA[big-data-analytics]]></category>
            <category><![CDATA[apache]]></category>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Dr. Virendra Kumar Shrivastava]]></dc:creator>
            <pubDate>Fri, 22 Jul 2022 06:43:09 GMT</pubDate>
            <atom:updated>2022-08-02T16:44:23.842Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*NjLoy5dgJte-7Qd7ttg6MQ.jpeg" /></figure><p>Hey guys,</p><p>In this article, I will briefly discuss how to start exploring Apache Pig for data analysis. In the previous post “<a href="https://profvkshrivastava.medium.com/how-to-install-apache-pig-on-a-hadoop-cluster-on-ubuntu-948412465948">How to install Apache Pig on a Hadoop cluster on Ubuntu</a>”, I discussed what Apache Pig is and explained how to install Apache Pig on a Hadoop cluster. So, I am sure now you are ready to explore Pig to make your hand dirty. In this post, I will briefly introduce the basic operations that you can perform on a Pig.</p><p><strong>So let’s begin…</strong></p><p>First of all, you need to run Pig. You can launch Pig in either local mode or MapReduce mode. Apache Pig Grunt is an interactive shell that lets users interface with HDFS and local file systems and allows users to interactively enter into Pig Latin.</p><p><strong>A.</strong> <strong>Local Mode</strong></p><p>If you want to run Pig in local mode.</p><p>$ pig -x local</p><p>You may now interact with your local filesystem via the Grunt shell. However, if you omit the -x local option and your cluster configuration is configured in PIG CLASSPATH, you will be placed in a Grunt shell that will communicate with your cluster’s HDFS if you do.</p><p><strong>B.</strong> <strong>MapReduce mode</strong></p><p>To run Pig in MapReduce mode, run the Hadoop cluster to read/write files from the HDFS cluster. By default, pig runs in MapReduce mode.</p><p>$ pig</p><p>Now, I will demonstrate some of the important operations you can perform on Pig.</p><p><strong>1.</strong> <strong>LOAD a data file into Pig</strong></p><p>You can read a data file like a CSV file into Pig. It reads data file from the file system. For example,</p><p>grunt&gt; record = LOAD ‘/home/hdoop/Downloads/data1.csv’ using PigStorage(‘,’) as (id.int, name:chararray, grade:int);</p><p><strong>2.</strong> <strong>DESCRIBE</strong></p><p>Describe returns the schema of a relation. For example</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*vPPDvFJAHgqccGpbzU7VcA.jpeg" /></figure><p><strong>3.</strong> <strong>DUMP</strong></p><p>It is used to display results. For example,</p><p>grunt&gt; dump record;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/293/1*eoxfqPyFRlwxRibveb82gQ.jpeg" /></figure><p><strong>4.</strong> <strong>STORE</strong></p><p>It is used to store/save results.</p><p>grunt&gt; STORE record INTO ‘myresult.txt’</p><p><strong>5.</strong> <strong>FILTER</strong></p><p>It is used to select tuples from a relation that satisfies the specified conditions. For example,</p><p>grunt&gt; record = LOAD ‘/home/hdoop/Downloads/data1.csv’ using PigStorage(‘,’) as (id.int, name:chararray, grade:int);</p><p>grunt&gt; grade_5 = FILTER record BY grade&gt;5;</p><p>grunt&gt; DUMP grade_5;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*zzuJsC5JBovg9oEhXnS4XA.jpeg" /></figure><p><strong>6.</strong> <strong>FOREACH</strong></p><p>You can use the foreach operator when you want to perform data transformation on columns of the dataset. For example,</p><p>grunt&gt; record = LOAD ‘/home/hdoop/Downloads/data1.csv’ using PigStorage(‘,’) as (id.int, name:chararray, grade:int);</p><p>grunt&gt; grade_per = FOREACH record GENERATE grade*10.0;</p><p>grunt&gt; DUMP grade_per;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*64GWKwfw73xIZxoLU_VZ8A.jpeg" /></figure><p><strong>7.</strong> <strong>GROUP</strong></p><p>It is used to group data on the specified column. For example, you can group students’ data based on their grades,</p><p>grunt&gt; record = LOAD ‘/home/hdoop/Downloads/data1.csv’ using PigStorage(‘,’) as (id.int, name:chararray, grade:int);</p><p>grunt&gt; grade_grp = GROUP record BY grade;</p><p>grunt&gt; DUMP grade_grp;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*t2IgdJu7z1E9c3_O0Eimvw.jpeg" /></figure><p><strong>8.</strong> <strong>ORDER BY</strong></p><p>It is used to sort a relation based on the specific column (ascending or descending order) values.</p><p>grunt&gt; record = LOAD ‘/home/hdoop/Downloads/data1.csv’ using PigStorage(‘,’) as (id.int, name:chararray, grade:int);</p><p>grunt&gt; name_ord = ORDER record BY name;</p><p>grunt&gt; DUMP name_ord;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*oJa02qAN429yiRBj2jcIAQ.jpeg" /></figure><p><strong>9.</strong> <strong>DISTINCT</strong></p><p>Distinct is a very useful operator. It is used to remove duplicate tuples and find the distinct values. For example, you want distinct values of the name column in the relation.</p><p>grunt&gt; record = LOAD ‘/home/hdoop/Downloads/data1.csv’ using PigStorage(‘,’) as (id.int, name:chararray, grade:int);</p><p>grunt&gt; name_1 = FOREACH record GENERATE name;</p><p>grunt&gt; name_dis = DISTINCT name;</p><p>grunt&gt; DUMP name_dis,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*q0phQn4dMrJH_7M-cjTd_w.jpeg" /></figure><p><strong>10.</strong> <strong>LIMIT</strong></p><p>It is used to restrict the number of output rows.</p><p>grunt&gt; record = LOAD ‘/home/hdoop/Downloads/data1.csv’ using PigStorage(‘,’) as (id.int, name:chararray, grade:int);</p><p>grunt&gt; rec_5 = LIMIT record 5;</p><p>grunt&gt; DUMP rec_5;</p><p><strong>11.</strong> <strong>JOIN</strong></p><p>The join operator is used to join two or more relations on the common column/ field. It performs inner join. For example, you can join two relations namely employee and dept based on their values contained in empid field. The schema of relations is given below:</p><p>employee(empid:int, empname:chararray, subject:chararray)</p><p>dept(deptid:int, deptname:chararray, empid:int)</p><p>grunt&gt; emp = LOAD ‘/home/hdoop/Downloads/employee.csv’ using PigStorage(‘,’) as (empid.int, empname:chararray, subject:chararray);</p><p>grunt&gt; dept = LOAD ‘/home/hdoop/Downloads/dept.csv’ using PigStorage(‘,’) as (deptpid:int, depname:chararray, empid:int);</p><p>grunt&gt; new = JOIN employee BY empid, dept BY empid;</p><p>grunt&gt; DUMP new;</p><p><strong>12.</strong> <strong>UNION</strong></p><p>A Union operator is used to merge the contents of the two relations. For example, you can merge the contents of two relations employee and dept.</p><p>grunt&gt; new = UNION employee, empid;</p><p>grunt&gt; DUMP new;</p><p><strong>13.</strong> <strong>SPLIT</strong></p><p>You can partition a relation into two or more relations. For example, you can partition students’ records based on their grades.</p><p>grunt&gt; record = LOAD ‘/home/hdoop/Downloads/data1.csv’ using PigStorage(‘,’) as (id.int, name:chararray, grade:int);</p><p>grunt&gt; SPLIT record INTO grade_b IF grade &lt;=b, grade_a IF grade &gt;5;</p><p>grunt&gt; DUMP grade_a;</p><p>grunt&gt; DUMP grade_b;</p><p><strong>14.</strong> <strong>SAMPLE</strong></p><p>A sample operator is used to create a sample of data based on a specified size.</p><p>grunt&gt; record = LOAD ‘/home/hdoop/Downloads/data1.csv’ using PigStorage(‘,’) as (id.int, name:chararray, grade:int);</p><p>grunt&gt; sample1 = SAMPLE record 0.01;</p><p>grunt&gt; DUMP sample1;</p><p><strong>15</strong>. <strong>AVG</strong></p><p>It is an evaluation function used to compute the average numeric values in a single column bag. For example, you can calculate the average grade of the class</p><p>grunt&gt; record = LOAD ‘/home/hdoop/Downloads/data2.csv’ using PigStorage(‘,’) as (id.int, name:chararray, subject:chararray,grade:int);</p><p>grunt&gt; grade_grp = GROUP record BY name;</p><p>grunt&gt; grade_avg = FOREACH grade_grp GENERATE record.name, record.subject, AVG(record.grade);</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*ejfKfe1ZOqYP8byyFzh8hA.jpeg" /></figure><p><strong>16.</strong> <strong>MAX</strong></p><p>It is an evaluation function and is used to find the maximum value in a single-column bag. For example, you can find the highest grade in the class.</p><p>grunt&gt; record = LOAD ‘/home/hdoop/Downloads/data2.csv’ using PigStorage(‘,’) as (id.int, name:chararray, subject:chararray,grade:int);</p><p>grunt&gt; grade_grp = GROUP record BY name;</p><p>grunt&gt; grade_avg = FOREACH grade_grp GENERATE record.name, record.subject, MAX(record.grade);</p><p><strong>17</strong>. <strong>COUNT</strong></p><p>It is an evaluation function and is used to count the number of elements in a bag. For example, the number of tuples in a bag.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*N1PAEebpmYs_xt4-ecViYg.jpeg" /></figure><p>grunt&gt; DUMP record;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*wftizlwYsyQm21vM7HPTwA.jpeg" /></figure><p><strong>18. Filter</strong></p><p>It is used to select the required tuples from a relation based on a specified condition.</p><p>grunt&gt; gpa = FILTER record BY grade &gt;6;</p><p>grunt&gt; DUMP gpa;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*2KCg3lHXrQPJOO5RAid3tw.jpeg" /></figure><p><strong>19. FOREACH</strong></p><p>It is used to generate specified data transformations based on the specified column data.</p><p>grunt&gt; names = FOREACH record generate UPPER(name);</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*pTBpbBjb456wKVxHLJ0xFg.jpeg" /></figure><p><strong>20. Group By</strong></p><p>It is used to group the data in one or more relations. It is used to collect the data having the same key.</p><p>grunt&gt; grades = GROUP record BY grade;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*7lvyuFcdUxC_9IGCHt-G4g.jpeg" /></figure><h4><strong>Conclusion</strong></h4><p>This article has discussed and demonstrated the basic operations you can perform on Apache Pig in a Hadoop cluster. It is super simple to operate on a data file using Apache Pig. If you are a novice user, this blog can be a stepping stone for you.</p><p>On winding up notes, feel free to share your comments. Your likes and comments will help me to present the contents in a better way.</p><p><em>More content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><em>. Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a>, <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><strong><em> </em></strong><em>and </em><a href="https://discord.gg/GtDtUAvyhW"><strong><em>Discord</em></strong></a><strong><em>.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d485f6415d9" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/apache-pig-illustrate-basic-operators-with-example-9d485f6415d9">Apache Pig: Illustrate Basic Operators with Example</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to install Apache Pig on a Hadoop cluster on Ubuntu]]></title>
            <link>https://aws.plainenglish.io/how-to-install-apache-pig-on-a-hadoop-cluster-on-ubuntu-948412465948?source=rss-11777614c8cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/948412465948</guid>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[apache]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[big-data-analytics]]></category>
            <dc:creator><![CDATA[Dr. Virendra Kumar Shrivastava]]></dc:creator>
            <pubDate>Wed, 22 Jun 2022 12:18:38 GMT</pubDate>
            <atom:updated>2022-07-01T07:29:28.911Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BzAJ9B4jhXgh7uBb9W2fSw.jpeg" /></figure><p>Hey guys,</p><p>In this article, I will briefly discuss what is Apache Pig and explain how to install Apache Pig in on a Hadoop cluster. I will provide step-by-step installation instructions to set up Pig on Ubuntu 16.04. So, if you have the curiosity to know about Pig and its installation process, this article can be a steppingstone for your Apache Pig journey. The agenda for discussion is as follows:</p><p><strong>What is Apache Pig?</strong></p><p><strong>Step-by-step Apache Pig installation</strong></p><p>So, let us begin…</p><h4><strong>What is Apache Pig?</strong></h4><p>Pig is a scripting platform that is designed to process and analyze large datasets which were developed in late 2006 by Yahoo researchers. Later, Pig became an Apache open-source project. Pig enables users/analysts to express complicated transformations in a quite simple manner. Pig is a scripting tool that runs on Hadoop clusters. Apache Pig interacts directly with data stored in the Hadoop cluster. Besides Java, Pig is another language in which MapReduce programs can be written. Apache Pig converts Pig scripts into MapReduce tasks, which may then be used with Hadoop YARN (Yet Another Resource Negotiator) to retrieve HDFS (Hadoop Distributed File System) datasets. Pig has two components:</p><p>1. Pig Latin script language</p><p>2. A runtime engine</p><p>To use Apache Pig, first set up a Hadoop cluster and then install Pig on the Hadoop cluster.</p><h4>Step-by-step installation of Apache Pig on Hadoop cluster on Ubuntu</h4><p><strong>Pre-requisite:</strong></p><p>· Ubuntu 16.04 or higher version running (I have installed Ubuntu on Oracle VM (Virtual Machine) VirtualBox),</p><p>· Run Hadoop on ubuntu (I have installed Hadoop 3.2.1 on Ubuntu 16.04). You may refer to my blog “How to install Hadoop installation” click <a href="https://medium.com/mlearning-ai/installation-of-apache-hadoop-3-2-1-on-ubuntu-67073ce208d7">here</a> for Hadoop installation).</p><p><strong>Pig installation steps</strong></p><p><strong>Step 1: </strong>Login into Ubuntu</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*QqFZUSE5MkbWzElQrhmvBA.jpeg" /></figure><p><strong>Step 2</strong>: Go to <a href="https://pig.apache.org/releases.html">https://pig.apache.org/releases.html</a> and copy the path of the latest version of pig that you want to install. Run the following comment to download Apache Pig in Ubuntu:</p><pre>$ wget <a href="https://dlcdn.apache.org/pig/pig-0.16.0/pig-0.16.0.tar.gz">https://dlcdn.apache.org/pig/pig-0.16.0/pig-0.16.0.tar.gz</a></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*fUNqn4VGW-CAHwPa74Kc2Q.jpeg" /></figure><p><strong>Step 3</strong>: To untar pig-0.16.0.tar.gz file run the following command:</p><pre>$ tar xvzf pig-0.16.0.tar.gz</pre><p><strong>Step 4: </strong>To create a pig folder and move pig-0.16.0 to the pig folder, execute the following command:</p><pre>$ sudo mv /home/hdoop/pig-0.16.0 /home/hdoop/pig</pre><p><strong>Step 5: </strong>Now open the .bashrc file to edit the path and variables/settings for pig. Run the following command:</p><pre>$ sudo nano .bashrc</pre><p>Add the below given to .bashrc file at the end and save the file.</p><pre>#PIG settings</pre><pre>export PIG_HOME=/home/hdoop/pig</pre><pre>export PATH=$PATH:$PIG_HOME/bin</pre><pre>export PIG_CLASSPATH=$PIG_HOME/conf:$HADOOP_INSTALL/etc/hadoop/</pre><pre>export PIG_CONF_DIR=$PIG_HOME/conf</pre><pre>export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64</pre><pre>export PIG_CLASSPATH=$PIG_CONF_DIR:$PATH</pre><pre>#PIG setting ends</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*9-868M6B7bhc1Lf4LgijpQ.jpeg" /></figure><p><strong>Step 6: </strong>Run the following command to make the changes effective in the .bashrc file:</p><pre>$ source .bashrc</pre><p><strong>Step 7:</strong> To start all Hadoop daemons, navigate to the hadoop-3.2.1/sbin folder and run the following commands:</p><pre>$ ./start-dfs.sh</pre><pre>$ ./start-yarn</pre><pre>$ jps</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*WiQPJVZ7HsCMfTydMaiIXA.jpeg" /></figure><p><strong>Step 8: </strong>Now you can launch pig by executing the following command:</p><pre>$ pig</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*ivot8eCtOUYdvtsYqw1LfQ.jpeg" /></figure><p><strong>Step 9:</strong> Now you are in pig and can perform your desired tasks on pig. You can come out of the pig by the quit command:</p><pre>&gt; quit;</pre><h4><strong>Conclusion</strong></h4><p>In this article, I have introduced what is Apache Pig and demonstrated how to install Pig on the Hadoop cluster in Ubuntu. So, friends, it is super simple to install Pig. Now, you can explore and perform desired operations on it.</p><p>On wrapping up notes, feel free to share your comments. Your claps and comments will help me to present content in a better way. See you next week.</p><p><em>More content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><em>. Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a><em> and </em><a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><em>. Check out our </em><a href="https://discord.gg/GtDtUAvyhW"><strong><em>Community Discord</em></strong></a><em> and join our </em><a href="https://inplainenglish.pallet.com/talent/welcome"><strong><em>Talent Collective</em></strong></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=948412465948" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/how-to-install-apache-pig-on-a-hadoop-cluster-on-ubuntu-948412465948">How to install Apache Pig on a Hadoop cluster on Ubuntu</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to install Apache HBase on Ubuntu?]]></title>
            <link>https://aws.plainenglish.io/how-to-install-apache-hbase-on-ubuntu-65a2a10cb71d?source=rss-11777614c8cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/65a2a10cb71d</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[big-data-analytics]]></category>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Dr. Virendra Kumar Shrivastava]]></dc:creator>
            <pubDate>Wed, 15 Jun 2022 08:23:46 GMT</pubDate>
            <atom:updated>2022-06-24T04:00:59.180Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*ElBplJeyCmZMzTwx7FQLjQ.jpeg" /></figure><p>Hey guys,</p><p>In this article, I will explain how to install Apache HBase in standalone mode on a Hadoop cluster. I will provide step by step instructions to set up HBase on Ubuntu 16.04. So, if you are looking to make your hands dirty on Apache HBase, this article can be a steppingstone for you. The agenda for discussion is as follows:</p><p>· What is Apache HBase?</p><p>· Step by step HBase installation</p><p>· Test HBase</p><p><strong>So, let us begin…</strong></p><h3><strong>What is Apache HBase?</strong></h3><p>Apache HBase is a distributed large data store that is open-source, and it is a NoSQL database. It offers real-time access to petabytes of data in a random, strictly consistent manner. HBase excels in dealing with huge, sparse datasets. HBase works on top of the Hadoop Distributed File System (HDFS) or Amazon S3 utilizing the Amazon Elastic MapReduce (EMR) file system and interacts easily with Apache Hadoop and the Hadoop ecosystem.</p><p>HBase does support SQL. But HBase can interact with Apache Phoenix to provide SQL-like queries over HBase tables and serves as a direct input and output to the Apache MapReduce framework for Hadoop. HBase is a non-relational database with a column-oriented design. Data is kept in distinct columns and is indexed using a unique row key. This design enables quick retrieval of individual rows and columns inside a table, as well as fast scans over individual columns. The data and queries in an HBase cluster are dispersed over all servers, allowing you to query results on petabytes of data in milliseconds.</p><p>HBase is best for storing non-relational data that can be retrieved using the HBase API (Application Programming Interface). Apache Phoenix is often used as a SQL layer on top of HBase, allowing you to insert, remove, and query data saved in HBase using familiar SQL syntax.</p><h3><strong>Step by step installation of HBase on Ubuntu</strong></h3><h3>Pre-requisite:</h3><p>· Ubuntu 16.04 or higher installed on a virtual machine.</p><p>· Hadoop installation (How to install Hadoop installation click <a href="https://medium.com/mlearning-ai/installation-of-apache-hadoop-3-2-1-on-ubuntu-67073ce208d7">here</a>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*iJSrBdbhPB7xiaTsv-03fg.jpeg" /></figure><h4>Steps for installation of Apache HBase</h4><h4>Step 1</h4><p>Download HBase: To download HBase stable version, run the following command:</p><pre>$ wget <a href="https://archive.apache.org/dist/hbase/stable/hbase-2.4.12-bin.tar.gz">https://archive.apache.org/dist/hbase/stable/hbase-2.4.12-bin.tar.gz</a></pre><h4>Step 2</h4><p>To untar the HBase file run the following command:</p><pre>$ tar xzvf <a href="https://archive.apache.org/dist/hbase/stable/hbase-2.4.12-bin.tar.gz">https://archive.apache.org/dist/hbase/stable/hbase-2.4.12-bin.tar.gz</a></pre><h4>Step 3</h4><p>Navigate to hbase-2.4.12/bin:</p><pre>$ cd hbase-2.4.12/bin</pre><p>Set Java path: To set java path for HBase environment run the following command:</p><pre>$ sudo nano conf/hbase-env.sh</pre><p>And add the java path (below given) at the end of the and save it.</p><pre>export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64</pre><h4>Step 4</h4><p>To start HBase: run the following commands:</p><pre>$ cd hbase-2.4.12/bin</pre><pre>$ ./start-hbase.sh</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*oDSN04YleoVpAZu1m2zA_A.jpeg" /></figure><p>Go to <a href="http://localhost:16010/">http://localhost:16010</a> to view the HBase web user interface.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*PKPB072R9uFQQ7kXBvFqUQ.jpeg" /></figure><h4>Step 5:</h4><p>Connect to HBase shell: Connect to the instance of HBase use the hbase shell command, located in the bin directory.</p><pre>$ ./hbase shell</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*nOPl-ZIhcOFD6WVlAdlYiA.jpeg" /></figure><h4>Step 6:</h4><p>To create a table: Use the create command to create a new table. It is mandatory to specify the table name and the ColumnFamily name.</p><pre>&gt;create ‘MyDatabase’, ‘cf’</pre><h4>Step 7:</h4><p>Use the list command to check that the table exists or not.</p><pre>&gt;list ‘MyDatabase’</pre><h4>Step 8:</h4><p>Use the describe command to see details of the table</p><pre>&gt;describe ‘MyDatabase’</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*pxMYOhV9o-uIKTAus9mn9w.jpeg" /></figure><h4>Step 9:</h4><p>To insert data into your table, use the put command:</p><pre>&gt;put ‘MyDatabase’, ‘row1’, ‘cf:a’, ‘10001’</pre><pre>&gt;put ‘MyDatabase’, ‘row2’, ‘cf:b’, ‘Virendra’</pre><pre>&gt;put ‘MyDatabase’, ‘row3’, ‘cf:c’, ‘200000’</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*oL9fiws-03FDFym-vnEXoA.jpeg" /></figure><h4>Step 10</h4><p>Use the scan command to scan the table for data.</p><pre>&gt;scan ‘MyDatabase’</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*oL9fiws-03FDFym-vnEXoA.jpeg" /></figure><h4>Step 11</h4><p>To get a single row of data at a time, use the get command.</p><pre>&gt;get ‘MyDatabase’, ‘row1’</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*PZA58ra0runkPYRnoqEmcQ.jpeg" /></figure><h4>Step 12:</h4><p>Disable a table — If you want to delete a table or change its settings, you need to disable the table first, so use the disable command for this purpose. You can re-enable the table using the enable command.</p><pre>&gt;disable ‘MyDatabase’</pre><pre>&gt;enable ‘MyDatabase’</pre><h4>Step 13:</h4><p>Drop the table: To drop (delete) a table, use the drop command.</p><pre>&gt;drop ‘MyDatabase’</pre><h4>Step 14</h4><p>Exit from HBase Shell: To exit from the HBase Shell and disconnect from your cluster, use the quit command. However, HBase is still running in the background</p><h4>Step 15</h4><p>Stop HBase: To stop all HBase daemons, you can run ./stop-hbase.sh script.</p><pre>$ ./stop-hbase.sh</pre><h3>Conclusion</h3><p>In this article, I have presented step by step process to set up Apache HBase on Ubuntu 16.04. It is super simple to install HBase on Hadoop cluster. Hope you would successfully set up Apache HBase on your system.</p><p>On wrapping up notes, feel free to share your comments. Your likes and comments will help me to present contents in better way. See you next week.</p><p><em>More content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><em>. Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a><em> and </em><a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><em>. Check out our </em><a href="https://discord.gg/GtDtUAvyhW"><strong><em>Community Discord</em></strong></a><em> and join our </em><a href="https://inplainenglish.pallet.com/talent/welcome"><strong><em>Talent Collective</em></strong></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=65a2a10cb71d" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/how-to-install-apache-hbase-on-ubuntu-65a2a10cb71d">How to install Apache HBase on Ubuntu?</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Spark VS Apache Hadoop]]></title>
            <link>https://profvkshrivastava.medium.com/apache-spark-vs-apache-hadoop-49aadd9df9b0?source=rss-11777614c8cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/49aadd9df9b0</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[big-data-analytics]]></category>
            <dc:creator><![CDATA[Dr. Virendra Kumar Shrivastava]]></dc:creator>
            <pubDate>Wed, 25 May 2022 06:36:01 GMT</pubDate>
            <atom:updated>2022-05-25T10:00:51.308Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>(Which is the best fit for you?)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/1*RuncHzo19O2nmub-aUeyhA.jpeg" /></figure><p>Hey guys,</p><p>In this article I am going to discuss Apache Spark vs Apache Hadoop and when to choose what? Big data analysis has become the secret to success in many businesses. Businesses are constantly attempting to get more insights from every piece of data they collect, which has resulted in a desire to use big data frameworks that can provide the most significant information for the business. Several organizations are switching to Apache Spark and Hadoop to tackle these problems but choosing the correct one can sometimes a bit difficult. I have authored this article to help you in making this decision super simple. Presently, spark and Hadoop are the two most popular big data processing frameworks in the market. This article will explore Apache Hadoop versus Apache Spark and will help you to decide which one is the right choice for your business. So, let us dive deep into the discussion. The agenda for the discussion is as follows:</p><p>· What is Apache Hadoop?</p><p>· What is Apache Spark?</p><p>· Spark VS Hadoop</p><p>· Which framework is the best fit for you?</p><h3><strong>What is Apache Hadoop?</strong></h3><p>Hadoop is a sophisticated platform that employs the MapReduce architecture to divide data into blocks, which are then distributed over a cluster to be processed across machines and combined. Hadoop is a notable framework for its fault tolerance since it replicates data across clusters and then gets data from healthy sets to recreate data that has been lost or corrupted due to hardware failure. It is based on on-disc processing and is the most suitable for batch processing.</p><h3><strong>What is Apache Spark?</strong></h3><p>Spark is a flexible framework that processes massive volumes of data by distributing tasks over several nodes, but it does so much quicker than Hadoop since it leverages in memory (RAM) data processing to do so. The Spark engine is famous as the Swiss army knife of frameworks for one simple reason. Use of Spark for batch processing and real-time stream processing, is increasing fast.</p><h3><strong>Spark VS Hadoop</strong></h3><p>I will explore spark vs Hadoop on the following parameters:</p><p>· <strong>Performance of the framework</strong></p><p>· <strong>Implementation cost</strong></p><p>· <strong>Easy to maintain and use</strong></p><h3><strong>Performance of the framework</strong></h3><p>Since it is not concerned with input-output issues every time it executes a component of a MapReduce operation, Spark runs 100 times faster in memory and ten times faster on disc. However, you need to have more RAM for in memory processing capabilities which may incur more cost. Hoop utilizes low-cost machines or commodity hardware which lacks the cyclical relationship between MapReduce phases, but Spark’s DAGs optimize between degrees better. If speed is not a primary need, choose MapReduce since it can handle massive data without consuming too much memory. Hadoop stores data before processing it. Its primary intention was to process data obtained from a variety of sources.</p><h3><strong>Implementation cost</strong></h3><p>Both platforms are free to use and open source. However, Spark requires a considerable amount of RAM to run, whereas Hadoop requires more disc capacity to function. In the near run, this makes Hadoop appear less expensive. However, when optimized for computation time, Spark completes the identical jobs far quicker than Hadoop. This is crucial since you pay for computers to browse the cloud.</p><h3><strong>Easy to maintain and use</strong></h3><p>Hadoop MapReduce is recognized to be more difficult to programme and lacks an interactive mode. With its pre-built APIs (Application Programming Interface) for Java, Scala, and Python, Spark features an interactive interface and significantly simpler building blocks, making it easier to develop user-defined functions. So, you can choose one of the platforms as per your expertise and business requirements which one is the best fit for you.</p><h3><strong>Which framework is the best fit for you?</strong></h3><p>These frameworks are widely used in combination and operate quite well when used in tandem. However, there are definite characteristics in which one is superior to the other. Spark is capable of efficiently handling real-time and bath data processing. Hadoop MapReduce is great for batch processing, but it falls short of the mark to real-time processing. Instead of separating your jobs over many platforms like in Hadoop, Spark deployment provides a one-size-fits-all platform.</p><p>If you are <strong>contemplating whether Apache Spark</strong> is right choice for you, consider the following scenarios:</p><p>· Analysis in real-time</p><p>· Instantaneous outcomes from the analysis</p><p>· Repetitive operations</p><p>· Algorithms for Machine Learning</p><p>Hadoop is better suited for you, consider the following scenarios:</p><p>· Archival data analysis</p><p>· Operation on commodity hardware</p><p>· Data analysis is not hurried.</p><p>· Linear data processing</p><h3>Conclusion</h3><p>Although, I have discussed the differences between Hadoop and Spark, as well as why Spark is quicker than Hadoop, a lot will rely on your project’s needs and the expertise and knowledge you have. At first sight, it may appear like Spark is a newer, better version of Hadoop, but that is not the case, and it really is a promising idea to do in-depth analysis based on your requirement and expertise into Spark versus Hadoop to see which is a good fit for you. In terms of skill, if your team is unfamiliar with Hadoop or has no prior experience with it, it may be tough to learn due to its significant reliance on Java, which is difficult to programme with. If that is the case, Spark is a superior option since it is easy to code and has an interactive capability that allows you to get immediate response after running commands.</p><p>Although I compared Hadoop and Spark, a great circumstance is when you do not really have to pick between the two and can utilize both. You may have observed that these frameworks have a healthy connection. Spark is Hadoop-compatible and works well with Hadoop’s distributed file system. As a result, if you choose one, think about combining the two for a more thorough study. You will be able to receive faster insights, save expenses, and avoid duplication by combining the two frameworks. Many large businesses, such as Amazon, Yahoo, and eBay, are already employing the two combinations, and it worked well for them based on their degree of development.</p><p>On wrapping up notes, feel free to share your comments. Your claps and comments will help me to present contents in better way. See you next week.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=49aadd9df9b0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why trade-off between Bias and Variance is important in Machine Learning models?]]></title>
            <link>https://aws.plainenglish.io/why-trade-off-between-bias-and-variance-is-important-in-machine-learning-models-32190b689159?source=rss-11777614c8cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/32190b689159</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[big-data-analytics]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Dr. Virendra Kumar Shrivastava]]></dc:creator>
            <pubDate>Wed, 18 May 2022 06:12:35 GMT</pubDate>
            <atom:updated>2022-05-19T21:24:32.117Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*GwJxvyYdla669URlbFuVCw.jpeg" /><figcaption><a href="http://www.iskalo.com/insights/what-are-zoning-variances-and-how-do-you-get-one/">www.iskalo.com/insights/what-are-zoning-variances-and-how-do-you-get-one/</a></figcaption></figure><p><strong>H</strong>ey guys,</p><p>In Machine Leaning (ML) based predictive modelling bias and variance plays vital roles. High bias model leads to under fitting whereas high variance starts to over fit of the model. However, ideally a ML model should achieve low bias and low variance which is impossible task. So, a good model should achieve trade-off between bias and variance. Bias and variance terms used to be very confusing for the people. In this article I will explore what is the difference between bias and variance? What is the cause? What are the ways of dealing with the trade-off?</p><p>So, let us dive into deep in this discussion.</p><p>When discussing ML Prediction, it is critical to distinguish between two sorts of prediction mistakes (bias and variance). The capacity of a model to minimize bias and variation is a tradeoff. Bias emerges when an algorithm has limited flexibility to discover the real indicator from the dataset.</p><h4>What is bias in machine learning model?</h4><p>Bias is a mistake caused by incorrect assumptions in the learning process of a ML model. Underfitting is when an algorithm misses the crucial relationships between features and target outputs due to high bias. Thus, the accuracy of our predictions is known as bias.</p><p>A high bias indicates that the prediction will be incorrect. Bias may be conceived of as having a bias toward other individuals. You are more prone to make incorrect assumptions about somebody if you are very prejudiced. An unfair dynamic is created by an oversimplified mindset. You describe them as biased. Bias is indeed the algorithm’s propensity to learn the wrong thing repeatedly by not taking all the key variables into account and it leads to underfitting.</p><p>The model’s simplifying assumptions simplify the target function, making it easier to estimate.</p><p><strong>Low Bias</strong> — Suggests that there are less assumptions regarding the target function’s shape.</p><p><strong>High-Bias</strong> — Suggests making additional assumptions about the target function’s shape.</p><h4>What is variance in machine learning model?</h4><p>Variance is the amount by which the target function’s estimate changes when different training data is used. The discrepancy in the predictions of many models is known as variance. Unlike the comparison before, we are putting complex models into action. As a result, the model may capture any noise in the dataset. When we utilize sophisticated models that potentially overfit our training sets, we get high variance.</p><p><strong>Low Variance</strong> — Suggests that tiny changes to the training dataset will result in slight changes to the target function estimate.</p><p><strong>High Variance </strong>— Suggests that changes to the training dataset cause considerable variations in the target function estimate.</p><h4>How does the Bias Variance tradeoff work in ML Model?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/489/1*f8EcyfGY0dV0uQYZNFNagw.jpeg" /></figure><p>Any ML model should aim for low variance and low bias. High bias leads to underfit models, whereas high variance leads to overfit models (refer diagram above). Both should be low in ideal conditions, but obtaining one comes at the expense of growing the other. The bias-variance tradeoff is what it is called. But there is no quantitative method for finding this balanced error point when both are at their lowest. Instead, you will need to use accuracy measurements and alter the complexity of your model until you discover the iteration that minimizes total error.</p><p>For illustration, a sophisticated model may show that people’s names are a good predictor of our hypothesis. Names, on the other hand, are completely random and should not be used to forecast anything. People with the name “Vijay” have been shown to be more likely to be criminals in one dataset. People with the name “Vijay” on the other hand, can imply that they are likely to be gentleman in another dataset. As a result, names should not be utilized as a predictor. If you use a simplistic model, you can get to the conclusion that every “Vijay” is a fantastic person. This is an issue of High Bias and Low Variance. Your data is skewed toward persons named Vijay. As a result, many predictions will be same, because you assume individuals named “Vijay” behave in a specific way. You try to rebuild the model. The model, however, is excessively complex. Diverse groups of people get different outcomes from your model. As a result, “Vijay” has the potential to be a gentleman, a criminal, a scholar and a thieve. You should achieve balance! The good news is that cross-validation allows you to train on many datasets and average the results. However, bias and variance cannot be minimized.</p><h4>Conclusion</h4><p>When a model is too basic or underfitting, bias results. Bias is the difference between what we expect and what we get. Regardless of how many data points you utilize, the model will consistently miss the mark with predictions. If we modify the data set several times, the fit does not vary significantly.</p><p>When a model is too sophisticated or overfitting, variance develops. In other words, it is the amount that a different data set will modify the goal estimate. The technique is quite sensitive to the data sets used in the model training. Noise in the data will be misinterpreted by the models as signals. Thus, to create effective predictions, you must strike a balance between bias and variance that reduces overall error and making this tradeoff is crucial for ML models.</p><p>On wrapping up notes, feel free to share your comments. Your claps and comments will surely help me to present contents in a better way. See you next week.</p><p><em>More content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><em>. Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a><em> and </em><a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><em>. Join our </em><a href="https://discord.gg/GtDtUAvyhW"><strong><em>community Discord</em></strong></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=32190b689159" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/why-trade-off-between-bias-and-variance-is-important-in-machine-learning-models-32190b689159">Why trade-off between Bias and Variance is important in Machine Learning models?</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Dimensionality Reduction Is Crucial in Machine Learning Models?]]></title>
            <link>https://profvkshrivastava.medium.com/why-dimensionality-reduction-is-crucial-in-machine-learning-models-ca766df37617?source=rss-11777614c8cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/ca766df37617</guid>
            <category><![CDATA[big-data-analytics]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[dimensionality-reduction]]></category>
            <dc:creator><![CDATA[Dr. Virendra Kumar Shrivastava]]></dc:creator>
            <pubDate>Wed, 11 May 2022 06:08:07 GMT</pubDate>
            <atom:updated>2022-05-11T06:08:07.572Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*9KgcSEBv5QKOa0daerTwvg.png" /></figure><p>Hey guys,</p><p>High dimensionality throws challenges in performance accuracy of machine learning models. In this article, I explored the importance of dimensionality reduction in building machine learning models and discussed why high dimensionality impacts the performance of machine learning models / statistical models. Here, I touched upon the following questions:</p><ul><li>What is high dimensionality?</li><li>What are the difficulties caused by high dimensionality?</li><li>What is curse of dimensionality?</li><li>What are the benefits of dimensionality reduction?</li></ul><h3><strong>What is high dimensionality?</strong></h3><p>The number of features or input variables that contribute to the prediction process to predict target or output in machine learning model is called dimensionality. However, it is observed that all feature variables do not have the same significance / contribution in the output. Machine learning models such as regression and classification are built on training dataset. The dimension of the dataset plays a crucial role in the performance of the model. There are machine learning models that use weight factors to predict the output for unknown data. The dimension or feature variables show significant impact in the accuracy of the model. Therefore, before training a machine learning model, it is necessary to identify relevant features that contribute to output / target and this process is called dimensionality reduction. Dimensionality reduction is essential because of the following reasons:</p><ul><li><strong>To reduce complexity of model</strong></li></ul><p>More number of features / high dimensionality will lead to build a complex model specially when there is a high correlation exits in the feature variables. Thus, it is useful to select the right set of features to overcome this challenge.</p><ul><li><strong>To prevent overfitting</strong></li></ul><p>A dataset with high dimensionality sometimes may lead to overfitting of a machine learning model because the model captures key features and noise as well. Thus, the model performs well during training of the model, but performance degrades on testing on unknown data.</p><ul><li><strong>To achieve computational efficiency</strong></li></ul><p>A machine learning model with low dimensionality takes less time in training of the model because it takes less time in computations.</p><p>The prime objective of dimensionality reduction is to find a low-dimensional representation of the dataset that preserves as much information as possible. In other words, dimensionality reduction is the means of lowering the number of arbitrary variables under consideration, by obtaining a set of key variables.</p><h3><strong>What are the difficulties caused by high dimensionality?</strong></h3><p>High dimensionality throws several challenges and adversely impacts on the performance accuracy of the model. Some of the notable points are:</p><ul><li>It may lead to high computational cost.</li><li>It may cause overfitting during training of the model that means the model performs well during training, but performance accuracy degrades during testing on new data samples.</li><li>The higher the number of features variables throws more difficulties to visualize the training set.</li><li>High dimensionality may also have more chances to high correlation in data.</li></ul><h3><strong>What is curse of dimensionality?</strong></h3><p>Dimensionality indicates the number of variables/features in the dataset that are used for training the machine learning model. When the number of variables/ features is large compared to the number of observations in the dataset which drastically degrades the performance of the model is called curse of dimensionality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/894/1*5ECWxAD2R5T7h6zvXvnlLw.png" /></figure><p>For example, if I build a linear regression model (a machine learning model) to predict price of house, I need to take into consideration the noteworthy features only such as area, number of bedrooms, number of bathrooms, furnishing, age of house, locality etc. Selection of nonsignificant feature such as loan amount, income of the owner, number of flats in the society etc. will make adverse impact on the performance accuracy of the regression model.</p><h3><strong>What are the benefits of dimensionality reduction?</strong></h3><p>There are several benefits of dimensionality reduction. Some of the most important benefits are:</p><ul><li>It eliminates insignificant features from the dataset which improves the performance of the model because irrelevant features or noise leaves an adverse impact on the performance accuracy of machine leaning model.</li><li>It is useful to lower model training time and lower the data storage requirement.</li><li>It prevents from curse of dimensionality</li><li>It eliminates multi collinearity which results in better performance of the model.</li></ul><h3><strong>Conclusion</strong></h3><p>In this article, I “explored what is high dimensionality?” And touched upon some of the important aspects of high dimensionality such as what are the difficulties caused by high dimensionality? what is curse of dimensionality? and what are the benefits of dimensionality reduction? I am sure that I will create a fundamental understanding of high dimensionality.</p><p>On wrapping up notes, feel free to share your comments and feedback. Your clapping and comments will surely help me to present contents in a better way. See you next week.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ca766df37617" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is Partitioning vs Bucketing in Apache Hive? (Partitioning vs Bucketing)]]></title>
            <link>https://python.plainenglish.io/what-is-partitioning-vs-bucketing-in-apache-hive-partitioning-vs-bucketing-88d31578bb34?source=rss-11777614c8cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/88d31578bb34</guid>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[big-data-analytics]]></category>
            <category><![CDATA[hive]]></category>
            <dc:creator><![CDATA[Dr. Virendra Kumar Shrivastava]]></dc:creator>
            <pubDate>Wed, 04 May 2022 10:22:08 GMT</pubDate>
            <atom:updated>2022-06-17T04:00:46.574Z</atom:updated>
            <content:encoded><![CDATA[<h4>Exploring partitioning vs clustering in the Hive table, and understanding when to do partitioning and when to do clustering</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/949/1*gGGIaiRo60CQMwT3JKSSpw.jpeg" /></figure><p>Hey guys,</p><p>Apache Hive is one of the popular data warehouses in distributed cluster environments. Apache hive is used to store massive amounts of data and it can be processed in a fast, parallel, and efficient manner in HDFS (Hadoop Distributed File System) environment. To improve the access time of Hive queries, Hive tables can be stored in diverse ways. Hive tables can be stored as partitions and bucketing. In this article, I am going to explore partitioning vs clustering in the Hive table, and I will discuss when to do partitioning and when to do clustering?</p><p>Let’s dive deeper to understand this detail.</p><p>A hive is an excellent tool for performing queries on large datasets. There are some instances where we want to filter the data on specific column values. In such cases, partitioning and bucketing play a vital role in improving the query performance/access time in Hive. We can improve query retrieval time using partitioning and bucketing appropriately.</p><p><strong>At a conceptual level, partitioning is a technique to divide a large table (in a hive warehouse) into smaller tables based on the distinct values of a specified column (one partition for each distinct value) whereas bucketing is a way to split the data based on a hash function in a manageable table (user can specify how many buckets he/she wants)</strong>. Let us see both in detail.</p><h3><strong>What is Partitioning in Hive table?</strong></h3><p>Partitioning is a technique of managing the data load horizontally in a more manageable way by diving data into directories and sub-directories. It allows us to categorize data in a more logical manner. In hive partition is of two types:</p><p>· <strong>Static partition (managed by user</strong>): In Static Partitioning, you need to specify the partition (the segment column) in which we want to load the data. In static partitioning, the user takes care of everything from defining the partitioning columns to loading data into the partition folders/directories. For example, run the following command to create static partition and load data:</p><p>&gt; create table student_detail_static_part(id int, name string, branch string) partitioned by (gpa float) row format delimited fields terminated by ‘,’;</p><p>&gt;insert overwrite table student_detail_static_part partition (gpa = 7.0) select id,name, branch from student_detail where gpa = 7.0;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YiF3Lewoua9ps4EzNWO91g.png" /></figure><p>Now you can run the following query to fetch data from a specified partition. So, rather than scanning entre data, the query will scan data from the partition which meets the criteria set in the where clause. It will improve query execution time.</p><p>&gt;select * from student_detail_static_part where gpa =7.0;</p><p>· <strong>Dynamic Partition (managed by Hive): </strong>In dynamic partitioning, the user is required to just state the column name on which partition is to be created. The hive will automatically create a partition based on the unique values in the column on which the partition is defined while the data load operation happens. In dynamic partitioning, the hive will control the partitioning operation and will figure out unique values in the partitioning column and segregate data in the respective partitions. So, no user intervention is required in dynamic partitioning. For example,</p><p>&gt;create table student_detail_dynamic_partition(id int, name string, branch string) partitioned by (gpa float) row format delimited fields terminated by ‘,’;</p><p>&gt;set hive.exec.dynamic.partition = true;</p><p>&gt;set hive.exec.dynamic.partition.mode = nonstrict;</p><p>&gt;insert overwrite table student_detail_dynamic_partition partition(gpa) select id, name, gpa from student_detail;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l58LxkZ6RZkkv4VHw9fpmg.png" /></figure><p>As you can see above, the hive has created partitions based on the distinct values in the gpa column, and data is loaded in the respective sub-directories of the student_detail_dynamic_partition directory.</p><h3>What is bucketing in Hive?</h3><p>Bucketing is like partitioning with some differences. In bucketing, Hive splits the data into a fixed number of buckets, according to a hash function over some set of columns. Hive ensures that all rows that have the same hash will store in the same bucket. However, a single bucket may contain multiple such groups. For example,</p><p>&gt; create table student_detail_bucket(id int, name string, branch string, gpa float) clustered by (gpa) into 2 buckets;</p><p>from student_detail insert overwrite table student_detail_bucket select id, name, branch, gpa;</p><p>The above code will create two buckets (two sub-directories) under the student_detail_bucket directory and data are segmented in each sub-directory in accordance with the hash function. See below,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8yv8xIyYlyXBQvQHeBtHaw.png" /></figure><h3>When to use partitioning in Hive?</h3><p>When you have a smaller number of distinct values in a column (a column has low cardinality), you should use partitioning. Partitioning (dynamic partitioning on the column) will create the number of partitions (sub-directories) equivalent to a number of distinct values in the column. It will enhance query response time because the query will filter data from the sub-directory which matches in the where clause rather than scanning the full table.</p><p>If a column has high cardinality, too many partitions will create too many sub-folders/sub-directories in a table folder/directory which will create overhead to NameNode. Because NameNode keeps all metadata for the file system in memory. So, be cautious.</p><h3>When to use bucketing in Hive?</h3><p>Partitioning is not recommended in case of a column has high cardinality because too many partitions (sub-directories) will be created. In other words, in a case of a column having high cardinality, bucketing is preferred. While you make bucketing in the hive, you need to specify the number of buckets to manage your data efficiently and the hash function will put data in the appropriate bucket. For example, the salary column in a table that has extremely high cardinality can be placed in 5 buckets using a hash function.</p><h3>Conclusion</h3><p>In the above discussion, it is super simple to understand what partitioning and bucketing are. Both are used to enhance query execution time/query optimization. If a column has low cardinality (a smaller number of distinct values in the column) partitioning is used. On the contrary, if a column has extremely high cardinality (a greater number of distinct values in the column) bucketing is preferred.</p><p>On winding up notes, feel free to share your comments. Your likes and comments will help me to present content in a better way. See you next week.</p><p><em>More content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><em>. Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a><em> and </em><a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><em>. Check out our </em><a href="https://discord.gg/GtDtUAvyhW"><strong><em>Community Discord</em></strong></a><em> and join our </em><a href="https://inplainenglish.pallet.com/talent/welcome"><strong><em>Talent Collective</em></strong></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88d31578bb34" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/what-is-partitioning-vs-bucketing-in-apache-hive-partitioning-vs-bucketing-88d31578bb34">What is Partitioning vs Bucketing in Apache Hive? (Partitioning vs Bucketing)</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Managed Table vs. External Table in Apache Hive (with Hands-on)]]></title>
            <link>https://aws.plainenglish.io/managed-table-vs-external-table-in-apache-hive-with-hands-on-2483205e86e1?source=rss-11777614c8cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/2483205e86e1</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[big-data-analytics]]></category>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[hive]]></category>
            <dc:creator><![CDATA[Dr. Virendra Kumar Shrivastava]]></dc:creator>
            <pubDate>Wed, 27 Apr 2022 11:44:19 GMT</pubDate>
            <atom:updated>2022-04-28T10:38:59.017Z</atom:updated>
            <content:encoded><![CDATA[<h4>Explaining managed tables vs. external tables in Apache Hive.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*i9FYQBuVGpq1VJcZCdp0kw.jpeg" /></figure><p>Hello everyone,</p><p>In this article I am going to explain managed tables vs. external tables in Apache Hive. This question is prominently featured in the interviews. So, if you want clarity on this aspect, this article is for you.</p><p>So, lets dive into deep in the discussion.</p><p>Apache Hive is a warehouse tool in distributed HDFS environment which is used to store huge dataset. You can use data ingestion tools to ingest dataset from variety of platform to Hive warehouse. Apache hive support two type of tables:</p><p>1. Managed or Internal table</p><p>2. External table</p><h3><strong>1. Managed or internal table</strong></h3><p>When you create a table in Apache hive, by default it is treated as managed or internal table. If you want to create an external table, you will have to use “external” keyword explicitly. By default, Hive stores the managed table in the warehouse folder under hive. In case of managed table, schema (meta data) and the data included in the tables are managed by <strong>Hive</strong>. You can perform HQL operation such as insert, update, and delete on the managed table<em>. But when you drop managed table, </em>the data along with schema will also get deleted.</p><p>For example, I have created a managed table iris_tab_managed in the students database. The see below given snippet:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*ukxfpJB-fmmm9Ehm73sUkw.jpeg" /></figure><p>So, by default iris_tab_managed is created in the warehouse folder under Hive. I have upload data in the managed table (see below given snippet).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*lXdWp9Jtu7ZoJoh9Q0uFgQ.jpeg" /></figure><p>So, When I load data in the iris_tab_managed table it will be stored in warehouse folder under studets.db/iris_tab_managed location. The see below given snippet:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*IApYxg6N9fi16n2D0R-33g.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*qW4MuEuFM3g2oEhFvTmrDA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/1*_lqz50qM1YQfnUQ_OHJiNA.jpeg" /></figure><p>When I execute drop table iris_tab_managed command it will drop data as well as meta data of iris_tab_managed table. The see below given snippet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*8ZOVek1jUbLsfrpxvyD0pA.jpeg" /></figure><p>It above snippet clearly confirms that when I dropped a managed table, the data along with schema get deleted.</p><h3><strong>2. External Table</strong></h3><p>You can create external table using <strong>external</strong> keyword and you can also specify <strong>location</strong> of the dataset in HDFS. See below given snippet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*USgpzM3jCXgK4d9otg0ECg.jpeg" /></figure><p>To confirm whether a table is managed or external you can run following command</p><p>&gt;describe formatted iris_tab_managed;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*3cNEvQZ7ka3Qby0MWCyStA.jpeg" /></figure><p>External tables can be stored at any location in the HDFS. Hive only manages meta data of the external table. In case of external table only schema is get deleted but data remains intact in the underlaying location.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*-E5rkblDXOziEZhrgf45TA.jpeg" /></figure><p>If you do not specify location of data, then by default external table will be created in the warehouse folder under hive. See below snippet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*aIlVTAcQ4AaYtoZik_ALCQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*3TufL88GuKXgG_FeX_VYiA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*_KFxSWQwdFo_e7vujy2pbQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*lEB95aM3zMxVmqj3tfVWdQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/1*7x0kzsO4ZdLsLbk99unNpw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*mzMIYAdwjNMsgvZhC-37Vw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*fOJ-2aTtASSNzM-1-GorGg.jpeg" /></figure><p>In case of <strong>External Tables</strong>, only the meta data of the table is managed by <strong>HIVE</strong>. The data present in these tables can sit on any storage locations in <strong>HDFS</strong>. You cannot perform update and delete operation on external tables. You can perform only select operation on external tables. When you <strong>drop</strong> external table, only the schema / table/ meta data gets deleted and not the data contained in the external table.</p><h3><strong>Conclusion</strong></h3><p>From the above discussion, it is super simple to understand that managed table and external table is quite different and used for different purpose. In case of managed table, schema and data is managed by Hive whereas to create external table you need to use external keyword in create table statement and you can also specify location of the dataset. In external table, Hive only manage meta data and data can be stored at any location in HDFS. Managed tables follow ACID properties whereas you cannot perform update and delete operations on external tables. Insert operation is slow in managed operation whereas it is fast in external tables. So, I am sure this stuff will clear your doubts about managed or internal tables VS managed tabled in Apache Hadoop.</p><p>On wrapping up notes, feel free to share your comments. Your claps and comments will surely help me to present contents in a better way. See you next week.</p><p><em>More content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><em>. Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a><em> and </em><a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><em>. Join our </em><a href="https://discord.gg/GtDtUAvyhW"><strong><em>community Discord</em></strong></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2483205e86e1" width="1" height="1" alt=""><hr><p><a href="https://aws.plainenglish.io/managed-table-vs-external-table-in-apache-hive-with-hands-on-2483205e86e1">Managed Table vs. External Table in Apache Hive (with Hands-on)</a> was originally published in <a href="https://aws.plainenglish.io/">AWS in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CURD Operations in MongoDB — NOSQL Database]]></title>
            <link>https://profvkshrivastava.medium.com/curd-operations-in-mongodb-nosql-database-494207e9d114?source=rss-11777614c8cc------2</link>
            <guid isPermaLink="false">https://medium.com/p/494207e9d114</guid>
            <category><![CDATA[big-data-analytics]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[mongodb]]></category>
            <category><![CDATA[nosql]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Dr. Virendra Kumar Shrivastava]]></dc:creator>
            <pubDate>Wed, 20 Apr 2022 05:43:28 GMT</pubDate>
            <atom:updated>2022-04-20T06:36:37.756Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>CURD Operations in MongoDB — NOSQL Database</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*3svLPgFArsDJouE2ur9_bQ.jpeg" /></figure><p>Hey guys,</p><p>In this article I am going present Create, Read, Update, and Delete (CURD) operation with examples which you can perform in MongoDB. Mongo DB is a document based NOSQL database which is quite popular in the industry. This article will create a basic understanding of how to perform CURD operations in MongoDB.</p><p><strong>So, lets dive into deep…</strong></p><p>In first step launch MongoDB. If you want to know how to install and run MongoDB, you can refer to my article <a href="https://aws.plainenglish.io/introduction-to-mongodb-a-nosql-database-3f55b7475e70">Introduction to MongoDB — a NoSQL Database (How to Install — MongoDB on Ubuntu?)</a></p><h3><strong>1. Create Operation:</strong></h3><p>Frist I will create a dataset. To create a database run following command:</p><p>&gt; use studentdb</p><p>To create collection in MongoDB, run the following command:</p><p>&gt; db.createCollection(“midterm”)</p><p>Once the collection is created, I can insert documents into the collection. To insert documents into the collection, run the following command:</p><p># Insert record in collection</p><p>&gt; db.midterm.insert(</p><p>{</p><p>_id: 1,</p><p>name:”Ayush”,</p><p>gpa:5.5,</p><p>DOA:”01-Jan-2022&quot;</p><p>}</p><p>);</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*EBpojXLJISoyNGhxiGTViQ.jpeg" /></figure><p>To insert more than one document or bulk documents, run the following command:</p><p>db.midterm.insert(</p><p>[</p><p>{</p><p>_id: 2,</p><p>name:”Jay”,</p><p>gpa:5.8,</p><p>DOA:”02-Jan-2022&quot;</p><p>},</p><p>{</p><p>_id: 3,</p><p>name:”Aakash”,</p><p>gpa:8.5,</p><p>DOA:”02-Jan-2022&quot;</p><p>}</p><p>]</p><p>);</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*sLgv4pgsEkpaowzC_AiodQ.jpeg" /></figure><h3><strong>2. Update</strong></h3><p>MongoDB allows to update documents inserted in the collection. To update existing document in the collection, run the following command:</p><p>&gt;db.midterm.update( {_id: 3}, {$set:{gpa:2.5}});</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*GHiDouDbFxwwKT7gKivQpg.jpeg" /></figure><p>In case of the document does not exists in the collection, it will be inserted otherwise updated. Use the following command to update record</p><p>db.midterm.update(</p><p>{</p><p>_id: 5,</p><p>name:”Rohit”,</p><p>gpa:6.8,</p><p>DOA:”02-Jan-2022&quot;</p><p>},</p><p>{$set:{gpa:6.5}</p><p>}, {upsert:true}</p><p>);</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*xSe27biwxectmJAF71k2hQ.jpeg" /></figure><p>To add an additional attribute in the collection, run the following command:</p><p>db.midterm.update( {_id: 3}, {$set:{class:”BDA”}});</p><h3><strong>3. Read Operation:</strong></h3><p>To read documents form collection is very simple in the MongoDB. The find() method allows to read / show records available in the collection.</p><p>&gt;db.midterm.find();</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*-1_jMuQDsWZl9r0d2ouAFg.jpeg" /></figure><p>To format read result, we can use pretty() method</p><p>&gt;db.midterm.find().pretty();</p><p>To find specific document in the collection, run the following command:</p><p>&gt;db.midterm.find({name:”Ayush”}).pretty()</p><p>To display only the name from all the documents from collection midterm and identifier id should not be displayed, run the following command:</p><p>&gt;db.midterm.find({}, {name:1, _id:0}).pretty()</p><p>Here, _id:0 indicate that the _id will not be included in the result.</p><p>&gt;db.midterm.find({}, {name:1,gpa:1, _id:0}).pretty()</p><p>&gt;db.midterm.find({_id:1}, {name:1,gpa:1}).pretty()</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*4ywi0g0A-rTmKyg3itt6PA.jpeg" /></figure><p>You can use conditional operator to perform desired output based on satisfying the search criteria. The Relational operators are</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/523/1*b3f8C4hENhaNR_-ckQyXVA.jpeg" /></figure><p>To select name, gpa from midterm collection whose id is 1, run the following command:</p><p>&gt;db.midterm.find({_id:1}, {{},name:1, gpa:1})</p><p>&gt;db.midterm.find({gpa:{$eq:5.5}}, {name:1})</p><p>&gt;db.midterm.find({gpa:{$lt:5}}).pretty()</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*qT7oc0BM-7o40RDQHE-kSA.jpeg" /></figure><p><strong>Other operators</strong></p><p>in for in the list</p><p>nin for not in the list</p><p>For example,</p><p>&gt;db.midterm.find({gpa:{$nin:[2.5,7.5]}} )</p><p>&gt;db.midterm.find({gpa:{$nin:[2.3,7.8]}} )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*LHjO7wVljH6RaeR6Lh_vxg.jpeg" /></figure><h3><strong>4. Delete Operation:</strong></h3><p>A document can be removed from collection by the following command:</p><p>&gt; db.midterm.find({name:”Rohit”})</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*TtuLW8iE4_J8GAkZ2HmwTg.jpeg" /></figure><p>To drop a document from the collection run the following command:</p><p>&gt;db.midterm.remove({_id=4})</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*_3koiWZ2OGGPO4lI3AhQTA.jpeg" /></figure><p>To drop a collection, you can use the following command:</p><p>&gt;db.temp.drop()</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*auAhxxAh5ht0MI0sSl_Yhg.jpeg" /></figure><h3><strong>Conclusion:</strong></h3><p>In this article I have explored the basic operations Create, Update, Read and Delete (CURD) in MongoDB. It is supper simple to perform these operations on NOSQL document database MongoDB. I am sure, it can be a stepping point for the novice MongoDB users. It is widely used nowadays.</p><p>On wrapping up notes, thank for the read and feel free to share your comments. Your comments will surely help me to present contents in a better way. See you next week.</p><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=494207e9d114" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>