<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shwetabh Dixit on Medium]]></title>
        <description><![CDATA[Stories by Shwetabh Dixit on Medium]]></description>
        <link>https://medium.com/@shwetabh.dixit?source=rss-2e5af0d62140------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*l95EUGQcjOEvT79Z.</url>
            <title>Stories by Shwetabh Dixit on Medium</title>
            <link>https://medium.com/@shwetabh.dixit?source=rss-2e5af0d62140------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 09:29:58 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shwetabh.dixit/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[My intern experience at Zeotap]]></title>
            <link>https://medium.com/@shwetabh.dixit/my-intern-experience-at-zeotap-e066c3de82a7?source=rss-2e5af0d62140------2</link>
            <guid isPermaLink="false">https://medium.com/p/e066c3de82a7</guid>
            <category><![CDATA[intern]]></category>
            <category><![CDATA[internship-experience]]></category>
            <category><![CDATA[internships]]></category>
            <category><![CDATA[internship-program]]></category>
            <dc:creator><![CDATA[Shwetabh Dixit]]></dc:creator>
            <pubDate>Sat, 06 Jun 2020 09:12:29 GMT</pubDate>
            <atom:updated>2020-06-06T09:34:04.260Z</atom:updated>
            <content:encoded><![CDATA[<p>Hey people, I am going to keep this short, no one wants a 20 min read about an internship. I will try and be as descriptive and concise as possible, that seems like an oxymoron, but let’s give it a try.</p><p>I got an internship with Zeotap after a coding round and an interview. I will not go into the details, but I would say the hardest part of the coding round was the SQL questions. I was able to solve only 1 of the 3. Feel free to get in touch in case you need more information.</p><p>A little about Zeotap, it is a customer intelligence platform that helps brands understand customers and predict behaviour. This is the link to the official website — <a href="https://zeotap.com/">Zeotap</a></p><p>The internship started in the 1st week of January and initially I was given migrations tasks as a part of the ongoing effort to switch from AWS to GCP. I got to learn quite a few things through this process, like the flow of data through different pipelines at Zeotap and GCP basics.</p><p>It was a month later that I along with my mentor, started work on a new module, which was a revamp of an existing pipeline and we got to build it from scratch. I still manage that particular module and I am the point of contact if something in the module fails. That is the best feeling, having the responsibility of an entire module, seeing it being used in production. I have never found it overwhelming to deal with any aspect of the module and have at times volunteered to fix any problems with it, because it gives me a sense of ownership.</p><p>I have to accept one thing though, I had read during college that a developer should not be allowed to test his own code and I always thought why would a developer not want his code to better. I understand it now, the design, the code, once you think it’s elegant, it becomes difficult to see other opinions, but enough about my struggles with by stubbornness.</p><p>Currently, I am working on a second module, completely different, its about populating statistics over each layer of pipeline for different products. Its still in progress, so let’s hope it fares out well too.</p><p>One more part of the job that I would like to mention is the on-call responsibility. It means that for a particular time frame, generally a week, you are the one responsible for dealing with any failures in the pipelines running under ZeoCore. It’s an arduous process no doubt, but it helps you understand different workflows, intricacies of each of them and in my view is the best way to get to know all the workflows better. What better way to understand a pipeline than fixing it, huh?</p><p>I have learned so much through this internship, be it Spark, Scala, design development, it has been wonderful. If I had to pick one, it would be JIRA based development, I had built projects before, but just the elegance of dividing your project into modules and having JIRAs that explain those modules, was a new experience for me. I know there must be better and more accurate definitions of the above development process, but this is how I understood it.</p><p>Talking about the culture at Zeotap, its very hard to put into words a culture, so I would help you out with a little anecdote. I once committed a mistake and deleted something from production, and the support I got after that from my manager and my senior is something that I will always remember.</p><p>“We are engineers, this should be taken as a learning experience” — My senior</p><p>“Don’t worry, let’s get on a call to understand the impact” — My manager</p><p>Although the data was recovered, but for me as an intern having made a mistake, it meant the world to get this support and helped me have confidence. I have taken help or advice from almost every person in my team, and to be honest some of those questions were not the best, not very well researched and yet I have gotten patient answers. It’s a good balance between expecting ownership from you and helping you when you make mistakes.</p><p>We also have monthly all-hands meeting, where the whole org. gets together to list out our achievements and celebrate each other. These meetings are semi-casual. Even during Covid, the trend has continued although no parties afterwards, haha.</p><p>This wasn’t the first time I did an internship, I had interned before and had very good experiences, and it was just a blessing that the cycle did not break.I am gonna end this by saying, it has been a wonderful experience, the level of responsibilities and access given to me was something I had never experience before.</p><p>Some noteworthy mentions, the food is great, especially the evening snacks. Also, we have a TT table and I am open to a challenge. <br>P.S — I don’t play all that well. Haha.</p><p>I hope this wasn’t too long of an article, feel free to reach out to me for any further questions. If people do request a more detailed view of the internship, I would be happy to do it.</p><p>Thanks for Reading and Stay safe !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e066c3de82a7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding tcpdump/windump]]></title>
            <link>https://medium.com/@shwetabh.dixit/understanding-tcpdump-windump-b9edb9f6cd61?source=rss-2e5af0d62140------2</link>
            <guid isPermaLink="false">https://medium.com/p/b9edb9f6cd61</guid>
            <category><![CDATA[computer-network]]></category>
            <category><![CDATA[tcp]]></category>
            <category><![CDATA[internet]]></category>
            <category><![CDATA[tcpdump]]></category>
            <category><![CDATA[network-security]]></category>
            <dc:creator><![CDATA[Shwetabh Dixit]]></dc:creator>
            <pubDate>Mon, 18 Mar 2019 12:50:38 GMT</pubDate>
            <atom:updated>2019-03-18T12:50:38.927Z</atom:updated>
            <content:encoded><![CDATA[<p>So, I had this assignment on tcpdump and when I started researching for it, I found that I had trouble understanding how exactly it works, what does the output indicate and how is it useful. This is my attempt to provide myself and everyone else some clarity on the process. Let us get on with it then.</p><p>So, the basic definition of tcpdump is -</p><p>“tcpdump is a common packet-analyzer. It allows the display of TCP/IP packets and other packets being transmitted over a network to which the computer is connected. “</p><p>Also, its a free software under the BSD license and runs on Unix like operating systems such as Linux, macOS, etc. It makes use of libpcap to do so.</p><p>So till this point when I first read this, I had very little clue as to what tcpdump was, having not heard the name before. You people may be smarter than me and might have understood this in one go. But I am going to assume you did not and break it down the way I did to understand this.</p><p>The terms that struck out were :</p><ul><li>packet-analyzer</li><li>network interface</li><li>libpcap</li></ul><p>Let us try and understand each of them, not in that exact order.</p><h3><strong>Network Interface</strong></h3><p>A <strong>network interface</strong> is a system’s (software and/or hardware) interface or point of connection between two pieces of equipment or protocol layers in the computer network. For our purposes it mostly takes the form of a NIC( Network Interface Controller). You can read more about NIC here-</p><p><a href="https://searchnetworking.techtarget.com/definition/network-interface-card">What is network interface card (NIC)? - Definition from WhatIs.com</a></p><h3>Packet Analyzer</h3><p>A <strong>packet analyzer</strong> or <strong>packet sniffer</strong> is a computer program or piece of computer hardware that can intercept and log traffic that passes over a digital network or part of a network.</p><p>Now that we have looked at the basics of what tcpdump is trying to achieve and where does it operate. I think its the right time to dive into libpcap.</p><h3>libpcap</h3><ul><li>pcap(Packet Capture Process) roughly means to grab a copy of packets off the wire before they are processed by the operating system.</li><li>libpcap is a c/c++ library of procedures which we will use to grab packets as they come out of the network interface card.</li></ul><p>But this still feels a little vague, let us see if we can make this clearer with the following statements.</p><ul><li>Capturing packets means collecting data being transmitted on the network. Every time a network card receives an Ethernet frame, it checks if its destination MAC address matches its own. If it does, it generates an interrupt request. The routine that handles this interrupt is the network card’s driver.</li><li>It copies the data from the card buffer to kernel space, then checks the ethertype field of the Ethernet header to determine the type of the packet, and passes it to the appropriate handler in the protocol stack. The data is passed up the layers until it reaches the user-space application, which consumes it.</li><li>When we are sniffing packets, the network driver also sends a copy of each received packet to the packet filter.</li></ul><p>So now that we have developed an understanding of how the process works and what makes it work. It would be good to look at some commands.</p><h3>NOTE : I will be using windump which is the windows port of tcpdump since I have a Windows system. The commands for both of them are identical barring from the intial “windump” instead of “tcpdump”.</h3><p>So first off, we need to install tcpdump or windump is my case.</p><p><strong>tcpdump</strong> : <a href="https://www.hugeserver.com/kb/install-use-tcpdump-capture-packets/">https://www.hugeserver.com/kb/install-use-tcpdump-capture-packets/</a></p><p><strong>windump</strong> : <a href="https://uwnthesis.wordpress.com/2014/05/26/windump-how-to-use-windump-tcpdump-on-windows-7-the-visual-guide/">https://uwnthesis.wordpress.com/2014/05/26/windump-how-to-use-windump-tcpdump-on-windows-7-the-visual-guide/</a></p><p>Now lets get our hands dirty -</p><p>First off we need to specify an interface from which we want to capture traffic.</p><ul><li><strong>tcpdump -D</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/1*2Gr4XWMm_F2xXA9SOR4KzA.png" /></figure><p>This command lists all the network interfaces present on your system. Now we would want to select a particular interface and capture the traffic.</p><ul><li><strong>Choosing the interface and number of packets</strong></li></ul><p><strong>tcpdump -i {interface of choice} — c {number of packets}</strong></p><p>Specifying the count tells the system to stop after collecting those many number of packets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/0*_cuyrNK5JknITuG8" /><figcaption>Specifying the count and interface</figcaption></figure><p>But if we are to use this data, we need to store this data to a file for later use, let us see how that can be accomplished.</p><ul><li><strong>Write output to a file</strong></li><li><strong>tcpdump -i {interface of choice} -c {number of packets} -w{absolute file path}.pcap</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/0*B_BBgEQ4fRqctBdz" /></figure><p>the .pcap is the file extension used to store the information. It can be read using tcpdump or <a href="https://en.wikipedia.org/wiki/Wireshark">WireShark</a>.</p><ul><li><strong>Reading the file —</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/0*uI2LesVwIpyDE601" /></figure><p>— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —</p><p>Now, what use is capturing this data if we don’t understand it. So let us go step by step and understand the output that is produced.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/1*pMyD7KWYl608cBYEpJ3qqw.png" /></figure><h3><strong>Interpreting the output</strong></h3><ol><li>The first information displayed is the timestamp at which the packet was captured.</li><li>Right after that the <strong>source IP: port &gt; destination IP:port</strong> have been displayed. Here you see that the IP has been resolved my desktop’s IP to its DNS form. If you don’t want that , add -n to the list of expressions</li></ol><p>i. e . tcpdump -i eth0 -c 20 -n</p><p>3. After that if you see a <strong>“S” , </strong>that indicates the SYN flag which means a session establishment request which is the first part of any TCP connection( 3-way handshake).</p><p>4. Next is the sequence numbers 3682678150:3682678150(0). This is the</p><p><strong>beginning TCP sequence number : ending TCP sequence number (data bytes)</strong></p><p>The initial sequence number(ISN) is selected as a unique number to mark the first byte of data. The ending sequence number is the beginning sequence plus the number of data bytes to be sent with this TCP segment. In this case there were 0 bytes sent, hence they both are same.</p><p>5. win 65535 : This is the receiving buffer size in bytes. The rest are the various flags and information on the set up of a connection.</p><p>6. Once they are connected and the ack (acknowledgment) flag is set from both sides , we see a P flag which means data is now being pushed.</p><p>So that brings us to the end of our article. But I will leave you with this -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/686/0*_CqJiOol1UFJC01K" /></figure><p>This specifies all the options available to you, to understand what each one of them does you can refer to the man page of tcpdump -</p><p><a href="https://www.tcpdump.org/manpages/tcpdump.1.html">tcpdump(1) man page</a></p><p>Feel free to contact me for any doubts. I will try and look into them. I am no expert, so if there are any mistakes, please do inform me. Happy reading :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9edb9f6cd61" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hadoop Installation on Windows]]></title>
            <link>https://medium.com/@shwetabh.dixit/hadoop-installation-on-windows-bf699ce088ab?source=rss-2e5af0d62140------2</link>
            <guid isPermaLink="false">https://medium.com/p/bf699ce088ab</guid>
            <category><![CDATA[mapreduce]]></category>
            <category><![CDATA[hadoop]]></category>
            <dc:creator><![CDATA[Shwetabh Dixit]]></dc:creator>
            <pubDate>Sat, 09 Jun 2018 13:56:52 GMT</pubDate>
            <atom:updated>2018-06-09T13:56:52.244Z</atom:updated>
            <content:encoded><![CDATA[<p>Hello, before I start, it should be established that I am in no way, an expert on Hadoop. I recently started learning about Hadoop and due to some reasons had to do the installation on Windows.</p><p>After a week of scavenging on the internet I was able to understand or I should say begin to understand the world of Hadoop. I would like to think of this article as being written by someone who has recently started exploring a field and knows the confusions others might face, rather than from someone who knows the process in and out.</p><p>Let us get started,</p><p>The moment you google hadoop installation, almost all the links that appear on the first page provide a guide for hadoop installation on Ubuntu. Why is that?</p><p>That is because Hadoop was built on top of Linux.</p><p>The question arises, can we install Hadoop on Windows?</p><p>The answer is “YES”. Hadoop versions following 2.2 support a windows installation. If you want to install Hadoop 1.x, you can follow the guide →</p><p><a href="http://saphanatutorial.com/hadoop-installation-on-windows-7-using-cygwin/">http://saphanatutorial.com/hadoop-installation-on-windows-7-using-cygwin/</a></p><p>Although I would not recommend using Hadoop 1.0 since it does not provide yarn as a tool, which is an integral part of using Hadoop now. Hadoop 2.0 replaces job tracker with resource manager and application manager. If you want to read in detail about the differences between the two, I would suggest the following link →</p><p><a href="https://www.journaldev.com/8806/differences-between-hadoop1-and-hadoop2">https://www.journaldev.com/8806/differences-between-hadoop1-and-hadoop2</a> .</p><p>Now, let us dive into the installation of Hadoop version 2.7.3 on Windows 10. A question that may arise here, is can other Windows versions be used?</p><p>Yes, I have personally successfully installed Hadoop 2.7.3 both on Windows 10 and Windows 7, and to the best of my knowledge it works fine on Windows 8 according to → <a href="https://hadoop.apache.org/docs/r2.8.2/hadoop-project-dist/hadoop-common/SingleCluster.html">https://hadoop.apache.org/docs/r2.8.2/hadoop-project-dist/hadoop-common/SingleCluster.html</a></p><p>One more information before proceeding forward, we are going to set up a psuedo-distributed single cluster with Namenode, Datanode and Yarn running on the same machine.</p><p>I will also try and cover some of the errors I faced while installation . Now, without any further adieu, let us look at the process step-by-step.</p><ol><li>Pre-requisite Softwares</li></ol><ul><li>Java Version 1.8</li></ul><p>If you don’t have Java installed, no worry, Java is completely free and can be downloaded from here → <a href="http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html">http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html</a></p><p>If you already have Java Installed, you can check your Java version by typing <strong>“java -version”</strong> on command prompt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/0*Wj8TNcWrtoIGAKeF" /></figure><p>If you do not have it installed, just follow the steps and you’ll be good to go, although it’s pretty basic but let me be thorough.</p><p>Go to <a href="http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html">http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/0*O3-HvP1COqVCsFiB" /></figure><p>Now that, we have that sorted out, let’s move ahead.</p><ul><li>A lot of tutorials say <strong>cygwin</strong> is needed to set up Hadoop, which is only partially true. Cygwin, in my experience was used only to ‘un-tar’ Hadoop from the format it is downloaded in and giving file permissions . If I am wrong please do tell me. But, given that you have Windows 10, you don’t need cygwin.</li></ul><p>Windows 10 provides an inbuilt Windows subsystem which allows you to basically work on Linux environment, I am guilty of being hazy in the specifics ,best you read it over here if you want to know -</p><p><a href="https://www.howtogeek.com/249966/how-to-install-and-use-the-linux-bash-shell-on-windows-10/">https://www.howtogeek.com/249966/how-to-install-and-use-the-linux-bash-shell-on-windows-10/</a></p><ul><li>Hadoop itself -</li></ul><p>I would recommend downloading Hadoop 2.7.3 for the sheer purpose of clarity through this tutorial. You can download it here-</p><p><a href="https://archive.apache.org/dist/hadoop/core/hadoop-2.7.3/">https://archive.apache.org/dist/hadoop/core/hadoop-2.7.3/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/0*ourPXCQ1QK8nEXYb" /></figure><p>Now, we have the tarred version of the Hadoop with us, we first of all need to un-tar it. People with Windows 10, can do it directly from their systems by following these steps-</p><ol><li>Open command prompt</li></ol><p>2. Type ‘bash’, if you already have the subsystem active you will see something like this-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*H8Vc3DS-bYQLPb3a" /></figure><p>If you don’t, just type the error on Google and windows the process kind of guides itself, if you do have any trouble, feel free to comment and I will try and help you.</p><p>Once you do see this, it means you can now run linux commands from your terminal.</p><p>For those using Windows 7, you have to download cygwin the guide for which can be found here -</p><p><a href="http://saphanatutorial.com/hadoop-installation-on-windows-7-using-cygwin/">http://saphanatutorial.com/hadoop-installation-on-windows-7-using-cygwin/</a></p><p>It’s the same for Hadoop 1.0 and 2.0 since as mentioned earlier, it is a tool for ‘un-tarring’ the downloaded Hadoop file. If you do use cygwin be very particular about the packages you download. Just follow the steps in the above tutorial and you should be fine.</p><p>Now, go to the terminal and go to the directory which contains your downloaded hadoop file</p><p>And type the command -&gt; tar xzf hadoop-2.7.3.tar.gz.</p><p>This will take some time to process and there won’t be a success message, but if you go the directory containing your .tar.gz version , you will now have an untarred version of the file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*V0qUSKyFWhHzodoi" /></figure><p>From here on, the steps for Windows 7 and Windows 10 are same.</p><p>So, now we have an untarred version of our Hadoop folder. When done successfully, it will look somewhat like this-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/0*FfsRSHLXqSabGtjk" /></figure><p>The folder that has been blacked out won’t be there, we will talk about that later.</p><p>Now, comes the configuration part of our assignment.</p><p>We need to set JAVA_HOME, HADOOP_HOME and append the path of bin of both java and Hadoop.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/0*aMIlxDKlhJ5ATZYd" /></figure><p>We go to the directory where Hadoop is installed and place the path of the directory as Hadoop home.</p><p>Same is done for <strong>JAVA.</strong></p><p>We now append the bin paths of both.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/391/0*WnS0lwx1I7MhAWqT" /></figure><p>Now, we need to configure Hadoop files, which are -</p><ul><li>Hadoop-env.cmd</li><li>Core-site.xml</li><li>Mapred-site.xml</li><li>Hdfs-site.xml</li><li>Yarn-site.xml</li></ul><p>Let us see all the files -</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/0*VhATUBqrbPlwTRk3" /></figure><p>Now, going to Hadoop-env.cmd . Open it with a text editor and edit the following lines -</p><ol><li>add “set JAVA_HOME=C:\Java\jdk1.8.0_92” at Line number 26 as shown below.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/0*o4D9uA7plt4royPY" /></figure><p>You should set your JAVA_HOME for your computer. <strong>One very important note here. Make sure that there are no spaces in the path of JAVA_HOME</strong>, that creates errors later on.</p><p>I would recommend keeping the Java folder directly inside C:/ rather than inside Program Files to avoid this.</p><p>Also, I have added these files to my Github repository. You can download them from there directly. But do read to get an understanding of how we configure Hadoop.</p><p>Now, before we go further, we have to create a data folder in Hadoop home directory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/0*I9czdad8yTGSescH" /></figure><p>Inside the data folder, create two folders by the name -</p><ol><li>Namenode</li><li>Datanode</li></ol><p>We will also have to give read permissions for the same.</p><p>This can be done using cmd</p><p>Open cmd, type bash</p><p>Then <strong>go to the directory of your HADOOP_HOME </strong>using the cd command .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/0*RWQMY3KZEjHtHoE6" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/0*sq_WqQT8BUsljcm-" /></figure><ul><li><strong>Hadoop-env.cmd</strong></li></ul><p>Also , add the following lines at the end of Hadoop-env.cmd</p><p>set HADOOP_PREFIX=C:\Hitesh\Hadoop\hadoop-2.7.1</p><p>set HADOOP_CONF_DIR=%HADOOP_PREFIX%\etc\hadoop</p><p>set YARN_CONF_DIR=%HADOOP_CONF_DIR%</p><p>set PATH=%PATH%;%HADOOP_PREFIX%\bin</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/0*qQL6Dh1JUft7P8AA" /></figure><ul><li><strong>Core-site.xml</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/635/0*7hYiDt-Gn8YOIGgQ" /></figure><p>What is this file :Specifies the filesystem settings for Hadoop commands.</p><p>Property Name :fs.defaultFS</p><p>value : hdfs://localhost:19000</p><p>Short Description: The name of the default file system and access port.</p><p>Property Name :hadoop.proxyuser.&lt;&lt;WINUSER&gt;&gt;.groups</p><p>Short Description:This property sets groups that &lt;&lt;WINUSER&gt;&gt; can impersonate a user belonging to the group provided in value while connecting HDFS.You can specify “*” for least security or specific hdfs user group name as shown above.</p><p>Property Name :hadoop.proxyuser.&lt;&lt;WINUSER&gt;&gt;.hosts</p><p>Value :127.0.0.1,192.168.55.3</p><p>Short Description:This property lists hosts from whre &lt;&lt;WINUSER&gt;&gt; can is allowed connect to HDFS and perform tasks.This property is required to be set when you need to connect to hadoop from Hive,Pig or Sqoop like applications.You can specify “*” for least security or specific IP address as shown above.</p><p>Property Name :hadoop.proxyuser.&lt;&lt;WINUSER&gt;&gt;.users</p><p>Short Description:This property lists users other then hdfs users who can connect to HDFS same as users listed in value list.This property is required when you execute hive,oozie or sqoop jobs even from same machine and user.value :&lt;&lt;WINUSER&gt;&gt;</p><p>You can refer sample file at :<a href="https://github.com/hiteshf/win10Hadoop/blob/master/core-site.xml">core-site.xml</a></p><p>&lt;&lt;WINUSER&gt;&gt; is windows user(s) from which you wants to connect and execute hadoop/hdfs or mapreduce commands.In my case it was same user.</p><ul><li><strong>Mapred-site.xml</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/0*hUk-ikfzP0TsTj4y" /></figure><p>What is this file :This file is required to run mapreduce applications on hadoop.This file contains configuration information that overrides the default values for MapReduce parameters.</p><p>Property Name :mapreduce.framework.name</p><p>Value : yarn</p><p>Short Description:This property decides where to execute mapreduce application.valid values are yarn,local or classic.</p><p>Property Name :mapreduce.jobtracker.address</p><p>Value : local</p><p>Short Description:The host and port that the MapReduce job tracker runs at.”local” is a default value and all jobs are run in-process as a single map and reduce task.</p><p>Property Name :mapreduce.map.memory.mb</p><p>Value : 512</p><p>Description:This property is used to define upper limit in MB that Haddop allocates to maper while executing any mapreduce application out of total RAM.With Total RAM of 8 GB keep this value equal to 512 or less.</p><p>Property Name :mapreduce.reduce.memory.mb</p><p>Value : 512</p><p>Description:This property is used to define upper limit in MB that Haddop allocates to reducer while executing any mapreduce application out of total RAM.With Total RAM of 8 GB keep this value equal to 512 or less.</p><p>Property Name :mapreduce.map.java.opts</p><p>Value : -Xmx256m</p><p>Description : This property is used to define upper limit for maper java process java heap size.Value of this parameter should be less than memory allocated in mapreduce.map.memory.mb so that java code can execute without outofmemory error.Atleast 20% to 30% less.</p><p>Property Name :mapreduce.reduce.java.opts</p><p>Value : -Xmx256m</p><p>Description :Same as mapreduce.map.java.opts this property is used to define upper limit for reducer java process java heap size.Value of this parameter should be less than memory allocated in mapreduce.reduce.memory.mb so that java code can execute without out-of-memory error.Atleast 20% to 30% less.</p><ul><li><strong>Yarn-site.xml</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*enmwgoezXGVy2GXq" /></figure><p>What is this file : This file contains yarn configuration.</p><p>Property Name :yarn.nodemanager.aux-services</p><p>Value : mapreduce_shuffle</p><p>Short Description: This property is for Nodemanager do see which aux-services needs to be used.”mapreduce_shuffle” is the default value.</p><p>Property Name :yarn.nodemanager.aux-services.mapreduce.shuffle.class</p><p>Value : org.apache.hadoop.mapred.ShuffleHandler</p><p>Short Description:This property is for Nodemanager to identify which class to use for shuffling when aux-services are set to “mapreduce_shuffle”.</p><p>Property Name :yarn.nodemanager.resource.memory-mb</p><p>Value : 10000</p><p>Short Description:This property is used to set Maximum Amount of physical memory, in MB, that can be allocated for containers.This is the maximum memory in MB which can be used by yarn to perform Hadoop operations.</p><p>Property Name :yarn.scheduler.minimum-allocation-mb</p><p>Value : 500</p><p>Short Description:This property is used to set Minimum Amount of physical memory, in MB allocated to yarn.</p><p>Property Name :yarn.application.classpath</p><p>Value:%HADOOP_CONF_DIR%,%HADOOP_COMMON_HOME%/share/hadoop/common/*,%HADOOP_COMMON_HOME%/share/hadoop/common/lib/*,%HADOOP_HDFS_HOME%/share/hadoop/hdfs/*,%HADOOP_HDFS_HOME%/share/hadoop/hdfs/lib/*,%HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/*,%HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/lib/*,%HADOOP_YARN_HOME%/share/hadoop/yarn/*,%HADOOP_YARN_HOME%/share/hadoop/yarn/lib/*</p><p>Short Description:Classpath for yarn.</p><ul><li><strong>Hdfs-site.xml</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/0*hJ7yyspb7Km3DZGa" /></figure><p>You will have to edit this file based on where your hadoop home is and where have you created the data folder for your namenode and datanode.</p><p>Other Files required for windows 10 :</p><p>Hadoop needs windows OS specific files which does not come with default download of hadoop. You can build those files using instructions given in -</p><p><a href="https://svn.apache.org/viewvc/hadoop/common/branches/branch-2/BUILDING.txt?view=markup">https://svn.apache.org/viewvc/hadoop/common/branches/branch-2/BUILDING.txt?view=markup</a> or you can download from my git-hub repository. <a href="https://github.com/Shwetabhdixit/Hadoop-2.7.3-Installation-Guide-for_windows">https://github.com/Shwetabhdixit/Hadoop-2.7.3-Installation-Guide-for_windows</a></p><p>I would not recommend going through the trouble of the steps involved in Building.txt, it helps build a package for windows using Maven.</p><p>Take backup of original BIN folder present in HADOOP_HOME and replace it with downloaded BIN folder.</p><p>Congratulations, you have now setup Hadoop on your windows system. Now let us start hadoop.</p><p>Open command prompt and go to the bin directory in HADOOP_HOME. And then type “ hdfs namenode -format”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/0*F6d9ys89Dz_GNd2l" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JpgCOGoMhGDxB1U0" /></figure><p>The result must be exiting with status 0.</p><p>Exiting with status 1 means there is an error and we will need to debug it. You can understand the error by the logs that appear on the command prompt.</p><p>Now go to the sbin directory in hadoop home from cmd and type the command “ start-dfs.cmd” .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/0*5sfEkD4dAXwu4g3W" /></figure><p>Two more cmd windows will open. The ends of which should like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JoIGrwmd8Q9AJ051" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Yt66sJZy7KxLTGZ4" /></figure><p>If you see a shutdown message, it means there is an error and you again have to check the logs to determine.</p><p>In case the error message is incompatible cluster IDs. Copy the cluster ID of data node from the error logs of datanode and then run the following command.</p><p>hdfs namenode -format -clusterID “clusterID of datanode copied “</p><p>Now again open a cmd and go the sbin directory of hadoop and run the command start-yarn.cmd</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/521/0*ehTui9vMYh5Ds4vY" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nmpASutpsk-CEzBK" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pIfuPOcv8U1mUyC4" /></figure><p>Now we have everything running on our system, to check you can type the command “jps” to confirm.</p><p>Let’s have a look at the web UI of hadoop now.</p><p>Hadoop provides userinterface to check details about HDFS (namenode and datanode).</p><p>By-default it runs on <a href="http://localhost:50070/">http://localhost:50070/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hJnm8wEcZcCCj1iD" /></figure><p>Also, Yarn also provides different interface to access information about resource manager,current map-reduce jobs in process,faild or successfull jobs completed till date</p><p>By-default it runs on <a href="http://localhost:8088/cluster/apps">http://localhost:8088/cluster/apps</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-CEJAtLziV1Z0vJP" /></figure><p>Now you have Hadoop set up on your system. I will be back with some basic operations you can perform on HDFS and later on with an explanation of Word-count program written in JAVA.</p><p>If you have any doubts, do ask me and I will try and solve them.</p><p>My email id — <a href="/service/mailto:shwetabh.dixit@gmail.com">shwetabh.dixit@gmail.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bf699ce088ab" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>