<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rishabh Miglani on Medium]]></title>
        <description><![CDATA[Stories by Rishabh Miglani on Medium]]></description>
        <link>https://medium.com/@rishabhmiglani?source=rss-d9a528243944------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Lc3J_PTkA-bZwhUD.jpg</url>
            <title>Stories by Rishabh Miglani on Medium</title>
            <link>https://medium.com/@rishabhmiglani?source=rss-d9a528243944------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 10:30:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@rishabhmiglani/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Flafka]]></title>
            <link>https://medium.com/@rishabhmiglani/flafka-641bc7a96558?source=rss-d9a528243944------2</link>
            <guid isPermaLink="false">https://medium.com/p/641bc7a96558</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[apache-flume]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Rishabh Miglani]]></dc:creator>
            <pubDate>Mon, 09 Apr 2018 12:45:37 GMT</pubDate>
            <atom:updated>2018-04-09T12:45:37.321Z</atom:updated>
            <content:encoded><![CDATA[<p>This Blog post will highlight the use case behind making a durable Ingestion system,that has the ability to handle data in velocity and with great simplicity.</p><p>Thinking about the previous tools in for ingestion , Kafka and flume were perfect for certain use cases and were working well.</p><p>Kafka had the power to harness all kinds of data but required a lot of custom code for the producers and as well as for the consumers.Until sometime back the proactive features of flume and Kafka were merged to give a new entity to the flume opensource project called as Flafka.</p><p>Kafka required a lot of custom code for producers and consumers ,also flume had a single source to a single sink limitation.Flume also has the drawback to be not able to retain data in case of an agent failure.</p><p>Combining the two we could use the following combinations:l</p><p>1. Flume sources to act as producers to write to kafka</p><p>2. Flume sinks to act as consumer reading from kafka</p><p>3. A combination of above</p><p>4. Kafka as a channel for flume source and sink Combination( Latest commit to FLUME : 2500)</p><p>To prepare a durable ingestion pipeline you can use the following combination :</p><p>Pre built flume sources → kafka based channel → <strong>Flume Based Sink</strong></p><p>Pre built flume sources → kafka based channel → <strong>HDFS</strong></p><p>Pre built flume sources→ kafka based channel → <strong>Multiple Sinks</strong></p><p>The ability to dump the data into multiple sinks is the reason of adding the kafka channel in the latest commit. The above combination allows you to ingest any kinds of data and do multiple transactions and processing with it at the same time.</p><p>Sample code that allows you to use the flume sources ( twitter in this case) and then using kafka channel to persist it in HDFS as well as a Spark Stream to analyse side by side.</p><p><strong><em>Flume.conf</em></strong></p><pre>TwitterAgent.sources = Twitter<br>TwitterAgent.channels = KafkaChannel<br>TwitterAgent.sinks = HDFS2 HDFS1</pre><pre>TwitterAgent.sources.Twitter.type = com.cloudera.flume.source.TwitterSource<br>TwitterAgent.sources.Twitter.consumerKey=&lt;required&gt;<br>TwitterAgent.sources.Twitter.consumerSecret=&lt;required&gt;<br>TwitterAgent.sources.Twitter.accessToken= &lt;required&gt;<br>TwitterAgent.sources.Twitter.accessTokenSecret= &lt;required&gt;<br>TwitterAgent.sources.Twitter.keywords = obama<br>TwitterAgent.sources.Twitter.channels=KafkaChannel<br>TwitterAgent.sources.Twitter.kafka.consumer.timeout = 100000</pre><pre>TwitterAgent.channels.KafkaChannel.channel = KafkaChannel<br>TwitterAgent.channels.KafkaChannel.type = org.apache.flume.channel.kafka.KafkaChannel<br>TwitterAgent.channels.KafkaChannel.capacity = 10000<br>TwitterAgent.channels.KafkaChannel.transactionCapacity = 1000<br>TwitterAgent.channels.KafkaChannel.brokerList = localhost:9092<br>TwitterAgent.channels.KafkaChannel.topic = test3<br>TwitterAgent.channels.KafkaChannel.zookeeperConnect = localhost:2181<br>TwitterAgent.channels.KafkaChannel.parseAsFlumeEvent = false</pre><pre>TwitterAgent.sinks.HDFS1.channel = KafkaChannel<br>TwitterAgent.sinks.HDFS1.type = hdfs<br>TwitterAgent.sinks.HDFS1.hdfs.path =hdfs://localhost:9000/user/flafka/tweets<br>TWitterAgent.sinks.HDFS1.hdfs.batchsize = 0<br>TwitterAgent.sinks.HDFS1.hdfs.fileType = DataStream<br>TwitterAgent.sinks.HDFS1.hdfs.filePrefix = Flumedata1</pre><pre>TwitterAgent.sinks.HDFS2.channel = KafkaChannel<br>TwitterAgent.sinks.HDFS2.type = hdfs<br>TwitterAgent.sinks.HDFS2.hdfs.path =hdfs://localhost:9000/users/data/twitter_data<br>TWitterAgent.sinks.HDFS2.hdfs.batchsize = 0<br>TwitterAgent.sinks.HDFS2.hdfs.fileType = DataStream<br>TwitterAgent.sinks.HDFS2.hdfs.filePrefix = Flumedata2</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=641bc7a96558" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kerberos : Authentication tool for Hadoop]]></title>
            <link>https://medium.com/@rishabhmiglani/kerberos-authentication-tool-for-hadoop-320549797b0d?source=rss-d9a528243944------2</link>
            <guid isPermaLink="false">https://medium.com/p/320549797b0d</guid>
            <category><![CDATA[hadoop-security]]></category>
            <category><![CDATA[authentication]]></category>
            <category><![CDATA[kerberos]]></category>
            <dc:creator><![CDATA[Rishabh Miglani]]></dc:creator>
            <pubDate>Tue, 13 Oct 2015 04:16:32 GMT</pubDate>
            <atom:updated>2016-12-04T03:58:03.152Z</atom:updated>
            <content:encoded><![CDATA[<p>The Scope of this blog is to make the user completely aware of Kerberos and how to implement it with a Hadoop cluster.</p><p>Hadoop by default does not perform any authentication for any user or service in the cluster.Therefore a need to have an authentication mechanism arises relating to amount of critical data that is being analyzed nowadays.</p><h4>Risks associated with the cluster :</h4><p><em>1. Does not authenticate any user or any service</em></p><ul><li>Any user can impersonate as any other user and hence the ACL’s can be easily circumvented.</li><li>Any user can masquerade as a Hadoop service, It can register as a task tracker in the cluster.</li></ul><p><em>2.Data Nodes do not enforce any access control on access of their data blocks.Any user with a valid Block ID have the ability to write any arbitrary data or ready from the block.</em></p><h4>Security related requirements for the cluster:</h4><ul><li>Users can access the HDFS files only that they have permission to access.</li><li>Users should be able to modify there own Map reduce jobs</li><li>User to service mutual authentication(Unauthorized NameNodes , DataNodes)</li><li>Service to service mutual authentication to prevent Unauthorized services from joining their MapReduce Or HDFS services.</li><li>A third party managing the credentials of the complete cluster that will be transparent to both the service and the user.</li><li>The degradation of GridMix performance should be less than 3%</li></ul><p><strong>Note</strong>: GridMix is a Benchmark for Hadoop cluster</p><h4>What is Kerberos?</h4><p><strong>Kerberos : An authentication Mechanism for Hadoop</strong></p><p>Kerberos is a faster and better approach to have a proper authentication mechanism set up for a cluster.</p><p>It works on the principle of Tickets and is better secured avoiding certain transmission related breaches found in other authentication mechanisms.</p><p>Kerberos enables single sign on capability for all the resources attached to the network without caching the password of the user. One of the major safety points in Kerberos is that it never allows the users password to travel in the clear over the network.It is a widely opted and standardized protocol.</p><blockquote><em>With a central repository for all the user credentials it minimizes the probability of a security breach</em></blockquote><h4>Important terms related to Kerberos:</h4><p><strong>1.Realm</strong>: A realm is an Administrative Authentication Domain.It can be though of as the network domain.</p><p><strong>2.Principal</strong>:Any entity within a realm is a principal in the KDC database.A principal may be a user,Service or server in the environment defined by the realm.</p><blockquote><em>The principal is made up of 3 parts</em></blockquote><ul><li><strong>Primary</strong>:The username is considered as the primary for example : <strong>rishabh@MYREALM.COM</strong> where<strong> rishabh</strong> is the username.</li><li><strong>Instance</strong> : When there is a need of a further qualification an instance is applied to it.For an addition to the above: <strong>rishabh/admin@MYREALM.COM.</strong></li><li><strong>Service Principal</strong>:For a service running on a host that is a part of the Kerberos configuration, the principal would be :<strong>hdfs/host1.cluster.com@MYREALM.COM</strong>, where in the name for the host is the<strong> fully qualified domain name</strong> being used.</li></ul><p><strong>3.Keys</strong>: The KDC is a centralized repository for all the keys. Every principal has a key associated to it and is just known to that entity and the KDC. The key is used foe the encryption and decryption of the tickets for the purpose of authentication.</p><p><strong>4.Keytab</strong>: A Keytab is a file that is synonymous to the /etc/passwd file that has all the list of keys related to a specific service.</p><h4>Kerberos Protocol :</h4><p>We will first understand the Kerberos protocol to make implement it on the Hadoop cluster.</p><p>We will have the scenario around three main components :</p><p><strong>1.KDC (who grants the access)</strong></p><p><strong>2.User (who needs the access)</strong></p><p><strong>3.File server (to which access is needed)</strong></p><p>The KDC is made up of the following components:</p><p><strong>1.Authentication Service:</strong>It will perform all authentication related processes.</p><p><strong>2.Database:</strong>It will store all client related sensitive information like the secret keys of the client and the services.</p><p><strong>3.Ticket Granting Server:</strong>It will grant the service tickets to users and to the services as well.</p><p><strong>Note:</strong>Every time a new user/service is added to the Network the administrator has to generate a secret password in the KDC and share it only once in the initial service/user network configuration.</p><h4>Implementation of Kerberos:</h4><p>The cluster requires a KDC at a machine and some other packages on all the other hosts.</p><blockquote><em>We generally choose the cloudera manager machine to setup the KDC on (it can be any machine that will perform the authentication store.)</em></blockquote><p><strong>Set up the KDC:</strong></p><ol><li>Install krb5-libs,krb5-server,and krb5-workstation packages on the KDC node.Perform the following operations as root on the machine:</li></ol><pre>$yum install krb5-libs<br> $yum install krb5-server<br> $yum install krb5-workstation</pre><p>2.Setup the realm and domain to realm mapping in the following files</p><pre>$ cat /etc/krb5.conf<br>[logging]<br>default = FILE:/var/log/krb5libs.log<br>kdc = FILE:/var/log/krb5kdc.log<br>admin_server = FILE:/var/log/kadmind.log</pre><pre>[libdefaults]<br>default_realm = MYREALM.COM<br>dns_lookup_kdc = false<br>ticket_lifetime = 24h<br>renew_lifetime = 7d<br>forwardable = true</pre><pre>[realms]<br>MYREALM.COM = {<br>kdc = node1.hcluster<br>admin_server = node1.hcluster<br>}</pre><pre>[domain_realm]<br>.example.com =  MYREALM.COM<br>example.com =  MYREALM.COM</pre><p><strong>NOTE </strong>: The name of our realm is <strong>MYREALM.COM</strong> and the KDC is the <strong>node1.hcluster.</strong></p><blockquote><em>All realm names are uppercase and all DNS hostnames and domain names are lowercase.</em></blockquote><p>3.Update the kdc.conf files in the /var/kerberos/krb5kdc/ folder from the root user as shown below:</p><pre>$ cat /var/kerberos/krb5kdc/kdc.conf<br>[kdcdefaults]<br>kdc_ports = 88<br>kdc_tcp_ports = 88</pre><pre>[realms]<br>MYREALM.COM = {<br>#master_key_type = aes256-cts<br>acl_file = /var/kerberos/krb5kdc/kadm5.acl<br>dict_file = /usr/share/dict/words<br>admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab</pre><pre>supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal</pre><pre>max_life = 24h 0m 0s<br>max_renewable_life = 7d 0h 0m 0s</pre><pre>}</pre><p><em>The parameters of</em><strong><em> max_life and max_renewable_life</em></strong><em> have been set depending on the various use cases that we obtain in a cluster.</em></p><p>4.Create a database for the KDC that stores the keys of all the principals in the realm.</p><pre>$ kdb5_util create -s</pre><pre>Loading random data</pre><pre>Initializing database &#39;/var/kerberos/krb5kdc/principal&#39; for realm &#39;MYREALM.COM&#39;,</pre><pre>master key name &#39;K/M@MYREALM.COM&#39;<br>You will be prompted for the database Master Password.<br>It is important that you NOT FORGET this password.<br>Enter KDC database master key:<br>Re-enter KDC database master key to verify:</pre><p>5.Edit the /var/kerberos/krb5kdc/kadm5.acl file to specify which principals have administrative access.</p><pre>$ cat /var/kerberos/krb5kdc/kadm5.acl<br>*/admin@MYREALM.COM *</pre><p>5.Create the first principal for the root user as follows:</p><pre>$ kadmin.local -q &quot;addprinc rishabh/admin&quot;<br>Authenticating as principal root/admin@MYREALM.COM with password.<br>WARNING: no policy specified for rishabh/admin@MYREALM.COM; defaulting to no policy<br>Enter password for principal &quot;rishabh/admin@MYREALM.COM&quot;:<br>Re-enter password for principal &quot;rishabh/admin@MYREALM.COM&quot;:<br>Principal &quot;rishabh/admin@MYREALM.COM&quot; created.</pre><p>6.Start the KDC service using the follwoing commands as the root user.</p><pre>$service krb5kdc start<br>$service kadmin start</pre><p>Testing the installation of the KDC</p><ol><li>Use the command kinit</li></ol><pre>kinit rishabh/admin@MYREALM.COM</pre><p>2.Verify the existence of the TGT using the klist command as shown below.</p><pre>$ kinit rishabh/admin<br>$ klist<br>Ticket cache: FILE:/tmp/krb5cc_0<br>Default principal: rishabh/admin@MYREALM.COM<br>Valid starting     Expires            Service principal<br>02/10/15 14:08:06  20/10/15 14:08:06  krbtgt/MYREALM.COM@MYREALM.COM<br>renew until 20/10/15 14:08:06</pre><blockquote><em>The KDC is successfully Setup.</em></blockquote><p>Configuring the Kerberos Clients</p><ol><li>Install the following client packages at the workers:</li></ol><pre>$yum install krb5-libs<br>$yum install krb5-workstation</pre><p>2.Copy /etc/krb5.conf from the KDC server as the root user to all the client nodes in the cluster.</p><blockquote><em>The client nodes are now configured to work with Kerberos.</em></blockquote><p><strong>In the classic example of a typical Hadoop cluster when the CDH5 installation is in progress the following users are created.</strong></p><p><strong>a.hdfs: </strong>This user runs the namenode, datanode and secondary namenode.</p><p><strong>b.mapred: </strong>This user runs the jobtracker and tasktracker daemons when using the MRv1, and the Job history server daemon when YARN(Mrv2)</p><p><strong>c.yarn:</strong>this user runs the resource manager and the node manager daemons.</p><p><strong>d.hue: </strong>This user runs the Hue Server , Beeswax Server, Authentication Manager and Job designer Daemons.</p><p><strong>e.cloudera-scm :</strong>For the cloudera-manager to function all throughout.</p><p>These users are created hence we need to setup the authentication for all these users in kerberos.</p><p>We do this using the cloudera-manager console which will be explained in our next blog.</p><h4>Kerberos protocol</h4><p>Having set up the KDC and the kerberos workstations in the clsuter now we are ready to understand the kerberos protocol in detail.</p><p>Using the below image we can easily understand the exchange of tickets in a Hadoop cluster.</p><blockquote><em>Kerberos V/s SSH</em></blockquote><p>Now to discuss how is it better than SSH on various parameters.Though both of them have certain features that is not provided by the latter. Kerberos vs SSH</p><p>The following simulation of facts has been done on various factors pertaining to the domain of Hadoop clusters.</p><p><strong>1. Infrastructure:</strong></p><p>The characteristic difference between both the technologies is that SSH is a lightweight, easily deployed package, designed specially for working with existing systems with minimal changes. KerberosKerberos in contrast , requires a significant infrastructure before use. The infrastructure reading further it will come out to be the driving mechanism behind the strong authentication mechanism created by Kerberos.</p><p>In Case of SSH we only need a SSH client on one machine and SSH server on the other machine, Where as the requirements for a Kerberos setup are:</p><ul><li>KDC(key distribution system)</li><li>Principal(per user per service)</li><li>A clock synchronization tool such as NTP.</li></ul><p><strong>Note:</strong>We will be taking SSH and SSL hand by hand believing on the fact that they both work on the same protocol under the hood.</p><p><strong>2.Security of Authenticators:</strong></p><p>Authenticators here are referred to as the Credentials like passwords, Secret keys etc.To demonstrate this we compare the Kerberos ticket system with the SSH password and password less communication.</p><p>SSH password authentication requires you to enter the password every time you try to login to a machine.The password is encrypted at the time of transmission but yet when it reaches the destination it is used in plain text and is present at the ssh server long enough to be compromised. Similarly in the scenario of crystallographic authentication it comes up with security and distribution problems.Having the stored keys protected by pass phrase,an adversary steals your encrypted key may subject your key to offline dictionary attack.</p><p>Kerberos on the other hand ensures that the users credentials travel the least and never outside the KDC. The.The credentials are immediately when the user identifies using the kinit command.Taking about the tickets issued per user per service are only readable by the user and the service. also they are set with an expiration time linked to them, making them even secure.The tickets present in the cache of the system are there for a shorter amount of time, but with the benefit they are secure because there are no passwords.</p><p><strong>3.Better performance:</strong> Uses Symmetric key operations that are orders of magnitude faster than Public key operations.</p><p><strong>4.Simple User management:</strong> In case of deleting a user can be directly done using the KDC and in case of SSL/SSH(keys) we have to a new certificate has to be updated at all the machines.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=320549797b0d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>