<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Punit Tiwan on Medium]]></title>
        <description><![CDATA[Stories by Punit Tiwan on Medium]]></description>
        <link>https://medium.com/@punittiwan?source=rss-78cee5366982------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*lau9Freljvg95WWmV7-P7Q.png</url>
            <title>Stories by Punit Tiwan on Medium</title>
            <link>https://medium.com/@punittiwan?source=rss-78cee5366982------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 21:03:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@punittiwan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Award is not the only way to recognise your employee]]></title>
            <link>https://medium.com/@punittiwan/award-is-not-the-only-way-to-recognise-your-employee-a6d725cce6fc?source=rss-78cee5366982------2</link>
            <guid isPermaLink="false">https://medium.com/p/a6d725cce6fc</guid>
            <category><![CDATA[company-culture]]></category>
            <category><![CDATA[people]]></category>
            <category><![CDATA[employee-engagement]]></category>
            <category><![CDATA[recognition-rewards]]></category>
            <category><![CDATA[management]]></category>
            <dc:creator><![CDATA[Punit Tiwan]]></dc:creator>
            <pubDate>Mon, 02 Jul 2018 18:31:41 GMT</pubDate>
            <atom:updated>2018-07-02T18:31:41.365Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dhKEYvg_i96QS__v." /><figcaption>Photo by <a href="https://unsplash.com/@rawpixel?utm_source=medium&amp;utm_medium=referral">rawpixel.com</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Everyone knows that to keep the employee motivated, happy and enthusiastic, you need to recognise their efforts. Mostly this recognition comes in the form of company’s annual or semi-anual award ceremony. I must say that giving an award to an employee is one the best way to keep him happy and enthusiastic. Nothing can beat that feeling of getting an award, that recognition, in front of everyone. Oh yeah!!</p><p>But how many days do you think it will take to drain out that feeling? a day? a week? or a month?<br>What about other months? How will you keep him enthusiastic and happy in those next months?</p><p>Let’s talk about another problem with this approach</p><p><strong><em>You can’t give award to everyone in the company</em></strong></p><p>So what about those other employees? Will they feel motivated to see others getting awards? They may while seeing their senior getting an award but they will surely be demotivated on seeing their colleague receiving an award. Here’s a famous quote from a popular movie 3 idiots</p><blockquote>We learnt something about human behaviour that day, It feels bad when a friend fails, but it feels even worse when he comes first.</blockquote><p>Although it is little embarrassing to admit it but it is true. People do feel little disappointed to see his/her colleague receiving an award instead of him. At that time they won’t think about the hardwork that other person must have put or what extra efforts he must have been putting in to receive such a recognition.<br>Or it could also happen that the person was very close to getting that award but there is no way of telling him that his efforts were actually mattered.</p><p>So what are the options? How can we keep the motivation going?</p><blockquote>By saying “Thank you”</blockquote><p>Value of award is not just the reward which comes with it, its what the award represent. A “Thank you” from the peer, “praise” from the manager gives the same effect.<br>These type of interactions needs to be encouraged within a company.</p><p>Let me share you a mail which i received from one of the senior management</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3VX2FVFMKRG5-G4FLMuF9Q.jpeg" /></figure><p>After reading this mail i smiled and thought “I am not just wasting my energy here. People are noticing my work and the efforts that i am putting in”<br>This is what a single mail can do to your employee. It shows that he matters, that it is not just during the appraisal process or award process that his work is evaluated.</p><p>Let me show you another example</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wXoZ_63dY-HrPscYcHfwog.jpeg" /></figure><p>In this example, people are getting recognised by their peer for their efforts and support. These type of recognitions not only improves the motivation of the employee but team spirit also. It encourage everyone to appreciate the efforts of their colleagues.</p><p>It also helps to create an environment where peers recognise each other’s effort and support them. These time to time recognitions act as a booster to the motivations and it keeps the momentum going.</p><p>If you feel that recognizing your employees is important and you’d like a tool to help make it happen, we want to introduce you to <a href="https://peoplehum.com/"><strong>peopleHum</strong></a><strong>.</strong></p><p>peopleHum is a platform where you can recognise employees at the organisation level, similar to a Wall of Fame where their tremendous efforts and contribution can be displayed. The module acknowledges 360-degree recognitions because anyone can appreciate good work regardless of their position or hierarchy. An unexpected recognition or positive feedback increases an employee’s motivation to perform actions that go above-and-beyond their normal responsibilities and tasks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6d725cce6fc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Don’t be afraid of the Storm (Part 3)]]></title>
            <link>https://medium.com/@punittiwan/dont-be-afraid-of-the-storm-part-3-bb734a3b804d?source=rss-78cee5366982------2</link>
            <guid isPermaLink="false">https://medium.com/p/bb734a3b804d</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[apache-storm]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Punit Tiwan]]></dc:creator>
            <pubDate>Thu, 26 Apr 2018 17:13:42 GMT</pubDate>
            <atom:updated>2018-04-26T17:13:42.461Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*MokOyBrg1QMeOyQG8Rsc_A.jpeg" /></figure><blockquote>This is the third part of the series on learning Apache Storm which includes description about tuple grouping. Here’s the link to <a href="https://medium.com/@punittiwan/dont-be-afraid-of-the-storm-part-1-e8193af5df22">Part 1</a> which has introduction and <a href="https://medium.com/@punittiwan/dont-be-afraid-of-the-storm-part-2-a05a6a6cae3e">Part 2</a> which has code example.</blockquote><p>In <a href="https://medium.com/@punittiwan/dont-be-afraid-of-the-storm-part-2-a05a6a6cae3e">Part 2</a>, we created a topology to read a stream of sentences and calculate counts of words. This post will cover challenges of running multiple bolts of same type and how to overcome them.</p><p>Let’s first discuss what is parallelism hint and what’s the need of it.</p><p>Parallelism hint is a concept of apache storm. It specify how many number of a particular spout/bolt should be running parallely. E.g. if a parallelism hint of 2 will specify supervisor that 2 instance of that particular spout/bolt should be running parallely in storm cluster.<br>One great thing about storm is that you can change the number of instances at runtime. You don’t have to redeploy your topology to change it.</p><p>Next question is, why do we need it?<br>It is needed when our system is not able to keep up with the incoming load i.e. incoming load is higher than what we can process. To actually solve this problem, we will first needs to find out the “bottleneck”. By “bottleneck” i mean that particular spout/bolt(s) which is not able to keep up with the incoming load. To increase the performance, we will then have to increase the number of instances of the bottleneck spout/bolt(s).</p><p>Now lets dig into our example and learn more about parallelism in storm.<br>Our topology consist of following spout and bolts:</p><ol><li>kafka-spout</li><li>sentence-splitter bolt</li><li>word counter bolt</li><li>output-bolt</li></ol><p>We will try to increase parallelism hint of each spout/bolt and analyse the effect of it. In between I will introduce the concept of tuple grouping and bolt stream.</p><h4>Increasing instances of spout</h4><p>Sometime it may happen that spout becomes the bottleneck i.e. our bolts are able to process load easily but they are not getting enough data to achieve our performance goal.</p><p>Our first instinct will be to increase the parallelism hint of spout. But don’t be surprised when you find that your performance remains the same.</p><p>This is because the <strong>spout depends on external entity</strong> for its input i.e. if your source can handle only spout then even if we increase the number to 10 or 20 it won’t make the difference.</p><p>Let’s take an example of Kafka as input source. <strong>Each spout of our topology will connect to each partition of Kafka</strong>. So if we our kafka topic has only one partition then even if you run multiple nodes of spout it won’t increase our performance.So our maximum performance depends on partition of kafka.</p><p>So increasing the performance by spouts will depend on your input source.</p><h4>Increasing instances of bolt</h4><p>Increasing the instances of bolt raises different challenges. This is because this is where we are doing processing and we may abide some storm’s principals because of our usecases. Let’s take the example of our topology.</p><p><strong>sentence-splitter bolt</strong><br>We will face no problem in increasing number of instances of “sentence-splitter” bolt because it’s just splitting the incoming sentence into words and forwarding them to next bolt.<br>It has no context of the previous sentence and does not depend on saved context to process current input.</p><p>But why it makes it easy to increase the number? What will happen if we increase the instance of a bolt which needs the previous context to process current input?</p><p>For that we need to know about different types of tuple grouping.</p><p>Let’s start with “What is a Tuple Grouping?”<br>Data gets transferred between two bolts in the form of tuple. Each tuple has a number of fields and tuple grouping specify how these tuples needs to be grouped to transfer it to other bolt.</p><p>Type of Field Grouping :</p><ol><li>Shuffle Grouping : We don’t specify field and tuples get randomly distributed between parallelly running bolts.</li><li>Field Grouping : We specify a field(s) which storm will use in hashing function to decide which tuple should go to which bolt. This way a particular bolt will always get same set of tuples.</li><li>Direct Grouping : In this grouping, producer of the tuple decides which tuple goes to which bolt.</li><li>Custom Grouping : Instead of specifying field statically, you pass on the implementation of CustomStreamGrouping and decide the distribution at the runtime.</li></ol><p>Coming back to original discussion, in our topology we have specified our grouping as “Shuffle Grouping” and the tuples can randomly go to any bolt.</p><p><strong>word-couter bolt<br></strong>Let’s take an example of our “word-couter” bolt. In that bolt we are keeping the count of each word which came since the topology is up and when new data comes in, we add that with the stored data and move the count to next bolt.</p><p>So let’s assume we have two instances of same bolt, WC1 and WC2, and we have selected shuffle grouping. Now a two tuple comes in with values</p><pre>tuple   : {&#39;word&#39;,count_in_sentence}<br>tuple 1 : {&quot;hello&quot;,1}<br>tuple 2 : {&quot;hello&quot;,1}</pre><p>storm send tuple1 to WC1 and tuple2 to WC2. When both bolt process them the output bolt will receive info that “hello” word has occured 1 time in that period of time. But in actual it occurred two twice.</p><p>This makes the perfect usecase for “Field Grouping”</p><p>There is <strong>one more usecase</strong> which fits the bill for “Field Grouping”.<br>Let’s say you suddenly got bombarded with huge amount of data from one user. Now if you use “userId” as one of your field grouping parameter then all the traffic will go into one bolt and only one set of users will get hampered and other’s data will get processed easily.</p><p>I hope by reading this article you get an understanding about tuple grouping in storm. You can comment on this post if you are facing any specific issue related to this.</p><p>Next post will be about how multiple bolts can use the same bolt’s data in their feed and when would you need that.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bb734a3b804d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dont be afraid of the Storm]]></title>
            <link>https://medium.com/@punittiwan/dont-be-afraid-of-the-storm-9ce5831bfd0c?source=rss-78cee5366982------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ce5831bfd0c</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[apache-storm]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Punit Tiwan]]></dc:creator>
            <pubDate>Wed, 25 Apr 2018 06:27:23 GMT</pubDate>
            <atom:updated>2018-04-25T06:27:23.593Z</atom:updated>
            <content:encoded><![CDATA[<h3>Don’t be afraid of the Storm (Part 1)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*lmo_qj441oAvxDpDJ-xr6A.png" /></figure><blockquote>This is the first part of the series on learning Apache Storm. Stay tuned for more.</blockquote><p>When i first heard that i have to build an platform which can handle stream of data coming in faster than a ferari , is guarenteed to scale and should be modular enough that one can increase the nodes of a particular node which is feeling most of the heat, I began to feel heat around my forehead. But fortunately for me i came to know the answer easily (but don’t tell my boss 😉 ).</p><h4>Apache Storm</h4><blockquote>It is a distributed real-time big data-processing system. Storm is designed to process vast amount of data in a fault-tolerant and horizontal scalable method.</blockquote><p>So basically Storm is designed to make our life easier by providing tools for handling stream of data and scaling while hiding the niche implmentation.</p><p>What exactly it means for us?</p><ol><li>we wont have anxiety issues about scaling</li><li>our BP will be low because of fault tolerant system</li><li>we will be calm even for high amount of streaming data</li></ol><p>and last but not the least is</p><blockquote>we can focus on our work</blockquote><p>So how does it work?</p><p>It makes use of the concept of directed acyclic graph(DAG) with spouts and bolts acts as nodes of the graph. Each edge represents data stream from one node to another. Spout and Bolt are know as <strong>Task</strong> and this whole graph is known as <strong>Topology</strong> in Storm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n74N_c-NOHD3Ix2kRWxiiQ.png" /><figcaption>Image credits to <a href="http://storm.apache.org/">http://storm.apache.org</a></figcaption></figure><p><strong>Spout</strong> acts as a provider. It receives data from external source and pass it forward to bolts. Spouts does not contain much business logic (mostly no logic at all).</p><p><strong>Bolt</strong> on the other hand contains business logic. But you don’t put whole logic in one bolt(that would violate the whole concept). They should perform a specific task and forward the data to another bolt for next task (if necessary else forward it to external source).</p><p>Nodes talk to each other using <strong>Stream</strong>. The stream is the core abstraction in Storm. A stream is an unbounded sequence of tuples that is processed and created in parallel in a distributed fashion. <strong>Tuple</strong> can contains Integer, String, Byte array or custom serializable classes.</p><p>I will explain how these things work with an example in next part of the series. For now lets talk about the Storm architecture.</p><p>Storm by default work like a cluster. Storm cluster consists of two things :</p><ul><li>nimbus : master node</li><li>supervisor : worker node</li></ul><p><strong>Nimbus</strong> acts as an orchestrator. We submit topology to nimbus and it breaks down the topology to tasks. After that it figure out which all worker nodes are free and distribute the task between them. Those worker nodes doesn’t have to be on the same server.</p><p><strong>Supervisor</strong> on the other hand acts as a worker node. It manages workers running on the same server. It gets the tasks from nimbus and distribute it to workers. Each worker has its own process.</p><p><strong>Worker</strong> wont execute those tasks by itself, they spawn threads (known as executors) and each thread execute its own task.</p><p>Storm can contain multiple nimbus instances but only one will be active at a time. If one goes down then other takes its place. This is not the same for supervisor, all supervisors remain active and gets their task from nimbus.</p><p>Storm uses Zookeeper to helps the supervisor to interact with the nimbus. Zookeeper is responsible to maintain the state of nimbus and supervisor. Since the state is available in ZooKeeper, a failed nimbus can be restarted and made to work from where it left.</p><p>In <a href="https://medium.com/@punittiwan/dont-be-afraid-of-the-storm-part-2-a05a6a6cae3e">Part 2</a> we will get our hands dirty and work on an example.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ce5831bfd0c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Don’t be afraid of the Storm (Part 2)]]></title>
            <link>https://medium.com/@punittiwan/dont-be-afraid-of-the-storm-part-2-a05a6a6cae3e?source=rss-78cee5366982------2</link>
            <guid isPermaLink="false">https://medium.com/p/a05a6a6cae3e</guid>
            <category><![CDATA[java]]></category>
            <category><![CDATA[apache-storm]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[parallel-processing]]></category>
            <dc:creator><![CDATA[Punit Tiwan]]></dc:creator>
            <pubDate>Wed, 25 Apr 2018 06:25:35 GMT</pubDate>
            <atom:updated>2018-04-26T17:14:13.908Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*g4BDW8b8MR6gxcUBamQ8yQ.jpeg" /></figure><blockquote>This is the second part of the series on learning Apache Storm which includes code example. Here’s the link to <a href="https://medium.com/@punittiwan/dont-be-afraid-of-the-storm-part-1-e8193af5df22">Part 1</a> which has introduction.</blockquote><p><em>I will be using Apache Storm 1.0.2, </em><a href="https://kafka.apache.org/quickstart"><em>Kafka</em></a><em>, Java 8, Gradle, IntelliJ IDE and Mac. I am assuming that you already have knowledge about Java, Gradle and IDE.</em></p><p>In this example we want to calculate number of repetition of different words in a stream of sentences. To achieve that we will need to do following tasks:</p><ol><li>Read sentence from Kafka</li><li>Split sentence into Map of word and count</li><li>Update the main Map with the new values.</li></ol><p>I don’t have to tell you how to implement this in a normal java application. This is a very simple problem to solve. But what if I tell you that you have to do all these tasks by using parallelism concept? It will become a trivial problem to solve.<br>That’s where <strong>Storm</strong> come’s into picture. It standardise the code structure in such a way that we just have to code our tasks in bolts and it will take care of running in parallely. We just needs to break down the problem into small tasks.<br>So without wasting anymore time, lets get to the example</p><p>First create a new project by in IntelliJ and select Gradle from left pane and Java in right. Add GroupId and ArtifactId as per your need. Set you project destination as needed.</p><p><em>I am using “com.article” as group id and “word-counter” as artifact id.</em></p><p>After the project is created, open “build.gradle”. IDE must have added some default values in that file. In “dependencies” add apache storm’s depedency by adding following lines.</p><pre>compile &#39;org.apache.storm:storm-core:1.0.2&#39;, {<br>    ext {<br>        fatJarExclude = true<br>    }<br>}</pre><blockquote>“ext” block tells gradle to <strong>not</strong> include storm-core into the final jar. We will only need this to test our code locally but we don’t need during submission to storm.</blockquote><p>Add these dependencies also</p><pre>#kafka client for integrating with storm<br>compile(&#39;org.apache.storm:storm-kafka:1.0.2&#39;)<br>compile(&#39;org.apache.storm:storm-kafka-client:1.2.1&#39;)<br>compile(&#39;com.fasterxml.jackson.core:jackson-databind:2.7.2&#39;)</pre><p>Now create folder hierarchy of src — &gt;main — &gt;java in the base folder. Add package named “com.article.word_count”.<br>Create another folder under “main” named “resources” and create a property file named “application.properties”. This will have necessary properties for our code.</p><p>Create new class “WordCountTopology” in the base package and add “public static void main” method. This will become our starter class. In this class we will construct our topology and submit it to storm.</p><p>Let’s look at the extract from the code( <a href="https://gitlab.com/tiwanpunit/word-counter">Click here</a> to see full code)</p><pre>public class WordCountTopology {<br>  public static void main(String[] args)<br>      throws InvalidTopologyException, AuthorizationException, AlreadyAliveException {</pre><pre>#1  Properties properties = <em>getResourceProperties</em>();<br><br>#2    String kafkaBootstrapServer =  properties.getProperty(&quot;kafka.bootstrap.server&quot;);<br>      String topicNameInput =   properties.getProperty(&quot;kafka.topic.input&quot;);<br>      String topicNameOutput = properties.getProperty(&quot;kafka.topic.output&quot;);<br><br>    //kafka spout configuration<br>#3  KafkaSpoutConfig&lt;String,String&gt; kafkaSpoutConfig = KafkaSpoutConfig.<em>builder</em>(kafkaBootstrapServer,topicNameInput).build();<br>#4  KafkaSpout&lt;String,String&gt; kafkaSpoutInput = new KafkaSpout&lt;&gt;(kafkaSpoutConfig);<br><br>    //Start building topology<br>#5  TopologyBuilder builder = new TopologyBuilder();<br><br>    //This will add spout to topology i.e. Kafka reader<br>#6  builder.setSpout(&quot;kafka-spout&quot;, kafkaSpoutInput, 1);<br><br>    //This will add bolt to topology which read output from spout<br>#7 builder.setBolt(&quot;sentence-splitter&quot;,new SentenceSplitterBolt(),1)<br>#8        .shuffleGrouping(&quot;kafka-spout&quot;);<br><br>#9 builder.setBolt(&quot;word-counter&quot;, new WordCounterBolt(), 1)<br>           .shuffleGrouping(&quot;sentence-splitter&quot;);<br><br>#10 KafkaBolt outputBolt = new KafkaBolt()<br>      .withProducerProperties(<em>readKafkaProperties</em>(properties))<br>      .withTopicSelector(new DefaultTopicSelector(topicNameOutput))<br>    .withTupleToKafkaMapper(new FieldNameBasedTupleToKafkaMapper());<br><br>#11 builder.setBolt(&quot;output-bolt&quot; , outputBolt , 1)<br>           .shuffleGrouping(&quot;word-counter&quot;);<br><br>#12 Config config = new Config();<br>    config.setDebug(false);<br>    config.setMessageTimeoutSecs(30);<br>    config.put(Config.<em>TOPOLOGY_MAX_SPOUT_PENDING</em>, 1000);<br>    config.put(Config.<em>TOPOLOGY_ACKER_EXECUTORS</em>,1);<br><br>#13    StormSubmitter.<em>submitTopology</em>(<br>           &quot;WordCountTopology&quot;,<br>           config,<br>           builder.createTopology());<br>  }<br>}</pre><p>Let’s go through the code :</p><ol><li>We are reading properties from property file</li><li>We are assigning kafka and zookeeper property values to constants</li><li>Storm developers has already created Spout and Bolt which can be used to connect to Kafka. We just need to initialise them configuration. So here, we are creating the KafkaSpoutConfig with Kafka server, input topic. Typecasting specify the input and output format of that spout.</li><li>We are creating Kafka spout with configuration</li><li>We are initialising TopologyBuilder. This will be used to construct our topology. By using this object we will setup bolt and spout arrangement.</li><li>We are setting up the <strong>spout</strong> for topology. Lets look at the arguments of “setSpout”:<br>* first argument specify unique name of the <br>* second argument specify object of spout<br>*third argument specify how many spouts should run in parallel</li><li>We are setting up Sentence Splitter <strong>bolt</strong> which will convert the incoming sentence to Map of word and count. Method argument of “setBolt” is same as “setSpout”.</li><li>The difference comes here. After setting up the “bolt” we will need to specify how this bolt will get its messages i.e how data will flow <strong>in</strong> to the bolt. For that we will need to specify <strong>name</strong> of the spout/bolt from which this bolt will read the messages. Here we have specified “kafka-spout” (name of our spout) as the feeder by calling “shuffleGrouping” method(We will discuss different types of grouping in later posts)</li><li>Here we are adding next bolt which will accumulate incoming Map of word and count and pass it into the next bolt as final result. This will have the total count of each word which came during the time our code was running. This bolt will read messages from “sentence-splitter” bolt.</li><li>Here we are reading the properties of kafka to create configuration of Kafka Bolt.</li><li>This bolt will read messages from “word-counter” bolt.</li><li>We are creating Config object to be used to set storm properties. These are some properties which I generally used. You can read more about these in apache documentation.<br>Another usecase of this config is to send your custom properties to bolts e.g. db url, redis-node info etc.<br>Note: This can be empty also if u want to use storm’s default properties.</li><li>Here we are submitting the topology to storm. Let’s look at the arguments:<br>* first we need to provide unique name for a topology<br>* second we need to provide storm configuration<br>* third we need to provide our topology object</li></ol><p>Let’s look at the code for “sentence-splitter” bolt. I won’t bore you with business logic but will provide you details which will be needed to create a bolt.</p><pre>public class SentenceSplitterBolt extends BaseRichBolt {<br><br>  OutputCollector collector;<br><br>  @Override<br>  public void prepare(Map stormConf, TopologyContext context, OutputCollector collector) {<br>    this.collector = collector;<br>  }<br><br>  @Override<br>  public void execute(Tuple input) {<br>    String sentence = input.getString(0);<br>    if (sentence != null) {<br>      System.<em>out</em>.println(&quot;Sentence: &quot; + sentence);<br>      String[] wordArray = sentence.split(&quot; &quot;);<br>      Map&lt;String, Integer&gt; wordMap = new HashMap&lt;&gt;();<br>      for (String word : wordArray) {<br>        Integer count = wordMap.get(word);<br>        if (count == null) {<br>          count = 0;<br>        }<br>        count++;<br>        wordMap.put(word, count);<br>      }<br>      // send the constructed Map to next bolt<br>      collector.emit(new Values(wordMap));<br>    }<br>    // acknowledge that the processing of this tuple is finished<br>    collector.ack(input);<br>  }<br><br>  @Override<br>  public void declareOutputFields(OutputFieldsDeclarer declarer) {<br>    declarer.declare(new Fields(&quot;wordMap&quot;));<br>  }<br>}</pre><p>To create a bolt class you can do two things:</p><ol><li>You can implement the bolt interface “IRichBolt”</li><li>You can extend the “BaseRichBolt”</li></ol><p>I am going with option 2 as it takes care of implementing methods which we don’t generally customize.</p><p>You will then need to Override three methods:</p><ol><li><strong>prepare</strong><br>This method will be called once in bolt’s lifetime. First parameter has the properties which is sent during creation of topology. Second parameter is the topology context and third is OutputCollector. OutputCollector will be used to acknowledging the processing of a tuple and sending the data to next tuple.</li><li><strong>execute</strong><br>This will be called each time a new tuple(i.e. message) is queued for this bolt. This is the method where will need to do our processing. We also need to do two more things in this method:<br>* acknowledge the tuple that came into this method. This notify storm that this tuple is successfully processed.<br>* emit(i.e. send) the data to next bolt if needed. We can do that by sending “Values” object. Values constructor takes variable number of “object” arguments. Note: we need to send the values in same order as we have defined in the schema in “<strong>declareOutputFields”</strong></li><li><strong>declareOutputFields</strong><br>This method will be called to know the format in which this bolt will transmit the data. We will need to use the passed object of OutputFieldsDeclarer to declare the fields. We can do that by creating “Fields” object. Fields constructor takes variable number of “string” arguments which will define the schema.<br>Note: We will need to send the values in same order as we are declaring them.</li></ol><p>To test your code locally, you will need to replace #13 with this</p><pre>LocalCluster localCluster = new LocalCluster();<br>localCluster.submitTopology(&quot;WordCountTopology&quot;,new Config(),builder.createTopology())</pre><p>This will create a local cluster of storm for you at runtime and submit the topology to it. That way you can test your code easily before submitting it to remote server.</p><p>After you have test your code, you will have to create your <strong>jar</strong> which you will submit to storm. I have added plugin “fatjar” in gradle file which i will be using to create that jar by executing this command in source folder of our project</p><pre>gradle build fatjar</pre><p>Before submitting the topology, you will need to specify “nimbus” host address in “storm.yaml” present in “conf” folder of your local apache storm directory. You can connect submit your topology by running the command locally by specifying nimbus host.<br>You can do that by adding this line to the config</p><pre>nimbus.seeds: [&quot;{ip-address}&quot;]</pre><p>Now go to the “bin” folder of your apache storm directory and execute following command</p><pre>#1    #2  #3            #4<br>storm jar {path of jar} com.article.word_count.WordCountTopology</pre><p>#1 “storm” is a script present in bin folder which take care of uploading our topology to storm cluster.</p><p>#2 “jar” is an argument which tells the script that we want to upload the topology</p><p>#3 we need to provide path of the jar file</p><p>#4 we need to specify classpath of our “main” class which takes care of building topology</p><p>I hope by reading this article you will be able to create your own topology and run it on storm. If you have any questions/suggestions, please comment and I will revert.</p><p>In the <a href="https://medium.com/@punittiwan/dont-be-afraid-of-the-storm-part-3-bb734a3b804d">Part 3</a> we will discuss the challenges in increasing the parallelism hint and how can we overcome them by using different type of “grouping”. We will also discuss about how multiple bolts can use the same bolt’s data in their feed.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a05a6a6cae3e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Don’t be afraid of the Storm (Part 1)]]></title>
            <link>https://medium.com/@punittiwan/dont-be-afraid-of-the-storm-part-1-e8193af5df22?source=rss-78cee5366982------2</link>
            <guid isPermaLink="false">https://medium.com/p/e8193af5df22</guid>
            <category><![CDATA[data-streaming]]></category>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[apache-storm]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Punit Tiwan]]></dc:creator>
            <pubDate>Fri, 18 Aug 2017 10:18:07 GMT</pubDate>
            <atom:updated>2018-01-16T04:40:09.927Z</atom:updated>
            <content:encoded><![CDATA[<h3>Don’t be afraid of the Storm (Part 1)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/0*kl306zxsDzBG6_Oy.jpg" /></figure><blockquote>This is the first part of the series on learning Apache Storm. This includes introduction and will include examples in next parts</blockquote><p>When I first heard that I have to build a platform which can handle a stream of data coming in faster than a Ferrari , is guaranteed to scale and should be modular enough that one can increase the nodes of a particular node which is feeling most of the heat, I began to feel heat around my forehead. But fortunately for me, I came to know the answer easily (but don’t tell my boss 😉 ).</p><h4>Apache Storm</h4><blockquote>It is a distributed real-time big data-processing system. Storm is designed to process vast amount of data in a fault-tolerant and horizontally scalable method.</blockquote><p>So basically it is designed to make our life easier by providing tools for handling the stream of data and scaling while hiding the implementation.</p><p>What does it mean for us?</p><ol><li>We won’t have anxiety issues about scaling</li><li>our BP will be stable because of fault tolerant system</li><li>we will be calm even with huge amount of streaming data</li></ol><p>and last but not the least “<strong>we can focus on our work</strong> 🙂”</p><h4>So how does it work?</h4><p>Apache Storm makes use of the concept of directed acyclic graph(DAG) with spouts and bolts acts as nodes of the graph. Each edge represents data stream from one node to another. Spout and Bolt are known as <strong>Task</strong> and this whole graph is known as <strong>Topology</strong> in Storm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6Ri3TvMKd1iO2BQQ.png" /></figure><p><strong>Spout</strong> acts as a provider. It receives data from the external source and passes it forward to bolts. Spouts do not contain much business logic (mostly no logic at all).</p><p><strong>Bolt</strong>, on the other hand, contains business logic. But you don’t put whole logic in one bolt(that would violate the whole concept). They should perform a specific task and forward the data to another bolt for next task (if necessary else forward it to the external source).</p><p>By making spout and bolt stateless, we make full use of storm’s concept of <strong>“parallelism hint</strong>”. This property tells storm, how many instances of a particular node (spout or bolt) should be running simultaneously. Each node can have its own value and can be set at runtime.</p><p>Nodes talk to each other using <strong>Stream</strong>. The stream is the core abstraction in Storm. A stream is an unbounded sequence of tuples that is processed and created in parallel in a distributed fashion. <strong>Tuple</strong> can contain Integer, String, Byte array or custom serializable classes.</p><p>I will explain how these things work with an example in next part of the series. For now, let’s talk about the Storm architecture.</p><h4>Apache Storm Architecture</h4><p>Storm by default work like a cluster and it mainly consists of two things :</p><ul><li>nimbus: master node</li><li>supervisor: worker node</li></ul><p><strong>Nimbus</strong> acts as an orchestrator. We submit topology to nimbus and it breaks down the topology to tasks. After that, it figures out which all worker nodes are free and distributes the task between them. Those worker nodes don’t have to be on the same server.</p><p><strong>Supervisor</strong>, on the other hand, acts as a worker node. It manages workers running on the same server. It gets the tasks from nimbus and distributes it to workers. Each worker has its own process.</p><p><strong>Worker</strong> won’t execute those tasks by itself, they spawn threads (known as executors) and each thread executes its own task.</p><p>Storm can contain multiple nimbus instances but only one will be active at a time. If one goes down, other takes its place. This is not the same for supervisor, all supervisors remain active and get their task from nimbus.</p><p>It uses Zookeeper to help the supervisor to interact with the nimbus. Zookeeper is responsible for maintaining the state of nimbus and supervisor. Since the state is available in ZooKeeper, a failed nimbus can be restarted and made to work from where it left.</p><p>Here’s the link for a step by step guide for <a href="http://storm.apache.org/releases/1.1.1/Setting-up-a-Storm-cluster.html">creating storm cluster</a>.</p><p>That’s all about the introduction, in the next part we will get our hands dirty and work on an example.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e8193af5df22" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simple steps to develop leadership skills]]></title>
            <link>https://medium.com/@punittiwan/simple-steps-to-develop-leadership-skills-cdd7a536736b?source=rss-78cee5366982------2</link>
            <guid isPermaLink="false">https://medium.com/p/cdd7a536736b</guid>
            <category><![CDATA[leadership-development]]></category>
            <category><![CDATA[life-lessons]]></category>
            <category><![CDATA[leadership]]></category>
            <dc:creator><![CDATA[Punit Tiwan]]></dc:creator>
            <pubDate>Thu, 18 Feb 2016 07:30:12 GMT</pubDate>
            <atom:updated>2016-02-18T07:39:04.076Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*gKCTKpSC8M-aWEZG3muhNw.png" /></figure><p>In MaaS360 (one of the security divisions of IBM), people at the director level hold various sessions (technical as well as non-technical) where we get opportunities to learn from their experience. Recently, I got an opportunity to be in a session where one of our directors, <strong>Abhijit Bhide, </strong>shared his experience about what it means to be a leader and how one can develop leadership skills.</p><p>When I got the invitation for this session, I was a little confused as to how relevant it will be for me as I am a developer with only 2 years of experience and who is not leading any team. But I went for the session anyways as I had heard that these sessions were really informative. This session totally blew me away and all thanks to Abhijit for making this session simple yet interesting.</p><p>This was originally scheduled to be an hour-long session but it became 2 sessions of one and a half hour each. After only the first session, I was so excited that I decided to share what I learnt from this session with everyone. Now, enough with the build-up, let us get to the point.</p><p>When you think about leadership, the first thought that comes into your mind is leading a team or a company. But it is not only about that. It is also about leading yourself to achieve goals that you have set for yourself. It not only applies to your professional life only but also to your personal life. I will describe different techniques and give examples where we can see how our professional as well as personal life are affected by it.</p><p>Leader should posses following qualities (not sorted by importance) :</p><ul><li><strong>Energy</strong> : A leader should be energetic who can lift himself and the team when things are not going according to plan.</li><li><strong>Ownership</strong> : A leader should take responsibility and be accountable for not only on what he worked upon but for the whole product which is going to be delivered to the customer.</li><li><strong>Curiosity</strong> : A leader should be very curious person. He should keep tabs on things related to his work, some in depth and some just to know.</li><li><strong>Understanding</strong> : A leader should have opinion about things which can affect his work directly or indirectly. To achieve that, he needs to have knowledge about the things which can affect his work directly or indirectly.</li><li><strong>Connection</strong> : A leader should be able to connect with the people around him, not only with people working under him but also with his peers and seniors.</li><li><strong>Organization</strong> : A leader should be very organized. This is very important as this can help you to develop other qualities.</li><li><strong>Followthrough</strong> : A leader shouldn’t be reactive but responsive. He should also take commitments seriously.</li></ul><p>Lets discuss each quality in detail and also discuss how one can develop this quality.</p><h3>Energy</h3><p>You should have that feeling that you can perform a particular task or you can take on that problem with enthusiasm. You should feel passionate about the thing which you are doing.</p><h4>Why is this needed?</h4><p>You will feel unhappy if you are not feeling energized while doing your work which will lead to negative effect on your work and personal life. If you lack energy then your team’s enthusiasm will also go down.</p><h4>What are the techniques you can use to develop energy?</h4><ul><li><strong>3 step technique</strong> : Think of things which can lift you when you are down e.g. listening to songs, talking to someone etc. Identify those things and distribute them to 3 parts: start, mid and end of the day. You should feel energetic when you start the day and end the day. So things which will lift you the most should be at the start or at the end of the day.</li><li>Create energy forums of like-minded people and initiate them yourself. If you think something can energize you, don’t wait for someone else to initiate it.</li><li>Find individuals who make you happy and give you energy and seek them out for debates and discussions.</li></ul><h3>Ownership</h3><p>First level of ownership is the work that has been given to you. But you need to think of ownership at a larger level also. You should think that the product on which you are working on is your ownership. By ownership I mean you should think how can we improve the whole product and take necessary action.</p><h4>Why is this important?</h4><p>Until you perform your responsibility properly no one is going to take you seriously. Another kind of ownership is your commitment to others. You should also fulfill your commitments that you make to others e.g. if you have told someone that you will get back on a particular thing and forget it then the other person won’t take your work seriously again.</p><h4>What technique you can to develop this quality?</h4><p>RACI (Responsibility, Accountability, Consulted and Information)</p><ul><li>You should take Responsibility of the work you have been doing.</li><li>You should be Accountable for the product on which you are working on.</li><li>You should only Consult to the work that you have delegated to the others. You should not micromanage that.</li><li>You should have Information about the work that is not directly connected to your work but affects your product as whole.</li></ul><h3>Curiosity</h3><p>You should have that zeal to learn things which are in your domain as well as things which are a little-bit related to your work. E.g. if you are working in iOS development then you should also be curious about the latest trends in iOS technology as well as different trends in UX etc.</p><h4>Why is this important?</h4><p>If you are curious then only will you be able to learn something new and grow your knowledge base. If you have more knowledge then you will be respected by others. This will also help in the things which you are working on.</p><p>It will also broaden the topics of discussion with other people. Due to which people will find you easy to talk and relate to.</p><h4>How can you increase curiosity(or say knowledge)?</h4><ul><li>Organize curiosity</li></ul><p>First choose what is the area you are (or needs to be) curious about e.g. iOS application development. Now break that down into tasks that needs to be done so that you can know about that thing e.g. iOS programming, design methodology, deployment technique etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/972/1*_wFqbuNAMiGomeEbsL68kQ.png" /></figure><p>Create circles (like earth crust), the innermost one represent task which are necessary ones and as you go out you reduce the level of necessity. You will need to have atleast 3 circles. Put tasks(which you computed earlier) in necessary circles.</p><p>Now prioritize tasks within circles and start thinking how you can get more information about these tasks.</p><blockquote>One interesting fact is that, when you learn more about a particular thing then it will automatically be removed from inner circle and new thing will come. Think about this.</blockquote><h3>Understanding</h3><p>When you are asked about your opinion regarding a subject in your work-area then you must have a reasonable answer. If you can answer, that means you have good understanding in that field.</p><h4>Why is this important?</h4><ul><li>Generally in a meeting or discussion, we fear to voice our opinions. One reason can be stage fear; that people will laugh at me. In reality, people are generally interested to know different opinions in a meeting. Another reason is that we don’t have confidence in our opinions. That can happen if you have less understanding in the subject.</li><li>If you have good understanding in your field then people will want to make you a part of their discussions thinking that you can be helpful.</li></ul><h4>Techniques for increasing understanding</h4><p>As you become more curious, you level of understanding will increase. You will understand more about things which are in the inner circle and less as we move out. But you won’t understand anything if that thing is not in your circle. So you need to increase your curiosity to increase your understanding.</p><h3>Connection</h3><p>By connection I am not talking about networking but feeling connected with the people you are working with. You should be able to talk to them without any hesitation and vice versa. You should also praise the people when they perform applaud able work. Sometime a pat on the back or asking them how they have done it makes a difference.</p><h4>Why is this important?</h4><p>Lets divide this into categories:</p><ul><li><strong>connection with peer</strong> : you should be comfortable with the person you are working with. If you are not connecting with that person then you won’t feel happy working with them and it will be hard to communicate ideas.</li><li><strong>connection with seniors</strong>: If there is anything that is bugging you then you should talk to your manager as it will give him some feedback about his own work and he will be able to help you. If you keep your grudges and just talk to him during year-end review then he will feel that you are doing only because you want to increase your salary or hide your failures.</li><li><strong>connection with people working under you</strong> : If they feel shy or restricted while talking to you then you won’t be able to understand what your team really wants and where you can improve. Also if you talk to them during year-end review and won’t give them time to improve then it will be unfair to them.</li></ul><h4>How to improve connections?</h4><p>You can connect at four levels :</p><ul><li><strong>casual talk</strong> : just have some chit-chat during coffee break, outings etc.</li><li><strong>one to one</strong> : generally one to one becomes very formal talk. Try to make it informal and you will find how much the other person will value this.</li><li><strong>Group Discussion </strong>: find similar minded people and talk with them so that they can also connect with each other. This will bring harmony in your team.</li><li><strong>Hard Discussion </strong>: This is one of those things which we tend to avoid. But to be successful as a leader, you will need to find a way to give some hard news to other people. By hard news I don’t mean salary discussion but giving positive feedback using which they can improve.</li></ul><h3>Organization</h3><p>You should keep tabs on things which are going on with your life or work. If you write things down which you need to do then you won’t forget anything.</p><h4>Why is this important?</h4><p>This is one of the important qualities. If you can organize your tasks and your life, then you will be able to manage things very easily.</p><p>Also, every technique that I mentioned earlier won’t be executed successfully until you are good at this. E.g. If you want to be curious then you need to make a list of things that you need to learn and keep tabs on the list. How will you do that? By becoming more organized. If you want to improve your ownership then you will need to keep tabs on the tasks that you are performing and also of the tasks which others are performing. How will you do that without becoming organized.</p><h4>How can i be organized?</h4><ul><li><strong>15–15–30 min rule</strong></li></ul><p>use 15 min daily morning to reflect what needs to be done</p><p>use 15 min daily evening to reflect what has been done</p><p>use 30 min weekly to review your week and prepare for next one</p><p>Create the matrix which has the execution plan for the priorities and track it in your organization tool</p><ul><li><strong>30 seconds rule</strong></li></ul><p>After every meeting or session, sit for 30 seconds and summarize the meeting. 30 seconds will give you enough time to write but it will also restrict you to write things which are not very important. You can find more about this in medium post</p><p><a href="https://medium.com/swlh/the-30-second-habit-with-a-lifelong-impact-2c3f948ead98#.i5jo8k6jk">https://medium.com/swlh/the-30-second-habit-with-a-lifelong-impact-2c3f948ead98</a></p><ul><li>There are various techniques to be organized. You will find hundreds of medium posts related to this.</li></ul><h3>Followthrough</h3><p>You should not be reactive but responsive. You should not work on things when they become very urgent. You should plan things ahead and have a plan to tackle the problem.</p><h4>Why is this important?</h4><p>If you fix any issues while you are panicking then you will just focus on one possibility of occurring that issue. But if you face that issue with cool and clear mind then chances of solving that problem with more ease and accuracy will be higher.</p><p>It will also increase your credibility as you will be delivering items on time.</p><h4>Techniques to be used</h4><p>If you are organized and keep honest with the list you have made then its done.</p><p>I would like to end this by saying, just learning these techniques is not important but applying those in your daily life is. I am also hopping to do the same.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cdd7a536736b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>