<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ana Suzuki on Medium]]></title>
        <description><![CDATA[Stories by Ana Suzuki on Medium]]></description>
        <link>https://medium.com/@minyodev?source=rss-e26380dd93fa------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*-Hh9kc9lv-5limoTJlbmgA.jpeg</url>
            <title>Stories by Ana Suzuki on Medium</title>
            <link>https://medium.com/@minyodev?source=rss-e26380dd93fa------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 07:48:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@minyodev/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Apache Flink on YARN with Kerberos Authentication]]></title>
            <link>https://minyodev.medium.com/apache-flink-on-yarn-with-kerberos-authentication-adeb62ef47d2?source=rss-e26380dd93fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/adeb62ef47d2</guid>
            <category><![CDATA[yarn]]></category>
            <category><![CDATA[apache-flink]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[kerberos]]></category>
            <dc:creator><![CDATA[Ana Suzuki]]></dc:creator>
            <pubDate>Tue, 17 Sep 2019 09:01:03 GMT</pubDate>
            <atom:updated>2022-05-23T08:16:54.359Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*2j6QHS1PcUpdc9CT1GVYkQ.png" /></figure><p>Setting up Flink on YARN is written pretty much straight forward on the <a href="https://ci.apache.org/projects/flink/flink-docs-stable/ops/deployment/yarn_setup.html">documentation</a>. But what if what you need to do is much more complex, say what if run Flink application on a secured YARN cluster, and consume records from a secured Kafka cluster?</p><p>The scope of this post is to give an example on this kind of situation. As a person coming from Apache Spark’s perspective — the way to do this on Apache Flink is quite different approach.</p><p>DISCLAIMER: There could be different solutions I’m not aware of. Though this is the only way was able to set it up. If you have any suggestions or other way to do it please let me and/or other users know! :)</p><p>Prerequisites:</p><ul><li>Apache Flink 1.9.0</li><li>Secured Kafka Cluster</li><li>Secured YARN Cluster</li></ul><p>Setup:</p><ul><li>Kafka Configuration — set the following <strong>programmatically</strong>:</li></ul><pre>security.protocol: &quot;SASL_PLAINTEXT&quot;<br>sasl.mechanism: &quot;GSSAPI&quot;<br>sasl.kerberos.service.name: &quot;kafka&quot;</pre><ul><li>Create a folder named /home/foouser/important_files, and put the following files on this directory:</li><li>1. kafka_client_jaas.cof</li><li>2. keytab file</li><li>3. krb5.conf</li><li>Set up <strong>flink-1.9.0/conf/flink-conf.yaml</strong> and point kafka_client_jaas.conf and krb5.conf configuration to the same path:</li></ul><pre>env.java.opts.jobmanager: -Djava.security.krb5.conf=/home/foouser/important_files/krb5.conf -Djava.security.auth.login.config=/home/foouser/important_files/kafka_client_jaas.conf<br>env.java.opts.taskmanager: -Djava.security.krb5.conf=/home/foouser/important_files/krb5.conf -Djava.security.auth.login.config=/home/foouser/important_files/kafka_client_jaas.conf</pre><ul><li>Flink command:</li></ul><pre>HADOOP_CLASSPATH=`hadoop classpath` /home/foouser/flink-1.9.0/bin/flink run \<br>-m yarn-cluster \<br>-yqu some_yarn_queue \<br>-ynm &quot;Flink on YARN&quot; \<br>-yt /home/foouser/important_files \<br>-p 4 -yjm 1024m -ytm 1024m \<br>--class HelloWorld \<br>/home/foouser/foo.jar</pre><p>Take note what you have to load the FOLDER_NAME upon running Flink job. If you look into YARN UI and check container.sh, you will see the value of FOLDER_NAME there.</p><p>To check if files are loaded on executors:</p><pre>ls /home/foouser/important_files<br><br># Inside &#39;important_files&#39; directory<br>krb5.conf<br>my.keytab<br>kafka_client_jaas.conf</pre><p>You will see on YARN UI &gt; application_id &gt; logs &gt; container.sh</p><pre>...<br>important_files/krb5.conf<br>important_files/my.keytab<br>important_files/kafka_client_jaas.conf<br>...</pre><p>That’s all. Happy coding!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=adeb62ef47d2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Spark Custom Logging]]></title>
            <link>https://minyodev.medium.com/apache-spark-custom-logging-da2e42fca6e?source=rss-e26380dd93fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/da2e42fca6e</guid>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[apache-spark]]></category>
            <dc:creator><![CDATA[Ana Suzuki]]></dc:creator>
            <pubDate>Tue, 11 Jun 2019 02:19:14 GMT</pubDate>
            <atom:updated>2019-06-11T02:19:14.871Z</atom:updated>
            <content:encoded><![CDATA[<p>The example below is only applicable if your spark job runs on yarn deployed on client.</p><p>Say we want to have a different logging for executor and driver, with the first one having normal logging, and the later being less verbose.</p><p>With this being said, we will prepare 2 files:</p><ul><li>log4j_executor.properties</li></ul><pre>log4j.rootLogger=WARN, stdout</pre><pre>log4j.appender.stdout=org.apache.log4j.ConsoleAppender<br>log4j.appender.stdout.layout=org.apache.log4j.PatternLayout<br>log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %p/%c{1}:%L - %m%n</pre><ul><li>log4j_driver.properties</li></ul><pre>log4j.rootLogger=WARN, stdout</pre><pre>log4j.appender.stdout=org.apache.log4j.ConsoleAppender<br>log4j.appender.stdout.layout=org.apache.log4j.PatternLayout<br>log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss,SSS} %p/%c{1}:%L - %m%n</pre><pre># print all information from my project<br>log4j.logger.com.foo=INFO</pre><pre># set <br>log4j.logger.org.apache.spark=WARN<br>log4j.logger.org.apache.spark.cluster=ERROR<br>log4j.logger.org.apache.spark.scheduler.cluster=ERROR<br>log4j.logger.org.apache.hadoop=ERROR<br>log4j.logger.io.netty=ERROR<br>log4j.logger.org.apache.zookeeper=ERROR</pre><pre>log4j.logger.org.apache.spark.streaming.scheduler.JobScheduler=INFO</pre><p>Upon submitting spark job, set it to run on YARN as client mode. Add -Dlog4j.configuration=&lt;path-to-configuration file&gt; to spark.driver.extraJavaOptions (for the driver) and spark.executor.extraJavaOptions (for executors).</p><p>Note that the <strong>file:// protocol should be explicitly provided</strong>, else it’s not going to work properly.</p><pre>/home/foo/spark-2.3.3-bin-hadoop2.7/bin/spark-submit<br>--master yarn<br>--deploy-mode client<br>...<br>--files /home/foo/my-sample-project/conf/log4j_executor.properties<br>--conf spark.driver.extraJavaOptions=&quot;&quot;-Dlog4j.configuration=file:///home/foo/my-sample-project/conf/log4j_driver.properties&quot;<br>--conf spark.executor.extraJavaOptions=&quot;&quot;-Dlog4j.configuration=log4j_executor.properties&quot;<br>...<br>--class com.foo.mysampleproject.MainClass<br>/home/foo/my-sample-project/target/scala-2.11/mysampleproject-1.0.0.jar</pre><p>Reference: <a href="https://spark.apache.org/docs/2.3.3/running-on-yarn.html">https://spark.apache.org/docs/2.3.3/running-on-yarn.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da2e42fca6e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Scala: List and Iterator Simulation]]></title>
            <link>https://minyodev.medium.com/scala-list-and-iterator-simulation-4a3314b211fb?source=rss-e26380dd93fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/4a3314b211fb</guid>
            <category><![CDATA[lists]]></category>
            <category><![CDATA[scala]]></category>
            <category><![CDATA[iteration]]></category>
            <category><![CDATA[collection]]></category>
            <dc:creator><![CDATA[Ana Suzuki]]></dc:creator>
            <pubDate>Mon, 27 May 2019 06:50:00 GMT</pubDate>
            <atom:updated>2019-05-27T06:50:00.768Z</atom:updated>
            <content:encoded><![CDATA[<p>A simple simulation on the difference between list and iterator. We all know that in Scala collection’s lineage, Iterable is very much the class being extended on other known collections.</p><p><strong>List</strong> finishes all elements before going on to next iteration.</p><pre>scala&gt; (0 to 5)<br>   .toList<br>   .map(x =&gt; { <br>      println(&quot;Sleeping...&quot;); <br>      Thread.sleep(2000L); <br>      println(&quot;Slept!&quot;); <br>      x;<br>   })<br>   .map(x =&gt; { <br>      println(s&quot;Adding 1 to $x..&quot;); <br>      x + &quot;1&quot;; })<br>   .foreach(println)</pre><pre>Sleeping...<br>Slept!<br>Sleeping...<br>Slept!<br>Sleeping...<br>Slept!<br>Sleeping...<br>Slept!<br>Sleeping...<br>Slept!<br>Sleeping...<br>Slept!<br>Adding 1 to 0..<br>Adding 1 to 1..<br>Adding 1 to 2..<br>Adding 1 to 3..<br>Adding 1 to 4..<br>Adding 1 to 5..<br>01<br>11<br>21<br>31<br>41<br>51</pre><p>Iterator works per element as it goes through the next iteration once it passed on the first one.</p><pre>scala&gt; (0 to 5)<br>   .toIterator<br>   .map(x =&gt; { <br>      println(&quot;Sleeping...&quot;); <br>      Thread.sleep(2000L); <br>      println(&quot;Slept!&quot;); <br>      x; <br>   })<br>   .map(x =&gt; { <br>      println(s&quot;Adding 1 to $x..&quot;); <br>      x + &quot;1&quot;; <br>   })<br>   .foreach(println)</pre><pre>Sleeping...<br>Slept!<br>Adding 1 to 0..<br>01<br>Sleeping...<br>Slept!<br>Adding 1 to 1..<br>11<br>Sleeping...<br>Slept!<br>Adding 1 to 2..<br>21<br>Sleeping...<br>Slept!<br>Adding 1 to 3..<br>31<br>Sleeping...<br>Slept!<br>Adding 1 to 4..<br>41<br>Sleeping...<br>Slept!<br>Adding 1 to 5..<br>51</pre><p>I’m not going to say 1 is good over the other since it depends on where do you want to use it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4a3314b211fb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AVRO vs Parquet — what to use?]]></title>
            <link>https://minyodev.medium.com/avro-vs-parquet-what-to-use-518ccbe8fb0c?source=rss-e26380dd93fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/518ccbe8fb0c</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[apache-avro]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[avro]]></category>
            <category><![CDATA[file-format]]></category>
            <dc:creator><![CDATA[Ana Suzuki]]></dc:creator>
            <pubDate>Wed, 16 Jan 2019 08:44:49 GMT</pubDate>
            <atom:updated>2021-06-08T07:48:48.849Z</atom:updated>
            <content:encoded><![CDATA[<h3>AVRO vs Parquet — what to use?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TyteRkt1WsgKF7o63-lKaw.jpeg" /></figure><p>I won’t say one is better and the other one is not as it totally depends where are they going to be used.</p><p><strong>Apache Avro</strong> is a remote procedure call and data serialization framework developed within Apache’s Hadoop project. It uses JSON for defining data types and protocols, and serializes data in a compact binary format. (by <a href="https://en.wikipedia.org/wiki/Apache_Avro">Wikipedia</a>)</p><ul><li>Since it’s a <strong>row based format</strong>, it’s better to use when all fields needs to be accessed</li><li>Files support block compression and are splittable</li><li>Suitable for write intensive operation</li></ul><p><strong>Apache Parquet</strong>, on the other hand,<strong> </strong>is a free and open-source column-oriented data storage format of the Apache Hadoop ecosystem. It is similar to the other columnar-storage file formats available in Hadoop namely RCFile and ORC. It provides efficient data compression and encoding schemes with enhanced performance to handle complex data in bulk. (by <a href="https://en.wikipedia.org/wiki/Apache_Parquet">Wikipedia</a>)</p><ul><li>Since it’s a <strong>column based format</strong>, it’s better to use when you only need to access specific fields</li><li>Each data file contains the values for a set of rows</li><li>Can’t be written from streaming data since it needs to wait for blocks to get finished. However, this will work using micro-batch or bulk sink</li><li>Suitable for data exploration — read intensive, complex or analytical querying, low latency data</li></ul><p>Both supports the following but depends on certain degree:</p><ul><li>Schema evolution</li><li>Nested datasets</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=518ccbe8fb0c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Git Command Cheatsheet]]></title>
            <link>https://minyodev.medium.com/git-command-cheatsheet-40e1d9a03590?source=rss-e26380dd93fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/40e1d9a03590</guid>
            <category><![CDATA[bash]]></category>
            <category><![CDATA[git]]></category>
            <dc:creator><![CDATA[Ana Suzuki]]></dc:creator>
            <pubDate>Tue, 11 Dec 2018 08:07:28 GMT</pubDate>
            <atom:updated>2021-07-03T23:19:06.569Z</atom:updated>
            <content:encoded><![CDATA[<p>You’re welcome!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*nHYKQEouL1o0vxEHZsWz5Q.gif" /><figcaption><a href="https://www.google.co.jp/url?sa=i&amp;source=images&amp;cd=&amp;ved=2ahUKEwjY5azXqJffAhVQdt4KHQKACksQjhx6BAgBEAM&amp;url=https%3A%2F%2Fgifer.com%2Fen%2FAvq1&amp;psig=AOvVaw3TqIcT9Xl64jFg0n5zzHMf&amp;ust=1544601919236916">Exciting GIF on GIFER — by Duzil</a></figcaption></figure><p>Git clone with a specific branch — just like starting new challenge in life:</p><pre>git clone -b develop <a href="https://ts-ana.esguerra@git.rakuten-it.com/scm/rat/khone.git">https://john.doe@gitfoo.com/scm/rat/khone.git</a></pre><p>Git merge — just like when Goku and Vegeta merge:</p><pre>git checkout feature<br>git pull origin feature<br>git merge develop<br>git push origin feature</pre><p>Create branch from develop — just like parting ways of Vin Diesel and Paul Walker:</p><pre>git checkout develop <br>git pull origin develop<br>git checkout -b feature/JIRA_PROJECT-#_your_task_name<br>git commit -m &quot;add spice to your life&quot;<br>git fetch origin develop<br>git rebase development<br>git push -u origin feature/JIRA_PROJECT-#_your_task_name</pre><p>Delete local branch — just like when you say a harsh joke to a new guy in your workplace and say “I’m just joking!” and move on with life:</p><pre>git branch -D some-branch</pre><p>How to delete remote branch — just like removing all clues on the crime scene:</p><pre>git push origin --delete some-branch</pre><p>Create and push hotfix — just like when there is a small crack of cement and workers patch it:</p><pre>git checkout -b hotfix/...<br>git add ..<br>git commit ..<br>git push ..</pre><p>Merge hotfix to master — just like proposing to your loved one and there’s no turning back:</p><pre>git checkout master <br>git merge hotfix/...<br>git push</pre><p>Create and push tag from master — just like when you decide on something and there’s no turning back:</p><pre>git tag -a v1.0.8 -m &quot;v1.0.8&quot;<br>git push origin v1.0.8</pre><p>Delete tag — just for um..nothing, maybe when you leave a chat room and don’t go back..?</p><pre>git tag -d v1.0.8<br>git push --delete origin v1.0.8</pre><p>Initialize and update user configuration — just like buying a second hand car and feel like you’re the first owner of this car, and remove everything that’s reminding it of it’s previous owner:</p><pre>git config --local user.email &quot;<a href="/service/mailto:esguerraama@gmail.com">john.doe@mail.com</a>&quot;<br>git init <br>git add <br>git log<br>git remote add origin <a href="https://github.com/aesguerra/ignite-pg.git">https://github.com/</a><a href="/service/mailto:esguerraama@gmail.com">john.doe</a><a href="https://github.com/aesguerra/ignite-pg.git">/foo.git</a><br>git push --force --set-upstream origin master</pre><p>Git revert — just like when you and your wife got into an intense fight, say sorry and promise to change, and wifey accepts (hopefully?):</p><pre>git push origin +da08bbc90da^:master<br>// da08bbc90da = commit</pre><p>Git tag — just like putting a name tag on your lovely dog:</p><pre>git tag -l<br>git tag -a v1.5.3<br>git push origin v1.5.3</pre><p>Git rename — just like buying a second hand car and feel like you’re the first owner of this car, and remove everything that’s reminding it of it’s previous owner:</p><pre>git branch -m new-name<br>git branch -m old-name new-name<br>git push origin :old-name new-name<br>git push origin -u new-name</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=40e1d9a03590" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Who’s this Spark Listener?]]></title>
            <link>https://minyodev.medium.com/whos-this-spark-listener-1f050edd87c6?source=rss-e26380dd93fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f050edd87c6</guid>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Ana Suzuki]]></dc:creator>
            <pubDate>Wed, 13 Jun 2018 08:39:03 GMT</pubDate>
            <atom:updated>2018-06-13T08:39:03.837Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-SparkListener.html">Jacek Laskowski</a> made a good documentation regarding spark listeners. I made this page since we keep on encountering this ERROR:</p><pre>2018-06-13 08:07:26 ERROR LiveListenerBus:70 - Dropping SparkListenerEvent because no remaining room in event queue. This likely means one of the SparkListeners is too slow and cannot keep up with the rate at which tasks are being started by the scheduler.<br>2018-06-13 08:07:26 WARN  LiveListenerBus:66 - Dropped 1 SparkListenerEvents since Thu Jan 01 00:00:00 UTC 1970<br>2018-06-13 08:08:26 WARN  LiveListenerBus:66 - Dropped 43737 SparkListenerEvents since Wed Jun 13 08:07:26 UTC 2018<br>...</pre><p>Question:</p><p>I encounter this error and my spark job ends successfully. Am I still assured that my output data is complete?</p><p>Answer:</p><p>Yes. As long as your spark job succeeds and you did not encounter error during the transformation and actions on stages, your data is still complete.</p><p>This error happens due to the fact that the event processor in ListenerBus is just single thread, and loops through all the Listeners for each event and processes each event synchronously. The onDropEvent is called when no further events can be added to the internal eventQueue queue. This kind of error pops up mostly on high latency event processing spark jobs.</p><p>One fix is to basically increase the listener bus event queue size using the following config:</p><pre>spark.scheduler.listenerbus.eventqueue.size=&lt;eventqueue_size&gt;</pre><p><em>Note: Default value is 10000</em></p><p>And of course, you need to increase the --driver-memory as well since these events will be located on driver.</p><p>Another way to solve this problem is to run using Apache Spark &gt;2.3.0. More info of fix <a href="https://issues-test.apache.org/jira/browse/SPARK-18838?page=com.googlecode.jira-suite-utilities%3Atransitions-summary-tabpanel">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f050edd87c6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10 Tips for Big Data Engineer Interviews]]></title>
            <link>https://minyodev.medium.com/10-tips-for-big-data-engineer-interviews-d2b34a6272a9?source=rss-e26380dd93fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2b34a6272a9</guid>
            <category><![CDATA[philippines]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[tips]]></category>
            <category><![CDATA[information-technology]]></category>
            <category><![CDATA[careers]]></category>
            <dc:creator><![CDATA[Ana Suzuki]]></dc:creator>
            <pubDate>Tue, 19 Dec 2017 12:19:58 GMT</pubDate>
            <atom:updated>2017-12-19T12:28:18.938Z</atom:updated>
            <content:encoded><![CDATA[<p>Alright here you go!</p><h4>Tip #1</h4><p>Here in the Philippines, the population of companies that are doing/using big data isn’t quite big yet.</p><p>The only company with large data are the telcos, so, obviously speaking, if you want to experience playing around a data cafe with hundreds of nodes plus hundreds of terabytes of data, those 2 telecommunication companies should be on your list.</p><h4>Tip #2</h4><p>Most of the companies here are just starting to migrate their current architecture to big data architecture. So they will ask questions if you already built one, or if you’re part of the team that established or developed data pipeline.</p><h4>Tip #3</h4><p>If you’re simply running a Spark job or MapReduce job without knowing the reason why you’re running it, well, now is the time you should! This is the most common pitfalls of applicants since they brag that they already experienced running Spark jobs, or MR, or whatever-it-is without knowing the reason why they are doing/using it. You should not tell them “because that’s what my boss told me to do.”, poor young thing your interviewer will be disappointed. They will ask you the projects and/or tasks you’re involved with, and how the organization benefits from it.</p><h4>Tip #4</h4><p>If you’re going on a team lead position, you should be mindful of the current data architecture of your company.</p><h4>Tip #5</h4><p>You must give them some of your experiences working as a team. They love to hear you’re a team player. Usually, the interviewers ask how many are you on a team and how you guys handle stress when under pressure.</p><h4>Tip #6</h4><p>You should show them that whenever there’s an issue/problem, you are open to tell your boss about it. It’s also a good thing that the interviewers are aware that you are open to discussing things with your boss or team lead (because normally the one interviewing you will be your future boss!).</p><h4>Tip #7</h4><p>Some of their questions came from <a href="https://www.dezyre.com/article/top-100-hadoop-interview-questions-and-answers-2017/159">here</a>. Guys, please don’t believe in blogs saying that <em>Hadoop is dead</em>. As what I’ve mentioned earlier, most of the companies here are just starting with big data, and Hadoop is on the top of what the would like to use.</p><h4>Tip #8</h4><p>Some will ask you these:</p><ul><li>RDBMS vs Hadoop</li><li>Hadoop distribution comparison</li><li>MapReduce vs Spark</li><li>Core components of Hadoop application</li><li>Programming languages</li><li>Reasons on leaving your current company</li><li>Algorithms or data structures</li><li>Concurrency</li><li>Problems you experience with running Spark applications</li><li>Structured vs unstructured data</li></ul><h4>Tip #9</h4><p>If the reason you’re leaving your current company is because you don’t like your boss/office mate, you should profile your interviewers as well. Their attitudes, the manner they ask/answer you, etc. That human will be your future office mate, and if he’s braggy, thinking high of himself, or all-knowing, have some doubt before accepting the JO my friend. You will hand your resignation due to same reasons.</p><h4>Tip #10</h4><p>Once they’re finished asking you questions, they will ask if you have questions. It’s better if you have some questions so they will feel your interest with the company. But guys, please do not ask them company benefits. It’s not appropriate to ask non-technical questions to technical interviewers. Things like salaries, health benefits, etc., should be directly discussed with HRs.</p><p>That’s all for now. Whoever you are reading this, I believe you’re now having some doubts with your current job that’s why you’re here. Haha JK. I wish you all the best! Shoot me some PMs if you have questions!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2b34a6272a9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Relocating Classes using Apache Maven Shade Plugin]]></title>
            <link>https://minyodev.medium.com/relocating-classes-using-apache-maven-shade-plugin-6957a1a8666d?source=rss-e26380dd93fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/6957a1a8666d</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[maven]]></category>
            <category><![CDATA[elasticsearch]]></category>
            <category><![CDATA[java]]></category>
            <dc:creator><![CDATA[Ana Suzuki]]></dc:creator>
            <pubDate>Tue, 24 Oct 2017 02:16:05 GMT</pubDate>
            <atom:updated>2017-10-24T02:16:05.247Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*_NMsPKzPQYtjJB4FHJjgwQ.png" /><figcaption>Photo Credit by <a href="https://genius.com/Eminem-the-real-slim-shady-lyrics">Genius</a></figcaption></figure><p>I just had a weird encounter upon running Spark Job.</p><p>There is a maven project that contains 2 modules: ModuleA and ModuleB.</p><p>To make the story short, ModuleA depends on ModuleB. There’s a class from ModuleB that is called to ModuleA.</p><p>Upon submitting the job, there is this error:</p><pre>Exception in thread “main” java.lang.NoSuchMethodError: io.netty.buffer.CompositeByteBuf.addComponents(ZLjava/lang/Iterable;)Lio/netty/buffer/CompositeByteBuf;<br>...<br>...<br>...</pre><p>Found out that there are multiple netty version:</p><ul><li>Module A: org.apache.spark:spark-core and org.apache.hbase:hbase-server</li><li>Module B: org.elasticsearch.client:transport</li></ul><p>To tell you the truth, I’ve been debugging this for hours I almost lost my humanity. I’ve tried excluding that package in pom.xml, assembly.xml and even in spark config but to no avail.</p><p>Finally, I met <a href="https://maven.apache.org/plugins/maven-shade-plugin/index.html">Apache Maven Shade Plugin</a> where one of its ability is to rename packages of some dependencies. Take a look at <a href="https://github.com/apache/spark/blob/6f1d0dea1cdda558c998179789b386f6e52b9e36/pom.xml">Apache Spark</a> and <a href="https://github.com/elastic/elasticsearch-parent/blob/master/pom.xml">Elasticsearch’s</a> pom.xml. They are also using this plugin.</p><p>The solution is this:</p><ol><li>Exclude netty package fororg.apache.hbase:hbase-server</li></ol><pre>&lt;!-- Module A&#39;s pom.xml --&gt;<br>&lt;<strong>dependency</strong>&gt;<br>    &lt;<strong>groupId</strong>&gt;org.apache.hbase&lt;/<strong>groupId</strong>&gt;<br>    &lt;<strong>artifactId</strong>&gt;hbase-server&lt;/<strong>artifactId</strong>&gt;<br>    &lt;<strong>version</strong>&gt;${hbase.version}&lt;/<strong>version</strong>&gt;<br>    &lt;<strong>exclusions</strong>&gt;<br>        &lt;<strong>exclusion</strong>&gt;<br>            &lt;<strong>groupId</strong>&gt;io.netty&lt;/<strong>groupId</strong>&gt;<br>            &lt;<strong>artifactId</strong>&gt;netty&lt;/<strong>artifactId</strong>&gt;<br>        &lt;/<strong>exclusion</strong>&gt;<br>        &lt;<strong>exclusion</strong>&gt;<br>            &lt;<strong>groupId</strong>&gt;io.netty&lt;/<strong>groupId</strong>&gt;<br>            &lt;<strong>artifactId</strong>&gt;netty-all&lt;/<strong>artifactId</strong>&gt;<br>        &lt;/<strong>exclusion</strong>&gt;<br>    &lt;/<strong>exclusions</strong>&gt;<br>&lt;/<strong>dependency</strong>&gt;</pre><p>2. Added maven-shade-plugin to org.elasticsearch.client:transport</p><pre>&lt;!-- ModuleB&#39;s pom.xml --&gt;<br>&lt;<strong>plugin</strong>&gt;<br>    &lt;<strong>groupId</strong>&gt;org.apache.maven.plugins&lt;/<strong>groupId</strong>&gt;<br>    &lt;<strong>artifactId</strong>&gt;maven-shade-plugin&lt;/<strong>artifactId</strong>&gt;<br>    &lt;<strong>version</strong>&gt;3.1.0&lt;/<strong>version</strong>&gt;<br>    &lt;<strong>configuration</strong>&gt;<br>        &lt;<strong>shadedArtifactAttached</strong>&gt;false&lt;/<strong>shadedArtifactAttached</strong>&gt;<br>        &lt;<strong>artifactSet</strong>&gt;<br>            &lt;<strong>includes</strong>&gt;<br>                &lt;<strong>include</strong>&gt;*:*&lt;/<strong>include</strong>&gt;<br>            &lt;/<strong>includes</strong>&gt;<br>        &lt;/<strong>artifactSet</strong>&gt;<br>        &lt;<strong>filters</strong>&gt;<br>            &lt;<strong>filter</strong>&gt;<br>                &lt;<strong>artifact</strong>&gt;*:*&lt;/<strong>artifact</strong>&gt;<br>                &lt;<strong>excludes</strong>&gt;<br>                    &lt;<strong>exclude</strong>&gt;META-INF/*.SF&lt;/<strong>exclude</strong>&gt;<br>                    &lt;<strong>exclude</strong>&gt;META-INF/*.DSA&lt;/<strong>exclude</strong>&gt;<br>                    &lt;<strong>exclude</strong>&gt;META-INF/*.RSA&lt;/<strong>exclude</strong>&gt;<br>                &lt;/<strong>excludes</strong>&gt;<br>            &lt;/<strong>filter</strong>&gt;<br>        &lt;/<strong>filters</strong>&gt;<br>        &lt;<strong>relocations combine.children=&quot;append&quot;</strong>&gt;<br>            &lt;<strong>relocation</strong>&gt;<br>                &lt;<strong>pattern</strong>&gt;io.netty&lt;/<strong>pattern</strong>&gt;<br>                &lt;<strong>shadedPattern</strong>&gt;<br>                  com.shaded_package.io.netty<br>                &lt;/<strong>shadedPattern</strong>&gt;<br>                &lt;<strong>includes</strong>&gt;<br>                   &lt;<strong>include</strong>&gt;io.netty.**&lt;/<strong>include</strong>&gt;<br>                &lt;/<strong>includes</strong>&gt;<br>            &lt;/<strong>relocation</strong>&gt;<br>        &lt;/<strong>relocations</strong>&gt;<br>    &lt;/<strong>configuration</strong>&gt;<br>    &lt;<strong>executions</strong>&gt;<br>        &lt;<strong>execution</strong>&gt;<br>            &lt;<strong>phase</strong>&gt;package&lt;/<strong>phase</strong>&gt;<br>            &lt;<strong>goals</strong>&gt;<br>                &lt;<strong>goal</strong>&gt;shade&lt;/<strong>goal</strong>&gt;<br>            &lt;/<strong>goals</strong>&gt;<br>        &lt;/<strong>execution</strong>&gt;<br>    &lt;/<strong>executions</strong>&gt;<br>&lt;/<strong>plugin</strong>&gt;</pre><p>It would basically append com.shaded_package to all that starts with io.netty . However if you doubt this and want proof, compile this to a jar file and do jar -tf moduleb.jar | grep shaded_package</p><p>3. I didn’t do anything with org.apache.spark:spark-core netty packages since this is my most priority package.</p><p>After all of that my Spark Job works like a charm.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6957a1a8666d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Import Mysql data to HDFS using Sqoop]]></title>
            <link>https://minyodev.medium.com/import-mysql-data-to-hdfs-using-sqoop-a3b0bac101dd?source=rss-e26380dd93fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/a3b0bac101dd</guid>
            <category><![CDATA[yarn]]></category>
            <category><![CDATA[hdfs]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[hadoop]]></category>
            <dc:creator><![CDATA[Ana Suzuki]]></dc:creator>
            <pubDate>Wed, 04 Oct 2017 07:20:55 GMT</pubDate>
            <atom:updated>2017-10-04T07:20:55.422Z</atom:updated>
            <content:encoded><![CDATA[<p>The trending topic in big data is now about AIs. It’s quite weird that I am posting something that’s not new in the big data world. This component already exists for hundred years. When compared to the bell curve, this topic is probably like, in the downward-slopping side of the curve.</p><p>In my previous company, we had a PoC using Sqoop to import/export files to/from HDFS. I’ve did not have the opportunity to appreciate this back then.</p><p>Requirements:</p><ul><li>Already installed mySQL 14.14</li><li>Java 1.8</li></ul><p>Download and install MySQL connector java <a href="https://dev.mysql.com/downloads/connector/j/">here</a>.</p><h4>Install and configure Hadoop (Pseudo-Distributed mode)</h4><ol><li>Download hadoop-2.7.3 <a href="https://archive.apache.org/dist/hadoop/core/hadoop-2.7.3/">here</a>.</li><li>Extract hadoop-2.7.3.tar.gz to /app/local/ and create symbolic link to /opt</li></ol><pre>$ ls /app/local<br>hadoop-2.7.3</pre><pre>$ cd /opt<br>$ ln -s /app/local/hadoop-2.7.3 hadoop<br>$ ls -lrt <br>lrwxrwxrwx 1 root root 23 Sep 21 09:40 hadoop -&gt; /app/local/hadoop-2.7.3</pre><p>3. Set up hadoop environment variables by appending the following commands to ~/.bashrc file</p><pre>export HADOOP_HOME=/opt/hadoop<br>export HADOOP_MAPRED_HOME=$HADOOP_HOME<br>export HADOOP_COMMON_HOME=$HADOOP_HOME<br>export HADOOP_HDFS_HOME=$HADOOP_HOME<br>export YARN_HOME=$HADOOP_HOME<br>export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native<br>export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin</pre><p>4. Now, apply all the changes into the current running system.</p><pre>source ~/.bashrc<br></pre><p>5. Let’s go to hadoop configuration directory</p><pre>$ cd $HADOOP_HOME/etc/hadoop</pre><p>6. Update hdfs-site.xml</p><pre>&lt;configuration&gt;<br>   &lt;property&gt;<br>      &lt;name&gt;dfs.replication&lt;/name&gt;<br>      &lt;value&gt;1&lt;/value&gt;<br>   &lt;/property&gt;<br>   &lt;property&gt;<br>      &lt;name&gt;dfs.name.dir&lt;/name&gt;<br>      &lt;value&gt;file:///app/data/hadoop/hdfs/namenode&lt;/value&gt;<br>   &lt;/property&gt;<br>   &lt;property&gt;<br>      &lt;name&gt;dfs.data.dir&lt;/name&gt;<br>      &lt;value&gt;file://app/data/hdfs/datanode&lt;/value&gt;<br>   &lt;/property&gt;<br>&lt;/configuration&gt;</pre><p>Note: Make sure that the directories are existing</p><p>7. Update mapred-site.xml</p><pre>&lt;configuration&gt;<br>   &lt;property&gt;<br>      &lt;name&gt;mapreduce.framework.name&lt;/name&gt;<br>      &lt;value&gt;yarn&lt;/value&gt;<br>   &lt;/property&gt;<br>&lt;/configuration&gt;</pre><p>8. Start hadoop dfs and yarn:</p><pre>$ start-dfs.sh<br>$ start-yarn.sh</pre><p>9. Access hadoop in your browser:</p><pre>http://localhost:50070/</pre><p>You will see something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WSK4Gazd4C3HD5KnVDFtqw.png" /></figure><p>10. Access all applications for hadoop cluster</p><pre><a href="http://localhost:8088/cluster">http://localhost:8088/cluster</a></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BS1DeNVZOIuAVXPq4iFgVA.png" /></figure><h4>Install and configure Sqoop</h4><ol><li>Download sqoop-1.4.6.bin__hadoop-2.0.4-alpha <a href="https://www-eu.apache.org/dist/sqoop/1.4.6/">here</a>.</li><li>Extract sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz to /app/local/ and create symbolic link to /opt</li></ol><pre>$ ls /app/local<br>sqoop-1.4.6.bin__hadoop-2.0.4-alpha</pre><pre>$ cd /opt<br>$ ln -s /app/local/sqoop-1.4.6.bin__hadoop-2.0.4-alpha sqoop<br>$ ls -lrt <br>lrwxrwxrwx 1 root root 23 Sep 21 09:50 sqoop -&gt; /app/local/sqoop-1.4.6.bin__hadoop-2.0.4-alpha</pre><p>3. Set up sqoop environment variables by appending the following commands to ~/.bashrc file</p><pre>export SQOOP_HOME=/opt/sqoop<br>export PATH=$PATH:$SQOOP_HOME/bin</pre><p>4. Configure sqoop-env.sh</p><pre>export HADOOP_COMMON_HOME=/opt/hadoop <br>export HADOOP_MAPRED_HOME=/opt/hadoop</pre><p>5. Copy mysql connector to sqoop lib/</p><pre>$ cp mysql-connector-java-5.1.40-bin.jar /usr/lib/sqoop/lib</pre><p>6. Verify Sqoop</p><pre>$ cd $SQOOP_HOME/bin<br>$ sqoop-version</pre><h4>How to import data from MySQL into HDFS</h4><ol><li>The following syntax is used to import to HDFS</li></ol><pre>$ sqoop-import (general-args) (import-args)</pre><p>2. Populate your mysql table</p><pre>CREATE DATABASE userdb;<br>CREATE TABLE emp (<br>   id int,<br>   name varchar(255),<br>   deg varchar(255),<br>   salary double,<br>   dept varchar(255)<br>);</pre><pre>INSERT INTO emp VALUES (1201, &quot;gopal&quot;, &quot;manager&quot;, 50000, &quot;TP&quot;);<br>INSERT INTO emp VALUES (1202, &quot;manisha&quot;, &quot;Proof reader&quot;, 50000, &quot;TP&quot;);<br>INSERT INTO emp VALUES (1203, &quot;khalil&quot;, &quot;php dev&quot;, 30000, &quot;AC&quot;);<br>INSERT INTO emp VALUES (1204, &quot;prasanth&quot;, &quot;php dev&quot;, 30000, &quot;AC&quot;);<br>INSERT INTO emp VALUES (1205, &quot;kranthi&quot;, &quot;admin&quot;, 20000, &quot;TP&quot;);</pre><p>3. Import a table</p><pre>sqoop import --connect jdbc:mysql://localhost/userdb --username myuser --password mypassword -table emp -m 1</pre><p>4. If it is executed successfully, then you get the following output.</p><pre>$ sqoop import --connect jdbc:mysql://localhost/userdb     --username myuser --password mypassword -table emp -m 1<br>17/10/04 14:07:40 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6<br>17/10/04 14:07:40 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.<br>17/10/04 14:07:40 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.<br>17/10/04 14:07:40 INFO tool.CodeGenTool: Beginning code generation<br>Wed Oct 04 14:07:40 PHT 2017 WARN: Establishing SSL connection without server&#39;s identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn&#39;t set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to &#39;false&#39;. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.<br>17/10/04 14:07:40 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `emp` AS t LIMIT 1<br>17/10/04 14:07:40 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `emp` AS t LIMIT 1<br>17/10/04 14:07:40 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/hadoop<br>Note: /tmp/sqoop-aesguerra/compile/4b9bbe3b5964e16fcc1d319522b1c929/emp.java uses or overrides a deprecated API.<br>Note: Recompile with -Xlint:deprecation for details.<br>17/10/04 14:07:42 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-aesguerra/compile/4b9bbe3b5964e16fcc1d319522b1c929/emp.jar<br>17/10/04 14:07:42 WARN manager.MySQLManager: It looks like you are importing from mysql.<br>17/10/04 14:07:42 WARN manager.MySQLManager: This transfer can be faster! Use the --direct<br>17/10/04 14:07:42 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.<br>17/10/04 14:07:42 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)<br>17/10/04 14:07:42 INFO mapreduce.ImportJobBase: Beginning import of emp<br>17/10/04 14:07:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable<br>17/10/04 14:07:42 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar<br>17/10/04 14:07:42 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps<br>17/10/04 14:07:43 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032<br>Wed Oct 04 14:07:47 PHT 2017 WARN: Establishing SSL connection without server&#39;s identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn&#39;t set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to &#39;false&#39;. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.<br>17/10/04 14:07:47 INFO db.DBInputFormat: Using read commited transaction isolation<br>17/10/04 14:07:47 INFO mapreduce.JobSubmitter: number of splits:1<br>17/10/04 14:07:47 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1507093844771_0001<br>17/10/04 14:07:47 INFO impl.YarnClientImpl: Submitted application application_1507093844771_0001<br>17/10/04 14:07:47 INFO mapreduce.Job: The url to track the job: <a href="http://localhost:8088/proxy/application_1507093844771_0001/">http://localhost:8088/proxy/application_1507093844771_0001/</a><br>17/10/04 14:07:47 INFO mapreduce.Job: Running job: job_1507093844771_0001<br>17/10/04 14:07:53 INFO mapreduce.Job: Job job_1507093844771_0001 running in uber mode : false<br>17/10/04 14:07:53 INFO mapreduce.Job:  map 0% reduce 0%<br>17/10/04 14:07:58 INFO mapreduce.Job:  map 100% reduce 0%<br>17/10/04 14:07:58 INFO mapreduce.Job: Job job_1507093844771_0001 completed successfully<br>17/10/04 14:07:59 INFO mapreduce.Job: Counters: 30<br> File System Counters<br>  FILE: Number of bytes read=0<br>  FILE: Number of bytes written=136966<br>  FILE: Number of read operations=0<br>  FILE: Number of large read operations=0<br>  FILE: Number of write operations=0<br>  HDFS: Number of bytes read=87<br>  HDFS: Number of bytes written=161<br>  HDFS: Number of read operations=4<br>  HDFS: Number of large read operations=0<br>  HDFS: Number of write operations=2<br> Job Counters <br>  Launched map tasks=1<br>  Other local map tasks=1<br>  Total time spent by all maps in occupied slots (ms)=2162<br>  Total time spent by all reduces in occupied slots (ms)=0<br>  Total time spent by all map tasks (ms)=2162<br>  Total vcore-milliseconds taken by all map tasks=2162<br>  Total megabyte-milliseconds taken by all map tasks=2213888<br> Map-Reduce Framework<br>  Map input records=5<br>  Map output records=5<br>  Input split bytes=87<br>  Spilled Records=0<br>  Failed Shuffles=0<br>  Merged Map outputs=0<br>  GC time elapsed (ms)=47<br>  CPU time spent (ms)=820<br>  Physical memory (bytes) snapshot=171454464<br>  Virtual memory (bytes) snapshot=1970679808<br>  Total committed heap usage (bytes)=123207680<br> File Input Format Counters <br>  Bytes Read=0<br> File Output Format Counters <br>  Bytes Written=161<br>17/10/04 14:07:59 INFO mapreduce.ImportJobBase: Transferred 161 bytes in 16.112 seconds (9.9926 bytes/sec)<br>17/10/04 14:07:59 INFO mapreduce.ImportJobBase: Retrieved 5 records.</pre><p>To verify the imported data in HDFS, use the following command.</p><pre>$ hdfs dfs -cat emp/*<br>17/10/04 14:09:29 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable<br>1201,gopal,manager,50000.0,TP<br>1202,manisha,Proof reader,50000.0,TP<br>1203,khalil,php dev,30000.0,AC<br>1204,prasanth,php dev,30000.0,AC<br>1205,kranthi,admin,20000.0,TP</pre><p>The MapReduce job can be seen via http://localhost:8088/cluster</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WUGXFB0dWDsX6qMWre4Gfg.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a3b0bac101dd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Streaming Kafka Receiver-Less Approach]]></title>
            <link>https://minyodev.medium.com/streaming-kafka-receiver-less-approach-df6390d90ef7?source=rss-e26380dd93fa------2</link>
            <guid isPermaLink="false">https://medium.com/p/df6390d90ef7</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[streaming]]></category>
            <category><![CDATA[scala]]></category>
            <dc:creator><![CDATA[Ana Suzuki]]></dc:creator>
            <pubDate>Wed, 05 Jul 2017 02:59:24 GMT</pubDate>
            <atom:updated>2017-07-05T08:51:18.676Z</atom:updated>
            <content:encoded><![CDATA[<p>Kindly read <a href="https://spark.apache.org/docs/latest/streaming-kafka-0-8-integration.html">https://spark.apache.org/docs/latest/streaming-kafka-0-8-integration.html</a> for more information about Receiver-Less Approach.</p><p>Here are some points to ponder on with this kind of approach:</p><ol><li>Periodically queries Kafka for the latest offsets in each topic+partition, and accordingly defines the offset ranges to process in each batch.</li><li>Spark Streaming will create as many RDD partitions as there are Kafka partitions to consume, which will all read data from Kafka in parallel.</li><li>There is a one-to-one mapping between Kafka and RDD partitions, which is easier to understand and tune.</li><li>As long as you have sufficient Kafka retention, messages can be recovered from Kafka.</li><li>Uses simple Kafka API that does not use Zookeeper. Offsets are tracked by Spark Streaming within its checkpoints.</li><li>Each record is received by Spark Streaming effectively exactly once despite failures.</li></ol><p>There are 2 files namely:</p><ol><li><strong>DirectKafkaWordCount.scala</strong> — Consume data from kafka and perform WordCount with records and prints the output</li><li><strong>KafkaWordCountProducer.scala</strong> — Produces randomly generated numbers to Kafka</li></ol><p>Note that this examples are readily available with <a href="https://github.com/apache/spark/blob/v2.1.1/examples/src/main/scala/org/apache/spark/examples/streaming/StreamingExamples.scala">spark’s github repository</a>.<br>Before you run both codes make sure that your Zookeeper and Kafka is up.</p><pre><strong>import </strong>java.util.HashMap<br><strong>import </strong>org.apache.kafka.clients.producer.{KafkaProducer, ProducerConfig, ProducerRecord}<br><br><em>/**<br>  * Produces some random words between 1 and 100 to Kafka.<br>  *<br>  * This program is readily available at <br>  * https://github.com/apache/spark/tree/v2.1.1/examples/src/main/scala/org/apache/spark/examples/streaming<br>  *<br>  */<br></em><strong>object </strong>KafkaWordCountProducer {<br><br>  <strong>def </strong>main(args: Array[String]) {<br>    <strong>val </strong><em>Array</em>(brokers, topic, messagesPerSec, wordsPerMessage) =<br>      <em>Array</em>(<strong>&quot;localhost:9092&quot;</strong>, <strong>&quot;sample-topic&quot;</strong>, <strong>&quot;20&quot;</strong>, <strong>&quot;2&quot;</strong>)<br><br>    <em>// Zookeeper connection properties<br>    </em><strong>val </strong>props = <strong>new </strong>HashMap[String, Object]()<br>    props.put(ProducerConfig.<em>BOOTSTRAP_SERVERS_CONFIG</em>, brokers)<br>    props.put(ProducerConfig.<em>VALUE_SERIALIZER_CLASS_CONFIG</em>,<br>      <strong>&quot;org.apache.kafka.common.serialization.StringSerializer&quot;</strong>)<br>    props.put(ProducerConfig.<em>KEY_SERIALIZER_CLASS_CONFIG</em>,<br>      <strong>&quot;org.apache.kafka.common.serialization.StringSerializer&quot;</strong>)<br><br>    <strong>val </strong>producer = <strong>new </strong>KafkaProducer[String, String](props)<br><br>    <em>// Send some messages<br>    </em><strong>while</strong>(<strong>true</strong>) {<br>      (1 to messagesPerSec.toInt).foreach { messageNum =&gt;<br>        <strong>val </strong>str = (1 to wordsPerMessage.toInt)<br>          .map(x =&gt; scala.util.Random.nextInt(10).toString)<br>          .mkString(<strong>&quot; &quot;</strong>)<br><br>        <strong>val </strong>message = <strong>new </strong>ProducerRecord[String, String](<br>            topic, <strong>null</strong>, str)<br>        producer.send(message)<br>      }<br><br>      Thread.<em>sleep</em>(1000)<br>    }<br>  }<br>}</pre><pre><strong>import </strong>kafka.serializer.StringDecoder<br><br><strong>import </strong>org.apache.spark.SparkConf<br><strong>import </strong>org.apache.spark.streaming.{Seconds, StreamingContext}<br><strong>import </strong>org.apache.spark.streaming.kafka._<br><br><em>/**<br>  * A Program that will consume records from Kafka, perform Wordcount and simply print out output<br>  *<br>  * This program is readily available at <br>  * https://github.com/apache/spark/tree/v2.1.1/examples/src/main/scala/org/apache/spark/examples/streaming<br>  *<br>  */<br></em><strong>object </strong>DirectKafkaWordCount {<br><br>  <strong>def </strong>main(args: Array[String]): Unit = {<br><br>    <strong>val </strong><em>Array</em>(brokers, topics) = <br>        <em>Array</em>(<strong>&quot;localhost:9092&quot;</strong>, <strong>&quot;sample-topic&quot;</strong>)<br><br>    <em>// Create context with 2 second batch interval<br>    </em><strong>val </strong>sparkConf = <strong>new </strong>SparkConf()<br>      .setAppName(<strong>&quot;DirectKafkaWordCount&quot;</strong>)<br>      .setMaster(<strong>&quot;local[*]&quot;</strong>)<br>      .setAppName(<strong>&quot;Sample&quot;</strong>)<br>    <strong>val </strong>ssc = <strong>new </strong>StreamingContext(sparkConf, <em>Seconds</em>(2))<br><br>    <em>// Create direct kafka stream with brokers and topics<br>    </em><strong>val </strong>topicsSet = topics.split(<strong>&quot;,&quot;</strong>).toSet<br>    <strong>val </strong>kafkaParams = <em>Map</em>[String, String](<br>        <strong>&quot;metadata.broker.list&quot; </strong>-&gt; brokers)<br>    <strong>val </strong>messages = KafkaUtils<br>       .<em>createDirectStream</em>[String, String, <br>             StringDecoder, StringDecoder](<br>      ssc, kafkaParams, topicsSet)<br><br>    <em>// Get the lines, split them into words, <br>    // count the words and print<br>    </em><strong>val </strong>lines = messages.map(_._2)<br>    <strong>val </strong>words = lines.flatMap(_.split(<strong>&quot; &quot;</strong>))<br>    <strong>val </strong>wordCounts = words<br>       .map(x =&gt; (x, 1L))<br>       .reduceByKey(_ + _)<br>    <br>    wordCounts.print()<br><br>    <em>// Start the computation<br>    </em>ssc.start()<br>    ssc.awaitTermination()<br>  }<br>}</pre><p>The following is my input:</p><pre>2 0<br>5 2<br>4 0<br>0 1<br>3 4<br>6 6<br>3 0<br>0 0<br>8 4<br>6 6<br>9 2<br>0 8<br>4 4<br>5 3<br>3 3<br>3 9<br>2 2<br>1 4<br>0 4<br>6 8</pre><p>And my output:</p><pre>(4,7)<br>(8,3)<br>(0,8)<br>(5,2)<br>(9,2)<br>(1,2)<br>(6,5)<br>(2,5)<br>(3,6)</pre><p>I used the following versions:</p><ul><li>Spark — 2.0.0</li><li>Kafka — 0.8.2.1</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df6390d90ef7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>